Re: [Numpy-discussion] PyArray_AsCArray (cfunction, in Array API) in Numpy User Guide
Hi,
I wrote sample code and it works fine.
This is my code, in case anyone else want to know how to use it:
#include Python.h
#include structmember.h
#include numpy/arrayobject.h
static PyObject *
print_a1(PyObject *dummy, PyObject *args)
{
npy_intp dims[3]; /* PyArray_AsCArray is for ndim = 3 */
int typenum;
int i, nd;
PyObject *o1;
double *d1;
PyArray_Descr *descr;
if (PyArg_ParseTuple(args, O:print_a1, o1) 0) {
PyErr_SetString( PyExc_TypeError, bad arguments);
return NULL;
}
nd = PyArray_NDIM(o1);
typenum = NPY_DOUBLE;
descr = PyArray_DescrFromType(typenum);
if (PyArray_AsCArray(o1, (void *)d1, dims, nd, descr) 0){
PyErr_SetString( PyExc_TypeError, error on getting c array);
return NULL;
}
printf([%d] , dims[0]);
for (i = 0; i dims[0]; ++i){
printf(%.2f , d1[i]);
}
printf(\n);
printf(if ( ((PyArrayObject *)o1)-data == d1): );
if ( ((PyArrayObject *)o1)-data == (char *)d1){
printf(True\n);
}else{
printf(False\n);
}
if (PyArray_Free(o1, (void *)d1) 0){
PyErr_SetString( PyExc_TypeError, PyArray_Free fail);
return NULL;
}
return Py_BuildValue(); /* return None */
}
static PyMethodDef module_methods[] = {
{print_a1, (PyCFunction)print_a1, METH_VARARGS, },
{NULL} /* Sentinel */
};
#ifndef PyMODINIT_FUNC /* declarations for DLL import/export */
#define PyMODINIT_FUNC void
#endif
PyMODINIT_FUNC
initaca(void)
{
(void) Py_InitModule(aca, module_methods);
import_array(); /* required NumPy initialization */
}
Thanks,
Takafumi
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] is ndarray.base the closest base or the ultimate base?
Hello, I cannot quite understand whether ndarray.base is the closest base, the one from which the view was made or the ultimate base, the one actually containing the data. I think the documentation and the actual behaviour mismatch. In [1]: import numpy as np In [2]: x = np.arange(12) In [3]: y = x[::2] In [4]: z = y[2:] In [5]: x.flags Out[5]: C_CONTIGUOUS : True F_CONTIGUOUS : True OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False In [6]: y.flags Out[6]: C_CONTIGUOUS : False F_CONTIGUOUS : False OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False In [7]: z.flags Out[7]: C_CONTIGUOUS : False F_CONTIGUOUS : False OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False In [8]: z.base Out[8]: array([ 0, 2, 4, 6, 8, 10]) It looks like the base of z is y, i.e. its closest base, the array from which the view z was created. But the documentation says: base : ndarray If the array is a view on another array, that array is its `base` (unless that array is also a view). The `base` array is where the array data is ultimately stored. and it looks like the base should be x, the array where the data is ultimately stored. I like the second one better. First, because this way I do not have to travel all the bases until I find an array with OWNDATA set. Second, because the current implementation keeps y alive because of z while in the end z only needs x. In [11]: del y In [12]: z.base Out[12]: array([ 0, 2, 4, 6, 8, 10]) Comments? Best, Luca ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] is ndarray.base the closest base or the ultimate base?
Mon, 21 Sep 2009 09:35:47 +0100, Citi, Luca wrote: I cannot quite understand whether ndarray.base is the closest base, the one from which the view was made or the ultimate base, the one actually containing the data. I think the documentation and the actual behaviour mismatch. The closest base. If the documentation says the opposite, it's wrong. That it's the closest base also causes crashes when the base chain becomes longer than the stack limit. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] string arrays - accessing data from C++
On Fri, Sep 18, 2009 at 10:26 PM, Christopher Barker
chris.bar...@noaa.gov wrote:
Jaroslav Hajek wrote:
string lengths determined
c-style null termination
Hmm, this didn't seem to work for me. But maybe I was doing something
else wrong. Thanks.
well, I notice that for a length-n string, if there are n real'
characters, then there is no null, so that may have messed up your code
somewhere.
As it happens, the problem was just in my brain :)
a = np.empty((3,4), dtype=np.character)
Are you sure? I think this is what I tried (I can't check at this
moment), and the result has descr-type equal to PyArray_STRING. Also,
note that even in the interpreter, the dtype shows itself as string:
numpy.dtype('c')
dtype('|S1')
Good point -- that is a length-one string, not the same thing. Running:
for n in dir(np):
if type(getattr(np, n)) == type(np.uint8): print n
give me what should be all the dtype objects, and these are the ones
that look to me like they might be char:
byte
character
chararray
int8
ubyte
uint8
but none of those seem to be quite right:
In [20]: for dtype in [np.byte, np.character, np.chararray, np.int8,
np.ubyte, np.uint8]:
: a = np.empty((1,1), dtype=dtype); print a.dtype
:
:
int8
|S1
object
int8
uint8
uint8
There was a discussion on the Cython list recently, and apparently
char is not as clearly defined as I thought -- some compilers use
signed, some unsigned.. who knew? So I'm not sure what PyArray_CHAR is.
This is what I suspected - there is no longer a true character array
type, and dtype(c) is just an alias for dtype(S1).
Similarly, creating a PyArray_CHAR array from the C API results in dtype(|S1).
yup-- it looks like the padding is maintained
That's great, because that's almost exactly the data Octave needs.
Only Octave typically uses space as the padding character for
compatibility with Matlab, but can cope with nulls as well.
NumPy string arrays are supported by Pytave now. Thanks for your help.
best regards
--
RNDr. Jaroslav Hajek
computing expert GNU Octave developer
Aeronautical Research and Test Institute (VZLU)
Prague, Czech Republic
url: www.highegg.matfyz.cz
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] is ndarray.base the closest base or the ultimate base?
Thanks for your quick answer.
Is there a reason for that?
Am I wrong or it only makes life harder, such as:
while (PyArray_Check(base) !PyArray_CHKFLAGS(base, NPY_OWNDATA)) {
base = PyArray_BASE(base);
}
plus the possible error you underlined, plus the fact that
this keeps a chain of zombies alive without reason.
Are there cases where the current behaviour is useful?
Best,
Luca
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] is ndarray.base the closest base or the ultimate base?
Mon, 21 Sep 2009 10:51:52 +0100, Citi, Luca wrote:
Thanks for your quick answer.
Is there a reason for that?
Am I wrong or it only makes life harder, such as:
while (PyArray_Check(base) !PyArray_CHKFLAGS(base, NPY_OWNDATA)) {
base = PyArray_BASE(base);
}
plus the possible error you underlined, plus the fact that this keeps a
chain of zombies alive without reason.
Are there cases where the current behaviour is useful?
I don't see real merits in the current behavior over doing the chain up-
walk on view creation. I don't know if anything in view creation requires
that the immediate base is alive afterwards, but that seems unlikely, so
I believe there's no reason not to make this change.
Some (bad) code might be broken if this was changed, for example
y = x[::-1]
y.base is x
but in practice this is probably negligible.
--
Pauli Virtanen
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] is ndarray.base the closest base or the ultimate base?
I think you do not need to do the chain up walk on view creation. If the assumption is that base is the ultimate base, on view creation you can do something like (pseudo-code): view.base = parent if parent.owndata else parent.base ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Best way to insert C code in numpy code
Xavier Gnata wrote: Hi, I have a large 2D numpy array as input and a 1D array as output. In between, I would like to use C code. C is requirement because it has to be fast and because the algorithm cannot be written in a numpy oriented way :( (no way...really). Which tool should I use to achieve that? waeve.inline? pyrex? What is the baseline? That's only a data point, but I almost always use cython in those cases, unless I need 'very advanced' features of the C API in which case I just do it manually. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] something wrong with docs?
I'm trying to read about subclassing. When I view http://docs.scipy.org/doc/numpy/user/basics.subclassing.html?highlight=subclass#module- numpy.doc.subclassing It seems the examples show the _outputs_ of tests, but I don't see the actual example code. e.g., the first example renders like this: Simple example - adding an extra attribute to ndarray¶ Using the object looks like this: ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] is ndarray.base the closest b ase or the ultimate base?
Hi! On Monday 21 September 2009 12:31:27 Citi, Luca wrote: I think you do not need to do the chain up walk on view creation. If the assumption is that base is the ultimate base, on view creation you can do something like (pseudo-code): view.base = parent if parent.owndata else parent.base Hmm. My impression was that .base was for refcounting purposes *only*. Thus, it is not even guaranteed that the attribute value is an array(-like) object. For example, I might want to allow direct access to some internal buffers of an object of mine in an extension module; then, I'd use .base to bind the lifetime of my object to the array (the lifetime of which I cannot control anymore). Ciao, Hans ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy question: Best hardware for Numpy?
David Warde-Farley a écrit : On 20-Sep-09, at 2:17 PM, Romain Brette wrote: Would anyone have thoughts about what the best hardware would be for Numpy? In particular, I am wondering about Intel Core i7 vs Xeon. Also, I feel that the limiting factor might be memory speed and cache rather than processor speed. What do you think? So, there are several different chips that bear the Xeon brand, you'd have to look at individual benchmarks. But if you're concerned about linear algebra performance, I'd say to go with the desktop version and spend some of the money you save on a license for the Intel Math Kernel Library to link NumPy against: http://software.intel.com/en-us/intel-mkl/ David Interesting, I might try Intel MKL. I use mostly element-wise operations (e.g. exp(x) or xx0, where x is a vector), do you think it would make a big difference? Romain ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] fixed-point arithmetic
One thing I'm really missing is something like matlab's fixed-pt toolbox. I'd love to see this added to numpy. A fixed point integer (fpi) type is based on an integer, but keeps track of where the 'binary point' is. When created, the fpi has a specified number of fractional bits and integer bits. When assigned to, the fpi will check for overflow. On overflow various actions can be taken, including raise exception and ignore (just wraparound). numpy arrays of fpi should support all numeric operations. Also mixed fpi/integer operations. I'm not sure how to go about implementing this. At first, I was thinking to just subclass numpy array. But, I don't think this provides fpi scalars, and their associated operations. I have code in c++ for a scalar fpi data type (not numpy scalar, just a c++ type) that I think has all the required behavior. It depends on boost::python and other boost code (and unreleased boost constrained_value), so probably would not be interesting to a larger numpy audience. I'm thinking this might all be implemented in cython. I haven't used cython yet, so this could be an opportunity. OTOH, I don't know if cython has all the capabilities to implement a new numpy scalar/array type. Thoughts? Interest? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy question: Best hardware for Numpy?
A Monday 21 September 2009 13:59:39 Romain Brette escrigué:
David Warde-Farley a écrit :
On 20-Sep-09, at 2:17 PM, Romain Brette wrote:
Would anyone have thoughts about what the best hardware would be for
Numpy? In
particular, I am wondering about Intel Core i7 vs Xeon. Also, I feel
that the
limiting factor might be memory speed and cache rather than
processor speed.
What do you think?
So, there are several different chips that bear the Xeon brand, you'd
have to look at individual benchmarks. But if you're concerned about
linear algebra performance, I'd say to go with the desktop version and
spend some of the money you save on a license for the Intel Math
Kernel Library to link NumPy against:
http://software.intel.com/en-us/intel-mkl/
David
Interesting, I might try Intel MKL. I use mostly element-wise operations
(e.g. exp(x) or xx0, where x is a vector), do you think it would make a
big difference?
MKL should represent a big advantage for the exp(x) operation. For example,
numexpr, that can optionally make use of MKL, gives these figures:
In [1]: import numpy as np
In [3]: a = np.random.rand(1e7)
In [4]: timeit np.exp(a)
10 loops, best of 3: 251 ms per loop
In [5]: import numpy as np
In [6]: import numexpr as ne
In [7]: timeit ne.evaluate(exp(a))
10 loops, best of 3: 78.7 ms per loop
That is, MKL's exp() is around 3x faster than plain C's exp().
You can also set different accuracy modes in MKL:
In [8]: ne.set_vml_accuracy_mode('low')
Out[8]: 'high'
In [9]: timeit ne.evaluate(exp(a))
10 loops, best of 3: 70.5 ms per loop # 3.5x speedup
In [10]: ne.set_vml_accuracy_mode('fast')
Out[10]: 'low'
In [11]: timeit ne.evaluate(exp(a))
10 loops, best of 3: 62.8 ms per loop # 4x speedup
For the xx0, you won't get any speed-up from using MKL, as this operation is
bounded by memory speed.
--
Francesc Alted
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy question: Best hardware for Numpy?
hi, Definitely memory speed is probably the biggest thing to consider. Also using 64bit if you need to do lots of calculations, and cache things. ACML is another alternative... but I've never tried linking it with numpy http://developer.amd.com/cpu/Libraries/acml/Pages/default.aspx AMD Phenom II is their latest chip, but I haven't tried that either. The chips in the latest mac pro run really quick :) Dual 4 core... with lower ghz, but faster memory speed makes my numpy stuff go faster than the higher ghz previous gen mac pro cpus. cu, ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy question: Best hardware for Numpy?
On Mon, Sep 21, 2009 at 8:59 PM, Romain Brette romain.bre...@ens.fr wrote: David Warde-Farley a écrit : On 20-Sep-09, at 2:17 PM, Romain Brette wrote: Would anyone have thoughts about what the best hardware would be for Numpy? In particular, I am wondering about Intel Core i7 vs Xeon. Also, I feel that the limiting factor might be memory speed and cache rather than processor speed. What do you think? So, there are several different chips that bear the Xeon brand, you'd have to look at individual benchmarks. But if you're concerned about linear algebra performance, I'd say to go with the desktop version and spend some of the money you save on a license for the Intel Math Kernel Library to link NumPy against: http://software.intel.com/en-us/intel-mkl/ David Interesting, I might try Intel MKL. I use mostly element-wise operations (e.g. exp(x) or xx0, where x is a vector), do you think it would make a big difference? It won't make any difference for most operations, at least by default, as we only support the MKL for BLAS/LAPACK. IF the MKL gives a C99 interface to the math library, it may be possible to tweak the build process such as to benefit from them. Concerning the hardware, I have just bought a core i7 (the cheapest model is ~ 200$ now, with 4 cores and 8 Mb of shared cache), and the thing flies for floating point computation. My last computer was a pentium 4 so I don't have a lot of reference, but you can compute ~ 300e6 exp (assuming a contiguous array), and ATLAS 3.8.3 built on it is extremely fast - using the threaded version, the asymptotic peak performances are quite impressive. It takes for example 14s to inverse a 5000x5000 matrix of double. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] fixed-point arithmetic
On Mon, Sep 21, 2009 at 9:00 PM, Neal Becker ndbeck...@gmail.com wrote: numpy arrays of fpi should support all numeric operations. Also mixed fpi/integer operations. I'm not sure how to go about implementing this. At first, I was thinking to just subclass numpy array. But, I don't think this provides fpi scalars, and their associated operations. Using dtype seems more straightforward. I would first try to see how far you could go using a pure python object as a dtype. For example (on python 2.6): from decimal import Decimal import numpy as np a = np.array([1, 2, 3], Decimal) b = np.array([2, 3, 4], Decimal) a + b works as expected. A lot of things won't work (e.g. most transcendent functions, which would require a specific implementation anyway), but arithmetic, etc... would work. Then, you could think about implementing the class in cython. If speed is an issue, then implementing your own dtype seems the way to go - I don't know exactly what kind of speed increase you could hope from going the object - dtype, though. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy question: Best hardware for Numpy?
Just because I have a ruler handy :) On my laptop with qx9300, I invert that 5000, 5000 double (float64) matrix in 14.67s. Granted my cpu cores were all at about 75 degrees during that process.. Cheers! Chris On Mon, Sep 21, 2009 at 4:53 PM, David Cournapeau courn...@gmail.com wrote: On Mon, Sep 21, 2009 at 8:59 PM, Romain Brette romain.bre...@ens.fr wrote: David Warde-Farley a écrit : On 20-Sep-09, at 2:17 PM, Romain Brette wrote: Would anyone have thoughts about what the best hardware would be for Numpy? In particular, I am wondering about Intel Core i7 vs Xeon. Also, I feel that the limiting factor might be memory speed and cache rather than processor speed. What do you think? So, there are several different chips that bear the Xeon brand, you'd have to look at individual benchmarks. But if you're concerned about linear algebra performance, I'd say to go with the desktop version and spend some of the money you save on a license for the Intel Math Kernel Library to link NumPy against: http://software.intel.com/en-us/intel-mkl/ David Interesting, I might try Intel MKL. I use mostly element-wise operations (e.g. exp(x) or xx0, where x is a vector), do you think it would make a big difference? It won't make any difference for most operations, at least by default, as we only support the MKL for BLAS/LAPACK. IF the MKL gives a C99 interface to the math library, it may be possible to tweak the build process such as to benefit from them. Concerning the hardware, I have just bought a core i7 (the cheapest model is ~ 200$ now, with 4 cores and 8 Mb of shared cache), and the thing flies for floating point computation. My last computer was a pentium 4 so I don't have a lot of reference, but you can compute ~ 300e6 exp (assuming a contiguous array), and ATLAS 3.8.3 built on it is extremely fast - using the threaded version, the asymptotic peak performances are quite impressive. It takes for example 14s to inverse a 5000x5000 matrix of double. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] fixed-point arithmetic
David Cournapeau wrote: On Mon, Sep 21, 2009 at 9:00 PM, Neal Becker ndbeck...@gmail.com wrote: numpy arrays of fpi should support all numeric operations. Also mixed fpi/integer operations. I'm not sure how to go about implementing this. At first, I was thinking to just subclass numpy array. But, I don't think this provides fpi scalars, and their associated operations. Using dtype seems more straightforward. I would first try to see how far you could go using a pure python object as a dtype. For example (on python 2.6): from decimal import Decimal import numpy as np a = np.array([1, 2, 3], Decimal) b = np.array([2, 3, 4], Decimal) a + b works as expected. A lot of things won't work (e.g. most transcendent functions, which would require a specific implementation anyway), but arithmetic, etc... would work. Then, you could think about implementing the class in cython. If speed is an issue, then implementing your own dtype seems the way to go - I don't know exactly what kind of speed increase you could hope from going the object - dtype, though. We don't want to create arrays of fixed-pt objects. That would be very wasteful. What I have in mind is that integer_bits, frac_bits are attributes of the entire arrays, not the individual elements. The array elements are just plain integers. At first I'm thinking that we could subclass numpy array, adding the int_bits and frac_bits attributes. The arithmetic operators would all have to be overloaded. The other aspect is that accessing an element of the array would return a fixed-pt object (not an integer). ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] masked arrays as array indices
Hello there, Given a masked array such as this one: In [19]: x = np.ma.masked_equal([-1, -1, 0, -1, 2], -1) In [20]: x Out[20]: masked_array(data = [-- -- 0 -- 2], mask = [ True True False True False], fill_value = 99) When you make an assignemnt in the vein of x[x == 0] = 25 the result can be a bit puzzling: In [21]: x[x == 0] = 25 In [22]: x Out[22]: masked_array(data = [25 25 25 25 2], mask = [False False False False False], fill_value = 99) Is this the correct result or have I found a bug? Cheers. -- Ernest ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] masked arrays as array indices
2009/9/21 Ernest Adrogué eadro...@gmx.net Hello there, Given a masked array such as this one: In [19]: x = np.ma.masked_equal([-1, -1, 0, -1, 2], -1) In [20]: x Out[20]: masked_array(data = [-- -- 0 -- 2], mask = [ True True False True False], fill_value = 99) When you make an assignemnt in the vein of x[x == 0] = 25 the result can be a bit puzzling: In [21]: x[x == 0] = 25 In [22]: x Out[22]: masked_array(data = [25 25 25 25 2], mask = [False False False False False], fill_value = 99) Is this the correct result or have I found a bug? I see the same here on 1.4.0.dev7400. Seems pretty odd to me. Then again, it's a bit more complex using masked boolean arrays for indexing since you have True, False, and masked values. Anyone have thoughts on what *should* happen here? Or is this it? Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] fixed-point arithmetic
On Mon, Sep 21, 2009 at 10:57, Neal Becker ndbeck...@gmail.com wrote: David Cournapeau wrote: On Mon, Sep 21, 2009 at 9:00 PM, Neal Becker ndbeck...@gmail.com wrote: numpy arrays of fpi should support all numeric operations. Also mixed fpi/integer operations. I'm not sure how to go about implementing this. At first, I was thinking to just subclass numpy array. But, I don't think this provides fpi scalars, and their associated operations. Using dtype seems more straightforward. I would first try to see how far you could go using a pure python object as a dtype. For example (on python 2.6): from decimal import Decimal import numpy as np a = np.array([1, 2, 3], Decimal) b = np.array([2, 3, 4], Decimal) a + b works as expected. A lot of things won't work (e.g. most transcendent functions, which would require a specific implementation anyway), but arithmetic, etc... would work. Then, you could think about implementing the class in cython. If speed is an issue, then implementing your own dtype seems the way to go - I don't know exactly what kind of speed increase you could hope from going the object - dtype, though. We don't want to create arrays of fixed-pt objects. That would be very wasteful. What I have in mind is that integer_bits, frac_bits are attributes of the entire arrays, not the individual elements. The array elements are just plain integers. At first I'm thinking that we could subclass numpy array, adding the int_bits and frac_bits attributes. The arithmetic operators would all have to be overloaded. The other aspect is that accessing an element of the array would return a fixed-pt object (not an integer). Actually, what you would do is create a new dtype, not a subclass of ndarray. The new datetime dtypes are similar in that they too are parameterized dtypes. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] numpy docstring sphinx pre-processors
Hi there -- I have been working on a small Python package whose central data object comes from Numpy (the record array object). I would like to produce documentation that looks like Numpy's, and am planning to follow Numpy's docstring standard. Numpy uses Sphinx to generate documentation (e.g. for HTML and LaTeX PDF docs). My understanding is that Numpy has its own pre-processors that modify the docstrings to format them in reStructuredText (reST) before using Sphinx to produce the final output (see http://projects.scipy.org/numpy/wiki/CodingStyleGuidelines#docstring-standard). Are these Numpy pre-processors available to the community? I would love to use them! Thanks very much, Elaine ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] fixed-point arithmetic
Robert Kern wrote: On Mon, Sep 21, 2009 at 10:57, Neal Becker ndbeck...@gmail.com wrote: David Cournapeau wrote: On Mon, Sep 21, 2009 at 9:00 PM, Neal Becker ndbeck...@gmail.com wrote: numpy arrays of fpi should support all numeric operations. Also mixed fpi/integer operations. I'm not sure how to go about implementing this. At first, I was thinking to just subclass numpy array. But, I don't think this provides fpi scalars, and their associated operations. Using dtype seems more straightforward. I would first try to see how far you could go using a pure python object as a dtype. For example (on python 2.6): from decimal import Decimal import numpy as np a = np.array([1, 2, 3], Decimal) b = np.array([2, 3, 4], Decimal) a + b works as expected. A lot of things won't work (e.g. most transcendent functions, which would require a specific implementation anyway), but arithmetic, etc... would work. Then, you could think about implementing the class in cython. If speed is an issue, then implementing your own dtype seems the way to go - I don't know exactly what kind of speed increase you could hope from going the object - dtype, though. We don't want to create arrays of fixed-pt objects. That would be very wasteful. What I have in mind is that integer_bits, frac_bits are attributes of the entire arrays, not the individual elements. The array elements are just plain integers. At first I'm thinking that we could subclass numpy array, adding the int_bits and frac_bits attributes. The arithmetic operators would all have to be overloaded. The other aspect is that accessing an element of the array would return a fixed-pt object (not an integer). Actually, what you would do is create a new dtype, not a subclass of ndarray. The new datetime dtypes are similar in that they too are parameterized dtypes. But doesn't this mean that each array element has it's own int_bits, frac_bits attributes? I don't want that. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy docstring sphinx pre-processors
On Mon, Sep 21, 2009 at 12:03, Elaine Angelino elaine.angel...@gmail.com wrote: Hi there -- I have been working on a small Python package whose central data object comes from Numpy (the record array object). I would like to produce documentation that looks like Numpy's, and am planning to follow Numpy's docstring standard. Numpy uses Sphinx to generate documentation (e.g. for HTML and LaTeX PDF docs). My understanding is that Numpy has its own pre-processors that modify the docstrings to format them in reStructuredText (reST) before using Sphinx to produce the final output (see http://projects.scipy.org/numpy/wiki/CodingStyleGuidelines#docstring-standard). Are these Numpy pre-processors available to the community? I would love to use them! http://svn.scipy.org/svn/numpy/trunk/doc/sphinxext/ -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] fixed-point arithmetic
On Mon, Sep 21, 2009 at 12:02, Neal Becker ndbeck...@gmail.com wrote: Robert Kern wrote: On Mon, Sep 21, 2009 at 10:57, Neal Becker ndbeck...@gmail.com wrote: David Cournapeau wrote: On Mon, Sep 21, 2009 at 9:00 PM, Neal Becker ndbeck...@gmail.com wrote: numpy arrays of fpi should support all numeric operations. Also mixed fpi/integer operations. I'm not sure how to go about implementing this. At first, I was thinking to just subclass numpy array. But, I don't think this provides fpi scalars, and their associated operations. Using dtype seems more straightforward. I would first try to see how far you could go using a pure python object as a dtype. For example (on python 2.6): from decimal import Decimal import numpy as np a = np.array([1, 2, 3], Decimal) b = np.array([2, 3, 4], Decimal) a + b works as expected. A lot of things won't work (e.g. most transcendent functions, which would require a specific implementation anyway), but arithmetic, etc... would work. Then, you could think about implementing the class in cython. If speed is an issue, then implementing your own dtype seems the way to go - I don't know exactly what kind of speed increase you could hope from going the object - dtype, though. We don't want to create arrays of fixed-pt objects. That would be very wasteful. What I have in mind is that integer_bits, frac_bits are attributes of the entire arrays, not the individual elements. The array elements are just plain integers. At first I'm thinking that we could subclass numpy array, adding the int_bits and frac_bits attributes. The arithmetic operators would all have to be overloaded. The other aspect is that accessing an element of the array would return a fixed-pt object (not an integer). Actually, what you would do is create a new dtype, not a subclass of ndarray. The new datetime dtypes are similar in that they too are parameterized dtypes. But doesn't this mean that each array element has it's own int_bits, frac_bits attributes? I don't want that. No, I'm suggesting that the dtype has the int_bits and frac_bits information just like the new datetime dtypes have their unit information. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy docstring sphinx pre-processors
On Mon, Sep 21, 2009 at 1:08 PM, Robert Kern robert.k...@gmail.com wrote: On Mon, Sep 21, 2009 at 12:03, Elaine Angelino elaine.angel...@gmail.com wrote: Hi there -- I have been working on a small Python package whose central data object comes from Numpy (the record array object). I would like to produce documentation that looks like Numpy's, and am planning to follow Numpy's docstring standard. Numpy uses Sphinx to generate documentation (e.g. for HTML and LaTeX PDF docs). My understanding is that Numpy has its own pre-processors that modify the docstrings to format them in reStructuredText (reST) before using Sphinx to produce the final output (see http://projects.scipy.org/numpy/wiki/CodingStyleGuidelines#docstring-standard). Are these Numpy pre-processors available to the community? I would love to use them! http://svn.scipy.org/svn/numpy/trunk/doc/sphinxext/ -- Robert Kern I just struggled through the same task, which required some adjustments to work on Windows. If you want to compare the versions, the sphinx doc directory of statsmodels is here: http://bazaar.launchpad.net/~scipystats/statsmodels/trunk/files/head%3A/scikits/statsmodels/docs/ This uses the numpy sphinxext, but requires a very recent sphinx and doesn't include any older sphinx compatibility, but works almost out of the box on both windows and linux. Josef I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] something wrong with docs?
On Mon, Sep 21, 2009 at 7:27 AM, Neal Becker ndbeck...@gmail.com wrote: I'm trying to read about subclassing. When I view http://docs.scipy.org/doc/numpy/user/basics.subclassing.html?highlight=subclass#module- numpy.doc.subclassing It seems the examples show the _outputs_ of tests, but I don't see the actual example code. e.g., the first example renders like this: Simple example - adding an extra attribute to ndarray¶ Using the object looks like this: I'd like to see this sorted as well. The problem is that the `testcode` directive http://docs.scipy.org/numpy/docs/numpy-docs/user/basics.subclassing.rst/ is not recognized. I was recently a bit confused by this, and I went to the rst file to view the code, but that's obviously not a fix for the rendering problem. Skipper ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy docstring sphinx pre-processors
thanks robert! yes i saw this (http://svn.scipy.org/svn/numpy/trunk/doc/sphinxext/) but is there a good description of how to use this? i'm looking for a standard recipe that could be followed by myself and others. e.g. what functions to call and in what order... i would like to emulate what numpy does as closely as possible. thanks again, elaine On Mon, Sep 21, 2009 at 1:08 PM, Robert Kern robert.k...@gmail.com wrote: On Mon, Sep 21, 2009 at 12:03, Elaine Angelino elaine.angel...@gmail.com wrote: Hi there -- I have been working on a small Python package whose central data object comes from Numpy (the record array object). I would like to produce documentation that looks like Numpy's, and am planning to follow Numpy's docstring standard. Numpy uses Sphinx to generate documentation (e.g. for HTML and LaTeX PDF docs). My understanding is that Numpy has its own pre-processors that modify the docstrings to format them in reStructuredText (reST) before using Sphinx to produce the final output (see http://projects.scipy.org/numpy/wiki/CodingStyleGuidelines#docstring-standard ). Are these Numpy pre-processors available to the community? I would love to use them! http://svn.scipy.org/svn/numpy/trunk/doc/sphinxext/ -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy docstring sphinx pre-processors
On Mon, Sep 21, 2009 at 12:20, Elaine Angelino elaine.angel...@gmail.com wrote: thanks robert! yes i saw this (http://svn.scipy.org/svn/numpy/trunk/doc/sphinxext/) but is there a good description of how to use this? i'm looking for a standard recipe that could be followed by myself and others. e.g. what functions to call and in what order... i would like to emulate what numpy does as closely as possible. http://svn.scipy.org/svn/numpy/trunk/doc/source/conf.py -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] np.take versus fancy indexing
Wes McKinney wrote: Any clue why I'm seeing this behavior? np.take's documentation says it does the same thing as fancy indexing, but from this example I'm not so sure: The code used to implement np.take is not the same as that used in fancy indexing. np.take's mission is simpler, so it uses type-specific code for each numeric type, generated using a template. The same type of optimization was done for putmask and clip. I haven't looked into the code used by fancy indexing. Presumably it could be sped up by using np.take (or the strategy used by np.take) in suitable cases, but I suspect that would be a big job, with plenty of opportunities for introducing bugs. Eric import numpy as np mat = np.random.randn(5000, 1000) selector = np.array(np.arange(5000)[::2]) In [95]: timeit submat = mat[selector] 10 loops, best of 3: 68.4 ms per loop In [96]: timeit submat = np.take(mat, selector, axis=0) 10 loops, best of 3: 21.5 ms per loop indeed the result is the same: In [97]: (mat[selector] == np.take(mat, selector, axis=0)).all() Out[97]: True In [98]: mat[selector].flags Out[98]: C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False In [99]: np.take(mat, selector, axis=0).flags Out[99]: C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False What's going on here / am I doing something wrong? Thanks, Wes ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy docstring sphinx pre-processors
On 21-Sep-09, at 1:20 PM, Elaine Angelino wrote: thanks robert! yes i saw this (http://svn.scipy.org/svn/numpy/trunk/doc/sphinxext/) but is there a good description of how to use this? i'm looking for a standard recipe that could be followed by myself and others. e.g. what functions to call and in what order... i would like to emulate what numpy does as closely as possible. You should have a look at matplotlib's sampledoc tutorial. It goes over how to use Sphinx extensions. http://matplotlib.sourceforge.net/sampledoc/ David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy docstring sphinx pre-processors
ok a couple more questions: 1) how does sphinxext relate to numpydoc? sphinxext in scipy source tree -- http://svn.scipy.org/svn/numpy/trunk/doc/sphinxext/ numpydoc on PyPI -- http://pypi.python.org/pypi?%3Aaction=searchterm=numpydocsubmit=search 2) what about postprocess.py, should i be using this too? ( http://svn.scipy.org/svn/numpy/trunk/doc/) thanks again elaine On Mon, Sep 21, 2009 at 1:23 PM, Robert Kern robert.k...@gmail.com wrote: On Mon, Sep 21, 2009 at 12:20, Elaine Angelino elaine.angel...@gmail.com wrote: thanks robert! yes i saw this (http://svn.scipy.org/svn/numpy/trunk/doc/sphinxext/) but is there a good description of how to use this? i'm looking for a standard recipe that could be followed by myself and others. e.g. what functions to call and in what order... i would like to emulate what numpy does as closely as possible. http://svn.scipy.org/svn/numpy/trunk/doc/source/conf.py -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] fixed-point arithmetic
Robert Kern wrote: On Mon, Sep 21, 2009 at 12:02, Neal Becker ndbeck...@gmail.com wrote: Robert Kern wrote: On Mon, Sep 21, 2009 at 10:57, Neal Becker ndbeck...@gmail.com wrote: David Cournapeau wrote: On Mon, Sep 21, 2009 at 9:00 PM, Neal Becker ndbeck...@gmail.com wrote: numpy arrays of fpi should support all numeric operations. Also mixed fpi/integer operations. I'm not sure how to go about implementing this. At first, I was thinking to just subclass numpy array. But, I don't think this provides fpi scalars, and their associated operations. Using dtype seems more straightforward. I would first try to see how far you could go using a pure python object as a dtype. For example (on python 2.6): from decimal import Decimal import numpy as np a = np.array([1, 2, 3], Decimal) b = np.array([2, 3, 4], Decimal) a + b works as expected. A lot of things won't work (e.g. most transcendent functions, which would require a specific implementation anyway), but arithmetic, etc... would work. Then, you could think about implementing the class in cython. If speed is an issue, then implementing your own dtype seems the way to go - I don't know exactly what kind of speed increase you could hope from going the object - dtype, though. We don't want to create arrays of fixed-pt objects. That would be very wasteful. What I have in mind is that integer_bits, frac_bits are attributes of the entire arrays, not the individual elements. The array elements are just plain integers. At first I'm thinking that we could subclass numpy array, adding the int_bits and frac_bits attributes. The arithmetic operators would all have to be overloaded. The other aspect is that accessing an element of the array would return a fixed-pt object (not an integer). Actually, what you would do is create a new dtype, not a subclass of ndarray. The new datetime dtypes are similar in that they too are parameterized dtypes. But doesn't this mean that each array element has it's own int_bits, frac_bits attributes? I don't want that. No, I'm suggesting that the dtype has the int_bits and frac_bits information just like the new datetime dtypes have their unit information. 1. Where would I find this new datetime dtype? 2. Don't know exactly what 'parameterized' dtypes are. Does this mean that the dtype for 8.1 format fixed-pt is different from the dtype for 6.2 format, for example? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Simple pattern recognition
I asked this question at http://stackoverflow.com/questions/1449139/simple-object-recognition and get lots of nice feedback, and finally I have managed to implement what I wanted. What I was looking for is named connected component labelling or analysis for my connected component extraction I have put the code (lab2.py) and the image (particles.png) under: http://code.google.com/p/ccnworks/source/browse/#svn/trunk/AtSc450/labs What do you think of improving that code and adding into scipy's ndimage library (like connected_components()) ? Comments and suggestions are welcome :) On Wed, Sep 16, 2009 at 7:22 PM, Gökhan Sever gokhanse...@gmail.com wrote: Hello all, I want to be able to count predefined simple rectangle shapes on an image as shown like in this one: http://img7.imageshack.us/img7/2327/particles.png Which is in my case to count all the blue pixels (they are ice-snow flake shadows in reality) in one of the column. What is the way to automate this task, which library or technique should I study to tackle it. Thanks. -- Gökhan -- Gökhan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy-User] Simple pattern recognition
I believe that pretty generic connected-component finding is already available with scipy.ndimage.label, as David suggested at the beginning of the thread... This function takes a binary array (e.g. zeros where the background is, non-zero where foreground is) and outputs an array where each connected component of non-background pixels has a unique non-zero label value. ndimage.find_objects will then give slices (e.g. bounding boxes) for each labeled object (or a subset of them as specified). There are also a ton of statistics you can calculate based on the labeled objects -- look at the entire ndimage.measurements namespace. Zach On Sep 21, 2009, at 1:45 PM, Gökhan Sever wrote: I asked this question at http://stackoverflow.com/questions/1449139/simple-object-recognition and get lots of nice feedback, and finally I have managed to implement what I wanted. What I was looking for is named connected component labelling or analysis for my connected component extraction I have put the code (lab2.py) and the image (particles.png) under: http://code.google.com/p/ccnworks/source/browse/#svn/trunk/AtSc450/ labs What do you think of improving that code and adding into scipy's ndimage library (like connected_components()) ? Comments and suggestions are welcome :) On Wed, Sep 16, 2009 at 7:22 PM, Gökhan Sever gokhanse...@gmail.com wrote: Hello all, I want to be able to count predefined simple rectangle shapes on an image as shown like in this one: http://img7.imageshack.us/img7/2327/particles.png Which is in my case to count all the blue pixels (they are ice-snow flake shadows in reality) in one of the column. What is the way to automate this task, which library or technique should I study to tackle it. Thanks. -- Gökhan -- Gökhan ___ SciPy-User mailing list scipy-u...@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy docstring sphinx pre-processors
ma, 2009-09-21 kello 13:35 -0400, Elaine Angelino kirjoitti: ok a couple more questions: 1) how does sphinxext relate to numpydoc? sphinxext in scipy source tree -- http://svn.scipy.org/svn/numpy/trunk/doc/sphinxext/ numpydoc on PyPI -- http://pypi.python.org/pypi?% 3Aaction=searchterm=numpydocsubmit=search They are the same. If you want to use easy_install, use numpydoc. 2) what about postprocess.py, should i be using this too? (http://svn.scipy.org/svn/numpy/trunk/doc/) It removes extra section headers from the Latex output. If you want to use it, you'll have to modify to match your module. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy-User] Simple pattern recognition
ndimage.label works differently than what I have done here. Later using find_objects you can get slices for row or column basis. Not possible to construct a dynamical structure to find objects that are in the in both axis. Could you look at the stackoverflow article once again and comment back? Thanks. On Mon, Sep 21, 2009 at 12:57 PM, Zachary Pincus zachary.pin...@yale.eduwrote: I believe that pretty generic connected-component finding is already available with scipy.ndimage.label, as David suggested at the beginning of the thread... This function takes a binary array (e.g. zeros where the background is, non-zero where foreground is) and outputs an array where each connected component of non-background pixels has a unique non-zero label value. ndimage.find_objects will then give slices (e.g. bounding boxes) for each labeled object (or a subset of them as specified). There are also a ton of statistics you can calculate based on the labeled objects -- look at the entire ndimage.measurements namespace. Zach On Sep 21, 2009, at 1:45 PM, Gökhan Sever wrote: I asked this question at http://stackoverflow.com/questions/1449139/simple-object-recognition and get lots of nice feedback, and finally I have managed to implement what I wanted. What I was looking for is named connected component labelling or analysis for my connected component extraction I have put the code (lab2.py) and the image (particles.png) under: http://code.google.com/p/ccnworks/source/browse/#svn/trunk/AtSc450/ labs What do you think of improving that code and adding into scipy's ndimage library (like connected_components()) ? Comments and suggestions are welcome :) On Wed, Sep 16, 2009 at 7:22 PM, Gökhan Sever gokhanse...@gmail.com wrote: Hello all, I want to be able to count predefined simple rectangle shapes on an image as shown like in this one: http://img7.imageshack.us/img7/2327/particles.png Which is in my case to count all the blue pixels (they are ice-snow flake shadows in reality) in one of the column. What is the way to automate this task, which library or technique should I study to tackle it. Thanks. -- Gökhan -- Gökhan ___ SciPy-User mailing list scipy-u...@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user ___ SciPy-User mailing list scipy-u...@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user -- Gökhan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Building problem on CentOS 5.3
Hi all,
I've installed python 2.5 on my centos 5.3 x86_64 machine (system
standard is 2.4), and now I want to install numpy for it. However, the
build fails. The final message is:
EnvironmentError: math library missing; rerun setup.py after setting
the MATHLIB env variable
However, from looking earlier in the output, it seems it's looking for
a library called libcpml. I'm not sure what this library is (there is
no yum package for it), and since numpy works for python 2.4 it seems
all libraries should be installed. Any help would be most appreciated.
The full build output is attached.
Thanks,
/Patrik Jonsson
Running from numpy source directory.
non-existing path in 'numpy/distutils': 'site.cfg'
F2PY Version 2
blas_opt_info:
blas_mkl_info:
libraries mkl,vml,guide not found in /usr/local/lib
libraries mkl,vml,guide not found in /usr/lib
NOT AVAILABLE
atlas_blas_threads_info:
Setting PTATLAS=ATLAS
libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
libraries ptf77blas,ptcblas,atlas not found in /usr/lib/sse2
libraries ptf77blas,ptcblas,atlas not found in /usr/lib
NOT AVAILABLE
atlas_blas_info:
libraries f77blas,cblas,atlas not found in /usr/local/lib
libraries f77blas,cblas,atlas not found in /usr/lib/sse2
libraries f77blas,cblas,atlas not found in /usr/lib
NOT AVAILABLE
/home/patrik/system-stuff/numpy-1.3.0/numpy/distutils/system_info.py:1383:
UserWarning:
Atlas (http://math-atlas.sourceforge.net/) libraries not found.
Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [atlas]) or by setting
the ATLAS environment variable.
warnings.warn(AtlasNotFoundError.__doc__)
blas_info:
libraries blas not found in /usr/local/lib
FOUND:
libraries = ['blas']
library_dirs = ['/usr/lib']
language = f77
FOUND:
libraries = ['blas']
library_dirs = ['/usr/lib']
define_macros = [('NO_ATLAS_INFO', 1)]
language = f77
lapack_opt_info:
lapack_mkl_info:
mkl_info:
libraries mkl,vml,guide not found in /usr/local/lib
libraries mkl,vml,guide not found in /usr/lib
NOT AVAILABLE
NOT AVAILABLE
atlas_threads_info:
Setting PTATLAS=ATLAS
libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
libraries lapack_atlas not found in /usr/local/lib
libraries ptf77blas,ptcblas,atlas not found in /usr/lib/sse2
libraries lapack_atlas not found in /usr/lib/sse2
libraries ptf77blas,ptcblas,atlas not found in /usr/lib
libraries lapack_atlas not found in /usr/lib
numpy.distutils.system_info.atlas_threads_info
NOT AVAILABLE
atlas_info:
libraries f77blas,cblas,atlas not found in /usr/local/lib
libraries lapack_atlas not found in /usr/local/lib
libraries f77blas,cblas,atlas not found in /usr/lib/sse2
libraries lapack_atlas not found in /usr/lib/sse2
libraries f77blas,cblas,atlas not found in /usr/lib
libraries lapack_atlas not found in /usr/lib
numpy.distutils.system_info.atlas_info
NOT AVAILABLE
/home/patrik/system-stuff/numpy-1.3.0/numpy/distutils/system_info.py:1290:
UserWarning:
Atlas (http://math-atlas.sourceforge.net/) libraries not found.
Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [atlas]) or by setting
the ATLAS environment variable.
warnings.warn(AtlasNotFoundError.__doc__)
lapack_info:
libraries lapack not found in /usr/local/lib
FOUND:
libraries = ['lapack']
library_dirs = ['/usr/lib']
language = f77
FOUND:
libraries = ['lapack', 'blas']
library_dirs = ['/usr/lib']
define_macros = [('NO_ATLAS_INFO', 1)]
language = f77
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands --compiler
options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands --fcompiler
options
running build_src
building py_modules sources
building library npymath sources
building extension numpy.core._sort sources
Generating build/src.linux-x86_64-2.5/numpy/core/include/numpy/config.h
customize Gnu95FCompiler
Found executable /usr/bin/gfortran
customize Gnu95FCompiler using config
C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic
-fasynchronous-unwind-tables -D_GNU_SOURCE -fPIC -fPIC
compile options: '-Inumpy/core/src -Inumpy/core/include
-I/usr/include/python2.5 -c'
gcc: _configtest.c
success!
removing: _configtest.c _configtest.o
C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic
-fasynchronous-unwind-tables -D_GNU_SOURCE -fPIC -fPIC
compile options: '-Inumpy/core/src -Inumpy/core/include
-I/usr/include/python2.5 -c'
gcc: _configtest.c
removing: _configtest.c _configtest.o
C compiler: gcc -pthread -fno-strict-aliasing
[Numpy-discussion] Numpy large array bug
Hello, I have downloaded numpy 1.3rc2 sources and compiled it on Ubuntu Hardy Linux x86_64. numpy.test() seems to run ok as well. Here is the bug I can reproduce import numpy as np a=np.zeros((2*1024*1024*1024 + 1), dtype=uint8) a[:]=1 # returns immediately a.mean() 0.0 print a [0 0 0 ..., 0 0 0] The bug only happens when the nElements 2G (2^31). So for dtype=uint16/32, the bug happens when size is greater thatn 2^31 as well. Can someone please tell me if I can find a patch for this? I checked the mailing list and trac and I cannot find any related bug. Thanks, Ashwin ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Simple pattern recognition
On Mon, Sep 21, 2009 at 6:45 PM, Gökhan Sever gokhanse...@gmail.com wrote: I asked this question at http://stackoverflow.com/questions/1449139/simple-object-recognition and get lots of nice feedback, and finally I have managed to implement what I wanted. What I was looking for is named connected component labelling or analysis for my connected component extraction I have put the code (lab2.py) and the image (particles.png) under: http://code.google.com/p/ccnworks/source/browse/#svn/trunk/AtSc450/labs What do you think of improving that code and adding into scipy's ndimage library (like connected_components()) ? Comments and suggestions are welcome :) On Wed, Sep 16, 2009 at 7:22 PM, Gökhan Sever gokhanse...@gmail.com wrote: Hello all, I want to be able to count predefined simple rectangle shapes on an image as shown like in this one: http://img7.imageshack.us/img7/2327/particles.png Which is in my case to count all the blue pixels (they are ice-snow flake shadows in reality) in one of the column. What is the way to automate this task, which library or technique should I study to tackle it. Thanks. -- Gökhan Hi, cool! I didn't even know there was an ndimage... :) Something similar(connected components) can be found in pygame.transform and pygame.mask. However that code is in C. cheers, ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] fixed-point arithmetic
On Mon, Sep 21, 2009 at 12:39, Neal Becker ndbeck...@gmail.com wrote: 1. Where would I find this new datetime dtype? It's in the SVN trunk. 2. Don't know exactly what 'parameterized' dtypes are. Does this mean that the dtype for 8.1 format fixed-pt is different from the dtype for 6.2 format, for example? Yes. The dtype code letter is the same, but the dtype object has metadata attached to it in the form of a dictionary. The ufunc loops get references to the array objects and will look at the dtype metadata in order to figure out exactly what to do. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy large array bug
A Monday 21 September 2009 19:45:27 Kashyap Ashwin escrigué: Hello, I have downloaded numpy 1.3rc2 sources and compiled it on Ubuntu Hardy Linux x86_64. numpy.test() seems to run ok as well. Here is the bug I can reproduce import numpy as np a=np.zeros((2*1024*1024*1024 + 1), dtype=uint8) a[:]=1 # returns immediately a.mean() 0.0 print a [0 0 0 ..., 0 0 0] The bug only happens when the nElements 2G (2^31). So for dtype=uint16/32, the bug happens when size is greater thatn 2^31 as well. Yup. I can reproduce your problem with NumPy 1.3.0 (final) and a 64-bit platform. I suppose that you should file a bug better. -- Francesc Alted ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] something wrong with docs?
ma, 2009-09-21 kello 13:15 -0400, Skipper Seabold kirjoitti: On Mon, Sep 21, 2009 at 7:27 AM, Neal Becker ndbeck...@gmail.com wrote: I'm trying to read about subclassing. When I view http://docs.scipy.org/doc/numpy/user/basics.subclassing.html?highlight=subclass#module- numpy.doc.subclassing It seems the examples show the _outputs_ of tests, but I don't see the actual example code. e.g., the first example renders like this: Simple example - adding an extra attribute to ndarray¶ Using the object looks like this: I'd like to see this sorted as well. The problem is that the `testcode` directive http://docs.scipy.org/numpy/docs/numpy-docs/user/basics.subclassing.rst/ is not recognized. I was recently a bit confused by this, and I went to the rst file to view the code, but that's obviously not a fix for the rendering problem. The `sphinx.ext.doctest` extension is not enabled, so the testcode:: etc. directives are not available. I'm not sure if it should be enabled -- it would be cleaner to just replace the testcode:: stuff with the ordinary example markup. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy-User] Simple pattern recognition
I think Zachary is right, ndimage does what you want: In [48]: image = array( [[0,0,0,1,1,0,0], [0,0,0,1,1,1,0], [0,0,0,1,0,0,0], [0,0,0,0,0,0,0], [0,1,0,0,0,0,0], [0,1,1,0,0,0,0], [0,0,0,0,1,1,0], [0,0,0,0,1,1,1]]) In [57]: import scipy.ndimage as ndimage In [58]: labels, num_found = ndimage.label(image) In [59]: object_slices = ndimage.find_objects(labels) In [60]: image[object_slices[0]] Out[60]: array([[1, 1, 0], [1, 1, 1], [1, 0, 0]]) In [61]: image[object_slices[1]] Out[61]: array([[1, 0], [1, 1]]) In [62]: image[object_slices[2]] Out[62]: array([[1, 1, 0], [1, 1, 1]]) David On 21-Sep-09, at 2:04 PM, Gökhan Sever wrote: ndimage.label works differently than what I have done here. Later using find_objects you can get slices for row or column basis. Not possible to construct a dynamical structure to find objects that are in the in both axis. Could you look at the stackoverflow article once again and comment back? Thanks. On Mon, Sep 21, 2009 at 12:57 PM, Zachary Pincus zachary.pin...@yale.edu wrote: I believe that pretty generic connected-component finding is already available with scipy.ndimage.label, as David suggested at the beginning of the thread... This function takes a binary array (e.g. zeros where the background is, non-zero where foreground is) and outputs an array where each connected component of non-background pixels has a unique non-zero label value. ndimage.find_objects will then give slices (e.g. bounding boxes) for each labeled object (or a subset of them as specified). There are also a ton of statistics you can calculate based on the labeled objects -- look at the entire ndimage.measurements namespace. Zach On Sep 21, 2009, at 1:45 PM, Gökhan Sever wrote: I asked this question at http://stackoverflow.com/questions/1449139/simple-object-recognition and get lots of nice feedback, and finally I have managed to implement what I wanted. What I was looking for is named connected component labelling or analysis for my connected component extraction I have put the code (lab2.py) and the image (particles.png) under: http://code.google.com/p/ccnworks/source/browse/#svn/trunk/AtSc450/ labs What do you think of improving that code and adding into scipy's ndimage library (like connected_components()) ? Comments and suggestions are welcome :) On Wed, Sep 16, 2009 at 7:22 PM, Gökhan Sever gokhanse...@gmail.com wrote: Hello all, I want to be able to count predefined simple rectangle shapes on an image as shown like in this one: http://img7.imageshack.us/img7/2327/particles.png Which is in my case to count all the blue pixels (they are ice-snow flake shadows in reality) in one of the column. What is the way to automate this task, which library or technique should I study to tackle it. Thanks. -- Gökhan -- Gökhan ___ SciPy-User mailing list scipy-u...@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user ___ SciPy-User mailing list scipy-u...@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user -- Gökhan ___ SciPy-User mailing list scipy-u...@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy large array bug
On Mon, Sep 21, 2009 at 12:30 PM, Francesc Alted fal...@pytables.orgwrote: A Monday 21 September 2009 19:45:27 Kashyap Ashwin escrigué: Hello, I have downloaded numpy 1.3rc2 sources and compiled it on Ubuntu Hardy Linux x86_64. numpy.test() seems to run ok as well. Here is the bug I can reproduce import numpy as np a=np.zeros((2*1024*1024*1024 + 1), dtype=uint8) a[:]=1 # returns immediately a.mean() 0.0 print a [0 0 0 ..., 0 0 0] The bug only happens when the nElements 2G (2^31). So for dtype=uint16/32, the bug happens when size is greater thatn 2^31 as well. Yup. I can reproduce your problem with NumPy 1.3.0 (final) and a 64-bit platform. I suppose that you should file a bug better. Does is persist for svn? IIRC, there is another ticket for a slicing bug for large arrays. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] masked arrays as array indices (is a bad idea)
On Sep 21, 2009, at 12:17 PM, Ryan May wrote: 2009/9/21 Ernest Adrogué eadro...@gmx.net Hello there, Given a masked array such as this one: In [19]: x = np.ma.masked_equal([-1, -1, 0, -1, 2], -1) In [20]: x Out[20]: masked_array(data = [-- -- 0 -- 2], mask = [ True True False True False], fill_value = 99) When you make an assignemnt in the vein of x[x == 0] = 25 the result can be a bit puzzling: In [21]: x[x == 0] = 25 In [22]: x Out[22]: masked_array(data = [25 25 25 25 2], mask = [False False False False False], fill_value = 99) Is this the correct result or have I found a bug? I see the same here on 1.4.0.dev7400. Seems pretty odd to me. Then again, it's a bit more complex using masked boolean arrays for indexing since you have True, False, and masked values. Anyone have thoughts on what *should* happen here? Or is this it? Using a masked array in fancy indexing is always a bad idea, as there's no way of guessing the behavior one would want for missing values: should they be evaluated as False ? True ? You should really use the `filled` method to control the behavior. x[(x==0).filled(False)] masked_array(data = [0], mask = [False], fill_value = 99) x[(x==0).filled(True)] masked_array(data = [-- -- 0 --], mask = [ True True False True], fill_value = 99) P. [If you're really interested: When testing for equality, a masked array is first filled with 0 (that was the behavior of the first implementation of numpy.ma), tested for equality, and the mask of the result set to the mask of the input. When used in fancy indexing, a masked array is viewed as a standard ndarray by dropping the mask. In the current case, the combination is therefore equivalent to (x.filled(0)==0), which explains why the missing values are treated as True... I agree that the prefilling may not be necessary...] ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Best way to insert C code in numpy code
David Cournapeau wrote: Xavier Gnata wrote: Hi, I have a large 2D numpy array as input and a 1D array as output. In between, I would like to use C code. C is requirement because it has to be fast and because the algorithm cannot be written in a numpy oriented way :( (no way...really). Which tool should I use to achieve that? waeve.inline? pyrex? What is the baseline? That's only a data point, but I almost always use cython in those cases, unless I need 'very advanced' features of the C API in which case I just do it manually. cheers, David Ok :) Should I read that to learn you cython and numpy interact? Or is there another best documentation (with examples...)? Xavier http://wiki.cython.org/tutorials/numpy ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy large array bug
Yes, it happens for the trunk as well. import numpy as np a=np.zeros((2*1024*1024*1024 + 1), dtype=uint8) a[:]=1 # returns immediately a.mean() 0.0 print a [0 0 0 ..., 0 0 0] The bug only happens when the nElements 2G (2^31). So for dtype=uint16/32, the bug happens when size is greater thatn 2^31 as well. Yup. I can reproduce your problem with NumPy 1.3.0 (final) and a 64-bit platform. I suppose that you should file a bug better. Does is persist for svn? IIRC, there is another ticket for a slicing bug for large arrays. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Best way to insert C code in numpy code
Xavier Gnata wrote: David Cournapeau wrote: That's only a data point, but I almost always use cython in those cases, I'm a second data point, but I think there are many more. Judging from the SciPy conference, Cython is the preferred method for new projects. Should I read that to learn you cython and numpy interact? http://wiki.cython.org/tutorials/numpy That's probably the best starting point. Also look online for the videos of the presentations at the SciPy2009 conference -- there were a few Cython ones. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Best way to insert C code in numpy code
On 21-Sep-09, at 2:55 PM, Xavier Gnata wrote: Should I read that to learn you cython and numpy interact? Or is there another best documentation (with examples...)? You should have a look at the Bresenham algorithm thread you posted. I went to the trouble of converting some Python code for Bresenham's algorithm to Cython, and a pointer to the Cython+NumPy tutorial: http://wiki.cython.org/tutorials/numpy David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy-User] Simple pattern recognition
Ahh my blindness and apologies :) The nice feeling of reinventing the wheel... Probably I forgot to reshape the image data in the first place before applying into ndimage.label(). However, this was a nice example to understand recursion, and get to know some basics of computer vision and few libraries (OpenCV, pygraph) during my research. Thanks again for all kind replies. On Mon, Sep 21, 2009 at 1:36 PM, David Warde-Farley d...@cs.toronto.eduwrote: I think Zachary is right, ndimage does what you want: In [48]: image = array( [[0,0,0,1,1,0,0], [0,0,0,1,1,1,0], [0,0,0,1,0,0,0], [0,0,0,0,0,0,0], [0,1,0,0,0,0,0], [0,1,1,0,0,0,0], [0,0,0,0,1,1,0], [0,0,0,0,1,1,1]]) In [57]: import scipy.ndimage as ndimage In [58]: labels, num_found = ndimage.label(image) In [59]: object_slices = ndimage.find_objects(labels) In [60]: image[object_slices[0]] Out[60]: array([[1, 1, 0], [1, 1, 1], [1, 0, 0]]) In [61]: image[object_slices[1]] Out[61]: array([[1, 0], [1, 1]]) In [62]: image[object_slices[2]] Out[62]: array([[1, 1, 0], [1, 1, 1]]) David On 21-Sep-09, at 2:04 PM, Gökhan Sever wrote: ndimage.label works differently than what I have done here. Later using find_objects you can get slices for row or column basis. Not possible to construct a dynamical structure to find objects that are in the in both axis. Could you look at the stackoverflow article once again and comment back? Thanks. On Mon, Sep 21, 2009 at 12:57 PM, Zachary Pincus zachary.pin...@yale.edu wrote: I believe that pretty generic connected-component finding is already available with scipy.ndimage.label, as David suggested at the beginning of the thread... This function takes a binary array (e.g. zeros where the background is, non-zero where foreground is) and outputs an array where each connected component of non-background pixels has a unique non-zero label value. ndimage.find_objects will then give slices (e.g. bounding boxes) for each labeled object (or a subset of them as specified). There are also a ton of statistics you can calculate based on the labeled objects -- look at the entire ndimage.measurements namespace. Zach On Sep 21, 2009, at 1:45 PM, Gökhan Sever wrote: I asked this question at http://stackoverflow.com/questions/1449139/simple-object-recognition and get lots of nice feedback, and finally I have managed to implement what I wanted. What I was looking for is named connected component labelling or analysis for my connected component extraction I have put the code (lab2.py) and the image (particles.png) under: http://code.google.com/p/ccnworks/source/browse/#svn/trunk/AtSc450/ labs What do you think of improving that code and adding into scipy's ndimage library (like connected_components()) ? Comments and suggestions are welcome :) On Wed, Sep 16, 2009 at 7:22 PM, Gökhan Sever gokhanse...@gmail.com wrote: Hello all, I want to be able to count predefined simple rectangle shapes on an image as shown like in this one: http://img7.imageshack.us/img7/2327/particles.png Which is in my case to count all the blue pixels (they are ice-snow flake shadows in reality) in one of the column. What is the way to automate this task, which library or technique should I study to tackle it. Thanks. -- Gökhan -- Gökhan ___ SciPy-User mailing list scipy-u...@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user ___ SciPy-User mailing list scipy-u...@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user -- Gökhan ___ SciPy-User mailing list scipy-u...@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Gökhan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy large array bug
I can confirm this bug for the last svn. Also: a.put([2*1024*1024*1024 + 100,], 8) IndexError: index out of range for array in this case, I think the error is that in numpy/core/src/multiarray/item_selection.c in PyArray_PutTo line 209 should be: intp i, chunk, ni, max_item, nv, tmp; instead of: int i, chunk, ni, max_item, nv, tmp; fixing it as suggested: a.put([2*1024*1024*1024 + 100,], 8) a.max() 8 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Indexing transposes the array?
Why does indexing seem to transpose this array? In [14]: x = arange(8).reshape((2,2,2)) In [15]: x[0,:,:] Out[15]: array([[0, 1], [2, 3]]) In [16]: x[0,:,[0,1]] Out[16]: array([[0, 2], [1, 3]]) Thanks, Jonathan. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy question: Best hardware for Numpy?
On 21-Sep-09, at 10:53 AM, David Cournapeau wrote: Concerning the hardware, I have just bought a core i7 (the cheapest model is ~ 200$ now, with 4 cores and 8 Mb of shared cache), and the thing flies for floating point computation. My last computer was a pentium 4 so I don't have a lot of reference, but you can compute ~ 300e6 exp (assuming a contiguous array), and ATLAS 3.8.3 built on it is extremely fast - using the threaded version, the asymptotic peak performances are quite impressive. It takes for example 14s to inverse a 5000x5000 matrix of double. I thought you had a Macbook too? The Core i5 750 seems like a good buy right now as well. A bit cheaper, 4 cores and 8Mb of shared cache though at a slightly lower clock speed. David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] masked arrays as array indices (is a bad idea)
21/09/09 @ 14:43 (-0400), thus spake Pierre GM: On Sep 21, 2009, at 12:17 PM, Ryan May wrote: 2009/9/21 Ernest Adrogué eadro...@gmx.net Hello there, Given a masked array such as this one: In [19]: x = np.ma.masked_equal([-1, -1, 0, -1, 2], -1) In [20]: x Out[20]: masked_array(data = [-- -- 0 -- 2], mask = [ True True False True False], fill_value = 99) When you make an assignemnt in the vein of x[x == 0] = 25 the result can be a bit puzzling: In [21]: x[x == 0] = 25 In [22]: x Out[22]: masked_array(data = [25 25 25 25 2], mask = [False False False False False], fill_value = 99) Is this the correct result or have I found a bug? I see the same here on 1.4.0.dev7400. Seems pretty odd to me. Then again, it's a bit more complex using masked boolean arrays for indexing since you have True, False, and masked values. Anyone have thoughts on what *should* happen here? Or is this it? Using a masked array in fancy indexing is always a bad idea, as there's no way of guessing the behavior one would want for missing values: should they be evaluated as False ? True ? You should really use the `filled` method to control the behavior. x[(x==0).filled(False)] masked_array(data = [0], mask = [False], fill_value = 99) x[(x==0).filled(True)] masked_array(data = [-- -- 0 --], mask = [ True True False True], fill_value = 99) P. [If you're really interested: When testing for equality, a masked array is first filled with 0 (that was the behavior of the first implementation of numpy.ma), tested for equality, and the mask of the result set to the mask of the input. When used in fancy indexing, a masked array is viewed as a standard ndarray by dropping the mask. In the current case, the combination is therefore equivalent to (x.filled(0)==0), which explains why the missing values are treated as True... I agree that the prefilling may not be necessary...] This explains why x[x == 3] = 4 works as expected, whereas x[x == 0] = 4 ruins everything. Basically, any condition that matches 0 will match every masked item as well. I don't know, but maybe it would be better to raise an exception when the index is a masked array then. The current behaviour seems a bit confusing to me. -- Ernest ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Indexing transposes the array?
On 21-Sep-09, at 3:36 PM, Jonathan Taylor wrote: Why does indexing seem to transpose this array? In [14]: x = arange(8).reshape((2,2,2)) In [15]: x[0,:,:] Out[15]: array([[0, 1], [2, 3]]) In [16]: x[0,:,[0,1]] Out[16]: array([[0, 2], [1, 3]]) The last example in this section (and the explanation) proves instructive: http://docs.scipy.org/doc/numpy/user/basics.indexing.html#indexing-multi-dimensional-arrays Also, notice: In [121]: x[0,:,0] Out[121]: array([0, 2]) In [122]: x[0,:,[0]] Out[122]: array([[0, 2]] The fancy indexing is basically going to look at x[0,:,0], x[0,:,1] and merge them along a new axis. If you used the fancy index along the second dimension, it would pull out the rows, like you want it to. David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Numpy large array bug
Also, what about PyArray_PutMask() That function also has a line like int i, chunk, ni, max_item, nv, tmp; Should that be changed as well? (Your patch does not fix my original issue.) BTW, in numpy 1.3, that is present in numpy/core/src/multiarraymodule.c. Can someone please give me a temporary patch to test? I am not familiar with numpy codebase! -Ashwin I can confirm this bug for the last svn. Also: a.put([2*1024*1024*1024 + 100,], 8) IndexError: index out of range for array in this case, I think the error is that in numpy/core/src/multiarray/item_selection.c in PyArray_PutTo line 209 should be: intp i, chunk, ni, max_item, nv, tmp; instead of: int i, chunk, ni, max_item, nv, tmp; fixing it as suggested: a.put([2*1024*1024*1024 + 100,], 8) a.max() 8 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] masked arrays as array indices (is a bad idea)
On Sep 21, 2009, at 4:23 PM, Ernest Adrogué wrote: This explains why x[x == 3] = 4 works as expected, whereas x[x == 0] = 4 ruins everything. Basically, any condition that matches 0 will match every masked item as well. There's room for improvement here indeed. I need to check first whether fixing the comparison methods doesn't break anything. I don't know, but maybe it would be better to raise an exception when the index is a masked array then. The current behaviour seems a bit confusing to me. That'd be modifying ndarray.__getitem__ and I don't see that happening. In the meantime, please just fill your masked array with the `filled` method or the corresponding function. Remmber that masked arrays are for convenience. As soon as you try to do some heavy computations, you're better processing data and mask yourself. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy large array bug
I think the original bug is due to line 535 of numpy/core/src/multiarray/ctors.c (svn) that should be: intp numcopies, nbytes; instead of: int numcopies, nbytes; To resume: in line 535 of numpy/core/src/multiarray/ctors.c and in line 209 of numpy/core/src/multiarray/item_selection.c int should be replaced with intp. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy large array bug
Hi, Luca, On Mon, Sep 21, 2009 at 4:52 PM, Citi, Luca lc...@essex.ac.uk wrote: I think the original bug is due to line 535 of numpy/core/src/multiarray/ctors.c (svn) that should be: intp numcopies, nbytes; instead of: int numcopies, nbytes; To resume: in line 535 of numpy/core/src/multiarray/ctors.c and in line 209 of numpy/core/src/multiarray/item_selection.c int should be replaced with intp. Please open a ticket for this. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] fixed-point arithmetic
On Tue, Sep 22, 2009 at 12:57 AM, Neal Becker ndbeck...@gmail.com wrote: David Cournapeau wrote: On Mon, Sep 21, 2009 at 9:00 PM, Neal Becker ndbeck...@gmail.com wrote: numpy arrays of fpi should support all numeric operations. Also mixed fpi/integer operations. I'm not sure how to go about implementing this. At first, I was thinking to just subclass numpy array. But, I don't think this provides fpi scalars, and their associated operations. Using dtype seems more straightforward. I would first try to see how far you could go using a pure python object as a dtype. For example (on python 2.6): from decimal import Decimal import numpy as np a = np.array([1, 2, 3], Decimal) b = np.array([2, 3, 4], Decimal) a + b works as expected. A lot of things won't work (e.g. most transcendent functions, which would require a specific implementation anyway), but arithmetic, etc... would work. Then, you could think about implementing the class in cython. If speed is an issue, then implementing your own dtype seems the way to go - I don't know exactly what kind of speed increase you could hope from going the object - dtype, though. We don't want to create arrays of fixed-pt objects. That would be very wasteful. Maybe, but that would be a good way to prototype the thing. What I have in mind is that integer_bits, frac_bits are attributes of the entire arrays, not the individual elements. The array elements are just plain integers. That's not really how numpy arrays are designed: type-specific info should be in the dtype, not the array class. As Robert mentioned, the recently added datetime dtype shows an example on how to do it. David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy large array bug
Here it is... http://projects.scipy.org/numpy/ticket/1229 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] something wrong with docs?
On Mon, Sep 21, 2009 at 11:32 AM, Pauli Virtanen p...@iki.fi wrote: The `sphinx.ext.doctest` extension is not enabled, so the testcode:: etc. directives are not available. I'm not sure if it should be enabled -- it would be cleaner to just replace the testcode:: stuff with the ordinary example markup. Why not enable it? It would be nice if we could move gradually towards docs whose examples (at least those marked as such) were always run via sphinx. The more we do this, the higher the chances of non-zero overlap between documentation and reality :) Cheers, f ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Multi-dimensional indexing
I forgot to mention that the second array, which I wish to conditionally select elements from using tmax_idx, has the same dimensions as the speed array, That is, (ntimes, nlon, nlat) = U.shape And tmax_idx has dimensions of (nlon, nlat). Daran -- My apology for the simplemindedness of my question. I've been a long time user of NumPy and its predecessor Numeric, but am struggling to understand fancy indexing for multi- dimensional arrays. Here is the problem I am trying to solve. Suppose I have an 3-D array, named speed whose first dimen- sion is time, and the second and third dimensions are latitude and longitude. Further suppose that I wish to find the time where the values at each point are at their maximum. This can easily be done with the following code: tmax_idx = np.argsort(speed, axis=0) I now wish to use this tmax_idx array to conditionally select the values from a separate array. How can this be done with fancy indexing? I've certainly done this sort of selection with index arrays in 1D, but I can not wrap my head round the multi-dimensionl index selection, even after carefully studying the excellent indexing documentation and examples on-line. I'd like to learn how to do this, to avoid the brute force looping solution of: mean_u = np.zeros((nlon, nlat), dtype=np.float32) for i in xrange(nlon): for j in xrange(nlat): mean_u[i,j] = U[max_spd_idx[i,j],i,j] As you know, this is reasonably fast for modest-sized arrays, but is far more expensive for large arrays. Thanks in advance for your help. Sincerely, Daran Rife ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [IPython-dev] Testing matplotlib on IPython trunk
Thanks Fernando for the quick response. Today this is the 3rd time I am hitting an unsupported feature in the Python lands. 1-) No attribute docstrings 2-) Look this question: http://stackoverflow.com/questions/1458203/reading-a-float-from-string and 3rd is this. However I think I influenced to guys in our campus to take a look Python. One using Matlab-Simulink and C on collision-detection system design, the latter uses C to design a small scale embedded acquisition system for UAV platforms. He uses an ARM Cortex A8 processor powered Gumstix boardhttp://www.gumstix.com/store/catalog/product_info.php?cPath=31products_id=228. Xubuntu 9.04 runs on it. I saw Python 2.6.2 installed, however not sure how easy would that be to bring rest of the scipy stack into that machine. Besides, tomorrow there is going to be a Matlab seminar here http://www.mathworks.com/company/events/seminars/seminar39323.html It is about a SciPy advanced tutorial long. Many similar subjects I see there: *Speeding Up MATLAB Applications:Tips and Tricks for Writing Efficient Code *Topics include: • Understanding preallocation and vectorization • Addressing bottlenecks • Efficient indexing and manipulations • JIT • Interpreter • Mex *Brief Introduction to Parallel Computing with MATLAB *• Task parallel applications for faster processing • Data parallel applications for handling large data sets • Scheduling your programs to run I hope I will not kick out from the session by keep commenting oh that is possible in Python, oh this is too :) On Tue, Sep 22, 2009 at 12:18 AM, Fernando Perez fperez@gmail.comwrote: 2009/9/21 Gökhan Sever gokhanse...@gmail.com: It's a very late reply but I am wondering how to make these appear in the Ipy dev loaded into the session but not visible to a whos listing? I don't think that's supported quite right now. IPython does one special thing to support a clean %whos listing: right before opening up the user mainloop, it checks all keys in the user namespace, and later on when %whos is run, those variables that were initially present are not displayed. So for now if you do this interactively, you will unfortunately pollute %whos. This is one thing we'll need to make sure works nicely again when the dust settles. Cheers, f -- Gökhan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
