Re: [Numpy-discussion] Removing masked arrays for 1.7? (Was 1.7 blockers)
Hi, I just discovered that the NA mask will modify the base ndarray object. So I read about it to find the consequences on our c code. Up to now I have fully read: http://docs.scipy.org/doc/numpy/reference/arrays.maskna.html and partially read: https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst https://github.com/njsmith/numpy/wiki/NA-discussion-status In those documents, I see a problem with legacy code that will receive an NA masked array as input. If I missed something, tell me. All our c functions check their inputs array with PyArray_Check and PyArray_ISALIGNED. If the NA mask array is set inside the ndarray c object, our c functions who don't know about it and will treat those inputs as not masked. So the user will have unexpected results. The output will be an ndarray without mask but the code will have used the masked value. This will also happen with all other c code that use ndarray. In our case, all the input check is done at the same place, so adding the check with PyArray_HasNASupport(PyArrayObject* obj) to raise an error will be easy for us. But I don't think this is the case for most other c code. So I would prefer a separate object to protect users from code not being updated to reject NA masked inputs. An alternative would be to have PyArray_Check return False for the NA masked array, but I don't like that as this break the semantic that it check for the class. A last option I see would be to make the NPY_ARRAY_BEHAVED flag also check that the array is not an NA marked array. I suppose many c code do this check. But this is not a bullet proof check as not all code (as ours) do not use it. Also, I don't mind the added pointers to the structure as we use big arrays. thanks Frédéric ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Removing masked arrays for 1.7? (Was 1.7 blockers)
On Fri, Apr 20, 2012 at 9:49 AM, Chris Barker chris.bar...@noaa.gov wrote: I recall discossion a couple times in the past of having some special-case numpy arrays for the simple, small cases -- perhaps 1-d or 2-d C-contiguous only, for instance. That might be a better way to address the small-array performance issue, and free us of concerns about minor growth to the core ndarray object. +1 on that: I once wrote such code in pyrex (years ago) and it worked extremely well for me. No fancy features, very small footprint and highly optimized codepaths that gave me excellent performance. Cheers, f ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] A 1.6.2 release?
Hi All, Given the amount of new stuff coming in 1.7 and the slip in it's schedule, I wonder if it would be worth putting out a 1.6.2 release with fixes for einsum, ticket 1578, perhaps some others. My reasoning is that the fall releases of Fedora, Ubuntu are likely to still use 1.6 and they might as well use a somewhat fixed up version. The downside is located and backporting fixes is likely to be a fair amount of work. A 1.7 release would be preferable, but I'm not sure when we can make that happen. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Command-line options for (Windows) NumPy Installer?
Hi, is there any documentation available on exactly which command line options are available from NumPy's 'superpack' installers on Windows? E.g., http://docs.scipy.org/doc/numpy/user/install.html mentions an /arch flag, but I'm not seeing anything else called out. Thanks! Dave ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] A 1.6.2 release?
Hi, On Fri, Apr 20, 2012 at 11:04 AM, Charles R Harris charlesr.har...@gmail.com wrote: Hi All, Given the amount of new stuff coming in 1.7 and the slip in it's schedule, I wonder if it would be worth putting out a 1.6.2 release with fixes for einsum, ticket 1578, perhaps some others. My reasoning is that the fall releases of Fedora, Ubuntu are likely to still use 1.6 and they might as well use a somewhat fixed up version. The downside is located and backporting fixes is likely to be a fair amount of work. A 1.7 release would be preferable, but I'm not sure when we can make that happen. Also, I believe Debian will very soon freeze testing in order to prepare to release the next stable. See you, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] (no subject)
What would be the best way to remove the maximum from a cube and collapse the remaining elements along the z-axis ? For example, I want to reduce Cube to NewCube: Cube array([[[ 13, 2, 3, 42], [ 5, 100, 7, 8], [ 9, 1, 11, 12]], [[ 25, 4, 15, 1], [ 17, 30, 9, 20], [ 21, 2, 23, 24]], [[ 1, 2, 27, 28], [ 29, 18, 31, 32], [ -1, 3, 35, 4]]]) NewCube array([[[ 13, 2, 3, 1], [ 5, 30, 7, 8], [ 9, 1, 11, 12]], [[ 1, 2, 15, 28], [ 17, 18, 9, 20], [ -1, 2, 23, 4]]]) I tried with argmax() and then roll() and delete() but these all work on 1-D arrays only. Thanks. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Removing masked arrays for 1.7? (Was 1.7 blockers)
Fernando Perez fperez@gmail.com wrote: On Fri, Apr 20, 2012 at 9:49 AM, Chris Barker chris.bar...@noaa.gov wrote: I recall discossion a couple times in the past of having some special-case numpy arrays for the simple, small cases -- perhaps 1-d or 2-d C-contiguous only, for instance. That might be a better way to address the small-array performance issue, and free us of concerns about minor growth to the core ndarray object. +1 on that: I once wrote such code in pyrex (years ago) and it worked extremely well for me. No fancy features, very small footprint and highly optimized codepaths that gave me excellent performance. I don't think you gain that much by using a different type though? Those optimized code paths could be plugged into NumPy as well. I always assumed that it would be possible to optimize NumPy, just that nobody invested time in it. Starting from scratch you gain that you don't have to work with and understand NumPy's codebase, but I honestly think that's a small price to pay for compatibility. Dag Cheers, f ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] (no subject)
On Fri, Apr 20, 2012 at 2:15 PM, Andre Martel soucoupevola...@yahoo.comwrote: What would be the best way to remove the maximum from a cube and collapse the remaining elements along the z-axis ? For example, I want to reduce Cube to NewCube: Cube array([[[ 13, 2, 3, 42], [ 5, 100, 7, 8], [ 9, 1, 11, 12]], [[ 25, 4, 15, 1], [ 17, 30, 9, 20], [ 21, 2, 23, 24]], [[ 1, 2, 27, 28], [ 29, 18, 31, 32], [ -1, 3, 35, 4]]]) NewCube array([[[ 13, 2, 3, 1], [ 5, 30, 7, 8], [ 9, 1, 11, 12]], [[ 1, 2, 15, 28], [ 17, 18, 9, 20], [ -1, 2, 23, 4]]]) I tried with argmax() and then roll() and delete() but these all work on 1-D arrays only. Thanks. Actually, those commands do work with n-dimensional arrays, but you'll have to specify the axis (the default for all these functions is `axis=None` which tell the function to operate on flattened the arrays). If you don't care about the order of the collapse, you can just do a simple sort (and drop the last---i.e. max---sub-array): np.sort(cube, axis=0)[:2] If you need to keep the order, you can probably use some combination of `np.argsort` and `np.choose`. Cheers, -Tony ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Removing masked arrays for 1.7? (Was 1.7 blockers)
On Fri, Apr 20, 2012 at 11:27 AM, Dag Sverre Seljebotn d.s.seljeb...@astro.uio.no wrote: I don't think you gain that much by using a different type though? Those optimized code paths could be plugged into NumPy as well. Could be: this was years ago, and the bottleneck for me was in the constructor and in basic arithmetic. I had to make millions of these vectors and I needed to do basic arithmetic, but they were always 1-d and had one to 6 entries only. So writing a very static constructor with very low overhead did make a huge difference in that project. Also, when I wrote this code numpy didn't exist, I was using Numeric. Perhaps the same results could be obtained in numpy itself with judicious coding, I don't know. But in that project, ~600 lines of really easy pyrex code (it would be cython today) made a *huge* performance difference for me. Cheers, f ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Removing masked arrays for 1.7? (Was 1.7 blockers)
On 04/20/2012 08:35 PM, Fernando Perez wrote: On Fri, Apr 20, 2012 at 11:27 AM, Dag Sverre Seljebotn d.s.seljeb...@astro.uio.no wrote: I don't think you gain that much by using a different type though? Those optimized code paths could be plugged into NumPy as well. Could be: this was years ago, and the bottleneck for me was in the constructor and in basic arithmetic. I had to make millions of these vectors and I needed to do basic arithmetic, but they were always 1-d and had one to 6 entries only. So writing a very static constructor with very low overhead did make a huge difference in that project. Oh, right. I was thinking small as in fits in L2 cache, not small as in a few dozen entries. You definitely still need a Cython class then. Dag Also, when I wrote this code numpy didn't exist, I was using Numeric. Perhaps the same results could be obtained in numpy itself with judicious coding, I don't know. But in that project, ~600 lines of really easy pyrex code (it would be cython today) made a *huge* performance difference for me. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Removing masked arrays for 1.7? (Was 1.7 blockers)
On Fri, Apr 20, 2012 at 11:39 AM, Dag Sverre Seljebotn d.s.seljeb...@astro.uio.no wrote: Oh, right. I was thinking small as in fits in L2 cache, not small as in a few dozen entries. or even two or three entries. I often use a (2,) or (3,) numpy array to represent an (x,y) point (usually pulled out from a Nx2 array). I like it 'cause i can do array math, etc. it makes the code cleaner, but it's actually faster to use tuples and do the indexing by hand :-( but yes, having something built-in, or at least very compatible with numpy would be best. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] A 1.6.2 release?
Also, I believe Debian will very soon freeze testing in order to prepare to release the next stable. yes, the estimates are around June (nor clear if the beginning of the end). Cheers, -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Removing masked arrays for 1.7? (Was 1.7 blockers)
On Fri, Apr 20, 2012 at 11:45 AM, Chris Barker chris.bar...@noaa.gov wrote: On Fri, Apr 20, 2012 at 11:39 AM, Dag Sverre Seljebotn d.s.seljeb...@astro.uio.no wrote: Oh, right. I was thinking small as in fits in L2 cache, not small as in a few dozen entries. Another example of a small array use-case: I've been using numpy for my research in multi-target tracking, which involves something like a bunch of entangled hidden markov models. I represent target states with small 2d arrays (e.g. 2x2, 4x4, ..) and observations with small 1d arrays (1 or 2 elements). It may be possible to combine a bunch of these small arrays into a single larger array and use indexing to extract views, but it is much cleaner and more intuitive to use separate, small arrays. It's also convenient to use numpy arrays rather than a custom class because I use the linear algebra functionality as well as integration with other libraries (e.g. matplotlib). I also work with approximate probabilistic inference in graphical models (belief propagation, etc), which is another area where it can be nice to work with many small arrays. In any case, I just wanted to chime in with my small bit of evidence for people wanting to use numpy for work with small arrays, even if they are currently pretty slow. If there were a special version of a numpy array that would be faster for cases like this, I would definitely make use of it. Drew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
