Re: [Numpy-discussion] better error message possible?
On Mon, Jun 4, 2012 at 11:49 PM, Nathaniel Smith n...@pobox.com wrote: On Mon, Jun 4, 2012 at 10:00 PM, Thouis (Ray) Jones tho...@gmail.com wrote: On Mon, Jun 4, 2012 at 4:27 PM, Thouis (Ray) Jones tho...@gmail.com wrote: I could look into this. There are only ~10 places the code generates this error, so it should be a pretty minor change. My initial estimate was low, but not overly so. An initial pass at adding index/dimension information to IndexErrors is here: https://github.com/thouis/numpy/tree/index_error_info Fabulous! I made a few comments there, but also: A typical result: numpy.zeros(3)[5] Traceback (most recent call last): File stdin, line 1, in module IndexError: index 5 out of bounds in dimension 0 I would say for, not in. index 5 is a bit ambiguous too... people might mis-read it as the dimension, like, the 5th index value I gave? Not sure how to make it unambiguous. Maybe: IndexError: dimension 0 index out of bounds: got 5, size is 3 ? How about: IndexError: 5 is out of bounds for dimension 0: must be in [-3, 3). to be maximally explicit about what values are allowed, and avoid the index confusion. Ray Jones ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] nditer_buffer_flag branch (was: Add data memory allocation tracing facilities. (#284))
On Tue, Jun 5, 2012 at 11:06 AM, Thouis (Ray) Jones wrote: All of the failing tests seem to have been caused by the buffer copy bug, fixed in https://github.com/mwiebe/numpy/tree/nditer_buffer_flag (but not yet pulled into numpy). I also have a version that implements tracing, with pure C in the allocation functions writing to a dynamically allocated buffer, which must then be fetched proactively by Python. However, I think this version is a little nicer to use from the Python perspective. --- Reply to this email directly or view it on GitHub: https://github.com/numpy/numpy/pull/284#issuecomment-6121817 Speaking of which, Mark - what's the status of that nditer_buffer_flag branch? Should there be a pull request? -N ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] lazy evaluation
Hey, Another discussion on lazy evaluation, given the recent activity here: https://github.com/ContinuumIO/numba/pull/6#issuecomment-6117091 A somewhat recent previous thread can be found here: http://mail.scipy.org/pipermail/numpy-discussion/2012-February/060862.html , and a NEP here: https://github.com/numpy/numpy/blob/master/doc/neps/deferred-ufunc-evaluation.rst I think trying to parse bytecode and build an expression graph for array expressions from that has disadvantages and is harder in general. For instance it won't be able to deal with branching at execution time, and things like inter-procedural analysis will be harder (not to mention you'd have to parse dtype creation). Instead, what you really want to do is hook into a lazy evaluating version of numpy, and generate your own code from the operations it records. It would be great if we implement the NEP listed above, but with a few extensions. I think Numpy should handle the lazy evaluation part, and determine when expressions should be evaluated, etc. However, for each user operation, Numpy will call back a user-installed hook implementing some interface, to allow various packages to provide their own hooks to evaluate vector operations however they want. This will include packages such as Theano, which could run things on the GPU, Numexpr, and in the future https://github.com/markflorisson88/minivect (which will likely have an LLVM backend in the future, and possibly integrated with Numba to allow inlining of numba ufuncs). The project above tries to bring together all the different array expression compilers together in a single framework, to provide efficient array expressions specialized for any data layout (nditer on steroids if you will, with SIMD, threaded and inlining capabilities). We could allow each hook to specify which dtypes it supports, and a minimal data size needed before it should be invoked (to avoid overhead for small arrays, like the openmp 'if' clause). If an operation is not supported, it will simply raise NotImplementedError, which means Numpy will evaluate the expression built so far and run its own implementation, resulting in a non-lazy array. E.g. if a library supports adding things together, but doesn't support the 'sin' function, np.sin(a + b) will result in the library executing a + b, and numpy evaluating sin on the result. So the idea is that the numpy lazy array will wrap an expression graph, which is built when the user performs operations and evaluated when needed (when a result is required or when someone tells numpy to evaluate all lazy arrays). Numpy will simply use the first hook willing to operate on data of the specified size and dtype, and will keep using that hook to build the expression until evaluated. Anyway, this is somewhat of a high-level overview. If there is any interest, we can flesh out the details and extend the NEP. Mark ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] better error message possible?
On Tue, Jun 5, 2012 at 12:15 PM, Thouis Jones thouis.jo...@curie.fr wrote: On Mon, Jun 4, 2012 at 11:49 PM, Nathaniel Smith n...@pobox.com wrote: On Mon, Jun 4, 2012 at 10:00 PM, Thouis (Ray) Jones tho...@gmail.com wrote: On Mon, Jun 4, 2012 at 4:27 PM, Thouis (Ray) Jones tho...@gmail.com wrote: I could look into this. There are only ~10 places the code generates this error, so it should be a pretty minor change. My initial estimate was low, but not overly so. An initial pass at adding index/dimension information to IndexErrors is here: https://github.com/thouis/numpy/tree/index_error_info Fabulous! I made a few comments there, but also: A typical result: numpy.zeros(3)[5] Traceback (most recent call last): File stdin, line 1, in module IndexError: index 5 out of bounds in dimension 0 I would say for, not in. index 5 is a bit ambiguous too... people might mis-read it as the dimension, like, the 5th index value I gave? Not sure how to make it unambiguous. Maybe: IndexError: dimension 0 index out of bounds: got 5, size is 3 ? How about: IndexError: 5 is out of bounds for dimension 0: must be in [-3, 3). to be maximally explicit about what values are allowed, and avoid the index confusion. Or perhaps axis instead of dimension, since this is how they are referred to in most numpy argument lists. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] varargs for logical_or, etc
I think it's unfortunate that functions like logical_or are limited to binary. As a workaround, I've been using this: def apply_binary (func, *args): if len (args) == 1: return args[0] elif len (args) == 2: return func (*args) else: return func ( apply_binary (func, *args[:len(args)/2]), apply_binary (func, *args[(len(args))/2:])) Then for example: punc2 = np.logical_and (u % 5 == 4, apply_binary (np.logical_or, u/5 == 3, u/5 == 8, u/5 == 13)) ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] lazy evaluation
On Tue, Jun 5, 2012 at 12:55 PM, mark florisson markflorisso...@gmail.com wrote: It would be great if we implement the NEP listed above, but with a few extensions. I think Numpy should handle the lazy evaluation part, and determine when expressions should be evaluated, etc. However, for each user operation, Numpy will call back a user-installed hook implementing some interface, to allow various packages to provide their own hooks to evaluate vector operations however they want. This will include packages such as Theano, which could run things on the GPU, Numexpr, and in the future https://github.com/markflorisson88/minivect (which will likely have an LLVM backend in the future, and possibly integrated with Numba to allow inlining of numba ufuncs). The project above tries to bring together all the different array expression compilers together in a single framework, to provide efficient array expressions specialized for any data layout (nditer on steroids if you will, with SIMD, threaded and inlining capabilities). A global hook sounds ugly and hard to control -- it's hard to tell which operations should be deferred and which should be forced, etc. While it would be less magical, I think a more explicit API would in the end be easier to use... something like a, b, c, d = deferred([a, b, c, d]) e = a + b * c # 'e' is a deferred object too f = np.dot(e, d) # so is 'f' g = force(f) # 'g' is an ndarray # or force(f, out=g) But at that point, this could easily be an external library, right? All we'd need from numpy would be some way for external types to override the evaluation of ufuncs, np.dot, etc.? We've recently seen several reasons to want that functionality, and it seems like developing these improved numexpr ideas would be much easier if they didn't require doing deep surgery to numpy itself... -N ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 1D array sorting ascending and descending by fields
On Mon, Jun 4, 2012 at 6:08 PM, Chris Barker chris.bar...@noaa.gov wrote: could you multiply the numeric field by -1, sort, then put it back Yeah, that works great for my situation. Thanks Chris! On Mon, Jun 4, 2012 at 8:17 PM, Benjamin Root ben.r...@ou.edu wrote: While that may work for this users case, that would not work for all dtypes. Some, such as timedelta, datetime and strings would not be able to be multiplied by a number. This is the reason why I thought there might be such a feature. Would be an interesting feature to add, but I am not certain if the negative sign notation would be best. Is it possible for a named field to start with a negative sign? I'm not sure about what is allowable in names, but I would be interested in getting involved with the NumPy project by helping to add this feature. I'll check out the contributing doc. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] varargs for logical_or, etc
On Tue, Jun 5, 2012 at 2:54 PM, Neal Becker ndbeck...@gmail.com wrote: I think it's unfortunate that functions like logical_or are limited to binary. As a workaround, I've been using this: def apply_binary (func, *args): if len (args) == 1: return args[0] elif len (args) == 2: return func (*args) else: return func ( apply_binary (func, *args[:len(args)/2]), apply_binary (func, *args[(len(args))/2:])) Then for example: punc2 = np.logical_and (u % 5 == 4, apply_binary (np.logical_or, u/5 == 3, u/5 == 8, u/5 == 13)) reduce(np.logical_and, args) -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] nditer_buffer_flag branch (was: Add data memory allocation tracing facilities. (#284))
On Tue, Jun 5, 2012 at 5:40 AM, Nathaniel Smith n...@pobox.com wrote: On Tue, Jun 5, 2012 at 11:06 AM, Thouis (Ray) Jones wrote: All of the failing tests seem to have been caused by the buffer copy bug, fixed in https://github.com/mwiebe/numpy/tree/nditer_buffer_flag(but not yet pulled into numpy). I also have a version that implements tracing, with pure C in the allocation functions writing to a dynamically allocated buffer, which must then be fetched proactively by Python. However, I think this version is a little nicer to use from the Python perspective. --- Reply to this email directly or view it on GitHub: https://github.com/numpy/numpy/pull/284#issuecomment-6121817 Speaking of which, Mark - what's the status of that nditer_buffer_flag branch? Should there be a pull request? Thanks for the nudge, I've made a PR. -Mark -N ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] lazy evaluation
On 5 June 2012 14:58, Nathaniel Smith n...@pobox.com wrote: On Tue, Jun 5, 2012 at 12:55 PM, mark florisson markflorisso...@gmail.com wrote: It would be great if we implement the NEP listed above, but with a few extensions. I think Numpy should handle the lazy evaluation part, and determine when expressions should be evaluated, etc. However, for each user operation, Numpy will call back a user-installed hook implementing some interface, to allow various packages to provide their own hooks to evaluate vector operations however they want. This will include packages such as Theano, which could run things on the GPU, Numexpr, and in the future https://github.com/markflorisson88/minivect (which will likely have an LLVM backend in the future, and possibly integrated with Numba to allow inlining of numba ufuncs). The project above tries to bring together all the different array expression compilers together in a single framework, to provide efficient array expressions specialized for any data layout (nditer on steroids if you will, with SIMD, threaded and inlining capabilities). A global hook sounds ugly and hard to control -- it's hard to tell which operations should be deferred and which should be forced, etc. Yes, but for the user the difference should not be visible (unless operations can raise exceptions, in which case you choose the safe path, or let the user configure what to do). While it would be less magical, I think a more explicit API would in the end be easier to use... something like a, b, c, d = deferred([a, b, c, d]) e = a + b * c # 'e' is a deferred object too f = np.dot(e, d) # so is 'f' g = force(f) # 'g' is an ndarray # or force(f, out=g) But at that point, this could easily be an external library, right? All we'd need from numpy would be some way for external types to override the evaluation of ufuncs, np.dot, etc.? We've recently seen several reasons to want that functionality, and it seems like developing these improved numexpr ideas would be much easier if they didn't require doing deep surgery to numpy itself... Definitely, but besides monkey-patch-chaining I think some modifications would be required, but they would be reasonably simple. Most of the functionality would be handled in one function, which most ufuncs (the ones you care about, as well as ufunc (methods) like add) call. E.g. if ((result = NPy_LazyEval(add, op1, op2)) return result; , which is inserted after argument unpacking and sanity checking. You could also do a per-module hook, and have the function look at sys._getframe(1).f_globals, but that is fragile and won't work from C or Cython code. How did you have overrides in mind? I also found this thread: http://mail.scipy.org/pipermail/numpy-discussion/2011-June/056945.html , but I think you want more than just to override ufuncs, you want numpy to govern when stuff is allowed to be lazy and when stuff should be evaluated (e.g. when it is indexed, slice assigned (although that itself may also be lazy), etc). You don't want some funny object back that doesn't work with things which are not overridden in numpy. -N ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] commit rights for Nathaniel
On Sun, Jun 3, 2012 at 12:04 PM, Ralf Gommers ralf.gomm...@googlemail.comwrote: On Sun, Jun 3, 2012 at 6:43 PM, Charles R Harris charlesr.har...@gmail.com wrote: Hi All, Numpy is approaching a time of transition. Ralf will be concentrating his efforts on Scipy I'll write a separate post on that asap. and I will be cutting back on my work on Numpy. I sincerely hope you don't cut back on your work too much Charles. You have done an excellent job as chief maintainer over the last years. The 1.7 release looks to be delayed and I suspect that the Continuum Analytics folks will become increasingly dedicated to the big data push. We need new people to carry things forward and I think Nathaniel can pick up part of the load. Assuming he wants them, I am definitely +1 on giving Nathaniel commit rights. His recent patches and debugging of issues were of high quality and very helpful. OK, I went ahead and added him whether he wants it or not ;) Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changes in PyArray_FromAny between 1.5.x and 1.6.x
On Mon, Jun 4, 2012 at 10:12 PM, Dag Sverre Seljebotn
d.s.seljeb...@astro.uio.no wrote:
On 06/04/2012 09:06 PM, Mike Hansen wrote:
On Mon, May 28, 2012 at 3:15 AM, Mike Hansenmhan...@gmail.com wrote:
In trying to upgrade NumPy within Sage, we notices some differences in
behavior between 1.5 and 1.6. In particular, in 1.5, we have
sage: f = 0.5
sage: f.__array_interface__
{'typestr': '=f8'}
sage: numpy.array(f)
array(0.5)
sage: numpy.array(float(f))
array(0.5)
In 1.6, we get the following,
sage: f = 0.5
sage: f.__array_interface__
{'typestr': '=f8'}
sage: numpy.array(f)
array(0.500, dtype=object)
This seems to be do to the changes in PyArray_FromAny introduced in
https://github.com/mwhansen/numpy/commit/2635398db3f26529ce2aaea4028a8118844f3c48
. In particular, _array_find_type used to be used to query our
__array_interface__ attribute, and it no longer seems to work. Is
there a way to get the old behavior with the current code?
No idea. If you want to spend the time to fix this properly, you could
implement PEP 3118 and use that instead to export your array data (which
can be done from Cython using __getbuffer__ on a Cython class).
I don't think that would work, because looking more closely, I don't
think they're actually doing anything like what
__array_interface__/PEP3118 are designed for. They just have some
custom class (sage.rings.real_mpfr.RealLiteral, I guess an arbitrary
precision floating point of some sort?), and they want instances that
are passed to np.array() to be automatically coerced to another type
(float64) by default. But there's no buffer sharing or anything like
that going on at all. Mike, does that sound right?
This automagic coercion seems... in very dubious taste to me. (Why
does creating an array object imply that you want to throw away
precision? You can already throw away precision explicitly by doing
np.array(f, dtype=float).) But if this automatic coercion feature is
useful, then wouldn't it be better to have a different interface
instead of kluging it into __array_interface__, like we should check
for an attribute called __numpy_preferred_dtype__ or something?
-n
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] commit rights for Nathaniel
On Tue, Jun 5, 2012 at 4:19 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Sun, Jun 3, 2012 at 12:04 PM, Ralf Gommers ralf.gomm...@googlemail.com wrote: On Sun, Jun 3, 2012 at 6:43 PM, Charles R Harris charlesr.har...@gmail.com wrote: Hi All, Numpy is approaching a time of transition. Ralf will be concentrating his efforts on Scipy I'll write a separate post on that asap. and I will be cutting back on my work on Numpy. I sincerely hope you don't cut back on your work too much Charles. You have done an excellent job as chief maintainer over the last years. The 1.7 release looks to be delayed and I suspect that the Continuum Analytics folks will become increasingly dedicated to the big data push. We need new people to carry things forward and I think Nathaniel can pick up part of the load. Assuming he wants them, I am definitely +1 on giving Nathaniel commit rights. His recent patches and debugging of issues were of high quality and very helpful. OK, I went ahead and added him whether he wants it or not ;) Hah. Thanks! Is there a committers guide anywhere? By default I would assume that the rules are pretty much -- continue sending pull requests for my own changes (unless a trivial typo fix in a comment or something), go ahead and merge anyone else's pull request where things seem okay and my best judgement is we have consensus, fix things if my judgement was wrong? But I don't want to step on any toes... -n ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] lazy evaluation
On Tue, Jun 5, 2012 at 4:12 PM, mark florisson markflorisso...@gmail.com wrote: On 5 June 2012 14:58, Nathaniel Smith n...@pobox.com wrote: On Tue, Jun 5, 2012 at 12:55 PM, mark florisson markflorisso...@gmail.com wrote: It would be great if we implement the NEP listed above, but with a few extensions. I think Numpy should handle the lazy evaluation part, and determine when expressions should be evaluated, etc. However, for each user operation, Numpy will call back a user-installed hook implementing some interface, to allow various packages to provide their own hooks to evaluate vector operations however they want. This will include packages such as Theano, which could run things on the GPU, Numexpr, and in the future https://github.com/markflorisson88/minivect (which will likely have an LLVM backend in the future, and possibly integrated with Numba to allow inlining of numba ufuncs). The project above tries to bring together all the different array expression compilers together in a single framework, to provide efficient array expressions specialized for any data layout (nditer on steroids if you will, with SIMD, threaded and inlining capabilities). A global hook sounds ugly and hard to control -- it's hard to tell which operations should be deferred and which should be forced, etc. Yes, but for the user the difference should not be visible (unless operations can raise exceptions, in which case you choose the safe path, or let the user configure what to do). While it would be less magical, I think a more explicit API would in the end be easier to use... something like a, b, c, d = deferred([a, b, c, d]) e = a + b * c # 'e' is a deferred object too f = np.dot(e, d) # so is 'f' g = force(f) # 'g' is an ndarray # or force(f, out=g) But at that point, this could easily be an external library, right? All we'd need from numpy would be some way for external types to override the evaluation of ufuncs, np.dot, etc.? We've recently seen several reasons to want that functionality, and it seems like developing these improved numexpr ideas would be much easier if they didn't require doing deep surgery to numpy itself... Definitely, but besides monkey-patch-chaining I think some modifications would be required, but they would be reasonably simple. Most of the functionality would be handled in one function, which most ufuncs (the ones you care about, as well as ufunc (methods) like add) call. E.g. if ((result = NPy_LazyEval(add, op1, op2)) return result; , which is inserted after argument unpacking and sanity checking. You could also do a per-module hook, and have the function look at sys._getframe(1).f_globals, but that is fragile and won't work from C or Cython code. How did you have overrides in mind? My vague idea is that core numpy operations are about as fundamental for scientific users as the Python builtin operations are, so they should probably be overrideable in a similar way. So we'd teach numpy functions to check for methods named like __numpy_ufunc__ or __numpy_dot__ and let themselves be overridden if found. Like how __gt__ and __add__ and stuff work. Or something along those lines. I also found this thread: http://mail.scipy.org/pipermail/numpy-discussion/2011-June/056945.html , but I think you want more than just to override ufuncs, you want numpy to govern when stuff is allowed to be lazy and when stuff should be evaluated (e.g. when it is indexed, slice assigned (although that itself may also be lazy), etc). You don't want some funny object back that doesn't work with things which are not overridden in numpy. My point is that probably numpy should *not* govern the decision about what stuff should be lazy and what should be evaluated; that should be governed by some combination of the user and Numba/Theano/minivect/whatever. The toy API I sketched out would make those decisions obvious and explicit. (And if the funny objects had an __array_interface__ attribute that automatically forced evaluation when accessed, then they'd work fine with code that was expecting an array, or if they were assigned to a real ndarray, etc.) -n ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 1D array sorting ascending and descending by fields
On Tue, Jun 5, 2012 at 10:49 AM, Nathaniel Smith n...@pobox.com wrote: On Tue, Jun 5, 2012 at 1:17 AM, Benjamin Root ben.r...@ou.edu wrote: On Monday, June 4, 2012, Chris Barker wrote: On Mon, Jun 4, 2012 at 11:10 AM, Patrick Redmond plredm...@gmail.com wrote: Here's how I sorted primarily by field 'a' descending and secondarily by field 'b' ascending: could you multiply the numeric field by -1, sort, then put it back -- somethign like: data *- -1 data_sorted = np.sort(data, order=['a','b']) data_sorted *= -1 (reverse if necessary -- I lost track...) -Chris While that may work for this users case, that would not work for all dtypes. Some, such as timedelta, datetime and strings would not be able to be multiplied by a number. Would be an interesting feature to add, but I am not certain if the negative sign notation would be best. Is it possible for a named field to start with a negative sign? Maybe add a reverse= argument (named after the corresponding argument to list.sort and __builtins__.sorted). # sorts in descending order, no fields required np.sort([10, 20, 0], reverse=True) # sorts in descending order np.sort(rec_array, order=(a, b), reverse=True) # ascending by a then descending by b np.sort(rec_array, order=(a, b), reverse=(False, True)) ? -n Clear, unambiguous, and works with the existing framework. +1 Ben Root ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] lazy evaluation
Would lazy eval be able to eliminate temps in doing operations such as: np.sum (u != 23)? That is, now ops involving selecting elements of matrixes are often performed by first constructing temp matrixes, and the operating on them. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changes in PyArray_FromAny between 1.5.x and 1.6.x
On Tue, Jun 5, 2012 at 8:34 AM, Nathaniel Smith n...@pobox.com wrote: I don't think that would work, because looking more closely, I don't think they're actually doing anything like what __array_interface__/PEP3118 are designed for. They just have some custom class (sage.rings.real_mpfr.RealLiteral, I guess an arbitrary precision floating point of some sort?), and they want instances that are passed to np.array() to be automatically coerced to another type (float64) by default. But there's no buffer sharing or anything like that going on at all. Mike, does that sound right? Yes, there's no buffer sharing going on at all. This automagic coercion seems... in very dubious taste to me. (Why does creating an array object imply that you want to throw away precision? The __array_interface__ attribute is a property which depends on the precision of the ring. If it floats have enough precision, you just get floats; otherwise you get objects. You can already throw away precision explicitly by doing np.array(f, dtype=float).) But if this automatic coercion feature is useful, then wouldn't it be better to have a different interface instead of kluging it into __array_interface__, like we should check for an attribute called __numpy_preferred_dtype__ or something? It isn't just the array() calls which end up getting problems. For example, in 1.5.x sage: f = 10; type(f) type 'sage.rings.integer.Integer' sage: numpy.arange(f) array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) #int64 while in 1.6.x sage: numpy.arange(f) array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=object) We also see problems with calls like sage: scipy.stats.uniform(0,15).ppf([0.5,0.7]) array([ 7.5, 10.5]) which work in 1.5.x, but fail with a traceback TypeError: array cannot be safely cast to required type in 1.6.x. --Mike ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] commit rights for Nathaniel
On Tue, Jun 5, 2012 at 10:25 AM, Nathaniel Smith n...@pobox.com wrote: On Tue, Jun 5, 2012 at 4:19 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Sun, Jun 3, 2012 at 12:04 PM, Ralf Gommers ralf.gomm...@googlemail.com wrote: On Sun, Jun 3, 2012 at 6:43 PM, Charles R Harris charlesr.har...@gmail.com wrote: Hi All, Numpy is approaching a time of transition. Ralf will be concentrating his efforts on Scipy I'll write a separate post on that asap. and I will be cutting back on my work on Numpy. I sincerely hope you don't cut back on your work too much Charles. You have done an excellent job as chief maintainer over the last years. The 1.7 release looks to be delayed and I suspect that the Continuum Analytics folks will become increasingly dedicated to the big data push. We need new people to carry things forward and I think Nathaniel can pick up part of the load. Assuming he wants them, I am definitely +1 on giving Nathaniel commit rights. His recent patches and debugging of issues were of high quality and very helpful. OK, I went ahead and added him whether he wants it or not ;) Hah. Thanks! Is there a committers guide anywhere? By default I would assume that the rules are pretty much -- continue sending pull requests for my own changes (unless a trivial typo fix in a comment or something), go ahead and merge anyone else's pull request where things seem okay and my best judgement is we have consensus, fix things if my judgement was wrong? But I don't want to step on any toes... You can commit your own stuff also if someone signs off on it or it seems uncontroversial and has sat there for a while. It's mostly a judgement call. For the commits themselves, the github button doesn't do fast forward or whitespace cleanup, so I have the following alias in .git/config getpatch = !sh -c 'git co -b pull-$1 master \ curl https://github.com/numpy/nump/pull/$1.patch|\ git am -3 --whitespace=strip' - which opens a new branch pull-nnn and is useful for the bigger commits so they can be tested and then merged with master before pushing. The non-trivial commits should be tested with at least Python 2.4, 2.7, and 3.2. I also suggest running the one-file build for changes in core since most developers do the separate file thing and sometimes fail to catch single file build problems. Keep an eye on coding style, otherwise it will drift. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] lazy evaluation
On 5 June 2012 18:21, Neal Becker ndbeck...@gmail.com wrote: Would lazy eval be able to eliminate temps in doing operations such as: np.sum (u != 23)? That is, now ops involving selecting elements of matrixes are often performed by first constructing temp matrixes, and the operating on them. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion Sure, yeah, it's pretty easy to generate a loop with an if statement and a reduction. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] commit rights for Nathaniel
On Tue, Jun 5, 2012 at 11:52 AM, Charles R Harris charlesr.har...@gmail.com wrote: On Tue, Jun 5, 2012 at 10:25 AM, Nathaniel Smith n...@pobox.com wrote: On Tue, Jun 5, 2012 at 4:19 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Sun, Jun 3, 2012 at 12:04 PM, Ralf Gommers ralf.gomm...@googlemail.com wrote: On Sun, Jun 3, 2012 at 6:43 PM, Charles R Harris charlesr.har...@gmail.com wrote: Hi All, Numpy is approaching a time of transition. Ralf will be concentrating his efforts on Scipy I'll write a separate post on that asap. and I will be cutting back on my work on Numpy. I sincerely hope you don't cut back on your work too much Charles. You have done an excellent job as chief maintainer over the last years. The 1.7 release looks to be delayed and I suspect that the Continuum Analytics folks will become increasingly dedicated to the big data push. We need new people to carry things forward and I think Nathaniel can pick up part of the load. Assuming he wants them, I am definitely +1 on giving Nathaniel commit rights. His recent patches and debugging of issues were of high quality and very helpful. OK, I went ahead and added him whether he wants it or not ;) Hah. Thanks! Is there a committers guide anywhere? By default I would assume that the rules are pretty much -- continue sending pull requests for my own changes (unless a trivial typo fix in a comment or something), go ahead and merge anyone else's pull request where things seem okay and my best judgement is we have consensus, fix things if my judgement was wrong? But I don't want to step on any toes... You can commit your own stuff also if someone signs off on it or it seems uncontroversial and has sat there for a while. It's mostly a judgement call. For the commits themselves, the github button doesn't do fast forward or whitespace cleanup, so I have the following alias in .git/config getpatch = !sh -c 'git co -b pull-$1 master \ curl https://github.com/numpy/nump/pull/$1.patch|\ git am -3 --whitespace=strip' - which opens a new branch pull-nnn and is useful for the bigger commits so they can be tested and then merged with master before pushing. The non-trivial commits should be tested with at least Python 2.4, 2.7, and 3.2. I also suggest running the one-file build for changes in core since most developers do the separate file thing and sometimes fail to catch single file build problems. Keep an eye on coding style, otherwise it will drift. And keep in mind that part of your job is to train new committers and help bring them up to speed. See yourself as a recruiter as well as a reviewer. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] lazy evaluation
On 5 June 2012 17:38, Nathaniel Smith n...@pobox.com wrote: On Tue, Jun 5, 2012 at 4:12 PM, mark florisson markflorisso...@gmail.com wrote: On 5 June 2012 14:58, Nathaniel Smith n...@pobox.com wrote: On Tue, Jun 5, 2012 at 12:55 PM, mark florisson markflorisso...@gmail.com wrote: It would be great if we implement the NEP listed above, but with a few extensions. I think Numpy should handle the lazy evaluation part, and determine when expressions should be evaluated, etc. However, for each user operation, Numpy will call back a user-installed hook implementing some interface, to allow various packages to provide their own hooks to evaluate vector operations however they want. This will include packages such as Theano, which could run things on the GPU, Numexpr, and in the future https://github.com/markflorisson88/minivect (which will likely have an LLVM backend in the future, and possibly integrated with Numba to allow inlining of numba ufuncs). The project above tries to bring together all the different array expression compilers together in a single framework, to provide efficient array expressions specialized for any data layout (nditer on steroids if you will, with SIMD, threaded and inlining capabilities). A global hook sounds ugly and hard to control -- it's hard to tell which operations should be deferred and which should be forced, etc. Yes, but for the user the difference should not be visible (unless operations can raise exceptions, in which case you choose the safe path, or let the user configure what to do). While it would be less magical, I think a more explicit API would in the end be easier to use... something like a, b, c, d = deferred([a, b, c, d]) e = a + b * c # 'e' is a deferred object too f = np.dot(e, d) # so is 'f' g = force(f) # 'g' is an ndarray # or force(f, out=g) But at that point, this could easily be an external library, right? All we'd need from numpy would be some way for external types to override the evaluation of ufuncs, np.dot, etc.? We've recently seen several reasons to want that functionality, and it seems like developing these improved numexpr ideas would be much easier if they didn't require doing deep surgery to numpy itself... Definitely, but besides monkey-patch-chaining I think some modifications would be required, but they would be reasonably simple. Most of the functionality would be handled in one function, which most ufuncs (the ones you care about, as well as ufunc (methods) like add) call. E.g. if ((result = NPy_LazyEval(add, op1, op2)) return result; , which is inserted after argument unpacking and sanity checking. You could also do a per-module hook, and have the function look at sys._getframe(1).f_globals, but that is fragile and won't work from C or Cython code. How did you have overrides in mind? My vague idea is that core numpy operations are about as fundamental for scientific users as the Python builtin operations are, so they should probably be overrideable in a similar way. So we'd teach numpy functions to check for methods named like __numpy_ufunc__ or __numpy_dot__ and let themselves be overridden if found. Like how __gt__ and __add__ and stuff work. Or something along those lines. I also found this thread: http://mail.scipy.org/pipermail/numpy-discussion/2011-June/056945.html , but I think you want more than just to override ufuncs, you want numpy to govern when stuff is allowed to be lazy and when stuff should be evaluated (e.g. when it is indexed, slice assigned (although that itself may also be lazy), etc). You don't want some funny object back that doesn't work with things which are not overridden in numpy. My point is that probably numpy should *not* govern the decision about what stuff should be lazy and what should be evaluated; that should be governed by some combination of the user and Numba/Theano/minivect/whatever. The toy API I sketched out would make those decisions obvious and explicit. (And if the funny objects had an __array_interface__ attribute that automatically forced evaluation when accessed, then they'd work fine with code that was expecting an array, or if they were assigned to a real ndarray, etc.) That's disappointing though, since the performance drawbacks can severely limit the usefulness for people with big data sets. Ideally, you would take your intuitive numpy code, and make it go fast, without jumping through hoops. Numpypy has lazy evaluation, I don't know how good a job it does, but it does mean you can finally get fast numpy code in an intuitive way (and even run it on a GPU if that is possible and beneficial). -n ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org
Re: [Numpy-discussion] Changes in PyArray_FromAny between 1.5.x and 1.6.x
On Tue, Jun 5, 2012 at 11:51 AM, Zachary Pincus zachary.pin...@yale.eduwrote: It isn't just the array() calls which end up getting problems. For example, in 1.5.x sage: f = 10; type(f) type 'sage.rings.integer.Integer' sage: numpy.arange(f) array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) #int64 while in 1.6.x sage: numpy.arange(f) array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=object) We also see problems with calls like sage: scipy.stats.uniform(0,15).ppf([0.5,0.7]) array([ 7.5, 10.5]) which work in 1.5.x, but fail with a traceback TypeError: array cannot be safely cast to required type in 1.6.x. I'm getting problems like this after a 1.6 upgrade as well. Lots of object arrays being created when previously there would either be an error, or an array of floats. Also, lots of the TypeError: array cannot be safely cast to required type are cropping up. Honestly, most of these are in places where my code was lax and so I just cleaned things up to use the right dtypes etc. But still a bit unexpected in terms of having more code to fix than I was used to for 0.X numpy revisions. There is a fine line here. We do need to make people clean up lax code in order to improve numpy, but hopefully we can keep the cleanups reasonable. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changes in PyArray_FromAny between 1.5.x and 1.6.x
There is a fine line here. We do need to make people clean up lax code in order to improve numpy, but hopefully we can keep the cleanups reasonable. Oh agreed. Somehow, though, I was surprised by this, even though I keep tabs on the numpy lists -- at no point did it become clear that big changes in how arrays get constructed and typecast are ahead that may require code fixes. That was my main point, but probably a PEBCAK issue more than anything. Zach ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changes in PyArray_FromAny between 1.5.x and 1.6.x
On Tue, Jun 5, 2012 at 8:41 PM, Zachary Pincus zachary.pin...@yale.eduwrote: There is a fine line here. We do need to make people clean up lax code in order to improve numpy, but hopefully we can keep the cleanups reasonable. Oh agreed. Somehow, though, I was surprised by this, even though I keep tabs on the numpy lists -- at no point did it become clear that big changes in how arrays get constructed and typecast are ahead that may require code fixes. That was my main point, but probably a PEBCAK issue more than anything. It was fairly extensively discussed when introduced, http://thread.gmane.org/gmane.comp.python.numeric.general/44206, and again at some later point. Ralf ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] varargs for logical_or, etc
On Tue, Jun 5, 2012 at 6:59 PM, Benjamin Root ben.r...@ou.edu wrote: On Tue, Jun 5, 2012 at 10:37 AM, Robert Kern robert.k...@gmail.comwrote: On Tue, Jun 5, 2012 at 2:54 PM, Neal Becker ndbeck...@gmail.com wrote: I think it's unfortunate that functions like logical_or are limited to binary. As a workaround, I've been using this: def apply_binary (func, *args): if len (args) == 1: return args[0] elif len (args) == 2: return func (*args) else: return func ( apply_binary (func, *args[:len(args)/2]), apply_binary (func, *args[(len(args))/2:])) Then for example: punc2 = np.logical_and (u % 5 == 4, apply_binary (np.logical_or, u/5 == 3, u/5 == 8, u/5 == 13)) reduce(np.logical_and, args) I would love it if we could add something like that to the doc-string of those functions because I don't think it is immediately obvious. How do we do that for ufuncs? Edit numpy/core/code_generators/ufunc_docstrings.py Ralf ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changes in PyArray_FromAny between 1.5.x and 1.6.x
On Tue, Jun 5, 2012 at 7:47 PM, Ralf Gommers ralf.gomm...@googlemail.com wrote: On Tue, Jun 5, 2012 at 8:41 PM, Zachary Pincus zachary.pin...@yale.edu wrote: There is a fine line here. We do need to make people clean up lax code in order to improve numpy, but hopefully we can keep the cleanups reasonable. Oh agreed. Somehow, though, I was surprised by this, even though I keep tabs on the numpy lists -- at no point did it become clear that big changes in how arrays get constructed and typecast are ahead that may require code fixes. That was my main point, but probably a PEBCAK issue more than anything. It was fairly extensively discussed when introduced, http://thread.gmane.org/gmane.comp.python.numeric.general/44206, and again at some later point. Those are the not-yet-finalized changes in 1.7; Zachary (I think) is talking about problems upgrading from ~1.5 to 1.6. -n ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changes in PyArray_FromAny between 1.5.x and 1.6.x
On Tue, Jun 5, 2012 at 8:41 PM, Zachary Pincus zachary.pin...@yale.edu wrote: There is a fine line here. We do need to make people clean up lax code in order to improve numpy, but hopefully we can keep the cleanups reasonable. Oh agreed. Somehow, though, I was surprised by this, even though I keep tabs on the numpy lists -- at no point did it become clear that big changes in how arrays get constructed and typecast are ahead that may require code fixes. That was my main point, but probably a PEBCAK issue more than anything. It was fairly extensively discussed when introduced, http://thread.gmane.org/gmane.comp.python.numeric.general/44206, and again at some later point. Those are the not-yet-finalized changes in 1.7; Zachary (I think) is talking about problems upgrading from ~1.5 to 1.6. Yes, unless I'm wrong I experienced these problems from 1.5.something to 1.6.1. I didn't take notes as it was in the middle of a deadline-crunch so I just fixed the code and moved on (long, stupid story about why the upgrade before a deadline...). It's just that the issues mentioned above seem to have hit me too and I wanted to mention that. But unhelpfully, I think, without code, and now I've hijacked this thread! Sorry. Zach ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] lazy evaluation
On Tue, Jun 5, 2012 at 7:08 PM, mark florisson markflorisso...@gmail.com wrote: On 5 June 2012 17:38, Nathaniel Smith n...@pobox.com wrote: On Tue, Jun 5, 2012 at 4:12 PM, mark florisson markflorisso...@gmail.com wrote: On 5 June 2012 14:58, Nathaniel Smith n...@pobox.com wrote: On Tue, Jun 5, 2012 at 12:55 PM, mark florisson markflorisso...@gmail.com wrote: It would be great if we implement the NEP listed above, but with a few extensions. I think Numpy should handle the lazy evaluation part, and determine when expressions should be evaluated, etc. However, for each user operation, Numpy will call back a user-installed hook implementing some interface, to allow various packages to provide their own hooks to evaluate vector operations however they want. This will include packages such as Theano, which could run things on the GPU, Numexpr, and in the future https://github.com/markflorisson88/minivect (which will likely have an LLVM backend in the future, and possibly integrated with Numba to allow inlining of numba ufuncs). The project above tries to bring together all the different array expression compilers together in a single framework, to provide efficient array expressions specialized for any data layout (nditer on steroids if you will, with SIMD, threaded and inlining capabilities). A global hook sounds ugly and hard to control -- it's hard to tell which operations should be deferred and which should be forced, etc. Yes, but for the user the difference should not be visible (unless operations can raise exceptions, in which case you choose the safe path, or let the user configure what to do). While it would be less magical, I think a more explicit API would in the end be easier to use... something like a, b, c, d = deferred([a, b, c, d]) e = a + b * c # 'e' is a deferred object too f = np.dot(e, d) # so is 'f' g = force(f) # 'g' is an ndarray # or force(f, out=g) But at that point, this could easily be an external library, right? All we'd need from numpy would be some way for external types to override the evaluation of ufuncs, np.dot, etc.? We've recently seen several reasons to want that functionality, and it seems like developing these improved numexpr ideas would be much easier if they didn't require doing deep surgery to numpy itself... Definitely, but besides monkey-patch-chaining I think some modifications would be required, but they would be reasonably simple. Most of the functionality would be handled in one function, which most ufuncs (the ones you care about, as well as ufunc (methods) like add) call. E.g. if ((result = NPy_LazyEval(add, op1, op2)) return result; , which is inserted after argument unpacking and sanity checking. You could also do a per-module hook, and have the function look at sys._getframe(1).f_globals, but that is fragile and won't work from C or Cython code. How did you have overrides in mind? My vague idea is that core numpy operations are about as fundamental for scientific users as the Python builtin operations are, so they should probably be overrideable in a similar way. So we'd teach numpy functions to check for methods named like __numpy_ufunc__ or __numpy_dot__ and let themselves be overridden if found. Like how __gt__ and __add__ and stuff work. Or something along those lines. I also found this thread: http://mail.scipy.org/pipermail/numpy-discussion/2011-June/056945.html , but I think you want more than just to override ufuncs, you want numpy to govern when stuff is allowed to be lazy and when stuff should be evaluated (e.g. when it is indexed, slice assigned (although that itself may also be lazy), etc). You don't want some funny object back that doesn't work with things which are not overridden in numpy. My point is that probably numpy should *not* govern the decision about what stuff should be lazy and what should be evaluated; that should be governed by some combination of the user and Numba/Theano/minivect/whatever. The toy API I sketched out would make those decisions obvious and explicit. (And if the funny objects had an __array_interface__ attribute that automatically forced evaluation when accessed, then they'd work fine with code that was expecting an array, or if they were assigned to a real ndarray, etc.) That's disappointing though, since the performance drawbacks can severely limit the usefulness for people with big data sets. Ideally, you would take your intuitive numpy code, and make it go fast, without jumping through hoops. Numpypy has lazy evaluation, I don't know how good a job it does, but it does mean you can finally get fast numpy code in an intuitive way (and even run it on a GPU if that is possible and beneficial). All of these proposals require the user to jump through hoops -- the deferred-ufunc NEP has the extra 'with deferredstate' thing, and more importantly, a set of rules that people have to learn and keep in mind for which numpy
Re: [Numpy-discussion] lazy evaluation
On 5 June 2012 20:17, Nathaniel Smith n...@pobox.com wrote: On Tue, Jun 5, 2012 at 7:08 PM, mark florisson markflorisso...@gmail.com wrote: On 5 June 2012 17:38, Nathaniel Smith n...@pobox.com wrote: On Tue, Jun 5, 2012 at 4:12 PM, mark florisson markflorisso...@gmail.com wrote: On 5 June 2012 14:58, Nathaniel Smith n...@pobox.com wrote: On Tue, Jun 5, 2012 at 12:55 PM, mark florisson markflorisso...@gmail.com wrote: It would be great if we implement the NEP listed above, but with a few extensions. I think Numpy should handle the lazy evaluation part, and determine when expressions should be evaluated, etc. However, for each user operation, Numpy will call back a user-installed hook implementing some interface, to allow various packages to provide their own hooks to evaluate vector operations however they want. This will include packages such as Theano, which could run things on the GPU, Numexpr, and in the future https://github.com/markflorisson88/minivect (which will likely have an LLVM backend in the future, and possibly integrated with Numba to allow inlining of numba ufuncs). The project above tries to bring together all the different array expression compilers together in a single framework, to provide efficient array expressions specialized for any data layout (nditer on steroids if you will, with SIMD, threaded and inlining capabilities). A global hook sounds ugly and hard to control -- it's hard to tell which operations should be deferred and which should be forced, etc. Yes, but for the user the difference should not be visible (unless operations can raise exceptions, in which case you choose the safe path, or let the user configure what to do). While it would be less magical, I think a more explicit API would in the end be easier to use... something like a, b, c, d = deferred([a, b, c, d]) e = a + b * c # 'e' is a deferred object too f = np.dot(e, d) # so is 'f' g = force(f) # 'g' is an ndarray # or force(f, out=g) But at that point, this could easily be an external library, right? All we'd need from numpy would be some way for external types to override the evaluation of ufuncs, np.dot, etc.? We've recently seen several reasons to want that functionality, and it seems like developing these improved numexpr ideas would be much easier if they didn't require doing deep surgery to numpy itself... Definitely, but besides monkey-patch-chaining I think some modifications would be required, but they would be reasonably simple. Most of the functionality would be handled in one function, which most ufuncs (the ones you care about, as well as ufunc (methods) like add) call. E.g. if ((result = NPy_LazyEval(add, op1, op2)) return result; , which is inserted after argument unpacking and sanity checking. You could also do a per-module hook, and have the function look at sys._getframe(1).f_globals, but that is fragile and won't work from C or Cython code. How did you have overrides in mind? My vague idea is that core numpy operations are about as fundamental for scientific users as the Python builtin operations are, so they should probably be overrideable in a similar way. So we'd teach numpy functions to check for methods named like __numpy_ufunc__ or __numpy_dot__ and let themselves be overridden if found. Like how __gt__ and __add__ and stuff work. Or something along those lines. I also found this thread: http://mail.scipy.org/pipermail/numpy-discussion/2011-June/056945.html , but I think you want more than just to override ufuncs, you want numpy to govern when stuff is allowed to be lazy and when stuff should be evaluated (e.g. when it is indexed, slice assigned (although that itself may also be lazy), etc). You don't want some funny object back that doesn't work with things which are not overridden in numpy. My point is that probably numpy should *not* govern the decision about what stuff should be lazy and what should be evaluated; that should be governed by some combination of the user and Numba/Theano/minivect/whatever. The toy API I sketched out would make those decisions obvious and explicit. (And if the funny objects had an __array_interface__ attribute that automatically forced evaluation when accessed, then they'd work fine with code that was expecting an array, or if they were assigned to a real ndarray, etc.) That's disappointing though, since the performance drawbacks can severely limit the usefulness for people with big data sets. Ideally, you would take your intuitive numpy code, and make it go fast, without jumping through hoops. Numpypy has lazy evaluation, I don't know how good a job it does, but it does mean you can finally get fast numpy code in an intuitive way (and even run it on a GPU if that is possible and beneficial). All of these proposals require the user to jump through hoops -- the deferred-ufunc NEP has the extra 'with deferredstate' thing, and more importantly, a set of
Re: [Numpy-discussion] lazy evaluation
On 06/05/2012 10:47 PM, mark florisson wrote: On 5 June 2012 20:17, Nathaniel Smithn...@pobox.com wrote: On Tue, Jun 5, 2012 at 7:08 PM, mark florisson markflorisso...@gmail.com wrote: On 5 June 2012 17:38, Nathaniel Smithn...@pobox.com wrote: On Tue, Jun 5, 2012 at 4:12 PM, mark florisson markflorisso...@gmail.com wrote: On 5 June 2012 14:58, Nathaniel Smithn...@pobox.com wrote: On Tue, Jun 5, 2012 at 12:55 PM, mark florisson markflorisso...@gmail.com wrote: It would be great if we implement the NEP listed above, but with a few extensions. I think Numpy should handle the lazy evaluation part, and determine when expressions should be evaluated, etc. However, for each user operation, Numpy will call back a user-installed hook implementing some interface, to allow various packages to provide their own hooks to evaluate vector operations however they want. This will include packages such as Theano, which could run things on the GPU, Numexpr, and in the future https://github.com/markflorisson88/minivect (which will likely have an LLVM backend in the future, and possibly integrated with Numba to allow inlining of numba ufuncs). The project above tries to bring together all the different array expression compilers together in a single framework, to provide efficient array expressions specialized for any data layout (nditer on steroids if you will, with SIMD, threaded and inlining capabilities). A global hook sounds ugly and hard to control -- it's hard to tell which operations should be deferred and which should be forced, etc. Yes, but for the user the difference should not be visible (unless operations can raise exceptions, in which case you choose the safe path, or let the user configure what to do). While it would be less magical, I think a more explicit API would in the end be easier to use... something like a, b, c, d = deferred([a, b, c, d]) e = a + b * c # 'e' is a deferred object too f = np.dot(e, d) # so is 'f' g = force(f) # 'g' is an ndarray # or force(f, out=g) But at that point, this could easily be an external library, right? All we'd need from numpy would be some way for external types to override the evaluation of ufuncs, np.dot, etc.? We've recently seen several reasons to want that functionality, and it seems like developing these improved numexpr ideas would be much easier if they didn't require doing deep surgery to numpy itself... Definitely, but besides monkey-patch-chaining I think some modifications would be required, but they would be reasonably simple. Most of the functionality would be handled in one function, which most ufuncs (the ones you care about, as well as ufunc (methods) like add) call. E.g. if ((result = NPy_LazyEval(add, op1, op2)) return result; , which is inserted after argument unpacking and sanity checking. You could also do a per-module hook, and have the function look at sys._getframe(1).f_globals, but that is fragile and won't work from C or Cython code. How did you have overrides in mind? My vague idea is that core numpy operations are about as fundamental for scientific users as the Python builtin operations are, so they should probably be overrideable in a similar way. So we'd teach numpy functions to check for methods named like __numpy_ufunc__ or __numpy_dot__ and let themselves be overridden if found. Like how __gt__ and __add__ and stuff work. Or something along those lines. I also found this thread: http://mail.scipy.org/pipermail/numpy-discussion/2011-June/056945.html , but I think you want more than just to override ufuncs, you want numpy to govern when stuff is allowed to be lazy and when stuff should be evaluated (e.g. when it is indexed, slice assigned (although that itself may also be lazy), etc). You don't want some funny object back that doesn't work with things which are not overridden in numpy. My point is that probably numpy should *not* govern the decision about what stuff should be lazy and what should be evaluated; that should be governed by some combination of the user and Numba/Theano/minivect/whatever. The toy API I sketched out would make those decisions obvious and explicit. (And if the funny objects had an __array_interface__ attribute that automatically forced evaluation when accessed, then they'd work fine with code that was expecting an array, or if they were assigned to a real ndarray, etc.) That's disappointing though, since the performance drawbacks can severely limit the usefulness for people with big data sets. Ideally, you would take your intuitive numpy code, and make it go fast, without jumping through hoops. Numpypy has lazy evaluation, I don't know how good a job it does, but it does mean you can finally get fast numpy code in an intuitive way (and even run it on a GPU if that is possible and beneficial). All of these proposals require the user to jump through hoops -- the deferred-ufunc NEP has the extra 'with
Re: [Numpy-discussion] lazy evaluation
On 5 June 2012 22:36, Dag Sverre Seljebotn d.s.seljeb...@astro.uio.no wrote: On 06/05/2012 10:47 PM, mark florisson wrote: On 5 June 2012 20:17, Nathaniel Smithn...@pobox.com wrote: On Tue, Jun 5, 2012 at 7:08 PM, mark florisson markflorisso...@gmail.com wrote: On 5 June 2012 17:38, Nathaniel Smithn...@pobox.com wrote: On Tue, Jun 5, 2012 at 4:12 PM, mark florisson markflorisso...@gmail.com wrote: On 5 June 2012 14:58, Nathaniel Smithn...@pobox.com wrote: On Tue, Jun 5, 2012 at 12:55 PM, mark florisson markflorisso...@gmail.com wrote: It would be great if we implement the NEP listed above, but with a few extensions. I think Numpy should handle the lazy evaluation part, and determine when expressions should be evaluated, etc. However, for each user operation, Numpy will call back a user-installed hook implementing some interface, to allow various packages to provide their own hooks to evaluate vector operations however they want. This will include packages such as Theano, which could run things on the GPU, Numexpr, and in the future https://github.com/markflorisson88/minivect (which will likely have an LLVM backend in the future, and possibly integrated with Numba to allow inlining of numba ufuncs). The project above tries to bring together all the different array expression compilers together in a single framework, to provide efficient array expressions specialized for any data layout (nditer on steroids if you will, with SIMD, threaded and inlining capabilities). A global hook sounds ugly and hard to control -- it's hard to tell which operations should be deferred and which should be forced, etc. Yes, but for the user the difference should not be visible (unless operations can raise exceptions, in which case you choose the safe path, or let the user configure what to do). While it would be less magical, I think a more explicit API would in the end be easier to use... something like a, b, c, d = deferred([a, b, c, d]) e = a + b * c # 'e' is a deferred object too f = np.dot(e, d) # so is 'f' g = force(f) # 'g' is an ndarray # or force(f, out=g) But at that point, this could easily be an external library, right? All we'd need from numpy would be some way for external types to override the evaluation of ufuncs, np.dot, etc.? We've recently seen several reasons to want that functionality, and it seems like developing these improved numexpr ideas would be much easier if they didn't require doing deep surgery to numpy itself... Definitely, but besides monkey-patch-chaining I think some modifications would be required, but they would be reasonably simple. Most of the functionality would be handled in one function, which most ufuncs (the ones you care about, as well as ufunc (methods) like add) call. E.g. if ((result = NPy_LazyEval(add, op1, op2)) return result; , which is inserted after argument unpacking and sanity checking. You could also do a per-module hook, and have the function look at sys._getframe(1).f_globals, but that is fragile and won't work from C or Cython code. How did you have overrides in mind? My vague idea is that core numpy operations are about as fundamental for scientific users as the Python builtin operations are, so they should probably be overrideable in a similar way. So we'd teach numpy functions to check for methods named like __numpy_ufunc__ or __numpy_dot__ and let themselves be overridden if found. Like how __gt__ and __add__ and stuff work. Or something along those lines. I also found this thread: http://mail.scipy.org/pipermail/numpy-discussion/2011-June/056945.html , but I think you want more than just to override ufuncs, you want numpy to govern when stuff is allowed to be lazy and when stuff should be evaluated (e.g. when it is indexed, slice assigned (although that itself may also be lazy), etc). You don't want some funny object back that doesn't work with things which are not overridden in numpy. My point is that probably numpy should *not* govern the decision about what stuff should be lazy and what should be evaluated; that should be governed by some combination of the user and Numba/Theano/minivect/whatever. The toy API I sketched out would make those decisions obvious and explicit. (And if the funny objects had an __array_interface__ attribute that automatically forced evaluation when accessed, then they'd work fine with code that was expecting an array, or if they were assigned to a real ndarray, etc.) That's disappointing though, since the performance drawbacks can severely limit the usefulness for people with big data sets. Ideally, you would take your intuitive numpy code, and make it go fast, without jumping through hoops. Numpypy has lazy evaluation, I don't know how good a job it does, but it does mean you can finally get fast numpy code in an intuitive way (and even run it on a GPU if that is possible and beneficial). All of these proposals require
Re: [Numpy-discussion] commit rights for Nathaniel
On Tue, Jun 5, 2012 at 4:59 PM, Fernando Perez fperez@gmail.com wrote: A couple of notes from the IPython workflow in case it's of use to you guys: On Tue, Jun 5, 2012 at 10:52 AM, Charles R Harris charlesr.har...@gmail.com wrote: For the commits themselves, the github button doesn't do fast forward or whitespace cleanup, so I have the following alias in .git/config getpatch = !sh -c 'git co -b pull-$1 master \ curl https://github.com/numpy/nump/pull/$1.patch|\ git am -3 --whitespace=strip' - which opens a new branch pull-nnn and is useful for the bigger commits so they can be tested and then merged with master before pushing. The non-trivial commits should be tested with at least Python 2.4, 2.7, and 3.2. I also suggest running the one-file build for changes in core since most developers do the separate file thing and sometimes fail to catch single file build problems. 1) We've settled on using the green button rather than something like the above, because we decided that having the no-ff was actually a *good* thing (and yes, this reverses my initial opinion on the matter). The reasoning that convinced me was that the merge commit in itself is signal, not noise: - it indicates who did the final reviewing and merging (which doesn't happen in a ff merge b/c there's no separate merge commit) - it serves as a good place to cleanly summarize the PR itself, which could possibly contain many commits. It's the job and responsibility of the person doing the merge to understand the PR enough to explain it succinctly, so that one can read just that message and get a realistic idea of what the say 100 commits that went in were meant to do. These merge commits are the right thing to read when building release notes, instead of having to slog through the individual commits. - this way, the DAG's topology immediately shows what went in with review and what was committed without review (hopefully only small/trivial/emergency fixes). - even if the PR has a single commit, it's still OK to do this, as it marks the reviewer (and credits the reviewer as well, which is actual work). For all these reasons, I'm very happy that we reversed our policy and now *only* use the green button to merge, and *never* do a FF merge. We only commit directly to master in the case of absolutely trivial typo fixes or emergency 'my god master is borked' scenarios. 2) I'd encourage you to steal/improve our 'test_pr / post_pr_test' as well as git-mrb tools: https://github.com/ipython/ipython/blob/master/tools/test_pr.py https://github.com/ipython/ipython/blob/master/tools/post_pr_test.py https://github.com/ipython/ipython/blob/master/tools/git-mrb In particular test_pr is a *huge* help. We now almost never merge something that doesn't have a test_pr report. Here's an example where test_pr revealed initially problems, later fixed: https://github.com/ipython/ipython/pull/1847 Once the fix was confirmed, it was easy to merge. It routinely catches python3 errors we put in because most of the core devs don't use python3 regularly. But now I'm not worried about it anymore, as I know the problems will be caught before merging (I used to feel guilty for constantly breaking py3 and having poor Thomas Kluyver have to clean up my messes). There are other advantages to pulling down the patch. Fixups can be merged together, commit comments enhanced, whitespace removed, style cleanups can be added, tests can be run, and the PR is automatically rebased. I still like fast forward for single commit merges, for larger merges I specify no-ff so that things come in as a well defined chunk. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] commit rights for Nathaniel
On Tue, Jun 5, 2012 at 4:15 PM, Charles R Harris charlesr.har...@gmail.com wrote: There are other advantages to pulling down the patch. Fixups can be merged together, commit comments enhanced, whitespace removed, style cleanups can be added, tests can be run, and the PR is automatically rebased. I still like fast forward for single commit merges, for larger merges I specify no-ff so that things come in as a well defined chunk. Sure, that's a decision each project can take as it prefers: we've taken the approach that the person doing the merge does *not* massage the history as presented in the PR; instead we have submitters fix things up when deemed necessary (and we help them out a bit with git-fu if needed). And for single commit merges, we use the merge commit as topological evidence that there was review, which is very useful when looking retrospectively at the project. But each project must find how it best wants to proceed, I'm only offering our perspective in case any of it is useful for numpy. You guys will cherrypick the pieces that merge cleanly for numpy ;) Cheers, f ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] numpy.clip behavior at max and min of dtypes
Can the following function be written using numpy.clip? In some other way? Does numpy.clip satisfy condition 4 below? Does numpy.clip satisfy some closely related condition? Define a function clipcast: output = clipcast(arr, dtype=None, out=None) 1. All arrays have int or float dtypes. 2. Exactly one of the keyword arguments dtype and out must be used. If dtype is given, then output has that dtype. 3. output has the same shape as arr. 4. Let ER be the set of all the real numbers that can be exactly represented by the output dtype. ER is finite and bounded. Let themin = min(ER) and themax = max(ER). For any real number x, define a function f(x) by If x is in ER, define f(x) = x. If x is between two consecutive numbers, u and v, in ER, then define f(x) = u or f(x) = v. Probably the choice would be made using a C cast. If x themin, define f(x) = themin. If x themax, define f(x) = themax. If x is an element of arr, say Arr[I], then output[I] == f(x) where I is any index that defines a single element of arr. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changes in PyArray_FromAny between 1.5.x and 1.6.x
During the original discussion, Gael pointed out that the changes would probably break some code (which might need to be cleaned up but still). I think it was underestimated how quickly people would upgrade and see the changes and therefore be able to report problems. We are talking about a 1.7 release, but there are still people who have not upgraded their code to use 1.6 (when some of the big changes occurred). This should probably guide our view of how long it takes to migrate behavior in NumPy and minimize migration difficulties for users. -Travis On Jun 5, 2012, at 2:01 PM, Zachary Pincus wrote: On Tue, Jun 5, 2012 at 8:41 PM, Zachary Pincus zachary.pin...@yale.edu wrote: There is a fine line here. We do need to make people clean up lax code in order to improve numpy, but hopefully we can keep the cleanups reasonable. Oh agreed. Somehow, though, I was surprised by this, even though I keep tabs on the numpy lists -- at no point did it become clear that big changes in how arrays get constructed and typecast are ahead that may require code fixes. That was my main point, but probably a PEBCAK issue more than anything. It was fairly extensively discussed when introduced, http://thread.gmane.org/gmane.comp.python.numeric.general/44206, and again at some later point. Those are the not-yet-finalized changes in 1.7; Zachary (I think) is talking about problems upgrading from ~1.5 to 1.6. Yes, unless I'm wrong I experienced these problems from 1.5.something to 1.6.1. I didn't take notes as it was in the middle of a deadline-crunch so I just fixed the code and moved on (long, stupid story about why the upgrade before a deadline...). It's just that the issues mentioned above seem to have hit me too and I wanted to mention that. But unhelpfully, I think, without code, and now I've hijacked this thread! Sorry. Zach ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changes in PyArray_FromAny between 1.5.x and 1.6.x
I don't think that would work, because looking more closely, I don't think they're actually doing anything like what __array_interface__/PEP3118 are designed for. They just have some custom class (sage.rings.real_mpfr.RealLiteral, I guess an arbitrary precision floating point of some sort?), and they want instances that are passed to np.array() to be automatically coerced to another type (float64) by default. But there's no buffer sharing or anything like that going on at all. Mike, does that sound right? This automagic coercion seems... in very dubious taste to me. (Why does creating an array object imply that you want to throw away precision? You can already throw away precision explicitly by doing np.array(f, dtype=float).) But if this automatic coercion feature is useful, then wouldn't it be better to have a different interface instead of kluging it into __array_interface__, like we should check for an attribute called __numpy_preferred_dtype__ or something? Interesting. It does look like off-label use of the __array_interface__ attribute. Given that array used to query the __array_interface__ attribute for type discovery, I still wonder why it was disabled in 1.6? -Travis ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
