On 5 June 2012 14:58, Nathaniel Smith <n...@pobox.com> wrote:
> On Tue, Jun 5, 2012 at 12:55 PM, mark florisson
> <markflorisso...@gmail.com> wrote:
>> It would be great if we implement the NEP listed above, but with a few
>> extensions. I think Numpy should handle the lazy evaluation part, and
>> determine when expressions should be evaluated, etc. However, for each
>> user operation, Numpy will call back a user-installed hook
>> implementing some interface, to allow various packages to provide
>> their own hooks to evaluate vector operations however they want. This
>> will include packages such as Theano, which could run things on the
>> GPU, Numexpr, and in the future
>> https://github.com/markflorisson88/minivect (which will likely have an
>> LLVM backend in the future, and possibly integrated with Numba to
>> allow inlining of numba ufuncs). The project above tries to bring
>> together all the different array expression compilers together in a
>> single framework, to provide efficient array expressions specialized
>> for any data layout (nditer on steroids if you will, with SIMD,
>> threaded and inlining capabilities).
>
> A global hook sounds ugly and hard to control -- it's hard to tell
> which operations should be deferred and which should be forced, etc.
Yes, but for the user the difference should not be visible (unless
operations can raise exceptions, in which case you choose the safe
path, or let the user configure what to do).
> While it would be less magical, I think a more explicit API would in
> the end be easier to use... something like
>
> a, b, c, d = deferred([a, b, c, d])
> e = a + b * c # 'e' is a deferred object too
> f = np.dot(e, d) # so is 'f'
> g = force(f) # 'g' is an ndarray
> # or
> force(f, out=g)
>
> But at that point, this could easily be an external library, right?
> All we'd need from numpy would be some way for external types to
> override the evaluation of ufuncs, np.dot, etc.? We've recently seen
> several reasons to want that functionality, and it seems like
> developing these "improved numexpr" ideas would be much easier if they
> didn't require doing deep surgery to numpy itself...
Definitely, but besides monkey-patch-chaining I think some
modifications would be required, but they would be reasonably simple.
Most of the functionality would be handled in one function, which most
ufuncs (the ones you care about, as well as ufunc (methods) like add)
call. E.g. if ((result = NPy_LazyEval("add", op1, op2)) return result;
, which is inserted after argument unpacking and sanity checking. You
could also do a per-module hook, and have the function look at
sys._getframe(1).f_globals, but that is fragile and won't work from C
or Cython code.
How did you have overrides in mind? I also found this thread:
http://mail.scipy.org/pipermail/numpy-discussion/2011-June/056945.html
, but I think you want more than just to override ufuncs, you want
numpy to govern when stuff is allowed to be lazy and when stuff should
be evaluated (e.g. when it is indexed, slice assigned (although that
itself may also be lazy), etc). You don't want some funny object back
that doesn't work with things which are not overridden in numpy.
> -N
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
