Francesc Alted writes:
> On Feb 20, 2012, at 6:18 PM, Dag Sverre Seljebotn wrote:
>> You need at least a slightly different Python API to get anywhere, so
>> numexpr/Theano is the right place to work on an implementation of this
>> idea. Of course it would be nice if numexpr/Theano offered something as
>> convenient as
>>
>> with lazy:
>> arr = A + B + C # with all of these NumPy arrays
>> # compute upon exiting…
> Hmm, that would be cute indeed. Do you have an idea on how the code in the
> with
> context could be passed to the Python AST compiler (à la numexpr.evaluate("A
> + B
> + C"))?
Well, I started writing some experiments to "almost transparently" translate
regular ndarray operations to numexpr strings (or others) using only python
code.
The concept is very simple:
# you only need the first one to start building the AST
a = lazy(np.arange(16))
b = np.arange(16)
res = a + b + 3
print evaluate(res)
# the actual evaluation can be delayed to something like __repr__ or __str__
print repr(res)
print res
# you could also delay evaluation until someone uses res to create a new
array
My target was to use this to also generate optimized GPU kernels in-flight using
pycuda, but I think some other relatively recent project already performed
something similar (w.r.t. generating cuda kernels out of python expressions).
The supporting code for numexpr was something like:
import numexpr
import numpy as np
def build_arg_expr (arg, args):
if isinstance(arg, Expr):
# recursively build the expression
arg_expr, arg_args = arg.build_expr()
args.update(arg_args)
return arg_expr
else:
# unique argument identifier
arg_id = "arg_%d" % id(arg)
args[arg_id] = arg
return arg_id
# generic expression builder
class Expr:
def evaluate(self):
expr, args = self.build_expr()
return numexpr.evaluate(expr, local_dict = args, global_dict = {})
def __repr__ (self):
return self.evaluate().__repr__()
def __str__ (self):
return self.evaluate().__str__()
def __add__ (self, other):
return ExprAdd(self, other)
# expression builder for adds
class ExprAdd(Expr):
def __init__(self, arg1, arg2):
self.arg1 = arg1
self.arg2 = arg2
def build_expr(self):
args = {}
expr1 = build_arg_expr(self.arg1, args)
expr2 = build_arg_expr(self.arg2, args)
return "("+expr1+") + ("+expr2+")", args
# ndarray-like class to generate expression builders
class LazyNdArray(np.ndarray):
def __add__ (self, other):
return ExprAdd(self, other)
# build a LazyNdArray
def lazy (arg):
return arg.view(LazyNdArray)
# evaluate with numexpr an arbitrary expression builder
def evaluate(arg):
return arg.evaluate()
The thing here is to always return to the user something that looks like an
ndarray.
As you can see the whole thing is not very complex, but some less funny code had
to be written meanwhile for work and I just dropped this :)
Lluis
--
"And it's much the same thing with knowledge, for whenever you learn
something new, the whole world becomes that much richer."
-- The Princess of Pure Reason, as told by Norton Juster in The Phantom
Tollbooth
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
