On Tue, Aug 25, 2009 at 9:16 PM, Kurt Smith<[email protected]> wrote:
> On Tue, Aug 25, 2009 at 4:21 PM, Fernando Perez<[email protected]> wrote:
>> Hi folks,
>>
>> On Tue, Aug 25, 2009 at 12:08 PM, Kurt Smith<[email protected]> wrote:
>>> Weave-like support in Cython -- Fernando Perez (not at the BoF)
>>> suggested that Cython could assume the same functionality of
>>> scipy.weave (see: http://www.scipy.org/Weave), since scipy.weave is
>>> poorly documented and could use more maintenance than it is currently
>>> getting. I'm personally very interested in this -- it seems that
>>> there is much overlap between scipy.weave and the combination of
>>> pyximport & the pure-python mode of Cython. It seems that with a bit
>>> of work from an interested user and some guidance from yours truly, we
>>> could improve pyximport & Cython's pure-python mode to incorporate the
>>> good stuff from scipy.weave, but better ;-) I intend to write-up a
>>> CEP with my thoughts sometime in the near future.
>>>
>>
>> Kurt, thanks for resurrecting this comment. Indeed, weave is an
>> extremely useful piece of functionality but one that has been somewhat
>> neglected for years. There's a lot of good in there though, and the
>> need for it is significant. It's *really* cool to do things like:
>>
>> def prod(m, v):
>> """Matrix-vector multiply via weave/blitz++"""
>> nrows, ncolumns = m.shape
>> assert v.ndim==1 and ncolumns==v.shape[0],"Shape mismatch in prod"
>>
>> res = np.zeros(nrows, float)
>> code = r"""
>> for (int i=0; i<nrows; i++)
>> {
>> for (int j=0; j<ncolumns; j++)
>> {
>> res(i) += m(i,j)*v(j);
>> }
>> }
>> """
>> err = inline(code,['nrows', 'ncolumns', 'res', 'm', 'v'], verbose=2,
>> type_converters=converters.blitz)
>> return res
>
> Here's what I'm envisioning for the above usecase -- I admit to not
> using the pure-python mode that much, so there may be errors (and some
> of what I'm using isn't implemented yet). But it shows how much is
> already there in Cython's pure-python mode.
>
> @cython.compile # this is new & where most of the work is required!
> @cython.locals(m=cython.float[:,::1], v=cython.float[::1])
> def prod(m, v):
> cython.declare(
> i=cython.int,
> j=cython.int,
> nrows=cython.int,
> ncolumns=cython.int)
> nrows = m.shape[0]; ncolumns = m.shape[1]
> cython.declare(res=cython.float[::1])
> res = np.zeros(nrows, np.float)
> for i in range(nrows):
> for j in range(ncolumns):
> res[i] += m[i,j] * v[j]
> return res
>
> Once the new array type / memoryview slice is merged into cython-devel
> and we get it integrated into pure-python mode, this will be close to
> working. If you just want things fast in the function body (what
> weave gives you currently) then the above is all you need.
>
> The hard part is getting things to compile & run automatically for
> just this function (the cython.compile decorator). This overlaps with
> pyximport a good amount, and it will take some thought as to how it
> will all work together. Perhaps the core devs won't like so much
> going on behind the scenes -- cython is principally a utility for
> generating a C extension module, not compiling & loading the extension
> module.
>
> Weave compiles individual snippets into their own extension module,
> whereas Cython has always compiled entire modules into extension
> modules, and requires a separate compilation step (whether through the
> commandline or through pyximport). The cython.compile decorator
> would be different, if it were to be just like weave -- it would
> compile a single function into its own extension module, without an
> explicit compilation step. Much more is automated.
>
> This seems to be the main thing that weave offers -- fast code only
> where you want it, without the explicit compilation step. Pyximport
> overlaps here, but pyximport still compiles an entire module, not just
> a part.
>
> So those are some of my thoughts. They'll be in a CEP sometime soon :-)
>
>>
>> This makes interactive experimentation and the writing of
>> loop/indexing-intensive code really easy. But by itself, weave is in
>> serious danger of bitrot, and right now Cython has the momentum and
>> concentrated effort for this problem domain.
>>
>> Everyone is extremely impressed with the amazing work you've all done,
>> and rooting for cython to continue to improve, as you are solving a
>> number of key problems to make python a really complete platform for
>> scientific computing. It's been lots of fun to see cython steadily
>> attack a number of problems that have been a real issue for years.
>>
>> So if Cython absorbs/extends/integrates/develops/whatever the good
>> ideas and code from weave, so that this type of workflow is also
>> supported, it would be great. I am really happy to see f2py grow
>> into fwrap, and the weave machinery (or at least parts of it) seems
>> like a logical addition to the cython-based toolkit.
>>
>> One particular use case for which weave.inline was invaluable to me in
>> the past was the generation of fast looping code for various
>> dimensions. I had to write function evaluation codes where the user
>> could supply a C expression (as a string, that would #define NDIM to
>> indicate the dimensionality of the problem). I could then easily
>> generate a few methods for D=1..6 (the range I needed to cover) that
>> would do certain key operations very fast. The bulk of the code used
>> numpy arrays to be dimension agnostic, but a few methods really needed
>> explicit loops and with weave, it was made completely transparent to
>> the user, all of it being done at runtime (for us, needing gcc at
>> runtime was OK). I mention this to give you some feedback on the use
>> cases for weave (there are others), but I hope that in the end, all of
>> this class of functionality will be available through a unified entry
>> point, and if that's cython, all the better.
>>
>
> Thanks for the encouragement, and for the examples. I don't have time
> to absorb all this just yet, but I'll return to it. And get other
> usecases for weave from you and others.
>
> Kurt
>
>> Thanks again to the team for your great work!
>>
>> Cheers,
>>
>> f
(A comment from a non-expert)
A more literal approach to imitating weave inline compilation, seems
already possible with cython and pyximport. To try out cython and to
reduce the time in debugging my (many) mistakes, a added the
compilation step directly to the script. This seems closer to the
usage of weave that Fernando was deswcribing and what I saw of weave
examples.
Cheers,
Josef
code = '''
import numpy as np
cimport numpy as np
__all__ = ["hist3d"]
DTYPE = ${imgtype}
DTYPE32 = ${outtype}
ctypedef ${imgtype}_t DTYPE_t
ctypedef ${outtype}_t DTYPE_t32
def hist3d(np.ndarray[DTYPE_t, ndim=3] img, int nbins):
# a variation on code by Chris Colbert
#if nbins are not a divisor of 256, all extra high values are added to the
last bin
cdef int x = img.shape[0]
cdef int y = img.shape[1]
#cdef int z = img.shape[2]
#I don't know how to get the correct width
cdef int bdist = np.max(img) / nbins #np.ceil(256 / (1.0* nbins))
cdef int addx
cdef int addy
cdef int addz
cdef np.ndarray[DTYPE_t32, ndim=3] out = np.zeros([nbins, nbins, nbins],
dtype=DTYPE32)
cdef int i, j, v0, v1, v2
for i in range(x):
for j in range(y):
v0 = img[i, j, 0]
v1 = img[i, j, 1]
v2 = img[i, j, 2]
addx = min((v0 - (v0 % bdist)) / bdist, nbins-1)
addy = min((v1 - (v1 % bdist)) / bdist, nbins-1)
addz = min((v2 - (v2 % bdist)) / bdist, nbins-1)
#print addx, addy, addz
out[addx, addy, addz] += 1
return out
'''
from string import Template
s = Template(code)
src = s.substitute({'imgtype': 'np.int', 'outtype': 'np.float64'})
open('histrgbintfl06.pyx','w').write(src)
import pyximport; pyximport.install()
import histrgbintfl06
import numpy as np
factors = np.random.randint(256,size=(480, 630, 3))
h = histrgbintfl06.hist3d(factors, 4)#.astype(np.uint8))
print h[:,:,0]
h = histrgbintfl06.hist3d(factors, 5)
_______________________________________________
Cython-dev mailing list
[email protected]
http://codespeak.net/mailman/listinfo/cython-dev
