Re: [Numpy-discussion] Question on numpy.ma.masked_values
Gökhan, By default, the mask of a MaskedArray is set to the special value `np.ma.nomask`. In other terms:: np.ma.array(...) <=> np.ma.array(..., mask=np.ma.nomask) In practice, np.ma.nomask lets us quickly check whether a MaskedArray has a masked value : if its .mask is np.ma.nomask, then no masked value, otherwise it's a full boolean array and we can use any. If you want to create a MaskedArray w/ a full boolean mask, just use:: np.ma.array(..., mask=False) In that case, the mask is automatically created as a boolean array with the same shape as the data, with False everywhere. If you used True, the mask would be full of True... Now, just to be clear, you'd want 'np.ma.masked_values(...,shrink=False) to create a maked array w/ a full boolean mask by default, right ? On 3/15/12, Gökhan Sever wrote: > Submitted the ticket at http://projects.scipy.org/numpy/ticket/2082 > > > > On Thu, Mar 15, 2012 at 1:24 PM, Gökhan Sever wrote: > >> >> >> On Thu, Mar 15, 2012 at 1:12 PM, Pierre GM wrote: >> >>> Ciao Gökhan, >>> AFAIR, shrink is used only to force a collapse of a mask full of False, >>> not to force the creation of such a mask. >>> Now, it should work as you expected, meaning that it needs to be fixed. >>> Could you open a ticket? And put me in copy, just in case. >>> Anyhow: >>> Your trick is a tad dangerous, as it erases the previous mask. I'd prefer >>> to create x w/ a full mask, then use masked_values w/ shrink=False... >>> Now, >>> if you're sure there's x= no masked values, go for it. >>> Cheers >>> >>> This condition checking should make it stronger: >> >> I7 x = np.array([1, 1.1, 2, 1.1, 3]) >> >> I8 y = np.ma.masked_values(x, 1.5) >> >> I9 if y.mask == False: >> y.mask = np.zeros(len(x), dtype=np.bool)*True >>...: >> >> I10 y.mask >> O10 array([False, False, False, False, False], dtype=bool) >> >> I11 y >> O11 >> masked_array(data = [1.0 1.1 2.0 1.1 3.0], >> mask = [False False False False False], >>fill_value = 1.5) >> >> How do you create "x w/ a full mask"? >> >> -- >> Gökhan >> > > > > -- > Gökhan > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Recovering from a STOP ?
Pauli, Chris, Thanks for your inputs. Pauli, I think that when f2py encounters a STOP statement, it just stops the execution of the process. Alas, it's the same process as the interpreter... So we need a trick not to interrupt the whole process. I eventually resorted to patching f2py as suggested in http://article.gmane.org/gmane.comp.python.f2py.user/1204/ which amounts to let f2py trap the SIGINT. That's good enough for what I need. However, that stresses the need to have a more robust way to deal with these interruptions and make sure that several objects calling one (or several temporary) fortran extension don't interact the ones with the others. And I'm coming to the same conclusion as Chris', that I have to use the `multiprocessing` module, with several processes calling their own fortran extension. But that's another story. Thanks y'all for your help P. On 3/14/12, Chris Barker wrote: > On Wed, Mar 14, 2012 at 9:25 AM, Pauli Virtanen wrote: >> Or, maybe the whole Fortran stuff can be run in a separate process, so >> that crashing doesn't matter. > > That's what I was going to suggest -- even if you can get it not to > crash, it may well be in a bad state -- memory leaks, and who know > what else. > > We did something similar with some C code that called the system > exit() function when it encountered errors -- it may not have been too > hard to replace those calls, but making sure the memory was all > cleaned up was going to be a trick -- so we just used the multiprocess > module to call it in another process. > > HTH, > > -Chris > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > chris.bar...@noaa.gov > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] NumPy-Discussion Digest, Vol 66, Issue 61
Hi Richard, Thanks for your answer and the related help ! In fact, I was hoping to have a less memory and more speed solution. Something equivalent to a "raster calculator" for numpy. Wouldn't it make sense to have some optimized function to work on more than 2 arrays for numpy anyway ? At the end, I am rather interested by more speed. I tried first a code-sparing version : array = numpy.asarray([(aBlueChannel < 1.0),(aNirChannel > aBlueChannel * 1.0),(aNirChannel < aBlueChannel * 1.8)]).all() But this one is at the end more than 2 times slower than : array1 = numpy.empty([3,6566,6682], dtype=numpy.bool) numpy.less(aBlueChannel, 1.0, out=array1[0]) numpy.greater(aNirChannel, (aBlueChannel * 1.0), out=array1[1]) numpy.less(aNirChannel, (aBlueChannel * 1.8), out=array1[2]) array = array1.all() (and this solution is about 30% faster than the original one) I could find another way which was fine for me too: array = (aBlueChannel < 1.0) * (aNirChannel > (aBlueChannel * 1.0)) * (aNirChannel < (aBlueChannel * 1.8)) But this one is only 5-10% faster than the original solution, even if probably using less memory than the 2 previous ones. (same was possible with operator +, but slower than operator *) Regards, Matthieu Rigal On Monday 19 March 2012 18:00:02 numpy-discussion-requ...@scipy.org wrote: > Message: 2 > Date: Mon, 19 Mar 2012 13:20:23 + > From: Richard Hattersley > Subject: Re: [Numpy-discussion] Using logical function on more than 2 > arrays, availability of a "between" function ? > To: Discussion of Numerical Python > Message-ID: > > Content-Type: text/plain; charset=ISO-8859-1 > > What do you mean by "efficient"? Are you trying to get it execute > faster? Or using less memory? Or have more concise source code? > > Less memory: > - numpy.vectorize would let you get to the end result without any > intermediate arrays but will be slow. > - Using the "out" parameter of numpy.logical_and will let you avoid > one of the intermediate arrays. > > More speed?: > Perhaps putting all three boolean temporary results into a single > boolean array (using the "out" parameter of numpy.greater, etc) and > using numpy.all might benefit from logical short-circuiting. > > And watch out for divide-by-zero from "aNirChannel/aBlueChannel". > > Regards, > Richard Hattersley > RapidEye AG Molkenmarkt 30 14776 Brandenburg an der Havel Germany Follow us on Twitter! www.twitter.com/rapideye_ag Head Office/Sitz der Gesellschaft: Brandenburg an der Havel Management Board/Vorstand: Ryan Johnson Chairman of Supervisory Board/Vorsitzender des Aufsichtsrates: Robert Johnson Commercial Register/Handelsregister Potsdam HRB 24742 P Tax Number/Steuernummer: 048/100/00053 VAT-Ident-Number/Ust.-ID: DE 199331235 DIN EN ISO 9001 certified ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Trying to read 500M txt file using numpy.genfromtxt within ipython shell
Dear all, I received a file from others which contains ~30 million lines and in size of ~500M. I try read it with numpy.genfromtxt in ipython interactive mode. Then ipython crashed. The data contains lat,lon,var1,year, the year ranges from 1001 to 2006. Finally I want to write the data to netcdf for separate years and feed them into the model. I guess I need a better way to do this? anyone would be any idea is highly appreciated. lon,lat,year,area_burned -180.0,65.0,1001,0 -180.0,65.0,1002,0 -180.0,65.0,1003,0 -180.0,65.0,1004,0 -180.0,65.0,1005,0 -180.0,65.0,1006,0 -180.0,65.0,1007,0 thanks and cheers, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Using logical function on more than 2 arrays, availability of a "between" function ?
Auto-answer, sorry, Well, actually I made a mistake lower... that you may have noticed... On the faster (your) solution, even with a cleaner use of the out parameter, the fact that the all has then to be used with parameter axis=0 takes more time and makes it actually slower than the initial solution... So I may go for the "multiplier" solution. Regards, Matthieu On Tuesday 20 March 2012 13:13:15 you wrote: > Hi Richard, > > Thanks for your answer and the related help ! > > In fact, I was hoping to have a less memory and more speed solution. > Something equivalent to a "raster calculator" for numpy. Wouldn't it make > sense to have some optimized function to work on more than 2 arrays for > numpy anyway ? > > At the end, I am rather interested by more speed. > > I tried first a code-sparing version : > array = numpy.asarray([(aBlueChannel < 1.0),(aNirChannel > aBlueChannel * > 1.0),(aNirChannel < aBlueChannel * 1.8)]).all() > > But this one is at the end more than 2 times slower than : > array1 = numpy.empty([3,6566,6682], dtype=numpy.bool) > numpy.less(aBlueChannel, 1.0, out=array1[0]) > numpy.greater(aNirChannel, (aBlueChannel * 1.0), out=array1[1]) > numpy.less(aNirChannel, (aBlueChannel * 1.8), out=array1[2]) > array = array1.all() > > (and this solution is about 30% faster than the original one) > > I could find another way which was fine for me too: > array = (aBlueChannel < 1.0) * (aNirChannel > (aBlueChannel * 1.0)) * > (aNirChannel < (aBlueChannel * 1.8)) > > But this one is only 5-10% faster than the original solution, even if > probably using less memory than the 2 previous ones. (same was possible > with operator +, but slower than operator *) > > Regards, > Matthieu Rigal > > On Monday 19 March 2012 18:00:02 numpy-discussion-requ...@scipy.org wrote: > > Message: 2 > > Date: Mon, 19 Mar 2012 13:20:23 + > > From: Richard Hattersley > > Subject: Re: [Numpy-discussion] Using logical function on more than 2 > > arrays, availability of a "between" function ? > > To: Discussion of Numerical Python > > Message-ID: > > > > > > > > Content-Type: text/plain; charset=ISO-8859-1 > > > > What do you mean by "efficient"? Are you trying to get it execute > > faster? Or using less memory? Or have more concise source code? > > > > Less memory: > > - numpy.vectorize would let you get to the end result without any > > intermediate arrays but will be slow. > > - Using the "out" parameter of numpy.logical_and will let you avoid > > one of the intermediate arrays. > > > > More speed?: > > Perhaps putting all three boolean temporary results into a single > > boolean array (using the "out" parameter of numpy.greater, etc) and > > using numpy.all might benefit from logical short-circuiting. > > > > And watch out for divide-by-zero from "aNirChannel/aBlueChannel". > > > > Regards, > > Richard Hattersley > RapidEye AG Molkenmarkt 30 14776 Brandenburg an der Havel Germany Follow us on Twitter! www.twitter.com/rapideye_ag Head Office/Sitz der Gesellschaft: Brandenburg an der Havel Management Board/Vorstand: Ryan Johnson Chairman of Supervisory Board/Vorsitzender des Aufsichtsrates: Robert Johnson Commercial Register/Handelsregister Potsdam HRB 24742 P Tax Number/Steuernummer: 048/100/00053 VAT-Ident-Number/Ust.-ID: DE 199331235 DIN EN ISO 9001 certified ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Trying to read 500M txt file using numpy.genfromtxt within ipython shell
Hi, I think writing a Python script that convert your txt file to one netcdf file, reading the txt file one line at a time, and then use the netcdf file normally would be a good solution! Best, David Excerpts from Chao YUE's message of mar. mars 20 13:33:56 +0100 2012: > Dear all, > > I received a file from others which contains ~30 million lines and in size > of ~500M. > I try read it with numpy.genfromtxt in ipython interactive mode. Then > ipython crashed. > The data contains lat,lon,var1,year, the year ranges from 1001 to 2006. > Finally I want to write the > data to netcdf for separate years and feed them into the model. I guess I > need a better way to do this? > anyone would be any idea is highly appreciated. > > > lon,lat,year,area_burned > -180.0,65.0,1001,0 > -180.0,65.0,1002,0 > -180.0,65.0,1003,0 > -180.0,65.0,1004,0 > -180.0,65.0,1005,0 > -180.0,65.0,1006,0 > -180.0,65.0,1007,0 > > thanks and cheers, > > Chao > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Trying to read 500M txt file using numpy.genfromtxt within ipython shell
I would be in agree. thanks! I use gawk to separate the file into many files by year, then it would be easier to handle. anyway, it's not a good practice to produce such huge line txt files Chao 2012/3/20 David Froger > Hi, > > I think writing a Python script that convert your txt file to one netcdf > file, > reading the txt file one line at a time, and then use the netcdf file > normally > would be a good solution! > > Best, > David > > Excerpts from Chao YUE's message of mar. mars 20 13:33:56 +0100 2012: > > Dear all, > > > > I received a file from others which contains ~30 million lines and in > size > > of ~500M. > > I try read it with numpy.genfromtxt in ipython interactive mode. Then > > ipython crashed. > > The data contains lat,lon,var1,year, the year ranges from 1001 to 2006. > > Finally I want to write the > > data to netcdf for separate years and feed them into the model. I guess I > > need a better way to do this? > > anyone would be any idea is highly appreciated. > > > > > > lon,lat,year,area_burned > > -180.0,65.0,1001,0 > > -180.0,65.0,1002,0 > > -180.0,65.0,1003,0 > > -180.0,65.0,1004,0 > > -180.0,65.0,1005,0 > > -180.0,65.0,1006,0 > > -180.0,65.0,1007,0 > > > > thanks and cheers, > > > > Chao > > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Trying to read 500M txt file using numpy.genfromtxt within ipython shell
On 20 Mar 2012, at 14:40, Chao YUE wrote:
> I would be in agree. thanks!
> I use gawk to separate the file into many files by year, then it would be
> easier to handle.
> anyway, it's not a good practice to produce such huge line txt files
Indeed it's not, but it's also not good practice to load the entire content
of text files as python lists into memory, as unfortunately all the numpy
readers are still doing. But this has been discussed on this list and
improvements are under way.
For your problem at hand the textreader Warren Weckesser recently
made known - can't find the post right now, but you can find it at
https://github.com/WarrenWeckesser/textreader
might be helpful. It is still under construction, but for a plain csv file such
as yours it should be working already. And since the text parsing is
implemented in C, it should also give you a huge speedup for your 1/2 GB!
For additional profiling, similar to what David suggested, it would certainly
be a good idea to read in smaller chunks of the file and write it directly to
the netCDF file. Note that you can already read single lines at a time with the
likes of
from StringIO import StringIO
f = open('file.txt'. 'r')
np.genfromtxt(StringIO(f.next()), delimiter=',')
but I don't think it would work this way with textreader, and iterating such a
small
loop over lines in Python would beat the point of using a fast reader.
As your actual data would be < 1GB in numpy, memory usage with textreader
should also not be critical yet.
Cheers,
Derek
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Using logical function on more than 2 arrays, availability of a "between" function ?
I didn't try it, but I think that Theano and numexpr should be able to make them faster. [1] http://deeplearning.net/software/theano/ [2] https://code.google.com/p/numexpr/ Fred On Tue, Mar 20, 2012 at 9:05 AM, Matthieu Rigal wrote: > Auto-answer, sorry, > > Well, actually I made a mistake lower... that you may have noticed... > On the faster (your) solution, even with a cleaner use of the out parameter, > the fact that the all has then to be used with parameter axis=0 takes more > time and makes it actually slower than the initial solution... > > So I may go for the "multiplier" solution. > > Regards, > Matthieu > > On Tuesday 20 March 2012 13:13:15 you wrote: >> Hi Richard, >> >> Thanks for your answer and the related help ! >> >> In fact, I was hoping to have a less memory and more speed solution. >> Something equivalent to a "raster calculator" for numpy. Wouldn't it make >> sense to have some optimized function to work on more than 2 arrays for >> numpy anyway ? >> >> At the end, I am rather interested by more speed. >> >> I tried first a code-sparing version : >> array = numpy.asarray([(aBlueChannel < 1.0),(aNirChannel > aBlueChannel * >> 1.0),(aNirChannel < aBlueChannel * 1.8)]).all() >> >> But this one is at the end more than 2 times slower than : >> array1 = numpy.empty([3,6566,6682], dtype=numpy.bool) >> numpy.less(aBlueChannel, 1.0, out=array1[0]) >> numpy.greater(aNirChannel, (aBlueChannel * 1.0), out=array1[1]) >> numpy.less(aNirChannel, (aBlueChannel * 1.8), out=array1[2]) >> array = array1.all() >> >> (and this solution is about 30% faster than the original one) >> >> I could find another way which was fine for me too: >> array = (aBlueChannel < 1.0) * (aNirChannel > (aBlueChannel * 1.0)) * >> (aNirChannel < (aBlueChannel * 1.8)) >> >> But this one is only 5-10% faster than the original solution, even if >> probably using less memory than the 2 previous ones. (same was possible >> with operator +, but slower than operator *) >> >> Regards, >> Matthieu Rigal >> >> On Monday 19 March 2012 18:00:02 numpy-discussion-requ...@scipy.org wrote: >> > Message: 2 >> > Date: Mon, 19 Mar 2012 13:20:23 + >> > From: Richard Hattersley >> > Subject: Re: [Numpy-discussion] Using logical function on more than 2 >> > arrays, availability of a "between" function ? >> > To: Discussion of Numerical Python >> > Message-ID: >> > >> > > > >> > > Content-Type: text/plain; charset=ISO-8859-1 >> > >> > What do you mean by "efficient"? Are you trying to get it execute >> > faster? Or using less memory? Or have more concise source code? >> > >> > Less memory: >> > - numpy.vectorize would let you get to the end result without any >> > intermediate arrays but will be slow. >> > - Using the "out" parameter of numpy.logical_and will let you avoid >> > one of the intermediate arrays. >> > >> > More speed?: >> > Perhaps putting all three boolean temporary results into a single >> > boolean array (using the "out" parameter of numpy.greater, etc) and >> > using numpy.all might benefit from logical short-circuiting. >> > >> > And watch out for divide-by-zero from "aNirChannel/aBlueChannel". >> > >> > Regards, >> > Richard Hattersley >> > > RapidEye AG > Molkenmarkt 30 > 14776 Brandenburg an der Havel > Germany > > Follow us on Twitter! www.twitter.com/rapideye_ag > > Head Office/Sitz der Gesellschaft: Brandenburg an der Havel > Management Board/Vorstand: Ryan Johnson > Chairman of Supervisory Board/Vorsitzender des Aufsichtsrates: > Robert Johnson > Commercial Register/Handelsregister Potsdam HRB 24742 P > Tax Number/Steuernummer: 048/100/00053 > VAT-Ident-Number/Ust.-ID: DE 199331235 > DIN EN ISO 9001 certified > > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Looking for people interested in helping with Python compiler to LLVM
On 13 March 2012 18:18, Travis Oliphant wrote: >>> >>> (Mark F., how does the above match how you feel about this?) >> >> I would like collaboration, but from a technical perspective I think >> this would be much more involved than just dumping the AST to an IR >> and generating some code from there. For vector expressions I think >> sharing code would be more feasible than arbitrary (parallel) loops, >> etc. Cython as a compiler can make many decisions that a Python >> (bytecode) compiler can't make (at least without annotations and a >> well-defined subset of the language (not so much the syntax as the >> semantics)). I think in numba, if parallelism is to be supported, you >> will want a prange-like construct, as proving independence between >> iterations can be very hard to near impossible for a compiler. > > I completely agree that you have to define some kind of syntax to get > parallelism. But, a prange construct would not be out of the question, of > course. > >> >> As for code generation, I'm not sure how llvm would do things like >> slicing arrays, reshaping, resizing etc (for vector expressions you >> can first evaluate all slicing and indexing operations and then >> compile the remaining vector expression), but for loops and array >> reassignment within loops this would have to invoke the actual slicing >> code from the llvm code (I presume). > > There could be some analysis on the byte-code, prior to emitting the llvm > code in order to handle lots of things. Basically, you have to "play" the > byte-code on a simple machine anyway in order to emit the correct code. The > big thing about Cython is you have to typedef too many things that are really > quite knowable from the code. If Cython could improve it's type inference, > then it would be a more suitable target. > >> There are many other things, like >> bounds checking, wraparound, etc, that are all supported in both numpy >> and Cython, but going through an llvm layer would as far as I can see, >> require re-implementing those, at least if you want top-notch >> performance. Personally, I think for non-trivial performance-critical >> code (for loops with indexing, slicing, function calls, etc) Cython is >> a better target. > > With libclang it is really quite possible to imagine a cython -> C target > that itself compiles to llvm so that you can do everything at that > intermediate layer. However, LLVM is a much better layer for optimization > than C now that there are a lot of people collaborating on that layer. I > think it would be great if Cython targeted LLVM actually instead of C. > >> >> Finally, as for non-vector-expression code, I really believe Cython is >> a better target. cython.inline can have high overhead (at least the >> first time it has to compile), but with better (numpy-aware) type >> inference or profile guided optimizations (see recent threads on the >> cython-dev mailing list), in addition to things like prange, I >> personally believe Cython targets most of the use cases where numba >> would be able to generate performing code. > > Cython and Numba certainly overlap. However, Cython requires: > > 1) learning another language > 2) creating an extension module --- loading bit-code files and > dynamically executing (even on a different machine from the one that > initially created them) can be a powerful alternative for run-time > compilation and distribution of code. > > These aren't show-stoppers obviously. But, I think some users would prefer > an even simpler approach to getting fast-code than Cython (which currently > doesn't do enought type-inference and requires building a dlopen extension > module). Dag and I have been discussing this at PyCon, and here is my take on it (at this moment :). Definitely, if you can avoid Cython then that is easier and more desirable in many ways. So perhaps we can create a third project called X (I'm not very creative, maybe ArrayExprOpt), that takes an abstract syntax tree in a rather simple form, performs code optimizations such as rewriting loops with array accesses to vector expressions, fusing vector expressions and loops, etc, and spits out a transformed AST containing these optimizations. If runtime information is given such as actual shape and stride information the transformations could figure out there and then whether to do things like collapsing, axes swapping, blocking (as in, introducing more axes or loops to retain discontiguous blocks in the cache), blocked memory copies to contiguous chunks, etc. The AST could then also say whether the final expressions are vectorizable. Part of this functionality is already in numpy's nditer, except that this would be implicit and do more (and hopefully with minimal overhead). So numba, Cython and maybe numexpr could use the functionality, simply by building the AST from Python and converting back (if necessary) to its own AST. As such, the AST optimizer would be only part of any (
Re: [Numpy-discussion] NumPy-Discussion Digest, Vol 66, Issue 61
On Tue, Mar 20, 2012 at 5:13 AM, Matthieu Rigal wrote: > In fact, I was hoping to have a less memory and more speed solution. which do often go together, at least for big problems -- pushingm emory around often takes more time than the computation itself. > At the end, I am rather interested by more speed. > > I tried first a code-sparing version : array = numpy.asarray([(aBlueChannel < 1.0),(aNirChannel > aBlueChannel * > 1.0),(aNirChannel < aBlueChannel * 1.8)]).all() (by the way -- it is MUCH better if you post example code with actual data (the example data can be much smaller) -- while with small data sets you can't test performance, you can test correctness -- the easier you make it for us to try stuff out, the more we can help you. re-formatting so I can read this, I get: array = numpy.asarray([(aBlueChannel < 1.0), (aNirChannel > aBlueChannel * 1.0), (aNirChannel < aBlueChannel * 1.8)]).all() a few notes: asarray will work, but is pointless here -- I'd just use "array" -- asarray() is for when you may or may not have an array as input, and you want to preserve it if you do. I'd probably use numpy.vstack or rstack here, rather than the array (or asarray) function -- there is a larger parsing overhead to array() I suppose this is a special case, but multiplying by 1.0 is kind of pointless there (or are you doing that to cast to a float array)? if you are doing an all() -- not much reason to put them all in the same array first anyway. > But this one is at the end more than 2 times slower than : array1 = numpy.empty([3,6566,6682], dtype=numpy.bool) numpy.less(aBlueChannel, 1.0, out=array1[0]) numpy.greater(aNirChannel, (aBlueChannel * 1.0), out=array1[1]) numpy.less(aNirChannel, (aBlueChannel * 1.8), out=array1[2]) array = array1.all() yup -- creating temporaries can be slow for big data -- there is the trade-off between compact code and performance some times. I think you can be more memory efficient here, though -- if in the end all you want is the final "all" check, no need to store all checks for each channel -- something like: #allocate a bool array: array1 = numpy.empty( (6566,6682), dtype=numpy.bool) result = numpy.less(aBlueChannel, 1.0, out=array1).all() result &= numpy.greater(aNirChannel, (aBlueChannel * 1.0), out=array1).all() result &= numpy.less(aNirChannel, (aBlueChannel * 1.8), out=array1[2]).all() three loops for the all(), but less memory to push around -- may be faster. I'd also take a look at numexpr for this, it could be very helpful: http://code.google.com/p/numexpr/ -Chris > (and this solution is about 30% faster than the original one) > > I could find another way which was fine for me too: > array = (aBlueChannel < 1.0) * (aNirChannel > (aBlueChannel * 1.0)) * > (aNirChannel < (aBlueChannel * 1.8)) > > But this one is only 5-10% faster than the original solution, even if probably > using less memory than the 2 previous ones. (same was possible with operator > +, but slower than operator *) > > Regards, > Matthieu Rigal > > > On Monday 19 March 2012 18:00:02 numpy-discussion-requ...@scipy.org wrote: >> Message: 2 >> Date: Mon, 19 Mar 2012 13:20:23 + >> From: Richard Hattersley >> Subject: Re: [Numpy-discussion] Using logical function on more than 2 >> arrays, availability of a "between" function ? >> To: Discussion of Numerical Python >> Message-ID: >> > > Content-Type: text/plain; charset=ISO-8859-1 >> >> What do you mean by "efficient"? Are you trying to get it execute >> faster? Or using less memory? Or have more concise source code? >> >> Less memory: >> - numpy.vectorize would let you get to the end result without any >> intermediate arrays but will be slow. >> - Using the "out" parameter of numpy.logical_and will let you avoid >> one of the intermediate arrays. >> >> More speed?: >> Perhaps putting all three boolean temporary results into a single >> boolean array (using the "out" parameter of numpy.greater, etc) and >> using numpy.all might benefit from logical short-circuiting. >> >> And watch out for divide-by-zero from "aNirChannel/aBlueChannel". >> >> Regards, >> Richard Hattersley >> > > RapidEye AG > Molkenmarkt 30 > 14776 Brandenburg an der Havel > Germany > > Follow us on Twitter! www.twitter.com/rapideye_ag > > Head Office/Sitz der Gesellschaft: Brandenburg an der Havel > Management Board/Vorstand: Ryan Johnson > Chairman of Supervisory Board/Vorsitzender des Aufsichtsrates: > Robert Johnson > Commercial Register/Handelsregister Potsdam HRB 24742 P > Tax Number/Steuernummer: 048/100/00053 > VAT-Ident-Number/Ust.-ID: DE 199331235 > DIN EN ISO 9001 certified > > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (20
Re: [Numpy-discussion] Looking for people interested in helping with Python compiler to LLVM
This sounds a lot like Theano, did you look into it? -=- Olivier Le 20 mars 2012 13:49, mark florisson a écrit : > On 13 March 2012 18:18, Travis Oliphant wrote: > >>> > >>> (Mark F., how does the above match how you feel about this?) > >> > >> I would like collaboration, but from a technical perspective I think > >> this would be much more involved than just dumping the AST to an IR > >> and generating some code from there. For vector expressions I think > >> sharing code would be more feasible than arbitrary (parallel) loops, > >> etc. Cython as a compiler can make many decisions that a Python > >> (bytecode) compiler can't make (at least without annotations and a > >> well-defined subset of the language (not so much the syntax as the > >> semantics)). I think in numba, if parallelism is to be supported, you > >> will want a prange-like construct, as proving independence between > >> iterations can be very hard to near impossible for a compiler. > > > > I completely agree that you have to define some kind of syntax to get > parallelism. But, a prange construct would not be out of the question, of > course. > > > >> > >> As for code generation, I'm not sure how llvm would do things like > >> slicing arrays, reshaping, resizing etc (for vector expressions you > >> can first evaluate all slicing and indexing operations and then > >> compile the remaining vector expression), but for loops and array > >> reassignment within loops this would have to invoke the actual slicing > >> code from the llvm code (I presume). > > > > There could be some analysis on the byte-code, prior to emitting the > llvm code in order to handle lots of things. Basically, you have to > "play" the byte-code on a simple machine anyway in order to emit the > correct code. The big thing about Cython is you have to typedef too many > things that are really quite knowable from the code. If Cython could > improve it's type inference, then it would be a more suitable target. > > > >> There are many other things, like > >> bounds checking, wraparound, etc, that are all supported in both numpy > >> and Cython, but going through an llvm layer would as far as I can see, > >> require re-implementing those, at least if you want top-notch > >> performance. Personally, I think for non-trivial performance-critical > >> code (for loops with indexing, slicing, function calls, etc) Cython is > >> a better target. > > > > With libclang it is really quite possible to imagine a cython -> C > target that itself compiles to llvm so that you can do everything at that > intermediate layer. However, LLVM is a much better layer for > optimization than C now that there are a lot of people collaborating on > that layer. I think it would be great if Cython targeted LLVM actually > instead of C. > > > >> > >> Finally, as for non-vector-expression code, I really believe Cython is > >> a better target. cython.inline can have high overhead (at least the > >> first time it has to compile), but with better (numpy-aware) type > >> inference or profile guided optimizations (see recent threads on the > >> cython-dev mailing list), in addition to things like prange, I > >> personally believe Cython targets most of the use cases where numba > >> would be able to generate performing code. > > > > Cython and Numba certainly overlap. However, Cython requires: > > > >1) learning another language > >2) creating an extension module --- loading bit-code files and > dynamically executing (even on a different machine from the one that > initially created them) can be a powerful alternative for run-time > compilation and distribution of code. > > > > These aren't show-stoppers obviously. But, I think some users would > prefer an even simpler approach to getting fast-code than Cython (which > currently doesn't do enought type-inference and requires building a dlopen > extension module). > > Dag and I have been discussing this at PyCon, and here is my take on > it (at this moment :). > > Definitely, if you can avoid Cython then that is easier and more > desirable in many ways. So perhaps we can create a third project > called X (I'm not very creative, maybe ArrayExprOpt), that takes an > abstract syntax tree in a rather simple form, performs code > optimizations such as rewriting loops with array accesses to vector > expressions, fusing vector expressions and loops, etc, and spits out a > transformed AST containing these optimizations. If runtime information > is given such as actual shape and stride information the > transformations could figure out there and then whether to do things > like collapsing, axes swapping, blocking (as in, introducing more axes > or loops to retain discontiguous blocks in the cache), blocked memory > copies to contiguous chunks, etc. The AST could then also say whether > the final expressions are vectorizable. Part of this functionality is > already in numpy's nditer, except that this would be implicit and do > more (and h
Re: [Numpy-discussion] Question on numpy.ma.masked_values
Yes, that's the behaviour that I expect setting the 'shrink' keyword to 'False' > Now, just to be clear, you'd want > 'np.ma.masked_values(...,shrink=False) to create a maked array w/ a > full boolean mask by default, right ? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Looking for people interested in helping with Python compiler to LLVM
On Mar 20, 2012, at 12:49 PM, mark florisson wrote: >> Cython and Numba certainly overlap. However, Cython requires: >> >>1) learning another language >>2) creating an extension module --- loading bit-code files and >> dynamically executing (even on a different machine from the one that >> initially created them) can be a powerful alternative for run-time >> compilation and distribution of code. >> >> These aren't show-stoppers obviously. But, I think some users would prefer >> an even simpler approach to getting fast-code than Cython (which currently >> doesn't do enought type-inference and requires building a dlopen extension >> module). > > Dag and I have been discussing this at PyCon, and here is my take on > it (at this moment :). > > Definitely, if you can avoid Cython then that is easier and more > desirable in many ways. So perhaps we can create a third project > called X (I'm not very creative, maybe ArrayExprOpt), that takes an > abstract syntax tree in a rather simple form, performs code > optimizations such as rewriting loops with array accesses to vector > expressions, fusing vector expressions and loops, etc, and spits out a > transformed AST containing these optimizations. If runtime information > is given such as actual shape and stride information the > transformations could figure out there and then whether to do things > like collapsing, axes swapping, blocking (as in, introducing more axes > or loops to retain discontiguous blocks in the cache), blocked memory > copies to contiguous chunks, etc. The AST could then also say whether > the final expressions are vectorizable. Part of this functionality is > already in numpy's nditer, except that this would be implicit and do > more (and hopefully with minimal overhead). > > So numba, Cython and maybe numexpr could use the functionality, simply > by building the AST from Python and converting back (if necessary) to > its own AST. As such, the AST optimizer would be only part of any > (runtime) compiler's pipeline, and it should be very flexible to > retain any information (metadata regarding actual types, control flow > information, etc) provided by the original AST. It would not do > control flow analysis, type inference or promotion, etc, but only deal > with abstract types like integers, reals and arrays (C, Fortran or > partly contiguous or strided). It would not deal with objects, but > would allow to insert nodes like UnreorderableNode and SideEffectNode > wrapping parts of the original AST. In short, it should be as easy as > possible to convert from an original AST to this project's AST and > back again afterwards. I think this is a very interesting project, and certainly projects like numba can benefit of it. So, in order to us have an idea on what you are after, can we assume that your project (call it X) would be kind of an compiler optimizer, and then the produced, optimized code could be feed into numba for optimized LLVM code generation (that on its turn, can be run on top of CPUs or GPUs or a combination)? Is that correct? Giving that my interpretation above is correct, it is bit more difficult to me to see how your X project could be of benefit for numexpr. In fact, I actually see this the other way round: once the optimizer has discovered the vectorization parts, then go one step further and generate code that uses numexpr automatically (call this, vectorization through numexpr). This is what you mean, or I'm missing something? > As the project matures many optimizations may be added that deal with > all sorts of loop restructuring and ways to efficiently utilize the > cache as well as enable vectorization and possibly parallelism. > Perhaps it could even generate a different AST depending on whether > execution target the CPU or the GPU (with optionally available > information such as cache sizes, GPU shared/local memory sizes, etc). > > Seeing that this would be a part of my master dissertation, my > supervisor would require me to write the code, so at least until > August I think I would have to write (at least the bulk of) this. > Otherwise I can also make other parts of my dissertation's project > more prominent to make up for it. Anyway, my question is, is there > interest from at least the numba and numexpr projects (if code can be > transformed into vector operations, it makes sense to use numexpr for > that, I'm not sure what numba's interest is in that). I'm definitely interested for the numexpr part. It is just that I'm still struggling to see the big picture on this. But the general idea is really appealing. Thanks, -- Francesc Alted ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Looking for people interested in helping with Python compiler to LLVM
We talked some about Theano. There are some differences in project goals which means that it makes sense to make this a seperate project: Cython wants to use this to generate C code up front from the Cython AST at compilation time; numba also has a different frontend (parsing of python bytecode) and a different backend (LLVM). However, it may very well be possible that Theano could be refactored so that the more essential algorithms working on the syntax tree could be pulled out and shared with cython and numba. Then the question is whether the core of Theano is smart enough to compete with Fortran compilers and support arbitraily strided inputs optimally. Otherwise one might as well start from scratch. I'll leave that for Mark to figure out... Dag -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. Olivier Delalleau wrote: This sounds a lot like Theano, did you look into it? -=- Olivier Le 20 mars 2012 13:49, mark florisson a écrit : On 13 March 2012 18:18, Travis Oliphant wrote: >>> >>> (Mark F., how does the above match how you feel about this?) >> >> I would like collaboration, but from a technical perspective I think >> this would be much more involved than just dumping the AST to an IR >> and generating some code from there. For vector expressions I think >> sharing code would be more feasible than arbitrary (parallel) loops, >> etc. Cython as a compiler can make many decisions that a Python >> (bytecode) compiler can't make (at least without annotations and a >> well-defined subset of the language (not so much the syntax as the >> semantics)). I think in numba, if parallelism is to be supported, you >> will want a prange-like construct, as proving independence between >> iterations can be very hard to near impossible for a compiler. > > I completely agree that you have to define some kind of syntax to get > parallelism. But, a prange construct would not be out of the question, of > course. > >> >> As for code generation, I'm not sure how llvm would do things like >> slicing arrays, reshaping, resizing etc (for vector expressions you >> can first evaluate all slicing and indexing operations and then >> compile the remaining vector expression), but for loops and array >> reassignment within loops this would have to invoke the actual slicing >> code from the llvm code (I presume). > > There could be some analysis on the byte-code, prior to emitting the llvm > code in order to handle lots of things. Basically, you have to "play" the > byte-code on a simple machine anyway in order to emit the correct code. The > big thing about Cython is you have to typedef too many things that are really > quite knowable from the code. If Cython could improve it's type inference, > then it would be a more suitable target. > >> There are many other things, like >> bounds checking, wraparound, etc, that are all supported in both numpy >> and Cython, but going through an llvm layer would as far as I can see, >> require re-implementing those, at least if you want top-notch >> performance. Personally, I think for non-trivial performance-critical >> code (for loops with indexing, slicing, function calls, etc) Cython is >> a better target. > > With libclang it is really quite possible to imagine a cython -> C target > that itself compiles to llvm so that you can do everything at that > intermediate layer. However, LLVM is a much better layer for optimization > than C now that there are a lot of people collaborating on that layer. I > think it would be great if Cython targeted LLVM actually instead of C. > >> >> Finally, as for non-vector-expression code, I really believe Cython is >> a better target. cython.inline can have high overhead (at least the >> first time it has to compile), but with better (numpy-aware) type >> inference or profile guided optimizations (see recent threads on the >> cython-dev mailing list), in addition to things like prange, I >> personally believe Cython targets most of the use cases where numba >> would be able to generate performing code. > > Cython and Numba certainly overlap. However, Cython requires: > >1) learning another language >2) creating an extension module --- loading bit-code files and > dynamically executing (even on a different machine from the one that > initially created them) can be a powerful alternative for run-time > compilation and distribution of code. > > These aren't show-stoppers obviously. But, I think some users would prefer > an even simpler approach to getting fast-code than Cython (which currently > doesn't do enought type-inference and requires building a dlopen extension > module). Dag and I have been discussing this at PyCon, and here is my take on it (at this moment :). Definitely, if you can avoid Cython then that is easier and more desirable in many ways. So perhaps we can create a third project called X (I'm not very creative, maybe ArrayExprOpt), that takes an abstract syn
Re: [Numpy-discussion] Looking for people interested in helping with Python compiler to LLVM
Sorry, forgot to CC list on this. Lines staring with single greater-than are mine. -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. Dag Sverre Seljebotn wrote: Francesc Alted wrote: >On Mar 20, 2012, at 12:49 PM, mark florisson wrote: >>> Cython and Numba certainly overlap. However, Cython requires: >>> >>> 1) learning another language >>> 2) creating an extension module --- loading bit-code files >and dynamically executing (even on a different machine from the one >that initially created them) can be a powerful alternative for run-time >compilation and distribution of code. >>> >>> These aren't show-stoppers obviously. But, I think some users >would prefer an even simpler approach to getting fast-code than Cython >(which currently doesn't do enought type-inference and requires >building a dlopen extension module). >> >> Dag and I have been discussing this at PyCon, and here is my take on >> it (at this moment :). >> >> Definitely, if you can avoid Cython then that is easier and more >> desirable in many ways. So perhaps we can create a third project >> called X (I'm not very creative, maybe ArrayExprOpt), that takes an >> abstract syntax tree in a rather simple form, performs code >> optimizations such as rewriting loops with array accesses to vector >> expressions, fusing vector expressions and loops, etc, and spits out >a >> transformed AST containing these optimizations. If runtime >information >> is given such as actual shape and stride information the >> transformations could figure out there and then whether to do things >> like collapsing, axes swapping, blocking (as in, introducing more >axes >> or loops to retain discontiguous blocks in the cache), blocked memory >> copies to contiguous chunks, etc. The AST could then also say whether >> the final expressions are vectorizable. Part of this functionality is >> already in numpy's nditer, except that this would be implicit and do >> more (and hopefully with minimal overhead). >> >> So numba, Cython and maybe numexpr could use the functionality, >simply >> by building the AST from Python and converting back (if necessary) to >> its own AST. As such, the AST optimizer would be only part of any >> (runtime) compiler's pipeline, and it should be very flexible to >> retain any information (metadata regarding actual types, control flow >> information, etc) provided by the original AST. It would not do >> control flow analysis, type inference or promotion, etc, but only >deal >> with abstract types like integers, reals and arrays (C, Fortran or >> partly contiguous or strided). It would not deal with objects, but >> would allow to insert nodes like UnreorderableNode and SideEffectNode >> wrapping parts of the original AST. In short, it should be as easy as >> possible to convert from an original AST to this project's AST and >> back again afterwards. > >I think this is a very interesting project, and certainly projects like >numba can benefit of it. So, in order to us have an idea on what you >are after, can we assume that your project (call it X) would be kind of >an compiler optimizer, and then the produced, optimized code could be >feed into numba for optimized LLVM code generation (that on its turn, >can be run on top of CPUs or GPUs or a combination)? Is that correct? I think so. Another way of thinking about it is that it is a reimplementation of the logic in the (good and closed source) Fortran 90 compilers, in a reusable component for inclusion in various compilers. Various c++ metaprogramming libraries (like Blitz++) are similar too. > >Giving that my interpretation above is correct, it is bit more >difficult to me to see how your X project could be of benefit for >numexpr. In fact, I actually see this the other way round: once the >optimizer has discovered the vectorization parts, then go one step >further and generate code that uses numexpr automatically (call this, >vectorization through numexpr). This is what you mean, or I'm missing >something? No. I think in some ways this is a competitor to numexpr -- you would gut out the middle of numexpr and keep the frontend and backend, but use this to optimize iteration order and blocking strategies. I think the goal is for higher performance than what I understand numexpr can provide (in some cases, not all!). For instance, can numexpr deal well with a + a.T where a is a c-contiguous array? Any numpy-like iteration order will not work well, one needs to use higher-dimensional (eg 2D) blocking, not 1D blocking. (if numexpr can do this then great, the task might then reduce to refactoring numexpr so that cython and numba can use the same logic) Dag > >> As the project matures many optimizations may be added that deal with >> all sorts of loop restructuring and ways to efficiently utilize the >> cache as well as enable vectorization and possibly parallelism. >> Perhaps it could even generate a different AST depending on whether >> execution target the
Re: [Numpy-discussion] Looking for people interested in helping with Python compiler to LLVM
I doubt Theano is already as smart as you'd want it to be right now, however the core mechanisms are there to perform graph optimizations and move computations to GPU. It may save time to start from there instead of starting all over from scratch. I'm not sure though, but it looks like it would be worth considering it at least. -=- Olivier Le 20 mars 2012 15:40, Dag Sverre Seljebotn a écrit : > ** We talked some about Theano. There are some differences in project > goals which means that it makes sense to make this a seperate project: > Cython wants to use this to generate C code up front from the Cython AST at > compilation time; numba also has a different frontend (parsing of python > bytecode) and a different backend (LLVM). > > However, it may very well be possible that Theano could be refactored so > that the more essential algorithms working on the syntax tree could be > pulled out and shared with cython and numba. Then the question is whether > the core of Theano is smart enough to compete with Fortran compilers and > support arbitraily strided inputs optimally. Otherwise one might as well > start from scratch. I'll leave that for Mark to figure out... > > Dag > -- > Sent from my Android phone with K-9 Mail. Please excuse my brevity. > > > Olivier Delalleau wrote: >> >> This sounds a lot like Theano, did you look into it? >> >> -=- Olivier >> >> Le 20 mars 2012 13:49, mark florisson a >> écrit : >> >>> On 13 March 2012 18:18, Travis Oliphant wrote: >>> >>> >>> >>> (Mark F., how does the above match how you feel about this?) >>> >> >>> >> I would like collaboration, but from a technical perspective I think >>> >> this would be much more involved than just dumping the AST to an IR >>> >> and generating some code from there. For vector expressions I think >>> >> sharing code would be more feasible than arbitrary (parallel) loops, >>> >> etc. Cython as a compiler can make many decisions that a Python >>> >> (bytecode) compiler can't make (at least without annotations and a >>> >> well-defined subset of the language (not so much the syntax as the >>> >> semantics)). I think in numba, if parallelism is to be supported, you >>> >> will want a prange-like construct, as proving independence between >>> >> iterations can be very hard to near impossible for a compiler. >>> > >>> > I completely agree that you have to define some kind of syntax to get >>> parallelism. But, a prange construct would not be out of the question, of >>> course. >>> > >>> >> >>> >> As for code generation, I'm not sure how llvm would do things like >>> >> slicing arrays, reshaping, resizing etc (for vector expressions you >>> >> can first evaluate all slicing and indexing operations and then >>> >> compile the remaining vector expression), but for loops and array >>> >> reassignment within loops this would have to invoke the actual slicing >>> >> code from the llvm code (I presume). >>> > >>> > There could be some analysis on the byte-code, prior to emitting the >>> llvm code in order to handle lots of things. Basically, you have to >>> "play" the byte-code on a simple machine anyway in order to emit the >>> correct code. The big thing about Cython is you have to typedef too many >>> things that are really quite knowable from the code. If Cython could >>> improve it's type inference, then it would be a more suitable target. >>> > >>> >> There are many other things, like >>> >> bounds checking, wraparound, etc, that are all supported in both numpy >>> >> and Cython, but going through an llvm layer would as far as I can see, >>> >> require re-implementing those, at least if you want top-notch >>> >> performance. Personally, I think for non-trivial performance-critical >>> >> code (for loops with indexing, slicing, function calls, etc) Cython is >>> >> a better target. >>> > >>> > With libclang it is really quite possible to imagine a cython -> C >>> target that itself compiles to llvm so that you can do everything at that >>> intermediate layer. However, LLVM is a much better layer for >>> optimization than C now that there are a lot of people collaborating on >>> that layer. I think it would be great if Cython targeted LLVM actually >>> instead of C. >>> > >>> >> >>> >> Finally, as for non-vector-expression code, I really believe Cython is >>> >> a better target. cython.inline can have high overhead (at least the >>> >> first time it has to compile), but with better (numpy-aware) type >>> >> inference or profile guided optimizations (see recent threads on the >>> >> cython-dev mailing list), in addition to things like prange, I >>> >> personally believe Cython targets most of the use cases where numba >>> >> would be able to generate performing code. >>> > >>> > Cython and Numba certainly overlap. However, Cython requires: >>> > >>> >1) learning another language >>> >2) creating an extension module --- loading bit-code files and >>> dynamically executing (even on a different machine from the one that
Re: [Numpy-discussion] Looking for people interested in helping with Python compiler to LLVM
On Mar 20, 2012, at 2:29 PM, Dag Sverre Seljebotn wrote: > Francesc Alted wrote: > >> On Mar 20, 2012, at 12:49 PM, mark florisson wrote: Cython and Numba certainly overlap. However, Cython requires: 1) learning another language 2) creating an extension module --- loading bit-code files >> and dynamically executing (even on a different machine from the one >> that initially created them) can be a powerful alternative for run-time >> compilation and distribution of code. These aren't show-stoppers obviously. But, I think some users >> would prefer an even simpler approach to getting fast-code than Cython >> (which currently doesn't do enought type-inference and requires >> building a dlopen extension module). >>> >>> Dag and I have been discussing this at PyCon, and here is my take on >>> it (at this moment :). >>> >>> Definitely, if you can avoid Cython then that is easier and more >>> desirable in many ways. So perhaps we can create a third project >>> called X (I'm not very creative, maybe ArrayExprOpt), that takes an >>> abstract syntax tree in a rather simple form, performs code >>> optimizations such as rewriting loops with array accesses to vector >>> expressions, fusing vector expressions and loops, etc, and spits out >> a >>> transformed AST containing these optimizations. If runtime >> information >>> is given such as actual shape and stride information the >>> transformations could figure out there and then whether to do things >>> like collapsing, axes swapping, blocking (as in, introducing more >> axes >>> or loops to retain discontiguous blocks in the cache), blocked memory >>> copies to contiguous chunks, etc. The AST could then also say whether >>> the final expressions are vectorizable. Part of this functionality is >>> already in numpy's nditer, except that this would be implicit and do >>> more (and hopefully with minimal overhead). >>> >>> So numba, Cython and maybe numexpr could use the functionality, >> simply >>> by building the AST from Python and converting back (if necessary) to >>> its own AST. As such, the AST optimizer would be only part of any >>> (runtime) compiler's pipeline, and it should be very flexible to >>> retain any information (metadata regarding actual types, control flow >>> information, etc) provided by the original AST. It would not do >>> control flow analysis, type inference or promotion, etc, but only >> deal >>> with abstract types like integers, reals and arrays (C, Fortran or >>> partly contiguous or strided). It would not deal with objects, but >>> would allow to insert nodes like UnreorderableNode and SideEffectNode >>> wrapping parts of the original AST. In short, it should be as easy as >>> possible to convert from an original AST to this project's AST and >>> back again afterwards. >> >> I think this is a very interesting project, and certainly projects like >> numba can benefit of it. So, in order to us have an idea on what you >> are after, can we assume that your project (call it X) would be kind of >> an compiler optimizer, and then the produced, optimized code could be >> feed into numba for optimized LLVM code generation (that on its turn, >> can be run on top of CPUs or GPUs or a combination)? Is that correct? > > I think so. Another way of thinking about it is that it is a reimplementation > of the logic in the (good and closed source) Fortran 90 compilers, in a > reusable component for inclusion in various compilers. > > Various c++ metaprogramming libraries (like Blitz++) are similar too. Aha, thanks. >> Giving that my interpretation above is correct, it is bit more >> difficult to me to see how your X project could be of benefit for >> numexpr. In fact, I actually see this the other way round: once the >> optimizer has discovered the vectorization parts, then go one step >> further and generate code that uses numexpr automatically (call this, >> vectorization through numexpr). This is what you mean, or I'm missing >> something? > > No. I think in some ways this is a competitor to numexpr -- you would gut out > the middle of numexpr and keep the frontend and backend, but use this to > optimize iteration order and blocking strategies. I see. Yes, I can easily see Mark's project X + numba more as a competitor (killer?) to numexpr too. > > I think the goal is for higher performance than what I understand numexpr can > provide (in some cases, not all!). For instance, can numexpr deal well with > > a + a.T > > where a is a c-contiguous array? Any numpy-like iteration order will not work > well, one needs to use higher-dimensional (eg 2D) blocking, not 1D blocking. No. numexpr cannot deal with the above problem efficiently. numexpr is about 1d blocking, so its approach is pretty naive (but effective for these 1d blocking tasks). > (if numexpr can do this then great, the task might then reduce to refactoring > numexpr so that cython and numba can use the same logic)
Re: [Numpy-discussion] Trouble building NumPy on PPC64
On Mon, Mar 19, 2012 at 6:45 PM, Andreas H. wrote: > Hi all, > > I have troube installing numpy in a virtual environment on a SuSE > Enterprise 11 server (ppc64). > > Here is what I did: > >curl -O https://raw.github.com/pypa/virtualenv/master/virtualenv.py >python virtualenv.py --distribute --no-site-packages .virtualenvs/pydoas >source .virtualenvs/pydoas/bin/activate >pip install numpy > > And here is the outcome: > >SystemError: Cannot compile 'Python.h'. Perhaps you need to install > python-dev|python-devel. > > However, Python.h exists, because I did install the python-devel package: > >(pydoas)hilboll@odin:~/.virtualenvs/pydoas/build/numpy> find > /usr/include/ | grep Python.h >/usr/include/python2.6/Python.h > > I also tried without the --distribute --no-site-packages flags, with the > same result. > > Any hints are very welcome :) > Hard to say what's going wrong. Perhaps pip is not for python2.6 but another python version? Can you install without pip, so normal "python setup.py install"? Can you post the full build log? Ralf ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers
Warren et al:
On Wed, Mar 7, 2012 at 7:49 AM, Warren Weckesser
wrote:
> If you are setup with Cython to build extension modules,
I am
> and you don't mind
> testing an unreleased and experimental reader,
and I don't.
> you can try the text reader
> that I'm working on: https://github.com/WarrenWeckesser/textreader
It just took me a while to get around to it!
First of all: this is pretty much exactly what I've been looking for
for years, and never got around to writing myself - thanks!
My comments/suggestions:
1) a docstring for the textreader module would be nice.
2) "tzoffset" -- this is tricky stuff. Ideally, it should be able to
parse an ISO datetime string timezone specifier, but short of that, I
think the default should be None or UTC -- time zones are too ugly to
presume anything!
3) it breaks with the old MacOS style line endings: \r only. Maybe no
need to support that, but it turns out one of my old test files still
had them!
4) when I try to read more rows than are in the file, I get:
File "textreader.pyx", line 247, in textreader.readrows
(python/textreader.c:3469)
ValueError: negative dimensions are not allowed
good to get an error, but it's not very informative!
5) for reading float64 values -- I get something different with
textreader than with the python "float()":
input: "678.901"
float("") : 678.900995
textreader : 678.901007
as close as the number of figures available, but curious...
5) Performance issue: in my case, I'm reading a big file that's in
chunks -- each one has a header indicating how many rows follow, then
the rows, so I parse it out bit by bit.
For smallish files, it's much faster than pure python, and almost as
fast as some old C code of mine that is far less flexible.
But for large files, -- it's much slower -- indeed slower than a pure
python version for my use case.
I did a simplified test -- with 10,000 rows:
total number of rows: 1
pure python took: 1.410408 seconds
pure python chunks took: 1.613094 seconds
textreader all at once took: 0.067098 seconds
textreader in chunks took : 0.131802 seconds
but with 1,000,000 rows:
total number of rows: 100
total number of chunks: 1000
pure python took: 30.712564 seconds
pure python chunks took: 31.313225 seconds
textreader all at once took: 1.314924 seconds
textreader in chunks took : 9.684819 seconds
then it gets even worse with the chunk size smaller:
total number of rows: 100
total number of chunks: 1
pure python took: 30.032246 seconds
pure python chunks took: 42.010589 seconds
textreader all at once took: 1.318613 seconds
textreader in chunks took : 87.743729 seconds
my code, which is C that essentially runs fscanf over the file, has
essentially no performance hit from doing it in chunks -- so I think
something is wrong here.
Sorry, I haven't dug into the code to try to figure out what yet --
does it rewind the file each time maybe?
Enclosed is my test code.
-Chris
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
chris.bar...@noaa.gov
test_performance.py
Description: Binary data
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Testsuite fails with Python 2.7.3rc1 and 3.2.3rc1 (Debian)
Hello, I've reported http://projects.scipy.org/numpy/ticket/2085 and Ralf asked for bringing that up here: is anyone able to replicate the problem described in that ticket? The debian bug tracking the problem is: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=664672 Cheers, -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Looking for people interested in helping with Python compiler to LLVM
On 03/20/2012 12:56 PM, Francesc Alted wrote: > On Mar 20, 2012, at 2:29 PM, Dag Sverre Seljebotn wrote: >> Francesc Alted wrote: >> >>> On Mar 20, 2012, at 12:49 PM, mark florisson wrote: > Cython and Numba certainly overlap. However, Cython requires: > >1) learning another language >2) creating an extension module --- loading bit-code files >>> and dynamically executing (even on a different machine from the one >>> that initially created them) can be a powerful alternative for run-time >>> compilation and distribution of code. > > These aren't show-stoppers obviously. But, I think some users >>> would prefer an even simpler approach to getting fast-code than Cython >>> (which currently doesn't do enought type-inference and requires >>> building a dlopen extension module). Dag and I have been discussing this at PyCon, and here is my take on it (at this moment :). Definitely, if you can avoid Cython then that is easier and more desirable in many ways. So perhaps we can create a third project called X (I'm not very creative, maybe ArrayExprOpt), that takes an abstract syntax tree in a rather simple form, performs code optimizations such as rewriting loops with array accesses to vector expressions, fusing vector expressions and loops, etc, and spits out >>> a transformed AST containing these optimizations. If runtime >>> information is given such as actual shape and stride information the transformations could figure out there and then whether to do things like collapsing, axes swapping, blocking (as in, introducing more >>> axes or loops to retain discontiguous blocks in the cache), blocked memory copies to contiguous chunks, etc. The AST could then also say whether the final expressions are vectorizable. Part of this functionality is already in numpy's nditer, except that this would be implicit and do more (and hopefully with minimal overhead). So numba, Cython and maybe numexpr could use the functionality, >>> simply by building the AST from Python and converting back (if necessary) to its own AST. As such, the AST optimizer would be only part of any (runtime) compiler's pipeline, and it should be very flexible to retain any information (metadata regarding actual types, control flow information, etc) provided by the original AST. It would not do control flow analysis, type inference or promotion, etc, but only >>> deal with abstract types like integers, reals and arrays (C, Fortran or partly contiguous or strided). It would not deal with objects, but would allow to insert nodes like UnreorderableNode and SideEffectNode wrapping parts of the original AST. In short, it should be as easy as possible to convert from an original AST to this project's AST and back again afterwards. >>> >>> I think this is a very interesting project, and certainly projects like >>> numba can benefit of it. So, in order to us have an idea on what you >>> are after, can we assume that your project (call it X) would be kind of >>> an compiler optimizer, and then the produced, optimized code could be >>> feed into numba for optimized LLVM code generation (that on its turn, >>> can be run on top of CPUs or GPUs or a combination)? Is that correct? >> >> I think so. Another way of thinking about it is that it is a >> reimplementation of the logic in the (good and closed source) Fortran 90 >> compilers, in a reusable component for inclusion in various compilers. >> >> Various c++ metaprogramming libraries (like Blitz++) are similar too. > > Aha, thanks. > >>> Giving that my interpretation above is correct, it is bit more >>> difficult to me to see how your X project could be of benefit for >>> numexpr. In fact, I actually see this the other way round: once the >>> optimizer has discovered the vectorization parts, then go one step >>> further and generate code that uses numexpr automatically (call this, >>> vectorization through numexpr). This is what you mean, or I'm missing >>> something? >> >> No. I think in some ways this is a competitor to numexpr -- you would gut >> out the middle of numexpr and keep the frontend and backend, but use this to >> optimize iteration order and blocking strategies. > > I see. Yes, I can easily see Mark's project X + numba more as a competitor > (killer?) to numexpr too. > >> >> I think the goal is for higher performance than what I understand numexpr >> can provide (in some cases, not all!). For instance, can numexpr deal well >> with >> >> a + a.T >> >> where a is a c-contiguous array? Any numpy-like iteration order will not >> work well, one needs to use higher-dimensional (eg 2D) blocking, not 1D >> blocking. > > No. numexpr cannot deal with the above problem efficiently. numexpr is > about 1d blocking, so its approach is pretty naive (but effective for these > 1d blocking task
Re: [Numpy-discussion] Looking for people interested in helping with Python compiler to LLVM
On 03/20/2012 09:20 PM, Dag Sverre Seljebotn wrote: > On 03/20/2012 12:56 PM, Francesc Alted wrote: >> On Mar 20, 2012, at 2:29 PM, Dag Sverre Seljebotn wrote: >>> Francesc Alted wrote: >>> On Mar 20, 2012, at 12:49 PM, mark florisson wrote: >> Cython and Numba certainly overlap. However, Cython requires: >> >> 1) learning another language >> 2) creating an extension module --- loading bit-code files and dynamically executing (even on a different machine from the one that initially created them) can be a powerful alternative for run-time compilation and distribution of code. >> >> These aren't show-stoppers obviously. But, I think some users would prefer an even simpler approach to getting fast-code than Cython (which currently doesn't do enought type-inference and requires building a dlopen extension module). > > Dag and I have been discussing this at PyCon, and here is my take on > it (at this moment :). > > Definitely, if you can avoid Cython then that is easier and more > desirable in many ways. So perhaps we can create a third project > called X (I'm not very creative, maybe ArrayExprOpt), that takes an > abstract syntax tree in a rather simple form, performs code > optimizations such as rewriting loops with array accesses to vector > expressions, fusing vector expressions and loops, etc, and spits out a > transformed AST containing these optimizations. If runtime information > is given such as actual shape and stride information the > transformations could figure out there and then whether to do things > like collapsing, axes swapping, blocking (as in, introducing more axes > or loops to retain discontiguous blocks in the cache), blocked memory > copies to contiguous chunks, etc. The AST could then also say whether > the final expressions are vectorizable. Part of this functionality is > already in numpy's nditer, except that this would be implicit and do > more (and hopefully with minimal overhead). > > So numba, Cython and maybe numexpr could use the functionality, simply > by building the AST from Python and converting back (if necessary) to > its own AST. As such, the AST optimizer would be only part of any > (runtime) compiler's pipeline, and it should be very flexible to > retain any information (metadata regarding actual types, control flow > information, etc) provided by the original AST. It would not do > control flow analysis, type inference or promotion, etc, but only deal > with abstract types like integers, reals and arrays (C, Fortran or > partly contiguous or strided). It would not deal with objects, but > would allow to insert nodes like UnreorderableNode and SideEffectNode > wrapping parts of the original AST. In short, it should be as easy as > possible to convert from an original AST to this project's AST and > back again afterwards. I think this is a very interesting project, and certainly projects like numba can benefit of it. So, in order to us have an idea on what you are after, can we assume that your project (call it X) would be kind of an compiler optimizer, and then the produced, optimized code could be feed into numba for optimized LLVM code generation (that on its turn, can be run on top of CPUs or GPUs or a combination)? Is that correct? >>> >>> I think so. Another way of thinking about it is that it is a >>> reimplementation of the logic in the (good and closed source) Fortran 90 >>> compilers, in a reusable component for inclusion in various compilers. >>> >>> Various c++ metaprogramming libraries (like Blitz++) are similar too. >> >> Aha, thanks. >> Giving that my interpretation above is correct, it is bit more difficult to me to see how your X project could be of benefit for numexpr. In fact, I actually see this the other way round: once the optimizer has discovered the vectorization parts, then go one step further and generate code that uses numexpr automatically (call this, vectorization through numexpr). This is what you mean, or I'm missing something? >>> >>> No. I think in some ways this is a competitor to numexpr -- you would gut >>> out the middle of numexpr and keep the frontend and backend, but use this >>> to optimize iteration order and blocking strategies. >> >> I see. Yes, I can easily see Mark's project X + numba more as a competitor >> (killer?) to numexpr too. >> >>> >>> I think the goal is for higher performance than what I understand numexpr >>> can provide (in some cases, not all!). For instance, can numexpr deal well >>> with >>> >>> a + a.T >>> >>> where a is a c-contiguous array? Any numpy-like iteration order will not >>> work well, one needs to use higher-dimensional (eg 2D) blocking, not 1D >>> blocking. >> >> No. numexpr cann
Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers
On Tue, Mar 20, 2012 at 5:59 PM, Chris Barker wrote:
> Warren et al:
>
> On Wed, Mar 7, 2012 at 7:49 AM, Warren Weckesser
> wrote:
> > If you are setup with Cython to build extension modules,
>
> I am
>
> > and you don't mind
> > testing an unreleased and experimental reader,
>
> and I don't.
>
> > you can try the text reader
> > that I'm working on: https://github.com/WarrenWeckesser/textreader
>
> It just took me a while to get around to it!
>
> First of all: this is pretty much exactly what I've been looking for
> for years, and never got around to writing myself - thanks!
>
> My comments/suggestions:
>
> 1) a docstring for the textreader module would be nice.
>
> 2) "tzoffset" -- this is tricky stuff. Ideally, it should be able to
> parse an ISO datetime string timezone specifier, but short of that, I
> think the default should be None or UTC -- time zones are too ugly to
> presume anything!
>
> 3) it breaks with the old MacOS style line endings: \r only. Maybe no
> need to support that, but it turns out one of my old test files still
> had them!
>
> 4) when I try to read more rows than are in the file, I get:
> File "textreader.pyx", line 247, in textreader.readrows
> (python/textreader.c:3469)
> ValueError: negative dimensions are not allowed
>
> good to get an error, but it's not very informative!
>
> 5) for reading float64 values -- I get something different with
> textreader than with the python "float()":
> input: "678.901"
> float("") : 678.900995
> textreader : 678.901007
>
> as close as the number of figures available, but curious...
>
>
> 5) Performance issue: in my case, I'm reading a big file that's in
> chunks -- each one has a header indicating how many rows follow, then
> the rows, so I parse it out bit by bit.
> For smallish files, it's much faster than pure python, and almost as
> fast as some old C code of mine that is far less flexible.
>
> But for large files, -- it's much slower -- indeed slower than a pure
> python version for my use case.
>
> I did a simplified test -- with 10,000 rows:
>
> total number of rows: 1
> pure python took: 1.410408 seconds
> pure python chunks took: 1.613094 seconds
> textreader all at once took: 0.067098 seconds
> textreader in chunks took : 0.131802 seconds
>
> but with 1,000,000 rows:
>
> total number of rows: 100
> total number of chunks: 1000
> pure python took: 30.712564 seconds
> pure python chunks took: 31.313225 seconds
> textreader all at once took: 1.314924 seconds
> textreader in chunks took : 9.684819 seconds
>
> then it gets even worse with the chunk size smaller:
>
> total number of rows: 100
> total number of chunks: 1
> pure python took: 30.032246 seconds
> pure python chunks took: 42.010589 seconds
> textreader all at once took: 1.318613 seconds
> textreader in chunks took : 87.743729 seconds
>
> my code, which is C that essentially runs fscanf over the file, has
> essentially no performance hit from doing it in chunks -- so I think
> something is wrong here.
>
> Sorry, I haven't dug into the code to try to figure out what yet --
> does it rewind the file each time maybe?
>
> Enclosed is my test code.
>
> -Chris
>
Chris,
Thanks! The feedback is great. I won't have time to get back to this for
another week or so, but then I'll look into the issues you reported.
Warren
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R(206) 526-6959 voice
> 7600 Sand Point Way NE (206) 526-6329 fax
> Seattle, WA 98115 (206) 526-6317 main reception
>
> chris.bar...@noaa.gov
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
