Re: [Numpy-discussion] Question on numpy.ma.masked_values

2012-03-20 Thread Pierre GM
Gökhan, By default, the mask of a MaskedArray is set to the special value `np.ma.nomask`. In other terms:: np.ma.array(...) = np.ma.array(..., mask=np.ma.nomask) In practice, np.ma.nomask lets us quickly check whether a MaskedArray has a masked value : if its .mask is np.ma.nomask, then no

Re: [Numpy-discussion] Recovering from a STOP ?

2012-03-20 Thread Pierre GM
Pauli, Chris, Thanks for your inputs. Pauli, I think that when f2py encounters a STOP statement, it just stops the execution of the process. Alas, it's the same process as the interpreter... So we need a trick not to interrupt the whole process. I eventually resorted to patching f2py as

Re: [Numpy-discussion] NumPy-Discussion Digest, Vol 66, Issue 61

2012-03-20 Thread Matthieu Rigal
Hi Richard, Thanks for your answer and the related help ! In fact, I was hoping to have a less memory and more speed solution. Something equivalent to a raster calculator for numpy. Wouldn't it make sense to have some optimized function to work on more than 2 arrays for numpy anyway ? At the

[Numpy-discussion] Trying to read 500M txt file using numpy.genfromtxt within ipython shell

2012-03-20 Thread Chao YUE
Dear all, I received a file from others which contains ~30 million lines and in size of ~500M. I try read it with numpy.genfromtxt in ipython interactive mode. Then ipython crashed. The data contains lat,lon,var1,year, the year ranges from 1001 to 2006. Finally I want to write the data to netcdf

Re: [Numpy-discussion] Using logical function on more than 2 arrays, availability of a between function ?

2012-03-20 Thread Matthieu Rigal
Auto-answer, sorry, Well, actually I made a mistake lower... that you may have noticed... On the faster (your) solution, even with a cleaner use of the out parameter, the fact that the all has then to be used with parameter axis=0 takes more time and makes it actually slower than the initial

Re: [Numpy-discussion] Trying to read 500M txt file using numpy.genfromtxt within ipython shell

2012-03-20 Thread David Froger
Hi, I think writing a Python script that convert your txt file to one netcdf file, reading the txt file one line at a time, and then use the netcdf file normally would be a good solution! Best, David Excerpts from Chao YUE's message of mar. mars 20 13:33:56 +0100 2012: Dear all, I received

Re: [Numpy-discussion] Trying to read 500M txt file using numpy.genfromtxt within ipython shell

2012-03-20 Thread Chao YUE
I would be in agree. thanks! I use gawk to separate the file into many files by year, then it would be easier to handle. anyway, it's not a good practice to produce such huge line txt files Chao 2012/3/20 David Froger david.fro...@gmail.com Hi, I think writing a Python script that

Re: [Numpy-discussion] Trying to read 500M txt file using numpy.genfromtxt within ipython shell

2012-03-20 Thread Derek Homeier
On 20 Mar 2012, at 14:40, Chao YUE wrote: I would be in agree. thanks! I use gawk to separate the file into many files by year, then it would be easier to handle. anyway, it's not a good practice to produce such huge line txt files Indeed it's not, but it's also not good practice to

Re: [Numpy-discussion] Using logical function on more than 2 arrays, availability of a between function ?

2012-03-20 Thread Frédéric Bastien
I didn't try it, but I think that Theano and numexpr should be able to make them faster. [1] http://deeplearning.net/software/theano/ [2] https://code.google.com/p/numexpr/ Fred On Tue, Mar 20, 2012 at 9:05 AM, Matthieu Rigal ri...@rapideye.net wrote: Auto-answer, sorry, Well, actually I

Re: [Numpy-discussion] Looking for people interested in helping with Python compiler to LLVM

2012-03-20 Thread mark florisson
On 13 March 2012 18:18, Travis Oliphant tra...@continuum.io wrote: (Mark F., how does the above match how you feel about this?) I would like collaboration, but from a technical perspective I think this would be much more involved than just dumping the AST to an IR and generating some code

Re: [Numpy-discussion] NumPy-Discussion Digest, Vol 66, Issue 61

2012-03-20 Thread Chris Barker
On Tue, Mar 20, 2012 at 5:13 AM, Matthieu Rigal ri...@rapideye.net wrote: In fact, I was hoping to have a less memory and more speed solution. which do often go together, at least for big problems -- pushingm emory around often takes more time than the computation itself. At the end, I am

Re: [Numpy-discussion] Looking for people interested in helping with Python compiler to LLVM

2012-03-20 Thread Olivier Delalleau
This sounds a lot like Theano, did you look into it? -=- Olivier Le 20 mars 2012 13:49, mark florisson markflorisso...@gmail.com a écrit : On 13 March 2012 18:18, Travis Oliphant tra...@continuum.io wrote: (Mark F., how does the above match how you feel about this?) I would like

Re: [Numpy-discussion] Question on numpy.ma.masked_values

2012-03-20 Thread Gökhan Sever
Yes, that's the behaviour that I expect setting the 'shrink' keyword to 'False' Now, just to be clear, you'd want 'np.ma.masked_values(...,shrink=False) to create a maked array w/ a full boolean mask by default, right ? ___ NumPy-Discussion mailing

Re: [Numpy-discussion] Looking for people interested in helping with Python compiler to LLVM

2012-03-20 Thread Francesc Alted
On Mar 20, 2012, at 12:49 PM, mark florisson wrote: Cython and Numba certainly overlap. However, Cython requires: 1) learning another language 2) creating an extension module --- loading bit-code files and dynamically executing (even on a different machine from the one that

Re: [Numpy-discussion] Looking for people interested in helping with Python compiler to LLVM

2012-03-20 Thread Dag Sverre Seljebotn
We talked some about Theano. There are some differences in project goals which means that it makes sense to make this a seperate project: Cython wants to use this to generate C code up front from the Cython AST at compilation time; numba also has a different frontend (parsing of python

Re: [Numpy-discussion] Looking for people interested in helping with Python compiler to LLVM

2012-03-20 Thread Dag Sverre Seljebotn
Sorry, forgot to CC list on this. Lines staring with single greater-than are mine. -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. Dag Sverre Seljebotn d.s.seljeb...@astro.uio.no wrote: Francesc Alted franc...@continuum.io wrote: On Mar 20, 2012, at 12:49 PM, mark

Re: [Numpy-discussion] Looking for people interested in helping with Python compiler to LLVM

2012-03-20 Thread Olivier Delalleau
I doubt Theano is already as smart as you'd want it to be right now, however the core mechanisms are there to perform graph optimizations and move computations to GPU. It may save time to start from there instead of starting all over from scratch. I'm not sure though, but it looks like it would be

Re: [Numpy-discussion] Looking for people interested in helping with Python compiler to LLVM

2012-03-20 Thread Francesc Alted
On Mar 20, 2012, at 2:29 PM, Dag Sverre Seljebotn wrote: Francesc Alted franc...@continuum.io wrote: On Mar 20, 2012, at 12:49 PM, mark florisson wrote: Cython and Numba certainly overlap. However, Cython requires: 1) learning another language 2) creating an extension module

Re: [Numpy-discussion] Trouble building NumPy on PPC64

2012-03-20 Thread Ralf Gommers
On Mon, Mar 19, 2012 at 6:45 PM, Andreas H. li...@hilboll.de wrote: Hi all, I have troube installing numpy in a virtual environment on a SuSE Enterprise 11 server (ppc64). Here is what I did: curl -O https://raw.github.com/pypa/virtualenv/master/virtualenv.py python virtualenv.py

Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers

2012-03-20 Thread Chris Barker
Warren et al: On Wed, Mar 7, 2012 at 7:49 AM, Warren Weckesser warren.weckes...@enthought.com wrote: If you are setup with Cython to build extension modules, I am and you don't mind testing an unreleased and experimental reader, and I don't. you can try the text reader that I'm working

[Numpy-discussion] Testsuite fails with Python 2.7.3rc1 and 3.2.3rc1 (Debian)

2012-03-20 Thread Sandro Tosi
Hello, I've reported http://projects.scipy.org/numpy/ticket/2085 and Ralf asked for bringing that up here: is anyone able to replicate the problem described in that ticket? The debian bug tracking the problem is: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=664672 Cheers, -- Sandro Tosi

Re: [Numpy-discussion] Looking for people interested in helping with Python compiler to LLVM

2012-03-20 Thread Dag Sverre Seljebotn
On 03/20/2012 12:56 PM, Francesc Alted wrote: On Mar 20, 2012, at 2:29 PM, Dag Sverre Seljebotn wrote: Francesc Altedfranc...@continuum.io wrote: On Mar 20, 2012, at 12:49 PM, mark florisson wrote: Cython and Numba certainly overlap. However, Cython requires: 1) learning another

Re: [Numpy-discussion] Looking for people interested in helping with Python compiler to LLVM

2012-03-20 Thread Dag Sverre Seljebotn
On 03/20/2012 09:20 PM, Dag Sverre Seljebotn wrote: On 03/20/2012 12:56 PM, Francesc Alted wrote: On Mar 20, 2012, at 2:29 PM, Dag Sverre Seljebotn wrote: Francesc Altedfranc...@continuum.io wrote: On Mar 20, 2012, at 12:49 PM, mark florisson wrote: Cython and Numba certainly overlap.

Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers

2012-03-20 Thread Warren Weckesser
On Tue, Mar 20, 2012 at 5:59 PM, Chris Barker chris.bar...@noaa.gov wrote: Warren et al: On Wed, Mar 7, 2012 at 7:49 AM, Warren Weckesser warren.weckes...@enthought.com wrote: If you are setup with Cython to build extension modules, I am and you don't mind testing an unreleased and