Re: [Numpy-discussion] argwhere does not accept py list

2009-07-08 Thread Scott Sinclair
2009/7/8 Robert Kern robert.k...@gmail.com: 2009/7/4 Stéfan van der Walt ste...@sun.ac.za: Thanks, Scott.  This should now be fixed in SVN. You should probably change that to asanyarray() before the masked array crowd gets upset. :-) I hadn't thought about that, but I'm don't think it

Re: [Numpy-discussion] Mathematica vs Matlab vs Python

2009-07-08 Thread Pauli Virtanen
Tue, 07 Jul 2009 21:30:06 -0500, alan kirjoitti: Mathematica vs Matlab vs Python http://www.larssono.com/musings/matmatpy/index.html The Python code there is not very idiomatic Numpy code. It's written for Numeric, and fancy indexing etc. are not used. Seems like the author also left it as

Re: [Numpy-discussion] Question for David re npy_funcs

2009-07-08 Thread David Cournapeau
On Wed, Jul 8, 2009 at 5:37 AM, Charles R Harrischarlesr.har...@gmail.com wrote: David, Should any standard c functions used in loops.c.src be the npy_* version? I've been using fabs, but I'm wondering if that should be npy_fabs. Yes. Although fabs is available on any platform in theory, we

[Numpy-discussion] Help with np.where and datetime functions

2009-07-08 Thread John [H2O]
Hello, I have several issues which require me to iterate through a fairly large array (30+ records). The first case is calculating and hourly average from non-regularly sampled data. The second is screening one array, based on data in the second array. The functions are defined below, but

Re: [Numpy-discussion] RFC: add a install_clib numpy.distutils command to install pure C libraries

2009-07-08 Thread David Cournapeau
On Fri, Jun 12, 2009 at 7:46 PM, David Cournapeauda...@ar.media.kyoto-u.ac.jp wrote: Hi,    I have finally spent some time so that we can install pure C libraries using numpy.distutils. With this, one could imagine having a C library for fft, special functions in numpy or scipy, so that the

Re: [Numpy-discussion] roots and high-order polynomial

2009-07-08 Thread Fabrice Silva
Le lundi 06 juillet 2009 à 17:57 +0200, Fabrice Silva a écrit : Le lundi 06 juillet 2009 à 17:13 +0200, Nils Wagner a écrit : IIRC, the coefficients of your polynomial are complex. So, you cannot guarantee that the roots are complex conjugate pairs. Correct! If the construction is done

Re: [Numpy-discussion] Help with np.where and datetime functions

2009-07-08 Thread Neil Crighton
John [H2O] washakie at gmail.com writes: What I am trying to do (obviously?) is find all the values of X that fall within a time range. Specifically, one point I do not understand is why the following two methods fail: -- 196 ind = np.where( (t1 Y[:,0] t2) ) #same result

[Numpy-discussion] performing operations in-place in numpy

2009-07-08 Thread Citi, Luca
Hello guys, I made a patch for numpy which allows performing operations in-place to save memory allocations. For example 'sqrt(a**2 + b**2)' can be performed allocating only two arrays instead of four. You find the details in ticket 1153 of numpy-core. I thought maybe you could be interested. I am

Re: [Numpy-discussion] Creating PyArrayObject in C++

2009-07-08 Thread Kenny Abernathy
Thanks, you two. That helps alot. The PyCObject_FromVoidPtr() trick is good to know--I still have to have my class because it holds other data, but this definitely points me in the right direction. On Tue, Jul 7, 2009 at 9:55 PM, Lisandro Dalcin dalc...@gmail.com wrote: 2009/7/7 Stéfan van

Re: [Numpy-discussion] argwhere does not accept py list

2009-07-08 Thread Pierre GM
On Jul 8, 2009, at 3:18 AM, Scott Sinclair wrote: 2009/7/8 Robert Kern robert.k...@gmail.com: 2009/7/4 Stéfan van der Walt ste...@sun.ac.za: Thanks, Scott. This should now be fixed in SVN. You should probably change that to asanyarray() before the masked array crowd gets upset. :-) I

Re: [Numpy-discussion] Help with np.where and datetime functions

2009-07-08 Thread Pierre GM
On Jul 8, 2009, at 7:03 AM, John [H2O] wrote: Hello, I have several issues which require me to iterate through a fairly large array (30+ records). The first case is calculating and hourly average from non-regularly sampled data. Would you like to give the scikits.timeseries

Re: [Numpy-discussion] Mathematica vs Matlab vs Python

2009-07-08 Thread Matthew Brett
Hi, Yup. It's not even very idiomatic Python. readlines() is probably a bad idea unless your file is trivial length, and even ignoring numpy.loadtxt(), all of this could be considerably simplified with the built-in csv module. or a 1-liner with scipy.io.loadmat ... Best, Matthew

Re: [Numpy-discussion] Mathematica vs Matlab vs Python

2009-07-08 Thread Gael Varoquaux
On Wed, Jul 08, 2009 at 12:48:17PM -0700, Matthew Brett wrote: Yup. It's not even very idiomatic Python. readlines() is probably a bad idea unless your file is trivial length, and even ignoring numpy.loadtxt(), all of this could be considerably simplified with the built-in csv module. or

Re: [Numpy-discussion] performing operations in-place in numpy

2009-07-08 Thread Stéfan van der Walt
Hi Luca 2009/7/8 Citi, Luca lc...@essex.ac.uk: Hello guys, I made a patch for numpy which allows performing operations in-place to save memory allocations. For example 'sqrt(a**2 + b**2)' can be performed allocating only two arrays instead of four. You find the details in ticket 1153 of

Re: [Numpy-discussion] argwhere does not accept py list

2009-07-08 Thread Stéfan van der Walt
2009/7/8 Robert Kern robert.k...@gmail.com: 2009/7/4 Stéfan van der Walt ste...@sun.ac.za: Thanks, Scott.  This should now be fixed in SVN. You should probably change that to asanyarray() before the masked array crowd gets upset. :-) Thanks, fixed! Stéfan

Re: [Numpy-discussion] Mathematica vs Matlab vs Python

2009-07-08 Thread Stéfan van der Walt
2009/7/8 Gael Varoquaux gael.varoqu...@normalesup.org: On Wed, Jul 08, 2009 at 12:48:17PM -0700, Matthew Brett wrote: Yup. It's not even very idiomatic Python. readlines() is probably a bad idea unless your file is trivial length, and even ignoring numpy.loadtxt(), all of this could be

Re: [Numpy-discussion] Problem with ML subscriptions?

2009-07-08 Thread Fons Adriaensen
On Sun, Jul 05, 2009 at 12:22:37AM +0200, Stéfan van der Walt wrote: 2009/7/5 Pauli Virtanen pav...@iki.fi: 2009-07-04 22:52 +0200, Fons Adriaensen f...@kokkinizita.net wrote: [clip] I subscribed to numpy-discussion almost two days ago. I do receive messages from the list, but the ones I

Re: [Numpy-discussion] performing operations in-place in numpy

2009-07-08 Thread Charles R Harris
On Wed, Jul 8, 2009 at 11:34 AM, Citi, Luca lc...@essex.ac.uk wrote: Hello guys, I made a patch for numpy which allows performing operations in-place to save memory allocations. For example 'sqrt(a**2 + b**2)' can be performed I think this particular function is already available as a

Re: [Numpy-discussion] performing operations in-place in numpy

2009-07-08 Thread Citi, Luca
Hi Stefan, I am afraid I did not explain myself clear enough. Of course c = a + b + d leaves a, b, and d unchanged. The only array that is overwritten is (a+b) which is a temporary array that would be destroyed anyway. Normally the operation above is performed like this: 1) allocation of a

Re: [Numpy-discussion] Problem with ML subscriptions?

2009-07-08 Thread Peter
On Wed, Jul 8, 2009 at 10:00 PM, Fons Adriaensenf...@kokkinizita.net wrote: On Sun, Jul 05, 2009 at 12:22:37AM +0200, Stéfan van der Walt wrote: 2009/7/5 Pauli Virtanen pav...@iki.fi: 2009-07-04 22:52 +0200, Fons Adriaensen f...@kokkinizita.net wrote: [clip] I subscribed to

Re: [Numpy-discussion] performing operations in-place in numpy

2009-07-08 Thread Charles R Harris
On Wed, Jul 8, 2009 at 3:57 PM, Citi, Luca lc...@essex.ac.uk wrote: Hi Stefan, I am afraid I did not explain myself clear enough. Of course c = a + b + d leaves a, b, and d unchanged. The only array that is overwritten is (a+b) which is a temporary array that would be destroyed anyway.

Re: [Numpy-discussion] performing operations in-place in numpy

2009-07-08 Thread Citi, Luca
@Charles R Harris For example 'sqrt(a**2 + b**2)' can be performed... I think this particular function is already available as a ufunc. I am not sure it is implemented as ufunc. But in any case it was just an example. Another example is sin(2*pi*w+phi) that is currently implemented

Re: [Numpy-discussion] performing operations in-place in numpy

2009-07-08 Thread Robert Kern
On Wed, Jul 8, 2009 at 17:10, Citi, Lucalc...@essex.ac.uk wrote: @Charles R Harris For example 'sqrt(a**2 + b**2)' can be performed... I think this particular function is already available as a ufunc. I am not sure it is implemented as ufunc. hypot(a, b) -- Robert Kern I have come to

Re: [Numpy-discussion] performing operations in-place in numpy

2009-07-08 Thread Charles R Harris
On Wed, Jul 8, 2009 at 4:10 PM, Citi, Luca lc...@essex.ac.uk wrote: @Charles R Harris For example 'sqrt(a**2 + b**2)' can be performed... I think this particular function is already available as a ufunc. I am not sure it is implemented as ufunc. But in any case it was just an example.

[Numpy-discussion] Optimizing reduction loops (sum(), prod(), et al.)

2009-07-08 Thread Pauli Virtanen
Hi, Ticket #1143 points out that Numpy's reduction operations are not always cache friendly. I worked a bit on tuning them. Just to tickle some interest, a pathological case before optimization: In [1]: import numpy as np In [2]: x = np.zeros((8, 256)) In [3]: %timeit

Re: [Numpy-discussion] performing operations in-place in numpy

2009-07-08 Thread Citi, Luca
On thing to keep in mind is that the inputs might be different views of the same array so the elements might accessed in an unexpected order. Only inputs owning their own data and with refcount 1 (i.e. no other array can be a view of it) are re-used as outputs.

Re: [Numpy-discussion] Optimizing reduction loops (sum(), prod(), et al.)

2009-07-08 Thread Charles R Harris
On Wed, Jul 8, 2009 at 4:16 PM, Pauli Virtanen pav...@iki.fipav%2...@iki.fi wrote: Hi, Ticket #1143 points out that Numpy's reduction operations are not always cache friendly. I worked a bit on tuning them. Just to tickle some interest, a pathological case before optimization: In

Re: [Numpy-discussion] performing operations in-place in numpy

2009-07-08 Thread Charles R Harris
On Wed, Jul 8, 2009 at 4:17 PM, Citi, Luca lc...@essex.ac.uk wrote: On thing to keep in mind is that the inputs might be different views of the same array so the elements might accessed in an unexpected order. Only inputs owning their own data and with refcount 1 (i.e. no other array can

Re: [Numpy-discussion] Optimizing reduction loops (sum(), prod(), et al.)

2009-07-08 Thread Stéfan van der Walt
Hi Pauli 2009/7/9 Pauli Virtanen pav...@iki.fi: Unfortunately, improving the performance using the above scheme comes at the cost of some slightly murky heuristics.  I didn't manage to come up with an optimal decision rule, so they are partly empirical. There is one parameter tuning the

Re: [Numpy-discussion] Optimizing reduction loops (sum(), prod(), et al.)

2009-07-08 Thread Pauli Virtanen
On 2009-07-08, Charles R Harris charlesr.har...@gmail.com wrote: [clip] How do the benchmarks compare with just making contiguous copies? Which is blocking of sort, I suppose. I think that's slower than just walking over the discontiguous array: 1) The new code: (on the Athlon

Re: [Numpy-discussion] Problem with ML subscriptions?

2009-07-08 Thread Fons Adriaensen
On Wed, Jul 08, 2009 at 11:01:55PM +0100, Peter wrote: On Wed, Jul 8, 2009 at 10:00 PM, Fons Adriaensenf...@kokkinizita.net wrote: On Sun, Jul 05, 2009 at 12:22:37AM +0200, Stéfan van der Walt wrote: 2009/7/5 Pauli Virtanen pav...@iki.fi: 2009-07-04 22:52 +0200, Fons Adriaensen

Re: [Numpy-discussion] Problem with ML subscriptions?

2009-07-08 Thread Robert Kern
On Wed, Jul 8, 2009 at 17:53, Fons Adriaensenf...@kokkinizita.net wrote: On Wed, Jul 08, 2009 at 11:01:55PM +0100, Peter wrote: Anyway - fingers crossed the list is working for you now... I'm not convinced... Will try posting again. And here you are. -- Robert Kern I have come to believe

Re: [Numpy-discussion] Optimizing reduction loops (sum(), prod(), et al.)

2009-07-08 Thread Pauli Virtanen
On 2009-07-08, Stéfan van der Walt ste...@sun.ac.za wrote: I know very little about cache optimality, so excuse the triviality of this question: Is it possible to design this loop optimally (taking into account certain build-time measurable parameters), or is it the kind of thing that can only

Re: [Numpy-discussion] Optimizing reduction loops (sum(), prod(), et al.)

2009-07-08 Thread Robert Kern
On Wed, Jul 8, 2009 at 18:02, Pauli Virtanenpav...@iki.fi wrote: On 2009-07-08, Stéfan van der Walt ste...@sun.ac.za wrote: I know very little about cache optimality, so excuse the triviality of this question: Is it possible to design this loop optimally (taking into account certain build-time

Re: [Numpy-discussion] Optimizing reduction loops (sum(), prod(), et al.)

2009-07-08 Thread David Cournapeau
On Thu, Jul 9, 2009 at 8:02 AM, Pauli Virtanenpav...@iki.fi wrote: I don't think we want to go the ATNumPy route, or even have tunable parameters chosen at build or compile time. Detecting things like cache size at compile time should not be too difficult, at least for common platforms. Even

[Numpy-discussion] ndarray and the buffer protocol

2009-07-08 Thread Fons Adriaensen
Hello all, (resending for the Nth time, as the previous attempts didn't make it to the list) I'm new to this list (and numpy is mostly new to me :-). Using python 2.6 and numpy 1.3. My plan is to write some C extensions that will perform rather specialised processing on multichannel digital

Re: [Numpy-discussion] Optimizing reduction loops (sum(), prod(), et al.)

2009-07-08 Thread Charles R Harris
On Wed, Jul 8, 2009 at 5:02 PM, Pauli Virtanen pav...@iki.fipav%2...@iki.fi wrote: On 2009-07-08, Stéfan van der Walt ste...@sun.ac.za wrote: I know very little about cache optimality, so excuse the triviality of this question: Is it possible to design this loop optimally (taking into

Re: [Numpy-discussion] performing operations in-place in numpy

2009-07-08 Thread Pauli Virtanen
On 2009-07-08, Charles R Harris charlesr.har...@gmail.com wrote: In that case I don't see a problem offhand. That said, I haven't looked at the code yet. I'm a bit worried about the problem that cropped up in the ticket with the complex ufuncs. As Luca noted in the ticket, obj3 =

Re: [Numpy-discussion] performing operations in-place in numpy

2009-07-08 Thread Robert Kern
On Wed, Jul 8, 2009 at 18:28, Pauli Virtanenpav...@iki.fi wrote: On 2009-07-08, Charles R Harris charlesr.har...@gmail.com wrote: In that case I don't see a problem offhand. That said, I haven't looked at the code yet. I'm a bit worried about the problem that cropped up in the ticket with

Re: [Numpy-discussion] Help with np.where and datetime functions

2009-07-08 Thread John [H2O]
Pierre GM-2 wrote: Would you like to give the scikits.timeseries package a try ? It's available at pytseries.sourceforge.net. Calculatng the hourly average should be straightforward. I would, in fact I have been investigating it, but I didn't have numpy1.3 up and running until just

Re: [Numpy-discussion] Help with np.where and datetime functions

2009-07-08 Thread John [H2O]
nhmc wrote: Also, if you don't need the indices, you can just use the conditional expression as a boolean mask: condition = (t1 Y[:,0]) (Y[:,0] t2) Y[:,0][condition] Neil 'condition' is not an index array? Wouldn't it just be the indices as well? Would it be possible to do

Re: [Numpy-discussion] Help with np.where and datetime functions

2009-07-08 Thread John [H2O]
Also, could someone please explain why: Tsub = T[ (T[:,0]t1) (T[:,0]t2) ] Works, but: Tsub = T[ (t1T[:,0]) (T[:,0]t2) ] Does not??? Thanks. -- View this message in context: http://www.nabble.com/Help-with-np.where-and-datetime-functions-tp24389447p24401687.html Sent from the

Re: [Numpy-discussion] Help with np.where and datetime functions

2009-07-08 Thread Robert Kern
On Wed, Jul 8, 2009 at 19:11, John [H2O]washa...@gmail.com wrote: Also, could someone please explain why: Tsub = T[ (T[:,0]t1) (T[:,0]t2) ] Works, but: Tsub = T[ (t1T[:,0]) (T[:,0]t2) ] Does not??? I'm not positive, but I think it boils down to this: Python tries to look up the method

Re: [Numpy-discussion] performing operations in-place in numpy

2009-07-08 Thread Citi, Luca
Hello The problem is not PyArray_Conjugate itself. The problem is that whenever you call a function from the C side and one of the inputs has ref_count 1, it can be overwritten. This is not a problem from the python side because if the ufunc sees a ref_count=1 it means that no python object is

Re: [Numpy-discussion] Bug in the F distribution?

2009-07-08 Thread alan
On Fri, Jul 3, 2009 at 10:21 PM, Alan Jacksona...@ajackson.org wrote: I don't see any problem here. If you can replicate your results, we would need more information about the versions. Josef ''' np.version.version '1.3.0' scipy.version.version '0.8.0.dev5789' ''' In [4]:

[Numpy-discussion] speeding up an array operation

2009-07-08 Thread Mag Gam
Hey All I am reading thru a file and trying to store the values into another array, but instead of storing the values 1 by 1, I would like to store them in bulk sets for optimization purposes. Here is what I have, which does it 1x1: z={} #dictionary r=csv.reader(file) for i,row in

Re: [Numpy-discussion] argwhere does not accept py list

2009-07-08 Thread Scott Sinclair
2009/7/8 Pierre GM pgmdevl...@gmail.com: On Jul 8, 2009, at 3:18 AM, Scott Sinclair wrote: 2009/7/8 Robert Kern robert.k...@gmail.com: 2009/7/4 Stéfan van der Walt ste...@sun.ac.za: Thanks, Scott.  This should now be fixed in SVN. You should probably change that to asanyarray() before the