Re: [Numpy-discussion] Fast threading solution thoughts

2009-02-11 Thread David Cournapeau
Brian Granger wrote: >> I am curious: would you know what would be different in numpy's case >> compared to matlab array model concerning locks ? Matlab, up to >> recently, only spreads BLAS/LAPACK on multi-cores, but since matlab 7.3 >> (or 7.4), it also uses multicore for mathematical functions (

Re: [Numpy-discussion] Fast threading solution thoughts

2009-02-11 Thread Brian Granger
>> Good point. Is it possible to tell what array size it switches over >> to using multiple threads? > > Yes. > > http://svn.scipy.org/svn/numpy/branches/multicore/numpy/core/threadapi.py Sorry, I was curious about what Matlab does in this respect. But, this is very useful and I will look at it.

Re: [Numpy-discussion] Fast threading solution thoughts

2009-02-11 Thread Robert Kern
On Thu, Feb 12, 2009 at 00:52, Brian Granger wrote: >> I am curious: would you know what would be different in numpy's case >> compared to matlab array model concerning locks ? Matlab, up to >> recently, only spreads BLAS/LAPACK on multi-cores, but since matlab 7.3 >> (or 7.4), it also uses multic

Re: [Numpy-discussion] Fast threading solution thoughts

2009-02-11 Thread Brian Granger
> I am curious: would you know what would be different in numpy's case > compared to matlab array model concerning locks ? Matlab, up to > recently, only spreads BLAS/LAPACK on multi-cores, but since matlab 7.3 > (or 7.4), it also uses multicore for mathematical functions (cos, > etc...). So at lea

Re: [Numpy-discussion] Fast threading solution thoughts

2009-02-11 Thread Robert Kern
On Thu, Feb 12, 2009 at 00:15, David Cournapeau wrote: > Gael Varoquaux wrote: >> >From a programmer's perspective, because, IMHO, openmp is implemented >> using pthreads. > > Since openmp also exists on windows, I doubt that it is required that > openmp uses pthread :) It is implemented using th

Re: [Numpy-discussion] Fast threading solution thoughts

2009-02-11 Thread David Cournapeau
Gael Varoquaux wrote: > >From a programmer's perspective, because, IMHO, openmp is implemented > using pthreads. Since openmp also exists on windows, I doubt that it is required that openmp uses pthread :) On linux, with gcc, using -fopenmp implies -pthread, so I guess it uses pthread (can you b

Re: [Numpy-discussion] Fast threading solution thoughts

2009-02-11 Thread David Cournapeau
Robert Kern wrote: > > Eric Jones tried to do this with pthreads in C some time ago. His work is > here: > > http://svn.scipy.org/svn/numpy/branches/multicore/ > > The lock overhead makes it usually not worthwhile. > I am curious: would you know what would be different in numpy's case compar

Re: [Numpy-discussion] Fast threading solution thoughts

2009-02-11 Thread Gael Varoquaux
On Wed, Feb 11, 2009 at 11:52:40PM -0600, Robert Kern wrote: > > This seem like pretty heavy solutions though. > >From a programmer's perspective, it seems to me like OpenMP is a muck > >lighter weight solution than pthreads. >From a programmer's perspective, because, IMHO, openmp is implemented

Re: [Numpy-discussion] Fast threading solution thoughts

2009-02-11 Thread Robert Kern
On Thu, Feb 12, 2009 at 00:03, Brian Granger wrote: >> Eric Jones tried to do this with pthreads in C some time ago. His work is >> here: >> >> http://svn.scipy.org/svn/numpy/branches/multicore/ >> >> The lock overhead makes it usually not worthwhile. > > I was under the impression that Eric's i

Re: [Numpy-discussion] [SciPy-user] Numpy 1.2.1 and Scipy 0.7.0; Ubuntu packages

2009-02-11 Thread Fernando Perez
On Wed, Feb 11, 2009 at 6:17 PM, David Cournapeau wrote: > Unfortunately, it does require some work, because hardy uses g77 > instead of gfortran, so the source package has to be different (once > hardy is done, all the one below would be easy, though). I am not sure > how to do that with PPA (th

Re: [Numpy-discussion] Fast threading solution thoughts

2009-02-11 Thread Brian Granger
> Eric Jones tried to do this with pthreads in C some time ago. His work is > here: > > http://svn.scipy.org/svn/numpy/branches/multicore/ > > The lock overhead makes it usually not worthwhile. I was under the impression that Eric's implementation didn't use a thread pool. Thus I thought the bo

Re: [Numpy-discussion] loadtxt issues

2009-02-11 Thread Scott Sinclair
> 2009/2/12 A B : > Actually, I was using two different machines and it appears that the > version of numpy available on Ubuntu is seriously out of date (1.0.4). > Wonder why ... See the recent post here http://projects.scipy.org/pipermail/numpy-discussion/2009-February/040252.html Cheers, Scott

Re: [Numpy-discussion] Fast threading solution thoughts

2009-02-11 Thread Robert Kern
On Wed, Feb 11, 2009 at 23:46, Brian Granger wrote: > Hi, > > This is relevant for anyone who would like to speed up array based > codes using threads. > > I have a simple loop that I have implemented using Cython: > > def backstep(np.ndarray opti, np.ndarray optf, > int istart, int ie

Re: [Numpy-discussion] numscons/numpy.distutils bug related to MACOSX_DEPLOYMENT_TARGET

2009-02-11 Thread Brian Granger
Thanks much! Brian On Wed, Feb 11, 2009 at 9:44 PM, Stéfan van der Walt wrote: > 2009/2/6 Brian Granger : >> Great, what is the best way of rolling this into numpy? > > I've committed your patch. > > Cheers > Stéfan > ___ > Numpy-discussion mailing lis

[Numpy-discussion] Fast threading solution thoughts

2009-02-11 Thread Brian Granger
Hi, This is relevant for anyone who would like to speed up array based codes using threads. I have a simple loop that I have implemented using Cython: def backstep(np.ndarray opti, np.ndarray optf, int istart, int iend, double p, double q): cdef int j cdef double *pi cde

Re: [Numpy-discussion] numscons/numpy.distutils bug related to MACOSX_DEPLOYMENT_TARGET

2009-02-11 Thread Stéfan van der Walt
2009/2/6 Brian Granger : > Great, what is the best way of rolling this into numpy? I've committed your patch. Cheers Stéfan ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Outer join ?

2009-02-11 Thread Robert Kern
On Wed, Feb 11, 2009 at 23:24, A B wrote: > Hi, > > I have the following data structure: > > col1 | col2 | col3 > > 20080101|key1|4 > 20080201|key1|6 > 20080301|key1|5 > 20080301|key2|3.4 > 20080601|key2|5.6 > > For each key in the second column, I would like to create an array > where for all uni

Re: [Numpy-discussion] FYI: New "select-multiple-fields" behavior

2009-02-11 Thread Travis E. Oliphant
Stéfan van der Walt wrote: > Hi Travis > > 2009/2/12 Travis E. Oliphant : > >> ary['field1', 'field3'] raises an error >> ary[['field1', 'field3']] is the correct spelling and returns a copy of >> the data in those fields in a new array. >> > > Is there absolutely no way of returning the r

[Numpy-discussion] Outer join ?

2009-02-11 Thread A B
Hi, I have the following data structure: col1 | col2 | col3 20080101|key1|4 20080201|key1|6 20080301|key1|5 20080301|key2|3.4 20080601|key2|5.6 For each key in the second column, I would like to create an array where for all unique values in the first column, there will be either a value or zer

Re: [Numpy-discussion] FYI: New "select-multiple-fields" behavior

2009-02-11 Thread Stéfan van der Walt
Hi Travis 2009/2/12 Travis E. Oliphant : > ary['field1', 'field3'] raises an error > ary[['field1', 'field3']] is the correct spelling and returns a copy of > the data in those fields in a new array. Is there absolutely no way of returning the result as a view? Regards Stéfan __

Re: [Numpy-discussion] loadtxt issues

2009-02-11 Thread A B
On Wed, Feb 11, 2009 at 6:27 PM, A B wrote: > On Tue, Feb 10, 2009 at 9:52 PM, Brent Pedersen wrote: >> On Tue, Feb 10, 2009 at 9:40 PM, A B wrote: >>> Hi, >>> >>> How do I write a loadtxt command to read in the following file and >>> store each data point as the appropriate data type: >>> >>> 1

[Numpy-discussion] FYI: New "select-multiple-fields" behavior

2009-02-11 Thread Travis E. Oliphant
Hi all, As of r6358, I checked in the functionality to allow selection by multiple fields along with a couple of tests. ary['field1', 'field3'] raises an error ary[['field1', 'field3']] is the correct spelling and returns a copy of the data in those fields in a new array. -Travis _

Re: [Numpy-discussion] genloadtxt: dtype=None and unpack=True

2009-02-11 Thread Pierre GM
On Feb 11, 2009, at 11:38 PM, Ryan May wrote: > Pierre, > > I noticed that using dtype=None with a heterogeneous set of data, > trying to use unpack=True to get the columns into separate arrays > (instead of a structured array) doesn't work. I've attached a patch > that, in the case of dty

[Numpy-discussion] genloadtxt: dtype=None and unpack=True

2009-02-11 Thread Ryan May
Pierre, I noticed that using dtype=None with a heterogeneous set of data, trying to use unpack=True to get the columns into separate arrays (instead of a structured array) doesn't work. I've attached a patch that, in the case of dtype=None, unpacks the fields in the final array into a list of sep

Re: [Numpy-discussion] loadtxt issues

2009-02-11 Thread A B
On Tue, Feb 10, 2009 at 9:52 PM, Brent Pedersen wrote: > On Tue, Feb 10, 2009 at 9:40 PM, A B wrote: >> Hi, >> >> How do I write a loadtxt command to read in the following file and >> store each data point as the appropriate data type: >> >> 12|h|34.5|44.5 >> 14552|bbb|34.5|42.5 >> >> Do the stri

Re: [Numpy-discussion] [SciPy-user] Numpy 1.2.1 and Scipy 0.7.0; Ubuntu packages

2009-02-11 Thread David Cournapeau
On Thu, Feb 12, 2009 at 8:11 AM, Fernando Perez wrote: > On Wed, Feb 11, 2009 at 4:46 AM, David Cournapeau wrote: >> Hi, >> >> I started to set up a PPA for scipy on launchpad, which enables to >> build ubuntu packages for various distributions/architectures. The >> link is there: >> >> https://e

Re: [Numpy-discussion] [SciPy-user] Numpy 1.2.1 and Scipy 0.7.0; Ubuntu packages

2009-02-11 Thread Fernando Perez
On Wed, Feb 11, 2009 at 4:46 AM, David Cournapeau wrote: > Hi, > > I started to set up a PPA for scipy on launchpad, which enables to > build ubuntu packages for various distributions/architectures. The > link is there: > > https://edge.launchpad.net/~scipy/+archive/ppa Cool, thanks. Is it easy

Re: [Numpy-discussion] PEP: named axis

2009-02-11 Thread Andrew Jaffe
Pauli Virtanen wrote: > Wed, 11 Feb 2009 22:21:30 +, Andrew Jaffe wrote: > [clip] >> Maybe I misunderstand the proposal, but, actually, I think this is >> completely the wrong semantics for "axis=" anyway. "axis=" in numpy >> refers to what is also a dimension, not a column. > > I think the

Re: [Numpy-discussion] PEP: named axis

2009-02-11 Thread Pauli Virtanen
Wed, 11 Feb 2009 22:21:30 +, Andrew Jaffe wrote: [clip] > Maybe I misunderstand the proposal, but, actually, I think this is > completely the wrong semantics for "axis=" anyway. "axis=" in numpy > refers to what is also a dimension, not a column. I think the proposal was to add the ability

Re: [Numpy-discussion] PEP: named axis

2009-02-11 Thread Andrew Jaffe
Robert Kern wrote: > On Fri, Feb 6, 2009 at 03:22, Stéfan van der Walt wrote: >> Hi Robert >> >> 2009/2/6 Robert Kern : This could be implemented but would require adding information to the NumPy array. >>> More than that, though. Every function and method that takes an axis >>> or reduc

Re: [Numpy-discussion] PyArray_SETITEM with object arrays in Cython

2009-02-11 Thread Wes McKinney
I actually got it to work-- the function prototype in the pxi file was wrong, needed to be: int PyArray_SETITEM(object obj, void* itemptr, object item) This still doesn't explain why the buffer interface was slow. The general problem here is an indexed array (by dates or strings, for example), t

Re: [Numpy-discussion] PyArray_SETITEM with object arrays in Cython

2009-02-11 Thread Dag Sverre Seljebotn
Wes McKinney wrote: > I am writing some Cython code and have noted that the buffer interface > offers very little speedup for PyObject arrays. In trying to rewrite the > same code using the C API in Cython, I find I can't get PyArray_SETITEM to > work, in a call like: > > PyArray_SETITEM(result, i

Re: [Numpy-discussion] PyArray_SETITEM with object arrays in Cython

2009-02-11 Thread Hanni Ali
Hi Wes, I do not profess to be an expert, but I have been off loading a fair number of loops to C from Python code and achieved significant improvements most have been of the following form (which I have found to be the fastest): size = *incomingArrayObj->dimensions; r_dptr = PyArray_DATA(result

[Numpy-discussion] PyArray_SETITEM with object arrays in Cython

2009-02-11 Thread Wes McKinney
Hello, I am writing some Cython code and have noted that the buffer interface offers very little speedup for PyObject arrays. In trying to rewrite the same code using the C API in Cython, I find I can't get PyArray_SETITEM to work, in a call like: PyArray_SETITEM(result, iterresult.dataptr, obj)

[Numpy-discussion] ANN: Numexpr 1.2 released

2009-02-11 Thread Francesc Alted
Announcing Numexpr 1.2 Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. The main feature added

Re: [Numpy-discussion] Numpy 1.2.1 and Scipy 0.7.0; Ubuntu packages

2009-02-11 Thread David Cournapeau
On Wed, Feb 11, 2009 at 9:46 PM, David Cournapeau wrote: > Hi, > > I started to set up a PPA for scipy on launchpad, which enables to > build ubuntu packages for various distributions/architectures. The > link is there: > > https://edge.launchpad.net/~scipy/+archive/ppa > > So you just need to add

[Numpy-discussion] Numpy 1.2.1 and Scipy 0.7.0; Ubuntu packages

2009-02-11 Thread David Cournapeau
Hi, I started to set up a PPA for scipy on launchpad, which enables to build ubuntu packages for various distributions/architectures. The link is there: https://edge.launchpad.net/~scipy/+archive/ppa So you just need to add one line to your /etc/apt/sources.list, and you will get uptodate numpy

Re: [Numpy-discussion] PEP: named axis

2009-02-11 Thread Lars Friedrich
Hello list, I am not sure, if I understood everything of the discussion on the named-axis-idea of numpy-arrays, since I am only a *user* of numpy. I never subclassed the numpy-array-class ;-) However, I have the need to store meta-information for my arrays. I do this with a stand-alone class w

Re: [Numpy-discussion] numpy-izing a loop

2009-02-11 Thread Paul Rudin
Stéfan van der Walt writes: > 2009/2/10 Stéfan van der Walt : >> x = np.arange(dim) >> y = np.arange(dim)[:, None] >> z = np.arange(dim)[:, None, None] > > Do not operate heavy machinery or attempt broadcasting while tired or > under the influence. That order was incorrect: > >> z = np.arange(di

[Numpy-discussion] ANN: SciPy 0.7.0

2009-02-11 Thread Jarrod Millman
I'm pleased to announce SciPy 0.7.0. SciPy is a package of tools for science and engineering for Python. It includes modules for statistics, optimization, integration, linear algebra, Fourier transforms, signal and image processing, ODE solvers, and more. This release comes sixteen months after