Re: [Numpy-discussion] Memory hungry reduce ops in Numpy

2011-11-15 Thread Bruce Southey
On Tue, Nov 15, 2011 at 11:57 AM, Robert Kern wrote: > On Tue, Nov 15, 2011 at 16:55, Gael Varoquaux > wrote: >> On Tue, Nov 15, 2011 at 05:46:52PM +0100, Andreas Müller wrote: >>>    My question was more along the lines of "why doesn't numpy do the online >>>    algorithm". >> >> It's probably a

Re: [Numpy-discussion] Memory hungry reduce ops in Numpy

2011-11-15 Thread Andreas Müller
On 11/15/2011 07:03 PM, Gael Varoquaux wrote: > On Tue, Nov 15, 2011 at 05:57:14PM +, Robert Kern wrote: >> Actually, last time I suggested it, it was brought up that the online >> algorithms can be worse numerically. I'll try to find the thread. > Indeed, especially for smallish datasets where

Re: [Numpy-discussion] Memory hungry reduce ops in Numpy

2011-11-15 Thread Gael Varoquaux
On Tue, Nov 15, 2011 at 05:57:14PM +, Robert Kern wrote: > Actually, last time I suggested it, it was brought up that the online > algorithms can be worse numerically. I'll try to find the thread. Indeed, especially for smallish datasets where the memory overhead is not an issue. I think that

Re: [Numpy-discussion] Memory hungry reduce ops in Numpy

2011-11-15 Thread Robert Kern
On Tue, Nov 15, 2011 at 16:55, Gael Varoquaux wrote: > On Tue, Nov 15, 2011 at 05:46:52PM +0100, Andreas Müller wrote: >>    My question was more along the lines of "why doesn't numpy do the online >>    algorithm". > > It's probably a matter of nobody having had the time and the urge to code > it

Re: [Numpy-discussion] Memory hungry reduce ops in Numpy

2011-11-15 Thread Andreas Müller
On 11/15/2011 06:02 PM, Warren Weckesser wrote: On Tue, Nov 15, 2011 at 10:48 AM, Andreas Müller mailto:amuel...@ais.uni-bonn.de>> wrote: On 11/15/2011 05:46 PM, Andreas Müller wrote: On 11/15/2011 04:28 PM, Bruce Southey wrote: On 11/14/2011 10:05 AM, Andreas Müller wrote:

Re: [Numpy-discussion] Memory hungry reduce ops in Numpy

2011-11-15 Thread Warren Weckesser
On Tue, Nov 15, 2011 at 10:48 AM, Andreas Müller wrote: > ** > On 11/15/2011 05:46 PM, Andreas Müller wrote: > > On 11/15/2011 04:28 PM, Bruce Southey wrote: > > On 11/14/2011 10:05 AM, Andreas Müller wrote: > > On 11/14/2011 04:23 PM, David Cournapeau wrote: > > On Mon, Nov 14, 2011 at 12:46 PM,

Re: [Numpy-discussion] Memory hungry reduce ops in Numpy

2011-11-15 Thread Gael Varoquaux
On Tue, Nov 15, 2011 at 05:46:52PM +0100, Andreas Müller wrote: >My question was more along the lines of "why doesn't numpy do the online >algorithm". It's probably a matter of nobody having had the time and the urge to code it, _and_ do all the extra steps necessary to integrate to the co

Re: [Numpy-discussion] Memory hungry reduce ops in Numpy

2011-11-15 Thread Andreas Müller
On 11/15/2011 05:46 PM, Andreas Müller wrote: On 11/15/2011 04:28 PM, Bruce Southey wrote: On 11/14/2011 10:05 AM, Andreas Müller wrote: On 11/14/2011 04:23 PM, David Cournapeau wrote: On Mon, Nov 14, 2011 at 12:46 PM, Andreas Müller wrote: Hi everybody. When I did some normalization using

Re: [Numpy-discussion] Memory hungry reduce ops in Numpy

2011-11-15 Thread Andreas Müller
On 11/15/2011 04:28 PM, Bruce Southey wrote: On 11/14/2011 10:05 AM, Andreas Müller wrote: On 11/14/2011 04:23 PM, David Cournapeau wrote: On Mon, Nov 14, 2011 at 12:46 PM, Andreas Müller wrote: Hi everybody. When I did some normalization using numpy, I noticed that numpy.std uses more ram

Re: [Numpy-discussion] Memory hungry reduce ops in Numpy

2011-11-15 Thread Bruce Southey
On 11/14/2011 10:05 AM, Andreas Müller wrote: On 11/14/2011 04:23 PM, David Cournapeau wrote: On Mon, Nov 14, 2011 at 12:46 PM, Andreas Müller wrote: Hi everybody. When I did some normalization using numpy, I noticed that numpy.std uses more ram than I was expecting. A quick google search ga

Re: [Numpy-discussion] Memory hungry reduce ops in Numpy

2011-11-14 Thread Andreas Müller
On 11/14/2011 04:23 PM, David Cournapeau wrote: > On Mon, Nov 14, 2011 at 12:46 PM, Andreas Müller > wrote: >> Hi everybody. >> When I did some normalization using numpy, I noticed that numpy.std uses >> more ram than I was expecting. >> A quick google search gave me this: >> http://luispedro.org

Re: [Numpy-discussion] Memory hungry reduce ops in Numpy

2011-11-14 Thread David Cournapeau
On Mon, Nov 14, 2011 at 12:46 PM, Andreas Müller wrote: > Hi everybody. > When I did some normalization using numpy, I noticed that numpy.std uses > more ram than I was expecting. > A quick google search gave me this: > http://luispedro.org/software/ncreduce > The site claims that std and other re

[Numpy-discussion] Memory hungry reduce ops in Numpy

2011-11-14 Thread Andreas Müller
Hi everybody. When I did some normalization using numpy, I noticed that numpy.std uses more ram than I was expecting. A quick google search gave me this: http://luispedro.org/software/ncreduce The site claims that std and other reduce operations are implemented naively with many temporaries. Is tha