Re: [Numpy-discussion] Fastest way to compute summary statistics for a specific axis

2015-03-17 Thread Dave Hirschfeld
Sebastian Berg sebastian at sipsolutions.net writes: On Mo, 2015-03-16 at 15:53 +, Dave Hirschfeld wrote: I have a number of large arrays for which I want to compute the mean and standard deviation over a particular axis - e.g. I want to compute the statistics for axis=1

[Numpy-discussion] Fastest way to compute summary statistics for a specific axis

2015-03-16 Thread Dave Hirschfeld
I have a number of large arrays for which I want to compute the mean and standard deviation over a particular axis - e.g. I want to compute the statistics for axis=1 as if the other axes were combined so that in the example below I get two values back In [1]: a = randn(30, 2, 1) For the

Re: [Numpy-discussion] Optimizing numpy's einsum expression (again)

2015-01-16 Thread Dave Hirschfeld
Daniel Smith dgasmith at icloud.com writes: Hello everyone,I originally brought an optimized einsum routine forward a few weeks back that attempts to contract numpy arrays together in an optimal way. This can greatly reduce the scaling and overall cost of the einsum expression for the cost

Re: [Numpy-discussion] help using np.einsum for stacked matrix multiplication

2014-10-29 Thread Dave Hirschfeld
Andrew Nelson writes: Dear list,I have a 4D array, A, that has the shape (NX, NY, 2, 2).  I wish to perform matrix multiplication of the 'NY' 2x2 matrices, resulting in the matrix B.  B would have shape (NX, 2, 2).  I believe that np.einsum would be up to the task, but I'm not quite sure of

Re: [Numpy-discussion] segfault in np.arange

2014-10-24 Thread Dave Hirschfeld
Julian Taylor jtaylor.debian at googlemail.com writes: On 23.10.2014 19:21, Dave Hirschfeld wrote: Hi, I accidentally passed a pandas DatetimeIndex to `np.arange` which caused it to segfault. It's a pretty dumb thing to do but I don't think it should cause a segfault! thanks

[Numpy-discussion] segfault in np.arange

2014-10-23 Thread Dave Hirschfeld
Hi, I accidentally passed a pandas DatetimeIndex to `np.arange` which caused it to segfault. It's a pretty dumb thing to do but I don't think it should cause a segfault! Python 2.7.5 |Continuum Analytics, Inc.| (default, Jul 1 2013, 12:37:52) [MSC v.1500 64 bit (AMD64)] on win32 Type help,

[Numpy-discussion] Website down!

2014-08-20 Thread Dave Hirschfeld
It seems that the docs website is down? http://docs.scipy.org/doc/ -Dave ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] simple way to denote unchanged dimension in reshape?

2014-05-29 Thread Dave Hirschfeld
Chao YUE chaoyuejoy at gmail.com writes: Dear all, I have a simple question. Is there a way to denote the unchanged dimension in the reshape function? like suppose I have an array named arr having three dims with the first dimension length as 48, I want to reshape the first dim into

Re: [Numpy-discussion] Fancy Indexing of Structured Arrays is Slow

2014-05-21 Thread Dave Hirschfeld
Julian Taylor jtaylor.debian at googlemail.com writes: On 16.05.2014 10:59, Dave Hirschfeld wrote: Julian Taylor jtaylor.debian at googlemail.com writes: Yes, I'd heard about the improvements and am very excited to try them out since indexing is one of the bottlenecks in our

Re: [Numpy-discussion] Fancy Indexing of Structured Arrays is Slow

2014-05-16 Thread Dave Hirschfeld
Sebastian Berg sebastian at sipsolutions.net writes: On Do, 2014-05-15 at 12:31 +, Dave Hirschfeld wrote: As can be seen from the code below (or in the notebook linked beneath) fancy indexing of a structured array is twice as slow as indexing both fields independently - making

Re: [Numpy-discussion] Fancy Indexing of Structured Arrays is Slow

2014-05-16 Thread Dave Hirschfeld
Julian Taylor jtaylor.debian at googlemail.com writes: if ~50% faster is fast enough a simple improvement would be to replace the use of PyArg_ParseTuple with manual tuple unpacking. The PyArg functions are incredibly slow and is not required in VOID_copyswap which just extracts 'Oi.

[Numpy-discussion] Fancy Indexing of Structured Arrays is Slow

2014-05-15 Thread Dave Hirschfeld
As can be seen from the code below (or in the notebook linked beneath) fancy indexing of a structured array is twice as slow as indexing both fields independently - making it 4x slower? I found that fancy indexing was a bottleneck in my application so I was hoping to reduce the overhead by

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-03-19 Thread Dave Hirschfeld
Sankarshan Mudkavi smudkavi at uwaterloo.ca writes: Hey all, It's been a while since the last datetime and timezones discussion thread was visited (linked below): http://thread.gmane.org/gmane.comp.python.numeric.general/53805 It looks like the best approach to follow is the UTC only

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-03-19 Thread Dave Hirschfeld
Jeff Reback jeffreback at gmail.com writes: Dave, your example is not a problem with numpy per se, rather that the default generation is in local timezone (same as what python datetime does). If you localize to UTC you get the results that you expect.  The problem is that the default

Re: [Numpy-discussion] svd error checking vs. speed

2014-02-17 Thread Dave Hirschfeld
alex argriffi at ncsu.edu writes: Hello list, Here's another idea resurrection from numpy github comments that I've been advised could be posted here for re-discussion. The proposal would be to make np.linalg.svd more like scipy.linalg.svd with respect to input checking. The argument

Re: [Numpy-discussion] svd error checking vs. speed

2014-02-17 Thread Dave Hirschfeld
Sturla Molden sturla.molden at gmail.com writes: josef.pktd at gmail.com wrote: I use official numpy release for development, Windows, 32bit python, i.e. MingW 3.5 and whatever old ATLAS the release includes. a constant 13% cpu usage is 1/8 th of my 8 virtual cores. Based on this

Re: [Numpy-discussion] Numpy 1.9 release date

2013-11-10 Thread Dave Hirschfeld
Ralf Gommers ralf.gommers at gmail.com writes: On Fri, Nov 8, 2013 at 8:22 PM, Charles R Harris charlesr.harris at gmail.com wrote: and think that the main thing missing at this point is fixing the datetime problems. Is anyone planning to work on this? If yes, you need a rough

Re: [Numpy-discussion] add .H attribute?

2013-07-24 Thread Dave Hirschfeld
josef.pktd at gmail.com writes: I think a H is feature creep and too specialized What's .H of a int a str a bool ? It's just .T and a view, so you cannot rely that conj() makes a copy if you don't work with complex. .T is just a reshape function and has **nothing** to do with matrix

Re: [Numpy-discussion] add .H attribute?

2013-07-24 Thread Dave Hirschfeld
Nathaniel Smith njs at pobox.com writes: As soon as you talk about attributes returning things you've already broken Python's mental model... attributes are things that sit there, not things that execute arbitrary code. Of course this is not how the actual implementation works, attribute

Re: [Numpy-discussion] add .H attribute?

2013-07-23 Thread Dave Hirschfeld
Alan G Isaac alan.isaac at gmail.com writes: On 7/22/2013 3:10 PM, Nathaniel Smith wrote: Having .T but not .H is an example of this split. Hate to do this but ... Readability counts. +10! A.conjugate().transpose() is unspeakably horrible IMHO. Since there's no way to avoid a copy

[Numpy-discussion] datetime64 constructor ignores dtype argument?

2013-06-12 Thread Dave Hirschfeld
The example below demonstrates the fact that the datetime64 constructor ignores the dtype argument if passed in. Is this conscious design decision or a bug/oversight? In [55]: from datetime import datetime ...: d = datetime.now() ...: In [56]: d Out[56]: datetime.datetime(2013, 6,

Re: [Numpy-discussion] 1.8 release

2013-04-25 Thread Dave Hirschfeld
Charles R Harris charlesr.harris at gmail.com writes: Hi All,I think it is time to start the runup to the 1.8 release. I don't know of any outstanding blockers but if anyone has a PR/issue that they feel needs to be in the next Numpy release now is the time to make it known.Chuck It would

Re: [Numpy-discussion] timezones and datetime64

2013-04-03 Thread Dave Hirschfeld
Andreas Hilboll lists at hilboll.de writes: I think your point about using current timezone in interpreting user input being dangerous is probably correct --- perhaps UTC all the way would be a safer (and simpler) choice? +1 +10 from me! I've recently come across a bug due to

Re: [Numpy-discussion] timezones and datetime64

2013-04-03 Thread Dave Hirschfeld
Nathaniel Smith njs at pobox.com writes: On Wed, Apr 3, 2013 at 2:26 PM, Dave Hirschfeld dave.hirschfeld at gmail.com wrote: This isn't acceptable for my use case (in a multinational company) and I found no reasonable way around it other than bypassing the numpy conversion entirely

Re: [Numpy-discussion] New numpy functions: filled, filled_like

2013-01-14 Thread Dave Hirschfeld
Robert Kern robert.kern at gmail.com writes: One alternative that does not expand the API with two-liners is to let the ndarray.fill() method return self: a = np.empty(...).fill(20.0) This violates the convention that in-place operations never return self, to avoid

Re: [Numpy-discussion] Bug in as_strided/reshape

2012-08-10 Thread Dave Hirschfeld
Sebastian Berg sebastian at sipsolutions.net writes: Hello, looking at the code, when only adding/removing dimensions with size 1, numpy takes a small shortcut, however it uses 0 stride lengths as value for the new one element dimensions temporarily, then replacing it again to ensure the

Re: [Numpy-discussion] Second try: possible bug in assignment to complex array

2012-08-10 Thread Dave Hirschfeld
Mark Bakker markbak at gmail.com writes: I think there is a problem with assigning a 1D complex array of length one to a position in another complex array. Example: a = ones(1,'D') b = ones(1,'D') a[0] = b ---

Re: [Numpy-discussion] Bug in as_strided/reshape

2012-08-09 Thread Dave Hirschfeld
Dave Hirschfeld dave.hirschfeld at gmail.com writes: It seems that reshape doesn't work correctly on an array which has been resized using the 0-stride trick e.g. In [73]: x = array([5]) In [74]: y = as_strided(x, shape=(10,), strides=(0,)) In [75]: y Out[75]: array([5, 5, 5, 5, 5

[Numpy-discussion] Bug in as_strided/reshape

2012-08-08 Thread Dave Hirschfeld
It seems that reshape doesn't work correctly on an array which has been resized using the 0-stride trick e.g. In [73]: x = array([5]) In [74]: y = as_strided(x, shape=(10,), strides=(0,)) In [75]: y Out[75]: array([5, 5, 5, 5, 5, 5, 5, 5, 5, 5]) In [76]: y.reshape([10,1]) Out[76]: array([[

Re: [Numpy-discussion] Moving lib.recfunctions?

2011-07-06 Thread Dave Hirschfeld
Pierre GM pgmdevlist at gmail.com writes: Hello, The idea behin having a lib.recfunctions and not a rec.recfunctions or whatever was to illustrate that the functions of this package are more generic than they appear. They work with regular structured ndarrays and don't need recarrays.

Re: [Numpy-discussion] fixing up datetime

2011-06-08 Thread Dave Hirschfeld
Wes McKinney wesmckinn at gmail.com writes: - Fundamental need to be able to work with multiple time series, especially performing operations involving cross-sectional data - I think it's a bit hard for lay people to use (read: ex-MATLAB/R users). This is just my opinion, but a few

Re: [Numpy-discussion] fixing up datetime

2011-06-08 Thread Dave Hirschfeld
Mark Wiebe mwwiebe at gmail.com writes: It appears to me that a structured dtype with some further NumPy extensions could entirely replace the 'events' metadata fairly cleanly. If the ufuncs are extended to operate on structured arrays, and integers modulo n are added as a new dtype, a

Re: [Numpy-discussion] fixing up datetime

2011-06-07 Thread Dave Hirschfeld
As a user of numpy/scipy in finance I thought I would put in my 2p worth as it's something which is of great importance in this area. I'm currently a heavy user of the scikits.timeseries package by Matt Pierre and I'm also following the development of statsmodels and pandas should we require

Re: [Numpy-discussion] fixing up datetime

2011-06-07 Thread Dave Hirschfeld
Robert Kern robert.kern at gmail.com writes: On Tue, Jun 7, 2011 at 07:34, Dave Hirschfeld dave.hirschfeld at gmail.com wrote: I'm not convinced about the events concept - it seems to add complexity for something which could be accomplished better in other ways. A [Y]//4 dtype

Re: [Numpy-discussion] fixing up datetime

2011-06-07 Thread Dave Hirschfeld
Christopher Barker Chris.Barker at noaa.gov writes: Dave Hirschfeld wrote: That would be one way of dealing with irregularly spaced data. I would argue that the example is somewhat back-to-front though. If something happens twice a month it's not occuring at a monthly frequency

Re: [Numpy-discussion] Changing the datetime operation unit rules

2011-06-07 Thread Dave Hirschfeld
Mark Wiebe mwwiebe at gmail.com writes: a = np.datetime64('today') a - a.astype('M8[Y]') numpy.timedelta64(157,'D') vs a = np.datetime64('today') a - a.astype('M8[Y]') Traceback (most recent call last): File stdin, line 1, in module TypeError: ufunc subtract cannot use

Re: [Numpy-discussion] numpy speed question

2010-11-25 Thread Dave Hirschfeld
Jean-Luc Menut jeanluc.menut at free.fr writes: I have a little question about the speed of numpy vs IDL 7.0. Here the IDL result: % Compiled module: $MAIN$. 2.837 The python code: from numpy import * from time import time time1 = time() for j in range(1): for

Re: [Numpy-discussion] How to import input data to make nda rray for batch processing?

2010-11-18 Thread Dave Hirschfeld
Venkat dvr002 at gmail.com writes: Hi All,I am new to Numpy (also Scipy).I am trying to reshape my text data which is in one single column (10,000 rows).I want the data to be in 100x100 array form.I have many files to convert like this. All of them are having file names like 0, 1, 2, 500.

Re: [Numpy-discussion] Printing formatted numerical values

2010-11-15 Thread Dave Hirschfeld
pv+numpy at math.duke.edu writes: Hi, what is the best way to print (to a file or to stdout) formatted numerical values? Analogously to C's printf(%d %g,x,y) etc? For stdout you can simply do: In [26]: w, x, y, z = np.randint(0,100,4) In [27]: type(w) Out[27]: type 'numpy.int32' In

Re: [Numpy-discussion] Unpleasant behavior with poly1d and numpy scalar multiplication

2010-02-15 Thread Dave Hirschfeld
Charles R Harris charlesr.harris at gmail.com writes: I was also thinking that someone might want to provide a better display at some point, drawing on a canvas, for instance. And what happens when the degree gets up over 100, which is quite reasonable with the Cheybshev polynomials? There