Re: [Numpy-discussion] Add a function to broadcast arrays to a given shape to numpy's stride_tricks?

2014-12-10 Thread Stephan Hoyer
On Sun, Dec 7, 2014 at 11:31 PM, Pierre Haessig pierre.haes...@crans.org wrote: Instead of putting this function in stride_tricks (which is quite hidden), could it be added instead as a boolean flag to the existing `reshape` method ? Something like: x.reshape(y.shape, broadcast=True) What

Re: [Numpy-discussion] Add a function to broadcast arrays to a given shape to numpy's stride_tricks?

2014-12-10 Thread Stephan Hoyer
On Wed, Dec 10, 2014 at 4:00 PM, Nathaniel Smith n...@pobox.com wrote: 2) Add a broadcast_to(arr, shape) function, which broadcasts the array to exactly the shape given, or else errors out if this is not possible. I like np.broadcast_to as a new function. We can document it alongside

[Numpy-discussion] Add a function to broadcast arrays to a given shape to numpy's stride_tricks?

2014-12-06 Thread Stephan Hoyer
I recently wrote function to manually broadcast an ndarray to a given shape according to numpy's broadcasting rules (using strides): https://github.com/xray/xray/commit/7aee4a3ed2dfd3b9aff7f3c5c6c68d51df2e3ff3 The same functionality can be done pretty straightforwardly with np.broadcast_arrays,

Re: [Numpy-discussion] Finding values in an array

2014-11-27 Thread Stephan Hoyer
On Thu, Nov 27, 2014 at 10:15 PM, Alexander Belopolsky ndar...@mac.com wrote: I probably miss something very basic, but how given two arrays a and b, can I find positions in a where elements of b are located? If a were sorted, I could use searchsorted, but I don't want to get valid positions

Re: [Numpy-discussion] Why ndarray provides four ways to flatten?

2014-10-28 Thread Stephan Hoyer
On Tue, Oct 28, 2014 at 10:25 AM, Nathaniel Smith n...@pobox.com wrote: I too would be curious to know why .flat exists (beyond it seemed like a good idea at the time ;-)). I've always treated it as some weird legacy thing and ignored it, and this has worked out well for me. Is there any

Re: [Numpy-discussion] Add an axis argument to generalized ufuncs?

2014-10-19 Thread Stephan Hoyer
On Sat, Oct 18, 2014 at 6:46 PM, Nathaniel Smith n...@pobox.com wrote: One thing we'll have to watch out for is that for reduction operations (which are basically gufuncs with (n)-() signatures), we already allow axis=(0,1) to mean reshape axes 0 and 1 together into one big axis, and then use

Re: [Numpy-discussion] Add an axis argument to generalized ufuncs?

2014-10-19 Thread Stephan Hoyer
On Sun, Oct 19, 2014 at 6:43 AM, Nathaniel Smith n...@pobox.com wrote: I feel strongly that we should come up with a syntax that is unambiguous even *without* looking at the gufunc signature. It's easy for the computer to disambiguate stuff like this, but it'd be cruel to ask people trying to

[Numpy-discussion] Add an axis argument to generalized ufuncs?

2014-10-17 Thread Stephan Hoyer
Yesterday I created a GitHub issue proposing adding an axis argument to numpy's gufuncs: https://github.com/numpy/numpy/issues/5197 I was told I should repost this on the mailing list, so here's the recap: I would like to write generalized ufuncs (probably using numba), to create fast functions

Re: [Numpy-discussion] numpy.mean slicing in a netCDF file

2014-10-14 Thread Stephan Hoyer
Hi Fadzil, My strong recommendation is that you don't just use numpy/netCDF4 to process your data, but rather use one of a multitude of packages that have been developed specifically to facilitate working with labeled data from netCDF files: - Iris: http://scitools.org.uk/iris/ - CDAT:

Re: [Numpy-discussion] Request for enhancement to numpy.random.shuffle

2014-10-12 Thread Stephan Hoyer
On Sun, Oct 12, 2014 at 10:56 AM, Jaime Fernández del Río jaime.f...@gmail.com wrote: Just to add some noise to a productive conversation: if you add a 'copy' flag to shuffle, then all the functionality is in one place, and 'permutation' can either be deprecated, or trivially implemented in

Re: [Numpy-discussion] use ufunc for arbitrary positional arguments?

2014-10-10 Thread Stephan Hoyer
On Fri, Oct 10, 2014 at 11:23 AM, Benjamin Root ben.r...@ou.edu wrote: I have a need to and together an arbitrary number of boolean arrays. np.logical_and() expects only two positional arguments. There has got to be some sort of easy way to just and these together using the ufunc mechanism,

Re: [Numpy-discussion] 0/0 == 0?

2014-10-03 Thread Stephan Hoyer
On Thu, Oct 2, 2014 at 11:29 PM, Nathaniel Smith n...@pobox.com wrote: The seterr warning system makes a lot of sense for IEEE754 floats, which are specifically designed so that 0/0 has a unique well-defined answer. For ints though this seems really broken to me. 0 / 0 = 0 is just the wrong

Re: [Numpy-discussion] Proposal: add ndarray.keys() to return dtype.names

2014-09-30 Thread Stephan Hoyer
I like this idea. But I am -1 on returning None if the array is unstructured. I expect .keys(), if present, to always return an iterable. In fact, this would break some of my existing code, which checks for the existence of keys as a way to do duck typed checks for dictionary like objects (e.g.,

Re: [Numpy-discussion] Proposal: add ndarray.keys() to return dtype.names

2014-09-30 Thread Stephan Hoyer
On Tue, Sep 30, 2014 at 1:22 PM, Eelco Hoogendoorn hoogendoorn.ee...@gmail.com wrote: On more careful reading of your words, I think we agree; indeed, if keys() is present is should return an iterable; but I don't think it should be present for non-structured arrays. Indeed, I think we do

Re: [Numpy-discussion] Custom dtypes without C -- or, a standard ndarray-like type

2014-09-23 Thread Stephan Hoyer
On Sun, Sep 21, 2014 at 8:31 PM, Nathaniel Smith n...@pobox.com wrote: For cases where people genuinely want to implement a new array-like types (e.g. DataFrame or scipy.sparse) then numpy provides a fair amount of support for this already (e.g., the various hooks that allow things like

[Numpy-discussion] Custom dtypes without C -- or, a standard ndarray-like type

2014-09-21 Thread Stephan Hoyer
pandas has some hacks to support custom types of data for which numpy can't handle well enough or at all. Examples include datetime and Categorical [1], and others like GeoArray [2] that haven't make it into pandas yet. Most of these look like numpy arrays but with custom dtypes and type specific

[Numpy-discussion] ANN: xray 0.3 released

2014-09-21 Thread Stephan Hoyer
I'm pleased to announce the v0.3 release for xray, N-D labeled arrays and datasets in Python. xray is an open source project and Python package that aims to bring the labeled data power of pandas to the physical sciences, by providing N-dimensional variants of the core pandas data structures,

Re: [Numpy-discussion] Generalize hstack/vstack -- stack; Block matrices like in matlab

2014-09-08 Thread Stephan Hoyer
On Mon, Sep 8, 2014 at 10:00 AM, Benjamin Root ben.r...@ou.edu wrote: Btw, on a somewhat related note, whoever can implement ndarray to be able to use views from other ndarrays stitched together would get a fruit basket from me come the holidays and possibly naming rights for the next kid...

Re: [Numpy-discussion] Proposed new feature for numpy.einsum: repeated output subscripts as diagonal

2014-08-14 Thread Stephan Hoyer
I think this would be very nice addition. On Thu, Aug 14, 2014 at 12:21 PM, Benjamin Root ben.r...@ou.edu wrote: You had me at Kronecker delta... :-) +1 On Thu, Aug 14, 2014 at 3:07 PM, Pierre-Andre Noel noel.pierre.an...@gmail.com wrote: (I created issue 4965 earlier today on this

Re: [Numpy-discussion] String type again.

2014-07-17 Thread Stephan Hoyer
On Mon, Jul 14, 2014 at 10:00 AM, Olivier Grisel olivier.gri...@ensta.org wrote: 2014-07-13 19:05 GMT+02:00 Alexander Belopolsky ndar...@mac.com: I've been toying with the idea of creating an array type for interned strings. In many applications dealing with large arrays of variable size

Re: [Numpy-discussion] Fast way to convert (nested) list to numpy object array?

2014-07-03 Thread Stephan Hoyer
On Thu, Jul 3, 2014 at 5:36 AM, Marc Hulsman m.huls...@tudelft.nl wrote: This can however go wrong. Say that we have nested variable length lists, what sometimes happens is that part of the data has (by chance) only fixed length nested lists, while another part has variable length nested

Re: [Numpy-discussion] Fwd: [Python-ideas] PEP pre-draft: Support for indexing with keyword arguments

2014-07-02 Thread Stephan Hoyer
NumPy doesn't have named axes, but perhaps it should. See, for example, Fernando Perez's datarray prototype (https://github.com/fperez/datarray) or my project, xray (https://github.com/xray/xray). Syntactical support for indexing an axis by name would makes using named axes much more readable.

Re: [Numpy-discussion] Find Daily max - create lists using date and add hourly data to that list for the day

2014-05-21 Thread Stephan Hoyer
Hello anonymous, I recently wrote a package xray (http://xray.readthedocs.org/) specifically to make it easier to work with high-dimensional labeled data, as often found in NetCDF files. Xray has a groupby method for grouping over subsets of your data, which would seem well suited to what you're

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-04-18 Thread Stephan Hoyer
On Mon, Apr 14, 2014 at 11:59 AM, Chris Barker chris.bar...@noaa.govwrote: - datetime64 objects with high precision (e.g., ns) can't compare to datetime objects. That's a problem, but how do you think it should be handled? My thought is that it should round to microseconds, and then compare

Re: [Numpy-discussion] min depth to nonzero in 3d array

2014-04-17 Thread Stephan Hoyer
Hi Alan, You can abuse np.argmax to calculate the first nonzero element in a vectorized manner: import numpy as np A = (2 * np.random.rand(100, 50, 50)).astype(int) Compare: np.argmax(A != 0, axis=0) np.array([[np.flatnonzero(A[:,i,j])[0] for j in range(50)] for i in range(50)]) You'll also

Re: [Numpy-discussion] Dates and times and Datetime64 (again)

2014-04-11 Thread Stephan Hoyer
On Fri, Apr 11, 2014 at 3:56 PM, Charles R Harris charlesr.har...@gmail.com wrote: Are we in a position to start looking at implementation? If so, it would be useful to have a collection of test cases, i.e., typical uses with specified results. That should also cover conversion from/(to?)

Re: [Numpy-discussion] Transparently reading complex arrays from netcdf4

2014-03-30 Thread Stephan Hoyer
smart about taking advantage of the mmap when possible. But perhaps your solution is the best compromise. Thanks again, Glenn On Mar 29, 2014 10:59 PM, Stephan Hoyer sho...@gmail.com wrote: Hi Glenn, My usual strategy for this sort of thing is to make a light-weight wrapper class which reads

Re: [Numpy-discussion] Transparently reading complex arrays from netcdf4

2014-03-30 Thread Stephan Hoyer
is built in for free. Thanks, Glenn On Mar 30, 2014 2:18 AM, Stephan Hoyer sho...@gmail.com wrote: Hi Glenn, Here is a full example of how we wrap a netCDF4.Variable object, implementing all of its ndarray-like methods: https://github.com/akleeman/xray/blob

Re: [Numpy-discussion] Transparently reading complex arrays from netcdf4

2014-03-29 Thread Stephan Hoyer
Hi Glenn, My usual strategy for this sort of thing is to make a light-weight wrapper class which reads and converts values when you access them. For example: class WrapComplex(object): def __init__(self, nc_var): self.nc_var = nc_var def __getitem__(self, item): return

Re: [Numpy-discussion] [RFC] should we argue for a matrix power operator, @@?

2014-03-15 Thread Stephan Hoyer
Speaking only for myself (and as someone who has regularly used matrix powers), I would not expect matrix power as @@ to follow from matrix multiplication as @. I do agree that matrix power is the only reasonable use for @@ (given @), but it's still not something I would be confident enough to

[Numpy-discussion] Allowing slices as arguments for ndarray.take

2014-01-16 Thread Stephan Hoyer
There was a discussion last year about slicing along specified axes in numpy arrays: http://mail.scipy.org/pipermail/numpy-discussion/2012-April/061632.html I'm finding that slicing along specified axes is a common task for me when writing code to manipulate N-D arrays. The method ndarray.take

<    1   2