Re: [Numpy-discussion] using loadtxt to load a text file in to a numpy array

2014-01-22 Thread Oscar Benjamin
On Tue, Jan 21, 2014 at 06:54:33PM -0700, Andrew Collette wrote: Hi Chris, it looks from here: http://www.hdfgroup.org/HDF5/doc/ADGuide/WhatsNew180.html that HDF uses utf-8 for unicode strings -- so you _could_ roundtrip with a lot of calls to encode/decode -- which could be pretty

Re: [Numpy-discussion] fromiter cannot create array of object - was: Creating an ndarray from an iterable, over sequences

2014-01-22 Thread Sebastian Berg
On Wed, 2014-01-22 at 07:58 +0100, Dr. Leo wrote: Hi, thanks. Both recarray and itertools.chain work just fine in the example case. However, the real purpose of this is to read strings from a large xml file into a pandas DataFrame. But fromiter cannot create arrays of dtype 'object'.

[Numpy-discussion] accumulation operation

2014-01-22 Thread Ralf Juengling
Executing the following code, import numpy as np a = np.zeros((3,)) w = np.array([0, 1, 0, 1, 2]) v = np.array([10.0, 1, 10.0, 2, 9]) a[w] += v I was expecting 'a' to be array([20., 3., 9.]. Instead I get a array([ 10., 2., 9.]) This with numpy version 1.6.1. Is there another way to

Re: [Numpy-discussion] accumulation operation

2014-01-22 Thread Sebastian Berg
On Wed, 2014-01-22 at 17:23 +, Ralf Juengling wrote: Executing the following code, import numpy as np a = np.zeros((3,)) w = np.array([0, 1, 0, 1, 2]) v = np.array([10.0, 1, 10.0, 2, 9]) a[w] += v I was expecting ‘a’ to be array([20., 3., 9.]. Instead I get

Re: [Numpy-discussion] accumulation operation

2014-01-22 Thread Julian Taylor
On 22.01.2014 18:23, Ralf Juengling wrote: Executing the following code, import numpy as np a = np.zeros((3,)) w = np.array([0, 1, 0, 1, 2]) v = np.array([10.0, 1, 10.0, 2, 9]) a[w] += v I was expecting ‘a’ to be array([20., 3., 9.]. Instead I get a array([

Re: [Numpy-discussion] using loadtxt to load a text file in to a numpy array

2014-01-22 Thread Andrew Collette
Hi Oscar, Is it fair to say that people should really be using vlen utf-8 strings for text? Is it problematic because of the need to interface with non-Python libraries using the same hdf5 file? The general recommendation has been to use fixed-width strings for exactly that reason; FORTRAN

Re: [Numpy-discussion] using loadtxt to load a text file in to a numpy array

2014-01-22 Thread Chris Barker
On Wed, Jan 22, 2014 at 2:46 AM, Oscar Benjamin oscar.j.benja...@gmail.comwrote: BTW, as much as the fixed-width 'S' dtype doesn't really work for str in Python 3 it's also a poor fit for bytes since it strips trailing nulls: a = np.array(['a\0s\0', 'qwert'], dtype='S') a

Re: [Numpy-discussion] using loadtxt to load a text file in to a numpy array

2014-01-22 Thread Oscar Benjamin
On Wed, Jan 22, 2014 at 12:07:28PM -0800, Chris Barker wrote: On Wed, Jan 22, 2014 at 2:46 AM, Oscar Benjamin oscar.j.benja...@gmail.comwrote: BTW, as much as the fixed-width 'S' dtype doesn't really work for str in Python 3 it's also a poor fit for bytes since it strips trailing nulls:

Re: [Numpy-discussion] (no subject)

2014-01-22 Thread Ralf Gommers
On Tue, Jan 21, 2014 at 5:46 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Tue, Jan 21, 2014 at 9:26 AM, jennifer stone jenny.stone...@gmail.comwrote: What are your interests and experience? If you use numpy, are there things you would like to fix, or enhancements you would

Re: [Numpy-discussion] using loadtxt to load a text file in to a numpy array

2014-01-22 Thread Chris Barker - NOAA Federal
On Jan 22, 2014, at 1:13 PM, Oscar Benjamin oscar.j.benja...@gmail.com wrote: It's not safe to stop removing the null bytes. This is how numpy determines the length of the strings in a dtype='S' array. The strings are not fixed-width but rather have a maximum width. Exactly--but folks have