On 9/14/11 1:01 PM, Christopher Barker wrote:
numpy.ndarray.resize is a different method, and I'm pretty sure it
should be as fast or faster that np.empty + np.append.
My profile:
In [25]: %timeit f1 # numpy.resize()
1000 loops, best of 3: 163 ns per loop
In [26]: %timeit f2
On Wed, Sep 14, 2011 at 4:25 PM, Christopher Barker
chris.bar...@noaa.govwrote:
On 9/14/11 1:01 PM, Christopher Barker wrote:
numpy.ndarray.resize is a different method, and I'm pretty sure it
should be as fast or faster that np.empty + np.append.
My profile:
In [25]: %timeit f1 #
On 9/14/11 2:41 PM, Benjamin Root wrote:
Are you sure the f2 code works? a.resize() takes only a shape tuple. As
coded, you should get an exception.
wow, what an idiot!
I think I just timed how long it takes to raise that exception...
And when I fix that, I get a memory error.
When I fix
On Wed, Sep 14, 2011 at 5:30 PM, Christopher Barker
chris.bar...@noaa.gov wrote:
On 9/14/11 2:41 PM, Benjamin Root wrote:
Are you sure the f2 code works? a.resize() takes only a shape tuple. As
coded, you should get an exception.
wow, what an idiot!
I think I just timed how long it takes
On Sep 13, 2011, at 01:38 , Christopher Jordan-Squire wrote:
I did some timings to see what the advantage would be, in the simplest
case possible, of taking multiple lines from the file to process at a
time. Assuming the dtype is already known. The code is attached. What
I found was I can't
On 9/12/11 4:38 PM, Christopher Jordan-Squire wrote:
I did some timings to see what the advantage would be, in the simplest
case possible, of taking multiple lines from the file to process at a
time.
Nice work, only a minor comment:
f6 and f7 use stripped down versions of Chris
Barker's
On Tue, Sep 13, 2011 at 2:41 PM, Chris.Barker chris.bar...@noaa.gov wrote:
On 9/12/11 4:38 PM, Christopher Jordan-Squire wrote:
I did some timings to see what the advantage would be, in the simplest
case possible, of taking multiple lines from the file to process at a
time.
Nice work, only a
I did some timings to see what the advantage would be, in the simplest
case possible, of taking multiple lines from the file to process at a
time. Assuming the dtype is already known. The code is attached. What
I found was I can't use generators to avoid constructing a list and
then making a tuple
Wed, 07 Sep 2011 12:52:44 -0700, Chris.Barker wrote:
[clip]
In [9]: temp['x'] = 3
In [10]: temp['y'] = 4
In [11]: temp['z'] = 5
[clip]
maybe it wouldn't be any faster, but with re-using temp, and one less
list-tuple conversion, and fewer python type to numpy type conversions,
maybe it
On Wed, Sep 7, 2011 at 2:52 PM, Chris.Barker chris.bar...@noaa.gov wrote:
On 9/2/11 2:45 PM, Christopher Jordan-Squire wrote:
It doesn't have to parse the entire file to determine the dtypes. It
builds up a regular expression for what it expects to see, in terms of
dtypes. Then it just loops
On 9/8/11 1:43 PM, Christopher Jordan-Squire wrote:
I just ran a quick test on my machine of this idea. With
dt = np.dtype([('x',np.float32),('y', np.int32),('z', np.float64)])
temp = np.empty((), dtype=dt)
temp2 = np.zeros(1,dtype=dt)
In [96]: def f():
...: l=[0]*3
...:
On 9/2/11 2:45 PM, Christopher Jordan-Squire wrote:
It doesn't have to parse the entire file to determine the dtypes. It
builds up a regular expression for what it expects to see, in terms of
dtypes. Then it just loops over the lines, only parsing if the regular
expression doesn't match. It
On 02.09.2011, at 11:45PM, Christopher Jordan-Squire wrote:
and unfortunately it's for 1D-arrays only).
That's not bad for this use -- make a row a struct dtype, and you've got
a 1-d array anyway -- you can optionally convert to a 2-d array after
the fact.
I don't know why I didn't
On Tue, Sep 6, 2011 at 9:32 AM, Derek Homeier
de...@astro.physik.uni-goettingen.de wrote:
On 02.09.2011, at 11:45PM, Christopher Jordan-Squire wrote:
and unfortunately it's for 1D-arrays only).
That's not bad for this use -- make a row a struct dtype, and you've got
a 1-d array anyway -- you
On 30.08.2011, at 6:21PM, Chris.Barker wrote:
I've submitted a pull request for a new method for loading data from
text files into a record array/masked record array.
Click on the link for more info, but the general idea is to create a
regular expression for what entries should look like
On 9/2/11 8:22 AM, Derek Homeier wrote:
I agree it would make a very nice addition, and could complement my
pre-allocation option for loadtxt - however there I've also been made
aware that this approach breaks streamed input etc., so the buffer.resize(…)
methods in accumulator would be the
Sorry I'm only now getting around to thinking more about this. Been
side-tracked by stats stuff.
On Fri, Sep 2, 2011 at 10:50 AM, Chris.Barker chris.bar...@noaa.gov wrote:
On 9/2/11 8:22 AM, Derek Homeier wrote:
I agree it would make a very nice addition, and could complement my
pre-allocation
On 02.09.2011, at 5:50PM, Chris.Barker wrote:
hmmm -- it seems you could jsut as well be building the array as you go,
and if you hit a change in the imput, re-set and start again.
In my tests, I'm pretty sure that the time spent file io and string
parsing swamp the time it takes to
On 02.09.2011, at 6:16PM, Christopher Jordan-Squire wrote:
I hadn't thought of that. Interesting idea. I'm surprised that
completely resetting the array could be faster.
I had experimented a bit with the fromiter function, which also increases
the output array as needed, and this creates
On 9/2/11 9:16 AM, Christopher Jordan-Squire wrote:
I agree it would make a very nice addition, and could complement my
pre-allocation option for loadtxt - however there I've also been made
aware that this approach breaks streamed input etc., so the buffer.resize(…)
methods in accumulator
On Fri, Sep 2, 2011 at 3:54 PM, Chris.Barker chris.bar...@noaa.gov wrote:
On 9/2/11 9:16 AM, Christopher Jordan-Squire wrote:
I agree it would make a very nice addition, and could complement my
pre-allocation option for loadtxt - however there I've also been made
aware that this approach
On 8/27/11 11:08 AM, Christopher Jordan-Squire wrote:
I've submitted a pull request for a new method for loading data from
text files into a record array/masked record array.
Click on the link for more info, but the general idea is to create a
regular expression for what entries should look
22 matches
Mail list logo
