On 5/18/07, David M. Cooke <[EMAIL PROTECTED]> wrote:
On Fri, May 18, 2007 at 03:10:23PM +0300, dmitrey wrote:
> hi all,
> what is best way for storing data in numpy array? (amount of memory for
> preallocating is unknown)
> Currently I use just a Python list, i.e.
>
> r = []
> for i in xrange(N)#N is very big
> ...
> r.append(some_value)
In the above, you know how big you need b/c you know N ;-) so empty is
a good choice:
r = empty((N,), dtype=float)
for i in xrange(N):
r[i] = some_value
empty() allocates the array, but doesn't clear it or anything (as
opposed to zeros(), which would set the elements to zero).
If you don't know N, then fromiter would be best:
def ivalues():
while some_condition():
...
yield some_value
r = fromiter(ivalues(), dtype=float)
It'll act like appending to a list, where it will grow the array (by
doubling, I think)
It uses 50% overallocation. It also reallocs at the end in an attempt to
give back any extra space. I'm not sure is that's actually effective though.
when it needs to, so appending each value is
amortized to O(1) time. A list though would use more memory
per element as each element is a full Python object.
Note that if you are pulling from an iterator, even if you know the length,
fromiter may well be better since you can specify a length with the optional
third argument.
r = fromiter(some_value_iter, float, N)
-tim
--
//=][=\\
[EMAIL PROTECTED]
_______________________________________________
Numpy-discussion mailing list
[email protected]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
