On 10 Aug 2011, at 22:03, Gael Varoquaux wrote:
> On Wed, Aug 10, 2011 at 04:01:37PM -0400, Anne Archibald wrote:
>> A 1 Gb text file is a miserable object anyway, so it might be desirable
>> to convert to (say) HDF5 and then throw away the text file.
>
> +1
There might be concerns about ensuring data accessibility agains throwing the
text file away, but converting to HDF5 would be an elegant for reading in
without the memory issues, too (I must confess though, I've regularly read ~
1GB ASCII files into memory - with decent virtual memory management it did not
turn out too bad...)
Cheers,
Derek
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
