[Numpy-discussion] R: fast numpy.fromfile skipping data chunks

2013-03-13 Thread Andrea Cimatoribus
This solution does not work for me since I have an offset before the data that is not a multiple of the datatype (it's a header containing various stuff). I'll at pytables. # Exploit the operating system's virtual memory manager to get a virtual copy of the entire file in memory # (This does not

[Numpy-discussion] R: fast numpy.fromfile skipping data chunks

2013-03-13 Thread Andrea Cimatoribus
I see that pytables deals with hdf5 data. It would be very nice if the data were in such a standard format, but that is not the case, and that can't be changed. Da: numpy-discussion-boun...@scipy.org [numpy-discussion-boun...@scipy.org] per conto di

Re: [Numpy-discussion] R: fast numpy.fromfile skipping data chunks

2013-03-13 Thread Nathaniel Smith
On Wed, Mar 13, 2013 at 2:18 PM, Andrea Cimatoribus andrea.cimatori...@nioz.nl wrote: This solution does not work for me since I have an offset before the data that is not a multiple of the datatype (it's a header containing various stuff). np.memmap takes an offset= argument. -n

[Numpy-discussion] R: fast numpy.fromfile skipping data chunks

2013-03-13 Thread Andrea Cimatoribus
Thanks a lot for the feedback, I'll try to modify my function to overcome this issue. Since I'm in the process of buying new hardware too, a slight OT (but definitely related). Does an ssd provide substantial improvement in these cases? Da:

Re: [Numpy-discussion] R: fast numpy.fromfile skipping data chunks

2013-03-13 Thread Daπid
On 13 March 2013 16:54, Andrea Cimatoribus andrea.cimatori...@nioz.nlwrote: Since I'm in the process of buying new hardware too, a slight OT (but definitely related). Does an ssd provide substantial improvement in these cases? It should help. Nevertheless, when talking about performance, it

Re: [Numpy-discussion] R: fast numpy.fromfile skipping data chunks

2013-03-13 Thread Charles R Harris
On Wed, Mar 13, 2013 at 9:54 AM, Andrea Cimatoribus andrea.cimatori...@nioz.nl wrote: Thanks a lot for the feedback, I'll try to modify my function to overcome this issue. Since I'm in the process of buying new hardware too, a slight OT (but definitely related). Does an ssd provide