Re: [Numpy-discussion] R: fast numpy.fromfile skipping data chunks
On Wed, Mar 13, 2013 at 2:18 PM, Andrea Cimatoribus andrea.cimatori...@nioz.nl wrote: This solution does not work for me since I have an offset before the data that is not a multiple of the datatype (it's a header containing various stuff). np.memmap takes an offset= argument. -n ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] R: fast numpy.fromfile skipping data chunks
On 13 March 2013 16:54, Andrea Cimatoribus andrea.cimatori...@nioz.nlwrote: Since I'm in the process of buying new hardware too, a slight OT (but definitely related). Does an ssd provide substantial improvement in these cases? It should help. Nevertheless, when talking about performance, it is difficult to predict, mainly because in a computer there are many things going on and many layers involved. I have a couple of computers equipped with SSD, if you want, if you send me some benchmarks I can run them and see if I get any speedup. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] R: fast numpy.fromfile skipping data chunks
On Wed, Mar 13, 2013 at 9:54 AM, Andrea Cimatoribus andrea.cimatori...@nioz.nl wrote: Thanks a lot for the feedback, I'll try to modify my function to overcome this issue. Since I'm in the process of buying new hardware too, a slight OT (but definitely related). Does an ssd provide substantial improvement in these cases? It should. Seek time on an ssd is quite low, and readout is fast. Skipping over items will probably not be as fast as a sequential read but I expect it will be substantially faster than a disk. Nathaniel's loop idea will probably work faster also. The sequential readout rate of a modern ssd will be about 500 MB/sec, so you can probably just divide that into your file size to get an estimate of the time needed. snip Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
