On 30/11/2016 16:16, Heli wrote:
Hi all,Writing my ASCII file once to either of pickle or npy or hdf data types and then working afterwards on the result binary file reduced the read time from 80(min) to 2 seconds.
240,000% faster? Something doesn't sound quite right! How big is the file now? (The one that had been 40GB of text and contained 100M lines or datasets, although file caching is just about a possibility.)
-- Bartc -- https://mail.python.org/mailman/listinfo/python-list
