Re: [Pytables-users] Simple fast array I/O with Pytables

2012-01-03 Thread Francesc Alted
2012/1/2 Gael Varoquaux : > On Mon, Jan 02, 2012 at 07:18:24PM +0100, Gael Varoquaux wrote: >> > BTW, another cleaner, faster way to empty the OS filesystem cache is this: > >> > sudo echo 3 | sudo tee /proc/sys/vm/drop_caches > >> > [ from http://ubuntuforums.org/showthread.php?t=589975 ] > >> Tha

Re: [Pytables-users] Simple fast array I/O with Pytables

2012-01-03 Thread Francesc Alted
2012/1/2 Gael Varoquaux : > On Mon, Jan 02, 2012 at 06:44:54PM +0100, Francesc Alted wrote: >> Perhaps you may get a bit more performance if you use the >> `[read,write]_vl_blosc2_hdf` functions that I have sent in my earlier >> post, but that adds the python-blosc dependency (available at >> http:

Re: [Pytables-users] Simple fast array I/O with Pytables

2012-01-02 Thread Gael Varoquaux
On Mon, Jan 02, 2012 at 07:18:24PM +0100, Gael Varoquaux wrote: > > BTW, another cleaner, faster way to empty the OS filesystem cache is this: > > sudo echo 3 | sudo tee /proc/sys/vm/drop_caches > > [ from http://ubuntuforums.org/showthread.php?t=589975 ] > That's useful! And it works much bett

Re: [Pytables-users] Simple fast array I/O with Pytables

2012-01-02 Thread Gael Varoquaux
On Mon, Jan 02, 2012 at 06:44:54PM +0100, Francesc Alted wrote: > Perhaps you may get a bit more performance if you use the > `[read,write]_vl_blosc2_hdf` functions that I have sent in my earlier > post, but that adds the python-blosc dependency (available at > http://pypi.python.org/pypi/blosc/1.0

Re: [Pytables-users] Simple fast array I/O with Pytables

2012-01-02 Thread Francesc Alted
2012/1/2 Gael Varoquaux : > On Fri, Dec 30, 2011 at 02:16:51PM -0800, Dav Clark wrote: >> I'll add that it would be cool to see what numbers you come up with >> (maybe with some loose specs on the machine CPU and disk you used). > > Yes, I have been working on this for a little while, and it is act

Re: [Pytables-users] Simple fast array I/O with Pytables

2012-01-02 Thread Gael Varoquaux
On Fri, Dec 30, 2011 at 02:16:51PM -0800, Dav Clark wrote: > I'll add that it would be cool to see what numbers you come up with > (maybe with some loose specs on the machine CPU and disk you used). Yes, I have been working on this for a little while, and it is actually full of surprises. I have p

Re: [Pytables-users] Simple fast array I/O with Pytables

2011-12-31 Thread Gael Varoquaux
Hi, Sorry for the slow reply, I was away from my computer for a little while. Quickly (I must leave in 5mn), I have found what the problem was: when giving a large 4D fortran-ordered array, performance was fairly bad. Making it C ordered gave me much better performance. I tried also CArrays, and w

Re: [Pytables-users] Simple fast array I/O with Pytables

2011-12-30 Thread Francesc Alted
2011/12/30 Dav Clark : > On Dec 30, 2011, at 8:40 AM, Francesc Alted wrote: > >> 2011/12/30 Gael Varoquaux : >>> Hi list, >>> >>> I am trying to do a simple comparison of various I/O libraries to save a >>> bunch of numpy arrays. I don't have time to actually invest in PyTables >>> now, but it has

Re: [Pytables-users] Simple fast array I/O with Pytables

2011-12-30 Thread Dav Clark
On Dec 30, 2011, at 8:40 AM, Francesc Alted wrote: > 2011/12/30 Gael Varoquaux : >> Hi list, >> >> I am trying to do a simple comparison of various I/O libraries to save a >> bunch of numpy arrays. I don't have time to actually invest in PyTables >> now, but it has always been on my radar. I want

Re: [Pytables-users] Simple fast array I/O with Pytables

2011-12-30 Thread Francesc Alted
2011/12/30 Gael Varoquaux : > Hi list, > > I am trying to do a simple comparison of various I/O libraries to save a > bunch of numpy arrays. I don't have time to actually invest in PyTables > now, but it has always been on my radar. I wanted to get a ball-park > estimate of what was achievable with