Re: [Pytables-users] Shuffle and performance

2007-08-28 Thread elias . collas
> > Aha, so you are doing a binary search in an 'index' first; then it is > almost sure that most of the time is spent in performing the look-up in > this rank-1 array. As you are doing binary search, and the minimum > amount of I/O chunk in HDF5 is precisely the chunksize, having small > chu

Re: [Pytables-users] Shuffle and performance

2007-08-28 Thread Francesc Altet
A Monday 27 August 2007, escriguéreu: > > Yeah, that's a bit strange. If 're-adding' shuffle is actually > > improving your search times, then perhaps it is not the actual > > problem. Now, I think that the main issue should be the length of > > the chunksize of 'new' files. Can you run the 'h5ls