Re: [Pytables-users] Organization of Simple Coordinate Data

2010-06-02 Thread Joshua Adelman
The data that I'm trying to store isn't linear in the sense that you have a single molecule and then you have a snapshot of it to store at each iteration. It's more like there is a variable number of molecules that need to be stored at each iteration, and each of those molecules carry a unique i

Re: [Pytables-users] Organization of Simple Coordinate Data

2010-06-02 Thread Francesc Alted
Ah, you are ending with a table that is more than 6 GB large, and that's probably more than the memory in your laptop, so the query has to traverse all the table *on-disk*, and this is why the query is slow. I suppose that I was thinking that you was going to use indexing, but forgot that this

Re: [Pytables-users] Organization of Simple Coordinate Data

2010-06-02 Thread Joshua Adelman
Hi Francesc, Thank you for your reply. I've done some testing of building a pytables file with a single table as set up as follows: class Seg(tables.IsDescription): n_iter = tables.UInt32Col(pos=0) seg_id = tables.UInt32Col(pos=1) x = tables.Float32Col(shape=(405,3),pos=2) hf

Re: [Pytables-users] Organization of Simple Coordinate Data

2010-06-02 Thread Francesc Alted
A Wednesday 02 June 2010 05:22:15 Joshua Adelman escrigué: > Looking through the archives there seems to be a number of suggestions on > organizing data, but all of the ones I could find are for data sets that > are more complicated than I need. > > I just need to store the 3D coordinates (~ 500

[Pytables-users] Organization of Simple Coordinate Data

2010-06-01 Thread Joshua Adelman
Looking through the archives there seems to be a number of suggestions on organizing data, but all of the ones I could find are for data sets that are more complicated than I need. I just need to store the 3D coordinates (~ 500x3 numpy array) of a number of instances of a system for each itera