Hello Faisal, A Thursday 03 December 2009 00:32:17 Faisal Moledina escrigué: > Hello everyone, > > This is my first project with pytables and from the manual and videos, > it seems like it should suit my application very well. I'm currently > trying to figure out how to go about organizing my groups, tables, and > arrays logically for my simulation and post-processing. > > I'm running a stochastic simulation of Brownian motion for a number of > particles, for which I'll present a simplified version here. At each > time step, I determine if some particles have left the system, > determine the next position of the remaining particles, and then > introduce new particles into the system at defined starting > coordinates. I have a defined set of particle sources, which also act > as sinks. I would like to store information about these sources as > well as all the x-y-z location data for the particles. > > My current thought is to have two groups: sources and particles. > sources would look like: > > /sources Group > /sources/parameters Table (info such as # particles > released+captured, and other parameters) > /sources/coordinates Array (x-y-z coordinates) > > During the simulation, some values in /sources/parameters would be > modified but the coordinates would be fixed. > > My _main problem_ is thinking about organizing my particles. Each > particle has information associated with it such as its origin, > endpoint, release-time, capture-time, etc, which I thought to store in > one table. Each particle also has a history of x-y-z coordinates at > given timepoints. For this, I think that an EArray() seems best for > the x-y-z coordinates, and I'm not sure if the timepoint should also > be in this array or not. Also, because the simulation will be dealing > with 1e4 to 1e5 particles, I don't think it would be efficient to > create separate tables for them.
In a few words, the rules of thumb for using PyTables efficiently in these situations is: - Don't create too many leaves. It is better to stick with 'long' tables or arrays. - Do not create too 'wide' (that is, with too many columns) tables. If you need a lot of fields in a row (taking more than, say, 16 KB/row), it is way better to save some of the variables into EArrays (if their number of entries per table row is fixed) or VLArrays (if there are a variable number of entries per row). > If anyone has any thoughts on how to organize the data, that would be > helpful. I have found the thread on this mailing list "Advice on > organizing data" (http://tinyurl.com/yd88ts5) and the linked PDF > (http://zeus.ws.dei.polimi.it/is-manet/Documenti/pap-isti-5.pdf). I'm > still trying to understand them but I think I'm getting there. I need > a way to link the particle information with its location history. Yeah. The advices there continue to apply for your situation. Keep trying to understand them and if you end with a possible implementation and want to optimize it still further, you may want to share it with us so that we can comment on it. Luck! -- Francesc Alted ------------------------------------------------------------------------------ Join us December 9, 2009 for the Red Hat Virtual Experience, a free event focused on virtualization and cloud computing. Attend in-depth sessions from your desk. Your couch. Anywhere. http://p.sf.net/sfu/redhat-sfdev2dev _______________________________________________ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
