A Wednesday 10 March 2010 08:48:33 Jorge Scandaliaris escrigué: > Hi, > Some time ago I started using pytables for organizing large amounts of > data, I it has worked really nice until now. Up to now I had an append > only usage pattern. Because of the nature of the data, for each > measurement there are a number of fixed fields plus a variable length > array, I was using a table plus a VLArray. My problem now is that I need > to change to an add/remove data usage pattern, and so the VLArray has > turned into a problem. > What would be my options now? That is, and add/remove usage pattern for > variable sized arrays. I searched the list yesterday, and it seems one > option is to defined a fixed size array large enough to hold the biggest > expected array and use only a table, but in case I overlooked something, I > decided to ask here.
Yes, having everything in a single table and using compression to reduce unused space is the simplest option. If your maximum length for variable length field is high (>1000 bytes), you can still use a VLArray for keeping them, and add another level of indirection in case you want to add/remove rows. The level of indirection can be just an external EArray that keeps track of the affected operations in the VLArray. For example, the EArray can have the contents: [0,2,3,4] that means that row 1 in VLArray has been deleted. Or: [3,2,1,0] where you decided to change the order of the rows in the referred VLArray. This will offer you the degree of flexibility you need, at the expense of more complexity. It is up to you to choose this approach of the simpler one above. Hope this helps, -- Francesc Alted ------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev _______________________________________________ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
