Hi Mike, A Dijous 16 Març 2006 18:43, vàreu escriure: > >> In looking for additional ways to make my files more efficient, > >> I came across the H5Fmount capability while perusing the HDF5 > >> documentation. I couldn't find this in the PyTables docs so can > >> you tell me if PyTables supports this? > >> > > Right now we have no plans to add this feature. Can you explain > > a little under which situations do you need this? > > I'll go ahead and chime in here sine I too would like to see this > feature in pytables. I am working in a group that has just started to > use pytables to store grids generated by weather models. Each model > run produces anywhere from 20mb to 2Gb of data. > > When a new model run comes in, we are storing it in pytables and > then removing the older run. It seems that the pytables file is > suffering a bit from fragmentation or something. The file size keeps > increasing (we are not using undo). So, it seems that pythables is > not reusing alot of the freed space. Or it might mean that we are not > doing something right :).
Mmm, right now, I think that PyTables can reclaim the space freed in a file only if it is reclaimed in the same opening session (this is ultimately an HDF5 limitation). So, perhaps you can get a better behaviour if you remove the older run *before* storing the new one. In any case, if you do this kind of things frequently, it is always good to run ptrepack (or h5repack for that matter) on the affected files in order to reclaim unused space. > > Anyway, I think it would be great if we could place each model run > in a seperate file and then use the hdf5 mount capability to place it > into a single "main" file. This way, most of the software could just > think that all data existed in the "main" file. And when a new model > run comes in then we just: > > - write the data to a new pytables file > - mount this new file in the main file > - unmount the old file > - remove the old file > > I think this would reduce the fragmentation of the main file. And > it would require fewer File.copyFile() calls to consolidate a single > main file. Yes, this is a valid point. We will add such a "mount capability" to our TODO list. BTW, I see that you discovered by yourself that running File.copyFile() is an equivalent way of running h5repack ;-) Chers, -- >0,0< Francesc Altet http://www.carabos.com/ V V Cárabos Coop. V. Enjoy Data "-" ------------------------------------------------------- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid0944&bid$1720&dat1642 _______________________________________________ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
