A Dissabte 07 Abril 2007 13:55, Michael Hoffman escrigué:
> PyTables has been a great help to my research. I was wondering if I
> could make my use somewhat more efficient.
>
> For a particular project, I produce about 22000 tables of 2000 rows
> each. These are initially produced by a distributing computing farm into
> 2500 files, but I concatenate them so that (a) I won't have so many
> files lying around, which our system administrators hate, and (b) I can
> randomly access tables by name easily.
>
> Of course, having about 4 GiB and 22000 tables in one file slows things
> down a bit, especially since it is stored on a remote Lustre file
> system. One thing I thought of was to find some middle ground and
> concatenate the original file set into a small number of files, but not
> just one. Then I could make a separate file for an index to provide
> random access.
>
> Is this a good idea? Any suggestions as to a target number of datasets
> (I know 4096 was once suggested as the max) or data size per file? Are
> there any facilities within PyTables or elsewhere to make this easier?

Well, it largely depends on your requeriments. In these days that disks do 
offer huge capacities at reasonable prices, I'd advocate by creating a 
monolithic file containing all your data.  You can use compression in order 
to reduce the size of the file.  Also, reducing the number of tables and 
enlarging them can also contribute to keep the demand for space low and, as a 
side effect, reduce the number of objects in the hierarchy too.  In 
principal, accessing a large number of objects in the tree shouldn't exhibit 
a very noticiable slowdown, specially when the LRU cache of objects starts to 
warm.

Regarding the limitation of 4096 nodes per group, it was removed some time ago 
(PyTables 1.2?), but it is true that keeping this number low will help making 
the object browsing faster, so in case you have too many objects, you can 
always endow them with a deeper tree.

In case this is definitely not feasible for you, then I'd go by splitting the  
file to the limit it is manageable and build a separate index in order to 
access them, as you suggest.  For doing this, another possibility is that 
PyTables would support the mounting of external files in one place of the 
hierarchy (very similar as mounting a filesystem in another). Unfortunately, 
and despite the fact that HDF5 already comes with such a support, it has not 
been implemented yet on PyTables (and to say the truth, we have no plans in 
the near end to implement it).

Hope that helps,

-- 
>0,0<   Francesc Altet     http://www.carabos.com/
V   V   Cárabos Coop. V.   Enjoy Data
 "-"

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users

Reply via email to