Francesc Altet wrote: > Hi James, > > A Monday 04 February 2008, James Philbin escrigué: >> Hi, >> >> I'm planning on using pytables for storing data on large image >> datasets (1M+) and while playing around with some code, came across a >> warning saying I had exceeded the maximum number of children (4096). >> My aim is to eventually have one child per image (ie millions of >> children), so i'm wondering where this limitation comes from? HDF5 >> itself seems to have no such limits. > > The maximum number of children is more a recommendation than a > limitation. Such recommendation is based on my experiences, and they > say that, when this limit is exceeded, HDF5 starts to use large amounts > of memory and access to metadata is slooow.
I also have this experience. And it's not just PyTables--the native utilities that come with HDF5 can have a lot of trouble dealing with large numbers of datasets. I found that for many uses I could speed things up by using a tree topology so that no node has more than 256 children at each step. I look forward to the fix. ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
