Hello, We have a ~30GB HDF5 file with something like 100 million small datasets in it. We need to iterate through all of them, and doing so is very slow as each one has to be loaded from disk. I also don't know if it is possible to find out a proper ordering to go through them, so I suspect that there might also be a lot of disk seeks necessary.
Maybe it isn't such a good idea to have so many small objects in the file, but I'm stuck with this format now. What options do I have? I'm working now on a machine with 128GB of RAM, so my file would fit comfortably inside. Is it possible to load the file completely into memory to avoid all of the above problems? Thanks, Thorben _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
