Daniel, But, the data is exactly what I want to see when I use h5dump, not the header. If I just want to traverse the data, I'll use h5ls -rv which I find to be lightning fast.
For instance, I store grid and mesh data in groups /grid and /mesh - this has info like grid spacing, Cartesian location of gridpoints, and the x y and z size of the 3d data in the file. A few hundred bytes at most. Sometime I just want to h5dump -g /mesh What I find is that it can take 10 or so seconds for this to complete with a ~ 2GB file which is 99.999% 3D floating point data. I am worried that perhaps the mesh data (which I write before the 3D data) is somehow at the end of the file. It could be an issue with h5dump. I just want to be sure before I write out a few hundred TB of data for analysis. FWIW I have looked at http://www.hdfgroup.org/HDF5/doc/H5.format.html but it's more of a spec sheet and I couldn't find anything addressing my specific question. Leigh On Wed, Jul 6, 2011 at 11:51 AM, <[email protected]> wrote: > Leigh Orf wrote: > > My concern - perhaps unfounded - is that the very small bits of what I > > call > > metadata (integers, lists of what variables are in the file, and other > > very > > small bits of data I write which describe stuff like the 3D data and is > > necessary for my reader code) will be placed after the huge 3d data such > > that accessing it will require long seeks through 3d data. The only > reason > > I > > am worried about this is I noticed doing a h5dump on one of my small > > metadata datasets that it took more than 10 seconds to output data on one > > of > > my files. I got the impression that perhaps h5dump was having to make its > > way through the 3d arrays before getting to the metadata. However, my C > > code > > seemed to access the metadata quickly; perhaps it's an issue with h5dump. > > I hit the same issue when inspecting metadata in some files. `h5dump -H` > was what I needed. > > - Daniel > > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org > -- Leigh Orf Associate Professor of Atmospheric Science Department of Earth and Atmospheric Science Central Michigan University
_______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
