Hi Leigh,
I'm guessing it's just a performance issue with how h5dump is creating
its internal data structures: it needs to walk the entire file at the
beginning, in order to properly handle forward references (in hard/soft links,
and in reference datatypes) and other internal statistics, etc. Normal file
opens don't traverse the entire file and only load metadata on demand.
Quincey
On Jul 6, 2011, at 11:13 AM, Leigh Orf wrote:
> Daniel,
>
> But, the data is exactly what I want to see when I use h5dump, not the
> header. If I just want to traverse the data, I'll use h5ls -rv which I find
> to be lightning fast.
>
> For instance, I store grid and mesh data in groups /grid and /mesh - this has
> info like grid spacing, Cartesian location of gridpoints, and the x y and z
> size of the 3d data in the file. A few hundred bytes at most.
>
> Sometime I just want to
>
> h5dump -g /mesh
>
> What I find is that it can take 10 or so seconds for this to complete with a
> ~ 2GB file which is 99.999% 3D floating point data.
>
> I am worried that perhaps the mesh data (which I write before the 3D data) is
> somehow at the end of the file. It could be an issue with h5dump. I just want
> to be sure before I write out a few hundred TB of data for analysis.
>
> FWIW I have looked at http://www.hdfgroup.org/HDF5/doc/H5.format.html but
> it's more of a spec sheet and I couldn't find anything addressing my specific
> question.
>
> Leigh
>
> On Wed, Jul 6, 2011 at 11:51 AM, <[email protected]> wrote:
> Leigh Orf wrote:
> > My concern - perhaps unfounded - is that the very small bits of what I
> > call
> > metadata (integers, lists of what variables are in the file, and other
> > very
> > small bits of data I write which describe stuff like the 3D data and is
> > necessary for my reader code) will be placed after the huge 3d data such
> > that accessing it will require long seeks through 3d data. The only reason
> > I
> > am worried about this is I noticed doing a h5dump on one of my small
> > metadata datasets that it took more than 10 seconds to output data on one
> > of
> > my files. I got the impression that perhaps h5dump was having to make its
> > way through the 3d arrays before getting to the metadata. However, my C
> > code
> > seemed to access the metadata quickly; perhaps it's an issue with h5dump.
>
> I hit the same issue when inspecting metadata in some files. `h5dump -H`
> was what I needed.
>
> - Daniel
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [email protected]
> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>
>
>
> --
> Leigh Orf
> Associate Professor of Atmospheric Science
> Department of Earth and Atmospheric Science
> Central Michigan University
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [email protected]
> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
