Hi John,
On Sep 22, 2010, at 2:36 PM, John Knutson wrote:
> I'm still trying to milk as much as I can, performance-wise, out of HDF5...
>
> My latest bit of confusion comes from the following seeming paradox. I have
> two files, poo2.h5 and poo3-d-1.h5. Both files contain exactly the same
> data, though poo2, being the larger data set, has more blank-filled elements.
> Also, because of the larger size of the data set in poo2, the data starts at
> (0, 0, 694080) or thereabouts, vs. (0,0,0) for poo3-d-1. My question is:
> "why is the smaller data set 10x larger in size (bytes?) than the larger data
> set with the same data and chunking?"
The "compression ratio" reported accounts for the sparseness of the
chunks in the dataset. You probably have written more data to the smaller
dataset.
> Is there any way to look at the details of what data is stored in the file,
> i.e. how many and maybe which chunks are stored, etc.?
We don't have a way to return a "map" of the chunks for a dataset
currently (although it is in our issue tracker).
Quincey
> HDF5 "poo2.h5" {
> DATASET "/Data/IS-GPS-200 ID 2 Ephemerides" {
> DATATYPE "/Types/Ephemeris IS-GPS-200 id 2"
> DATASPACE SIMPLE { ( 26, 160, 1051200 ) / ( H5S_UNLIMITED, H5S_UNLIMITED,
> 1051200 ) }
> STORAGE_LAYOUT {
> CHUNKED ( 1, 15, 250 )
> SIZE 23228 (52713869.468:1 COMPRESSION)
> }
>
> vs.
>
> HDF5 "poo3-d-1.h5" {
> DATASET "/Data/IS-GPS-200 ID 2 Ephemerides" {
> DATATYPE "/Types/Ephemeris IS-GPS-200 id 2"
> DATASPACE SIMPLE { ( 1, 160, 2880 ) / ( H5S_UNLIMITED, H5S_UNLIMITED, 2880
> ) }
> STORAGE_LAYOUT {
> CHUNKED ( 1, 15, 250 )
> SIZE 251461 (513.097:1 COMPRESSION)
> }
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [email protected]
> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
