Hi,
I'm trying to create a hdf5 file with some compound datatypes with GZIP
compression. The development is done in C# using the HDF5DotNet dll.
I need these compression options: shuffle & gzip=9 and I would like to
achieve the same compression ration as h5repack.
The problem however is that the compressed file is the same size as the not
compressed file. If I use h5repack on that file, this size is 10 times
smaller. Can someone see what I am doing wrong?
Part of my implementation:
// We want to write a compound datatype, which is a struct containing a int
and some bytes
DataStruct[] data = new DataStruct[]{...}; //data has been filled
// Create the compound datatype for memory
H5DataTypeId dataTypeMem = H5T.create(H5T.CreateClass.COMPOUND,
(int)Marshal.SizeOf(default(DataStruct)));
H5T.insert(dataTypeMem, "A", (int)Marshal.OffsetOf(typeof( DataStruct ),
"A"), H5T.H5Type.NATIVE_INT);
H5T.insert(dataTypeMem, "B", (int)Marshal.OffsetOf(typeof( DataStruct ),
"B"), H5T.H5Type.NATIVE_UCHAR);
H5T.insert(dataTypeMem, "C", (int)Marshal.OffsetOf(typeof( DataStruct ),
"C"), H5T.H5Type.NATIVE_UCHAR);
H5T.insert(dataTypeMem, "D", (int)Marshal.OffsetOf(typeof( DataStruct ),
"D"), H5T.H5Type.NATIVE_UCHAR);
H5T.insert(dataTypeMem, "E", (int)Marshal.OffsetOf(typeof( DataStruct ),
"E"), H5T.H5Type.NATIVE_UCHAR);
// Create the compound datatype for the file. Because the standard
// types we are using for the file may have different sizes than
// the corresponding native types, we must manually calculate the
// offset of each member.
int offset = 0;
H5DataTypeId dataTypeFile = H5T.create(H5T.CreateClass.COMPOUND, (int)(4 + 1
+ 1 + 1 + 1));
H5T.insert(dataTypeFile, "A", offset, H5T.H5Type.STD_U32BE);
offset += 4;
H5T.insert(dataTypeFile, "B", offset, H5T.H5Type.STD_U8BE);
offset += 1;
H5T.insert(dataTypeFile, "C", offset, H5T.H5Type.STD_U8BE);
offset += 1;
H5T.insert(dataTypeFile, "D", offset, H5T.H5Type.STD_U8BE);
offset += 1;
H5T.insert(dataTypeFile, "E", offset, H5T.H5Type.STD_U8BE);
offset += 1;
long[] dims = { (long) data.Count() };
try
{
// Create dataspace, with maximum = current
H5DataSpaceId dataSpace = H5S.create_simple(1, dims);
//Create compression properties
long[] chunk = dims; //What value should be used as chunk?
H5PropertyListId compressProperty =
H5P.create(H5P.PropertyListClass.DATASET_CREATE);
H5P.setShuffle(compressProperty);
H5P.setDeflate(compressProperty, 9)
H5P.setChunk(compressProperty, chunk);
// Create the data set
H5DataSetId dataSet = H5D.create(fileId, "NAME", dataTypeFile, dataSpace,
new H5PropertyListId(H5P.Template.DEFAULT), compressProperty, new
H5PropertyListId(H5P.Template.DEFAULT));
// Write data to it
H5D.write(dataSet, dataTypeMem, new H5DataSpaceId(H5S.H5SType.ALL), new
H5DataSpaceId(H5S.H5SType.ALL), new H5PropertyListId(H5P.Template.DEFAULT),
new H5Array< DataStruct >(data));
// Cleanup
H5T.close(dataTypeMem);
H5T.close(dataTypeFile);
H5D.close(dataSet);
H5P.close(compressProperty);
H5S.close(dataSpace);
}
catch
{
...
}
All steps are: creating datatype for both file and memory, creating
dataspace, creating the dataset with shuffle and compression creation
properties and finally writing the data to file.
It might be a bit difficult to check this code, but are there any steps
missing/incorrect?
Help appreciated.
Best regards,
Bas Schoen
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
