Hi Bas,
I have a couple questions.
1. About the size differences between h5repack and your code, is it
the size of HDF5 file or dataset?
2. About the chunk, what the size of chunk used for h5repack and your
code?
Jonathan
On 7/7/2011 10:00 AM, Bas Schoen wrote:
Hi,
I'm trying to create a hdf5 file with some compound datatypes with
GZIP compression. The development is done in C# using the HDF5DotNet dll.
I need these compression options: shuffle & gzip=9 and I would like to
achieve the same compression ration as h5repack.
The problem however is that the compressed file is the same size as
the not compressed file. If I use h5repack on that file, this size is
10 times smaller. Can someone see what I am doing wrong?
Part of my implementation:
// We want to write a compound datatype, which is a struct containing
a int and some bytes
DataStruct[] data = new DataStruct[]{...}; //data has been filled
// Create the compound datatype for memory
H5DataTypeId dataTypeMem = H5T.create(H5T.CreateClass.COMPOUND,
(int)Marshal.SizeOf(default(DataStruct)));
H5T.insert(dataTypeMem, "A", (int)Marshal.OffsetOf(typeof(
DataStruct ), "A"), H5T.H5Type.NATIVE_INT);
H5T.insert(dataTypeMem, "B", (int)Marshal.OffsetOf(typeof( DataStruct
), "B"), H5T.H5Type.NATIVE_UCHAR);
H5T.insert(dataTypeMem, "C", (int)Marshal.OffsetOf(typeof( DataStruct
), "C"), H5T.H5Type.NATIVE_UCHAR);
H5T.insert(dataTypeMem, "D", (int)Marshal.OffsetOf(typeof( DataStruct
), "D"), H5T.H5Type.NATIVE_UCHAR);
H5T.insert(dataTypeMem, "E", (int)Marshal.OffsetOf(typeof( DataStruct
), "E"), H5T.H5Type.NATIVE_UCHAR);
// Create the compound datatype for the file. Because the standard
// types we are using for the file may have different sizes than
// the corresponding native types, we must manually calculate the
// offset of each member.
int offset = 0;
H5DataTypeId dataTypeFile = H5T.create(H5T.CreateClass.COMPOUND,
(int)(4 + 1 + 1 + 1 + 1));
H5T.insert(dataTypeFile, "A", offset, H5T.H5Type.STD_U32BE);
offset += 4;
H5T.insert(dataTypeFile, "B", offset, H5T.H5Type.STD_U8BE);
offset += 1;
H5T.insert(dataTypeFile, "C", offset, H5T.H5Type.STD_U8BE);
offset += 1;
H5T.insert(dataTypeFile, "D", offset, H5T.H5Type.STD_U8BE);
offset += 1;
H5T.insert(dataTypeFile, "E", offset, H5T.H5Type.STD_U8BE);
offset += 1;
long[] dims = { (long) data.Count() };
try
{
// Create dataspace, with maximum = current
H5DataSpaceId dataSpace = H5S.create_simple(1, dims);
//Create compression properties
long[] chunk = dims; //What value should be used as chunk?
H5PropertyListId compressProperty =
H5P.create(H5P.PropertyListClass.DATASET_CREATE);
H5P.setShuffle(compressProperty);
H5P.setDeflate(compressProperty, 9)
H5P.setChunk(compressProperty, chunk);
// Create the data set
H5DataSetId dataSet = H5D.create(fileId, "NAME", dataTypeFile,
dataSpace, new H5PropertyListId(H5P.Template.DEFAULT),
compressProperty, new H5PropertyListId(H5P.Template.DEFAULT));
// Write data to it
H5D.write(dataSet, dataTypeMem, new H5DataSpaceId(H5S.H5SType.ALL),
new H5DataSpaceId(H5S.H5SType.ALL), new
H5PropertyListId(H5P.Template.DEFAULT), new H5Array< DataStruct >(data));
// Cleanup
H5T.close(dataTypeMem);
H5T.close(dataTypeFile);
H5D.close(dataSet);
H5P.close(compressProperty);
H5S.close(dataSpace);
}
catch
{
...
}
All steps are: creating datatype for both file and memory, creating
dataspace, creating the dataset with shuffle and compression creation
properties and finally writing the data to file.
It might be a bit difficult to check this code, but are there any
steps missing/incorrect?
Help appreciated.
Best regards,
Bas Schoen
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
