Some thing is not right in your program or in H5PTappend(). I posted an
Java example
writing 25M variable length strings on March 23. It took about 60
seconds to write
1.2GB data.
Thanks
--pc
Stamminger, Johannes wrote:
I will be very interested in your performance results. Also, check the
memory usage
as you test the performance.
I keep you informed ;-).
My test setup was to split up 488MB binary data to varying long portions
and writing each as hex string (as I fail to write binary up to now)
directly
a) a zip file (on the fly inflation)
b) to a hdf5 file (variable strings to packet table, chunk size 100,
compression 9)
Writing to the zip takes 66s, to hdf 432s.
Resulting zip file is 28MB, hdf file is 961MB.
As it is a java app the only thing I can tell concerning memory
consumption is that I did not see increasing mem, does not look like
eating the mem (garbage collecting returned needed mem to a size as used
at startup).
Both, output file size and performance, look like an issue to me:
Concerning performance I found
http://www.mail-archive.com/[email protected]/msg00024.html
Therefore I expect hopefully some improvements with the mentioned 1.8.5
release.
The compression of the packet table seems to take *no* effect: sum of
sizes of all strings as written to the zip is 938M ... :-(
Any hints of what I might have done the wrong here?
My initialisation code:
fH5fileId = Hdf5Hl.H5Fcreate(
fHdf5File.toString(),
HDF5Constants.H5F_ACC_TRUNC,
HDF5Constants.H5P_DEFAULT,
HDF5Constants.H5P_DEFAULT);
fStrTypeId = Hdf5Hl.H5Tcopy (HDF5Constants.H5T_C_S1);
status = Hdf5Hl.H5Tset_size (fStrTypeId,
HDF5Constants.H5T_VARIABLE);
fVlTableId = Hdf5Hl.H5PTcreate_fl(fH5fileId, "/Packets As
Received", fStrTypeId, 100, 9);
For writing the string H5PTappend is used with only 1 string being
written each time (~1.100.000 calls).
I'm preparing a test with writing blocks of 100 strings ...
Any hints appreciated,
Johannes Stamminger
------------------------------------------------------------------------
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
