On Tue, 2011-10-11 at 12:33 -0400, Zaak Beekman wrote: > Out of curiosity, did you try the szip filter provided with HDF5? (If > you are able to use it under the licensing terms.)
We took as close a look as possible at the algorithm but concluded that it cannot hope to compete with lzma due to the suboptimality of the rice coder used for entropy coding. Range coders, like the one lzma uses, are already optimal in terms of storage space while rice coding makes assumptions about the frequency of symbols that are generally not met. On Wed, 2011-10-12 at 00:07 -0700, Mark Miller wrote: > Hello Nathanael, > > A while back, I added some application specific compression filters to a > library I support that uses HDF5. I spent a lot of time reading the > following... > > http://www.hdfgroup.org/HDF5/doc/RM/RM_H5Z.html#Compression-Register > > http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetFilter > > http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-FilterBehavior Yes, I know these pages by heart already (well, almost), since we already have a moded version of the h5repack utility and an extremely minimalistic API (one function call) to include support into any application using HDF5 files. You can find a patch with the modified h5repack utility at: http://files.wr.informatik.uni-hamburg.de/hdf5-dkrz-compressor.patch So the question is not, how we can use our algorithm in HDF5 files, we can already do it; the question is, whether you want support for lzma and/or our algorithm in the standard library, which we would be glad to contribute. Best regards, Nathanael Hübbe _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
