From: Hdf-forum <hdf-forum-boun...@lists.hdfgroup.org<mailto:hdf-forum-boun...@lists.hdfgroup.org>> on behalf of houssen <hous...@ipgp.fr<mailto:hous...@ipgp.fr>> Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>> Date: Monday, January 11, 2016 2:00 AM To: "hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>" <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>> Subject: [Hdf-forum] Mixing use of ZLib and SZip compression
When using HDF5, is it possible to mix ZLib and SZip compression ? If yes how ? Yes. You can have a single hdf5 file with some datasets that are compressed with zlib and others compressed with szip. A single dataset compressed with both zlib and szip? I imagine that *might* be possible. Never tried it. Not sure why you'd want to do it. But, I can't think if a reason HDF5 might balk at it except if they've added logic to explicitly forbid it. For reasons you mention below, don't think szip *after* zlib would "work" at all. But, zlib *after* szip might. My understanding is that: 1. ZLib can compress any data (char *, int, double, ...) Yes, its a byte-level compressor. Doesn't care if those bytes comprise an array of floats, doubles, ints, chars, etc. 2. SZip is dedicated to number compression only (float, double) I honestly can't recall but that sounds plausible/right. According to HDF5 doc, we have: 1. to use H5Pset_deflate to use ZLib (deflate algorithm) 2. to use H5Pset_szip to use SZip Yes. Though, take care to read licensing limitations regarding szip and confirm you're workflows involving it meet its requirements. My understanding is that, if I use both H5Pset_deflate and H5Pset_szip, then: 1. data (whatever they are) which are NOT numbers will be compressed with ZLib 2. numbers will be compressed with SZip I don't think it works that way. If you apply *both* filters to a dataset, HDF5 will apply each filter in order. Though, since zlib and szip are sort of built-in compressors, maybe HDF5 library has some logic to handle them specially? If not *and* if you want the behavior you describe here. Its easy to impliment your own sort of merged zlib/szip filter yourself that does something like… 1. Check data type. If type is double or float, apply szip, else apply zlib. Hope that helps. I am 99% certain what I've just written is accurate ;) Mark
_______________________________________________ Hdf-forum is for HDF software users discussion. Hdf-forum@lists.hdfgroup.org http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
