2016-10-28 14:17 GMT+02:00 Peter Steinbach <[email protected]>: > Hi Elvis, > > interesting I am mostly looking into 3D optical tomography images (which > exclusively use voxels represented by integers). > >> >> This may not be of interest, and does not include zstd, but I'm >> attaching an excerpt from some of the results I got when back when >> doing our basic benchmarking of some algorithms (all lossless). > > > We've seen a rough factor of (2.+/-0.5) with lz4 r131 in compression as well > with unfiltered data. In my cases we are mostly interested in high > compression bandwidth and high compression ratio. lz4 so far gives > compression bandwidths up to 1GB/s depending on the quality aspired (of > course the compression ratios tend to be lower then). > >> >> It was based on those that we settled on Blosc_LZ4HC at level 4, since >> we were looking for very fast decompression times, while longer >> compression times and slightly larger file size was acceptable up to >> certain points. The gzip results are included mostly because that's >> what we were using at the time and I wanted them as a comparison, but >> we knew we wanted something else. The input for those benchmarks was a >> 500x300x300 float dataset containing a tomographic 3D image. > > > to be honest, I am still surprised that hdf5 doesn't contain these > state-of-the-art encoders, but rather ships bzip2 et al. which are painfully > slow and don't make any account of computer architectures (lz4 is cache > aware AFAIK). But hey, coming up with a hdf5 compressor is straight forward > after one wrangled with the docs. I just don't know how contributing to hdf5 > works.
Yea me too, but I believe the HDF5 group has as a goal to open up the development a bit more, which would be very welcome. So lets hope for that. Elvis > >> >> I might try to dig up the script I used for the benchmark and see if >> we still have the input I used, and do a test with lossy ZFP. It could >> be very interesting for creating 3D "thumbnails" in our application. > > > indeed, that would be interesting to see. > Best, > Peter > > >> >> Elvis >> >>> >>> P >>> >>> >>> On 10/28/2016 01:12 PM, Elvis Stansvik wrote: >>>> >>>> >>>> 2016-10-28 1:53 GMT+02:00 Miller, Mark C. <[email protected]>: >>>>> >>>>> >>>>> Hi All, >>>>> >>>>> Just wanted to mention a new HDF5 floating point compression plugin >>>>> available on github... >>>>> >>>>> https://github.com/LLNL/H5Z-ZFP >>>>> >>>>> This plugin will come embedded in the next release of the Silo library >>>>> as >>>>> well. >>>> >>>> >>>> >>>> Thanks for the pointer. That's very interesting. I had not heard about >>>> ZFP before. The ability to set a bound on the error in the lossless >>>> case seems very useful. >>>> >>>> Do you know if there has been any comparative benchmarks of ZFP >>>> against other compressors? >>>> >>>> After some basic benchmarking, we recently settled on Blosc_LZ4HC at >>>> level 4 for our datasets (3D float tomography data), but maybe it >>>> would be worthwhile to look at ZFP as well.. >>>> >>>> Best regards, >>>> Elvis >>>> >>>>> >>>>> -- >>>>> Mark C. Miller, LLNL >>>>> >>>>> _______________________________________________ >>>>> Hdf-forum is for HDF software users discussion. >>>>> [email protected] >>>>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >>>>> Twitter: https://twitter.com/hdf5 >>>> >>>> >>>> >>>> _______________________________________________ >>>> Hdf-forum is for HDF software users discussion. >>>> [email protected] >>>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >>>> Twitter: https://twitter.com/hdf5 >>>> >>> >>> _______________________________________________ >>> Hdf-forum is for HDF software users discussion. >>> [email protected] >>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >>> Twitter: https://twitter.com/hdf5 >>> >>> >>> _______________________________________________ >>> Hdf-forum is for HDF software users discussion. >>> [email protected] >>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >>> Twitter: https://twitter.com/hdf5 > > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > Twitter: https://twitter.com/hdf5 _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
