Sorry my last email was a bit off topic. I guess I latched on to the "...we just need to know (in)formal requirements to integrate a new filter" part.
So, related to this, I thought I recall a move afoot a couple of years back now to formalize submission, repository of 3rd party HDF5 'filters' of various sorts and compression in particular. What happened with that? I don't recall seeing much response on the forum to it but this sounds like a good test case ;) Mark On Tue, 2011-10-11 at 08:18 -0700, Nathanael Huebbe wrote: > Hello all, > > here at the DKRZ (German Climate Computing Center) we have to store > large volumes of climate data, some of which is stored in HDF5-files. > So, during the last few month we have been doing some research into > climate data compression. > > With very interesting results: We have been able to shrink our test data > set to 38.76% of the original file size. This is a compression factor of > more than 2.5 and it is significantly better than the performance of all > the standard methods we tested (bzip2 = 54%, gzip = 58%, sldc = 66% and > lzma = 46%). Also, we have seen that the lzma-algorithm performs much > better than the other standard algorithms. > > Even though we have constructed our methods to fit climate data, the > features we exploited for compression are very general and likely to > apply to other scientific data as well. This is why we are confident > that many of you could profit from these methods as well, and we would > be happy to share our results with the rest of the community. > > The filtering mechanism in HDF5 predestines it to be the first place for > us to share our algorithms. But first we would be very interested to see > the lzma algorithm integrated as an optional filtering method, something > that should be very easy to do and offers large benefits to all users. > > Since we would be willing to do the necessary work, we just need to know > the (in)formal requirements to integrate a new filter. And, of course, > we would be very interested to hear about other recent work which > adresses compression in HDF5, and to get in touch with whoever works on > it. > > Best regards, > Nathanael Hübbe > > http://wr.informatik.uni-hamburg.de/ > http://wr.informatik.uni-hamburg.de/people/nathanael_huebbe > > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org -- Mark C. Miller, Lawrence Livermore National Laboratory ================!!LLNL BUSINESS ONLY!!================ [email protected] urgent: [email protected] T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511 _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
