Please sned it to [email protected] Elena On Oct 14, 2011, at 6:37 AM, Jerome BENOIT wrote:
> Is there a place where a patch can be sent ? > > On 13/10/11 10:58, Jerome BENOIT wrote: >> >> >> -------- Original Message -------- >> Subject: Re: [Hdf-forum] Compression filters >> Date: Thu, 13 Oct 2011 10:57:29 +0200 >> From: Jerome BENOIT <[email protected]> >> Reply-To: [email protected] >> Organization: none >> To: [email protected] >> >> Hello List: >> >> On 13/10/11 07:51, Ger van Diepen wrote: >>> I think it would be nice if a 3rd party filter is loaded dynamically using >>> dlopen. >> >> It can be done easily actually: >> I use some filters of mime in this way as xz comprressor. >> >> hth, >> Jerome >> >>> >>> The name of the filter needs to be reflected in the library name in some >>> standard way. The init function of the library can register the filter in >>> HDF5's filter registry. I think it only requires a small addition to the >>> way HDF5 finds a filter. >>> >>> In this way one the filter repository can be a set of shared libraries or >>> dlls; one does not need to change HDF5 to use a new filter. Also all HDF5 >>> tools can work with all filters. >>> >>> >>> This is the way python works and how query languages execute user defined >>> functions. >>> >>> I've used it myself successfully several times. >>> >>> >>> Cheers, >>> >>> Ger >>> >>> >>> Mark Miller <[email protected]> 10/13/2011 4:32 AM >>> >>> Sorry my last email was a bit off topic. I guess I latched on to the >>> "...we just need to know (in)formal requirements to integrate a new >>> filter" part. >>> >>> So, related to this, I thought I recall a move afoot a couple of years >>> back now to formalize submission, repository of 3rd party HDF5 'filters' >>> of various sorts and compression in particular. What happened with that? >>> I don't recall seeing much response on the forum to it but this sounds >>> like a good test case ;) >>> >>> Mark >>> >>> >>> On Tue, 2011-10-11 at 08:18 -0700, Nathanael Huebbe wrote: >>> > Hello all, >>> > >>> > here at the DKRZ (German Climate Computing Center) we have to store >>> > large volumes of climate data, some of which is stored in HDF5-files. >>> > So, during the last few month we have been doing some research into >>> > climate data compression. >>> > >>> > With very interesting results: We have been able to shrink our test data >>> > set to 38.76% of the original file size. This is a compression factor of >>> > more than 2.5 and it is significantly better than the performance ofall >>> > the standard methods we tested (bzip2 = 54%, gzip = 58%, sldc =66% and >>> > lzma = 46%). Also, we have seen that the lzma-algorithm performs much >>> > better than the other standard algorithms. >>> > >>> > Even though we have constructed our methods to fit climate data, the >>> > features we exploited for compression are very general and likely to >>> > apply to other scientific data as well. This is why we are confident >>> > that many of you could profit from these methods as well, and we would >>> > be happy to share our results with the rest of the community. >>> > >>> > The filtering mechanism in HDF5 predestines it to be the first placefor >>> > us to share our algorithms. But first we would be very interested tosee >>> > the lzma algorithm integrated as an optional filtering method, something >>> > that should be very easy to do and offers large benefits to all users. >>> > >>> > Since we would be willing to do the necessary work, we just need to know >>> > the (in)formal requirements to integrate a new filter. And, of course, >>> > we would be very interested to hear about other recent work which >>> > adresses compression in HDF5, and to get in touch with whoever workson >>> > it. >>> > >>> > Best regards, >>> > Nathanael Hübbe >>> > >>> > http://wr.informatik.uni-hamburg.de/ >>> > http://wr.informatik.uni-hamburg.de/people/nathanael_huebbe >>> > >>> > >>> > _______________________________________________ >>> > Hdf-forum is for HDF software users discussion. >>> > [email protected] >>> > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org >>> -- >>> Mark C. Miller, Lawrence Livermore National Laboratory >>> ================!!LLNL BUSINESS ONLY!!================ >>> [email protected] urgent: [email protected] >>> T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511 >>> >>> >>> _______________________________________________ >>> Hdf-forum is for HDF software users discussion. >>> [email protected] >>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org >>> >>> >>> >>> _______________________________________________ >>> Hdf-forum is for HDF software users discussion. >>> [email protected] >>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org >> >> >> _______________________________________________ >> Hdf-forum is for HDF software users discussion. >> [email protected] >> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org > > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
