Mark and All, On Oct 13, 2011, at 1:11 AM, Mark Miller wrote:
> Well, my recollection is something that I think Elena Pourmal was trying > to start where source code for (open source) 3rd party filters somehow > got submitted to HDF Group's web pages along with example data files > containing the 'filtered' data and information about the filter such as > Point of contact, purpose, performance characteristics, etc. > This information is available at The HDF Group Web site http://www.hdfgroup.org/services/contributions.html In the past, developers who supported the filters had access to PBWiki and could update corresponding information. Currently this functionality is missing. Please contact [email protected] if you need help with updates. > I like the idea of making it 'easy' to install such 3rd party filters > with (or next to) the main HDF5 library so they can be loaded as needed. > But, that is a step beyond what I was trying to open a dialog about > here ;) > This will be a great feature and we will be happy to consider and accept the patches to the library. Elena > Mark > > On Wed, 2011-10-12 at 22:51 -0700, Ger van Diepen wrote: >> I think it would be nice if a 3rd party filter is loaded dynamically >> using dlopen. > >> The name of the filter needs to be reflected in the library name in >> some standard way. The init function of the library can register the >> filter in HDF5's filter registry. I think it only requires a small >> addition to the way HDF5 finds a filter. >> >> In this way one the filter repository can be a set of shared libraries >> or dlls; one does not need to change HDF5 to use a new filter. Also >> all HDF5 tools can work with all filters. >> >> >> This is the way python works and how query languages execute user >> defined functions. >> >> I've used it myself successfully several times. >> >> >> Cheers, >> >> Ger >> >>>>> Mark Miller <[email protected]> 10/13/2011 4:32 AM >>> >> Sorry my last email was a bit off topic. I guess I latched on to the >> "...we just need to know (in)formal requirements to integrate a new >> filter" part. >> >> So, related to this, I thought I recall a move afoot a couple of years >> back now to formalize submission, repository of 3rd party HDF5 >> 'filters' >> of various sorts and compression in particular. What happened with >> that? >> I don't recall seeing much response on the forum to it but this sounds >> like a good test case ;) >> >> Mark >> >> >> On Tue, 2011-10-11 at 08:18 -0700, Nathanael Huebbe wrote: >>> Hello all, >>> >>> here at the DKRZ (German Climate Computing Center) we have to store >>> large volumes of climate data, some of which is stored in >> HDF5-files. >>> So, during the last few month we have been doing some research into >>> climate data compression. >>> >>> With very interesting results: We have been able to shrink our test >> data >>> set to 38.76% of the original file size. This is a compression >> factor of >>> more than 2.5 and it is significantly better than the performance of >> all >>> the standard methods we tested (bzip2 = 54%, gzip = 58%, sldc = 66% >> and >>> lzma = 46%). Also, we have seen that the lzma-algorithm performs >> much >>> better than the other standard algorithms. >>> >>> Even though we have constructed our methods to fit climate data, the >>> features we exploited for compression are very general and likely to >>> apply to other scientific data as well. This is why we are confident >>> that many of you could profit from these methods as well, and we >> would >>> be happy to share our results with the rest of the community. >>> >>> The filtering mechanism in HDF5 predestines it to be the first place >> for >>> us to share our algorithms. But first we would be very interested to >> see >>> the lzma algorithm integrated as an optional filtering method, >> something >>> that should be very easy to do and offers large benefits to all >> users. >>> >>> Since we would be willing to do the necessary work, we just need to >> know >>> the (in)formal requirements to integrate a new filter. And, of >> course, >>> we would be very interested to hear about other recent work which >>> adresses compression in HDF5, and to get in touch with whoever works >> on >>> it. >>> >>> Best regards, >>> Nathanael Hübbe >>> >>> http://wr.informatik.uni-hamburg.de/ >>> http://wr.informatik.uni-hamburg.de/people/nathanael_huebbe >>> >>> >>> _______________________________________________ >>> Hdf-forum is for HDF software users discussion. >>> [email protected] >>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org >> -- >> Mark C. Miller, Lawrence Livermore National Laboratory >> ================!!LLNL BUSINESS ONLY!!================ >> [email protected] urgent: [email protected] >> T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511 >> >> >> _______________________________________________ >> Hdf-forum is for HDF software users discussion. >> [email protected] >> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org >> >> > -- > Mark C. Miller, Lawrence Livermore National Laboratory > ================!!LLNL BUSINESS ONLY!!================ > [email protected] urgent: [email protected] > T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511 > > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
