There is a public source code repository for HDF5-addons, hosted
at Origo, which is something like the Swiss version of Sourceforge:
http://hdf5-addons.origo.ethz.ch/
This one could be used for such filters as well. As of now it just
contains the HDF5 Streaming VFD, which was outsourced from HDF5 1.6.4
some time ago. Origo is a pretty nice platform offering a couple
of useful tools beyond just SVN management. Currently it's Quincey
and myself being project managers there, we could easily add other
project members to allow them adding their 3rd party extensions
there. This platform could also be a first step of publishing
HDF5 addons before official integration into HDF5 itself, in case
such is on the roadmap, which may or may not be the case for
HDF5 addons. If licensing schemes are incompatible, it might be
better to have another, accessible place - Origo is very flexible,
in contrast to sourceforge it does not require open source licensing
for projects hosted there, so it can also host commercial code
which is still publicly available under a non-OS license.
Werner
On Thu, 13 Oct 2011 04:32:33 +0200, Mark Miller <[email protected]> wrote:
Sorry my last email was a bit off topic. I guess I latched on to the
"...we just need to know (in)formal requirements to integrate a new
filter" part.
So, related to this, I thought I recall a move afoot a couple of years
back now to formalize submission, repository of 3rd party HDF5 'filters'
of various sorts and compression in particular. What happened with that?
I don't recall seeing much response on the forum to it but this sounds
like a good test case ;)
Mark
On Tue, 2011-10-11 at 08:18 -0700, Nathanael Huebbe wrote:
Hello all,
here at the DKRZ (German Climate Computing Center) we have to store
large volumes of climate data, some of which is stored in HDF5-files.
So, during the last few month we have been doing some research into
climate data compression.
With very interesting results: We have been able to shrink our test data
set to 38.76% of the original file size. This is a compression factor of
more than 2.5 and it is significantly better than the performance of all
the standard methods we tested (bzip2 = 54%, gzip = 58%, sldc = 66% and
lzma = 46%). Also, we have seen that the lzma-algorithm performs much
better than the other standard algorithms.
Even though we have constructed our methods to fit climate data, the
features we exploited for compression are very general and likely to
apply to other scientific data as well. This is why we are confident
that many of you could profit from these methods as well, and we would
be happy to share our results with the rest of the community.
The filtering mechanism in HDF5 predestines it to be the first place for
us to share our algorithms. But first we would be very interested to see
the lzma algorithm integrated as an optional filtering method, something
that should be very easy to do and offers large benefits to all users.
Since we would be willing to do the necessary work, we just need to know
the (in)formal requirements to integrate a new filter. And, of course,
we would be very interested to hear about other recent work which
adresses compression in HDF5, and to get in touch with whoever works on
it.
Best regards,
Nathanael Hübbe
http://wr.informatik.uni-hamburg.de/
http://wr.informatik.uni-hamburg.de/people/nathanael_huebbe
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
