I have just sent it.
On 14/10/11 16:11, Elena Pourmal wrote:
Please sned it to [email protected]
Elena
On Oct 14, 2011, at 6:37 AM, Jerome BENOIT wrote:
Is there a place where a patch can be sent ?
On 13/10/11 10:58, Jerome BENOIT wrote:
-------- Original Message --------
Subject: Re: [Hdf-forum] Compression filters
Date: Thu, 13 Oct 2011 10:57:29 +0200
From: Jerome BENOIT<[email protected]>
Reply-To: [email protected]
Organization: none
To: [email protected]
Hello List:
On 13/10/11 07:51, Ger van Diepen wrote:
I think it would be nice if a 3rd party filter is loaded dynamically using
dlopen.
It can be done easily actually:
I use some filters of mime in this way as xz comprressor.
hth,
Jerome
The name of the filter needs to be reflected in the library name in some
standard way. The init function of the library can register the filter in
HDF5's filter registry. I think it only requires a small addition to the way
HDF5 finds a filter.
In this way one the filter repository can be a set of shared libraries or dlls;
one does not need to change HDF5 to use a new filter. Also all HDF5 tools can
work with all filters.
This is the way python works and how query languages execute user defined
functions.
I've used it myself successfully several times.
Cheers,
Ger
Mark Miller<[email protected]> 10/13/2011 4:32 AM>>>
Sorry my last email was a bit off topic. I guess I latched on to the
"...we just need to know (in)formal requirements to integrate a new
filter" part.
So, related to this, I thought I recall a move afoot a couple of years
back now to formalize submission, repository of 3rd party HDF5 'filters'
of various sorts and compression in particular. What happened with that?
I don't recall seeing much response on the forum to it but this sounds
like a good test case ;)
Mark
On Tue, 2011-10-11 at 08:18 -0700, Nathanael Huebbe wrote:
Hello all,
here at the DKRZ (German Climate Computing Center) we have to store
large volumes of climate data, some of which is stored in HDF5-files.
So, during the last few month we have been doing some research into
climate data compression.
With very interesting results: We have been able to shrink our test data
set to 38.76% of the original file size. This is a compression factor of
more than 2.5 and it is significantly better than the performance ofall
the standard methods we tested (bzip2 = 54%, gzip = 58%, sldc =66% and
lzma = 46%). Also, we have seen that the lzma-algorithm performs much
better than the other standard algorithms.
Even though we have constructed our methods to fit climate data, the
features we exploited for compression are very general and likely to
apply to other scientific data as well. This is why we are confident
that many of you could profit from these methods as well, and we would
be happy to share our results with the rest of the community.
The filtering mechanism in HDF5 predestines it to be the first placefor
us to share our algorithms. But first we would be very interested tosee
the lzma algorithm integrated as an optional filtering method, something
that should be very easy to do and offers large benefits to all users.
Since we would be willing to do the necessary work, we just need to know
the (in)formal requirements to integrate a new filter. And, of course,
we would be very interested to hear about other recent work which
adresses compression in HDF5, and to get in touch with whoever workson
it.
Best regards,
Nathanael Hübbe
http://wr.informatik.uni-hamburg.de/
http://wr.informatik.uni-hamburg.de/people/nathanael_huebbe
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
[email protected] urgent: [email protected]
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
