Hi Leigh!
On Jun 26, 2010, at 8:06 AM, Leigh Orf wrote:
> I am a fan of the scale-offset filter followed by the gzip filter to
> really reduce the size of big 3D datasets of weather model data. I am
> using this compression strategy with HDF5 to do massively parallel
> simulations and writing out one HDF5 file per MPI process.
Glad they are turning out to be useful to you. Adding a "shuffle"
filter preprocessing step may improve the compression ratio further.
> I recently discovered when rendering data spanning multiple files that
> there is a boundary issue as you hop from one dataset to the next.
> There is a slight discontinuity in the uncompressed floating point
> data between values as you go from one file to the next. I would
> imagine this has to do with the internal parameters chosen by the
> filter algorithm which must look for the maximum and minimum values in
> the dataset being operated upon, which will vary from file to file
> (from MPI proc. to MPI proc).
Hmm, yes, I would expect that...
> Is there some way to have the scale offset filter use global
> parameters such that the discontinuities vanish? Before I used HDF5 I
> used HDF4 and wrote my own scale/offset filter which used the global
> max and min values (using a collective MPI call to determine this) and
> this worked fine. However I like the transparency of the HDF5 filters
> and would prefer to not write my own.
It's definitely a good idea, but since each dataset is compressed
independently, there isn't a way to have a global set of min/max values, at
least currently. However, I don't imagine it would be too difficult to add a
new "scale type" to the filter... I'll add an issue to our bugtracker and
Elena can prioritize it with the other work there. If you'd like to submit a
patch or find a little bit of funding for us to perform this work, that'll
speed things up. :-)
Quincey
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
