I would think szip compressor might do well on this, at least for 2D stuff. Not sure about 3D. Have you tried gzip? I always try that to see where it leads.
I saw a recent email posted to hdf-forum announcing a new compressor named 'Blosc 1.0'. But the announcement came in relation to pytables and so I don't know if it is avaiable as a separate HDF5 filter that you can just grab and use. And, I don't know if by 'shuffle' that is the compressor you were talking about. Regarding data organization to 'encourage' compression, I would think that if things don't '...change much over time...', then making that axis the 'slowest varying dimension' of the dataset in storage would be best. That's all I can think of. Hope it helps. Good luck. Mark On Tue, 2010-07-13 at 08:06 -0700, John Knutson wrote: > I'm trying to find the optimal compression for 2- and 3-dimensional > matrices of recorded data. These data sets contain data that doesn't > change much over time (and time is being used as the first axis). I > thought that by using shuffle, I might get better compression, but > instead the resulting files were larger than without shuffle. > > Is shuffle meant to work with compound types? Are there things I need > to be considering in the organization of the axes of the data set in > order to better encourage compression? > > > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://*mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org > -- Mark C. Miller, Lawrence Livermore National Laboratory ================!!LLNL BUSINESS ONLY!!================ [email protected] urgent: [email protected] T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511 _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
