Does the hdfgroup have any kind of plan/schedule for enabling compression of chunks when using parallel IO?
The use case being that each process compresses its own chunk at write time and the overall file size is reduced. (I understand that chunks are preallocated and this makes it hard to implement compressed chunking with Parallel IO). Thanks JB -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Frank Baker Sent: 04 October 2010 22:31 To: HDF Users Discussion List Subject: [Hdf-forum] New "Chunking in HDF5" document Several users have raised questions regarding chunking in HDF5. Partly in response to these questions, the initial draft of a new "Chunking in HDF5" document is now available on The HDF Group's website: http://www.hdfgroup.org/HDF5/doc/_topic/Chunking/ This draft includes sections on the following topics: General description of chunks Storage and access order Partial I/O Chunk caching I/O filters and compression Pitfalls and errors to avoid Additional Resources Future directions Several suggestions for tuning chunking in an application are provided along the way. As a draft, this remains a work in progress; your feedback will be appreciated and will be very useful in the document's development. For example, let us know if there are additional questions that you would like to see treated. Regards, -- Frank Baker HDF Documentation [email protected] _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
