Peter, there's an API call that lets you write chunks directly into the file including chunks which you have compressed outside the HDF5 filter pipeline. Have a look at:
http://www.hdfgroup.org/HDF5/doc/HL/RM_HDF5Optimized.html#H5DOwrite_chunk See how fast you can write with H5DOwrite_chunk and then do a back-of-the-envelope calculation to see how elaborate a queueing mechanism you want. G. From: Hdf-forum [mailto:[email protected]] On Behalf Of Peter Majer Sent: Monday, March 16, 2015 11:53 AM To: [email protected] Subject: [Hdf-forum] Queuing chunks for compression and writing Dear All We have been experiencing and suffering from the fact that writing compressed files with hdf is significantly slower than writing uncompressed. I have been asking myself for a while whether there is a simple remedy. Would it be possible to have two queues of chunks when writing a file, one for compression and one for actual writing to achieve the following: 1) I enqueue N chunks for CompressionAndWriting. They initially enter CompressQueue. 2) The chunks from CompressQueue are concurrently compressed by multiple compression threads and subsequently enqueued in a WriteQueue. 3) A WriteThread sequentially writes all compressed chunks from WriteQueue to the file system. This should allow to keep the WriteThread constantly busy and it should allow compressed writing to be faster than uncompressed writing by a factor that is more or less identical to the compression rate. Interfacewise it would be nice to have "StartWrite" and "FinishWrite" methods where "Startwrite" simply copies the data into the CompressQueue and returns immediately thereafter while FinishWrite would be blocking until the write operation for the corresponding chunk has actually completed. Would this be possible? Would it be feasible? Would it be easy? Thanks, Peter Dr. Peter Majer Image Analysis Scientist and Software Architect Bitplane AG www.bitplane.com<http://www.bitplane.com/> This message is intended only for the use of the addressee and may contain information that is confidential and/or subject to copyright. If you are not the intended recipient, you are hereby notified that any dissemination, copying, or redistribution of this message is strictly prohibited. If you have received this message in error please delete all copies immediately. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of Andor Technology Limited Companies. Andor Technology Limited has taken reasonable precautions to ensure that no viruses are contained in this email, but does not accept any responsibility once this email has been transmitted. Andor Technology Limited is a registered company in Northern Ireland, registration number: NI022466. Registered Office: Andor Technology, 7 Millennium Way, Springvale Business Park, Belfast, BT12 7AL. ___________________________________________________________________________ Please refer to www.oxinst.com/email-statement<http://www.oxinst.com/email-statement> for regulatory information.
_______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
