Hello Peter!
16.03.2015 19:53, Peter Majer пишет:
Dear All
We have been experiencing and suffering from the fact that writing
compressed files with hdf is significantly slower than writing
uncompressed.
My experience suggests that this depends greatly on the compression
algorithm of choice. For example, gzip compression typically slowed data
write by a factor of 10x-20x, while lz4 compression was even a bit
faster compared to uncompressed data, for my type of data (mass spec).
I have been asking myself for a while whether there is a
simple remedy.
It seems so. Please try lz4 compression plugin. For mass spec data,
applying lz4 compression delivered the following, simultaneously:
1) Smaller file size (5x-10x compression).
2) Slightly faster data write.
3) Slightly faster data read.
The only problem with lz4 is that it's not available in default HDF5
installation. However, I found using dynamic lz4 plugin reasonably easy.
Best wishes,
Andrey Paramonov
--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
