On Mon, 2011-07-25 at 17:49 -0700, Pedro Silva Vicente wrote:
> However, due to the way the NeXus API was used, performance was very > slow. It was several orders of magnitude slower > than using a plain binary file to save the experiment results, so one > question that come up was > > “Why should I use HDF5 instead of a binary file, if it’s several > orders of magnitude slower?” > > So, I implemented the 2 solutions explained. Ok, thanks for context. That helps. If I recall though, the best of your two solutions resulted in only about a 65% speedup. Thats maybe a 2.2x speedup which is certainly a step in the right direction but hardly solves the 'orders of magnitude' problem you mention. Now, I've been using and comparing raw binary I/O and products like HDF5 for many years. In all my experiences and depending on the size of the read/write requests, I have observed and have been willing to tolerate at most a 2-3x performance hit for using a 'high level' I/O library over what is achievable using raw binary I/O interfaces. And, honestly, the performance hit is usually less than 20-25% of raw binary bandwidth. That, that assumes the HDF5 library is being used properly and, in turn, the HDF5 library is using the underlying filesystem properly. I have seen situations where neither or both are not the case and indeed I have also seen orders of magnitude loss of performance. In fact, within the last year, we needed to write our a specialized Virtual File Driver (VFD) to get good performance on our BG/P system. Writing a new VFD for HDF5 wasn't necessarily simple. But, it was possible and doing so resulted in 30-50x performance improvement. On top of that, getting the applications to use HDF5 slightly differently I think can gives us another 2-3x performance improvement. So, I guess what I am saying is that I've got to believe it is possible to change the way NeXus uses HDF5 and/or the way HDF5 interacts with underlying storage (maybe by writing your own VFD), to address the 'orders of magnitude' performance hit. Otherwise, I agree with you, why should anyone pay that kind of a price to use it? Mark _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
