Did another test on kraken, ended up with approximately 270 MB/s performance. This appears to be in line with the "baseline" results of the "Tuning HDF5 for Lustre File Systems" paper. I double checked to verify that I was using version 1.8.5 and have H5FD_MPIO_COLLECTIVE set.
For this particular test, I wrote 12 files spanning 30,000 cores simultaneously. I watched the data come in (ls -l every 5 seconds) and noticed the data came in in 'fits and starts' and towards the end of the writes, only a few hundreds of bytes remained to be written, and it took a long time for those bytes to get written. Something is weird. I put some stuff on line if anyone wants to take a glance at it. The output of h5stat and h5ls on one of the files is included, the output of lfs getstripe is included, and a typescript file showing the ls -l output every 5 seconds is included to show how the files grew over time. You can view the files here: http://orf5.com/hdf5/kraken I have one question that may be at the root of this performance issue. The Tuning paper talked about how chunks should be aligned. I have chosen my own chunk dimensions, which are the same size as the array dimensions. h5ls -rv shows that those chunk dimensions are preserved (and these chunk dimensions are of course not aligned). Does this mean I am overriding an internal mechanism in hdf5 which chooses its own chunk dimensions based upon the lustre strip size? If I do not write chunked data, will pHDF5 choose chunk dimensions for me which are aligned? Thanks, Leigh -- Leigh Orf Associate Professor of Atmospheric Science Department of Geology and Meteorology Central Michigan University Currently on sabbatical at the National Center for Atmospheric Research in Boulder, CO NCAR office phone: (303) 497-8200 _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
