Hi Leigh, On Mar 5, 2011, at 12:49 PM, Leigh Orf wrote:
> Did another test on kraken, ended up with approximately 270 MB/s > performance. This appears to be in line with the "baseline" results of > the "Tuning HDF5 for Lustre File Systems" paper. I double checked to > verify that I was using version 1.8.5 and have H5FD_MPIO_COLLECTIVE > set. > > For this particular test, I wrote 12 files spanning 30,000 cores > simultaneously. I watched the data come in (ls -l every 5 seconds) and > noticed the data came in in 'fits and starts' and towards the end of > the writes, only a few hundreds of bytes remained to be written, and > it took a long time for those bytes to get written. Something is > weird. > > I put some stuff on line if anyone wants to take a glance at it. The > output of h5stat and h5ls on one of the files is included, the output > of lfs getstripe is included, and a typescript file showing the ls -l > output every 5 seconds is included to show how the files grew over > time. > > You can view the files here: http://orf5.com/hdf5/kraken Interesting... So, your datasets are all fixed size, with no filters. If you are writing the entire dataset in one I/O operation (via collective parallel I/O, or with serial I/O), you should try switching to using contiguous storage for all your datasets. Quincey > I have one question that may be at the root of this performance issue. > The Tuning paper talked about how chunks should be aligned. I have > chosen my own chunk dimensions, which are the same size as the array > dimensions. h5ls -rv shows that those chunk dimensions are preserved > (and these chunk dimensions are of course not aligned). Does this mean > I am overriding an internal mechanism in hdf5 which chooses its own > chunk dimensions based upon the lustre strip size? If I do not write > chunked data, will pHDF5 choose chunk dimensions for me which are > aligned? > > Thanks, > > Leigh > > > -- > Leigh Orf > Associate Professor of Atmospheric Science > Department of Geology and Meteorology > Central Michigan University > Currently on sabbatical at the National Center for Atmospheric > Research in Boulder, CO > NCAR office phone: (303) 497-8200 > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
