On Mon, Mar 7, 2011 at 9:16 AM, Quincey Koziol <[email protected]> wrote: > Hi Leigh, > >> >> open(unit=50,file=trim(filename),form='unformatted',status='unknown') >> write(50) ua >> write(50) va >> write(50) wa >> write(50) ppi >> write(50) tha >> >> etc. where each array is 3d (some 1d arrays are written as well). > > Yow! Very 70's... :-)
Don't blame me, it's not my code :) >> I am assuming I am getting less than optimal results primarily because >> the writes are unaligned. This brings me to my question: Should I >> bother to rewrite the checkpoint writing/reading code using hdf5 in >> order to increase performance? I understand with pHDF5 and collective >> I/O, it automatically does aligned writes, presumably being able to >> detect the strip size on the lustre filesystem (is this true?). > > Unless the MPI-IO layer is doing this, HDF5 doesn't do this by default. Good to know, I am going to set alignment manually from now on. > >> With serial HDF5, I see there is a H5Pset_alignment command. I also >> assume that with serial HDF5, I would need to manually set the >> alignment, as it defaults to unaligned writes. Would I benefit from >> using H5Pset_alignment to the stripe size on the lustre filesystem? > > Yes, almost certainly. This is one of the ways that Mark Howison and > I worked out to improve the I/O performance on the NERSC machines. Good.... > >> My arrays are roughly 28x21x330 with some slight variation. 16 these 4 >> byte floating point arrays are written, giving approximately 12 MB per >> file. >> >> So, as a rough guess, I am thinking of trying the following: >> >> Set stripe size to 4 MB (4194304 bytes) >> >> Try something like: >> >> H5Pset_alignment(fapl, 1000, 4194304) >> >> (I didn't set the second argument to 0 because I really don't want to >> align my 4 byte integers etc, that comprise some of the restart data, >> right?) > > Yes, that looks fine. > >> Chunk in Z only, so my chunk dimensions would be something like >> 28x21x30 (it's never been clear to me what chunk size to pick to >> optimize I/O). >> >> And keep the other parameters the same (1 stripe, and 3,000 files per >> directory). >> >> I guess what I'm mostly looking for is assurance that I will get >> faster I/O going down this kind of route than the current way I am >> doing unformatted I/O. > > This looks like a fruitful direction to go it. Do you really need > chunking though? Not sure, It's never been super clear to me what chunking gets you beyond (1) the ability to do compression (2) faster seeking through large datasets when you want to access space towards the end of the file. I may just forego chunking and see where that gets me first. Thanks again for your help, I am more optimistic today... Leigh > > Quincey > > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org > -- Leigh Orf Associate Professor of Atmospheric Science Department of Geology and Meteorology Central Michigan University Currently on sabbatical at the National Center for Atmospheric Research in Boulder, CO NCAR office phone: (303) 497-8200 _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
