On Sat, Mar 5, 2011 at 12:14 PM, Tjf (mobile) <[email protected]> wrote: > I'm not up on hdf5 internals, but I can't imagine any API would effectively > deal with such small writes, because the os/disks aren't going to cope with > them effectively. > > If hdf5 can coalesce writes, try enabling that. Otherwise, forward your data > to a subset of nodes for writing, such that each write is large. Generally > larger is better, but I would say shoot for 16 megs per write.
As I understand from Mark & Quincey when you write in collective mode, it assigns writers and collects data to the writers so that the chunks are larger, and aligns the data to the underlying FS stripe size (at least with lustre, what I am using). However the details of this are a mystery to me. Leigh > > -tom > > Am Mar 4, 2011 um 5:03 PM schrieb Leigh Orf <[email protected]>: > >> What is the size of a "write operation" with parallel hdf5? That >> terminology comes up a lot on my sole source of guidance for lustre on >> the machine I'm running on ( http://www.nics.tennessee.edu/io-tips ) >> >> I am trying to choose ideal parameters for the lustre file system. >> >> I experienced abysmal performance with my first attempt at writing 1 >> file containing 3D data with 30,000 cores, and I want to choose better >> parameters. After 11 minutes 62 GB had been written, and I killed the >> job. >> >> Each 3D array that I write from a core is 435,600 bytes. I have my >> chunk dimensions the same as my array dimension. Does that mean that >> each core writes a chunk of data 435,600 bytes long? Would I therefore >> wish to set my stripe size to 435,600 bytes? That is smaller than the >> default of 1 MB. >> >> It seems that lustre performs best when each "write operation" is >> large (say 32 MB) and the stripe size matches it. However our cores >> each are writing comparatively much smaller chunks of data. >> >> I am going to see if the folks on the kraken machine can help me with >> optimizing lustre, but want to understand as much as possible about >> how pHDF5 works before I do. >> >> Thanks, >> >> Leigh >> >> -- >> Leigh Orf >> Associate Professor of Atmospheric Science >> Department of Geology and Meteorology >> Central Michigan University >> Currently on sabbatical at the National Center for Atmospheric >> Research in Boulder, CO >> NCAR office phone: (303) 497-8200 >> >> _______________________________________________ >> Hdf-forum is for HDF software users discussion. >> [email protected] >> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org > -- Leigh Orf Associate Professor of Atmospheric Science Department of Geology and Meteorology Central Michigan University Currently on sabbatical at the National Center for Atmospheric Research in Boulder, CO NCAR office phone: (303) 497-8200 _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
