I'm not up on hdf5 internals, but I can't imagine any API would effectively deal with such small writes, because the os/disks aren't going to cope with them effectively.
If hdf5 can coalesce writes, try enabling that. Otherwise, forward your data to a subset of nodes for writing, such that each write is large. Generally larger is better, but I would say shoot for 16 megs per write. -tom Am Mar 4, 2011 um 5:03 PM schrieb Leigh Orf <[email protected]>: > What is the size of a "write operation" with parallel hdf5? That > terminology comes up a lot on my sole source of guidance for lustre on > the machine I'm running on ( http://www.nics.tennessee.edu/io-tips ) > > I am trying to choose ideal parameters for the lustre file system. > > I experienced abysmal performance with my first attempt at writing 1 > file containing 3D data with 30,000 cores, and I want to choose better > parameters. After 11 minutes 62 GB had been written, and I killed the > job. > > Each 3D array that I write from a core is 435,600 bytes. I have my > chunk dimensions the same as my array dimension. Does that mean that > each core writes a chunk of data 435,600 bytes long? Would I therefore > wish to set my stripe size to 435,600 bytes? That is smaller than the > default of 1 MB. > > It seems that lustre performs best when each "write operation" is > large (say 32 MB) and the stripe size matches it. However our cores > each are writing comparatively much smaller chunks of data. > > I am going to see if the folks on the kraken machine can help me with > optimizing lustre, but want to understand as much as possible about > how pHDF5 works before I do. > > Thanks, > > Leigh > > -- > Leigh Orf > Associate Professor of Atmospheric Science > Department of Geology and Meteorology > Central Michigan University > Currently on sabbatical at the National Center for Atmospheric > Research in Boulder, CO > NCAR office phone: (303) 497-8200 > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
