Hi Rob, I think the blocks should be aligned. The detector blocks are exactly 1MB in size and we chunk 4 of them together in the layout and set the alignment to be 4 MB to correspond to the GPFS block size. If we don't do this performance is dire. Do we have to do anything else?
Having said this, our Lustre experience is larger than our GPFS experience, so any ideas are welcome. We have tried asking support but we haven't had many ideas from DDN either. We do know that with simple IOR tests GPFS is great, but not with our real data at the moment. Thanks for the ADIOS pointers. Where is a good link to start from to find out about it? Cheers, Nick Rees Principal Software Engineer Phone: +44 (0)1235-778430 Diamond Light Source Fax: +44 (0)1235-446713 -------- Original message -------- From: Rob Latham <[email protected]> Date: To: HDF Users Discussion List <[email protected]> Subject: Re: [Hdf-forum] Very poor performance of pHDF5 when using single (shared) file On Thu, Sep 19, 2013 at 08:43:48AM +0000, [email protected] wrote: > I have been following this thread with interest since we have the same issue > in the synchrotron community, with new detectors generating 100's-1000's of > 2D frames/sec and total rates approaching 10 GB/sec using multiple parallel > 10 GbE streams from different detector nodes. What we have found is: > > - Lustre is better at managing the pHDF5 contention between nodes than GPFS > is. > - GPFS is better at streaming data from one node, if there is no contention. > - Having the nodes write to separate files is better than using pHDF5 to > enable all nodes to write to one. I would wager a tasty beverage or a box of donuts that the reason you seen poor performance with GPFS to a shared file is because your writes are not aligned to file system block boundaries. On large HPC systems, the MPI-IO layer will often take care of that file system block boundary alignment for you -- *if* you turn on collective I/O. If you are using independent POSIX i/o then there won't be much HDF5 or MPI-IO can do to help you out. > What we are doing is working with The HDF Group to define a work package > dubbed "Virtual Datasets" where you can have a virtual dataset in a master > file which is composed of datasets in underlying files. It is a bit like > extending the soft-link mechanism to allow unions. The method of mapping the > underlying datasets onto the virtual dataset is very flexible and so we hope > it can be used in a number of circumstances. The two main requirements are: > > - The use of the virtual dataset is transparent to any program reading the > data later. > - The writing nodes can write their files independently, so don't need pHDF5. > > An additional benefit is the data can be compressed, so data rates may be > able to be reduced drastically by compression, depending on your situation. You're proposing something akin to ADIOS, except the interface continues to be the community-standard HDF5. how interesting! This approach will make it impossible to benefit from several collective MPI-I/O optimizations, but it does open the door to another family of optimizations (one would likely trawl the many ADIOS publications for ideas). ==rob -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
