Hi Mohamad, thanks for your quick response. Aim of our modifications were to demonstrate optimal parallel write performance, e.g. as achieved by IOR / POSIX, is possible. We could use it not only for checkpointing but for all periodic output of the climate models. Therefore, we chose to realize a lightweight and easy solution.
Currently, on our parallel file system, performance degradation of parallel I/O and some high-level libraries leads to the absurd situation in which scientists use sequential I/O for performance reasons. We have not compared performance to PLFS due to time restrictions in the project and we really believe a general solution, i.e. fixing the underlying file systems is important. I think indeed the PLFS approach is a good general solution. There are differences and minor drawbacks of using PLFS compared to HDF5/multifile. 1) PLFS requires to deal with an extra mountpoint which must be managed by someone. This modification here just requires to select the driver (it could be also a drop-in replacement of the existing driver for a library, thus no modifications would be needed). 2) Due to FUSE, PLFS adds overhead compared to direct operations, which is not the case for the multifile approach. The post-mortem conversion takes time, but the conversion can be done by only one thread reducing required CPU resources drastically. 3) Due to the involvement of VFS and POSIX, the access pattern given by PLFS to the parallel file system underneath may be suboptimal. Our file system is very sensible to the access pattern, therefore we decided to have full control of the pattern without rewriting I/O handling in more complex software. 4) Once the file has been accessed for reading once it (should be) bit-identical with a regular HDF5 file thus it can be used without any library / file system modifications... 5) Finally a very important one: FUSE does not work under AIX :-) If you are interested we could continue investigating the results on another platform. Thanks & Best regards, julian 2013/9/20 Mohamad Chaarawi <[email protected]>: > Hi Nathanael, > > I'll try and spend some time looking at the patch. Thanks for sharing! > > This sounds like you are optimizing your checkpointing phase. > Is there any advantage from doing this rather than using PLFS? > > Mohamad > > -----Original Message----- > From: Hdf-forum [mailto:[email protected]] On Behalf Of > huebbe > Sent: Friday, September 20, 2013 6:34 AM > To: [email protected] > Cc: Julian Kunkel > Subject: Re: [Hdf-forum] Very poor performance of pHDF5 when using single > (shared) file > > On 09/19/2013 10:43 AM, [email protected] wrote: >> What we are doing is working with The HDF Group to define a work package >> dubbed "Virtual Datasets" where you can have a virtual dataset in a master >> file which is composed of datasets in underlying files. It is a bit like >> extending the soft-link mechanism to allow unions. The method of mapping the >> underlying datasets onto the virtual dataset is very flexible and so we hope >> it can be used in a number of circumstances. The two main requirements are: >> >> - The use of the virtual dataset is transparent to any program reading the >> data later. >> - The writing nodes can write their files independently, so don't need >> pHDF5. > > As a matter of fact, this is pretty much what we did already for our own > research: We, too, patched the HDF5 library to provide writing of multiple > files and reading them back in a way entirely transparent to the application. > You can find our patch, along with a much more detailed description, on our > website: > http://www.wr.informatik.uni-hamburg.de/research/projects/icomex/multifilehdf5 > > On our system, we could actually see an improvement in wall-clock time for > the entire process of writing-reconstructing-reading as opposed to writing to > a shared file and reading it single stream. This may be different on other > systems, but at least we expect a huge benefit in CPU-time since the > multifile approach allows the parallel part of the workflow to be fast. > > Of course, we are very interested to hear about other people's experiences > with transparent multifiles. > > Cheers, > Nathanael Hübbe > -- http://wr.informatik.uni-hamburg.de/people/julian_kunkel _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
