Hi Daniel, Yes the numbers look very bad, but from your use case description, I'm pretty sure something can be done to improve the performance.
What is the file system you are working with? Did you make sure to set the stripe size and count of your file to something larger than the default (usually the default stripe count is something small like 1 or 2)? From the amount of data you are writing, I would try something like the maximum, then you can tune more if you want. The stripe size can be increased to something larger than the default too. The above usually solve 90% of users problems when seeing bad performance when accessing a single shared file. If the above does not help or you have done that already, then I'll need to take a deeper look at how your processes are accessing the datasets. It sounds like each process is writing a good large piece of data so you should not see this poor performance; although the separate file thing is always going to be the fastest strategy (not convenient though). Thanks, Mohamad -----Original Message----- From: Hdf-forum [mailto:[email protected]] On Behalf Of Daniel Langr Sent: Friday, August 30, 2013 9:05 AM To: [email protected] Subject: [Hdf-forum] Very poor performance of pHDF5 when using single (shared) file I've run some benchmark, where within an MPI program, each process wrote 3 plain 1D arrays to 3 datasets of an HDF5 file. I've used the following writing strategies: 1) each process writes to its own file, 2) each process writes to the same file to its own dataset, 3) each process writes to the same file to a same dataset. I've tested 1)-3) for both fixed/chunked datasets (chunk size 1024), and I've tested 2)-3) for both independent/collective options of the MPI driver. I've also used 3 different clusters for measurements (all quite modern). As a result, the running (storage) times of the same-file strategy, i.e. 2) and 3), were of orders of magnitudes longer than the running times of the separate-files strategy. For illustration: cluster #1, 512 MPI processes, each process stores 100 MB of data, fixed data sets: 1) separate files: 2.73 [s] 2) single file, independent calls, separate data sets: 88.54[s] cluster #2, 256 MPI processes, each process stores 100 MB of data, chunked data sets (chunk size 1024): 1) separate files: 10.40 [s] 2) single file, independent calls, shared data sets: 295 [s] 3) single file, collective calls, shared data sets: 3275 [s] Any idea why the single-file strategy gives so poor writing performance? Daniel _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
