Hi,
I am trying to write a ~24GB large array of floats to a file with PHDF5.
I am running on a Lustre PFS, with IB networking. I am running the
software on 128 processes, spread amongst 16 nodes of 8 cores each. The
MPI implementation is OpenMPI 1.6.3, and HDF5 is 1.8.10.
Each process is writing one regular hyperslab with a various offset. Not
every process has a hyperslab of the same size, but they are close in
size. Each process should therefore be writing around 192MB of data.
For some reason, it seems that if I set
H5Pset_dxpl_mpio(plist_id, H5FD_MPIO_COLLECTIVE);
only the master node writes anything into the resulting file (and it
takes ~10 minutes to write it).
If instead I set
H5Pset_dxpl_mpio(plist_id, H5FD_MPIO_INDEPENDENT);
all nodes write data, and it takes ~4-5 minutes to write the whole file.
I am expecting two things that I don't see happening :
1) With Collective IOs, I would expect all ranks to write.
2) With our lustre filesystem, I would expect way more than 100MB/s for
such collective IOs (at least around 1GB/s).
Any tips on what might be going one ?
Thanks,
--
---------------------------------
Maxime Boissonneault
Analyste de calcul - Calcul Québec, Université Laval
Ph. D. en physique
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
