On Wed, Nov 09, 2011 at 04:36:30PM -0500, Zaak Beekman wrote: > Here are some more details about when the bug shows up: The bug/error only > occurs when performing collective h5dwrite_f operations when some of the > processes have selected null hyperslabs (h5sselect_none_f) some of the time > (it appears to depend on the topology of MPI ranks with null selections). > If the data transfer property is set to individual, and only the MPI ranks > with data to write make calls to h5dwrite_f then the data is written > successfully. Individual IO is prohibitively slow, more than 2 orders of > magnitude slower than collective IO and would cause this portion of the IO > to take as long, or longer than the computation portion of the simulation.
MVAPICH-1.1 is based on mpich-1.2.7 (that's mpich, not mpich2: it's 7 years old). You should probably check with valgrind just to make sure you are not doing anything bad with memory. Probably you are ok on that regard, but valgrind will tell you for sure (mpiexec -np whatever valgrind --log-file=myprogram.%p.vg myprogram ) Since you are stuck with mvapich-1.1 you will have to go out of your way a bit to make collective writes work: - you know which processors have data and which ones do not (or else you would not be able to call h5sselect_none_f). - With this information you can call MPI_COMM_SPLIT MPI_COMM_SPLIT(COMM, COLOR, KEY, NEWCOMM, IERROR) INTEGER COMM, COLOR, KEY, NEWCOMM, IERROR where "color" would be either "have data" or "don't have data". Processors that don't have data get to sit out this iteration. Processors that have data participate in collective I/O: instead of passing in MPI_COMM_WORLD, pass in the NEWCOMM from MPI_COMM_SPLIT. I supsect the "don't have data" processors change from iteration to iteration. I guess you'll have to do the benchmark to see if "open, write collectively, close" is still faster than "write independently) ==rob -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
