On Thu 2009-03-26 10:54, Roy Stogner wrote: > I can't say I'm happy about not having perfectly deterministic apps, > but I'm not sure what to do about it. Maybe someone familiar with > PETSc MatSetValues can chime in with "We're deterministic! Roy > doesn't know what he's talking about" or "We *can be deterministic*, > if you just configure or run with --some-obscure-option"; otherwise > I'm stuck.
No, there isn't a magic option to make this deterministic. There is a chance you could make it so by hacking the profiling interface for MPI_Waitany. For example, set a global before MatAssemblyEnd and intercept MPI_Waitany to receive in rank order when this global is set. The "correct" way to do this is to sort entries in ascending order prior to summing. In matrix assembly, that would likely kill performance and certainly injects a lot of complexity. It's much less complicated for VecAssembly, see the discussion on this thread http://lists.mcs.anl.gov/pipermail/petsc-users/2009-January/003875.html and the proposed patch http://lists.mcs.anl.gov/pipermail/petsc-dev/2009-January/001186.html The patch has not hit petsc-dev, likely due to it being a bit of work to incorporate and the need seems to be rare. Note that if you want to replace MatAssemblyEnd, you can implement you own PetscErrorCode MatAssemblyEnd_MPIAIJ_Deterministic(Mat,MatAssemblyType); and use it with your matrix by calling MatShellSetOperation(A,MATOP_ASSEMBLY_END,(void(*)(void))&MatAssemblyEnd_MPIAIJ_Deterministic); A lighter weight, though less performant, option would be to make matrix assembly sequential with something like for (int r=0; r<size; r++) { if (r == rank) { // matrix assembly code using MatSetValues(P,...); } MatAssemblyBegin(P,MAT_FLUSH_ASSEMBLY); MatAssemblyEnd(P,MAT_FLUSH_ASSEMBLY); } MatAssemblyBegin(P,MAT_FINAL_ASSEMBLY); MatAssemblyEnd(P,MAT_FINAL_ASSEMBLY); I hope this helps. Jed
pgpUH4sEaBBEK.pgp
Description: PGP signature
------------------------------------------------------------------------------
_______________________________________________ Libmesh-devel mailing list Libmesh-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/libmesh-devel
