Hello,
We are using a SNES to solve a steady-state FSI problem.
The operator is defined as a MatNest with multiple fields.
Some submatrices are entirely defined on a subset of processes (but they are
still created on the same communicator as the MatNest).
The preconditioner is defined as a FieldSplit.
During the first call to KSPSolve within SNESSolve, I’m getting this (with
debugging turned on):
[0]PETSC ERROR: VecGetSubVector() line 1243 in
/Users/jolivet/Documents/repositories/petsc/src/vec/vec/interface/rvector.c
MPI_Allreduce() called in different locations (code lines) on different
processors
[2]PETSC ERROR: VecNorm_MPI() line 57 in
/Users/jolivet/Documents/repositories/petsc/src/vec/vec/impls/mpi/pvec2.c
MPI_Allreduce() called in different locations (code lines) on different
processors
[1]PETSC ERROR: VecNorm_MPI() line 57 in
/Users/jolivet/Documents/repositories/petsc/src/vec/vec/impls/mpi/pvec2.c
MPI_Allreduce() called in different locations (code lines) on different
processors
As you may have guessed, process 0 (resp. 1–2) is where the structure (resp.
fluid) is handled.
I’m attaching both stacks. I don’t see what could trigger such an error from my
side, since everything is delegated to PETSc in the SNESSolve.
Is there an easy way to debug this?
Is there some way to dump _everything_ related to a KSP (Mat + PC + ISes) for
“easier” debugging?
Thank you,
Pierre
frame #5: 0x000000011a8d29fd
libpetsc.3.011.dylib`::PetscAttachDebuggerErrorHandler(comm=1140850689,
line=57, fun="VecNorm_MPI",
file="/Users/jolivet/Documents/repositories/petsc/src/vec/vec/impls/mpi/pvec2.c",
num=77, p=PETSC_ERROR_INITIAL, mess="MPI_Allreduce() called in different
locations (code lines) on different processors", ctx=0x0000000000000000) at
adebug.c:444:10
frame #6: 0x000000011a8d4e32
libpetsc.3.011.dylib`::PetscError(comm=1140850689, line=57, func="VecNorm_MPI",
file="/Users/jolivet/Documents/repositories/petsc/src/vec/vec/impls/mpi/pvec2.c",
n=77, p=PETSC_ERROR_INITIAL, mess="MPI_Allreduce() called in different
locations (code lines) on different processors") at err.c:355:19
frame #7: 0x000000011a7a7c6d
libpetsc.3.011.dylib`::PetscAllreduceBarrierCheck(comm=-2080374780, ctn=1,
line=57, func="VecNorm_MPI",
file="/Users/jolivet/Documents/repositories/petsc/src/vec/vec/impls/mpi/pvec2.c")
at pbarrier.c:28:31
frame #8: 0x000000011aa9af9b
libpetsc.3.011.dylib`::VecNorm_MPI(xin=0x00007fd157aa4060, type=NORM_2,
z=0x00007ffee2afe588) at pvec2.c:57:12
frame #9: 0x000000011aabd056
libpetsc.3.011.dylib`::VecNorm(x=0x00007fd157aa4060, type=NORM_2,
val=0x00007ffee2afe588) at rvector.c:221:10
frame #10: 0x000000011bc15a10
libpetsc.3.011.dylib`KSPSolve_PREONLY(ksp=0x00007fd156ab9660) at preonly.c:32:14
frame #11: 0x000000011bc8f88d
libpetsc.3.011.dylib`::KSPSolve(ksp=0x00007fd156ab9660, b=0x00007fd1579ee460,
x=0x00007fd157aa4060) at itfunc.c:764:10
frame #12: 0x000000011ba3c965
libpetsc.3.011.dylib`PCApply_FieldSplit(pc=0x00007fd15820ec60,
x=0x00007fd158321a60, y=0x00007fd15832c860) at fieldsplit.c:1266:14
frame #13: 0x000000011bb2fb19
libpetsc.3.011.dylib`::PCApply(pc=0x00007fd15820ec60, x=0x00007fd158321a60,
y=0x00007fd15832c860) at precon.c:444:10
frame #14: 0x000000011bc5496b
libpetsc.3.011.dylib`KSP_PCApply(ksp=0x00007fd157afd660, x=0x00007fd158321a60,
y=0x00007fd15832c860) at kspimpl.h:281:12
frame #15: 0x000000011bc532ff
libpetsc.3.011.dylib`KSPFGMRESCycle(itcount=0x00007ffee2afe9e8,
ksp=0x00007fd157afd660) at fgmres.c:166:12
frame #16: 0x000000011bc56099
libpetsc.3.011.dylib`KSPSolve_FGMRES(ksp=0x00007fd157afd660) at fgmres.c:291:10
frame #17: 0x000000011bc8f88d
libpetsc.3.011.dylib`::KSPSolve(ksp=0x00007fd157afd660, b=0x00007fd157b66260,
x=0x00007fd157bd3260) at itfunc.c:764:10
frame #18: 0x000000011be2d916
libpetsc.3.011.dylib`SNESSolve_NEWTONLS(snes=0x00007fd157b6f460) at ls.c:225:12
frame #19: 0x000000011bda9830
libpetsc.3.011.dylib`::SNESSolve(snes=0x00007fd157b6f460, b=0x0000000000000000,
x=0x00007fd157b90060) at snes.c:4433:12
frame #5: 0x00000001112039fd
libpetsc.3.011.dylib`::PetscAttachDebuggerErrorHandler(comm=1140850689,
line=1243, fun="VecGetSubVector",
file="/Users/jolivet/Documents/repositories/petsc/src/vec/vec/interface/rvector.c",
num=77, p=PETSC_ERROR_INITIAL, mess="MPI_Allreduce() called in different
locations (code lines) on different processors", ctx=0x0000000000000000) at
adebug.c:444:10
frame #6: 0x0000000111205e32
libpetsc.3.011.dylib`::PetscError(comm=1140850689, line=1243,
func="VecGetSubVector",
file="/Users/jolivet/Documents/repositories/petsc/src/vec/vec/interface/rvector.c",
n=77, p=PETSC_ERROR_INITIAL, mess="MPI_Allreduce() called in different
locations (code lines) on different processors") at err.c:355:19
frame #7: 0x00000001110d8c6d
libpetsc.3.011.dylib`::PetscAllreduceBarrierCheck(comm=-2080374777, ctn=1,
line=1243, func="VecGetSubVector",
file="/Users/jolivet/Documents/repositories/petsc/src/vec/vec/interface/rvector.c")
at pbarrier.c:28:31
frame #8: 0x00000001114007fb
libpetsc.3.011.dylib`::VecGetSubVector(X=0x00007fe49da14a60,
is=0x00007fe49d298260, Y=0x00007fe49e18ee60) at rvector.c:1243:12
frame #9: 0x0000000111814e6a
libpetsc.3.011.dylib`MatMult_Nest(A=0x00007fe49d286860, x=0x00007fe49e17ee60,
y=0x00007fe49da14a60) at matnest.c:45:32
frame #10: 0x00000001114c1246
libpetsc.3.011.dylib`::MatMult(mat=0x00007fe49d286860, x=0x00007fe49e17ee60,
y=0x00007fe49da14a60) at matrix.c:2370:10
frame #11: 0x000000011236d642
libpetsc.3.011.dylib`PCApply_FieldSplit(pc=0x00007fe49ea1b060,
x=0x00007fe49e1c1a60, y=0x00007fe49e17ee60) at fieldsplit.c:1261:14
frame #12: 0x0000000112460b19
libpetsc.3.011.dylib`::PCApply(pc=0x00007fe49ea1b060, x=0x00007fe49e1c1a60,
y=0x00007fe49e17ee60) at precon.c:444:10
frame #13: 0x000000011258596b
libpetsc.3.011.dylib`KSP_PCApply(ksp=0x00007fe49dabe260, x=0x00007fe49e1c1a60,
y=0x00007fe49e17ee60) at kspimpl.h:281:12
frame #14: 0x00000001125842ff
libpetsc.3.011.dylib`KSPFGMRESCycle(itcount=0x00007ffeec1319e8,
ksp=0x00007fe49dabe260) at fgmres.c:166:12
frame #15: 0x0000000112587099
libpetsc.3.011.dylib`KSPSolve_FGMRES(ksp=0x00007fe49dabe260) at fgmres.c:291:10
frame #16: 0x00000001125c088d
libpetsc.3.011.dylib`::KSPSolve(ksp=0x00007fe49dabe260, b=0x00007fe49daeca60,
x=0x00007fe49ea54260) at itfunc.c:764:10
frame #17: 0x000000011275e916
libpetsc.3.011.dylib`SNESSolve_NEWTONLS(snes=0x00007fe49ea5ec60) at ls.c:225:12
frame #18: 0x00000001126da830
libpetsc.3.011.dylib`::SNESSolve(snes=0x00007fe49ea5ec60, b=0x0000000000000000,
x=0x00007fe49dae3a60) at snes.c:4433:12
