Hello Barry, Matthew, thanks for the replies ! Yes, it is our custom code, and it also happens when setting -pc_type bjacobi. Before testing an iterative solver, we were using MUMPS (-ksp_type preonly -ksp_pc_type lu -pc_factor_mat_solver_type mumps) without issues.
Running the ex19 (as "mpirun -n 4 ex19 -da_refine 5") did not produce any problem. To reproduce the situation on my computer, I was able to reproduce the error for a small case and -pc_type bjacobi. For that particular case, when running in the cluster the error appears at the very last iteration: ===== 27 KSP Residual norm 8.230378644666e-06 [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Invalid argument [0]PETSC ERROR: Scalar value must be same on all processes, argument # 3 ==== whereas running on my computer the error is not launched and convergence is reached instead: ==== Linear interp_ solve converged due to CONVERGED_RTOL iterations 27 ==== I will run valgrind to seek for possible memory corruptions. thank you Alfredo On Mon, Aug 24, 2020 at 9:00 PM Barry Smith <[email protected]> wrote: > > Oh yes, it could happen with Nan. > > KSPGMRESClassicalGramSchmidtOrthogonalization() > calls KSPCheckDot(ksp,lhh[j]); so should detect any NAN that appear and > set ksp->convergedreason but the call to MAXPY() is still made before > returning and hence producing the error message. > > We should circuit the orthogonalization as soon as it sees a Nan/Inf > and return immediately for GMRES to cleanup and produce a very useful error > message. > > Alfredo, > > It is also possible that the hypre preconditioners are producing a Nan > because your matrix is too difficult for them to handle, but it would be > odd to happen after many iterations. > > As I suggested before run with -pc_type bjacobi to see if you get the > same problem. > > Barry > > > On Aug 24, 2020, at 6:38 PM, Matthew Knepley <[email protected]> wrote: > > On Mon, Aug 24, 2020 at 6:27 PM Barry Smith <[email protected]> wrote: > >> >> Alfredo, >> >> This should never happen. The input to the VecMAXPY in gmres is >> computed via VMDot which produces the same result on all processes. >> >> If you run with -pc_type bjacobi does it also happen? >> >> Is this your custom code or does it happen in PETSc examples also? >> Like src/snes/tutorials/ex19 -da_refine 5 >> >> Could be memory corruption, can you run under valgrind? >> > > Couldn't it happen if something generates a NaN? That also should not > happen, but I was allowing that pilut might do it. > > Thanks, > > Matt > > >> Barry >> >> >> > On Aug 24, 2020, at 4:05 PM, Alfredo Jaramillo < >> [email protected]> wrote: >> > >> > Dear PETSc developers, >> > >> > I'm trying to solve a linear problem with GMRES preconditioned with >> pilut from HYPRE. For this I'm using the options: >> > >> > -ksp_type gmres -pc_type hypre -pc_hypre_type pilut -ksp_monitor >> > >> > If I use a single core, GMRES (+ pilut or euclid) converges. However, >> when using multiple cores the next error appears after some number of >> iterations: >> > >> > [0]PETSC ERROR: Scalar value must be same on all processes, argument # 3 >> > >> > relative to the function VecMAXPY. I attached a screenshot with more >> detailed output. The same happens when using euclid. Can you please give me >> some insight on this? >> > >> > best regards >> > Alfredo >> > <Screenshot from 2020-08-24 17-57-52.png> >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > <http://www.cse.buffalo.edu/~knepley/> > > >
