On one system you get this error, on another system with the identical code and test case you do not get the error?
You get it with three iterative methods but not with MUMPS? Barry > On Aug 24, 2020, at 8:35 PM, Alfredo Jaramillo <[email protected]> > wrote: > > Hello Barry, Matthew, thanks for the replies ! > > Yes, it is our custom code, and it also happens when setting -pc_type > bjacobi. Before testing an iterative solver, we were using MUMPS (-ksp_type > preonly -ksp_pc_type lu -pc_factor_mat_solver_type mumps) without issues. > > Running the ex19 (as "mpirun -n 4 ex19 -da_refine 5") did not produce any > problem. > > To reproduce the situation on my computer, I was able to reproduce the error > for a small case and -pc_type bjacobi. For that particular case, when running > in the cluster the error appears at the very last iteration: > > ===== > 27 KSP Residual norm 8.230378644666e-06 > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Invalid argument > [0]PETSC ERROR: Scalar value must be same on all processes, argument # 3 > ==== > > whereas running on my computer the error is not launched and convergence is > reached instead: > > ==== > Linear interp_ solve converged due to CONVERGED_RTOL iterations 27 > ==== > > I will run valgrind to seek for possible memory corruptions. > > thank you > Alfredo > > On Mon, Aug 24, 2020 at 9:00 PM Barry Smith <[email protected] > <mailto:[email protected]>> wrote: > > Oh yes, it could happen with Nan. > > KSPGMRESClassicalGramSchmidtOrthogonalization() calls > KSPCheckDot(ksp,lhh[j]); so should detect any NAN that appear and set > ksp->convergedreason but the call to MAXPY() is still made before returning > and hence producing the error message. > > We should circuit the orthogonalization as soon as it sees a Nan/Inf and > return immediately for GMRES to cleanup and produce a very useful error > message. > > Alfredo, > > It is also possible that the hypre preconditioners are producing a Nan > because your matrix is too difficult for them to handle, but it would be odd > to happen after many iterations. > > As I suggested before run with -pc_type bjacobi to see if you get the same > problem. > > Barry > > >> On Aug 24, 2020, at 6:38 PM, Matthew Knepley <[email protected] >> <mailto:[email protected]>> wrote: >> >> On Mon, Aug 24, 2020 at 6:27 PM Barry Smith <[email protected] >> <mailto:[email protected]>> wrote: >> >> Alfredo, >> >> This should never happen. The input to the VecMAXPY in gmres is >> computed via VMDot which produces the same result on all processes. >> >> If you run with -pc_type bjacobi does it also happen? >> >> Is this your custom code or does it happen in PETSc examples also? >> Like src/snes/tutorials/ex19 -da_refine 5 >> >> Could be memory corruption, can you run under valgrind? >> >> Couldn't it happen if something generates a NaN? That also should not >> happen, but I was allowing that pilut might do it. >> >> Thanks, >> >> Matt >> >> Barry >> >> >> > On Aug 24, 2020, at 4:05 PM, Alfredo Jaramillo <[email protected] >> > <mailto:[email protected]>> wrote: >> > >> > Dear PETSc developers, >> > >> > I'm trying to solve a linear problem with GMRES preconditioned with pilut >> > from HYPRE. For this I'm using the options: >> > >> > -ksp_type gmres -pc_type hypre -pc_hypre_type pilut -ksp_monitor >> > >> > If I use a single core, GMRES (+ pilut or euclid) converges. However, when >> > using multiple cores the next error appears after some number of >> > iterations: >> > >> > [0]PETSC ERROR: Scalar value must be same on all processes, argument # 3 >> > >> > relative to the function VecMAXPY. I attached a screenshot with more >> > detailed output. The same happens when using euclid. Can you please give >> > me some insight on this? >> > >> > best regards >> > Alfredo >> > <Screenshot from 2020-08-24 17-57-52.png> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments >> is infinitely more interesting than any results to which their experiments >> lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/> >
