thank you, Barry, I wasn't able to reproduce the error on my computer, neither on a second cluster. On the first cluster, I requested to activate X11 at some node for attaching a debugger, and that activation (if possible) should take some time. I will inform you of any news on that.
kind regards Alfredo On Tue, Aug 25, 2020 at 6:46 PM Barry Smith <[email protected]> wrote: > > I have submitted a merge request > https://gitlab.com/petsc/petsc/-/merge_requests/3096 that will make the > error handling and message clearer in the future. > > Barry > > > On Aug 25, 2020, at 8:55 AM, Alfredo Jaramillo <[email protected]> > wrote: > > In fact, on my machine the code is compiled with gnu, and on the cluster > it is compiled with intel (2015) compilers. I just run the program with > "-fp_trap" and got: > > =============================================================== > |> Assembling interface problem. Unk # 56 > |> Solving interface problem > Residual norms for interp_ solve. > 0 KSP Residual norm 3.642615470862e+03 > [0]PETSC ERROR: *** unknown floating point error occurred *** > [0]PETSC ERROR: The specific exception can be determined by running in a > debugger. When the > [0]PETSC ERROR: debugger traps the signal, the exception can be found with > fetestexcept(0x3f) > [0]PETSC ERROR: where the result is a bitwise OR of the following flags: > [0]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 > FE_UNDERFLOW=0x10 FE_INEXACT=0x20 > [0]PETSC ERROR: Try option -start_in_debugger > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [1]PETSC ERROR: [2]PETSC ERROR: *** unknown floating point error occurred > *** > [3]PETSC ERROR: *** unknown floating point error occurred *** > [3]PETSC ERROR: The specific exception can be determined by running in a > debugger. When the > [4]PETSC ERROR: *** unknown floating point error occurred *** > [4]PETSC ERROR: The specific exception can be determined by running in a > debugger. When the > [4]PETSC ERROR: [5]PETSC ERROR: *** unknown floating point error occurred > *** > [5]PETSC ERROR: The specific exception can be determined by running in a > debugger. When the > [5]PETSC ERROR: debugger traps the signal, the exception can be found with > fetestexcept(0x3f) > [5]PETSC ERROR: where the result is a bitwise OR of the following flags: > [6]PETSC ERROR: *** unknown floating point error occurred *** > [6]PETSC ERROR: The specific exception can be determined by running in a > debugger. When the > [6]PETSC ERROR: debugger traps the signal, the exception can be found with > fetestexcept(0x3f) > [6]PETSC ERROR: where the result is a bitwise OR of the following flags: > [6]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 > FE_UNDERFLOW=0x10 FE_INEXACT=0x20 > [7]PETSC ERROR: *** unknown floating point error occurred *** > [7]PETSC ERROR: The specific exception can be determined by running in a > debugger. When the > [7]PETSC ERROR: debugger traps the signal, the exception can be found with > fetestexcept(0x3f) > [7]PETSC ERROR: where the result is a bitwise OR of the following flags: > [7]PETSC ERROR: FE_INVALID=0x1 FE_DIVBYZERO=0x4 FE_OVERFLOW=0x8 > FE_UNDERFLOW=0x10 FE_INEXACT=0x20 > [7]PETSC ERROR: Try option -start_in_debugger > [7]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] PetscDefaultFPTrap line 355 > /mnt/lustre/home/ajaramillo/petsc-3.13.0/src/sys/error/fp.c > [0]PETSC ERROR: [0] VecMDot line 1154 > /mnt/lustre/home/ajaramillo/petsc-3.13.0/src/vec/vec/interface/rvector.c > [0]PETSC ERROR: [0] KSPGMRESClassicalGramSchmidtOrthogonalization line 44 > /mnt/lustre/home/ajaramillo/petsc-3.13.0/src/ksp/ksp/impls/gmres/borthog2.c > [0]PETSC ERROR: [0] KSPGMRESCycle line 122 > /mnt/lustre/home/ajaramillo/petsc-3.13.0/src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: [0] KSPSolve_GMRES line 225 > /mnt/lustre/home/ajaramillo/petsc-3.13.0/src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: [0] KSPSolve_Private line 590 > /mnt/lustre/home/ajaramillo/petsc-3.13.0/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: *** unknown floating point error occurred *** > =============================================================== > > So it seems that in fact a division by 0 is taking place. I will try to > run this in debug mode. > > thanks > Alfredo > > On Tue, Aug 25, 2020 at 10:23 AM Barry Smith <[email protected]> wrote: > >> >> Sounds like it might be a compiler problem generating bad code. >> >> On the machine where it fails you can run with -fp_trap to have it >> error out as soon as a Nan or Inf appears. If you can use the debugger on >> that machine you can tell the debugger to catch floating point exceptions >> and see the exact line an values of variables where a Nan or Inf appear. >> >> As Matt conjectured it is likely there is a divide by zero before >> PETSc detects and it may be helpful to find out exactly where that happens. >> >> Barry >> >> >> On Aug 25, 2020, at 8:03 AM, Alfredo Jaramillo <[email protected]> >> wrote: >> >> Yes, Barry, that is correct. >> >> >> >> On Tue, Aug 25, 2020 at 1:02 AM Barry Smith <[email protected]> wrote: >> >>> >>> On one system you get this error, on another system with the identical >>> code and test case you do not get the error? >>> >>> You get it with three iterative methods but not with MUMPS? >>> >>> Barry >>> >>> >>> On Aug 24, 2020, at 8:35 PM, Alfredo Jaramillo < >>> [email protected]> wrote: >>> >>> Hello Barry, Matthew, thanks for the replies ! >>> >>> Yes, it is our custom code, and it also happens when setting -pc_type >>> bjacobi. Before testing an iterative solver, we were using MUMPS (-ksp_type >>> preonly -ksp_pc_type lu -pc_factor_mat_solver_type mumps) without issues. >>> >>> Running the ex19 (as "mpirun -n 4 ex19 -da_refine 5") did not produce >>> any problem. >>> >>> To reproduce the situation on my computer, I was able to reproduce the >>> error for a small case and -pc_type bjacobi. For that particular case, when >>> running in the cluster the error appears at the very last iteration: >>> >>> ===== >>> 27 KSP Residual norm 8.230378644666e-06 >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: Invalid argument >>> [0]PETSC ERROR: Scalar value must be same on all processes, argument # 3 >>> ==== >>> >>> whereas running on my computer the error is not launched and convergence >>> is reached instead: >>> >>> ==== >>> Linear interp_ solve converged due to CONVERGED_RTOL iterations 27 >>> ==== >>> >>> I will run valgrind to seek for possible memory corruptions. >>> >>> thank you >>> Alfredo >>> >>> On Mon, Aug 24, 2020 at 9:00 PM Barry Smith <[email protected]> wrote: >>> >>>> >>>> Oh yes, it could happen with Nan. >>>> >>>> KSPGMRESClassicalGramSchmidtOrthogonalization() >>>> calls KSPCheckDot(ksp,lhh[j]); so should detect any NAN that appear and >>>> set ksp->convergedreason but the call to MAXPY() is still made before >>>> returning and hence producing the error message. >>>> >>>> We should circuit the orthogonalization as soon as it sees a Nan/Inf >>>> and return immediately for GMRES to cleanup and produce a very useful error >>>> message. >>>> >>>> Alfredo, >>>> >>>> It is also possible that the hypre preconditioners are producing a >>>> Nan because your matrix is too difficult for them to handle, but it would >>>> be odd to happen after many iterations. >>>> >>>> As I suggested before run with -pc_type bjacobi to see if you get >>>> the same problem. >>>> >>>> Barry >>>> >>>> >>>> On Aug 24, 2020, at 6:38 PM, Matthew Knepley <[email protected]> wrote: >>>> >>>> On Mon, Aug 24, 2020 at 6:27 PM Barry Smith <[email protected]> wrote: >>>> >>>>> >>>>> Alfredo, >>>>> >>>>> This should never happen. The input to the VecMAXPY in gmres is >>>>> computed via VMDot which produces the same result on all processes. >>>>> >>>>> If you run with -pc_type bjacobi does it also happen? >>>>> >>>>> Is this your custom code or does it happen in PETSc examples >>>>> also? Like src/snes/tutorials/ex19 -da_refine 5 >>>>> >>>>> Could be memory corruption, can you run under valgrind? >>>>> >>>> >>>> Couldn't it happen if something generates a NaN? That also should not >>>> happen, but I was allowing that pilut might do it. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Barry >>>>> >>>>> >>>>> > On Aug 24, 2020, at 4:05 PM, Alfredo Jaramillo < >>>>> [email protected]> wrote: >>>>> > >>>>> > Dear PETSc developers, >>>>> > >>>>> > I'm trying to solve a linear problem with GMRES preconditioned with >>>>> pilut from HYPRE. For this I'm using the options: >>>>> > >>>>> > -ksp_type gmres -pc_type hypre -pc_hypre_type pilut -ksp_monitor >>>>> > >>>>> > If I use a single core, GMRES (+ pilut or euclid) converges. >>>>> However, when using multiple cores the next error appears after some >>>>> number >>>>> of iterations: >>>>> > >>>>> > [0]PETSC ERROR: Scalar value must be same on all processes, argument >>>>> # 3 >>>>> > >>>>> > relative to the function VecMAXPY. I attached a screenshot with more >>>>> detailed output. The same happens when using euclid. Can you please give >>>>> me >>>>> some insight on this? >>>>> > >>>>> > best regards >>>>> > Alfredo >>>>> > <Screenshot from 2020-08-24 17-57-52.png> >>>>> >>>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> <http://www.cse.buffalo.edu/~knepley/> >>>> >>>> >>>> >>> >> >
