Hi Mark, I don’t have an account on KNL and have not tested hypre there with OpenMP. I believe other people have used it however, but I haven’t heard about any non-convergence issues before now. Ulrike
From: Mark Adams <[email protected]> Sent: Friday, June 22, 2018 4:28 PM To: Smith, Barry F. <[email protected]> Cc: For users of the development version of PETSc <[email protected]>; Yang, Ulrike Meier <[email protected]>; [email protected] Trebotich <[email protected]> Subject: Re: [petsc-dev] problem with hypre with '--with-openmp=1' On Fri, Jun 22, 2018 at 4:39 PM Smith, Barry F. <[email protected]<mailto:[email protected]>> wrote: > On Jun 22, 2018, at 3:33 PM, Mark Adams > <[email protected]<mailto:[email protected]>> wrote: > > We are using KNL (Cori) and hypre is not working when configured with > '--with-openmp=1', even when not using threads (as far as I can tell, I never > use threads). It does seem to run correctly without the --with-openmp option? Yes. And I've run it through valgrind. Ulrike: have you tested on KNL with openMP? PETSc code does not know about --with-openmp at all so my guess is some bug in the compiler when building hypre with OpenMP. Barry > > Hypre is not converging, for instance with an optimized build: > > srun -n 1 ./ex56 -pc_type hypre -ksp_monitor -ksp_converged_reason -ksp_type > cg -pc_hypre_type boomeramg > OMP: Warning #239: KMP_AFFINITY: granularity=fine will be used. > 0 KSP Residual norm 7.366251922394e+22 > 1 KSP Residual norm 3.676434682799e+22 > Linear solve did not converge due to DIVERGED_INDEFINITE_PC iterations 2 > > Interestingly in debug mode it almost looks good but it is dying: > > 05:09 nid02516 maint *= ~/petsc_install/petsc/src/ksp/ksp/examples/tutorials$ > make > PETSC_DIR=/global/homes/m/madams/petsc_install/petsc-cori-knl-dbg64-intel-omp > PETSC_ARCH="" run > srun -n 1 ./ex56 -pc_type hypre -ksp_monitor -ksp_converged_reason -ksp_type > cg -pc_hypre_type boomeramg > OMP: Warning #239: KMP_AFFINITY: granularity=fine will be used. > 0 KSP Residual norm 7.882081712007e+02 > 1 KSP Residual norm 2.500214073037e+02 > 2 KSP Residual norm 3.371746347713e+01 > 3 KSP Residual norm 2.918759396143e+00 > 4 KSP Residual norm 9.006505495017e-01 > Linear solve did not converge due to DIVERGED_INDEFINITE_PC iterations 5 > > This test runs fine on Xeon nodes. I assume that Hypre has been tested on > KNL. GAMG runs fine, of coarse and the initial residual is similar to this > debug run. > > Could PETSc be messing up the matrix conversion to hypre '--with-openmp=1' ? > > Any ideas? > > Thanks, > Mark >
