On Tue, Jun 26, 2018 at 8:26 AM, Mark Adams <[email protected]> wrote: > > > On Tue, Jun 26, 2018 at 12:19 AM Junchao Zhang <[email protected]> > wrote: > >> Mark, >> Your email reminded me my recent experiments. My PETSc was configured >> --with-openmp=1. >> With hypre, my job ran out of time. That was on an Argonne Xeon cluster. >> > > Interesting. I tested on Cori's Haswell nodes and it looked fine. I did > not time it but seemed OK. > > >> I repeated the experiments on Cori's Haswell nodes. --with-openmp=1, >> "Linear solve converged due to CONVERGED_RTOL iterations 5". But it took >> very long time (10 mins). Without --with-openmp=1, it took less than 1 >> second. >> > > Humm. I seemed to run OK on Cori's Haswell nodes. Where you running a > significant sized job? I was test small serial runs. >
I ran with 27 processors and each had 30^3 unknowns. > > >> >> --Junchao Zhang >> >> On Fri, Jun 22, 2018 at 3:33 PM, Mark Adams <[email protected]> wrote: >> >>> We are using KNL (Cori) and hypre is not working when configured >>> with '--with-openmp=1', even when not using threads (as far as I can tell, >>> I never use threads). >>> >>> Hypre is not converging, for instance with an optimized build: >>> >>> srun -n 1 ./ex56 -pc_type hypre -ksp_monitor -ksp_converged_reason >>> -ksp_type cg -pc_hypre_type boomeramg >>> OMP: Warning #239: KMP_AFFINITY: granularity=fine will be used. >>> 0 KSP Residual norm 7.366251922394e+22 >>> 1 KSP Residual norm 3.676434682799e+22 >>> Linear solve did not converge due to DIVERGED_INDEFINITE_PC iterations 2 >>> >>> Interestingly in debug mode it almost looks good but it is dying: >>> >>> 05:09 nid02516 maint *= >>> ~/petsc_install/petsc/src/ksp/ksp/examples/tutorials$ >>> make >>> PETSC_DIR=/global/homes/m/madams/petsc_install/petsc-cori-knl-dbg64-intel-omp >>> PETSC_ARCH="" run >>> srun -n 1 ./ex56 -pc_type hypre -ksp_monitor -ksp_converged_reason >>> -ksp_type cg -pc_hypre_type boomeramg >>> OMP: Warning #239: KMP_AFFINITY: granularity=fine will be used. >>> 0 KSP Residual norm 7.882081712007e+02 >>> 1 KSP Residual norm 2.500214073037e+02 >>> 2 KSP Residual norm 3.371746347713e+01 >>> 3 KSP Residual norm 2.918759396143e+00 >>> 4 KSP Residual norm 9.006505495017e-01 >>> Linear solve did not converge due to DIVERGED_INDEFINITE_PC iterations 5 >>> >>> This test runs fine on Xeon nodes. I assume that Hypre has been tested >>> on KNL. GAMG runs fine, of coarse and the initial residual is similar to >>> this debug run. >>> >>> Could PETSc be messing up the matrix conversion to hypre >>> '--with-openmp=1' ? >>> >>> Any ideas? >>> >>> Thanks, >>> Mark >>> >>> >>
