I did not set OMP_NUM_THREADS in my .bashrc or job script. The job ran out of time. If I did export OMP_NUM_THREADS=1 in job script on Cori, the job ran very slowly, i.e., finished in 200 seconds compared to 1s without --with-openmp.
--Junchao Zhang On Tue, Jun 26, 2018 at 12:05 PM, Balay, Satish <[email protected]> wrote: > I wonder if these jobs are scheduled in such a way so that they are not > oversubscribed. > > i.e number_mpi_jobs_per_node * number_of_openmp_threads_per_node <= > no_of_cores_per_node > > Satish > > On Tue, 26 Jun 2018, Mark Adams wrote: > > > Interesting, I am seeing the same thing with ksp/ex56 (elasticity) with > > 30^3 grid on each process. One process runs fine (1.5 sec) but 8 > processes > > with 30^3 on each process took 156 sec. > > > > And, PETSc's log_view is running extremely slow. I have the total time > > (156) but each event is taking like a minute or more to come out. > > > > On Tue, Jun 26, 2018 at 10:13 AM Junchao Zhang <[email protected]> > wrote: > > > > > > > > On Tue, Jun 26, 2018 at 8:26 AM, Mark Adams <[email protected]> wrote: > > > > > >> > > >> > > >> On Tue, Jun 26, 2018 at 12:19 AM Junchao Zhang <[email protected]> > > >> wrote: > > >> > > >>> Mark, > > >>> Your email reminded me my recent experiments. My PETSc was > configured --with-openmp=1. > > >>> With hypre, my job ran out of time. That was on an Argonne Xeon > cluster. > > >>> > > >> > > >> Interesting. I tested on Cori's Haswell nodes and it looked fine. I > did > > >> not time it but seemed OK. > > >> > > >> > > >>> I repeated the experiments on Cori's Haswell nodes. > --with-openmp=1, > > >>> "Linear solve converged due to CONVERGED_RTOL iterations 5". But it > took > > >>> very long time (10 mins). Without --with-openmp=1, it took less than > 1 > > >>> second. > > >>> > > >> > > >> Humm. I seemed to run OK on Cori's Haswell nodes. Where you running a > > >> significant sized job? I was test small serial runs. > > >> > > > > > > I ran with 27 processors and each had 30^3 unknowns. > > > > > >> > > >> > > >>> > > >>> --Junchao Zhang > > >>> > > >>> On Fri, Jun 22, 2018 at 3:33 PM, Mark Adams <[email protected]> wrote: > > >>> > > >>>> We are using KNL (Cori) and hypre is not working when configured > > >>>> with '--with-openmp=1', even when not using threads (as far as I > can tell, > > >>>> I never use threads). > > >>>> > > >>>> Hypre is not converging, for instance with an optimized build: > > >>>> > > >>>> srun -n 1 ./ex56 -pc_type hypre -ksp_monitor -ksp_converged_reason > > >>>> -ksp_type cg -pc_hypre_type boomeramg > > >>>> OMP: Warning #239: KMP_AFFINITY: granularity=fine will be used. > > >>>> 0 KSP Residual norm 7.366251922394e+22 > > >>>> 1 KSP Residual norm 3.676434682799e+22 > > >>>> Linear solve did not converge due to DIVERGED_INDEFINITE_PC > iterations 2 > > >>>> > > >>>> Interestingly in debug mode it almost looks good but it is dying: > > >>>> > > >>>> 05:09 nid02516 maint *= > > >>>> ~/petsc_install/petsc/src/ksp/ksp/examples/tutorials$ make > > >>>> PETSC_DIR=/global/homes/m/madams/petsc_install/petsc- > cori-knl-dbg64-intel-omp > > >>>> PETSC_ARCH="" run > > >>>> srun -n 1 ./ex56 -pc_type hypre -ksp_monitor -ksp_converged_reason > > >>>> -ksp_type cg -pc_hypre_type boomeramg > > >>>> OMP: Warning #239: KMP_AFFINITY: granularity=fine will be used. > > >>>> 0 KSP Residual norm 7.882081712007e+02 > > >>>> 1 KSP Residual norm 2.500214073037e+02 > > >>>> 2 KSP Residual norm 3.371746347713e+01 > > >>>> 3 KSP Residual norm 2.918759396143e+00 > > >>>> 4 KSP Residual norm 9.006505495017e-01 > > >>>> Linear solve did not converge due to DIVERGED_INDEFINITE_PC > iterations 5 > > >>>> > > >>>> This test runs fine on Xeon nodes. I assume that Hypre has been > tested > > >>>> on KNL. GAMG runs fine, of coarse and the initial residual is > similar to > > >>>> this debug run. > > >>>> > > >>>> Could PETSc be messing up the matrix conversion to hypre > > >>>> '--with-openmp=1' ? > > >>>> > > >>>> Any ideas? > > >>>> > > >>>> Thanks, > > >>>> Mark > > >>>> > > >>>> > > >>> > > > > > > >
