Re: [petsc-dev] problem with hypre with '--with-openmp=1'
On Fri, Jun 22, 2018 at 4:39 PM Smith, Barry F. wrote: > > > > On Jun 22, 2018, at 3:33 PM, Mark Adams wrote: > > > > We are using KNL (Cori) and hypre is not working when configured with > '--with-openmp=1', even when not using threads (as far as I can tell, I > never use threads). > >It does seem to run correctly without the --with-openmp option? > Yes. And I've run it through valgrind. Ulrike: have you tested on KNL with openMP? > >PETSc code does not know about --with-openmp at all so my guess is some > bug in the compiler when building hypre with OpenMP. > >Barry > > > > > Hypre is not converging, for instance with an optimized build: > > > > srun -n 1 ./ex56 -pc_type hypre -ksp_monitor -ksp_converged_reason > -ksp_type cg -pc_hypre_type boomeramg > > OMP: Warning #239: KMP_AFFINITY: granularity=fine will be used. > > 0 KSP Residual norm 7.366251922394e+22 > > 1 KSP Residual norm 3.676434682799e+22 > > Linear solve did not converge due to DIVERGED_INDEFINITE_PC iterations 2 > > > > Interestingly in debug mode it almost looks good but it is dying: > > > > 05:09 nid02516 maint *= > ~/petsc_install/petsc/src/ksp/ksp/examples/tutorials$ make > PETSC_DIR=/global/homes/m/madams/petsc_install/petsc-cori-knl-dbg64-intel-omp > PETSC_ARCH="" run > > srun -n 1 ./ex56 -pc_type hypre -ksp_monitor -ksp_converged_reason > -ksp_type cg -pc_hypre_type boomeramg > > OMP: Warning #239: KMP_AFFINITY: granularity=fine will be used. > > 0 KSP Residual norm 7.882081712007e+02 > > 1 KSP Residual norm 2.500214073037e+02 > > 2 KSP Residual norm 3.371746347713e+01 > > 3 KSP Residual norm 2.918759396143e+00 > > 4 KSP Residual norm 9.006505495017e-01 > > Linear solve did not converge due to DIVERGED_INDEFINITE_PC iterations 5 > > > > This test runs fine on Xeon nodes. I assume that Hypre has been tested > on KNL. GAMG runs fine, of coarse and the initial residual is similar to > this debug run. > > > > Could PETSc be messing up the matrix conversion to hypre > '--with-openmp=1' ? > > > > Any ideas? > > > > Thanks, > > Mark > > > >
Re: [petsc-dev] problem with hypre with '--with-openmp=1'
> On Jun 22, 2018, at 3:33 PM, Mark Adams wrote: > > We are using KNL (Cori) and hypre is not working when configured with > '--with-openmp=1', even when not using threads (as far as I can tell, I never > use threads). It does seem to run correctly without the --with-openmp option? PETSc code does not know about --with-openmp at all so my guess is some bug in the compiler when building hypre with OpenMP. Barry > > Hypre is not converging, for instance with an optimized build: > > srun -n 1 ./ex56 -pc_type hypre -ksp_monitor -ksp_converged_reason -ksp_type > cg -pc_hypre_type boomeramg > OMP: Warning #239: KMP_AFFINITY: granularity=fine will be used. > 0 KSP Residual norm 7.366251922394e+22 > 1 KSP Residual norm 3.676434682799e+22 > Linear solve did not converge due to DIVERGED_INDEFINITE_PC iterations 2 > > Interestingly in debug mode it almost looks good but it is dying: > > 05:09 nid02516 maint *= ~/petsc_install/petsc/src/ksp/ksp/examples/tutorials$ > make > PETSC_DIR=/global/homes/m/madams/petsc_install/petsc-cori-knl-dbg64-intel-omp > PETSC_ARCH="" run > srun -n 1 ./ex56 -pc_type hypre -ksp_monitor -ksp_converged_reason -ksp_type > cg -pc_hypre_type boomeramg > OMP: Warning #239: KMP_AFFINITY: granularity=fine will be used. > 0 KSP Residual norm 7.882081712007e+02 > 1 KSP Residual norm 2.500214073037e+02 > 2 KSP Residual norm 3.371746347713e+01 > 3 KSP Residual norm 2.918759396143e+00 > 4 KSP Residual norm 9.006505495017e-01 > Linear solve did not converge due to DIVERGED_INDEFINITE_PC iterations 5 > > This test runs fine on Xeon nodes. I assume that Hypre has been tested on > KNL. GAMG runs fine, of coarse and the initial residual is similar to this > debug run. > > Could PETSc be messing up the matrix conversion to hypre '--with-openmp=1' ? > > Any ideas? > > Thanks, > Mark >
[petsc-dev] problem with hypre with '--with-openmp=1'
We are using KNL (Cori) and hypre is not working when configured with '--with-openmp=1', even when not using threads (as far as I can tell, I never use threads). Hypre is not converging, for instance with an optimized build: srun -n 1 ./ex56 -pc_type hypre -ksp_monitor -ksp_converged_reason -ksp_type cg -pc_hypre_type boomeramg OMP: Warning #239: KMP_AFFINITY: granularity=fine will be used. 0 KSP Residual norm 7.366251922394e+22 1 KSP Residual norm 3.676434682799e+22 Linear solve did not converge due to DIVERGED_INDEFINITE_PC iterations 2 Interestingly in debug mode it almost looks good but it is dying: 05:09 nid02516 maint *= ~/petsc_install/petsc/src/ksp/ksp/examples/tutorials$ make PETSC_DIR=/global/homes/m/madams/petsc_install/petsc-cori-knl-dbg64-intel-omp PETSC_ARCH="" run srun -n 1 ./ex56 -pc_type hypre -ksp_monitor -ksp_converged_reason -ksp_type cg -pc_hypre_type boomeramg OMP: Warning #239: KMP_AFFINITY: granularity=fine will be used. 0 KSP Residual norm 7.882081712007e+02 1 KSP Residual norm 2.500214073037e+02 2 KSP Residual norm 3.371746347713e+01 3 KSP Residual norm 2.918759396143e+00 4 KSP Residual norm 9.006505495017e-01 Linear solve did not converge due to DIVERGED_INDEFINITE_PC iterations 5 This test runs fine on Xeon nodes. I assume that Hypre has been tested on KNL. GAMG runs fine, of coarse and the initial residual is similar to this debug run. Could PETSc be messing up the matrix conversion to hypre '--with-openmp=1' ? Any ideas? Thanks, Mark
Re: [petsc-dev] building on Titan
Well these external tools are decoupled from petsc configure [they don't need mpi etc] - so they find their own suitable compilers [which has a better chance of success]. Sure this isn't fail safe. For sowing we provide --download-sowing-cc= C compiler for sowing configure --download-sowing-cxx= CXX compiler for sowing configure --download-sowing-cpp= CPP for sowing configure --download-sowing-cxxcpp= CXX CPP for sowing configure --download-sowing-configure-options= We could add that for all such packages. But then - if its not convinent to install with --download-package - why not manually install it [per packages instructions for this machine/compiler?] --download-cmake --download-make etc are convinence options for systems that dont' have suitable versions preinstalled anyway. No point is petsc installing this on all machines [esp LCFs] Satish On Fri, 22 Jun 2018, Smith, Barry F. wrote: > >Should cmake.py be changed to have > > self.requirescxx11= 1 > >Then at least you get an error message instead of having to wade into the > configure output? > > Barry > > > > On Jun 21, 2018, at 9:41 PM, Balay, Satish wrote: > > > > Hm - we've generally used preinstalled cmake on LCFs - so its not > > clear to me when cmake switched to requring C++11. [this issue never > > came up with so far] > > > > So - suggest avoiding --download-cmake - and use 'module load cmake' > > or whatever is appropriate on titan. > > > > Satish > > > > On Thu, 21 Jun 2018, Smith, Barry F. wrote: > > > >> We now require C++11 to build basic packages like metis? Come on, is that > >> really necessary? Do we really need this recent a version of cmake? > >> > >> > >> > >> > >> Error running configure on CMAKE: Could not execute "cd > >> /autofs/nccs-svm1_home1/adams/petsc/arch-titan-opt64-pgi/externalpackages/cmake-3.11.1 > >> && ./configure > >> --prefix=/autofs/nccs-svm1_home1/adams/petsc/arch-titan-opt64-pgi > >> --parallel=24": > >> - > >> CMake 3.11.1, Copyright 2000-2018 Kitware, Inc. and Contributors > >> Found GNU toolchain > >> C compiler on this system is: gcc > >> - > >> Error when bootstrapping CMake: > >> Cannot find a C++ compiler supporting C++11 on this system. > >> Please specify one using environment variable CXX. > >> See cmake_bootstrap.log for compilers attempted. > >> > >> > >>> On Jun 21, 2018, at 5:54 PM, Mark Adams wrote: > >>> > >>> I am getting a cmake error on Titan... > >>> > >> > > >
Re: [petsc-dev] building on Titan
Should cmake.py be changed to have self.requirescxx11= 1 Then at least you get an error message instead of having to wade into the configure output? Barry > On Jun 21, 2018, at 9:41 PM, Balay, Satish wrote: > > Hm - we've generally used preinstalled cmake on LCFs - so its not > clear to me when cmake switched to requring C++11. [this issue never > came up with so far] > > So - suggest avoiding --download-cmake - and use 'module load cmake' > or whatever is appropriate on titan. > > Satish > > On Thu, 21 Jun 2018, Smith, Barry F. wrote: > >> We now require C++11 to build basic packages like metis? Come on, is that >> really necessary? Do we really need this recent a version of cmake? >> >> >> >> >> Error running configure on CMAKE: Could not execute "cd >> /autofs/nccs-svm1_home1/adams/petsc/arch-titan-opt64-pgi/externalpackages/cmake-3.11.1 >> && ./configure >> --prefix=/autofs/nccs-svm1_home1/adams/petsc/arch-titan-opt64-pgi >> --parallel=24": >> - >> CMake 3.11.1, Copyright 2000-2018 Kitware, Inc. and Contributors >> Found GNU toolchain >> C compiler on this system is: gcc >> - >> Error when bootstrapping CMake: >> Cannot find a C++ compiler supporting C++11 on this system. >> Please specify one using environment variable CXX. >> See cmake_bootstrap.log for compilers attempted. >> >> >>> On Jun 21, 2018, at 5:54 PM, Mark Adams wrote: >>> >>> I am getting a cmake error on Titan... >>> >> >
[petsc-dev] running test harness under batch system
Has anyone run the entire test harness under a batch system? Is this possible, does it require specific commands that should be documented in the users manual? Thanks Barry
Re: [petsc-dev] GAMG and custom MatMults in smoothers
> On Jun 22, 2018, at 5:43 AM, Pierre Jolivet > wrote: > > Hello, > I’m solving a system using a MATSHELL and PCGAMG. > The MPIAIJ Mat I’m giving to GAMG has a specific structure (inherited from > the MATSHELL) I’d like to exploit during the solution phase when the smoother > on the finest level is doing MatMults. > > Is there some way to: > 1) decouple in -log_view the time spent in the MATSHELL MatMult and in the > smoothers MatMult You can register a new event and then inside your MATSHELL MatMult() call PetscLogEventBegin/End on your new event. Note that the MatMult() like will still contain the time for your MatShell mult so you will need to subtract it off to get the time for your non-shell matmults. > 2) hardwire a specific MatMult implementation for the smoother on the finest > level In the latest release you do MatSetOperation() to override the normal matrix vector product with anything else you want. > > Thanks in advance, > Pierre > > PS : here is what I have right now, > MatMult 118 1.0 1.0740e+02 1.6 1.04e+13 1.6 1.7e+06 6.1e+05 > 0.0e+00 47100 90 98 0 47100 90 98 0 81953703 > […] > PCSetUp2 1.0 8.6513e+00 1.0 1.01e+09 1.7 2.6e+05 4.0e+05 > 1.8e+02 5 0 14 10 66 5 0 14 10 68 94598 > PCApply 14 1.0 8.0373e+01 1.1 9.06e+12 1.6 1.3e+06 6.0e+05 > 2.1e+01 45 87 72 78 8 45 87 72 78 8 95365211 // I’m guessing a lot of time > here is being wasted in doing inefficient MatMults on the finest level but > this is only speculation > > Same code with -pc_type none -ksp_max_it 13, > MatMult 14 1.0 1.2936e+01 1.7 1.35e+12 1.6 2.0e+05 6.1e+05 > 0.0e+00 15100 78 93 0 15100 78 93 0 88202079 > > The grid itself is rather simple (two levels, extremely aggressive > coarsening), >type is MULTIPLICATIVE, levels=2 cycles=v >KSP Object: (mg_coarse_) 1024 MPI processes > linear system matrix = precond matrix: > Mat Object: 1024 MPI processes >type: mpiaij >rows=775, cols=775 >total: nonzeros=1793, allocated nonzeros=1793 > > linear system matrix followed by preconditioner matrix: > Mat Object: 1024 MPI processes >type: shell >rows=1369307136, cols=1369307136 > Mat Object: 1024 MPI processes >type: mpiaij >rows=1369307136, cols=1369307136 >total: nonzeros=19896719360, allocated nonzeros=19896719360
Re: [petsc-dev] build error on KNL
On Fri, 22 Jun 2018, Mark Adams wrote: > I get this make error on Cori/KNL > /global/u2/m/madams/petsc_install/petsc/src/mat/impls/hypre/mhypre.c(1453): error #55: too many arguments in invocation of macro "hypre_TFree" hypre_TFree(ptr,HYPRE_MEMORY_HOST); ^ You have a stale version of hypre. Redo the build after: rm -rf arch-cori-knl-opt64-intel Satish
[petsc-dev] GAMG and custom MatMults in smoothers
Hello, I’m solving a system using a MATSHELL and PCGAMG. The MPIAIJ Mat I’m giving to GAMG has a specific structure (inherited from the MATSHELL) I’d like to exploit during the solution phase when the smoother on the finest level is doing MatMults. Is there some way to: 1) decouple in -log_view the time spent in the MATSHELL MatMult and in the smoothers MatMult 2) hardwire a specific MatMult implementation for the smoother on the finest level Thanks in advance, Pierre PS : here is what I have right now, MatMult 118 1.0 1.0740e+02 1.6 1.04e+13 1.6 1.7e+06 6.1e+05 0.0e+00 47100 90 98 0 47100 90 98 0 81953703 […] PCSetUp2 1.0 8.6513e+00 1.0 1.01e+09 1.7 2.6e+05 4.0e+05 1.8e+02 5 0 14 10 66 5 0 14 10 68 94598 PCApply 14 1.0 8.0373e+01 1.1 9.06e+12 1.6 1.3e+06 6.0e+05 2.1e+01 45 87 72 78 8 45 87 72 78 8 95365211 // I’m guessing a lot of time here is being wasted in doing inefficient MatMults on the finest level but this is only speculation Same code with -pc_type none -ksp_max_it 13, MatMult 14 1.0 1.2936e+01 1.7 1.35e+12 1.6 2.0e+05 6.1e+05 0.0e+00 15100 78 93 0 15100 78 93 0 88202079 The grid itself is rather simple (two levels, extremely aggressive coarsening), type is MULTIPLICATIVE, levels=2 cycles=v KSP Object: (mg_coarse_) 1024 MPI processes linear system matrix = precond matrix: Mat Object: 1024 MPI processes type: mpiaij rows=775, cols=775 total: nonzeros=1793, allocated nonzeros=1793 linear system matrix followed by preconditioner matrix: Mat Object: 1024 MPI processes type: shell rows=1369307136, cols=1369307136 Mat Object: 1024 MPI processes type: mpiaij rows=1369307136, cols=1369307136 total: nonzeros=19896719360, allocated nonzeros=19896719360
