On Sun, Jun 24, 2018 at 4:18 PM Pierre Jolivet <[email protected]> wrote:
> > On 24 Jun 2018, at 6:15 PM, Mark Adams <[email protected]> wrote: > > First, -pc_gamg_threshold 0.8 is crazy. 0.08 is very high. probably best > to start with 0. > > > I know it’s high, but that is what works for my application (otherwise the > setup takes forever for just one or two iterations less). > The problem is related to my previous email from Friday ([petsc-dev] GAMG > and custom MatMults in smoothers, thanks Barry for the thorough answer), I > basically need only two or three levels (extremely aggressive coarsening). > OK, you are coarsening somehow. The normal GAMG solver with 0.8 will coarsen so slow that it is crazy. > > I don’t know if this can help, but depending on the value I use for the > threshold, I also get this error: > ksp/pc/impls/gamg/agg.c: if (flid < 0) > SETERRQ(PETSC_COMM_SELF,PETSC_ERR_PLIB,"Cannot find gid1 in table”); > > This could also be from the messed up hash function. > Of course, if I switch to an even crazier threshold, e.g., 0.999999, and > end-up with a one-level method, I don’t see such an error. > Yea, I put logic in there to switch silently because it happens if you use GAMG for a mass matrix (happens all the time). If you run with -info and grep on GAMG you will see a message that it is doing this. > > I know I am to blame here but my application does not use PetscInt as > default integers (plain int instead). I use a lot > of MatMPIAIJSetPreallocationCSR, so it is not as easy as simply recompiling > with --with-64-bit-indices for me. > I’d gladly try anything that you suggest which would still use 32 bit > indices as "primary PetscInt type". > If you feel like remaking PETSc (no reconfigure) with line 89 of pcgamgimpl.h changed as Barry suggested. I think it will fix your problems. Also, you are using a MatShell. GAMG requires AIJ because it digs into its data structures. Not properly abstracted. > > Thanks, > Pierre > > This threshold will delete all of your edges in the coarsening graph, > probably, and GAMG will switch to a one level solve (silently although > -info will tell you). > > And yes, this looks like a 64 bit indices problem, we have: > > #define GAMG_HASH(key) ((((PetscInt)7)*key)%a_tab->size) > > We should probably cast everything to (long long) or (unsigned PetscInt) I > guess ... I'm not sure how this should be done exactly. Any ideas from the > peanut gallery? > > > > On Sun, Jun 24, 2018 at 2:34 AM Pierre Jolivet <[email protected]> > wrote: > >> Hello, >> Any ideas on how to debug this? >> [406]PETSC ERROR: key 328214936 not found in table >> [406]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [406]PETSC ERROR: Petsc Development GIT revision: v3.9.2-603-gceafe64 >> GIT Date: 2018-06-10 12:46:16 -0500 >> [406]PETSC ERROR: >> /ccc/work/cont003/rndm/rndm/FreeFem-sources/src/mpi/FreeFem++-mpi on a >> arch-linux2-c-debug-bullxmpi named curie2698 by jolivetp Sun Jun 24 >> 07:06:07 2018 >> [406]PETSC ERROR: Configure options --with-scalar-type=real >> --with-hypre-dir=arch-linux2-c-opt-bullxmpi >> --with-metis-dir=arch-linux2-c-opt-bullxmpi >> --with-mumps-dir=arch-linux2-c-opt-bullxmpi --with-mkl_cpardiso=1 >> --with-mkl_pardiso=1 --with-parmetis-dir=arch-linux2-c-opt-bullxmpi >> --with-ptscotch-dir=arch-linux2-c-opt-bullxmpi >> --with-sowing-dir=arch-linux2-c-opt-bullxmpi >> --with-blacs-include=/ccc/products/mkl-18.0.1.163/default/ >> 18.0.1.163/mkl/include >> --with-blacs-lib=/ccc/products/mkl-18.0.1.163/default/ >> 18.0.1.163/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.so >> --with-blaslapack-dir=/ccc/products/mkl-18.0.1.163/default/ >> 18.0.1.163/mkl/lib/intel64 --with-x=0 --with-debugging=1 >> --with-errorchecking=1 >> --with-mkl_pardiso-dir=/ccc/products/mkl-18.0.1.163/default/ >> 18.0.1.163/mkl >> --with-mkl_cpardiso-dir=/ccc/products/mkl-18.0.1.163/default/ >> 18.0.1.163/mkl >> --with-scalapack-include=/ccc/products/mkl-18.0.1.163/default/ >> 18.0.1.163/mkl/include >> --with-scalapack-lib="[/ccc/products/mkl-18.0.1.163/default/ >> 18.0.1.163/mkl/lib/intel64/libmkl_scalapack_lp64.so,/ccc/products/mkl-18.0.1.163/default/18.0.1.163/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.so]" >> --with-fortran-bindings=0 PETSC_ARCH=arch-linux2-c-debug-bullxmpi >> [406]PETSC ERROR: #1 PCGAMGHashTableFind() line 106 in >> /ccc/work/cont003/rndm/rndm/petsc/include/petsc/private/pcgamgimpl.h >> [406]PETSC ERROR: #2 smoothAggs() line 526 in >> /ccc/work/cont003/rndm/rndm/petsc/src/ksp/pc/impls/gamg/agg.c >> [406]PETSC ERROR: #3 PCGAMGCoarsen_AGG() line 970 in >> /ccc/work/cont003/rndm/rndm/petsc/src/ksp/pc/impls/gamg/agg.c >> [406]PETSC ERROR: #4 PCSetUp_GAMG() line 518 in >> /ccc/work/cont003/rndm/rndm/petsc/src/ksp/pc/impls/gamg/gamg.c >> [406]PETSC ERROR: #5 PCSetUp() line 932 in >> /ccc/work/cont003/rndm/rndm/petsc/src/ksp/pc/interface/precon.c >> [406]PETSC ERROR: #6 KSPSetUp() line 381 in >> /ccc/work/cont003/rndm/rndm/petsc/src/ksp/ksp/interface/itfunc.c >> >> For smaller problems, this works fine. Should I compile using 64 bit >> indices? >> I’m using -pc_type gamg -pc_gamg_threshold 0.8 -pc_gamg_agg_nsmooths 1 >> -pc_gamg_sym_graph true -pc_gamg_square_graph 10 >> >> Thanks, >> Pierre > > >
