Hong, Yes this looks like a regression in these MatMatMult routines, they now require unscalable memory
> On Jan 11, 2017, at 2:15 PM, Satish Balay <[email protected]> wrote: > >>>> > aN=A->cmap->N > ierr = PetscTableCreate(aN,aN,&ta);CHKERRQ(ierr); > <<< > > Looks like thats a 'global' size. That doesn't look right. > > https://bitbucket.org/petsc/petsc/commits/c373ccc6#chg-src/mat/impls/aij/mpi/mpimatmatmult.c > > I see Hong had previously updated this code. Will have to recheck.. > > Satish > > On Wed, 11 Jan 2017, Mark Adams wrote: > >>> >>> >>> Must be the same place Fande encountered this error. >>> >>> >> Satish, here is our stack trace. Do you have any thoughts vis-a-vis Fande's >> experience what the problem is here? My best guess is that the local >> matrix size is huge, but this code can have some very large stencils. >> >> Treb: about how many processors have this error? Is is it just one, a few, >> or tons? >> >> [5]PETSC ERROR: Argument out of range >> [5]PETSC ERROR: A really huge hash is being requested.. cannot process: >> 67108864 >> [5]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for >> trouble shooting. >> [5]PETSC ERROR: Petsc Development GIT revision: v3.7.4-2584-g7175242 GIT >> Date: 2016-12-21 13:19:16 -0600 >> [5]PETSC ERROR: /tmp/targetExec on a arch-xc30-dbg64-intel named nid00013 >> by u6338 Sat Jan 7 18:35:34 2017 >> [5]PETSC ERROR: Configure options --COPTFLAGS="-O0 -g -no-ipo" >> --CXXOPTFLAGS="-O0 -g -no-ipo" --FOPTFLAGS="-O0 -g -ipo" >> --download-parmetis --download-metis --download-hypre --with-ssl=0 >> --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC >> --with-cxxlib-autodetect=0 --with-debugging=1 --with-fc=ftn >> --with-fortranlib-autodetect=0 --with-shared-libraries=0 --with-x=0 >> --with-mpiexec=srun LIBS=-lstdc++ --with-64-bit-indices >> --prefix=/global/homes/m/madams/petsc_install/petscv3.7-xc3[1983]PETSC >> ERROR: --------------------- Error Message ------------------------------ >> -------------------------------- >> >> [5]PETSC ERROR: [522]PETSC ERROR: --------------------- Error Message >> ---------- >> >> [5]PETSC ERROR: #2 PetscTableCreate() line 52 in >> /global/u2/m/madams/petsc_install/petsc/src/sys/utils/ctable.c >> [5]PETSC ERROR: #3 MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ() line 1691 in >> /global/u2/m/madams/petsc_install/petsc/src/mat/impls/aij/ >> mpi/mpimatmatmult.c >> [5]PETSC ERROR: #4 MatTransposeMatMult_MPIAIJ_MPIAIJ() line 917 in >> /global/u2/m/madams/petsc_install/petsc/src/mat/impls/aij/ >> mpi/mpimatmatmult.c >> [5]PETSC ERROR: [608]PETSC ERROR: --------------------- Error Message >> ---------- >> >> [5]PETSC ERROR: #6 PCGAMGCoarsen_AGG() line 955 in >> /global/u2/m/madams/petsc_install/petsc/src/ksp/pc/impls/gamg/agg.c >> [5]PETSC ERROR: #7 PCSetUp_GAMG() line 524 in /global/u2/m/madams/petsc_inst >> all/petsc/src/ksp/pc/impls/gamg/gamg.c >> [5]PETSC ERROR: #8 PCSetUp() line 968 in /global/u2/m/madams/petsc_inst >> all/petsc/src/ksp/pc/interface/precon.c >> [5]PETSC ERROR: #9 KSPSetUp() line 393 in /global/u2/m/madams/petsc_inst >> all/petsc/src/ksp/ksp/interface/itfunc.c >> [5]PETSC ERROR: [1640]PETSC ERROR: --------------------- Error Message ----- >> >> >>> MatSetUpMultiply_MPIAIJ() does: >>> >>> ierr = PetscTableCreate(aij->B->rmap->n,mat->cmap->N+1,&gid1_lid1); >>> CHKERRQ(ierr); >>> >>> So the local matrix size (n) must be: 4294967296 >>> >>> [assuming atleast one nonzero per row - the local matrix itself must use >>> more memory than the hash-table..] >>> >>> Satish >>> >> >
