This works. Thanks again for your help,
Fabian Am Freitag, den 11.07.2014, 14:10 -0500 schrieb Barry Smith: > Fabian, > > I have fixed this in the branch barry/fix-gamg-preallocation-block and > next. It worked for the matrix you provided after my fix. > > Please let me know if you have any additional problems > > Barry > > > On Jul 8, 2014, at 10:51 AM, Fabian.Jakub > <[email protected]> wrote: > > > Thank you very much for the swift reply! > > > > I am running > > git describe: v3.5-25-g0ace994 > > and configured with > > > > ./configure \ > > --with-cc=$(which mpicc) \ > > --with-fc=$(which mpif90) \ > > --with-cxx=$(which mpic++) \ > > --with-fortran \ > > --with-shared-libraries \ > > --with-hdf5 \ > > --with-hdf5-dir="$HDF5_DIR" \ > > --with-cmake=$(which cmake) \ > > \ > > > > I run the model with: > > mpirun -np 2 -map-by socket bin/petsc_solver -pc_type gamg > > -pc_gamg_agg_nsmooths 1 -pc_gamg_verbose 1 -info > > > > Should I append the output when creating the DMDA? > > For the KSP setup it says: > > > > [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter > > [0] VecScatterCreate(): General case: MPI to Seq > > [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 6359040 X 131520; storage space: > > 0 unneeded,420864 used > > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > > [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 8 > > [1] MatCheckCompressedRow(): Found the ratio (num_zerorows > > 6227520)/(num_localrows 6359040) > 0.6. Use CompressedRow routines. > > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 6359040 X 131520; storage space: > > 0 unneeded,420864 used > > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 8 > > [0] MatCheckCompressedRow(): Found the ratio (num_zerorows > > 6227520)/(num_localrows 6359040) > 0.6. Use CompressedRow routines. > > [1] PetscCommDuplicate(): Using internal PETSc communicator 47323925895584 > > 15967792 > > Setup KSP > > [0] PetscCommDuplicate(): Using internal PETSc communicator 47292596238752 > > 42057616 > > [0] PCSetUp(): Setting up PC for first time [0]PCSetUp_GAMG level 0 > > N=12718080, n data rows=10, n data cols=10, nnz/row (ave)=10, np=2 > > [1] PetscCommDuplicate(): Using internal PETSc communicator 47323925894432 > > 16179040 > > [0] PetscCommDuplicate(): Using internal PETSc communicator 47292596237600 > > 41749680 > > [1] PetscCommDuplicate(): Using internal PETSc communicator 47323925894432 > > 16179040 > > [0] PetscCommDuplicate(): Using internal PETSc communicator 47292596237600 > > 41749680 > > [1]PETSC ERROR: --------------------- Error Message > > -------------------------------------------------------------- > > [1]PETSC ERROR: Argument out of range > > [1]PETSC ERROR: New nonzero at (4704,4609) caused a malloc > > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn > > off this check > > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for > > trouble shooting. > > [1]PETSC ERROR: Petsc Development GIT revision: v3.5-25-g0ace994 GIT Date: > > 2014-07-07 14:50:15 +0200 > > [1]PETSC ERROR: bin/petsc_solver on a debug named lx001 by jakub Tue Jul 8 > > 17:47:31 2014 > > [1]PETSC ERROR: Configure options > > --with-cc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpicc > > > > --with-fc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpif90 > > > > --with-cxx=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpic++ > > --with-fortran --with-shared-libraries --with-hdf5 > > --with-hdf5-dir=/home/opt/cosmo_tica_lib//ompi1.8.1/hdf5/HDF_Group/HDF5/1.8.13/ > > --with-cmake=/usr/bin/cmake > > [1]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 530 in > > /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/impls/aij/mpi/mpiaij.c > > [1]PETSC ERROR: #2 MatSetValues() line 1136 in > > /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/interface/matrix.c > > [1]PETSC ERROR: #3 PCGAMGCreateGraph() line 72 in > > /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/tools.c > > [1]PETSC ERROR: #4 PCGAMGgraph_AGG() line 936 in > > /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/agg.c > > [1]PETSC ERROR: #5 PCSetUp_GAMG() line 595 in > > /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/gamg.c > > [1]PETSC ERROR: #6 PCSetUp() line 902 in > > /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/interface/precon.c > > [1]PETSC ERROR: #7 KSPSetUp() line 305 in > > /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: --------------------- Error Message > > -------------------------------------------------------------- > > [0]PETSC ERROR: Argument out of range > > [0]PETSC ERROR: New nonzero at (4704,4609) caused a malloc > > Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn > > off this check > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for > > trouble shooting. > > [0]PETSC ERROR: Petsc Development GIT revision: v3.5-25-g0ace994 GIT Date: > > 2014-07-07 14:50:15 +0200 > > [0]PETSC ERROR: bin/petsc_solver on a debug named lx001 by jakub Tue Jul 8 > > 17:47:31 2014 > > [0]PETSC ERROR: Configure options > > --with-cc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpicc > > > > --with-fc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpif90 > > > > --with-cxx=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpic++ > > --with-fortran --with-shared-libraries --with-hdf5 > > --with-hdf5-dir=/home/opt/cosmo_tica_lib//ompi1.8.1/hdf5/HDF_Group/HDF5/1.8.13/ > > --with-cmake=/usr/bin/cmake > > [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 530 in > > /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/impls/aij/mpi/mpiaij.c > > [0]PETSC ERROR: #2 MatSetValues() line 1136 in > > /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/interface/matrix.c > > [0]PETSC ERROR: #3 PCGAMGCreateGraph() line 72 in > > /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/tools.c > > [0]PETSC ERROR: #4 PCGAMGgraph_AGG() line 936 in > > /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/agg.c > > [0]PETSC ERROR: #5 PCSetUp_GAMG() line 595 in > > /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/gamg.c > > [0]PETSC ERROR: #6 PCSetUp() line 902 in > > /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: #7 KSPSetUp() line 305 in > > /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/ksp/interface/itfunc.c > > -------------------------------------------------------------------------- > > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD > > with errorcode 63. > > > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > > You may or may not see output from other processes, depending on > > exactly when Open MPI kills them. > > ------------------------------------------------------------------------- > > > > Am 08.07.2014 17:14, schrieb Barry Smith: > >> GAMG should not give a preallocation error regardless of the > >> preallocation of the original matrix; so presumably you’ve hit on an error > >> in GAMG. What version of PETSc are you using and what is the ENTIRE error > >> message from GAMG? If you are using PETSc 3.4 please upgrade to 3.5 and > >> see if GAMG still produces an error. > >> > >> Barry > >> > >> > >> > >> On Jul 8, 2014, at 7:39 AM, Fabian.Jakub > >> <[email protected]> wrote: > >> > >>> Hi, > >>> I have a question regarding gamg where I get a wrong preallocation i.e. a > >>> MAT_NEW_NONZERO_ALLOCATION_ERR **. > >>> > >>> I use a 3d DMDA with 10 dof but the coupling only ever needs 2 dof on a > >>> star stencil. > >>> > >>> The code to setup the matrix is like this: > >>> > >>> call DMSetMatrixPreallocateOnly(C%da, PETSC_TRUE,ierr) > >>> call DMCreateMatrix(C%da, A, ierr) > >>> call MatSetFromOptions(A,ierr) > >>> > >>> call MatMPIAIJSetPreallocation(A, PETSC_NULL_INTEGER,d_nnz, > >>> PETSC_NULL_INTEGER, o_nnz, ierr) > >>> > >>> insert matrix values & assemble > >>> > >>> then solve. > >>> > >>> If I solve the system with any ''normal'' KSP/PC there is no problem and > >>> matrix info actually confirms that the preallocation is good. > >>> However if I use gamg I get an allocation error when it tries to create a > >>> coarse grid. > >>> > >>> If I use the DMDA preallocation, it works but uses way more memory... > >>> > >>> Is there a possibility to use custom preallocation and at the same time > >>> let gamg create the coarse grid? > >>> Is that even the problem or am I missing something? > >>> > >>> Thank you so very much. > >>> > >>> Sincerely, > >>> Fabian > >>> > >>> > >>> > >>> > >>> > >>> > >>> > > >
