Fabian,
I have fixed this in the branch barry/fix-gamg-preallocation-block and
next. It worked for the matrix you provided after my fix.
Please let me know if you have any additional problems
Barry
On Jul 8, 2014, at 10:51 AM, Fabian.Jakub <[email protected]>
wrote:
> Thank you very much for the swift reply!
>
> I am running
> git describe: v3.5-25-g0ace994
> and configured with
>
> ./configure \
> --with-cc=$(which mpicc) \
> --with-fc=$(which mpif90) \
> --with-cxx=$(which mpic++) \
> --with-fortran \
> --with-shared-libraries \
> --with-hdf5 \
> --with-hdf5-dir="$HDF5_DIR" \
> --with-cmake=$(which cmake) \
> \
>
> I run the model with:
> mpirun -np 2 -map-by socket bin/petsc_solver -pc_type gamg
> -pc_gamg_agg_nsmooths 1 -pc_gamg_verbose 1 -info
>
> Should I append the output when creating the DMDA?
> For the KSP setup it says:
>
> [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter
> [0] VecScatterCreate(): General case: MPI to Seq
> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 6359040 X 131520; storage space: 0
> unneeded,420864 used
> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 8
> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 6227520)/(num_localrows 6359040) > 0.6. Use CompressedRow routines.
> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 6359040 X 131520; storage space: 0
> unneeded,420864 used
> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 8
> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 6227520)/(num_localrows 6359040) > 0.6. Use CompressedRow routines.
> [1] PetscCommDuplicate(): Using internal PETSc communicator 47323925895584
> 15967792
> Setup KSP
> [0] PetscCommDuplicate(): Using internal PETSc communicator 47292596238752
> 42057616
> [0] PCSetUp(): Setting up PC for first time [0]PCSetUp_GAMG level 0
> N=12718080, n data rows=10, n data cols=10, nnz/row (ave)=10, np=2
> [1] PetscCommDuplicate(): Using internal PETSc communicator 47323925894432
> 16179040
> [0] PetscCommDuplicate(): Using internal PETSc communicator 47292596237600
> 41749680
> [1] PetscCommDuplicate(): Using internal PETSc communicator 47323925894432
> 16179040
> [0] PetscCommDuplicate(): Using internal PETSc communicator 47292596237600
> 41749680
> [1]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [1]PETSC ERROR: Argument out of range
> [1]PETSC ERROR: New nonzero at (4704,4609) caused a malloc
> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off
> this check
> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
> trouble shooting.
> [1]PETSC ERROR: Petsc Development GIT revision: v3.5-25-g0ace994 GIT Date:
> 2014-07-07 14:50:15 +0200
> [1]PETSC ERROR: bin/petsc_solver on a debug named lx001 by jakub Tue Jul 8
> 17:47:31 2014
> [1]PETSC ERROR: Configure options
> --with-cc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpicc
> --with-fc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpif90
> --with-cxx=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpic++
> --with-fortran --with-shared-libraries --with-hdf5
> --with-hdf5-dir=/home/opt/cosmo_tica_lib//ompi1.8.1/hdf5/HDF_Group/HDF5/1.8.13/
> --with-cmake=/usr/bin/cmake
> [1]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 530 in
> /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/impls/aij/mpi/mpiaij.c
> [1]PETSC ERROR: #2 MatSetValues() line 1136 in
> /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/interface/matrix.c
> [1]PETSC ERROR: #3 PCGAMGCreateGraph() line 72 in
> /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/tools.c
> [1]PETSC ERROR: #4 PCGAMGgraph_AGG() line 936 in
> /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/agg.c
> [1]PETSC ERROR: #5 PCSetUp_GAMG() line 595 in
> /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/gamg.c
> [1]PETSC ERROR: #6 PCSetUp() line 902 in
> /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/interface/precon.c
> [1]PETSC ERROR: #7 KSPSetUp() line 305 in
> /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Argument out of range
> [0]PETSC ERROR: New nonzero at (4704,4609) caused a malloc
> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off
> this check
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
> trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision: v3.5-25-g0ace994 GIT Date:
> 2014-07-07 14:50:15 +0200
> [0]PETSC ERROR: bin/petsc_solver on a debug named lx001 by jakub Tue Jul 8
> 17:47:31 2014
> [0]PETSC ERROR: Configure options
> --with-cc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpicc
> --with-fc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpif90
> --with-cxx=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpic++
> --with-fortran --with-shared-libraries --with-hdf5
> --with-hdf5-dir=/home/opt/cosmo_tica_lib//ompi1.8.1/hdf5/HDF_Group/HDF5/1.8.13/
> --with-cmake=/usr/bin/cmake
> [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 530 in
> /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/impls/aij/mpi/mpiaij.c
> [0]PETSC ERROR: #2 MatSetValues() line 1136 in
> /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/interface/matrix.c
> [0]PETSC ERROR: #3 PCGAMGCreateGraph() line 72 in
> /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/tools.c
> [0]PETSC ERROR: #4 PCGAMGgraph_AGG() line 936 in
> /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/agg.c
> [0]PETSC ERROR: #5 PCSetUp_GAMG() line 595 in
> /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/gamg.c
> [0]PETSC ERROR: #6 PCSetUp() line 902 in
> /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: #7 KSPSetUp() line 305 in
> /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/ksp/interface/itfunc.c
> --------------------------------------------------------------------------
> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
> with errorcode 63.
>
> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> You may or may not see output from other processes, depending on
> exactly when Open MPI kills them.
> -------------------------------------------------------------------------
>
> Am 08.07.2014 17:14, schrieb Barry Smith:
>> GAMG should not give a preallocation error regardless of the
>> preallocation of the original matrix; so presumably you’ve hit on an error
>> in GAMG. What version of PETSc are you using and what is the ENTIRE error
>> message from GAMG? If you are using PETSc 3.4 please upgrade to 3.5 and see
>> if GAMG still produces an error.
>>
>> Barry
>>
>>
>>
>> On Jul 8, 2014, at 7:39 AM, Fabian.Jakub
>> <[email protected]> wrote:
>>
>>> Hi,
>>> I have a question regarding gamg where I get a wrong preallocation i.e. a
>>> MAT_NEW_NONZERO_ALLOCATION_ERR **.
>>>
>>> I use a 3d DMDA with 10 dof but the coupling only ever needs 2 dof on a
>>> star stencil.
>>>
>>> The code to setup the matrix is like this:
>>>
>>> call DMSetMatrixPreallocateOnly(C%da, PETSC_TRUE,ierr)
>>> call DMCreateMatrix(C%da, A, ierr)
>>> call MatSetFromOptions(A,ierr)
>>>
>>> call MatMPIAIJSetPreallocation(A, PETSC_NULL_INTEGER,d_nnz,
>>> PETSC_NULL_INTEGER, o_nnz, ierr)
>>>
>>> insert matrix values & assemble
>>>
>>> then solve.
>>>
>>> If I solve the system with any ''normal'' KSP/PC there is no problem and
>>> matrix info actually confirms that the preallocation is good.
>>> However if I use gamg I get an allocation error when it tries to create a
>>> coarse grid.
>>>
>>> If I use the DMDA preallocation, it works but uses way more memory...
>>>
>>> Is there a possibility to use custom preallocation and at the same time let
>>> gamg create the coarse grid?
>>> Is that even the problem or am I missing something?
>>>
>>> Thank you so very much.
>>>
>>> Sincerely,
>>> Fabian
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>
