On Jul 9, 2014, at 6:06 AM, Fabian.Jakub <[email protected]>
wrote:
> If I do, the generated <binaryoutput.info> file is empty and the
> <binaryoutput> just states the solvers...
>
> cat binaryoutput:
> {Wgmres{Vilu
>
> Do I have an error in my Petscbuild?
No this was my mistake I told you the wrong directions
> This by the way happens with all matrices, even tried a couple of examples
> from petsc src.
>
> Anyway, I uploaded the matrix dump (-mat_view binary)
> I hope this helps.
> <http://homepages.physik.uni-muenchen.de/~Fabian.Jakub/>
Thanks. I have reproduced the problem with your matrix. I will work on
figuring out how to fix it. From the code
/* very pessimistic but could be too low in theory */ so likely you just
provided a matrix that Mark’s original algorithm could not handle.
Barry
>
> To be thorough, I further put up a second file, with 2 matrices,
> The second matrix is the same as before (the one that is not working with
> gamg).
> The first one with dof 8 is solved and the solution is somewhat the source
> term for the second.
> I just thought I would append this one too because it does indeed work with
> gamg.
> Maybe that gives any hints.
>
> I really appreciate you looking into this.
>
> Thank you!
>
> Sincerely,
> Fabian
>
>
>
> Am 09.07.2014 07:37, schrieb Barry Smith:
>> Can you run with -ksp_view_pre binary and email the file binaryoutput
>>
>> With this we should be able to reproduce the problem and debug it.
>>
>> Thanks
>>
>>
>> Barry
>>
>> On Jul 8, 2014, at 10:51 AM, Fabian.Jakub
>> <[email protected]> wrote:
>>
>>> Thank you very much for the swift reply!
>>>
>>> I am running
>>> git describe: v3.5-25-g0ace994
>>> and configured with
>>>
>>> ./configure \
>>> --with-cc=$(which mpicc) \
>>> --with-fc=$(which mpif90) \
>>> --with-cxx=$(which mpic++) \
>>> --with-fortran \
>>> --with-shared-libraries \
>>> --with-hdf5 \
>>> --with-hdf5-dir="$HDF5_DIR" \
>>> --with-cmake=$(which cmake) \
>>> \
>>>
>>> I run the model with:
>>> mpirun -np 2 -map-by socket bin/petsc_solver -pc_type gamg
>>> -pc_gamg_agg_nsmooths 1 -pc_gamg_verbose 1 -info
>>>
>>> Should I append the output when creating the DMDA?
>>> For the KSP setup it says:
>>>
>>> [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter
>>> [0] VecScatterCreate(): General case: MPI to Seq
>>> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 6359040 X 131520; storage space:
>>> 0 unneeded,420864 used
>>> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
>>> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 8
>>> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>> 6227520)/(num_localrows 6359040) > 0.6. Use CompressedRow routines.
>>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 6359040 X 131520; storage space:
>>> 0 unneeded,420864 used
>>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
>>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 8
>>> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>> 6227520)/(num_localrows 6359040) > 0.6. Use CompressedRow routines.
>>> [1] PetscCommDuplicate(): Using internal PETSc communicator 47323925895584
>>> 15967792
>>> Setup KSP
>>> [0] PetscCommDuplicate(): Using internal PETSc communicator 47292596238752
>>> 42057616
>>> [0] PCSetUp(): Setting up PC for first time [0]PCSetUp_GAMG level 0
>>> N=12718080, n data rows=10, n data cols=10, nnz/row (ave)=10, np=2
>>> [1] PetscCommDuplicate(): Using internal PETSc communicator 47323925894432
>>> 16179040
>>> [0] PetscCommDuplicate(): Using internal PETSc communicator 47292596237600
>>> 41749680
>>> [1] PetscCommDuplicate(): Using internal PETSc communicator 47323925894432
>>> 16179040
>>> [0] PetscCommDuplicate(): Using internal PETSc communicator 47292596237600
>>> 41749680
>>> [1]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [1]PETSC ERROR: Argument out of range
>>> [1]PETSC ERROR: New nonzero at (4704,4609) caused a malloc
>>> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn
>>> off this check
>>> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
>>> trouble shooting.
>>> [1]PETSC ERROR: Petsc Development GIT revision: v3.5-25-g0ace994 GIT Date:
>>> 2014-07-07 14:50:15 +0200
>>> [1]PETSC ERROR: bin/petsc_solver on a debug named lx001 by jakub Tue Jul 8
>>> 17:47:31 2014
>>> [1]PETSC ERROR: Configure options
>>> --with-cc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpicc
>>>
>>> --with-fc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpif90
>>>
>>> --with-cxx=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpic++
>>> --with-fortran --with-shared-libraries --with-hdf5
>>> --with-hdf5-dir=/home/opt/cosmo_tica_lib//ompi1.8.1/hdf5/HDF_Group/HDF5/1.8.13/
>>> --with-cmake=/usr/bin/cmake
>>> [1]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 530 in
>>> /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/impls/aij/mpi/mpiaij.c
>>> [1]PETSC ERROR: #2 MatSetValues() line 1136 in
>>> /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/interface/matrix.c
>>> [1]PETSC ERROR: #3 PCGAMGCreateGraph() line 72 in
>>> /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/tools.c
>>> [1]PETSC ERROR: #4 PCGAMGgraph_AGG() line 936 in
>>> /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/agg.c
>>> [1]PETSC ERROR: #5 PCSetUp_GAMG() line 595 in
>>> /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/gamg.c
>>> [1]PETSC ERROR: #6 PCSetUp() line 902 in
>>> /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/interface/precon.c
>>> [1]PETSC ERROR: #7 KSPSetUp() line 305 in
>>> /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/ksp/interface/itfunc.c
>>> [0]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [0]PETSC ERROR: Argument out of range
>>> [0]PETSC ERROR: New nonzero at (4704,4609) caused a malloc
>>> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn
>>> off this check
>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
>>> trouble shooting.
>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.5-25-g0ace994 GIT Date:
>>> 2014-07-07 14:50:15 +0200
>>> [0]PETSC ERROR: bin/petsc_solver on a debug named lx001 by jakub Tue Jul 8
>>> 17:47:31 2014
>>> [0]PETSC ERROR: Configure options
>>> --with-cc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpicc
>>>
>>> --with-fc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpif90
>>>
>>> --with-cxx=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpic++
>>> --with-fortran --with-shared-libraries --with-hdf5
>>> --with-hdf5-dir=/home/opt/cosmo_tica_lib//ompi1.8.1/hdf5/HDF_Group/HDF5/1.8.13/
>>> --with-cmake=/usr/bin/cmake
>>> [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 530 in
>>> /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/impls/aij/mpi/mpiaij.c
>>> [0]PETSC ERROR: #2 MatSetValues() line 1136 in
>>> /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/interface/matrix.c
>>> [0]PETSC ERROR: #3 PCGAMGCreateGraph() line 72 in
>>> /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/tools.c
>>> [0]PETSC ERROR: #4 PCGAMGgraph_AGG() line 936 in
>>> /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/agg.c
>>> [0]PETSC ERROR: #5 PCSetUp_GAMG() line 595 in
>>> /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/gamg.c
>>> [0]PETSC ERROR: #6 PCSetUp() line 902 in
>>> /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/interface/precon.c
>>> [0]PETSC ERROR: #7 KSPSetUp() line 305 in
>>> /home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/ksp/interface/itfunc.c
>>> --------------------------------------------------------------------------
>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
>>> with errorcode 63.
>>>
>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
>>> You may or may not see output from other processes, depending on
>>> exactly when Open MPI kills them.
>>> -------------------------------------------------------------------------
>>>
>>> Am 08.07.2014 17:14, schrieb Barry Smith:
>>>> GAMG should not give a preallocation error regardless of the
>>>> preallocation of the original matrix; so presumably you’ve hit on an error
>>>> in GAMG. What version of PETSc are you using and what is the ENTIRE error
>>>> message from GAMG? If you are using PETSc 3.4 please upgrade to 3.5 and
>>>> see if GAMG still produces an error.
>>>>
>>>> Barry
>>>>
>>>>
>>>>
>>>> On Jul 8, 2014, at 7:39 AM, Fabian.Jakub
>>>> <[email protected]> wrote:
>>>>
>>>>> Hi,
>>>>> I have a question regarding gamg where I get a wrong preallocation i.e. a
>>>>> MAT_NEW_NONZERO_ALLOCATION_ERR **.
>>>>>
>>>>> I use a 3d DMDA with 10 dof but the coupling only ever needs 2 dof on a
>>>>> star stencil.
>>>>>
>>>>> The code to setup the matrix is like this:
>>>>>
>>>>> call DMSetMatrixPreallocateOnly(C%da, PETSC_TRUE,ierr)
>>>>> call DMCreateMatrix(C%da, A, ierr)
>>>>> call MatSetFromOptions(A,ierr)
>>>>>
>>>>> call MatMPIAIJSetPreallocation(A, PETSC_NULL_INTEGER,d_nnz,
>>>>> PETSC_NULL_INTEGER, o_nnz, ierr)
>>>>>
>>>>> insert matrix values & assemble
>>>>>
>>>>> then solve.
>>>>>
>>>>> If I solve the system with any ''normal'' KSP/PC there is no problem and
>>>>> matrix info actually confirms that the preallocation is good.
>>>>> However if I use gamg I get an allocation error when it tries to create a
>>>>> coarse grid.
>>>>>
>>>>> If I use the DMDA preallocation, it works but uses way more memory...
>>>>>
>>>>> Is there a possibility to use custom preallocation and at the same time
>>>>> let gamg create the coarse grid?
>>>>> Is that even the problem or am I missing something?
>>>>>
>>>>> Thank you so very much.
>>>>>
>>>>> Sincerely,
>>>>> Fabian
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>
