You might also test with Jacobi as a sanity check. On Oct 16, 2013, at 11:19 AM, Barry Smith <[email protected]> wrote:
> > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > especially the GAMG version which you also try in the debugger with the > argument -start_in_debugger > > > > On Oct 16, 2013, at 9:32 AM, Dave May <[email protected]> wrote: > >> Sounds like a memory error. >> I'd run your code through valgrind to double check. The error could be >> completely unconnected to the nullspaces. >> >> Cheers, >> Dave >> >> >> On 16 October 2013 16:18, Bishesh Khanal <[email protected]> wrote: >> Dear all, >> I'm trying to solve a stokes flow with constant viscosity but with non-zero >> divergence prescribed in the rhs. >> >> I have a matrix created from DMDA (mDa) of 4 dofs: vx, vy, vz and p >> respectively. >> I have another DMDA (mDaP) of same size but of 1 dof corresponding to only p. >> I have assigned the null space for constant pressure inside the code. I have >> assigned two nullspace basis: One corresponding to vector created from mDa >> that is assigned to outer ksp. Second corresponding to vector created from >> mDaP that is assigned to a ksp obtained from the fieldsplit corresponding to >> the schur complement. >> >> Now when running the code, the solver converges for up to certain size, e.g. >> 92X110 X 92 (the results for this convegent case with -ksp_view is given at >> the end of the emal. >> But when I double the size of the grid in each dimension, it gives me a >> run-time error. >> >> The options I've used are of the kind: >> -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_dm_splits 0 >> -pc_fieldsplit_0_fields 0,1,2 -pc_fieldsplit_1_fields 3 >> -fieldsplit_0_pc_type hypre -fieldsplit_0_ksp_converged_reason >> -fieldsplit_1_ksp_converged_reason -ksp_converged_reason -ksp_view >> >> Here are: >> 1. Error message when using hypre for fieldsplit_0 >> 2. Error message when using gamg for fieldsplit_0 >> 3. -ksp_view of the working case using hypre for filedsplit_0 >> >> I get following error when I use hypre : >> 1. >> ****************************************************************************************************** >> [5]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [5]PETSC ERROR: Signal received! >> [5]PETSC ERROR: >> ------------------------------------------------------------------------ >> [5]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013 >> [5]PETSC ERROR: See docs/changes/index.html for recent updates. >> [5]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [5]PETSC ERROR: See docs/index.html for manual pages. >> [5]PETSC ERROR: >> ------------------------------------------------------------------------ >> [5]PETSC ERROR: /epi/asclepios2/bkhanal/works/AdLemModel/build/src/AdLemMain >> on a arch-linux2-cxx-debug named nef001 by bkhanal Wed Oct 16 15:08:42 2013 >> [5]PETSC ERROR: Libraries linked from /epi/asclepios2/bkhanal/petscDebug/lib >> [5]PETSC ERROR: Configure run at Wed Oct 16 14:18:48 2013 >> [5]PETSC ERROR: Configure options --with-mpi-dir=/opt/openmpi-gcc/current/ >> --with-shared-libraries --prefix=/epi/asclepios2/bkhanal/petscDebug >> -download-f-blas-lapack=1 --download-metis --download-parmetis >> --download-superlu_dist --download-scalapack --download-mumps >> --download-hypre --with-clanguage=cxx >> [5]PETSC ERROR: >> ------------------------------------------------------------------------ >> [5]PETSC ERROR: User provided function() line 0 in unknown directory unknown >> file >> [6]PETSC ERROR: >> ------------------------------------------------------------------------ >> [6]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the >> batch system) has told this process to end >> [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> [6]PETSC ERROR: or see >> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[6]PETSC ERROR: >> or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory >> corruption errors >> [6]PETSC ERROR: likely location of problem given in stack below >> [6]PETSC ERROR: --------------------- Stack Frames >> ------------------------------------ >> [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >> [6]PETSC ERROR: INSTEAD the line number of the start of the function >> [6]PETSC ERROR: is given. >> [6]PETSC ERROR: [6] HYPRE_SetupXXX line 130 >> /tmp/petsc-3.4.3/src/ksp/pc/impls/hypre/hypre.c >> [6]PETSC ERROR: [6] PCSetUp_HYPRE line 94 >> /tmp/petsc-3.4.3/src/ksp/pc/impls/hypre/hypre.c >> [6]PETSC ERROR: [6] PCSetUp line 868 >> /tmp/petsc-3.4.3/src/ksp/pc/interface/precon.c >> [6]PETSC ERROR: [6] KSPSetUp line 192 >> /tmp/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c >> [6]PETSC ERROR: [6] KSPSolve line 356 >> /tmp/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c >> [6]PETSC ERROR: [6] MatMult_SchurComplement line 75 >> /tmp/petsc-3.4.3/src/ksp/ksp/utils/schurm.c >> [6]PETSC ERROR: [6] MatNullSpaceTest line 408 >> /tmp/petsc-3.4.3/src/mat/interface/matnull.c >> [6]PETSC ERROR: [6] solveModel line 113 >> "unknowndirectory/"/epi/asclepios2/bkhanal/works/AdLemModel/src/PetscAdLemTaras3D.cxx >> >> >> 2. >> **************************************************************************************************** >> Using gamg instead has errors like following: >> >> [5]PETSC ERROR: --------------------- Stack Frames >> ------------------------------------ >> [5]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >> [5]PETSC ERROR: INSTEAD the line number of the start of the function >> [5]PETSC ERROR: is given. >> [5]PETSC ERROR: [5] PetscLLCondensedAddSorted line 1202 >> /tmp/petsc-3.4.3/include/petsc-private/matimpl.h >> [5]PETSC ERROR: [5] MatPtAPSymbolic_MPIAIJ_MPIAIJ line 124 >> /tmp/petsc-3.4.3/src/mat/impls/aij/mpi/mpiptap.c >> [5]PETSC ERROR: [5] MatPtAP_MPIAIJ_MPIAIJ line 80 >> /tmp/petsc-3.4.3/src/mat/impls/aij/mpi/mpiptap.c >> [5]PETSC ERROR: [5] MatPtAP line 8223 >> /tmp/petsc-3.4.3/src/mat/interface/matrix.c >> [5]PETSC ERROR: [5] createLevel line 144 >> /tmp/petsc-3.4.3/src/ksp/pc/impls/gamg/gamg.c >> [5]PETSC ERROR: [5] PCSetUp_GAMG line 545 >> /tmp/petsc-3.4.3/src/ksp/pc/impls/gamg/gamg.c >> [5]PETSC ERROR: [5] PCSetUp line 868 >> /tmp/petsc-3.4.3/src/ksp/pc/interface/precon.c >> [5]PETSC ERROR: [5] KSPSetUp line 192 >> /tmp/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c >> [5]PETSC ERROR: [5] KSPSolve line 356 >> /tmp/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c >> [5]PETSC ERROR: [5] MatMult_SchurComplement line 75 >> /tmp/petsc-3.4.3/src/ksp/ksp/utils/schurm.c >> [5]PETSC ERROR: [5] MatNullSpaceTest line 408 >> /tmp/petsc-3.4.3/src/mat/interface/matnull.c >> [5]PETSC ERROR: [5] solveModel line 113 >> "unknowndirectory/"/epi/asclepios2/bkhanal/works/AdLemModel/src/PetscAdLemTaras3D.cxx >> >> >> 3. >> ******************************************************************************************************** >> >> BUT, It does give me results when I use a domain of size: 91X109 X 91 (half >> sized in each dimension) The result along with ksp view in this case is as >> follows: >> >> Linear solve converged due to CONVERGED_RTOL iterations 2 >> KSP Object: 64 MPI processes >> type: gmres >> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt >> Orthogonalization with no iterative refinement >> GMRES: happy breakdown tolerance 1e-30 >> maximum iterations=10000, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> has attached null space >> using PRECONDITIONED norm type for convergence test >> PC Object: 64 MPI processes >> type: fieldsplit >> FieldSplit with Schur preconditioner, blocksize = 4, factorization FULL >> Preconditioner for the Schur complement formed from user provided matrix >> Split info: >> Split number 0 Fields 0, 1, 2 >> Split number 1 Fields 3 >> KSP solver for A00 block >> KSP Object: (fieldsplit_0_) 64 MPI processes >> type: gmres >> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt >> Orthogonalization with no iterative refinement >> GMRES: happy breakdown tolerance 1e-30 >> maximum iterations=10000, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> using PRECONDITIONED norm type for convergence test >> PC Object: (fieldsplit_0_) 64 MPI processes >> type: hypre >> HYPRE BoomerAMG preconditioning >> HYPRE BoomerAMG: Cycle type V >> HYPRE BoomerAMG: Maximum number of levels 25 >> HYPRE BoomerAMG: Maximum number of iterations PER hypre call 1 >> HYPRE BoomerAMG: Convergence tolerance PER hypre call 0 >> HYPRE BoomerAMG: Threshold for strong coupling 0.25 >> HYPRE BoomerAMG: Interpolation truncation factor 0 >> HYPRE BoomerAMG: Interpolation: max elements per row 0 >> HYPRE BoomerAMG: Number of levels of aggressive coarsening 0 >> HYPRE BoomerAMG: Number of paths for aggressive coarsening 1 >> HYPRE BoomerAMG: Maximum row sums 0.9 >> HYPRE BoomerAMG: Sweeps down 1 >> HYPRE BoomerAMG: Sweeps up 1 >> HYPRE BoomerAMG: Sweeps on coarse 1 >> HYPRE BoomerAMG: Relax down symmetric-SOR/Jacobi >> HYPRE BoomerAMG: Relax up symmetric-SOR/Jacobi >> HYPRE BoomerAMG: Relax on coarse Gaussian-elimination >> HYPRE BoomerAMG: Relax weight (all) 1 >> HYPRE BoomerAMG: Outer relax weight (all) 1 >> HYPRE BoomerAMG: Using CF-relaxation >> HYPRE BoomerAMG: Measure type local >> HYPRE BoomerAMG: Coarsen type Falgout >> HYPRE BoomerAMG: Interpolation type classical >> linear system matrix = precond matrix: >> Matrix Object: 64 MPI processes >> type: mpiaij >> rows=2793120, cols=2793120 >> total: nonzeros=221624352, allocated nonzeros=221624352 >> total number of mallocs used during MatSetValues calls =0 >> using I-node (on process 0) routines: found 14812 nodes, limit >> used is 5 >> KSP solver for S = A11 - A10 inv(A00) A01 >> KSP Object: (fieldsplit_1_) 64 MPI processes >> type: gmres >> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt >> Orthogonalization with no iterative refinement >> GMRES: happy breakdown tolerance 1e-30 >> maximum iterations=10000, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> has attached null space >> using PRECONDITIONED norm type for convergence test >> PC Object: (fieldsplit_1_) 64 MPI processes >> type: bjacobi >> block Jacobi: number of blocks = 64 >> Local solve is same for all blocks, in the following KSP and PC >> objects: >> KSP Object: (fieldsplit_1_sub_) 1 MPI processes >> type: preonly >> maximum iterations=10000, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> using NONE norm type for convergence test >> PC Object: (fieldsplit_1_sub_) 1 MPI processes >> type: ilu >> ILU: out-of-place factorization >> 0 levels of fill >> tolerance for zero pivot 2.22045e-14 >> using diagonal shift on blocks to prevent zero pivot [INBLOCKS] >> matrix ordering: natural >> factor fill ratio given 1, needed 1 >> Factored matrix follows: >> Matrix Object: 1 MPI processes >> type: seqaij >> rows=14812, cols=14812 >> package used to perform factorization: petsc >> total: nonzeros=368098, allocated nonzeros=368098 >> total number of mallocs used during MatSetValues calls =0 >> not using I-node routines >> linear system matrix = precond matrix: >> Matrix Object: 1 MPI processes >> type: seqaij >> rows=14812, cols=14812 >> total: nonzeros=368098, allocated nonzeros=368098 >> total number of mallocs used during MatSetValues calls =0 >> not using I-node routines >> >> linear system matrix followed by preconditioner matrix: >> Matrix Object: 64 MPI processes >> type: schurcomplement >> rows=931040, cols=931040 >> Schur complement A11 - A10 inv(A00) A01 >> A11 >> Matrix Object: 64 MPI processes >> type: mpiaij >> rows=931040, cols=931040 >> total: nonzeros=24624928, allocated nonzeros=24624928 >> total number of mallocs used during MatSetValues calls =0 >> not using I-node (on process 0) routines >> A10 >> Matrix Object: 64 MPI processes >> type: mpiaij >> rows=931040, cols=2793120 >> total: nonzeros=73874784, allocated nonzeros=73874784 >> total number of mallocs used during MatSetValues calls =0 >> not using I-node (on process 0) routines >> KSP of A00 >> KSP Object: (fieldsplit_0_) 64 MPI >> processes >> type: gmres >> GMRES: restart=30, using Classical (unmodified) >> Gram-Schmidt Orthogonalization with no iterative refinement >> GMRES: happy breakdown tolerance 1e-30 >> maximum iterations=10000, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >> left preconditioning >> using PRECONDITIONED norm type for convergence test >> PC Object: (fieldsplit_0_) 64 MPI >> processes >> type: hypre >> HYPRE BoomerAMG preconditioning >> HYPRE BoomerAMG: Cycle type V >> HYPRE BoomerAMG: Maximum number of levels 25 >> HYPRE BoomerAMG: Maximum number of iterations PER hypre >> call 1 >> HYPRE BoomerAMG: Convergence tolerance PER hypre call 0 >> HYPRE BoomerAMG: Threshold for strong coupling 0.25 >> HYPRE BoomerAMG: Interpolation truncation factor 0 >> HYPRE BoomerAMG: Interpolation: max elements per row 0 >> HYPRE BoomerAMG: Number of levels of aggressive coarsening 0 >> HYPRE BoomerAMG: Number of paths for aggressive coarsening 1 >> HYPRE BoomerAMG: Maximum row sums 0.9 >> HYPRE BoomerAMG: Sweeps down 1 >> HYPRE BoomerAMG: Sweeps up 1 >> HYPRE BoomerAMG: Sweeps on coarse 1 >> HYPRE BoomerAMG: Relax down symmetric-SOR/Jacobi >> HYPRE BoomerAMG: Relax up symmetric-SOR/Jacobi >> HYPRE BoomerAMG: Relax on coarse Gaussian-elimination >> HYPRE BoomerAMG: Relax weight (all) 1 >> HYPRE BoomerAMG: Outer relax weight (all) 1 >> HYPRE BoomerAMG: Using CF-relaxation >> HYPRE BoomerAMG: Measure type local >> HYPRE BoomerAMG: Coarsen type Falgout >> HYPRE BoomerAMG: Interpolation type classical >> linear system matrix = precond matrix: >> Matrix Object: 64 MPI processes >> type: mpiaij >> rows=2793120, cols=2793120 >> total: nonzeros=221624352, allocated nonzeros=221624352 >> total number of mallocs used during MatSetValues calls =0 >> using I-node (on process 0) routines: found 14812 nodes, >> limit used is 5 >> A01 >> Matrix Object: 64 MPI processes >> type: mpiaij >> rows=2793120, cols=931040 >> total: nonzeros=73874784, allocated nonzeros=73874784 >> total number of mallocs used during MatSetValues calls =0 >> using I-node (on process 0) routines: found 14812 nodes, >> limit used is 5 >> Matrix Object: 64 MPI processes >> type: mpiaij >> rows=931040, cols=931040 >> total: nonzeros=24624928, allocated nonzeros=24624928 >> total number of mallocs used during MatSetValues calls =0 >> not using I-node (on process 0) routines >> linear system matrix = precond matrix: >> Matrix Object: 64 MPI processes >> type: mpiaij >> rows=3724160, cols=3724160, bs=4 >> total: nonzeros=393998848, allocated nonzeros=393998848 >> total number of mallocs used during MatSetValues calls =0 >> >> ****************************************************************************************************** >> What could be going wrong here ? Is it something related to null-space >> setting ? But I do not know why it does not arise for smaller domain sizes! >> >
