Finally got it. My failing memory. I had to add the line
call KSPSetDMActive(ksp,PETSC_FALSE,ierr)
immediately after KSPSetDM() and
change
call DMCreateMatrix(da,MATMPIAIJ,A,ierr)
to
call DMCreateMatrix(da,MATAIJ,A,ierr)
so it will work in both parallel and sequential then
ksp_monitor -ksp_converged_reason -pc_type mg -ksp_view -pc_mg_galerkin
-pc_mg_levels 2
works great with 2 levels.
Barry
On Aug 1, 2013, at 6:29 PM, Michele Rosso <mro...@uci.edu> wrote:
> Barry,
>
> no problem. I attached the full code in test_poisson_solver.tar.gz.
> My test code is a very reduced version of my productive code (incompressible
> DNS code) thus fftw3 and the library 2decomp&fft are needed to run it.
> I attached the 2decomp&fft version I used: it is a matter of minutes to
> install it, so you should not have any problem.
> Please, contact me for any question/suggestion.
> I the mean time I will try to debug it.
>
> Michele
>
>
>
>
> On 08/01/2013 04:19 PM, Barry Smith wrote:
>> Run on one process until this is debugged. You can try the option
>>
>> -start_in_debugger noxterm
>>
>> and then call VecView(vec,0) in the debugger when it gives the error below.
>> It seems like some objects are not getting their initial values set
>> properly. Are you able to email the code so we can run it and figure out
>> what is going on?
>>
>> Barry
>>
>> On Aug 1, 2013, at 5:52 PM, Michele Rosso
>> <mro...@uci.edu>
>> wrote:
>>
>>
>>> Barry,
>>>
>>> I checked the matrix: the element (0,0) is not zero, nor any other diagonal
>>> element is.
>>> The matrix is symmetric positive define (i.e. the standard Poisson matrix).
>>> Also, -da_refine is never used (see previous output).
>>> I tried to run with -pc_type mg -pc_mg_galerkin -mg_levels_pc_type jacobi
>>> -mg_levels_ksp_type chebyshev -mg_levels_ksp_chebyshev_estimate_eigenvalues
>>> -ksp_view -options_left
>>>
>>> and now the error is different:
>>> 0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message
>>> ------------------------------------
>>> [1]PETSC ERROR: Floating point exception!
>>> [1]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite
>>> at beginning of function: Parameter number 2!
>>> [1]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [1]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
>>> [1]PETSC ERROR: See docs/changes/index.html for recent updates.
>>> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>> [2]PETSC ERROR: --------------------- Error Message
>>> ------------------------------------
>>> [2]PETSC ERROR: Floating point exception!
>>> [2]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite
>>> at beginning of function: Parameter number 2!
>>> [2]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [2]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
>>> [2]PETSC ERROR: See docs/changes/index.html for recent updates.
>>> [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>> [2]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message
>>> ------------------------------------
>>> [3]PETSC ERROR: Floating point exception!
>>> [3]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite
>>> at beginning of function: Parameter number 2!
>>> [3]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [3]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
>>> [3]PETSC ERROR: See docs/changes/index.html for recent updates.
>>> [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>> [3]PETSC ERROR: See docs/index.html for manual pages.
>>> [3]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [1]PETSC ERROR: See docs/index.html for manual pages.
>>> [1]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [1]PETSC ERROR: ./test on a linux-gnu-dbg named enterprise-A by mic Thu Aug
>>> 1 15:43:16 2013
>>> [1]PETSC ERROR: Libraries linked from
>>> /opt/petsc/petsc-3.4.2/linux-gnu-dbg/lib
>>> [1]PETSC ERROR: See docs/index.html for manual pages.
>>> [2]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [2]PETSC ERROR: ./test on a linux-gnu-dbg named enterprise-A by mic Thu Aug
>>> 1 15:43:16 2013
>>> [2]PETSC ERROR: Libraries linked from
>>> /opt/petsc/petsc-3.4.2/linux-gnu-dbg/lib
>>> [2]PETSC ERROR: Configure run at Thu Aug 1 12:01:44 2013
>>> [2]PETSC ERROR: [3]PETSC ERROR: ./test on a linux-gnu-dbg named
>>> enterprise-A by mic Thu Aug 1 15:43:16 2013
>>> [3]PETSC ERROR: Libraries linked from
>>> /opt/petsc/petsc-3.4.2/linux-gnu-dbg/lib
>>> [3]PETSC ERROR: Configure run at Thu Aug 1 12:01:44 2013
>>> [3]PETSC ERROR: Configure options
>>> Configure run at Thu Aug 1 12:01:44 2013
>>> [1]PETSC ERROR: Configure options
>>> [1]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [1]PETSC ERROR: VecValidValues() line 28 in src/vec/vec/interface/rvector.c
>>> Configure options
>>> [2]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [2]PETSC ERROR: VecValidValues() line 28 in src/vec/vec/interface/rvector.c
>>> [3]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [3]PETSC ERROR: VecValidValues() line 28 in src/vec/vec/interface/rvector.c
>>> [3]PETSC ERROR: [1]PETSC ERROR: MatMult() line 2174 in
>>> src/mat/interface/matrix.c
>>> [1]PETSC ERROR: [2]PETSC ERROR: MatMult() line 2174 in
>>> src/mat/interface/matrix.c
>>> [2]PETSC ERROR: KSP_MatMult() line 204 in
>>> src/ksp/ksp/impls/cheby//opt/petsc/petsc-3.4.2/include/petsc-private/kspimpl.h
>>> MatMult() line 2174 in src/mat/interface/matrix.c
>>> [3]PETSC ERROR: KSP_MatMult() line 204 in
>>> src/ksp/ksp/impls/cheby//opt/petsc/petsc-3.4.2/include/petsc-private/kspimpl.h
>>> [3]PETSC ERROR: KSP_MatMult() line 204 in
>>> src/ksp/ksp/impls/cheby//opt/petsc/petsc-3.4.2/include/petsc-private/kspimpl.h
>>> [1]PETSC ERROR: KSPSolve_Chebyshev() line 504 in
>>> src/ksp/ksp/impls/cheby/cheby.c
>>> [2]PETSC ERROR: KSPSolve_Chebyshev() line 504 in
>>> src/ksp/ksp/impls/cheby/cheby.c
>>> [2]PETSC ERROR: KSPSolve_Chebyshev() line 504 in
>>> src/ksp/ksp/impls/cheby/cheby.c
>>> [3]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c
>>> [1]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c
>>> [1]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c
>>> [2]PETSC ERROR: PCMGMCycle_Private() line 19 in src/ksp/pc/impls/mg/mg.c
>>> [3]PETSC ERROR: PCMGMCycle_Private() line 19 in src/ksp/pc/impls/mg/mg.c
>>> PCMGMCycle_Private() line 19 in src/ksp/pc/impls/mg/mg.c
>>> [1]PETSC ERROR: PCApply_MG() line 330 in src/ksp/pc/impls/mg/mg.c
>>> [2]PETSC ERROR: PCApply_MG() line 330 in src/ksp/pc/impls/mg/mg.c
>>> [2]PETSC ERROR: PCApply() line 442 in src/ksp/pc/interface/precon.c
>>> [3]PETSC ERROR: PCApply_MG() line 330 in src/ksp/pc/impls/mg/mg.c
>>> [3]PETSC ERROR: PCApply() line 442 in src/ksp/pc/interface/precon.c
>>> [1]PETSC ERROR: PCApply() line 442 in src/ksp/pc/interface/precon.c
>>> [1]PETSC ERROR: KSP_PCApply() line 227 in
>>> src/ksp/ksp/impls/cg//opt/petsc/petsc-3.4.2/include/petsc-private/kspimpl.h
>>> [2]PETSC ERROR: KSP_PCApply() line 227 in
>>> src/ksp/ksp/impls/cg//opt/petsc/petsc-3.4.2/include/petsc-private/kspimpl.h
>>> [2]PETSC ERROR: [3]PETSC ERROR: KSP_PCApply() line 227 in
>>> src/ksp/ksp/impls/cg//opt/petsc/petsc-3.4.2/include/petsc-private/kspimpl.h
>>> [3]PETSC ERROR: KSPSolve_CG() line 175 in src/ksp/ksp/impls/cg/cg.c
>>> [1]PETSC ERROR: KSPSolve_CG() line 175 in src/ksp/ksp/impls/cg/cg.c
>>> [1]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c
>>> KSPSolve_CG() line 175 in src/ksp/ksp/impls/cg/cg.c
>>> [2]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c
>>> [3]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c
>>> --------------------- Error Message ------------------------------------
>>> [0]PETSC ERROR: Floating point exception!
>>> [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite
>>> at beginning of function: Parameter number 2!
>>> [0]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>> [0]PETSC ERROR: See docs/index.html for manual pages.
>>> [0]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [0]PETSC ERROR: ./test on a linux-gnu-dbg named enterprise-A by mic Thu Aug
>>> 1 15:43:16 2013
>>> [0]PETSC ERROR: Libraries linked from
>>> /opt/petsc/petsc-3.4.2/linux-gnu-dbg/lib
>>> [0]PETSC ERROR: Configure run at Thu Aug 1 12:01:44 2013
>>> [0]PETSC ERROR: Configure options
>>> [0]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [0]PETSC ERROR: VecValidValues() line 28 in src/vec/vec/interface/rvector.c
>>> [0]PETSC ERROR: MatMult() line 2174 in src/mat/interface/matrix.c
>>> [0]PETSC ERROR: KSP_MatMult() line 204 in
>>> src/ksp/ksp/impls/cheby//opt/petsc/petsc-3.4.2/include/petsc-private/kspimpl.h
>>> [0]PETSC ERROR: KSPSolve_Chebyshev() line 504 in
>>> src/ksp/ksp/impls/cheby/cheby.c
>>> [0]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c
>>> [0]PETSC ERROR: PCMGMCycle_Private() line 19 in src/ksp/pc/impls/mg/mg.c
>>> [0]PETSC ERROR: PCApply_MG() line 330 in src/ksp/pc/impls/mg/mg.c
>>> [0]PETSC ERROR: PCApply() line 442 in src/ksp/pc/interface/precon.c
>>> [0]PETSC ERROR: KSP_PCApply() line 227 in
>>> src/ksp/ksp/impls/cg//opt/petsc/petsc-3.4.2/include/petsc-private/kspimpl.h
>>> [0]PETSC ERROR: KSPSolve_CG() line 175 in src/ksp/ksp/impls/cg/cg.c
>>> [0]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c
>>>
>>> #PETSc Option Table entries:
>>> -ksp_view
>>> -mg_levels_ksp_chebyshev_estimate_eigenvalues
>>> -mg_levels_ksp_type chebyshev
>>> -mg_levels_pc_type jacobi
>>> -options_left
>>> -pc_mg_galerkin
>>> -pc_type mg
>>> #End of PETSc Option Table entries
>>> There are no unused options.
>>>
>>> Michele
>>>
>>>
>>> On 08/01/2013 03:27 PM, Barry Smith wrote:
>>>
>>>> Do a MatView() on A before the solve (remove the -da_refine 4) so it is
>>>> small. Is the 0,0 entry 0? If the matrix has zero on the diagonals you
>>>> cannot us Gauss-Seidel as the smoother. You can start with
>>>> -mg_levels_pc_type jacobi -mg_levels_ksp_type chebychev
>>>> -mg_levels_ksp_chebyshev_estimate_eigenvalues
>>>>
>>>> Is the matrix a Stokes-like matrix? If so then different
>>>> preconditioners are in order.
>>>>
>>>> Barry
>>>>
>>>> On Aug 1, 2013, at 5:21 PM, Michele Rosso
>>>>
>>>> <mro...@uci.edu>
>>>>
>>>> wrote:
>>>>
>>>>
>>>>
>>>>> Barry,
>>>>>
>>>>> here it is the fraction of code where I set the rhs term and the matrix.
>>>>>
>>>>> ! Create matrix
>>>>> call form_matrix( A , qrho, lsf, head )
>>>>> call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr)
>>>>> call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr)
>>>>>
>>>>> ! Create rhs term
>>>>> call form_rhs(work, qrho, lsf, b , head)
>>>>>
>>>>> ! Solve system
>>>>> call KSPSetFromOptions(ksp,ierr)
>>>>> call KSPSetUp(ksp,ierr)
>>>>> call KSPSolve(ksp,b,x,ierr)
>>>>> call KSPGetIterationNumber(ksp, iiter ,ierr)
>>>>>
>>>>> The subroutine form_matrix returns the Mat object A that is filled by
>>>>> using MatSetValuesStencil.
>>>>> qrho, lsf and head are additional arguments that are needed to compute
>>>>> the matrix value.
>>>>>
>>>>>
>>>>> Michele
>>>>>
>>>>>
>>>>>
>>>>> On 08/01/2013 03:11 PM, Barry Smith wrote:
>>>>>
>>>>>
>>>>>> Where are you putting the values into the matrix? It seems the matrix
>>>>>> has no values in it? The code is stopping because in the Gauss-Seidel
>>>>>> smoothing it has detected zero diagonals.
>>>>>>
>>>>>> Barry
>>>>>>
>>>>>>
>>>>>> On Aug 1, 2013, at 4:47 PM, Michele Rosso
>>>>>>
>>>>>> <mro...@uci.edu>
>>>>>>
>>>>>> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Barry,
>>>>>>>
>>>>>>> I run with : -pc_type mg -pc_mg_galerkin -da_refine 4 -ksp_view
>>>>>>> -options_left
>>>>>>>
>>>>>>> For the test I use a 64^3 grid and 4 processors.
>>>>>>>
>>>>>>> The output is:
>>>>>>>
>>>>>>> [2]PETSC ERROR: --------------------- Error Message
>>>>>>> ------------------------------------
>>>>>>> [2]PETSC ERROR: Arguments are incompatible!
>>>>>>> [2]PETSC ERROR: Zero diagonal on row 0!
>>>>>>> [2]PETSC ERROR:
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [2]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
>>>>>>> [2]PETSC ERROR: See docs/changes/index.html for recent updates.
>>>>>>> [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>>>>>> [2]PETSC ERROR: See docs/index.html for manual pages.
>>>>>>> [2]PETSC ERROR:
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [2]PETSC ERROR: ./test on a linux-gnu-dbg named enterprise-A by mic Thu
>>>>>>> Aug 1 14:44:04 2013
>>>>>>> [0]PETSC ERROR: [2]PETSC ERROR: Libraries linked from
>>>>>>> /opt/petsc/petsc-3.4.2/linux-gnu-dbg/lib
>>>>>>> [2]PETSC ERROR: Configure run at Thu Aug 1 12:01:44 2013
>>>>>>> [2]PETSC ERROR: Configure options
>>>>>>> [2]PETSC ERROR:
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [2]PETSC ERROR: MatInvertDiagonal_SeqAIJ() line 1457 in
>>>>>>> src/mat/impls/aij/seq/aij.c
>>>>>>> [2]PETSC ERROR: MatSOR_SeqAIJ() line 1489 in src/mat/impls/aij/seq/aij.c
>>>>>>> --------------------- Error Message ------------------------------------
>>>>>>> [2]PETSC ERROR: MatSOR_MPIAIJ() line 1623 in
>>>>>>> src/mat/impls/aij/mpi/mpiaij.c
>>>>>>> [2]PETSC ERROR: MatSOR() line 3649 in src/mat/interface/matrix.c
>>>>>>> [2]PETSC ERROR: [0]PETSC ERROR: PCApply_SOR() line 35 in
>>>>>>> src/ksp/pc/impls/sor/sor.c
>>>>>>> [2]PETSC ERROR: PCApply() line 442 in src/ksp/pc/interface/precon.c
>>>>>>> [2]PETSC ERROR: KSP_PCApply() line 227 in
>>>>>>> src/ksp/ksp/interface//opt/petsc/petsc-3.4.2/include/petsc-private/kspimpl.h
>>>>>>> Arguments are incompatible!
>>>>>>> [2]PETSC ERROR: KSPInitialResidual() line 64 in
>>>>>>> src/ksp/ksp/interface/itres.c
>>>>>>> [2]PETSC ERROR: KSPSolve_GMRES() line 239 in
>>>>>>> src/ksp/ksp/impls/gmres/gmres.c
>>>>>>> [2]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c
>>>>>>> [2]PETSC ERROR: [0]PETSC ERROR: KSPSolve_Chebyshev() line 409 in
>>>>>>> src/ksp/ksp/impls/cheby/cheby.c
>>>>>>> [2]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c
>>>>>>> [2]PETSC ERROR: PCMGMCycle_Private() line 19 in src/ksp/pc/impls/mg/mg.c
>>>>>>> Zero diagonal on row 0!
>>>>>>> [2]PETSC ERROR: PCApply_MG() line 330 in src/ksp/pc/impls/mg/mg.c
>>>>>>> [2]PETSC ERROR: PCApply() line 442 in src/ksp/pc/interface/precon.c
>>>>>>> [0]PETSC ERROR: [2]PETSC ERROR: KSP_PCApply() line 227 in
>>>>>>> src/ksp/ksp/impls/cg//opt/petsc/petsc-3.4.2/include/petsc-private/kspimpl.h
>>>>>>> [2]PETSC ERROR: KSPSolve_CG() line 175 in src/ksp/ksp/impls/cg/cg.c
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [2]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c
>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
>>>>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>>>>>> [3]PETSC ERROR: [0]PETSC ERROR: See docs/faq.html for hints about
>>>>>>> trouble shooting.
>>>>>>> [0]PETSC ERROR: --------------------- Error Message
>>>>>>> ------------------------------------
>>>>>>> [3]PETSC ERROR: Arguments are incompatible!
>>>>>>> [3]PETSC ERROR: Zero diagonal on row 0!
>>>>>>> [3]PETSC ERROR:
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [3]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
>>>>>>> [3]PETSC ERROR: See docs/changes/index.html for recent updates.
>>>>>>> [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>>>>>> [3]PETSC ERROR: See docs/index.html for manual pages.
>>>>>>> [3]PETSC ERROR:
>>>>>>> ------------------------------------------------------------------------
>>>>>>> See docs/index.html for manual pages.
>>>>>>> [3]PETSC ERROR: ./test on a linux-gnu-dbg named enterprise-A by mic Thu
>>>>>>> Aug 1 14:44:04 2013
>>>>>>> [3]PETSC ERROR: Libraries linked from
>>>>>>> /opt/petsc/petsc-3.4.2/linux-gnu-dbg/lib
>>>>>>> [1]PETSC ERROR: [3]PETSC ERROR: Configure run at Thu Aug 1 12:01:44
>>>>>>> 2013
>>>>>>> [3]PETSC ERROR: Configure options
>>>>>>> [3]PETSC ERROR:
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [3]PETSC ERROR: --------------------- Error Message
>>>>>>> ------------------------------------
>>>>>>> MatInvertDiagonal_SeqAIJ() line 1457 in src/mat/impls/aij/seq/aij.c
>>>>>>> [3]PETSC ERROR: MatSOR_SeqAIJ() line 1489 in src/mat/impls/aij/seq/aij.c
>>>>>>> [3]PETSC ERROR: [0]PETSC ERROR: MatSOR_MPIAIJ() line 1623 in
>>>>>>> src/mat/impls/aij/mpi/mpiaij.c
>>>>>>> [1]PETSC ERROR: Arguments are incompatible!
>>>>>>> [1]PETSC ERROR: Zero diagonal on row 0!
>>>>>>> [1]PETSC ERROR:
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [1]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
>>>>>>> [1]PETSC ERROR: See docs/changes/index.html for recent updates.
>>>>>>> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>>>>>> [1]PETSC ERROR: See docs/index.html for manual pages.
>>>>>>> [1]PETSC ERROR:
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [1]PETSC ERROR: ./test on a linux-gnu-dbg named enterprise-A by mic Thu
>>>>>>> Aug 1 14:44:04 2013
>>>>>>> [1]PETSC ERROR: Libraries linked from
>>>>>>> /opt/petsc/petsc-3.4.2/linux-gnu-dbg/lib
>>>>>>> [1]PETSC ERROR: Configure run at Thu Aug 1 12:01:44 2013
>>>>>>> [1]PETSC ERROR: Configure options
>>>>>>> [1]PETSC ERROR:
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [1]PETSC ERROR: MatInvertDiagonal_SeqAIJ() line 1457 in
>>>>>>> src/mat/impls/aij/seq/aij.c
>>>>>>> [1]PETSC ERROR: [3]PETSC ERROR: MatSOR() line 3649 in
>>>>>>> src/mat/interface/matrix.c
>>>>>>> [3]PETSC ERROR: PCApply_SOR() line 35 in src/ksp/pc/impls/sor/sor.c
>>>>>>> [3]PETSC ERROR: PCApply() line 442 in src/ksp/pc/interface/precon.c
>>>>>>> [3]PETSC ERROR: KSP_PCApply() line 227 in
>>>>>>> src/ksp/ksp/interface//opt/petsc/petsc-3.4.2/include/petsc-private/kspimpl.h
>>>>>>> [3]PETSC ERROR: KSPInitialResidual() line 64 in
>>>>>>> src/ksp/ksp/interface/itres.c
>>>>>>> [3]PETSC ERROR: KSPSolve_GMRES() line 239 in
>>>>>>> src/ksp/ksp/impls/gmres/gmres.c
>>>>>>> [3]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c
>>>>>>> [3]PETSC ERROR: KSPSolve_Chebyshev() line 409 in
>>>>>>> src/ksp/ksp/impls/cheby/cheby.c
>>>>>>> [3]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c
>>>>>>> [3]PETSC ERROR: PCMGMCycle_Private() line 19 in src/ksp/pc/impls/mg/mg.c
>>>>>>> [3]PETSC ERROR: PCApply_MG() line 330 in src/ksp/pc/impls/mg/mg.c
>>>>>>> [3]PETSC ERROR: PCApply() line 442 in src/ksp/pc/interface/precon.c
>>>>>>> [3]PETSC ERROR: KSP_PCApply() line 227 in
>>>>>>> src/ksp/ksp/impls/cg//opt/petsc/petsc-3.4.2/include/petsc-private/kspimpl.h
>>>>>>> [3]PETSC ERROR: KSPSolve_CG() line 175 in src/ksp/ksp/impls/cg/cg.c
>>>>>>> [3]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c
>>>>>>> MatSOR_SeqAIJ() line 1489 in src/mat/impls/aij/seq/aij.c
>>>>>>> [1]PETSC ERROR: MatSOR_MPIAIJ() line 1623 in
>>>>>>> src/mat/impls/aij/mpi/mpiaij.c
>>>>>>> [1]PETSC ERROR: MatSOR() line 3649 in src/mat/interface/matrix.c
>>>>>>> [1]PETSC ERROR: PCApply_SOR() line 35 in src/ksp/pc/impls/sor/sor.c
>>>>>>> [1]PETSC ERROR: PCApply() line 442 in src/ksp/pc/interface/precon.c
>>>>>>> [1]PETSC ERROR: KSP_PCApply() line 227 in
>>>>>>> src/ksp/ksp/interface//opt/petsc/petsc-3.4.2/include/petsc-private/kspimpl.h
>>>>>>> [1]PETSC ERROR: KSPInitialResidual() line 64 in
>>>>>>> src/ksp/ksp/interface/itres.c
>>>>>>> [1]PETSC ERROR: KSPSolve_GMRES() line 239 in
>>>>>>> src/ksp/ksp/impls/gmres/gmres.c
>>>>>>> [1]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c
>>>>>>> [1]PETSC ERROR: KSPSolve_Chebyshev() line 409 in
>>>>>>> src/ksp/ksp/impls/cheby/cheby.c
>>>>>>> [1]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c
>>>>>>> [1]PETSC ERROR: PCMGMCycle_Private() line 19 in src/ksp/pc/impls/mg/mg.c
>>>>>>> [1]PETSC ERROR: PCApply_MG() line 330 in src/ksp/pc/impls/mg/mg.c
>>>>>>> [1]PETSC ERROR: PCApply() line 442 in src/ksp/pc/interface/precon.c
>>>>>>> [1]PETSC ERROR: KSP_PCApply() line 227 in
>>>>>>> src/ksp/ksp/impls/cg//opt/petsc/petsc-3.4.2/include/petsc-private/kspimpl.h
>>>>>>> [1]PETSC ERROR: KSPSolve_CG() line 175 in src/ksp/ksp/impls/cg/cg.c
>>>>>>> [1]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [0]PETSC ERROR: ./test on a linux-gnu-dbg named enterprise-A by mic Thu
>>>>>>> Aug 1 14:44:04 2013
>>>>>>> [0]PETSC ERROR: Libraries linked from
>>>>>>> /opt/petsc/petsc-3.4.2/linux-gnu-dbg/lib
>>>>>>> [0]PETSC ERROR: Configure run at Thu Aug 1 12:01:44 2013
>>>>>>> [0]PETSC ERROR: Configure options
>>>>>>> [0]PETSC ERROR:
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [0]PETSC ERROR: MatInvertDiagonal_SeqAIJ() line 1457 in
>>>>>>> src/mat/impls/aij/seq/aij.c
>>>>>>> [0]PETSC ERROR: MatSOR_SeqAIJ() line 1489 in src/mat/impls/aij/seq/aij.c
>>>>>>> [0]PETSC ERROR: MatSOR_MPIAIJ() line 1623 in
>>>>>>> src/mat/impls/aij/mpi/mpiaij.c
>>>>>>> [0]PETSC ERROR: MatSOR() line 3649 in src/mat/interface/matrix.c
>>>>>>> [0]PETSC ERROR: PCApply_SOR() line 35 in src/ksp/pc/impls/sor/sor.c
>>>>>>> [0]PETSC ERROR: PCApply() line 442 in src/ksp/pc/interface/precon.c
>>>>>>> [0]PETSC ERROR: KSP_PCApply() line 227 in
>>>>>>> src/ksp/ksp/interface//opt/petsc/petsc-3.4.2/include/petsc-private/kspimpl.h
>>>>>>> [0]PETSC ERROR: KSPInitialResidual() line 64 in
>>>>>>> src/ksp/ksp/interface/itres.c
>>>>>>> [0]PETSC ERROR: KSPSolve_GMRES() line 239 in
>>>>>>> src/ksp/ksp/impls/gmres/gmres.c
>>>>>>> [0]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c
>>>>>>> [0]PETSC ERROR: KSPSolve_Chebyshev() line 409 in
>>>>>>> src/ksp/ksp/impls/cheby/cheby.c
>>>>>>> [0]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c
>>>>>>> [0]PETSC ERROR: PCMGMCycle_Private() line 19 in src/ksp/pc/impls/mg/mg.c
>>>>>>> [0]PETSC ERROR: PCApply_MG() line 330 in src/ksp/pc/impls/mg/mg.c
>>>>>>> [0]PETSC ERROR: PCApply() line 442 in src/ksp/pc/interface/precon.c
>>>>>>> [0]PETSC ERROR: KSP_PCApply() line 227 in
>>>>>>> src/ksp/ksp/impls/cg//opt/petsc/petsc-3.4.2/include/petsc-private/kspimpl.h
>>>>>>> [0]PETSC ERROR: KSPSolve_CG() line 175 in src/ksp/ksp/impls/cg/cg.c
>>>>>>> [0]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c
>>>>>>> #PETSc Option Table entries:
>>>>>>> -da_refine 4
>>>>>>> -ksp_view
>>>>>>> -options_left
>>>>>>> -pc_mg_galerkin
>>>>>>> -pc_type mg
>>>>>>> #End of PETSc Option Table entries
>>>>>>> There is one unused database option. It is:
>>>>>>> Option left: name:-da_refine value: 4
>>>>>>>
>>>>>>>
>>>>>>> Here is the code I use to setup DMDA and KSP:
>>>>>>>
>>>>>>> call DMDACreate3d( PETSC_COMM_WORLD ,
>>>>>>> &
>>>>>>> & DMDA_BOUNDARY_PERIODIC , DMDA_BOUNDARY_PERIODIC,
>>>>>>> &
>>>>>>> & DMDA_BOUNDARY_PERIODIC , DMDA_STENCIL_STAR,
>>>>>>> &
>>>>>>> & N_Z , N_Y , N_X , N_B3 , N_B2 , 1_ip, 1_ip ,
>>>>>>> 1_ip , &
>>>>>>> & int(NNZ,ip) ,int(NNY,ip) , NNX, da , ierr)
>>>>>>> ! Create Global Vectors
>>>>>>> call DMCreateGlobalVector(da,b,ierr)
>>>>>>> call VecDuplicate(b,x,ierr)
>>>>>>> ! Set initial guess for first use of the module to 0
>>>>>>> call VecSet(x,0.0_rp,ierr)
>>>>>>> ! Create matrix
>>>>>>> call DMCreateMatrix(da,MATMPIAIJ,A,ierr)
>>>>>>> ! Create solver
>>>>>>> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr)
>>>>>>> call KSPSetDM(ksp,da,ierr)
>>>>>>> call KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN,ierr)
>>>>>>> ! call KSPSetOperators(ksp,A,A,SAME_NONZERO_PATTERN,ierr)
>>>>>>> call KSPSetType(ksp,KSPCG,ierr)
>>>>>>> call KSPSetNormType(ksp,KSP_NORM_UNPRECONDITIONED,ierr) ! Real
>>>>>>> residual
>>>>>>> call KSPSetInitialGuessNonzero(ksp,PETSC_TRUE,ierr)
>>>>>>> call KSPSetTolerances(ksp, tol ,PETSC_DEFAULT_DOUBLE_PRECISION,&
>>>>>>> & PETSC_DEFAULT_DOUBLE_PRECISION,PETSC_DEFAULT_INTEGER,ierr)
>>>>>>>
>>>>>>> ! To allow using option from command line
>>>>>>> call KSPSetFromOptions(ksp,ierr)
>>>>>>>
>>>>>>>
>>>>>>> Michele
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 08/01/2013 01:04 PM, Barry Smith wrote:
>>>>>>>
>>>>>>>
>>>>>>>> You can use the option -pc_mg_galerkin and then MG will compute
>>>>>>>> the coarser matrices with a sparse matrix matrix matrix product so you
>>>>>>>> should not need to change your code to try it out. You still need to
>>>>>>>> use the KSPSetDM() and -da_refine n to get it working
>>>>>>>>
>>>>>>>> If it doesn't work, send us all the output.
>>>>>>>>
>>>>>>>> Barry
>>>>>>>>
>>>>>>>>
>>>>>>>> On Aug 1, 2013, at 2:47 PM, Michele Rosso
>>>>>>>>
>>>>>>>>
>>>>>>>> <mro...@uci.edu>
>>>>>>>>
>>>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Barry,
>>>>>>>>> you are correct, I did not use it. I think I get now where is the
>>>>>>>>> problem. Correct me if I am wrong, but for the
>>>>>>>>> geometric multigrid to work, ksp must be provided with subroutines to
>>>>>>>>> compute the matrix and the rhs at any level through
>>>>>>>>> KSPSetComputeOperators and KSPSetComputeRHS.
>>>>>>>>> I do not do that, I simply build a rhs vector and a matrix and then I
>>>>>>>>> solve the system.
>>>>>>>>> If you confirm what I just wrote, I will try to modify my code
>>>>>>>>> accordingly and get back to you.
>>>>>>>>> Thank you,
>>>>>>>>> Michele
>>>>>>>>> On 08/01/2013 11:48 AM, Barry Smith wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Do you use KSPSetDM(ksp,da); ? See
>>>>>>>>>> src/ksp/ksp/examples/tutorials/ex19.c
>>>>>>>>>>
>>>>>>>>>> Barry
>>>>>>>>>>
>>>>>>>>>> On Aug 1, 2013, at 1:35 PM, Michele Rosso
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> <mro...@uci.edu>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Barry,
>>>>>>>>>>>
>>>>>>>>>>> I am using a finite difference Cartesian uniform grid and DMDA and
>>>>>>>>>>> so far it has not given me any problem.
>>>>>>>>>>> I am using a ksp solver (not snes). In a previous thread, I was
>>>>>>>>>>> told an odd number of grid points was needed for the geometric
>>>>>>>>>>> multigrid, is that correct?
>>>>>>>>>>> I tried to run my case with
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> -pc_type mg -da_refine 4
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> but it does not seem to use the -da_refine option:
>>>>>>>>>>>
>>>>>>>>>>> mpiexec -np 4 ./test -pc_type mg -da_refine 4 -ksp_view
>>>>>>>>>>> -options_left
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> KSP Object: 4 MPI processes
>>>>>>>>>>> type: cg
>>>>>>>>>>> maximum iterations=10000
>>>>>>>>>>> tolerances: relative=1e-08, absolute=1e-50, divergence=10000
>>>>>>>>>>> left preconditioning
>>>>>>>>>>> using nonzero initial guess
>>>>>>>>>>> using UNPRECONDITIONED norm type for convergence test
>>>>>>>>>>> PC Object: 4 MPI processes
>>>>>>>>>>> type: mg
>>>>>>>>>>> MG: type is MULTIPLICATIVE, levels=1 cycles=v
>>>>>>>>>>> Cycles per PCApply=1
>>>>>>>>>>> Not using Galerkin computed coarse grid matrices
>>>>>>>>>>> Coarse grid solver -- level -------------------------------
>>>>>>>>>>> KSP Object: (mg_levels_0_) 4 MPI processes
>>>>>>>>>>> type: chebyshev
>>>>>>>>>>> Chebyshev: eigenvalue estimates: min = 0.134543, max =
>>>>>>>>>>> 1.47998
>>>>>>>>>>> Chebyshev: estimated using: [0 0.1; 0 1.1]
>>>>>>>>>>> KSP Object: (mg_levels_0_est_) 4 MPI processes
>>>>>>>>>>> type: gmres
>>>>>>>>>>> GMRES: restart=30, using Classical (unmodified)
>>>>>>>>>>> Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>>>>>> GMRES: happy breakdown tolerance 1e-30
>>>>>>>>>>> maximum iterations=10, initial guess is zero
>>>>>>>>>>> tolerances: relative=1e-05, absolute=1e-50,
>>>>>>>>>>> divergence=10000
>>>>>>>>>>> left preconditioning
>>>>>>>>>>> using NONE norm type for convergence test
>>>>>>>>>>> PC Object: (mg_levels_0_) 4 MPI processes
>>>>>>>>>>> type: sor
>>>>>>>>>>> SOR: type = local_symmetric, iterations = 1, local
>>>>>>>>>>> iterations = 1, omega = 1
>>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>>> Matrix Object: 4 MPI processes
>>>>>>>>>>> type: mpiaij
>>>>>>>>>>> rows=262144, cols=262144
>>>>>>>>>>> total: nonzeros=1835008, allocated nonzeros=1835008
>>>>>>>>>>> total number of mallocs used during MatSetValues calls =0
>>>>>>>>>>> maximum iterations=1, initial guess is zero
>>>>>>>>>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>>>>>> left preconditioning
>>>>>>>>>>> using NONE norm type for convergence test
>>>>>>>>>>> PC Object: (mg_levels_0_) 4 MPI processes
>>>>>>>>>>> type: sor
>>>>>>>>>>> SOR: type = local_symmetric, iterations = 1, local
>>>>>>>>>>> iterations = 1, omega = 1
>>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>>> Matrix Object: 4 MPI processes
>>>>>>>>>>> type: mpiaij
>>>>>>>>>>> rows=262144, cols=262144
>>>>>>>>>>> total: nonzeros=1835008, allocated nonzeros=1835008
>>>>>>>>>>> total number of mallocs used during MatSetValues calls =0
>>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>>> Matrix Object: 4 MPI processes
>>>>>>>>>>> type: mpiaij
>>>>>>>>>>> rows=262144, cols=262144
>>>>>>>>>>> total: nonzeros=1835008, allocated nonzeros=1835008
>>>>>>>>>>> total number of mallocs used during MatSetValues calls =0
>>>>>>>>>>> Solution = 1.53600013 sec
>>>>>>>>>>> #PETSc Option Table entries:
>>>>>>>>>>> -da_refine 4
>>>>>>>>>>> -ksp_view
>>>>>>>>>>> -options_left
>>>>>>>>>>> -pc_type mg
>>>>>>>>>>> #End of PETSc Option Table entries
>>>>>>>>>>> There is one unused database option. It is:
>>>>>>>>>>> Option left: name:-da_refine value: 4
>>>>>>>>>>>
>>>>>>>>>>> Michele
>>>>>>>>>>>
>>>>>>>>>>> On 08/01/2013 11:21 AM, Barry Smith wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> What kind of mesh are you using? Are you using DMDA? If you are
>>>>>>>>>>>> using DMDA (and have written your code to use it "correctly") then
>>>>>>>>>>>> it should be trivial to run with geometric multigrid and geometric
>>>>>>>>>>>> multigrid should be a bit faster.
>>>>>>>>>>>>
>>>>>>>>>>>> For example on src/snes/examples/tutorials/ex19.c I run with
>>>>>>>>>>>> ./ex19 -pc_type mg -da_refine 4 and it refines the original DMDA 4
>>>>>>>>>>>> times and uses geometric multigrid with 5 levels.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Barry
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Aug 1, 2013, at 1:14 PM, Michele Rosso
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> <mro...@uci.edu>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I am successfully using PETSc (v3.4.2) to solve a 3D Poisson's
>>>>>>>>>>>>> equation with CG + GAMG as I was suggested to do in a previous
>>>>>>>>>>>>> thread.
>>>>>>>>>>>>> So far I am using GAMG with the default settings, i.e.
>>>>>>>>>>>>>
>>>>>>>>>>>>> -pc_type gamg -pc_gamg_agg_nsmooths 1
>>>>>>>>>>>>>
>>>>>>>>>>>>> The speed of the solution is satisfactory, but I would like to
>>>>>>>>>>>>> know if you have any suggestions to further speed it up,
>>>>>>>>>>>>> particularly
>>>>>>>>>>>>> if there is any parameters worth looking into to achieve an even
>>>>>>>>>>>>> faster solution, for example number of levels and so on.
>>>>>>>>>>>>> So far I am using Dirichlet's BCs for my test case, but I will
>>>>>>>>>>>>> soon have periodic conditions: in this case, does GAMG require
>>>>>>>>>>>>> particular settings?
>>>>>>>>>>>>> Finally, I did not try geometric multigrid: do you think it is
>>>>>>>>>>>>> worth a shot?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Here are my current settings:
>>>>>>>>>>>>>
>>>>>>>>>>>>> I run with
>>>>>>>>>>>>>
>>>>>>>>>>>>> -pc_type gamg -pc_gamg_agg_nsmooths 1 -ksp_view -options_left
>>>>>>>>>>>>>
>>>>>>>>>>>>> and the output is:
>>>>>>>>>>>>>
>>>>>>>>>>>>> KSP Object: 4 MPI processes
>>>>>>>>>>>>> type: cg
>>>>>>>>>>>>> maximum iterations=10000
>>>>>>>>>>>>> tolerances: relative=1e-08, absolute=1e-50, divergence=10000
>>>>>>>>>>>>> left preconditioning
>>>>>>>>>>>>> using nonzero initial guess
>>>>>>>>>>>>> using UNPRECONDITIONED norm type for convergence test
>>>>>>>>>>>>> PC Object: 4 MPI processes
>>>>>>>>>>>>> type: gamg
>>>>>>>>>>>>> MG: type is MULTIPLICATIVE, levels=3 cycles=v
>>>>>>>>>>>>> Cycles per PCApply=1
>>>>>>>>>>>>> Using Galerkin computed coarse grid matrices
>>>>>>>>>>>>> Coarse grid solver -- level -------------------------------
>>>>>>>>>>>>> KSP Object: (mg_coarse_) 4 MPI processes
>>>>>>>>>>>>> type: preonly
>>>>>>>>>>>>> maximum iterations=1, initial guess is zero
>>>>>>>>>>>>> tolerances: relative=1e-05, absolute=1e-50,
>>>>>>>>>>>>> divergence=10000
>>>>>>>>>>>>> left preconditioning
>>>>>>>>>>>>> using NONE norm type for convergence test
>>>>>>>>>>>>> PC Object: (mg_coarse_) 4 MPI processes
>>>>>>>>>>>>> type: bjacobi
>>>>>>>>>>>>> block Jacobi: number of blocks = 4
>>>>>>>>>>>>> Local solve info for each block is in the following KSP
>>>>>>>>>>>>> and PC objects:
>>>>>>>>>>>>> [0] number of local blocks = 1, first local block number = 0
>>>>>>>>>>>>> [0] local block number 0
>>>>>>>>>>>>> KSP Object: (mg_coarse_sub_) 1 MPI processes
>>>>>>>>>>>>> type: preonly
>>>>>>>>>>>>> maximum iterations=1, initial guess is zero
>>>>>>>>>>>>> tolerances: relative=1e-05, absolute=1e-50,
>>>>>>>>>>>>> divergence=10000
>>>>>>>>>>>>> KSP Object: (mg_coarse_sub_) left
>>>>>>>>>>>>> preconditioning
>>>>>>>>>>>>> using NONE norm type for convergence test
>>>>>>>>>>>>> PC Object: (mg_coarse_sub_) 1 MPI processes
>>>>>>>>>>>>> type: preonly
>>>>>>>>>>>>> 1 MPI processes
>>>>>>>>>>>>> type: lu
>>>>>>>>>>>>> maximum iterations=1, initial guess is zero
>>>>>>>>>>>>> tolerances: relative=1e-05, absolute=1e-50,
>>>>>>>>>>>>> divergence=10000
>>>>>>>>>>>>> LU: out-of-place factorization
>>>>>>>>>>>>> left preconditioning
>>>>>>>>>>>>> using NONE norm type for convergence test
>>>>>>>>>>>>> PC Object: (mg_coarse_sub_) 1 MPI
>>>>>>>>>>>>> processes
>>>>>>>>>>>>> type: lu
>>>>>>>>>>>>> tolerance for zero pivot 2.22045e-14
>>>>>>>>>>>>> using diagonal shift on blocks to prevent zero pivot
>>>>>>>>>>>>> matrix ordering: nd
>>>>>>>>>>>>> LU: out-of-place factorization
>>>>>>>>>>>>> tolerance for zero pivot 2.22045e-14
>>>>>>>>>>>>> using diagonal shift on blocks to prevent zero pivot
>>>>>>>>>>>>> matrix ordering: nd
>>>>>>>>>>>>> factor fill ratio given 5, needed 0
>>>>>>>>>>>>> Factored matrix follows:
>>>>>>>>>>>>> factor fill ratio given 5, needed 4.13207
>>>>>>>>>>>>> Factored matrix follows:
>>>>>>>>>>>>> Matrix Object: Matrix Object:
>>>>>>>>>>>>> 1 MPI processes
>>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>>> rows=395, cols=395
>>>>>>>>>>>>> package used to perform factorization: petsc
>>>>>>>>>>>>> total: nonzeros=132379, allocated
>>>>>>>>>>>>> nonzeros=132379
>>>>>>>>>>>>> total number of mallocs used during
>>>>>>>>>>>>> MatSetValues calls =0
>>>>>>>>>>>>> not using I-node routines
>>>>>>>>>>>>> 1 MPI processes
>>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>>>>> rows=0, cols=0
>>>>>>>>>>>>> package used to perform factorization: petsc
>>>>>>>>>>>>> total: nonzeros=1, allocated nonzeros=1
>>>>>>>>>>>>> total number of mallocs used during
>>>>>>>>>>>>> MatSetValues calls =0
>>>>>>>>>>>>> not using I-node routines
>>>>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>>>>> Matrix Object: 1 MPI processes
>>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>>> Matrix Object:KSP Object: 1 MPI processes
>>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>>> rows=0, cols=0
>>>>>>>>>>>>> total: nonzeros=0, allocated nonzeros=0
>>>>>>>>>>>>> total number of mallocs used during MatSetValues
>>>>>>>>>>>>> calls =0
>>>>>>>>>>>>> not using I-node routines
>>>>>>>>>>>>> rows=395, cols=395
>>>>>>>>>>>>> total: nonzeros=32037, allocated nonzeros=32037
>>>>>>>>>>>>> total number of mallocs used during MatSetValues
>>>>>>>>>>>>> calls =0
>>>>>>>>>>>>> not using I-node routines
>>>>>>>>>>>>> - - - - - - - - - - - - - - - - - -
>>>>>>>>>>>>> KSP Object: (mg_coarse_sub_) 1 MPI
>>>>>>>>>>>>> processes
>>>>>>>>>>>>> type: preonly
>>>>>>>>>>>>> maximum iterations=1, initial guess is zero
>>>>>>>>>>>>> tolerances: relative=1e-05, absolute=1e-50,
>>>>>>>>>>>>> divergence=10000
>>>>>>>>>>>>> left preconditioning
>>>>>>>>>>>>> using NONE norm type for convergence test
>>>>>>>>>>>>> PC Object: (mg_coarse_sub_) 1 MPI processes
>>>>>>>>>>>>> type: lu
>>>>>>>>>>>>> LU: out-of-place factorization
>>>>>>>>>>>>> tolerance for zero pivot 2.22045e-14
>>>>>>>>>>>>> using diagonal shift on blocks to prevent zero pivot
>>>>>>>>>>>>> matrix ordering: nd
>>>>>>>>>>>>> factor fill ratio given 5, needed 0
>>>>>>>>>>>>> Factored matrix follows:
>>>>>>>>>>>>> Matrix Object: 1 MPI processes
>>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>>> rows=0, cols=0
>>>>>>>>>>>>> package used to perform factorization: petsc
>>>>>>>>>>>>> total: nonzeros=1, allocated nonzeros=1
>>>>>>>>>>>>> total number of mallocs used during
>>>>>>>>>>>>> MatSetValues calls =0
>>>>>>>>>>>>> not using I-node routines
>>>>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>>>>> Matrix Object: 1 MPI processes
>>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>>> rows=0, cols=0
>>>>>>>>>>>>> total: nonzeros=0, allocated nonzeros=0
>>>>>>>>>>>>> total number of mallocs used during MatSetValues
>>>>>>>>>>>>> calls =0
>>>>>>>>>>>>> not using I-node routines
>>>>>>>>>>>>> (mg_coarse_sub_) 1 MPI processes
>>>>>>>>>>>>> type: preonly
>>>>>>>>>>>>> maximum iterations=1, initial guess is zero
>>>>>>>>>>>>> tolerances: relative=1e-05, absolute=1e-50,
>>>>>>>>>>>>> divergence=10000
>>>>>>>>>>>>> left preconditioning
>>>>>>>>>>>>> using NONE norm type for convergence test
>>>>>>>>>>>>> PC Object: (mg_coarse_sub_) 1 MPI processes
>>>>>>>>>>>>> type: lu
>>>>>>>>>>>>> LU: out-of-place factorization
>>>>>>>>>>>>> tolerance for zero pivot 2.22045e-14
>>>>>>>>>>>>> using diagonal shift on blocks to prevent zero pivot
>>>>>>>>>>>>> matrix ordering: nd
>>>>>>>>>>>>> factor fill ratio given 5, needed 0
>>>>>>>>>>>>> Factored matrix follows:
>>>>>>>>>>>>> Matrix Object: 1 MPI processes
>>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>>> rows=0, cols=0
>>>>>>>>>>>>> package used to perform factorization: petsc
>>>>>>>>>>>>> total: nonzeros=1, allocated nonzeros=1
>>>>>>>>>>>>> total number of mallocs used during
>>>>>>>>>>>>> MatSetValues calls =0
>>>>>>>>>>>>> not using I-node routines
>>>>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>>>>> Matrix Object: 1 MPI processes
>>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>>> rows=0, cols=0
>>>>>>>>>>>>> total: nonzeros=0, allocated nonzeros=0
>>>>>>>>>>>>> total number of mallocs used during MatSetValues
>>>>>>>>>>>>> calls =0
>>>>>>>>>>>>> not using I-node routines
>>>>>>>>>>>>> [1] number of local blocks = 1, first local block number = 1
>>>>>>>>>>>>> [1] local block number 0
>>>>>>>>>>>>> - - - - - - - - - - - - - - - - - -
>>>>>>>>>>>>> [2] number of local blocks = 1, first local block number = 2
>>>>>>>>>>>>> [2] local block number 0
>>>>>>>>>>>>> - - - - - - - - - - - - - - - - - -
>>>>>>>>>>>>> [3] number of local blocks = 1, first local block number = 3
>>>>>>>>>>>>> [3] local block number 0
>>>>>>>>>>>>> - - - - - - - - - - - - - - - - - -
>>>>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>>>>> Matrix Object: 4 MPI processes
>>>>>>>>>>>>> type: mpiaij
>>>>>>>>>>>>> rows=395, cols=395
>>>>>>>>>>>>> total: nonzeros=32037, allocated nonzeros=32037
>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls =0
>>>>>>>>>>>>> not using I-node (on process 0) routines
>>>>>>>>>>>>> Down solver (pre-smoother) on level 1
>>>>>>>>>>>>> -------------------------------
>>>>>>>>>>>>> KSP Object: (mg_levels_1_) 4 MPI processes
>>>>>>>>>>>>> type: chebyshev
>>>>>>>>>>>>> Chebyshev: eigenvalue estimates: min = 0.0636225, max =
>>>>>>>>>>>>> 1.33607
>>>>>>>>>>>>> maximum iterations=2
>>>>>>>>>>>>> tolerances: relative=1e-05, absolute=1e-50,
>>>>>>>>>>>>> divergence=10000
>>>>>>>>>>>>> left preconditioning
>>>>>>>>>>>>> using nonzero initial guess
>>>>>>>>>>>>> using NONE norm type for convergence test
>>>>>>>>>>>>> PC Object: (mg_levels_1_) 4 MPI processes
>>>>>>>>>>>>> type: jacobi
>>>>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>>>>> Matrix Object: 4 MPI processes
>>>>>>>>>>>>> type: mpiaij
>>>>>>>>>>>>> rows=23918, cols=23918
>>>>>>>>>>>>> total: nonzeros=818732, allocated nonzeros=818732
>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls =0
>>>>>>>>>>>>> not using I-node (on process 0) routines
>>>>>>>>>>>>> Up solver (post-smoother) same as down solver (pre-smoother)
>>>>>>>>>>>>> Down solver (pre-smoother) on level 2
>>>>>>>>>>>>> -------------------------------
>>>>>>>>>>>>> KSP Object: (mg_levels_2_) 4 MPI processes
>>>>>>>>>>>>> type: chebyshev
>>>>>>>>>>>>> Chebyshev: eigenvalue estimates: min = 0.0971369, max =
>>>>>>>>>>>>> 2.03987
>>>>>>>>>>>>> maximum iterations=2
>>>>>>>>>>>>> tolerances: relative=1e-05, absolute=1e-50,
>>>>>>>>>>>>> divergence=10000
>>>>>>>>>>>>> left preconditioning
>>>>>>>>>>>>> using nonzero initial guess
>>>>>>>>>>>>> using NONE norm type for convergence test
>>>>>>>>>>>>> PC Object: (mg_levels_2_) 4 MPI processes
>>>>>>>>>>>>> type: jacobi
>>>>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>>>>> Matrix Object: 4 MPI processes
>>>>>>>>>>>>> type: mpiaij
>>>>>>>>>>>>> rows=262144, cols=262144
>>>>>>>>>>>>> total: nonzeros=1835008, allocated nonzeros=1835008
>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls =0
>>>>>>>>>>>>> Up solver (post-smoother) same as down solver (pre-smoother)
>>>>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>>>>> Matrix Object: 4 MPI processes
>>>>>>>>>>>>> type: mpiaij
>>>>>>>>>>>>> rows=262144, cols=262144
>>>>>>>>>>>>> total: nonzeros=1835008, allocated nonzeros=1835008
>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls =0
>>>>>>>>>>>>> #PETSc Option Table entries:
>>>>>>>>>>>>> -ksp_view
>>>>>>>>>>>>> -options_left
>>>>>>>>>>>>> -pc_gamg_agg_nsmooths 1
>>>>>>>>>>>>> -pc_type gamg
>>>>>>>>>>>>> #End of PETSc Option Table entries
>>>>>>>>>>>>> There are no unused options.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>> Michele
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>
>
> <test_poisson_solver.tar.gz><2decomp_fft-1.5.847-modified.tar.gz>
