On Aug 2, 2013, at 6:11 PM, Barry Smith <bsm...@mcs.anl.gov> wrote: > > On Aug 2, 2013, at 4:52 PM, Michele Rosso <mro...@uci.edu> wrote: > >> Barry, >> >> thank you very much for your help. I was trying to debug the error with no >> success! >> Now it works like a charm for me too! >> I have still two questions for you: >> >> 1) How did you choose the number of levels to use: trial and error? > > I just used 2 because it is more than one level :-). When you use a finer > mesh you can use more levels. > >> 2) For a singular system (periodic), besides the nullspace removal, should I >> change any parameter? > > I don't know of anything. >
Don't you need an iterative coarse grid solver? > But there is a possible problem with -pc_mg_galerkin, PETSc does not > transfer the null space information from the fine mesh to the other meshes > and technically we really want the multigrid to remove the null space on all > the levels but usually it will work without worrying about that. I've found that the one outer cleaning is plenty and in fact can work w/o it. > > Barry > >> >> Again, thank you very much! >> >> Michele >> >> On 08/02/2013 02:38 PM, Barry Smith wrote: >>> Finally got it. My failing memory. I had to add the line >>> >>> call KSPSetDMActive(ksp,PETSC_FALSE,ierr) >>> >>> immediately after KSPSetDM() and >>> >>> change >>> >>> call DMCreateMatrix(da,MATMPIAIJ,A,ierr) >>> >>> to >>> >>> call DMCreateMatrix(da,MATAIJ,A,ierr) >>> >>> so it will work in both parallel and sequential then >>> >>> ksp_monitor -ksp_converged_reason -pc_type mg -ksp_view -pc_mg_galerkin >>> -pc_mg_levels 2 >>> >>> works great with 2 levels. >>> >>> Barry >>> >>> >>> >>> >>> On Aug 1, 2013, at 6:29 PM, Michele Rosso >>> <mro...@uci.edu> >>> wrote: >>> >>> >>>> Barry, >>>> >>>> no problem. I attached the full code in test_poisson_solver.tar.gz. >>>> My test code is a very reduced version of my productive code >>>> (incompressible DNS code) thus fftw3 and the library 2decomp&fft are >>>> needed to run it. >>>> I attached the 2decomp&fft version I used: it is a matter of minutes to >>>> install it, so you should not have any problem. >>>> Please, contact me for any question/suggestion. >>>> I the mean time I will try to debug it. >>>> >>>> Michele >>>> >>>> >>>> >>>> >>>> On 08/01/2013 04:19 PM, Barry Smith wrote: >>>> >>>>> Run on one process until this is debugged. You can try the option >>>>> >>>>> -start_in_debugger noxterm >>>>> >>>>> and then call VecView(vec,0) in the debugger when it gives the error >>>>> below. It seems like some objects are not getting their initial values >>>>> set properly. Are you able to email the code so we can run it and figure >>>>> out what is going on? >>>>> >>>>> Barry >>>>> >>>>> On Aug 1, 2013, at 5:52 PM, Michele Rosso >>>>> >>>>> <mro...@uci.edu> >>>>> >>>>> wrote: >>>>> >>>>> >>>>> >>>>>> Barry, >>>>>> >>>>>> I checked the matrix: the element (0,0) is not zero, nor any other >>>>>> diagonal element is. >>>>>> The matrix is symmetric positive define (i.e. the standard Poisson >>>>>> matrix). >>>>>> Also, -da_refine is never used (see previous output). >>>>>> I tried to run with -pc_type mg -pc_mg_galerkin -mg_levels_pc_type >>>>>> jacobi -mg_levels_ksp_type chebyshev >>>>>> -mg_levels_ksp_chebyshev_estimate_eigenvalues -ksp_view -options_left >>>>>> >>>>>> and now the error is different: >>>>>> 0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message >>>>>> ------------------------------------ >>>>>> [1]PETSC ERROR: Floating point exception! >>>>>> [1]PETSC ERROR: Vec entry at local location 0 is not-a-number or >>>>>> infinite at beginning of function: Parameter number 2! >>>>>> [1]PETSC ERROR: >>>>>> ------------------------------------------------------------------------ >>>>>> [1]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 >>>>>> [1]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>> [2]PETSC ERROR: --------------------- Error Message >>>>>> ------------------------------------ >>>>>> [2]PETSC ERROR: Floating point exception! >>>>>> [2]PETSC ERROR: Vec entry at local location 0 is not-a-number or >>>>>> infinite at beginning of function: Parameter number 2! >>>>>> [2]PETSC ERROR: >>>>>> ------------------------------------------------------------------------ >>>>>> [2]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 >>>>>> [2]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>> [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>> [2]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message >>>>>> ------------------------------------ >>>>>> [3]PETSC ERROR: Floating point exception! >>>>>> [3]PETSC ERROR: Vec entry at local location 0 is not-a-number or >>>>>> infinite at beginning of function: Parameter number 2! >>>>>> [3]PETSC ERROR: >>>>>> ------------------------------------------------------------------------ >>>>>> [3]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 >>>>>> [3]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>> [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>> [3]PETSC ERROR: See docs/index.html for manual pages. >>>>>> [3]PETSC ERROR: >>>>>> ------------------------------------------------------------------------ >>>>>> [1]PETSC ERROR: See docs/index.html for manual pages. >>>>>> [1]PETSC ERROR: >>>>>> ------------------------------------------------------------------------ >>>>>> [1]PETSC ERROR: ./test on a linux-gnu-dbg named enterprise-A by mic Thu >>>>>> Aug 1 15:43:16 2013 >>>>>> [1]PETSC ERROR: Libraries linked from >>>>>> /opt/petsc/petsc-3.4.2/linux-gnu-dbg/lib >>>>>> [1]PETSC ERROR: See docs/index.html for manual pages. >>>>>> [2]PETSC ERROR: >>>>>> ------------------------------------------------------------------------ >>>>>> [2]PETSC ERROR: ./test on a linux-gnu-dbg named enterprise-A by mic Thu >>>>>> Aug 1 15:43:16 2013 >>>>>> [2]PETSC ERROR: Libraries linked from >>>>>> /opt/petsc/petsc-3.4.2/linux-gnu-dbg/lib >>>>>> [2]PETSC ERROR: Configure run at Thu Aug 1 12:01:44 2013 >>>>>> [2]PETSC ERROR: [3]PETSC ERROR: ./test on a linux-gnu-dbg named >>>>>> enterprise-A by mic Thu Aug 1 15:43:16 2013 >>>>>> [3]PETSC ERROR: Libraries linked from >>>>>> /opt/petsc/petsc-3.4.2/linux-gnu-dbg/lib >>>>>> [3]PETSC ERROR: Configure run at Thu Aug 1 12:01:44 2013 >>>>>> [3]PETSC ERROR: Configure options >>>>>> Configure run at Thu Aug 1 12:01:44 2013 >>>>>> [1]PETSC ERROR: Configure options >>>>>> [1]PETSC ERROR: >>>>>> ------------------------------------------------------------------------ >>>>>> [1]PETSC ERROR: VecValidValues() line 28 in >>>>>> src/vec/vec/interface/rvector.c >>>>>> Configure options >>>>>> [2]PETSC ERROR: >>>>>> ------------------------------------------------------------------------ >>>>>> [2]PETSC ERROR: VecValidValues() line 28 in >>>>>> src/vec/vec/interface/rvector.c >>>>>> [3]PETSC ERROR: >>>>>> ------------------------------------------------------------------------ >>>>>> [3]PETSC ERROR: VecValidValues() line 28 in >>>>>> src/vec/vec/interface/rvector.c >>>>>> [3]PETSC ERROR: [1]PETSC ERROR: MatMult() line 2174 in >>>>>> src/mat/interface/matrix.c >>>>>> [1]PETSC ERROR: [2]PETSC ERROR: MatMult() line 2174 in >>>>>> src/mat/interface/matrix.c >>>>>> [2]PETSC ERROR: KSP_MatMult() line 204 in >>>>>> src/ksp/ksp/impls/cheby//opt/petsc/petsc-3.4.2/include/petsc-private/kspimpl.h >>>>>> MatMult() line 2174 in src/mat/interface/matrix.c >>>>>> [3]PETSC ERROR: KSP_MatMult() line 204 in >>>>>> src/ksp/ksp/impls/cheby//opt/petsc/petsc-3.4.2/include/petsc-private/kspimpl.h >>>>>> [3]PETSC ERROR: KSP_MatMult() line 204 in >>>>>> src/ksp/ksp/impls/cheby//opt/petsc/petsc-3.4.2/include/petsc-private/kspimpl.h >>>>>> [1]PETSC ERROR: KSPSolve_Chebyshev() line 504 in >>>>>> src/ksp/ksp/impls/cheby/cheby.c >>>>>> [2]PETSC ERROR: KSPSolve_Chebyshev() line 504 in >>>>>> src/ksp/ksp/impls/cheby/cheby.c >>>>>> [2]PETSC ERROR: KSPSolve_Chebyshev() line 504 in >>>>>> src/ksp/ksp/impls/cheby/cheby.c >>>>>> [3]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c >>>>>> [1]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c >>>>>> [1]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c >>>>>> [2]PETSC ERROR: PCMGMCycle_Private() line 19 in src/ksp/pc/impls/mg/mg.c >>>>>> [3]PETSC ERROR: PCMGMCycle_Private() line 19 in src/ksp/pc/impls/mg/mg.c >>>>>> PCMGMCycle_Private() line 19 in src/ksp/pc/impls/mg/mg.c >>>>>> [1]PETSC ERROR: PCApply_MG() line 330 in src/ksp/pc/impls/mg/mg.c >>>>>> [2]PETSC ERROR: PCApply_MG() line 330 in src/ksp/pc/impls/mg/mg.c >>>>>> [2]PETSC ERROR: PCApply() line 442 in src/ksp/pc/interface/precon.c >>>>>> [3]PETSC ERROR: PCApply_MG() line 330 in src/ksp/pc/impls/mg/mg.c >>>>>> [3]PETSC ERROR: PCApply() line 442 in src/ksp/pc/interface/precon.c >>>>>> [1]PETSC ERROR: PCApply() line 442 in src/ksp/pc/interface/precon.c >>>>>> [1]PETSC ERROR: KSP_PCApply() line 227 in >>>>>> src/ksp/ksp/impls/cg//opt/petsc/petsc-3.4.2/include/petsc-private/kspimpl.h >>>>>> [2]PETSC ERROR: KSP_PCApply() line 227 in >>>>>> src/ksp/ksp/impls/cg//opt/petsc/petsc-3.4.2/include/petsc-private/kspimpl.h >>>>>> [2]PETSC ERROR: [3]PETSC ERROR: KSP_PCApply() line 227 in >>>>>> src/ksp/ksp/impls/cg//opt/petsc/petsc-3.4.2/include/petsc-private/kspimpl.h >>>>>> [3]PETSC ERROR: KSPSolve_CG() line 175 in src/ksp/ksp/impls/cg/cg.c >>>>>> [1]PETSC ERROR: KSPSolve_CG() line 175 in src/ksp/ksp/impls/cg/cg.c >>>>>> [1]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c >>>>>> KSPSolve_CG() line 175 in src/ksp/ksp/impls/cg/cg.c >>>>>> [2]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c >>>>>> [3]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c >>>>>> --------------------- Error Message ------------------------------------ >>>>>> [0]PETSC ERROR: Floating point exception! >>>>>> [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or >>>>>> infinite at beginning of function: Parameter number 2! >>>>>> [0]PETSC ERROR: >>>>>> ------------------------------------------------------------------------ >>>>>> [0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 >>>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>>>> [0]PETSC ERROR: >>>>>> ------------------------------------------------------------------------ >>>>>> [0]PETSC ERROR: ./test on a linux-gnu-dbg named enterprise-A by mic Thu >>>>>> Aug 1 15:43:16 2013 >>>>>> [0]PETSC ERROR: Libraries linked from >>>>>> /opt/petsc/petsc-3.4.2/linux-gnu-dbg/lib >>>>>> [0]PETSC ERROR: Configure run at Thu Aug 1 12:01:44 2013 >>>>>> [0]PETSC ERROR: Configure options >>>>>> [0]PETSC ERROR: >>>>>> ------------------------------------------------------------------------ >>>>>> [0]PETSC ERROR: VecValidValues() line 28 in >>>>>> src/vec/vec/interface/rvector.c >>>>>> [0]PETSC ERROR: MatMult() line 2174 in src/mat/interface/matrix.c >>>>>> [0]PETSC ERROR: KSP_MatMult() line 204 in >>>>>> src/ksp/ksp/impls/cheby//opt/petsc/petsc-3.4.2/include/petsc-private/kspimpl.h >>>>>> [0]PETSC ERROR: KSPSolve_Chebyshev() line 504 in >>>>>> src/ksp/ksp/impls/cheby/cheby.c >>>>>> [0]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c >>>>>> [0]PETSC ERROR: PCMGMCycle_Private() line 19 in src/ksp/pc/impls/mg/mg.c >>>>>> [0]PETSC ERROR: PCApply_MG() line 330 in src/ksp/pc/impls/mg/mg.c >>>>>> [0]PETSC ERROR: PCApply() line 442 in src/ksp/pc/interface/precon.c >>>>>> [0]PETSC ERROR: KSP_PCApply() line 227 in >>>>>> src/ksp/ksp/impls/cg//opt/petsc/petsc-3.4.2/include/petsc-private/kspimpl.h >>>>>> [0]PETSC ERROR: KSPSolve_CG() line 175 in src/ksp/ksp/impls/cg/cg.c >>>>>> [0]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c >>>>>> >>>>>> #PETSc Option Table entries: >>>>>> -ksp_view >>>>>> -mg_levels_ksp_chebyshev_estimate_eigenvalues >>>>>> -mg_levels_ksp_type chebyshev >>>>>> -mg_levels_pc_type jacobi >>>>>> -options_left >>>>>> -pc_mg_galerkin >>>>>> -pc_type mg >>>>>> #End of PETSc Option Table entries >>>>>> There are no unused options. >>>>>> >>>>>> Michele >>>>>> >>>>>> >>>>>> On 08/01/2013 03:27 PM, Barry Smith wrote: >>>>>> >>>>>> >>>>>>> Do a MatView() on A before the solve (remove the -da_refine 4) so it >>>>>>> is small. Is the 0,0 entry 0? If the matrix has zero on the diagonals >>>>>>> you cannot us Gauss-Seidel as the smoother. You can start with >>>>>>> -mg_levels_pc_type jacobi -mg_levels_ksp_type chebychev >>>>>>> -mg_levels_ksp_chebyshev_estimate_eigenvalues >>>>>>> >>>>>>> Is the matrix a Stokes-like matrix? If so then different >>>>>>> preconditioners are in order. >>>>>>> >>>>>>> Barry >>>>>>> >>>>>>> On Aug 1, 2013, at 5:21 PM, Michele Rosso >>>>>>> >>>>>>> >>>>>>> <mro...@uci.edu> >>>>>>> >>>>>>> >>>>>>> wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Barry, >>>>>>>> >>>>>>>> here it is the fraction of code where I set the rhs term and the >>>>>>>> matrix. >>>>>>>> >>>>>>>> ! Create matrix >>>>>>>> call form_matrix( A , qrho, lsf, head ) >>>>>>>> call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr) >>>>>>>> call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr) >>>>>>>> >>>>>>>> ! Create rhs term >>>>>>>> call form_rhs(work, qrho, lsf, b , head) >>>>>>>> >>>>>>>> ! Solve system >>>>>>>> call KSPSetFromOptions(ksp,ierr) >>>>>>>> call KSPSetUp(ksp,ierr) >>>>>>>> call KSPSolve(ksp,b,x,ierr) >>>>>>>> call KSPGetIterationNumber(ksp, iiter ,ierr) >>>>>>>> >>>>>>>> The subroutine form_matrix returns the Mat object A that is filled by >>>>>>>> using MatSetValuesStencil. >>>>>>>> qrho, lsf and head are additional arguments that are needed to compute >>>>>>>> the matrix value. >>>>>>>> >>>>>>>> >>>>>>>> Michele >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 08/01/2013 03:11 PM, Barry Smith wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> Where are you putting the values into the matrix? It seems the >>>>>>>>> matrix has no values in it? The code is stopping because in the >>>>>>>>> Gauss-Seidel smoothing it has detected zero diagonals. >>>>>>>>> >>>>>>>>> Barry >>>>>>>>> >>>>>>>>> >>>>>>>>> On Aug 1, 2013, at 4:47 PM, Michele Rosso >>>>>>>>> >>>>>>>>> >>>>>>>>> <mro...@uci.edu> >>>>>>>>> >>>>>>>>> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> Barry, >>>>>>>>>> >>>>>>>>>> I run with : -pc_type mg -pc_mg_galerkin -da_refine 4 -ksp_view >>>>>>>>>> -options_left >>>>>>>>>> >>>>>>>>>> For the test I use a 64^3 grid and 4 processors. >>>>>>>>>> >>>>>>>>>> The output is: >>>>>>>>>> >>>>>>>>>> [2]PETSC ERROR: --------------------- Error Message >>>>>>>>>> ------------------------------------ >>>>>>>>>> [2]PETSC ERROR: Arguments are incompatible! >>>>>>>>>> [2]PETSC ERROR: Zero diagonal on row 0! >>>>>>>>>> [2]PETSC ERROR: >>>>>>>>>> ------------------------------------------------------------------------ >>>>>>>>>> [2]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 >>>>>>>>>> [2]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>>>>>> [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>>>>>> [2]PETSC ERROR: See docs/index.html for manual pages. >>>>>>>>>> [2]PETSC ERROR: >>>>>>>>>> ------------------------------------------------------------------------ >>>>>>>>>> [2]PETSC ERROR: ./test on a linux-gnu-dbg named enterprise-A by mic >>>>>>>>>> Thu Aug 1 14:44:04 2013 >>>>>>>>>> [0]PETSC ERROR: [2]PETSC ERROR: Libraries linked from >>>>>>>>>> /opt/petsc/petsc-3.4.2/linux-gnu-dbg/lib >>>>>>>>>> [2]PETSC ERROR: Configure run at Thu Aug 1 12:01:44 2013 >>>>>>>>>> [2]PETSC ERROR: Configure options >>>>>>>>>> [2]PETSC ERROR: >>>>>>>>>> ------------------------------------------------------------------------ >>>>>>>>>> [2]PETSC ERROR: MatInvertDiagonal_SeqAIJ() line 1457 in >>>>>>>>>> src/mat/impls/aij/seq/aij.c >>>>>>>>>> [2]PETSC ERROR: MatSOR_SeqAIJ() line 1489 in >>>>>>>>>> src/mat/impls/aij/seq/aij.c >>>>>>>>>> --------------------- Error Message >>>>>>>>>> ------------------------------------ >>>>>>>>>> [2]PETSC ERROR: MatSOR_MPIAIJ() line 1623 in >>>>>>>>>> src/mat/impls/aij/mpi/mpiaij.c >>>>>>>>>> [2]PETSC ERROR: MatSOR() line 3649 in src/mat/interface/matrix.c >>>>>>>>>> [2]PETSC ERROR: [0]PETSC ERROR: PCApply_SOR() line 35 in >>>>>>>>>> src/ksp/pc/impls/sor/sor.c >>>>>>>>>> [2]PETSC ERROR: PCApply() line 442 in src/ksp/pc/interface/precon.c >>>>>>>>>> [2]PETSC ERROR: KSP_PCApply() line 227 in >>>>>>>>>> src/ksp/ksp/interface//opt/petsc/petsc-3.4.2/include/petsc-private/kspimpl.h >>>>>>>>>> Arguments are incompatible! >>>>>>>>>> [2]PETSC ERROR: KSPInitialResidual() line 64 in >>>>>>>>>> src/ksp/ksp/interface/itres.c >>>>>>>>>> [2]PETSC ERROR: KSPSolve_GMRES() line 239 in >>>>>>>>>> src/ksp/ksp/impls/gmres/gmres.c >>>>>>>>>> [2]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c >>>>>>>>>> [2]PETSC ERROR: [0]PETSC ERROR: KSPSolve_Chebyshev() line 409 in >>>>>>>>>> src/ksp/ksp/impls/cheby/cheby.c >>>>>>>>>> [2]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c >>>>>>>>>> [2]PETSC ERROR: PCMGMCycle_Private() line 19 in >>>>>>>>>> src/ksp/pc/impls/mg/mg.c >>>>>>>>>> Zero diagonal on row 0! >>>>>>>>>> [2]PETSC ERROR: PCApply_MG() line 330 in src/ksp/pc/impls/mg/mg.c >>>>>>>>>> [2]PETSC ERROR: PCApply() line 442 in src/ksp/pc/interface/precon.c >>>>>>>>>> [0]PETSC ERROR: [2]PETSC ERROR: KSP_PCApply() line 227 in >>>>>>>>>> src/ksp/ksp/impls/cg//opt/petsc/petsc-3.4.2/include/petsc-private/kspimpl.h >>>>>>>>>> [2]PETSC ERROR: KSPSolve_CG() line 175 in src/ksp/ksp/impls/cg/cg.c >>>>>>>>>> ------------------------------------------------------------------------ >>>>>>>>>> [2]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c >>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 >>>>>>>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>>>>>> [3]PETSC ERROR: [0]PETSC ERROR: See docs/faq.html for hints about >>>>>>>>>> trouble shooting. >>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>>>>> ------------------------------------ >>>>>>>>>> [3]PETSC ERROR: Arguments are incompatible! >>>>>>>>>> [3]PETSC ERROR: Zero diagonal on row 0! >>>>>>>>>> [3]PETSC ERROR: >>>>>>>>>> ------------------------------------------------------------------------ >>>>>>>>>> [3]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 >>>>>>>>>> [3]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>>>>>> [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>>>>>> [3]PETSC ERROR: See docs/index.html for manual pages. >>>>>>>>>> [3]PETSC ERROR: >>>>>>>>>> ------------------------------------------------------------------------ >>>>>>>>>> See docs/index.html for manual pages. >>>>>>>>>> [3]PETSC ERROR: ./test on a linux-gnu-dbg named enterprise-A by mic >>>>>>>>>> Thu Aug 1 14:44:04 2013 >>>>>>>>>> [3]PETSC ERROR: Libraries linked from >>>>>>>>>> /opt/petsc/petsc-3.4.2/linux-gnu-dbg/lib >>>>>>>>>> [1]PETSC ERROR: [3]PETSC ERROR: Configure run at Thu Aug 1 12:01:44 >>>>>>>>>> 2013 >>>>>>>>>> [3]PETSC ERROR: Configure options >>>>>>>>>> [3]PETSC ERROR: >>>>>>>>>> ------------------------------------------------------------------------ >>>>>>>>>> [3]PETSC ERROR: --------------------- Error Message >>>>>>>>>> ------------------------------------ >>>>>>>>>> MatInvertDiagonal_SeqAIJ() line 1457 in src/mat/impls/aij/seq/aij.c >>>>>>>>>> [3]PETSC ERROR: MatSOR_SeqAIJ() line 1489 in >>>>>>>>>> src/mat/impls/aij/seq/aij.c >>>>>>>>>> [3]PETSC ERROR: [0]PETSC ERROR: MatSOR_MPIAIJ() line 1623 in >>>>>>>>>> src/mat/impls/aij/mpi/mpiaij.c >>>>>>>>>> [1]PETSC ERROR: Arguments are incompatible! >>>>>>>>>> [1]PETSC ERROR: Zero diagonal on row 0! >>>>>>>>>> [1]PETSC ERROR: >>>>>>>>>> ------------------------------------------------------------------------ >>>>>>>>>> [1]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 >>>>>>>>>> [1]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>>>>>> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>>>>>> [1]PETSC ERROR: See docs/index.html for manual pages. >>>>>>>>>> [1]PETSC ERROR: >>>>>>>>>> ------------------------------------------------------------------------ >>>>>>>>>> [1]PETSC ERROR: ./test on a linux-gnu-dbg named enterprise-A by mic >>>>>>>>>> Thu Aug 1 14:44:04 2013 >>>>>>>>>> [1]PETSC ERROR: Libraries linked from >>>>>>>>>> /opt/petsc/petsc-3.4.2/linux-gnu-dbg/lib >>>>>>>>>> [1]PETSC ERROR: Configure run at Thu Aug 1 12:01:44 2013 >>>>>>>>>> [1]PETSC ERROR: Configure options >>>>>>>>>> [1]PETSC ERROR: >>>>>>>>>> ------------------------------------------------------------------------ >>>>>>>>>> [1]PETSC ERROR: MatInvertDiagonal_SeqAIJ() line 1457 in >>>>>>>>>> src/mat/impls/aij/seq/aij.c >>>>>>>>>> [1]PETSC ERROR: [3]PETSC ERROR: MatSOR() line 3649 in >>>>>>>>>> src/mat/interface/matrix.c >>>>>>>>>> [3]PETSC ERROR: PCApply_SOR() line 35 in src/ksp/pc/impls/sor/sor.c >>>>>>>>>> [3]PETSC ERROR: PCApply() line 442 in src/ksp/pc/interface/precon.c >>>>>>>>>> [3]PETSC ERROR: KSP_PCApply() line 227 in >>>>>>>>>> src/ksp/ksp/interface//opt/petsc/petsc-3.4.2/include/petsc-private/kspimpl.h >>>>>>>>>> [3]PETSC ERROR: KSPInitialResidual() line 64 in >>>>>>>>>> src/ksp/ksp/interface/itres.c >>>>>>>>>> [3]PETSC ERROR: KSPSolve_GMRES() line 239 in >>>>>>>>>> src/ksp/ksp/impls/gmres/gmres.c >>>>>>>>>> [3]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c >>>>>>>>>> [3]PETSC ERROR: KSPSolve_Chebyshev() line 409 in >>>>>>>>>> src/ksp/ksp/impls/cheby/cheby.c >>>>>>>>>> [3]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c >>>>>>>>>> [3]PETSC ERROR: PCMGMCycle_Private() line 19 in >>>>>>>>>> src/ksp/pc/impls/mg/mg.c >>>>>>>>>> [3]PETSC ERROR: PCApply_MG() line 330 in src/ksp/pc/impls/mg/mg.c >>>>>>>>>> [3]PETSC ERROR: PCApply() line 442 in src/ksp/pc/interface/precon.c >>>>>>>>>> [3]PETSC ERROR: KSP_PCApply() line 227 in >>>>>>>>>> src/ksp/ksp/impls/cg//opt/petsc/petsc-3.4.2/include/petsc-private/kspimpl.h >>>>>>>>>> [3]PETSC ERROR: KSPSolve_CG() line 175 in src/ksp/ksp/impls/cg/cg.c >>>>>>>>>> [3]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c >>>>>>>>>> MatSOR_SeqAIJ() line 1489 in src/mat/impls/aij/seq/aij.c >>>>>>>>>> [1]PETSC ERROR: MatSOR_MPIAIJ() line 1623 in >>>>>>>>>> src/mat/impls/aij/mpi/mpiaij.c >>>>>>>>>> [1]PETSC ERROR: MatSOR() line 3649 in src/mat/interface/matrix.c >>>>>>>>>> [1]PETSC ERROR: PCApply_SOR() line 35 in src/ksp/pc/impls/sor/sor.c >>>>>>>>>> [1]PETSC ERROR: PCApply() line 442 in src/ksp/pc/interface/precon.c >>>>>>>>>> [1]PETSC ERROR: KSP_PCApply() line 227 in >>>>>>>>>> src/ksp/ksp/interface//opt/petsc/petsc-3.4.2/include/petsc-private/kspimpl.h >>>>>>>>>> [1]PETSC ERROR: KSPInitialResidual() line 64 in >>>>>>>>>> src/ksp/ksp/interface/itres.c >>>>>>>>>> [1]PETSC ERROR: KSPSolve_GMRES() line 239 in >>>>>>>>>> src/ksp/ksp/impls/gmres/gmres.c >>>>>>>>>> [1]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c >>>>>>>>>> [1]PETSC ERROR: KSPSolve_Chebyshev() line 409 in >>>>>>>>>> src/ksp/ksp/impls/cheby/cheby.c >>>>>>>>>> [1]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c >>>>>>>>>> [1]PETSC ERROR: PCMGMCycle_Private() line 19 in >>>>>>>>>> src/ksp/pc/impls/mg/mg.c >>>>>>>>>> [1]PETSC ERROR: PCApply_MG() line 330 in src/ksp/pc/impls/mg/mg.c >>>>>>>>>> [1]PETSC ERROR: PCApply() line 442 in src/ksp/pc/interface/precon.c >>>>>>>>>> [1]PETSC ERROR: KSP_PCApply() line 227 in >>>>>>>>>> src/ksp/ksp/impls/cg//opt/petsc/petsc-3.4.2/include/petsc-private/kspimpl.h >>>>>>>>>> [1]PETSC ERROR: KSPSolve_CG() line 175 in src/ksp/ksp/impls/cg/cg.c >>>>>>>>>> [1]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c >>>>>>>>>> ------------------------------------------------------------------------ >>>>>>>>>> [0]PETSC ERROR: ./test on a linux-gnu-dbg named enterprise-A by mic >>>>>>>>>> Thu Aug 1 14:44:04 2013 >>>>>>>>>> [0]PETSC ERROR: Libraries linked from >>>>>>>>>> /opt/petsc/petsc-3.4.2/linux-gnu-dbg/lib >>>>>>>>>> [0]PETSC ERROR: Configure run at Thu Aug 1 12:01:44 2013 >>>>>>>>>> [0]PETSC ERROR: Configure options >>>>>>>>>> [0]PETSC ERROR: >>>>>>>>>> ------------------------------------------------------------------------ >>>>>>>>>> [0]PETSC ERROR: MatInvertDiagonal_SeqAIJ() line 1457 in >>>>>>>>>> src/mat/impls/aij/seq/aij.c >>>>>>>>>> [0]PETSC ERROR: MatSOR_SeqAIJ() line 1489 in >>>>>>>>>> src/mat/impls/aij/seq/aij.c >>>>>>>>>> [0]PETSC ERROR: MatSOR_MPIAIJ() line 1623 in >>>>>>>>>> src/mat/impls/aij/mpi/mpiaij.c >>>>>>>>>> [0]PETSC ERROR: MatSOR() line 3649 in src/mat/interface/matrix.c >>>>>>>>>> [0]PETSC ERROR: PCApply_SOR() line 35 in src/ksp/pc/impls/sor/sor.c >>>>>>>>>> [0]PETSC ERROR: PCApply() line 442 in src/ksp/pc/interface/precon.c >>>>>>>>>> [0]PETSC ERROR: KSP_PCApply() line 227 in >>>>>>>>>> src/ksp/ksp/interface//opt/petsc/petsc-3.4.2/include/petsc-private/kspimpl.h >>>>>>>>>> [0]PETSC ERROR: KSPInitialResidual() line 64 in >>>>>>>>>> src/ksp/ksp/interface/itres.c >>>>>>>>>> [0]PETSC ERROR: KSPSolve_GMRES() line 239 in >>>>>>>>>> src/ksp/ksp/impls/gmres/gmres.c >>>>>>>>>> [0]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c >>>>>>>>>> [0]PETSC ERROR: KSPSolve_Chebyshev() line 409 in >>>>>>>>>> src/ksp/ksp/impls/cheby/cheby.c >>>>>>>>>> [0]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c >>>>>>>>>> [0]PETSC ERROR: PCMGMCycle_Private() line 19 in >>>>>>>>>> src/ksp/pc/impls/mg/mg.c >>>>>>>>>> [0]PETSC ERROR: PCApply_MG() line 330 in src/ksp/pc/impls/mg/mg.c >>>>>>>>>> [0]PETSC ERROR: PCApply() line 442 in src/ksp/pc/interface/precon.c >>>>>>>>>> [0]PETSC ERROR: KSP_PCApply() line 227 in >>>>>>>>>> src/ksp/ksp/impls/cg//opt/petsc/petsc-3.4.2/include/petsc-private/kspimpl.h >>>>>>>>>> [0]PETSC ERROR: KSPSolve_CG() line 175 in src/ksp/ksp/impls/cg/cg.c >>>>>>>>>> [0]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c >>>>>>>>>> #PETSc Option Table entries: >>>>>>>>>> -da_refine 4 >>>>>>>>>> -ksp_view >>>>>>>>>> -options_left >>>>>>>>>> -pc_mg_galerkin >>>>>>>>>> -pc_type mg >>>>>>>>>> #End of PETSc Option Table entries >>>>>>>>>> There is one unused database option. It is: >>>>>>>>>> Option left: name:-da_refine value: 4 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Here is the code I use to setup DMDA and KSP: >>>>>>>>>> >>>>>>>>>> call DMDACreate3d( PETSC_COMM_WORLD , >>>>>>>>>> & >>>>>>>>>> & DMDA_BOUNDARY_PERIODIC , >>>>>>>>>> DMDA_BOUNDARY_PERIODIC, & >>>>>>>>>> & DMDA_BOUNDARY_PERIODIC , DMDA_STENCIL_STAR, >>>>>>>>>> & >>>>>>>>>> & N_Z , N_Y , N_X , N_B3 , N_B2 , 1_ip, 1_ip , >>>>>>>>>> 1_ip , & >>>>>>>>>> & int(NNZ,ip) ,int(NNY,ip) , NNX, da , ierr) >>>>>>>>>> ! Create Global Vectors >>>>>>>>>> call DMCreateGlobalVector(da,b,ierr) >>>>>>>>>> call VecDuplicate(b,x,ierr) >>>>>>>>>> ! Set initial guess for first use of the module to 0 >>>>>>>>>> call VecSet(x,0.0_rp,ierr) >>>>>>>>>> ! Create matrix >>>>>>>>>> call DMCreateMatrix(da,MATMPIAIJ,A,ierr) >>>>>>>>>> ! Create solver >>>>>>>>>> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr) >>>>>>>>>> call KSPSetDM(ksp,da,ierr) >>>>>>>>>> call KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN,ierr) >>>>>>>>>> ! call KSPSetOperators(ksp,A,A,SAME_NONZERO_PATTERN,ierr) >>>>>>>>>> call KSPSetType(ksp,KSPCG,ierr) >>>>>>>>>> call KSPSetNormType(ksp,KSP_NORM_UNPRECONDITIONED,ierr) ! Real >>>>>>>>>> residual >>>>>>>>>> call KSPSetInitialGuessNonzero(ksp,PETSC_TRUE,ierr) >>>>>>>>>> call KSPSetTolerances(ksp, tol ,PETSC_DEFAULT_DOUBLE_PRECISION,& >>>>>>>>>> & PETSC_DEFAULT_DOUBLE_PRECISION,PETSC_DEFAULT_INTEGER,ierr) >>>>>>>>>> >>>>>>>>>> ! To allow using option from command line >>>>>>>>>> call KSPSetFromOptions(ksp,ierr) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Michele >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 08/01/2013 01:04 PM, Barry Smith wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> You can use the option -pc_mg_galerkin and then MG will compute >>>>>>>>>>> the coarser matrices with a sparse matrix matrix matrix product so >>>>>>>>>>> you should not need to change your code to try it out. You still >>>>>>>>>>> need to use the KSPSetDM() and -da_refine n to get it working >>>>>>>>>>> >>>>>>>>>>> If it doesn't work, send us all the output. >>>>>>>>>>> >>>>>>>>>>> Barry >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Aug 1, 2013, at 2:47 PM, Michele Rosso >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> <mro...@uci.edu> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Barry, >>>>>>>>>>>> you are correct, I did not use it. I think I get now where is the >>>>>>>>>>>> problem. Correct me if I am wrong, but for the >>>>>>>>>>>> geometric multigrid to work, ksp must be provided with subroutines >>>>>>>>>>>> to compute the matrix and the rhs at any level through >>>>>>>>>>>> KSPSetComputeOperators and KSPSetComputeRHS. >>>>>>>>>>>> I do not do that, I simply build a rhs vector and a matrix and >>>>>>>>>>>> then I solve the system. >>>>>>>>>>>> If you confirm what I just wrote, I will try to modify my code >>>>>>>>>>>> accordingly and get back to you. >>>>>>>>>>>> Thank you, >>>>>>>>>>>> Michele >>>>>>>>>>>> On 08/01/2013 11:48 AM, Barry Smith wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> Do you use KSPSetDM(ksp,da); ? See >>>>>>>>>>>>> src/ksp/ksp/examples/tutorials/ex19.c >>>>>>>>>>>>> >>>>>>>>>>>>> Barry >>>>>>>>>>>>> >>>>>>>>>>>>> On Aug 1, 2013, at 1:35 PM, Michele Rosso >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> <mro...@uci.edu> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> Barry, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I am using a finite difference Cartesian uniform grid and DMDA >>>>>>>>>>>>>> and so far it has not given me any problem. >>>>>>>>>>>>>> I am using a ksp solver (not snes). In a previous thread, I was >>>>>>>>>>>>>> told an odd number of grid points was needed for the geometric >>>>>>>>>>>>>> multigrid, is that correct? >>>>>>>>>>>>>> I tried to run my case with >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -pc_type mg -da_refine 4 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> but it does not seem to use the -da_refine option: >>>>>>>>>>>>>> >>>>>>>>>>>>>> mpiexec -np 4 ./test -pc_type mg -da_refine 4 -ksp_view >>>>>>>>>>>>>> -options_left >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> KSP Object: 4 MPI processes >>>>>>>>>>>>>> type: cg >>>>>>>>>>>>>> maximum iterations=10000 >>>>>>>>>>>>>> tolerances: relative=1e-08, absolute=1e-50, divergence=10000 >>>>>>>>>>>>>> left preconditioning >>>>>>>>>>>>>> using nonzero initial guess >>>>>>>>>>>>>> using UNPRECONDITIONED norm type for convergence test >>>>>>>>>>>>>> PC Object: 4 MPI processes >>>>>>>>>>>>>> type: mg >>>>>>>>>>>>>> MG: type is MULTIPLICATIVE, levels=1 cycles=v >>>>>>>>>>>>>> Cycles per PCApply=1 >>>>>>>>>>>>>> Not using Galerkin computed coarse grid matrices >>>>>>>>>>>>>> Coarse grid solver -- level ------------------------------- >>>>>>>>>>>>>> KSP Object: (mg_levels_0_) 4 MPI processes >>>>>>>>>>>>>> type: chebyshev >>>>>>>>>>>>>> Chebyshev: eigenvalue estimates: min = 0.134543, max = >>>>>>>>>>>>>> 1.47998 >>>>>>>>>>>>>> Chebyshev: estimated using: [0 0.1; 0 1.1] >>>>>>>>>>>>>> KSP Object: (mg_levels_0_est_) 4 MPI >>>>>>>>>>>>>> processes >>>>>>>>>>>>>> type: gmres >>>>>>>>>>>>>> GMRES: restart=30, using Classical (unmodified) >>>>>>>>>>>>>> Gram-Schmidt Orthogonalization with no iterative refinement >>>>>>>>>>>>>> GMRES: happy breakdown tolerance 1e-30 >>>>>>>>>>>>>> maximum iterations=10, initial guess is zero >>>>>>>>>>>>>> tolerances: relative=1e-05, absolute=1e-50, >>>>>>>>>>>>>> divergence=10000 >>>>>>>>>>>>>> left preconditioning >>>>>>>>>>>>>> using NONE norm type for convergence test >>>>>>>>>>>>>> PC Object: (mg_levels_0_) 4 MPI processes >>>>>>>>>>>>>> type: sor >>>>>>>>>>>>>> SOR: type = local_symmetric, iterations = 1, local >>>>>>>>>>>>>> iterations = 1, omega = 1 >>>>>>>>>>>>>> linear system matrix = precond matrix: >>>>>>>>>>>>>> Matrix Object: 4 MPI processes >>>>>>>>>>>>>> type: mpiaij >>>>>>>>>>>>>> rows=262144, cols=262144 >>>>>>>>>>>>>> total: nonzeros=1835008, allocated nonzeros=1835008 >>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls >>>>>>>>>>>>>> =0 >>>>>>>>>>>>>> maximum iterations=1, initial guess is zero >>>>>>>>>>>>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 >>>>>>>>>>>>>> left preconditioning >>>>>>>>>>>>>> using NONE norm type for convergence test >>>>>>>>>>>>>> PC Object: (mg_levels_0_) 4 MPI processes >>>>>>>>>>>>>> type: sor >>>>>>>>>>>>>> SOR: type = local_symmetric, iterations = 1, local >>>>>>>>>>>>>> iterations = 1, omega = 1 >>>>>>>>>>>>>> linear system matrix = precond matrix: >>>>>>>>>>>>>> Matrix Object: 4 MPI processes >>>>>>>>>>>>>> type: mpiaij >>>>>>>>>>>>>> rows=262144, cols=262144 >>>>>>>>>>>>>> total: nonzeros=1835008, allocated nonzeros=1835008 >>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls =0 >>>>>>>>>>>>>> linear system matrix = precond matrix: >>>>>>>>>>>>>> Matrix Object: 4 MPI processes >>>>>>>>>>>>>> type: mpiaij >>>>>>>>>>>>>> rows=262144, cols=262144 >>>>>>>>>>>>>> total: nonzeros=1835008, allocated nonzeros=1835008 >>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls =0 >>>>>>>>>>>>>> Solution = 1.53600013 sec >>>>>>>>>>>>>> #PETSc Option Table entries: >>>>>>>>>>>>>> -da_refine 4 >>>>>>>>>>>>>> -ksp_view >>>>>>>>>>>>>> -options_left >>>>>>>>>>>>>> -pc_type mg >>>>>>>>>>>>>> #End of PETSc Option Table entries >>>>>>>>>>>>>> There is one unused database option. It is: >>>>>>>>>>>>>> Option left: name:-da_refine value: 4 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Michele >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 08/01/2013 11:21 AM, Barry Smith wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> What kind of mesh are you using? Are you using DMDA? If you >>>>>>>>>>>>>>> are using DMDA (and have written your code to use it >>>>>>>>>>>>>>> "correctly") then it should be trivial to run with geometric >>>>>>>>>>>>>>> multigrid and geometric multigrid should be a bit faster. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> For example on src/snes/examples/tutorials/ex19.c I run >>>>>>>>>>>>>>> with ./ex19 -pc_type mg -da_refine 4 and it refines the >>>>>>>>>>>>>>> original DMDA 4 times and uses geometric multigrid with 5 >>>>>>>>>>>>>>> levels. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Barry >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Aug 1, 2013, at 1:14 PM, Michele Rosso >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> <mro...@uci.edu> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I am successfully using PETSc (v3.4.2) to solve a 3D >>>>>>>>>>>>>>>> Poisson's equation with CG + GAMG as I was suggested to do in >>>>>>>>>>>>>>>> a previous thread. >>>>>>>>>>>>>>>> So far I am using GAMG with the default settings, i.e. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -pc_type gamg -pc_gamg_agg_nsmooths 1 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The speed of the solution is satisfactory, but I would like to >>>>>>>>>>>>>>>> know if you have any suggestions to further speed it up, >>>>>>>>>>>>>>>> particularly >>>>>>>>>>>>>>>> if there is any parameters worth looking into to achieve an >>>>>>>>>>>>>>>> even faster solution, for example number of levels and so on. >>>>>>>>>>>>>>>> So far I am using Dirichlet's BCs for my test case, but I will >>>>>>>>>>>>>>>> soon have periodic conditions: in this case, does GAMG require >>>>>>>>>>>>>>>> particular settings? >>>>>>>>>>>>>>>> Finally, I did not try geometric multigrid: do you think it is >>>>>>>>>>>>>>>> worth a shot? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Here are my current settings: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I run with >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -pc_type gamg -pc_gamg_agg_nsmooths 1 -ksp_view -options_left >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> and the output is: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> KSP Object: 4 MPI processes >>>>>>>>>>>>>>>> type: cg >>>>>>>>>>>>>>>> maximum iterations=10000 >>>>>>>>>>>>>>>> tolerances: relative=1e-08, absolute=1e-50, divergence=10000 >>>>>>>>>>>>>>>> left preconditioning >>>>>>>>>>>>>>>> using nonzero initial guess >>>>>>>>>>>>>>>> using UNPRECONDITIONED norm type for convergence test >>>>>>>>>>>>>>>> PC Object: 4 MPI processes >>>>>>>>>>>>>>>> type: gamg >>>>>>>>>>>>>>>> MG: type is MULTIPLICATIVE, levels=3 cycles=v >>>>>>>>>>>>>>>> Cycles per PCApply=1 >>>>>>>>>>>>>>>> Using Galerkin computed coarse grid matrices >>>>>>>>>>>>>>>> Coarse grid solver -- level ------------------------------- >>>>>>>>>>>>>>>> KSP Object: (mg_coarse_) 4 MPI processes >>>>>>>>>>>>>>>> type: preonly >>>>>>>>>>>>>>>> maximum iterations=1, initial guess is zero >>>>>>>>>>>>>>>> tolerances: relative=1e-05, absolute=1e-50, >>>>>>>>>>>>>>>> divergence=10000 >>>>>>>>>>>>>>>> left preconditioning >>>>>>>>>>>>>>>> using NONE norm type for convergence test >>>>>>>>>>>>>>>> PC Object: (mg_coarse_) 4 MPI processes >>>>>>>>>>>>>>>> type: bjacobi >>>>>>>>>>>>>>>> block Jacobi: number of blocks = 4 >>>>>>>>>>>>>>>> Local solve info for each block is in the following KSP >>>>>>>>>>>>>>>> and PC objects: >>>>>>>>>>>>>>>> [0] number of local blocks = 1, first local block number >>>>>>>>>>>>>>>> = 0 >>>>>>>>>>>>>>>> [0] local block number 0 >>>>>>>>>>>>>>>> KSP Object: (mg_coarse_sub_) 1 MPI processes >>>>>>>>>>>>>>>> type: preonly >>>>>>>>>>>>>>>> maximum iterations=1, initial guess is zero >>>>>>>>>>>>>>>> tolerances: relative=1e-05, absolute=1e-50, >>>>>>>>>>>>>>>> divergence=10000 >>>>>>>>>>>>>>>> KSP Object: (mg_coarse_sub_) left >>>>>>>>>>>>>>>> preconditioning >>>>>>>>>>>>>>>> using NONE norm type for convergence test >>>>>>>>>>>>>>>> PC Object: (mg_coarse_sub_) 1 MPI >>>>>>>>>>>>>>>> processes >>>>>>>>>>>>>>>> type: preonly >>>>>>>>>>>>>>>> 1 MPI processes >>>>>>>>>>>>>>>> type: lu >>>>>>>>>>>>>>>> maximum iterations=1, initial guess is zero >>>>>>>>>>>>>>>> tolerances: relative=1e-05, absolute=1e-50, >>>>>>>>>>>>>>>> divergence=10000 >>>>>>>>>>>>>>>> LU: out-of-place factorization >>>>>>>>>>>>>>>> left preconditioning >>>>>>>>>>>>>>>> using NONE norm type for convergence test >>>>>>>>>>>>>>>> PC Object: (mg_coarse_sub_) 1 MPI >>>>>>>>>>>>>>>> processes >>>>>>>>>>>>>>>> type: lu >>>>>>>>>>>>>>>> tolerance for zero pivot 2.22045e-14 >>>>>>>>>>>>>>>> using diagonal shift on blocks to prevent zero pivot >>>>>>>>>>>>>>>> matrix ordering: nd >>>>>>>>>>>>>>>> LU: out-of-place factorization >>>>>>>>>>>>>>>> tolerance for zero pivot 2.22045e-14 >>>>>>>>>>>>>>>> using diagonal shift on blocks to prevent zero pivot >>>>>>>>>>>>>>>> matrix ordering: nd >>>>>>>>>>>>>>>> factor fill ratio given 5, needed 0 >>>>>>>>>>>>>>>> Factored matrix follows: >>>>>>>>>>>>>>>> factor fill ratio given 5, needed 4.13207 >>>>>>>>>>>>>>>> Factored matrix follows: >>>>>>>>>>>>>>>> Matrix Object: Matrix Object: >>>>>>>>>>>>>>>> 1 MPI processes >>>>>>>>>>>>>>>> type: seqaij >>>>>>>>>>>>>>>> rows=395, cols=395 >>>>>>>>>>>>>>>> package used to perform factorization: petsc >>>>>>>>>>>>>>>> total: nonzeros=132379, allocated >>>>>>>>>>>>>>>> nonzeros=132379 >>>>>>>>>>>>>>>> total number of mallocs used during >>>>>>>>>>>>>>>> MatSetValues calls =0 >>>>>>>>>>>>>>>> not using I-node routines >>>>>>>>>>>>>>>> 1 MPI processes >>>>>>>>>>>>>>>> type: seqaij >>>>>>>>>>>>>>>> linear system matrix = precond matrix: >>>>>>>>>>>>>>>> rows=0, cols=0 >>>>>>>>>>>>>>>> package used to perform factorization: petsc >>>>>>>>>>>>>>>> total: nonzeros=1, allocated nonzeros=1 >>>>>>>>>>>>>>>> total number of mallocs used during >>>>>>>>>>>>>>>> MatSetValues calls =0 >>>>>>>>>>>>>>>> not using I-node routines >>>>>>>>>>>>>>>> linear system matrix = precond matrix: >>>>>>>>>>>>>>>> Matrix Object: 1 MPI processes >>>>>>>>>>>>>>>> type: seqaij >>>>>>>>>>>>>>>> Matrix Object:KSP Object: 1 MPI processes >>>>>>>>>>>>>>>> type: seqaij >>>>>>>>>>>>>>>> rows=0, cols=0 >>>>>>>>>>>>>>>> total: nonzeros=0, allocated nonzeros=0 >>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues >>>>>>>>>>>>>>>> calls =0 >>>>>>>>>>>>>>>> not using I-node routines >>>>>>>>>>>>>>>> rows=395, cols=395 >>>>>>>>>>>>>>>> total: nonzeros=32037, allocated nonzeros=32037 >>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues >>>>>>>>>>>>>>>> calls =0 >>>>>>>>>>>>>>>> not using I-node routines >>>>>>>>>>>>>>>> - - - - - - - - - - - - - - - - - - >>>>>>>>>>>>>>>> KSP Object: (mg_coarse_sub_) 1 MPI >>>>>>>>>>>>>>>> processes >>>>>>>>>>>>>>>> type: preonly >>>>>>>>>>>>>>>> maximum iterations=1, initial guess is zero >>>>>>>>>>>>>>>> tolerances: relative=1e-05, absolute=1e-50, >>>>>>>>>>>>>>>> divergence=10000 >>>>>>>>>>>>>>>> left preconditioning >>>>>>>>>>>>>>>> using NONE norm type for convergence test >>>>>>>>>>>>>>>> PC Object: (mg_coarse_sub_) 1 MPI >>>>>>>>>>>>>>>> processes >>>>>>>>>>>>>>>> type: lu >>>>>>>>>>>>>>>> LU: out-of-place factorization >>>>>>>>>>>>>>>> tolerance for zero pivot 2.22045e-14 >>>>>>>>>>>>>>>> using diagonal shift on blocks to prevent zero pivot >>>>>>>>>>>>>>>> matrix ordering: nd >>>>>>>>>>>>>>>> factor fill ratio given 5, needed 0 >>>>>>>>>>>>>>>> Factored matrix follows: >>>>>>>>>>>>>>>> Matrix Object: 1 MPI processes >>>>>>>>>>>>>>>> type: seqaij >>>>>>>>>>>>>>>> rows=0, cols=0 >>>>>>>>>>>>>>>> package used to perform factorization: petsc >>>>>>>>>>>>>>>> total: nonzeros=1, allocated nonzeros=1 >>>>>>>>>>>>>>>> total number of mallocs used during >>>>>>>>>>>>>>>> MatSetValues calls =0 >>>>>>>>>>>>>>>> not using I-node routines >>>>>>>>>>>>>>>> linear system matrix = precond matrix: >>>>>>>>>>>>>>>> Matrix Object: 1 MPI processes >>>>>>>>>>>>>>>> type: seqaij >>>>>>>>>>>>>>>> rows=0, cols=0 >>>>>>>>>>>>>>>> total: nonzeros=0, allocated nonzeros=0 >>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues >>>>>>>>>>>>>>>> calls =0 >>>>>>>>>>>>>>>> not using I-node routines >>>>>>>>>>>>>>>> (mg_coarse_sub_) 1 MPI processes >>>>>>>>>>>>>>>> type: preonly >>>>>>>>>>>>>>>> maximum iterations=1, initial guess is zero >>>>>>>>>>>>>>>> tolerances: relative=1e-05, absolute=1e-50, >>>>>>>>>>>>>>>> divergence=10000 >>>>>>>>>>>>>>>> left preconditioning >>>>>>>>>>>>>>>> using NONE norm type for convergence test >>>>>>>>>>>>>>>> PC Object: (mg_coarse_sub_) 1 MPI >>>>>>>>>>>>>>>> processes >>>>>>>>>>>>>>>> type: lu >>>>>>>>>>>>>>>> LU: out-of-place factorization >>>>>>>>>>>>>>>> tolerance for zero pivot 2.22045e-14 >>>>>>>>>>>>>>>> using diagonal shift on blocks to prevent zero pivot >>>>>>>>>>>>>>>> matrix ordering: nd >>>>>>>>>>>>>>>> factor fill ratio given 5, needed 0 >>>>>>>>>>>>>>>> Factored matrix follows: >>>>>>>>>>>>>>>> Matrix Object: 1 MPI processes >>>>>>>>>>>>>>>> type: seqaij >>>>>>>>>>>>>>>> rows=0, cols=0 >>>>>>>>>>>>>>>> package used to perform factorization: petsc >>>>>>>>>>>>>>>> total: nonzeros=1, allocated nonzeros=1 >>>>>>>>>>>>>>>> total number of mallocs used during >>>>>>>>>>>>>>>> MatSetValues calls =0 >>>>>>>>>>>>>>>> not using I-node routines >>>>>>>>>>>>>>>> linear system matrix = precond matrix: >>>>>>>>>>>>>>>> Matrix Object: 1 MPI processes >>>>>>>>>>>>>>>> type: seqaij >>>>>>>>>>>>>>>> rows=0, cols=0 >>>>>>>>>>>>>>>> total: nonzeros=0, allocated nonzeros=0 >>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues >>>>>>>>>>>>>>>> calls =0 >>>>>>>>>>>>>>>> not using I-node routines >>>>>>>>>>>>>>>> [1] number of local blocks = 1, first local block number >>>>>>>>>>>>>>>> = 1 >>>>>>>>>>>>>>>> [1] local block number 0 >>>>>>>>>>>>>>>> - - - - - - - - - - - - - - - - - - >>>>>>>>>>>>>>>> [2] number of local blocks = 1, first local block number >>>>>>>>>>>>>>>> = 2 >>>>>>>>>>>>>>>> [2] local block number 0 >>>>>>>>>>>>>>>> - - - - - - - - - - - - - - - - - - >>>>>>>>>>>>>>>> [3] number of local blocks = 1, first local block number >>>>>>>>>>>>>>>> = 3 >>>>>>>>>>>>>>>> [3] local block number 0 >>>>>>>>>>>>>>>> - - - - - - - - - - - - - - - - - - >>>>>>>>>>>>>>>> linear system matrix = precond matrix: >>>>>>>>>>>>>>>> Matrix Object: 4 MPI processes >>>>>>>>>>>>>>>> type: mpiaij >>>>>>>>>>>>>>>> rows=395, cols=395 >>>>>>>>>>>>>>>> total: nonzeros=32037, allocated nonzeros=32037 >>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls >>>>>>>>>>>>>>>> =0 >>>>>>>>>>>>>>>> not using I-node (on process 0) routines >>>>>>>>>>>>>>>> Down solver (pre-smoother) on level 1 >>>>>>>>>>>>>>>> ------------------------------- >>>>>>>>>>>>>>>> KSP Object: (mg_levels_1_) 4 MPI processes >>>>>>>>>>>>>>>> type: chebyshev >>>>>>>>>>>>>>>> Chebyshev: eigenvalue estimates: min = 0.0636225, max >>>>>>>>>>>>>>>> = 1.33607 >>>>>>>>>>>>>>>> maximum iterations=2 >>>>>>>>>>>>>>>> tolerances: relative=1e-05, absolute=1e-50, >>>>>>>>>>>>>>>> divergence=10000 >>>>>>>>>>>>>>>> left preconditioning >>>>>>>>>>>>>>>> using nonzero initial guess >>>>>>>>>>>>>>>> using NONE norm type for convergence test >>>>>>>>>>>>>>>> PC Object: (mg_levels_1_) 4 MPI processes >>>>>>>>>>>>>>>> type: jacobi >>>>>>>>>>>>>>>> linear system matrix = precond matrix: >>>>>>>>>>>>>>>> Matrix Object: 4 MPI processes >>>>>>>>>>>>>>>> type: mpiaij >>>>>>>>>>>>>>>> rows=23918, cols=23918 >>>>>>>>>>>>>>>> total: nonzeros=818732, allocated nonzeros=818732 >>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls >>>>>>>>>>>>>>>> =0 >>>>>>>>>>>>>>>> not using I-node (on process 0) routines >>>>>>>>>>>>>>>> Up solver (post-smoother) same as down solver (pre-smoother) >>>>>>>>>>>>>>>> Down solver (pre-smoother) on level 2 >>>>>>>>>>>>>>>> ------------------------------- >>>>>>>>>>>>>>>> KSP Object: (mg_levels_2_) 4 MPI processes >>>>>>>>>>>>>>>> type: chebyshev >>>>>>>>>>>>>>>> Chebyshev: eigenvalue estimates: min = 0.0971369, max >>>>>>>>>>>>>>>> = 2.03987 >>>>>>>>>>>>>>>> maximum iterations=2 >>>>>>>>>>>>>>>> tolerances: relative=1e-05, absolute=1e-50, >>>>>>>>>>>>>>>> divergence=10000 >>>>>>>>>>>>>>>> left preconditioning >>>>>>>>>>>>>>>> using nonzero initial guess >>>>>>>>>>>>>>>> using NONE norm type for convergence test >>>>>>>>>>>>>>>> PC Object: (mg_levels_2_) 4 MPI processes >>>>>>>>>>>>>>>> type: jacobi >>>>>>>>>>>>>>>> linear system matrix = precond matrix: >>>>>>>>>>>>>>>> Matrix Object: 4 MPI processes >>>>>>>>>>>>>>>> type: mpiaij >>>>>>>>>>>>>>>> rows=262144, cols=262144 >>>>>>>>>>>>>>>> total: nonzeros=1835008, allocated nonzeros=1835008 >>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls >>>>>>>>>>>>>>>> =0 >>>>>>>>>>>>>>>> Up solver (post-smoother) same as down solver (pre-smoother) >>>>>>>>>>>>>>>> linear system matrix = precond matrix: >>>>>>>>>>>>>>>> Matrix Object: 4 MPI processes >>>>>>>>>>>>>>>> type: mpiaij >>>>>>>>>>>>>>>> rows=262144, cols=262144 >>>>>>>>>>>>>>>> total: nonzeros=1835008, allocated nonzeros=1835008 >>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls =0 >>>>>>>>>>>>>>>> #PETSc Option Table entries: >>>>>>>>>>>>>>>> -ksp_view >>>>>>>>>>>>>>>> -options_left >>>>>>>>>>>>>>>> -pc_gamg_agg_nsmooths 1 >>>>>>>>>>>>>>>> -pc_type gamg >>>>>>>>>>>>>>>> #End of PETSc Option Table entries >>>>>>>>>>>>>>>> There are no unused options. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thank you, >>>>>>>>>>>>>>>> Michele >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>> <test_poisson_solver.tar.gz><2decomp_fft-1.5.847-modified.tar.gz> >>>> >>> >> >
