Hi Mark,
We added the Jacobi line and it appears to accept that flag now. However, we
are getting all zeros for the solution vector. And PETSc is claiming to have
converged in 7 iterations to the relative tolerance.
-Ari
----- Original Message -----
From: "Mark Adams" <[email protected]>
To: "Ari Rappaport" <[email protected]>, "For users of the development version
of PETSc" <[email protected]>
Sent: Wednesday, July 27, 2016 3:26:03 PM
Subject: Re: [petsc-dev] Algebraic Multigrid
Please keep this on the Petsc list.
We seem to have lost Jacobi smoother again. I'm suspecting that you're some
funny character in your line with Jacobi that is Confusing the parser. Get back
to the old file with the two two Jacobi entries and delete the other line and
get Jacobi in the KSP view output. There should be no SOR in the output.
On Wednesday, July 27, 2016, Ari Rappaport < [email protected] > wrote:
Hi Mark,
I added all these new things. The PCAMG is now ending very quickly but the
residual is unreasonably large by about 10 orders of magnitude. I noticed the
line "Linear solve did not converge due to DIVERGED_INDEFINITE_PC iterations 2"
in the output, could this be causing the problem? It only appears to be going
for 2 iterations now.
-Ari
----- Original Message -----
From: "Mark Adams" < [email protected] >
To: "Ari Rappaport" < [email protected] >
Cc: "For users of the development version of PETSc" < [email protected] >
Sent: Tuesday, July 26, 2016 5:07:34 PM
Subject: Re: [petsc-dev] Algebraic Multigrid
Ari, I would also check that your operator is not messed up in parallel. The
solver is looking pretty solid.
Also, you can configure PETSc with hypre and use '-pc_type hypre'. If hypre is
also good in serial but hosed on multi-proc then it is most probably your
operator.
On Tue, Jul 26, 2016 at 6:58 PM, Mark Adams < [email protected] > wrote:
So remove one of the -mg_levels_pc_type jacobi and add -mg_coarse_ksp_type
preonly, then verify that this works on one proc and then try two procs.
On Tue, Jul 26, 2016 at 6:56 PM, Mark Adams < [email protected] > wrote:
Oh, actually this worked. You have this ...pc_type jacobi in there twice, so
one of them was "unused".
Try this with 2 processors now.
On Tue, Jul 26, 2016 at 6:42 PM, Mark Adams < [email protected] > wrote:
On Tue, Jul 26, 2016 at 6:24 PM, Ari Rappaport < [email protected] > wrote:
So I commented out the line PCSetType(pc, PCGAMG). The line
KSPSetFromOptions(ksp) was already in the code at the end of our initialization
routine. I also added .petscrc to the working dir. Here is the current output.
It seems as if Option left: name:-mg_levels_pc_type jacobi (no value) is still
present in the output..I dunno.
Yea, I dunno either. If you use -help you will get printout of the available
options. If you do this you will see stuff like -mg_levels_1_ ... you can also
see this in the ksp_view. There is a shortcut that lets you _not_ put "_1" in.
Try putting this in for each level like so:
-mg_levels_1_pc_type jacobi
-mg_levels_2_pc_type jacobi
-mg_levels_3_pc_type jacobi
I also notice that the coarse grid ksp is GMRES. This is our fault. It should
be preonly. Add:
-mg_coarse_ksp_type preonly
-Ari
----- Original Message -----
From: "Mark Adams" < [email protected] >
To: "Ari Rappaport" < [email protected] >, "For users of the development version
of PETSc" < [email protected] >
Sent: Tuesday, July 26, 2016 4:03:03 PM
Subject: Re: [petsc-dev] Algebraic Multigrid
At the end of this you have:
#PETSc Option Table entries:
-ksp_view
-mg_levels_pc_type jacobi
-options_left
#End of PETSc Option Table entries
There is one unused database option. It is:
Option left: name:-mg_levels_pc_type jacobi (no value)
So this jacobi parameter is not being used.
Do you call KPSSetFromOptions? Do you set solver parameters in the code? Like
PCGAMG?
You should not set anything in the code, it just confuses things at this point.
Use KSPSetFromOptions(). You can hardwire stuff before this call, this just
lets you set the defaults, but you should always call this last to let command
line parameters override the defaults.
You can put this in a .petscrc file in the working directory and try again.
-ksp_type cg
-ksp_max_it 50
-ksp_rtol 1.e-6
-ksp_converged_reason
-pc_type gamg
-pc_gamg_type agg
-pc_gamg_agg_nsmooths 1
-pc_gamg_coarse_eq_limit 10
-pc_gamg_reuse_interpolation true
-pc_gamg_square_graph 1
-pc_gamg_threshold -0.05
-mg_levels_ksp_max_it 2
-mg_levels_ksp_type chebyshev
-mg_levels_esteig_ksp_type cg
-mg_levels_esteig_ksp_max_it 10
-mg_levels_ksp_chebyshev_esteig 0,.05,0,1.05
-mg_levels_pc_type jacobi
-pc_hypre_type boomeramg
-pc_hypre_boomeramg_no_CF
-pc_hypre_boomeramg_agg_nl 1
-pc_hypre_boomeramg_coarsen_type HMIS
-pc_hypre_boomeramg_interp_type ext+i
LSP version 12.7, revision LSP_160707
Compiled Wed Jul 27 16:07:01 MDT 2016 on squash from /home/mantis/lsp
Compiler flags: -g -Werror -Wmissing-declarations -Wmissing-prototypes -Wcomment -Wuninitialized
Compiler options defined by user: -DCAR_X_Y -DSTATIC_FIELDS -DMATRIX_SOLUTION -DMULTI_PROCESS
Code options defined at compile-time:
CAR_X_Y
CHARGE_DENSITY
IMPLICIT_FIELDS
MATRIX_SOLUTION
MULTI_PROCESS
STATIC_FIELDS
STATIC_FIELDS_ADI
STATIC_IMPLICIT
Coordinate system used:
CARTESIAN
Input data file: sim.lsp
Simulation started on Wed Jul 27 16:13:48 2016
Standard courant limit for timestep is 4.717309e-03
R1 D0 #cells=832 #particles=100 (13 cell thickness)
R1 D1 #cells=884 #particles=0 (13 cell thickness)
R1 D2 #cells=832 #particles=0 (12 cell thickness)
R1 D3 #cells=780 #particles=0 (12 cell thickness)
Linear solve converged due to CONVERGED_RTOL iterations 7
KSP Object: 4 MPI processes
type: cg
maximum iterations=50
tolerances: relative=1e-06, absolute=1e-50, divergence=10000
left preconditioning
using nonzero initial guess
using PRECONDITIONED norm type for convergence test
PC Object: 4 MPI processes
type: gamg
MG: type is MULTIPLICATIVE, levels=4 cycles=v
Cycles per PCApply=1
Using Galerkin computed coarse grid matrices
GAMG specific options
Threshold for dropping small values from graph -0.05
AGG specific options
Symmetric graph false
Coarse grid solver -- level -------------------------------
KSP Object: (mg_coarse_) 4 MPI processes
type: gmres
GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
GMRES: happy breakdown tolerance 1e-30
maximum iterations=1, initial guess is zero
tolerances: relative=1e-05, absolute=1e-50, divergence=10000
left preconditioning
using NONE norm type for convergence test
PC Object: (mg_coarse_) 4 MPI processes
type: bjacobi
block Jacobi: number of blocks = 4
Local solve is same for all blocks, in the following KSP and PC objects:
KSP Object: (mg_coarse_sub_) 1 MPI processes
type: preonly
maximum iterations=1, initial guess is zero
tolerances: relative=1e-05, absolute=1e-50, divergence=10000
left preconditioning
using NONE norm type for convergence test
PC Object: (mg_coarse_sub_) 1 MPI processes
type: lu
LU: out-of-place factorization
tolerance for zero pivot 2.22045e-14
using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
matrix ordering: nd
factor fill ratio given 5, needed 1.03333
Factored matrix follows:
Mat Object: 1 MPI processes
type: seqaij
rows=8, cols=8
package used to perform factorization: petsc
total: nonzeros=62, allocated nonzeros=62
total number of mallocs used during MatSetValues calls =0
using I-node routines: found 4 nodes, limit used is 5
linear system matrix = precond matrix:
Mat Object: 1 MPI processes
type: seqaij
rows=8, cols=8
total: nonzeros=60, allocated nonzeros=60
total number of mallocs used during MatSetValues calls =0
using I-node routines: found 6 nodes, limit used is 5
linear system matrix = precond matrix:
Mat Object: 4 MPI processes
type: mpiaij
rows=8, cols=8
total: nonzeros=60, allocated nonzeros=60
total number of mallocs used during MatSetValues calls =0
using I-node (on process 0) routines: found 6 nodes, limit used is 5
Down solver (pre-smoother) on level 1 -------------------------------
KSP Object: (mg_levels_1_) 4 MPI processes
type: chebyshev
Chebyshev: eigenvalue estimates: min = 0.0793488, max = 1.66632
Chebyshev: eigenvalues estimated using cg with translations [0 0.05; 0 1.05]
KSP Object: (mg_levels_1_esteig_) 4 MPI processes
type: cg
maximum iterations=10, initial guess is zero
tolerances: relative=1e-05, absolute=1e-50, divergence=10000
left preconditioning
using NONE norm type for convergence test
maximum iterations=2
tolerances: relative=1e-05, absolute=1e-50, divergence=10000
left preconditioning
using nonzero initial guess
using NONE norm type for convergence test
PC Object: (mg_levels_1_) 4 MPI processes
type: jacobi
linear system matrix = precond matrix:
Mat Object: 4 MPI processes
type: mpiaij
rows=71, cols=71
total: nonzeros=1281, allocated nonzeros=1281
total number of mallocs used during MatSetValues calls =0
not using I-node (on process 0) routines
Up solver (post-smoother) same as down solver (pre-smoother)
Down solver (pre-smoother) on level 2 -------------------------------
KSP Object: (mg_levels_2_) 4 MPI processes
type: chebyshev
Chebyshev: eigenvalue estimates: min = 0.0865493, max = 1.81754
Chebyshev: eigenvalues estimated using cg with translations [0 0.05; 0 1.05]
KSP Object: (mg_levels_2_esteig_) 4 MPI processes
type: cg
maximum iterations=10, initial guess is zero
tolerances: relative=1e-05, absolute=1e-50, divergence=10000
left preconditioning
using NONE norm type for convergence test
maximum iterations=2
tolerances: relative=1e-05, absolute=1e-50, divergence=10000
left preconditioning
using nonzero initial guess
using NONE norm type for convergence test
PC Object: (mg_levels_2_) 4 MPI processes
type: jacobi
linear system matrix = precond matrix:
Mat Object: 4 MPI processes
type: mpiaij
rows=391, cols=391
total: nonzeros=3981, allocated nonzeros=3981
total number of mallocs used during MatSetValues calls =0
not using I-node (on process 0) routines
Up solver (post-smoother) same as down solver (pre-smoother)
Down solver (pre-smoother) on level 3 -------------------------------
KSP Object: (mg_levels_3_) 4 MPI processes
type: chebyshev
Chebyshev: eigenvalue estimates: min = 0.0945854, max = 1.98629
Chebyshev: eigenvalues estimated using cg with translations [0 0.05; 0 1.05]
KSP Object: (mg_levels_3_esteig_) 4 MPI processes
type: cg
maximum iterations=10, initial guess is zero
tolerances: relative=1e-05, absolute=1e-50, divergence=10000
left preconditioning
using NONE norm type for convergence test
maximum iterations=2
tolerances: relative=1e-05, absolute=1e-50, divergence=10000
left preconditioning
using nonzero initial guess
using NONE norm type for convergence test
PC Object: (mg_levels_3_) 4 MPI processes
type: jacobi
linear system matrix = precond matrix:
Mat Object: 4 MPI processes
type: mpiaij
rows=2601, cols=2601
total: nonzeros=12801, allocated nonzeros=15606
total number of mallocs used during MatSetValues calls =0
not using I-node (on process 0) routines
Up solver (post-smoother) same as down solver (pre-smoother)
linear system matrix = precond matrix:
Mat Object: 4 MPI processes
type: mpiaij
rows=2601, cols=2601
total: nonzeros=12801, allocated nonzeros=15606
total number of mallocs used during MatSetValues calls =0
not using I-node (on process 0) routines
Matrix static solution iteration=7
Step=1 Time=4.246e-03 Run-time=0.16(s) Number of particles=100
Step=1 Time=4.246e-03(ns) Run-time=0.16(s) at finish
R1 D0 #cells=832 #particles=100 (13 cell thickness)
R1 D1 #cells=884 #particles=0 (13 cell thickness)
R1 D2 #cells=832 #particles=0 (12 cell thickness)
R1 D3 #cells=780 #particles=0 (12 cell thickness)
Simulation terminated on Wed Jul 27 16:13:48 2016
#PETSc Option Table entries:
-ksp_converged_reason
-ksp_max_it 50
-ksp_rtol 1.e-6
-ksp_type cg
-ksp_view
-mg_coarse_ksp_type preonly
-mg_levels_esteig_ksp_max_it 10
-mg_levels_esteig_ksp_type cg
-mg_levels_ksp_chebyshev_esteig 0,.05,0,1.05
-mg_levels_ksp_max_it 2
-mg_levels_ksp_type chebyshev
-mg_levels_pc_type jacobi
-options_left
-pc_gamg_agg_nsmooths 1
-pc_gamg_coarse_eq_limit 10
-pc_gamg_reuse_interpolation true
-pc_gamg_square_graph 1
-pc_gamg_threshold -0.05
-pc_gamg_type agg
-pc_hypre_boomeramg_agg_nl 1
-pc_hypre_boomeramg_coarsen_type HMIS
-pc_hypre_boomeramg_interp_type ext+i
-pc_hypre_boomeramg_no_CF
-pc_hypre_type boomeramg
-pc_type gamg
#End of PETSc Option Table entries
There are 5 unused database options. They are:
Option left: name:-pc_hypre_boomeramg_agg_nl value: 1
Option left: name:-pc_hypre_boomeramg_coarsen_type value: HMIS
Option left: name:-pc_hypre_boomeramg_interp_type value: ext+i
Option left: name:-pc_hypre_boomeramg_no_CF (no value)
Option left: name:-pc_hypre_type value: boomeramg
