Re: [petsc-users] GAMG for the unsymmetrical matrix

[Kong, Fande]

Thanks, Mark,

Now, the total compute time using GAMG is competitive with ASM.  Looks like
I could not use something like: "-mg_level_1_ksp_type gmres" because this
option makes the compute time much worse.

Fande,

On Thu, Apr 13, 2017 at 9:14 AM, Mark Adams  wrote:

>
>
> On Wed, Apr 12, 2017 at 7:04 PM, Kong, Fande  wrote:
>
>>
>>
>> On Sun, Apr 9, 2017 at 6:04 AM, Mark Adams  wrote:
>>
>>> You seem to have two levels here and 3M eqs on the fine grid and 37 on
>>> the coarse grid. I don't understand that.
>>>
>>> You are also calling the AMG setup a lot, but not spending much time
>>> in it. Try running with -info and grep on "GAMG".
>>>
>>
>> I got the following output:
>>
>> [0] PCSetUp_GAMG(): level 0) N=3020875, n data rows=1, n data cols=1,
>> nnz/row (ave)=71, np=384
>> [0] PCGAMGFilterGraph():  100.% nnz after filtering, with threshold
>> 0., 73.6364 nnz ave. (N=3020875)
>> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
>> [0] PCGAMGProlongator_AGG(): New grid 18162 nodes
>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.978702e+00
>> min=2.559747e-02 PC=jacobi
>> [0] PCGAMGCreateLevel_GAMG(): Aggregate processors noop: new_size=384,
>> neq(loc)=40
>> [0] PCSetUp_GAMG(): 1) N=18162, n data cols=1, nnz/row (ave)=94, 384
>> active pes
>> [0] PCSetUp_GAMG(): 2 levels, grid complexity = 1.00795
>> [0] PCSetUp_GAMG(): level 0) N=3020875, n data rows=1, n data cols=1,
>> nnz/row (ave)=71, np=384
>> [0] PCGAMGFilterGraph():  100.% nnz after filtering, with threshold
>> 0., 73.6364 nnz ave. (N=3020875)
>> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
>> [0] PCGAMGProlongator_AGG(): New grid 18145 nodes
>> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.978584e+00
>> min=2.557887e-02 PC=jacobi
>> [0] PCGAMGCreateLevel_GAMG(): Aggregate processors noop: new_size=384,
>> neq(loc)=37
>> [0] PCSetUp_GAMG(): 1) N=18145, n data cols=1, nnz/row (ave)=94, 384
>> active pes
>>
>
> You are still doing two levels. Just use the parameters that I told you
> and you should see that 1) this coarsest (last) grid has "1 active pes" and
> 2) the overall solve time and overall convergence rate is much better.
>
>
>> [0] PCSetUp_GAMG(): 2 levels, grid complexity = 1.00792
>> GAMG specific options
>> PCGAMGGraph_AGG   40 1.0 8.0759e+00 1.0 3.56e+07 2.3 1.6e+06 1.9e+04
>> 7.6e+02  2  0  2  4  2   2  0  2  4  2  1170
>> PCGAMGCoarse_AGG  40 1.0 7.1698e+01 1.0 4.05e+09 2.3 4.0e+06 5.1e+04
>> 1.2e+03 18 37  5 27  3  18 37  5 27  3 14632
>> PCGAMGProl_AGG40 1.0 9.2650e-01 1.2 0.00e+00 0.0 9.8e+05 2.9e+03
>> 9.6e+02  0  0  1  0  2   0  0  1  0  2 0
>> PCGAMGPOpt_AGG40 1.0 2.4484e+00 1.0 4.72e+08 2.3 3.1e+06 2.3e+03
>> 1.9e+03  1  4  4  1  4   1  4  4  1  4 51328
>> GAMG: createProl  40 1.0 8.3786e+01 1.0 4.56e+09 2.3 9.6e+06 2.5e+04
>> 4.8e+03 21 42 12 32 10  21 42 12 32 10 14134
>> GAMG: partLevel   40 1.0 6.7755e+00 1.1 2.59e+08 2.3 2.9e+06 2.5e+03
>> 1.5e+03  2  2  4  1  3   2  2  4  1  3  9431
>>
>>
>>
>>
>>
>>
>>
>>
>>>
>>>
>>> On Fri, Apr 7, 2017 at 5:29 PM, Kong, Fande  wrote:
>>> > Thanks, Barry.
>>> >
>>> > It works.
>>> >
>>> > GAMG is three times better than ASM in terms of the number of linear
>>> > iterations, but it is five times slower than ASM. Any suggestions to
>>> improve
>>> > the performance of GAMG? Log files are attached.
>>> >
>>> > Fande,
>>> >
>>> > On Thu, Apr 6, 2017 at 3:39 PM, Barry Smith 
>>> wrote:
>>> >>
>>> >>
>>> >> > On Apr 6, 2017, at 9:39 AM, Kong, Fande  wrote:
>>> >> >
>>> >> > Thanks, Mark and Barry,
>>> >> >
>>> >> > It works pretty wells in terms of the number of linear iterations
>>> (using
>>> >> > "-pc_gamg_sym_graph true"), but it is horrible in the compute time.
>>> I am
>>> >> > using the two-level method via "-pc_mg_levels 2". The reason why
>>> the compute
>>> >> > time is larger than other preconditioning options is that a matrix
>>> free
>>> >> > method is used in the fine level and in my particular problem the
>>> function
>>> >> > evaluation is expensive.
>>> >> >
>>> >> > I am using "-snes_mf_operator 1" to turn on the Jacobian-free
>>> Newton,
>>> >> > but I do not think I want to make the preconditioning part
>>> matrix-free.  Do
>>> >> > you guys know how to turn off the matrix-free method for GAMG?
>>> >>
>>> >>-pc_use_amat false
>>> >>
>>> >> >
>>> >> > Here is the detailed solver:
>>> >> >
>>> >> > SNES Object: 384 MPI processes
>>> >> >   type: newtonls
>>> >> >   maximum iterations=200, maximum function evaluations=1
>>> >> >   tolerances: relative=1e-08, absolute=1e-08, solution=1e-50
>>> >> >   total number of linear solver iterations=20
>>> >> >   total number of function evaluations=166
>>> >> >   norm schedule ALWAYS
>>> >> >   SNESLineSearch Object:   384 MPI processes
>>> >> > type: bt
>>> >> >   interpolation: cubic
>>> >> >   

Re: [petsc-users] GAMG for the unsymmetrical matrix

[Mark Adams]

On Wed, Apr 12, 2017 at 7:04 PM, Kong, Fande  wrote:

>
>
> On Sun, Apr 9, 2017 at 6:04 AM, Mark Adams  wrote:
>
>> You seem to have two levels here and 3M eqs on the fine grid and 37 on
>> the coarse grid. I don't understand that.
>>
>> You are also calling the AMG setup a lot, but not spending much time
>> in it. Try running with -info and grep on "GAMG".
>>
>
> I got the following output:
>
> [0] PCSetUp_GAMG(): level 0) N=3020875, n data rows=1, n data cols=1,
> nnz/row (ave)=71, np=384
> [0] PCGAMGFilterGraph():  100.% nnz after filtering, with threshold
> 0., 73.6364 nnz ave. (N=3020875)
> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
> [0] PCGAMGProlongator_AGG(): New grid 18162 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.978702e+00
> min=2.559747e-02 PC=jacobi
> [0] PCGAMGCreateLevel_GAMG(): Aggregate processors noop: new_size=384,
> neq(loc)=40
> [0] PCSetUp_GAMG(): 1) N=18162, n data cols=1, nnz/row (ave)=94, 384
> active pes
> [0] PCSetUp_GAMG(): 2 levels, grid complexity = 1.00795
> [0] PCSetUp_GAMG(): level 0) N=3020875, n data rows=1, n data cols=1,
> nnz/row (ave)=71, np=384
> [0] PCGAMGFilterGraph():  100.% nnz after filtering, with threshold
> 0., 73.6364 nnz ave. (N=3020875)
> [0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
> [0] PCGAMGProlongator_AGG(): New grid 18145 nodes
> [0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.978584e+00
> min=2.557887e-02 PC=jacobi
> [0] PCGAMGCreateLevel_GAMG(): Aggregate processors noop: new_size=384,
> neq(loc)=37
> [0] PCSetUp_GAMG(): 1) N=18145, n data cols=1, nnz/row (ave)=94, 384
> active pes
>

You are still doing two levels. Just use the parameters that I told you and
you should see that 1) this coarsest (last) grid has "1 active pes" and 2)
the overall solve time and overall convergence rate is much better.


> [0] PCSetUp_GAMG(): 2 levels, grid complexity = 1.00792
> GAMG specific options
> PCGAMGGraph_AGG   40 1.0 8.0759e+00 1.0 3.56e+07 2.3 1.6e+06 1.9e+04
> 7.6e+02  2  0  2  4  2   2  0  2  4  2  1170
> PCGAMGCoarse_AGG  40 1.0 7.1698e+01 1.0 4.05e+09 2.3 4.0e+06 5.1e+04
> 1.2e+03 18 37  5 27  3  18 37  5 27  3 14632
> PCGAMGProl_AGG40 1.0 9.2650e-01 1.2 0.00e+00 0.0 9.8e+05 2.9e+03
> 9.6e+02  0  0  1  0  2   0  0  1  0  2 0
> PCGAMGPOpt_AGG40 1.0 2.4484e+00 1.0 4.72e+08 2.3 3.1e+06 2.3e+03
> 1.9e+03  1  4  4  1  4   1  4  4  1  4 51328
> GAMG: createProl  40 1.0 8.3786e+01 1.0 4.56e+09 2.3 9.6e+06 2.5e+04
> 4.8e+03 21 42 12 32 10  21 42 12 32 10 14134
> GAMG: partLevel   40 1.0 6.7755e+00 1.1 2.59e+08 2.3 2.9e+06 2.5e+03
> 1.5e+03  2  2  4  1  3   2  2  4  1  3  9431
>
>
>
>
>
>
>
>
>>
>>
>> On Fri, Apr 7, 2017 at 5:29 PM, Kong, Fande  wrote:
>> > Thanks, Barry.
>> >
>> > It works.
>> >
>> > GAMG is three times better than ASM in terms of the number of linear
>> > iterations, but it is five times slower than ASM. Any suggestions to
>> improve
>> > the performance of GAMG? Log files are attached.
>> >
>> > Fande,
>> >
>> > On Thu, Apr 6, 2017 at 3:39 PM, Barry Smith  wrote:
>> >>
>> >>
>> >> > On Apr 6, 2017, at 9:39 AM, Kong, Fande  wrote:
>> >> >
>> >> > Thanks, Mark and Barry,
>> >> >
>> >> > It works pretty wells in terms of the number of linear iterations
>> (using
>> >> > "-pc_gamg_sym_graph true"), but it is horrible in the compute time.
>> I am
>> >> > using the two-level method via "-pc_mg_levels 2". The reason why the
>> compute
>> >> > time is larger than other preconditioning options is that a matrix
>> free
>> >> > method is used in the fine level and in my particular problem the
>> function
>> >> > evaluation is expensive.
>> >> >
>> >> > I am using "-snes_mf_operator 1" to turn on the Jacobian-free Newton,
>> >> > but I do not think I want to make the preconditioning part
>> matrix-free.  Do
>> >> > you guys know how to turn off the matrix-free method for GAMG?
>> >>
>> >>-pc_use_amat false
>> >>
>> >> >
>> >> > Here is the detailed solver:
>> >> >
>> >> > SNES Object: 384 MPI processes
>> >> >   type: newtonls
>> >> >   maximum iterations=200, maximum function evaluations=1
>> >> >   tolerances: relative=1e-08, absolute=1e-08, solution=1e-50
>> >> >   total number of linear solver iterations=20
>> >> >   total number of function evaluations=166
>> >> >   norm schedule ALWAYS
>> >> >   SNESLineSearch Object:   384 MPI processes
>> >> > type: bt
>> >> >   interpolation: cubic
>> >> >   alpha=1.00e-04
>> >> > maxstep=1.00e+08, minlambda=1.00e-12
>> >> > tolerances: relative=1.00e-08, absolute=1.00e-15,
>> >> > lambda=1.00e-08
>> >> > maximum iterations=40
>> >> >   KSP Object:   384 MPI processes
>> >> > type: gmres
>> >> >   GMRES: restart=100, using Classical (unmodified) Gram-Schmidt
>> >> > Orthogonalization with no iterative refinement
>> >> >   GMRES: 

Re: [petsc-users] GAMG for the unsymmetrical matrix

[Mark Adams]

On Wed, Apr 12, 2017 at 1:31 PM, Kong, Fande  wrote:

> Hi Mark,
>
> Thanks for your reply.
>
> On Wed, Apr 12, 2017 at 9:16 AM, Mark Adams  wrote:
>
>> The problem comes from setting the number of MG levels (-pc_mg_levels 2).
>> Not your fault, it looks like the GAMG logic is faulty, in your version at
>> least.
>>
>
> What I want is that GAMG coarsens the fine matrix once and then stops
> doing anything.  I did not see any benefits to have more levels if the
> number of processors is small.
>

The number of levels is a math issue and has nothing to do with
parallelism. If you do just one level your coarse grid is very large and
expensive to solve, so you want to keep coarsening. There is rarely a need
to use -pc_mg_levels


>
>
>>
>> GAMG will force the coarsest grid to one processor by default, in newer
>> versions. You can override the default with:
>>
>> -pc_gamg_use_parallel_coarse_grid_solver
>>
>> Your coarse grid solver is ASM with these 37 equation per process and 512
>> processes. That is bad.
>>
>
> Why this is bad? The subdomain problem is too small?
>

Because ASM with 512 blocks is a weak solver. You want the coarse grid to
be solved exactly.


>
>
>> Note, you could run this on one process to see the proper convergence
>> rate.
>>
>
> Convergence rate for which part? coarse solver, subdomain solver?
>

The overall converge rate.


>
>
>> You can fix this with parameters:
>>
>> >   -pc_gamg_process_eq_limit <50>: Limit (goal) on number of equations
>> per process on coarse grids (PCGAMGSetProcEqLim)
>> >   -pc_gamg_coarse_eq_limit <50>: Limit on number of equations for the
>> coarse grid (PCGAMGSetCoarseEqLim)
>>
>> If you really want two levels then set something like
>> -pc_gamg_coarse_eq_limit 18145 (or higher) -pc_gamg_coarse_eq_limit 18145
>> (or higher).
>>
>
>
> May have something like: make the coarse problem 1/8 large as the original
> problem? Otherwise, this number is just problem dependent.
>

GAMG will stop automatically so that you do not need problem dependant
parameters.


>
>
>
>> You can run with -info and grep on GAMG and you will meta-data for each
>> level. you should see "npe=1" for the coarsest, last, grid. Or use a
>> parallel direct solver.
>>
>
> I will try.
>
>
>>
>> Note, you should not see much degradation as you increase the number of
>> levels. 18145 eqs on a 3D problem will probably be noticeable. I generally
>> aim for about 3000.
>>
>
> It should be fine as long as the coarse problem is solved by a parallel
> solver.
>

>
> Fande,
>
>
>>
>>
>> On Mon, Apr 10, 2017 at 12:17 PM, Kong, Fande  wrote:
>>
>>>
>>>
>>> On Sun, Apr 9, 2017 at 6:04 AM, Mark Adams  wrote:
>>>
 You seem to have two levels here and 3M eqs on the fine grid and 37 on
 the coarse grid.
>>>
>>>
>>> 37 is on the sub domain.
>>>
>>>  rows=18145, cols=18145 on the entire coarse grid.
>>>
>>>
>>>
>>>
>>>
 I don't understand that.

 You are also calling the AMG setup a lot, but not spending much time
 in it. Try running with -info and grep on "GAMG".


 On Fri, Apr 7, 2017 at 5:29 PM, Kong, Fande  wrote:
 > Thanks, Barry.
 >
 > It works.
 >
 > GAMG is three times better than ASM in terms of the number of linear
 > iterations, but it is five times slower than ASM. Any suggestions to
 improve
 > the performance of GAMG? Log files are attached.
 >
 > Fande,
 >
 > On Thu, Apr 6, 2017 at 3:39 PM, Barry Smith 
 wrote:
 >>
 >>
 >> > On Apr 6, 2017, at 9:39 AM, Kong, Fande 
 wrote:
 >> >
 >> > Thanks, Mark and Barry,
 >> >
 >> > It works pretty wells in terms of the number of linear iterations
 (using
 >> > "-pc_gamg_sym_graph true"), but it is horrible in the compute
 time. I am
 >> > using the two-level method via "-pc_mg_levels 2". The reason why
 the compute
 >> > time is larger than other preconditioning options is that a matrix
 free
 >> > method is used in the fine level and in my particular problem the
 function
 >> > evaluation is expensive.
 >> >
 >> > I am using "-snes_mf_operator 1" to turn on the Jacobian-free
 Newton,
 >> > but I do not think I want to make the preconditioning part
 matrix-free.  Do
 >> > you guys know how to turn off the matrix-free method for GAMG?
 >>
 >>-pc_use_amat false
 >>
 >> >
 >> > Here is the detailed solver:
 >> >
 >> > SNES Object: 384 MPI processes
 >> >   type: newtonls
 >> >   maximum iterations=200, maximum function evaluations=1
 >> >   tolerances: relative=1e-08, absolute=1e-08, solution=1e-50
 >> >   total number of linear solver iterations=20
 >> >   total number of function evaluations=166
 >> >   norm schedule ALWAYS
 >> >   SNESLineSearch Object:   384 MPI processes
 >> 

Re: [petsc-users] GAMG for the unsymmetrical matrix

[Kong, Fande]

On Sun, Apr 9, 2017 at 6:04 AM, Mark Adams  wrote:

> You seem to have two levels here and 3M eqs on the fine grid and 37 on
> the coarse grid. I don't understand that.
>
> You are also calling the AMG setup a lot, but not spending much time
> in it. Try running with -info and grep on "GAMG".
>

I got the following output:

[0] PCSetUp_GAMG(): level 0) N=3020875, n data rows=1, n data cols=1,
nnz/row (ave)=71, np=384
[0] PCGAMGFilterGraph():  100.% nnz after filtering, with threshold 0.,
73.6364 nnz ave. (N=3020875)
[0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
[0] PCGAMGProlongator_AGG(): New grid 18162 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.978702e+00
min=2.559747e-02 PC=jacobi
[0] PCGAMGCreateLevel_GAMG(): Aggregate processors noop: new_size=384,
neq(loc)=40
[0] PCSetUp_GAMG(): 1) N=18162, n data cols=1, nnz/row (ave)=94, 384 active
pes
[0] PCSetUp_GAMG(): 2 levels, grid complexity = 1.00795
[0] PCSetUp_GAMG(): level 0) N=3020875, n data rows=1, n data cols=1,
nnz/row (ave)=71, np=384
[0] PCGAMGFilterGraph():  100.% nnz after filtering, with threshold 0.,
73.6364 nnz ave. (N=3020875)
[0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
[0] PCGAMGProlongator_AGG(): New grid 18145 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.978584e+00
min=2.557887e-02 PC=jacobi
[0] PCGAMGCreateLevel_GAMG(): Aggregate processors noop: new_size=384,
neq(loc)=37
[0] PCSetUp_GAMG(): 1) N=18145, n data cols=1, nnz/row (ave)=94, 384 active
pes
[0] PCSetUp_GAMG(): 2 levels, grid complexity = 1.00792
GAMG specific options
PCGAMGGraph_AGG   40 1.0 8.0759e+00 1.0 3.56e+07 2.3 1.6e+06 1.9e+04
7.6e+02  2  0  2  4  2   2  0  2  4  2  1170
PCGAMGCoarse_AGG  40 1.0 7.1698e+01 1.0 4.05e+09 2.3 4.0e+06 5.1e+04
1.2e+03 18 37  5 27  3  18 37  5 27  3 14632
PCGAMGProl_AGG40 1.0 9.2650e-01 1.2 0.00e+00 0.0 9.8e+05 2.9e+03
9.6e+02  0  0  1  0  2   0  0  1  0  2 0
PCGAMGPOpt_AGG40 1.0 2.4484e+00 1.0 4.72e+08 2.3 3.1e+06 2.3e+03
1.9e+03  1  4  4  1  4   1  4  4  1  4 51328
GAMG: createProl  40 1.0 8.3786e+01 1.0 4.56e+09 2.3 9.6e+06 2.5e+04
4.8e+03 21 42 12 32 10  21 42 12 32 10 14134
GAMG: partLevel   40 1.0 6.7755e+00 1.1 2.59e+08 2.3 2.9e+06 2.5e+03
1.5e+03  2  2  4  1  3   2  2  4  1  3  9431








>
>
> On Fri, Apr 7, 2017 at 5:29 PM, Kong, Fande  wrote:
> > Thanks, Barry.
> >
> > It works.
> >
> > GAMG is three times better than ASM in terms of the number of linear
> > iterations, but it is five times slower than ASM. Any suggestions to
> improve
> > the performance of GAMG? Log files are attached.
> >
> > Fande,
> >
> > On Thu, Apr 6, 2017 at 3:39 PM, Barry Smith  wrote:
> >>
> >>
> >> > On Apr 6, 2017, at 9:39 AM, Kong, Fande  wrote:
> >> >
> >> > Thanks, Mark and Barry,
> >> >
> >> > It works pretty wells in terms of the number of linear iterations
> (using
> >> > "-pc_gamg_sym_graph true"), but it is horrible in the compute time. I
> am
> >> > using the two-level method via "-pc_mg_levels 2". The reason why the
> compute
> >> > time is larger than other preconditioning options is that a matrix
> free
> >> > method is used in the fine level and in my particular problem the
> function
> >> > evaluation is expensive.
> >> >
> >> > I am using "-snes_mf_operator 1" to turn on the Jacobian-free Newton,
> >> > but I do not think I want to make the preconditioning part
> matrix-free.  Do
> >> > you guys know how to turn off the matrix-free method for GAMG?
> >>
> >>-pc_use_amat false
> >>
> >> >
> >> > Here is the detailed solver:
> >> >
> >> > SNES Object: 384 MPI processes
> >> >   type: newtonls
> >> >   maximum iterations=200, maximum function evaluations=1
> >> >   tolerances: relative=1e-08, absolute=1e-08, solution=1e-50
> >> >   total number of linear solver iterations=20
> >> >   total number of function evaluations=166
> >> >   norm schedule ALWAYS
> >> >   SNESLineSearch Object:   384 MPI processes
> >> > type: bt
> >> >   interpolation: cubic
> >> >   alpha=1.00e-04
> >> > maxstep=1.00e+08, minlambda=1.00e-12
> >> > tolerances: relative=1.00e-08, absolute=1.00e-15,
> >> > lambda=1.00e-08
> >> > maximum iterations=40
> >> >   KSP Object:   384 MPI processes
> >> > type: gmres
> >> >   GMRES: restart=100, using Classical (unmodified) Gram-Schmidt
> >> > Orthogonalization with no iterative refinement
> >> >   GMRES: happy breakdown tolerance 1e-30
> >> > maximum iterations=100, initial guess is zero
> >> > tolerances:  relative=0.001, absolute=1e-50, divergence=1.
> >> > right preconditioning
> >> > using UNPRECONDITIONED norm type for convergence test
> >> >   PC Object:   384 MPI processes
> >> > type: gamg
> >> >   MG: type is MULTIPLICATIVE, levels=2 cycles=v
> >> > Cycles per PCApply=1
> >> > Using Galerkin computed coarse 

Re: [petsc-users] GAMG for the unsymmetrical matrix

[Kong, Fande]

Hi Mark,

Thanks for your reply.

On Wed, Apr 12, 2017 at 9:16 AM, Mark Adams  wrote:

> The problem comes from setting the number of MG levels (-pc_mg_levels 2).
> Not your fault, it looks like the GAMG logic is faulty, in your version at
> least.
>

What I want is that GAMG coarsens the fine matrix once and then stops doing
anything.  I did not see any benefits to have more levels if the number of
processors is small.


>
> GAMG will force the coarsest grid to one processor by default, in newer
> versions. You can override the default with:
>
> -pc_gamg_use_parallel_coarse_grid_solver
>
> Your coarse grid solver is ASM with these 37 equation per process and 512
> processes. That is bad.
>

Why this is bad? The subdomain problem is too small?


> Note, you could run this on one process to see the proper convergence
> rate.
>

Convergence rate for which part? coarse solver, subdomain solver?


> You can fix this with parameters:
>
> >   -pc_gamg_process_eq_limit <50>: Limit (goal) on number of equations
> per process on coarse grids (PCGAMGSetProcEqLim)
> >   -pc_gamg_coarse_eq_limit <50>: Limit on number of equations for the
> coarse grid (PCGAMGSetCoarseEqLim)
>
> If you really want two levels then set something like
> -pc_gamg_coarse_eq_limit 18145 (or higher) -pc_gamg_coarse_eq_limit 18145
> (or higher).
>


May have something like: make the coarse problem 1/8 large as the original
problem? Otherwise, this number is just problem dependent.



> You can run with -info and grep on GAMG and you will meta-data for each
> level. you should see "npe=1" for the coarsest, last, grid. Or use a
> parallel direct solver.
>

I will try.


>
> Note, you should not see much degradation as you increase the number of
> levels. 18145 eqs on a 3D problem will probably be noticeable. I generally
> aim for about 3000.
>

It should be fine as long as the coarse problem is solved by a parallel
solver.


Fande,


>
>
> On Mon, Apr 10, 2017 at 12:17 PM, Kong, Fande  wrote:
>
>>
>>
>> On Sun, Apr 9, 2017 at 6:04 AM, Mark Adams  wrote:
>>
>>> You seem to have two levels here and 3M eqs on the fine grid and 37 on
>>> the coarse grid.
>>
>>
>> 37 is on the sub domain.
>>
>>  rows=18145, cols=18145 on the entire coarse grid.
>>
>>
>>
>>
>>
>>> I don't understand that.
>>>
>>> You are also calling the AMG setup a lot, but not spending much time
>>> in it. Try running with -info and grep on "GAMG".
>>>
>>>
>>> On Fri, Apr 7, 2017 at 5:29 PM, Kong, Fande  wrote:
>>> > Thanks, Barry.
>>> >
>>> > It works.
>>> >
>>> > GAMG is three times better than ASM in terms of the number of linear
>>> > iterations, but it is five times slower than ASM. Any suggestions to
>>> improve
>>> > the performance of GAMG? Log files are attached.
>>> >
>>> > Fande,
>>> >
>>> > On Thu, Apr 6, 2017 at 3:39 PM, Barry Smith 
>>> wrote:
>>> >>
>>> >>
>>> >> > On Apr 6, 2017, at 9:39 AM, Kong, Fande  wrote:
>>> >> >
>>> >> > Thanks, Mark and Barry,
>>> >> >
>>> >> > It works pretty wells in terms of the number of linear iterations
>>> (using
>>> >> > "-pc_gamg_sym_graph true"), but it is horrible in the compute time.
>>> I am
>>> >> > using the two-level method via "-pc_mg_levels 2". The reason why
>>> the compute
>>> >> > time is larger than other preconditioning options is that a matrix
>>> free
>>> >> > method is used in the fine level and in my particular problem the
>>> function
>>> >> > evaluation is expensive.
>>> >> >
>>> >> > I am using "-snes_mf_operator 1" to turn on the Jacobian-free
>>> Newton,
>>> >> > but I do not think I want to make the preconditioning part
>>> matrix-free.  Do
>>> >> > you guys know how to turn off the matrix-free method for GAMG?
>>> >>
>>> >>-pc_use_amat false
>>> >>
>>> >> >
>>> >> > Here is the detailed solver:
>>> >> >
>>> >> > SNES Object: 384 MPI processes
>>> >> >   type: newtonls
>>> >> >   maximum iterations=200, maximum function evaluations=1
>>> >> >   tolerances: relative=1e-08, absolute=1e-08, solution=1e-50
>>> >> >   total number of linear solver iterations=20
>>> >> >   total number of function evaluations=166
>>> >> >   norm schedule ALWAYS
>>> >> >   SNESLineSearch Object:   384 MPI processes
>>> >> > type: bt
>>> >> >   interpolation: cubic
>>> >> >   alpha=1.00e-04
>>> >> > maxstep=1.00e+08, minlambda=1.00e-12
>>> >> > tolerances: relative=1.00e-08, absolute=1.00e-15,
>>> >> > lambda=1.00e-08
>>> >> > maximum iterations=40
>>> >> >   KSP Object:   384 MPI processes
>>> >> > type: gmres
>>> >> >   GMRES: restart=100, using Classical (unmodified) Gram-Schmidt
>>> >> > Orthogonalization with no iterative refinement
>>> >> >   GMRES: happy breakdown tolerance 1e-30
>>> >> > maximum iterations=100, initial guess is zero
>>> >> > tolerances:  relative=0.001, absolute=1e-50, divergence=1.
>>> >> > right preconditioning

Re: [petsc-users] GAMG for the unsymmetrical matrix

[Mark Adams]

The problem comes from setting the number of MG levels (-pc_mg_levels 2).
Not your fault, it looks like the GAMG logic is faulty, in your version at
least.

GAMG will force the coarsest grid to one processor by default, in newer
versions. You can override the default with:

-pc_gamg_use_parallel_coarse_grid_solver

Your coarse grid solver is ASM with these 37 equation per process and 512
processes. That is bad. Note, you could run this on one process to see the
proper convergence rate.  You can fix this with parameters:

>   -pc_gamg_process_eq_limit <50>: Limit (goal) on number of equations per
process on coarse grids (PCGAMGSetProcEqLim)
>   -pc_gamg_coarse_eq_limit <50>: Limit on number of equations for the
coarse grid (PCGAMGSetCoarseEqLim)

If you really want two levels then set something like
-pc_gamg_coarse_eq_limit 18145 (or higher) -pc_gamg_coarse_eq_limit 18145
(or higher). You can run with -info and grep on GAMG and you will meta-data
for each level. you should see "npe=1" for the coarsest, last, grid. Or use
a parallel direct solver.

Note, you should not see much degradation as you increase the number of
levels. 18145 eqs on a 3D problem will probably be noticeable. I generally
aim for about 3000.


On Mon, Apr 10, 2017 at 12:17 PM, Kong, Fande  wrote:

>
>
> On Sun, Apr 9, 2017 at 6:04 AM, Mark Adams  wrote:
>
>> You seem to have two levels here and 3M eqs on the fine grid and 37 on
>> the coarse grid.
>
>
> 37 is on the sub domain.
>
>  rows=18145, cols=18145 on the entire coarse grid.
>
>
>
>
>
>> I don't understand that.
>>
>> You are also calling the AMG setup a lot, but not spending much time
>> in it. Try running with -info and grep on "GAMG".
>>
>>
>> On Fri, Apr 7, 2017 at 5:29 PM, Kong, Fande  wrote:
>> > Thanks, Barry.
>> >
>> > It works.
>> >
>> > GAMG is three times better than ASM in terms of the number of linear
>> > iterations, but it is five times slower than ASM. Any suggestions to
>> improve
>> > the performance of GAMG? Log files are attached.
>> >
>> > Fande,
>> >
>> > On Thu, Apr 6, 2017 at 3:39 PM, Barry Smith  wrote:
>> >>
>> >>
>> >> > On Apr 6, 2017, at 9:39 AM, Kong, Fande  wrote:
>> >> >
>> >> > Thanks, Mark and Barry,
>> >> >
>> >> > It works pretty wells in terms of the number of linear iterations
>> (using
>> >> > "-pc_gamg_sym_graph true"), but it is horrible in the compute time.
>> I am
>> >> > using the two-level method via "-pc_mg_levels 2". The reason why the
>> compute
>> >> > time is larger than other preconditioning options is that a matrix
>> free
>> >> > method is used in the fine level and in my particular problem the
>> function
>> >> > evaluation is expensive.
>> >> >
>> >> > I am using "-snes_mf_operator 1" to turn on the Jacobian-free Newton,
>> >> > but I do not think I want to make the preconditioning part
>> matrix-free.  Do
>> >> > you guys know how to turn off the matrix-free method for GAMG?
>> >>
>> >>-pc_use_amat false
>> >>
>> >> >
>> >> > Here is the detailed solver:
>> >> >
>> >> > SNES Object: 384 MPI processes
>> >> >   type: newtonls
>> >> >   maximum iterations=200, maximum function evaluations=1
>> >> >   tolerances: relative=1e-08, absolute=1e-08, solution=1e-50
>> >> >   total number of linear solver iterations=20
>> >> >   total number of function evaluations=166
>> >> >   norm schedule ALWAYS
>> >> >   SNESLineSearch Object:   384 MPI processes
>> >> > type: bt
>> >> >   interpolation: cubic
>> >> >   alpha=1.00e-04
>> >> > maxstep=1.00e+08, minlambda=1.00e-12
>> >> > tolerances: relative=1.00e-08, absolute=1.00e-15,
>> >> > lambda=1.00e-08
>> >> > maximum iterations=40
>> >> >   KSP Object:   384 MPI processes
>> >> > type: gmres
>> >> >   GMRES: restart=100, using Classical (unmodified) Gram-Schmidt
>> >> > Orthogonalization with no iterative refinement
>> >> >   GMRES: happy breakdown tolerance 1e-30
>> >> > maximum iterations=100, initial guess is zero
>> >> > tolerances:  relative=0.001, absolute=1e-50, divergence=1.
>> >> > right preconditioning
>> >> > using UNPRECONDITIONED norm type for convergence test
>> >> >   PC Object:   384 MPI processes
>> >> > type: gamg
>> >> >   MG: type is MULTIPLICATIVE, levels=2 cycles=v
>> >> > Cycles per PCApply=1
>> >> > Using Galerkin computed coarse grid matrices
>> >> > GAMG specific options
>> >> >   Threshold for dropping small values from graph 0.
>> >> >   AGG specific options
>> >> > Symmetric graph true
>> >> > Coarse grid solver -- level ---
>> >> >   KSP Object:  (mg_coarse_)   384 MPI processes
>> >> > type: preonly
>> >> > maximum iterations=1, initial guess is zero
>> >> > tolerances:  relative=1e-05, absolute=1e-50,
>> divergence=1.
>> >> > left 

Re: [petsc-users] GAMG for the unsymmetrical matrix

[Kong, Fande]

On Sun, Apr 9, 2017 at 6:04 AM, Mark Adams  wrote:

> You seem to have two levels here and 3M eqs on the fine grid and 37 on
> the coarse grid.


37 is on the sub domain.

 rows=18145, cols=18145 on the entire coarse grid.





> I don't understand that.
>
> You are also calling the AMG setup a lot, but not spending much time
> in it. Try running with -info and grep on "GAMG".
>
>
> On Fri, Apr 7, 2017 at 5:29 PM, Kong, Fande  wrote:
> > Thanks, Barry.
> >
> > It works.
> >
> > GAMG is three times better than ASM in terms of the number of linear
> > iterations, but it is five times slower than ASM. Any suggestions to
> improve
> > the performance of GAMG? Log files are attached.
> >
> > Fande,
> >
> > On Thu, Apr 6, 2017 at 3:39 PM, Barry Smith  wrote:
> >>
> >>
> >> > On Apr 6, 2017, at 9:39 AM, Kong, Fande  wrote:
> >> >
> >> > Thanks, Mark and Barry,
> >> >
> >> > It works pretty wells in terms of the number of linear iterations
> (using
> >> > "-pc_gamg_sym_graph true"), but it is horrible in the compute time. I
> am
> >> > using the two-level method via "-pc_mg_levels 2". The reason why the
> compute
> >> > time is larger than other preconditioning options is that a matrix
> free
> >> > method is used in the fine level and in my particular problem the
> function
> >> > evaluation is expensive.
> >> >
> >> > I am using "-snes_mf_operator 1" to turn on the Jacobian-free Newton,
> >> > but I do not think I want to make the preconditioning part
> matrix-free.  Do
> >> > you guys know how to turn off the matrix-free method for GAMG?
> >>
> >>-pc_use_amat false
> >>
> >> >
> >> > Here is the detailed solver:
> >> >
> >> > SNES Object: 384 MPI processes
> >> >   type: newtonls
> >> >   maximum iterations=200, maximum function evaluations=1
> >> >   tolerances: relative=1e-08, absolute=1e-08, solution=1e-50
> >> >   total number of linear solver iterations=20
> >> >   total number of function evaluations=166
> >> >   norm schedule ALWAYS
> >> >   SNESLineSearch Object:   384 MPI processes
> >> > type: bt
> >> >   interpolation: cubic
> >> >   alpha=1.00e-04
> >> > maxstep=1.00e+08, minlambda=1.00e-12
> >> > tolerances: relative=1.00e-08, absolute=1.00e-15,
> >> > lambda=1.00e-08
> >> > maximum iterations=40
> >> >   KSP Object:   384 MPI processes
> >> > type: gmres
> >> >   GMRES: restart=100, using Classical (unmodified) Gram-Schmidt
> >> > Orthogonalization with no iterative refinement
> >> >   GMRES: happy breakdown tolerance 1e-30
> >> > maximum iterations=100, initial guess is zero
> >> > tolerances:  relative=0.001, absolute=1e-50, divergence=1.
> >> > right preconditioning
> >> > using UNPRECONDITIONED norm type for convergence test
> >> >   PC Object:   384 MPI processes
> >> > type: gamg
> >> >   MG: type is MULTIPLICATIVE, levels=2 cycles=v
> >> > Cycles per PCApply=1
> >> > Using Galerkin computed coarse grid matrices
> >> > GAMG specific options
> >> >   Threshold for dropping small values from graph 0.
> >> >   AGG specific options
> >> > Symmetric graph true
> >> > Coarse grid solver -- level ---
> >> >   KSP Object:  (mg_coarse_)   384 MPI processes
> >> > type: preonly
> >> > maximum iterations=1, initial guess is zero
> >> > tolerances:  relative=1e-05, absolute=1e-50, divergence=1.
> >> > left preconditioning
> >> > using NONE norm type for convergence test
> >> >   PC Object:  (mg_coarse_)   384 MPI processes
> >> > type: bjacobi
> >> >   block Jacobi: number of blocks = 384
> >> >   Local solve is same for all blocks, in the following KSP and
> >> > PC objects:
> >> > KSP Object:(mg_coarse_sub_) 1 MPI processes
> >> >   type: preonly
> >> >   maximum iterations=1, initial guess is zero
> >> >   tolerances:  relative=1e-05, absolute=1e-50,
> divergence=1.
> >> >   left preconditioning
> >> >   using NONE norm type for convergence test
> >> > PC Object:(mg_coarse_sub_) 1 MPI processes
> >> >   type: lu
> >> > LU: out-of-place factorization
> >> > tolerance for zero pivot 2.22045e-14
> >> > using diagonal shift on blocks to prevent zero pivot
> >> > [INBLOCKS]
> >> > matrix ordering: nd
> >> > factor fill ratio given 5., needed 1.31367
> >> >   Factored matrix follows:
> >> > Mat Object: 1 MPI processes
> >> >   type: seqaij
> >> >   rows=37, cols=37
> >> >   package used to perform factorization: petsc
> >> >   total: nonzeros=913, allocated nonzeros=913
> >> >   

Re: [petsc-users] GAMG for the unsymmetrical matrix

[Mark Adams]

You seem to have two levels here and 3M eqs on the fine grid and 37 on
the coarse grid. I don't understand that.

You are also calling the AMG setup a lot, but not spending much time
in it. Try running with -info and grep on "GAMG".


On Fri, Apr 7, 2017 at 5:29 PM, Kong, Fande  wrote:
> Thanks, Barry.
>
> It works.
>
> GAMG is three times better than ASM in terms of the number of linear
> iterations, but it is five times slower than ASM. Any suggestions to improve
> the performance of GAMG? Log files are attached.
>
> Fande,
>
> On Thu, Apr 6, 2017 at 3:39 PM, Barry Smith  wrote:
>>
>>
>> > On Apr 6, 2017, at 9:39 AM, Kong, Fande  wrote:
>> >
>> > Thanks, Mark and Barry,
>> >
>> > It works pretty wells in terms of the number of linear iterations (using
>> > "-pc_gamg_sym_graph true"), but it is horrible in the compute time. I am
>> > using the two-level method via "-pc_mg_levels 2". The reason why the 
>> > compute
>> > time is larger than other preconditioning options is that a matrix free
>> > method is used in the fine level and in my particular problem the function
>> > evaluation is expensive.
>> >
>> > I am using "-snes_mf_operator 1" to turn on the Jacobian-free Newton,
>> > but I do not think I want to make the preconditioning part matrix-free.  Do
>> > you guys know how to turn off the matrix-free method for GAMG?
>>
>>-pc_use_amat false
>>
>> >
>> > Here is the detailed solver:
>> >
>> > SNES Object: 384 MPI processes
>> >   type: newtonls
>> >   maximum iterations=200, maximum function evaluations=1
>> >   tolerances: relative=1e-08, absolute=1e-08, solution=1e-50
>> >   total number of linear solver iterations=20
>> >   total number of function evaluations=166
>> >   norm schedule ALWAYS
>> >   SNESLineSearch Object:   384 MPI processes
>> > type: bt
>> >   interpolation: cubic
>> >   alpha=1.00e-04
>> > maxstep=1.00e+08, minlambda=1.00e-12
>> > tolerances: relative=1.00e-08, absolute=1.00e-15,
>> > lambda=1.00e-08
>> > maximum iterations=40
>> >   KSP Object:   384 MPI processes
>> > type: gmres
>> >   GMRES: restart=100, using Classical (unmodified) Gram-Schmidt
>> > Orthogonalization with no iterative refinement
>> >   GMRES: happy breakdown tolerance 1e-30
>> > maximum iterations=100, initial guess is zero
>> > tolerances:  relative=0.001, absolute=1e-50, divergence=1.
>> > right preconditioning
>> > using UNPRECONDITIONED norm type for convergence test
>> >   PC Object:   384 MPI processes
>> > type: gamg
>> >   MG: type is MULTIPLICATIVE, levels=2 cycles=v
>> > Cycles per PCApply=1
>> > Using Galerkin computed coarse grid matrices
>> > GAMG specific options
>> >   Threshold for dropping small values from graph 0.
>> >   AGG specific options
>> > Symmetric graph true
>> > Coarse grid solver -- level ---
>> >   KSP Object:  (mg_coarse_)   384 MPI processes
>> > type: preonly
>> > maximum iterations=1, initial guess is zero
>> > tolerances:  relative=1e-05, absolute=1e-50, divergence=1.
>> > left preconditioning
>> > using NONE norm type for convergence test
>> >   PC Object:  (mg_coarse_)   384 MPI processes
>> > type: bjacobi
>> >   block Jacobi: number of blocks = 384
>> >   Local solve is same for all blocks, in the following KSP and
>> > PC objects:
>> > KSP Object:(mg_coarse_sub_) 1 MPI processes
>> >   type: preonly
>> >   maximum iterations=1, initial guess is zero
>> >   tolerances:  relative=1e-05, absolute=1e-50, divergence=1.
>> >   left preconditioning
>> >   using NONE norm type for convergence test
>> > PC Object:(mg_coarse_sub_) 1 MPI processes
>> >   type: lu
>> > LU: out-of-place factorization
>> > tolerance for zero pivot 2.22045e-14
>> > using diagonal shift on blocks to prevent zero pivot
>> > [INBLOCKS]
>> > matrix ordering: nd
>> > factor fill ratio given 5., needed 1.31367
>> >   Factored matrix follows:
>> > Mat Object: 1 MPI processes
>> >   type: seqaij
>> >   rows=37, cols=37
>> >   package used to perform factorization: petsc
>> >   total: nonzeros=913, allocated nonzeros=913
>> >   total number of mallocs used during MatSetValues calls
>> > =0
>> > not using I-node routines
>> >   linear system matrix = precond matrix:
>> >   Mat Object:   1 MPI processes
>> > type: seqaij
>> > rows=37, cols=37
>> > total: nonzeros=695, allocated nonzeros=695
>> > total number 

Re: [petsc-users] GAMG for the unsymmetrical matrix

[Kong, Fande]

On Fri, Apr 7, 2017 at 3:52 PM, Barry Smith  wrote:

>
> > On Apr 7, 2017, at 4:46 PM, Kong, Fande  wrote:
> >
> >
> >
> > On Fri, Apr 7, 2017 at 3:39 PM, Barry Smith  wrote:
> >
> >   Using Petsc Release Version 3.7.5, unknown
> >
> >So are you using the release or are you using master branch?
> >
> > I am working on the maint branch.
> >
> > I did something two months ago:
> >
> >  git clone -b maint https://urldefense.proofpoint.
> com/v2/url?u=https-3A__bitbucket.org_petsc_petsc=DwIFAg=
> 54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00=DUUt3SRGI0_
> JgtNaS3udV68GRkgV4ts7XKfj2opmiCY=c92UNplDTVgzFrXIn_
> 70buWa2rXPGUKN083_aJYI0FQ=yrulwZxJiduZc-703r7PJOUApPDehsFIkhS0BTrroXc=
> petsc.
> >
> >
> > I am interested to improve the GAMG performance.
>
>   Why, why not use the best solver for your problem?
>

I am just curious. I want to understand the potential of interesting
preconditioners.



>
> > Is it possible? It can not beat ASM at all? The multilevel method should
> be better than the one-level if the number of processor cores is large.
>
>The ASM is taking 30 iterations, this is fantastic, it is really going
> to be tough to get GAMG to be faster (set up time for GAMG is high).
>
>What happens to both with 10 times as many processes? 100 times as many?
>


Did not try many processes yet.

Fande,



>
>
>Barry
>
> >
> > Fande,
> >
> >
> >If you use master the ASM will be even faster.
> >
> > What's new in master?
> >
> >
> > Fande,
> >
> >
> >
> > > On Apr 7, 2017, at 4:29 PM, Kong, Fande  wrote:
> > >
> > > Thanks, Barry.
> > >
> > > It works.
> > >
> > > GAMG is three times better than ASM in terms of the number of linear
> iterations, but it is five times slower than ASM. Any suggestions to
> improve the performance of GAMG? Log files are attached.
> > >
> > > Fande,
> > >
> > > On Thu, Apr 6, 2017 at 3:39 PM, Barry Smith 
> wrote:
> > >
> > > > On Apr 6, 2017, at 9:39 AM, Kong, Fande  wrote:
> > > >
> > > > Thanks, Mark and Barry,
> > > >
> > > > It works pretty wells in terms of the number of linear iterations
> (using "-pc_gamg_sym_graph true"), but it is horrible in the compute time.
> I am using the two-level method via "-pc_mg_levels 2". The reason why the
> compute time is larger than other preconditioning options is that a matrix
> free method is used in the fine level and in my particular problem the
> function evaluation is expensive.
> > > >
> > > > I am using "-snes_mf_operator 1" to turn on the Jacobian-free
> Newton, but I do not think I want to make the preconditioning part
> matrix-free.  Do you guys know how to turn off the matrix-free method for
> GAMG?
> > >
> > >-pc_use_amat false
> > >
> > > >
> > > > Here is the detailed solver:
> > > >
> > > > SNES Object: 384 MPI processes
> > > >   type: newtonls
> > > >   maximum iterations=200, maximum function evaluations=1
> > > >   tolerances: relative=1e-08, absolute=1e-08, solution=1e-50
> > > >   total number of linear solver iterations=20
> > > >   total number of function evaluations=166
> > > >   norm schedule ALWAYS
> > > >   SNESLineSearch Object:   384 MPI processes
> > > > type: bt
> > > >   interpolation: cubic
> > > >   alpha=1.00e-04
> > > > maxstep=1.00e+08, minlambda=1.00e-12
> > > > tolerances: relative=1.00e-08, absolute=1.00e-15,
> lambda=1.00e-08
> > > > maximum iterations=40
> > > >   KSP Object:   384 MPI processes
> > > > type: gmres
> > > >   GMRES: restart=100, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> > > >   GMRES: happy breakdown tolerance 1e-30
> > > > maximum iterations=100, initial guess is zero
> > > > tolerances:  relative=0.001, absolute=1e-50, divergence=1.
> > > > right preconditioning
> > > > using UNPRECONDITIONED norm type for convergence test
> > > >   PC Object:   384 MPI processes
> > > > type: gamg
> > > >   MG: type is MULTIPLICATIVE, levels=2 cycles=v
> > > > Cycles per PCApply=1
> > > > Using Galerkin computed coarse grid matrices
> > > > GAMG specific options
> > > >   Threshold for dropping small values from graph 0.
> > > >   AGG specific options
> > > > Symmetric graph true
> > > > Coarse grid solver -- level ---
> > > >   KSP Object:  (mg_coarse_)   384 MPI processes
> > > > type: preonly
> > > > maximum iterations=1, initial guess is zero
> > > > tolerances:  relative=1e-05, absolute=1e-50,
> divergence=1.
> > > > left preconditioning
> > > > using NONE norm type for convergence test
> > > >   PC Object:  (mg_coarse_)   384 MPI processes
> > > > type: bjacobi
> > > >   block Jacobi: number of blocks = 384
> > > >   Local solve is 

Re: [petsc-users] GAMG for the unsymmetrical matrix

[Barry Smith]

> On Apr 7, 2017, at 4:46 PM, Kong, Fande  wrote:
> 
> 
> 
> On Fri, Apr 7, 2017 at 3:39 PM, Barry Smith  wrote:
> 
>   Using Petsc Release Version 3.7.5, unknown
> 
>So are you using the release or are you using master branch?
> 
> I am working on the maint branch. 
> 
> I did something two months ago:
> 
>  git clone -b maint https://bitbucket.org/petsc/petsc petsc.
> 
> 
> I am interested to improve the GAMG performance.

  Why, why not use the best solver for your problem?

> Is it possible? It can not beat ASM at all? The multilevel method should be 
> better than the one-level if the number of processor cores is large.

   The ASM is taking 30 iterations, this is fantastic, it is really going to be 
tough to get GAMG to be faster (set up time for GAMG is high).

   What happens to both with 10 times as many processes? 100 times as many?


   Barry

> 
> Fande,
>  
> 
>If you use master the ASM will be even faster.
> 
> What's new in master?
> 
> 
> Fande,
>  
> 
> 
> > On Apr 7, 2017, at 4:29 PM, Kong, Fande  wrote:
> >
> > Thanks, Barry.
> >
> > It works.
> >
> > GAMG is three times better than ASM in terms of the number of linear 
> > iterations, but it is five times slower than ASM. Any suggestions to 
> > improve the performance of GAMG? Log files are attached.
> >
> > Fande,
> >
> > On Thu, Apr 6, 2017 at 3:39 PM, Barry Smith  wrote:
> >
> > > On Apr 6, 2017, at 9:39 AM, Kong, Fande  wrote:
> > >
> > > Thanks, Mark and Barry,
> > >
> > > It works pretty wells in terms of the number of linear iterations (using 
> > > "-pc_gamg_sym_graph true"), but it is horrible in the compute time. I am 
> > > using the two-level method via "-pc_mg_levels 2". The reason why the 
> > > compute time is larger than other preconditioning options is that a 
> > > matrix free method is used in the fine level and in my particular problem 
> > > the function evaluation is expensive.
> > >
> > > I am using "-snes_mf_operator 1" to turn on the Jacobian-free Newton, but 
> > > I do not think I want to make the preconditioning part matrix-free.  Do 
> > > you guys know how to turn off the matrix-free method for GAMG?
> >
> >-pc_use_amat false
> >
> > >
> > > Here is the detailed solver:
> > >
> > > SNES Object: 384 MPI processes
> > >   type: newtonls
> > >   maximum iterations=200, maximum function evaluations=1
> > >   tolerances: relative=1e-08, absolute=1e-08, solution=1e-50
> > >   total number of linear solver iterations=20
> > >   total number of function evaluations=166
> > >   norm schedule ALWAYS
> > >   SNESLineSearch Object:   384 MPI processes
> > > type: bt
> > >   interpolation: cubic
> > >   alpha=1.00e-04
> > > maxstep=1.00e+08, minlambda=1.00e-12
> > > tolerances: relative=1.00e-08, absolute=1.00e-15, 
> > > lambda=1.00e-08
> > > maximum iterations=40
> > >   KSP Object:   384 MPI processes
> > > type: gmres
> > >   GMRES: restart=100, using Classical (unmodified) Gram-Schmidt 
> > > Orthogonalization with no iterative refinement
> > >   GMRES: happy breakdown tolerance 1e-30
> > > maximum iterations=100, initial guess is zero
> > > tolerances:  relative=0.001, absolute=1e-50, divergence=1.
> > > right preconditioning
> > > using UNPRECONDITIONED norm type for convergence test
> > >   PC Object:   384 MPI processes
> > > type: gamg
> > >   MG: type is MULTIPLICATIVE, levels=2 cycles=v
> > > Cycles per PCApply=1
> > > Using Galerkin computed coarse grid matrices
> > > GAMG specific options
> > >   Threshold for dropping small values from graph 0.
> > >   AGG specific options
> > > Symmetric graph true
> > > Coarse grid solver -- level ---
> > >   KSP Object:  (mg_coarse_)   384 MPI processes
> > > type: preonly
> > > maximum iterations=1, initial guess is zero
> > > tolerances:  relative=1e-05, absolute=1e-50, divergence=1.
> > > left preconditioning
> > > using NONE norm type for convergence test
> > >   PC Object:  (mg_coarse_)   384 MPI processes
> > > type: bjacobi
> > >   block Jacobi: number of blocks = 384
> > >   Local solve is same for all blocks, in the following KSP and PC 
> > > objects:
> > > KSP Object:(mg_coarse_sub_) 1 MPI processes
> > >   type: preonly
> > >   maximum iterations=1, initial guess is zero
> > >   tolerances:  relative=1e-05, absolute=1e-50, divergence=1.
> > >   left preconditioning
> > >   using NONE norm type for convergence test
> > > PC Object:(mg_coarse_sub_) 1 MPI processes
> > >   type: lu
> > > LU: out-of-place factorization
> > > tolerance for zero pivot 

Re: [petsc-users] GAMG for the unsymmetrical matrix

[Barry Smith]

  Using Petsc Release Version 3.7.5, unknown 

   So are you using the release or are you using master branch?

   If you use master the ASM will be even faster.


> On Apr 7, 2017, at 4:29 PM, Kong, Fande  wrote:
> 
> Thanks, Barry.
> 
> It works.
> 
> GAMG is three times better than ASM in terms of the number of linear 
> iterations, but it is five times slower than ASM. Any suggestions to improve 
> the performance of GAMG? Log files are attached.
> 
> Fande,
> 
> On Thu, Apr 6, 2017 at 3:39 PM, Barry Smith  wrote:
> 
> > On Apr 6, 2017, at 9:39 AM, Kong, Fande  wrote:
> >
> > Thanks, Mark and Barry,
> >
> > It works pretty wells in terms of the number of linear iterations (using 
> > "-pc_gamg_sym_graph true"), but it is horrible in the compute time. I am 
> > using the two-level method via "-pc_mg_levels 2". The reason why the 
> > compute time is larger than other preconditioning options is that a matrix 
> > free method is used in the fine level and in my particular problem the 
> > function evaluation is expensive.
> >
> > I am using "-snes_mf_operator 1" to turn on the Jacobian-free Newton, but I 
> > do not think I want to make the preconditioning part matrix-free.  Do you 
> > guys know how to turn off the matrix-free method for GAMG?
> 
>-pc_use_amat false
> 
> >
> > Here is the detailed solver:
> >
> > SNES Object: 384 MPI processes
> >   type: newtonls
> >   maximum iterations=200, maximum function evaluations=1
> >   tolerances: relative=1e-08, absolute=1e-08, solution=1e-50
> >   total number of linear solver iterations=20
> >   total number of function evaluations=166
> >   norm schedule ALWAYS
> >   SNESLineSearch Object:   384 MPI processes
> > type: bt
> >   interpolation: cubic
> >   alpha=1.00e-04
> > maxstep=1.00e+08, minlambda=1.00e-12
> > tolerances: relative=1.00e-08, absolute=1.00e-15, 
> > lambda=1.00e-08
> > maximum iterations=40
> >   KSP Object:   384 MPI processes
> > type: gmres
> >   GMRES: restart=100, using Classical (unmodified) Gram-Schmidt 
> > Orthogonalization with no iterative refinement
> >   GMRES: happy breakdown tolerance 1e-30
> > maximum iterations=100, initial guess is zero
> > tolerances:  relative=0.001, absolute=1e-50, divergence=1.
> > right preconditioning
> > using UNPRECONDITIONED norm type for convergence test
> >   PC Object:   384 MPI processes
> > type: gamg
> >   MG: type is MULTIPLICATIVE, levels=2 cycles=v
> > Cycles per PCApply=1
> > Using Galerkin computed coarse grid matrices
> > GAMG specific options
> >   Threshold for dropping small values from graph 0.
> >   AGG specific options
> > Symmetric graph true
> > Coarse grid solver -- level ---
> >   KSP Object:  (mg_coarse_)   384 MPI processes
> > type: preonly
> > maximum iterations=1, initial guess is zero
> > tolerances:  relative=1e-05, absolute=1e-50, divergence=1.
> > left preconditioning
> > using NONE norm type for convergence test
> >   PC Object:  (mg_coarse_)   384 MPI processes
> > type: bjacobi
> >   block Jacobi: number of blocks = 384
> >   Local solve is same for all blocks, in the following KSP and PC 
> > objects:
> > KSP Object:(mg_coarse_sub_) 1 MPI processes
> >   type: preonly
> >   maximum iterations=1, initial guess is zero
> >   tolerances:  relative=1e-05, absolute=1e-50, divergence=1.
> >   left preconditioning
> >   using NONE norm type for convergence test
> > PC Object:(mg_coarse_sub_) 1 MPI processes
> >   type: lu
> > LU: out-of-place factorization
> > tolerance for zero pivot 2.22045e-14
> > using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
> > matrix ordering: nd
> > factor fill ratio given 5., needed 1.31367
> >   Factored matrix follows:
> > Mat Object: 1 MPI processes
> >   type: seqaij
> >   rows=37, cols=37
> >   package used to perform factorization: petsc
> >   total: nonzeros=913, allocated nonzeros=913
> >   total number of mallocs used during MatSetValues calls =0
> > not using I-node routines
> >   linear system matrix = precond matrix:
> >   Mat Object:   1 MPI processes
> > type: seqaij
> > rows=37, cols=37
> > total: nonzeros=695, allocated nonzeros=695
> > total number of mallocs used during MatSetValues calls =0
> >   not using I-node routines
> > linear system matrix = precond matrix:
> > Mat Object: 

Re: [petsc-users] GAMG for the unsymmetrical matrix

[Kong, Fande]

Thanks, Barry.

It works.

GAMG is three times better than ASM in terms of the number of linear
iterations, but it is five times slower than ASM. Any suggestions to
improve the performance of GAMG? Log files are attached.

Fande,

On Thu, Apr 6, 2017 at 3:39 PM, Barry Smith  wrote:

>
> > On Apr 6, 2017, at 9:39 AM, Kong, Fande  wrote:
> >
> > Thanks, Mark and Barry,
> >
> > It works pretty wells in terms of the number of linear iterations (using
> "-pc_gamg_sym_graph true"), but it is horrible in the compute time. I am
> using the two-level method via "-pc_mg_levels 2". The reason why the
> compute time is larger than other preconditioning options is that a matrix
> free method is used in the fine level and in my particular problem the
> function evaluation is expensive.
> >
> > I am using "-snes_mf_operator 1" to turn on the Jacobian-free Newton,
> but I do not think I want to make the preconditioning part matrix-free.  Do
> you guys know how to turn off the matrix-free method for GAMG?
>
>-pc_use_amat false
>
> >
> > Here is the detailed solver:
> >
> > SNES Object: 384 MPI processes
> >   type: newtonls
> >   maximum iterations=200, maximum function evaluations=1
> >   tolerances: relative=1e-08, absolute=1e-08, solution=1e-50
> >   total number of linear solver iterations=20
> >   total number of function evaluations=166
> >   norm schedule ALWAYS
> >   SNESLineSearch Object:   384 MPI processes
> > type: bt
> >   interpolation: cubic
> >   alpha=1.00e-04
> > maxstep=1.00e+08, minlambda=1.00e-12
> > tolerances: relative=1.00e-08, absolute=1.00e-15,
> lambda=1.00e-08
> > maximum iterations=40
> >   KSP Object:   384 MPI processes
> > type: gmres
> >   GMRES: restart=100, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> >   GMRES: happy breakdown tolerance 1e-30
> > maximum iterations=100, initial guess is zero
> > tolerances:  relative=0.001, absolute=1e-50, divergence=1.
> > right preconditioning
> > using UNPRECONDITIONED norm type for convergence test
> >   PC Object:   384 MPI processes
> > type: gamg
> >   MG: type is MULTIPLICATIVE, levels=2 cycles=v
> > Cycles per PCApply=1
> > Using Galerkin computed coarse grid matrices
> > GAMG specific options
> >   Threshold for dropping small values from graph 0.
> >   AGG specific options
> > Symmetric graph true
> > Coarse grid solver -- level ---
> >   KSP Object:  (mg_coarse_)   384 MPI processes
> > type: preonly
> > maximum iterations=1, initial guess is zero
> > tolerances:  relative=1e-05, absolute=1e-50, divergence=1.
> > left preconditioning
> > using NONE norm type for convergence test
> >   PC Object:  (mg_coarse_)   384 MPI processes
> > type: bjacobi
> >   block Jacobi: number of blocks = 384
> >   Local solve is same for all blocks, in the following KSP and
> PC objects:
> > KSP Object:(mg_coarse_sub_) 1 MPI processes
> >   type: preonly
> >   maximum iterations=1, initial guess is zero
> >   tolerances:  relative=1e-05, absolute=1e-50, divergence=1.
> >   left preconditioning
> >   using NONE norm type for convergence test
> > PC Object:(mg_coarse_sub_) 1 MPI processes
> >   type: lu
> > LU: out-of-place factorization
> > tolerance for zero pivot 2.22045e-14
> > using diagonal shift on blocks to prevent zero pivot
> [INBLOCKS]
> > matrix ordering: nd
> > factor fill ratio given 5., needed 1.31367
> >   Factored matrix follows:
> > Mat Object: 1 MPI processes
> >   type: seqaij
> >   rows=37, cols=37
> >   package used to perform factorization: petsc
> >   total: nonzeros=913, allocated nonzeros=913
> >   total number of mallocs used during MatSetValues calls
> =0
> > not using I-node routines
> >   linear system matrix = precond matrix:
> >   Mat Object:   1 MPI processes
> > type: seqaij
> > rows=37, cols=37
> > total: nonzeros=695, allocated nonzeros=695
> > total number of mallocs used during MatSetValues calls =0
> >   not using I-node routines
> > linear system matrix = precond matrix:
> > Mat Object: 384 MPI processes
> >   type: mpiaij
> >   rows=18145, cols=18145
> >   total: nonzeros=1709115, allocated nonzeros=1709115
> >   total number of mallocs used during MatSetValues calls =0
> > not using I-node (on process 0) routines
> >  

Re: [petsc-users] GAMG for the unsymmetrical matrix

[Barry Smith]

> On Apr 6, 2017, at 9:39 AM, Kong, Fande  wrote:
> 
> Thanks, Mark and Barry,
> 
> It works pretty wells in terms of the number of linear iterations (using 
> "-pc_gamg_sym_graph true"), but it is horrible in the compute time. I am 
> using the two-level method via "-pc_mg_levels 2". The reason why the compute 
> time is larger than other preconditioning options is that a matrix free 
> method is used in the fine level and in my particular problem the function 
> evaluation is expensive. 
> 
> I am using "-snes_mf_operator 1" to turn on the Jacobian-free Newton, but I 
> do not think I want to make the preconditioning part matrix-free.  Do you 
> guys know how to turn off the matrix-free method for GAMG?

   -pc_use_amat false

> 
> Here is the detailed solver:
> 
> SNES Object: 384 MPI processes
>   type: newtonls
>   maximum iterations=200, maximum function evaluations=1
>   tolerances: relative=1e-08, absolute=1e-08, solution=1e-50
>   total number of linear solver iterations=20
>   total number of function evaluations=166
>   norm schedule ALWAYS
>   SNESLineSearch Object:   384 MPI processes
> type: bt
>   interpolation: cubic
>   alpha=1.00e-04
> maxstep=1.00e+08, minlambda=1.00e-12
> tolerances: relative=1.00e-08, absolute=1.00e-15, 
> lambda=1.00e-08
> maximum iterations=40
>   KSP Object:   384 MPI processes
> type: gmres
>   GMRES: restart=100, using Classical (unmodified) Gram-Schmidt 
> Orthogonalization with no iterative refinement
>   GMRES: happy breakdown tolerance 1e-30
> maximum iterations=100, initial guess is zero
> tolerances:  relative=0.001, absolute=1e-50, divergence=1.
> right preconditioning
> using UNPRECONDITIONED norm type for convergence test
>   PC Object:   384 MPI processes
> type: gamg
>   MG: type is MULTIPLICATIVE, levels=2 cycles=v
> Cycles per PCApply=1
> Using Galerkin computed coarse grid matrices
> GAMG specific options
>   Threshold for dropping small values from graph 0.
>   AGG specific options
> Symmetric graph true
> Coarse grid solver -- level ---
>   KSP Object:  (mg_coarse_)   384 MPI processes
> type: preonly
> maximum iterations=1, initial guess is zero
> tolerances:  relative=1e-05, absolute=1e-50, divergence=1.
> left preconditioning
> using NONE norm type for convergence test
>   PC Object:  (mg_coarse_)   384 MPI processes
> type: bjacobi
>   block Jacobi: number of blocks = 384
>   Local solve is same for all blocks, in the following KSP and PC 
> objects:
> KSP Object:(mg_coarse_sub_) 1 MPI processes
>   type: preonly
>   maximum iterations=1, initial guess is zero
>   tolerances:  relative=1e-05, absolute=1e-50, divergence=1.
>   left preconditioning
>   using NONE norm type for convergence test
> PC Object:(mg_coarse_sub_) 1 MPI processes
>   type: lu
> LU: out-of-place factorization
> tolerance for zero pivot 2.22045e-14
> using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
> matrix ordering: nd
> factor fill ratio given 5., needed 1.31367
>   Factored matrix follows:
> Mat Object: 1 MPI processes
>   type: seqaij
>   rows=37, cols=37
>   package used to perform factorization: petsc
>   total: nonzeros=913, allocated nonzeros=913
>   total number of mallocs used during MatSetValues calls =0
> not using I-node routines
>   linear system matrix = precond matrix:
>   Mat Object:   1 MPI processes
> type: seqaij
> rows=37, cols=37
> total: nonzeros=695, allocated nonzeros=695
> total number of mallocs used during MatSetValues calls =0
>   not using I-node routines
> linear system matrix = precond matrix:
> Mat Object: 384 MPI processes
>   type: mpiaij
>   rows=18145, cols=18145
>   total: nonzeros=1709115, allocated nonzeros=1709115
>   total number of mallocs used during MatSetValues calls =0
> not using I-node (on process 0) routines
> Down solver (pre-smoother) on level 1 ---
>   KSP Object:  (mg_levels_1_)   384 MPI processes
> type: chebyshev
>   Chebyshev: eigenvalue estimates:  min = 0.19, max = 1.46673
>   Chebyshev: eigenvalues estimated using gmres with translations  [0. 
> 0.1; 0. 1.1]
>   KSP Object:  (mg_levels_1_esteig_)   384 MPI 
> processes
> type: gmres
>   GMRES: 

Re: [petsc-users] GAMG for the unsymmetrical matrix

[Mark Adams]

On Thu, Apr 6, 2017 at 7:39 AM, Kong, Fande  wrote:
> Thanks, Mark and Barry,
>
> It works pretty wells in terms of the number of linear iterations (using
> "-pc_gamg_sym_graph true"), but it is horrible in the compute time. I am
> using the two-level method via "-pc_mg_levels 2". The reason why the compute
> time is larger than other preconditioning options is that a matrix free
> method is used in the fine level and in my particular problem the function
> evaluation is expensive.
>
> I am using "-snes_mf_operator 1" to turn on the Jacobian-free Newton, but I
> do not think I want to make the preconditioning part matrix-free.  Do you
> guys know how to turn off the matrix-free method for GAMG?

You do have an option to use the operator or the preconditioner
operator (matrix) for the fine grid smoother, but I thought it uses
the PC matrix by default. I don't recall the parameters nor do I see
this in the view output.  Others should be able to help.

Re: [petsc-users] GAMG for the unsymmetrical matrix

[Kong, Fande]

Thanks, Mark and Barry,

It works pretty wells in terms of the number of linear iterations (using
"-pc_gamg_sym_graph true"), but it is horrible in the compute time. I am
using the two-level method via "-pc_mg_levels 2". The reason why the
compute time is larger than other preconditioning options is that a matrix
free method is used in the fine level and in my particular problem the
function evaluation is expensive.

I am using "-snes_mf_operator 1" to turn on the Jacobian-free Newton, but I
do not think I want to make the preconditioning part matrix-free.  Do you
guys know how to turn off the matrix-free method for GAMG?

Here is the detailed solver:































































































































*SNES Object: 384 MPI processes  type: newtonls  maximum iterations=200,
maximum function evaluations=1  tolerances: relative=1e-08,
absolute=1e-08, solution=1e-50  total number of linear solver
iterations=20  total number of function evaluations=166  norm schedule
ALWAYS  SNESLineSearch Object:   384 MPI processestype: bt
interpolation: cubic  alpha=1.00e-04maxstep=1.00e+08,
minlambda=1.00e-12tolerances: relative=1.00e-08,
absolute=1.00e-15, lambda=1.00e-08maximum iterations=40  KSP
Object:   384 MPI processestype: gmres  GMRES: restart=100, using
Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative
refinement  GMRES: happy breakdown tolerance 1e-30maximum
iterations=100, initial guess is zerotolerances:  relative=0.001,
absolute=1e-50, divergence=1.right preconditioningusing
UNPRECONDITIONED norm type for convergence test  PC Object:   384 MPI
processestype: gamg  MG: type is MULTIPLICATIVE, levels=2
cycles=vCycles per PCApply=1Using Galerkin computed coarse
grid matricesGAMG specific options  Threshold for dropping
small values from graph 0.  AGG specific options
Symmetric graph trueCoarse grid solver -- level
---  KSP Object:  (mg_coarse_)
384 MPI processestype: preonlymaximum iterations=1,
initial guess is zerotolerances:  relative=1e-05, absolute=1e-50,
divergence=1.left preconditioningusing NONE norm type
for convergence test  PC Object:  (mg_coarse_)   384 MPI
processestype: bjacobi  block Jacobi: number of blocks =
384  Local solve is same for all blocks, in the following KSP and
PC objects:KSP Object:(mg_coarse_sub_) 1 MPI
processes  type: preonly  maximum iterations=1, initial
guess is zero  tolerances:  relative=1e-05, absolute=1e-50,
divergence=1.  left preconditioning  using NONE norm
type for convergence testPC Object:(mg_coarse_sub_)
1 MPI processes  type: luLU: out-of-place
factorizationtolerance for zero pivot 2.22045e-14
using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
matrix ordering: ndfactor fill ratio given 5., needed
1.31367  Factored matrix follows:Mat
Object: 1 MPI processes  type:
seqaij  rows=37, cols=37  package used to
perform factorization: petsc  total: nonzeros=913,
allocated nonzeros=913  total number of mallocs used during
MatSetValues calls =0not using I-node routines
linear system matrix = precond matrix:  Mat Object:   1 MPI
processestype: seqaijrows=37, cols=37
total: nonzeros=695, allocated nonzeros=695total number of
mallocs used during MatSetValues calls =0  not using I-node
routineslinear system matrix = precond matrix:Mat
Object: 384 MPI processes  type: mpiaij
rows=18145, cols=18145  total: nonzeros=1709115, allocated
nonzeros=1709115  total number of mallocs used during MatSetValues
calls =0not using I-node (on process 0) routinesDown solver
(pre-smoother) on level 1 ---  KSP
Object:  (mg_levels_1_)   384 MPI processestype:
chebyshev  Chebyshev: eigenvalue estimates:  min = 0.19, max =
1.46673  Chebyshev: eigenvalues estimated using gmres with
translations  [0. 0.1; 0. 1.1]  KSP Object:
(mg_levels_1_esteig_)   384 MPI processestype:
gmres  GMRES: restart=30, using Classical (unmodified)
Gram-Schmidt Orthogonalization with no iterative refinement
GMRES: happy breakdown tolerance 1e-30maximum iterations=10,
initial guess is zerotolerances:  relative=1e-12,
absolute=1e-50, divergence=1.left
preconditioningusing PRECONDITIONED norm type for convergence
test   

Re: [petsc-users] GAMG for the unsymmetrical matrix

[Mark Adams]

On Tue, Apr 4, 2017 at 10:10 AM, Barry Smith  wrote:
>
>> Does this mean that GAMG works for the symmetrical matrix only?
>
>   No, it means that for non symmetric nonzero structure you need the extra 
> flag. So use the extra flag. The reason we don't always use the flag is 
> because it adds extra cost and isn't needed if the matrix already has a 
> symmetric nonzero structure.

BTW, if you have symmetric non-zero structure you can just set
-pc_gamg_threshold -1.0', note the "or" in the message.

If you want to mess with the threshold then you need to use the
symmetrized flag.

Re: [petsc-users] GAMG for the unsymmetrical matrix

[Barry Smith]

> Does this mean that GAMG works for the symmetrical matrix only?

  No, it means that for non symmetric nonzero structure you need the extra 
flag. So use the extra flag. The reason we don't always use the flag is because 
it adds extra cost and isn't needed if the matrix already has a symmetric 
nonzero structure.


  Barry

> On Apr 4, 2017, at 11:46 AM, Kong, Fande  wrote:
> 
> Hi All,
> 
> I am using GAMG to solve a group of coupled diffusion equations, but the 
> resulting matrix is not symmetrical. I got the following error messages:
> 
> 
> [0]PETSC ERROR: Petsc has generated inconsistent data
> [0]PETSC ERROR: Have un-symmetric graph (apparently). Use '-pc_gamg_sym_graph 
> true' to symetrize the graph or '-pc_gamg_threshold -1.0' if the matrix is 
> structurally symmetric.
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for 
> trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.7.5, unknown 
> [0]PETSC ERROR: /home/kongf/workhome/projects/yak/yak-opt on a 
> arch-linux2-c-opt named r2i2n0 by kongf Mon Apr  3 16:19:59 2017
> [0]PETSC ERROR: /home/kongf/workhome/projects/yak/yak-opt on a 
> arch-linux2-c-opt named r2i2n0 by kongf Mon Apr  3 16:19:59 2017
> [0]PETSC ERROR: #1 smoothAggs() line 462 in 
> /home/kongf/workhome/projects/petsc/src/ksp/pc/impls/gamg/agg.c
> [0]PETSC ERROR: #2 PCGAMGCoarsen_AGG() line 998 in 
> /home/kongf/workhome/projects/petsc/src/ksp/pc/impls/gamg/agg.c
> [0]PETSC ERROR: #3 PCSetUp_GAMG() line 571 in 
> /home/kongf/workhome/projects/petsc/src/ksp/pc/impls/gamg/gamg.c
> [0]PETSC ERROR: #3 PCSetUp_GAMG() line 571 in 
> /home/kongf/workhome/projects/petsc/src/ksp/pc/impls/gamg/gamg.c
> 
> Does this mean that GAMG works for the symmetrical matrix only?
> 
> Fande,

[petsc-users] GAMG for the unsymmetrical matrix

[Kong, Fande]

Hi All,

I am using GAMG to solve a group of coupled diffusion equations, but the
resulting matrix is not symmetrical. I got the following error messages:













*[0]PETSC ERROR: Petsc has generated inconsistent data[0]PETSC ERROR: Have
un-symmetric graph (apparently). Use '-pc_gamg_sym_graph true' to symetrize
the graph or '-pc_gamg_threshold -1.0' if the matrix is structurally
symmetric.[0]PETSC ERROR: See
http://www.mcs.anl.gov/petsc/documentation/faq.html
 for trouble
shooting.[0]PETSC ERROR: Petsc Release Version 3.7.5, unknown [0]PETSC
ERROR: /home/kongf/workhome/projects/yak/yak-opt on a arch-linux2-c-opt
named r2i2n0 by kongf Mon Apr  3 16:19:59 2017[0]PETSC ERROR:
/home/kongf/workhome/projects/yak/yak-opt on a arch-linux2-c-opt named
r2i2n0 by kongf Mon Apr  3 16:19:59 2017[0]PETSC ERROR: #1 smoothAggs()
line 462 in
/home/kongf/workhome/projects/petsc/src/ksp/pc/impls/gamg/agg.c[0]PETSC
ERROR: #2 PCGAMGCoarsen_AGG() line 998 in
/home/kongf/workhome/projects/petsc/src/ksp/pc/impls/gamg/agg.c[0]PETSC
ERROR: #3 PCSetUp_GAMG() line 571 in
/home/kongf/workhome/projects/petsc/src/ksp/pc/impls/gamg/gamg.c[0]PETSC
ERROR: #3 PCSetUp_GAMG() line 571 in
/home/kongf/workhome/projects/petsc/src/ksp/pc/impls/gamg/gamg.c*
Does this mean that GAMG works for the symmetrical matrix only?

Fande,