Re: [petsc-users] KSP has an extra iteration when use shell matrix

[Barry Smith]

   It is not running an extra KSP iteration. This "extra" matmult is normal and 
occurs in many of the SNESLineSearchApply_* functions, for example, 
https://petsc.org/release/src/snes/linesearch/impls/bt/linesearchbt.c.html#SNESLineSearchApply_BT
 It is used to decide if the Newton step results in sufficient decrease of the 
function value.

  Barry



> On Jan 30, 2024, at 3:19 PM, Yi Hu  wrote:
> 
> Hello Barry,
> 
> Thanks for your reply. The monitor options are fine. I actually meant my 
> modification of snes tutorial ex1f.F90 does not work and has some unexpected 
> behavior. I basically wanted to test if I can use a shell matrix as my 
> jacobian (code is here 
> https://github.com/hyharry/small_petsc_test/blob/master/test_shell_jac/ex1f.F90).
>  After compile my modified version and run with these monitor options, it 
> gives me the following,
> 
>  ( in rhs )
>  ( leave rhs )
>   0 SNES Function norm 6.041522986797e+00 
>   in jac shell +++
> 0 KSP Residual norm 6.041522986797e+00 
>  === start mymult ===
>  === done mymult ===
> 1 KSP Residual norm 5.065392549852e-16 
>   Linear solve converged due to CONVERGED_RTOL iterations 1
>  === start mymult ===
>  === done mymult ===
>  ( in rhs )
>  ( leave rhs )
>   1 SNES Function norm 3.512662245652e+00 
>   in jac shell +++
> 0 KSP Residual norm 3.512662245652e+00 
>  === start mymult ===
>  === done mymult ===
> 1 KSP Residual norm 6.230314124713e-16 
>   Linear solve converged due to CONVERGED_RTOL iterations 1
>  === start mymult ===
>  === done mymult ===
>  ( in rhs )
>  ( leave rhs )
>  ( in rhs )
>  ( leave rhs )
>   2 SNES Function norm 8.969285922373e-01 
>   in jac shell +++
> 0 KSP Residual norm 8.969285922373e-01 
>  === start mymult ===
>  === done mymult ===
> 1 KSP Residual norm 0.e+00 
>   Linear solve converged due to CONVERGED_ATOL iterations 1
>  === start mymult ===
>  === done mymult ===
>  ( in rhs )
>  ( leave rhs )
>  ( in rhs )
>  ( leave rhs )
>   3 SNES Function norm 4.863816734540e-01 
>   in jac shell +++
> 0 KSP Residual norm 4.863816734540e-01 
>  === start mymult ===
>  === done mymult ===
> 1 KSP Residual norm 0.e+00 
>   Linear solve converged due to CONVERGED_ATOL iterations 1
>  === start mymult ===
>  === done mymult ===
>  ( in rhs )
>  ( leave rhs )
>  ( in rhs )
>  ( leave rhs )
>   4 SNES Function norm 3.512070785520e-01 
>   in jac shell +++
> 0 KSP Residual norm 3.512070785520e-01 
>  === start mymult ===
>  === done mymult ===
> 1 KSP Residual norm 0.e+00 
>   Linear solve converged due to CONVERGED_ATOL iterations 1
>  === start mymult ===
>  === done mymult ===
>  ( in rhs )
>  ( leave rhs )
>  ( in rhs )
>  ( leave rhs )
>   5 SNES Function norm 2.769700293115e-01 
>   in jac shell +++
> 0 KSP Residual norm 2.769700293115e-01 
>  === start mymult ===
>  === done mymult ===
> 1 KSP Residual norm 1.104778916974e-16 
>   Linear solve converged due to CONVERGED_RTOL iterations 1
>  === start mymult ===
>  === done mymult ===
>  ( in rhs )
>  ( leave rhs )
>  ( in rhs )
>  ( leave rhs )
>   6 SNES Function norm 2.055345318150e-01 
>   in jac shell +++
> 0 KSP Residual norm 2.055345318150e-01 
>  === start mymult ===
>  === done mymult ===
> 1 KSP Residual norm 0.e+00 
>   Linear solve converged due to CONVERGED_ATOL iterations 1
>  === start mymult ===
>  === done mymult ===
>  ( in rhs )
>  ( leave rhs )
>  ( in rhs )
>  ( leave rhs )
>   7 SNES Function norm 1.267482220786e-01 
>   in jac shell +++
> 0 KSP Residual norm 1.267482220786e-01 
>  === start mymult ===
>  === done mymult ===
> 1 KSP Residual norm 1.498679601680e-17 
>   Linear solve converged due to CONVERGED_RTOL iterations 1
>  === start mymult ===
>  === done mymult ===
>  ( in rhs )
>  ( leave rhs )
>  ( in rhs )
>  ( leave rhs )
>   8 SNES Function norm 3.468150619264e-02 
>   in jac shell +++
> 0 KSP Residual norm 3.468150619264e-02 
>  === start mymult ===
>  === done mymult ===
> 1 KSP Residual norm 5.944160522951e-18 
>   Linear solve converged due to CONVERGED_RTOL iterations 1
>  === start mymult ===
>  === done mymult ===
>  ( in rhs )
>  ( leave rhs )
>  ( in rhs )
>  ( leave rhs )
>  ( in rhs )
>  ( leave rhs )
>  ( in rhs )
>  ( leave rhs )
>  ( in rhs )
>  ( leave rhs )
>  ( in rhs )
>  ( leave rhs )
>  ( in rhs )
>  ( leave rhs )
>  ( in rhs )
>  ( leave rhs )
>  ( in rhs )
>  ( leave rhs )
>  ( in rhs )
>  ( leave rhs )
>  ( in rhs )
>  ( leave rhs )
>  ( in rhs )
>  ( leave rhs )
>  ( in rhs )
>  ( leave rhs )
>  ( in rhs )
>  ( leave rhs )
>  === start mymult ===
>  === done mymult ===
> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 8
> Number of SNES iterations = 8
> 
> After each "Linear solve 

Re: [petsc-users] KSP has an extra iteration when use shell matrix

[Yi Hu]

Hello Barry,

Thanks for your reply. The monitor options are fine. I actually meant my 
modification of snes tutorial ex1f.F90 does not work and has some 
unexpected behavior. I basically wanted to test if I can use a shell 
matrix as my jacobian (code is here 
https://github.com/hyharry/small_petsc_test/blob/master/test_shell_jac/ex1f.F90). 
After compile my modified version and run with these monitor options, it 
gives me the following,


 ( in rhs )
 ( leave rhs )
  0 SNES Function norm 6.041522986797e+00
  in jac shell +++
    0 KSP Residual norm 6.041522986797e+00
 === start mymult ===
 === done mymult ===
    1 KSP Residual norm 5.065392549852e-16
  Linear solve converged due to CONVERGED_RTOL iterations 1
 === start mymult ===
 === done mymult ===
 ( in rhs )
 ( leave rhs )
  1 SNES Function norm 3.512662245652e+00
  in jac shell +++
    0 KSP Residual norm 3.512662245652e+00
 === start mymult ===
 === done mymult ===
    1 KSP Residual norm 6.230314124713e-16
  Linear solve converged due to CONVERGED_RTOL iterations 1
 === start mymult ===
 === done mymult ===
 ( in rhs )
 ( leave rhs )
 ( in rhs )
 ( leave rhs )
  2 SNES Function norm 8.969285922373e-01
  in jac shell +++
    0 KSP Residual norm 8.969285922373e-01
 === start mymult ===
 === done mymult ===
    1 KSP Residual norm 0.e+00
  Linear solve converged due to CONVERGED_ATOL iterations 1
 === start mymult ===
 === done mymult ===
 ( in rhs )
 ( leave rhs )
 ( in rhs )
 ( leave rhs )
  3 SNES Function norm 4.863816734540e-01
  in jac shell +++
    0 KSP Residual norm 4.863816734540e-01
 === start mymult ===
 === done mymult ===
    1 KSP Residual norm 0.e+00
  Linear solve converged due to CONVERGED_ATOL iterations 1
 === start mymult ===
 === done mymult ===
 ( in rhs )
 ( leave rhs )
 ( in rhs )
 ( leave rhs )
  4 SNES Function norm 3.512070785520e-01
  in jac shell +++
    0 KSP Residual norm 3.512070785520e-01
 === start mymult ===
 === done mymult ===
    1 KSP Residual norm 0.e+00
  Linear solve converged due to CONVERGED_ATOL iterations 1
 === start mymult ===
 === done mymult ===
 ( in rhs )
 ( leave rhs )
 ( in rhs )
 ( leave rhs )
  5 SNES Function norm 2.769700293115e-01
  in jac shell +++
    0 KSP Residual norm 2.769700293115e-01
 === start mymult ===
 === done mymult ===
    1 KSP Residual norm 1.104778916974e-16
  Linear solve converged due to CONVERGED_RTOL iterations 1
 === start mymult ===
 === done mymult ===
 ( in rhs )
 ( leave rhs )
 ( in rhs )
 ( leave rhs )
  6 SNES Function norm 2.055345318150e-01
  in jac shell +++
    0 KSP Residual norm 2.055345318150e-01
 === start mymult ===
 === done mymult ===
    1 KSP Residual norm 0.e+00
  Linear solve converged due to CONVERGED_ATOL iterations 1
 === start mymult ===
 === done mymult ===
 ( in rhs )
 ( leave rhs )
 ( in rhs )
 ( leave rhs )
  7 SNES Function norm 1.267482220786e-01
  in jac shell +++
    0 KSP Residual norm 1.267482220786e-01
 === start mymult ===
 === done mymult ===
    1 KSP Residual norm 1.498679601680e-17
  Linear solve converged due to CONVERGED_RTOL iterations 1
 === start mymult ===
 === done mymult ===
 ( in rhs )
 ( leave rhs )
 ( in rhs )
 ( leave rhs )
  8 SNES Function norm 3.468150619264e-02
  in jac shell +++
    0 KSP Residual norm 3.468150619264e-02
 === start mymult ===
 === done mymult ===
    1 KSP Residual norm 5.944160522951e-18
  Linear solve converged due to CONVERGED_RTOL iterations 1
 === start mymult ===
 === done mymult ===
 ( in rhs )
 ( leave rhs )
 ( in rhs )
 ( leave rhs )
 ( in rhs )
 ( leave rhs )
 ( in rhs )
 ( leave rhs )
 ( in rhs )
 ( leave rhs )
 ( in rhs )
 ( leave rhs )
 ( in rhs )
 ( leave rhs )
 ( in rhs )
 ( leave rhs )
 ( in rhs )
 ( leave rhs )
 ( in rhs )
 ( leave rhs )
 ( in rhs )
 ( leave rhs )
 ( in rhs )
 ( leave rhs )
 ( in rhs )
 ( leave rhs )
 ( in rhs )
 ( leave rhs )
 === start mymult ===
 === done mymult ===
Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 8
Number of SNES iterations = 8

After each "Linear solve converged due to CONVERGED_ATOL iterations", 
the code starts to do mymult again. So I thought it did an extra 
(unwanted) KSP iteration. I would like to ask if this extra iteration 
could be disabled, or maybe I am wrong about it.


Best regards,

Yi

On 1/30/24 18:35, Barry Smith wrote:


  How do I see a difference? What does "hence ruin my previous 
converged KSP result" mean? A different answer at the end of the KSP 
solve?


$ ./joe > joe.basic

~/Src/petsc/src/ksp/ksp/tutorials*(barry/2023-09-15/fix-log-pcmpi=)*arch-fix-log-pcmpi

$ ./joe -ksp_monitor -ksp_converged_reason -snes_monitor > joe.monitor

~/Src/petsc/src/ksp/ksp/tutorials*(barry/2023-09-15/fix-log-pcmpi=)*arch-fix-log-pcmpi

$ diff joe.basic joe.monitor

0a1,36

>   0 

Re: [petsc-users] KSP has an extra iteration when use shell matrix

[Barry Smith]

  How do I see a difference? What does "hence ruin my previous converged KSP 
result" mean? A different answer at the end of the KSP solve?

$ ./joe > joe.basic
~/Src/petsc/src/ksp/ksp/tutorials (barry/2023-09-15/fix-log-pcmpi=) 
arch-fix-log-pcmpi
$ ./joe -ksp_monitor -ksp_converged_reason -snes_monitor > joe.monitor
~/Src/petsc/src/ksp/ksp/tutorials (barry/2023-09-15/fix-log-pcmpi=) 
arch-fix-log-pcmpi
$ diff joe.basic joe.monitor 
0a1,36
>   0 SNES Function norm 6.041522986797e+00 
> 0 KSP Residual norm 6.041522986797e+00 
> 1 KSP Residual norm 5.065392549852e-16 
>   Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1
>   1 SNES Function norm 3.512662245652e+00 
> 0 KSP Residual norm 3.512662245652e+00 
> 1 KSP Residual norm 6.230314124713e-16 
>   Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1
>   2 SNES Function norm 8.969285922373e-01 
> 0 KSP Residual norm 8.969285922373e-01 
> 1 KSP Residual norm 0.e+00 
>   Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1
>   3 SNES Function norm 4.863816734540e-01 
> 0 KSP Residual norm 4.863816734540e-01 
> 1 KSP Residual norm 0.e+00 
>   Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1
>   4 SNES Function norm 3.512070785520e-01 
> 0 KSP Residual norm 3.512070785520e-01 
> 1 KSP Residual norm 0.e+00 
>   Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1
>   5 SNES Function norm 2.769700293115e-01 
> 0 KSP Residual norm 2.769700293115e-01 
> 1 KSP Residual norm 1.104778916974e-16 
>   Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1
>   6 SNES Function norm 2.055345318150e-01 
> 0 KSP Residual norm 2.055345318150e-01 
> 1 KSP Residual norm 1.535110861002e-17 
>   Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1
>   7 SNES Function norm 1.267482220786e-01 
> 0 KSP Residual norm 1.267482220786e-01 
> 1 KSP Residual norm 1.498679601680e-17 
>   Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1
>   8 SNES Function norm 3.468150619264e-02 
> 0 KSP Residual norm 3.468150619264e-02 
> 1 KSP Residual norm 5.944160522951e-18 
>   Linear solve converged due to CONVERGED_RTOL_NORMAL iterations 1



> On Jan 30, 2024, at 11:19 AM, Yi Hu  wrote:
> 
> Dear PETSc team,
>  
> I am still trying to sort out my previous thread 
> https://lists.mcs.anl.gov/pipermail/petsc-users/2024-January/050079.html 
> using a minimal working example. However, I encountered another problem. 
> Basically I combined the basic usage of SNES solver and shell matrix and 
> tried to make it work. The jacobian of my snes is replaced by a customized 
> MATOP_MULT. The minimal example code can be viewed here 
> https://github.com/hyharry/small_petsc_test/blob/master/test_shell_jac/ex1f.F90
>  
> When running with -ksp_monitor -ksp_converged_reason, it shows an extra 
> mymult step, and hence ruin my previous converged KSP result. Implement a 
> customized converged call-back also does not help. I am wondering how to skip 
> this extra ksp iteration. Could anyone help me on this?
>  
> Thanks for your help.
>  
> Best wishes,
> Yi
> 
> 
> -
> Stay up to date and follow us on LinkedIn, Twitter and YouTube.
> 
> Max-Planck-Institut für Eisenforschung GmbH
> Max-Planck-Straße 1
> D-40237 Düsseldorf
>  
> Handelsregister B 2533 
> Amtsgericht Düsseldorf
>  
> Geschäftsführung
> Prof. Dr. Gerhard Dehm
> Prof. Dr. Jörg Neugebauer
> Prof. Dr. Dierk Raabe
> Dr. Kai de Weldige
>  
> Ust.-Id.-Nr.: DE 11 93 58 514 
> Steuernummer: 105 5891 1000
> 
> 
> Please consider that invitations and e-mails of our institute are 
> only valid if they end with …@mpie.de. 
> If you are not sure of the validity please contact r...@mpie.de 
> 
> 
> Bitte beachten Sie, dass Einladungen zu Veranstaltungen und E-Mails
> aus unserem Haus nur mit der Endung …@mpie.de gültig sind. 
> In Zweifelsfällen wenden Sie sich bitte an r...@mpie.de 
> -

Re: [petsc-users] Parallel vector layout for TAO optimization with separable state/design structure

[Jed Brown]

For a bit of assistance, you can use DMComposite and DMRedundantCreate; see 
src/snes/tutorials/ex21.c and ex22.c.

Note that when computing redundantly, it's critical that the computation be 
deterministic (i.e., not using atomics or randomness without matching seeds) so 
the logic stays collective.

This merge request may also be relevant and comments related to your needs 
would be welcome in the discussion. 

https://gitlab.com/petsc/petsc/-/merge_requests/6531

Barry Smith  writes:

>This is a problem with MPI programming and optimization; I am unaware of a 
> perfect solution.
>
>Put the design variables into the solution vector on MPI rank 0, and when 
> doing your objective/gradient, send the values to all the MPI processes where 
> you use them. You can use a VecScatter to handle the communication you need 
> or MPI_Scatter() etc whatever makes the most sense in your code. 
>
>Barry
>
>
>> On Jan 30, 2024, at 10:53 AM, Guenther, Stefanie via petsc-users 
>>  wrote:
>> 
>> Hi Petsc team, 
>>  
>> I have a question regarding parallel layout of a Petsc vector to be used in 
>> TAO optimizers for cases where the optimization variables split into 
>> ‘design’ and ‘state’ variables (e.g. such as in PDE-constrained optimization 
>> as in tao_lcl). In our case, the state variable naturally parallelizes 
>> evenly amongst multiple processors and this distribution is fixed. The 
>> ‘design’ vector however does not, it is very short compared to the state 
>> vector and it is required on all state-processors when evaluating the 
>> objective function and gradient. My question would be how the TAO 
>> optimization vector x = [design,state] should be created in such a way that 
>> the ‘state’ part is distributed as needed in our solver, while the design 
>> part is not.
>>  
>> My only idea so far was to copy the design variables to all processors and 
>> augment / interleave the optimization vector as x = [state_proc1,design, 
>> state_proc2, design, … ] . When creating this vector in parallel on 
>> PETSC_COMM_WORLD, each processor would then own the same number of variables 
>> ( [state_proc, design] ), as long as the numbers match up, and I would 
>> only need to be careful when gathering the gradient wrt the design parts 
>> from all processors.
>>  
>> This seems cumbersome however, and I would be worried whether the 
>> optimization problem is harder to solve this way. Is there any other way to 
>> achieve this splitting, that I am missing here? Note that the distribution 
>> of the state itself is given and can not be changed, and that the state vs 
>> design vectors have very different (and independent) dimensions.
>>  
>> Thanks for your help and thoughts!
>> Best,
>> Stefanie

Re: [petsc-users] Parallel vector layout for TAO optimization with separable state/design structure

[Barry Smith]

   This is a problem with MPI programming and optimization; I am unaware of a 
perfect solution.

   Put the design variables into the solution vector on MPI rank 0, and when 
doing your objective/gradient, send the values to all the MPI processes where 
you use them. You can use a VecScatter to handle the communication you need or 
MPI_Scatter() etc whatever makes the most sense in your code. 

   Barry


> On Jan 30, 2024, at 10:53 AM, Guenther, Stefanie via petsc-users 
>  wrote:
> 
> Hi Petsc team, 
>  
> I have a question regarding parallel layout of a Petsc vector to be used in 
> TAO optimizers for cases where the optimization variables split into ‘design’ 
> and ‘state’ variables (e.g. such as in PDE-constrained optimization as in 
> tao_lcl). In our case, the state variable naturally parallelizes evenly 
> amongst multiple processors and this distribution is fixed. The ‘design’ 
> vector however does not, it is very short compared to the state vector and it 
> is required on all state-processors when evaluating the objective function 
> and gradient. My question would be how the TAO optimization vector x = 
> [design,state] should be created in such a way that the ‘state’ part is 
> distributed as needed in our solver, while the design part is not.
>  
> My only idea so far was to copy the design variables to all processors and 
> augment / interleave the optimization vector as x = [state_proc1,design, 
> state_proc2, design, … ] . When creating this vector in parallel on 
> PETSC_COMM_WORLD, each processor would then own the same number of variables 
> ( [state_proc, design] ), as long as the numbers match up, and I would 
> only need to be careful when gathering the gradient wrt the design parts from 
> all processors.
>  
> This seems cumbersome however, and I would be worried whether the 
> optimization problem is harder to solve this way. Is there any other way to 
> achieve this splitting, that I am missing here? Note that the distribution of 
> the state itself is given and can not be changed, and that the state vs 
> design vectors have very different (and independent) dimensions.
>  
> Thanks for your help and thoughts!
> Best,
> Stefanie

[petsc-users] KSP has an extra iteration when use shell matrix

[Yi Hu]

Dear PETSc team,
 
I am still trying to sort out my previous thread 
https://lists.mcs.anl.gov/pipermail/petsc-users/2024-January/050079.html using 
a minimal working example. However, I encountered another problem. Basically I 
combined the basic usage of SNES solver and shell matrix and tried to make it 
work. The jacobian of my snes is replaced by a customized MATOP_MULT. The 
minimal example code can be viewed here 
https://github.com/hyharry/small_petsc_test/blob/master/test_shell_jac/ex1f.F90 
 
When running with -ksp_monitor -ksp_converged_reason, it shows an extra mymult 
step, and hence ruin my previous converged KSP result. Implement a customized 
converged call-back also does not help. I am wondering how to skip this extra 
ksp iteration. Could anyone help me on this?
 
Thanks for your help.
 
Best wishes,
Yi

-
Stay up to date and follow us on LinkedIn, Twitter and YouTube.

Max-Planck-Institut für Eisenforschung GmbH
Max-Planck-Straße 1
D-40237 Düsseldorf
 
Handelsregister B 2533 
Amtsgericht Düsseldorf
 
Geschäftsführung
Prof. Dr. Gerhard Dehm
Prof. Dr. Jörg Neugebauer
Prof. Dr. Dierk Raabe
Dr. Kai de Weldige
 
Ust.-Id.-Nr.: DE 11 93 58 514 
Steuernummer: 105 5891 1000


Please consider that invitations and e-mails of our institute are 
only valid if they end with …@mpie.de. 
If you are not sure of the validity please contact r...@mpie.de

Bitte beachten Sie, dass Einladungen zu Veranstaltungen und E-Mails
aus unserem Haus nur mit der Endung …@mpie.de gültig sind. 
In Zweifelsfällen wenden Sie sich bitte an r...@mpie.de
-

[petsc-users] Parallel vector layout for TAO optimization with separable state/design structure

[Guenther, Stefanie via petsc-users]

Hi Petsc team,

I have a question regarding parallel layout of a Petsc vector to be used in TAO 
optimizers for cases where the optimization variables split into ‘design’ and 
‘state’ variables (e.g. such as in PDE-constrained optimization as in tao_lcl). 
In our case, the state variable naturally parallelizes evenly amongst multiple 
processors and this distribution is fixed. The ‘design’ vector however does 
not, it is very short compared to the state vector and it is required on all 
state-processors when evaluating the objective function and gradient. My 
question would be how the TAO optimization vector x = [design,state] should be 
created in such a way that the ‘state’ part is distributed as needed in our 
solver, while the design part is not.

My only idea so far was to copy the design variables to all processors and 
augment / interleave the optimization vector as x = [state_proc1,design, 
state_proc2, design, … ] . When creating this vector in parallel on 
PETSC_COMM_WORLD, each processor would then own the same number of variables ( 
[state_proc, design] ), as long as the numbers match up, and I would only 
need to be careful when gathering the gradient wrt the design parts from all 
processors.

This seems cumbersome however, and I would be worried whether the optimization 
problem is harder to solve this way. Is there any other way to achieve this 
splitting, that I am missing here? Note that the distribution of the state 
itself is given and can not be changed, and that the state vs design vectors 
have very different (and independent) dimensions.

Thanks for your help and thoughts!
Best,
Stefanie