Re: [petsc-users] Help using FAS as an initial guess

[tt73]

Thanks, Barry. I'll look into it. 
 Original message From: Barry Smith  Date: 
5/2/23  6:10 PM  (GMT-05:00) To: "Takahashi, Tadanaga"  Cc: 
PETSc  Subject: Re: [petsc-users] Help using FAS as an 
initial guess   You might consider 
https://petsc.org/release/manualpages/SNES/SNESSetGridSequence/ it does exactly 
what I think you want to do.  FAS is a bit more subtle than that. The "coarse 
grid problem" that FAS builds and solves are dependent on the current fine grid 
solution so you need an "approximate" fine grid solution already in order to 
create a FAS coarse problem. Of course FAS can, and often should, be used also 
with grid sequencing to boot-strap the fine grid solutions.  BarryOn May 2, 
2023, at 5:40 PM, Takahashi, Tadanaga  wrote:Hi, I want to know 
how to configure the FAS so that it solves a problem on a coarse grid of size 
4h, interpolate the solution, and then stop. Here is the context: I am using 
Newton LS to solve a problem on square domain discretized with DMDA meshed with 
step size h. I have a subroutine to compute the initial guess. I want this 
subroutine to first do a Newton solve on a coarse grid of size 4h. Then it 
interpolates the solution to the main mesh. I think this is achievable by using 
one iteration of FAS. Below is the gist of what my subroutine looks like: 
PetscErrorCode InitialState(DM da, Vec u) { // SNES snes; 
SNESCreate(PETSC_COMM_WORLD,);  
SNESSetDM(snes,da);SNESSetType(snes,SNESFAS); // solve with 
multigridSNESSetTolerances(snes,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT,1,PETSC_DEFAULT);
 // just one iterationVecSet(u,0.0);            // start with zeros 
SNESSolve(snes,NULL,u);   // cheap solve SNESGetSolution(snes,); // extract 
solution }For some reason, my initial guess is too accurate. The initial guess 
produced by this subroutine looks exactly like the final solution. My guess is 
that it's doing more than what I want it to do. I'm still new to FAS. How can I 
tell FAS to do just one crude solve and an interpolation?

Re: [petsc-users] Help using FAS as an initial guess

[Barry Smith]

  You might consider 
https://petsc.org/release/manualpages/SNES/SNESSetGridSequence/ it does exactly 
what I think you want to do.

  FAS is a bit more subtle than that. The "coarse grid problem" that FAS builds 
and solves are dependent on the current fine grid solution so you need an 
"approximate" fine grid solution already in order to create a FAS coarse 
problem. Of course FAS can, and often should, be used also with grid sequencing 
to boot-strap the fine grid solutions.

  Barry


> On May 2, 2023, at 5:40 PM, Takahashi, Tadanaga  wrote:
> 
> Hi, 
> 
> I want to know how to configure the FAS so that it solves a problem on a 
> coarse grid of size 4h, interpolate the solution, and then stop. 
> 
> Here is the context: I am using Newton LS to solve a problem on square domain 
> discretized with DMDA meshed with step size h. I have a subroutine to compute 
> the initial guess. I want this subroutine to first do a Newton solve on a 
> coarse grid of size 4h. Then it interpolates the solution to the main mesh. I 
> think this is achievable by using one iteration of FAS. Below is the gist of 
> what my subroutine looks like: 
> 
> PetscErrorCode InitialState(DM da, Vec u) { // 
> SNES snes; 
> SNESCreate(PETSC_COMM_WORLD,);  
> SNESSetDM(snes,da);
> SNESSetType(snes,SNESFAS); // solve with multigrid
> SNESSetTolerances(snes,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT,1,PETSC_DEFAULT);
>  // just one iteration
> VecSet(u,0.0);// start with zeros 
> SNESSolve(snes,NULL,u);   // cheap solve 
> SNESGetSolution(snes,); // extract solution 
> }
> 
> For some reason, my initial guess is too accurate. The initial guess produced 
> by this subroutine looks exactly like the final solution. My guess is that 
> it's doing more than what I want it to do. I'm still new to FAS. How can I 
> tell FAS to do just one crude solve and an interpolation?

[petsc-users] Help using FAS as an initial guess

[Takahashi, Tadanaga]

Hi,

I want to know how to configure the FAS so that it solves a problem
on a coarse grid of size 4h, interpolate the solution, and then stop.

Here is the context: I am using Newton LS to solve a problem on square
domain discretized with DMDA meshed with step size h. I have a subroutine
to compute the initial guess. I want this subroutine to first do a Newton
solve on a coarse grid of size 4h. Then it interpolates the solution to the
main mesh. I think this is achievable by using one iteration of FAS. Below
is the gist of what my subroutine looks like:

PetscErrorCode InitialState(DM da, Vec u) { //
SNES snes;
SNESCreate(PETSC_COMM_WORLD,);
SNESSetDM(snes,da);
SNESSetType(snes,SNESFAS); // solve with multigrid
SNESSetTolerances(snes,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT,1,PETSC_DEFAULT);
// just one iteration
VecSet(u,0.0);// start with zeros
SNESSolve(snes,NULL,u);   // cheap solve
SNESGetSolution(snes,); // extract solution
}

For some reason, my initial guess is too accurate. The initial guess
produced by this subroutine looks exactly like the final solution. My guess
is that it's doing more than what I want it to do. I'm still new to FAS.
How can I tell FAS to do just one crude solve and an interpolation?

Re: [petsc-users] Scalable Solver for Incompressible Flow

[Matthew Knepley]

On Tue, May 2, 2023 at 2:29 PM Jed Brown  wrote:

> Sebastian Blauth  writes:
>
> > I agree with your comment for the Stokes equations - for these, I have
> > already tried and used the pressure mass matrix as part of a (additive)
> > block preconditioner and it gave mesh independent results.
> >
> > However, for the Navier Stokes equations, is the Schur complement really
> > spectrally equivalent to the pressure mass matrix?
>
> No, it's not. You'd want something like PCD (better, but not algebraic) or
> LSC.
>

I think you can do a better job than that using something like

  https://arxiv.org/abs/1810.03315

Basically, you use an augmented Lagrangian thing to make the Schur
complement well-conditioned,
and then use a special smoother to handle that perturbation.


> > And even if it is, the convergence is only good for small Reynolds
> numbers, for moderately high ones the convergence really deteriorates. This
> is why I am trying to make fieldsplit_schur_precondition selfp work better
> (this is, if I understand it correctly, a SIMPLE type preconditioner).
>
> SIMPLE is for short time steps (not too far from resolving CFL) and bad
> for steady. This taxonomy is useful, though the problems are super academic
> and they don't use high aspect ratio.
>
> https://doi.org/10.1016/j.jcp.2007.09.026
>

   Thanks,

  Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Scalable Solver for Incompressible Flow

[Jed Brown]

Sebastian Blauth  writes:

> I agree with your comment for the Stokes equations - for these, I have 
> already tried and used the pressure mass matrix as part of a (additive) 
> block preconditioner and it gave mesh independent results.
>
> However, for the Navier Stokes equations, is the Schur complement really 
> spectrally equivalent to the pressure mass matrix? 

No, it's not. You'd want something like PCD (better, but not algebraic) or LSC.

> And even if it is, the convergence is only good for small Reynolds numbers, 
> for moderately high ones the convergence really deteriorates. This is why I 
> am trying to make fieldsplit_schur_precondition selfp work better (this is, 
> if I understand it correctly, a SIMPLE type preconditioner).

SIMPLE is for short time steps (not too far from resolving CFL) and bad for 
steady. This taxonomy is useful, though the problems are super academic and 
they don't use high aspect ratio.

https://doi.org/10.1016/j.jcp.2007.09.026

Re: [petsc-users] Scalable Solver for Incompressible Flow

[Sebastian Blauth]

On 02.05.2023 15:12, Matthew Knepley wrote:
On Tue, May 2, 2023 at 9:07 AM Blauth, Sebastian 
> wrote:


Hello,

__ __

I am having a problem using / configuring PETSc to obtain a scalable
solver for the incompressible Navier Stokes equations. I am
discretizing the equations using FEM (with the library fenics) and I
am using the stable P2-P1 Taylor-Hood elements. I have read and
tried a lot regarding preconditioners for incompressible Navier
Stokes and I am aware that this is very much an active research
field, but maybe I can get some hints / tips. 

I am interested in solving large-scale 3D problems, but I cannot
even set up a scaleable 2D solver for the problems. All of my
approaches at the moment are trying to use a Schur Complement
approach, but I cannot get a “good” preconditioner for the Schur
complement matrix. For the velocity block, I am using the AMG
provided by hypre (which seems to work fine and is most likely not
the problem).

__ __

To test the solver, I am using a simple 2D channel flow problem with
do-nothing conditions at the outlet.

__ __

I am facing the following difficulties at the moment:

__ __

- First, I am having trouble with using
-pc_fieldsplit_schur_precondition selfp. With this setup, the cost
for solving the Schur complement part in the fieldsplit
preconditioner (approximately) increase when the mesh is refined. I
am using the following options for this setup (note that I am using
exact solves for the velocity part to debug, but using, e.g., gmres
with hypre boomeramg reaches a given tolerance with a number of
iterations that is independent of the mesh)


The diagonal of the momentum block is a bad preconditioner for the Schur 
complement, because S is spectrally equivalent to the mass matrix. You 
should build the mass matrix and use that as the preconditioning matrix 
for the Schur part. The FEniCS people can show you how to do that. This 
will provide mesh-independent convergence (you can see me doing this in 
SNES ex69).


   Thanks,

      Matt


I agree with your comment for the Stokes equations - for these, I have 
already tried and used the pressure mass matrix as part of a (additive) 
block preconditioner and it gave mesh independent results.


However, for the Navier Stokes equations, is the Schur complement really 
spectrally equivalent to the pressure mass matrix? And even if it is, 
the convergence is only good for small Reynolds numbers, for moderately 
high ones the convergence really deteriorates. This is why I am trying 
to make fieldsplit_schur_precondition selfp work better (this is, if I 
understand it correctly, a SIMPLE type preconditioner).


Best regards,
Sebastian




     -ksp_type fgmres

     -ksp_rtol 1e-6

-ksp_atol 1e-30

     -pc_type fieldsplit

     -pc_fieldsplit_type schur

     -pc_fieldsplit_schur_fact_type full

     -pc_fieldsplit_schur_precondition selfp

     -fieldsplit_0_ksp_type preonly

     -fieldsplit_0_pc_type lu

     -fieldsplit_1_ksp_type gmres

     -fieldsplit_1_ksp_pc_side right

     -fieldsplit_1_ksp_max_it 1000

     -fieldsplit_1_ksp_rtol 1e-1

     -fieldsplit_1_ksp_atol 1e-30

     -fieldsplit_1_pc_type lu

     -fieldsplit_1_ksp_converged_reason

     -ksp_converged_reason

__ __

Note, that I use direct solvers for the subproblems to get an
“ideal” convergence. Even if I replace the direct solver with
boomeramg, the behavior is the same and the number of iterations
does not change much. 

In particular, I get the following behavior:

For a 8x8 mesh, I need, on average, 25 iterations to solve
fieldsplit_1

For a 16x16 mesh, I need 40 iterations

For a 32x32 mesh, I need 70 iterations

For a 64x64 mesh, I need 100 iterations

__ __

However, the outer fgmres requires, as expected, always the same
number of iterations to reach convergence (as expected).

I do understand that the selfp preconditioner for the Schur
complement is expected to deteriorate as the Reynolds number
increases and the problem becomes more convective in nature, but I
had hoped that I can at least get a scaleable preconditioner with
respect to the mesh size out of it. Are there any tips on how to
achieve this?

__ __

My second problem is concerning the LSC preconditioner. When I am
using this, again both with exact solves of the linear problems or
when using boomeramg, I do not get a scalable solver with respect to
the mesh size. On the contrary, here the number of solves required
for solving fieldsplit_1 to a fixed relative tolerance seem to
behave linearly w.r.t. the problem size. For 

Re: [petsc-users] DMSWARM with DMDA and KSP

[Matthew Young]

Yup -- I realized that I had '-pc_type mg' in the script I was using to
build and run as I developed. I guess that was causing the KSP to coarsen
its DM, which made me think I had to force the density DM to be consistent.
Still, refactoring my original grid DM into one for Vlasov components and
one for the potential was useful.

Thanks for the help!

--Matt
==
Matthew Young, PhD (he/him)
Research Scientist II
Space Science Center
University of New Hampshire
matthew.yo...@unh.edu
==


On Mon, May 1, 2023 at 10:51 PM Dave May  wrote:

>
>
> On Mon 1. May 2023 at 18:57, Matthew Young 
> wrote:
>
>> Thanks for the suggestion to keep DMs separate, and for pointing me
>> toward that example. I now have a DM for the particle quantities (i.e.,
>> density and flux) and another for the potential. I'm hoping to use
>> KSPSetComputeOperators with PCGAMG, so I packed the density DM into the
>> application context and set the potential DM on the KSP, but I'm not sure
>> how to communicate changes in the KSP DM (e.g., coarsening) to the density
>> DM inside my operator function.
>>
>
> I don’t think you need to.
>
> GAMG only requires the fine grid operator - this will be the matrix
> assembled from KSPSetComputeOperators. Hence density DM and potential DM
> fields only need to be managed by you on the finest level.
>
> However, if you wanted to use PCMG with rediscretized operators on every
> level, then you would need the density DM field defined on each level of
> your geometric multigrid hierarchy. This could be done (possibly less than
> ideally) by calling DMCreateInterpolation() and then using the Mat to
> interpolate the density from the  finest level to next coarsest level (and
> so on).
>
> Thanks,
> Dave
>
>
>>
>> --Matt
>> ==
>> Matthew Young, PhD (he/him)
>> Research Scientist II
>> Space Science Center
>> University of New Hampshire
>> matthew.yo...@unh.edu
>> ==
>>
>>
>> On Sun, Apr 30, 2023 at 1:52 PM Matthew Knepley 
>> wrote:
>>
>>> On Sun, Apr 30, 2023 at 1:12 PM Matthew Young <
>>> myoung.space.scie...@gmail.com> wrote:
>>>
 Hi all,

 I am developing a particle-in-cell code that models ions as particles
 and electrons as an inertialess fluid. I use a PIC DMSWARM for the ions,
 which I gather into density and flux before solving a linear system for the
 electrostatic potential (phi). I currently have one DMDA with 5 degrees of
 freedom -- one each for density, 3 flux components, and phi.

 When setting up the linear system to solve for phi, I've been following
 examples like KSP ex34.c and ex42.c when writing the KSP operator and RHS
 functions but I'm not sure I have the right approach, since 4 of the DOFs
 are known and 1 is unknown.

 I saw this thread
 
 that recommended using DMDAGetReducedDMDA, which I gather has been
 deprecated in favor of DMDACreateCompatibleDMDA. Is that a good approach
 for managing a regular grid with known and unknown quantities on each node?
 Could a composite DM be useful? Has anyone else worked on a problem like
 this?

>>>
>>> I recommend making a different DM for each kind of solve you want.
>>> DMDACreateCompatibleDMDA() should be the implementation of DMClone(), but
>>> we have yet to harmonize all things for all DMs. I would create one DM for
>>> your Vlasov components and one for the Poisson.
>>> We follow this strategy in our Vlasov-Poisson test for Landau damping:
>>> https://gitlab.com/petsc/petsc/-/blob/main/src/dm/impls/swarm/tests/ex9.c
>>>
>>>   Thanks,
>>>
>>>  Matt
>>>
>>>
 --Matt
 ==
 Matthew Young, PhD (he/him)
 Research Scientist II
 Space Science Center
 University of New Hampshire
 matthew.yo...@unh.edu
 ==

>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> 
>>>
>>

Re: [petsc-users] Node numbering in parallel partitioned mesh

[Matthew Knepley]

On Tue, May 2, 2023 at 11:03 AM Karthikeyan Chockalingam - STFC UKRI <
karthikeyan.chockalin...@stfc.ac.uk> wrote:

> Thank you Matt.
>
>
>
> I will look to find out those shared nodes. Sorry, I didn’t get it when
> you say “Roots are owned, and leaves are not owned”
>

That is the nomenclature from PetscSF.


>
>
> My question was specifically related to numbering – how do I start
> numbering in a partition from where I left off from the previous partition
> without double counting so that the node numbers are unique?
>

1) Determine the local sizes

Run over the local nodes. If any are not owned, do not count them.

2) Get the local offset nStart

Add up the local sizes to get the offset for each process using
MPI_Scan()

3) Number locally

   Run over local nodes and number each owned node, starting with nStart

  Thanks,

 Matt


>
>
Let's say I have a VECMPI which is distributed among the partitions. When I
> try to retrieve the data using VecGetValues, I often run into problems
> accessing non-local data (so, for now, I scatter the vector). When some
> nodes are shared, will I not always have this problem accessing those nodes
> from the wrong partition unless those nodes are ghosted? Maybe I am not
> thinking about it correctly.
>
>
>
> Kind regards,
>
> Karthik.
>
>
>
>
>
> *From: *Matthew Knepley 
> *Date: *Tuesday, 2 May 2023 at 13:35
> *To: *Chockalingam, Karthikeyan (STFC,DL,HC) <
> karthikeyan.chockalin...@stfc.ac.uk>
> *Cc: *petsc-users@mcs.anl.gov 
> *Subject: *Re: [petsc-users] Node numbering in parallel partitioned mesh
>
> On Tue, May 2, 2023 at 8:25 AM Karthikeyan Chockalingam - STFC UKRI via
> petsc-users  wrote:
>
> Hello,
>
>
>
> This is not exactly a PETSc question. I have a parallel partitioned finite
> element mesh. What are the steps involved in having a contiguous but unique
> set of node numbering from one partition to the next? There are nodes which
> are shared between different partitions. Moreover, this partition has to
> coincide parallel partition of PETSc Vec/Mat, which ensures data locality.
>
>
>
> If you can post the algorithm or cite a reference, it will prove helpful.
>
>
>
> Somehow, you have to know what "nodes" are shared. Once you know this, you
> can make a rule for numbering, such
>
> as "the lowest rank gets the shared nodes". We encapsulate this ownership
> relation in the PetscSF. Roots are owned,
>
> and leaves are not owned. The rule above is not great for load balance, so
> we have an optimization routine for the
>
> simple PetscSF:
> https://petsc.org/main/manualpages/DMPlex/DMPlexRebalanceSharedPoints/
>
>
>
>   Thanks,
>
>
>
>  Matt
>
>
>
> Many thanks.
>
>
>
> Kind regards,
>
> Karthik.
>
>
>
>
>
>
> --
>
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>
> https://www.cse.buffalo.edu/~knepley/
> 
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Node numbering in parallel partitioned mesh

[Karthikeyan Chockalingam - STFC UKRI via petsc-users]

Thank you Matt.

I will look to find out those shared nodes. Sorry, I didn’t get it when you say 
“Roots are owned, and leaves are not owned”

My question was specifically related to numbering – how do I start numbering in 
a partition from where I left off from the previous partition without double 
counting so that the node numbers are unique?

Let's say I have a VECMPI which is distributed among the partitions. When I try 
to retrieve the data using VecGetValues, I often run into problems accessing 
non-local data (so, for now, I scatter the vector). When some nodes are shared, 
will I not always have this problem accessing those nodes from the wrong 
partition unless those nodes are ghosted? Maybe I am not thinking about it 
correctly.

Kind regards,
Karthik.


From: Matthew Knepley 
Date: Tuesday, 2 May 2023 at 13:35
To: Chockalingam, Karthikeyan (STFC,DL,HC) 
Cc: petsc-users@mcs.anl.gov 
Subject: Re: [petsc-users] Node numbering in parallel partitioned mesh
On Tue, May 2, 2023 at 8:25 AM Karthikeyan Chockalingam - STFC UKRI via 
petsc-users mailto:petsc-users@mcs.anl.gov>> wrote:
Hello,

This is not exactly a PETSc question. I have a parallel partitioned finite 
element mesh. What are the steps involved in having a contiguous but unique set 
of node numbering from one partition to the next? There are nodes which are 
shared between different partitions. Moreover, this partition has to coincide 
parallel partition of PETSc Vec/Mat, which ensures data locality.

If you can post the algorithm or cite a reference, it will prove helpful.

Somehow, you have to know what "nodes" are shared. Once you know this, you can 
make a rule for numbering, such
as "the lowest rank gets the shared nodes". We encapsulate this ownership 
relation in the PetscSF. Roots are owned,
and leaves are not owned. The rule above is not great for load balance, so we 
have an optimization routine for the
simple PetscSF: 
https://petsc.org/main/manualpages/DMPlex/DMPlexRebalanceSharedPoints/

  Thanks,

 Matt

Many thanks.

Kind regards,
Karthik.



--
What most experimenters take for granted before they begin their experiments is 
infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/

Re: [petsc-users] Node numbering in parallel partitioned mesh

[Barry Smith]

   Assuming you have generated your renumbering, you can use 
https://petsc.org/release/manualpages/AO/AO/#ao to convert lists in the old (or 
new) numbering to the new (or old) numbering.

  Barry


> On May 2, 2023, at 8:34 AM, Matthew Knepley  wrote:
> 
> On Tue, May 2, 2023 at 8:25 AM Karthikeyan Chockalingam - STFC UKRI via 
> petsc-users mailto:petsc-users@mcs.anl.gov>> wrote:
>> Hello,
>> 
>>  
>> 
>> This is not exactly a PETSc question. I have a parallel partitioned finite 
>> element mesh. What are the steps involved in having a contiguous but unique 
>> set of node numbering from one partition to the next? There are nodes which 
>> are shared between different partitions. Moreover, this partition has to 
>> coincide parallel partition of PETSc Vec/Mat, which ensures data locality.
>> 
>>  
>> 
>> If you can post the algorithm or cite a reference, it will prove helpful.
>> 
> 
> Somehow, you have to know what "nodes" are shared. Once you know this, you 
> can make a rule for numbering, such
> as "the lowest rank gets the shared nodes". We encapsulate this ownership 
> relation in the PetscSF. Roots are owned,
> and leaves are not owned. The rule above is not great for load balance, so we 
> have an optimization routine for the
> simple PetscSF: 
> https://petsc.org/main/manualpages/DMPlex/DMPlexRebalanceSharedPoints/
> 
>   Thanks,
> 
>  Matt
>  
>> Many thanks.
>> 
>>  
>> 
>> Kind regards,
>> 
>> Karthik.
>> 
>>  
>> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments 
> is infinitely more interesting than any results to which their experiments 
> lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] 'mpirun' run not found error

[Barry Smith]

   For any PETSc install you can run 

make getmpiexec 

in the PETSC_DIR directory to see how to use mpiexec for that PETSc install.

  Barry


> On May 2, 2023, at 2:56 AM, ­권승리 / 학생 / 항공우주공학과  wrote:
> 
> Dear developers
> 
> I'm trying to use the mpi, but I'm encountering error messages like below:
> 
> 
> Command 'mpirun' not found, but can be installed with:
> sudo apt install lam-runtime   # version 7.1.4-6build2, or
> sudo apt install mpich # version 3.3.2-2build1
> sudo apt install openmpi-bin   # version 4.0.3-0ubuntu1
> sudo apt install slurm-wlm-torque  # version 19.05.5-1
> //
> 
> However, I've already installed the mpich.
> cd $PETSC_DIR
> ./configure --download-mpich --with-debugging=0 COPTFLAGS='-O3 -march=native 
> -mtune=native' CXXOPTFLAGS='-O3 -march=native -mtune=native' FOPTFLAGS='-O3 
> -march=native -mtune=native' --download-mumps --download-scalapack 
> --download-parmetis --download-metis --download-parmetis --download-hpddm 
> --download-slepc
> 
> Could you recommend some advice related to this?
> 
> Best,
> Seung Lee Kwon
> -- 
> Seung Lee Kwon, Ph.D.Candidate
> Aerospace Structures and Materials Laboratory
> Department of Mechanical and Aerospace Engineering
> Seoul National University
> Building 300 Rm 503, Gwanak-ro 1, Gwanak-gu, Seoul, South Korea, 08826
> E-mail : ksl7...@snu.ac.kr 
> Office : +82-2-880-7389
> C. P : +82-10-4695-1062

Re: [petsc-users] Scalable Solver for Incompressible Flow

[Matthew Knepley]

On Tue, May 2, 2023 at 9:07 AM Blauth, Sebastian <
sebastian.bla...@itwm.fraunhofer.de> wrote:

> Hello,
>
>
>
> I am having a problem using / configuring PETSc to obtain a scalable
> solver for the incompressible Navier Stokes equations. I am discretizing
> the equations using FEM (with the library fenics) and I am using the stable
> P2-P1 Taylor-Hood elements. I have read and tried a lot regarding
> preconditioners for incompressible Navier Stokes and I am aware that this
> is very much an active research field, but maybe I can get some hints /
> tips.
>
> I am interested in solving large-scale 3D problems, but I cannot even set
> up a scaleable 2D solver for the problems. All of my approaches at the
> moment are trying to use a Schur Complement approach, but I cannot get a
> “good” preconditioner for the Schur complement matrix. For the velocity
> block, I am using the AMG provided by hypre (which seems to work fine and
> is most likely not the problem).
>
>
>
> To test the solver, I am using a simple 2D channel flow problem with
> do-nothing conditions at the outlet.
>
>
>
> I am facing the following difficulties at the moment:
>
>
>
> - First, I am having trouble with using -pc_fieldsplit_schur_precondition
> selfp. With this setup, the cost for solving the Schur complement part in
> the fieldsplit preconditioner (approximately) increase when the mesh is
> refined. I am using the following options for this setup (note that I am
> using exact solves for the velocity part to debug, but using, e.g., gmres
> with hypre boomeramg reaches a given tolerance with a number of iterations
> that is independent of the mesh)
>

The diagonal of the momentum block is a bad preconditioner for the Schur
complement, because S is spectrally equivalent to the mass matrix. You
should build the mass matrix and use that as the preconditioning matrix for
the Schur part. The FEniCS people can show you how to do that. This will
provide mesh-independent convergence (you can see me doing this in SNES
ex69).

  Thanks,

 Matt


> -ksp_type fgmres
>
> -ksp_rtol 1e-6
>
> -ksp_atol 1e-30
>
> -pc_type fieldsplit
>
> -pc_fieldsplit_type schur
>
> -pc_fieldsplit_schur_fact_type full
>
> -pc_fieldsplit_schur_precondition selfp
>
> -fieldsplit_0_ksp_type preonly
>
> -fieldsplit_0_pc_type lu
>
> -fieldsplit_1_ksp_type gmres
>
> -fieldsplit_1_ksp_pc_side right
>
> -fieldsplit_1_ksp_max_it 1000
>
> -fieldsplit_1_ksp_rtol 1e-1
>
> -fieldsplit_1_ksp_atol 1e-30
>
> -fieldsplit_1_pc_type lu
>
> -fieldsplit_1_ksp_converged_reason
>
> -ksp_converged_reason
>
>
>
> Note, that I use direct solvers for the subproblems to get an “ideal”
> convergence. Even if I replace the direct solver with boomeramg, the
> behavior is the same and the number of iterations does not change much.
>
> In particular, I get the following behavior:
>
> For a 8x8 mesh, I need, on average, 25 iterations to solve fieldsplit_1
>
> For a 16x16 mesh, I need 40 iterations
>
> For a 32x32 mesh, I need 70 iterations
>
> For a 64x64 mesh, I need 100 iterations
>
>
>
> However, the outer fgmres requires, as expected, always the same number of
> iterations to reach convergence (as expected).
>
> I do understand that the selfp preconditioner for the Schur complement is
> expected to deteriorate as the Reynolds number increases and the problem
> becomes more convective in nature, but I had hoped that I can at least get
> a scaleable preconditioner with respect to the mesh size out of it. Are
> there any tips on how to achieve this?
>
>
>
> My second problem is concerning the LSC preconditioner. When I am using
> this, again both with exact solves of the linear problems or when using
> boomeramg, I do not get a scalable solver with respect to the mesh size. On
> the contrary, here the number of solves required for solving fieldsplit_1
> to a fixed relative tolerance seem to behave linearly w.r.t. the problem
> size. For this problem, I suspect that the issue lies in the scaling of the
> LSC preconditioner matrices (in the book of Elman, Sylvester and Wathen,
> the matrices are scaled with the inverse of the diagonal velocity mass
> matrix). Is it possible to achieve this with PETSc? I started experimenting
> with supplying the velocity mass matrix as preconditioner matrix and using
> “use_amat”, but I am not sure where / how to do it this way.
>
>
>
> And finally, more of an observation and question: I noticed that the AMG
> approximations for the velocity block became worse with increase of the
> Reynolds number when using the default options. However, when using
> -pc_hypre_boomeramg_relax_weight_all 0.0 I noticed that boomeramg performed
> way more robustly w.r.t. the Reynolds number. Are there any other ways to
> improve the AMG performance in this regard?
>
>
>
> Thanks a lot in advance and I am looking forward to your reply,
>
> Sebastian
>
>
>
> --
>
> Dr. Sebastian Blauth
>
> Fraunhofer-Institut für
>
> 

[petsc-users] Scalable Solver for Incompressible Flow

[Blauth, Sebastian]

Hello,

 

I am having a problem using / configuring PETSc to obtain a scalable solver
for the incompressible Navier Stokes equations. I am discretizing the
equations using FEM (with the library fenics) and I am using the stable
P2-P1 Taylor-Hood elements. I have read and tried a lot regarding
preconditioners for incompressible Navier Stokes and I am aware that this is
very much an active research field, but maybe I can get some hints / tips. 

I am interested in solving large-scale 3D problems, but I cannot even set up
a scaleable 2D solver for the problems. All of my approaches at the moment
are trying to use a Schur Complement approach, but I cannot get a “good”
preconditioner for the Schur complement matrix. For the velocity block, I am
using the AMG provided by hypre (which seems to work fine and is most likely
not the problem).

 

To test the solver, I am using a simple 2D channel flow problem with
do-nothing conditions at the outlet.

 

I am facing the following difficulties at the moment:

 

- First, I am having trouble with using -pc_fieldsplit_schur_precondition
selfp. With this setup, the cost for solving the Schur complement part in
the fieldsplit preconditioner (approximately) increase when the mesh is
refined. I am using the following options for this setup (note that I am
using exact solves for the velocity part to debug, but using, e.g., gmres
with hypre boomeramg reaches a given tolerance with a number of iterations
that is independent of the mesh)

 

-ksp_type fgmres

-ksp_rtol 1e-6

-ksp_atol 1e-30

-pc_type fieldsplit

-pc_fieldsplit_type schur

-pc_fieldsplit_schur_fact_type full

-pc_fieldsplit_schur_precondition selfp

-fieldsplit_0_ksp_type preonly

-fieldsplit_0_pc_type lu

-fieldsplit_1_ksp_type gmres

-fieldsplit_1_ksp_pc_side right

-fieldsplit_1_ksp_max_it 1000

-fieldsplit_1_ksp_rtol 1e-1

-fieldsplit_1_ksp_atol 1e-30

-fieldsplit_1_pc_type lu

-fieldsplit_1_ksp_converged_reason

-ksp_converged_reason

 

Note, that I use direct solvers for the subproblems to get an “ideal”
convergence. Even if I replace the direct solver with boomeramg, the
behavior is the same and the number of iterations does not change much. 

In particular, I get the following behavior:

For a 8x8 mesh, I need, on average, 25 iterations to solve fieldsplit_1

For a 16x16 mesh, I need 40 iterations

For a 32x32 mesh, I need 70 iterations

For a 64x64 mesh, I need 100 iterations

 

However, the outer fgmres requires, as expected, always the same number of
iterations to reach convergence (as expected).

I do understand that the selfp preconditioner for the Schur complement is
expected to deteriorate as the Reynolds number increases and the problem
becomes more convective in nature, but I had hoped that I can at least get a
scaleable preconditioner with respect to the mesh size out of it. Are there
any tips on how to achieve this?

 

My second problem is concerning the LSC preconditioner. When I am using
this, again both with exact solves of the linear problems or when using
boomeramg, I do not get a scalable solver with respect to the mesh size. On
the contrary, here the number of solves required for solving fieldsplit_1 to
a fixed relative tolerance seem to behave linearly w.r.t. the problem size.
For this problem, I suspect that the issue lies in the scaling of the LSC
preconditioner matrices (in the book of Elman, Sylvester and Wathen, the
matrices are scaled with the inverse of the diagonal velocity mass matrix).
Is it possible to achieve this with PETSc? I started experimenting with
supplying the velocity mass matrix as preconditioner matrix and using
“use_amat”, but I am not sure where / how to do it this way.

 

And finally, more of an observation and question: I noticed that the AMG
approximations for the velocity block became worse with increase of the
Reynolds number when using the default options. However, when using
-pc_hypre_boomeramg_relax_weight_all 0.0 I noticed that boomeramg performed
way more robustly w.r.t. the Reynolds number. Are there any other ways to
improve the AMG performance in this regard?

 

Thanks a lot in advance and I am looking forward to your reply,

Sebastian

 

--

Dr. Sebastian Blauth

Fraunhofer-Institut für

Techno- und Wirtschaftsmathematik ITWM

Abteilung Transportvorgänge

Fraunhofer-Platz 1, 67663 Kaiserslautern

Telefon: +49 631 31600-4968

sebastian.bla...@itwm.fraunhofer.de

https://www.itwm.fraunhofer.de

 



smime.p7s
Description: S/MIME cryptographic signature

Re: [petsc-users] Node numbering in parallel partitioned mesh

[Matthew Knepley]

On Tue, May 2, 2023 at 8:25 AM Karthikeyan Chockalingam - STFC UKRI via
petsc-users  wrote:

> Hello,
>
>
>
> This is not exactly a PETSc question. I have a parallel partitioned finite
> element mesh. What are the steps involved in having a contiguous but unique
> set of node numbering from one partition to the next? There are nodes which
> are shared between different partitions. Moreover, this partition has to
> coincide parallel partition of PETSc Vec/Mat, which ensures data locality.
>
>
>
> If you can post the algorithm or cite a reference, it will prove helpful.
>

Somehow, you have to know what "nodes" are shared. Once you know this, you
can make a rule for numbering, such
as "the lowest rank gets the shared nodes". We encapsulate this ownership
relation in the PetscSF. Roots are owned,
and leaves are not owned. The rule above is not great for load balance, so
we have an optimization routine for the
simple PetscSF:
https://petsc.org/main/manualpages/DMPlex/DMPlexRebalanceSharedPoints/

  Thanks,

 Matt


> Many thanks.
>
>
>
> Kind regards,
>
> Karthik.
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

[petsc-users] Node numbering in parallel partitioned mesh

[Karthikeyan Chockalingam - STFC UKRI via petsc-users]

Hello,

This is not exactly a PETSc question. I have a parallel partitioned finite 
element mesh. What are the steps involved in having a contiguous but unique set 
of node numbering from one partition to the next? There are nodes which are 
shared between different partitions. Moreover, this partition has to coincide 
parallel partition of PETSc Vec/Mat, which ensures data locality.

If you can post the algorithm or cite a reference, it will prove helpful.

Many thanks.

Kind regards,
Karthik.

Re: [petsc-users] 'mpirun' run not found error

[Pierre Jolivet]

> On 2 May 2023, at 8:56 AM, ­권승리 / 학생 / 항공우주공학과  wrote:
> 
> Dear developers
> 
> I'm trying to use the mpi, but I'm encountering error messages like below:
> 
> 
> Command 'mpirun' not found, but can be installed with:
> sudo apt install lam-runtime   # version 7.1.4-6build2, or
> sudo apt install mpich # version 3.3.2-2build1
> sudo apt install openmpi-bin   # version 4.0.3-0ubuntu1
> sudo apt install slurm-wlm-torque  # version 19.05.5-1
> //
> 
> However, I've already installed the mpich.
> cd $PETSC_DIR
> ./configure --download-mpich --with-debugging=0 COPTFLAGS='-O3 -march=native 
> -mtune=native' CXXOPTFLAGS='-O3 -march=native -mtune=native' FOPTFLAGS='-O3 
> -march=native -mtune=native' --download-mumps --download-scalapack 
> --download-parmetis --download-metis --download-parmetis --download-hpddm 
> --download-slepc
> 
> Could you recommend some advice related to this?

Most likely you do not want to run mpirun, but 
${PETSC_DIR}/${PETSC_ARCH}/bin/mpirun instead.
Or add ${PETSC_DIR}/${PETSC_ARCH}/bin to your PATH environment variable.

Thanks,
Pierre

> Best,
> Seung Lee Kwon
> -- 
> Seung Lee Kwon, Ph.D.Candidate
> Aerospace Structures and Materials Laboratory
> Department of Mechanical and Aerospace Engineering
> Seoul National University
> Building 300 Rm 503, Gwanak-ro 1, Gwanak-gu, Seoul, South Korea, 08826
> E-mail : ksl7...@snu.ac.kr 
> Office : +82-2-880-7389
> C. P : +82-10-4695-1062

[petsc-users] 'mpirun' run not found error

[­권승리 / 학생 / 항공우주공학과]

Dear developers

I'm trying to use the mpi, but I'm encountering error messages like below:


Command 'mpirun' not found, but can be installed with:
sudo apt install lam-runtime   # version 7.1.4-6build2, or
sudo apt install mpich # version 3.3.2-2build1
sudo apt install openmpi-bin   # version 4.0.3-0ubuntu1
sudo apt install slurm-wlm-torque  # version 19.05.5-1
//

However, I've already installed the mpich.
cd $PETSC_DIR
./configure --download-mpich --with-debugging=0 COPTFLAGS='-O3
-march=native -mtune=native' CXXOPTFLAGS='-O3 -march=native -mtune=native'
FOPTFLAGS='-O3 -march=native -mtune=native' --download-mumps
--download-scalapack --download-parmetis --download-metis
--download-parmetis --download-hpddm --download-slepc

Could you recommend some advice related to this?

Best,
Seung Lee Kwon
-- 
Seung Lee Kwon, Ph.D.Candidate
Aerospace Structures and Materials Laboratory
Department of Mechanical and Aerospace Engineering
Seoul National University
Building 300 Rm 503, Gwanak-ro 1, Gwanak-gu, Seoul, South Korea, 08826
E-mail : ksl7...@snu.ac.kr
Office : +82-2-880-7389
C. P : +82-10-4695-1062