Re: [petsc-users] DMLocalToLocal() and DMCoarsen() for DMPlex with Fortran

[Matthew Knepley]

On Thu, Dec 15, 2022 at 10:10 PM Mike Michell  wrote:

> Hello PETSc developer team,
>
> I am a user of DMPlex in PETSc with Fortran. I have two questions:
>
> - Is DMLocalToLocal() now available for DMPlex with Fortran? I made
> similar inquiry before:
> https://www.mail-archive.com/petsc-users@mcs.anl.gov/msg44500.html
>

There is a DMDA example (
https://gitlab.com/petsc/petsc/-/blob/main/src/dm/tutorials/ex13f90.F90) so
the Fortran binding works. However, there is no Plex implementation. I can
add it to the TODO list.


> - Is there any example that can see how DMCoarsen() works? I can see
> either src/dm/impls/stag/tutorials/ex4.c  or  src/ksp/ksp/tutorials/ex65.c
> from the example folder. However, it is a bit tough to get an idea of how
> DMCoarsen() works. What can be the "coarsening" criteria? Is it uniformly
> coarsening over the domain? or Can it be variable-gradient based? Having
> more examples would be very helpful.
>

DMCoarsen really only applies to more structured grids. There is a
definition for unstructured grids, and we have written a paper
about it (https://arxiv.org/abs/1104.0261), but that code could not be
maintained and was difficult to generalize. Right now, if you want
unstructured coarsening, I would recommend trying DMPlexAdaptMetric() which
works with MMG, or DMPlexAdaptLabel() which works
with p4est.

  Thanks,

 Matt


> Thanks,
> Mike
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] dmplex normal vector incorrect for periodic gmsh grids

[Matthew Knepley]

On Fri, Dec 16, 2022 at 12:22 AM Praveen C  wrote:

> Thank you very much. I do see correct normals now.
>
> Is there a way to set the option
>
> -dm_localize_height 1
>>
>
> within the code ?
>

The problem is that the localization happens within the Gmsh construction,
so it is difficult to insert an API.
We could use a callback, but that rapidly becomes unwieldy. If you want to
do it programmatically, I would use
PetscOptionsSetValue().

  Thanks,

  Matt


> best
> praveen
>
> On 15-Dec-2022, at 9:53 PM, Matthew Knepley  wrote:
>
> On Wed, Dec 14, 2022 at 9:20 AM Praveen C  wrote:
>
>> Hello
>>
>> I tried this with the attached code but I still get wrong normals.
>>
>> ./dmplex -dm_plex_filename ug_periodic.msh -dm_localize_height 1
>>
>> Code and grid file are attached
>>
>
> I have fixed the code. The above options should now work for you.
>
>   Thanks,
>
>   Matt
>
>
>> Thank you
>> praveen
>>
>>
>>
>> On 14-Dec-2022, at 6:40 PM, Matthew Knepley  wrote:
>>
>> On Wed, Dec 14, 2022 at 2:38 AM Praveen C  wrote:
>>
>>> Thank you, this MR works if I generate the mesh within the code.
>>>
>>> But using a mesh made in gmsh, I see the same issue.
>>>
>>
>> Because the operations happen in a different order. If you read in your
>> mesh with
>>
>>   -dm_plex_filename mymesh.gmsh -dm_localize_height 1
>>
>> then it should work.
>>
>>   Thanks,
>>
>>  Matt
>>
>>
>>> Thanks
>>> praveen
>>>
>>> On 14-Dec-2022, at 12:51 AM, Matthew Knepley  wrote:
>>>
>>> On Tue, Dec 13, 2022 at 10:57 AM Matthew Knepley 
>>> wrote:
>>>
 On Tue, Dec 13, 2022 at 6:11 AM Praveen C  wrote:

> Hello
>
> In the attached test, I read a small grid made in gmsh with periodic
> bc.
>
> This is a 2d mesh.
>
> The cell numbers are shown in the figure.
>
> All faces have length = 2.5
>
> But using PetscFVFaceGeom I am getting length of 7.5 for some faces.
> E.g.,
>
> face: 59, centroid = 3.75, 2.50, normal = 0.00, -7.50
> ===> Face length incorrect = 7.50, should be 2.5
> support[0] = 11, cent = 8.75, 3.75, area = 6.25
> support[1] = 15, cent = 8.75, 1.25, area = 6.25
>
> There are also errors in the orientation of normal.
>
> If we disable periodicity in geo file, this error goes away.
>

 Yes, by default we only localize coordinates for cells. I can put in
 code to localize faces.

>>>
>>> Okay, I now have a MR for this:
>>> https://gitlab.com/petsc/petsc/-/merge_requests/5917
>>>
>>> I am attaching your code, slightly modified. You can run
>>>
>>>   ./dmplex -malloc_debug 0 -dm_plex_box_upper 10,10 -dm_plex_box_faces
>>> 4,4 -dm_plex_simplex 0 -dm_view ::ascii_info_detail -draw_pause 3
>>> -dm_plex_box_bd periodic,periodic -dm_localize_height 0
>>>
>>> which shows incorrect edges and
>>>
>>> ./dmplex -malloc_debug 0 -dm_plex_box_upper 10,10 -dm_plex_box_faces 4,4
>>> -dm_plex_simplex 0 -dm_view ::ascii_info_detail -draw_pause 3
>>> -dm_plex_box_bd periodic,periodic -dm_localize_height 1
>>>
>>> which is correct. If you want to control things yourself, instead of
>>> using the option you can call DMPlexSetMaxProjectionHeight() on the
>>> coordinate DM yourself.
>>>
>>>   Thanks,
>>>
>>>  Matt
>>>
>>>
   Thanks,

 Matt


> Thanks
> praveen
>
 --


>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> 
>>
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> 
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

[petsc-users] Help with input construction hang on 2-GPU CG Solve

[Rohan Yadav]

Hi,

I'm developing a microbenchmark that runs a CG solve using PETSc on a mesh
using a 5-point stencil matrix. My code (linked here:
https://github.com/rohany/petsc-pde-benchmark/blob/main/main.cpp, only 120
lines) works on 1 GPU and has great performance. When I move to 2 GPUs, the
program appears to get stuck in the input generation. I've literred the
code with print statements and have found out the following clues:

* The first rank progresses through this loop:
https://github.com/rohany/petsc-pde-benchmark/blob/main/main.cpp#L44, but
then does not exit (it seems to get stuck right before rowStart == rowEnd)
* The second rank makes very few iterations through the loop for its
allotted rows.

Therefore, neither rank makes it to the call to MatAssemblyBegin.

I'm running the code using the following command line on the Summit
supercomputer:
```
jsrun -n 2 -g 1 -c 1 -b rs -r 2
/gpfs/alpine/scratch/rohany/csc335/petsc-pde-benchmark/main -ksp_max_it 200
-ksp_type cg -pc_type none -ksp_atol 1e-10 -ksp_rtol 1e-10 -vec_type cuda
-mat_type aijcusparse -use_gpu_aware_mpi 0 -nx 8485 -ny 8485
```

Any suggestions will be appreciated! I feel that I have applied many of the
common petsc optimizations of preallocating my matrix row counts, so I'm
not sure what's going on with this input generation.

Thanks,

Rohan Yadav