On Tue, May 31, 2022 at 10:26 AM Mike Michell <mi.mike1...@gmail.com> wrote:
> Thank you. But, it seems that PetscSFCreateSectionSF() also requires > petscsf.h file. Which header file I should include to call > PetscSFCreateSectionSF() from Fortran? > I will have to write a binding. I will send you the MR when I finish. THanks, Matt > Thanks, > Mike > > > On Tue, May 31, 2022 at 10:04 AM Mike Michell <mi.mike1...@gmail.com> >> wrote: >> >>> As a follow-up question on your example, is it possible to call >>> PetscSFCreateRemoteOffsets() from Fortran? >>> >>> My code is written in .F90 and in "petsc/finclude/" there is no >>> petscsf.h so that the code currently cannot find >>> PetscSFCreateRemoteOffsets(). >>> >> >> I believe if you pass in NULL for remoteOffsets, that function will be >> called internally. >> >> Thanks, >> >> Matt >> >> >>> Thanks, >>> Mike >>> >>> >>> I will also point out that Toby has created a nice example showing how >>>> to create an SF for halo exchange between local vectors. >>>> >>>> https://gitlab.com/petsc/petsc/-/merge_requests/5267 >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> On Sun, May 22, 2022 at 9:47 PM Matthew Knepley <knep...@gmail.com> >>>> wrote: >>>> >>>>> On Sun, May 22, 2022 at 4:28 PM Mike Michell <mi.mike1...@gmail.com> >>>>> wrote: >>>>> >>>>>> Thanks for the reply. The diagram makes sense and is helpful for >>>>>> understanding 1D representation. >>>>>> >>>>>> However, something is still unclear. From your diagram, the number of >>>>>> roots per process seems to vary according to run arguments, such as >>>>>> "-dm_distribute_overlap", because "the number of roots for a DMPlex is >>>>>> the >>>>>> number of mesh points in the local portion of the mesh (cited from your >>>>>> answer to my question (1))" will end up change according to that >>>>>> argument. >>>>>> However, from my mock-up code, number of roots is independent to >>>>>> -dm_distribute_overlap argument. The summation of "number of roots" >>>>>> through >>>>>> processes was always equal to number of physical vertex on my mesh, if I >>>>>> define the section layout on vertex with 1DOF. But in your diagram >>>>>> example, >>>>>> the summation of "nroots" is larger than the actual number of mesh >>>>>> points, >>>>>> which is 13. >>>>>> >>>>> >>>>> I do not understand your question. Notice the -dm_distribute_overlap >>>>> does _not_ change the owned points for any process. It only puts in new >>>>> leaves, so it also never >>>>> changes the roots for this way of using the SF. >>>>> >>>>> >>>>>> Also, it is still unclear how to get the size of "roots" from the >>>>>> PetscSection & PetscSF on distributed DMPlex? >>>>>> >>>>> >>>>> For an SF mapping ghost dofs in a global vector, the number of roots >>>>> is just the size of the local portion of the vector. >>>>> >>>>> >>>>>> In your diagram, how can you tell your code and make it allocate the >>>>>> "nroots=7 for P0, nroots=9 for P1, and nroots=7 for P2" arrays before you >>>>>> call PetscSFBcastBegin/End()? It seems that we need to define arrays >>>>>> having >>>>>> the size of nroots & nleaves before calling PetscSFBcastBegin/End(). >>>>>> >>>>> >>>>> I just want to note that this usage is different from the canonical >>>>> usage in Plex. It is fine to do this, but this will not match what I do in >>>>> the library if you look. >>>>> In Plex, I distinguish two linear spaces: >>>>> >>>>> 1) Global space: This is the vector space for the solvers. Each >>>>> point is uniquely represented and owned by some process >>>>> >>>>> 2) Local space: This is the vector space for assembly. Some points >>>>> are represented multiple times. >>>>> >>>>> I create an SF that maps from the global space (roots) to the local >>>>> space (leaves), and it is called in DMGlobalToLocal() (and >>>>> associated functions). This >>>>> is more natural in FEM. You seem to want an SF that maps between >>>>> global vectors. This will also work. The roots would be the local dofs, >>>>> and >>>>> the leaves >>>>> would be shared dofs. >>>>> >>>>> Does this make sense? >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Thanks, >>>>>> Mike >>>>>> >>>>>> Here's a diagram of a 1D mesh with overlap and 3 partitions, showing >>>>>>> what the petscsf data is for each. The number of roots is the number of >>>>>>> mesh points in the local representation, and the number of leaves is the >>>>>>> number of mesh points that are duplicates of mesh points on other >>>>>>> processes. With that in mind, answering your questions >>>>>>> >>>>>>> > (1) It seems that the "roots" means the number of vertex not >>>>>>> considering overlap layer, and "leaves" seems the number of distributed >>>>>>> vertex for each processor that includes overlap layer. Can you >>>>>>> acknowledge >>>>>>> that this is correct understanding? I have tried to find clearer >>>>>>> examples >>>>>>> from PETSc team's articles relevant to Star Forest, but I am still >>>>>>> unclear >>>>>>> about the exact relation & graphical notation of roots & leaves in SF if >>>>>>> it's the case of DMPlex solution arrays. >>>>>>> >>>>>>> No, the number of roots for a DMPlex is the number of mesh points in >>>>>>> the local portion of the mesh >>>>>>> >>>>>>> > (2) If it is so, there is an issue that I cannot define "root >>>>>>> data" and "leave data" generally. I am trying to following >>>>>>> "src/vec/is/sf/tutorials/ex1f.F90", however, in that example, size of >>>>>>> roots >>>>>>> and leaves are predefined as 6. How can I generalize that? Because I can >>>>>>> get size of leaves using DAG depth(or height), which is equal to number >>>>>>> of >>>>>>> vertices each proc has. But, how can I get the size of my "roots" region >>>>>>> from SF? Any example about that? This question is connected to how can I >>>>>>> define "rootdata" for "PetscSFBcastBegin/End()". >>>>>>> >>>>>>> Does the diagram help you generalize? >>>>>>> >>>>>>> > (3) More importantly, with the attached PetscSection & SF layout, >>>>>>> my vector is only resolved for the size equal to "number of roots" for >>>>>>> each >>>>>>> proc, but not for the overlapping area(i.e., "leaves"). What I wish to >>>>>>> do >>>>>>> is to exchange (or reduce) the solution data between each proc, in the >>>>>>> overlapping region. Can I get some advices why my vector does not >>>>>>> encompass >>>>>>> the "leaves" regime? Is there any example doing similar things? >>>>>>> Going back to my first response: if you use a section to say how >>>>>>> many pieces of data are associated with each local mesh point, then a >>>>>>> PetscSF is constructed that requires no more manipulation from you. >>>>>>> >>>>>>> >>>>>>> On Sun, May 22, 2022 at 10:47 AM Mike Michell <mi.mike1...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Thank you for the reply. >>>>>>>> The PetscSection and PetscSF objects are defined as in the attached >>>>>>>> mock-up code (Q_PetscSF_1.tar). 1-DOF is defined on vertex as my >>>>>>>> solution >>>>>>>> is determined on each vertex with 1-DOF from a finite-volume method. >>>>>>>> >>>>>>>> As follow up questions: >>>>>>>> (1) It seems that the "roots" means the number of vertex not >>>>>>>> considering overlap layer, and "leaves" seems the number of distributed >>>>>>>> vertex for each processor that includes overlap layer. Can you >>>>>>>> acknowledge >>>>>>>> that this is correct understanding? I have tried to find clearer >>>>>>>> examples >>>>>>>> from PETSc team's articles relevant to Star Forest, but I am still >>>>>>>> unclear >>>>>>>> about the exact relation & graphical notation of roots & leaves in SF >>>>>>>> if >>>>>>>> it's the case of DMPlex solution arrays. >>>>>>>> >>>>>>>> (2) If it is so, there is an issue that I cannot define "root data" >>>>>>>> and "leave data" generally. I am trying to following >>>>>>>> "src/vec/is/sf/tutorials/ex1f.F90", however, in that example, size of >>>>>>>> roots >>>>>>>> and leaves are predefined as 6. How can I generalize that? Because I >>>>>>>> can >>>>>>>> get size of leaves using DAG depth(or height), which is equal to >>>>>>>> number of >>>>>>>> vertices each proc has. But, how can I get the size of my "roots" >>>>>>>> region >>>>>>>> from SF? Any example about that? This question is connected to how can >>>>>>>> I >>>>>>>> define "rootdata" for "PetscSFBcastBegin/End()". >>>>>>>> >>>>>>>> (3) More importantly, with the attached PetscSection & SF layout, >>>>>>>> my vector is only resolved for the size equal to "number of roots" for >>>>>>>> each >>>>>>>> proc, but not for the overlapping area(i.e., "leaves"). What I wish to >>>>>>>> do >>>>>>>> is to exchange (or reduce) the solution data between each proc, in the >>>>>>>> overlapping region. Can I get some advices why my vector does not >>>>>>>> encompass >>>>>>>> the "leaves" regime? Is there any example doing similar things? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Mike >>>>>>>> >>>>>>>> >>>>>>>>> On Fri, May 20, 2022 at 4:45 PM Mike Michell < >>>>>>>>> mi.mike1...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Thanks for the reply. >>>>>>>>>> >>>>>>>>>> > "What I want to do is to exchange data (probably just >>>>>>>>>> MPI_Reduce)" which confuses me, because halo exchange is a >>>>>>>>>> point-to-point >>>>>>>>>> exchange and not a reduction. Can you clarify? >>>>>>>>>> PetscSFReduceBegin/End seems to be the function that do reduction >>>>>>>>>> for PetscSF object. What I intended to mention was either reduction >>>>>>>>>> or >>>>>>>>>> exchange, not specifically intended "reduction". >>>>>>>>>> >>>>>>>>>> As a follow-up question: >>>>>>>>>> Assuming that the code has its own local solution arrays (not >>>>>>>>>> Petsc type), and if the plex's DAG indices belong to the halo region >>>>>>>>>> are >>>>>>>>>> the only information that I want to know (not the detailed section >>>>>>>>>> description, such as degree of freedom on vertex, cells, etc.). I >>>>>>>>>> have >>>>>>>>>> another PetscSection for printing out my solution. >>>>>>>>>> Also if I can convert that DAG indices into my local cell/vertex >>>>>>>>>> index, can I just use the PetscSF object created from DMGetPointSF(), >>>>>>>>>> instead of "creating PetscSection + DMGetSectionSF()"? In other >>>>>>>>>> words, can >>>>>>>>>> I use the PetscSF object declared from DMGetPointSF() for the halo >>>>>>>>>> communication? >>>>>>>>>> >>>>>>>>> >>>>>>>>> No, because that point SF will index information by point number. >>>>>>>>> You would need to build a new SF that indexes your dofs. The steps >>>>>>>>> you would >>>>>>>>> go through are exactly the same as you would if you just told us >>>>>>>>> what the Section is that indexes your data. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Matt >>>>>>>>> >>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Mike >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> The PetscSF that is created automatically is the "point sf" ( >>>>>>>>>>> https://petsc.org/main/docs/manualpages/DM/DMGetPointSF/): it >>>>>>>>>>> says which mesh points (cells, faces, edges and vertices) are >>>>>>>>>>> duplicates of >>>>>>>>>>> others. >>>>>>>>>>> >>>>>>>>>>> In a finite volume application we typically want to assign >>>>>>>>>>> degrees of freedom just to cells: some applications may only have >>>>>>>>>>> one >>>>>>>>>>> degree of freedom, others may have multiple. >>>>>>>>>>> >>>>>>>>>>> You encode where you want degrees of freedom in a PetscSection >>>>>>>>>>> and set that as the section for the DM in DMSetLocalSection() ( >>>>>>>>>>> https://petsc.org/release/docs/manualpages/DM/DMSetLocalSection.html >>>>>>>>>>> ) >>>>>>>>>>> >>>>>>>>>>> (A c example of these steps that sets degrees of freedom for >>>>>>>>>>> *vertices* instead of cells is `src/dm/impls/plex/tutorials/ex7.c`) >>>>>>>>>>> >>>>>>>>>>> After that you can call DMGetSectionSF() ( >>>>>>>>>>> https://petsc.org/main/docs/manualpages/DM/DMGetSectionSF/) to >>>>>>>>>>> the the PetscSF that you want for halo exchange: the one for your >>>>>>>>>>> solution >>>>>>>>>>> variables. >>>>>>>>>>> >>>>>>>>>>> After that, the only calls you typically need in a finite volume >>>>>>>>>>> code is PetscSFBcastBegin() to start a halo exchange and >>>>>>>>>>> PetscSFBcastEnd() >>>>>>>>>>> to complete it. >>>>>>>>>>> >>>>>>>>>>> You say >>>>>>>>>>> >>>>>>>>>>> > What I want to do is to exchange data (probably just >>>>>>>>>>> MPI_Reduce) >>>>>>>>>>> >>>>>>>>>>> which confuses me, because halo exchange is a point-to-point >>>>>>>>>>> exchange and not a reduction. Can you clarify? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Fri, May 20, 2022 at 8:35 PM Mike Michell < >>>>>>>>>>> mi.mike1...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Dear PETSc developer team, >>>>>>>>>>>> >>>>>>>>>>>> Hi, I am using DMPlex for a finite-volume code and trying to >>>>>>>>>>>> understand the usage of PetscSF. What is a typical procedure for >>>>>>>>>>>> doing halo >>>>>>>>>>>> data exchange at parallel boundary using PetscSF object on DMPlex? >>>>>>>>>>>> Is there >>>>>>>>>>>> any example that I can refer to usage of PetscSF with distributed >>>>>>>>>>>> DMPlex? >>>>>>>>>>>> >>>>>>>>>>>> Assuming to use the attached mock-up code and mesh, if I give >>>>>>>>>>>> "-dm_distribute_overlap 1 -over_dm_view" to run the code, I can >>>>>>>>>>>> see a >>>>>>>>>>>> PetscSF object is already created, although I have not called >>>>>>>>>>>> "PetscSFCreate" in the code. How can I import & use that PetscSF >>>>>>>>>>>> already >>>>>>>>>>>> created by the code to do the halo data exchange? >>>>>>>>>>>> >>>>>>>>>>>> What I want to do is to exchange data (probably just >>>>>>>>>>>> MPI_Reduce) in a parallel boundary region using PetscSF and its >>>>>>>>>>>> functions. >>>>>>>>>>>> I might need to have an overlapping layer or not. >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Mike >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>> experiments is infinitely more interesting than any results to which >>>>>>>>> their >>>>>>>>> experiments lead. >>>>>>>>> -- Norbert Wiener >>>>>>>>> >>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>>> <http://www.cse.buffalo.edu/~knepley/> >>>>>>>>> >>>>>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> <http://www.cse.buffalo.edu/~knepley/> >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> <http://www.cse.buffalo.edu/~knepley/> >>>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> <http://www.cse.buffalo.edu/~knepley/> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>