Re: [petsc-users] DMLocalToLocal with DMPlex in Fortran

[Mike Michell]

Thanks a lot for the reply.

It is possible that it will make a difference. An explicit code, running
completely from local vectors, which assembles its own residuals without
mapping to global vectors,
could probably realize some gain. It would be interesting to see the
numbers.
=> Agree with this. Even with an implicit scheme, I need the halo exchange
of local-to-local while constructing the local matrix object (for example,
summation over control volume components). Thus I expect some computational
gain with this.

In December, will try to remind you via this email chain.

Thanks,
Mike


> On Tue, Oct 11, 2022 at 6:04 PM Mike Michell 
> wrote:
>
>> Thank you for the reply and checking.
>> Indeed, it seems that local-to-local halo is still not implemented for
>> DMPlex. But I believe this is a very required feature for large 3D
>> simulations with DMPlex.
>>
>
> It is possible that it will make a difference. An explicit code, running
> completely from local vectors, which assembles its own residuals without
> mapping to global vectors,
> could probably realize some gain. It would be interesting to see the
> numbers.
>
>
>> Would you mind if I ask for any estimated timeline to fix this issue and
>> put it on the official version of PETSc? If I remember correctly, we had a
>> similar discussion a few months ago.
>>
>
> I cannot do it until November since we have a big review coming up.
>
>   Thanks,
>
>  Matt
>
>
>> Thanks,
>> Mike
>>
>> On Mon, Oct 10, 2022 at 10:41 PM Mike Michell 
>>> wrote:
>>>
 Hi, I was wondering if there is any comment on the example file that I
 can refer to.

>>>
>>> I see the problem. Local2Local is not implemented for Plex. I thought we
>>> had this automated, but it was only
>>> coded for DMDA. It is a fairly mechanical transformation of the
>>> Global2Local, just remapping indices, but it
>>> will take some time since there is a lot of backlog this semester.
>>>
>>> I have fixed the error message so now it is obvious what the problem is.
>>>
>>>   Thanks,
>>>
>>>   Matt
>>>
>>>
 Thanks,
 Mike


> Thank you for the reply.
> Sure, a short example code is attached here with a square box mesh and
> a run script.
> Inside the source, you may find two versions of halo exchange; one is
> for local to global (Version-1) and another one is for local to local
> (Version-2), which is not working in my case.
> In the output.vtu, you will see the halo exchanged vector resolved to
> each vertex with (myrank + 1), so if the code is running with 2procs, at
> the parallel boundary, you will see 3. In this example, there is no ghost
> layer.
>
> Thanks,
> Mike
>
>
>> On Sat, Oct 1, 2022 at 8:51 PM Mike Michell 
>> wrote:
>>
>>> Thank you for the reply. There is that file in
>>> src/dm/interface/ftn-auto/ for me, instead of the path you mentioned.
>>>
>>> After "make allfortranstubs" was done and, PETSc reconfigured and
>>> reinstalled.
>>>
>>> However, I still have the same problem at the line in which
>>> DMLocalToLocalBegin() is used. What I am doing to setup halo exchange 
>>> is as
>>> follows;
>>> - declare DMPlex
>>> - PetscSectionCreate()
>>> - PetscSectionSetNumFields()
>>> - PetscSectionSetFieldComponents()
>>> - PetscSectionSetChart()
>>> - do loop over dofs: PetscSectionSetDof() and
>>> PetscSectionSetFieldDof()
>>> - PetscSectionSetUp()
>>> - DMSetLocalSection()
>>> - PetscSectionDestroy()
>>> - DMGetSectionSF()
>>> - PetscSFSetUp()
>>>
>>> Then, the halo exchange is performed as follows;
>>> - DMGetLocalVector()
>>> - Fill the local vector
>>> - DMLocalToLocalBegin() --(*)
>>> - DMLocalToLocalEnd()
>>> - DMRestoreLocalVector()
>>>
>>> Then, the code crashes at (*).
>>>
>>
>> Can you send something I can run? Then I will find the problem and
>> fix it.
>>
>>   Thanks,
>>
>>  Matt
>>
>>
>>> Previously(at the time PETSc did not support LocalToLocal for DMPlex
>>> in Fortran), the part above, "DMLocalToLocalBegin() and
>>> DMLocalToLocalEnd()", consisted of;
>>> - DMLocalToGlobalBegin()
>>> - DMLocalToGlobalEnd()
>>> - DMGlobalToLocalBegin()
>>> - DMGlobalToLocalEnd()
>>> and it worked okay.
>>>
>>> I am unclear which part is causing the problem. Shall I define the
>>> PetscSection and PetscSF in different ways in case of Local to Local 
>>> Halo
>>> exchange?
>>> Any comment will be appreciated.
>>>
>>> Thanks,
>>> Mike
>>>
>>>
>>>
 On Fri, Sep 30, 2022 at 4:14 PM Mike Michell 
 wrote:

> Hi,
>
> As a follow-up to this email thread,
> https://www.mail-archive.com/petsc-users@mcs.anl.gov/msg44070.html
>
> Are DMLocalToLocalBegin() and DMLocalToLocalEnd() really 

Re: [petsc-users] DMLocalToLocal with DMPlex in Fortran

[Matthew Knepley]

On Tue, Oct 11, 2022 at 6:04 PM Mike Michell  wrote:

> Thank you for the reply and checking.
> Indeed, it seems that local-to-local halo is still not implemented for
> DMPlex. But I believe this is a very required feature for large 3D
> simulations with DMPlex.
>

It is possible that it will make a difference. An explicit code, running
completely from local vectors, which assembles its own residuals without
mapping to global vectors,
could probably realize some gain. It would be interesting to see the
numbers.


> Would you mind if I ask for any estimated timeline to fix this issue and
> put it on the official version of PETSc? If I remember correctly, we had a
> similar discussion a few months ago.
>

I cannot do it until November since we have a big review coming up.

  Thanks,

 Matt


> Thanks,
> Mike
>
> On Mon, Oct 10, 2022 at 10:41 PM Mike Michell 
>> wrote:
>>
>>> Hi, I was wondering if there is any comment on the example file that I
>>> can refer to.
>>>
>>
>> I see the problem. Local2Local is not implemented for Plex. I thought we
>> had this automated, but it was only
>> coded for DMDA. It is a fairly mechanical transformation of the
>> Global2Local, just remapping indices, but it
>> will take some time since there is a lot of backlog this semester.
>>
>> I have fixed the error message so now it is obvious what the problem is.
>>
>>   Thanks,
>>
>>   Matt
>>
>>
>>> Thanks,
>>> Mike
>>>
>>>
 Thank you for the reply.
 Sure, a short example code is attached here with a square box mesh and
 a run script.
 Inside the source, you may find two versions of halo exchange; one is
 for local to global (Version-1) and another one is for local to local
 (Version-2), which is not working in my case.
 In the output.vtu, you will see the halo exchanged vector resolved to
 each vertex with (myrank + 1), so if the code is running with 2procs, at
 the parallel boundary, you will see 3. In this example, there is no ghost
 layer.

 Thanks,
 Mike


> On Sat, Oct 1, 2022 at 8:51 PM Mike Michell 
> wrote:
>
>> Thank you for the reply. There is that file in
>> src/dm/interface/ftn-auto/ for me, instead of the path you mentioned.
>>
>> After "make allfortranstubs" was done and, PETSc reconfigured and
>> reinstalled.
>>
>> However, I still have the same problem at the line in which
>> DMLocalToLocalBegin() is used. What I am doing to setup halo exchange is 
>> as
>> follows;
>> - declare DMPlex
>> - PetscSectionCreate()
>> - PetscSectionSetNumFields()
>> - PetscSectionSetFieldComponents()
>> - PetscSectionSetChart()
>> - do loop over dofs: PetscSectionSetDof() and
>> PetscSectionSetFieldDof()
>> - PetscSectionSetUp()
>> - DMSetLocalSection()
>> - PetscSectionDestroy()
>> - DMGetSectionSF()
>> - PetscSFSetUp()
>>
>> Then, the halo exchange is performed as follows;
>> - DMGetLocalVector()
>> - Fill the local vector
>> - DMLocalToLocalBegin() --(*)
>> - DMLocalToLocalEnd()
>> - DMRestoreLocalVector()
>>
>> Then, the code crashes at (*).
>>
>
> Can you send something I can run? Then I will find the problem and fix
> it.
>
>   Thanks,
>
>  Matt
>
>
>> Previously(at the time PETSc did not support LocalToLocal for DMPlex
>> in Fortran), the part above, "DMLocalToLocalBegin() and
>> DMLocalToLocalEnd()", consisted of;
>> - DMLocalToGlobalBegin()
>> - DMLocalToGlobalEnd()
>> - DMGlobalToLocalBegin()
>> - DMGlobalToLocalEnd()
>> and it worked okay.
>>
>> I am unclear which part is causing the problem. Shall I define the
>> PetscSection and PetscSF in different ways in case of Local to Local Halo
>> exchange?
>> Any comment will be appreciated.
>>
>> Thanks,
>> Mike
>>
>>
>>
>>> On Fri, Sep 30, 2022 at 4:14 PM Mike Michell 
>>> wrote:
>>>
 Hi,

 As a follow-up to this email thread,
 https://www.mail-archive.com/petsc-users@mcs.anl.gov/msg44070.html

 Are DMLocalToLocalBegin() and DMLocalToLocalEnd() really available
 for DMPlex with Fortran on the latest version of PETSc (3.17.99 from
 GitLab)? Matt commented that the Fortran bindings were updated so that
 those functions must be available in the latest version of PETSc, 
 however,
 it seems still they are not working from my test with DMPlex in 
 Fortran.
 Can anyone provide some comments? Probably I am missing some mandatory
 header file? Currently, I have headers;

 #include "petsc/finclude/petscvec.h"
 #include "petsc/finclude/petscdmplex.h"
 #include "petsc/finclude/petscdmlabel.h"
 #include "petsc/finclude/petscdm.h"

>>>
>>> The source for these functions is in
>>>
>>>   

Re: [petsc-users] DMLocalToLocal with DMPlex in Fortran

[Mike Michell]

Thank you for the reply and checking.
Indeed, it seems that local-to-local halo is still not implemented for
DMPlex. But I believe this is a very required feature for large 3D
simulations with DMPlex.
Would you mind if I ask for any estimated timeline to fix this issue and
put it on the official version of PETSc? If I remember correctly, we had a
similar discussion a few months ago.

Thanks,
Mike

On Mon, Oct 10, 2022 at 10:41 PM Mike Michell  wrote:
>
>> Hi, I was wondering if there is any comment on the example file that I
>> can refer to.
>>
>
> I see the problem. Local2Local is not implemented for Plex. I thought we
> had this automated, but it was only
> coded for DMDA. It is a fairly mechanical transformation of the
> Global2Local, just remapping indices, but it
> will take some time since there is a lot of backlog this semester.
>
> I have fixed the error message so now it is obvious what the problem is.
>
>   Thanks,
>
>   Matt
>
>
>> Thanks,
>> Mike
>>
>>
>>> Thank you for the reply.
>>> Sure, a short example code is attached here with a square box mesh and a
>>> run script.
>>> Inside the source, you may find two versions of halo exchange; one is
>>> for local to global (Version-1) and another one is for local to local
>>> (Version-2), which is not working in my case.
>>> In the output.vtu, you will see the halo exchanged vector resolved to
>>> each vertex with (myrank + 1), so if the code is running with 2procs, at
>>> the parallel boundary, you will see 3. In this example, there is no ghost
>>> layer.
>>>
>>> Thanks,
>>> Mike
>>>
>>>
 On Sat, Oct 1, 2022 at 8:51 PM Mike Michell 
 wrote:

> Thank you for the reply. There is that file in
> src/dm/interface/ftn-auto/ for me, instead of the path you mentioned.
>
> After "make allfortranstubs" was done and, PETSc reconfigured and
> reinstalled.
>
> However, I still have the same problem at the line in which
> DMLocalToLocalBegin() is used. What I am doing to setup halo exchange is 
> as
> follows;
> - declare DMPlex
> - PetscSectionCreate()
> - PetscSectionSetNumFields()
> - PetscSectionSetFieldComponents()
> - PetscSectionSetChart()
> - do loop over dofs: PetscSectionSetDof() and PetscSectionSetFieldDof()
> - PetscSectionSetUp()
> - DMSetLocalSection()
> - PetscSectionDestroy()
> - DMGetSectionSF()
> - PetscSFSetUp()
>
> Then, the halo exchange is performed as follows;
> - DMGetLocalVector()
> - Fill the local vector
> - DMLocalToLocalBegin() --(*)
> - DMLocalToLocalEnd()
> - DMRestoreLocalVector()
>
> Then, the code crashes at (*).
>

 Can you send something I can run? Then I will find the problem and fix
 it.

   Thanks,

  Matt


> Previously(at the time PETSc did not support LocalToLocal for DMPlex
> in Fortran), the part above, "DMLocalToLocalBegin() and
> DMLocalToLocalEnd()", consisted of;
> - DMLocalToGlobalBegin()
> - DMLocalToGlobalEnd()
> - DMGlobalToLocalBegin()
> - DMGlobalToLocalEnd()
> and it worked okay.
>
> I am unclear which part is causing the problem. Shall I define the
> PetscSection and PetscSF in different ways in case of Local to Local Halo
> exchange?
> Any comment will be appreciated.
>
> Thanks,
> Mike
>
>
>
>> On Fri, Sep 30, 2022 at 4:14 PM Mike Michell 
>> wrote:
>>
>>> Hi,
>>>
>>> As a follow-up to this email thread,
>>> https://www.mail-archive.com/petsc-users@mcs.anl.gov/msg44070.html
>>>
>>> Are DMLocalToLocalBegin() and DMLocalToLocalEnd() really available
>>> for DMPlex with Fortran on the latest version of PETSc (3.17.99 from
>>> GitLab)? Matt commented that the Fortran bindings were updated so that
>>> those functions must be available in the latest version of PETSc, 
>>> however,
>>> it seems still they are not working from my test with DMPlex in Fortran.
>>> Can anyone provide some comments? Probably I am missing some mandatory
>>> header file? Currently, I have headers;
>>>
>>> #include "petsc/finclude/petscvec.h"
>>> #include "petsc/finclude/petscdmplex.h"
>>> #include "petsc/finclude/petscdmlabel.h"
>>> #include "petsc/finclude/petscdm.h"
>>>
>>
>> The source for these functions is in
>>
>>   src/dm/ftn-auto/dmf.c
>>
>> Is it there for you? If not, you can run
>>
>>   make allfortranstubs
>>
>> Fortran functions are not declared, so the header should not matter
>> for compilation, just the libraries for linking.
>>
>>   Thanks,
>>
>>  Matt
>>
>>
>>> Thanks,
>>> Mike
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which 
>> their
>> experiments lead.
>> 

Re: [petsc-users] DMLocalToLocal with DMPlex in Fortran

[Matthew Knepley]

On Mon, Oct 10, 2022 at 10:41 PM Mike Michell  wrote:

> Hi, I was wondering if there is any comment on the example file that I can
> refer to.
>

I see the problem. Local2Local is not implemented for Plex. I thought we
had this automated, but it was only
coded for DMDA. It is a fairly mechanical transformation of the
Global2Local, just remapping indices, but it
will take some time since there is a lot of backlog this semester.

I have fixed the error message so now it is obvious what the problem is.

  Thanks,

  Matt


> Thanks,
> Mike
>
>
>> Thank you for the reply.
>> Sure, a short example code is attached here with a square box mesh and a
>> run script.
>> Inside the source, you may find two versions of halo exchange; one is for
>> local to global (Version-1) and another one is for local to local
>> (Version-2), which is not working in my case.
>> In the output.vtu, you will see the halo exchanged vector resolved to
>> each vertex with (myrank + 1), so if the code is running with 2procs, at
>> the parallel boundary, you will see 3. In this example, there is no ghost
>> layer.
>>
>> Thanks,
>> Mike
>>
>>
>>> On Sat, Oct 1, 2022 at 8:51 PM Mike Michell 
>>> wrote:
>>>
 Thank you for the reply. There is that file in
 src/dm/interface/ftn-auto/ for me, instead of the path you mentioned.

 After "make allfortranstubs" was done and, PETSc reconfigured and
 reinstalled.

 However, I still have the same problem at the line in which
 DMLocalToLocalBegin() is used. What I am doing to setup halo exchange is as
 follows;
 - declare DMPlex
 - PetscSectionCreate()
 - PetscSectionSetNumFields()
 - PetscSectionSetFieldComponents()
 - PetscSectionSetChart()
 - do loop over dofs: PetscSectionSetDof() and PetscSectionSetFieldDof()
 - PetscSectionSetUp()
 - DMSetLocalSection()
 - PetscSectionDestroy()
 - DMGetSectionSF()
 - PetscSFSetUp()

 Then, the halo exchange is performed as follows;
 - DMGetLocalVector()
 - Fill the local vector
 - DMLocalToLocalBegin() --(*)
 - DMLocalToLocalEnd()
 - DMRestoreLocalVector()

 Then, the code crashes at (*).

>>>
>>> Can you send something I can run? Then I will find the problem and fix
>>> it.
>>>
>>>   Thanks,
>>>
>>>  Matt
>>>
>>>
 Previously(at the time PETSc did not support LocalToLocal for DMPlex in
 Fortran), the part above, "DMLocalToLocalBegin() and DMLocalToLocalEnd()",
 consisted of;
 - DMLocalToGlobalBegin()
 - DMLocalToGlobalEnd()
 - DMGlobalToLocalBegin()
 - DMGlobalToLocalEnd()
 and it worked okay.

 I am unclear which part is causing the problem. Shall I define the
 PetscSection and PetscSF in different ways in case of Local to Local Halo
 exchange?
 Any comment will be appreciated.

 Thanks,
 Mike



> On Fri, Sep 30, 2022 at 4:14 PM Mike Michell 
> wrote:
>
>> Hi,
>>
>> As a follow-up to this email thread,
>> https://www.mail-archive.com/petsc-users@mcs.anl.gov/msg44070.html
>>
>> Are DMLocalToLocalBegin() and DMLocalToLocalEnd() really available
>> for DMPlex with Fortran on the latest version of PETSc (3.17.99 from
>> GitLab)? Matt commented that the Fortran bindings were updated so that
>> those functions must be available in the latest version of PETSc, 
>> however,
>> it seems still they are not working from my test with DMPlex in Fortran.
>> Can anyone provide some comments? Probably I am missing some mandatory
>> header file? Currently, I have headers;
>>
>> #include "petsc/finclude/petscvec.h"
>> #include "petsc/finclude/petscdmplex.h"
>> #include "petsc/finclude/petscdmlabel.h"
>> #include "petsc/finclude/petscdm.h"
>>
>
> The source for these functions is in
>
>   src/dm/ftn-auto/dmf.c
>
> Is it there for you? If not, you can run
>
>   make allfortranstubs
>
> Fortran functions are not declared, so the header should not matter
> for compilation, just the libraries for linking.
>
>   Thanks,
>
>  Matt
>
>
>> Thanks,
>> Mike
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> 
>

>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> 
>>>
>>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their

Re: [petsc-users] DMLocalToLocal with DMPlex in Fortran

[Mike Michell]

Hi, I was wondering if there is any comment on the example file that I can
refer to.

Thanks,
Mike


> Thank you for the reply.
> Sure, a short example code is attached here with a square box mesh and a
> run script.
> Inside the source, you may find two versions of halo exchange; one is for
> local to global (Version-1) and another one is for local to local
> (Version-2), which is not working in my case.
> In the output.vtu, you will see the halo exchanged vector resolved to each
> vertex with (myrank + 1), so if the code is running with 2procs, at the
> parallel boundary, you will see 3. In this example, there is no ghost layer.
>
> Thanks,
> Mike
>
>
>> On Sat, Oct 1, 2022 at 8:51 PM Mike Michell 
>> wrote:
>>
>>> Thank you for the reply. There is that file in
>>> src/dm/interface/ftn-auto/ for me, instead of the path you mentioned.
>>>
>>> After "make allfortranstubs" was done and, PETSc reconfigured and
>>> reinstalled.
>>>
>>> However, I still have the same problem at the line in which
>>> DMLocalToLocalBegin() is used. What I am doing to setup halo exchange is as
>>> follows;
>>> - declare DMPlex
>>> - PetscSectionCreate()
>>> - PetscSectionSetNumFields()
>>> - PetscSectionSetFieldComponents()
>>> - PetscSectionSetChart()
>>> - do loop over dofs: PetscSectionSetDof() and PetscSectionSetFieldDof()
>>> - PetscSectionSetUp()
>>> - DMSetLocalSection()
>>> - PetscSectionDestroy()
>>> - DMGetSectionSF()
>>> - PetscSFSetUp()
>>>
>>> Then, the halo exchange is performed as follows;
>>> - DMGetLocalVector()
>>> - Fill the local vector
>>> - DMLocalToLocalBegin() --(*)
>>> - DMLocalToLocalEnd()
>>> - DMRestoreLocalVector()
>>>
>>> Then, the code crashes at (*).
>>>
>>
>> Can you send something I can run? Then I will find the problem and fix it.
>>
>>   Thanks,
>>
>>  Matt
>>
>>
>>> Previously(at the time PETSc did not support LocalToLocal for DMPlex in
>>> Fortran), the part above, "DMLocalToLocalBegin() and DMLocalToLocalEnd()",
>>> consisted of;
>>> - DMLocalToGlobalBegin()
>>> - DMLocalToGlobalEnd()
>>> - DMGlobalToLocalBegin()
>>> - DMGlobalToLocalEnd()
>>> and it worked okay.
>>>
>>> I am unclear which part is causing the problem. Shall I define the
>>> PetscSection and PetscSF in different ways in case of Local to Local Halo
>>> exchange?
>>> Any comment will be appreciated.
>>>
>>> Thanks,
>>> Mike
>>>
>>>
>>>
 On Fri, Sep 30, 2022 at 4:14 PM Mike Michell 
 wrote:

> Hi,
>
> As a follow-up to this email thread,
> https://www.mail-archive.com/petsc-users@mcs.anl.gov/msg44070.html
>
> Are DMLocalToLocalBegin() and DMLocalToLocalEnd() really available for
> DMPlex with Fortran on the latest version of PETSc (3.17.99 from GitLab)?
> Matt commented that the Fortran bindings were updated so that those
> functions must be available in the latest version of PETSc, however, it
> seems still they are not working from my test with DMPlex in Fortran. Can
> anyone provide some comments? Probably I am missing some mandatory header
> file? Currently, I have headers;
>
> #include "petsc/finclude/petscvec.h"
> #include "petsc/finclude/petscdmplex.h"
> #include "petsc/finclude/petscdmlabel.h"
> #include "petsc/finclude/petscdm.h"
>

 The source for these functions is in

   src/dm/ftn-auto/dmf.c

 Is it there for you? If not, you can run

   make allfortranstubs

 Fortran functions are not declared, so the header should not matter for
 compilation, just the libraries for linking.

   Thanks,

  Matt


> Thanks,
> Mike
>


 --
 What most experimenters take for granted before they begin their
 experiments is infinitely more interesting than any results to which their
 experiments lead.
 -- Norbert Wiener

 https://www.cse.buffalo.edu/~knepley/
 

>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> 
>>
>

Re: [petsc-users] DMLocalToLocal with DMPlex in Fortran

[Matthew Knepley]

On Sat, Oct 1, 2022 at 8:51 PM Mike Michell  wrote:

> Thank you for the reply. There is that file in src/dm/interface/ftn-auto/
> for me, instead of the path you mentioned.
>
> After "make allfortranstubs" was done and, PETSc reconfigured and
> reinstalled.
>
> However, I still have the same problem at the line in which
> DMLocalToLocalBegin() is used. What I am doing to setup halo exchange is as
> follows;
> - declare DMPlex
> - PetscSectionCreate()
> - PetscSectionSetNumFields()
> - PetscSectionSetFieldComponents()
> - PetscSectionSetChart()
> - do loop over dofs: PetscSectionSetDof() and PetscSectionSetFieldDof()
> - PetscSectionSetUp()
> - DMSetLocalSection()
> - PetscSectionDestroy()
> - DMGetSectionSF()
> - PetscSFSetUp()
>
> Then, the halo exchange is performed as follows;
> - DMGetLocalVector()
> - Fill the local vector
> - DMLocalToLocalBegin() --(*)
> - DMLocalToLocalEnd()
> - DMRestoreLocalVector()
>
> Then, the code crashes at (*).
>

Can you send something I can run? Then I will find the problem and fix it.

  Thanks,

 Matt


> Previously(at the time PETSc did not support LocalToLocal for DMPlex in
> Fortran), the part above, "DMLocalToLocalBegin() and DMLocalToLocalEnd()",
> consisted of;
> - DMLocalToGlobalBegin()
> - DMLocalToGlobalEnd()
> - DMGlobalToLocalBegin()
> - DMGlobalToLocalEnd()
> and it worked okay.
>
> I am unclear which part is causing the problem. Shall I define the
> PetscSection and PetscSF in different ways in case of Local to Local Halo
> exchange?
> Any comment will be appreciated.
>
> Thanks,
> Mike
>
>
>
>> On Fri, Sep 30, 2022 at 4:14 PM Mike Michell 
>> wrote:
>>
>>> Hi,
>>>
>>> As a follow-up to this email thread,
>>> https://www.mail-archive.com/petsc-users@mcs.anl.gov/msg44070.html
>>>
>>> Are DMLocalToLocalBegin() and DMLocalToLocalEnd() really available for
>>> DMPlex with Fortran on the latest version of PETSc (3.17.99 from GitLab)?
>>> Matt commented that the Fortran bindings were updated so that those
>>> functions must be available in the latest version of PETSc, however, it
>>> seems still they are not working from my test with DMPlex in Fortran. Can
>>> anyone provide some comments? Probably I am missing some mandatory header
>>> file? Currently, I have headers;
>>>
>>> #include "petsc/finclude/petscvec.h"
>>> #include "petsc/finclude/petscdmplex.h"
>>> #include "petsc/finclude/petscdmlabel.h"
>>> #include "petsc/finclude/petscdm.h"
>>>
>>
>> The source for these functions is in
>>
>>   src/dm/ftn-auto/dmf.c
>>
>> Is it there for you? If not, you can run
>>
>>   make allfortranstubs
>>
>> Fortran functions are not declared, so the header should not matter for
>> compilation, just the libraries for linking.
>>
>>   Thanks,
>>
>>  Matt
>>
>>
>>> Thanks,
>>> Mike
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> 
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] DMLocalToLocal with DMPlex in Fortran

[Mike Michell]

Thank you for the reply. There is that file in src/dm/interface/ftn-auto/
for me, instead of the path you mentioned.

After "make allfortranstubs" was done and, PETSc reconfigured and
reinstalled.

However, I still have the same problem at the line in which
DMLocalToLocalBegin() is used. What I am doing to setup halo exchange is as
follows;
- declare DMPlex
- PetscSectionCreate()
- PetscSectionSetNumFields()
- PetscSectionSetFieldComponents()
- PetscSectionSetChart()
- do loop over dofs: PetscSectionSetDof() and PetscSectionSetFieldDof()
- PetscSectionSetUp()
- DMSetLocalSection()
- PetscSectionDestroy()
- DMGetSectionSF()
- PetscSFSetUp()

Then, the halo exchange is performed as follows;
- DMGetLocalVector()
- Fill the local vector
- DMLocalToLocalBegin() --(*)
- DMLocalToLocalEnd()
- DMRestoreLocalVector()

Then, the code crashes at (*).

Previously(at the time PETSc did not support LocalToLocal for DMPlex in
Fortran), the part above, "DMLocalToLocalBegin() and DMLocalToLocalEnd()",
consisted of;
- DMLocalToGlobalBegin()
- DMLocalToGlobalEnd()
- DMGlobalToLocalBegin()
- DMGlobalToLocalEnd()
and it worked okay.

I am unclear which part is causing the problem. Shall I define the
PetscSection and PetscSF in different ways in case of Local to Local Halo
exchange?
Any comment will be appreciated.

Thanks,
Mike



> On Fri, Sep 30, 2022 at 4:14 PM Mike Michell 
> wrote:
>
>> Hi,
>>
>> As a follow-up to this email thread,
>> https://www.mail-archive.com/petsc-users@mcs.anl.gov/msg44070.html
>>
>> Are DMLocalToLocalBegin() and DMLocalToLocalEnd() really available for
>> DMPlex with Fortran on the latest version of PETSc (3.17.99 from GitLab)?
>> Matt commented that the Fortran bindings were updated so that those
>> functions must be available in the latest version of PETSc, however, it
>> seems still they are not working from my test with DMPlex in Fortran. Can
>> anyone provide some comments? Probably I am missing some mandatory header
>> file? Currently, I have headers;
>>
>> #include "petsc/finclude/petscvec.h"
>> #include "petsc/finclude/petscdmplex.h"
>> #include "petsc/finclude/petscdmlabel.h"
>> #include "petsc/finclude/petscdm.h"
>>
>
> The source for these functions is in
>
>   src/dm/ftn-auto/dmf.c
>
> Is it there for you? If not, you can run
>
>   make allfortranstubs
>
> Fortran functions are not declared, so the header should not matter for
> compilation, just the libraries for linking.
>
>   Thanks,
>
>  Matt
>
>
>> Thanks,
>> Mike
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> 
>

Re: [petsc-users] DMLocalToLocal with DMPlex in Fortran

[Matthew Knepley]

On Fri, Sep 30, 2022 at 4:14 PM Mike Michell  wrote:

> Hi,
>
> As a follow-up to this email thread,
> https://www.mail-archive.com/petsc-users@mcs.anl.gov/msg44070.html
>
> Are DMLocalToLocalBegin() and DMLocalToLocalEnd() really available for
> DMPlex with Fortran on the latest version of PETSc (3.17.99 from GitLab)?
> Matt commented that the Fortran bindings were updated so that those
> functions must be available in the latest version of PETSc, however, it
> seems still they are not working from my test with DMPlex in Fortran. Can
> anyone provide some comments? Probably I am missing some mandatory header
> file? Currently, I have headers;
>
> #include "petsc/finclude/petscvec.h"
> #include "petsc/finclude/petscdmplex.h"
> #include "petsc/finclude/petscdmlabel.h"
> #include "petsc/finclude/petscdm.h"
>

The source for these functions is in

  src/dm/ftn-auto/dmf.c

Is it there for you? If not, you can run

  make allfortranstubs

Fortran functions are not declared, so the header should not matter for
compilation, just the libraries for linking.

  Thanks,

 Matt


> Thanks,
> Mike
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

[petsc-users] DMLocalToLocal with DMPlex in Fortran

[Mike Michell]

Hi,

As a follow-up to this email thread,
https://www.mail-archive.com/petsc-users@mcs.anl.gov/msg44070.html

Are DMLocalToLocalBegin() and DMLocalToLocalEnd() really available for
DMPlex with Fortran on the latest version of PETSc (3.17.99 from GitLab)?
Matt commented that the Fortran bindings were updated so that those
functions must be available in the latest version of PETSc, however, it
seems still they are not working from my test with DMPlex in Fortran. Can
anyone provide some comments? Probably I am missing some mandatory header
file? Currently, I have headers;

#include "petsc/finclude/petscvec.h"
#include "petsc/finclude/petscdmplex.h"
#include "petsc/finclude/petscdmlabel.h"
#include "petsc/finclude/petscdm.h"

Thanks,
Mike

[petsc-users] DMLocalToLocal for DMPlex

[Adrian Croucher via petsc-users]

hi

If I have a local vector and I want to update all its ghost values from 
its non-ghost values, should I use DMLocalToLocalBegin()/End() ?


I have tried it and it gives me an error: "This DM does not support 
local to local maps".


The DM is a DMPlex. Is the local-to-local operation not implemented for 
DMPlex?


Or should I be using something else to do this?

- Adrian

--
Dr Adrian Croucher
Senior Research Fellow
Department of Engineering Science
University of Auckland, New Zealand
email: a.crouc...@auckland.ac.nz
tel: +64 (0)9 923 4611