Re: [petsc-users] Vec Ownership ranges with Global Section Offsets

[Matthew Knepley]

On Fri, Jan 6, 2023 at 12:18 PM Nicholas Arnold-Medabalimi <
narno...@umich.edu> wrote:

> I am doing that as well (although not for vertex dofs). And I had it
> working quite well for purely cell-associated DOFs. But I realized later
> that I also wanted to transmit some DOFs associated with faces so I suspect
> I'm messing something up there.
>
> Something we discussed back on 12/26 (email subject:Getting a vector from
> a DM to output VTK)  was associating a vector generated this way with the
> DM so it can be more easily visualized using vtk files is
>
> VecSetOperation(state_dist, VECOP_VIEW, (void(*)(void))VecView_Plex);
>
> If there is an easy way to get this working in Fortran I would very much
> appreciate it as it was very helpful when I was debugging in C.
>

This is a workaround for Fortran (the problem is that Fortran cannot easily
refer to the function pointers). You can

  1) Set the new section into the distributed DM

  2) Make a new global or local vector

  3) Call VecCopy() to move the data

The new vector will have the right viewer.

  Thanks,

 Matt


> Thanks
> Nicholas
>
> On Fri, Jan 6, 2023 at 11:37 AM Matthew Knepley  wrote:
>
>> On Fri, Jan 6, 2023 at 11:32 AM Nicholas Arnold-Medabalimi <
>> narno...@umich.edu> wrote:
>>
>>> Hi Matt
>>>
>>> This was generated using the DMPlexDistributeField which we discussed a
>>> while back. Everything seemed to be working fine when I only had cells dofs
>>> but I recently added face dofs, which seems to have caused some issues.
>>> Whats weird is that I'm feeding the same distribution SF and inputting a
>>> vector and section that are consistent to the DMPlexDistributeField so I'd
>>> expect the vectors and section output to be consistent. I'll take a closer
>>> look at that.
>>>
>>
>> The way I use DistributeField(), for example to distribute coordinates,
>> is that I give it the migrationSF, the local coordinate section, and the
>> local coordinate vector. It gives me back the new local coordinate section
>> (which I set into the distributed DM), and the local coordinate vector. It
>> seems like you are doing something else. Maybe the documentation is
>> misleading.
>>
>>   Thanks,
>>
>> Matt
>>
>>
>>> Thanks
>>> Nicholas
>>>
>>> On Fri, Jan 6, 2023 at 10:59 AM Matthew Knepley 
>>> wrote:
>>>
 On Fri, Jan 6, 2023 at 10:41 AM Nicholas Arnold-Medabalimi <
 narno...@umich.edu> wrote:

> Hi Matt
>
> I appreciate the help. The section view is quite extensive because
> each cell has 55 dofs located at the cells and on certain faces. I've
> appended the first of these which corresponds with the output in the first
> email, to save space. The following 54 are exactly the same but offset
> incremented by 1. (or negative 1 for negative offsets)
>

 Okay, from the output it is clear that this vector does not match your
 global section. Did you get stateVec by calling DMCreateGlobalVector()?

   Thanks,

  Matt


> Thanks for your time
> Nicholas
>
> On Fri, Jan 6, 2023 at 10:23 AM Matthew Knepley 
> wrote:
>
>> On Fri, Jan 6, 2023 at 10:10 AM Nicholas Arnold-Medabalimi <
>> narno...@umich.edu> wrote:
>>
>>> Hi Matt
>>>
>>> I apologize for any lack of clarity in the initial email.
>>>
>>> looking at the initial output on rank 1
>>> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart
>>> cell   0 offset2475 oStart2640-165
>>>  cell   1 offset2530 oStart2640-110
>>>  cell   2 offset2585 oStart2640 -55
>>>  cell   3 offset2640 oStart2640   0
>>> .
>>> cell  15 offset-771 oStart2640   -3411
>>>
>>>
>>> cell 15 provides a negative offset because it is the overlap cell
>>> (that is unowned)
>>> The remained of cells are all owned. However, the first 3 cells
>>> (0,1,2) return an offset that is less than the starting ownership 
>>> range. I
>>> would expect cell 0 to start at offset 2640 at minimum.
>>>
>>
>> Send the output for this section
>>
>>   call PetscSectionView(section, PETSC_VIEWER_STDOUT_WORLD);
>>
>>   Thanks,
>>
>>  Matt
>>
>>
>>> Sincerely
>>> Nicholas
>>>
>>>
>>>
>>>
>>> On Fri, Jan 6, 2023 at 10:05 AM Matthew Knepley 
>>> wrote:
>>>
 On Fri, Jan 6, 2023 at 9:56 AM Nicholas Arnold-Medabalimi <
 narno...@umich.edu> wrote:

> Apologies. If it helps, there is one cell of overlap in this small
> test case for a 2D mesh that is 1 cell in height and a number of 
> cells in
> length. .
>
> process 0
>  Petsc VecGetLocalSize2750
>  size(stateVecV)2750
>
> process 1
> 

Re: [petsc-users] Vec Ownership ranges with Global Section Offsets

[Nicholas Arnold-Medabalimi]

I am doing that as well (although not for vertex dofs). And I had it
working quite well for purely cell-associated DOFs. But I realized later
that I also wanted to transmit some DOFs associated with faces so I suspect
I'm messing something up there.

Something we discussed back on 12/26 (email subject:Getting a vector from a
DM to output VTK)  was associating a vector generated this way with the DM
so it can be more easily visualized using vtk files is

VecSetOperation(state_dist, VECOP_VIEW, (void(*)(void))VecView_Plex);

If there is an easy way to get this working in Fortran I would very much
appreciate it as it was very helpful when I was debugging in C.

Thanks
Nicholas

On Fri, Jan 6, 2023 at 11:37 AM Matthew Knepley  wrote:

> On Fri, Jan 6, 2023 at 11:32 AM Nicholas Arnold-Medabalimi <
> narno...@umich.edu> wrote:
>
>> Hi Matt
>>
>> This was generated using the DMPlexDistributeField which we discussed a
>> while back. Everything seemed to be working fine when I only had cells dofs
>> but I recently added face dofs, which seems to have caused some issues.
>> Whats weird is that I'm feeding the same distribution SF and inputting a
>> vector and section that are consistent to the DMPlexDistributeField so I'd
>> expect the vectors and section output to be consistent. I'll take a closer
>> look at that.
>>
>
> The way I use DistributeField(), for example to distribute coordinates, is
> that I give it the migrationSF, the local coordinate section, and the local
> coordinate vector. It gives me back the new local coordinate section (which
> I set into the distributed DM), and the local coordinate vector. It seems
> like you are doing something else. Maybe the documentation is misleading.
>
>   Thanks,
>
> Matt
>
>
>> Thanks
>> Nicholas
>>
>> On Fri, Jan 6, 2023 at 10:59 AM Matthew Knepley 
>> wrote:
>>
>>> On Fri, Jan 6, 2023 at 10:41 AM Nicholas Arnold-Medabalimi <
>>> narno...@umich.edu> wrote:
>>>
 Hi Matt

 I appreciate the help. The section view is quite extensive because each
 cell has 55 dofs located at the cells and on certain faces. I've appended
 the first of these which corresponds with the output in the first email, to
 save space. The following 54 are exactly the same but offset incremented by
 1. (or negative 1 for negative offsets)

>>>
>>> Okay, from the output it is clear that this vector does not match your
>>> global section. Did you get stateVec by calling DMCreateGlobalVector()?
>>>
>>>   Thanks,
>>>
>>>  Matt
>>>
>>>
 Thanks for your time
 Nicholas

 On Fri, Jan 6, 2023 at 10:23 AM Matthew Knepley 
 wrote:

> On Fri, Jan 6, 2023 at 10:10 AM Nicholas Arnold-Medabalimi <
> narno...@umich.edu> wrote:
>
>> Hi Matt
>>
>> I apologize for any lack of clarity in the initial email.
>>
>> looking at the initial output on rank 1
>> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart
>> cell   0 offset2475 oStart2640-165
>>  cell   1 offset2530 oStart2640-110
>>  cell   2 offset2585 oStart2640 -55
>>  cell   3 offset2640 oStart2640   0
>> .
>> cell  15 offset-771 oStart2640   -3411
>>
>>
>> cell 15 provides a negative offset because it is the overlap cell
>> (that is unowned)
>> The remained of cells are all owned. However, the first 3 cells
>> (0,1,2) return an offset that is less than the starting ownership range. 
>> I
>> would expect cell 0 to start at offset 2640 at minimum.
>>
>
> Send the output for this section
>
>   call PetscSectionView(section, PETSC_VIEWER_STDOUT_WORLD);
>
>   Thanks,
>
>  Matt
>
>
>> Sincerely
>> Nicholas
>>
>>
>>
>>
>> On Fri, Jan 6, 2023 at 10:05 AM Matthew Knepley 
>> wrote:
>>
>>> On Fri, Jan 6, 2023 at 9:56 AM Nicholas Arnold-Medabalimi <
>>> narno...@umich.edu> wrote:
>>>
 Apologies. If it helps, there is one cell of overlap in this small
 test case for a 2D mesh that is 1 cell in height and a number of cells 
 in
 length. .

 process 0
  Petsc VecGetLocalSize2750
  size(stateVecV)2750

 process 1
  Petsc VecGetLocalSize2640
  size(stateVecV)2640

>>>
>>> The offsets shown below are well-within these sizes. I do not
>>> understand the problem.
>>>
>>>   Thanks,
>>>
>>>  Matt
>>>
>>>
 On Fri, Jan 6, 2023 at 9:51 AM Matthew Knepley 
 wrote:

> On Fri, Jan 6, 2023 at 9:37 AM Nicholas Arnold-Medabalimi <
> narno...@umich.edu> wrote:
>
>> Hi Matt
>>
>> I made a typo on the line statVecV(offset) = 

Re: [petsc-users] Vec Ownership ranges with Global Section Offsets

[Matthew Knepley]

On Fri, Jan 6, 2023 at 11:32 AM Nicholas Arnold-Medabalimi <
narno...@umich.edu> wrote:

> Hi Matt
>
> This was generated using the DMPlexDistributeField which we discussed a
> while back. Everything seemed to be working fine when I only had cells dofs
> but I recently added face dofs, which seems to have caused some issues.
> Whats weird is that I'm feeding the same distribution SF and inputting a
> vector and section that are consistent to the DMPlexDistributeField so I'd
> expect the vectors and section output to be consistent. I'll take a closer
> look at that.
>

The way I use DistributeField(), for example to distribute coordinates, is
that I give it the migrationSF, the local coordinate section, and the local
coordinate vector. It gives me back the new local coordinate section (which
I set into the distributed DM), and the local coordinate vector. It seems
like you are doing something else. Maybe the documentation is misleading.

  Thanks,

Matt


> Thanks
> Nicholas
>
> On Fri, Jan 6, 2023 at 10:59 AM Matthew Knepley  wrote:
>
>> On Fri, Jan 6, 2023 at 10:41 AM Nicholas Arnold-Medabalimi <
>> narno...@umich.edu> wrote:
>>
>>> Hi Matt
>>>
>>> I appreciate the help. The section view is quite extensive because each
>>> cell has 55 dofs located at the cells and on certain faces. I've appended
>>> the first of these which corresponds with the output in the first email, to
>>> save space. The following 54 are exactly the same but offset incremented by
>>> 1. (or negative 1 for negative offsets)
>>>
>>
>> Okay, from the output it is clear that this vector does not match your
>> global section. Did you get stateVec by calling DMCreateGlobalVector()?
>>
>>   Thanks,
>>
>>  Matt
>>
>>
>>> Thanks for your time
>>> Nicholas
>>>
>>> On Fri, Jan 6, 2023 at 10:23 AM Matthew Knepley 
>>> wrote:
>>>
 On Fri, Jan 6, 2023 at 10:10 AM Nicholas Arnold-Medabalimi <
 narno...@umich.edu> wrote:

> Hi Matt
>
> I apologize for any lack of clarity in the initial email.
>
> looking at the initial output on rank 1
> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart
> cell   0 offset2475 oStart2640-165
>  cell   1 offset2530 oStart2640-110
>  cell   2 offset2585 oStart2640 -55
>  cell   3 offset2640 oStart2640   0
> .
> cell  15 offset-771 oStart2640   -3411
>
>
> cell 15 provides a negative offset because it is the overlap cell
> (that is unowned)
> The remained of cells are all owned. However, the first 3 cells
> (0,1,2) return an offset that is less than the starting ownership range. I
> would expect cell 0 to start at offset 2640 at minimum.
>

 Send the output for this section

   call PetscSectionView(section, PETSC_VIEWER_STDOUT_WORLD);

   Thanks,

  Matt


> Sincerely
> Nicholas
>
>
>
>
> On Fri, Jan 6, 2023 at 10:05 AM Matthew Knepley 
> wrote:
>
>> On Fri, Jan 6, 2023 at 9:56 AM Nicholas Arnold-Medabalimi <
>> narno...@umich.edu> wrote:
>>
>>> Apologies. If it helps, there is one cell of overlap in this small
>>> test case for a 2D mesh that is 1 cell in height and a number of cells 
>>> in
>>> length. .
>>>
>>> process 0
>>>  Petsc VecGetLocalSize2750
>>>  size(stateVecV)2750
>>>
>>> process 1
>>>  Petsc VecGetLocalSize2640
>>>  size(stateVecV)2640
>>>
>>
>> The offsets shown below are well-within these sizes. I do not
>> understand the problem.
>>
>>   Thanks,
>>
>>  Matt
>>
>>
>>> On Fri, Jan 6, 2023 at 9:51 AM Matthew Knepley 
>>> wrote:
>>>
 On Fri, Jan 6, 2023 at 9:37 AM Nicholas Arnold-Medabalimi <
 narno...@umich.edu> wrote:

> Hi Matt
>
> I made a typo on the line statVecV(offset) =  in
> my example, I agree. (I wrote that offhand since the actual 
> assignment is
> much larger) I should be statVecV(offset+1) =  so I'm 
> confident
> it's not a 1 0 indexing thing.
>
> My question is more related to what is happening in the offsets.
> c0 and c1 are pulled using DMplexgetheight stratum, so they are
> zero-indexed (which is why I loop from c0 to (c1-1)).
>
> For the size inquiries. on processor 0
>  Petsc VecGetSize(stateVec)5390
>

 I need to see VecGetLocalSize()

   Matt


>  size(stateVecV)   2640
>
> on processor 1
> Petsc VecGetSize 5390
> size(stateVecV)2750
>
> It's quite weird to me that processor one can have a positive
> 

Re: [petsc-users] Vec Ownership ranges with Global Section Offsets

[Nicholas Arnold-Medabalimi]

Hi Matt

This was generated using the DMPlexDistributeField which we discussed a
while back. Everything seemed to be working fine when I only had cells dofs
but I recently added face dofs, which seems to have caused some issues.
Whats weird is that I'm feeding the same distribution SF and inputting a
vector and section that are consistent to the DMPlexDistributeField so I'd
expect the vectors and section output to be consistent. I'll take a closer
look at that.

Thanks
Nicholas

On Fri, Jan 6, 2023 at 10:59 AM Matthew Knepley  wrote:

> On Fri, Jan 6, 2023 at 10:41 AM Nicholas Arnold-Medabalimi <
> narno...@umich.edu> wrote:
>
>> Hi Matt
>>
>> I appreciate the help. The section view is quite extensive because each
>> cell has 55 dofs located at the cells and on certain faces. I've appended
>> the first of these which corresponds with the output in the first email, to
>> save space. The following 54 are exactly the same but offset incremented by
>> 1. (or negative 1 for negative offsets)
>>
>
> Okay, from the output it is clear that this vector does not match your
> global section. Did you get stateVec by calling DMCreateGlobalVector()?
>
>   Thanks,
>
>  Matt
>
>
>> Thanks for your time
>> Nicholas
>>
>> On Fri, Jan 6, 2023 at 10:23 AM Matthew Knepley 
>> wrote:
>>
>>> On Fri, Jan 6, 2023 at 10:10 AM Nicholas Arnold-Medabalimi <
>>> narno...@umich.edu> wrote:
>>>
 Hi Matt

 I apologize for any lack of clarity in the initial email.

 looking at the initial output on rank 1
 write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart
 cell   0 offset2475 oStart2640-165
  cell   1 offset2530 oStart2640-110
  cell   2 offset2585 oStart2640 -55
  cell   3 offset2640 oStart2640   0
 .
 cell  15 offset-771 oStart2640   -3411


 cell 15 provides a negative offset because it is the overlap cell (that
 is unowned)
 The remained of cells are all owned. However, the first 3 cells (0,1,2)
 return an offset that is less than the starting ownership range. I would
 expect cell 0 to start at offset 2640 at minimum.

>>>
>>> Send the output for this section
>>>
>>>   call PetscSectionView(section, PETSC_VIEWER_STDOUT_WORLD);
>>>
>>>   Thanks,
>>>
>>>  Matt
>>>
>>>
 Sincerely
 Nicholas




 On Fri, Jan 6, 2023 at 10:05 AM Matthew Knepley 
 wrote:

> On Fri, Jan 6, 2023 at 9:56 AM Nicholas Arnold-Medabalimi <
> narno...@umich.edu> wrote:
>
>> Apologies. If it helps, there is one cell of overlap in this small
>> test case for a 2D mesh that is 1 cell in height and a number of cells in
>> length. .
>>
>> process 0
>>  Petsc VecGetLocalSize2750
>>  size(stateVecV)2750
>>
>> process 1
>>  Petsc VecGetLocalSize2640
>>  size(stateVecV)2640
>>
>
> The offsets shown below are well-within these sizes. I do not
> understand the problem.
>
>   Thanks,
>
>  Matt
>
>
>> On Fri, Jan 6, 2023 at 9:51 AM Matthew Knepley 
>> wrote:
>>
>>> On Fri, Jan 6, 2023 at 9:37 AM Nicholas Arnold-Medabalimi <
>>> narno...@umich.edu> wrote:
>>>
 Hi Matt

 I made a typo on the line statVecV(offset) =  in
 my example, I agree. (I wrote that offhand since the actual assignment 
 is
 much larger) I should be statVecV(offset+1) =  so I'm 
 confident
 it's not a 1 0 indexing thing.

 My question is more related to what is happening in the offsets. c0
 and c1 are pulled using DMplexgetheight stratum, so they are 
 zero-indexed
 (which is why I loop from c0 to (c1-1)).

 For the size inquiries. on processor 0
  Petsc VecGetSize(stateVec)5390

>>>
>>> I need to see VecGetLocalSize()
>>>
>>>   Matt
>>>
>>>
  size(stateVecV)   2640

 on processor 1
 Petsc VecGetSize 5390
 size(stateVecV)2750

 It's quite weird to me that processor one can have a positive
 offset that is less than its starting ownership index (in the initial 
 email
 output).

 Thanks for the assistance
 Nicholas


 On Fri, Jan 6, 2023 at 9:20 AM Matthew Knepley 
 wrote:

> On Fri, Jan 6, 2023 at 2:28 AM Nicholas Arnold-Medabalimi <
> narno...@umich.edu> wrote:
>
>> Hi Petsc Users,
>>
>> I'm working with a dmplex system with a subsampled mesh
>> distributed with an overlap of 1.
>>
>> I'm encountering unusual situations when using
>> VecGetOwnershipRange to adjust 

Re: [petsc-users] Vec Ownership ranges with Global Section Offsets

[Matthew Knepley]

On Fri, Jan 6, 2023 at 10:41 AM Nicholas Arnold-Medabalimi <
narno...@umich.edu> wrote:

> Hi Matt
>
> I appreciate the help. The section view is quite extensive because each
> cell has 55 dofs located at the cells and on certain faces. I've appended
> the first of these which corresponds with the output in the first email, to
> save space. The following 54 are exactly the same but offset incremented by
> 1. (or negative 1 for negative offsets)
>

Okay, from the output it is clear that this vector does not match your
global section. Did you get stateVec by calling DMCreateGlobalVector()?

  Thanks,

 Matt


> Thanks for your time
> Nicholas
>
> On Fri, Jan 6, 2023 at 10:23 AM Matthew Knepley  wrote:
>
>> On Fri, Jan 6, 2023 at 10:10 AM Nicholas Arnold-Medabalimi <
>> narno...@umich.edu> wrote:
>>
>>> Hi Matt
>>>
>>> I apologize for any lack of clarity in the initial email.
>>>
>>> looking at the initial output on rank 1
>>> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart
>>> cell   0 offset2475 oStart2640-165
>>>  cell   1 offset2530 oStart2640-110
>>>  cell   2 offset2585 oStart2640 -55
>>>  cell   3 offset2640 oStart2640   0
>>> .
>>> cell  15 offset-771 oStart2640   -3411
>>>
>>>
>>> cell 15 provides a negative offset because it is the overlap cell (that
>>> is unowned)
>>> The remained of cells are all owned. However, the first 3 cells (0,1,2)
>>> return an offset that is less than the starting ownership range. I would
>>> expect cell 0 to start at offset 2640 at minimum.
>>>
>>
>> Send the output for this section
>>
>>   call PetscSectionView(section, PETSC_VIEWER_STDOUT_WORLD);
>>
>>   Thanks,
>>
>>  Matt
>>
>>
>>> Sincerely
>>> Nicholas
>>>
>>>
>>>
>>>
>>> On Fri, Jan 6, 2023 at 10:05 AM Matthew Knepley 
>>> wrote:
>>>
 On Fri, Jan 6, 2023 at 9:56 AM Nicholas Arnold-Medabalimi <
 narno...@umich.edu> wrote:

> Apologies. If it helps, there is one cell of overlap in this small
> test case for a 2D mesh that is 1 cell in height and a number of cells in
> length. .
>
> process 0
>  Petsc VecGetLocalSize2750
>  size(stateVecV)2750
>
> process 1
>  Petsc VecGetLocalSize2640
>  size(stateVecV)2640
>

 The offsets shown below are well-within these sizes. I do not
 understand the problem.

   Thanks,

  Matt


> On Fri, Jan 6, 2023 at 9:51 AM Matthew Knepley 
> wrote:
>
>> On Fri, Jan 6, 2023 at 9:37 AM Nicholas Arnold-Medabalimi <
>> narno...@umich.edu> wrote:
>>
>>> Hi Matt
>>>
>>> I made a typo on the line statVecV(offset) =  in
>>> my example, I agree. (I wrote that offhand since the actual assignment 
>>> is
>>> much larger) I should be statVecV(offset+1) =  so I'm 
>>> confident
>>> it's not a 1 0 indexing thing.
>>>
>>> My question is more related to what is happening in the offsets. c0
>>> and c1 are pulled using DMplexgetheight stratum, so they are 
>>> zero-indexed
>>> (which is why I loop from c0 to (c1-1)).
>>>
>>> For the size inquiries. on processor 0
>>>  Petsc VecGetSize(stateVec)5390
>>>
>>
>> I need to see VecGetLocalSize()
>>
>>   Matt
>>
>>
>>>  size(stateVecV)   2640
>>>
>>> on processor 1
>>> Petsc VecGetSize 5390
>>> size(stateVecV)2750
>>>
>>> It's quite weird to me that processor one can have a positive offset
>>> that is less than its starting ownership index (in the initial email
>>> output).
>>>
>>> Thanks for the assistance
>>> Nicholas
>>>
>>>
>>> On Fri, Jan 6, 2023 at 9:20 AM Matthew Knepley 
>>> wrote:
>>>
 On Fri, Jan 6, 2023 at 2:28 AM Nicholas Arnold-Medabalimi <
 narno...@umich.edu> wrote:

> Hi Petsc Users,
>
> I'm working with a dmplex system with a subsampled mesh
> distributed with an overlap of 1.
>
> I'm encountering unusual situations when using
> VecGetOwnershipRange to adjust the offset received from a global 
> section.
> The logic of the following code is first to get the offset needed to 
> index
> a global vector while still being able to check if it is an 
> overlapped cell
> and skip if needed while counting the owned cells.
>


>
> call DMGetGlobalSection(dmplex,section,ierr)
> call VecGetArrayF90(stateVec,stateVecV,ierr)
> call VecGetOwnershipRange(stateVec,oStart,oEnd,ierr)
> do i = c0, (c1-1)
>
> call PetscSectionGetOffset(section,i,offset,ierr)
> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-
> 

Re: [petsc-users] Vec Ownership ranges with Global Section Offsets

[Nicholas Arnold-Medabalimi]

Hi Matt

I appreciate the help. The section view is quite extensive because each
cell has 55 dofs located at the cells and on certain faces. I've appended
the first of these which corresponds with the output in the first email, to
save space. The following 54 are exactly the same but offset incremented by
1. (or negative 1 for negative offsets)

Thanks for your time
Nicholas

On Fri, Jan 6, 2023 at 10:23 AM Matthew Knepley  wrote:

> On Fri, Jan 6, 2023 at 10:10 AM Nicholas Arnold-Medabalimi <
> narno...@umich.edu> wrote:
>
>> Hi Matt
>>
>> I apologize for any lack of clarity in the initial email.
>>
>> looking at the initial output on rank 1
>> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart
>> cell   0 offset2475 oStart2640-165
>>  cell   1 offset2530 oStart2640-110
>>  cell   2 offset2585 oStart2640 -55
>>  cell   3 offset2640 oStart2640   0
>> .
>> cell  15 offset-771 oStart2640   -3411
>>
>>
>> cell 15 provides a negative offset because it is the overlap cell (that
>> is unowned)
>> The remained of cells are all owned. However, the first 3 cells (0,1,2)
>> return an offset that is less than the starting ownership range. I would
>> expect cell 0 to start at offset 2640 at minimum.
>>
>
> Send the output for this section
>
>   call PetscSectionView(section, PETSC_VIEWER_STDOUT_WORLD);
>
>   Thanks,
>
>  Matt
>
>
>> Sincerely
>> Nicholas
>>
>>
>>
>>
>> On Fri, Jan 6, 2023 at 10:05 AM Matthew Knepley 
>> wrote:
>>
>>> On Fri, Jan 6, 2023 at 9:56 AM Nicholas Arnold-Medabalimi <
>>> narno...@umich.edu> wrote:
>>>
 Apologies. If it helps, there is one cell of overlap in this small test
 case for a 2D mesh that is 1 cell in height and a number of cells in
 length. .

 process 0
  Petsc VecGetLocalSize2750
  size(stateVecV)2750

 process 1
  Petsc VecGetLocalSize2640
  size(stateVecV)2640

>>>
>>> The offsets shown below are well-within these sizes. I do not understand
>>> the problem.
>>>
>>>   Thanks,
>>>
>>>  Matt
>>>
>>>
 On Fri, Jan 6, 2023 at 9:51 AM Matthew Knepley 
 wrote:

> On Fri, Jan 6, 2023 at 9:37 AM Nicholas Arnold-Medabalimi <
> narno...@umich.edu> wrote:
>
>> Hi Matt
>>
>> I made a typo on the line statVecV(offset) =  in my
>> example, I agree. (I wrote that offhand since the actual assignment is 
>> much
>> larger) I should be statVecV(offset+1) =  so I'm confident 
>> it's
>> not a 1 0 indexing thing.
>>
>> My question is more related to what is happening in the offsets. c0
>> and c1 are pulled using DMplexgetheight stratum, so they are zero-indexed
>> (which is why I loop from c0 to (c1-1)).
>>
>> For the size inquiries. on processor 0
>>  Petsc VecGetSize(stateVec)5390
>>
>
> I need to see VecGetLocalSize()
>
>   Matt
>
>
>>  size(stateVecV)   2640
>>
>> on processor 1
>> Petsc VecGetSize 5390
>> size(stateVecV)2750
>>
>> It's quite weird to me that processor one can have a positive offset
>> that is less than its starting ownership index (in the initial email
>> output).
>>
>> Thanks for the assistance
>> Nicholas
>>
>>
>> On Fri, Jan 6, 2023 at 9:20 AM Matthew Knepley 
>> wrote:
>>
>>> On Fri, Jan 6, 2023 at 2:28 AM Nicholas Arnold-Medabalimi <
>>> narno...@umich.edu> wrote:
>>>
 Hi Petsc Users,

 I'm working with a dmplex system with a subsampled mesh distributed
 with an overlap of 1.

 I'm encountering unusual situations when using VecGetOwnershipRange
 to adjust the offset received from a global section. The logic of the
 following code is first to get the offset needed to index a global 
 vector
 while still being able to check if it is an overlapped cell and skip if
 needed while counting the owned cells.

>>>
>>>

 call DMGetGlobalSection(dmplex,section,ierr)
 call VecGetArrayF90(stateVec,stateVecV,ierr)
 call VecGetOwnershipRange(stateVec,oStart,oEnd,ierr)
 do i = c0, (c1-1)

 call PetscSectionGetOffset(section,i,offset,ierr)
 write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-
 oStart

 if(offset<0) then
 cycle
 endif
 offset=offset-oStart
 plexcells=plexcells+1
 stateVecV(offset)=  enddo

 I'm noticing some very weird results that I've appended below. The
 GetOffset documentation notes that a negative offset indicates an 
 unowned
 point (which I use to cycle). However, the offset subtraction with 
 

Re: [petsc-users] Vec Ownership ranges with Global Section Offsets

[Matthew Knepley]

On Fri, Jan 6, 2023 at 10:10 AM Nicholas Arnold-Medabalimi <
narno...@umich.edu> wrote:

> Hi Matt
>
> I apologize for any lack of clarity in the initial email.
>
> looking at the initial output on rank 1
> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart
> cell   0 offset2475 oStart2640-165
>  cell   1 offset2530 oStart2640-110
>  cell   2 offset2585 oStart2640 -55
>  cell   3 offset2640 oStart2640   0
> .
> cell  15 offset-771 oStart2640   -3411
>
>
> cell 15 provides a negative offset because it is the overlap cell (that is
> unowned)
> The remained of cells are all owned. However, the first 3 cells (0,1,2)
> return an offset that is less than the starting ownership range. I would
> expect cell 0 to start at offset 2640 at minimum.
>

Send the output for this section

  call PetscSectionView(section, PETSC_VIEWER_STDOUT_WORLD);

  Thanks,

 Matt


> Sincerely
> Nicholas
>
>
>
>
> On Fri, Jan 6, 2023 at 10:05 AM Matthew Knepley  wrote:
>
>> On Fri, Jan 6, 2023 at 9:56 AM Nicholas Arnold-Medabalimi <
>> narno...@umich.edu> wrote:
>>
>>> Apologies. If it helps, there is one cell of overlap in this small test
>>> case for a 2D mesh that is 1 cell in height and a number of cells in
>>> length. .
>>>
>>> process 0
>>>  Petsc VecGetLocalSize2750
>>>  size(stateVecV)2750
>>>
>>> process 1
>>>  Petsc VecGetLocalSize2640
>>>  size(stateVecV)2640
>>>
>>
>> The offsets shown below are well-within these sizes. I do not understand
>> the problem.
>>
>>   Thanks,
>>
>>  Matt
>>
>>
>>> On Fri, Jan 6, 2023 at 9:51 AM Matthew Knepley 
>>> wrote:
>>>
 On Fri, Jan 6, 2023 at 9:37 AM Nicholas Arnold-Medabalimi <
 narno...@umich.edu> wrote:

> Hi Matt
>
> I made a typo on the line statVecV(offset) =  in my
> example, I agree. (I wrote that offhand since the actual assignment is 
> much
> larger) I should be statVecV(offset+1) =  so I'm confident 
> it's
> not a 1 0 indexing thing.
>
> My question is more related to what is happening in the offsets. c0
> and c1 are pulled using DMplexgetheight stratum, so they are zero-indexed
> (which is why I loop from c0 to (c1-1)).
>
> For the size inquiries. on processor 0
>  Petsc VecGetSize(stateVec)5390
>

 I need to see VecGetLocalSize()

   Matt


>  size(stateVecV)   2640
>
> on processor 1
> Petsc VecGetSize 5390
> size(stateVecV)2750
>
> It's quite weird to me that processor one can have a positive offset
> that is less than its starting ownership index (in the initial email
> output).
>
> Thanks for the assistance
> Nicholas
>
>
> On Fri, Jan 6, 2023 at 9:20 AM Matthew Knepley 
> wrote:
>
>> On Fri, Jan 6, 2023 at 2:28 AM Nicholas Arnold-Medabalimi <
>> narno...@umich.edu> wrote:
>>
>>> Hi Petsc Users,
>>>
>>> I'm working with a dmplex system with a subsampled mesh distributed
>>> with an overlap of 1.
>>>
>>> I'm encountering unusual situations when using VecGetOwnershipRange
>>> to adjust the offset received from a global section. The logic of the
>>> following code is first to get the offset needed to index a global 
>>> vector
>>> while still being able to check if it is an overlapped cell and skip if
>>> needed while counting the owned cells.
>>>
>>
>>
>>>
>>> call DMGetGlobalSection(dmplex,section,ierr)
>>> call VecGetArrayF90(stateVec,stateVecV,ierr)
>>> call VecGetOwnershipRange(stateVec,oStart,oEnd,ierr)
>>> do i = c0, (c1-1)
>>>
>>> call PetscSectionGetOffset(section,i,offset,ierr)
>>> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-
>>> oStart
>>>
>>> if(offset<0) then
>>> cycle
>>> endif
>>> offset=offset-oStart
>>> plexcells=plexcells+1
>>> stateVecV(offset)=  enddo
>>>
>>> I'm noticing some very weird results that I've appended below. The
>>> GetOffset documentation notes that a negative offset indicates an 
>>> unowned
>>> point (which I use to cycle). However, the offset subtraction with 
>>> oStart
>>> will yield an illegal index for the Vector access. I see that on the
>>> documentation for GetOwnershipRange, it notes that this may be
>>> "ill-defined"  but I wanted to see if this is type of ill-defined I can
>>> expect or there is just something terribly wrong with my 
>>> PetscSection.(both
>>> the Vec and Section were produced from DMPlexDistributeField so should 
>>> by
>>> definition have synchronized section information) I was wondering if 
>>> there
>>> is a possible output and/or the best way to index the vector. I'm 
>>> 

Re: [petsc-users] Vec Ownership ranges with Global Section Offsets

[Nicholas Arnold-Medabalimi]

Hi Matt

I apologize for any lack of clarity in the initial email.

looking at the initial output on rank 1
write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart
cell   0 offset2475 oStart2640-165
 cell   1 offset2530 oStart2640-110
 cell   2 offset2585 oStart2640 -55
 cell   3 offset2640 oStart2640   0
.
cell  15 offset-771 oStart2640   -3411


cell 15 provides a negative offset because it is the overlap cell (that is
unowned)
The remained of cells are all owned. However, the first 3 cells (0,1,2)
return an offset that is less than the starting ownership range. I would
expect cell 0 to start at offset 2640 at minimum.

Sincerely
Nicholas




On Fri, Jan 6, 2023 at 10:05 AM Matthew Knepley  wrote:

> On Fri, Jan 6, 2023 at 9:56 AM Nicholas Arnold-Medabalimi <
> narno...@umich.edu> wrote:
>
>> Apologies. If it helps, there is one cell of overlap in this small test
>> case for a 2D mesh that is 1 cell in height and a number of cells in
>> length. .
>>
>> process 0
>>  Petsc VecGetLocalSize2750
>>  size(stateVecV)2750
>>
>> process 1
>>  Petsc VecGetLocalSize2640
>>  size(stateVecV)2640
>>
>
> The offsets shown below are well-within these sizes. I do not understand
> the problem.
>
>   Thanks,
>
>  Matt
>
>
>> On Fri, Jan 6, 2023 at 9:51 AM Matthew Knepley  wrote:
>>
>>> On Fri, Jan 6, 2023 at 9:37 AM Nicholas Arnold-Medabalimi <
>>> narno...@umich.edu> wrote:
>>>
 Hi Matt

 I made a typo on the line statVecV(offset) =  in my
 example, I agree. (I wrote that offhand since the actual assignment is much
 larger) I should be statVecV(offset+1) =  so I'm confident it's
 not a 1 0 indexing thing.

 My question is more related to what is happening in the offsets. c0 and
 c1 are pulled using DMplexgetheight stratum, so they are zero-indexed
 (which is why I loop from c0 to (c1-1)).

 For the size inquiries. on processor 0
  Petsc VecGetSize(stateVec)5390

>>>
>>> I need to see VecGetLocalSize()
>>>
>>>   Matt
>>>
>>>
  size(stateVecV)   2640

 on processor 1
 Petsc VecGetSize 5390
 size(stateVecV)2750

 It's quite weird to me that processor one can have a positive offset
 that is less than its starting ownership index (in the initial email
 output).

 Thanks for the assistance
 Nicholas


 On Fri, Jan 6, 2023 at 9:20 AM Matthew Knepley 
 wrote:

> On Fri, Jan 6, 2023 at 2:28 AM Nicholas Arnold-Medabalimi <
> narno...@umich.edu> wrote:
>
>> Hi Petsc Users,
>>
>> I'm working with a dmplex system with a subsampled mesh distributed
>> with an overlap of 1.
>>
>> I'm encountering unusual situations when using VecGetOwnershipRange
>> to adjust the offset received from a global section. The logic of the
>> following code is first to get the offset needed to index a global vector
>> while still being able to check if it is an overlapped cell and skip if
>> needed while counting the owned cells.
>>
>
>
>>
>> call DMGetGlobalSection(dmplex,section,ierr)
>> call VecGetArrayF90(stateVec,stateVecV,ierr)
>> call VecGetOwnershipRange(stateVec,oStart,oEnd,ierr)
>> do i = c0, (c1-1)
>>
>> call PetscSectionGetOffset(section,i,offset,ierr)
>> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-
>> oStart
>>
>> if(offset<0) then
>> cycle
>> endif
>> offset=offset-oStart
>> plexcells=plexcells+1
>> stateVecV(offset)=  enddo
>>
>> I'm noticing some very weird results that I've appended below. The
>> GetOffset documentation notes that a negative offset indicates an unowned
>> point (which I use to cycle). However, the offset subtraction with oStart
>> will yield an illegal index for the Vector access. I see that on the
>> documentation for GetOwnershipRange, it notes that this may be
>> "ill-defined"  but I wanted to see if this is type of ill-defined I can
>> expect or there is just something terribly wrong with my 
>> PetscSection.(both
>> the Vec and Section were produced from DMPlexDistributeField so should by
>> definition have synchronized section information) I was wondering if 
>> there
>> is a possible output and/or the best way to index the vector. I'm 
>> thinking
>> of subtracting the offset of cell 0 perhaps?
>>
>
> Can you show your vector sizes? Are you sure it is not the fact that
> F90 arrays use 1-based indices, but these are 0-based offsets?
>
>   Thanks,
>
>  Matt
>
>
>> on rank 0
>>
>>  cell   0 offset   0 oStart   0   0
>>  cell   1 offset  55 

Re: [petsc-users] Vec Ownership ranges with Global Section Offsets

[Matthew Knepley]

On Fri, Jan 6, 2023 at 9:56 AM Nicholas Arnold-Medabalimi <
narno...@umich.edu> wrote:

> Apologies. If it helps, there is one cell of overlap in this small test
> case for a 2D mesh that is 1 cell in height and a number of cells in
> length. .
>
> process 0
>  Petsc VecGetLocalSize2750
>  size(stateVecV)2750
>
> process 1
>  Petsc VecGetLocalSize2640
>  size(stateVecV)2640
>

The offsets shown below are well-within these sizes. I do not understand
the problem.

  Thanks,

 Matt


> On Fri, Jan 6, 2023 at 9:51 AM Matthew Knepley  wrote:
>
>> On Fri, Jan 6, 2023 at 9:37 AM Nicholas Arnold-Medabalimi <
>> narno...@umich.edu> wrote:
>>
>>> Hi Matt
>>>
>>> I made a typo on the line statVecV(offset) =  in my
>>> example, I agree. (I wrote that offhand since the actual assignment is much
>>> larger) I should be statVecV(offset+1) =  so I'm confident it's
>>> not a 1 0 indexing thing.
>>>
>>> My question is more related to what is happening in the offsets. c0 and
>>> c1 are pulled using DMplexgetheight stratum, so they are zero-indexed
>>> (which is why I loop from c0 to (c1-1)).
>>>
>>> For the size inquiries. on processor 0
>>>  Petsc VecGetSize(stateVec)5390
>>>
>>
>> I need to see VecGetLocalSize()
>>
>>   Matt
>>
>>
>>>  size(stateVecV)   2640
>>>
>>> on processor 1
>>> Petsc VecGetSize 5390
>>> size(stateVecV)2750
>>>
>>> It's quite weird to me that processor one can have a positive offset
>>> that is less than its starting ownership index (in the initial email
>>> output).
>>>
>>> Thanks for the assistance
>>> Nicholas
>>>
>>>
>>> On Fri, Jan 6, 2023 at 9:20 AM Matthew Knepley 
>>> wrote:
>>>
 On Fri, Jan 6, 2023 at 2:28 AM Nicholas Arnold-Medabalimi <
 narno...@umich.edu> wrote:

> Hi Petsc Users,
>
> I'm working with a dmplex system with a subsampled mesh distributed
> with an overlap of 1.
>
> I'm encountering unusual situations when using VecGetOwnershipRange to
> adjust the offset received from a global section. The logic of the
> following code is first to get the offset needed to index a global vector
> while still being able to check if it is an overlapped cell and skip if
> needed while counting the owned cells.
>


>
> call DMGetGlobalSection(dmplex,section,ierr)
> call VecGetArrayF90(stateVec,stateVecV,ierr)
> call VecGetOwnershipRange(stateVec,oStart,oEnd,ierr)
> do i = c0, (c1-1)
>
> call PetscSectionGetOffset(section,i,offset,ierr)
> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart
>
> if(offset<0) then
> cycle
> endif
> offset=offset-oStart
> plexcells=plexcells+1
> stateVecV(offset)=  enddo
>
> I'm noticing some very weird results that I've appended below. The
> GetOffset documentation notes that a negative offset indicates an unowned
> point (which I use to cycle). However, the offset subtraction with oStart
> will yield an illegal index for the Vector access. I see that on the
> documentation for GetOwnershipRange, it notes that this may be
> "ill-defined"  but I wanted to see if this is type of ill-defined I can
> expect or there is just something terribly wrong with my 
> PetscSection.(both
> the Vec and Section were produced from DMPlexDistributeField so should by
> definition have synchronized section information) I was wondering if there
> is a possible output and/or the best way to index the vector. I'm thinking
> of subtracting the offset of cell 0 perhaps?
>

 Can you show your vector sizes? Are you sure it is not the fact that
 F90 arrays use 1-based indices, but these are 0-based offsets?

   Thanks,

  Matt


> on rank 0
>
>  cell   0 offset   0 oStart   0   0
>  cell   1 offset  55 oStart   0  55
>  cell   2 offset 110 oStart   0 110
>  cell   3 offset 165 oStart   0 165
>  cell   4 offset 220 oStart   0 220
>  cell   5 offset 275 oStart   0 275
>  cell   6 offset 330 oStart   0 330
>  cell   7 offset 385 oStart   0 385
>  cell   8 offset 440 oStart   0 440
>  cell   9 offset 495 oStart   0 495
>  cell  10 offset 550 oStart   0 550
>  cell  11 offset 605 oStart   0 605
>  cell  12 offset 660 oStart   0 660
>  cell  13 offset 715 oStart   0 715
>
> and on rank one
> cell   0 offset2475 oStart2640-165
>  cell 

Re: [petsc-users] Vec Ownership ranges with Global Section Offsets

[Nicholas Arnold-Medabalimi]

Apologies. If it helps, there is one cell of overlap in this small test
case for a 2D mesh that is 1 cell in height and a number of cells in
length. .

process 0
 Petsc VecGetLocalSize2750
 size(stateVecV)2750

process 1
 Petsc VecGetLocalSize2640
 size(stateVecV)2640

On Fri, Jan 6, 2023 at 9:51 AM Matthew Knepley  wrote:

> On Fri, Jan 6, 2023 at 9:37 AM Nicholas Arnold-Medabalimi <
> narno...@umich.edu> wrote:
>
>> Hi Matt
>>
>> I made a typo on the line statVecV(offset) =  in my
>> example, I agree. (I wrote that offhand since the actual assignment is much
>> larger) I should be statVecV(offset+1) =  so I'm confident it's
>> not a 1 0 indexing thing.
>>
>> My question is more related to what is happening in the offsets. c0 and
>> c1 are pulled using DMplexgetheight stratum, so they are zero-indexed
>> (which is why I loop from c0 to (c1-1)).
>>
>> For the size inquiries. on processor 0
>>  Petsc VecGetSize(stateVec)5390
>>
>
> I need to see VecGetLocalSize()
>
>   Matt
>
>
>>  size(stateVecV)   2640
>>
>> on processor 1
>> Petsc VecGetSize 5390
>> size(stateVecV)2750
>>
>> It's quite weird to me that processor one can have a positive offset that
>> is less than its starting ownership index (in the initial email output).
>>
>> Thanks for the assistance
>> Nicholas
>>
>>
>> On Fri, Jan 6, 2023 at 9:20 AM Matthew Knepley  wrote:
>>
>>> On Fri, Jan 6, 2023 at 2:28 AM Nicholas Arnold-Medabalimi <
>>> narno...@umich.edu> wrote:
>>>
 Hi Petsc Users,

 I'm working with a dmplex system with a subsampled mesh distributed
 with an overlap of 1.

 I'm encountering unusual situations when using VecGetOwnershipRange to
 adjust the offset received from a global section. The logic of the
 following code is first to get the offset needed to index a global vector
 while still being able to check if it is an overlapped cell and skip if
 needed while counting the owned cells.

>>>
>>>

 call DMGetGlobalSection(dmplex,section,ierr)
 call VecGetArrayF90(stateVec,stateVecV,ierr)
 call VecGetOwnershipRange(stateVec,oStart,oEnd,ierr)
 do i = c0, (c1-1)

 call PetscSectionGetOffset(section,i,offset,ierr)
 write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart

 if(offset<0) then
 cycle
 endif
 offset=offset-oStart
 plexcells=plexcells+1
 stateVecV(offset)=  enddo

 I'm noticing some very weird results that I've appended below. The
 GetOffset documentation notes that a negative offset indicates an unowned
 point (which I use to cycle). However, the offset subtraction with oStart
 will yield an illegal index for the Vector access. I see that on the
 documentation for GetOwnershipRange, it notes that this may be
 "ill-defined"  but I wanted to see if this is type of ill-defined I can
 expect or there is just something terribly wrong with my PetscSection.(both
 the Vec and Section were produced from DMPlexDistributeField so should by
 definition have synchronized section information) I was wondering if there
 is a possible output and/or the best way to index the vector. I'm thinking
 of subtracting the offset of cell 0 perhaps?

>>>
>>> Can you show your vector sizes? Are you sure it is not the fact that F90
>>> arrays use 1-based indices, but these are 0-based offsets?
>>>
>>>   Thanks,
>>>
>>>  Matt
>>>
>>>
 on rank 0

  cell   0 offset   0 oStart   0   0
  cell   1 offset  55 oStart   0  55
  cell   2 offset 110 oStart   0 110
  cell   3 offset 165 oStart   0 165
  cell   4 offset 220 oStart   0 220
  cell   5 offset 275 oStart   0 275
  cell   6 offset 330 oStart   0 330
  cell   7 offset 385 oStart   0 385
  cell   8 offset 440 oStart   0 440
  cell   9 offset 495 oStart   0 495
  cell  10 offset 550 oStart   0 550
  cell  11 offset 605 oStart   0 605
  cell  12 offset 660 oStart   0 660
  cell  13 offset 715 oStart   0 715

 and on rank one
 cell   0 offset2475 oStart2640-165
  cell   1 offset2530 oStart2640-110
  cell   2 offset2585 oStart2640 -55
  cell   3 offset2640 oStart2640   0
  cell   4 offset2695 oStart2640  55
  cell   5 offset2750 oStart2640 110

Re: [petsc-users] Vec Ownership ranges with Global Section Offsets

[Matthew Knepley]

On Fri, Jan 6, 2023 at 9:37 AM Nicholas Arnold-Medabalimi <
narno...@umich.edu> wrote:

> Hi Matt
>
> I made a typo on the line statVecV(offset) =  in my
> example, I agree. (I wrote that offhand since the actual assignment is much
> larger) I should be statVecV(offset+1) =  so I'm confident it's
> not a 1 0 indexing thing.
>
> My question is more related to what is happening in the offsets. c0 and c1
> are pulled using DMplexgetheight stratum, so they are zero-indexed (which
> is why I loop from c0 to (c1-1)).
>
> For the size inquiries. on processor 0
>  Petsc VecGetSize(stateVec)5390
>

I need to see VecGetLocalSize()

  Matt


>  size(stateVecV)   2640
>
> on processor 1
> Petsc VecGetSize 5390
> size(stateVecV)2750
>
> It's quite weird to me that processor one can have a positive offset that
> is less than its starting ownership index (in the initial email output).
>
> Thanks for the assistance
> Nicholas
>
>
> On Fri, Jan 6, 2023 at 9:20 AM Matthew Knepley  wrote:
>
>> On Fri, Jan 6, 2023 at 2:28 AM Nicholas Arnold-Medabalimi <
>> narno...@umich.edu> wrote:
>>
>>> Hi Petsc Users,
>>>
>>> I'm working with a dmplex system with a subsampled mesh distributed with
>>> an overlap of 1.
>>>
>>> I'm encountering unusual situations when using VecGetOwnershipRange to
>>> adjust the offset received from a global section. The logic of the
>>> following code is first to get the offset needed to index a global vector
>>> while still being able to check if it is an overlapped cell and skip if
>>> needed while counting the owned cells.
>>>
>>
>>
>>>
>>> call DMGetGlobalSection(dmplex,section,ierr)
>>> call VecGetArrayF90(stateVec,stateVecV,ierr)
>>> call VecGetOwnershipRange(stateVec,oStart,oEnd,ierr)
>>> do i = c0, (c1-1)
>>>
>>> call PetscSectionGetOffset(section,i,offset,ierr)
>>> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart
>>>
>>> if(offset<0) then
>>> cycle
>>> endif
>>> offset=offset-oStart
>>> plexcells=plexcells+1
>>> stateVecV(offset)=  enddo
>>>
>>> I'm noticing some very weird results that I've appended below. The
>>> GetOffset documentation notes that a negative offset indicates an unowned
>>> point (which I use to cycle). However, the offset subtraction with oStart
>>> will yield an illegal index for the Vector access. I see that on the
>>> documentation for GetOwnershipRange, it notes that this may be
>>> "ill-defined"  but I wanted to see if this is type of ill-defined I can
>>> expect or there is just something terribly wrong with my PetscSection.(both
>>> the Vec and Section were produced from DMPlexDistributeField so should by
>>> definition have synchronized section information) I was wondering if there
>>> is a possible output and/or the best way to index the vector. I'm thinking
>>> of subtracting the offset of cell 0 perhaps?
>>>
>>
>> Can you show your vector sizes? Are you sure it is not the fact that F90
>> arrays use 1-based indices, but these are 0-based offsets?
>>
>>   Thanks,
>>
>>  Matt
>>
>>
>>> on rank 0
>>>
>>>  cell   0 offset   0 oStart   0   0
>>>  cell   1 offset  55 oStart   0  55
>>>  cell   2 offset 110 oStart   0 110
>>>  cell   3 offset 165 oStart   0 165
>>>  cell   4 offset 220 oStart   0 220
>>>  cell   5 offset 275 oStart   0 275
>>>  cell   6 offset 330 oStart   0 330
>>>  cell   7 offset 385 oStart   0 385
>>>  cell   8 offset 440 oStart   0 440
>>>  cell   9 offset 495 oStart   0 495
>>>  cell  10 offset 550 oStart   0 550
>>>  cell  11 offset 605 oStart   0 605
>>>  cell  12 offset 660 oStart   0 660
>>>  cell  13 offset 715 oStart   0 715
>>>
>>> and on rank one
>>> cell   0 offset2475 oStart2640-165
>>>  cell   1 offset2530 oStart2640-110
>>>  cell   2 offset2585 oStart2640 -55
>>>  cell   3 offset2640 oStart2640   0
>>>  cell   4 offset2695 oStart2640  55
>>>  cell   5 offset2750 oStart2640 110
>>>  cell   6 offset2805 oStart2640 165
>>>  cell   7 offset2860 oStart2640 220
>>>  cell   8 offset2915 oStart2640 275
>>>  cell   9 offset2970 oStart2640 330
>>>  cell  10 offset3025 oStart2640 385
>>>  cell  11 offset3080 oStart2640 440
>>>  cell  12 offset3135 oStart 

Re: [petsc-users] Vec Ownership ranges with Global Section Offsets

[Nicholas Arnold-Medabalimi]

Hi Matt

I made a typo on the line statVecV(offset) =  in my
example, I agree. (I wrote that offhand since the actual assignment is much
larger) I should be statVecV(offset+1) =  so I'm confident it's
not a 1 0 indexing thing.

My question is more related to what is happening in the offsets. c0 and c1
are pulled using DMplexgetheight stratum, so they are zero-indexed (which
is why I loop from c0 to (c1-1)).

For the size inquiries. on processor 0
 Petsc VecGetSize(stateVec)5390
 size(stateVecV)   2640

on processor 1
Petsc VecGetSize 5390
size(stateVecV)2750

It's quite weird to me that processor one can have a positive offset that
is less than its starting ownership index (in the initial email output).

Thanks for the assistance
Nicholas


On Fri, Jan 6, 2023 at 9:20 AM Matthew Knepley  wrote:

> On Fri, Jan 6, 2023 at 2:28 AM Nicholas Arnold-Medabalimi <
> narno...@umich.edu> wrote:
>
>> Hi Petsc Users,
>>
>> I'm working with a dmplex system with a subsampled mesh distributed with
>> an overlap of 1.
>>
>> I'm encountering unusual situations when using VecGetOwnershipRange to
>> adjust the offset received from a global section. The logic of the
>> following code is first to get the offset needed to index a global vector
>> while still being able to check if it is an overlapped cell and skip if
>> needed while counting the owned cells.
>>
>
>
>>
>> call DMGetGlobalSection(dmplex,section,ierr)
>> call VecGetArrayF90(stateVec,stateVecV,ierr)
>> call VecGetOwnershipRange(stateVec,oStart,oEnd,ierr)
>> do i = c0, (c1-1)
>>
>> call PetscSectionGetOffset(section,i,offset,ierr)
>> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart
>>
>> if(offset<0) then
>> cycle
>> endif
>> offset=offset-oStart
>> plexcells=plexcells+1
>> stateVecV(offset)=  enddo
>>
>> I'm noticing some very weird results that I've appended below. The
>> GetOffset documentation notes that a negative offset indicates an unowned
>> point (which I use to cycle). However, the offset subtraction with oStart
>> will yield an illegal index for the Vector access. I see that on the
>> documentation for GetOwnershipRange, it notes that this may be
>> "ill-defined"  but I wanted to see if this is type of ill-defined I can
>> expect or there is just something terribly wrong with my PetscSection.(both
>> the Vec and Section were produced from DMPlexDistributeField so should by
>> definition have synchronized section information) I was wondering if there
>> is a possible output and/or the best way to index the vector. I'm thinking
>> of subtracting the offset of cell 0 perhaps?
>>
>
> Can you show your vector sizes? Are you sure it is not the fact that F90
> arrays use 1-based indices, but these are 0-based offsets?
>
>   Thanks,
>
>  Matt
>
>
>> on rank 0
>>
>>  cell   0 offset   0 oStart   0   0
>>  cell   1 offset  55 oStart   0  55
>>  cell   2 offset 110 oStart   0 110
>>  cell   3 offset 165 oStart   0 165
>>  cell   4 offset 220 oStart   0 220
>>  cell   5 offset 275 oStart   0 275
>>  cell   6 offset 330 oStart   0 330
>>  cell   7 offset 385 oStart   0 385
>>  cell   8 offset 440 oStart   0 440
>>  cell   9 offset 495 oStart   0 495
>>  cell  10 offset 550 oStart   0 550
>>  cell  11 offset 605 oStart   0 605
>>  cell  12 offset 660 oStart   0 660
>>  cell  13 offset 715 oStart   0 715
>>
>> and on rank one
>> cell   0 offset2475 oStart2640-165
>>  cell   1 offset2530 oStart2640-110
>>  cell   2 offset2585 oStart2640 -55
>>  cell   3 offset2640 oStart2640   0
>>  cell   4 offset2695 oStart2640  55
>>  cell   5 offset2750 oStart2640 110
>>  cell   6 offset2805 oStart2640 165
>>  cell   7 offset2860 oStart2640 220
>>  cell   8 offset2915 oStart2640 275
>>  cell   9 offset2970 oStart2640 330
>>  cell  10 offset3025 oStart2640 385
>>  cell  11 offset3080 oStart2640 440
>>  cell  12 offset3135 oStart2640 495
>>  cell  13 offset3190 oStart2640 550
>>  cell  14 offset3245 oStart2640 605
>>  cell  15 offset-771 oStart2640   -3411
>>
>>
>> Sincerely
>> 

Re: [petsc-users] Vec Ownership ranges with Global Section Offsets

[Matthew Knepley]

On Fri, Jan 6, 2023 at 2:28 AM Nicholas Arnold-Medabalimi <
narno...@umich.edu> wrote:

> Hi Petsc Users,
>
> I'm working with a dmplex system with a subsampled mesh distributed with
> an overlap of 1.
>
> I'm encountering unusual situations when using VecGetOwnershipRange to
> adjust the offset received from a global section. The logic of the
> following code is first to get the offset needed to index a global vector
> while still being able to check if it is an overlapped cell and skip if
> needed while counting the owned cells.
>


>
> call DMGetGlobalSection(dmplex,section,ierr)
> call VecGetArrayF90(stateVec,stateVecV,ierr)
> call VecGetOwnershipRange(stateVec,oStart,oEnd,ierr)
> do i = c0, (c1-1)
>
> call PetscSectionGetOffset(section,i,offset,ierr)
> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart
>
> if(offset<0) then
> cycle
> endif
> offset=offset-oStart
> plexcells=plexcells+1
> stateVecV(offset)=  enddo
>
> I'm noticing some very weird results that I've appended below. The
> GetOffset documentation notes that a negative offset indicates an unowned
> point (which I use to cycle). However, the offset subtraction with oStart
> will yield an illegal index for the Vector access. I see that on the
> documentation for GetOwnershipRange, it notes that this may be
> "ill-defined"  but I wanted to see if this is type of ill-defined I can
> expect or there is just something terribly wrong with my PetscSection.(both
> the Vec and Section were produced from DMPlexDistributeField so should by
> definition have synchronized section information) I was wondering if there
> is a possible output and/or the best way to index the vector. I'm thinking
> of subtracting the offset of cell 0 perhaps?
>

Can you show your vector sizes? Are you sure it is not the fact that F90
arrays use 1-based indices, but these are 0-based offsets?

  Thanks,

 Matt


> on rank 0
>
>  cell   0 offset   0 oStart   0   0
>  cell   1 offset  55 oStart   0  55
>  cell   2 offset 110 oStart   0 110
>  cell   3 offset 165 oStart   0 165
>  cell   4 offset 220 oStart   0 220
>  cell   5 offset 275 oStart   0 275
>  cell   6 offset 330 oStart   0 330
>  cell   7 offset 385 oStart   0 385
>  cell   8 offset 440 oStart   0 440
>  cell   9 offset 495 oStart   0 495
>  cell  10 offset 550 oStart   0 550
>  cell  11 offset 605 oStart   0 605
>  cell  12 offset 660 oStart   0 660
>  cell  13 offset 715 oStart   0 715
>
> and on rank one
> cell   0 offset2475 oStart2640-165
>  cell   1 offset2530 oStart2640-110
>  cell   2 offset2585 oStart2640 -55
>  cell   3 offset2640 oStart2640   0
>  cell   4 offset2695 oStart2640  55
>  cell   5 offset2750 oStart2640 110
>  cell   6 offset2805 oStart2640 165
>  cell   7 offset2860 oStart2640 220
>  cell   8 offset2915 oStart2640 275
>  cell   9 offset2970 oStart2640 330
>  cell  10 offset3025 oStart2640 385
>  cell  11 offset3080 oStart2640 440
>  cell  12 offset3135 oStart2640 495
>  cell  13 offset3190 oStart2640 550
>  cell  14 offset3245 oStart2640 605
>  cell  15 offset-771 oStart2640   -3411
>
>
> Sincerely
> Nicholas
>
> --
> Nicholas Arnold-Medabalimi
>
> Ph.D. Candidate
> Computational Aeroscience Lab
> University of Michigan
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/