Hi Matt I apologize for any lack of clarity in the initial email.
looking at the initial output on rank 1 write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart cell 0 offset 2475 oStart 2640 -165 cell 1 offset 2530 oStart 2640 -110 cell 2 offset 2585 oStart 2640 -55 cell 3 offset 2640 oStart 2640 0 ..... cell 15 offset -771 oStart 2640 -3411 cell 15 provides a negative offset because it is the overlap cell (that is unowned) The remained of cells are all owned. However, the first 3 cells (0,1,2) return an offset that is less than the starting ownership range. I would expect cell 0 to start at offset 2640 at minimum. Sincerely Nicholas On Fri, Jan 6, 2023 at 10:05 AM Matthew Knepley <knep...@gmail.com> wrote: > On Fri, Jan 6, 2023 at 9:56 AM Nicholas Arnold-Medabalimi < > narno...@umich.edu> wrote: > >> Apologies. If it helps, there is one cell of overlap in this small test >> case for a 2D mesh that is 1 cell in height and a number of cells in >> length. . >> >> process 0 >> Petsc VecGetLocalSize 2750 >> size(stateVecV) 2750 >> >> process 1 >> Petsc VecGetLocalSize 2640 >> size(stateVecV) 2640 >> > > The offsets shown below are well-within these sizes. I do not understand > the problem. > > Thanks, > > Matt > > >> On Fri, Jan 6, 2023 at 9:51 AM Matthew Knepley <knep...@gmail.com> wrote: >> >>> On Fri, Jan 6, 2023 at 9:37 AM Nicholas Arnold-Medabalimi < >>> narno...@umich.edu> wrote: >>> >>>> Hi Matt >>>> >>>> I made a typo on the line statVecV(offset) = <set to something> in my >>>> example, I agree. (I wrote that offhand since the actual assignment is much >>>> larger) I should be statVecV(offset+1) = <assignment> so I'm confident it's >>>> not a 1 0 indexing thing. >>>> >>>> My question is more related to what is happening in the offsets. c0 and >>>> c1 are pulled using DMplexgetheight stratum, so they are zero-indexed >>>> (which is why I loop from c0 to (c1-1)). >>>> >>>> For the size inquiries. on processor 0 >>>> Petsc VecGetSize(stateVec) 5390 >>>> >>> >>> I need to see VecGetLocalSize() >>> >>> Matt >>> >>> >>>> size(stateVecV) 2640 >>>> >>>> on processor 1 >>>> Petsc VecGetSize 5390 >>>> size(stateVecV) 2750 >>>> >>>> It's quite weird to me that processor one can have a positive offset >>>> that is less than its starting ownership index (in the initial email >>>> output). >>>> >>>> Thanks for the assistance >>>> Nicholas >>>> >>>> >>>> On Fri, Jan 6, 2023 at 9:20 AM Matthew Knepley <knep...@gmail.com> >>>> wrote: >>>> >>>>> On Fri, Jan 6, 2023 at 2:28 AM Nicholas Arnold-Medabalimi < >>>>> narno...@umich.edu> wrote: >>>>> >>>>>> Hi Petsc Users, >>>>>> >>>>>> I'm working with a dmplex system with a subsampled mesh distributed >>>>>> with an overlap of 1. >>>>>> >>>>>> I'm encountering unusual situations when using VecGetOwnershipRange >>>>>> to adjust the offset received from a global section. The logic of the >>>>>> following code is first to get the offset needed to index a global vector >>>>>> while still being able to check if it is an overlapped cell and skip if >>>>>> needed while counting the owned cells. >>>>>> >>>>> >>>>> >>>>>> >>>>>> call DMGetGlobalSection(dmplex,section,ierr) >>>>>> call VecGetArrayF90(stateVec,stateVecV,ierr) >>>>>> call VecGetOwnershipRange(stateVec,oStart,oEnd,ierr) >>>>>> do i = c0, (c1-1) >>>>>> >>>>>> call PetscSectionGetOffset(section,i,offset,ierr) >>>>>> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset- >>>>>> oStart >>>>>> >>>>>> if(offset<0) then >>>>>> cycle >>>>>> endif >>>>>> offset=offset-oStart >>>>>> plexcells=plexcells+1 >>>>>> stateVecV(offset)= <set to something> enddo >>>>>> >>>>>> I'm noticing some very weird results that I've appended below. The >>>>>> GetOffset documentation notes that a negative offset indicates an unowned >>>>>> point (which I use to cycle). However, the offset subtraction with oStart >>>>>> will yield an illegal index for the Vector access. I see that on the >>>>>> documentation for GetOwnershipRange, it notes that this may be >>>>>> "ill-defined" but I wanted to see if this is type of ill-defined I can >>>>>> expect or there is just something terribly wrong with my >>>>>> PetscSection.(both >>>>>> the Vec and Section were produced from DMPlexDistributeField so should by >>>>>> definition have synchronized section information) I was wondering if >>>>>> there >>>>>> is a possible output and/or the best way to index the vector. I'm >>>>>> thinking >>>>>> of subtracting the offset of cell 0 perhaps? >>>>>> >>>>> >>>>> Can you show your vector sizes? Are you sure it is not the fact that >>>>> F90 arrays use 1-based indices, but these are 0-based offsets? >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> on rank 0 >>>>>> >>>>>> cell 0 offset 0 oStart 0 0 >>>>>> cell 1 offset 55 oStart 0 55 >>>>>> cell 2 offset 110 oStart 0 110 >>>>>> cell 3 offset 165 oStart 0 165 >>>>>> cell 4 offset 220 oStart 0 220 >>>>>> cell 5 offset 275 oStart 0 275 >>>>>> cell 6 offset 330 oStart 0 330 >>>>>> cell 7 offset 385 oStart 0 385 >>>>>> cell 8 offset 440 oStart 0 440 >>>>>> cell 9 offset 495 oStart 0 495 >>>>>> cell 10 offset 550 oStart 0 550 >>>>>> cell 11 offset 605 oStart 0 605 >>>>>> cell 12 offset 660 oStart 0 660 >>>>>> cell 13 offset 715 oStart 0 715 >>>>>> >>>>>> and on rank one >>>>>> cell 0 offset 2475 oStart 2640 -165 >>>>>> cell 1 offset 2530 oStart 2640 -110 >>>>>> cell 2 offset 2585 oStart 2640 -55 >>>>>> cell 3 offset 2640 oStart 2640 0 >>>>>> cell 4 offset 2695 oStart 2640 55 >>>>>> cell 5 offset 2750 oStart 2640 110 >>>>>> cell 6 offset 2805 oStart 2640 165 >>>>>> cell 7 offset 2860 oStart 2640 220 >>>>>> cell 8 offset 2915 oStart 2640 275 >>>>>> cell 9 offset 2970 oStart 2640 330 >>>>>> cell 10 offset 3025 oStart 2640 385 >>>>>> cell 11 offset 3080 oStart 2640 440 >>>>>> cell 12 offset 3135 oStart 2640 495 >>>>>> cell 13 offset 3190 oStart 2640 550 >>>>>> cell 14 offset 3245 oStart 2640 605 >>>>>> cell 15 offset -771 oStart 2640 -3411 >>>>>> >>>>>> >>>>>> Sincerely >>>>>> Nicholas >>>>>> >>>>>> -- >>>>>> Nicholas Arnold-Medabalimi >>>>>> >>>>>> Ph.D. Candidate >>>>>> Computational Aeroscience Lab >>>>>> University of Michigan >>>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> <http://www.cse.buffalo.edu/~knepley/> >>>>> >>>> >>>> >>>> -- >>>> Nicholas Arnold-Medabalimi >>>> >>>> Ph.D. Candidate >>>> Computational Aeroscience Lab >>>> University of Michigan >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> <http://www.cse.buffalo.edu/~knepley/> >>> >> >> >> -- >> Nicholas Arnold-Medabalimi >> >> Ph.D. Candidate >> Computational Aeroscience Lab >> University of Michigan >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > <http://www.cse.buffalo.edu/~knepley/> > -- Nicholas Arnold-Medabalimi Ph.D. Candidate Computational Aeroscience Lab University of Michigan