Hi Matt This was generated using the DMPlexDistributeField which we discussed a while back. Everything seemed to be working fine when I only had cells dofs but I recently added face dofs, which seems to have caused some issues. Whats weird is that I'm feeding the same distribution SF and inputting a vector and section that are consistent to the DMPlexDistributeField so I'd expect the vectors and section output to be consistent. I'll take a closer look at that.
Thanks Nicholas On Fri, Jan 6, 2023 at 10:59 AM Matthew Knepley <[email protected]> wrote: > On Fri, Jan 6, 2023 at 10:41 AM Nicholas Arnold-Medabalimi < > [email protected]> wrote: > >> Hi Matt >> >> I appreciate the help. The section view is quite extensive because each >> cell has 55 dofs located at the cells and on certain faces. I've appended >> the first of these which corresponds with the output in the first email, to >> save space. The following 54 are exactly the same but offset incremented by >> 1. (or negative 1 for negative offsets) >> > > Okay, from the output it is clear that this vector does not match your > global section. Did you get stateVec by calling DMCreateGlobalVector()? > > Thanks, > > Matt > > >> Thanks for your time >> Nicholas >> >> On Fri, Jan 6, 2023 at 10:23 AM Matthew Knepley <[email protected]> >> wrote: >> >>> On Fri, Jan 6, 2023 at 10:10 AM Nicholas Arnold-Medabalimi < >>> [email protected]> wrote: >>> >>>> Hi Matt >>>> >>>> I apologize for any lack of clarity in the initial email. >>>> >>>> looking at the initial output on rank 1 >>>> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart >>>> cell 0 offset 2475 oStart 2640 -165 >>>> cell 1 offset 2530 oStart 2640 -110 >>>> cell 2 offset 2585 oStart 2640 -55 >>>> cell 3 offset 2640 oStart 2640 0 >>>> ..... >>>> cell 15 offset -771 oStart 2640 -3411 >>>> >>>> >>>> cell 15 provides a negative offset because it is the overlap cell (that >>>> is unowned) >>>> The remained of cells are all owned. However, the first 3 cells (0,1,2) >>>> return an offset that is less than the starting ownership range. I would >>>> expect cell 0 to start at offset 2640 at minimum. >>>> >>> >>> Send the output for this section >>> >>> call PetscSectionView(section, PETSC_VIEWER_STDOUT_WORLD); >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Sincerely >>>> Nicholas >>>> >>>> >>>> >>>> >>>> On Fri, Jan 6, 2023 at 10:05 AM Matthew Knepley <[email protected]> >>>> wrote: >>>> >>>>> On Fri, Jan 6, 2023 at 9:56 AM Nicholas Arnold-Medabalimi < >>>>> [email protected]> wrote: >>>>> >>>>>> Apologies. If it helps, there is one cell of overlap in this small >>>>>> test case for a 2D mesh that is 1 cell in height and a number of cells in >>>>>> length. . >>>>>> >>>>>> process 0 >>>>>> Petsc VecGetLocalSize 2750 >>>>>> size(stateVecV) 2750 >>>>>> >>>>>> process 1 >>>>>> Petsc VecGetLocalSize 2640 >>>>>> size(stateVecV) 2640 >>>>>> >>>>> >>>>> The offsets shown below are well-within these sizes. I do not >>>>> understand the problem. >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> On Fri, Jan 6, 2023 at 9:51 AM Matthew Knepley <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> On Fri, Jan 6, 2023 at 9:37 AM Nicholas Arnold-Medabalimi < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi Matt >>>>>>>> >>>>>>>> I made a typo on the line statVecV(offset) = <set to something> in >>>>>>>> my example, I agree. (I wrote that offhand since the actual assignment >>>>>>>> is >>>>>>>> much larger) I should be statVecV(offset+1) = <assignment> so I'm >>>>>>>> confident >>>>>>>> it's not a 1 0 indexing thing. >>>>>>>> >>>>>>>> My question is more related to what is happening in the offsets. c0 >>>>>>>> and c1 are pulled using DMplexgetheight stratum, so they are >>>>>>>> zero-indexed >>>>>>>> (which is why I loop from c0 to (c1-1)). >>>>>>>> >>>>>>>> For the size inquiries. on processor 0 >>>>>>>> Petsc VecGetSize(stateVec) 5390 >>>>>>>> >>>>>>> >>>>>>> I need to see VecGetLocalSize() >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>>> size(stateVecV) 2640 >>>>>>>> >>>>>>>> on processor 1 >>>>>>>> Petsc VecGetSize 5390 >>>>>>>> size(stateVecV) 2750 >>>>>>>> >>>>>>>> It's quite weird to me that processor one can have a positive >>>>>>>> offset that is less than its starting ownership index (in the initial >>>>>>>> email >>>>>>>> output). >>>>>>>> >>>>>>>> Thanks for the assistance >>>>>>>> Nicholas >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Jan 6, 2023 at 9:20 AM Matthew Knepley <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> On Fri, Jan 6, 2023 at 2:28 AM Nicholas Arnold-Medabalimi < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi Petsc Users, >>>>>>>>>> >>>>>>>>>> I'm working with a dmplex system with a subsampled mesh >>>>>>>>>> distributed with an overlap of 1. >>>>>>>>>> >>>>>>>>>> I'm encountering unusual situations when using >>>>>>>>>> VecGetOwnershipRange to adjust the offset received from a global >>>>>>>>>> section. >>>>>>>>>> The logic of the following code is first to get the offset needed to >>>>>>>>>> index >>>>>>>>>> a global vector while still being able to check if it is an >>>>>>>>>> overlapped cell >>>>>>>>>> and skip if needed while counting the owned cells. >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> call DMGetGlobalSection(dmplex,section,ierr) >>>>>>>>>> call VecGetArrayF90(stateVec,stateVecV,ierr) >>>>>>>>>> call VecGetOwnershipRange(stateVec,oStart,oEnd,ierr) >>>>>>>>>> do i = c0, (c1-1) >>>>>>>>>> >>>>>>>>>> call PetscSectionGetOffset(section,i,offset,ierr) >>>>>>>>>> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset- >>>>>>>>>> oStart >>>>>>>>>> >>>>>>>>>> if(offset<0) then >>>>>>>>>> cycle >>>>>>>>>> endif >>>>>>>>>> offset=offset-oStart >>>>>>>>>> plexcells=plexcells+1 >>>>>>>>>> stateVecV(offset)= <set to something> enddo >>>>>>>>>> >>>>>>>>>> I'm noticing some very weird results that I've appended below. >>>>>>>>>> The GetOffset documentation notes that a negative offset indicates an >>>>>>>>>> unowned point (which I use to cycle). However, the offset >>>>>>>>>> subtraction with >>>>>>>>>> oStart will yield an illegal index for the Vector access. I see that >>>>>>>>>> on the >>>>>>>>>> documentation for GetOwnershipRange, it notes that this may be >>>>>>>>>> "ill-defined" but I wanted to see if this is type of ill-defined I >>>>>>>>>> can >>>>>>>>>> expect or there is just something terribly wrong with my >>>>>>>>>> PetscSection.(both >>>>>>>>>> the Vec and Section were produced from DMPlexDistributeField so >>>>>>>>>> should by >>>>>>>>>> definition have synchronized section information) I was wondering if >>>>>>>>>> there >>>>>>>>>> is a possible output and/or the best way to index the vector. I'm >>>>>>>>>> thinking >>>>>>>>>> of subtracting the offset of cell 0 perhaps? >>>>>>>>>> >>>>>>>>> >>>>>>>>> Can you show your vector sizes? Are you sure it is not the fact >>>>>>>>> that F90 arrays use 1-based indices, but these are 0-based offsets? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Matt >>>>>>>>> >>>>>>>>> >>>>>>>>>> on rank 0 >>>>>>>>>> >>>>>>>>>> cell 0 offset 0 oStart 0 >>>>>>>>>> 0 >>>>>>>>>> cell 1 offset 55 oStart 0 >>>>>>>>>> 55 >>>>>>>>>> cell 2 offset 110 oStart 0 >>>>>>>>>> 110 >>>>>>>>>> cell 3 offset 165 oStart 0 >>>>>>>>>> 165 >>>>>>>>>> cell 4 offset 220 oStart 0 >>>>>>>>>> 220 >>>>>>>>>> cell 5 offset 275 oStart 0 >>>>>>>>>> 275 >>>>>>>>>> cell 6 offset 330 oStart 0 >>>>>>>>>> 330 >>>>>>>>>> cell 7 offset 385 oStart 0 >>>>>>>>>> 385 >>>>>>>>>> cell 8 offset 440 oStart 0 >>>>>>>>>> 440 >>>>>>>>>> cell 9 offset 495 oStart 0 >>>>>>>>>> 495 >>>>>>>>>> cell 10 offset 550 oStart 0 >>>>>>>>>> 550 >>>>>>>>>> cell 11 offset 605 oStart 0 >>>>>>>>>> 605 >>>>>>>>>> cell 12 offset 660 oStart 0 >>>>>>>>>> 660 >>>>>>>>>> cell 13 offset 715 oStart 0 >>>>>>>>>> 715 >>>>>>>>>> >>>>>>>>>> and on rank one >>>>>>>>>> cell 0 offset 2475 oStart 2640 -165 >>>>>>>>>> cell 1 offset 2530 oStart 2640 >>>>>>>>>> -110 >>>>>>>>>> cell 2 offset 2585 oStart 2640 >>>>>>>>>> -55 >>>>>>>>>> cell 3 offset 2640 oStart 2640 >>>>>>>>>> 0 >>>>>>>>>> cell 4 offset 2695 oStart 2640 >>>>>>>>>> 55 >>>>>>>>>> cell 5 offset 2750 oStart 2640 >>>>>>>>>> 110 >>>>>>>>>> cell 6 offset 2805 oStart 2640 >>>>>>>>>> 165 >>>>>>>>>> cell 7 offset 2860 oStart 2640 >>>>>>>>>> 220 >>>>>>>>>> cell 8 offset 2915 oStart 2640 >>>>>>>>>> 275 >>>>>>>>>> cell 9 offset 2970 oStart 2640 >>>>>>>>>> 330 >>>>>>>>>> cell 10 offset 3025 oStart 2640 >>>>>>>>>> 385 >>>>>>>>>> cell 11 offset 3080 oStart 2640 >>>>>>>>>> 440 >>>>>>>>>> cell 12 offset 3135 oStart 2640 >>>>>>>>>> 495 >>>>>>>>>> cell 13 offset 3190 oStart 2640 >>>>>>>>>> 550 >>>>>>>>>> cell 14 offset 3245 oStart 2640 >>>>>>>>>> 605 >>>>>>>>>> cell 15 offset -771 oStart 2640 >>>>>>>>>> -3411 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Sincerely >>>>>>>>>> Nicholas >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Nicholas Arnold-Medabalimi >>>>>>>>>> >>>>>>>>>> Ph.D. Candidate >>>>>>>>>> Computational Aeroscience Lab >>>>>>>>>> University of Michigan >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>> experiments is infinitely more interesting than any results to which >>>>>>>>> their >>>>>>>>> experiments lead. >>>>>>>>> -- Norbert Wiener >>>>>>>>> >>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>>> <http://www.cse.buffalo.edu/~knepley/> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Nicholas Arnold-Medabalimi >>>>>>>> >>>>>>>> Ph.D. Candidate >>>>>>>> Computational Aeroscience Lab >>>>>>>> University of Michigan >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which >>>>>>> their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>> <http://www.cse.buffalo.edu/~knepley/> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Nicholas Arnold-Medabalimi >>>>>> >>>>>> Ph.D. Candidate >>>>>> Computational Aeroscience Lab >>>>>> University of Michigan >>>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> <http://www.cse.buffalo.edu/~knepley/> >>>>> >>>> >>>> >>>> -- >>>> Nicholas Arnold-Medabalimi >>>> >>>> Ph.D. Candidate >>>> Computational Aeroscience Lab >>>> University of Michigan >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> <http://www.cse.buffalo.edu/~knepley/> >>> >> >> >> -- >> Nicholas Arnold-Medabalimi >> >> Ph.D. Candidate >> Computational Aeroscience Lab >> University of Michigan >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > <http://www.cse.buffalo.edu/~knepley/> > -- Nicholas Arnold-Medabalimi Ph.D. Candidate Computational Aeroscience Lab University of Michigan
