Greg,
The general mechanism for moving elements of vector between processes is
with VecScatterCreate() and VecScatterBegin/End() with it you can indicate
exactly what values are to move and to where, but it is all based on “one
dimension” indexing into the vector.
DMDA provides a subset of communication for “structured meshes”
(essentially 1,2, 3 dimensional arrays split among processes with horizontal
and vertical cuts). It is pretty much all or nothing in terms of communicating
neighbor information.
VecCreateGhost() uses the VecScatter mechanism to set up a SINGLE
communication pattern between a Vec and its ghosted partner Vec.
Based on your email: “However, when communicating ghost values, not all of
those degrees of freedom should be exchanged at once.” it sounds like you need
several (many) communication patterns requiring different ghost entries in
each.
1) The hard general case: if for each set of ghost points you may need several
entries from one grid point and a different number of entries from another grid
point VecScatterCreate() (called multiple times, one for each pattern) and
VecScatterBegin/End() are intended for this purpose. However if you are working
with a 2 dimension grid/array you want to do ghosting for you need to initially
map your ghosting patterns from the 2d indexing to the 1d indexing of the
VecScatterCreate() which is a bit painful. You can see the routine I mentioned
before DMSetUp_DA_2D() but it would need to be heavily modified.
2) The easy case: if, for example, you need just the first index from each
point communicated as a ghost and then next the second and then next the third
you can avoid all custom communication patterns and just create 2 DMDA, one
with a dof of 100 (or what ever it is for your case) and one with dof of 1 then
use VecStrideGather to pull out the one component of the global vector (with
the dof of 100) into a Vec obtained from DMCreateGlobalVector() from the dof of
1 DMDA then use the DMGlobalToLocalBegin/End on the 1 dof DMDA and now you have
your single ghost point at each point vector that you want.
Barry
On Feb 21, 2014, at 1:12 PM, Fischer, Greg A. <[email protected]> wrote:
> Barry,
>
> Thanks! I have another question. The user manual says:
>
> PETSc currently provides no container for multiple arrays
> sharing the same distributed array communication; note, however, that the dof
> parameter handles many cases of interest.
>
> In my application, each space location will have on the order of hundreds of
> values associated with it (which I believe translates to dof=O(100) – I don’t
> see the “degrees of freedom” explicitly defined anywhere). However, when
> communicating ghost values, not all of those degrees of freedom should be
> exchanged at once. I need to be able to exchange one at a time.
>
> It sounds like what I may want to do is use VecCreateGhost(), which would
> allow me to define exactly where the ghost points are, and then duplicate
> that vector using VecDuplicateVecs() for each DOF. I can then scatter the
> vectors individually as the need arises. Does that sound reasonable?
>
> Greg
>
> >-----Original Message-----
> >From: Barry Smith [mailto:[email protected]]
> >Sent: Friday, February 21, 2014 12:21 PM
> >To: Fischer, Greg A.
> >Cc: [email protected]
> >Subject: Re: [petsc-users] partial stencil in DMDA?
> >
> >
> >On Feb 21, 2014, at 11:02 AM, Fischer, Greg A. <[email protected]>
> >wrote:
> >
> >> Hello,
> >>
> >> I’m interested in using PETSc to manage distributed arrays. Based on my
> >reading about the DMDA objects, I see that ghost points can be
> >communicated in box-type stencils or star-type stencils.
> >>
> >> For my application, assume that I have a 2D DMDA object with star-type
> >stencils. For a typical local calculation, I only need to access ghost values
> >from two of the four directions at a time. For example, I’d like access to
> >ghost
> >values in the South and East directions, but not in the North or West
> >directions. Communicating North and West data would seem to be wasting
> >bandwidth. Is there any way to accomplish this?
> >
> > Greg,
> >
> > There is not anything built in. Here is what I suggest:
> >
> >1) write your application code not worrying about the fact that the
> >DMGlobalToLocalBegin/End() is moving values you don’t need.
> >
> >2) when your code is running correctly for your problem and giving useful
> >results if the communication times are impacting how long it takes to run you
> >can provide a custom communication pattern. It would involve little
> >additional coding essentially taking DMSetUp_DA_2D() which creates the list
> >of ghost points and removing the unneeded ghost points. But it would be
> >premature to do this optimization until you have a full working application
> >code.
> >
> > Barry
> >
> >>
> >> Thanks,
> >> Greg
> >
> >
