Darn, of course! This works! My sincere apologies and many thanks for the swift reply!
Fabian On 05.06.20 14:04, Dave May wrote: > > > On Fri 5. Jun 2020 at 13:52, Fabian Jakub > <[email protected] > <mailto:[email protected]>> wrote: > > Dear Petsc list, > > I have a question regarding reverse vec-scatters: > > I have a shared memory solver that I want to use on a distributed > DMDA and average its results. > > The shared mem solver needs some of the global state. > > So I want to create a full copy of a global vec on each master > rank of a machine, compute some result > > and average the results back into the global vec. > > > For example: the parallel layout for a global vector is > > 3 ranks in comm_world, each with Ndof = 6, so global size is 18 > > Then I create a mpi sub communicator with MPI_COMM_TYPE_SHARED > > Then I create a local vec on each rank, with sizes 9 on each > rank_0 on the sub comm, size 0 otherwise > > if ( sub_id == 0 ) then > call VecGetSize(gvec, Nlocal, ierr); call CHKERR(ierr) > else > Nlocal = 0 > endif > call VecCreateSeq(PETSC_COMM_SELF, Nlocal, lvec, ierr) > > This yields for example (first 2 ranks in a shared mem subcomm, > 3rd rank on another machine): > > Global rank_id | sub_rank_id | local_size_gvec | > local_size_lvec > > 0 | 0 | > 6 | 18 > > 1 | 1 | > 6 | 0 > > 2 | 0 | > 6 | 18 > > > To copy the global vec on each shared mem master, I do: > > ISCreateStride(PETSC_COMM_SELF, Nlocal, 0, 1, is, ierr) > > VecScatterCreate(gvec, is, lvec, is, ctx, ierr) > > VecScatterBegin(ctx, gvec, lvec, INSERT_VALUES, SCATTER_FORWARD, ierr) > > VecScatterEnd (ctx, gvec, lvec, INSERT_VALUES, SCATTER_FORWARD, ierr) > > > That works fine. > > Then I want to do the reverse, i.e. add all values from the local > vec to the global vec on comm_world to generate the average of the > results. > > I tried: > > VecSet(gvec, zero, ierr) > > VecScatterBegin(ctx, gvec, lvec, ADD_VALUES, SCATTER_REVERSE, ierr) > > > I was hoping to get the sum of svec in the global vec, so that > gvec / comm_size(sub_comm_id==0) gives the mean. > > However, I get the following error: > > > Nonconforming object sizes > > Vector wrong size 18 for scatter 6 (scatter reverse and vector to > != ctx from size) > > > Going with the same approach with VecScatterCreateToAll > > <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecScatterCreateToAll.html#VecScatterCreateToAll> > leads to the same issue. > > Do you have suggestions on how I could/should achieve it? > > > > You have to flip the lvec, gvec args in reverse mode (as per the man > pages states under Notes) > > Thanks > Dave > > > > Many thanks! > > Fabian >
signature.asc
Description: OpenPGP digital signature
