I am not sure if VecDuplicate is problematic. But we should avoid collective calls in general. They do more harm than VecScatters, which are often neighborhood and do not sync all processors.
--Junchao Zhang On Tue, Jun 19, 2018 at 9:16 PM, Smith, Barry F. <[email protected]> wrote: > > Jed, > > You added these two lines. Any idea why they are needed? > > Thanks > > Barry > > 3e08d2bebf99 src/dm/impls/da/dadist.c (Barry Smith 2012-09-18 > 14) PetscFunctionBegin; > c688c0463f6c src/dm/impls/da/dadist.c (Matthew Knepley 2012-11-11 > 15) ierr = VecGetDM(g, &da);CHKERRQ(ierr); > 2dcb2ebcaafe src/dm/impls/da/dadist.c (Barry Smith 2010-10-31 > 16) ierr = DMCreateGlobalVector(da,gg);CHKERRQ(ierr); > 077aedaf8cb5 src/dm/impls/da/dadist.c (Jed Brown 2013-02-11 > 17) ierr = VecGetLayout(g,&map);CHKERRQ(ierr); > 077aedaf8cb5 src/dm/impls/da/dadist.c (Jed Brown 2013-02-11 > 18) ierr = VecSetLayout(*gg,map);CHKERRQ(ierr); > 2dcb2ebcaafe src/dm/impls/da/dadist.c (Barry Smith 2010-10-31 > 19) PetscFunctionReturn(0) > > > If we were a bit more clever we could probably avoid all communication > in VecDuplicate() but is it worth bothering unless we know it is problematic > > > On Jun 19, 2018, at 4:30 PM, Junchao Zhang <[email protected]> wrote: > > > > I met an expensive VecDuplicate. See the attached call stack. > VecDuplicate_MPI_DA indirectly calls MPI_Allreduce/Allgather to build the > vector's layout, but it then immediately destroys it with VecGetLayout & > VecSetLayout. Is it wrong? > > src/dm/impls/da/dadist.c > > PetscErrorCode VecDuplicate_MPI_DA(Vec g,Vec *gg) > > { > > PetscErrorCode ierr; > > DM da; > > PetscLayout map; > > > > PetscFunctionBegin; > > ierr = VecGetDM(g, &da);CHKERRQ(ierr); > > ierr = DMCreateGlobalVector(da,gg);CHKERRQ(ierr); > > ierr = VecGetLayout(g,&map);CHKERRQ(ierr); > > ierr = VecSetLayout(*gg,map);CHKERRQ(ierr); > > PetscFunctionReturn(0); > > } > > > > --Junchao Zhang > > <VecDup.png> > >
