On Thu, Feb 18, 2021 at 1:55 PM Junchao Zhang <[email protected]> wrote:
> VecScatter (i.e., SF, the two are the same thing) setup (building various > index lists, rank lists) is done on the CPU. is1, is2 must be host data. > Just out of curiosity, is1 and is2 can not be created on a GPU device in the first place? That being said, it is technically impossible? Or we just did not implement them yet? Fande, > When the SF is used to communicate device data, indices are copied to the > device.. > > --Junchao Zhang > > > On Thu, Feb 18, 2021 at 11:50 AM Patrick Sanan <[email protected]> > wrote: > >> I'm trying to understand how VecScatters work with GPU-native Kokkos >> Vecs. >> >> Specifically, I'm interested in what will happen in code like in >> src/vec/vec/tests/ex22.c, >> >> ierr = VecScatterCreate(x,is1,y,is2,&ctx);CHKERRQ(ierr); >> >> (from >> https://gitlab.com/petsc/petsc/-/blob/master/src/vec/vec/tests/ex22.c#L44 >> ) >> >> Here, x and y can be set to type KOKKOS using -vec_type kokkos at the >> command line. But is1 and is2 are (I think), always >> CPU/host data. Assuming that the scatter itself can happen on the GPU, >> the indices must make it to the device somehow - are they copied there when >> the scatter is created? Is there a way to create the scatter using indices >> already on the GPU (Maybe using SF more directly)? >> >>
