Re: [petsc-dev] Understanding Vecscatter with Kokkos Vecs

2021-02-20 Thread Barry Smith
Using the git history one can presumably get the version a function was introduced (appeared in the source). Changes are a bit trickier but I guess one could take the current function, march back in time in git until there is a diff appear. Of course many diffs are trivial. In theory one

Re: [petsc-dev] Understanding Vecscatter with Kokkos Vecs

2021-02-19 Thread Jed Brown
Patrick Sanan writes: > Aside: it's been on the list of good things to do, docs-wise, to be able to > label parts of the API as more or less stable, so I'm hoping we'll get to > that (though I think it makes sense to wait until we've finished some of the > current migrations tasks). Yes, I

Re: [petsc-dev] Understanding Vecscatter with Kokkos Vecs

2021-02-19 Thread Patrick Sanan
We ended up doing it just as you say - copy the data to the host, use it to build the IS, and build the scatter. Would be fun to optimize further, maybe, but as you say that might be premature since there's ongoing work. Happy to get to play with it a bit, though! Aside: it's been on the list

Re: [petsc-dev] Understanding Vecscatter with Kokkos Vecs

2021-02-19 Thread Jed Brown
ISCUDA isn't even right (perhaps ISGENERALCUDA, ISBLOCKCUDA). I agree that this isn't a priority, but I could see it being needed in the next few years to avoid bottlenecks in adaptive mesh refinement or other adaptive algorithms. It's not a small amount of work, but I think all the index

Re: [petsc-dev] Understanding Vecscatter with Kokkos Vecs

2021-02-19 Thread Junchao Zhang
Even ISCUDA is simple to add, the PetscSFSetUp algorithm and many functions involved are done on host (and are not simple to be parallelized on GPU) The indices passed to VecScatter are analyzed and re-grouped. Even they are copied to device eventually, they are likely not in their original form.

Re: [petsc-dev] Understanding Vecscatter with Kokkos Vecs

2021-02-19 Thread Patrick Sanan
True that it's not a huge efficiency concern, as this only affects the setup stage. This is more wondering if there's a simpler way to do the setup (sounds like there isn't at the moment). > Am 19.02.2021 um 11:41 schrieb Stefano Zampini : > > I dont understand the issue. I assume your

Re: [petsc-dev] Understanding Vecscatter with Kokkos Vecs

2021-02-19 Thread Stefano Zampini
I dont understand the issue. I assume your vecscatter is not a throw away object but you will reuse it many times. Once the setup is done, the indices used internally by the implementation should be on gpu , or not? Il Ven 19 Feb 2021, 13:33 Patrick Sanan ha scritto: > Thanks! That helps a

Re: [petsc-dev] Understanding Vecscatter with Kokkos Vecs

2021-02-19 Thread Patrick Sanan
Thanks! That helps a lot. I assume "no," but is ISCUDA simple to add? More on what I'm trying to do, in case I'm missing an obvious approach: I'm working on a demo code that uses an external library, based on Kokkos, as a solver - I create a Vec of type KOKKOS and populate it with the

Re: [petsc-dev] Understanding Vecscatter with Kokkos Vecs

2021-02-18 Thread Junchao Zhang
On Thu, Feb 18, 2021 at 4:04 PM Fande Kong wrote: > > > On Thu, Feb 18, 2021 at 1:55 PM Junchao Zhang > wrote: > >> VecScatter (i.e., SF, the two are the same thing) setup (building various >> index lists, rank lists) is done on the CPU. is1, is2 must be host data. >> > > Just out of

Re: [petsc-dev] Understanding Vecscatter with Kokkos Vecs

2021-02-18 Thread Fande Kong
On Thu, Feb 18, 2021 at 1:55 PM Junchao Zhang wrote: > VecScatter (i.e., SF, the two are the same thing) setup (building various > index lists, rank lists) is done on the CPU. is1, is2 must be host data. > Just out of curiosity, is1 and is2 can not be created on a GPU device in the first

Re: [petsc-dev] Understanding Vecscatter with Kokkos Vecs

2021-02-18 Thread Junchao Zhang
VecScatter (i.e., SF, the two are the same thing) setup (building various index lists, rank lists) is done on the CPU. is1, is2 must be host data. When the SF is used to communicate device data, indices are copied to the device.. --Junchao Zhang On Thu, Feb 18, 2021 at 11:50 AM Patrick Sanan

[petsc-dev] Understanding Vecscatter with Kokkos Vecs

2021-02-18 Thread Patrick Sanan
I'm trying to understand how VecScatters work with GPU-native Kokkos Vecs. Specifically, I'm interested in what will happen in code like in src/vec/vec/tests/ex22.c, ierr = VecScatterCreate(x,is1,y,is2,);CHKERRQ(ierr); (from