Well, I think it's a moderately crazy idea unless it's less painful to implement than I'm thinking. Is there a use case for a mixed device system where one petsc executable might be addressing both a HIP and CUDA device beyond some frankenstein test system somebody cooked up? In all my code I implicitly assume I have either have one host with one device or one host with zero devices. I guess you can support these weird scenarios, but why? Life is hard enough supporting one device compiler with one host compiler.
Many thanks Junchao -- with combinations of SetPreallocation I was able to grab allocated pointers out of petsc. Now I have all the jacobian construction on device with no copies. On Fri, Jan 6, 2023 at 12:27 AM Barry Smith <[email protected]> wrote: > > So Jed's "everyone" now consists of "no one" and Jed can stop > complaining that "everyone" thinks it is a bad idea. > > > > On Jan 5, 2023, at 11:50 PM, Junchao Zhang <[email protected]> > wrote: > > > > > On Thu, Jan 5, 2023 at 10:32 PM Barry Smith <[email protected]> wrote: > >> >> >> > On Jan 5, 2023, at 3:42 PM, Jed Brown <[email protected]> wrote: >> > >> > Mark Adams <[email protected]> writes: >> > >> >> Support of HIP and CUDA hardware together would be crazy, >> > >> > I don't think it's remotely crazy. libCEED supports both together and >> it's very convenient when testing on a development machine that has one of >> each brand GPU and simplifies binary distribution for us and every package >> that uses us. Every day I wish PETSc could build with both simultaneously, >> but everyone tells me it's silly. >> >> Not everyone at all; just a subset of everyone. Junchao is really the >> hold-out :-) >> > I am not, instead I think we should try (I fully agree it can ease binary > distribution). But satish needs to install such a machine first :) > There are issues out of our control if we want to mix GPUs in execution. > For example, how to do VexAXPY on a cuda vector and a hip vector? Shall we > do it on the host? Also, there are no gpu-aware MPI implementations > supporting messages between cuda memory and hip memory. > >> >> I just don't care about "binary packages" :-); I think they are an >> archaic and bad way of thinking about code distribution (but yes the >> alternatives need lots of work to make them flawless, but I think that is >> where the work should go in the packaging world.) >> >> I go further and think one should be able to automatically use a CUDA >> vector on a HIP device as well, it is not hard in theory but requires >> thinking about how we handle classes and subclasses a little to make it >> straightforward; or perhaps Jacob has fixed that also? > > > > >
