Thanks, '-mat_type hypre' was failing for me. I could not find a test that used it and I was not sure it was considered functional. I will look at it again and collect a bug report if needed.
On Sat, Jan 22, 2022 at 11:31 AM Stefano Zampini <[email protected]> wrote: > Mark > > the two options are only there to test the code in CI, and are not needed > in general > > '--download-hypre-configure-arguments=--enable-unified-memory', > This is only here to test the unified memory code path > > '--with-hypre-gpuarch=gfx90a', > This is not needed if rocminfo is in PATH > > Our interface code with HYPRE GPU works fine for HIP, it is tested in CI. > The -mat_type hypre assembling for ex19 does not work because ex19 uses > FDColoring. Just assemble in mpiaij format (look at runex19_hypre_hip in > src/snes/tutorials/makefile); the interface code will copy the matrix to > the GPU > > Il giorno ven 21 gen 2022 alle ore 19:24 Mark Adams <[email protected]> ha > scritto: > >> >> >> On Fri, Jan 21, 2022 at 11:14 AM Jed Brown <[email protected]> wrote: >> >>> "Paul T. Bauman" <[email protected]> writes: >>> >>> > On Fri, Jan 21, 2022 at 8:52 AM Paul T. Bauman <[email protected]> >>> wrote: >>> >> Yes. The way HYPRE's memory model is setup is that ALL GPU >>> allocations are >>> >> "native" (i.e. [cuda,hip]Malloc) or, if unified memory is enabled, >>> then ALL >>> >> GPU allocations are unified memory (i.e. [cuda,hip]MallocManaged). >>> >> Regarding HIP, there is an HMM implementation of hipMallocManaged >>> planned, >>> >> but is it not yet delivered AFAIK (and it will *not* support gfx906, >>> e.g. >>> >> RVII, FYI), so, today, under the covers, hipMallocManaged is calling >>> >> hipHostMalloc. So, today, all your unified memory allocations in >>> HYPRE on >>> >> HIP are doing CPU-pinned memory accesses. And performance is just >>> truly >>> >> terrible (as you might expect). >>> >>> Thanks for this important bit of information. >>> >>> And it sounds like when we add support to hand off Kokkos matrices and >>> vectors (our current support for matrices on ROCm devices uses Kokkos) or >>> add direct support for hipSparse, we'll avoid touching host memory in >>> assembly-to-solve with hypre. >>> >> >> It does not look like anyone has made Hypre work with HIP. Stafano added >> a runex19_hypre_hip target 4 months ago and hypre.py has some HIP things. >> >> I have a user that would like to try this, no hurry but, can I get an >> idea of a plan for this? >> >> Thanks, >> Mark >> >> > > > -- > Stefano >
