*>> You can first fix on what MPI to use - and build everyting else with
it. (and avoid mixing in multiple MPIs) *
It sounds wise to make PETSC point to CUDA MPI.
I changed PETSC configure to:
$ ./configure --prefix=/usr/local/petsc --PETSC_ARCH=arch-linux-c-opt
--with-debugging=0 --with-mpi
*--with-mpi-dir=${NVHPC_ROOT}/comm_libs/12.9/openmpi4/latest*
--download-hypre --with-blaslapack --with-cuda
--with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/25.5/cuda && make all &&
make install
And now it seems hypre is not happy:"Cannot use hypre with 64-bit
BLAS/LAPACK indices".
Any idea on that one?
* >> Or use 'cuda toolkit' instead of 'nvhpc-sdk'*
We are using nvfortran, which is shipped with nvhpc-sdk AFAIK
*>> E4S has containers with PETSc+CUDA installed (via Spack) - I guess you
can give it a try *
I'll take a look. Thanks!
On Sat, Aug 2, 2025 at 12:20 PM Satish Balay <balay....@fastmail.org> wrote:
> You can first fix on what MPI to use - and build everyting else with it.
> (and avoid mixing in multiple MPIs)
>
> I've had issues with Ubuntu built MPI - but if it works for you - thats
> great!
>
> You can make sure /usr/bin is ahead in your PATH [before nvhpc-sdk so that
> this MPI gets picked up] - or specify explicitly to PETSc with:
>
> Or use 'cuda toolkit' instead of 'nvhpc-sdk'
>
> ./configure --with-cc=/usr/bin/mpicc --with-cxx=/usr/bin/mpicxx
> --with-fc=/usr/bin/mpif90 --download-hypre
>
> etc..
>
> BTW: PETSc configure attempts to run some binaries to detect
> compiler/library features, and such run tests (when linked in with cuda
> libraries) can fail if a cuda build is done on a box (container?) without
> nvidia driver installed.
>
> Satish
>
>
> On Sat, 2 Aug 2025, Renato E Poli wrote:
>
> > Understood.
> >
> > I believe I am running in circles.
> >
> > I managed to create the container, but it ended up with multiple MPI
> > implementations.
> > It seems that CUDA SDK has its own MPI, and polutes the PATH.
> > Is there any docker container PETSC+CUDA you would recommend?
> > This is a snippet of my Dockerfile, but I am not 100% happy with it.
> >
> > <<<<<<
> > FROM docker.io/ubuntu:24.04
> > # Install necessary build tools and dependencies
> > RUN apt-get update && apt-get install -y libmpich-dev gfortran
> > libomp-dev (...)
> > (...)
> >
> > ## NVIDIA HPC SDK
> > RUN wget
> >
> https://urldefense.us/v3/__https://developer.download.nvidia.com/hpc-sdk/25.5/nvhpc_2025_255_Linux_x86_64_cuda_12.9.tar.gz__;!!G_uCfscf7eWS!ft2AAfoKdKf-j-6E6nTqtzXHqQEmnbsH7as3hylOIpz7OiONASpD-qNuRNcnDapAFok1YeTP2lLQfH7ne1cFSw$
> > &&
> > tar xpzfv nvhpc_2025_255_Linux_x86_64_cuda_12.9.tar.gz &&
> > nvhpc_2025_255_Linux_x86_64_cuda_12.9/install --silent
> --accept-license
> > --instal-dir /opt/nvidia/hpc_sdk
> >
> > RUN cd petsc-3.23.5 &&\
> > ./configure --prefix=/usr/local/petsc --PETSC_ARCH=arch-linux-c-opt
> > --with-debugging=0 \
> > --with-mpi --download-hypre --with-blaslapack --with-cuda
> > --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/25.5/cuda && \
> > make all install && \
> > cd ..
> >
> > (...)
> > ENV
> PATH=${NVHPC_ROOT}/compilers/bin:${NVHPC_ROOT}/comm_libs/mpi/bin:${PATH}
> > ENV MANPATH=${NVHPC_ROOT}/compilers/man:${MANPATH}
> > ENV LD_LIBRARY_PATH=${NVHPC_ROOT}/compilers/lib:${LD_LIBRARY_PATH}
> > >>>>>>
> >
> > Renato
> >
> > On Sat, Aug 2, 2025 at 11:11 AM Satish Balay <balay....@fastmail.org>
> wrote:
> >
> > > Please send configure.log
> > >
> > > presumably you are using a pre-installed mpi from hpc-sdk - that uses
> NV
> > > compilers. [and PETSc configure is defaulting to mpicc/mpif90 in PATH]
> > >
> > > If you do not have MPI with gcc/gfortran - you can do something like:
> > >
> > > ./configure --with-cuda=1 --with-cc=gcc --with-cxx=g++
> --with-fc=gfortran
> > > --download-mpich [other options]
> > >
> > > When no options are specified - configure guesses - and picks up what
> it
> > > can find - and sometimes that might not be what you intend to use..
> > >
> > > Satish
> > >
> > > On Sat, 2 Aug 2025, Renato E Poli wrote:
> > >
> > > > It seems that the installer is the one choosing to use nvfortran
> instead
> > > of
> > > > gfortran.
> > > >
> > > > I am building a docker container (clean environment).
> > > > I simply install CUDA SDK from
> > > >
> > >
> https://urldefense.us/v3/__https://developer.download.nvidia.com/hpc-sdk/25.5/nvhpc_2025_255_Linux_x86_64_cuda_12.9.tar.gz__;!!G_uCfscf7eWS!cBTymZpXcpFZ7BHpCGxn0ooq_STD1f9CioiFgRypIYNWg9lpwnmLpddrWk0-J-wQoeTw9Ipu2FPm037mQg_C2w$
> > > > , set paths and call petsc configure.
> > > >
> > > > RUN cd petsc-3.23.5 &&\
> > > > ./configure --prefix=/usr/local/petsc --with-debugging=0
> --with-mpi
> > > > --download-hdf5 --download-hypre --with-blaslapack --with-cuda
> && \
> > > > make all && make install
> > > >
> > > > Do you see any reason for using the wrong compiler?
> > > >
> > > > Renato
> > > >
> > > > On Sat, Aug 2, 2025 at 12:28 AM Satish Balay <balay....@fastmail.org
> >
> > > wrote:
> > > >
> > > > > NVFORTRAN (based on flang-17) does not support some F2008 features
> that
> > > > > are required here (flang-19+ should work)
> > > > >
> > > > > Suggest using gfortran instead.
> > > > >
> > > > > Satish
> > > > >
> > > > > On Fri, 1 Aug 2025, Renato E Poli wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I see the error below when compiling with --with-cuda.
> > > > > >
> > > > > > Any suggestion?
> > > > > >
> > > > > >
> > > > > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/inode2.o
> > > > > > CC
> > > arch-linux-c-opt/obj/src/mat/impls/aij/seq/aijperm/aijperm.o
> > > > > > CC
> > > > > arch-linux-c-opt/obj/src/mat/impls/aij/seq/mattransposematmult.o
> > > > > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/matptap.o
> > > > > > CC arch-linux-c-opt/obj/src/mat/impls/aij/seq/matrart.o
> > > > > > FC arch-linux-c-opt/obj/src/sys/ftn-mod/petscsysmod.o
> > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > >
> > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> 6)
> > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > >
> > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > > 18)
> > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > >
> > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > > 30)
> > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > >
> > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > > 42)
> > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > >
> > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > > 54)
> > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > >
> > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > > 65)
> > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > >
> > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > > 76)
> > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > >
> > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > > 88)
> > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > >
> > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > > 100)
> > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > >
> > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > > 112)
> > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > >
> > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > > 124)
> > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > >
> > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > > 136)
> > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > >
> > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > > 148)
> > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > >
> > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscsys.h:
> > > > > 160)
> > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > >
> > > > >
> > >
> (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscviewer.h: 6)
> > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > >
> > >
> (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscviewer.h:
> > > > > > 17)
> > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > >
> > > > >
> > >
> (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h:
> > > > > > 6)
> > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > >
> > > > >
> > >
> (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h:
> > > > > > 18)
> > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > >
> > > > >
> > >
> (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscoptions.h:
> > > > > > 30)
> > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > >
> > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h:
> 5)
> > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > >
> > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h:
> > > > > 17)
> > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > >
> > > (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petsclog.h:
> > > > > 29)
> > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > >
> > > > >
> > >
> (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscdevice.h: 6)
> > > > > > NVFORTRAN-S-0034-Syntax error at or near end of line
> > > > > >
> > >
> (/opt/src/petsc-3.23.5/arch-linux-c-opt/include/../ftn/sys/petscdevice.h:
> > > > > > 17)
> > > > > > 0 inform, 0 warnings, 24 severes, 0 fatal for petscsysdef
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
