Thanks, however, CUDAFLAGS doesn't work. Even I removed all -fopenmp string from the make file, it still complains nvcc doesn't know -fopenmp. It seems by enabling --with-openmp, there is a background -fopenmp option added without -Xcompiler to any compiler.
On Wed, Oct 25, 2023 at 11:44 AM Satish Balay <[email protected]> wrote: > I guess the flag you are looking for is CUDAFLAGS > > >>> > balay@petsc-gpu-01:/scratch/balay/petsc/src/vec/vec/tests$ make ex100 > CUDAFLAGS="-Xcompiler -fopenmp" LDFLAGS=-fopenmp > /usr/local/cuda/bin/nvcc -o ex100.o -c > -I/nfs/gce/projects/petsc/soft/u22.04/mpich-4.0.2/include -ccbin mpicxx > -std=c++17 -Xcompiler -fPIC -Xcompiler -fvisibility=hidden -g -lineinfo > -gencode arch=compute_86,code=sm_86 -Xcompiler -fopenmp > -I/scratch/balay/petsc/include > -I/scratch/balay/petsc/arch-linux-c-debug/include > -I/usr/local/cuda/include `pwd`/ex100.cu > mpicc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas > -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector > -fvisibility=hidden -g3 -O0 -fopenmp -Wl,-export-dynamic ex100.o > -Wl,-rpath,/scratch/balay/petsc/arch-linux-c-debug/lib > -L/scratch/balay/petsc/arch-linux-c-debug/lib > -Wl,-rpath,/usr/local/cuda/lib64 -L/usr/local/cuda/lib64 > -L/usr/local/cuda/lib64/stubs > -Wl,-rpath,/nfs/gce/projects/petsc/soft/u22.04/mpich-4.0.2/lib > -L/nfs/gce/projects/petsc/soft/u22.04/mpich-4.0.2/lib > -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/11 > -L/usr/lib/gcc/x86_64-linux-gnu/11 -lpetsc -llapack -lblas -lm -lcudart > -lnvToolsExt -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda -lX11 > -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++ > -lquadmath -o ex100 > rm ex100.o > balay@petsc-gpu-01:/scratch/balay/petsc/src/vec/vec/tests$ > <<< > > Satish > > On Wed, 25 Oct 2023, Qiyue Lu wrote: > > > Even with > > CXXFLAGS=-Xcompiler -fopenmp -std=c++17 > > LDFLAGS= -Xcompiler -fopenmp > > CXXPPFLAGS=-I/u/qiyuelu1/cuda/cuda-samples/Common > > include ${PETSC_DIR}/lib/petsc/conf/variables > > include ${PETSC_DIR}/lib/petsc/conf/rules > > > > won't work. > > > > On Wed, Oct 25, 2023 at 11:06 AM Qiyue Lu <[email protected]> wrote: > > > > > Thanks for your reply, using this configurations: > > > > > > *--with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > > > --download-f2cblaslapack=1 --with-cudac=nvcc --with-cuda=1 > --with-openmp=1 > > > --with-threadsafety=1* > > > However, I got an error like: > > > *nvcc fatal : Unknown option '-fopenmp'* > > > Previously, when I don't have --with-openmp for the configuration, the > > > PETSc make system can build my *.cu code using nvcc and g++, of course, > > > OpenMP doesn't work. Now with this --with-openmp option, it cannot even > > > build. The interesting thing is, I got this error even after removing > the > > > *-fopenmp* from *CXXFLAGS* contents: > > > CXXFLAGS=-std=c++17 > > > LDFLAGS= > > > CXXPPFLAGS=-I/u/qiyuelu1/cuda/cuda-samples/Common > > > include ${PETSC_DIR}/lib/petsc/conf/variables > > > include ${PETSC_DIR}/lib/petsc/conf/rules > > > > > > > > > > > > Thanks, > > > Qiyue Lu > > > > > > On Wed, Oct 25, 2023 at 10:54 AM Satish Balay <[email protected]> > wrote: > > > > > >> > > >> On Wed, 25 Oct 2023, Qiyue Lu wrote: > > >> > > >> > Hello, > > >> > I have an in-house code enabled OpenMP and it works. Now I am > trying to > > >> > incorporate PETSc as the linear solver and build together using the > > >> > building rules in $PETSC_HOME/lib/petsc/conf/rules. However, I > found the > > >> > OpenMP part doesn't work anymore. > > >> > > >> If you are looking at building only your sources with openmp - using > > >> petsc formatted makefile [using petsc build rules], > > >> you can specify it via CFLAGS - either in makefile - or on command > line. > > >> > > >> >>>>>>> > > >> For ex: [this example is using src/ksp/ksp/tutorials/makefile - with > the > > >> corresponding make fules] > > >> > > >> [balay@pj01 tutorials]$ make ex2 > > >> mpicc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas > > >> -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector > > >> -fvisibility=hidden -g3 -O0 -I/home/balay/petsc/include > > >> -I/home/balay/petsc/arch-linux-c-debug/include -Wl,-export-dynamic > > >> ex2.c -Wl,-rpath,/home/balay/petsc/arch-linux-c-debug/lib > > >> -L/home/balay/petsc/arch-linux-c-debug/lib > > >> -Wl,-rpath,/software/mpich-4.1.1/lib -L/software/mpich-4.1.1/lib > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/13 > > >> -L/usr/lib/gcc/x86_64-redhat-linux/13 -lpetsc -llapack -lblas -lm > -lX11 > > >> -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath > -lstdc++ > > >> -lquadmath -o ex2 > > >> [balay@pj01 tutorials]$ make clean > > >> [balay@pj01 tutorials]$ make ex2 CFLAGS=-fopenmp > > >> mpicc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas > > >> -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector > > >> -fvisibility=hidden -g3 -O0 -fopenmp -I/home/balay/petsc/include > > >> -I/home/balay/petsc/arch-linux-c-debug/include -Wl,-export-dynamic > > >> ex2.c -Wl,-rpath,/home/balay/petsc/arch-linux-c-debug/lib > > >> -L/home/balay/petsc/arch-linux-c-debug/lib > > >> -Wl,-rpath,/software/mpich-4.1.1/lib -L/software/mpich-4.1.1/lib > > >> -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/13 > > >> -L/usr/lib/gcc/x86_64-redhat-linux/13 -lpetsc -llapack -lblas -lm > -lX11 > > >> -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath > -lstdc++ > > >> -lquadmath -o ex2 > > >> [balay@pj01 tutorials]$ > > >> <<<<< > > >> > > >> Satish > > >> > > >> > > >> > Should I re-configure the petsc installation with --with-openmp=1 > > >> option? I > > >> > wonder are the building rules affected by this missing option? > > >> > > > >> > Thanks, > > >> > Qiyue Lu > > >> > > > >> > > >> > > >
