Hi Jacob I tried with "--with-cxx-dialect=auto" which did look to clean up all the references to c++14 / c++11 from the options used to configure superlu-dist but it still didnt work and looked to generate the same "nvcc fatal" errors.
Thanks Byron On Fri, Aug 26, 2022 at 2:47 PM byron <[email protected]> wrote: > Hi Barry > > I dont think I see the same thing. > > When I do a build using the release branch petsc passes the following > arguements to superlu_dist (taken from > /home/blahblah/Projects/220826-vanilla-petsc-release-branch/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/superlu_dist.petscconf) > which has only one -O3 entry > > -DCMAKE_INSTALL_PREFIX=/home/blahblah/Projects/220826-vanilla-petsc-release-branch/installation > -DCMAKE_INSTALL_NAME_DIR:STRING="/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib" > -DCMAKE_INSTALL_LIBDIR:STRING="lib" -DCMAKE_VERBOSE_MAKEFILE=1 > -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER="mpicc" > -DMPI_C_COMPILER="/usr/bin/mpicc" -DCMAKE_AR=/usr/bin/ar > -DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_C_FLAGS:STRING="-fPIC > -Wno-lto-type-mismatch -fstack-protector -g -O -fopenmp -DDEBUGlevel=0 > -DPRNTlevel=0" -DCMAKE_C_FLAGS_DEBUG:STRING="-fPIC -Wno-lto-type-mismatch > -fstack-protector -g -O -fopenmp -DDEBUGlevel=0 -DPRNTlevel=0" > -DCMAKE_C_FLAGS_RELEASE:STRING="-fPIC -Wno-lto-type-mismatch > -fstack-protector -g -O -fopenmp -DDEBUGlevel=0 -DPRNTlevel=0" > -DCMAKE_CXX_COMPILER="mpicxx" -DMPI_CXX_COMPILER="/usr/bin/mpicxx" > -DCMAKE_CXX_FLAGS:STRING="-Wno-lto-type-mismatch -fstack-protector -g -O > -fopenmp -std=c++11 -fPIC -fopenmp -DDEBUGlevel=0 -DPRNTlevel=0" > -DCMAKE_CXX_FLAGS_DEBUG:STRING="-Wno-lto-type-mismatch -fstack-protector -g > -O -fopenmp -std=c++11 -fPIC -fopenmp -DDEBUGlevel=0 -DPRNTlevel=0" > -DCMAKE_CXX_FLAGS_RELEASE:STRING="-Wno-lto-type-mismatch -fstack-protector > -g -O -fopenmp -std=c++11 -fPIC -fopenmp -DDEBUGlevel=0 -DPRNTlevel=0" > -DCMAKE_Fortran_COMPILER="mpif90" -DMPI_Fortran_COMPILER="/usr/bin/mpif90" > -DCMAKE_Fortran_FLAGS:STRING="-fPIC -ffree-line-length-0 > -Wno-lto-type-mismatch -g -O -fopenmp" > -DCMAKE_Fortran_FLAGS_DEBUG:STRING="-fPIC -ffree-line-length-0 > -Wno-lto-type-mismatch -g -O -fopenmp" > -DCMAKE_Fortran_FLAGS_RELEASE:STRING="-fPIC -ffree-line-length-0 > -Wno-lto-type-mismatch -g -O -fopenmp" -DCMAKE_EXE_LINKER_FLAGS:STRING=" > -fopenmp -fopenmp" -DBUILD_SHARED_LIBS:BOOL=ON -DTPL_ENABLE_CUDALIB=TRUE > -DTPL_CUDA_LIBRARIES="-lcudart -lcufft -lcublas -lcusparse -lcusolver > -lcurand -lcuda" -DCUDA_ARCH_FLAGS="-arch=sm_75" > -DCMAKE_CUDA_COMPILER="nvcc" -DCMAKE_CUDA_FLAGS=" -ccbin mpicxx -std=c++14 > -Xcompiler -fPIC *-O3 *-gencode arch=compute_75,code=sm_75 > -Wno-deprecated-gpu-targets > -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi > -I/usr/lib/x86_64-linux-gnu/openmpi/include -DDEBUGlevel=0 -DPRNTlevel=0" > -DUSE_XSDK_DEFAULTS=YES > -DTPL_BLAS_LIBRARIES="-Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib > -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib > -lflapack > -Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib > -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib > -lfblas -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda -lm > -lstdc++ -ldl -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib > -L/usr/lib/x86_64-linux-gnu/openmpi/lib -lmpi_usempif08 > -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm > -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/9 > -L/usr/lib/gcc/x86_64-linux-gnu/9 -lgfortran -lm -lgcc_s -lquadmath > -lpthread" > -DTPL_LAPACK_LIBRARIES="-Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib > -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib > -lflapack > -Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib > -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib > -lfblas -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda -lm > -lstdc++ -ldl -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib > -L/usr/lib/x86_64-linux-gnu/openmpi/lib -lmpi_usempif08 > -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm > -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/9 > -L/usr/lib/gcc/x86_64-linux-gnu/9 -lgfortran -lm -lgcc_s -lquadmath > -lpthread" > -DTPL_PARMETIS_INCLUDE_DIRS="/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/include;/usr/lib/x86_64-linux-gnu/openmpi/include;/usr/lib/x86_64-linux-gnu/openmpi/lib" > -DTPL_PARMETIS_LIBRARIES="-Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib > -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib > -lparmetis > -Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib > -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib > -lmetis -lm -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand > -lcuda" -DXSDK_INDEX_SIZE=64 -DXSDK_ENABLE_Fortran=ON -Denable_tests=0 > -Denable_examples=0 -DMPI_C_COMPILE_FLAGS:STRING="" > -DMPI_C_INCLUDE_PATH:STRING="" -DMPI_C_HEADER_DIR:STRING="" > -DMPI_C_LIBRARIES:STRING="" > > Or am I looking at the wrong thing? > > I also tried doing the build downloading petsc from the main branch but > that died at the same point and looked to give the same "redefinition of > argument 'optimize'" errors > > Thanks > Byron > > > > On Thu, Aug 25, 2022 at 4:15 PM Jacob Faibussowitsch <[email protected]> > wrote: > >> > Sadly each one makes different -std=c++11 vs -std=c++14 choices as >> well. >> >> Main should hopefully *also* warn about this, but unless you really >> really want -std=c++11 or -std=c++14, it is better to leave out this >> configure flag (--with-cxx-dialect). >> >> If --with-cxx-dialect is unspecified (or set to “auto”) configure will >> pick the most appropriate flag for you, taking your package environment >> into account when choosing. If you specify the flag configure must assume >> the user knows something it doesn’t and will blindly take what is given. >> >> Best regards, >> >> Jacob Faibussowitsch >> (Jacob Fai - booss - oh - vitch) >> >> > On Aug 25, 2022, at 11:05, Barry Smith <[email protected]> wrote: >> > >> > >> > Oh yes, my mistake I misinterpreted the error message. It is actually >> the nvcc compiler complaining about multiple command line arguments related >> to optimization (which it is really picky about even if the two arguments >> are identical.) I reformatted some of the configure output to show this >> > >> > -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3 -gencode >> arch=compute_75,code=sm_75 -Wno-deprecated-gpu-targets >> > -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi >> -I/usr/lib/x86_64-linux-gnu/openmpi/include >> > -DDEBUGlevel=0 -DPRNTlevel=0 >> > >> > the next fragment is >> > >> > -O3 -DNDEBUG >> > >> > --generate-code=arch=compute_75,code=[compute_75,sm_75] >> > >> > -Xcompiler=-fPIC -std=c++11 -x cu -rdc=true -c >> /home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/SRC/ >> superlu_gpu_utils.cu -o >> > >> > Hence the -O3 appears twice, once from the CMAKE argument PETSc passes >> to superlu_dist >> > >> > -DCMAKE_CUDA_FLAGS=" -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3 >> -gencode arch=compute_75,code=sm_75 -Wno-deprecated-gpu-targets >> -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi >> -I/usr/lib/x86_64-linux-gnu/openmpi/include -DDEBUGlevel=0 -DPRNTlevel=0" >> > >> > and then presumably, the superlu_dist cmake process adds its own >> arguments which overlap. >> > >> > If I remember correctly this is fixed in the main branch of PETSc, >> could you do an identical build with the main branch? If that works we will >> just need to figure out how to get MOOSE to work with main PETSc branch. >> > >> > Barry >> > >> > Sadly each one makes different -std=c++11 vs -std=c++14 choices as >> well. >> > >> > >> > -------------------- >> > >> > [ 15%] Building CUDA object >> SRC/CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu.o >> > cd >> /home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/petsc-build/SRC >> && /usr/bin/nvcc >> > >> > -DSUPERLU_DIST_EXPORTS -Dsuperlu_dist_EXPORTS >> > >> > >> -I/home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/petsc-build/SRC >> > >> > -I/home/blahblah/Projects/220824-petsc-build/installation/include >> -I/usr/lib/x86_64-linux-gnu/openmpi/lib >> > >> > -DUSE_VENDOR_BLAS -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3 >> -gencode arch=compute_75,code=sm_75 -Wno-deprecated-gpu-targets >> > >> > -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi >> -I/usr/lib/x86_64-linux-gnu/openmpi/include >> > >> > -DDEBUGlevel=0 -DPRNTlevel=0 -O3 -DNDEBUG >> > >> > --generate-code=arch=compute_75,code=[compute_75,sm_75] >> > >> > -Xcompiler=-fPIC -std=c++11 -x cu -rdc=true -c >> /home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/SRC/ >> superlu_gpu_utils.cu -o >> > >> > CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu >> > >> > >> > >> >> On Aug 25, 2022, at 9:38 AM, byron <[email protected]> wrote: >> >> >> >> Hi Barry >> >> >> >> There doesn't appear to be any reference to 'optimize' in the only >> copy of superlu_gpu_utils.cu I could find, I've attached a copy. >> >> >> >> Thanks >> >> >> >> >> >> >> >> On Thu, Aug 25, 2022 at 1:44 PM Barry Smith <[email protected]> wrote: >> >> >> >> nvcc fatal : redefinition of argument 'optimize' >> >> make[2]: *** [SRC/CMakeFiles/superlu_dist.dir/build.make:499: >> SRC/CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu.o] Error 1 >> >> make[2]: *** Waiting for unfinished jobs.... >> >> nvcc fatal : redefinition of argument 'optimize' >> >> make[2]: *** [SRC/CMakeFiles/superlu_dist-static.dir/build.make:499: >> SRC/CMakeFiles/superlu_dist-static.dir/superlu_gpu_utils.cu.o] Error 1 >> >> make[2]: *** Waiting for unfinished jobs.... >> >> >> >> It seems that nvcc is unhappy with something in superlu_gpu_utils.cu >> Could you check that file for any "redefinition of optimize"? Unfortunately >> the nvcc compiler isn't kind enough to print the line number of the >> redefinition or that information is lost somehow in the build process. >> >> >> >> Barry >> >> >> >>> On Aug 25, 2022, at 6:50 AM, byron <[email protected]> wrote: >> >>> >> >>> Hi All >> >>> >> >>> I am having trouble getting petsc to build with cuda / superlu-dist >> enabled. I have been trying to build it as part of another software tool, >> moose, hence the choice of options being passed to configure >> >>> >> >>> python3 ./configure --download-hypre=1 \ >> >>> --with-shared-libraries=1 \ >> >>> --download-hdf5=1 \ >> >>> --download-hdf5-fortran-bindings=0 \ >> >>> --with-debugging=no \ >> >>> --download-fblaslapack=1 \ >> >>> --download-metis=1 \ >> >>> --download-ptscotch=1 \ >> >>> --download-parmetis=1 \ >> >>> --download-superlu_dist=1 \ >> >>> --download-mumps=1 \ >> >>> --download-strumpack=1 \ >> >>> --download-scalapack=1 \ >> >>> --download-slepc=1 \ >> >>> --with-mpi=1 \ >> >>> --with-openmp=1 \ >> >>> --with-cxx-dialect=C++11 \ >> >>> --with-fortran-bindings=0 \ >> >>> --with-sowing=0 \ >> >>> --with-64-bit-indices \ >> >>> --with-cuda \ >> >>> --prefix=/home/blah/blah >> >>> >> >>> it fails with >> >>> >> >>> >> ============================================================================================= >> >> Installing PTScotch; this may take several >> minutes >> >> ============================================================================================= >> >> >> >> ============================================================================================= >> >> Trying to download git:// >> https://github.com/xiaoyeli/superlu_dist for SUPERLU_DIST >> >> >> ============================================================================================= >> >> >> >> ============================================================================================= >> >> Configuring SUPERLU_DIST with cmake; this >> may take several minutes >> >> ============================================================================================= >> >> >> >> ============================================================================================= >> >> Compiling and installing SUPERLU_DIST; this >> may take several minutes >> >> ============================================================================================= >> >> >> >> ******************************************************************************* >> >>> UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log >> for details): >> >>> >> ------------------------------------------------------------------------------- >> >>> Error running make on SUPERLU_DIST >> >>> >> ******************************************************************************* >> >>> >> >>> I've attached the configure.log. >> >>> >> >>> I've tried a few different things but nothing that resulted in a >> successfull build. The most promising was I tried to build superlu_dist >> separately with the configure options that petsc passes it and then kept >> trying the build over and over droping out different configure options >> until I got it to work. In that case removing a "-std=c++14" entry >> appeared to fix the problem but then when I made the same change in how >> petsc builds superlu-dist the make failed on superlu-dist with a different >> error. I get the feeling I'm dealing with symptoms of the problem rather >> than the cause. >> >>> >> >>> Thanks >> >>> Byron >> >>> >> >>> >> >>> >> >>> >> >>> <configure.log> >> >> >> >> <superlu_gpu_utils.cu> >> > >> >>
