Glad it worked. The plan is the next release will not require these hacks.
> On Aug 26, 2022, at 12:13 PM, byron <[email protected]> wrote: > > It looks like I've (we've) cracked it, at least the petsc stage of what I'm > trying to do. To get the configure to work I used the following > > python3 ./configure --download-hypre=1 --with-shared-libraries=1 > --download-hdf5=1 --download-hdf5-fortran-bindings=0 --with-debugging=no > --download-fblaslapack=1 --download-metis=1 --download-ptscotch=1 > --download-parmetis=1 --download-superlu_dist=1 --download-mumps=1 > --download-strumpack=1 --download-scalapack=1 --download-slepc=1 --with-mpi=1 > --with-openmp=1 --with-cxx-dialect=C++11 -with-fortran-bindings=0 > --with-sowing=0 --with-64-bit-indices --with-cuda -CUDAOPTFLAGS="" > --with-cuda-dialect=cxx11 > --prefix=/home/lgoodman/Projects/petscExamples/main-build-CUDAOPTFLAGS-zero-cxx-dialect-unspc-cuda-dial-11/installation > > Where -CUDAOPTFLAGS="" was the solutions to getting rid of the duplicate -O3 > > then I was getting errors about 'redefinition of argument 'std'" so i added > > --with-cxx-dialect=C++11 > and > --with-cuda-dialect=cxx11 > > and that got rid of those errors. I image there are cleaner / more elegant > ways of doing this but I'll settle for this for now. > > Thanks a lot for the help guys, much appreciated > > Best regards > Byron > > On Fri, Aug 26, 2022 at 4:06 PM Barry Smith <[email protected] > <mailto:[email protected]>> wrote: > > Yes, PETSc sends down a single -O3 to SuperLU_DIST CMake but internally > SuperLU_DIST CMake is adding its own set of nvcc compiler flags to use and > these that get added to the one's PETSc passes in (they do not replace them). > A little further down in the file it displays a compile line with nvcc that > has two sets of -O3 (as well as two sets of other things as well). > > I think we changed the main branch PETSc to not pass down the optimization > flags to the CMake CUDA to prevent the problem from happening. In other > words, we let the package decide its optimization flags instead of trying to > tell it what to use. > > Barry > > The root of the problem is that compiler APIs are a bunch of strings passed > on the command line to the compiler without any structure. > > >> On Aug 26, 2022, at 9:47 AM, byron <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hi Barry >> >> I dont think I see the same thing. >> >> When I do a build using the release branch petsc passes the following >> arguements to superlu_dist (taken from >> /home/blahblah/Projects/220826-vanilla-petsc-release-branch/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/superlu_dist.petscconf) >> which has only one -O3 entry >> >> -DCMAKE_INSTALL_PREFIX=/home/blahblah/Projects/220826-vanilla-petsc-release-branch/installation >> >> -DCMAKE_INSTALL_NAME_DIR:STRING="/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib" >> -DCMAKE_INSTALL_LIBDIR:STRING="lib" -DCMAKE_VERBOSE_MAKEFILE=1 >> -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER="mpicc" >> -DMPI_C_COMPILER="/usr/bin/mpicc" -DCMAKE_AR=/usr/bin/ar >> -DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_C_FLAGS:STRING="-fPIC >> -Wno-lto-type-mismatch -fstack-protector -g -O -fopenmp -DDEBUGlevel=0 >> -DPRNTlevel=0" -DCMAKE_C_FLAGS_DEBUG:STRING="-fPIC -Wno-lto-type-mismatch >> -fstack-protector -g -O -fopenmp -DDEBUGlevel=0 -DPRNTlevel=0" >> -DCMAKE_C_FLAGS_RELEASE:STRING="-fPIC -Wno-lto-type-mismatch >> -fstack-protector -g -O -fopenmp -DDEBUGlevel=0 -DPRNTlevel=0" >> -DCMAKE_CXX_COMPILER="mpicxx" -DMPI_CXX_COMPILER="/usr/bin/mpicxx" >> -DCMAKE_CXX_FLAGS:STRING="-Wno-lto-type-mismatch -fstack-protector -g -O >> -fopenmp -std=c++11 -fPIC -fopenmp -DDEBUGlevel=0 -DPRNTlevel=0" >> -DCMAKE_CXX_FLAGS_DEBUG:STRING="-Wno-lto-type-mismatch -fstack-protector -g >> -O -fopenmp -std=c++11 -fPIC -fopenmp -DDEBUGlevel=0 -DPRNTlevel=0" >> -DCMAKE_CXX_FLAGS_RELEASE:STRING="-Wno-lto-type-mismatch -fstack-protector >> -g -O -fopenmp -std=c++11 -fPIC -fopenmp -DDEBUGlevel=0 -DPRNTlevel=0" >> -DCMAKE_Fortran_COMPILER="mpif90" -DMPI_Fortran_COMPILER="/usr/bin/mpif90" >> -DCMAKE_Fortran_FLAGS:STRING="-fPIC -ffree-line-length-0 >> -Wno-lto-type-mismatch -g -O -fopenmp" >> -DCMAKE_Fortran_FLAGS_DEBUG:STRING="-fPIC -ffree-line-length-0 >> -Wno-lto-type-mismatch -g -O -fopenmp" >> -DCMAKE_Fortran_FLAGS_RELEASE:STRING="-fPIC -ffree-line-length-0 >> -Wno-lto-type-mismatch -g -O -fopenmp" -DCMAKE_EXE_LINKER_FLAGS:STRING=" >> -fopenmp -fopenmp" -DBUILD_SHARED_LIBS:BOOL=ON -DTPL_ENABLE_CUDALIB=TRUE >> -DTPL_CUDA_LIBRARIES="-lcudart -lcufft -lcublas -lcusparse -lcusolver >> -lcurand -lcuda" -DCUDA_ARCH_FLAGS="-arch=sm_75" >> -DCMAKE_CUDA_COMPILER="nvcc" -DCMAKE_CUDA_FLAGS=" -ccbin mpicxx -std=c++14 >> -Xcompiler -fPIC -O3 -gencode arch=compute_75,code=sm_75 >> -Wno-deprecated-gpu-targets >> -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi >> -I/usr/lib/x86_64-linux-gnu/openmpi/include -DDEBUGlevel=0 -DPRNTlevel=0" >> -DUSE_XSDK_DEFAULTS=YES >> -DTPL_BLAS_LIBRARIES="-Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib >> >> -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib >> -lflapack >> -Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib >> >> -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib >> -lfblas -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda -lm >> -lstdc++ -ldl -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib >> -L/usr/lib/x86_64-linux-gnu/openmpi/lib -lmpi_usempif08 >> -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/9 -L/usr/lib/gcc/x86_64-linux-gnu/9 >> -lgfortran -lm -lgcc_s -lquadmath -lpthread" >> -DTPL_LAPACK_LIBRARIES="-Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib >> >> -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib >> -lflapack >> -Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib >> >> -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib >> -lfblas -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda -lm >> -lstdc++ -ldl -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib >> -L/usr/lib/x86_64-linux-gnu/openmpi/lib -lmpi_usempif08 >> -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm >> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/9 -L/usr/lib/gcc/x86_64-linux-gnu/9 >> -lgfortran -lm -lgcc_s -lquadmath -lpthread" >> -DTPL_PARMETIS_INCLUDE_DIRS="/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/include;/usr/lib/x86_64-linux-gnu/openmpi/include;/usr/lib/x86_64-linux-gnu/openmpi/lib" >> >> -DTPL_PARMETIS_LIBRARIES="-Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib >> >> -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib >> -lparmetis >> -Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib >> >> -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib >> -lmetis -lm -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand >> -lcuda" -DXSDK_INDEX_SIZE=64 -DXSDK_ENABLE_Fortran=ON -Denable_tests=0 >> -Denable_examples=0 -DMPI_C_COMPILE_FLAGS:STRING="" >> -DMPI_C_INCLUDE_PATH:STRING="" -DMPI_C_HEADER_DIR:STRING="" >> -DMPI_C_LIBRARIES:STRING="" >> >> Or am I looking at the wrong thing? >> >> I also tried doing the build downloading petsc from the main branch but that >> died at the same point and looked to give the same "redefinition of argument >> 'optimize'" errors >> >> Thanks >> Byron >> >> >> >> On Thu, Aug 25, 2022 at 4:15 PM Jacob Faibussowitsch <[email protected] >> <mailto:[email protected]>> wrote: >> > Sadly each one makes different -std=c++11 vs -std=c++14 choices as well. >> >> Main should hopefully *also* warn about this, but unless you really really >> want -std=c++11 or -std=c++14, it is better to leave out this configure flag >> (--with-cxx-dialect). >> >> If --with-cxx-dialect is unspecified (or set to “auto”) configure will pick >> the most appropriate flag for you, taking your package environment into >> account when choosing. If you specify the flag configure must assume the >> user knows something it doesn’t and will blindly take what is given. >> >> Best regards, >> >> Jacob Faibussowitsch >> (Jacob Fai - booss - oh - vitch) >> >> > On Aug 25, 2022, at 11:05, Barry Smith <[email protected] >> > <mailto:[email protected]>> wrote: >> > >> > >> > Oh yes, my mistake I misinterpreted the error message. It is actually >> > the nvcc compiler complaining about multiple command line arguments >> > related to optimization (which it is really picky about even if the two >> > arguments are identical.) I reformatted some of the configure output to >> > show this >> > >> > -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3 -gencode >> > arch=compute_75,code=sm_75 -Wno-deprecated-gpu-targets >> > -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi >> > -I/usr/lib/x86_64-linux-gnu/openmpi/include >> > -DDEBUGlevel=0 -DPRNTlevel=0 >> > >> > the next fragment is >> > >> > -O3 -DNDEBUG >> > >> > --generate-code=arch=compute_75,code=[compute_75,sm_75] >> > >> > -Xcompiler=-fPIC -std=c++11 -x cu -rdc=true -c >> > /home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/SRC/superlu_gpu_utils.cu >> > <http://superlu_gpu_utils.cu/> -o >> > >> > Hence the -O3 appears twice, once from the CMAKE argument PETSc passes to >> > superlu_dist >> > >> > -DCMAKE_CUDA_FLAGS=" -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3 >> > -gencode arch=compute_75,code=sm_75 -Wno-deprecated-gpu-targets >> > -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi >> > -I/usr/lib/x86_64-linux-gnu/openmpi/include -DDEBUGlevel=0 -DPRNTlevel=0" >> > >> > and then presumably, the superlu_dist cmake process adds its own arguments >> > which overlap. >> > >> > If I remember correctly this is fixed in the main branch of PETSc, could >> > you do an identical build with the main branch? If that works we will just >> > need to figure out how to get MOOSE to work with main PETSc branch. >> > >> > Barry >> > >> > Sadly each one makes different -std=c++11 vs -std=c++14 choices as well. >> > >> > >> > -------------------- >> > >> > [ 15%] Building CUDA object >> > SRC/CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu.o >> > cd >> > /home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/petsc-build/SRC >> > && /usr/bin/nvcc >> > >> > -DSUPERLU_DIST_EXPORTS -Dsuperlu_dist_EXPORTS >> > >> > -I/home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/petsc-build/SRC >> > >> > -I/home/blahblah/Projects/220824-petsc-build/installation/include >> > -I/usr/lib/x86_64-linux-gnu/openmpi/lib >> > >> > -DUSE_VENDOR_BLAS -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3 -gencode >> > arch=compute_75,code=sm_75 -Wno-deprecated-gpu-targets >> > >> > -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi >> > -I/usr/lib/x86_64-linux-gnu/openmpi/include >> > >> > -DDEBUGlevel=0 -DPRNTlevel=0 -O3 -DNDEBUG >> > >> > --generate-code=arch=compute_75,code=[compute_75,sm_75] >> > >> > -Xcompiler=-fPIC -std=c++11 -x cu -rdc=true -c >> > /home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/SRC/superlu_gpu_utils.cu >> > <http://superlu_gpu_utils.cu/> -o >> > >> > CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu >> > <http://superlu_gpu_utils.cu/> >> > >> > >> > >> >> On Aug 25, 2022, at 9:38 AM, byron <[email protected] >> >> <mailto:[email protected]>> wrote: >> >> >> >> Hi Barry >> >> >> >> There doesn't appear to be any reference to 'optimize' in the only copy >> >> of superlu_gpu_utils.cu <http://superlu_gpu_utils.cu/> I could find, >> >> I've attached a copy. >> >> >> >> Thanks >> >> >> >> >> >> >> >> On Thu, Aug 25, 2022 at 1:44 PM Barry Smith <[email protected] >> >> <mailto:[email protected]>> wrote: >> >> >> >> nvcc fatal : redefinition of argument 'optimize' >> >> make[2]: *** [SRC/CMakeFiles/superlu_dist.dir/build.make:499: >> >> SRC/CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu.o] Error 1 >> >> make[2]: *** Waiting for unfinished jobs.... >> >> nvcc fatal : redefinition of argument 'optimize' >> >> make[2]: *** [SRC/CMakeFiles/superlu_dist-static.dir/build.make:499: >> >> SRC/CMakeFiles/superlu_dist-static.dir/superlu_gpu_utils.cu.o] Error 1 >> >> make[2]: *** Waiting for unfinished jobs.... >> >> >> >> It seems that nvcc is unhappy with something in superlu_gpu_utils.cu >> >> <http://superlu_gpu_utils.cu/> Could you check that file for any >> >> "redefinition of optimize"? Unfortunately the nvcc compiler isn't kind >> >> enough to print the line number of the redefinition or that information >> >> is lost somehow in the build process. >> >> >> >> Barry >> >> >> >>> On Aug 25, 2022, at 6:50 AM, byron <[email protected] >> >>> <mailto:[email protected]>> wrote: >> >>> >> >>> Hi All >> >>> >> >>> I am having trouble getting petsc to build with cuda / superlu-dist >> >>> enabled. I have been trying to build it as part of another software >> >>> tool, moose, hence the choice of options being passed to configure >> >>> >> >>> python3 ./configure --download-hypre=1 \ >> >>> --with-shared-libraries=1 \ >> >>> --download-hdf5=1 \ >> >>> --download-hdf5-fortran-bindings=0 \ >> >>> --with-debugging=no \ >> >>> --download-fblaslapack=1 \ >> >>> --download-metis=1 \ >> >>> --download-ptscotch=1 \ >> >>> --download-parmetis=1 \ >> >>> --download-superlu_dist=1 \ >> >>> --download-mumps=1 \ >> >>> --download-strumpack=1 \ >> >>> --download-scalapack=1 \ >> >>> --download-slepc=1 \ >> >>> --with-mpi=1 \ >> >>> --with-openmp=1 \ >> >>> --with-cxx-dialect=C++11 \ >> >>> --with-fortran-bindings=0 \ >> >>> --with-sowing=0 \ >> >>> --with-64-bit-indices \ >> >>> --with-cuda \ >> >>> --prefix=/home/blah/blah >> >>> >> >>> it fails with >> >>> >> >>> ============================================================================================= >> >>> >> >>> Installing PTScotch; this may take >> >>> several minutes >> >>> >> >>> >> >>> ============================================================================================= >> >>> >> >>> >> >>> ============================================================================================= >> >>> >> >>> Trying to download >> >>> git://https://github.com/xiaoyeli/superlu_dist >> >>> <https://github.com/xiaoyeli/superlu_dist> for SUPERLU_DIST >> >>> >> >>> >> >>> ============================================================================================= >> >>> >> >>> >> >>> ============================================================================================= >> >>> >> >>> Configuring SUPERLU_DIST with cmake; >> >>> this may take several minutes >> >>> >> >>> >> >>> ============================================================================================= >> >>> >> >>> >> >>> ============================================================================================= >> >>> >> >>> Compiling and installing >> >>> SUPERLU_DIST; this may take several minutes >> >>> >> >>> >> >>> ============================================================================================= >> >>> >> >>> >> >>> ******************************************************************************* >> >>> UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log >> >>> for details): >> >>> ------------------------------------------------------------------------------- >> >>> Error running make on SUPERLU_DIST >> >>> ******************************************************************************* >> >>> >> >>> I've attached the configure.log. >> >>> >> >>> I've tried a few different things but nothing that resulted in a >> >>> successfull build. The most promising was I tried to build superlu_dist >> >>> separately with the configure options that petsc passes it and then kept >> >>> trying the build over and over droping out different configure options >> >>> until I got it to work. In that case removing a "-std=c++14" entry >> >>> appeared to fix the problem but then when I made the same change in how >> >>> petsc builds superlu-dist the make failed on superlu-dist with a >> >>> different error. I get the feeling I'm dealing with symptoms of the >> >>> problem rather than the cause. >> >>> >> >>> Thanks >> >>> Byron >> >>> >> >>> >> >>> >> >>> >> >>> <configure.log> >> >> >> >> <superlu_gpu_utils.cu <http://superlu_gpu_utils.cu/>> >> > >> >
