The following is working for me on the compute node [its completed kokkos (core) build - and is now building kokkos-kernels
~/petsc.save/arch-arm/lib/petsc/conf/reconfigure-arch-arm.py --download-p4est --download-zlib --download-kokkos --download-kokkos-kernels --download-kokkos-commit=origin/develop --download-cmake=https://github.com/Kitware/CMake/releases/download/v3.20.1/cmake-3.20.1.tar.gz -DKokkos_ENABLE_OPENMP=ON give me some issue - so I didn't try any of the additional options.. Satish On Wed, 14 Apr 2021, Mark Adams wrote: > Satish, I get this error. > > I wonder if this syntax is wrong: > > '--download-kokkos-cmake-arguments=-DCMAKE_BUILD_TYPE=Release > -DBUILD_TESTING=OFF -DKokkos_ENABLE_LIBDL=OFF -DKokkos_ENABLE_OPENMP=ON > -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON', > > > ============================================================================================= > Configuring KOKKOS with cmake; this may take > several minutes > > ============================================================================================= > > Executing: /usr/bin/cmake .. > -DCMAKE_INSTALL_PREFIX=/home/ra010009/a04199/petsc/arch-arm > -DCMAKE_INSTALL_NAME_DIR:STRING="/home/ra010009/a04199/petsc/arch-arm/lib" > -DCMAKE_INSTALL_LIBDIR:STRING="lib" -DCMAKE_VERBOSE_MAKEFILE=1 > -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER="mpifcc" > -DMPI_C_COMPILER="mpifcc" -DCMAKE_AR=/usr/bin/ar > -DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_C_FLAGS:STRING="-fPIC -Kfast,openmp > -fopenmp" -DCMAKE_C_FLAGS_DEBUG:STRING="-fPIC -Kfast,openmp -fopenmp" > -DCMAKE_C_FLAGS_RELEASE:STRING="-fPIC -Kfast,openmp -fopenmp" > -DCMAKE_CXX_COMPILER="mpiFCC" -DMPI_CXX_COMPILER="mpiFCC" > -DCMAKE_CXX_FLAGS:STRING="-Kfast,openmp -fopenmp -fPIC -fopenmp" > -DCMAKE_CXX_FLAGS_DEBUG:STRING="-Kfast,openmp -fopenmp -fPIC -fopenmp" > -DCMAKE_CXX_FLAGS_RELEASE:STRING="-Kfast,openmp -fopenmp -fPIC -fopenmp" > -DCMAKE_Fortran_COMPILER="mpifrt" -DMPI_Fortran_COMPILER="mpifrt" > -DCMAKE_Fortran_FLAGS:STRING="-fPIC -O -fopenmp" > -DCMAKE_Fortran_FLAGS_DEBUG:STRING="-fPIC -O -fopenmp" > -DCMAKE_Fortran_FLAGS_RELEASE:STRING="-fPIC -O -fopenmp" > -DCMAKE_EXE_LINKER_FLAGS:STRING=" -fopenmp" -DBUILD_SHARED_LIBS:BOOL=ON > -DUSE_XSDK_DEFAULTS=YES -DXSDK_ENABLE_DEBUG=NO > -DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=ON > -DCMAKE_BUILD_WITH_INSTALL_RPATH:BOOL=ON -DKokkos_ENABLE_MPI=ON > -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_OPENMP=ON > -DCMAKE_CXX_STANDARD="14" -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=OFF > -DKokkos_ENABLE_LIBDL=OFF -DKokkos_ENABLE_OPENMP=ON > -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON > stdout: > -- Setting default Kokkos CXX standard to 14 > -- The CXX compiler identification is Fujitsu > -- Check for working CXX compiler: /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC > -- Check for working CXX compiler: > /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC -- works > -- Detecting CXX compiler ABI info > -- Detecting CXX compiler ABI info - done > -- The project name is: Kokkos > -- Configuring incomplete, errors occurred! > See also > "/vol0004/ra010009/a04199/petsc/arch-arm/externalpackages/git.kokkos/petsc-build/CMakeFiles/CMakeOutput.log". > Error configuring KOKKOS with cmake Could not execute > "['/usr/bin/cmake .. > -DCMAKE_INSTALL_PREFIX=/home/ra010009/a04199/petsc/arch-arm > -DCMAKE_INSTALL_NAME_DIR:STRING="/home/ra010009/a04199/petsc/arch-arm/lib" > -DCMAKE_INSTALL_LIBDIR:STRING="lib" -DCMAKE_VERBOSE_MAKEFILE=1 > -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER="mpifcc" > -DMPI_C_COMPILER="mpifcc" -DCMAKE_AR=/usr/bin/ar > -DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_C_FLAGS:STRING="-fPIC -Kfast,openmp > -fopenmp" -DCMAKE_C_FLAGS_DEBUG:STRING="-fPIC -Kfast,openmp -fopenmp" > -DCMAKE_C_FLAGS_RELEASE:STRING="-fPIC -Kfast,openmp -fopenmp" > -DCMAKE_CXX_COMPILER="mpiFCC" -DMPI_CXX_COMPILER="mpiFCC" > -DCMAKE_CXX_FLAGS:STRING="-Kfast,openmp -fopenmp -fPIC -fopenmp" > -DCMAKE_CXX_FLAGS_DEBUG:STRING="-Kfast,openmp -fopenmp -fPIC -fopenmp" > -DCMAKE_CXX_FLAGS_RELEASE:STRING="-Kfast,openmp -fopenmp -fPIC -fopenmp" > -DCMAKE_Fortran_COMPILER="mpifrt" -DMPI_Fortran_COMPILER="mpifrt" > -DCMAKE_Fortran_FLAGS:STRING="-fPIC -O -fopenmp" > -DCMAKE_Fortran_FLAGS_DEBUG:STRING="-fPIC -O -fopenmp" > -DCMAKE_Fortran_FLAGS_RELEASE:STRING="-fPIC -O -fopenmp" > -DCMAKE_EXE_LINKER_FLAGS:STRING=" -fopenmp" -DBUILD_SHARED_LIBS:BOOL=ON > -DUSE_XSDK_DEFAULTS=YES -DXSDK_ENABLE_DEBUG=NO > -DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=ON > -DCMAKE_BUILD_WITH_INSTALL_RPATH:BOOL=ON -DKokkos_ENABLE_MPI=ON > -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_OPENMP=ON > -DCMAKE_CXX_STANDARD="14" -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=OFF > -DKokkos_ENABLE_LIBDL=OFF -DKokkos_ENABLE_OPENMP=ON > -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON']": > -- Setting default Kokkos CXX standard to 14 > -- The CXX compiler identification is Fujitsu > -- Check for working CXX compiler: /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC > -- Check for working CXX compiler: > /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC -- works > -- Detecting CXX compiler ABI info > -- Detecting CXX compiler ABI info - done > -- The project name is: Kokkos > -- Configuring incomplete, errors occurred! > See also > "/vol0004/ra010009/a04199/petsc/arch-arm/externalpackages/git.kokkos/petsc-build/CMakeFiles/CMakeOutput.log".CMake > Error at cmake/kokkos_compiler_id.cmake:129 (STRING): > > * STRING sub-command REPLACE requires at least four arguments.*Call Stack > (most recent call first): > cmake/kokkos_tribits.cmake:174 (INCLUDE) > CMakeLists.txt:166 (KOKKOS_SETUP_BUILD_ENVIRONMENT) > > On Wed, Apr 14, 2021 at 3:34 PM Satish Balay <[email protected]> wrote: > > > Additional kokkos cmake arguments can be passed in via > > --download-kokkos-cmake-arguments=string option. > > > > Satish > > > > On Wed, 14 Apr 2021, Mark Adams wrote: > > > > > Satish, > > > > > > For the fujitsu compiler OMP is -Kopenmp. > > > > > > Sarat (cc'ed) tells me that he built Kokkos with: > > > > > > cmake -DCMAKE_BUILD_TYPE=Release \ > > > -DCMAKE_INSTALL_PREFIX=${KOKKOS_SRC_DIR}/install \ > > > -DBUILD_TESTING=OFF \ > > > -DKokkos_ENABLE_LIBDL=OFF \ > > > -DKokkos_ENABLE_OPENMP=ON \ > > > -DKokkos_ENABLE_SERIAL=ON \ > > > -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON \ > > > .. > > > > > > How might I make this happen in PETSc? > > > > > > Thanks, > > > Mark > > > > > > > > > On Wed, Apr 14, 2021 at 2:44 PM Satish Balay <[email protected]> wrote: > > > > > > > On Wed, 14 Apr 2021, Mark Adams wrote: > > > > > > > > > I have this building now. > > > > > Do you know anything about OpenMP? > > > > > I can add --with-openmp > > > > > That should get Kokkos to be made with OpenMP. > > > > > Should PETSc deal with the compilers correctly? > > > > > > > > Well it tries the following compiler options for openmp. > > > > > > > > oflags = ["-fopenmp", # Gnu > > > > "-qsmp=omp",# IBM XL C/C++ > > > > "-h omp", # Cray. Must come after XL because XL > > interprets > > > > this option as meaning "-soname omp" > > > > "-mp", # Portland Group > > > > "-Qopenmp", # Intel windows > > > > "-openmp", # Intel > > > > "-xopenmp", # Sun > > > > "+Oopenmp", # HP > > > > "/openmp" # Microsoft Visual Studio > > > > ] > > > > > > > > I don't know what the flag for fugaku compiler is. > > > > > > > > Satish > > > > > > > > > > > > > Thanks, > > > > > Mark > > > > > > > > > > On Wed, Apr 14, 2021 at 1:45 PM Mark Adams <[email protected]> wrote: > > > > > > > > > > > Thanks, > > > > > > If you feel inspired you could try Kokkos :|| > > > > > > I am in a parking lot waiting for my daughter but can try this > > when I > > > > get > > > > > > home, > > > > > > Thanks again, > > > > > > Mark > > > > > > > > > > > > On Wed, Apr 14, 2021 at 1:33 PM Satish Balay <[email protected]> > > > > wrote: > > > > > > > > > > > >> I think I allocated a single node - and did the build on it. > > > > > >> > > > > > >> Now I'm getting an error - don't know what changed.. > > > > > >> > > > > > >> login6$ pjsub --interact -L "node=1" -L "rscunit=rscunit_ft01" -L > > > > > >> "rscgrp=eap-int" -L "elapse=1:00:00" --sparam "wait-time=600" > > > > > >> [ERR.] PJM 0059 pjsub rscgrp=eap-int is disabled. > > > > > >> > > > > > >> Ok - the following worked.. > > > > > >> > > > > > >> login6$ pjsub --interact -L "node=1" -L "rscunit=rscunit_ft01" -L > > > > > >> "elapse=1:00:00" --sparam "wait-time=600" > > > > > >> [INFO] PJM 0000 pjsub Job 6301572 submitted. > > > > > >> [INFO] PJM 0081 .connected. > > > > > >> [INFO] PJM 0082 pjsub Interactive job 6301572 started. > > > > > >> [a04201@j31-3110s petsc]$ > > > > > >> > > > > > >> Ok - trying this build now. > > > > > >> > > > > > >> [a04201@j31-3110s petsc]$ cat > > > > > >> ~/petsc.save/arch-arm/lib/petsc/conf/reconfigure-arch-arm.py > > > > > >> #!/usr/bin/python3 > > > > > >> if __name__ == '__main__': > > > > > >> import sys > > > > > >> import os > > > > > >> sys.path.insert(0, os.path.abspath('config')) > > > > > >> import configure > > > > > >> configure_options = [ > > > > > >> '--with-blaslapack-lib=-lfjlapack', > > > > > >> '--with-debugging=0', > > > > > >> 'CC=mpifcc', > > > > > >> 'CXX=mpiFCC', > > > > > >> 'FC=mpifrt', > > > > > >> 'PETSC_ARCH=arch-arm', > > > > > >> ] > > > > > >> configure.petsc_configure(configure_options) > > > > > >> [a04201@j31-3110s petsc]$ > > > > > >> ~/petsc.save/arch-arm/lib/petsc/conf/reconfigure-arch-arm.py > > > > > >> --download-p4est --download-zlib > > > > > >> <snip> > > > > > >> p4est: > > > > > >> Includes: -I/vol0004/ra010009/a04201/petsc/arch-arm/include > > > > > >> Library: -Wl,-rpath,/vol0004/ra010009/a04201/petsc/arch-arm/lib > > > > > >> -L/vol0004/ra010009/a04201/petsc/arch-arm/lib -lp4est -lsc > > > > > >> > > > > > >> Ok - this worked for me. > > > > > >> > > > > > >> Satish > > > > > >> > > > > > >> On Wed, 14 Apr 2021, Mark Adams wrote: > > > > > >> > > > > > >> > Do you recall what nodes you use to build on a "compute" node, > > to > > > > avoid > > > > > >> > cross compilation? > > > > > >> > > > > > > >> > On Wed, Apr 14, 2021 at 12:08 PM Satish Balay < > > [email protected]> > > > > > >> wrote: > > > > > >> > > > > > > >> > > looks like p4est cannot be cross-compiled. > > > > > >> > > > > > > > >> > > Satish > > > > > >> > > > > > > > >> > > On Wed, 14 Apr 2021, Mark Adams wrote: > > > > > >> > > > > > > > >> > > > I get this error with p4est on Fugaku. > > > > > >> > > > It is a Fortran error. Odd. > > > > > >> > > > Mark > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > > > > >
