Mark, I can reproduce this error with PrgEnv-cray, i.e., using the Cray compiler (clang-11). Previously I used PrgEnv-gnu, which did not have this error. Probably it is a problem of Spock. But I am not sure.
--Junchao Zhang On Sat, Jul 17, 2021 at 10:17 AM Mark Adams <mfad...@lbl.gov> wrote: > And I can run a fortran test, with warnings, but C tests fail: > > 11:15 jczhang/fix-cray-mpicxx-includes/main= > /gpfs/alpine/csc314/scratch/adams/petsc$ make > PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc > PETSC_ARCH=arch-spock-opt-cray-kokkos -f gmakefile test > search='ts_utils_dmplexlandau_tutorials-ex1f90_0' > Using MAKEFLAGS: -- search=ts_utils_dmplexlandau_tutorials-ex1f90_0 > PETSC_ARCH=arch-spock-opt-cray-kokkos > PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc > FC > arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o > FLINKER > arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90 > /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: > warning: alignment 128 of symbol > `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in > /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in > arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o > /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: > warning: alignment 64 of symbol `$data_init$iso_c_binding_' in > /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in > arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o > TEST > arch-spock-opt-cray-kokkos/tests/counts/ts_utils_dmplexlandau_tutorials-ex1f90_0.counts > ok ts_utils_dmplexlandau_tutorials-ex1f90_0 > ok diff-ts_utils_dmplexlandau_tutorials-ex1f90_0 > > > On Sat, Jul 17, 2021 at 10:53 AM Mark Adams <mfad...@lbl.gov> wrote: > >> HUmm, I can not reproduce this. >> >> 10:49 jczhang/fix-cray-mpicxx-includes/main= >> /gpfs/alpine/csc314/scratch/adams/petsc$ make >> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >> PETSC_ARCH=arch-spock-dbg-kokkos check >> Running check examples to verify correct installation >> Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and >> PETSC_ARCH=arch-spock-dbg-kokkos >> gmake[3]: >> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: >> ex19.PETSc] Error 2 (ignored) >> *******************Error detected during compile or >> link!******************* >> See http://www.mcs.anl.gov/petsc/documentation/faq.html >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex19 >> >> ********************************************************************************* >> cc -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 >> -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas >> -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O0 -fPIC >> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas >> -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O0 >> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >> -I/gpfs/alpine/csc314/scratch/adams/petsc/include >> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include >> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex19.c >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >> -lquadmath -lstdc++ -ldl -o ex19 >> ld.lld: error: >> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa >> [--no-allow-shlib-undefined] >> ld.lld: error: >> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >> undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa >> [--no-allow-shlib-undefined] >> ld.lld: error: >> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >> undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa >> [--no-allow-shlib-undefined] >> clang-11: error: linker command failed with exit code 1 (use -v to see >> invocation) >> gmake[4]: *** [<builtin>: ex19] Error 1 >> gmake[3]: >> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: >> ex3k.PETSc] Error 2 (ignored) >> *******************Error detected during compile or >> link!******************* >> See http://www.mcs.anl.gov/petsc/documentation/faq.html >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex3k >> >> ********************************************************************************* >> hipcc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing >> -Wno-unknown-pragmas -fvisibility=hidden -g -O0 -std=c++14 >> -I/gpfs/alpine/csc314/scratch/adams/petsc/include >> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include >> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >> -I/sw/spock/spack-envs/views/rocm-4.1.0/include -I/opt/cray/pe/libsci/ >> 21.04.1.1/CRAY/9.0/x86_64/include >> -I/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/include >> -I/opt/cray/pe/pmi/6.0.10/include -I/opt/cray/pe/dsmml/0.1.4/dsmml//include >> -I/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/include >> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/pmi/6.0.10/lib >> -L/opt/cray/pe/dsmml/0.1.4/dsmml//lib >> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib/pkgconfig/../ >> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> -Wl,--as-needed,-lsci_cray_mpi,--no-as-needed >> -Wl,--as-needed,-lsci_cray,--no-as-needed >> -Wl,--as-needed,-lmpi_cray,--no-as-needed >> -Wl,--as-needed,-ldsmml,--no-as-needed >> -Wl,--as-needed,-lpgas-shmem,--no-as-needed -lpmi -ldl -lmpi_gtl_hsa >> -lxpmem -lquadmath -lmodules -lfi -lcraymath -lf -lu -lcsup >> ex3k.kokkos.cxx >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >> -lquadmath -lstdc++ -ldl -o ex3k >> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: >> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >> undefined reference to `.omp_offloading.img_size.cray_amdgcn-amd-amdhsa' >> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: >> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >> undefined reference to `.omp_offloading.img_start.cray_amdgcn-amd-amdhsa' >> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: >> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >> undefined reference to `.omp_offloading.img_cache.cray_amdgcn-amd-amdhsa' >> clang-12: error: linker command failed with exit code 1 (use -v to see >> invocation) >> gmake[4]: *** >> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/test:26: ex3k] >> Error 1 >> gmake[3]: [makefile:102: runex3k_kokkos] Error 2 (ignored) >> 1,25c1,4 >> < atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000 >> < Vec Object: Exact Solution 2 MPI processes >> < type: mpikokkos >> < Process [0] >> < 0. >> < 0.015625 >> < 0.125 >> < Process [1] >> < 0.421875 >> < 1. >> < Vec Object: Forcing function 2 MPI processes >> < type: mpikokkos >> < Process [0] >> < 1e-72 >> < 1.50024 >> < 3.01563 >> < Process [1] >> < 4.67798 >> < 7. >> < 0 SNES Function norm 5.414682427127e+00 >> < 1 SNES Function norm 2.952582418265e-01 >> < 2 SNES Function norm 4.502293658739e-04 >> < 3 SNES Function norm 1.389665806646e-09 >> < Number of SNES iterations = 3 >> < Norm of error 1.49752e-10 Iterations 3 >> --- >> > slurmstepd: error: execve(): >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such >> file or directory >> > slurmstepd: error: execve(): >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such >> file or directory >> > srun: error: spock01: tasks 0-1: Exited with exit code 2 >> > srun: launch/slurm: _step_signal: Terminating StepId=272979.0 >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials >> Possible problem with ex3k running with kokkos-kernels, diffs above >> ========================================= >> *******************Error detected during compile or >> link!******************* >> See http://www.mcs.anl.gov/petsc/documentation/faq.html >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex5f >> ********************************************************* >> ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 >> -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0 >> -I/gpfs/alpine/csc314/scratch/adams/petsc/include >> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include >> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex5f.F90 >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >> -lquadmath -lstdc++ -ldl -o ex5f >> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: >> warning: alignment 128 of symbol >> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in >> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in >> /tmp/pe_30429/ex5f_1.o >> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: >> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in >> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in >> /tmp/pe_30429/ex5f_1.o >> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI >> process >> Completed test examples >> 10:49 jczhang/fix-cray-mpicxx-includes/main= >> /gpfs/alpine/csc314/scratch/adams/petsc$ ll >> >> On Fri, Jul 16, 2021 at 11:28 PM Junchao Zhang <junchao.zh...@gmail.com> >> wrote: >> >>> Mark, I configured with Fortran binding enabled with main + MR !4175 >>> <https://gitlab.com/petsc/petsc/-/merge_requests/4175> >>> >>> '--with-mpiexec=srun', >>> '--with-shared-libraries=1', >>> '--with-cc=cc', >>> '--with-cxx=CC', >>> '--with-fc=ftn', >>> '--with-fortran-bindings', >>> '--with-hip', >>> '--with-hipc=hipcc', >>> '--with-debugging', >>> '--CPPFLAGS=-I${ROCM_PATH}/include', >>> '--CXXPPFLAGS=-I${ROCM_PATH}/include', >>> '--CC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', >>> '--CXX_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', >>> '--FC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', >>> '--COPTFLAGS=-g -O0', >>> '--CXXOPTFLAGS=-g -O0', >>> '--FOPTFLAGS=-g -O0', >>> '--download-kokkos', >>> '--download-kokkos-kernels', >>> '--download-kokkos-commit=3.4.01', >>> '--download-kokkos-kernels-commit=3.4.01', >>> '--with-kokkos-hip-arch=VEGA908', >>> >>> and 'make check' ran smoothly on a compute node >>> >>> $ make check >>> Running check examples to verify correct installation >>> Using PETSC_DIR=/ccs/home/jczhang/petsc and >>> PETSC_ARCH=arch-spock-cray-kokkos-dbg >>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process >>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI >>> processes >>> C/C++ example src/snes/tutorials/ex3k run successfully with >>> kokkos-kernels >>> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI >>> process >>> >>> Completed test examples >>> >>> >>> --Junchao Zhang >>> >>> >>> On Fri, Jul 16, 2021 at 6:04 PM Mark Adams <mfad...@lbl.gov> wrote: >>> >>>> And I find that this error, on non-Kokkos C tests, is fixed by turning >>>> the fortran bindings off: >>>> >>>> ld.lld: error: >>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: >>>> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa >>>> [--no-allow-shlib-undefined] >>>> ld.lld: error: >>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: >>>> undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa >>>> [--no-allow-shlib-undefined] >>>> ld.lld: error: >>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: >>>> undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa >>>> [--no-allow-shlib-undefined] >>>> >>>> On Fri, Jul 16, 2021 at 3:53 PM Mark Adams <mfad...@lbl.gov> wrote: >>>> >>>>> Not complex. THis has some overlap with my problem w/o Kokkos. >>>>> >>>>> On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang < >>>>> junchao.zh...@gmail.com> wrote: >>>>> >>>>>> Do you use complex? post your configure.log. >>>>>> >>>>>> --Junchao Zhang >>>>>> >>>>>> >>>>>> On Fri, Jul 16, 2021 at 9:47 AM Mark Adams <mfad...@lbl.gov> wrote: >>>>>> >>>>>>> The simple Kokkos example is failing for me on Spock. >>>>>>> Any ideas? >>>>>>> Thanks, >>>>>>> >>>>>>> 10:44 main *= >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make >>>>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >>>>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k >>>>>>> MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC >>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64 >>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 >>>>>>> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas >>>>>>> -fstack-protector -fvisibility=hidden -g -O2 -fPIC -Wall >>>>>>> -Wwrite-strings >>>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3 >>>>>>> -std=c++14 -I/gpfs/alpine/csc314/scratch/adams/petsc/include >>>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include >>>>>>> -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include >>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex3k.kokkos.cxx >>>>>>> >>>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib >>>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib >>>>>>> -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib >>>>>>> -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib >>>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >>>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >>>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib >>>>>>> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >>>>>>> -L/opt/cray/pe/pmi/6.0.10/lib >>>>>>> -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >>>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >>>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >>>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >>>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >>>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >>>>>>> -lquadmath -lstdc++ -ldl -o ex3k >>>>>>> In file included from ex3k.kokkos.cxx:3: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>>>>> In file included from >>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>>>>> In file included from >>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11: >>>>>>> error: no member named 'lock_address_host_space' in namespace >>>>>>> 'Kokkos::Impl::Kokkos::Impl'; did you mean simply >>>>>>> 'lock_address_host_space'? >>>>>>> while (!Impl::lock_address_host_space((void*)dest)) >>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>> lock_address_host_space >>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6: >>>>>>> note: 'lock_address_host_space' declared here >>>>>>> bool lock_address_host_space(void* ptr); >>>>>>> ^ >>>>>>> In file included from ex3k.kokkos.cxx:3: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>>>>> In file included from >>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>>>>> In file included from >>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3: >>>>>>> error: no member named 'memory_fence' in namespace >>>>>>> 'Kokkos::Impl::Kokkos'; >>>>>>> did you mean simply 'memory_fence'? >>>>>>> Kokkos::memory_fence(); >>>>>>> ^~~~~~~~~~~~~~~~~~~~ >>>>>>> memory_fence >>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: >>>>>>> note: 'memory_fence' declared here >>>>>>> void memory_fence() { >>>>>>> ^ >>>>>>> In file included from ex3k.kokkos.cxx:3: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>>>>> In file included from >>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>>>>> In file included from >>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3: >>>>>>> error: no member named 'memory_fence' in namespace >>>>>>> 'Kokkos::Impl::Kokkos'; >>>>>>> did you mean simply 'memory_fence'? >>>>>>> Kokkos::memory_fence(); >>>>>>> ^~~~~~~~~~~~~~~~~~~~ >>>>>>> memory_fence >>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: >>>>>>> note: 'memory_fence' declared here >>>>>>> void memory_fence() { >>>>>>> ^ >>>>>>> >>>>>>
