On Mon, Jul 19, 2021 at 6:41 PM Scott Kruger <kru...@txcorp.com> wrote:
> > Mark, > > On tulip, things with fortran went wonky when a `-fast` flag snuck into > the flags (snuck in being me copying a previous configure file and not > questioning the flags). The reason is that for clang, `-fast` implies > link time optimization (lto) for C/C++ code, but flang doesn't support > lto so things got weird. I suspect that gfortran does not either (but > perhaps the real question is why not use flang?) > It does work with GNU. I am doing this for an application and I want to avoid dictating the program env. I'm (re)trying with -O0 just to check. > > I look at configure and I don't see anything in your flags that would > trigger lto, but as Junchao says, it might be picking something up from > Kokkos, so perhaps this is the issue. > > The configure files that I sent did not have Kokkos. They were stripped down. I have asked ORNL, but don't expect help. The person (Matt) that is helping me for other things is very helpful so perhaps he can come up with some ideas. Otherwise I will tell the app that they need to use GNU for now. Thanks, Mark > Scott > > > On 2021-07-19 07:19, Mark Adams did write: > > Thanks, but this happens w/o Kokkos. > > I've stripped this down and attached good/bad logs without/with Fortran > > bindings. > > Hope this helps, > > Thanks again, > > Mark > > > > On Sun, Jul 18, 2021 at 12:00 PM Stefano Zampini < > stefano.zamp...@gmail.com> > > wrote: > > > > > This is probably kokkos pulling in the dependencies where compiling > kokkos > > > source within PETSc. > > > > > > Il Dom 18 Lug 2021, 16:29 Mark Adams <mfad...@lbl.gov> ha scritto: > > > > > >> Whoops, this error was just from not telling gfortran to allow long > lines. > > >> > > >> Anway, I did find that the when fortran bindings are enabled this OMP > > >> thing gets into the library. No idea how. I'll askm ORNL. > > >> > > >> 10:26 jczhang/fix-cray-mpicxx-includes/main= > > >> /gpfs/alpine/csc314/scratch/adams/petsc2$ nm > > >> > /gpfs/alpine/csc314/scratch/adams/petsc2/arch-spock-opt-cray-kokkos/lib/libpetsc.so > > >> |g offload > > >> U .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa > > >> U .omp_offloading.img_size.cray_amdgcn-amd-amdhsa > > >> U .omp_offloading.img_start.cray_amdgcn-amd-amdhsa > > >> 0000000001d457b0 T vecgetoffloadmask_ > > >> > > >> > > >> On Sun, Jul 18, 2021 at 8:43 AM Mark Adams <mfad...@lbl.gov> wrote: > > >> > > >>> Ah, your test was not on Spock ... I have something working but this > is > > >>> strange. > > >>> > > >>> I switched to the GNU ProgEnv. and it passed the Fortran test in > > >>> 'check', but this failed: > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> *08:26 2 jczhang/fix-cray-mpicxx-includes/main= > > >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make > > >>> > PETSC_DIR=/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu > > >>> PETSC_ARCH="" ex40f90ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib > > >>> -lamdhip64 -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0 > > >>> > -I/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/include > > >>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex40f90.F90 > > >>> > -Wl,-rpath,/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib > > >>> > -L/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib > > >>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib > > >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib > > >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib > > >>> -L/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib > > >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib > > >>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib > > >>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib > > >>> <http://21.04.1.1/GNU/9.1/x86_64/lib> > > >>> -L/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib > > >>> <http://21.04.1.1/GNU/9.1/x86_64/lib> > > >>> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib > > >>> -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib > > >>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib > > >>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > > >>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > > >>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0 > > >>> -L/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0 > > >>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib64 -L/opt/gcc/10.2.0/snos/lib64 > > >>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib > > >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib > > >>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib -L/opt/gcc/10.2.0/snos/lib > -lpetsc > > >>> -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 > > >>> -lhsa-runtime64 -lstdc++ -ldl -lpmi -lmpi_gtl_hsa -lxpmem -lgfortran > -lm > > >>> -lsci_gnu_82_mpi -lsci_gnu_82 -lmpifort_gnu_91 -lmpi_gnu_91 -ldsmml > > >>> -lgfortran -lquadmath -lpthread -lm -lgcc_s -lquadmath -lstdc++ -ldl > -o > > >>> ex40f90ex40f90.F90:35:132: 35 | call > > >>> > DMDACreate2d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_BOX,ten,ten,PETSC_DECIDE,PETSC_DECIDE,two,one, > > >>> & | > > >>> > 1Error: > > >>> Line truncated at (1) [-Werror=line-truncation]ex40f90.F90:35:132:* > > >>> * ....* > > >>> > > >>> But ex5f (the Fortran check test) does work manually also (sanity > check). > > >>> > > >>> Oddly I seem to have the cray compilers: > > >>> > > >>> 08:34 jczhang/fix-cray-mpicxx-includes/main= > > >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ which ftn > > >>> /opt/cray/pe/craype/2.7.6/bin/ftn > > >>> > > >>> yet it is GNU: > > >>> > > >>> 08:34 jczhang/fix-cray-mpicxx-includes/main= > > >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ ftn > --version > > >>> GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.) > > >>> Copyright (C) 2020 Free Software Foundation, Inc. > > >>> This is free software; see the source for copying conditions. There > is > > >>> NO > > >>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR > > >>> PURPOSE. > > >>> > > >>> The documentation says ftn should be here: > > >>> > > >>> 08:34 jczhang/fix-cray-mpicxx-includes/main= > > >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ > > >>> $GCC_PATH/bin/gfortran --version > > >>> GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.) > > >>> Copyright (C) 2020 Free Software Foundation, Inc. > > >>> This is free software; see the source for copying conditions. There > is > > >>> NO > > >>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR > > >>> PURPOSE. > > >>> > > >>> Same, but the path is not the same: > > >>> > > >>> 08:39 jczhang/fix-cray-mpicxx-includes/main= > > >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ echo > $GCC_PATH > > >>> /opt/gcc/10.2.0 > > >>> > > >>> Anyway, thanks for your help, > > >>> Mark > > >>> > > >>> On Sat, Jul 17, 2021 at 11:26 PM Junchao Zhang < > junchao.zh...@gmail.com> > > >>> wrote: > > >>> > > >>>> Mark, I can reproduce this error with PrgEnv-cray, i.e., using the > > >>>> Cray compiler (clang-11). Previously I used PrgEnv-gnu, which did > not have > > >>>> this error. > > >>>> Probably it is a problem of Spock. But I am not sure. > > >>>> > > >>>> --Junchao Zhang > > >>>> > > >>>> > > >>>> On Sat, Jul 17, 2021 at 10:17 AM Mark Adams <mfad...@lbl.gov> > wrote: > > >>>> > > >>>>> And I can run a fortran test, with warnings, but C tests fail: > > >>>>> > > >>>>> 11:15 jczhang/fix-cray-mpicxx-includes/main= > > >>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ make > > >>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc > > >>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos -f gmakefile test > > >>>>> search='ts_utils_dmplexlandau_tutorials-ex1f90_0' > > >>>>> Using MAKEFLAGS: -- search=ts_utils_dmplexlandau_tutorials-ex1f90_0 > > >>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos > > >>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc > > >>>>> FC > > >>>>> > arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o > > >>>>> FLINKER > > >>>>> > arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90 > > >>>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: > > >>>>> warning: alignment 128 of symbol > > >>>>> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in > > >>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller > than 256 in > > >>>>> > arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o > > >>>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: > > >>>>> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in > > >>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller > than 256 in > > >>>>> > arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o > > >>>>> TEST > > >>>>> > arch-spock-opt-cray-kokkos/tests/counts/ts_utils_dmplexlandau_tutorials-ex1f90_0.counts > > >>>>> ok ts_utils_dmplexlandau_tutorials-ex1f90_0 > > >>>>> ok diff-ts_utils_dmplexlandau_tutorials-ex1f90_0 > > >>>>> > > >>>>> > > >>>>> On Sat, Jul 17, 2021 at 10:53 AM Mark Adams <mfad...@lbl.gov> > wrote: > > >>>>> > > >>>>>> HUmm, I can not reproduce this. > > >>>>>> > > >>>>>> 10:49 jczhang/fix-cray-mpicxx-includes/main= > > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ make > > >>>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc > > >>>>>> PETSC_ARCH=arch-spock-dbg-kokkos check > > >>>>>> Running check examples to verify correct installation > > >>>>>> Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and > > >>>>>> PETSC_ARCH=arch-spock-dbg-kokkos > > >>>>>> gmake[3]: > > >>>>>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: > > >>>>>> ex19.PETSc] Error 2 (ignored) > > >>>>>> *******************Error detected during compile or > > >>>>>> link!******************* > > >>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html > > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex19 > > >>>>>> > > >>>>>> > ********************************************************************************* > > >>>>>> cc -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 > > >>>>>> -lhsa-runtime64 -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing > > >>>>>> -Wno-unknown-pragmas -fstack-protector -Qunused-arguments > > >>>>>> -fvisibility=hidden -g -O0 -fPIC -Wall -Wwrite-strings > > >>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector > > >>>>>> -Qunused-arguments -fvisibility=hidden -g -O0 > > >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include > > >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include > > >>>>>> > -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include > > >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include > > >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex19.c > > >>>>>> > -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > > >>>>>> > -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > > >>>>>> > -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > > >>>>>> > -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > > >>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib > > >>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib > > >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 > > >>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > > >>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > > >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > > >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > > >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib > > >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib > -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib > > >>>>>> -L/opt/cray/pe/pmi/6.0.10/lib > -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib > > >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib > > >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > > >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > > >>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > > >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > > >>>>>> > -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > > >>>>>> > -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > > >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > > >>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > > >>>>>> > -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > > >>>>>> > -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > > >>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore > -lhipsparse > > >>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 > -lhsa-runtime64 > > >>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray > -lmpi_cray > > >>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath > -lcrayacc_amdgpu > > >>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran > -lpthread > > >>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 > > >>>>>> -lquadmath -lstdc++ -ldl -o ex19 > > >>>>>> ld.lld: error: > > >>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: > > >>>>>> undefined reference to > .omp_offloading.img_start.cray_amdgcn-amd-amdhsa > > >>>>>> [--no-allow-shlib-undefined] > > >>>>>> ld.lld: error: > > >>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: > > >>>>>> undefined reference to > .omp_offloading.img_size.cray_amdgcn-amd-amdhsa > > >>>>>> [--no-allow-shlib-undefined] > > >>>>>> ld.lld: error: > > >>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: > > >>>>>> undefined reference to > .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa > > >>>>>> [--no-allow-shlib-undefined] > > >>>>>> clang-11: error: linker command failed with exit code 1 (use -v to > > >>>>>> see invocation) > > >>>>>> gmake[4]: *** [<builtin>: ex19] Error 1 > > >>>>>> gmake[3]: > > >>>>>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: > > >>>>>> ex3k.PETSc] Error 2 (ignored) > > >>>>>> *******************Error detected during compile or > > >>>>>> link!******************* > > >>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html > > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex3k > > >>>>>> > > >>>>>> > ********************************************************************************* > > >>>>>> hipcc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing > > >>>>>> -Wno-unknown-pragmas -fvisibility=hidden -g -O0 -std=c++14 > > >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include > > >>>>>> > -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include > > >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include > > >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include > -I/opt/cray/pe/libsci/ > > >>>>>> 21.04.1.1/CRAY/9.0/x86_64/include > > >>>>>> -I/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/include > > >>>>>> -I/opt/cray/pe/pmi/6.0.10/include > -I/opt/cray/pe/dsmml/0.1.4/dsmml//include > > >>>>>> -I/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/include > > >>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > > >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > > >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/pmi/6.0.10/lib > > >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml//lib > > >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib/pkgconfig/../ > > >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > > >>>>>> -Wl,--as-needed,-lsci_cray_mpi,--no-as-needed > > >>>>>> -Wl,--as-needed,-lsci_cray,--no-as-needed > > >>>>>> -Wl,--as-needed,-lmpi_cray,--no-as-needed > > >>>>>> -Wl,--as-needed,-ldsmml,--no-as-needed > > >>>>>> -Wl,--as-needed,-lpgas-shmem,--no-as-needed -lpmi -ldl > -lmpi_gtl_hsa > > >>>>>> -lxpmem -lquadmath -lmodules -lfi -lcraymath -lf -lu -lcsup > > >>>>>> ex3k.kokkos.cxx > > >>>>>> > -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > > >>>>>> > -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > > >>>>>> > -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > > >>>>>> > -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > > >>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib > > >>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib > > >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 > > >>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > > >>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > > >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > > >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > > >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib > > >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib > -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib > > >>>>>> -L/opt/cray/pe/pmi/6.0.10/lib > -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib > > >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib > > >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > > >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > > >>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > > >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > > >>>>>> > -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > > >>>>>> > -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > > >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > > >>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > > >>>>>> > -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > > >>>>>> > -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > > >>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore > -lhipsparse > > >>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 > -lhsa-runtime64 > > >>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray > -lmpi_cray > > >>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath > -lcrayacc_amdgpu > > >>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran > -lpthread > > >>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 > > >>>>>> -lquadmath -lstdc++ -ldl -o ex3k > > >>>>>> > /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: > > >>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: > > >>>>>> undefined reference to > `.omp_offloading.img_size.cray_amdgcn-amd-amdhsa' > > >>>>>> > /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: > > >>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: > > >>>>>> undefined reference to > `.omp_offloading.img_start.cray_amdgcn-amd-amdhsa' > > >>>>>> > /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: > > >>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: > > >>>>>> undefined reference to > `.omp_offloading.img_cache.cray_amdgcn-amd-amdhsa' > > >>>>>> clang-12: error: linker command failed with exit code 1 (use -v to > > >>>>>> see invocation) > > >>>>>> gmake[4]: *** > > >>>>>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/test:26: > ex3k] > > >>>>>> Error 1 > > >>>>>> gmake[3]: [makefile:102: runex3k_kokkos] Error 2 (ignored) > > >>>>>> 1,25c1,4 > > >>>>>> < atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000 > > >>>>>> < Vec Object: Exact Solution 2 MPI processes > > >>>>>> < type: mpikokkos > > >>>>>> < Process [0] > > >>>>>> < 0. > > >>>>>> < 0.015625 > > >>>>>> < 0.125 > > >>>>>> < Process [1] > > >>>>>> < 0.421875 > > >>>>>> < 1. > > >>>>>> < Vec Object: Forcing function 2 MPI processes > > >>>>>> < type: mpikokkos > > >>>>>> < Process [0] > > >>>>>> < 1e-72 > > >>>>>> < 1.50024 > > >>>>>> < 3.01563 > > >>>>>> < Process [1] > > >>>>>> < 4.67798 > > >>>>>> < 7. > > >>>>>> < 0 SNES Function norm 5.414682427127e+00 > > >>>>>> < 1 SNES Function norm 2.952582418265e-01 > > >>>>>> < 2 SNES Function norm 4.502293658739e-04 > > >>>>>> < 3 SNES Function norm 1.389665806646e-09 > > >>>>>> < Number of SNES iterations = 3 > > >>>>>> < Norm of error 1.49752e-10 Iterations 3 > > >>>>>> --- > > >>>>>> > slurmstepd: error: execve(): > > >>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such > > >>>>>> file or directory > > >>>>>> > slurmstepd: error: execve(): > > >>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such > > >>>>>> file or directory > > >>>>>> > srun: error: spock01: tasks 0-1: Exited with exit code 2 > > >>>>>> > srun: launch/slurm: _step_signal: Terminating StepId=272979.0 > > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials > > >>>>>> Possible problem with ex3k running with kokkos-kernels, diffs > above > > >>>>>> ========================================= > > >>>>>> *******************Error detected during compile or > > >>>>>> link!******************* > > >>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html > > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex5f > > >>>>>> ********************************************************* > > >>>>>> ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 > > >>>>>> -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0 > > >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include > > >>>>>> > -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include > > >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex5f.F90 > > >>>>>> > -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > > >>>>>> > -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > > >>>>>> > -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > > >>>>>> > -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > > >>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib > > >>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib > > >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 > > >>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > > >>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > > >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > > >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > > >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib > > >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib > -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib > > >>>>>> -L/opt/cray/pe/pmi/6.0.10/lib > -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib > > >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib > > >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > > >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > > >>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > > >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > > >>>>>> > -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > > >>>>>> > -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > > >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > > >>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > > >>>>>> > -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > > >>>>>> > -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > > >>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore > -lhipsparse > > >>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 > -lhsa-runtime64 > > >>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray > -lmpi_cray > > >>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath > -lcrayacc_amdgpu > > >>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran > -lpthread > > >>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 > > >>>>>> -lquadmath -lstdc++ -ldl -o ex5f > > >>>>>> > /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: > > >>>>>> warning: alignment 128 of symbol > > >>>>>> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in > > >>>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller > than 256 in > > >>>>>> /tmp/pe_30429/ex5f_1.o > > >>>>>> > /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: > > >>>>>> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in > > >>>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller > than 256 in > > >>>>>> /tmp/pe_30429/ex5f_1.o > > >>>>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 > MPI > > >>>>>> process > > >>>>>> Completed test examples > > >>>>>> 10:49 jczhang/fix-cray-mpicxx-includes/main= > > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ ll > > >>>>>> > > >>>>>> On Fri, Jul 16, 2021 at 11:28 PM Junchao Zhang < > > >>>>>> junchao.zh...@gmail.com> wrote: > > >>>>>> > > >>>>>>> Mark, I configured with Fortran binding enabled with main + MR > !4175 > > >>>>>>> <https://gitlab.com/petsc/petsc/-/merge_requests/4175> > > >>>>>>> > > >>>>>>> '--with-mpiexec=srun', > > >>>>>>> '--with-shared-libraries=1', > > >>>>>>> '--with-cc=cc', > > >>>>>>> '--with-cxx=CC', > > >>>>>>> '--with-fc=ftn', > > >>>>>>> '--with-fortran-bindings', > > >>>>>>> '--with-hip', > > >>>>>>> '--with-hipc=hipcc', > > >>>>>>> '--with-debugging', > > >>>>>>> '--CPPFLAGS=-I${ROCM_PATH}/include', > > >>>>>>> '--CXXPPFLAGS=-I${ROCM_PATH}/include', > > >>>>>>> '--CC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 > -lhsa-runtime64', > > >>>>>>> '--CXX_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 > -lhsa-runtime64', > > >>>>>>> '--FC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 > -lhsa-runtime64', > > >>>>>>> '--COPTFLAGS=-g -O0', > > >>>>>>> '--CXXOPTFLAGS=-g -O0', > > >>>>>>> '--FOPTFLAGS=-g -O0', > > >>>>>>> '--download-kokkos', > > >>>>>>> '--download-kokkos-kernels', > > >>>>>>> '--download-kokkos-commit=3.4.01', > > >>>>>>> '--download-kokkos-kernels-commit=3.4.01', > > >>>>>>> '--with-kokkos-hip-arch=VEGA908', > > >>>>>>> > > >>>>>>> and 'make check' ran smoothly on a compute node > > >>>>>>> > > >>>>>>> $ make check > > >>>>>>> Running check examples to verify correct installation > > >>>>>>> Using PETSC_DIR=/ccs/home/jczhang/petsc and > > >>>>>>> PETSC_ARCH=arch-spock-cray-kokkos-dbg > > >>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI > > >>>>>>> process > > >>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI > > >>>>>>> processes > > >>>>>>> C/C++ example src/snes/tutorials/ex3k run successfully with > > >>>>>>> kokkos-kernels > > >>>>>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 > MPI > > >>>>>>> process > > >>>>>>> > > >>>>>>> Completed test examples > > >>>>>>> > > >>>>>>> > > >>>>>>> --Junchao Zhang > > >>>>>>> > > >>>>>>> > > >>>>>>> On Fri, Jul 16, 2021 at 6:04 PM Mark Adams <mfad...@lbl.gov> > wrote: > > >>>>>>> > > >>>>>>>> And I find that this error, on non-Kokkos C tests, is fixed by > > >>>>>>>> turning the fortran bindings off: > > >>>>>>>> > > >>>>>>>> ld.lld: error: > > >>>>>>>> > /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: > > >>>>>>>> undefined reference to > .omp_offloading.img_start.cray_amdgcn-amd-amdhsa > > >>>>>>>> [--no-allow-shlib-undefined] > > >>>>>>>> ld.lld: error: > > >>>>>>>> > /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: > > >>>>>>>> undefined reference to > .omp_offloading.img_size.cray_amdgcn-amd-amdhsa > > >>>>>>>> [--no-allow-shlib-undefined] > > >>>>>>>> ld.lld: error: > > >>>>>>>> > /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: > > >>>>>>>> undefined reference to > .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa > > >>>>>>>> [--no-allow-shlib-undefined] > > >>>>>>>> > > >>>>>>>> On Fri, Jul 16, 2021 at 3:53 PM Mark Adams <mfad...@lbl.gov> > wrote: > > >>>>>>>> > > >>>>>>>>> Not complex. THis has some overlap with my problem w/o Kokkos. > > >>>>>>>>> > > >>>>>>>>> On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang < > > >>>>>>>>> junchao.zh...@gmail.com> wrote: > > >>>>>>>>> > > >>>>>>>>>> Do you use complex? post your configure.log. > > >>>>>>>>>> > > >>>>>>>>>> --Junchao Zhang > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> On Fri, Jul 16, 2021 at 9:47 AM Mark Adams <mfad...@lbl.gov> > > >>>>>>>>>> wrote: > > >>>>>>>>>> > > >>>>>>>>>>> The simple Kokkos example is failing for me on Spock. > > >>>>>>>>>>> Any ideas? > > >>>>>>>>>>> Thanks, > > >>>>>>>>>>> > > >>>>>>>>>>> 10:44 main *= > > >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ > make > > >>>>>>>>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc > > >>>>>>>>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k > > >>>>>>>>>>> MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC > > >>>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64 > > >>>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 > -lhsa-runtime64 > > >>>>>>>>>>> -Wall -Wwrite-strings -Wno-strict-aliasing > -Wno-unknown-pragmas > > >>>>>>>>>>> -fstack-protector -fvisibility=hidden -g -O2 -fPIC -Wall > -Wwrite-strings > > >>>>>>>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas > -fvisibility=hidden -g -O3 > > >>>>>>>>>>> -std=c++14 -I/gpfs/alpine/csc314/scratch/adams/petsc/include > > >>>>>>>>>>> > -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include > > >>>>>>>>>>> > -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include > > >>>>>>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include > > >>>>>>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include > ex3k.kokkos.cxx > > >>>>>>>>>>> > -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib > > >>>>>>>>>>> > -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib > > >>>>>>>>>>> > -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib > > >>>>>>>>>>> > -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib > > >>>>>>>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib > > >>>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib > > >>>>>>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 > -L/opt/gcc/8.1.0/snos/lib64 > > >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > > >>>>>>>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > > >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > > >>>>>>>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > > >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib > > >>>>>>>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib > -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib > > >>>>>>>>>>> -L/opt/cray/pe/pmi/6.0.10/lib > -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib > > >>>>>>>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib > > >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > > >>>>>>>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > > >>>>>>>>>>> > -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > > >>>>>>>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > > >>>>>>>>>>> > -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > > >>>>>>>>>>> > -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > > >>>>>>>>>>> > -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > > >>>>>>>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > > >>>>>>>>>>> > -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > > >>>>>>>>>>> > -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > > >>>>>>>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore > -lhipsparse > > >>>>>>>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 > -lhsa-runtime64 > > >>>>>>>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray > -lmpi_cray > > >>>>>>>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath > -lcrayacc_amdgpu > > >>>>>>>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup > -lgfortran -lpthread > > >>>>>>>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 > -lclang_rt.builtins-x86_64 > > >>>>>>>>>>> -lquadmath -lstdc++ -ldl -o ex3k > > >>>>>>>>>>> In file included from ex3k.kokkos.cxx:3: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> > /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> > /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: > > >>>>>>>>>>> > /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11: > > >>>>>>>>>>> error: no member named 'lock_address_host_space' in namespace > > >>>>>>>>>>> 'Kokkos::Impl::Kokkos::Impl'; did you mean simply > 'lock_address_host_space'? > > >>>>>>>>>>> while (!Impl::lock_address_host_space((void*)dest)) > > >>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > >>>>>>>>>>> lock_address_host_space > > >>>>>>>>>>> > /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6: > > >>>>>>>>>>> note: 'lock_address_host_space' declared here > > >>>>>>>>>>> bool lock_address_host_space(void* ptr); > > >>>>>>>>>>> ^ > > >>>>>>>>>>> In file included from ex3k.kokkos.cxx:3: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> > /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> > /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: > > >>>>>>>>>>> > /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3: > > >>>>>>>>>>> error: no member named 'memory_fence' in namespace > 'Kokkos::Impl::Kokkos'; > > >>>>>>>>>>> did you mean simply 'memory_fence'? > > >>>>>>>>>>> Kokkos::memory_fence(); > > >>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~ > > >>>>>>>>>>> memory_fence > > >>>>>>>>>>> > /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: > > >>>>>>>>>>> note: 'memory_fence' declared here > > >>>>>>>>>>> void memory_fence() { > > >>>>>>>>>>> ^ > > >>>>>>>>>>> In file included from ex3k.kokkos.cxx:3: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> > /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> > /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: > > >>>>>>>>>>> > /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3: > > >>>>>>>>>>> error: no member named 'memory_fence' in namespace > 'Kokkos::Impl::Kokkos'; > > >>>>>>>>>>> did you mean simply 'memory_fence'? > > >>>>>>>>>>> Kokkos::memory_fence(); > > >>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~ > > >>>>>>>>>>> memory_fence > > >>>>>>>>>>> > /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: > > >>>>>>>>>>> note: 'memory_fence' declared here > > >>>>>>>>>>> void memory_fence() { > > >>>>>>>>>>> ^ > > >>>>>>>>>>> > > >>>>>>>>>> > > > > > -- > Scott Kruger > Tech-X Corporation kru...@txcorp.com > 5621 Arapahoe Ave, Suite A Phone: (720) 466-3196 > Boulder, CO 80303 Fax: (303) 448-7756 >
