Re: [petsc-dev] Kokkos make error on Spock

2021-07-20 Thread Scott Kruger 
Cray puts the runtime somewhere and then uses their library to put it
in.

I should have mentioned this earlier, but we have
config/examples/arch-olcf-frontier-hip-beta.py

for trying to keep up with Cray's default hip, but I think it's
different than what Junchao does for Kokkos which is why I didn't
mention it earlier.

Scott


On 2021-07-20 11:26, Mark Adams did write:
> OK, it looks like this flag caused the problem and does not seem to be
> necessary:
> 
> '--LDFLAGS=-L'+os.environ['ROCM_PATH']+'/lib -lhsa-runtime64',
> 
> On Tue, Jul 20, 2021 at 9:00 AM Mark Adams  wrote:
> 
> >
> >
> > On Mon, Jul 19, 2021 at 6:41 PM Scott Kruger  wrote:
> >
> >>
> >> Mark,
> >>
> >> On tulip, things with fortran went wonky when a `-fast` flag snuck into
> >> the flags (snuck in being me copying a previous configure file and not
> >> questioning the flags).  The reason is that for clang, `-fast` implies
> >> link time optimization (lto) for C/C++ code, but flang doesn't support
> >> lto so things got weird.  I suspect that gfortran does not either (but
> >> perhaps the real question is why not use flang?)
> >>
> >
> > It does work with GNU. I am doing this for an application and I want to
> > avoid dictating the program env.
> >
> > I'm (re)trying with -O0 just to check.
> >
> >
> >>
> >> I look at configure and I don't see anything in your flags that would
> >> trigger lto, but as Junchao says, it might be picking something up from
> >> Kokkos, so perhaps this is the issue.
> >>
> >>
> > The configure files that I sent did not have Kokkos. They were stripped
> > down.
> >
> > I have asked ORNL, but don't expect help.
> > The person (Matt) that is helping me for other things is very helpful so
> > perhaps he can come up with some ideas.
> > Otherwise I will tell the app that they need to use GNU for now.
> >
> > Thanks,
> > Mark
> >
> >
> >
> >> Scott
> >>
> >>
> >> On 2021-07-19 07:19, Mark Adams did write:
> >> > Thanks, but this happens w/o Kokkos.
> >> > I've stripped this down and attached good/bad logs without/with Fortran
> >> > bindings.
> >> > Hope this helps,
> >> > Thanks again,
> >> > Mark
> >> >
> >> > On Sun, Jul 18, 2021 at 12:00 PM Stefano Zampini <
> >> stefano.zamp...@gmail.com>
> >> > wrote:
> >> >
> >> > > This is probably kokkos pulling in the dependencies where compiling
> >> kokkos
> >> > > source within PETSc.
> >> > >
> >> > > Il Dom 18 Lug 2021, 16:29 Mark Adams  ha scritto:
> >> > >
> >> > >> Whoops, this error was just from not telling gfortran to allow long
> >> lines.
> >> > >>
> >> > >> Anway, I did find that the when fortran bindings are enabled this OMP
> >> > >> thing gets into the library. No idea how. I'll askm ORNL.
> >> > >>
> >> > >> 10:26 jczhang/fix-cray-mpicxx-includes/main=
> >> > >> /gpfs/alpine/csc314/scratch/adams/petsc2$ nm
> >> > >>
> >> /gpfs/alpine/csc314/scratch/adams/petsc2/arch-spock-opt-cray-kokkos/lib/libpetsc.so
> >> > >> |g offload
> >> > >>  U .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa
> >> > >>  U .omp_offloading.img_size.cray_amdgcn-amd-amdhsa
> >> > >>  U .omp_offloading.img_start.cray_amdgcn-amd-amdhsa
> >> > >> 01d457b0 T vecgetoffloadmask_
> >> > >>
> >> > >>
> >> > >> On Sun, Jul 18, 2021 at 8:43 AM Mark Adams  wrote:
> >> > >>
> >> > >>> Ah, your test was not on Spock ... I have something working but
> >> this is
> >> > >>> strange.
> >> > >>>
> >> > >>> I switched to the GNU ProgEnv. and it passed the Fortran test in
> >> > >>> 'check', but this failed:
> >> > >>>
> >> > >>>
> >> > >>>
> >> > >>>
> >> > >>>
> >> > >>>
> >> > >>>
> >> > >>>
> >> > >>>
> >> > >>> *08:26 2 jczhang/fix-cray-mpicxx-includes/main=
> >> > >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make
> >> > >>>
> >> PETSC_DIR=/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu
> >> > >>> PETSC_ARCH="" ex40f90ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
> >> > >>> -lamdhip64 -lhsa-runtime64  -fPIC -g -O0   -fPIC -g -O0
> >> > >>>
> >> -I/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/include
> >> > >>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex40f90.F90
> >> > >>>
> >> -Wl,-rpath,/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib
> >> > >>>
> >> -L/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib
> >> > >>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
> >> > >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
> >> > >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib
> >> > >>> -L/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib
> >> > >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
> >> > >>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib
> >> > >>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib
> >> > >>> 
> >> > >>> -L/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib
> >> > >>> 
> >> > >>> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib

Re: [petsc-dev] Kokkos make error on Spock

2021-07-20 Thread Mark Adams 
OK, it looks like this flag caused the problem and does not seem to be
necessary:

'--LDFLAGS=-L'+os.environ['ROCM_PATH']+'/lib -lhsa-runtime64',

On Tue, Jul 20, 2021 at 9:00 AM Mark Adams  wrote:

>
>
> On Mon, Jul 19, 2021 at 6:41 PM Scott Kruger  wrote:
>
>>
>> Mark,
>>
>> On tulip, things with fortran went wonky when a `-fast` flag snuck into
>> the flags (snuck in being me copying a previous configure file and not
>> questioning the flags).  The reason is that for clang, `-fast` implies
>> link time optimization (lto) for C/C++ code, but flang doesn't support
>> lto so things got weird.  I suspect that gfortran does not either (but
>> perhaps the real question is why not use flang?)
>>
>
> It does work with GNU. I am doing this for an application and I want to
> avoid dictating the program env.
>
> I'm (re)trying with -O0 just to check.
>
>
>>
>> I look at configure and I don't see anything in your flags that would
>> trigger lto, but as Junchao says, it might be picking something up from
>> Kokkos, so perhaps this is the issue.
>>
>>
> The configure files that I sent did not have Kokkos. They were stripped
> down.
>
> I have asked ORNL, but don't expect help.
> The person (Matt) that is helping me for other things is very helpful so
> perhaps he can come up with some ideas.
> Otherwise I will tell the app that they need to use GNU for now.
>
> Thanks,
> Mark
>
>
>
>> Scott
>>
>>
>> On 2021-07-19 07:19, Mark Adams did write:
>> > Thanks, but this happens w/o Kokkos.
>> > I've stripped this down and attached good/bad logs without/with Fortran
>> > bindings.
>> > Hope this helps,
>> > Thanks again,
>> > Mark
>> >
>> > On Sun, Jul 18, 2021 at 12:00 PM Stefano Zampini <
>> stefano.zamp...@gmail.com>
>> > wrote:
>> >
>> > > This is probably kokkos pulling in the dependencies where compiling
>> kokkos
>> > > source within PETSc.
>> > >
>> > > Il Dom 18 Lug 2021, 16:29 Mark Adams  ha scritto:
>> > >
>> > >> Whoops, this error was just from not telling gfortran to allow long
>> lines.
>> > >>
>> > >> Anway, I did find that the when fortran bindings are enabled this OMP
>> > >> thing gets into the library. No idea how. I'll askm ORNL.
>> > >>
>> > >> 10:26 jczhang/fix-cray-mpicxx-includes/main=
>> > >> /gpfs/alpine/csc314/scratch/adams/petsc2$ nm
>> > >>
>> /gpfs/alpine/csc314/scratch/adams/petsc2/arch-spock-opt-cray-kokkos/lib/libpetsc.so
>> > >> |g offload
>> > >>  U .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa
>> > >>  U .omp_offloading.img_size.cray_amdgcn-amd-amdhsa
>> > >>  U .omp_offloading.img_start.cray_amdgcn-amd-amdhsa
>> > >> 01d457b0 T vecgetoffloadmask_
>> > >>
>> > >>
>> > >> On Sun, Jul 18, 2021 at 8:43 AM Mark Adams  wrote:
>> > >>
>> > >>> Ah, your test was not on Spock ... I have something working but
>> this is
>> > >>> strange.
>> > >>>
>> > >>> I switched to the GNU ProgEnv. and it passed the Fortran test in
>> > >>> 'check', but this failed:
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> > >>> *08:26 2 jczhang/fix-cray-mpicxx-includes/main=
>> > >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make
>> > >>>
>> PETSC_DIR=/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu
>> > >>> PETSC_ARCH="" ex40f90ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
>> > >>> -lamdhip64 -lhsa-runtime64  -fPIC -g -O0   -fPIC -g -O0
>> > >>>
>> -I/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/include
>> > >>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex40f90.F90
>> > >>>
>> -Wl,-rpath,/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib
>> > >>>
>> -L/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib
>> > >>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
>> > >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
>> > >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib
>> > >>> -L/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib
>> > >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
>> > >>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib
>> > >>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib
>> > >>> 
>> > >>> -L/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib
>> > >>> 
>> > >>> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib
>> > >>> -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>> > >>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>> > >>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>> > >>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>> > >>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0
>> > >>> -L/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0
>> > >>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib64 -L/opt/gcc/10.2.0/snos/lib64
>> > >>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib
>> > >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib
>> > >>>

Re: [petsc-dev] Kokkos make error on Spock

2021-07-20 Thread Mark Adams 
On Mon, Jul 19, 2021 at 6:41 PM Scott Kruger  wrote:

>
> Mark,
>
> On tulip, things with fortran went wonky when a `-fast` flag snuck into
> the flags (snuck in being me copying a previous configure file and not
> questioning the flags).  The reason is that for clang, `-fast` implies
> link time optimization (lto) for C/C++ code, but flang doesn't support
> lto so things got weird.  I suspect that gfortran does not either (but
> perhaps the real question is why not use flang?)
>

It does work with GNU. I am doing this for an application and I want to
avoid dictating the program env.

I'm (re)trying with -O0 just to check.


>
> I look at configure and I don't see anything in your flags that would
> trigger lto, but as Junchao says, it might be picking something up from
> Kokkos, so perhaps this is the issue.
>
>
The configure files that I sent did not have Kokkos. They were stripped
down.

I have asked ORNL, but don't expect help.
The person (Matt) that is helping me for other things is very helpful so
perhaps he can come up with some ideas.
Otherwise I will tell the app that they need to use GNU for now.

Thanks,
Mark



> Scott
>
>
> On 2021-07-19 07:19, Mark Adams did write:
> > Thanks, but this happens w/o Kokkos.
> > I've stripped this down and attached good/bad logs without/with Fortran
> > bindings.
> > Hope this helps,
> > Thanks again,
> > Mark
> >
> > On Sun, Jul 18, 2021 at 12:00 PM Stefano Zampini <
> stefano.zamp...@gmail.com>
> > wrote:
> >
> > > This is probably kokkos pulling in the dependencies where compiling
> kokkos
> > > source within PETSc.
> > >
> > > Il Dom 18 Lug 2021, 16:29 Mark Adams  ha scritto:
> > >
> > >> Whoops, this error was just from not telling gfortran to allow long
> lines.
> > >>
> > >> Anway, I did find that the when fortran bindings are enabled this OMP
> > >> thing gets into the library. No idea how. I'll askm ORNL.
> > >>
> > >> 10:26 jczhang/fix-cray-mpicxx-includes/main=
> > >> /gpfs/alpine/csc314/scratch/adams/petsc2$ nm
> > >>
> /gpfs/alpine/csc314/scratch/adams/petsc2/arch-spock-opt-cray-kokkos/lib/libpetsc.so
> > >> |g offload
> > >>  U .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa
> > >>  U .omp_offloading.img_size.cray_amdgcn-amd-amdhsa
> > >>  U .omp_offloading.img_start.cray_amdgcn-amd-amdhsa
> > >> 01d457b0 T vecgetoffloadmask_
> > >>
> > >>
> > >> On Sun, Jul 18, 2021 at 8:43 AM Mark Adams  wrote:
> > >>
> > >>> Ah, your test was not on Spock ... I have something working but this
> is
> > >>> strange.
> > >>>
> > >>> I switched to the GNU ProgEnv. and it passed the Fortran test in
> > >>> 'check', but this failed:
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> *08:26 2 jczhang/fix-cray-mpicxx-includes/main=
> > >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make
> > >>>
> PETSC_DIR=/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu
> > >>> PETSC_ARCH="" ex40f90ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
> > >>> -lamdhip64 -lhsa-runtime64  -fPIC -g -O0   -fPIC -g -O0
> > >>>
> -I/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/include
> > >>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex40f90.F90
> > >>>
> -Wl,-rpath,/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib
> > >>>
> -L/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib
> > >>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
> > >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
> > >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib
> > >>> -L/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib
> > >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
> > >>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib
> > >>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib
> > >>> 
> > >>> -L/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib
> > >>> 
> > >>> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib
> > >>> -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
> > >>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
> > >>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
> > >>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
> > >>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0
> > >>> -L/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0
> > >>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib64 -L/opt/gcc/10.2.0/snos/lib64
> > >>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib
> > >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib
> > >>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib -L/opt/gcc/10.2.0/snos/lib
> -lpetsc
> > >>> -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64
> > >>> -lhsa-runtime64 -lstdc++ -ldl -lpmi -lmpi_gtl_hsa -lxpmem -lgfortran
> -lm
> > >>> -lsci_gnu_82_mpi -lsci_gnu_82 -lmpifort_gnu_91 -lmpi_gnu_91 -ldsmml
> > >>> -lgfortran -lquadmath -lpthread -lm -lgcc_s -lquadmath

Re: [petsc-dev] Kokkos make error on Spock

2021-07-19 Thread Scott Kruger 


Mark,

On tulip, things with fortran went wonky when a `-fast` flag snuck into
the flags (snuck in being me copying a previous configure file and not
questioning the flags).  The reason is that for clang, `-fast` implies
link time optimization (lto) for C/C++ code, but flang doesn't support
lto so things got weird.  I suspect that gfortran does not either (but
perhaps the real question is why not use flang?)

I look at configure and I don't see anything in your flags that would
trigger lto, but as Junchao says, it might be picking something up from
Kokkos, so perhaps this is the issue.

Scott


On 2021-07-19 07:19, Mark Adams did write:
> Thanks, but this happens w/o Kokkos.
> I've stripped this down and attached good/bad logs without/with Fortran
> bindings.
> Hope this helps,
> Thanks again,
> Mark
> 
> On Sun, Jul 18, 2021 at 12:00 PM Stefano Zampini 
> wrote:
> 
> > This is probably kokkos pulling in the dependencies where compiling kokkos
> > source within PETSc.
> >
> > Il Dom 18 Lug 2021, 16:29 Mark Adams  ha scritto:
> >
> >> Whoops, this error was just from not telling gfortran to allow long lines.
> >>
> >> Anway, I did find that the when fortran bindings are enabled this OMP
> >> thing gets into the library. No idea how. I'll askm ORNL.
> >>
> >> 10:26 jczhang/fix-cray-mpicxx-includes/main=
> >> /gpfs/alpine/csc314/scratch/adams/petsc2$ nm
> >> /gpfs/alpine/csc314/scratch/adams/petsc2/arch-spock-opt-cray-kokkos/lib/libpetsc.so
> >> |g offload
> >>  U .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa
> >>  U .omp_offloading.img_size.cray_amdgcn-amd-amdhsa
> >>  U .omp_offloading.img_start.cray_amdgcn-amd-amdhsa
> >> 01d457b0 T vecgetoffloadmask_
> >>
> >>
> >> On Sun, Jul 18, 2021 at 8:43 AM Mark Adams  wrote:
> >>
> >>> Ah, your test was not on Spock ... I have something working but this is
> >>> strange.
> >>>
> >>> I switched to the GNU ProgEnv. and it passed the Fortran test in
> >>> 'check', but this failed:
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> *08:26 2 jczhang/fix-cray-mpicxx-includes/main=
> >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make
> >>> PETSC_DIR=/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu
> >>> PETSC_ARCH="" ex40f90ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
> >>> -lamdhip64 -lhsa-runtime64  -fPIC -g -O0   -fPIC -g -O0
> >>>  
> >>> -I/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/include
> >>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex40f90.F90
> >>>  
> >>> -Wl,-rpath,/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib
> >>> -L/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib
> >>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
> >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
> >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib
> >>> -L/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib
> >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
> >>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib
> >>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib
> >>> 
> >>> -L/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib
> >>> 
> >>> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib
> >>> -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
> >>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
> >>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
> >>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
> >>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0
> >>> -L/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0
> >>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib64 -L/opt/gcc/10.2.0/snos/lib64
> >>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib
> >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib
> >>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib -L/opt/gcc/10.2.0/snos/lib -lpetsc
> >>> -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64
> >>> -lhsa-runtime64 -lstdc++ -ldl -lpmi -lmpi_gtl_hsa -lxpmem -lgfortran -lm
> >>> -lsci_gnu_82_mpi -lsci_gnu_82 -lmpifort_gnu_91 -lmpi_gnu_91 -ldsmml
> >>> -lgfortran -lquadmath -lpthread -lm -lgcc_s -lquadmath -lstdc++ -ldl -o
> >>> ex40f90ex40f90.F90:35:132:   35 |   call
> >>> DMDACreate2d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_BOX,ten,ten,PETSC_DECIDE,PETSC_DECIDE,two,one,
> >>> &  |
> >>>  1Error:
> >>> Line truncated at (1) [-Werror=line-truncation]ex40f90.F90:35:132:*
> >>> * *
> >>>
> >>> But ex5f (the Fortran check test) does work manually also (sanity check).
> >>>
> >>> Oddly I seem to have the cray compilers:
> >>>
> >>> 08:34 jczhang/fix-cray-mpicxx-includes/main=
> >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ which ftn
> >>> /opt/cray/pe/craype/2.7.6/bin/ftn
> >>>
> >>> yet it is GNU:
> >>>
> >>> 08:34

Re: [petsc-dev] Kokkos make error on Spock

2021-07-18 Thread Stefano Zampini 
This is probably kokkos pulling in the dependencies where compiling kokkos
source within PETSc.

Il Dom 18 Lug 2021, 16:29 Mark Adams  ha scritto:

> Whoops, this error was just from not telling gfortran to allow long lines.
>
> Anway, I did find that the when fortran bindings are enabled this OMP
> thing gets into the library. No idea how. I'll askm ORNL.
>
> 10:26 jczhang/fix-cray-mpicxx-includes/main=
> /gpfs/alpine/csc314/scratch/adams/petsc2$ nm
> /gpfs/alpine/csc314/scratch/adams/petsc2/arch-spock-opt-cray-kokkos/lib/libpetsc.so
> |g offload
>  U .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa
>  U .omp_offloading.img_size.cray_amdgcn-amd-amdhsa
>  U .omp_offloading.img_start.cray_amdgcn-amd-amdhsa
> 01d457b0 T vecgetoffloadmask_
>
>
> On Sun, Jul 18, 2021 at 8:43 AM Mark Adams  wrote:
>
>> Ah, your test was not on Spock ... I have something working but this is
>> strange.
>>
>> I switched to the GNU ProgEnv. and it passed the Fortran test in 'check',
>> but this failed:
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *08:26 2 jczhang/fix-cray-mpicxx-includes/main=
>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make
>> PETSC_DIR=/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu
>> PETSC_ARCH="" ex40f90ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
>> -lamdhip64 -lhsa-runtime64  -fPIC -g -O0   -fPIC -g -O0
>>  -I/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/include
>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex40f90.F90
>>  
>> -Wl,-rpath,/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib
>> -L/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib
>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib
>> -L/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib
>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib
>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib
>> 
>> -L/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib
>> 
>> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib
>> -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0
>> -L/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0
>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib64 -L/opt/gcc/10.2.0/snos/lib64
>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib
>> -L/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib
>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib -L/opt/gcc/10.2.0/snos/lib -lpetsc
>> -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64
>> -lhsa-runtime64 -lstdc++ -ldl -lpmi -lmpi_gtl_hsa -lxpmem -lgfortran -lm
>> -lsci_gnu_82_mpi -lsci_gnu_82 -lmpifort_gnu_91 -lmpi_gnu_91 -ldsmml
>> -lgfortran -lquadmath -lpthread -lm -lgcc_s -lquadmath -lstdc++ -ldl -o
>> ex40f90ex40f90.F90:35:132:   35 |   call
>> DMDACreate2d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_BOX,ten,ten,PETSC_DECIDE,PETSC_DECIDE,two,one,
>> &  |
>>  1Error:
>> Line truncated at (1) [-Werror=line-truncation]ex40f90.F90:35:132:*
>> * *
>>
>> But ex5f (the Fortran check test) does work manually also (sanity check).
>>
>> Oddly I seem to have the cray compilers:
>>
>> 08:34 jczhang/fix-cray-mpicxx-includes/main=
>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ which ftn
>> /opt/cray/pe/craype/2.7.6/bin/ftn
>>
>> yet it is GNU:
>>
>> 08:34 jczhang/fix-cray-mpicxx-includes/main=
>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ ftn --version
>> GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.)
>> Copyright (C) 2020 Free Software Foundation, Inc.
>> This is free software; see the source for copying conditions.  There is NO
>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
>> PURPOSE.
>>
>> The documentation says ftn should be here:
>>
>> 08:34 jczhang/fix-cray-mpicxx-includes/main=
>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$
>> $GCC_PATH/bin/gfortran --version
>> GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.)
>> Copyright (C) 2020 Free Software Foundation, Inc.
>> This is free software; see the source for copying conditions.  There is NO
>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
>> PURPOSE.
>>
>> Same, but the path is not the same:
>>
>> 08:39 jczhang/fix-cray-mpicxx-includes/main=
>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ echo $GCC_PATH
>> /opt/gcc/10.2.0
>>
>> Anyway, thanks for your help,
>> Mark
>>
>> On Sat, Jul 17, 2021 at 11:26 PM Junchao Zhang 
>> wrote:
>>
>>> Mark,  I can reproduce

Re: [petsc-dev] Kokkos make error on Spock

2021-07-18 Thread Mark Adams 
Whoops, this error was just from not telling gfortran to allow long lines.

Anway, I did find that the when fortran bindings are enabled this OMP thing
gets into the library. No idea how. I'll askm ORNL.

10:26 jczhang/fix-cray-mpicxx-includes/main=
/gpfs/alpine/csc314/scratch/adams/petsc2$ nm
/gpfs/alpine/csc314/scratch/adams/petsc2/arch-spock-opt-cray-kokkos/lib/libpetsc.so
|g offload
 U .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa
 U .omp_offloading.img_size.cray_amdgcn-amd-amdhsa
 U .omp_offloading.img_start.cray_amdgcn-amd-amdhsa
01d457b0 T vecgetoffloadmask_


On Sun, Jul 18, 2021 at 8:43 AM Mark Adams  wrote:

> Ah, your test was not on Spock ... I have something working but this is
> strange.
>
> I switched to the GNU ProgEnv. and it passed the Fortran test in 'check',
> but this failed:
>
>
>
>
>
>
>
>
>
> *08:26 2 jczhang/fix-cray-mpicxx-includes/main=
> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make
> PETSC_DIR=/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu
> PETSC_ARCH="" ex40f90ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
> -lamdhip64 -lhsa-runtime64  -fPIC -g -O0   -fPIC -g -O0
>  -I/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/include
> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex40f90.F90
>  
> -Wl,-rpath,/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib
> -L/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib
> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib
> -L/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib
> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
> -L/opt/cray/pe/mpich/8.1.4/gtl/lib
> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib
> 
> -L/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib
> 
> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib
> -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0
> -L/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0
> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib64 -L/opt/gcc/10.2.0/snos/lib64
> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib
> -L/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib
> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib -L/opt/gcc/10.2.0/snos/lib -lpetsc
> -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64
> -lhsa-runtime64 -lstdc++ -ldl -lpmi -lmpi_gtl_hsa -lxpmem -lgfortran -lm
> -lsci_gnu_82_mpi -lsci_gnu_82 -lmpifort_gnu_91 -lmpi_gnu_91 -ldsmml
> -lgfortran -lquadmath -lpthread -lm -lgcc_s -lquadmath -lstdc++ -ldl -o
> ex40f90ex40f90.F90:35:132:   35 |   call
> DMDACreate2d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_BOX,ten,ten,PETSC_DECIDE,PETSC_DECIDE,two,one,
> &  |
>  1Error:
> Line truncated at (1) [-Werror=line-truncation]ex40f90.F90:35:132:*
> * *
>
> But ex5f (the Fortran check test) does work manually also (sanity check).
>
> Oddly I seem to have the cray compilers:
>
> 08:34 jczhang/fix-cray-mpicxx-includes/main=
> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ which ftn
> /opt/cray/pe/craype/2.7.6/bin/ftn
>
> yet it is GNU:
>
> 08:34 jczhang/fix-cray-mpicxx-includes/main=
> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ ftn --version
> GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.)
> Copyright (C) 2020 Free Software Foundation, Inc.
> This is free software; see the source for copying conditions.  There is NO
> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>
> The documentation says ftn should be here:
>
> 08:34 jczhang/fix-cray-mpicxx-includes/main=
> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$
> $GCC_PATH/bin/gfortran --version
> GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.)
> Copyright (C) 2020 Free Software Foundation, Inc.
> This is free software; see the source for copying conditions.  There is NO
> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>
> Same, but the path is not the same:
>
> 08:39 jczhang/fix-cray-mpicxx-includes/main=
> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ echo $GCC_PATH
> /opt/gcc/10.2.0
>
> Anyway, thanks for your help,
> Mark
>
> On Sat, Jul 17, 2021 at 11:26 PM Junchao Zhang 
> wrote:
>
>> Mark,  I can reproduce this error with PrgEnv-cray, i.e., using the Cray
>> compiler (clang-11).  Previously I used PrgEnv-gnu, which did not have this
>> error.
>> Probably it is a problem of Spock.  But I am not sure.
>>
>> --Junchao Zhang
>>
>>
>> On Sat, Jul 17, 2021 at 10:17 AM Mark Adams  wrote:

Re: [petsc-dev] Kokkos make error on Spock

2021-07-18 Thread Mark Adams 
Ah, your test was not on Spock ... I have something working but this is
strange.

I switched to the GNU ProgEnv. and it passed the Fortran test in 'check',
but this failed:









*08:26 2 jczhang/fix-cray-mpicxx-includes/main=
/gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make
PETSC_DIR=/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu
PETSC_ARCH="" ex40f90ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
-lamdhip64 -lhsa-runtime64  -fPIC -g -O0   -fPIC -g -O0
 -I/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/include
-I/sw/spock/spack-envs/views/rocm-4.1.0/include ex40f90.F90
 -Wl,-rpath,/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib
-L/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib
-Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
-L/sw/spock/spack-envs/views/rocm-4.1.0/lib
-Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib
-L/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib
-Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
-L/opt/cray/pe/mpich/8.1.4/gtl/lib
-Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib

-L/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib

-Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib
-Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
-L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
-Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
-L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
-Wl,-rpath,/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0
-L/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0
-Wl,-rpath,/opt/gcc/10.2.0/snos/lib64 -L/opt/gcc/10.2.0/snos/lib64
-Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib
-L/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib
-Wl,-rpath,/opt/gcc/10.2.0/snos/lib -L/opt/gcc/10.2.0/snos/lib -lpetsc
-lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64
-lhsa-runtime64 -lstdc++ -ldl -lpmi -lmpi_gtl_hsa -lxpmem -lgfortran -lm
-lsci_gnu_82_mpi -lsci_gnu_82 -lmpifort_gnu_91 -lmpi_gnu_91 -ldsmml
-lgfortran -lquadmath -lpthread -lm -lgcc_s -lquadmath -lstdc++ -ldl -o
ex40f90ex40f90.F90:35:132:   35 |   call
DMDACreate2d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_BOX,ten,ten,PETSC_DECIDE,PETSC_DECIDE,two,one,
&  |
 1Error:
Line truncated at (1) [-Werror=line-truncation]ex40f90.F90:35:132:*
* *

But ex5f (the Fortran check test) does work manually also (sanity check).

Oddly I seem to have the cray compilers:

08:34 jczhang/fix-cray-mpicxx-includes/main=
/gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ which ftn
/opt/cray/pe/craype/2.7.6/bin/ftn

yet it is GNU:

08:34 jczhang/fix-cray-mpicxx-includes/main=
/gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ ftn --version
GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.)
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

The documentation says ftn should be here:

08:34 jczhang/fix-cray-mpicxx-includes/main=
/gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$
$GCC_PATH/bin/gfortran --version
GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.)
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Same, but the path is not the same:

08:39 jczhang/fix-cray-mpicxx-includes/main=
/gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ echo $GCC_PATH
/opt/gcc/10.2.0

Anyway, thanks for your help,
Mark

On Sat, Jul 17, 2021 at 11:26 PM Junchao Zhang 
wrote:

> Mark,  I can reproduce this error with PrgEnv-cray, i.e., using the Cray
> compiler (clang-11).  Previously I used PrgEnv-gnu, which did not have this
> error.
> Probably it is a problem of Spock.  But I am not sure.
>
> --Junchao Zhang
>
>
> On Sat, Jul 17, 2021 at 10:17 AM Mark Adams  wrote:
>
>> And I can run a fortran test, with warnings, but C tests fail:
>>
>> 11:15 jczhang/fix-cray-mpicxx-includes/main=
>> /gpfs/alpine/csc314/scratch/adams/petsc$ make
>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
>> PETSC_ARCH=arch-spock-opt-cray-kokkos -f gmakefile test
>> search='ts_utils_dmplexlandau_tutorials-ex1f90_0'
>> Using MAKEFLAGS: -- search=ts_utils_dmplexlandau_tutorials-ex1f90_0
>> PETSC_ARCH=arch-spock-opt-cray-kokkos
>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
>>   FC
>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o
>>  FLINKER
>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90
>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:
>> warning: alignment 128 of symbol
>> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in
>>

Re: [petsc-dev] Kokkos make error on Spock

2021-07-17 Thread Junchao Zhang 
Mark,  I can reproduce this error with PrgEnv-cray, i.e., using the Cray
compiler (clang-11).  Previously I used PrgEnv-gnu, which did not have this
error.
Probably it is a problem of Spock.  But I am not sure.

--Junchao Zhang


On Sat, Jul 17, 2021 at 10:17 AM Mark Adams  wrote:

> And I can run a fortran test, with warnings, but C tests fail:
>
> 11:15 jczhang/fix-cray-mpicxx-includes/main=
> /gpfs/alpine/csc314/scratch/adams/petsc$ make
> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
> PETSC_ARCH=arch-spock-opt-cray-kokkos -f gmakefile test
> search='ts_utils_dmplexlandau_tutorials-ex1f90_0'
> Using MAKEFLAGS: -- search=ts_utils_dmplexlandau_tutorials-ex1f90_0
> PETSC_ARCH=arch-spock-opt-cray-kokkos
> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
>   FC
> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o
>  FLINKER
> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90
> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:
> warning: alignment 128 of symbol
> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in
> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in
> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o
> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:
> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in
> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in
> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o
> TEST
> arch-spock-opt-cray-kokkos/tests/counts/ts_utils_dmplexlandau_tutorials-ex1f90_0.counts
>  ok ts_utils_dmplexlandau_tutorials-ex1f90_0
>  ok diff-ts_utils_dmplexlandau_tutorials-ex1f90_0
>
>
> On Sat, Jul 17, 2021 at 10:53 AM Mark Adams  wrote:
>
>> HUmm, I can not reproduce this.
>>
>> 10:49 jczhang/fix-cray-mpicxx-includes/main=
>> /gpfs/alpine/csc314/scratch/adams/petsc$ make
>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
>> PETSC_ARCH=arch-spock-dbg-kokkos check
>> Running check examples to verify correct installation
>> Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and
>> PETSC_ARCH=arch-spock-dbg-kokkos
>> gmake[3]:
>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301:
>> ex19.PETSc] Error 2 (ignored)
>> ***Error detected during compile or
>> link!***
>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex19
>>
>> *
>> cc -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64
>>  -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas
>> -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O0  -fPIC
>> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas
>> -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O0
>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include
>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include
>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
>>  -I/sw/spock/spack-envs/views/rocm-4.1.0/includeex19.c
>>  -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64
>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib
>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse
>>

Re: [petsc-dev] Kokkos make error on Spock

2021-07-17 Thread Mark Adams 
Also, the application is a C++ code with PETSc Fortran methods.
Thanks,
Mark

On Sat, Jul 17, 2021 at 11:17 AM Mark Adams  wrote:

> And I can run a fortran test, with warnings, but C tests fail:
>
> 11:15 jczhang/fix-cray-mpicxx-includes/main=
> /gpfs/alpine/csc314/scratch/adams/petsc$ make
> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
> PETSC_ARCH=arch-spock-opt-cray-kokkos -f gmakefile test
> search='ts_utils_dmplexlandau_tutorials-ex1f90_0'
> Using MAKEFLAGS: -- search=ts_utils_dmplexlandau_tutorials-ex1f90_0
> PETSC_ARCH=arch-spock-opt-cray-kokkos
> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
>   FC
> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o
>  FLINKER
> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90
> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:
> warning: alignment 128 of symbol
> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in
> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in
> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o
> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:
> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in
> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in
> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o
> TEST
> arch-spock-opt-cray-kokkos/tests/counts/ts_utils_dmplexlandau_tutorials-ex1f90_0.counts
>  ok ts_utils_dmplexlandau_tutorials-ex1f90_0
>  ok diff-ts_utils_dmplexlandau_tutorials-ex1f90_0
>
>
> On Sat, Jul 17, 2021 at 10:53 AM Mark Adams  wrote:
>
>> HUmm, I can not reproduce this.
>>
>> 10:49 jczhang/fix-cray-mpicxx-includes/main=
>> /gpfs/alpine/csc314/scratch/adams/petsc$ make
>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
>> PETSC_ARCH=arch-spock-dbg-kokkos check
>> Running check examples to verify correct installation
>> Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and
>> PETSC_ARCH=arch-spock-dbg-kokkos
>> gmake[3]:
>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301:
>> ex19.PETSc] Error 2 (ignored)
>> ***Error detected during compile or
>> link!***
>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex19
>>
>> *
>> cc -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64
>>  -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas
>> -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O0  -fPIC
>> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas
>> -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O0
>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include
>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include
>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
>>  -I/sw/spock/spack-envs/views/rocm-4.1.0/includeex19.c
>>  -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64
>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib
>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse
>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64
>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray
>>

Re: [petsc-dev] Kokkos make error on Spock

2021-07-17 Thread Mark Adams 
And I can run a fortran test, with warnings, but C tests fail:

11:15 jczhang/fix-cray-mpicxx-includes/main=
/gpfs/alpine/csc314/scratch/adams/petsc$ make
PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
PETSC_ARCH=arch-spock-opt-cray-kokkos -f gmakefile test
search='ts_utils_dmplexlandau_tutorials-ex1f90_0'
Using MAKEFLAGS: -- search=ts_utils_dmplexlandau_tutorials-ex1f90_0
PETSC_ARCH=arch-spock-opt-cray-kokkos
PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
  FC
arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o
 FLINKER
arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90
/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:
warning: alignment 128 of symbol
`$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in
/opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in
arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o
/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:
warning: alignment 64 of symbol `$data_init$iso_c_binding_' in
/opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in
arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o
TEST
arch-spock-opt-cray-kokkos/tests/counts/ts_utils_dmplexlandau_tutorials-ex1f90_0.counts
 ok ts_utils_dmplexlandau_tutorials-ex1f90_0
 ok diff-ts_utils_dmplexlandau_tutorials-ex1f90_0


On Sat, Jul 17, 2021 at 10:53 AM Mark Adams  wrote:

> HUmm, I can not reproduce this.
>
> 10:49 jczhang/fix-cray-mpicxx-includes/main=
> /gpfs/alpine/csc314/scratch/adams/petsc$ make
> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
> PETSC_ARCH=arch-spock-dbg-kokkos check
> Running check examples to verify correct installation
> Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and
> PETSC_ARCH=arch-spock-dbg-kokkos
> gmake[3]:
> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301:
> ex19.PETSc] Error 2 (ignored)
> ***Error detected during compile or
> link!***
> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex19
>
> *
> cc -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64
>  -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas
> -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O0  -fPIC
> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas
> -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O0
> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
> -I/gpfs/alpine/csc314/scratch/adams/petsc/include
> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include
> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
>  -I/sw/spock/spack-envs/views/rocm-4.1.0/includeex19.c
>  -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64
> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib
> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse
> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64
> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray
> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu
> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread
> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64
> -lquadmath -lstdc++ -ldl -o ex19
>

Re: [petsc-dev] Kokkos make error on Spock

2021-07-16 Thread Junchao Zhang 
Mark, I configured with Fortran binding enabled with main + MR !4175


'--with-mpiexec=srun',
'--with-shared-libraries=1',
'--with-cc=cc',
'--with-cxx=CC',
'--with-fc=ftn',
'--with-fortran-bindings',
'--with-hip',
'--with-hipc=hipcc',
'--with-debugging',
'--CPPFLAGS=-I${ROCM_PATH}/include',
'--CXXPPFLAGS=-I${ROCM_PATH}/include',
'--CC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64',
'--CXX_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64',
'--FC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64',
'--COPTFLAGS=-g -O0',
'--CXXOPTFLAGS=-g -O0',
'--FOPTFLAGS=-g -O0',
'--download-kokkos',
'--download-kokkos-kernels',
'--download-kokkos-commit=3.4.01',
'--download-kokkos-kernels-commit=3.4.01',
'--with-kokkos-hip-arch=VEGA908',

and 'make check' ran smoothly on a compute node

$ make check
Running check examples to verify correct installation
Using PETSC_DIR=/ccs/home/jczhang/petsc and
PETSC_ARCH=arch-spock-cray-kokkos-dbg
C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process
C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes
C/C++ example src/snes/tutorials/ex3k run successfully with kokkos-kernels
Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process

Completed test examples


--Junchao Zhang


On Fri, Jul 16, 2021 at 6:04 PM Mark Adams  wrote:

> And I find that this error, on non-Kokkos C tests, is fixed by turning the
> fortran bindings off:
>
> ld.lld: error:
> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so:
> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa
> [--no-allow-shlib-undefined]
> ld.lld: error:
> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so:
> undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa
> [--no-allow-shlib-undefined]
> ld.lld: error:
> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so:
> undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa
> [--no-allow-shlib-undefined]
>
> On Fri, Jul 16, 2021 at 3:53 PM Mark Adams  wrote:
>
>> Not complex. THis has some overlap with my problem w/o Kokkos.
>>
>> On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang 
>> wrote:
>>
>>> Do you use complex? post your configure.log.
>>>
>>> --Junchao Zhang
>>>
>>>
>>> On Fri, Jul 16, 2021 at 9:47 AM Mark Adams  wrote:
>>>
 The simple Kokkos example is failing for me on Spock.
 Any ideas?
 Thanks,

 10:44 main *=
 /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make
 PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
 PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k
 MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC
 -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64
 -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64
  -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas
 -fstack-protector -fvisibility=hidden -g -O2   -fPIC -Wall -Wwrite-strings
 -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3
 -std=c++14  -I/gpfs/alpine/csc314/scratch/adams/petsc/include
 -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include
 -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include
 -I/sw/spock/spack-envs/views/rocm-4.1.0/include
  -I/sw/spock/spack-envs/views/rocm-4.1.0/includeex3k.kokkos.cxx
  
 -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib
 -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib
 -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib
 -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib
 -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
 -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
 -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64
 -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
 -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
 -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
 -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
 -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
 -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib
 -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
 -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
 -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
 -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
 -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
 -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
 -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
 -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux

Re: [petsc-dev] Kokkos make error on Spock

2021-07-16 Thread Mark Adams 
And I find that this error, on non-Kokkos C tests, is fixed by turning the
fortran bindings off:

ld.lld: error:
/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so:
undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa
[--no-allow-shlib-undefined]
ld.lld: error:
/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so:
undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa
[--no-allow-shlib-undefined]
ld.lld: error:
/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so:
undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa
[--no-allow-shlib-undefined]

On Fri, Jul 16, 2021 at 3:53 PM Mark Adams  wrote:

> Not complex. THis has some overlap with my problem w/o Kokkos.
>
> On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang 
> wrote:
>
>> Do you use complex? post your configure.log.
>>
>> --Junchao Zhang
>>
>>
>> On Fri, Jul 16, 2021 at 9:47 AM Mark Adams  wrote:
>>
>>> The simple Kokkos example is failing for me on Spock.
>>> Any ideas?
>>> Thanks,
>>>
>>> 10:44 main *=
>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make
>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
>>> PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k
>>> MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC
>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64
>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64
>>>  -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas
>>> -fstack-protector -fvisibility=hidden -g -O2   -fPIC -Wall -Wwrite-strings
>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3
>>> -std=c++14  -I/gpfs/alpine/csc314/scratch/adams/petsc/include
>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include
>>> -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include
>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
>>>  -I/sw/spock/spack-envs/views/rocm-4.1.0/includeex3k.kokkos.cxx
>>>  
>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib
>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib
>>> -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib
>>> -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib
>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64
>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib
>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse
>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64
>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray
>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu
>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread
>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64
>>> -lquadmath -lstdc++ -ldl -o ex3k
>>> In file included from ex3k.kokkos.cxx:3:
>>> In file included from
>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:
>>> In file included from
>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:
>>> In file included from
>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:
>>> In file included from
>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:
>>> In file included from
>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:
>>> In file included from
>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:
>>> In file included from
>>>

Re: [petsc-dev] Kokkos make error on Spock

2021-07-16 Thread Mark Adams 
Ha, Landau is working with the Kokkos backend and it is running the the GPU!

My Kokkos build was clean, but let me try it again.

On Fri, Jul 16, 2021 at 5:19 PM Junchao Zhang 
wrote:

> I don't understand this when linking ex19
>ld.lld: error:
> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib/libpetsc.so:
> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa
> [--no-allow-shlib-undefined]
> Do you have a stale libpetsc.so?
>
> For ex3k, it is a makefile problem (I am fixing). If you do not directly
> build an executable from *.kokkos.cxx, then you can avoid this problem.
>  For example,  snes/tests/ex13 works with kokkos options on Spock.
>
> --Junchao Zhang
>
>
> On Fri, Jul 16, 2021 at 2:53 PM Mark Adams  wrote:
>
>> Not complex. THis has some overlap with my problem w/o Kokkos.
>>
>> On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang 
>> wrote:
>>
>>> Do you use complex? post your configure.log.
>>>
>>> --Junchao Zhang
>>>
>>>
>>> On Fri, Jul 16, 2021 at 9:47 AM Mark Adams  wrote:
>>>
 The simple Kokkos example is failing for me on Spock.
 Any ideas?
 Thanks,

 10:44 main *=
 /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make
 PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
 PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k
 MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC
 -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64
 -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64
  -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas
 -fstack-protector -fvisibility=hidden -g -O2   -fPIC -Wall -Wwrite-strings
 -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3
 -std=c++14  -I/gpfs/alpine/csc314/scratch/adams/petsc/include
 -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include
 -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include
 -I/sw/spock/spack-envs/views/rocm-4.1.0/include
  -I/sw/spock/spack-envs/views/rocm-4.1.0/includeex3k.kokkos.cxx
  
 -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib
 -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib
 -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib
 -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib
 -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
 -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
 -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64
 -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
 -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
 -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
 -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
 -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
 -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib
 -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
 -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
 -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
 -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
 -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
 -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
 -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
 -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
 -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
 -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
 -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
 -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
 -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse
 -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64
 -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray
 -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu
 -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread
 -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64
 -lquadmath -lstdc++ -ldl -o ex3k
 In file included from ex3k.kokkos.cxx:3:
 In file included from
 /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:
 In file included from
 /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:
 In file included from
 /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:
 In file included from
 /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:
 In file included from
 /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:
 In file included from

Re: [petsc-dev] Kokkos make error on Spock

2021-07-16 Thread Junchao Zhang 
I don't understand this when linking ex19
   ld.lld: error:
/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib/libpetsc.so:
undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa
[--no-allow-shlib-undefined]
Do you have a stale libpetsc.so?

For ex3k, it is a makefile problem (I am fixing). If you do not directly
build an executable from *.kokkos.cxx, then you can avoid this problem.
 For example,  snes/tests/ex13 works with kokkos options on Spock.

--Junchao Zhang


On Fri, Jul 16, 2021 at 2:53 PM Mark Adams  wrote:

> Not complex. THis has some overlap with my problem w/o Kokkos.
>
> On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang 
> wrote:
>
>> Do you use complex? post your configure.log.
>>
>> --Junchao Zhang
>>
>>
>> On Fri, Jul 16, 2021 at 9:47 AM Mark Adams  wrote:
>>
>>> The simple Kokkos example is failing for me on Spock.
>>> Any ideas?
>>> Thanks,
>>>
>>> 10:44 main *=
>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make
>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
>>> PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k
>>> MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC
>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64
>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64
>>>  -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas
>>> -fstack-protector -fvisibility=hidden -g -O2   -fPIC -Wall -Wwrite-strings
>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3
>>> -std=c++14  -I/gpfs/alpine/csc314/scratch/adams/petsc/include
>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include
>>> -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include
>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
>>>  -I/sw/spock/spack-envs/views/rocm-4.1.0/includeex3k.kokkos.cxx
>>>  
>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib
>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib
>>> -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib
>>> -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib
>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64
>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib
>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse
>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64
>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray
>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu
>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread
>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64
>>> -lquadmath -lstdc++ -ldl -o ex3k
>>> In file included from ex3k.kokkos.cxx:3:
>>> In file included from
>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:
>>> In file included from
>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:
>>> In file included from
>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:
>>> In file included from
>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:
>>> In file included from
>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:
>>> In file included from
>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:
>>> In file included from
>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:
>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11:
>>> error: no member

Re: [petsc-dev] Kokkos make error on Spock

2021-07-16 Thread Junchao Zhang 
Do you use complex? post your configure.log.

--Junchao Zhang


On Fri, Jul 16, 2021 at 9:47 AM Mark Adams  wrote:

> The simple Kokkos example is failing for me on Spock.
> Any ideas?
> Thanks,
>
> 10:44 main *= /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$
> make PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
> PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k
> MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC
> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64
> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64
>  -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas
> -fstack-protector -fvisibility=hidden -g -O2   -fPIC -Wall -Wwrite-strings
> -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3
> -std=c++14  -I/gpfs/alpine/csc314/scratch/adams/petsc/include
> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include
> -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include
> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
>  -I/sw/spock/spack-envs/views/rocm-4.1.0/includeex3k.kokkos.cxx
>  
> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib
> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib
> -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib
> -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib
> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64
> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib
> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse
> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64
> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray
> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu
> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread
> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64
> -lquadmath -lstdc++ -ldl -o ex3k
> In file included from ex3k.kokkos.cxx:3:
> In file included from
> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:
> In file included from
> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:
> In file included from
> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:
> In file included from
> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:
> In file included from
> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:
> In file included from
> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:
> In file included from
> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:
> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11:
> error: no member named 'lock_address_host_space' in namespace
> 'Kokkos::Impl::Kokkos::Impl'; did you mean simply 'lock_address_host_space'?
>   while (!Impl::lock_address_host_space((void*)dest))
>   ^
>   lock_address_host_space
> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6:
> note: 'lock_address_host_space' declared here
> bool lock_address_host_space(void* ptr);
>  ^
> In file included from ex3k.kokkos.cxx:3:
> In file included from
> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:
> In file included from
> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:
> In file included from
> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:
> In file included from
>