Re: [OMPI users] OpenMPI 4.1.1, CentOS 7.9, nVidia HPC-SDk, build hints?

2021-09-29 Thread Gilles Gouaillardet via users 
Ray,

note there is a bug in nvc compilers since 21.3
(it has been reported and is documented at
https://github.com/open-mpi/ompi/issues/9402)

For the time being, I suggest you use gcc, g++ and nvfortran

FWIW, the AVX2 issue is likely caused by nvc **not** defining some macros
(that are both defined by at least GNU and LLVM compilers),
I will take a look at it when I get some time (you won't face this issue if
you use GNU compilers for C/C++)


Cheers,

Gilles

On Thu, Sep 30, 2021 at 2:31 AM Ray Muno via users 
wrote:

>
> Tried this
>
> configure   CC='nvc -fPIC' CXX='nvc++ -fPIC' FC='nvfortran -fPIC'
>
> Configure completes. Compiles quite a way through. Dies in a different
> place. It does get past the
> first error, however with libmpi_usempif08.la
>
>
>FCLD libmpi_usempif08.la
> make[2]: Leaving directory
>
> `/project/muno/OpenMPI/BUILD/4.1.1/ROME/NV-HPC/21.9/ompi/mpi/fortran/use-mpi-f08'
> Making all in mpi/fortran/mpiext-use-mpi-f08
> make[2]: Entering directory
>
> `/project/muno/OpenMPI/BUILD/4.1.1/ROME/NV-HPC/21.9/ompi/mpi/fortran/mpiext-use-mpi-f08'
>PPFC mpi-f08-ext-module.lo
>FCLD libforce_usempif08_module_to_be_built.la
> make[2]: Leaving directory
>
> `/project/muno/OpenMPI/BUILD/4.1.1/ROME/NV-HPC/21.9/ompi/mpi/fortran/mpiext-use-mpi-f08'
>
> Dies here now.
>
>   CCLD liblocal_ops_avx512.la
>CCLD mca_op_avx.la
> ./.libs/liblocal_ops_avx512.a(liblocal_ops_avx512_la-op_avx_functions.o):(.data+0x0):
> multiple
> definition of `ompi_op_avx_functions_avx2'
> ./.libs/liblocal_ops_avx2.a(liblocal_ops_avx2_la-op_avx_functions.o):(.data+0x0):
> first defined here
> ./.libs/liblocal_ops_avx512.a(liblocal_ops_avx512_la-op_avx_functions.o):
> In function
> `ompi_op_avx_2buff_min_uint16_t_avx2':
> /project/muno/OpenMPI/BUILD/SRC/openmpi-4.1.1/ompi/mca/op/avx/op_avx_functions.c:651:
> multiple
> definition of `ompi_op_avx_3buff_functions_avx2'
> ./.libs/liblocal_ops_avx2.a(liblocal_ops_avx2_la-op_avx_functions.o):/project/muno/OpenMPI/BUILD/SRC/openmpi-4.1.1/ompi/mca/op/avx/op_avx_functions.c:651:
>
> first defined here
> make[2]: *** [mca_op_avx.la] Error 2
> make[2]: Leaving directory
> `/project/muno/OpenMPI/BUILD/4.1.1/ROME/NV-HPC/21.9/ompi/mca/op/avx'
> make[1]: *** [all-recursive] Error 1
> make[1]: Leaving directory
> `/project/muno/OpenMPI/BUILD/4.1.1/ROME/NV-HPC/21.9/ompi'
> make: *** [all-recursive] Error 1
>
>
> On 9/29/21 11:42 AM, Bennet Fauber via users wrote:
> > Ray,
> >
> > If all the errors about not being compiled with -fPIC are still
> appearing, there may be a bug that
> > is preventing the option from getting through to the compiler(s).  It
> might be worth looking through
> > the logs to see the full compile command for one or more of them to see
> whether that is true?  Say,
> > libs/comm_spawn_multiple_f08.o for example?
> >
> > If -fPIC is missing, you may be able to recompile that manually with the
> -fPIC in place, then remake
> > and see if that also causes the link error to go away, that would be a
> good start.
> >
> > Hope this helps,-- bennet
> >
> >
> >
> > On Wed, Sep 29, 2021 at 12:29 PM Ray Muno via users <
> users@lists.open-mpi.org
> > > wrote:
> >
> > I did try that and it fails at the same place.
> >
> > Which version of the nVidia HPC-SDK are you using? I a m using
> 21.7.  I see there is an upgrade to
> > 21.9, which came out since I installed.  I have that installed and
> will try to see if they changed
> > anything. Not much in the releases notes to indicate any major
> changes.
> >
> > -Ray Muno
> >
> >
> > On 9/29/21 10:54 AM, Jing Gong wrote:
> >  > Hi,
> >  >
> >  >
> >  > Before Nvidia persons look into details,pProbably you can try to
> add the flag "-fPIC" to the
> >  > nvhpc compiler likes cc="nvc -fPIC", which at least worked with
> me.
> >  >
> >  >
> >  >
> >  > /Jing
> >  >
> >  >
> >
>  
> 
> >  > *From:* users  users-boun...@lists.open-mpi.org>> on
> > behalf of Ray Muno via users
> >  > mailto:users@lists.open-mpi.org>>
> >  > *Sent:* Wednesday, September 29, 2021 17:22
> >  > *To:* Open MPI User's List
> >  > *Cc:* Ray Muno
> >  > *Subject:* Re: [OMPI users] OpenMPI 4.1.1, CentOS 7.9, nVidia
> HPC-SDk, build hints?
> >  > Thanks, I looked through previous emails here in the user list.
> I guess I need to subscribe
> > to the
> >  > Developers list.
> >  >
> >  > -Ray Muno
> >  >
> >  > On 9/29/21 9:58 AM, Jeff Squyres (jsquyres) wrote:
> >  >> Ray --
> >  >>
> >  >> Looks like this is a dup of
> https://github.com/open-mpi/ompi/issues/8919
> >  <
> https://github.com/open-mpi/ompi/issues/8919
> > >
> >  >>

Re: [OMPI users] OpenMPI 4.1.1, CentOS 7.9, nVidia HPC-SDk, build hints?

2021-09-29 Thread Ray Muno via users 



Tried this

configure   CC='nvc -fPIC' CXX='nvc++ -fPIC' FC='nvfortran -fPIC'

Configure completes. Compiles quite a way through. Dies in a different place. It does get past the 
first error, however with libmpi_usempif08.la



  FCLD libmpi_usempif08.la
make[2]: Leaving directory 
`/project/muno/OpenMPI/BUILD/4.1.1/ROME/NV-HPC/21.9/ompi/mpi/fortran/use-mpi-f08'

Making all in mpi/fortran/mpiext-use-mpi-f08
make[2]: Entering directory 
`/project/muno/OpenMPI/BUILD/4.1.1/ROME/NV-HPC/21.9/ompi/mpi/fortran/mpiext-use-mpi-f08'

  PPFC mpi-f08-ext-module.lo
  FCLD libforce_usempif08_module_to_be_built.la
make[2]: Leaving directory 
`/project/muno/OpenMPI/BUILD/4.1.1/ROME/NV-HPC/21.9/ompi/mpi/fortran/mpiext-use-mpi-f08'


Dies here now.

 CCLD liblocal_ops_avx512.la
  CCLD mca_op_avx.la
./.libs/liblocal_ops_avx512.a(liblocal_ops_avx512_la-op_avx_functions.o):(.data+0x0): multiple 
definition of `ompi_op_avx_functions_avx2'

./.libs/liblocal_ops_avx2.a(liblocal_ops_avx2_la-op_avx_functions.o):(.data+0x0):
 first defined here
./.libs/liblocal_ops_avx512.a(liblocal_ops_avx512_la-op_avx_functions.o): In function 
`ompi_op_avx_2buff_min_uint16_t_avx2':
/project/muno/OpenMPI/BUILD/SRC/openmpi-4.1.1/ompi/mca/op/avx/op_avx_functions.c:651: multiple 
definition of `ompi_op_avx_3buff_functions_avx2'
./.libs/liblocal_ops_avx2.a(liblocal_ops_avx2_la-op_avx_functions.o):/project/muno/OpenMPI/BUILD/SRC/openmpi-4.1.1/ompi/mca/op/avx/op_avx_functions.c:651: 
first defined here

make[2]: *** [mca_op_avx.la] Error 2
make[2]: Leaving directory 
`/project/muno/OpenMPI/BUILD/4.1.1/ROME/NV-HPC/21.9/ompi/mca/op/avx'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory 
`/project/muno/OpenMPI/BUILD/4.1.1/ROME/NV-HPC/21.9/ompi'
make: *** [all-recursive] Error 1


On 9/29/21 11:42 AM, Bennet Fauber via users wrote:

Ray,

If all the errors about not being compiled with -fPIC are still appearing, there may be a bug that 
is preventing the option from getting through to the compiler(s).  It might be worth looking through 
the logs to see the full compile command for one or more of them to see whether that is true?  Say, 
libs/comm_spawn_multiple_f08.o for example?


If -fPIC is missing, you may be able to recompile that manually with the -fPIC in place, then remake 
and see if that also causes the link error to go away, that would be a good start.


Hope this helps,    -- bennet



On Wed, Sep 29, 2021 at 12:29 PM Ray Muno via users > wrote:


I did try that and it fails at the same place.

Which version of the nVidia HPC-SDK are you using? I a m using 21.7.  I see 
there is an upgrade to
21.9, which came out since I installed.  I have that installed and will try 
to see if they changed
anything. Not much in the releases notes to indicate any major changes.

-Ray Muno


On 9/29/21 10:54 AM, Jing Gong wrote:
 > Hi,
 >
 >
 > Before Nvidia persons look into details,pProbably you can try to add the flag 
"-fPIC" to the
 > nvhpc compiler likes cc="nvc -fPIC", which at least worked with me.
 >
 >
 >
 > /Jing
 >
 >


 > *From:* users mailto:users-boun...@lists.open-mpi.org>> on
behalf of Ray Muno via users
 > mailto:users@lists.open-mpi.org>>
 > *Sent:* Wednesday, September 29, 2021 17:22
 > *To:* Open MPI User's List
 > *Cc:* Ray Muno
 > *Subject:* Re: [OMPI users] OpenMPI 4.1.1, CentOS 7.9, nVidia HPC-SDk, 
build hints?
 > Thanks, I looked through previous emails here in the user list.  I guess 
I need to subscribe
to the
 > Developers list.
 >
 > -Ray Muno
 >
 > On 9/29/21 9:58 AM, Jeff Squyres (jsquyres) wrote:
 >> Ray --
 >>
 >> Looks like this is a dup of https://github.com/open-mpi/ompi/issues/8919
 
>
 >> 
>>.
 >>
 >>

 >
-- 


   Ray Muno
   IT Systems Administrator
   e-mail: m...@umn.edu 

   University of Minnesota
   Aerospace Engineering and Mechanics




--

 Ray Muno
 IT Systems Administrator
 e-mail:   m...@umn.edu
 Phone:   (612) 625-9531

 University of Minnesota
 Aerospace Engineering and Mechanics
 110 Union St. S.E.
 Minneapolis, MN 55455

Re: [OMPI users] OpenMPI 4.1.1, CentOS 7.9, nVidia HPC-SDk, build hints?

2021-09-29 Thread Ray Muno via users 

In config.log, it appears to be passing the flag in the tests.

It also fails with version 21.9 of the HPC-SDK.

On 9/29/21 11:42 AM, Bennet Fauber via users wrote:

Ray,

If all the errors about not being compiled with -fPIC are still appearing, there may be a bug that 
is preventing the option from getting through to the compiler(s).  It might be worth looking through 
the logs to see the full compile command for one or more of them to see whether that is true?  Say, 
libs/comm_spawn_multiple_f08.o for example?


If -fPIC is missing, you may be able to recompile that manually with the -fPIC in place, then remake 
and see if that also causes the link error to go away, that would be a good start.


Hope this helps,    -- bennet



On Wed, Sep 29, 2021 at 12:29 PM Ray Muno via users > wrote:


I did try that and it fails at the same place.

Which version of the nVidia HPC-SDK are you using? I a m using 21.7.  I see 
there is an upgrade to
21.9, which came out since I installed.  I have that installed and will try 
to see if they changed
anything. Not much in the releases notes to indicate any major changes.

-Ray Muno


On 9/29/21 10:54 AM, Jing Gong wrote:
 > Hi,
 >
 >
 > Before Nvidia persons look into details,pProbably you can try to add the flag 
"-fPIC" to the
 > nvhpc compiler likes cc="nvc -fPIC", which at least worked with me.
 >
 >
 >
 > /Jing
 >
 >


 > *From:* users mailto:users-boun...@lists.open-mpi.org>> on
behalf of Ray Muno via users
 > mailto:users@lists.open-mpi.org>>
 > *Sent:* Wednesday, September 29, 2021 17:22
 > *To:* Open MPI User's List
 > *Cc:* Ray Muno
 > *Subject:* Re: [OMPI users] OpenMPI 4.1.1, CentOS 7.9, nVidia HPC-SDk, 
build hints?
 > Thanks, I looked through previous emails here in the user list.  I guess 
I need to subscribe
to the
 > Developers list.
 >
 > -Ray Muno
 >
 > On 9/29/21 9:58 AM, Jeff Squyres (jsquyres) wrote:
 >> Ray --
 >>
 >> Looks like this is a dup of https://github.com/open-mpi/ompi/issues/8919
 
>
 >> 
>>.
 >>
 >>

 >
-- 


   Ray Muno
   IT Systems Administrator
   e-mail: m...@umn.edu 

   University of Minnesota
   Aerospace Engineering and Mechanics




--

 Ray Muno
 IT Systems Administrator
 e-mail:   m...@umn.edu
 Phone:   (612) 625-9531

 University of Minnesota
 Aerospace Engineering and Mechanics
 110 Union St. S.E.
 Minneapolis, MN 55455

Re: [OMPI users] OpenMPI 4.1.1, CentOS 7.9, nVidia HPC-SDk, build hints?

2021-09-29 Thread Bennet Fauber via users 
Ray,

If all the errors about not being compiled with -fPIC are still appearing,
there may be a bug that is preventing the option from getting through to
the compiler(s).  It might be worth looking through the logs to see the
full compile command for one or more of them to see whether that is true?
Say, libs/comm_spawn_multiple_f08.o for example?

If -fPIC is missing, you may be able to recompile that manually with the
-fPIC in place, then remake and see if that also causes the link error to
go away, that would be a good start.

Hope this helps,-- bennet



On Wed, Sep 29, 2021 at 12:29 PM Ray Muno via users <
users@lists.open-mpi.org> wrote:

> I did try that and it fails at the same place.
>
> Which version of the nVidia HPC-SDK are you using? I a m using 21.7.  I
> see there is an upgrade to
> 21.9, which came out since I installed.  I have that installed and will
> try to see if they changed
> anything. Not much in the releases notes to indicate any major changes.
>
> -Ray Muno
>
>
> On 9/29/21 10:54 AM, Jing Gong wrote:
> > Hi,
> >
> >
> > Before Nvidia persons look into details,pProbably you can try to add the
> flag "-fPIC" to the
> > nvhpc compiler likes cc="nvc -fPIC", which at least worked with me.
> >
> >
> >
> > /Jing
> >
> >
> 
> > *From:* users  on behalf of Ray Muno
> via users
> > 
> > *Sent:* Wednesday, September 29, 2021 17:22
> > *To:* Open MPI User's List
> > *Cc:* Ray Muno
> > *Subject:* Re: [OMPI users] OpenMPI 4.1.1, CentOS 7.9, nVidia HPC-SDk,
> build hints?
> > Thanks, I looked through previous emails here in the user list.  I guess
> I need to subscribe to the
> > Developers list.
> >
> > -Ray Muno
> >
> > On 9/29/21 9:58 AM, Jeff Squyres (jsquyres) wrote:
> >> Ray --
> >>
> >> Looks like this is a dup of
> https://github.com/open-mpi/ompi/issues/8919 <
> https://github.com/open-mpi/ompi/issues/8919>
> >>  https://github.com/open-mpi/ompi/issues/8919>>.
> >>
> >>
>
> >
> --
>
>   Ray Muno
>   IT Systems Administrator
>   e-mail:   m...@umn.edu
>
>   University of Minnesota
>   Aerospace Engineering and Mechanics
>
>

Re: [OMPI users] OpenMPI 4.1.1, CentOS 7.9, nVidia HPC-SDk, build hints?

2021-09-29 Thread Ray Muno via users 

I did try that and it fails at the same place.

Which version of the nVidia HPC-SDK are you using? I a m using 21.7.  I see there is an upgrade to 
21.9, which came out since I installed.  I have that installed and will try to see if they changed 
anything. Not much in the releases notes to indicate any major changes.


-Ray Muno


On 9/29/21 10:54 AM, Jing Gong wrote:

Hi,


Before Nvidia persons look into details,pProbably you can try to add the flag "-fPIC" to the 
nvhpc compiler likes cc="nvc -fPIC", which at least worked with me.




/Jing


*From:* users  on behalf of Ray Muno via users 


*Sent:* Wednesday, September 29, 2021 17:22
*To:* Open MPI User's List
*Cc:* Ray Muno
*Subject:* Re: [OMPI users] OpenMPI 4.1.1, CentOS 7.9, nVidia HPC-SDk, build 
hints?
Thanks, I looked through previous emails here in the user list.  I guess I need 
to subscribe to the
Developers list.

-Ray Muno

On 9/29/21 9:58 AM, Jeff Squyres (jsquyres) wrote:

Ray --

Looks like this is a dup of https://github.com/open-mpi/ompi/issues/8919 

>.







--

 Ray Muno
 IT Systems Administrator
 e-mail:   m...@umn.edu

 University of Minnesota
 Aerospace Engineering and Mechanics

Re: [OMPI users] OpenMPI 4.1.1, CentOS 7.9, nVidia HPC-SDk, build hints?

2021-09-29 Thread Ray Muno via users 

On 9/29/21 9:58 AM, Jeff Squyres (jsquyres) wrote:

Ray --

Looks like this is a dup of https://github.com/open-mpi/ompi/issues/8919 
.





Looking at the page on the OpenMPI Developer list, it says

"A known workaround is to add '-fPIC' to the CFLAGS, CXXFLAGS, FCFLAGS (maybe not need to all of 
those)."


Tried adding these, still fails at the same place.



--

 Ray Muno
 IT Systems Administrator
 e-mail:   m...@umn.edu

 University of Minnesota
 Aerospace Engineering and Mechanics

Re: [OMPI users] OpenMPI 4.1.1, CentOS 7.9, nVidia HPC-SDk, build hints?

2021-09-29 Thread Jing Gong via users 
Hi,


Before Nvidia persons look into details,pProbably you can try to add the flag 
"-fPIC" to the nvhpc compiler likes cc="nvc -fPIC", which at least worked with 
me.


/Jing


From: users  on behalf of Ray Muno via users 

Sent: Wednesday, September 29, 2021 17:22
To: Open MPI User's List
Cc: Ray Muno
Subject: Re: [OMPI users] OpenMPI 4.1.1, CentOS 7.9, nVidia HPC-SDk, build 
hints?

Thanks, I looked through previous emails here in the user list.  I guess I need 
to subscribe to the
Developers list.

-Ray Muno

On 9/29/21 9:58 AM, Jeff Squyres (jsquyres) wrote:
> Ray --
>
> Looks like this is a dup of https://github.com/open-mpi/ompi/issues/8919
> .
>
>
>> On Sep 29, 2021, at 10:47 AM, Ray Muno via users > > wrote:
>>
>> Looking to compile
>>
>> OpenMPI 4.1.1
>>
>> under Centos 7.9, (with Mellanox OFED 5.3 stack)
>>
>> with the nVidia HPC-SDK, version 21.7.
>>
>> Configure works, build fails.
>>
>> The nvidia HPC-SDK is using their environmental module which sets these 
>> variables, etc.
>>
>>
>> setenvNVHPC /stage/opt/NV_hpc_sdk
>> setenvCC /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/bin/nvc
>> setenvCXX /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/bin/nvc++
>> setenvFC /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/bin/nvfortran
>> setenvF90 /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/bin/nvfortran
>> setenvF77 /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/bin/nvfortran
>> setenvCPP cpp
>> prepend-pathPATH /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/cuda/bin
>> prepend-pathPATH /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/bin
>> prepend-pathPATH 
>> /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/extras/qd/bin
>> prepend-pathLD_LIBRARY_PATH 
>> /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/cuda/lib64
>> prepend-pathLD_LIBRARY_PATH 
>> /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/cuda/extras/CUPTI/lib64
>> prepend-pathLD_LIBRARY_PATH 
>> /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/extras/qd/lib
>> prepend-pathLD_LIBRARY_PATH 
>> /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/lib
>> prepend-pathLD_LIBRARY_PATH 
>> /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/math_libs/lib64
>> prepend-pathLD_LIBRARY_PATH 
>> /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/comm_libs/nccl/lib
>> prepend-pathLD_LIBRARY_PATH 
>> /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/comm_libs/nvshmem/lib
>> prepend-pathCPATH /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/math_libs/include
>> prepend-pathCPATH 
>> /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/comm_libs/nccl/include
>> prepend-pathCPATH 
>> /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/comm_libs/nvshmem/include
>> prepend-pathCPATH 
>> /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/extras/qd/include/qd
>> prepend-pathMANPATH /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/man
>> ---
>>
>>
>> configure  --prefix=/stage/opt/OpenMPI/ROME/4.1.1/NV-HPC/21.7 CC=nvc 
>> CXX=nvc++ FC=nvfortran
>>
>> ...
>> FCLD libmpi_usempif08.la 
>> /usr/bin/ld: .libs/comm_spawn_multiple_f08.o: relocation R_X86_64_32S 
>> against `.rodata' can not be
>> used when making a shared object; recompile with -fPIC
>> /usr/bin/ld: .libs/startall_f08.o: relocation R_X86_64_32S against `.rodata' 
>> can not be used when
>> making a shared object; recompile with -fPIC
>> /usr/bin/ld: .libs/testall_f08.o: relocation R_X86_64_32S against `.rodata' 
>> can not be used when
>> making a shared object; recompile with -fPIC
>> /usr/bin/ld: .libs/testany_f08.o: relocation R_X86_64_32S against `.rodata' 
>> can not be used when
>> making a shared object; recompile with -fPIC
>> /usr/bin/ld: .libs/testsome_f08.o: relocation R_X86_64_32S against `.rodata' 
>> can not be used when
>> making a shared object; recompile with -fPIC
>> /usr/bin/ld: .libs/type_create_struct_f08.o: relocation R_X86_64_32S against 
>> `.rodata' can not be
>> used when making a shared object; recompile with -fPIC
>> /usr/bin/ld: .libs/type_get_contents_f08.o: relocation R_X86_64_32S against 
>> `.rodata' can not be
>> used when making a shared object; recompile with -fPIC
>> /usr/bin/ld: .libs/waitall_f08.o: relocation R_X86_64_32S against `.rodata' 
>> can not be used when
>> making a shared object; recompile with -fPIC
>> /usr/bin/ld: .libs/waitany_f08.o: relocation R_X86_64_32S against `.rodata' 
>> can not be used when
>> making a shared object; recompile with -fPIC
>> /usr/bin/ld: .libs/waitsome_f08.o: relocation R_X86_64_32S against `.rodata' 
>> can not be used when
>> making a shared object; recompile with -fPIC
>> /usr/bin/ld: profile/.libs/pcomm_spawn_multiple_f08.o: relocation 
>> R_X86_64_32S against `.rodata'
>> can not be used when making a shared object; recompile with -fPIC
>> /usr/bin/ld: profile/.libs/pstartall_f08.o: relocation R_X86_64_32S against 
>> `.rodata' can not be
>> used when making a shared object; recompile with

Re: [OMPI users] OpenMPI 4.1.1, CentOS 7.9, nVidia HPC-SDk, build hints?

2021-09-29 Thread Ray Muno via users 
Thanks, I looked through previous emails here in the user list.  I guess I need to subscribe to the 
Developers list.


-Ray Muno

On 9/29/21 9:58 AM, Jeff Squyres (jsquyres) wrote:

Ray --

Looks like this is a dup of https://github.com/open-mpi/ompi/issues/8919 
.




On Sep 29, 2021, at 10:47 AM, Ray Muno via users mailto:users@lists.open-mpi.org>> wrote:

Looking to compile

OpenMPI 4.1.1

under Centos 7.9, (with Mellanox OFED 5.3 stack)

with the nVidia HPC-SDK, version 21.7.

Configure works, build fails.

The nvidia HPC-SDK is using their environmental module which sets these 
variables, etc.


setenvNVHPC /stage/opt/NV_hpc_sdk
setenvCC /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/bin/nvc
setenvCXX /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/bin/nvc++
setenvFC /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/bin/nvfortran
setenvF90 /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/bin/nvfortran
setenvF77 /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/bin/nvfortran
setenvCPP cpp
prepend-pathPATH /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/cuda/bin
prepend-pathPATH /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/bin
prepend-pathPATH /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/extras/qd/bin
prepend-pathLD_LIBRARY_PATH /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/cuda/lib64
prepend-pathLD_LIBRARY_PATH 
/stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/cuda/extras/CUPTI/lib64
prepend-pathLD_LIBRARY_PATH 
/stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/extras/qd/lib
prepend-pathLD_LIBRARY_PATH 
/stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/lib
prepend-pathLD_LIBRARY_PATH 
/stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/math_libs/lib64
prepend-pathLD_LIBRARY_PATH 
/stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/comm_libs/nccl/lib
prepend-pathLD_LIBRARY_PATH 
/stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/comm_libs/nvshmem/lib
prepend-pathCPATH /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/math_libs/include
prepend-pathCPATH /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/comm_libs/nccl/include
prepend-pathCPATH 
/stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/comm_libs/nvshmem/include
prepend-pathCPATH 
/stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/extras/qd/include/qd
prepend-pathMANPATH /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/man
---


configure  --prefix=/stage/opt/OpenMPI/ROME/4.1.1/NV-HPC/21.7 CC=nvc CXX=nvc++ 
FC=nvfortran

...
FCLD libmpi_usempif08.la 
/usr/bin/ld: .libs/comm_spawn_multiple_f08.o: relocation R_X86_64_32S against `.rodata' can not be 
used when making a shared object; recompile with -fPIC
/usr/bin/ld: .libs/startall_f08.o: relocation R_X86_64_32S against `.rodata' can not be used when 
making a shared object; recompile with -fPIC
/usr/bin/ld: .libs/testall_f08.o: relocation R_X86_64_32S against `.rodata' can not be used when 
making a shared object; recompile with -fPIC
/usr/bin/ld: .libs/testany_f08.o: relocation R_X86_64_32S against `.rodata' can not be used when 
making a shared object; recompile with -fPIC
/usr/bin/ld: .libs/testsome_f08.o: relocation R_X86_64_32S against `.rodata' can not be used when 
making a shared object; recompile with -fPIC
/usr/bin/ld: .libs/type_create_struct_f08.o: relocation R_X86_64_32S against `.rodata' can not be 
used when making a shared object; recompile with -fPIC
/usr/bin/ld: .libs/type_get_contents_f08.o: relocation R_X86_64_32S against `.rodata' can not be 
used when making a shared object; recompile with -fPIC
/usr/bin/ld: .libs/waitall_f08.o: relocation R_X86_64_32S against `.rodata' can not be used when 
making a shared object; recompile with -fPIC
/usr/bin/ld: .libs/waitany_f08.o: relocation R_X86_64_32S against `.rodata' can not be used when 
making a shared object; recompile with -fPIC
/usr/bin/ld: .libs/waitsome_f08.o: relocation R_X86_64_32S against `.rodata' can not be used when 
making a shared object; recompile with -fPIC
/usr/bin/ld: profile/.libs/pcomm_spawn_multiple_f08.o: relocation R_X86_64_32S against `.rodata' 
can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: profile/.libs/pstartall_f08.o: relocation R_X86_64_32S against `.rodata' can not be 
used when making a shared object; recompile with -fPIC
/usr/bin/ld: profile/.libs/ptestall_f08.o: relocation R_X86_64_32S against `.rodata' can not be 
used when making a shared object; recompile with -fPIC
/usr/bin/ld: profile/.libs/ptestany_f08.o: relocation R_X86_64_32S against `.rodata' can not be 
used when making a shared object; recompile with -fPIC
/usr/bin/ld: profile/.libs/ptestsome_f08.o: relocation R_X86_64_32S against `.rodata' can not be 
used when making a shared object; recompile with -fPIC
/usr/bin/ld: profile/.libs/ptype_create_struct_f08.o: relocation R_X86_64_32S against `.rodata' 
can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: profile/.libs/ptype_get_contents_f08.o: relocation R_X86_64_32S

Re: [OMPI users] OpenMPI 4.1.1, CentOS 7.9, nVidia HPC-SDk, build hints?

2021-09-29 Thread Jeff Squyres (jsquyres) via users 
Ray --

Looks like this is a dup of https://github.com/open-mpi/ompi/issues/8919.


On Sep 29, 2021, at 10:47 AM, Ray Muno via users 
mailto:users@lists.open-mpi.org>> wrote:

Looking to compile

OpenMPI 4.1.1

under Centos 7.9, (with Mellanox OFED 5.3 stack)

with the nVidia HPC-SDK, version 21.7.

Configure works, build fails.

The nvidia HPC-SDK is using their environmental module which sets these 
variables, etc.


setenv NVHPC /stage/opt/NV_hpc_sdk
setenv CC /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/bin/nvc
setenv CXX /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/bin/nvc++
setenv FC /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/bin/nvfortran
setenv F90 /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/bin/nvfortran
setenv F77 /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/bin/nvfortran
setenv CPP cpp
prepend-path PATH /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/cuda/bin
prepend-path PATH /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/bin
prepend-path PATH 
/stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/extras/qd/bin
prepend-path LD_LIBRARY_PATH /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/cuda/lib64
prepend-path LD_LIBRARY_PATH 
/stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/cuda/extras/CUPTI/lib64
prepend-path LD_LIBRARY_PATH 
/stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/extras/qd/lib
prepend-path LD_LIBRARY_PATH 
/stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/lib
prepend-path LD_LIBRARY_PATH 
/stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/math_libs/lib64
prepend-path LD_LIBRARY_PATH 
/stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/comm_libs/nccl/lib
prepend-path LD_LIBRARY_PATH 
/stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/comm_libs/nvshmem/lib
prepend-path CPATH /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/math_libs/include
prepend-path CPATH 
/stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/comm_libs/nccl/include
prepend-path CPATH 
/stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/comm_libs/nvshmem/include
prepend-path CPATH 
/stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/extras/qd/include/qd
prepend-path MANPATH /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/man
---


configure  --prefix=/stage/opt/OpenMPI/ROME/4.1.1/NV-HPC/21.7 CC=nvc CXX=nvc++ 
FC=nvfortran

...
FCLD libmpi_usempif08.la
/usr/bin/ld: .libs/comm_spawn_multiple_f08.o: relocation R_X86_64_32S against 
`.rodata' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: .libs/startall_f08.o: relocation R_X86_64_32S against `.rodata' 
can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: .libs/testall_f08.o: relocation R_X86_64_32S against `.rodata' can 
not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: .libs/testany_f08.o: relocation R_X86_64_32S against `.rodata' can 
not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: .libs/testsome_f08.o: relocation R_X86_64_32S against `.rodata' 
can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: .libs/type_create_struct_f08.o: relocation R_X86_64_32S against 
`.rodata' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: .libs/type_get_contents_f08.o: relocation R_X86_64_32S against 
`.rodata' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: .libs/waitall_f08.o: relocation R_X86_64_32S against `.rodata' can 
not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: .libs/waitany_f08.o: relocation R_X86_64_32S against `.rodata' can 
not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: .libs/waitsome_f08.o: relocation R_X86_64_32S against `.rodata' 
can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: profile/.libs/pcomm_spawn_multiple_f08.o: relocation R_X86_64_32S 
against `.rodata' can not be used when making a shared object; recompile with 
-fPIC
/usr/bin/ld: profile/.libs/pstartall_f08.o: relocation R_X86_64_32S against 
`.rodata' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: profile/.libs/ptestall_f08.o: relocation R_X86_64_32S against 
`.rodata' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: profile/.libs/ptestany_f08.o: relocation R_X86_64_32S against 
`.rodata' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: profile/.libs/ptestsome_f08.o: relocation R_X86_64_32S against 
`.rodata' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: profile/.libs/ptype_create_struct_f08.o: relocation R_X86_64_32S 
against `.rodata' can not be used when making a shared object; recompile with 
-fPIC
/usr/bin/ld: profile/.libs/ptype_get_contents_f08.o: relocation R_X86_64_32S 
against `.rodata' can not be used when making a shared object; recompile with 
-fPIC
/usr/bin/ld: profile/.libs/pwaitall_f08.o: relocation R_X86_64_32S against 
`.rodata' can not be used when making a

[OMPI users] OpenMPI 4.1.1, CentOS 7.9, nVidia HPC-SDk, build hints?

2021-09-29 Thread Ray Muno via users 

Looking to compile

OpenMPI 4.1.1

under Centos 7.9, (with Mellanox OFED 5.3 stack)

with the nVidia HPC-SDK, version 21.7.

Configure works, build fails.

The nvidia HPC-SDK is using their environmental module which sets these 
variables, etc.


setenv   NVHPC /stage/opt/NV_hpc_sdk
setenv   CC /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/bin/nvc
setenv   CXX /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/bin/nvc++
setenv   FC 
/stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/bin/nvfortran
setenv   F90 
/stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/bin/nvfortran
setenv   F77 
/stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/bin/nvfortran
setenv   CPP cpp
prepend-path PATH /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/cuda/bin
prepend-path PATH /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/bin
prepend-path PATH 
/stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/extras/qd/bin
prepend-path LD_LIBRARY_PATH 
/stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/cuda/lib64
prepend-path LD_LIBRARY_PATH 
/stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/cuda/extras/CUPTI/lib64
prepend-path LD_LIBRARY_PATH 
/stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/extras/qd/lib
prepend-path LD_LIBRARY_PATH 
/stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/lib
prepend-path LD_LIBRARY_PATH 
/stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/math_libs/lib64
prepend-path LD_LIBRARY_PATH 
/stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/comm_libs/nccl/lib
prepend-path LD_LIBRARY_PATH 
/stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/comm_libs/nvshmem/lib
prepend-path CPATH /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/math_libs/include
prepend-path CPATH 
/stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/comm_libs/nccl/include
prepend-path CPATH 
/stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/comm_libs/nvshmem/include
prepend-path CPATH 
/stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/extras/qd/include/qd
prepend-path MANPATH /stage/opt/NV_hpc_sdk/Linux_x86_64/21.7/compilers/man
---


configure  --prefix=/stage/opt/OpenMPI/ROME/4.1.1/NV-HPC/21.7 CC=nvc CXX=nvc++ 
FC=nvfortran

...
 FCLD libmpi_usempif08.la
/usr/bin/ld: .libs/comm_spawn_multiple_f08.o: relocation R_X86_64_32S against `.rodata' can not be 
used when making a shared object; recompile with -fPIC
/usr/bin/ld: .libs/startall_f08.o: relocation R_X86_64_32S against `.rodata' can not be used when 
making a shared object; recompile with -fPIC
/usr/bin/ld: .libs/testall_f08.o: relocation R_X86_64_32S against `.rodata' can not be used when 
making a shared object; recompile with -fPIC
/usr/bin/ld: .libs/testany_f08.o: relocation R_X86_64_32S against `.rodata' can not be used when 
making a shared object; recompile with -fPIC
/usr/bin/ld: .libs/testsome_f08.o: relocation R_X86_64_32S against `.rodata' can not be used when 
making a shared object; recompile with -fPIC
/usr/bin/ld: .libs/type_create_struct_f08.o: relocation R_X86_64_32S against `.rodata' can not be 
used when making a shared object; recompile with -fPIC
/usr/bin/ld: .libs/type_get_contents_f08.o: relocation R_X86_64_32S against `.rodata' can not be 
used when making a shared object; recompile with -fPIC
/usr/bin/ld: .libs/waitall_f08.o: relocation R_X86_64_32S against `.rodata' can not be used when 
making a shared object; recompile with -fPIC
/usr/bin/ld: .libs/waitany_f08.o: relocation R_X86_64_32S against `.rodata' can not be used when 
making a shared object; recompile with -fPIC
/usr/bin/ld: .libs/waitsome_f08.o: relocation R_X86_64_32S against `.rodata' can not be used when 
making a shared object; recompile with -fPIC
/usr/bin/ld: profile/.libs/pcomm_spawn_multiple_f08.o: relocation R_X86_64_32S against `.rodata' can 
not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: profile/.libs/pstartall_f08.o: relocation R_X86_64_32S against `.rodata' can not be 
used when making a shared object; recompile with -fPIC
/usr/bin/ld: profile/.libs/ptestall_f08.o: relocation R_X86_64_32S against `.rodata' can not be used 
when making a shared object; recompile with -fPIC
/usr/bin/ld: profile/.libs/ptestany_f08.o: relocation R_X86_64_32S against `.rodata' can not be used 
when making a shared object; recompile with -fPIC
/usr/bin/ld: profile/.libs/ptestsome_f08.o: relocation R_X86_64_32S against `.rodata' can not be 
used when making a shared object; recompile with -fPIC
/usr/bin/ld: profile/.libs/ptype_create_struct_f08.o: relocation R_X86_64_32S against `.rodata' can 
not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: profile/.libs/ptype_get_contents_f08.o: relocation R_X86_64_32S against `.rodata' can 
not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: profile/.libs/pwaitall_f08.o: relocation R_X86_64_32S against `.rodata' can not be used 
when making a shared object; recompile with -fPIC
/usr/bin/ld: