Re: [OMPI users] [EXTERNAL] Confusions on building and running OpenMPI over Slingshot 10 on Cray EX HPC

[Pritchard Jr., Howard via users]

Hi Jerry,

Cray EX HPC with slingshot 10 (NOT 11!!!) is basically a Mellanox IB cluster 
using RoCE rather than IB.
For this sort of interconnect, don’t use OFI, use UCX.  Although UCX 1.12.0 is 
getting a bit old.
I’d recommend 1.14.0 or newer, esp. if your system has nodes with GPUs.

CXI is the name of the vendor libfabric provider and doesn’t function on this 
system – unless parts of the cluster are wired up with slingshot 11 (nics).
For the node where you ran lspci this doesn’t seem to be the case.  You’d see 
something like this if you had Slingshot 11:

27:00.0 Ethernet controller: Cray Inc Device 0501 (rev 02)
a8:00.0 Ethernet controller: Cray Inc Device 0501 (rev 02)


For your first question, you want to double check the final output from a 
configure run and make sure that the summary says UCX support is enabled.

Please see 
https://docs.open-mpi.org/en/v5.0.x/tuning-apps/networking/ib-and-roce.html for 
answers to some of your other questions below.
Note there are some RoCE specific items in the doc page you may want to check.

The PMIx slingshot config option is getting you confused. Just ignore it for 
this network.

I’d suggest tweaking your configure options to the following:

  --enable-mpi-fortran \
  --enable-shared \
  --with-pic \
  --with-ofi=no \
  --with-ucx=/project/app/ucx/1.12.1 \
  --with-pmix=internal \
  --with-pbs \
  --with-tm=/opt/pbs \
  --with-singularity=/project/app/singularity/3.10.3 \
  --with-lustre=/usr \
  CC=icc \
  FC=ifort \
  CXX=icpc

This will end up with a build of Open MPI that uses UCX – which is what you 
want.

You are getting the error message from the btl framework because the OFI BTL 
can’t find a suitable/workable OFI provider.

If you really need to build with OFI support, add –with-ofi, but set the 
following MCA parameters (here shown using env. Variables):

export OMPI_MCA_pml=ucx
export OMPI_MCA_osc=ucx
export OMPI_MCA_btl=^ofi

when running applications built using this Open MPI installation.

Hope this helps,

Howard


From: users  on behalf of Jianyu Liu via 
users 
Reply-To: Open MPI Users 
Date: Wednesday, May 8, 2024 at 7:41 PM
To: "users@lists.open-mpi.org" 
Cc: Jianyu Liu 
Subject: [EXTERNAL] [OMPI users] Confusions on building and running OpenMPI 
over Slingshot 10 on Cray EX HPC


Hi,

I'm trying to build an OpenMPI 5.0.3 environment on the Cray EX HPC with 
Slingshot 10 support.

General speaking,  there were error messages while building OpenMPI,  and make 
check also didn't report any failure.

While tested OpenMPI Env. with a simple 'hello world' MPI Fortran codes,  it 
threw out these error messages and caught  signal 11 with libucs if specified 
'-mca btl ofi'.

No components were able to be opened in the btl framework.

This typically means that either no components of this type were
installed, or none of the installed components can be loaded.
Sometimes this means that shared libraries required by these
components are unable to be found/loaded

Host: x3001c027b4n0
Framework: btl
-
Caught signal 11 ( Segmentation fault: address not mapped to object at address 
(nil))

/project/app/ucx/1.12.1/lib/libucs.so.0 (ucs_handle_error+0x134)


This made me confused and not sure if got OpenMPI built with full Slingshot 10 
support successfully and run over Slingshot 10 properly.


Here are the building env.  on Cray EX HPC with SLES 15 SP3

OpenMPI 5.0.3 + Intel 2022.0.2 + UCX 1.12.1 + libfabric 
1.11.0.4.125-SSHOT2.0.0 + mlnx-ofed 5.5.1

Here are my configurations

  --enable-mpi-fortran \
  --enable-shared \
  --with-pic \
  --with-ofi=/opt/cray/libfabric/1.11.0.4.125 \
  --with-ofi-libdir=/opt/cray/libfabric/1.11.0.4.125/lib64 \
  --with-ucx=/project/app/ucx/1.12.1 \
  --with-pmix=internal \
  --with-slingshot \
  --with-pbs \
  --with-tm=/opt/pbs \
  --with-singularity=/project/app/singularity/3.10.3 \
  --with-lustre=/usr \
  CC=icc \
  FC=ifort \
  CXX=icpc

Here are output of lspci on computing nodes

03:00.0 Ethernet controller: Mellanox Technologies MT27800 Family 
[ConnectX-5]
24:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network 
Connection (rev 01)

Here are what I'm confusing

  1. After the configuration completed, the pmix summary didn't tell slingshot 
support is turned on for the transports
  2. config.log didn't show any checking info. against slingshot while 
conducting mca checking,  just showed --with-slingshot was passed as an 
argument.
  3. Further looked into the configure script,  the only script which will 
check Slingshot support is 3rd-party/openmix/src/mca/pnet/sshot/configure.m4,  
but looked like it's never called,  as config.log didn't show any checking 
info. against appropriate dependencies, such as CXI, JANSSON, and I believed 
that CXI library was not installed on the machine.


Here are my questions

1.  How it could tell OpenMPI was built with full Slingshot 10 support 

Re: [OMPI users] [EXTERNAL] Helping interpreting error output

[Pritchard Jr., Howard via users]

Hi Jeffrey,

I would suggest trying to debug what may be going wrong with UCX on your DGX 
box.

There are several things to try from the UCX faq - 
https://openucx.readthedocs.io/en/master/faq.html

I’d suggest setting the UCX_LOG_LEVEL environment variable to info or debug and 
see if UCX says something about what’s going wrong.

Also add --mca plm_base_verbose 10 to the mpirun command line.

Have you used DGX boxes with only a single NIC successfully?

Howard


From: users  on behalf of Jeffrey Layton via 
users 
Reply-To: Open MPI Users 
Date: Tuesday, April 16, 2024 at 12:30 PM
To: Open MPI Users 
Cc: Jeffrey Layton 
Subject: [EXTERNAL] [OMPI users] Helping interpreting error output

Good afternoon MPI fans of all ages,

Yet again, I'm getting an error that I'm having trouble interpreting. This 
time, I'm trying to run ior. I've done it a thousand times but not on an NVIDIA 
DGX A100 with multiple NICs.

The ultimate command is the following:


/cm/shared/apps/openmpi4/gcc/4.1.5/bin/mpirun --mca btl '^openib' -np 4 -map-by 
ppr:4:node --allow-run-as-root --mca btl_openib_warn_default_gid_prefix 0 --mca 
btl_openib_if_exclude mlx5_0,mlx5_5,mlx5_6 --mca plm_base_verbose 0 --mca plm 
rsh /home/bcm/bin/bin/ior -w -r -z -e -C -t 1m -b 1g -s 1000 -o /mnt/test


It was suggested to me to use these MPI options. The error I get is the 
following.

--
A requested component was not found, or was unable to be opened.  This
means that this component is either not installed or is unable to be
used on your system (e.g., sometimes this means that shared libraries
that the component requires are unable to be found/loaded).  Note that
Open MPI stopped checking at the first component that it did not find.

Host:  dgx-02
Framework: pml
Component: ucx
--
--
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems.  This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):

  mca_pml_base_open() failed
  --> Returned "Not found" (-13) instead of "Success" (0)
--
[dgx-02:2399932] *** An error occurred in MPI_Init
[dgx-02:2399932] *** reported by process [2099773441,3]
[dgx-02:2399932] *** on a NULL communicator
[dgx-02:2399932] *** Unknown error
[dgx-02:2399932] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will 
now abort,
[dgx-02:2399932] ***and potentially your MPI job)


My first inclination was that it couldn't find ucx. So I loaded that module and 
re-ran it. I get the exact same error message. I'm still checking if the ucx 
module gets loaded when I run via Slurm, but mdtest ran without issue. But I'm 
checking that.

Any thoughts?

Thanks!

Jeff

Re: [OMPI users] [EXTERNAL] Help deciphering error message

[Pritchard Jr., Howard via users]

Hello Jeffrey,

A couple of things to try first.

Try running without UCX.  Add –-mca pml ^ucx to the mpirun command line.  If 
the app functions without ucx, then the next thing is to see what may be going 
wrong with UCX and the Open MPI components that use it.

You may want to set the UCX_LOG_LEVEL environment variable to see if Open MPI’s 
UCX PML component is actually able to initialize UCX and start trying to use it.

See https://openucx.readthedocs.io/en/master/faq.html  for an example to do 
this using mpirun and the type of output you should be getting.

Another simple thing to try is

mpirun -np 1 ucx_info -v


and see it you get something like this back on stdout:

 Library version: 1.14.0

# Library path: /usr/lib64/libucs.so.0

# API headers version: 1.14.0

# Git branch '', revision f8877c5

# Configured with: --build=aarch64-redhat-linux-gnu 
--host=aarch64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking 
--prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin 
--sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include 
--libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var 
--sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info 
--disable-optimizations --disable-logging --disable-debug --disable-assertions 
--enable-mt --disable-params-check --without-go --without-java --enable-cma 
--with-cuda --with-gdrcopy --with-verbs --with-knem --with-rdmacm 
--without-rocm --with-xpmem --without-fuse3 --without-ugni 
--with-cuda=/usr/local/cuda-11.7
Are you running the mpirun command on dgx-14?  If that’s a different host a 
likely problem is that for some reason, the information in your ucx/1.10.1 is 
not getting picked up on dgx-14.

One other thing, if the module UCX module name is indicating the version of 
UCX, its rather old.  I’d suggest, if possible, updating to a newer version, 
like 1.14.1 or newer.  There are many enhancements in more recent versions of 
UCX for GPU support and I would bet you’d want that for your DGX boxes.

Howard

From: users  on behalf of Jeffrey Layton via 
users 
Reply-To: Open MPI Users 
Date: Thursday, March 7, 2024 at 11:53 AM
To: Open MPI Users 
Cc: Jeffrey Layton 
Subject: [EXTERNAL] [OMPI users] Help deciphering error message

Good afternoon,

I'm getting an error message I'm not sure how to use to debug an issue. I'll 
try to give you all of the pertinent about the setup, but I didn't build the 
system nor install the software. It's an NVIDIA SuperPod system with Base 
Command Manager 10.0.

I'm building IOR but I'm really interested in mdtest. "module list" says I'm 
using the following modules:

gcc/64/4.1.5a1
ucx/1.10.1
openmpi4/gcc/4.1.5

There are no problems building the code.

I'm using Slurm to run mdtest using a script. The output from the script and 
Slurm is the following (the command to run it is included).


/cm/shared/apps/openmpi4/gcc/4.1.5/bin/mpirun --mca btl '^openib' -np 1 -map-by 
ppr:1:node --allow-run-as-root --mca btl_openib_warn_default_gid_prefix 0 --mca 
btl_openib_if_exclude mlx5_0,mlx5_5,mlx5_6 --mca plm_base_verbose 0
 --mca plm rsh /home/bcm/bin/bin/mdtest -i 3 -I 4 -z 3 -b 8 -u -u -d 
/raid/bcm/mdtest
--
A requested component was not found, or was unable to be opened.  This
means that this component is either not installed or is unable to be
used on your system (e.g., sometimes this means that shared libraries
that the component requires are unable to be found/loaded).  Note that
Open MPI stopped checking at the first component that it did not find.

Host:  dgx-14
Framework: pml
Component: ucx
--
[dgx-14:4055623] [[42340,0],0] ORTE_ERROR_LOG: Data unpack would read past end 
of buffer in file util/show_help.c at line 501
[dgx-14:4055632] *** An error occurred in MPI_Init
[dgx-14:4055632] *** reported by process [2774794241,0]
[dgx-14:4055632] *** on a NULL communicator
[dgx-14:4055632] *** Unknown error
[dgx-14:4055632] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will 
now abort,
[dgx-14:4055632] ***and potentially your MPI job)


Any pointers/help is greatly appreciated.

Thanks!

Jeff




[Image removed by 
sender.]
Virus-free.www.avast.com

Re: [OMPI users] [EXTERNAL] Re: MPI_Init_thread error

[Pritchard Jr., Howard via users]

HI Aziz,

Oh I see you referenced the faq.  That section of the faq is discussing how to 
make the Open MPI 4 series (and older)  job launcher “know” about the batch 
scheduler you are using.
The relevant section for launching with srun is covered by this faq - 
https://www-lb.open-mpi.org/faq/?category=slurm

Howard

From: "Pritchard Jr., Howard" 
Date: Tuesday, July 25, 2023 at 8:26 AM
To: Open MPI Users 
Cc: Aziz Ogutlu 
Subject: Re: [EXTERNAL] Re: [OMPI users] MPI_Init_thread error

HI Aziz,

Did you include –with-pmi2 on your Open MPI configure line?

Howard

From: users  on behalf of Aziz Ogutlu via 
users 
Organization: Eduline Bilisim
Reply-To: Open MPI Users 
Date: Tuesday, July 25, 2023 at 8:18 AM
To: Open MPI Users 
Cc: Aziz Ogutlu 
Subject: [EXTERNAL] Re: [OMPI users] MPI_Init_thread error


Hi Gilles,

Thank you for your response.

When I run srun --mpi=list, I get only pmi2.

When I run command with --mpi=pmi2 parameter, I got same error.

OpenMPI automatically support slurm after 4.x version. 
https://www.open-mpi.org/faq/?category=building#build-rte


On 7/25/23 12:55, Gilles Gouaillardet via users wrote:
Aziz,

When using direct run (e.g. srun), OpenMPI has to interact with SLURM.
This is typically achieved via PMI2 or PMIx

You can
srun --mpi=list
to list the available options on your system

if PMIx is available, you can
srun --mpi=pmix ...

if only PMI2 is available, you need to make sure Open MPI was built with SLURM 
support (e.g. configure --with-slurm ...)
and then
srun --mpi=pmi2 ...


Cheers,

Gilles

On Tue, Jul 25, 2023 at 5:07 PM Aziz Ogutlu via users 
mailto:users@lists.open-mpi.org>> wrote:
Hi there all,
We're using Slurm 21.08 on Redhat 7.9 HPC cluster with OpenMPI 4.0.3 + gcc 
8.5.0.
When we run command below for call SU2, we get an error message:

$ srun -p defq --nodes=1 --ntasks-per-node=1 --time=01:00:00 --pty bash -i
$ module load su2/7.5.1
$ SU2_CFD config.cfg

*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***and potentially your MPI job)
[cnode003.hpc:17534] Local abort before MPI_INIT completed completed 
successfully, but am not able to aggregate error messages, and not able to 
guarantee that all other processes were killed!

--

Best regards,

Aziz Öğütlü



Eduline Bilişim Sanayi ve Ticaret Ltd. Şti.  
www.eduline.com.tr

Merkez Mah. Ayazma Cad. No:37 Papirus Plaza

Kat:6 Ofis No:118 Kağıthane -  İstanbul - Türkiye 34406

Tel : +90 212 324 60 61 Cep: +90 541 350 40 72

--

İyi çalışmalar,

Aziz Öğütlü



Eduline Bilişim Sanayi ve Ticaret Ltd. Şti.  
www.eduline.com.tr

Merkez Mah. Ayazma Cad. No:37 Papirus Plaza

Kat:6 Ofis No:118 Kağıthane -  İstanbul - Türkiye 34406

Tel : +90 212 324 60 61 Cep: +90 541 350 40 72

Re: [OMPI users] [EXTERNAL] Re: MPI_Init_thread error

[Pritchard Jr., Howard via users]

HI Aziz,

Did you include –with-pmi2 on your Open MPI configure line?

Howard

From: users  on behalf of Aziz Ogutlu via 
users 
Organization: Eduline Bilisim
Reply-To: Open MPI Users 
Date: Tuesday, July 25, 2023 at 8:18 AM
To: Open MPI Users 
Cc: Aziz Ogutlu 
Subject: [EXTERNAL] Re: [OMPI users] MPI_Init_thread error


Hi Gilles,

Thank you for your response.

When I run srun --mpi=list, I get only pmi2.

When I run command with --mpi=pmi2 parameter, I got same error.

OpenMPI automatically support slurm after 4.x version. 
https://www.open-mpi.org/faq/?category=building#build-rte


On 7/25/23 12:55, Gilles Gouaillardet via users wrote:
Aziz,

When using direct run (e.g. srun), OpenMPI has to interact with SLURM.
This is typically achieved via PMI2 or PMIx

You can
srun --mpi=list
to list the available options on your system

if PMIx is available, you can
srun --mpi=pmix ...

if only PMI2 is available, you need to make sure Open MPI was built with SLURM 
support (e.g. configure --with-slurm ...)
and then
srun --mpi=pmi2 ...


Cheers,

Gilles

On Tue, Jul 25, 2023 at 5:07 PM Aziz Ogutlu via users 
mailto:users@lists.open-mpi.org>> wrote:
Hi there all,
We're using Slurm 21.08 on Redhat 7.9 HPC cluster with OpenMPI 4.0.3 + gcc 
8.5.0.
When we run command below for call SU2, we get an error message:

$ srun -p defq --nodes=1 --ntasks-per-node=1 --time=01:00:00 --pty bash -i
$ module load su2/7.5.1
$ SU2_CFD config.cfg

*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***and potentially your MPI job)
[cnode003.hpc:17534] Local abort before MPI_INIT completed completed 
successfully, but am not able to aggregate error messages, and not able to 
guarantee that all other processes were killed!

--

Best regards,

Aziz Öğütlü



Eduline Bilişim Sanayi ve Ticaret Ltd. Şti.  
www.eduline.com.tr

Merkez Mah. Ayazma Cad. No:37 Papirus Plaza

Kat:6 Ofis No:118 Kağıthane -  İstanbul - Türkiye 34406

Tel : +90 212 324 60 61 Cep: +90 541 350 40 72

--

İyi çalışmalar,

Aziz Öğütlü



Eduline Bilişim Sanayi ve Ticaret Ltd. Şti.  
www.eduline.com.tr

Merkez Mah. Ayazma Cad. No:37 Papirus Plaza

Kat:6 Ofis No:118 Kağıthane -  İstanbul - Türkiye 34406

Tel : +90 212 324 60 61 Cep: +90 541 350 40 72

Re: [OMPI users] [EXTERNAL] Re: How to use hugetlbfs with openmpi and ucx

[Pritchard Jr., Howard via users]

HI Arun,

Interesting.  For problem b) I would suggest one of two things
- if you want to dig deeper yourself, and its possible on your system, I'd look 
at the output of dmesg -H -w on the node where the job is hitting this failure 
(you'll need to rerun the job)
- ping the UCX group mail list (see 
https://elist.ornl.gov/mailman/listinfo/ucx-group . 

As for your more general question, I would suggest keeping it simple and 
letting the applications use large pages via the usual libhugetlbfs mechanism 
(LD_PRELOAD libhugetlbfs and set libhugetlbfs env variables for specifying what 
type of process memory to try and map to large pages).But I'm no expert in 
the ways UCX may be able to take advantage of internally allocated large pages 
nor the extent to which such use of large pages has led to demonstrable 
application speedups.

Howard

On 7/21/23, 8:37 AM, "Chandran, Arun" mailto:arun.chand...@amd.com>> wrote:


Hi Howard,


Thank you very much for the reply.


Ucx is trying to setup the FIFO for shared memory communication using both sysv 
and posix.
By default, these allocations are failing when tried with hugetlbfs


a) Failure log from strace(Pasting only for rank0):
[pid 3541286] shmget(IPC_PRIVATE, 6291456, IPC_CREAT|IPC_EXCL|SHM_HUGETLB|0660) 
= -1 EPERM (Operation not permitted)
[pid 3541286] mmap(NULL, 6291456, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_HUGETLB, 
29, 0) = -1 EINVAL (Invalid argument)


b) I was able to overcome the failure for shmget allocation with hugetlbfs by 
adding my gid to "/proc/sys/vm/hugetlb_shm_group"
[pid 3541465] shmget(IPC_PRIVATE, 6291456, IPC_CREAT|IPC_EXCL|SHM_HUGETLB|0660) 
= 2916410--> success
[pid 3541465] mmap(NULL, 6291456, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_HUGETLB, 
29, 0) = -1 EINVAL (Invalid argument) --> still fail


But mmap with " MAP_SHARED|MAP_HUGETLB" is still failing. Any clues?


I am aware of the advantages of huge pagetables, I am asking from the openmpi 
library perspective,
Should I use it for openmpi internal buffers and data structures or leave it 
for the applications to use?
What are the community recommendations in this regard?


--Arun


-Original Message-
From: Pritchard Jr., Howard mailto:howa...@lanl.gov>> 
Sent: Thursday, July 20, 2023 9:36 PM
To: Open MPI Users mailto:users@lists.open-mpi.org>>; Florent GERMAIN mailto:florent.germ...@eviden.com>>
Cc: Chandran, Arun mailto:arun.chand...@amd.com>>
Subject: Re: [EXTERNAL] Re: [OMPI users] How to use hugetlbfs with openmpi and 
ucx


HI Arun,


Its going to be chatty, but you may want to see if strace helps in diagnosing:


mpirun -np 2 (all your favorite mpi args) strace -f send_recv 1000 1


huge pages often helps reduce pressure on a NIC's I/O MMU widget and speeds up 
resolving va to pa memory addresses.


On 7/19/23, 9:24 PM, "users on behalf of Chandran, Arun via users" 
mailto:users-boun...@lists.open-mpi.org> 
> on behalf of 
users@lists.open-mpi.org  
>> wrote:


Good luck,


Howard


Hi,




I am trying to use static huge pages, not transparent huge pages.




Ucx is allowed to allocate via hugetlbfs.




$ ./bin/ucx_info -c | grep -i huge
UCX_SELF_ALLOC=huge,thp,md,mmap,heap
UCX_TCP_ALLOC=huge,thp,md,mmap,heap
UCX_SYSV_HUGETLB_MODE=try --->It is trying this and failing 
UCX_SYSV_FIFO_HUGETLB=n UCX_POSIX_HUGETLB_MODE=try---> it is trying this and 
failing UCX_POSIX_FIFO_HUGETLB=n 
UCX_ALLOC_PRIO=md:sysv,md:posix,huge,thp,md:*,mmap,heap
UCX_CMA_ALLOC=huge,thp,mmap,heap




It is failing even though I have static hugepages available in my system.




$ cat /proc/meminfo | grep HugePages_Total
HugePages_Total: 20




THP is also enabled:
$ cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never




--Arun




-Original Message-
From: Florent GERMAIN mailto:florent.germ...@eviden.com> >>
Sent: Wednesday, July 19, 2023 7:51 PM
To: Open MPI Users mailto:users@lists.open-mpi.org> 
>>; Chandran, 
Arun mailto:arun.chand...@amd.com> 
>>
Subject: RE: How to use hugetlbfs with openmpi and ucx




Hi,
You can check if there are dedicated huge pages on your system or if 
transparent huge pages are allowed.




Transparent huge pages on rhel systems :
$cat /sys/kernel/mm/transparent_hugepage/enabled
always [madvise] never
-> this means that transparent huge pages are selected through mmap + 
-> madvise always = always try to aggregate pages on thp (for large 
-> enough allocation with good alignment) never = never try to aggregate 
-> pages on thp




Dedicated huge pages on rhel systems :
$ cat /proc/meminfo | grep HugePages_Total
HugePages_Total: 0
-> no dedicated huge pages here

Re: [OMPI users] [EXTERNAL] Re: How to use hugetlbfs with openmpi and ucx

[Pritchard Jr., Howard via users]

HI Arun,

Its going to be chatty, but you may want to see if strace helps in diagnosing:

mpirun -np 2 (all your favorite mpi args) strace -f send_recv 1000 1

 huge pages often helps reduce pressure on a NIC's I/O MMU widget and speeds up 
resolving va to pa memory addresses.

On 7/19/23, 9:24 PM, "users on behalf of Chandran, Arun via users" 
mailto:users-boun...@lists.open-mpi.org> on 
behalf of users@lists.open-mpi.org > wrote:

Good luck,

Howard

Hi,


I am trying to use static huge pages, not transparent huge pages.


Ucx is allowed to allocate via hugetlbfs.


$ ./bin/ucx_info -c | grep -i huge
UCX_SELF_ALLOC=huge,thp,md,mmap,heap
UCX_TCP_ALLOC=huge,thp,md,mmap,heap
UCX_SYSV_HUGETLB_MODE=try --->It is trying this and failing
UCX_SYSV_FIFO_HUGETLB=n
UCX_POSIX_HUGETLB_MODE=try---> it is trying this and failing
UCX_POSIX_FIFO_HUGETLB=n
UCX_ALLOC_PRIO=md:sysv,md:posix,huge,thp,md:*,mmap,heap
UCX_CMA_ALLOC=huge,thp,mmap,heap


It is failing even though I have static hugepages available in my system.


$ cat /proc/meminfo | grep HugePages_Total
HugePages_Total: 20


THP is also enabled:
$ cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never


--Arun


-Original Message-
From: Florent GERMAIN mailto:florent.germ...@eviden.com>> 
Sent: Wednesday, July 19, 2023 7:51 PM
To: Open MPI Users mailto:users@lists.open-mpi.org>>; Chandran, Arun mailto:arun.chand...@amd.com>>
Subject: RE: How to use hugetlbfs with openmpi and ucx


Hi,
You can check if there are dedicated huge pages on your system or if 
transparent huge pages are allowed.


Transparent huge pages on rhel systems :
$cat /sys/kernel/mm/transparent_hugepage/enabled
always [madvise] never
-> this means that transparent huge pages are selected through mmap + 
-> madvise always = always try to aggregate pages on thp (for large 
-> enough allocation with good alignment) never = never try to aggregate 
-> pages on thp


Dedicated huge pages on rhel systems :
$ cat /proc/meminfo | grep HugePages_Total
HugePages_Total: 0
-> no dedicated huge pages here


It seems that ucx tries to use dedicated huge pages (mmap(addr=(nil), 
length=6291456, flags= HUGETLB, fd=29)).
If there are no dedicated huge pages available, mmap fails.


Huge pages can accelerate virtual address to physical address translation and 
reduce TLB consumption.
It may be useful for large and frequently used buffers.


Regards,
Florent


-Message d'origine-
De : users mailto:users-boun...@lists.open-mpi.org>> De la part de Chandran, Arun via 
users Envoyé : mercredi 19 juillet 2023 15:44 À : users@lists.open-mpi.org 
 Cc : Chandran, Arun mailto:arun.chand...@amd.com>> Objet : [OMPI users] How to use hugetlbfs with 
openmpi and ucx


Hi All,


I am trying to see whether hugetlbfs is improving the latency of communication 
with a small send receive program.


mpirun -np 2 --map-by core --bind-to core --mca pml ucx --mca 
opal_common_ucx_tls any --mca opal_common_ucx_devices any -mca pml_base_verbose 
10 --mca mtl_base_verbose 10 -x OMPI_MCA_pml_ucx_verbose=10 -x 
UCX_LOG_LEVEL=debu
g -x UCX_PROTO_INFO=y send_recv 1000 1




But the internal buffer allocation in ucx is unable to select the hugetlbfs.


[1688297246.205092] [lib-ssp-04:4022755:0] ucp_context.c:1979 UCX DEBUG 
allocation method[2] is 'huge'
[1688297246.208660] [lib-ssp-04:4022755:0] mm_sysv.c:97 UCX DEBUG mm failed to 
allocate 8447 bytes with hugetlb-> I checked the code, this is a valid 
failure as the size is small compared to huge page size of 2MB
[1688297246.208704] [lib-ssp-04:4022755:0] mm_sysv.c:97 UCX DEBUG mm failed to 
allocate 4292720 bytes with hugetlb
[1688297246.210048] [lib-ssp-04:4022755:0] mm_posix.c:332 UCX DEBUG shared 
memory mmap(addr=(nil), length=6291456, flags= HUGETLB, fd=29) failed: Invalid 
argument
[1688297246.211451] [lib-ssp-04:4022754:0] ucp_context.c:1979 UCX DEBUG 
allocation method[2] is 'huge'
[1688297246.214849] [lib-ssp-04:4022754:0] mm_sysv.c:97 UCX DEBUG mm failed to 
allocate 8447 bytes with hugetlb
[1688297246.214888] [lib-ssp-04:4022754:0] mm_sysv.c:97 UCX DEBUG mm failed to 
allocate 4292720 bytes with hugetlb
[1688297246.216235] [lib-ssp-04:4022754:0] mm_posix.c:332 UCX DEBUG shared 
memory mmap(addr=(nil), length=6291456, flags= HUGETLB, fd=29) failed: Invalid 
argument


Can someone suggest what are the steps to be done to enable hugetlbfs [I cannot 
run my application as root] ? Is using hugetlbfs for the internal buffers is 
recommended?


--Arun

Re: [OMPI users] [EXTERNAL] Requesting information about MPI_T events

[Pritchard Jr., Howard via users]

Hi Kingshuk,

Looks like the MPI_T Events feature is parked in this PR - 
https://github.com/open-mpi/ompi/pull/8057 - at the moment.

Hoawrd


From: users  on behalf of Kingshuk Haldar via 
users 
Reply-To: Open MPI Users 
Date: Wednesday, March 15, 2023 at 4:00 AM
To: OpenMPI-lists-users 
Cc: Kingshuk Haldar 
Subject: [EXTERNAL] [OMPI users] Requesting information about MPI_T events

Hi all,

is there any public branch of OpenMPI with which one can test the MPI_T Events 
interface?

Alternatively, any information about its potential availability in next 
releases would be good to know.

Best,
--
Kingshuk Haldar email: kingshuk.hal...@hlrs.de

Re: [OMPI users] [EXTERNAL] OFI, destroy_vni_context(1137).......: OFI domain close failed (ofi_init.c:1137:destroy_vni_context:Device or resource busy)

[Pritchard Jr., Howard via users]

HI,

You are using MPICH or a vendor derivative of MPICH.  You probably want to 
resend this email to the mpich users/help mail list.

Howard



From: users  on behalf of mrlong via users 

Reply-To: Open MPI Users 
Date: Tuesday, November 1, 2022 at 11:26 AM
To: "de...@lists.open-mpi.org" , 
"users@lists.open-mpi.org" 
Cc: mrlong 
Subject: [EXTERNAL] [OMPI users] OFI, destroy_vni_context(1137)...: OFI 
domain close failed (ofi_init.c:1137:destroy_vni_context:Device or resource 
busy)


Hi, teachers

code:

import mpi4py
import time
import numpy as np
from mpi4py import MPI

comm = MPI.COMM_WORLD
rank = comm.Get_rank()
print("rank",rank)


if __name__ == '__main__':
if rank == 0:
mem = np.array([0], dtype='i')
win = MPI.Win.Create(mem, comm=comm)
else:
win = MPI.Win.Create(None, comm=comm)
print(rank, "end")



(py3.6.8) ➜  ~  mpirun -n 2 python -u 
test.py
rank 0
rank 1
0 end
1 end
Abort(806449679): Fatal error in internal_Finalize: Other MPI error, error 
stack:
internal_Finalize(50)...: MPI_Finalize failed
MPII_Finalize(345)..:
MPID_Finalize(511)..:
MPIDI_OFI_mpi_finalize_hook(895):
destroy_vni_context(1137)...: OFI domain close failed 
(ofi_init.c:1137:destroy_vni_context:Device or resource busy)

Why is this happening? How to debug? This error is not reported on the other 
machine.

Re: [OMPI users] [EXTERNAL] Beginner Troubleshooting OpenMPI Installation - pmi.h Error

[Pritchard Jr., Howard via users]

Hi Jeff,

I think you are now in the “send the system admin an email to install RPMs, in 
particular ask that the numa and udev devel rpms be installed”.  They will need 
to install these rpms on the compute node image(s) as well.

Howard


From: "Jeffrey D. (JD) Tamucci" 
Date: Wednesday, October 5, 2022 at 9:20 AM
To: "Pritchard Jr., Howard" 
Cc: "bbarr...@amazon.com" , Open MPI Users 

Subject: Re: [EXTERNAL] [OMPI users] Beginner Troubleshooting OpenMPI 
Installation - pmi.h Error

Gladly, I tried it that way and it worked in that it was able to find pmi.h. 
Unfortunately there's a new error about finding lnuma and ludev.

make[2]: Entering directory '/shared/maylab/src/openmpi-4.1.4/opal'
  CCLD 
libopen-pal.la
/usr/bin/ld: cannot find -lnuma
/usr/bin/ld: cannot find -ludev
collect2: error: ld returned 1 exit status
make[2]: *** [Makefile:2249: 
libopen-pal.la]
 Error 1
make[2]: Leaving directory '/shared/maylab/src/openmpi-4.1.4/opal'
make[1]: *** [Makefile:2394: install-recursive] Error 1
make[1]: Leaving directory '/shared/maylab/src/openmpi-4.1.4/opal'
make: *** [Makefile:1912: install-recursive] Error 1

Here is a dropbox link to the full output: 
https://www.dropbox.com/s/4rv8n2yp320ix08/ompi-output_Oct4_2022.tar.bz2?dl=0

Thank you for your help!

JD



Jeffrey D. (JD) Tamucci
University of Connecticut
Molecular & Cell Biology
RA in Lab of Eric R. May
PhD / MPH Candidate
he/him


On Tue, Oct 4, 2022 at 1:51 PM Pritchard Jr., Howard 
mailto:howa...@lanl.gov>> wrote:
*Message sent from a system outside of UConn.*

Could you change the –with-pmi to be
--with-pmi=/cm/shared/apps/slurm21.08.8

?


From: "Jeffrey D. (JD) Tamucci" 
mailto:jeffrey.tamu...@uconn.edu>>
Date: Tuesday, October 4, 2022 at 10:40 AM
To: "Pritchard Jr., Howard" mailto:howa...@lanl.gov>>, 
"bbarr...@amazon.com" 
mailto:bbarr...@amazon.com>>
Cc: Open MPI Users mailto:users@lists.open-mpi.org>>
Subject: Re: [EXTERNAL] [OMPI users] Beginner Troubleshooting OpenMPI 
Installation - pmi.h Error

Hi Howard and Brian,

Of course. Here's a dropbox link to the full folder: 
https://www.dropbox.com/s/raqlcnpgk9wz78b/ompi-output_Sep30_2022.tar.bz2?dl=0

This was the configure and make commands:

./configure \
--prefix=/shared/maylab/mayapps/mpi/openmpi/4.1.4 \
--with-slurm \
--with-lsf=no \
--with-pmi=/cm/shared/apps/slurm/21.08.8/include/slurm \
--with-pmi-libdir=/cm/shared/apps/slurm/21.08.8/lib64 \
--with-hwloc=/cm/shared/apps/hwloc/1.11.11 \
--with-cuda=/gpfs/sharedfs1/admin/hpc2.0/apps/cuda/11.6 \
--enable-shared \
--enable-static &&
make -j 32 &&
make -j 32 check
make install

The output of the make command is in the install_open-mpi_4.1.4_hpc2.log file.



Jeffrey D. (JD) Tamucci
University of Connecticut
Molecular & Cell Biology
RA in Lab of Eric R. May
PhD / MPH Candidate
he/him


On Tue, Oct 4, 2022 at 12:33 PM Pritchard Jr., Howard 
mailto:howa...@lanl.gov>> wrote:
*Message sent from a system outside of UConn.*

HI JD,

Could you post the configure options your script uses to build Open MPI?

Howard

From: users 
mailto:users-boun...@lists.open-mpi.org>> on 
behalf of "Jeffrey D. (JD) Tamucci via users" 
mailto:users@lists.open-mpi.org>>
Reply-To: Open MPI Users 
mailto:users@lists.open-mpi.org>>
Date: Tuesday, October 4, 2022 at 10:07 AM
To: "users@lists.open-mpi.org" 
mailto:users@lists.open-mpi.org>>
Cc: "Jeffrey D. (JD) Tamucci" 
mailto:jeffrey.tamu...@uconn.edu>>
Subject: [EXTERNAL] [OMPI users] Beginner Troubleshooting OpenMPI Installation 
- pmi.h Error

Hi,

I have been trying to install OpenMPI v4.1.4 on a university HPC cluster. We 
use the Bright cluster manager and have SLURM v21.08.8 and RHEL 8.6. I used a 
script to install OpenMPI that a former co-worker had used to successfully 
install OpenMPI v3.0.0 previously. I updated it to include new versions of the 
dependencies and new paths to those installs.

Each time, it fails in the make install step. There is a fatal error about 
finding pmi.h. It specifically says:

make[2]: Entering directory '/shared/maylab/src/openmpi-4.1.4/opal/mca/pmix/s1'
  CC   

Re: [OMPI users] [EXTERNAL] Beginner Troubleshooting OpenMPI Installation - pmi.h Error

[Pritchard Jr., Howard via users]

Could you change the –with-pmi to be
--with-pmi=/cm/shared/apps/slurm21.08.8

?


From: "Jeffrey D. (JD) Tamucci" 
Date: Tuesday, October 4, 2022 at 10:40 AM
To: "Pritchard Jr., Howard" , "bbarr...@amazon.com" 

Cc: Open MPI Users 
Subject: Re: [EXTERNAL] [OMPI users] Beginner Troubleshooting OpenMPI 
Installation - pmi.h Error

Hi Howard and Brian,

Of course. Here's a dropbox link to the full folder: 
https://www.dropbox.com/s/raqlcnpgk9wz78b/ompi-output_Sep30_2022.tar.bz2?dl=0

This was the configure and make commands:

./configure \
--prefix=/shared/maylab/mayapps/mpi/openmpi/4.1.4 \
--with-slurm \
--with-lsf=no \
--with-pmi=/cm/shared/apps/slurm/21.08.8/include/slurm \
--with-pmi-libdir=/cm/shared/apps/slurm/21.08.8/lib64 \
--with-hwloc=/cm/shared/apps/hwloc/1.11.11 \
--with-cuda=/gpfs/sharedfs1/admin/hpc2.0/apps/cuda/11.6 \
--enable-shared \
--enable-static &&
make -j 32 &&
make -j 32 check
make install

The output of the make command is in the install_open-mpi_4.1.4_hpc2.log file.



Jeffrey D. (JD) Tamucci
University of Connecticut
Molecular & Cell Biology
RA in Lab of Eric R. May
PhD / MPH Candidate
he/him


On Tue, Oct 4, 2022 at 12:33 PM Pritchard Jr., Howard 
mailto:howa...@lanl.gov>> wrote:
*Message sent from a system outside of UConn.*

HI JD,

Could you post the configure options your script uses to build Open MPI?

Howard

From: users 
mailto:users-boun...@lists.open-mpi.org>> on 
behalf of "Jeffrey D. (JD) Tamucci via users" 
mailto:users@lists.open-mpi.org>>
Reply-To: Open MPI Users 
mailto:users@lists.open-mpi.org>>
Date: Tuesday, October 4, 2022 at 10:07 AM
To: "users@lists.open-mpi.org" 
mailto:users@lists.open-mpi.org>>
Cc: "Jeffrey D. (JD) Tamucci" 
mailto:jeffrey.tamu...@uconn.edu>>
Subject: [EXTERNAL] [OMPI users] Beginner Troubleshooting OpenMPI Installation 
- pmi.h Error

Hi,

I have been trying to install OpenMPI v4.1.4 on a university HPC cluster. We 
use the Bright cluster manager and have SLURM v21.08.8 and RHEL 8.6. I used a 
script to install OpenMPI that a former co-worker had used to successfully 
install OpenMPI v3.0.0 previously. I updated it to include new versions of the 
dependencies and new paths to those installs.

Each time, it fails in the make install step. There is a fatal error about 
finding pmi.h. It specifically says:

make[2]: Entering directory '/shared/maylab/src/openmpi-4.1.4/opal/mca/pmix/s1'
  CC   libmca_pmix_s1_la-pmix_s1_component.lo
  CC   libmca_pmix_s1_la-pmix_s1.lo
pmix_s1.c:29:10: fatal error: pmi.h: No such file or directory
   29 | #include 

I've looked through the archives and seen others face similar errors in years 
past but I couldn't understand the solutions. One person suggested that SLURM 
may be missing PMI libraries. I think I've verified that SLURM has PMI. I 
include paths to those files and it seems to find them earlier in the process.

I'm not sure what the next step is in troubleshooting this. I have included a 
bz2 file containing my install script, a log file containing the script output 
(from build, make, make install), the config.log, and the opal_config.h file. 
If anyone could provide any guidance, I'd  sincerely appreciate it.

Best,
JD

Re: [OMPI users] [EXTERNAL] Beginner Troubleshooting OpenMPI Installation - pmi.h Error

[Pritchard Jr., Howard via users]

HI JD,

Could you post the configure options your script uses to build Open MPI?

Howard

From: users  on behalf of "Jeffrey D. (JD) 
Tamucci via users" 
Reply-To: Open MPI Users 
Date: Tuesday, October 4, 2022 at 10:07 AM
To: "users@lists.open-mpi.org" 
Cc: "Jeffrey D. (JD) Tamucci" 
Subject: [EXTERNAL] [OMPI users] Beginner Troubleshooting OpenMPI Installation 
- pmi.h Error

Hi,

I have been trying to install OpenMPI v4.1.4 on a university HPC cluster. We 
use the Bright cluster manager and have SLURM v21.08.8 and RHEL 8.6. I used a 
script to install OpenMPI that a former co-worker had used to successfully 
install OpenMPI v3.0.0 previously. I updated it to include new versions of the 
dependencies and new paths to those installs.

Each time, it fails in the make install step. There is a fatal error about 
finding pmi.h. It specifically says:

make[2]: Entering directory '/shared/maylab/src/openmpi-4.1.4/opal/mca/pmix/s1'
  CC   libmca_pmix_s1_la-pmix_s1_component.lo
  CC   libmca_pmix_s1_la-pmix_s1.lo
pmix_s1.c:29:10: fatal error: pmi.h: No such file or directory
   29 | #include 

I've looked through the archives and seen others face similar errors in years 
past but I couldn't understand the solutions. One person suggested that SLURM 
may be missing PMI libraries. I think I've verified that SLURM has PMI. I 
include paths to those files and it seems to find them earlier in the process.

I'm not sure what the next step is in troubleshooting this. I have included a 
bz2 file containing my install script, a log file containing the script output 
(from build, make, make install), the config.log, and the opal_config.h file. 
If anyone could provide any guidance, I'd  sincerely appreciate it.

Best,
JD

Re: [OMPI users] [EXTERNAL] Problem with Mellanox ConnectX3 (FDR) and openmpi 4

[Pritchard Jr., Howard via users]

Hi Boyrie,

The warning message is coming from the older ibverbs component of the Open MPI 
4.0/4.1 releases.
You can make this message using several ways.  One at configure time is to add

--disable-verbs

to the configure options.

At runtime you can set

export OMPI_MCA_btl=^openib

The ucx messages are just being chatty about which ucx transport type is being 
selected.

The VASP hang may be something else.

Howard

From: users  on behalf of Boyrie Fabrice via 
users 
Reply-To: Open MPI Users 
Date: Friday, August 19, 2022 at 9:51 AM
To: "users@lists.open-mpi.org" 
Cc: Boyrie Fabrice 
Subject: [EXTERNAL] [OMPI users] Problem with Mellanox ConnectX3 (FDR) and 
openmpi 4


Hi



I had to reinstall a cluster in AlmaLinux 8.6



I am unable to make openmpi 4 working with infiniband. I have the following 
message in a trivial pingpong test


mpirun --hostfile hostfile -np 2 pingpong
--
WARNING: There was an error initializing an OpenFabrics device.

 Local host:   node2
 Local device: mlx4_0
--
[node2:12431] common_ucx.c:107 using OPAL memory hooks as external events
[node2:12431] pml_ucx.c:197 mca_pml_ucx_open: UCX version 1.11.2
[node1:13188] common_ucx.c:174 using OPAL memory hooks as external events
[node1:13188] pml_ucx.c:197 mca_pml_ucx_open: UCX version 1.11.2
[node2:12431] pml_ucx.c:289 mca_pml_ucx_init
[node1:13188] common_ucx.c:333 posix/memory: did not match transport list
[node1:13188] common_ucx.c:333 sysv/memory: did not match transport list
[node1:13188] common_ucx.c:333 self/memory0: did not match transport list
[node1:13188] common_ucx.c:333 tcp/lo: did not match transport list
[node1:13188] common_ucx.c:333 tcp/eno1: did not match transport list
[node1:13188] common_ucx.c:333 tcp/ib0: did not match transport list
[node1:13188] common_ucx.c:228 driver '../../../../bus/pci/drivers/mlx4_core' 
matched by 'mlx*'
[node1:13188] common_ucx.c:324 rc_verbs/mlx4_0:1: matched both transport and 
device list
[node1:13188] common_ucx.c:337 support level is transports and devices
[node1:13188] pml_ucx.c:289 mca_pml_ucx_init
[node2:12431] pml_ucx.c:114 Pack remote worker address, size 155
[node2:12431] pml_ucx.c:114 Pack local worker address, size 291
[node2:12431] pml_ucx.c:351 created ucp context 0xf832a0, worker 0x109fc50
[node1:13188] pml_ucx.c:114 Pack remote worker address, size 155
[node1:13188] pml_ucx.c:114 Pack local worker address, size 291
[node1:13188] pml_ucx.c:351 created ucp context 0x1696320, worker 0x16c9ce0
[node1:13188] pml_ucx_component.c:147 returning priority 51
[node2:12431] pml_ucx.c:182 Got proc 0 address, size 291
[node2:12431] pml_ucx.c:411 connecting to proc. 0
[node1:13188] pml_ucx.c:182 Got proc 1 address, size 291
[node1:13188] pml_ucx.c:411 connecting to proc. 1
length   time/message (usec)transfer rate (Gbyte/sec)
[node2:12431] pml_ucx.c:182 Got proc 1 address, size 155
[node2:12431] pml_ucx.c:411 connecting to proc. 1
[node1:13188] pml_ucx.c:182 Got proc 0 address, size 155
[node1:13188] pml_ucx.c:411 connecting to proc. 0
1 45.683729   0.88
1001  4.286029   0.934198
2001  5.755391   1.390696
3001  6.902443   1.739095
4001  8.485305   1.886084
5001  9.596994   2.084403
6001  11.055146   2.171297
7001  11.977093   2.338130
8001  13.324408   2.401908
9001  14.471116   2.487991
10001 15.806676   2.530829
[node2:12431] common_ucx.c:240 disconnecting from rank 0
[node2:12431] common_ucx.c:240 disconnecting from rank 1
[node2:12431] common_ucx.c:204 waiting for 1 disconnect requests
[node2:12431] common_ucx.c:204 waiting for 0 disconnect requests
[node1:13188] common_ucx.c:466 disconnecting from rank 0
[node1:13188] common_ucx.c:430 waiting for 1 disconnect requests
[node1:13188] common_ucx.c:466 disconnecting from rank 1
[node1:13188] common_ucx.c:430 waiting for 0 disconnect requests
[node2:12431] pml_ucx.c:367 mca_pml_ucx_cleanup
[node1:13188] pml_ucx.c:367 mca_pml_ucx_cleanup
[node2:12431] pml_ucx.c:268 mca_pml_ucx_close
[node1:13188] pml_ucx.c:268 mca_pml_ucx_close



cat hostfile
node1 slots=1
node2 slots=1


And with a real program (Vasp) it stops.

Infinband seems to be working. I can ssh over infiniband and qperf works in 
rdma mode

qperf  -t 10 ibnode1 ud_lat ud_bw
ud_lat:
   latency  =  18.2 us
ud_bw:
   send_bw  =  2.81 GB/sec
   recv_bw  =  2.81 GB/sec


I use the standard AlmaLinux module for infiniband

82:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]

I can not install MLNX_OFED_LINUX-5.6-2.0.9.0-rhel8.6-x86_64 because it does 
not supports ConnectX-3
And I can not install MLNX_OFED_LINUX-4.9-5.1.0.0-rhel8.6-x86_64 because the 
module compilation fails.

I have 

Re: [OMPI users] [EXTERNAL] Java Segentation Fault

[Pritchard Jr., Howard via users]

HI Janek,

A few questions.

First which version of Open MPI are you using?

Did you compile your code with the Open MPI mpijavac wrapper?

Howard

From: users  on behalf of "Laudan, Janek via 
users" 
Reply-To: "Laudan, Janek" , Open MPI Users 

Date: Thursday, March 17, 2022 at 9:52 AM
To: "users@lists.open-mpi.org" 
Cc: "Laudan, Janek" 
Subject: [EXTERNAL] [OMPI users] Java Segentation Fault

Hi,

I am trying to extend an existing Java-Project to be run with open-mpi. I have 
managed to successfully set up open-mpi and my project on my local machine to 
conduct some test runs.

However, when I tried to set up things on our cluster I ran into some problems. 
I was able to run some trivial examples such as "HelloWorld" and "Ring" which I 
found on in the ompi-Github-repo. Unfortunately, when I try to run our app 
wrapped between MPI.Init(args) and MPI.Finalize() I get the following 
segmentation fault:

$ mpirun -np 1 java -cp matsim-p-1.0-SNAPSHOT.jar 
org.matsim.parallel.RunMinimalMPIExample
Java-Version: 11.0.2
before getTestScenario
before load config
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will 
impact performance.
[cluster-i:1272 :0:1274] Caught signal 11 (Segmentation fault: address not 
mapped to object at address 0xc)
 backtrace (tid:   1274) 
=
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x14a85752fdf4, pid=1272, tid=1274
#
# JRE version: Java(TM) SE Runtime Environment (11.0.2+9) (build 11.0.2+9-LTS)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (11.0.2+9-LTS, mixed mode, tiered, 
compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# J 612 c2 
java.lang.StringBuilder.append(Ljava/lang/String;)Ljava/lang/StringBuilder; 
java.base@11.0.2 (8 bytes) @ 0x14a85752fdf4 
[0x14a85752fdc0+0x0034]
#
# No core dump will be written. Core dumps have been disabled. To enable core 
dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /net/ils/laudan/mpi-test/matsim-p/hs_err_pid1272.log
Compiled method (c2)1052  612   4   java.lang.StringBuilder::append 
(8 bytes)
 total in heap  [0x14a85752fc10,0x14a8575306a8] = 2712
 relocation [0x14a85752fd88,0x14a85752fdb8] = 48
 main code  [0x14a85752fdc0,0x14a857530360] = 1440
 stub code  [0x14a857530360,0x14a857530378] = 24
 metadata   [0x14a857530378,0x14a8575303c0] = 72
 scopes data[0x14a8575303c0,0x14a857530578] = 440
 scopes pcs [0x14a857530578,0x14a857530658] = 224
 dependencies   [0x14a857530658,0x14a857530660] = 8
 handler table  [0x14a857530660,0x14a857530678] = 24
 nul chk table  [0x14a857530678,0x14a8575306a8] = 48
Compiled method (c1)1053  263   3   java.lang.StringBuilder:: 
(7 bytes)
 total in heap  [0x14a850102790,0x14a850102b30] = 928
 relocation [0x14a850102908,0x14a850102940] = 56
 main code  [0x14a850102940,0x14a850102a20] = 224
 stub code  [0x14a850102a20,0x14a850102ac8] = 168
 metadata   [0x14a850102ac8,0x14a850102ad0] = 8
 scopes data[0x14a850102ad0,0x14a850102ae8] = 24
 scopes pcs [0x14a850102ae8,0x14a850102b28] = 64
 dependencies   [0x14a850102b28,0x14a850102b30] = 8
Could not load hsdis-amd64.so; library not loadable; PrintAssembly is disabled
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
[cluster-i:01272] *** Process received signal ***
[cluster-i:01272] Signal: Aborted (6)
[cluster-i:01272] Signal code:  (-6)
[cluster-i:01272] [ 0] /usr/lib64/libpthread.so.0(+0xf630)[0x14a86e477630]
[cluster-i:01272] [ 1] /usr/lib64/libc.so.6(gsignal+0x37)[0x14a86dcbb387]
[cluster-i:01272] [ 2] /usr/lib64/libc.so.6(abort+0x148)[0x14a86dcbca78]
[cluster-i:01272] [ 3] 
/afs/math.tu-berlin.de/software/java/jdk-11.0.2/lib/server/libjvm.so(+0xc00be9)[0x14a86d3f8be9]
[cluster-i:01272] [ 4] 
/afs/math.tu-berlin.de/software/java/jdk-11.0.2/lib/server/libjvm.so(+0xe29619)[0x14a86d621619]
[cluster-i:01272] [ 5] 
/afs/math.tu-berlin.de/software/java/jdk-11.0.2/lib/server/libjvm.so(+0xe29e9b)[0x14a86d621e9b]
[cluster-i:01272] [ 6] 
/afs/math.tu-berlin.de/software/java/jdk-11.0.2/lib/server/libjvm.so(+0xe29ece)[0x14a86d621ece]
[cluster-i:01272] [ 7] 
/afs/math.tu-berlin.de/software/java/jdk-11.0.2/lib/server/libjvm.so(JVM_handle_linux_signal+0x1c0)[0x14a86d403a00]
[cluster-i:01272] [ 8] 
/afs/math.tu-berlin.de/software/java/jdk-11.0.2/lib/server/libjvm.so(+0xbff5e8)[0x14a86d3f75e8]
[cluster-i:01272] [ 9] /usr/lib64/libpthread.so.0(+0xf630)[0x14a86e477630]
[cluster-i:01272] [10] [0x14a85752fdf4]
[cluster-i:01272] *** End of error message ***
--
Primary job  terminated normally, but 1 process returned
a non-zero 

Re: [OMPI users] [EXTERNAL] OpenMPI, Slurm and MPI_Comm_spawn

[Pritchard Jr., Howard via users]

Hi Kurt,

This documentation is rather slurm-centric.  If you build Open MPI 4.1.x series 
the default way, it will build its internal pmix package and use that when 
launching your app using mpirun.
In that case, you can use MPI_comm_spawn within a slurm allocation as long as 
there are sufficient slots in the allocation to hold both the spawner processes 
and the spawnee processes.
Note the slurm pmix implementation doesn’t support spawn – at least currently – 
so the documentation is accurate if you are building Open MPI against the SLURM 
PMix library.
In any case, you can’t use MPI_Comm_spawn if you use srun to launch the 
application.

Hope this helps,

Howard

From: users  on behalf of "Mccall, Kurt E. 
(MSFC-EV41) via users" 
Reply-To: Open MPI Users 
Date: Tuesday, March 8, 2022 at 7:49 AM
To: "OpenMpi User List (users@lists.open-mpi.org)" 
Cc: "Mccall, Kurt E. (MSFC-EV41)" 
Subject: [EXTERNAL] [OMPI users] OpenMPI, Slurm and MPI_Comm_spawn

The Slurm MPI User’s Guide at https://slurm.schedmd.com/mpi_guide.html#open_mpi 
has a note that states:

NOTE: OpenMPI has a limitation that does not support calls to MPI_Comm_spawn() 
from within a Slurm allocation. If you need to use the MPI_Comm_spawn() 
function you will need to use another MPI implementation combined with PMI-2 
since PMIx doesn't support it either.

Is this still true in OpenMPI 4.1?

Thanks,
Kurt

Re: [OMPI users] [EXTERNAL] openib BTL disabled when using MPI_Init_thread

[Pritchard Jr., Howard via users]

HI Jose,

I bet this device has not been tested with ucx.  

You may want to join the ucx users mail list at

https://elist.ornl.gov/mailman/listinfo/ucx-group

and ask whether this Marvell device has been tested and workarounds for 
disabling features that this device doesn't support.

Again though, you really may want to first see if the TCP btl will be good 
enough for your cluster. 

Howard

On 2/4/22, 8:03 AM, "Jose E. Roman"  wrote:

Howard,

I don't have much time now to try with --enable-debug.

The RoCE device we have is FastLinQ QL41000 Series 10/25/40/50GbE Controller
The output of ibv_devinfo is:
hca_id: qedr0
transport:  InfiniBand (0)
fw_ver: 8.20.0.0
node_guid:  2267:7cff:fe11:4a50
sys_image_guid: 2267:7cff:fe11:4a50
vendor_id:  0x1077
vendor_part_id: 32880
hw_ver: 0x0
phys_port_cnt:  1
port:   1
state:  PORT_ACTIVE (4)
max_mtu:4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
port_lid:   0
port_lmc:   0x00
link_layer: Ethernet

hca_id: qedr1
transport:  InfiniBand (0)
fw_ver: 8.20.0.0
node_guid:  2267:7cff:fe11:4a51
sys_image_guid: 2267:7cff:fe11:4a51
vendor_id:  0x1077
vendor_part_id: 32880
hw_ver: 0x0
phys_port_cnt:  1
port:   1
state:  PORT_DOWN (1)
max_mtu:4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
port_lid:   0
port_lmc:   0x00
link_layer: Ethernet

Regarding UCX, we have tried with the latest version. Compilation goes 
through, but the ucv_info command gives an error:

# Memory domain: qedr0
# Component: ib
# register: unlimited, cost: 180 nsec
#   remote key: 8 bytes
#   local memory handle is required for zcopy
#
#  Transport: rc_verbs
# Device: qedr0:1
#   Type: network
#  System device: qedr0 (0)
[1643982133.674556] [kahan01:8217 :0]rc_iface.c:505  UCX ERROR 
ibv_create_srq() failed: Function not implemented
#   < failed to open interface >
#
#  Transport: ud_verbs
# Device: qedr0:1
#   Type: network
#  System device: qedr0 (0)
[qelr_create_qp:545]create qp: failed on ibv_cmd_create_qp with 22
[1643982133.681169] [kahan01:8217 :0]ib_iface.c:994  UCX ERROR 
iface=0x56074944bf10: failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX 
wr:4096 sge:1 resp:0: Invalid argument
#   < failed to open interface >
#
# Memory domain: qedr1
# Component: ib
# register: unlimited, cost: 180 nsec
#   remote key: 8 bytes
#   local memory handle is required for zcopy
#   < no supported devices found >


Any idea what the error in ibv_create_srq() means?

Thanks for your help.
Jose



> El 3 feb 2022, a las 17:52, Pritchard Jr., Howard  
escribió:
> 
> Hi Jose,
> 
> A number of things.  
> 
> First for recent versions of Open MPI including the 4.1.x release stream, 
MPI_THREAD_MULTIPLE is supported by default.  However, some transport options 
available when using MPI_Init may not be available when requesting 
MPI_THREAD_MULTIPLE.
> You may want to let Open MPI trundle along with tcp used for inter-node 
messaging and see if your application performs well enough. For a small system 
tcp may well suffice. 
> 
> Second, if you want to pursue this further, you want to rebuild Open MPI 
with --enable-debug.  The debug output will be considerably more verbose and 
provides more info.  I think you will get  a message saying rdmacm CPC is 
excluded owing to the requested thread support level.  There may be info about 
why udcm is not selected as well.
> 
> Third, what sort of RoCE devices are available on your system?  The 
output from ibv_devinfo may be useful. 
> 
> As for UCX,  if it’s the version that came with your ubuntu release 
18.0.4 it may be pretty old.  It's likely that UCX has not been tested on the 
RoCE devices on your system.
> 
> You can run 
> 
> ucx_info -v
> 
> to 

Re: [OMPI users] [EXTERNAL] openib BTL disabled when using MPI_Init_thread

[Pritchard Jr., Howard via users]

Hi Jose,

A number of things.  

First for recent versions of Open MPI including the 4.1.x release stream, 
MPI_THREAD_MULTIPLE is supported by default.  However, some transport options 
available when using MPI_Init may not be available when requesting 
MPI_THREAD_MULTIPLE.
You may want to let Open MPI trundle along with tcp used for inter-node 
messaging and see if your application performs well enough. For a small system 
tcp may well suffice. 

Second, if you want to pursue this further, you want to rebuild Open MPI with 
--enable-debug.  The debug output will be considerably more verbose and 
provides more info.  I think you will get  a message saying rdmacm CPC is 
excluded owing to the requested thread support level.  There may be info about 
why udcm is not selected as well.

Third, what sort of RoCE devices are available on your system?  The output from 
ibv_devinfo may be useful. 

As for UCX,  if it’s the version that came with your ubuntu release 18.0.4 it 
may be pretty old.  It's likely that UCX has not been tested on the RoCE 
devices on your system.

You can run 

ucx_info -v

to check the version number of UCX that you are picking up.

You can download the latest release of UCX at

https://github.com/openucx/ucx/releases/tag/v1.12.0

Instructions for how to build are in the README.md at 
https://github.com/openucx/ucx.
You will want to configure with 

contrib/configure-release-mt --enable-gtest

You want to add the --enable-gtest to the configure options so that you can run 
the ucx sanity checks.   Note this takes quite a while to run but is pretty 
thorough at validating your UCX build. 
You'll want to run this test on one of the nodes with a RoCE device -  

ucx_info -d

This will show which UCX transports/devices are available.

See the Running internal unit tests section of the README.md

Hope this helps,

Howard


On 2/3/22, 8:46 AM, "Jose E. Roman"  wrote:

Thanks. The verbose output is:

[kahan01.upvnet.upv.es:29732] mca: base: components_register: registering 
framework btl components
[kahan01.upvnet.upv.es:29732] mca: base: components_register: found loaded 
component self
[kahan01.upvnet.upv.es:29732] mca: base: components_register: component 
self register function successful
[kahan01.upvnet.upv.es:29732] mca: base: components_register: found loaded 
component sm
[kahan01.upvnet.upv.es:29732] mca: base: components_register: found loaded 
component openib
[kahan01.upvnet.upv.es:29732] mca: base: components_register: component 
openib register function successful
[kahan01.upvnet.upv.es:29732] mca: base: components_register: found loaded 
component vader
[kahan01.upvnet.upv.es:29732] mca: base: components_register: component 
vader register function successful
[kahan01.upvnet.upv.es:29732] mca: base: components_register: found loaded 
component tcp
[kahan01.upvnet.upv.es:29732] mca: base: components_register: component tcp 
register function successful
[kahan01.upvnet.upv.es:29732] mca: base: components_open: opening btl 
components
[kahan01.upvnet.upv.es:29732] mca: base: components_open: found loaded 
component self
[kahan01.upvnet.upv.es:29732] mca: base: components_open: component self 
open function successful
[kahan01.upvnet.upv.es:29732] mca: base: components_open: found loaded 
component openib
[kahan01.upvnet.upv.es:29732] mca: base: components_open: component openib 
open function successful
[kahan01.upvnet.upv.es:29732] mca: base: components_open: found loaded 
component vader
[kahan01.upvnet.upv.es:29732] mca: base: components_open: component vader 
open function successful
[kahan01.upvnet.upv.es:29732] mca: base: components_open: found loaded 
component tcp
[kahan01.upvnet.upv.es:29732] mca: base: components_open: component tcp 
open function successful
[kahan01.upvnet.upv.es:29732] select: initializing btl component self
[kahan01.upvnet.upv.es:29732] select: init of component self returned 
success
[kahan01.upvnet.upv.es:29732] select: initializing btl component openib
[kahan01.upvnet.upv.es:29732] Checking distance from this process to 
device=qedr0
[kahan01.upvnet.upv.es:29732] hwloc_distances->nbobjs=4
[kahan01.upvnet.upv.es:29732] hwloc_distances->values[0]=10
[kahan01.upvnet.upv.es:29732] hwloc_distances->values[1]=16
[kahan01.upvnet.upv.es:29732] hwloc_distances->values[2]=16
[kahan01.upvnet.upv.es:29732] hwloc_distances->values[3]=16
[kahan01.upvnet.upv.es:29732] ibv_obj->type set to NULL
[kahan01.upvnet.upv.es:29732] Process is bound: distance to device is 
0.00
[kahan01.upvnet.upv.es:29732] Checking distance from this process to 
device=qedr1
[kahan01.upvnet.upv.es:29732] hwloc_distances->nbobjs=4
[kahan01.upvnet.upv.es:29732] hwloc_distances->values[0]=10
[kahan01.upvnet.upv.es:29732] hwloc_distances->values[1]=16
[kahan01.upvnet.upv.es:29732] hwloc_distances->values[2]=16
[kahan01.upvnet.upv.es:29732] 

Re: [OMPI users] [EXTERNAL] openib BTL disabled when using MPI_Init_thread

[Pritchard Jr., Howard via users]

Hello Jose,

I suspect the issue here is that the OpenIB BTl isn't finding a connection 
module when you are requesting MPI_THREAD_MULTIPLE.
The rdmacm connection is deselected if MPI_THREAD_MULTIPLE thread support level 
is being requested.

If you run the test in a shell with

export OMPI_MCA_btl_base_verbose=100

there may be some more info to help diagnose what's going on.

Another option would be to build Open MPI with UCX support.  That's the better 
way to use Open MPI over IB/RoCE.

Howard

On 2/2/22, 10:52 AM, "users on behalf of Jose E. Roman via users" 
 wrote:

Hi.

I am using Open MPI 4.1.1 with the openib BTL on a 4-node cluster with 
Ethernet 10/25Gb (RoCE). It is using libibverbs from Ubuntu 18.04 (kernel 
4.15.0-166-generic).

With this hello world example:

#include 
#include 
int main (int argc, char *argv[])
{
 int rank, size, provided;
 MPI_Init_thread(, , MPI_THREAD_FUNNELED, );
 MPI_Comm_rank(MPI_COMM_WORLD, );
 MPI_Comm_size(MPI_COMM_WORLD, );
 printf("Hello world from process %d of %d, provided=%d\n", rank, size, 
provided);
 MPI_Finalize();
 return 0;
}

I get the following output when run on one node:

$ ./hellow
--
No OpenFabrics connection schemes reported that they were able to be
used on a specific port.  As such, the openib BTL (OpenFabrics
support) will be disabled for this port.

 Local host:   kahan01
 Local device: qedr0
 Local port:   1
 CPCs attempted:   rdmacm, udcm
--
Hello world from process 0 of 1, provided=1


The message does not appear if I run on the front-end (does not have RoCE 
network) or if I run it on the node either using MPI_Init() instead of 
MPI_Init_thread() or using MPI_THREAD_SINGLE instead of MPI_THREAD_FUNNELED.

Is there any reason why MPI_Init_thread() is behaving differently to 
MPI_Init()? Note that I am not using threads, and just one MPI process.


The question has a second part: is there a way to determine (without 
running an MPI program) that MPI_Init_thread() won't work but MPI_Init() will 
work? I am asking this because PETSc programs default to use MPI_Init_thread() 
when PETSc's configure script finds the MPI_Init_thread() symbol in the MPI 
library. But in situations like the one reported here, it would be better to 
revert to MPI_Init() since MPI_Init_thread() will not work as expected. [The 
configure script cannot run an MPI program due to batch systems.]

Thanks for your help.
Jose

[OMPI users] Open MPI v4.0.7rc2 available for testing

[Pritchard Jr., Howard via users]

A second release candidate for Open MPI v4.0.7 is now available for testing:

https://www.open-mpi.org/software/ompi/v4.0/


New fixes with this release candidate:



- Fix an issue with MPI_IALLREDUCE_SCATTER when using large count arguments.

- Fixed an issue with POST/START/COMPLETE/WAIT when using subsets of processes. 
 Thanks to Thomas Gilles for reporting.

Your Open MPI release team.


—

[signature_61897647]

Howard Pritchard
Research Scientist
HPC-ENV

Los Alamos National Laboratory
howa...@lanl.gov

[signature_1468325140][signature_524373090][signature_1595424545][signature_371999348]

[OMPI users] Open MPI v4.0.7rc1 available for testing

[Pritchard Jr., Howard via users]

The first release candidate for Open MPI v4.0.7 is now available for testing:

https://www.open-mpi.org/software/ompi/v4.0/


Some fixes include:



- Numerous fixes from vendor partners.

- Fix a problem with a couple of MPI_IALLREDUCE algorithms.  Thanks to

  John Donners for reporting.

- Fix an edge case where MPI_Reduce is invoked with zero count and NULL

  source and destination buffers.

- Use the mfence instruction in opal_atomic_rmb on x86_64 cpus.  Thanks

  to George Katevenis for proposing a fix.

- Fix an issue with the Open MPI build system using the SLURM provided

  PMIx when not requested by the user.  Thanks to Alexander Grund for

  reporting.

- Fix a problem compiling Open MPI with clang on case-insensitive

  file systems.  Thanks to @srpgilles for reporting.

- Fix some OFI usNIC/OFI MTL interaction problems.  Thanks to

  @roguephysicist for reporting this issue.

- Fix a problem with the Posix fbtl component failing to load.

  Thanks to Honggang Li for reporting.

Your Open MPI release team.

—

[signature_61897647]

Howard Pritchard
Research Scientist
HPC-ENV

Los Alamos National Laboratory
howa...@lanl.gov

[signature_629633375][signature_843372916][signature_178570432][signature_1871057199]

Re: [OMPI users] [EXTERNAL] OpenMPI 3.1.6 openib failure: "mlx4_0 errno says Success"

[Pritchard Jr., Howard via users]

HI Greg,

I believe so concerning your TCP question.

I think the patch probably isn’t actually being used otherwise you would have 
noticed the curious print statement.
Sorry about that.  I’m out of ideas on what may be happening.

Howard

From: "Fischer, Greg A." 
Date: Friday, October 15, 2021 at 9:17 AM
To: "Pritchard Jr., Howard" , Open MPI Users 

Cc: "Fischer, Greg A." 
Subject: RE: [EXTERNAL] [OMPI users] OpenMPI 3.1.6 openib failure: "mlx4_0 
errno says Success"

I tried the patch, but I get the same result:

error obtaining device attributes for mlx4_0 errno says Success

I’m getting (what I think are) good transfer rates using “--mca btl self,tcp” 
on the osu_bw test (~7000 MB/s). It seems to me that the only way that could be 
happening is if the infiniband interfaces are being used over TCP, correct? 
Would such an arrangement preclude the ability to do RDMA or openib? Perhaps 
the network is setup in such a way that the IB hardware is not discoverable by 
openib?

(I’m not a network admin, and I wasn’t involved in the setup of the network. 
Unfortunately, the person who knows the most has recently left the 
organization.)

Greg

From: Pritchard Jr., Howard 
Sent: Thursday, October 14, 2021 5:45 PM
To: Fischer, Greg A. ; Open MPI Users 

Subject: Re: [EXTERNAL] [OMPI users] OpenMPI 3.1.6 openib failure: "mlx4_0 
errno says Success"

[External Email]
HI Greg,

Oh yes that’s not good about rdmacm.

Yes the OFED looks pretty old.

Did you by any chance apply that patch?  I generated that for a sysadmin here 
who was in the situation where they needed to maintain Open MPI 3.1.6 but had 
to also upgrade to some newer RHEL release, but the Open MPi wasn’t compiling 
after the RHEL upgrade.

Howard


From: "Fischer, Greg A." 
mailto:fisch...@westinghouse.com>>
Date: Thursday, October 14, 2021 at 1:47 PM
To: "Pritchard Jr., Howard" mailto:howa...@lanl.gov>>, Open 
MPI Users mailto:users@lists.open-mpi.org>>
Cc: "Fischer, Greg A." 
mailto:fisch...@westinghouse.com>>
Subject: RE: [EXTERNAL] [OMPI users] OpenMPI 3.1.6 openib failure: "mlx4_0 
errno says Success"

I added –enable-mt and re-installed UCX. Same result. (I didn’t re-compile 
OpenMPI.)

A conspicuous warning I see in my UCX configure output is:

checking for rdma_establish in -lrdmacm... no
configure: WARNING: RDMACM requested but librdmacm is not found or does not 
provide rdma_establish() API

The version of librdmacm we have comes from 
librdmacm-devel-41mlnx1-OFED.4.1.0.1.0.41102.x86_64, which seems to date from 
mid-2017. I wonder if that’s too old?

Greg

From: Pritchard Jr., Howard mailto:howa...@lanl.gov>>
Sent: Thursday, October 14, 2021 3:31 PM
To: Fischer, Greg A. 
mailto:fisch...@westinghouse.com>>; Open MPI Users 
mailto:users@lists.open-mpi.org>>
Subject: Re: [EXTERNAL] [OMPI users] OpenMPI 3.1.6 openib failure: "mlx4_0 
errno says Success"

[External Email]
Hi Greg,

I think the UCX PML may be discomfited by the lack of thread safety.

Could you try using the contrib/configure-release-mt  in your ucx folder?  You 
want to add –enable-mt.
That’s what stands out in your configure output from the one I usually get when 
building on a MLNX connectx5 cluster with
MLNX_OFED_LINUX-4.5-1.0.1.0

Here’s the output from one of my UCX configs:

configure: =
configure: UCX build configuration:
configure: Build prefix:   /ucx_testing/ucx/test_install
configure:Configuration dir:   ${prefix}/etc/ucx
configure:   Preprocessor flags:   -DCPU_FLAGS="" -I${abs_top_srcdir}/src 
-I${abs_top_builddir} -I${abs_top_builddir}/src
configure:   C compiler:   
/users/hpritchard/spack/opt/spack/linux-rhel7-aarch64/gcc-4.8.5/gcc-9.1.0-nhd4fe4i6jtn2hncfzumegojm6hsznxy/bin/gcc
 -O3 -g -Wall -Werror -funwind-tables -Wno-missing-field-initializers 
-Wno-unused-parameter -Wno-unused-label -Wno-long-long -Wno-endif-labels 
-Wno-sign-compare -Wno-multichar -Wno-deprecated-declarations -Winvalid-pch 
-Wno-pointer-sign -Werror-implicit-function-declaration -Wno-format-zero-length 
-Wnested-externs -Wshadow -Werror=declaration-after-statement
configure: C++ compiler:   
/users/hpritchard/spack/opt/spack/linux-rhel7-aarch64/gcc-4.8.5/gcc-9.1.0-nhd4fe4i6jtn2hncfzumegojm6hsznxy/bin/g++
 -O3 -g -Wall -Werror -funwind-tables -Wno-missing-field-initializers 
-Wno-unused-parameter -Wno-unused-label -Wno-long-long -Wno-endif-labels 
-Wno-sign-compare -Wno-multichar -Wno-deprecated-declarations -Winvalid-pch
configure: Multi-thread:   enabled
configure: NUMA support:   disabled
configure:MPI tests:   disabled
configure:  VFS support:   no
configure:Devel headers:   no
configure: io_demo CUDA support:   no
configure: Bindings:   < >
configure:  UCS modules:   < >
configure:  UCT modules:   < ib cma knem >
configure: CUDA modules:   < >
configure: ROCM modules:   < >
configure:   IB modules:   < >

Re: [OMPI users] [EXTERNAL] OpenMPI 3.1.6 openib failure: "mlx4_0 errno says Success"

[Pritchard Jr., Howard via users]

HI Greg,

Oh yes that’s not good about rdmacm.

Yes the OFED looks pretty old.

Did you by any chance apply that patch?  I generated that for a sysadmin here 
who was in the situation where they needed to maintain Open MPI 3.1.6 but had 
to also upgrade to some newer RHEL release, but the Open MPi wasn’t compiling 
after the RHEL upgrade.

Howard


From: "Fischer, Greg A." 
Date: Thursday, October 14, 2021 at 1:47 PM
To: "Pritchard Jr., Howard" , Open MPI Users 

Cc: "Fischer, Greg A." 
Subject: RE: [EXTERNAL] [OMPI users] OpenMPI 3.1.6 openib failure: "mlx4_0 
errno says Success"

I added –enable-mt and re-installed UCX. Same result. (I didn’t re-compile 
OpenMPI.)

A conspicuous warning I see in my UCX configure output is:

checking for rdma_establish in -lrdmacm... no
configure: WARNING: RDMACM requested but librdmacm is not found or does not 
provide rdma_establish() API

The version of librdmacm we have comes from 
librdmacm-devel-41mlnx1-OFED.4.1.0.1.0.41102.x86_64, which seems to date from 
mid-2017. I wonder if that’s too old?

Greg

From: Pritchard Jr., Howard 
Sent: Thursday, October 14, 2021 3:31 PM
To: Fischer, Greg A. ; Open MPI Users 

Subject: Re: [EXTERNAL] [OMPI users] OpenMPI 3.1.6 openib failure: "mlx4_0 
errno says Success"

[External Email]
Hi Greg,

I think the UCX PML may be discomfited by the lack of thread safety.

Could you try using the contrib/configure-release-mt  in your ucx folder?  You 
want to add –enable-mt.
That’s what stands out in your configure output from the one I usually get when 
building on a MLNX connectx5 cluster with
MLNX_OFED_LINUX-4.5-1.0.1.0

Here’s the output from one of my UCX configs:

configure: =
configure: UCX build configuration:
configure: Build prefix:   /ucx_testing/ucx/test_install
configure:Configuration dir:   ${prefix}/etc/ucx
configure:   Preprocessor flags:   -DCPU_FLAGS="" -I${abs_top_srcdir}/src 
-I${abs_top_builddir} -I${abs_top_builddir}/src
configure:   C compiler:   
/users/hpritchard/spack/opt/spack/linux-rhel7-aarch64/gcc-4.8.5/gcc-9.1.0-nhd4fe4i6jtn2hncfzumegojm6hsznxy/bin/gcc
 -O3 -g -Wall -Werror -funwind-tables -Wno-missing-field-initializers 
-Wno-unused-parameter -Wno-unused-label -Wno-long-long -Wno-endif-labels 
-Wno-sign-compare -Wno-multichar -Wno-deprecated-declarations -Winvalid-pch 
-Wno-pointer-sign -Werror-implicit-function-declaration -Wno-format-zero-length 
-Wnested-externs -Wshadow -Werror=declaration-after-statement
configure: C++ compiler:   
/users/hpritchard/spack/opt/spack/linux-rhel7-aarch64/gcc-4.8.5/gcc-9.1.0-nhd4fe4i6jtn2hncfzumegojm6hsznxy/bin/g++
 -O3 -g -Wall -Werror -funwind-tables -Wno-missing-field-initializers 
-Wno-unused-parameter -Wno-unused-label -Wno-long-long -Wno-endif-labels 
-Wno-sign-compare -Wno-multichar -Wno-deprecated-declarations -Winvalid-pch
configure: Multi-thread:   enabled
configure: NUMA support:   disabled
configure:MPI tests:   disabled
configure:  VFS support:   no
configure:Devel headers:   no
configure: io_demo CUDA support:   no
configure: Bindings:   < >
configure:  UCS modules:   < >
configure:  UCT modules:   < ib cma knem >
configure: CUDA modules:   < >
configure: ROCM modules:   < >
configure:   IB modules:   < >
configure:  UCM modules:   < >
configure: Perf modules:   < >
configure: =


Howard

From: "Fischer, Greg A." 
mailto:fisch...@westinghouse.com>>
Date: Thursday, October 14, 2021 at 12:46 PM
To: "Pritchard Jr., Howard" mailto:howa...@lanl.gov>>, Open 
MPI Users mailto:users@lists.open-mpi.org>>
Cc: "Fischer, Greg A." 
mailto:fisch...@westinghouse.com>>
Subject: RE: [EXTERNAL] [OMPI users] OpenMPI 3.1.6 openib failure: "mlx4_0 
errno says Success"

Thanks, Howard.

I downloaded a current version of UCX (1.11.2) and installed it with OpenMPI 
4.1.1. When I try to specify the “-mca pml ucx” for a simple, 2-process 
benchmark problem, I get:

--
No components were able to be opened in the pml framework.

This typically means that either no components of this type were
installed, or none of the installed components can be loaded.
Sometimes this means that shared libraries required by these
components are unable to be found/loaded.

  Host:  bl1311
  Framework: pml
--
[bl1311:20168] PML ucx cannot be selected
[bl1311:20169] PML ucx cannot be selected


I’ve attached my ucx_info -d output, as well as the ucx configuration 
information. I’m not sure I follow everything on the UCX FAQ page, but it seems 
like everything is being routed over TCP, which is probably not what I want. 
Any thoughts as to what I might be 

Re: [OMPI users] [EXTERNAL] OpenMPI 3.1.6 openib failure: "mlx4_0 errno says Success"

[Pritchard Jr., Howard via users]

Hi Greg,

I think the UCX PML may be discomfited by the lack of thread safety.

Could you try using the contrib/configure-release-mt  in your ucx folder?  You 
want to add –enable-mt.
That’s what stands out in your configure output from the one I usually get when 
building on a MLNX connectx5 cluster with
MLNX_OFED_LINUX-4.5-1.0.1.0

Here’s the output from one of my UCX configs:

configure: =
configure: UCX build configuration:
configure: Build prefix:   /ucx_testing/ucx/test_install
configure:Configuration dir:   ${prefix}/etc/ucx
configure:   Preprocessor flags:   -DCPU_FLAGS="" -I${abs_top_srcdir}/src 
-I${abs_top_builddir} -I${abs_top_builddir}/src
configure:   C compiler:   
/users/hpritchard/spack/opt/spack/linux-rhel7-aarch64/gcc-4.8.5/gcc-9.1.0-nhd4fe4i6jtn2hncfzumegojm6hsznxy/bin/gcc
 -O3 -g -Wall -Werror -funwind-tables -Wno-missing-field-initializers 
-Wno-unused-parameter -Wno-unused-label -Wno-long-long -Wno-endif-labels 
-Wno-sign-compare -Wno-multichar -Wno-deprecated-declarations -Winvalid-pch 
-Wno-pointer-sign -Werror-implicit-function-declaration -Wno-format-zero-length 
-Wnested-externs -Wshadow -Werror=declaration-after-statement
configure: C++ compiler:   
/users/hpritchard/spack/opt/spack/linux-rhel7-aarch64/gcc-4.8.5/gcc-9.1.0-nhd4fe4i6jtn2hncfzumegojm6hsznxy/bin/g++
 -O3 -g -Wall -Werror -funwind-tables -Wno-missing-field-initializers 
-Wno-unused-parameter -Wno-unused-label -Wno-long-long -Wno-endif-labels 
-Wno-sign-compare -Wno-multichar -Wno-deprecated-declarations -Winvalid-pch
configure: Multi-thread:   enabled
configure: NUMA support:   disabled
configure:MPI tests:   disabled
configure:  VFS support:   no
configure:Devel headers:   no
configure: io_demo CUDA support:   no
configure: Bindings:   < >
configure:  UCS modules:   < >
configure:  UCT modules:   < ib cma knem >
configure: CUDA modules:   < >
configure: ROCM modules:   < >
configure:   IB modules:   < >
configure:  UCM modules:   < >
configure: Perf modules:   < >
configure: =


Howard

From: "Fischer, Greg A." 
Date: Thursday, October 14, 2021 at 12:46 PM
To: "Pritchard Jr., Howard" , Open MPI Users 

Cc: "Fischer, Greg A." 
Subject: RE: [EXTERNAL] [OMPI users] OpenMPI 3.1.6 openib failure: "mlx4_0 
errno says Success"

Thanks, Howard.

I downloaded a current version of UCX (1.11.2) and installed it with OpenMPI 
4.1.1. When I try to specify the “-mca pml ucx” for a simple, 2-process 
benchmark problem, I get:

--
No components were able to be opened in the pml framework.

This typically means that either no components of this type were
installed, or none of the installed components can be loaded.
Sometimes this means that shared libraries required by these
components are unable to be found/loaded.

  Host:  bl1311
  Framework: pml
--
[bl1311:20168] PML ucx cannot be selected
[bl1311:20169] PML ucx cannot be selected


I’ve attached my ucx_info -d output, as well as the ucx configuration 
information. I’m not sure I follow everything on the UCX FAQ page, but it seems 
like everything is being routed over TCP, which is probably not what I want. 
Any thoughts as to what I might be doing wrong?

Thanks,
Greg

From: Pritchard Jr., Howard 
Sent: Wednesday, October 13, 2021 12:28 PM
To: Open MPI Users 
Cc: Fischer, Greg A. 
Subject: Re: [EXTERNAL] [OMPI users] OpenMPI 3.1.6 openib failure: "mlx4_0 
errno says Success"

[External Email]
HI Greg,

It’s the aging of the openib btl.

You may be able to apply the attached patch.  Note the 3.1.x release stream is 
no longer supported.

You may want to try using the 4.1.1 release, in which case you’ll want to use 
UCX.

Howard


From: users 
mailto:users-boun...@lists.open-mpi.org>> on 
behalf of "Fischer, Greg A. via users" 
mailto:users@lists.open-mpi.org>>
Reply-To: Open MPI Users 
mailto:users@lists.open-mpi.org>>
Date: Wednesday, October 13, 2021 at 10:06 AM
To: "users@lists.open-mpi.org" 
mailto:users@lists.open-mpi.org>>
Cc: "Fischer, Greg A." 
mailto:fisch...@westinghouse.com>>
Subject: [EXTERNAL] [OMPI users] OpenMPI 3.1.6 openib failure: "mlx4_0 errno 
says Success"


Hello,



I have compiled OpenMPI 3.1.6 from source on SLES12-SP3, and I am seeing the 
following errors when I try to use the openib btl:



WARNING: There was an error initializing an OpenFabrics device.



  Local host:   bl1308

  Local device: mlx4_0

--

Re: [OMPI users] [EXTERNAL] OpenMPI 3.1.6 openib failure: "mlx4_0 errno says Success"

[Pritchard Jr., Howard via users]

HI Greg,

It’s the aging of the openib btl.

You may be able to apply the attached patch.  Note the 3.1.x release stream is 
no longer supported.

You may want to try using the 4.1.1 release, in which case you’ll want to use 
UCX.

Howard


From: users  on behalf of "Fischer, Greg A. 
via users" 
Reply-To: Open MPI Users 
Date: Wednesday, October 13, 2021 at 10:06 AM
To: "users@lists.open-mpi.org" 
Cc: "Fischer, Greg A." 
Subject: [EXTERNAL] [OMPI users] OpenMPI 3.1.6 openib failure: "mlx4_0 errno 
says Success"


Hello,



I have compiled OpenMPI 3.1.6 from source on SLES12-SP3, and I am seeing the 
following errors when I try to use the openib btl:



WARNING: There was an error initializing an OpenFabrics device.



  Local host:   bl1308

  Local device: mlx4_0

--

[bl1308][[44866,1],5][../../../../../openmpi-3.1.6/opal/mca/btl/openib/btl_openib_component.c:1671:init_one_device]
 error obtaining device attributes for mlx4_0 errno says Success



I have disabled UCX ("--without-ucx") because the UCX installation we have 
seems to be too out-of-date. ofed_info says "MLNX_OFED_LINUX-4.1-1.0.2.0". I've 
attached the detailed output of ofed_info and ompi_info.



This issue seems similar to Issue #7461 
(https://github.com/open-mpi/ompi/issues/7461), which I don't see a resolution 
for.



Does anyone know what the likely explanation is? Is the version of OFED on the 
system badly out-of-sync with contemporary OpenMPI?



Thanks,

Greg




This e-mail may contain proprietary information of the sending organization. 
Any unauthorized or improper disclosure, copying, distribution, or use of the 
contents of this e-mail and attached document(s) is prohibited. The information 
contained in this e-mail and attached document(s) is intended only for the 
personal and private use of the recipient(s) named above. If you have received 
this communication in error, please notify the sender immediately by email and 
delete the original e-mail and attached document(s).



0001-patch-ibv_exp_dev_query-function-call.patch
Description: 0001-patch-ibv_exp_dev_query-function-call.patch

Re: [OMPI users] [EXTERNAL] Error Signal code: Address not mapped (1)

[Pritchard Jr., Howard via users]

Hello Arturo,

Would you mind filing an issue against Open MPI and use the template to provide 
info we could use to help triage this problem?

https://github.com/open-mpi/ompi/issues/new

Thanks,

Howard


From: users  on behalf of Arturo Fernandez 
via users 
Reply-To: Open MPI Users 
Date: Monday, June 21, 2021 at 3:33 PM
To: Open MPI Users 
Cc: Arturo Fernandez 
Subject: [EXTERNAL] [OMPI users] Error Signal code: Address not mapped (1)

Hello,
I'm getting the error message (with either v4.1.0 or v4.1.1)
 *** Process received signal ***
 Signal: Segmentation fault (11)
 Signal code: Address not mapped (1)
 Failing at address: (nil)
 *** End of error message ***
Segmentation fault (core dumped)
The AWS system is running CentOS8 but I don't think that is the problem. After 
some troubleshooting, the error seems to appear and disappear depending on the 
libfabric version. When the system uses libfabric-aws-1.10.2g everything sails 
smoothly, the problems appear when libfabric-aws is upgraded to 1.11.2. I've 
tried to understand the differences between these versions but it's beyond my 
expertise.
Thanks,
Arturo

Re: [OMPI users] [EXTERNAL] Linker errors in Fedora 34 Docker container

[Pritchard Jr., Howard via users]

Hi John,

Good to know.  For the record were you using a docker container unmodified from 
docker hub?

Howard

From: John Haiducek 
Date: Wednesday, May 26, 2021 at 9:35 AM
To: "Pritchard Jr., Howard" 
Cc: "users@lists.open-mpi.org" 
Subject: Re: [EXTERNAL] [OMPI users] Linker errors in Fedora 34 Docker container

That was it, thank you! After installing findutils it builds successfully.

John


On May 26, 2021, at 10:49 AM, Pritchard Jr., Howard 
mailto:howa...@lanl.gov>> wrote:

Hi John,

I don’t like this in the make output:

../../libtool: line 5705: find: command not found

Maybe you need to install findutils or relevant fedora rpm in your container?

Howard


From: John Haiducek mailto:jhaid...@gmail.com>>
Date: Wednesday, May 26, 2021 at 7:29 AM
To: "Pritchard Jr., Howard" mailto:howa...@lanl.gov>>, 
"users@lists.open-mpi.org" 
mailto:users@lists.open-mpi.org>>
Subject: Re: [EXTERNAL] [OMPI users] Linker errors in Fedora 34 Docker container





On May 25, 2021, at 6:53 PM, Pritchard Jr., Howard 
mailto:howa...@lanl.gov>> wrote:

In your build area, do you see any .lo files in opal/util/coeval?

That directory doesn’t exist in my build area. In opal/util/keyval I have 
keyval_lex.lo.




Which compiler are you using?

gcc 11.1.1



 Also, are you building from the tarballs at 
https://www.open-mpi.org/software/ompi/v4.1/ ?

Yes; specifically I’m using the tarball from 
https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-4.1.1.tar.bz2

John

Re: [OMPI users] [EXTERNAL] Linker errors in Fedora 34 Docker container

[Pritchard Jr., Howard via users]

Hi John,

I don’t like this in the make output:


../../libtool: line 5705: find: command not found

Maybe you need to install findutils or relevant fedora rpm in your container?

Howard


From: John Haiducek 
Date: Wednesday, May 26, 2021 at 7:29 AM
To: "Pritchard Jr., Howard" , "users@lists.open-mpi.org" 

Subject: Re: [EXTERNAL] [OMPI users] Linker errors in Fedora 34 Docker container




On May 25, 2021, at 6:53 PM, Pritchard Jr., Howard 
mailto:howa...@lanl.gov>> wrote:

In your build area, do you see any .lo files in opal/util/coeval?

That directory doesn’t exist in my build area. In opal/util/keyval I have 
keyval_lex.lo.



Which compiler are you using?

gcc 11.1.1


 Also, are you building from the tarballs at 
https://www.open-mpi.org/software/ompi/v4.1/ ?

Yes; specifically I’m using the tarball from 
https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-4.1.1.tar.bz2

John

Re: [OMPI users] [EXTERNAL] Linker errors in Fedora 34 Docker container

[Pritchard Jr., Howard via users]

Hi John,

I don’t think an external dependency is going to fix this.

In your build area, do you see any .lo files in

opal/util/keyval

?

Which compiler are you using?

Also, are you building from the tarballs at 
https://www.open-mpi.org/software/ompi/v4.1/ ?

Howard

From: users  on behalf of John Haiducek via 
users 
Reply-To: Open MPI Users 
Date: Tuesday, May 25, 2021 at 3:49 PM
To: "users@lists.open-mpi.org" 
Cc: John Haiducek 
Subject: [EXTERNAL] [OMPI users] Linker errors in Fedora 34 Docker container

Hi,

When attempting to build OpenMPI in a Fedora 34 Docker image I get the 
following linker errors:


#22 77.36 make[2]: Entering directory '/build/openmpi-4.1.1/opal/tools/wrappers'

#22 77.37   CC   opal_wrapper.o

#22 77.67   CCLD opal_wrapper

#22 77.81 /usr/bin/ld: ../../../opal/.libs/libopen-pal.so: undefined reference 
to `opal_util_keyval_yytext'

#22 77.81 /usr/bin/ld: ../../../opal/.libs/libopen-pal.so: undefined reference 
to `opal_util_keyval_yyin'

#22 77.81 /usr/bin/ld: ../../../opal/.libs/libopen-pal.so: undefined reference 
to `opal_util_keyval_yylineno'

#22 77.81 /usr/bin/ld: ../../../opal/.libs/libopen-pal.so: undefined reference 
to `opal_util_keyval_yynewlines'

#22 77.81 /usr/bin/ld: ../../../opal/.libs/libopen-pal.so: undefined reference 
to `opal_util_keyval_yylex'

#22 77.81 /usr/bin/ld: ../../../opal/.libs/libopen-pal.so: undefined reference 
to `opal_util_keyval_parse_done'

#22 77.81 /usr/bin/ld: ../../../opal/.libs/libopen-pal.so: undefined reference 
to `opal_util_keyval_yylex_destroy'

#22 77.81 /usr/bin/ld: ../../../opal/.libs/libopen-pal.so: undefined reference 
to `opal_util_keyval_init_buffer'

#22 77.81 collect2: error: ld returned 1 exit status
My configure command is just ./configure --prefix=/usr/local/openmpi.
I also tried ./configure --prefix=/usr/local/openmpi --disable-silent-rules 
--enable-builtin-atomics --with-hwloc=/usr --with-libevent=external 
--with-pmix=external --with-valgrind (similar to what is in the Fedora spec 
file for OpenMPI) but that produces the same errors.

Is there a third-party library I need to install or an additional configure 
option I can set that will fix these?

John

Re: [OMPI users] [EXTERNAL] Re: Newbie With Issues

[Pritchard Jr., Howard via users]

Hi Ben,

You're heading down the right path

On our HPC systems, we use modules to handle things like setting 
LD_LIBRARY_PATH etc. when using Intel 21.x.y and other Intel compilers.
For example, for the Intel/21.1.1 the following were added to LD_LIBRARY_PATH 
(edited to avoid posting explicit paths on our systems)

prepend-path LD_LIBRARY_PATH /path_to_compiler_install 
/x86_64/oneapi/2021.1.0.2684/compiler/2021.1.1/linux/lib:path_to_compiler_install
 /x86_64/oneapi/2021.1.0.2684/compiler/2021.1.1/linux/compiler/lib/intel64_lin 
prepend-path PATH 
/path_to_compiler_install/x86_64/oneapi/2021.1.0.2684/compiler/2021.1.1/linux/bin
 
prepend-path LD_LIBRARY_PATH /path_to_compiler_install 
/x86_64/oneapi/2021.1.0.2684/compiler/2021.1.1/linux/lib/emu 
prepend-path LD_LIBRARY_PATH /path_to_compiler_install 
/x86_64/oneapi/2021.1.0.2684/compiler/2021.1.1/linux/lib/x64 
prepend-path LD_LIBRARY_PATH /path_to_compiler_isnstall 
/x86_64/oneapi/2021.1.0.2684/compiler/2021.1.1/linux/lib

You should check which intel compiler libraries you installed and make sure 
you're prepending the relevant folders to LD_LIBRARY_PATH.

We have tested building Open MPI with the Intel OneAPI compilers and except for 
ifx, things went okay. 

Howard

On 3/30/21, 11:12 AM, "users on behalf of bend linux4ms.net via users" 
 wrote:

I think I have found one of the issues. I took the check c program from 
openmpi
and tried to compile and got the following:

[root@jean-r8-sch24 benchmarks]# icc dummy.c 
ld: cannot find -lstdc++
[root@jean-r8-sch24 benchmarks]# cat dummy.c 
int
main ()
 {

  ;
  return 0;
}
[root@jean-r8-sch24 benchmarks]# 

Ben Duncan - Business Network Solutions, Inc. 336 Elton Road Jackson MS, 
39212
"Never attribute to malice, that which can be adequately explained by 
stupidity"
- Hanlon's Razor





From: users  on behalf of bend 
linux4ms.net via users 
Sent: Tuesday, March 30, 2021 12:00 PM
To: Open MPI Users
Cc: bend linux4ms.net
Subject: Re: [OMPI users] Newbie With Issues

Thanks Mr Heinz for responding.

It maybe the case with clang, but doing a intel setvars.sh then issuing the 
following
compile gives me the message:

[root@jean-r8-sch24 openmpi-4.1.0]# icc
icc: command line error: no files specified; for help type "icc -help"
[root@jean-r8-sch24 openmpi-4.1.0]# icc -v
icc version 2021.1 (gcc version 8.3.1 compatibility)
[root@jean-r8-sch24 openmpi-4.1.0]#

Would lead me to believe that icc is still available to use.

This is a government contract and they want the latest and greatest.

Ben Duncan - Business Network Solutions, Inc. 336 Elton Road Jackson MS, 
39212
"Never attribute to malice, that which can be adequately explained by 
stupidity"
- Hanlon's Razor





From: Heinz, Michael  William 
Sent: Tuesday, March 30, 2021 11:52 AM
To: Open MPI Users
Cc: bend linux4ms.net
Subject: RE: Newbie With Issues

It looks like you're trying to build Open MPI with the Intel C compiler. 
TBH - I think that icc isn't included with the latest release of oneAPI, I 
think they've switched to including clang instead. I had a similar issue to 
yours but I resolved it by installing a 2020 version of the Intel HPC software. 
Unfortunately, those versions require purchasing a license.

-Original Message-
From: users  On Behalf Of bend 
linux4ms.net via users
Sent: Tuesday, March 30, 2021 12:42 PM
To: Open MPI Open MPI 
Cc: bend linux4ms.net 
Subject: [OMPI users] Newbie With Issues

Hello group, My name is Ben Duncan. I have been tasked with installing 
openMPI and Intel compiler on a HPC systems. I am new to the the whole HPC and 
MPI environment so be patient with me.

I have successfully gotten the Intel compiler (oneapi version from  
l_HPCKit_p_2021.1.0.2684_offline.sh installed without any errors.

I am trying to install and configure the openMPI version 4.1.0 however 
trying to run configuration for openmpi gives me the following error:


== Configuring Open MPI


*** Startup tests
checking build system type... x86_64-unknown-linux-gnu checking host system 
type... x86_64-unknown-linux-gnu checking target system type... 
x86_64-unknown-linux-gnu checking for gcc... icc checking whether the C 
compiler works... no
configure: error: in `/p/app/openmpi-4.1.0':
configure: error: C compiler cannot create executables See `config.log' for 
more details

With the error in config.log being:

configure:6499: $? = 0
configure:6488: icc -qversion >&5
icc: command line warning #10006: ignoring unknown option '-qversion'
icc: 

Re: [OMPI users] [EXTERNAL] building openshem on opa

[Pritchard Jr., Howard via users]

HI Michael,

You may want to try

https://github.com/Sandia-OpenSHMEM/SOS

if you want to use OpenSHMEM over OPA.

If you have lots of cycles for development work, you could write an OFI SPML 
for the  OSHMEM component of Open MPI.  

Howard


On 3/22/21, 8:56 AM, "users on behalf of Michael Di Domenico via users" 
 wrote:

i can build and run openmpi on an opa network just fine, but it turns
out building openshmem fails.  the message is (no spml) found

looking at the config log it looks like it tries to build spml ikrit
and ucx which fail.  i turn ucx off because it doesn't support opa and
isn't needed.

so this message is really just a confirmation that openshmem and opa
are not capable of being built or did i do something wrong

and a curiosity if anyone knows what kind of effort would be involved
in getting it to work

Re: [OMPI users] [EXTERNAL] Re: OpenMPI 4.0.5 error with Omni-path

[Pritchard Jr., Howard via users]

Hi Folks,

I'm also have problems reproducing this on one of our OPA clusters:

libpsm2-11.2.78-1.el7.x86_64
libpsm2-devel-11.2.78-1.el7.x86_64

cluster runs RHEL 7.8

hca_id: hfi1_0
transport:  InfiniBand (0)
fw_ver: 1.27.0
node_guid:  0011:7501:0179:e2d7
sys_image_guid: 0011:7501:0179:e2d7
vendor_id:  0x1175
vendor_part_id: 9456
hw_ver: 0x11
board_id:   Intel Omni-Path Host Fabric Interface 
Adapter 100 Series
phys_port_cnt:  1
port:   1
state:  PORT_ACTIVE (4)
max_mtu:4096 (5)
active_mtu: 4096 (5)
sm_lid: 1
port_lid:   99
port_lmc:   0x00
link_layer: InfiniBand

using gcc/gfortran 9.3.0

Built Open MPI 4.0.5 without any special configure options.

Howard

On 1/27/21, 9:47 AM, "users on behalf of Michael Di Domenico via users" 
 wrote:

for whatever it's worth running the test program on my OPA cluster
seems to work.  well it keeps spitting out [INFO MEMORY] lines, not
sure if it's supposed to stop at some point

i'm running rhel7, gcc 10.1, openmpi 4.0.5rc2, with-ofi, 
without-{psm,ucx,verbs}

On Tue, Jan 26, 2021 at 3:44 PM Patrick Begou via users
 wrote:
>
> Hi Michael
>
> indeed I'm a little bit lost with all these parameters in OpenMPI, mainly 
because for years it works just fine out of the box in all my deployments on 
various architectures, interconnects and linux flavor. Some weeks ago I deploy 
OpenMPI4.0.5 in Centos8 with gcc10, slurm and UCX on an AMD epyc2 cluster with 
connectX6, and it just works fine.  It is the first time I've such trouble to 
deploy this library.
>
> If you have my mail posted  the 25/01/2021 in this discussion at 18h54 
(may be Paris TZ) there is a small test case attached that show the problem. 
Did you got it or did the list strip these attachments ? I can provide it again.
>
> Many thanks
>
> Patrick
>
> Le 26/01/2021 à 19:25, Heinz, Michael William a écrit :
>
> Patrick how are you using original PSM if you’re using Omni-Path 
hardware? The original PSM was written for QLogic DDR and QDR Infiniband 
adapters.
>
> As far as needing openib - the issue is that the PSM2 MTL doesn’t support 
a subset of MPI operations that we previously used the pt2pt BTL for. For 
recent version of OMPI, the preferred BTL to use with PSM2 is OFI.
>
> Is there any chance you can give us a sample MPI app that reproduces the 
problem? I can’t think of another way I can give you more help without being 
able to see what’s going on. It’s always possible there’s a bug in the PSM2 MTL 
but it would be surprising at this point.
>
> Sent from my iPad
>
> On Jan 26, 2021, at 1:13 PM, Patrick Begou via users 
 wrote:
>
> 
> Hi all,
>
> I ran many tests today. I saw that an older 4.0.2 version of OpenMPI 
packaged with Nix was running using openib. So I add the --with-verbs option to 
setup this module.
>
> That I can see now is that:
>
> mpirun -hostfile $OAR_NODEFILE  --mca mtl psm -mca btl_openib_allow_ib 
true 
>
> - the testcase test_layout_array is running without error
>
> - the bandwidth measured with osu_bw is half of thar it should be:
>
> # OSU MPI Bandwidth Test v5.7
> # Size  Bandwidth (MB/s)
> 1   0.54
> 2   1.13
> 4   2.26
> 8   4.51
> 16  9.06
> 32 17.93
> 64 33.87
> 12869.29
> 256   161.24
> 512   333.82
> 1024  682.66
> 2048 1188.63
> 4096 1760.14
> 8192 2166.08
> 163842036.95
> 327683466.63
> 655366296.73
> 131072   7509.43
> 262144   9104.78
> 524288   6908.55
> 1048576  5530.37
> 2097152  4489.16
> 4194304  3498.14
>
> mpirun -hostfile $OAR_NODEFILE  --mca mtl psm2 -mca btl_openib_allow_ib 
true ...
>
> - the testcase test_layout_array is not giving correct results
>
> - the bandwidth measured with osu_bw is the right one:
>
> # OSU MPI Bandwidth Test v5.7
> # Size  Bandwidth (MB/s)
> 1   3.73
> 2   7.96
> 4 

Re: [OMPI users] [EXTERNAL] OpenMPI 4.0.5 error with Omni-path

[Pritchard Jr., Howard via users]

Hi Patrick,

Also it might not hurt to disable the Open IB BTL by setting

export OMPI_MCA_btl=^openib

in your shell prior to invoking mpirun

Howard


From: users  on behalf of "Heinz, Michael 
William via users" 
Reply-To: Open MPI Users 
Date: Monday, January 25, 2021 at 8:47 AM
To: "users@lists.open-mpi.org" 
Cc: "Heinz, Michael William" 
Subject: [EXTERNAL] [OMPI users] OpenMPI 4.0.5 error with Omni-path

Patrick,

You really have to provide us some detailed information if you want assistance. 
At a minimum we need to know if you’re using the PSM2 MTL or the OFI MTL and 
what the actual error is.

Please provide the actual command line you are having problems with, along with 
any errors. In addition, I recommend adding the following to your command line:

-mca mtl_base_verbose 99

If you have a way to reproduce the problem quickly you might also want to add:

-x PSM2_TRACEMASK=11

But that will add very detailed debug output to your command and you haven’t 
mentioned that PSM2 is failing, so it may not be useful.

Re: [OMPI users] [EXTERNAL] RMA breakage

[Pritchard Jr., Howard via users]

Hello Dave,

There's an issue opened about this -

https://github.com/open-mpi/ompi/issues/8252

However, I'm not observing failures with IMB RMA on a IB/aarch64 system and UCX 
1.9.0 using OMPI 4.0.x at 6ea9d98.
This cluster is running RHEL 7.6 and MLNX_OFED_LINUX-4.5-1.0.1.0.

Howard

On 12/7/20, 7:21 AM, "users on behalf of Dave Love via users" 
 wrote:

After seeing several failures with RMA with the change needed to get
4.0.5 through IMB, I looked for simple tests.  So, I built the mpich
3.4b1 tests -- or the ones that would build, and I haven't checked why
some fail -- and ran the rma set.

Three out of 180 passed.  Many (most?) aborted in ucx, like I saw with
production code, with a backtrace like below; others at least reported
an MPI error.  This was on two nodes of a ppc64le RHEL7 IB system with
4.0.5, ucx 1.9, and MCA parameters from the ucx FAQ (though I got the
same result without those parameters).  I haven't tried to reproduce it
on x86_64, but it seems unlikely to be CPU-specific.

Is there anything we can do to run RMA without just moving to mpich?  Do
releases actually get tested on run-of-the-mill IB+Lustre systems?

+ mpirun -n 2 winname
[gpu005:50906:0:50906]  ucp_worker.c:183  Fatal: failed to set active 
message handler id 1: Invalid parameter
 backtrace (tid:  50906) 
 0 0x0005453c ucs_debug_print_backtrace()  
.../src/ucs/debug/debug.c:656
 1 0x00028218 ucp_worker_set_am_handlers()  
.../src/ucp/core/ucp_worker.c:182
 2 0x00029ae0 ucp_worker_iface_deactivate()  
.../src/ucp/core/ucp_worker.c:816
 3 0x00029ae0 ucp_worker_iface_check_events()  
.../src/ucp/core/ucp_worker.c:766
 4 0x00029ae0 ucp_worker_iface_deactivate()  
.../src/ucp/core/ucp_worker.c:819
 5 0x00029ae0 ucp_worker_iface_unprogress_ep()  
.../src/ucp/core/ucp_worker.c:841
 6 0x000582a8 ucp_wireup_ep_t_cleanup()  
.../src/ucp/wireup/wireup_ep.c:381
 7 0x00068124 ucs_class_call_cleanup_chain()  
.../src/ucs/type/class.c:56
 8 0x00057420 ucp_wireup_ep_t_delete()  
.../src/ucp/wireup/wireup_ep.c:28
 9 0x00013de8 uct_ep_destroy()  .../src/uct/base/uct_iface.c:546
10 0x000252f4 ucp_proxy_ep_replace()  
.../src/ucp/core/ucp_proxy_ep.c:236
11 0x00057b88 ucp_wireup_ep_progress()  
.../src/ucp/wireup/wireup_ep.c:89
12 0x00049820 ucs_callbackq_slow_proxy()  
.../src/ucs/datastruct/callbackq.c:400
13 0x0002ca04 ucs_callbackq_dispatch()  
.../src/ucs/datastruct/callbackq.h:211
14 0x0002ca04 uct_worker_progress()  .../src/uct/api/uct.h:2346
15 0x0002ca04 ucp_worker_progress()  
.../src/ucp/core/ucp_worker.c:2040
16 0xc144 progress_callback()  osc_ucx_component.c:0
17 0x000374ac opal_progress()  ???:0
18 0x0006cc74 ompi_request_default_wait()  ???:0
19 0x000e6fcc ompi_coll_base_sendrecv_actual()  ???:0
20 0x000e5530 ompi_coll_base_allgather_intra_two_procs()  ???:0
21 0x6c44 ompi_coll_tuned_allgather_intra_dec_fixed()  ???:0
22 0xdc20 component_select()  osc_ucx_component.c:0
23 0x00115b90 ompi_osc_base_select()  ???:0
24 0x00075264 ompi_win_create()  ???:0
25 0x000cb4e8 PMPI_Win_create()  ???:0
26 0x10006ecc MTestGetWin()  
.../mpich-3.4b1/test/mpi/util/mtest.c:1173
27 0x10002e40 main()  .../mpich-3.4b1/test/mpi/rma/winname.c:25
28 0x00025200 generic_start_main.isra.0()  libc-start.c:0
29 0x000253f4 __libc_start_main()  ???:0

followed by the abort backtrace