Hi Boyrie,
The warning message is coming from the older ibverbs component of the Open MPI
4.0/4.1 releases.
You can make this message using several ways. One at configure time is to add
--disable-verbs
to the configure options.
At runtime you can set
export OMPI_MCA_btl=^openib
The ucx messages are just being chatty about which ucx transport type is being
selected.
The VASP hang may be something else.
Howard
From: users <users-boun...@lists.open-mpi.org> on behalf of Boyrie Fabrice via
users <users@lists.open-mpi.org>
Reply-To: Open MPI Users <users@lists.open-mpi.org>
Date: Friday, August 19, 2022 at 9:51 AM
To: "users@lists.open-mpi.org" <users@lists.open-mpi.org>
Cc: Boyrie Fabrice <fabrice.boy...@umontpellier.fr>
Subject: [EXTERNAL] [OMPI users] Problem with Mellanox ConnectX3 (FDR) and
openmpi 4
Hi
I had to reinstall a cluster in AlmaLinux 8.6
I am unable to make openmpi 4 working with infiniband. I have the following
message in a trivial pingpong test
mpirun --hostfile hostfile -np 2 pingpong
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.
Local host: node2
Local device: mlx4_0
--------------------------------------------------------------------------
[node2:12431] common_ucx.c:107 using OPAL memory hooks as external events
[node2:12431] pml_ucx.c:197 mca_pml_ucx_open: UCX version 1.11.2
[node1:13188] common_ucx.c:174 using OPAL memory hooks as external events
[node1:13188] pml_ucx.c:197 mca_pml_ucx_open: UCX version 1.11.2
[node2:12431] pml_ucx.c:289 mca_pml_ucx_init
[node1:13188] common_ucx.c:333 posix/memory: did not match transport list
[node1:13188] common_ucx.c:333 sysv/memory: did not match transport list
[node1:13188] common_ucx.c:333 self/memory0: did not match transport list
[node1:13188] common_ucx.c:333 tcp/lo: did not match transport list
[node1:13188] common_ucx.c:333 tcp/eno1: did not match transport list
[node1:13188] common_ucx.c:333 tcp/ib0: did not match transport list
[node1:13188] common_ucx.c:228 driver '../../../../bus/pci/drivers/mlx4_core'
matched by 'mlx*'
[node1:13188] common_ucx.c:324 rc_verbs/mlx4_0:1: matched both transport and
device list
[node1:13188] common_ucx.c:337 support level is transports and devices
[node1:13188] pml_ucx.c:289 mca_pml_ucx_init
[node2:12431] pml_ucx.c:114 Pack remote worker address, size 155
[node2:12431] pml_ucx.c:114 Pack local worker address, size 291
[node2:12431] pml_ucx.c:351 created ucp context 0xf832a0, worker 0x109fc50
[node1:13188] pml_ucx.c:114 Pack remote worker address, size 155
[node1:13188] pml_ucx.c:114 Pack local worker address, size 291
[node1:13188] pml_ucx.c:351 created ucp context 0x1696320, worker 0x16c9ce0
[node1:13188] pml_ucx_component.c:147 returning priority 51
[node2:12431] pml_ucx.c:182 Got proc 0 address, size 291
[node2:12431] pml_ucx.c:411 connecting to proc. 0
[node1:13188] pml_ucx.c:182 Got proc 1 address, size 291
[node1:13188] pml_ucx.c:411 connecting to proc. 1
length time/message (usec) transfer rate (Gbyte/sec)
[node2:12431] pml_ucx.c:182 Got proc 1 address, size 155
[node2:12431] pml_ucx.c:411 connecting to proc. 1
[node1:13188] pml_ucx.c:182 Got proc 0 address, size 155
[node1:13188] pml_ucx.c:411 connecting to proc. 0
1 45.683729 0.000088
1001 4.286029 0.934198
2001 5.755391 1.390696
3001 6.902443 1.739095
4001 8.485305 1.886084
5001 9.596994 2.084403
6001 11.055146 2.171297
7001 11.977093 2.338130
8001 13.324408 2.401908
9001 14.471116 2.487991
10001 15.806676 2.530829
[node2:12431] common_ucx.c:240 disconnecting from rank 0
[node2:12431] common_ucx.c:240 disconnecting from rank 1
[node2:12431] common_ucx.c:204 waiting for 1 disconnect requests
[node2:12431] common_ucx.c:204 waiting for 0 disconnect requests
[node1:13188] common_ucx.c:466 disconnecting from rank 0
[node1:13188] common_ucx.c:430 waiting for 1 disconnect requests
[node1:13188] common_ucx.c:466 disconnecting from rank 1
[node1:13188] common_ucx.c:430 waiting for 0 disconnect requests
[node2:12431] pml_ucx.c:367 mca_pml_ucx_cleanup
[node1:13188] pml_ucx.c:367 mca_pml_ucx_cleanup
[node2:12431] pml_ucx.c:268 mca_pml_ucx_close
[node1:13188] pml_ucx.c:268 mca_pml_ucx_close
cat hostfile
node1 slots=1
node2 slots=1
And with a real program (Vasp) it stops.
Infinband seems to be working. I can ssh over infiniband and qperf works in
rdma mode
qperf -t 10 ibnode1 ud_lat ud_bw
ud_lat:
latency = 18.2 us
ud_bw:
send_bw = 2.81 GB/sec
recv_bw = 2.81 GB/sec
I use the standard AlmaLinux module for infiniband
82:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
I can not install MLNX_OFED_LINUX-5.6-2.0.9.0-rhel8.6-x86_64 because it does
not supports ConnectX-3
And I can not install MLNX_OFED_LINUX-4.9-5.1.0.0-rhel8.6-x86_64 because the
module compilation fails.
I have compiled with --without-verbs as asked in some web pages.
Could someone help me ?
Thanks
Fabrice Boyrie
ucx_info -v
# UCT version=1.11.2 revision ef2bbcf
# configured with: --build=x86_64-redhat-linux-gnu
--host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking
--prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin
--sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include
--libdir=/usr/lib64 --libexecdir=/usr/libexec --lo
calstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man
--infodir=/usr/share/info --disable-optimizations --disable-logging
--disable-debug --disable-assertions --disable-params-check --without-java
--enable-cma --without-cuda --without-gdrcopy --with-verbs --without-cm
--without-knem --with-rdmacm --wit
hout-rocm --without-xpmem --without-fuse3 --without-ugni
ompi_info
Package: Open MPI
fboy...@bipbip.d5.icgm.fr<mailto:fboy...@bipbip.d5.icgm.fr> Distribution
Open MPI: 4.1.4
Open MPI repo revision: v4.1.4
Open MPI release date: May 26, 2022
Open RTE: 4.1.4
Open RTE repo revision: v4.1.4
Open RTE release date: May 26, 2022
OPAL: 4.1.4
OPAL repo revision: v4.1.4
OPAL release date: May 26, 2022
MPI API: 3.1.0
Ident string: 4.1.4
Prefix: /opt/intel-small
Configured architecture: x86_64-pc-linux-gnu
Configure host:
bipbip.d5.icgm.fr<https://urldefense.com/v3/__http:/bipbip.d5.icgm.fr__;!!Bt8fGhp8LhKGRg!CeO_nWZkoeam0WLm0bIckB-wvA5-uuRYPqd7W2jJFBVX1pvueYkRA0r_UXBh-I_74UTvb4wcxtJ0q59_xa-_$>
Configured by: fboyrie
Configured on: Fri Aug 19 14:54:00 UTC 2022
Configure host:
bipbip.d5.icgm.fr<https://urldefense.com/v3/__http:/bipbip.d5.icgm.fr__;!!Bt8fGhp8LhKGRg!CeO_nWZkoeam0WLm0bIckB-wvA5-uuRYPqd7W2jJFBVX1pvueYkRA0r_UXBh-I_74UTvb4wcxtJ0q59_xa-_$>
Configure command line: '--prefix=/opt/intel-small/' '--without-verbs'
Built by: fboyrie
Built on: ven. août 19 15:11:38 UTC 2022
Built host:
bipbip.d5.icgm.fr<https://urldefense.com/v3/__http:/bipbip.d5.icgm.fr__;!!Bt8fGhp8LhKGRg!CeO_nWZkoeam0WLm0bIckB-wvA5-uuRYPqd7W2jJFBVX1pvueYkRA0r_UXBh-I_74UTvb4wcxtJ0q59_xa-_$>
C bindings: yes
C++ bindings: no
Fort mpif.h: yes (all)
Fort use mpi: yes (full: ignore TKR)
Fort use mpi size: deprecated-ompi-info-value
Fort use mpi_f08: yes
Fort mpi_f08 compliance: The mpi_f08 module is available, but due to
limitations in the ifort compiler and/or Open MPI,
does not support the following: array subsections,
direct passthru (where possible) to underlying Open
MPI's C functionality
Fort mpi_f08 subarrays: no
Java bindings: no
Wrapper compiler rpath: runpath
C compiler: gcc
C compiler absolute: /usr/bin/gcc
C compiler family name: GNU
C compiler version: 8.5.0
C++ compiler: g++
C++ compiler absolute: /usr/bin/g++
Fort compiler: ifort
Fort compiler abs:
/opt/intel/oneapi/compiler/2021.3.0/linux/bin/intel64/ifort
Fort ignore TKR: yes (!DEC$ ATTRIBUTES NO_ARG_CHECK ::)
Fort 08 assumed shape: yes
Fort optional args: yes
Fort INTERFACE: yes
Fort ISO_FORTRAN_ENV: yes
Fort STORAGE_SIZE: yes
Fort BIND(C) (all): yes
Fort ISO_C_BINDING: yes
Fort SUBROUTINE BIND(C): yes
Fort TYPE,BIND(C): yes
Fort T,BIND(C,name="a"): yes
Fort PRIVATE: yes
Fort PROTECTED: yes
Fort ABSTRACT: yes
Fort ASYNCHRONOUS: yes
Fort PROCEDURE: yes
Fort USE...ONLY: yes
Fort C_FUNLOC: yes
Fort f08 using wrappers: yes
Fort MPI_SIZEOF: yes
C profiling: yes
C++ profiling: no
Fort mpif.h profiling: yes
Fort use mpi profiling: yes
Fort use mpi_f08 prof: yes
C++ exceptions: no
Thread support: posix (MPI_THREAD_MULTIPLE: yes, OPAL support: yes,
OMPI progress: no, ORTE progress: yes, Event lib:
yes)
Sparse Groups: no
Internal debug support: no
MPI interface warnings: yes
MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
dl support: yes
Heterogeneous support: no
mpirun default --prefix: no
MPI_WTIME support: native
Symbol vis. support: yes
Host topology support: yes
IPv6 support: no
MPI1 compatibility: no
MPI extensions: affinity, cuda, pcollreq
FT Checkpoint support: no (checkpoint thread: no)
C/R Enabled Debugging: no
MPI_MAX_PROCESSOR_NAME: 256
MPI_MAX_ERROR_STRING: 256
MPI_MAX_OBJECT_NAME: 64
MPI_MAX_INFO_KEY: 36
MPI_MAX_INFO_VAL: 256
MPI_MAX_PORT_NAME: 1024
MPI_MAX_DATAREP_STRING: 128
MCA allocator: bucket (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA allocator: basic (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA backtrace: execinfo (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA btl: tcp (MCA v2.1.0, API v3.1.0, Component v4.1.4)
MCA btl: ofi (MCA v2.1.0, API v3.1.0, Component v4.1.4)
MCA btl: usnic (MCA v2.1.0, API v3.1.0, Component v4.1.4)
MCA btl: vader (MCA v2.1.0, API v3.1.0, Component v4.1.4)
MCA btl: self (MCA v2.1.0, API v3.1.0, Component v4.1.4)
MCA compress: bzip (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA compress: gzip (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA crs: none (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA dl: dlopen (MCA v2.1.0, API v1.0.0, Component v4.1.4)
MCA event: libevent2022 (MCA v2.1.0, API v2.0.0, Component
v4.1.4)
MCA hwloc: external (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA if: linux_ipv6 (MCA v2.1.0, API v2.0.0, Component
v4.1.4)
MCA if: posix_ipv4 (MCA v2.1.0, API v2.0.0, Component
v4.1.4)
MCA installdirs: env (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA installdirs: config (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA memory: patcher (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA mpool: hugepage (MCA v2.1.0, API v3.0.0, Component v4.1.4)
MCA patcher: overwrite (MCA v2.1.0, API v1.0.0, Component
v4.1.4)
MCA pmix: isolated (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA pmix: pmix3x (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA pmix: flux (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA pstat: linux (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA rcache: grdma (MCA v2.1.0, API v3.3.0, Component v4.1.4)
MCA reachable: weighted (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA shmem: posix (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA shmem: mmap (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA shmem: sysv (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA timer: linux (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA errmgr: default_app (MCA v2.1.0, API v3.0.0, Component
v4.1.4)
MCA errmgr: default_hnp (MCA v2.1.0, API v3.0.0, Component
v4.1.4)
MCA errmgr: default_tool (MCA v2.1.0, API v3.0.0, Component
v4.1.4)
MCA errmgr: default_orted (MCA v2.1.0, API v3.0.0, Component
v4.1.4)
MCA ess: hnp (MCA v2.1.0, API v3.0.0, Component v4.1.4)
MCA ess: pmi (MCA v2.1.0, API v3.0.0, Component v4.1.4)
MCA ess: slurm (MCA v2.1.0, API v3.0.0, Component v4.1.4)
MCA ess: env (MCA v2.1.0, API v3.0.0, Component v4.1.4)
MCA ess: singleton (MCA v2.1.0, API v3.0.0, Component
v4.1.4)
MCA ess: tool (MCA v2.1.0, API v3.0.0, Component v4.1.4)
MCA filem: raw (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA grpcomm: direct (MCA v2.1.0, API v3.0.0, Component v4.1.4)
MCA iof: hnp (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA iof: orted (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA iof: tool (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA odls: pspawn (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA odls: default (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA oob: tcp (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA plm: rsh (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA plm: isolated (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA plm: slurm (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA ras: simulator (MCA v2.1.0, API v2.0.0, Component
v4.1.4)
MCA ras: slurm (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA regx: fwd (MCA v2.1.0, API v1.0.0, Component v4.1.4)
MCA regx: reverse (MCA v2.1.0, API v1.0.0, Component v4.1.4)
MCA regx: naive (MCA v2.1.0, API v1.0.0, Component v4.1.4)
MCA rmaps: mindist (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA rmaps: ppr (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA rmaps: round_robin (MCA v2.1.0, API v2.0.0, Component
v4.1.4)
MCA rmaps: seq (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA rmaps: rank_file (MCA v2.1.0, API v2.0.0, Component
v4.1.4)
MCA rmaps: resilient (MCA v2.1.0, API v2.0.0, Component
v4.1.4)
MCA rml: oob (MCA v2.1.0, API v3.0.0, Component v4.1.4)
MCA routed: radix (MCA v2.1.0, API v3.0.0, Component v4.1.4)
MCA routed: binomial (MCA v2.1.0, API v3.0.0, Component v4.1.4)
MCA routed: direct (MCA v2.1.0, API v3.0.0, Component v4.1.4)
MCA rtc: hwloc (MCA v2.1.0, API v1.0.0, Component v4.1.4)
MCA schizo: flux (MCA v2.1.0, API v1.0.0, Component v4.1.4)
MCA schizo: ompi (MCA v2.1.0, API v1.0.0, Component v4.1.4)
MCA schizo: slurm (MCA v2.1.0, API v1.0.0, Component v4.1.4)
MCA schizo: orte (MCA v2.1.0, API v1.0.0, Component v4.1.4)
MCA schizo: jsm (MCA v2.1.0, API v1.0.0, Component v4.1.4)
MCA state: tool (MCA v2.1.0, API v1.0.0, Component v4.1.4)
MCA state: hnp (MCA v2.1.0, API v1.0.0, Component v4.1.4)
MCA state: orted (MCA v2.1.0, API v1.0.0, Component v4.1.4)
MCA state: novm (MCA v2.1.0, API v1.0.0, Component v4.1.4)
MCA state: app (MCA v2.1.0, API v1.0.0, Component v4.1.4)
MCA bml: r2 (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA coll: han (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA coll: sm (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA coll: basic (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA coll: libnbc (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA coll: monitoring (MCA v2.1.0, API v2.0.0, Component
v4.1.4)
MCA coll: inter (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA coll: tuned (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA coll: sync (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA coll: adapt (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA coll: self (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA fbtl: posix (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA fcoll: dynamic_gen2 (MCA v2.1.0, API v2.0.0, Component
v4.1.4)
MCA fcoll: two_phase (MCA v2.1.0, API v2.0.0, Component
v4.1.4)
MCA fcoll: vulcan (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA fcoll: individual (MCA v2.1.0, API v2.0.0, Component
v4.1.4)
MCA fcoll: dynamic (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA fs: ufs (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA io: ompio (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA io: romio321 (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA mtl: ofi (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA op: avx (MCA v2.1.0, API v1.0.0, Component v4.1.4)
MCA osc: monitoring (MCA v2.1.0, API v3.0.0, Component
v4.1.4)
MCA osc: sm (MCA v2.1.0, API v3.0.0, Component v4.1.4)
MCA osc: ucx (MCA v2.1.0, API v3.0.0, Component v4.1.4)
MCA osc: rdma (MCA v2.1.0, API v3.0.0, Component v4.1.4)
MCA osc: pt2pt (MCA v2.1.0, API v3.0.0, Component v4.1.4)
MCA pml: v (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA pml: ob1 (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA pml: monitoring (MCA v2.1.0, API v2.0.0, Component
v4.1.4)
MCA pml: ucx (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA pml: cm (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA rte: orte (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA sharedfp: sm (MCA v2.1.0, API v2.0.0, Component v4.1.4)
MCA sharedfp: lockedfile (MCA v2.1.0, API v2.0.0, Component
v4.1.4)
MCA sharedfp: individual (MCA v2.1.0, API v2.0.0, Component
v4.1.4)
MCA topo: basic (MCA v2.1.0, API v2.2.0, Component v4.1.4)
MCA topo: treematch (MCA v2.1.0, API v2.2.0, Component
v4.1.4)
MCA vprotocol: pessimist (MCA v2.1.0, API v2.0.0, Component
v4.1.4)
