One other update. I compiled OpenMPI-4.0.4 The outcome was the same but
there is no mention of ibv_obj this time.
Tony
--
Tony Ladd
Chemical Engineering Department
University of Florida
Gainesville, Florida 32611-6005
USA
Email: tladd-"(AT)"-che.ufl.edu
Web http://ladd.che.ufl.edu
Tel: (352)-392-6509
FAX: (352)-392-9514
f34:tladd(~)> mpirun -d --report-bindings --mca btl_openib_allow_ib 1 --mca btl
openib,self --mca btl_base_verbose 30 -np 2
mpi-benchmarks-IMB-v2019.3/src_c/IMB-MPI1 SendRecv
[f34:24079] procdir: /tmp/ompi.f34.501/pid.24079/0/0
[f34:24079] jobdir: /tmp/ompi.f34.501/pid.24079/0
[f34:24079] top: /tmp/ompi.f34.501/pid.24079
[f34:24079] top: /tmp/ompi.f34.501
[f34:24079] tmp: /tmp
[f34:24079] sess_dir_cleanup: job session dir does not exist
[f34:24079] sess_dir_cleanup: top session dir not empty - leaving
[f34:24079] procdir: /tmp/ompi.f34.501/pid.24079/0/0
[f34:24079] jobdir: /tmp/ompi.f34.501/pid.24079/0
[f34:24079] top: /tmp/ompi.f34.501/pid.24079
[f34:24079] top: /tmp/ompi.f34.501
[f34:24079] tmp: /tmp
[f34:24079] [[62672,0],0] Releasing job data for [INVALID]
[f34:24079] MCW rank 0 bound to socket 0[core 0[hwt 0]]: [B/././.][./././.]
[f34:24079] MCW rank 1 bound to socket 0[core 1[hwt 0]]: [./B/./.][./././.]
MPIR_being_debugged = 0
MPIR_debug_state = 1
MPIR_partial_attach_ok = 1
MPIR_i_am_starter = 0
MPIR_forward_output = 0
MPIR_proctable_size = 2
MPIR_proctable:
(i, host, exe, pid) = (0, f34,
/home/tladd/mpi-benchmarks-IMB-v2019.3/src_c/IMB-MPI1, 24083)
(i, host, exe, pid) = (1, f34,
/home/tladd/mpi-benchmarks-IMB-v2019.3/src_c/IMB-MPI1, 24084)
MPIR_executable_path: NULL
MPIR_server_arguments: NULL
[f34:24084] procdir: /tmp/ompi.f34.501/pid.24079/1/1
[f34:24084] jobdir: /tmp/ompi.f34.501/pid.24079/1
[f34:24084] top: /tmp/ompi.f34.501/pid.24079
[f34:24084] top: /tmp/ompi.f34.501
[f34:24084] tmp: /tmp
[f34:24083] procdir: /tmp/ompi.f34.501/pid.24079/1/0
[f34:24083] jobdir: /tmp/ompi.f34.501/pid.24079/1
[f34:24083] top: /tmp/ompi.f34.501/pid.24079
[f34:24083] top: /tmp/ompi.f34.501
[f34:24083] tmp: /tmp
[f34:24084] mca: base: components_register: registering framework btl components
[f34:24084] mca: base: components_register: found loaded component self
[f34:24084] mca: base: components_register: component self register function
successful
[f34:24084] mca: base: components_register: found loaded component openib
[f34:24084] mca: base: components_register: component openib register function
successful
[f34:24084] mca: base: components_open: opening btl components
[f34:24084] mca: base: components_open: found loaded component self
[f34:24084] mca: base: components_open: component self open function successful
[f34:24084] mca: base: components_open: found loaded component openib
[f34:24084] mca: base: components_open: component openib open function
successful
[f34:24084] select: initializing btl component self
[f34:24084] select: init of component self returned success
[f34:24084] select: initializing btl component openib
[f34:24083] mca: base: components_register: registering framework btl components
[f34:24083] mca: base: components_register: found loaded component self
[f34:24083] mca: base: components_register: component self register function
successful
[f34:24083] mca: base: components_register: found loaded component openib
[f34:24083] mca: base: components_register: component openib register function
successful
[f34:24083] mca: base: components_open: opening btl components
[f34:24083] mca: base: components_open: found loaded component self
[f34:24083] mca: base: components_open: component self open function successful
[f34:24083] mca: base: components_open: found loaded component openib
[f34:24083] mca: base: components_open: component openib open function
successful
[f34:24083] select: initializing btl component self
[f34:24083] select: init of component self returned success
[f34:24083] select: initializing btl component openib
[f34:24084] Checking distance from this process to device=mlx4_0
[f34:24084] Process is not bound: distance to device is 0.000000
[f34:24083] Checking distance from this process to device=mlx4_0
[f34:24083] Process is not bound: distance to device is 0.000000
[f34:24083] [rank=0] openib: using port mlx4_0:1
[f34:24083] select: init of component openib returned success
[f34:24084] [rank=1] openib: using port mlx4_0:1
[f34:24084] select: init of component openib returned success
[f34:24083] mca: bml: Using self btl for send to [[62672,1],0] on node f34
[f34:24084] mca: bml: Using self btl for send to [[62672,1],1] on node f34
^C
[f34:24079] sess_dir_finalize: proc session dir does not exist
[f34:24079] sess_dir_finalize: job session dir does not exist
[f34:24079] sess_dir_finalize: jobfam session dir not empty - leaving
[f34:24079] sess_dir_finalize: jobfam session dir not empty - leaving
[f34:24079] sess_dir_finalize: top session dir not empty - leaving
[f34:24079] sess_dir_finalize: proc session dir does not exist
[f34:24079] sess_dir_finalize: job session dir does not exist
[f34:24079] sess_dir_finalize: jobfam session dir not empty - leaving
[f34:24079] sess_dir_finalize: jobfam session dir not empty - leaving
[f34:24079] sess_dir_finalize: top session dir not empty - leaving
[f34:24079] Job UNKNOWN has launched
[f34:24079] [[62672,0],0] Releasing job data for [62672,1]
[f34:24079] sess_dir_finalize: proc session dir does not exist
[f34:24079] sess_dir_finalize: job session dir does not exist
[f34:24079] sess_dir_finalize: jobfam session dir does not exist
[f34:24079] sess_dir_finalize: jobfam session dir does not exist
[f34:24079] sess_dir_finalize: top session dir not empty - leaving
[f34:24079] sess_dir_cleanup: job session dir does not exist
[f34:24079] sess_dir_cleanup: top session dir not empty - leaving
[f34:24079] [[62672,0],0] Releasing job data for [62672,0]
[f34:24079] sess_dir_cleanup: job session dir does not exist
[f34:24079] sess_dir_cleanup: top session dir not empty - leaving
exiting with status 1
foam:tladd(~)> mpirun -d --report-bindings --mca btl_openib_allow_ib 1 --mca
btl openib,self --mca btl_base_verbose 30 -np 2
mpi-benchmarks-IMB-v2019.3/src_c/IMB-MPI1 SendRecv
[foam:07245] procdir: /tmp/ompi.foam.501/pid.7245/0/0
[foam:07245] jobdir: /tmp/ompi.foam.501/pid.7245/0
[foam:07245] top: /tmp/ompi.foam.501/pid.7245
[foam:07245] top: /tmp/ompi.foam.501
[foam:07245] tmp: /tmp
[foam:07245] sess_dir_cleanup: job session dir does not exist
[foam:07245] sess_dir_cleanup: top session dir does not exist
[foam:07245] procdir: /tmp/ompi.foam.501/pid.7245/0/0
[foam:07245] jobdir: /tmp/ompi.foam.501/pid.7245/0
[foam:07245] top: /tmp/ompi.foam.501/pid.7245
[foam:07245] top: /tmp/ompi.foam.501
[foam:07245] tmp: /tmp
[foam:07245] [[2911,0],0] Releasing job data for [INVALID]
[foam:07245] MCW rank 0 bound to socket 0[core 0[hwt 0-1]]:
[BB/../../../../../../../../..][../../../../../../../../../..]
[foam:07245] MCW rank 1 bound to socket 0[core 1[hwt 0-1]]:
[../BB/../../../../../../../..][../../../../../../../../../..]
MPIR_being_debugged = 0
MPIR_debug_state = 1
MPIR_partial_attach_ok = 1
MPIR_i_am_starter = 0
MPIR_forward_output = 0
MPIR_proctable_size = 2
MPIR_proctable:
(i, host, exe, pid) = (0, foam,
/home/tladd/mpi-benchmarks-IMB-v2019.3/src_c/IMB-MPI1, 7250)
(i, host, exe, pid) = (1, foam,
/home/tladd/mpi-benchmarks-IMB-v2019.3/src_c/IMB-MPI1, 7251)
MPIR_executable_path: NULL
MPIR_server_arguments: NULL
[foam:07250] procdir: /tmp/ompi.foam.501/pid.7245/1/0
[foam:07250] jobdir: /tmp/ompi.foam.501/pid.7245/1
[foam:07250] top: /tmp/ompi.foam.501/pid.7245
[foam:07250] top: /tmp/ompi.foam.501
[foam:07250] tmp: /tmp
[foam:07251] procdir: /tmp/ompi.foam.501/pid.7245/1/1
[foam:07251] jobdir: /tmp/ompi.foam.501/pid.7245/1
[foam:07251] top: /tmp/ompi.foam.501/pid.7245
[foam:07251] top: /tmp/ompi.foam.501
[foam:07251] tmp: /tmp
[foam:07250] mca: base: components_register: registering framework btl
components
[foam:07250] mca: base: components_register: found loaded component self
[foam:07251] mca: base: components_register: registering framework btl
components
[foam:07251] mca: base: components_register: found loaded component self
[foam:07251] mca: base: components_register: component self register function
successful
[foam:07250] mca: base: components_register: component self register function
successful
[foam:07251] mca: base: components_register: found loaded component openib
[foam:07250] mca: base: components_register: found loaded component openib
[foam:07251] mca: base: components_register: component openib register function
successful
[foam:07251] mca: base: components_open: opening btl components
[foam:07251] mca: base: components_open: found loaded component self
[foam:07251] mca: base: components_open: component self open function successful
[foam:07251] mca: base: components_open: found loaded component openib
[foam:07251] mca: base: components_open: component openib open function
successful
[foam:07251] select: initializing btl component self
[foam:07251] select: init of component self returned success
[foam:07251] select: initializing btl component openib
[foam:07250] mca: base: components_register: component openib register function
successful
[foam:07250] mca: base: components_open: opening btl components
[foam:07250] mca: base: components_open: found loaded component self
[foam:07250] mca: base: components_open: component self open function successful
[foam:07250] mca: base: components_open: found loaded component openib
[foam:07250] mca: base: components_open: component openib open function
successful
[foam:07250] select: initializing btl component self
[foam:07250] select: init of component self returned success
[foam:07250] select: initializing btl component openib
[foam:07251] Checking distance from this process to device=mlx4_0
[foam:07251] Process is not bound: distance to device is 0.000000
[foam:07250] Checking distance from this process to device=mlx4_0
[foam:07250] Process is not bound: distance to device is 0.000000
[foam:07251] [rank=1] openib: using port mlx4_0:1
[foam:07251] select: init of component openib returned success
[foam:07250] [rank=0] openib: using port mlx4_0:1
[foam:07250] select: init of component openib returned success
[foam:07250] mca: bml: Using self btl for send to [[2911,1],0] on node foam
[foam:07251] mca: bml: Using self btl for send to [[2911,1],1] on node foam
[foam:07251] mca: bml: Using openib btl for send to [[2911,1],0] on node foam
[foam:07250] mca: bml: Using openib btl for send to [[2911,1],1] on node foam
#------------------------------------------------------------
# Intel(R) MPI Benchmarks 2018, MPI-1 part
#------------------------------------------------------------
# Date : Wed Aug 19 12:27:33 2020
# Machine : x86_64
# System : Linux
# Release : 4.15.0-112-generic
# Version : #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020
# MPI Version : 3.1
# MPI Thread Environment:
# Calling sequence was:
# mpi-benchmarks-IMB-v2019.3/src_c/IMB-MPI1 SendRecv
# Minimum message length in bytes: 0
# Maximum message length in bytes: 4194304
#
# MPI_Datatype : MPI_BYTE
# MPI_Datatype for reductions : MPI_FLOAT
# MPI_Op : MPI_SUM
#
#
# List of Benchmarks to run:
# Sendrecv
#-----------------------------------------------------------------------------
# Benchmarking Sendrecv
# #processes = 2
#-----------------------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec] Mbytes/sec
0 1000 1.30 1.30 1.30 0.00
1 1000 1.37 1.37 1.37 1.46
2 1000 1.36 1.36 1.36 2.93
4 1000 1.35 1.35 1.35 5.91
8 1000 1.38 1.38 1.38 11.57
16 1000 1.39 1.39 1.39 22.98
32 1000 1.42 1.42 1.42 45.02
64 1000 1.53 1.53 1.53 83.49
128 1000 2.76 2.76 2.76 92.70
256 1000 2.96 2.96 2.96 173.20
512 1000 3.50 3.50 3.50 292.56
1024 1000 4.12 4.12 4.12 496.52
2048 1000 5.43 5.43 5.43 753.95
4096 1000 6.52 6.52 6.52 1255.64
8192 1000 9.37 9.38 9.37 1747.40
16384 1000 15.62 15.62 15.62 2097.40
32768 1000 25.31 25.31 25.31 2589.36
65536 640 45.79 45.79 45.79 2862.59
131072 320 85.48 85.48 85.48 3066.83
262144 160 168.19 168.19 168.19 3117.16
524288 80 330.78 330.78 330.78 3170.01
1048576 40 667.66 667.67 667.66 3141.01
2097152 20 1336.98 1336.99 1336.98 3137.13
4194304 10 3025.18 3025.25 3025.21 2772.86
# All processes entering MPI_Finalize
[foam:07251] mca: base: close: component self closed
[foam:07251] mca: base: close: unloading component self
[foam:07251] mca: base: close: component openib closed
[foam:07251] mca: base: close: unloading component openib
[foam:07250] mca: base: close: component self closed
[foam:07250] mca: base: close: unloading component self
[foam:07250] mca: base: close: component openib closed
[foam:07250] mca: base: close: unloading component openib
[foam:07245] sess_dir_finalize: proc session dir does not exist
[foam:07245] sess_dir_finalize: job session dir does not exist
[foam:07245] sess_dir_finalize: jobfam session dir not empty - leaving
[foam:07245] sess_dir_finalize: jobfam session dir not empty - leaving
[foam:07245] sess_dir_finalize: top session dir not empty - leaving
[foam:07245] sess_dir_finalize: proc session dir does not exist
[foam:07245] sess_dir_finalize: job session dir does not exist
[foam:07245] sess_dir_finalize: jobfam session dir not empty - leaving
[foam:07245] sess_dir_finalize: jobfam session dir not empty - leaving
[foam:07245] sess_dir_finalize: top session dir not empty - leaving
[foam:07245] Job UNKNOWN has launched
[foam:07245] [[2911,0],0] Releasing job data for [2911,1]
[foam:07245] sess_dir_finalize: proc session dir does not exist
[foam:07245] sess_dir_finalize: job session dir does not exist
[foam:07245] sess_dir_finalize: jobfam session dir does not exist
[foam:07245] sess_dir_finalize: jobfam session dir does not exist
[foam:07245] sess_dir_finalize: top session dir does not exist
[foam:07245] sess_dir_cleanup: job session dir does not exist
[foam:07245] sess_dir_cleanup: top session dir does not exist
[foam:07245] [[2911,0],0] Releasing job data for [2911,0]
[foam:07245] sess_dir_cleanup: job session dir does not exist
[foam:07245] sess_dir_cleanup: top session dir does not exist
exiting with status 0
