I performed some tests on our Omnipath cluster, and I have a mixed bag of 
results with 4.0.0rc1

1.       Good news, the problems with the psm2 mtl that I reported in June/July 
seem to be fixed. I still get however a warning every time I run a job with 
4.0.0, e.g.

compute-1-1.local.4351PSM2 has not been initialized

compute-1-0.local.3826PSM2 has not been initialized

although based on the performance, it is very clear that psm2 is being used. I 
double checked with 3.0 series, I do not get the same warnings on the same
set of nodes. The unfortunate part about this error  message is, that it seems 
that applications seem to return an error (although tests and applications seem 
finish correctly otherwise)

mpirun detected that one or more processes exited with non-zero status, thus 
the job to be terminated. The first process to do so was:

              Process name: [[38418,1],1]
              Exit code:    255

2.       The ofi mtl does not work at all on our Omnipath cluster. If I try to 
force it using ‘mpirun –mca mtl ofi …’ I get the following error message.

[compute-1-0:03988] *** An error occurred in MPI_Barrier

[compute-1-0:03988] *** reported by process [2712141825,0]

[compute-1-0:03988] *** on communicator MPI_COMM_WORLD

[compute-1-0:03988] *** MPI_ERR_OTHER: known error not in list

[compute-1-0:03988] *** MPI_ERRORS_ARE_FATAL (processes in this communicator 
will now abort,

[compute-1-0:03988] ***    and potentially your MPI job)

[sabine.cacds.uh.edu:21046] 1 more process has sent help message 
help-mpi-errors.txt / mpi_errors_are_fatal

[sabine.cacds.uh.edu:21046] Set MCA parameter "orte_base_help_aggregate" to 0 
to see all help / error messages

I once again double checked that this works correctly in the 3.0 (and 3.1, 
although I did not run that test this time).

3.       The openib btl component is always getting in the way with annoying 
warnings. It is not really used, but constantly complains:

[sabine.cacds.uh.edu:25996] 1 more process has sent help message 
help-mpi-btl-openib.txt / ib port not selected
[sabine.cacds.uh.edu:25996] Set MCA parameter "orte_base_help_aggregate" to 0 
to see all help / error messages
[sabine.cacds.uh.edu:25996] 1 more process has sent help message 
help-mpi-btl-openib.txt / error in device init

So bottom line, if I do

mpirun –mca btl^openib –mca mtl^ofi ….

my tests finish correctly, although mpirun will still return an error.


From: devel [mailto:devel-boun...@lists.open-mpi.org] On Behalf Of Geoffrey 
Sent: Sunday, September 16, 2018 2:31 PM
To: devel@lists.open-mpi.org
Subject: [OMPI devel] Announcing Open MPI v4.0.0rc1

The first release candidate for the Open MPI v4.0.0 release is posted at


Major changes include:

4.0.0 -- September, 2018


- OSHMEM updated to the OpenSHMEM 1.4 API.

- Do not build Open SHMEM layer when there are no SPMLs available.

  Currently, this means the Open SHMEM layer will only build if

  a MXM or UCX library is found.

- A UCX BTL was added for enhanced MPI RMA support using UCX

- With this release,  OpenIB BTL now only supports iWarp and RoCE by default.

- Updated internal HWLOC to 2.0.1

- Updated internal PMIx to 3.0.1

- Change the priority for selecting external verses internal HWLOC

  and PMIx packages to build.  Starting with this release, configure

  by default selects available external HWLOC and PMIx packages over

  the internal ones.

- Updated internal ROMIO to 3.2.1.

- Removed support for the MXM MTL.

- Improved CUDA support when using UCX.

- Improved support for two phase MPI I/O operations when using OMPIO.

- Added support for Software-based Performance Counters, see

 Various improvements to MPI RMA performance when using RDMA

  capable interconnects.

- Update memkind component to use the memkind 1.6 public API.

- Fix problems with use of newer map-by mpirun options.  Thanks to

  Tony Reina for reporting.

- Fix rank-by algorithms to properly rank by object and span

- Allow for running as root of two environment variables are set.

  Requested by Axel Huebl.

- Fix a problem with building the Java bindings when using Java 10.

  Thanks to Bryce Glover for reporting.

Our goal is to release 4.0.0 by mid Oct, so any testing is appreciated.

devel mailing list

Reply via email to