On Wed, 19 Sep 2018 16:24:53 +0000
"Gabriel, Edgar" <egabr...@central.uh.edu> wrote:

> I performed some tests on our Omnipath cluster, and I have a mixed
> bag of results with 4.0.0rc1

I've also tried it on our OPA cluster (skylake+centos-7+inbox) with
very similar results.

> compute-1-1.local.4351PSM2 has not been initialized
> compute-1-0.local.3826PSM2 has not been initialized

yup I too see these.
> mpirun detected that one or more processes exited with non-zero
> status, thus causing the job to be terminated. The first process to
> do so was:
>               Process name: [[38418,1],1]
>               Exit code:    255
> ----------------------------------------------------------------------------

> 2.       The ofi mtl does not work at all on our Omnipath cluster. If
> I try to force it using ‘mpirun –mca mtl ofi …’ I get the following
> error message.

Yes ofi seems broken. But not even disabling it helps me completely (I
see "mca_btl_ofi.so           [.] mca_btl_ofi_component_progress" in my
perf top...

> 3.       The openib btl component is always getting in the way with
> annoying warnings. It is not really used, but constantly complains:
> [sabine.cacds.uh.edu:25996] 1 more process has sent help message
> help-mpi-btl-openib.txt / ib port not selected


> So bottom line, if I do
> mpirun –mca btl^openib –mca mtl^ofi ….
> my tests finish correctly, although mpirun will still return an error.

I get some things to work with this approach (two ranks on two nodes
for example). But a lot of things crash rahter hard:

 $ mpirun -mca btl ^openib -mca mtl
^ofi ./openmpi-4.0.0rc1/imb.openmpi-4.0.0rc1
PSM2 was unable to open an endpoint. Please make sure that the network
link is active on the node and the hardware is functioning.

  Error: Failure in initializing endpoint
n909.279895hfi_userinit: assign_context command failed: Device or
resource busy n909.279895psmi_context_open: hfi_userinit: failed,
trying again (1/3)
  PML add procs failed
  --> Returned "Error" (-1) instead of "Success" (0)
[n908:298761] *** An error occurred in MPI_Init
[n908:298761] *** reported by process [4092002305,59]
[n908:298761] *** on a NULL communicator
[n908:298761] *** Unknown error
[n908:298761] *** MPI_ERRORS_ARE_FATAL (processes in this communicator
  will now abort, [n908:298761] ***    and potentially your MPI job)
[n907:407748] 255 more processes have sent help message
  help-mtl-psm2.txt / unable to open endpoint [n907:407748] Set MCA
  parameter "orte_base_help_aggregate" to 0 to see all help / error
  messages [n907:407748] 127 more processes have sent help message
  help-mpi-runtime.txt / mpi_init:startup:internal-failure
  [n907:407748] 56 more processes have sent help message
  help-mpi-errors.txt / mpi_errors_are_fatal unknown handle

If I disable psm2 too I get it to run (apparantly on vader?)

/Peter K
devel mailing list

Reply via email to