Staring at this some more, I do have the following questions:
* in your first case, it looks like "prte" was started from microway3 - correct?
Yes, "prte" was started from microway3.
No, "mpirun" was executed from microway3.
* in the second case, that worked, it looks like "mpirun" was executed from microway1 - correct?
Yes, "mpirun" was started from microway3.
* in the third case, you state that "mpirun" was again executed from microway3, and the process output confirms that
I'm wondering if the issue here might actually be that PRRTE expects the ordering of hosts in the hostfile to start with the host it is sitting on - i.e., if the node index number between the various daemons is getting confused. Can you perhaps see what happens with the failing cases if you put microway3 at the top of the hostfile and execute prte/mpirun from microway3 as before?
OK, the first failing case:
mic:/amd/home/jdelsign/PMIx>pterm
pterm failed to initialize, likely due to no DVM being available
mic:/amd/home/jdelsign/PMIx>cat myhostfile3
microway3 slots=16
microway1 slots=16
microway2 slots=16
mic:/amd/home/jdelsign/PMIx>prte --hostfile ./myhostfile3 --daemonize
mic:/amd/home/jdelsign/PMIx>prun -n 3 --map-by node -x MESSAGE=name --personality ompi ./tx_basic_mpi
tx_basic_mpi
Hello from proc (0)
MESSAGE: microway3.totalviewtech.com
Hello from proc (1): microway1
Hello from proc (2): microway2.totalviewtech.com
--------------------------------------------------------------------------
WARNING: Open MPI failed to TCP connect to a peer MPI process. This
should not happen.
Your Open MPI job may now hang or fail.
Local host: microway1
PID: 292266
Message: connect() to 10.71.2.58:1024 failed
Error: No route to host (113)
--------------------------------------------------------------------------
[microway1:292266] ../../../../../../ompi/ompi/mca/pml/ob1/pml_ob1_sendreq.c:189 FATAL
mic:/amd/home/jdelsign/PMIx>hostname
microway3.totalviewtech.com
mic:/amd/home/jdelsign/PMIx>
And the second failing test case:
mic:/amd/home/jdelsign/PMIx>pterm
pterm failed to initialize, likely due to no DVM being available
mic:/amd/home/jdelsign/PMIx>cat myhostfile3+2
microway3 slots=16
microway2 slots=16
mic:/amd/home/jdelsign/PMIx>
mic:/amd/home/jdelsign/PMIx>mpirun -n 3 --map-by node -x MESSAGE=name --personality ompi --hostfile myhostfile3+2 ./tx_basic_mpi
tx_basic_mpi
Hello from proc (0)
MESSAGE: microway3.totalviewtech.com
Hello from proc (1): microway3.totalviewtech.com
--------------------------------------------------------------------------
WARNING: Open MPI failed to TCP connect to a peer MPI process. This
should not happen.
Your Open MPI job may now hang or fail.
Local host: microway3
PID: 271144
Message: connect() to 10.71.2.58:1024 failed
Error: No route to host (113)
--------------------------------------------------------------------------
[microway3.totalviewtech.com:271144] ../../../../../../ompi/ompi/mca/pml/ob1/pml_ob1_sendreq.c:189 FATAL
Hello from proc (2): microway2.totalviewtech.com
mic:/amd/home/jdelsign/PMIx>
So, AFAICT, host name order didn't matter.
Cheers, John D.
On May 4, 2020, at 7:34 AM, John DelSignore via devel <devel@lists.open-mpi.org> wrote:
Hi folks,
I cloned a fresh copy of OMPI master this morning at ~8:30am EDT and rebuilt. I'm running a very simple test code on three Centos 7.[56] nodes named microway[123] over TCP. I'm seeing a fatal error similar to the following:
[microway3.totalviewtech.com:227713] ../../../../../../ompi/ompi/mca/pml/ob1/pml_ob1_sendreq.c:189 FATAL
The case of prun launching an OMPI code does not work correctly. The MPI processes seem to launch OK, but there is the follwoing OMPI error at the point where the processes communicate. In the following case, I have DVM running on three nodes "microway[123]":
If I use mpirun to launch the program it works whether or not a DVM is already running (first without a DVM, then with a DVM):mic:/amd/home/jdelsign/PMIx>prun -n 3 --map-by node -x MESSAGE=name --personality ompi ./tx_basic_mpi
tx_basic_mpi
Hello from proc (0)
MESSAGE: microway3.totalviewtech.com
Hello from proc (1): microway1
Hello from proc (2): microway2.totalviewtech.com
--------------------------------------------------------------------------
WARNING: Open MPI failed to TCP connect to a peer MPI process. This
should not happen.
Your Open MPI job may now hang or fail.
Local host: microway1
PID: 282716
Message: connect() to 10.71.2.58:1024 failed
Error: No route to host (113)
--------------------------------------------------------------------------
[microway1:282716] ../../../../../../ompi/ompi/mca/pml/ob1/pml_ob1_sendreq.c:189 FATAL
--------------------------------------------------------------------------
An MPI communication peer process has unexpectedly disconnected. This
usually indicates a failure in the peer process (e.g., a crash or
otherwise exiting without calling MPI_FINALIZE first).
Although this local MPI process will likely now behave unpredictably
(it may even hang or crash), the root cause of this problem is the
failure of the peer -- that is what you need to investigate. For
example, there may be a core file that you can examine. More
generally: such peer hangups are frequently caused by application bugs
or other external events.
Local host: microway3
Local PID: 214271
Peer host: microway1
--------------------------------------------------------------------------
mic:/amd/home/jdelsign/PMIx>
mic:/amd/home/jdelsign/PMIx>mpirun -n 3 --map-by node -x MESSAGE=name --personality ompi --hostfile myhostfile ./tx_basic_mpi
tx_basic_mpi
Hello from proc (0)
MESSAGE: microway1
Hello from proc (1): microway2.totalviewtech.com
Hello from proc (2): microway3.totalviewtech.com
All Done!
mic:/amd/home/jdelsign/PMIx>
mic:/amd/home/jdelsign/PMIx>prte --hostfile ./myhostfile --daemonize
mic:/amd/home/jdelsign/PMIx>mpirun -n 3 --map-by node -x MESSAGE=name --personality ompi --hostfile myhostfile ./tx_basic_mpi
tx_basic_mpi
Hello from proc (0)
MESSAGE: microway1
Hello from proc (1): microway2.totalviewtech.com
Hello from proc (2): microway3.totalviewtech.com
All Done!
mic:/amd/home/jdelsign/PMIx>But if I use mpirun to launch 3 processes from microway3 and use a hostfile that contains only microway[23], I get a similar failure as the prun case:
mic:/amd/home/jdelsign/PMIx>hostname
microway3.totalviewtech.com
mic:/amd/home/jdelsign/PMIx>cat myhostfile2
microway2 slots=16
microway3 slots=16
mic:/amd/home/jdelsign/PMIx>mpirun -n 3 --map-by node -x MESSAGE=name --personality ompi --hostfile myhostfile2 ./tx_basic_mpi
tx_basic_mpi
Hello from proc (0)
MESSAGE: microway2.totalviewtech.com
Hello from proc (1): microway2.totalviewtech.com
--------------------------------------------------------------------------
WARNING: Open MPI failed to TCP connect to a peer MPI process. This
should not happen.
Your Open MPI job may now hang or fail.
Local host: microway3
PID: 227713
Message: connect() to 10.71.2.58:1024 failed
Error: No route to host (113)
--------------------------------------------------------------------------
[microway3.totalviewtech.com:227713] ../../../../../../ompi/ompi/mca/pml/ob1/pml_ob1_sendreq.c:189 FATAL
[microway2][[32270,1],1][../../../../../../ompi/opal/mca/btl/tcp/btl_tcp.c:566:mca_btl_tcp_recv_blocking] recv(13) failed: Connection reset by peer (104)
mic:/amd/home/jdelsign/PMIx>
I asked my therapist (Ralph) about it, and he said,
"It looks to me like the btl/tcp component is having trouble correctly selecting a route to use when opening communications across hosts. I've seen this in my docker setup too, but thought perhaps it was just a docker-related issue.
What's weird in your last example is that both procs are on the same node, and therefore they should only be using shared memory to communicate - the btl/tcp component shouldn't be trying to create a connection at all."
Cheers, John D.
This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
