Attached the output from openmpi/1.7.5a1r30708$ $MPI_BINDIR/mpiexec -mca oob_tcp_if_include ib0 -mca oob_base_verbose 100 -H linuxscc004 -np 1 hostname 2>&1 | tee oob_base_verbose-linuxbmc0008-175a1r29587.txt
Well, some 5 lines added.(The ib0 on linuxscc004 is not reachable from linuxbmc0008 - this lead to TCP shutdown? cf. line 36-37)
On 02/13/14 01:28, Ralph Castain wrote:
Could you please give the nightly 1.7.5 tarball a try using the same cmd line options and send me the output? I see the problem, but am trying to understand how it happens. I've added a bunch of diagnostic statements that should help me track it down. Thanks Ralph On Feb 12, 2014, at 1:26 AM, Paul Kapinos <kapi...@rz.rwth-aachen.de> wrote:As said, the change in behaviour is new in 1.7.4 - all previous versions has been worked. Moreover, setting "-mca oob_tcp_if_include ib0" is a workaround for older versions of Open MPI for some 60-seconds timeout when starting the same command (which is still sucessfull); or for infinite waiting in same cases. Attached are logs of the commands: $ export | grep OMPI | tee export_OMPI-linuxbmc0008.txt $ $MPI_BINDIR/mpiexec -mca oob_tcp_if_include ib0 -mca oob_base_verbose 100 -H linuxscc004 -np 1 hostname 2>&1 | tee oob_base_verbose-linuxbmc0008-173.txt (and -174 for appropriate versions 1.7.3 and 1.7.4) $ ifconfig 2>&1 | tee ifconfig-linuxbmc0008.txt (and -linuxscc004 for the two nodes; linuxscc004 is in (h) fabric and 'mpiexec' was called from node linuxbmc0008 which is in the (b) fabric where the 'ib0' is configured to be the main interface) and the OMPI environment on linuxbmc0008. Maybe you can see something from this. Best Paul On 02/11/14 20:29, Ralph Castain wrote:I've added better error messages in the trunk, scheduled to move over to 1.7.5. I don't see anything in the code that would explain why we don't pickup and use ib0 if it is present and specified in if_include - we should be doing it. For now, can you run this with "-mca oob_base_verbose 100" on your cmd line and send me the output? Might help debug the behavior. Thanks Ralph On Feb 11, 2014, at 1:22 AM, Paul Kapinos <kapi...@rz.rwth-aachen.de> wrote:Dear Open MPI developer, I. we see peculiar behaviour in the new 1.7.4 version of Open MPI which is a change to previous versions: - when calling "mpiexec", it returns "1" and exits silently. The behaviour is reproducible; well not that easy reproducible. We have multiple InfiniBand islands in our cluster. All nodes are passwordless reachable from each other in somehow way; some via IPoIB, for some routing you also have to use ethernet cards and IB/TCP gateways. One island (b) is configured to use the IB card as the main TCP interface. In this island, the variable OMPI_MCA_oob_tcp_if_include is set to "ib0" (*) Another island (h) is configured in convenient way: IB cards also are here and may be used for IPoIB in the island, but the "main interface" used for DNS and Hostname binds is eth0. When calling 'mpiexec' from (b) to start a process on (h), and OpenMPI version is 1.7.4, and OMPI_MCA_oob_tcp_if_include is set to "ib0", mpiexec just exits with return value "1" and no error/warning. When OMPI_MCA_oob_tcp_if_include is unset it works pretty fine. All previously versions of Open MPI (1.6.x, 1.7.3) ) did not have this behaviour; so this is aligned to v1.7.4 only. See log below. You ask why to hell starting MPI processes on other IB island? Because our front-end nodes are in the island (b) but we sometimes need to start something also on island (h), which has been worced perfectly until 1.7.4. (*) This is another Spaghetti Western long story. In short, we set OMPI_MCA_oob_tcp_if_include to 'ib0' in the subcluster where the IB card is configured to be the main network interface, in order to stop Open MPI trying to connect via (possibly unconfigured) ethernet cards - which lead to endless waiting, sometimes. Cf. http://www.open-mpi.org/community/lists/users/2011/11/17824.php ------------------------------------------------------------------------------ pk224850@cluster:~[523]$ module switch $_LAST_MPI openmpi/1.7.3 Unloading openmpi 1.7.3 [ OK ] Loading openmpi 1.7.3 for intel compiler [ OK ] pk224850@cluster:~[524]$ $MPI_BINDIR/mpiexec -H linuxscc004 -np 1 hostname ; echo $? linuxscc004.rz.RWTH-Aachen.DE 0 pk224850@cluster:~[525]$ module switch $_LAST_MPI openmpi/1.7.4 Unloading openmpi 1.7.3 [ OK ] Loading openmpi 1.7.4 for intel compiler [ OK ] pk224850@cluster:~[526]$ $MPI_BINDIR/mpiexec -H linuxscc004 -np 1 hostname ; echo $? 1 pk224850@cluster:~[527]$ ------------------------------------------------------------------------------ II. During some experiments with envvars and v1.7.4, got the below messages. -------------------------------------------------------------------------- Sorry! You were supposed to get help about: no-included-found But I couldn't open the help file: /opt/MPI/openmpi-1.7.4/linux/intel/share/openmpi/help-oob-tcp.txt: No such file or directory. Sorry! -------------------------------------------------------------------------- [linuxc2.rz.RWTH-Aachen.DE:13942] [[63331,0],0] ORTE_ERROR_LOG: Not available in file ess_hnp_module.c at line 314 -------------------------------------------------------------------------- Reproducing: $MPI_BINDIR/mpiexec -mca oob_tcp_if_include ib0 -H linuxscc004 -np 1 hostname *frome one node with no 'ib0' card*, also without infiniband. Yessir this is a bad idea, and the 1.7.3 has said more understanding "you do wrong thing": -------------------------------------------------------------------------- None of the networks specified to be included for out-of-band communications could be found: Value given: ib0 Please revise the specification and try again. -------------------------------------------------------------------------- No idea, why the file share/openmpi/help-oob-tcp.txt has not been installed in 1.7.4, as we compile this version in pretty the same way as previous versions.. Best, Paul Kapinos -- Dipl.-Inform. Paul Kapinos - High Performance Computing, RWTH Aachen University, IT Center Seffenter Weg 23, D 52074 Aachen (Germany) Tel: +49 241/80-24915-- Dipl.-Inform. Paul Kapinos - High Performance Computing, RWTH Aachen University, IT Center Seffenter Weg 23, D 52074 Aachen (Germany) Tel: +49 241/80-24915 <oob_base_verbose-linuxbmc0008-165.txt><oob_base_verbose-linuxbmc0008-173.txt><oob_base_verbose-linuxbmc0008-174.txt><export_OMPI-linuxbmc0008.txt><ifconfig-linuxbmc0008.txt><ifconfig-linuxscc004.txt>
-- Dipl.-Inform. Paul Kapinos - High Performance Computing, RWTH Aachen University, IT Center Seffenter Weg 23, D 52074 Aachen (Germany) Tel: +49 241/80-24915
[linuxbmc0008.rz.RWTH-Aachen.DE:15723] mca: base: components_register: registering oob components [linuxbmc0008.rz.RWTH-Aachen.DE:15723] mca: base: components_register: found loaded component tcp [linuxbmc0008.rz.RWTH-Aachen.DE:15723] mca: base: components_register: component tcp register function successful [linuxbmc0008.rz.RWTH-Aachen.DE:15723] mca: base: components_open: opening oob components [linuxbmc0008.rz.RWTH-Aachen.DE:15723] mca: base: components_open: found loaded component tcp [linuxbmc0008.rz.RWTH-Aachen.DE:15723] mca: base: components_open: component tcp open function successful [linuxbmc0008.rz.RWTH-Aachen.DE:15723] mca:oob:select: checking available component tcp [linuxbmc0008.rz.RWTH-Aachen.DE:15723] mca:oob:select: Querying component [tcp] [linuxbmc0008.rz.RWTH-Aachen.DE:15723] oob:tcp: component_available called [linuxbmc0008.rz.RWTH-Aachen.DE:15723] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4 [linuxbmc0008.rz.RWTH-Aachen.DE:15723] [[11240,0],0] oob:tcp:init rejecting interface lo (not in include list) [linuxbmc0008.rz.RWTH-Aachen.DE:15723] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4 [linuxbmc0008.rz.RWTH-Aachen.DE:15723] [[11240,0],0] oob:tcp:init rejecting interface eth0 (not in include list) [linuxbmc0008.rz.RWTH-Aachen.DE:15723] WORKING INTERFACE 3 KERNEL INDEX 4 FAMILY: V4 [linuxbmc0008.rz.RWTH-Aachen.DE:15723] [[11240,0],0] oob:tcp:init creating module for V4 address on interface ib0 [linuxbmc0008.rz.RWTH-Aachen.DE:15723] [[11240,0],0] oob:tcp:init adding 134.61.202.7 to our list of V4 connections [linuxbmc0008.rz.RWTH-Aachen.DE:15723] [[11240,0],0] TCP STARTUP [linuxbmc0008.rz.RWTH-Aachen.DE:15723] [[11240,0],0] attempting to bind to IPv4 port 0 [linuxbmc0008.rz.RWTH-Aachen.DE:15723] [[11240,0],0] assigned IPv4 port 61399 [linuxbmc0008.rz.RWTH-Aachen.DE:15723] mca:oob:select: Adding component to end [linuxbmc0008.rz.RWTH-Aachen.DE:15723] mca:oob:select: Found 1 active transports [linuxscc004.rz.RWTH-Aachen.DE:29141] mca: base: components_register: registering oob components [linuxscc004.rz.RWTH-Aachen.DE:29141] mca: base: components_register: found loaded component tcp [linuxscc004.rz.RWTH-Aachen.DE:29141] mca: base: components_register: component tcp register function successful [linuxscc004.rz.RWTH-Aachen.DE:29141] mca: base: components_open: opening oob components [linuxscc004.rz.RWTH-Aachen.DE:29141] mca: base: components_open: found loaded component tcp [linuxscc004.rz.RWTH-Aachen.DE:29141] mca: base: components_open: component tcp open function successful [linuxscc004.rz.RWTH-Aachen.DE:29141] mca:oob:select: checking available component tcp [linuxscc004.rz.RWTH-Aachen.DE:29141] mca:oob:select: Querying component [tcp] [linuxscc004.rz.RWTH-Aachen.DE:29141] oob:tcp: component_available called [linuxscc004.rz.RWTH-Aachen.DE:29141] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4 [linuxscc004.rz.RWTH-Aachen.DE:29141] [[11240,0],1] oob:tcp:init rejecting interface lo (not in include list) [linuxscc004.rz.RWTH-Aachen.DE:29141] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4 [linuxscc004.rz.RWTH-Aachen.DE:29141] [[11240,0],1] oob:tcp:init rejecting interface eth0 (not in include list) [linuxscc004.rz.RWTH-Aachen.DE:29141] WORKING INTERFACE 3 KERNEL INDEX 4 FAMILY: V4 [linuxscc004.rz.RWTH-Aachen.DE:29141] [[11240,0],1] oob:tcp:init creating module for V4 address on interface ib0 [linuxscc004.rz.RWTH-Aachen.DE:29141] [[11240,0],1] oob:tcp:init adding 192.168.222.4 to our list of V4 connections [linuxscc004.rz.RWTH-Aachen.DE:29141] [[11240,0],1] TCP STARTUP [linuxscc004.rz.RWTH-Aachen.DE:29141] [[11240,0],1] attempting to bind to IPv4 port 0 [linuxscc004.rz.RWTH-Aachen.DE:29141] [[11240,0],1] assigned IPv4 port 17812 [linuxscc004.rz.RWTH-Aachen.DE:29141] mca:oob:select: Adding component to end [linuxscc004.rz.RWTH-Aachen.DE:29141] mca:oob:select: Found 1 active transports [linuxscc004.rz.RWTH-Aachen.DE:29141] [[11240,0],1]: set_addr to uri 736624640.0;tcp://134.61.202.7:55279 [linuxscc004.rz.RWTH-Aachen.DE:29141] [[11240,0],1]:set_addr checking if peer [[11240,0],0] is reachable via component tcp [linuxscc004.rz.RWTH-Aachen.DE:29141] [[11240,0],1] oob:tcp: working peer [[11240,0],0] address tcp://134.61.202.7:55279 [linuxscc004.rz.RWTH-Aachen.DE:29141] [[11240,0],1] NO MODULE AT KINDEX 2 FOR ADDRESS 134.61.202.7 [linuxscc004.rz.RWTH-Aachen.DE:29141] [[11240,0],1] oob:base:send to target [[11240,0],0] [linuxscc004.rz.RWTH-Aachen.DE:29141] [[11240,0],1] oob:base:send no path to target [[11240,0],0] [linuxscc004.rz.RWTH-Aachen.DE:29141] [[11240,0],1] TCP SHUTDOWN [linuxscc004.rz.RWTH-Aachen.DE:29141] mca: base: close: component tcp closed [linuxscc004.rz.RWTH-Aachen.DE:29141] mca: base: close: unloading component tcp [linuxbmc0008.rz.RWTH-Aachen.DE:15723] [[11240,0],0] TCP SHUTDOWN [linuxbmc0008.rz.RWTH-Aachen.DE:15723] mca: base: close: component tcp closed [linuxbmc0008.rz.RWTH-Aachen.DE:15723] mca: base: close: unloading component tcp
smime.p7s
Description: S/MIME Cryptographic Signature
