Thanks, Ralph! When I add --mca oob_tcp_if_include ib0 (where ib0 is infiniband interface from ifconfig) to mpirun it starts working correct! Why OpenMPI doesn't do it itself?
Tue, 22 Jul 2014 11:26:16 -0700 от Ralph Castain <r...@open-mpi.org>: >Okay, the problem is that the connection back to mpirun isn't getting thru. We >are trying on the 10.0.251.53 address - is that blocked, or should we be using >something else? If so, you might want to direct us by adding "-mca >oob_tcp_if_include foo", where foo is the interface you want us to use > > >On Jul 20, 2014, at 10:24 PM, Timur Ismagilov < tismagi...@mail.ru > wrote: >>NIC = network interface controller? >>There is QDR Infiniband 4x/10G Ethernet/Gigabit Ethernet. >>I want to use QDR Infiniband. >>Here is a new output: >>$ mpirun -mca mca_base_env_list 'LD_PRELOAD' --debug-daemons --mca >>plm_base_verbose 5 -mca oob_base_verbose 10 -mca rml_base_verbose 10 -np 2 >>hello_c |tee hello.out >>Warning: Conflicting CPU frequencies detected, using: 2927.000000. >>[compiler-2:30735] mca:base:select:( plm) Querying component [isolated] >>[compiler-2:30735] mca:base:select:( plm) Query of component [isolated] set >>priority to 0 >>[compiler-2:30735] mca:base:select:( plm) Querying component [rsh] >>[compiler-2:30735] mca:base:select:( plm) Query of component [rsh] set >>priority to 10 >>[compiler-2:30735] mca:base:select:( plm) Querying component [slurm] >>[compiler-2:30735] mca:base:select:( plm) Query of component [slurm] set >>priority to 75 >>[compiler-2:30735] mca:base:select:( plm) Selected component [slurm] >>[compiler-2:30735] mca: base: components_register: registering oob components >>[compiler-2:30735] mca: base: components_register: found loaded component tcp >>[compiler-2:30735] mca: base: components_register: component tcp register >>function successful >>[compiler-2:30735] mca: base: components_open: opening oob components >>[compiler-2:30735] mca: base: components_open: found loaded component tcp >>[compiler-2:30735] mca: base: components_open: component tcp open function >>successful >>[compiler-2:30735] mca:oob:select: checking available component tcp >>[compiler-2:30735] mca:oob:select: Querying component [tcp] >>[compiler-2:30735] oob:tcp: component_available called >>[compiler-2:30735] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4 >>[compiler-2:30735] WORKING INTERFACE 2 KERNEL INDEX 3 FAMILY: V4 >>[compiler-2:30735] [[65177,0],0] oob:tcp:init adding 10.0.251.53 to our list >>of V4 connections >>[compiler-2:30735] WORKING INTERFACE 3 KERNEL INDEX 4 FAMILY: V4 >>[compiler-2:30735] [[65177,0],0] oob:tcp:init adding 10.0.0.4 to our list of >>V4 connections >>[compiler-2:30735] WORKING INTERFACE 4 KERNEL INDEX 5 FAMILY: V4 >>[compiler-2:30735] [[65177,0],0] oob:tcp:init adding 10.2.251.14 to our list >>of V4 connections >>[compiler-2:30735] WORKING INTERFACE 5 KERNEL INDEX 6 FAMILY: V4 >>[compiler-2:30735] [[65177,0],0] oob:tcp:init adding 10.128.0.4 to our list >>of V4 connections >>[compiler-2:30735] WORKING INTERFACE 6 KERNEL INDEX 7 FAMILY: V4 >>[compiler-2:30735] [[65177,0],0] oob:tcp:init adding 93.180.7.38 to our list >>of V4 connections >>[compiler-2:30735] [[65177,0],0] TCP STARTUP >>[compiler-2:30735] [[65177,0],0] attempting to bind to IPv4 port 0 >>[compiler-2:30735] [[65177,0],0] assigned IPv4 port 49759 >>[compiler-2:30735] mca:oob:select: Adding component to end >>[compiler-2:30735] mca:oob:select: Found 1 active transports >>[compiler-2:30735] mca: base: components_register: registering rml components >>[compiler-2:30735] mca: base: components_register: found loaded component oob >>[compiler-2:30735] mca: base: components_register: component oob has no >>register or open function >>[compiler-2:30735] mca: base: components_open: opening rml components >>[compiler-2:30735] mca: base: components_open: found loaded component oob >>[compiler-2:30735] mca: base: components_open: component oob open function >>successful >>[compiler-2:30735] orte_rml_base_select: initializing rml component oob >>[compiler-2:30735] [[65177,0],0] posting recv >>[compiler-2:30735] [[65177,0],0] posting persistent recv on tag 30 for peer >>[[WILDCARD],WILDCARD] >>[compiler-2:30735] [[65177,0],0] posting recv >>[compiler-2:30735] [[65177,0],0] posting persistent recv on tag 15 for peer >>[[WILDCARD],WILDCARD] >>[compiler-2:30735] [[65177,0],0] posting recv >>[compiler-2:30735] [[65177,0],0] posting persistent recv on tag 32 for peer >>[[WILDCARD],WILDCARD] >>[compiler-2:30735] [[65177,0],0] posting recv >>[compiler-2:30735] [[65177,0],0] posting persistent recv on tag 33 for peer >>[[WILDCARD],WILDCARD] >>[compiler-2:30735] [[65177,0],0] posting recv >>[compiler-2:30735] [[65177,0],0] posting persistent recv on tag 5 for peer >>[[WILDCARD],WILDCARD] >>[compiler-2:30735] [[65177,0],0] posting recv >>[compiler-2:30735] [[65177,0],0] posting persistent recv on tag 10 for peer >>[[WILDCARD],WILDCARD] >>[compiler-2:30735] [[65177,0],0] posting recv >>[compiler-2:30735] [[65177,0],0] posting persistent recv on tag 12 for peer >>[[WILDCARD],WILDCARD] >>[compiler-2:30735] [[65177,0],0] posting recv >>[compiler-2:30735] [[65177,0],0] posting persistent recv on tag 9 for peer >>[[WILDCARD],WILDCARD] >>[compiler-2:30735] [[65177,0],0] posting recv >>[compiler-2:30735] [[65177,0],0] posting persistent recv on tag 34 for peer >>[[WILDCARD],WILDCARD] >>[compiler-2:30735] [[65177,0],0] posting recv >>[compiler-2:30735] [[65177,0],0] posting persistent recv on tag 2 for peer >>[[WILDCARD],WILDCARD] >>[compiler-2:30735] [[65177,0],0] posting recv >>[compiler-2:30735] [[65177,0],0] posting persistent recv on tag 21 for peer >>[[WILDCARD],WILDCARD] >>[compiler-2:30735] [[65177,0],0] posting recv >>[compiler-2:30735] [[65177,0],0] posting persistent recv on tag 22 for peer >>[[WILDCARD],WILDCARD] >>[compiler-2:30735] [[65177,0],0] posting recv >>[compiler-2:30735] [[65177,0],0] posting persistent recv on tag 45 for peer >>[[WILDCARD],WILDCARD] >>[compiler-2:30735] [[65177,0],0] posting recv >>[compiler-2:30735] [[65177,0],0] posting persistent recv on tag 46 for peer >>[[WILDCARD],WILDCARD] >>[compiler-2:30735] [[65177,0],0] posting recv >>[compiler-2:30735] [[65177,0],0] posting persistent recv on tag 1 for peer >>[[WILDCARD],WILDCARD] >>[compiler-2:30735] [[65177,0],0] posting recv >>[compiler-2:30735] [[65177,0],0] posting persistent recv on tag 27 for peer >>[[WILDCARD],WILDCARD] >>Daemon was launched on node1-128-17 - beginning to initialize >>Daemon was launched on node1-128-18 - beginning to initialize >>[node1-128-17:14779] mca: base: components_register: registering oob >>components >>[node1-128-17:14779] mca: base: components_register: found loaded component >>tcp >>[node1-128-17:14779] mca: base: components_register: component tcp register >>function successful >>[node1-128-17:14779] mca: base: components_open: opening oob components >>[node1-128-17:14779] mca: base: components_open: found loaded component tcp >>[node1-128-17:14779] mca: base: components_open: component tcp open function >>successful >>[node1-128-17:14779] mca:oob:select: checking available component tcp >>[node1-128-17:14779] mca:oob:select: Querying component [tcp] >>[node1-128-17:14779] oob:tcp: component_available called >>[node1-128-17:14779] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4 >>[node1-128-17:14779] WORKING INTERFACE 2 KERNEL INDEX 3 FAMILY: V4 >>[node1-128-17:14779] [[65177,0],1] oob:tcp:init adding 10.0.128.17 to our >>list of V4 connections >>[node1-128-17:14779] WORKING INTERFACE 3 KERNEL INDEX 4 FAMILY: V4 >>[node1-128-17:14779] [[65177,0],1] oob:tcp:init adding 10.128.128.17 to our >>list of V4 connections >>[node1-128-17:14779] [[65177,0],1] TCP STARTUP >>[node1-128-17:14779] [[65177,0],1] attempting to bind to IPv4 port 0 >>[node1-128-17:14779] [[65177,0],1] assigned IPv4 port 46441 >>[node1-128-17:14779] mca:oob:select: Adding component to end >>[node1-128-17:14779] mca:oob:select: Found 1 active transports >>[node1-128-17:14779] mca: base: components_register: registering rml >>components >>[node1-128-17:14779] mca: base: components_register: found loaded component >>oob >>[node1-128-17:14779] mca: base: components_register: component oob has no >>register or open function >>[node1-128-17:14779] mca: base: components_open: opening rml components >>[node1-128-17:14779] mca: base: components_open: found loaded component oob >>[node1-128-17:14779] mca: base: components_open: component oob open function >>successful >>[node1-128-17:14779] orte_rml_base_select: initializing rml component oob >>[node1-128-18:17849] mca: base: components_register: registering oob >>components >>[node1-128-18:17849] mca: base: components_register: found loaded component >>tcp >>[node1-128-18:17849] mca: base: components_register: component tcp register >>function successful >>[node1-128-18:17849] mca: base: components_open: opening oob components >>[node1-128-18:17849] mca: base: components_open: found loaded component tcp >>[node1-128-18:17849] mca: base: components_open: component tcp open function >>successful >>[node1-128-18:17849] mca:oob:select: checking available component tcp >>[node1-128-18:17849] mca:oob:select: Querying component [tcp] >>[node1-128-18:17849] oob:tcp: component_available called >>[node1-128-18:17849] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4 >>[node1-128-18:17849] WORKING INTERFACE 2 KERNEL INDEX 3 FAMILY: V4 >>[node1-128-18:17849] [[65177,0],2] oob:tcp:init adding 10.0.128.18 to our >>list of V4 connections >>[node1-128-18:17849] WORKING INTERFACE 3 KERNEL INDEX 4 FAMILY: V4 >>[node1-128-18:17849] [[65177,0],2] oob:tcp:init adding 10.128.128.18 to our >>list of V4 connections >>[node1-128-18:17849] [[65177,0],2] TCP STARTUP >>[node1-128-18:17849] [[65177,0],2] attempting to bind to IPv4 port 0 >>[node1-128-18:17849] [[65177,0],2] assigned IPv4 port 60695 >>[node1-128-18:17849] mca:oob:select: Adding component to end >>[node1-128-18:17849] mca:oob:select: Found 1 active transports >>[node1-128-18:17849] mca: base: components_register: registering rml >>components >>[node1-128-18:17849] mca: base: components_register: found loaded component >>oob >>[node1-128-18:17849] mca: base: components_register: component oob has no >>register or open function >>[node1-128-18:17849] mca: base: components_open: opening rml components >>[node1-128-18:17849] mca: base: components_open: found loaded component oob >>[node1-128-18:17849] mca: base: components_open: component oob open function >>successful >>[node1-128-18:17849] orte_rml_base_select: initializing rml component oob >>Daemon [[65177,0],1] checking in as pid 14779 on host node1-128-17 >>[node1-128-17:14779] [[65177,0],1] orted: up and running - waiting for >>commands! >>[node1-128-17:14779] [[65177,0],1] posting recv >>[node1-128-17:14779] [[65177,0],1] posting persistent recv on tag 30 for peer >>[[WILDCARD],WILDCARD] >>[node1-128-17:14779] [[65177,0],1] posting recv >>[node1-128-17:14779] [[65177,0],1] posting persistent recv on tag 15 for peer >>[[WILDCARD],WILDCARD] >>[node1-128-17:14779] [[65177,0],1] posting recv >>[node1-128-17:14779] [[65177,0],1] posting persistent recv on tag 32 for peer >>[[WILDCARD],WILDCARD] >>[node1-128-17:14779] [[65177,0],1] posting recv >>[node1-128-17:14779] [[65177,0],1] posting persistent recv on tag 11 for peer >>[[WILDCARD],WILDCARD] >>[node1-128-17:14779] [[65177,0],1] posting recv >>[node1-128-17:14779] [[65177,0],1] posting persistent recv on tag 9 for peer >>[[WILDCARD],WILDCARD] >>[node1-128-17:14779] [[65177,0],1]: set_addr to uri 4271439872.0; >>tcp://10.0.251.53,10.0.0.4,10.2.251.14,10.128.0.4,93.180.7.38:49759 >>[node1-128-17:14779] [[65177,0],1]:set_addr checking if peer [[65177,0],0] is >>reachable via component tcp >>[node1-128-17:14779] [[65177,0],1] oob:tcp: working peer [[65177,0],0] >>address tcp://10.0.251.53,10.0.0.4,10.2.251.14,10.128.0.4,93.180.7.38:49759 >>[node1-128-17:14779] [[65177,0],1] PASSING ADDR 10.0.251.53 TO MODULE >>[node1-128-17:14779] [[65177,0],1]:tcp set addr for peer [[65177,0],0] >>[node1-128-17:14779] [[65177,0],1] PASSING ADDR 10.0.0.4 TO MODULE >>[node1-128-17:14779] [[65177,0],1]:tcp set addr for peer [[65177,0],0] >>[node1-128-17:14779] [[65177,0],1] PASSING ADDR 10.2.251.14 TO MODULE >>[node1-128-17:14779] [[65177,0],1]:tcp set addr for peer [[65177,0],0] >>[node1-128-17:14779] [[65177,0],1] PASSING ADDR 10.128.0.4 TO MODULE >>[node1-128-17:14779] [[65177,0],1]:tcp set addr for peer [[65177,0],0] >>[node1-128-17:14779] [[65177,0],1] PASSING ADDR 93.180.7.38 TO MODULE >>[node1-128-17:14779] [[65177,0],1]:tcp set addr for peer [[65177,0],0] >>[node1-128-17:14779] [[65177,0],1]: peer [[65177,0],0] is reachable via >>component tcp >>[node1-128-17:14779] [[65177,0],1] posting recv >>[node1-128-17:14779] [[65177,0],1] posting persistent recv on tag 3 for peer >>[[WILDCARD],WILDCARD] >>[node1-128-17:14779] [[65177,0],1] posting recv >>[node1-128-17:14779] [[65177,0],1] posting persistent recv on tag 21 for peer >>[[WILDCARD],WILDCARD] >>[node1-128-17:14779] [[65177,0],1] posting recv >>[node1-128-17:14779] [[65177,0],1] posting persistent recv on tag 45 for peer >>[[WILDCARD],WILDCARD] >>[node1-128-17:14779] [[65177,0],1] posting recv >>[node1-128-17:14779] [[65177,0],1] posting persistent recv on tag 46 for peer >>[[WILDCARD],WILDCARD] >>[node1-128-17:14779] [[65177,0],1] posting recv >>[node1-128-17:14779] [[65177,0],1] posting persistent recv on tag 1 for peer >>[[WILDCARD],WILDCARD] >>[node1-128-17:14779] [[65177,0],1] OOB_SEND: rml_oob_send.c:199 >>[node1-128-17:14779] [[65177,0],1]:tcp:processing set_peer cmd >>[node1-128-17:14779] [[65177,0],1]:tcp:processing set_peer cmd >>[node1-128-17:14779] [[65177,0],1]:tcp:processing set_peer cmd >>[node1-128-17:14779] [[65177,0],1]:tcp:processing set_peer cmd >>[node1-128-17:14779] [[65177,0],1]:tcp:processing set_peer cmd >>[node1-128-17:14779] [[65177,0],1] oob:base:send to target [[65177,0],0] >>[node1-128-17:14779] [[65177,0],1] oob:tcp:send_nb to peer [[65177,0],0]:10 >>[node1-128-17:14779] [[65177,0],1] tcp:send_nb to peer [[65177,0],0] >>[node1-128-17:14779] [[65177,0],1]:[oob_tcp.c:484] post send to [[65177,0],0] >>[node1-128-17:14779] [[65177,0],1]:[oob_tcp.c:421] processing send to peer >>[[65177,0],0]:10 >>[node1-128-17:14779] [[65177,0],1]:[oob_tcp.c:455] queue pending to >>[[65177,0],0] >>[node1-128-17:14779] [[65177,0],1] tcp:send_nb: initiating connection to >>[[65177,0],0] >>[node1-128-17:14779] [[65177,0],1]:[oob_tcp.c:469] connect to [[65177,0],0] >>[node1-128-17:14779] [[65177,0],1] orte_tcp_peer_try_connect: attempting to >>connect to proc [[65177,0],0] >>[node1-128-17:14779] [[65177,0],1] orte_tcp_peer_try_connect: attempting to >>connect to proc [[65177,0],0] on socket 10 >>[node1-128-17:14779] [[65177,0],1] orte_tcp_peer_try_connect: attempting to >>connect to proc [[65177,0],0] on 10.0.251.53:49759 - 0 retries >>[node1-128-17:14779] [[65177,0],1] waiting for connect completion to >>[[65177,0],0] - activating send event >>Daemon [[65177,0],2] checking in as pid 17849 on host node1-128-18 >>[node1-128-18:17849] [[65177,0],2] orted: up and running - waiting for >>commands! >>[node1-128-18:17849] [[65177,0],2] posting recv >>[node1-128-18:17849] [[65177,0],2] posting persistent recv on tag 30 for peer >>[[WILDCARD],WILDCARD] >>[node1-128-18:17849] [[65177,0],2] posting recv >>[node1-128-18:17849] [[65177,0],2] posting persistent recv on tag 15 for peer >>[[WILDCARD],WILDCARD] >>[node1-128-18:17849] [[65177,0],2] posting recv >>[node1-128-18:17849] [[65177,0],2] posting persistent recv on tag 32 for peer >>[[WILDCARD],WILDCARD] >>[node1-128-18:17849] [[65177,0],2] posting recv >>[node1-128-18:17849] [[65177,0],2] posting persistent recv on tag 11 for peer >>[[WILDCARD],WILDCARD] >>[node1-128-18:17849] [[65177,0],2] posting recv >>[node1-128-18:17849] [[65177,0],2] posting persistent recv on tag 9 for peer >>[[WILDCARD],WILDCARD] >>[node1-128-18:17849] [[65177,0],2]: set_addr to uri 4271439872.0; >>tcp://10.0.251.53,10.0.0.4,10.2.251.14,10.128.0.4,93.180.7.38:49759 >>[node1-128-18:17849] [[65177,0],2]:set_addr checking if peer [[65177,0],0] is >>reachable via component tcp >>[node1-128-18:17849] [[65177,0],2] oob:tcp: working peer [[65177,0],0] >>address tcp://10.0.251.53,10.0.0.4,10.2.251.14,10.128.0.4,93.180.7.38:49759 >>[node1-128-18:17849] [[65177,0],2] PASSING ADDR 10.0.251.53 TO MODULE >>[node1-128-18:17849] [[65177,0],2]:tcp set addr for peer [[65177,0],0] >>[node1-128-18:17849] [[65177,0],2] PASSING ADDR 10.0.0.4 TO MODULE >>[node1-128-18:17849] [[65177,0],2]:tcp set addr for peer [[65177,0],0] >>[node1-128-18:17849] [[65177,0],2] PASSING ADDR 10.2.251.14 TO MODULE >>[node1-128-18:17849] [[65177,0],2]:tcp set addr for peer [[65177,0],0] >>[node1-128-18:17849] [[65177,0],2] PASSING ADDR 10.128.0.4 TO MODULE >>[node1-128-18:17849] [[65177,0],2]:tcp set addr for peer [[65177,0],0] >>[node1-128-18:17849] [[65177,0],2] PASSING ADDR 93.180.7.38 TO MODULE >>[node1-128-18:17849] [[65177,0],2]:tcp set addr for peer [[65177,0],0] >>[node1-128-18:17849] [[65177,0],2]: peer [[65177,0],0] is reachable via >>component tcp >>[node1-128-18:17849] [[65177,0],2] posting recv >>[node1-128-18:17849] [[65177,0],2] posting persistent recv on tag 3 for peer >>[[WILDCARD],WILDCARD] >>[node1-128-18:17849] [[65177,0],2] posting recv >>[node1-128-18:17849] [[65177,0],2] posting persistent recv on tag 21 for peer >>[[WILDCARD],WILDCARD] >>[node1-128-18:17849] [[65177,0],2] posting recv >>[node1-128-18:17849] [[65177,0],2] posting persistent recv on tag 45 for peer >>[[WILDCARD],WILDCARD] >>[node1-128-18:17849] [[65177,0],2] posting recv >>[node1-128-18:17849] [[65177,0],2] posting persistent recv on tag 46 for peer >>[[WILDCARD],WILDCARD] >>[node1-128-18:17849] [[65177,0],2] posting recv >>[node1-128-18:17849] [[65177,0],2] posting persistent recv on tag 1 for peer >>[[WILDCARD],WILDCARD] >>[node1-128-18:17849] [[65177,0],2] OOB_SEND: rml_oob_send.c:199 >>[node1-128-18:17849] [[65177,0],2]:tcp:processing set_peer cmd >>[node1-128-18:17849] [[65177,0],2]:tcp:processing set_peer cmd >>[node1-128-18:17849] [[65177,0],2]:tcp:processing set_peer cmd >>[node1-128-18:17849] [[65177,0],2]:tcp:processing set_peer cmd >>[node1-128-18:17849] [[65177,0],2]:tcp:processing set_peer cmd >>[node1-128-18:17849] [[65177,0],2] oob:base:send to target [[65177,0],0] >>[node1-128-18:17849] [[65177,0],2] oob:tcp:send_nb to peer [[65177,0],0]:10 >>[node1-128-18:17849] [[65177,0],2] tcp:send_nb to peer [[65177,0],0] >>[node1-128-18:17849] [[65177,0],2]:[oob_tcp.c:484] post send to [[65177,0],0] >>[node1-128-18:17849] [[65177,0],2]:[oob_tcp.c:421] processing send to peer >>[[65177,0],0]:10 >>[node1-128-18:17849] [[65177,0],2]:[oob_tcp.c:455] queue pending to >>[[65177,0],0] >>[node1-128-18:17849] [[65177,0],2] tcp:send_nb: initiating connection to >>[[65177,0],0] >>[node1-128-18:17849] [[65177,0],2]:[oob_tcp.c:469] connect to [[65177,0],0] >>[node1-128-18:17849] [[65177,0],2] orte_tcp_peer_try_connect: attempting to >>connect to proc [[65177,0],0] >>[node1-128-18:17849] [[65177,0],2] orte_tcp_peer_try_connect: attempting to >>connect to proc [[65177,0],0] on socket 10 >>[node1-128-18:17849] [[65177,0],2] orte_tcp_peer_try_connect: attempting to >>connect to proc [[65177,0],0] on 10.0.251.53:49759 - 0 retries >>[node1-128-18:17849] [[65177,0],2] waiting for connect completion to >>[[65177,0],0] - activating send event >>[node1-128-18:17837] [[61806,0],2] tcp:send_handler called to send to peer >>[[61806,0],0] >>[node1-128-18:17837] [[61806,0],2] tcp:send_handler CONNECTING >>[node1-128-18:17837] [[61806,0],2]:tcp:complete_connect called for peer >>[[61806,0],0] on socket 10 >>[node1-128-18:17837] [[61806,0],2]-[[61806,0],0] tcp_peer_complete_connect: >>connection failed: Connection timed out (110) >>[node1-128-18:17837] [[61806,0],2] tcp_peer_close for [[61806,0],0] sd 10 >>state CONNECTING >>[node1-128-18:17837] [[61806,0],2] tcp:lost connection called for peer >>[[61806,0],0] >>[node1-128-18:17837] mca: base: close: component oob closed >>[node1-128-18:17837] mca: base: close: unloading component oob >>[node1-128-18:17837] [[61806,0],2] TCP SHUTDOWN >>[node1-128-18:17837] [[61806,0],2] RELEASING PEER OBJ [[61806,0],0] >>[node1-128-18:17837] [[61806,0],2] CLOSING SOCKET 10 >>[node1-128-18:17837] mca: base: close: component tcp closed >>[node1-128-18:17837] mca: base: close: unloading component tcp >>srun: error: node1-128-18: task 1: Exited with exit code 1 >>srun: Terminating job step 647191.1 >>[node1-128-17:14767] [[61806,0],1] tcp:send_handler called to send to peer >>[[61806,0],0] >>[node1-128-17:14767] [[61806,0],1] tcp:send_handler CONNECTING >>[node1-128-17:14767] [[61806,0],1]:tcp:complete_connect called for peer >>[[61806,0],0] on socket 10 >>[node1-128-17:14767] [[61806,0],1]-[[61806,0],0] tcp_peer_complete_connect: >>connection failed: Connection timed out (110) >>[node1-128-17:14767] [[61806,0],1] tcp_peer_close for [[61806,0],0] sd 10 >>state CONNECTING >>[node1-128-17:14767] [[61806,0],1] tcp:lost connection called for peer >>[[61806,0],0] >>[node1-128-17:14767] mca: base: close: component oob closed >>[node1-128-17:14767] mca: base: close: unloading component oob >>[node1-128-17:14767] [[61806,0],1] TCP SHUTDOWN >>[node1-128-17:14767] [[61806,0],1] RELEASING PEER OBJ [[61806,0],0] >>[node1-128-17:14767] [[61806,0],1] CLOSING SOCKET 10 >>[node1-128-17:14767] mca: base: close: component tcp closed >>[node1-128-17:14767] mca: base: close: unloading component tcp >>srun: error: node1-128-17: task 0: Exited with exit code 1 >>[node1-128-17:14779] [[65177,0],1] tcp:send_handler called to send to peer >>[[65177,0],0] >>[node1-128-17:14779] [[65177,0],1] tcp:send_handler CONNECTING >>[node1-128-17:14779] [[65177,0],1]:tcp:complete_connect called for peer >>[[65177,0],0] on socket 10 >>[node1-128-17:14779] [[65177,0],1]-[[65177,0],0] tcp_peer_complete_connect: >>connection failed: Connection timed out (110) >>[node1-128-17:14779] [[65177,0],1] tcp_peer_close for [[65177,0],0] sd 10 >>state CONNECTING >>[node1-128-17:14779] [[65177,0],1] tcp:lost connection called for peer >>[[65177,0],0] >>[node1-128-17:14779] mca: base: close: component oob closed >>[node1-128-17:14779] mca: base: close: unloading component oob >>[node1-128-17:14779] [[65177,0],1] TCP SHUTDOWN >>[node1-128-17:14779] [[65177,0],1] RELEASING PEER OBJ [[65177,0],0] >>[node1-128-17:14779] [[65177,0],1] CLOSING SOCKET 10 >>[node1-128-17:14779] mca: base: close: component tcp closed >>[node1-128-17:14779] mca: base: close: unloading component tcp >>[node1-128-18:17849] [[65177,0],2] tcp:send_handler called to send to peer >>[[65177,0],0] >>[node1-128-18:17849] [[65177,0],2] tcp:send_handler CONNECTING >>[node1-128-18:17849] [[65177,0],2]:tcp:complete_connect called for peer >>[[65177,0],0] on socket 10 >>[node1-128-18:17849] [[65177,0],2]-[[65177,0],0] tcp_peer_complete_connect: >>connection failed: Connection timed out (110) >>[node1-128-18:17849] [[65177,0],2] tcp_peer_close for [[65177,0],0] sd 10 >>state CONNECTING >>[node1-128-18:17849] [[65177,0],2] tcp:lost connection called for peer >>[[65177,0],0] >>[node1-128-18:17849] mca: base: close: component oob closed >>[node1-128-18:17849] mca: base: close: unloading component oob >>[node1-128-18:17849] [[65177,0],2] TCP SHUTDOWN >>[node1-128-18:17849] [[65177,0],2] RELEASING PEER OBJ [[65177,0],0] >>[node1-128-18:17849] [[65177,0],2] CLOSING SOCKET 10 >>[node1-128-18:17849] mca: base: close: component tcp closed >>[node1-128-18:17849] mca: base: close: unloading component tcp >>srun: error: node1-128-17: task 0: Exited with exit code 1 >>srun: Terminating job step 647191.2 >>srun: error: node1-128-18: task 1: Exited with exit code 1 >>-------------------------------------------------------------------------- >>An ORTE daemon has unexpectedly failed after launch and before >>communicating back to mpirun. This could be caused by a number >>of factors, including an inability to create a connection back >>to mpirun due to a lack of common network interfaces and/or no >>route found between them. Please check network connectivity >>(including firewalls and network routing requirements). >>-------------------------------------------------------------------------- >>[compiler-2:30735] [[65177,0],0] orted_cmd: received halt_vm cmd >>[compiler-2:30735] mca: base: close: component oob closed >>[compiler-2:30735] mca: base: close: unloading component oob >>[compiler-2:30735] [[65177,0],0] TCP SHUTDOWN >>[compiler-2:30735] mca: base: close: component tcp closed >>[compiler-2:30735] mca: base: close: unloading component tcp >> >> >>Sun, 20 Jul 2014 13:11:19 -0700 от Ralph Castain < r...@open-mpi.org >: >>>Yeah, we aren't connecting back - is there a firewall running? You need to >>>leave the "--debug-daemons --mca plm_base_verbose 5" on there as well to see >>>the entire problem. >>> >>>What you can see here is that mpirun is listening on several interfaces: >>>>[access1:24264] [[55095,0],0] oob:tcp:init adding 10.0.251.51 to our list >>>>of V4 connections >>>>[access1:24264] [[55095,0],0] oob:tcp:init adding 10.2.251.11 to our list >>>>of V4 connections >>>>[access1:24264] [[55095,0],0] oob:tcp:init adding 10.0.0.111 to our list of >>>>V4 connections >>>>[access1:24264] [[55095,0],0] oob:tcp:init adding 10.128.0.1 to our list of >>>>V4 connections >>>>[access1:24264] [[55095,0],0] oob:tcp:init adding 93.180.7.36 to our list >>>>of V4 connections >>> >>>It looks like you have multiple interfaces connected to the same subnet - >>>this is generally a bad idea. I also saw that the last one in the list shows >>>up twice in the kernel array - not sure why, but is there something special >>>about that NIC? >>> >>>What do the NICs look like on the remote hosts? >>> >>>On Jul 20, 2014, at 10:59 AM, Timur Ismagilov < tismagi...@mail.ru > wrote: >>>> >>>> >>>> >>>>-------- Пересылаемое сообщение -------- >>>>От кого: Timur Ismagilov < tismagi...@mail.ru > >>>>Кому: Ralph Castain < r...@open-mpi.org > >>>>Дата: Sun, 20 Jul 2014 21:58:41 +0400 >>>>Тема: Re[2]: [OMPI users] Fwd: Re[4]: Salloc and mpirun problem >>>> >>>>Here it is: >>>>$ salloc -N2 --exclusive -p test -J ompi >>>>salloc: Granted job allocation 647049 >>>> >>>>$ mpirun -mca mca_base_env_list 'LD_PRELOAD' -mca oob_base_verbose 10 -mca >>>>rml_base_verbose 10 -np 2 hello_c >>>>[access1:24264] mca: base: components_register: registering oob components >>>>[access1:24264] mca: base: components_register: found loaded component tcp >>>>[access1:24264] mca: base: components_register: component tcp register >>>>function successful >>>>[access1:24264] mca: base: components_open: opening oob components >>>>[access1:24264] mca: base: components_open: found loaded component tcp >>>>[access1:24264] mca: base: components_open: component tcp open function >>>>successful >>>>[access1:24264] mca:oob:select: checking available component tcp >>>>[access1:24264] mca:oob:select: Querying component [tcp] >>>>[access1:24264] oob:tcp: component_available called >>>>[access1:24264] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4 >>>>[access1:24264] WORKING INTERFACE 2 KERNEL INDEX 3 FAMILY: V4 >>>>[access1:24264] [[55095,0],0] oob:tcp:init adding 10.0.251.51 to our list >>>>of V4 connections >>>>[access1:24264] WORKING INTERFACE 3 KERNEL INDEX 4 FAMILY: V4 >>>>[access1:24264] [[55095,0],0] oob:tcp:init adding 10.0.0.111 to our list of >>>>V4 connections >>>>[access1:24264] WORKING INTERFACE 4 KERNEL INDEX 5 FAMILY: V4 >>>>[access1:24264] [[55095,0],0] oob:tcp:init adding 10.2.251.11 to our list >>>>of V4 connections >>>>[access1:24264] WORKING INTERFACE 5 KERNEL INDEX 6 FAMILY: V4 >>>>[access1:24264] [[55095,0],0] oob:tcp:init adding 10.128.0.1 to our list of >>>>V4 connections >>>>[access1:24264] WORKING INTERFACE 6 KERNEL INDEX 7 FAMILY: V4 >>>>[access1:24264] [[55095,0],0] oob:tcp:init adding 93.180.7.36 to our list >>>>of V4 connections >>>>[access1:24264] WORKING INTERFACE 7 KERNEL INDEX 7 FAMILY: V4 >>>>[access1:24264] [[55095,0],0] oob:tcp:init adding 93.180.7.36 to our list >>>>of V4 connections >>>>[access1:24264] [[55095,0],0] TCP STARTUP >>>>[access1:24264] [[55095,0],0] attempting to bind to IPv4 port 0 >>>>[access1:24264] [[55095,0],0] assigned IPv4 port 47756 >>>>[access1:24264] mca:oob:select: Adding component to end >>>>[access1:24264] mca:oob:select: Found 1 active transports >>>>[access1:24264] mca: base: components_register: registering rml components >>>>[access1:24264] mca: base: components_register: found loaded component oob >>>>[access1:24264] mca: base: components_register: component oob has no >>>>register or open function >>>>[access1:24264] mca: base: components_open: opening rml components >>>>[access1:24264] mca: base: components_open: found loaded component oob >>>>[access1:24264] mca: base: components_open: component oob open function >>>>successful >>>>[access1:24264] orte_rml_base_select: initializing rml component oob >>>>[access1:24264] [[55095,0],0] posting recv >>>>[access1:24264] [[55095,0],0] posting persistent recv on tag 30 for peer >>>>[[WILDCARD],WILDCARD] >>>>[access1:24264] [[55095,0],0] posting recv >>>>[access1:24264] [[55095,0],0] posting persistent recv on tag 15 for peer >>>>[[WILDCARD],WILDCARD] >>>>[access1:24264] [[55095,0],0] posting recv >>>>[access1:24264] [[55095,0],0] posting persistent recv on tag 32 for peer >>>>[[WILDCARD],WILDCARD] >>>>[access1:24264] [[55095,0],0] posting recv >>>>[access1:24264] [[55095,0],0] posting persistent recv on tag 33 for peer >>>>[[WILDCARD],WILDCARD] >>>>[access1:24264] [[55095,0],0] posting recv >>>>[access1:24264] [[55095,0],0] posting persistent recv on tag 5 for peer >>>>[[WILDCARD],WILDCARD] >>>>[access1:24264] [[55095,0],0] posting recv >>>>[access1:24264] [[55095,0],0] posting persistent recv on tag 10 for peer >>>>[[WILDCARD],WILDCARD] >>>>[access1:24264] [[55095,0],0] posting recv >>>>[access1:24264] [[55095,0],0] posting persistent recv on tag 12 for peer >>>>[[WILDCARD],WILDCARD] >>>>[access1:24264] [[55095,0],0] posting recv >>>>[access1:24264] [[55095,0],0] posting persistent recv on tag 9 for peer >>>>[[WILDCARD],WILDCARD] >>>>[access1:24264] [[55095,0],0] posting recv >>>>[access1:24264] [[55095,0],0] posting persistent recv on tag 34 for peer >>>>[[WILDCARD],WILDCARD] >>>>[access1:24264] [[55095,0],0] posting recv >>>>[access1:24264] [[55095,0],0] posting persistent recv on tag 2 for peer >>>>[[WILDCARD],WILDCARD] >>>>[access1:24264] [[55095,0],0] posting recv >>>>[access1:24264] [[55095,0],0] posting persistent recv on tag 21 for peer >>>>[[WILDCARD],WILDCARD] >>>>[access1:24264] [[55095,0],0] posting recv >>>>[access1:24264] [[55095,0],0] posting persistent recv on tag 22 for peer >>>>[[WILDCARD],WILDCARD] >>>>[access1:24264] [[55095,0],0] posting recv >>>>[access1:24264] [[55095,0],0] posting persistent recv on tag 45 for peer >>>>[[WILDCARD],WILDCARD] >>>>[access1:24264] [[55095,0],0] posting recv >>>>[access1:24264] [[55095,0],0] posting persistent recv on tag 46 for peer >>>>[[WILDCARD],WILDCARD] >>>>[access1:24264] [[55095,0],0] posting recv >>>>[access1:24264] [[55095,0],0] posting persistent recv on tag 1 for peer >>>>[[WILDCARD],WILDCARD] >>>>[access1:24264] [[55095,0],0] posting recv >>>>[access1:24264] [[55095,0],0] posting persistent recv on tag 27 for peer >>>>[[WILDCARD],WILDCARD] >>>>-------------------------------------------------------------------------- >>>>An ORTE daemon has unexpectedly failed after launch and before >>>>communicating back to mpirun. This could be caused by a number >>>>of factors, including an inability to create a connection back >>>>to mpirun due to a lack of common network interfaces and/or no >>>>route found between them. Please check network connectivity >>>>(including firewalls and network routing requirements). >>>>-------------------------------------------------------------------------- >>>>[access1:24264] mca: base: close: component oob closed >>>>[access1:24264] mca: base: close: unloading component oob >>>>[access1:24264] [[55095,0],0] TCP SHUTDOWN >>>>[access1:24264] mca: base: close: component tcp closed >>>>[access1:24264] mca: base: close: unloading component tcp >>>>When i use srun i got: >>>>$ salloc -N2 --exclusive -p test -J ompi >>>>.... >>>>$srun -N 2 ./hello_c >>>>Hello, world, I am 0 of 1, (Open MPI v1.9a1, package: Open MPI >>>>semenov@compiler-2 Distribution, ident: 1.9a1r32252, repo rev: r32252, Jul >>>>16, 2014 (nightly snapshot tarball), 146) >>>>Hello, world, I am 0 of 1, (Open MPI v1.9a1, package: Open MPI >>>>semenov@compiler-2 Distribution, ident: 1.9a1r32252, repo rev: r32252, Jul >>>>16, 2014 (nightly snapshot tarball), 146) >>>> >>>>Sun, 20 Jul 2014 09:28:13 -0700 от Ralph Castain < r...@open-mpi.org >: >>>>>Try adding -mca oob_base_verbose 10 -mca rml_base_verbose 10 to your cmd >>>>>line. It looks to me like we are unable to connect back to the node where >>>>>you are running mpirun for some reason. >>>>> >>>>> >>>>>On Jul 20, 2014, at 9:16 AM, Timur Ismagilov < tismagi...@mail.ru > wrote: >>>>>>I have the same problem in openmpi 1.8.1( Apr 23, 2014 ). >>>>>>Does the srun command have a --map-by<foo> mpirun parameter, or can i >>>>>>chage it from bash enviroment? >>>>>> >>>>>> >>>>>>-------- Пересылаемое сообщение -------- >>>>>>От кого: Timur Ismagilov < tismagi...@mail.ru > >>>>>>Кому: Mike Dubman < mi...@dev.mellanox.co.il > >>>>>>Копия: Open MPI Users < us...@open-mpi.org > >>>>>>Дата: Thu, 17 Jul 2014 16:42:24 +0400 >>>>>>Тема: Re[4]: [OMPI users] Salloc and mpirun problem >>>>>> >>>>>>With Open MPI 1.9a1r32252 (Jul 16, 2014 (nightly snapshot tarball)) i got >>>>>>this output (same?): >>>>>>$ salloc -N2 --exclusive -p test -J ompi >>>>>>salloc: Granted job allocation 645686 >>>>>> >>>>>>$LD_PRELOAD=/mnt/data/users/dm2/vol3/semenov/_scratch/mxm/mxm-3.0/lib/libmxm.so >>>>>> mpirun -mca mca_base_env_list 'LD_PRELOAD' --mca plm_base_verbose 10 >>>>>>--debug-daemons -np 1 hello_c >>>>>>[access1:04312] mca: base: components_register: registering plm components >>>>>>[access1:04312] mca: base: components_register: found loaded component >>>>>>isolated >>>>>>[access1:04312] mca: base: components_register: component isolated has no >>>>>>register or open function >>>>>>[access1:04312] mca: base: components_register: found loaded component rsh >>>>>>[access1:04312] mca: base: components_register: component rsh register >>>>>>function successful >>>>>>[access1:04312] mca: base: components_register: found loaded component >>>>>>slurm >>>>>>[access1:04312] mca: base: components_register: component slurm register >>>>>>function successful >>>>>>[access1:04312] mca: base: components_open: opening plm components >>>>>>[access1:04312] mca: base: components_open: found loaded component >>>>>>isolated >>>>>>[access1:04312] mca: base: components_open: component isolated open >>>>>>function successful >>>>>>[access1:04312] mca: base: components_open: found loaded component rsh >>>>>>[access1:04312] mca: base: components_open: component rsh open function >>>>>>successful >>>>>>[access1:04312] mca: base: components_open: found loaded component slurm >>>>>>[access1:04312] mca: base: components_open: component slurm open function >>>>>>successful >>>>>>[access1:04312] mca:base:select: Auto-selecting plm components >>>>>>[access1:04312] mca:base:select:( plm) Querying component [isolated] >>>>>>[access1:04312] mca:base:select:( plm) Query of component [isolated] set >>>>>>priority to 0 >>>>>>[access1:04312] mca:base:select:( plm) Querying component [rsh] >>>>>>[access1:04312] mca:base:select:( plm) Query of component [rsh] set >>>>>>priority to 10 >>>>>>[access1:04312] mca:base:select:( plm) Querying component [slurm] >>>>>>[access1:04312] mca:base:select:( plm) Query of component [slurm] set >>>>>>priority to 75 >>>>>>[access1:04312] mca:base:select:( plm) Selected component [slurm] >>>>>>[access1:04312] mca: base: close: component isolated closed >>>>>>[access1:04312] mca: base: close: unloading component isolated >>>>>>[access1:04312] mca: base: close: component rsh closed >>>>>>[access1:04312] mca: base: close: unloading component rsh >>>>>>Daemon was launched on node1-128-09 - beginning to initialize >>>>>>Daemon was launched on node1-128-15 - beginning to initialize >>>>>>Daemon [[39207,0],1] checking in as pid 26240 on host node1-128-09 >>>>>>[node1-128-09:26240] [[39207,0],1] orted: up and running - waiting for >>>>>>commands! >>>>>>Daemon [[39207,0],2] checking in as pid 30129 on host node1-128-15 >>>>>>[node1-128-15:30129] [[39207,0],2] orted: up and running - waiting for >>>>>>commands! >>>>>>srun: error: node1-128-09: task 0: Exited with exit code 1 >>>>>>srun: Terminating job step 645686.3 >>>>>>srun: error: node1-128-15: task 1: Exited with exit code 1 >>>>>>-------------------------------------------------------------------------- >>>>>>An ORTE daemon has unexpectedly failed after launch and before >>>>>>communicating back to mpirun. This could be caused by a number >>>>>>of factors, including an inability to create a connection back >>>>>>to mpirun due to a lack of common network interfaces and/or no >>>>>>route found between them. Please check network connectivity >>>>>>(including firewalls and network routing requirements). >>>>>>-------------------------------------------------------------------------- >>>>>>[access1:04312] [[39207,0],0] orted_cmd: received halt_vm cmd >>>>>>[access1:04312] mca: base: close: component slurm closed >>>>>>[access1:04312] mca: base: close: unloading component slurm >>>>>> >>>>>> >>>> >>>> >>>>---------------------------------------------------------------------- >>>> >>>> >>>>_______________________________________________ >>>>users mailing list >>>>us...@open-mpi.org >>>>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>Link to this post: >>>>http://www.open-mpi.org/community/lists/users/2014/07/24828.php >> >> >>
