Edgar, First, make sure your master includes https://github.com/open-mpi/ompi/commit/05af80b3025dbb95bdd4280087450791291d7219
If this is not enough, try with --mca coll ^ml Hope this helps Gilles. Edgar Gabriel <gabr...@cs.uh.edu>さんのメール: >I have some problems running jobs with ompi-master on one of our >clusters (after doing a major software update). Here are scenarios that >work and don't work. > >1. Everything still seems to work with 1.8.x series without any issues >2. With master, I can run without any issues single node, multi-process jobs >3. With master, I can run without any issues multi node jobs, as long as >there is only a single MPI process per node > >4. I can not run multi-node jobs with multiple processes per node, ompi >hangs for that scenario. This is independent of whether I enforce using >openib or tcp, and just for the sake of simplicity I attach the output >for tcp (there is another openib parameter issue that still linguers, >but I will report that later). > >Here is the output that I receive if setting btl_base_verbose > >---------------snip------------------ >gabriel@crill:~> salloc -N 2 -n 4 >gabriel@crill:~> mpirun --mca btl tcp,self --mca btl_base_verbose 30 -np > 4 ./hello_world >[crill-004:18161] mca: base: components_register: registering btl components >[crill-004:18161] mca: base: components_register: found loaded component >self >[crill-004:18161] mca: base: components_register: component self >register function successful >[crill-004:18161] mca: base: components_register: found loaded component tcp >[crill-004:18161] mca: base: components_register: component tcp register >function successful >[crill-004:18161] mca: base: components_open: opening btl components >[crill-004:18161] mca: base: components_open: found loaded component self >[crill-004:18161] mca: base: components_open: component self open >function successful >[crill-004:18161] mca: base: components_open: found loaded component tcp >[crill-004:18161] mca: base: components_open: component tcp open >function successful >[crill-004:18161] select: initializing btl component self >[crill-004:18161] select: init of component self returned success >[crill-004:18161] select: initializing btl component tcp >[crill-004:18161] btl: tcp: Searching for exclude address+prefix: >127.0.0.1 / 8 >[crill-004:18161] btl: tcp: Found match: 127.0.0.1 (lo) >[crill-004:18161] select: init of component tcp returned success >[crill-003:18962] mca: base: components_register: registering btl components >[crill-003:18962] mca: base: components_register: found loaded component >self >[crill-003:18962] mca: base: components_register: component self >register function successful >[crill-003:18962] mca: base: components_register: found loaded component tcp >[crill-003:18962] mca: base: components_register: component tcp register >function successful >[crill-003:18962] mca: base: components_open: opening btl components >[crill-003:18962] mca: base: components_open: found loaded component self >[crill-003:18962] mca: base: components_open: component self open >function successful >[crill-003:18962] mca: base: components_open: found loaded component tcp >[crill-003:18962] mca: base: components_open: component tcp open >function successful >[crill-003:18963] mca: base: components_register: registering btl components >[crill-003:18963] mca: base: components_register: found loaded component >self >[crill-003:18963] mca: base: components_register: component self >register function successful >[crill-003:18963] mca: base: components_register: found loaded component tcp >[crill-003:18963] mca: base: components_register: component tcp register >function successful >[crill-003:18963] mca: base: components_open: opening btl components >[crill-003:18963] mca: base: components_open: found loaded component self >[crill-003:18963] mca: base: components_open: component self open >function successful >[crill-003:18963] mca: base: components_open: found loaded component tcp >[crill-003:18963] mca: base: components_open: component tcp open >function successful >[crill-003:18964] mca: base: components_register: registering btl components >[crill-003:18964] mca: base: components_register: found loaded component >self >[crill-003:18964] mca: base: components_register: component self >register function successful >[crill-003:18964] mca: base: components_register: found loaded component tcp >[crill-003:18964] mca: base: components_register: component tcp register >function successful >[crill-003:18964] mca: base: components_open: opening btl components >[crill-003:18964] mca: base: components_open: found loaded component self >[crill-003:18964] mca: base: components_open: component self open >function successful >[crill-003:18964] mca: base: components_open: found loaded component tcp >[crill-003:18964] mca: base: components_open: component tcp open >function successful >[crill-003:18962] select: initializing btl component self >[crill-003:18962] select: init of component self returned success >[crill-003:18962] select: initializing btl component tcp >[crill-003:18962] btl: tcp: Searching for exclude address+prefix: >127.0.0.1 / 8 >[crill-003:18962] btl: tcp: Found match: 127.0.0.1 (lo) >[crill-003:18962] select: init of component tcp returned success >[crill-003:18963] select: initializing btl component self >[crill-003:18963] select: init of component self returned success >[crill-003:18963] select: initializing btl component tcp >[crill-003:18963] btl: tcp: Searching for exclude address+prefix: >127.0.0.1 / 8 >[crill-003:18963] btl: tcp: Found match: 127.0.0.1 (lo) >[crill-003:18963] select: init of component tcp returned success >[crill-003:18964] select: initializing btl component self >[crill-003:18964] select: init of component self returned success >[crill-003:18964] select: initializing btl component tcp >[crill-003:18964] btl: tcp: Searching for exclude address+prefix: >127.0.0.1 / 8 >[crill-003:18964] btl: tcp: Found match: 127.0.0.1 (lo) >[crill-003:18964] select: init of component tcp returned success >[crill-003:18964] mca: bml: Using self btl to [[3417,1],2] on node crill-003 >[crill-003:18964] mca: bml: Using tcp btl to [[3417,1],0] on node crill-003 >[crill-003:18964] mca: bml: Using tcp btl to [[3417,1],1] on node crill-003 >[crill-003:18964] mca: bml: Using tcp btl to [[3417,1],3] on node crill-004 >[crill-004:18161] mca: bml: Using self btl to [[3417,1],3] on node crill-004 >[crill-004:18161] mca: bml: Using tcp btl to [[3417,1],0] on node crill-003 >[crill-004:18161] mca: bml: Using tcp btl to [[3417,1],1] on node crill-003 >[crill-004:18161] mca: bml: Using tcp btl to [[3417,1],2] on node crill-003 >^C > >------------ >and than it just hangs. > >Does anybody have an idea/suggestion what to try or look for? > >Thanks >Edgar > >-- >Edgar Gabriel >Associate Professor >Parallel Software Technologies Lab http://pstl.cs.uh.edu >Department of Computer Science University of Houston >Philip G. Hoffman Hall, Room 524 Houston, TX-77204, USA >Tel: +1 (713) 743-3857 Fax: +1 (713) 743-3335 >_______________________________________________ >devel mailing list >de...@open-mpi.org >Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >Link to this post: >http://www.open-mpi.org/community/lists/devel/2014/12/16727.php
