Thanks! It fixes the problem with tcp. Best regards, Elena
On Thu, Nov 6, 2014 at 10:44 AM, George Bosilca <bosi...@icl.utk.edu> wrote: > I pushed a slightly better patch for the TCP BTL > (54ddb0aece0892dcdb1a1293a3bd3902b5f3acdc). The correct scheme would be to > OBJ_RETAIN the proc once it is attached to the btl_proc and release it upon > destruction of the btl_proc. However, for some obscure reason this doesn't > quite works, as the main opel_proc is apparently already destroyed when we > try to finalize the TCP BTL. > > George. > > PS: Sorry Dave I also pushed a master branch merge ... > > > On Thu, Nov 6, 2014 at 1:10 AM, Gilles Gouaillardet < > gilles.gouaillar...@iferc.org> wrote: > >> Ralph, >> >> i updated the MODEX flag to PMIX_GLOBAL >> >> https://github.com/open-mpi/ompi/commit/d542c9ff2dc57ca5d260d0578fd5c1c556c598c7 >> >> Elena, >> >> i was able to reproduce the issue (salloc -N 5 mpirun -np 2 is enough). >> i was "lucky" to reproduce the issue : it happened because one of node >> was misconfigured >> with two interfaces in the same subnet (!) >> >> could you please give a try to the attached patch ? >> i did not commit it because i do not know if this is the right fix or a >> simple workaround >> /* for example, should the opal_proc_t be OBJ_RETAINed before invoking >> add_procs and >> then be OBJ_RELEASEd by the btl add_proc if it is unreachable ? */ >> >> Cheers, >> >> Gilles >> >> On 2014/11/06 12:46, Ralph Castain wrote: >> >> On Nov 5, 2014, at 6:11 PM, Gilles Gouaillardet >> <gilles.gouaillar...@iferc.org> <gilles.gouaillar...@iferc.org> wrote: >> >> Elena, >> >> the first case (-mca btl tcp,self) crashing is a bug and i will have a look >> at it. >> >> the second case (-mca sm,self) is a feature : the sm btl cannot be used with >> tasks >> having different jobids (this is the case after a spawn), and obviously, >> self cannot be used also, >> so the behaviour and error message is correct. >> /* i am not aware of any plans to make the sm btl work with tasks from >> different jobids */\ >> >> That is correct - I’m also unaware of any plans to extend it at this point, >> though IIRC Nathan at one time mentioned perhaps extending vader for that >> purpose >> >> >> the third case (-mca openib,self) is more controversial ... >> i previously posted >> http://www.open-mpi.org/community/lists/devel/2014/10/16136.php >> <http://www.open-mpi.org/community/lists/devel/2014/10/16136.php> >> <http://www.open-mpi.org/community/lists/devel/2014/10/16136.php> >> >> what happens in your case (simple_spawn) is the openib modex is sent with >> PMIX_REMOTE, >> that means openib btl cannot be used between tasks on the same node. >> i am still waiting for some feedback since i cannot figure out whether this >> is a feature or an >> undesired side effect / bug >> >> I believe it is a bug - I provided some initial values for the modex scope >> with the expectation (and request when we committed it) that people would >> review and modify them as appropriate. I recall setting the openib scope as >> “remote” only because I wasn’t aware of anyone using it for local comm. >> Since Mellanox obviously is testing for that case, a scope of PMIX_GLOBAL >> would be more appropriate >> >> >> the last cast (-mca ^sm,openib) does make sense to me : >> the tcp and self btls are used and they work just like they should. >> >> bottom line, i will investigate the first crash, wait for feedback about the >> openib btl. >> >> Cheers, >> >> Gilles >> >> On 2014/11/06 1:08, Elena Elkina wrote: >> >> Hi, >> >> It looks like there is a problem in trunk which reproduces with >> simple_spawn test (orte/test/mpi/simple_spawn.c). It seems to be a n issue >> with pmix. It doesn't reproduce with default set of btls. But it reproduces >> with several btls specified. For example, >> >> salloc -N5 $OMPI_HOME/install/bin/mpirun -np 33 --map-by node -mca coll ^ml >> -display-map -mca orte_debug_daemons true --leave-session-attached >> --debug-daemons -mca pml ob1 -mca btl *tcp,self* >> ./orte/test/mpi/simple_spawn >> >> gets >> >> simple_spawn: ../../ompi/group/group_init.c:215: >> ompi_group_increment_proc_count: Assertion `((0xdeafbeedULL << 32) + >> 0xdeafbeedULL) == ((opal_object_t *) (proc_pointer))->obj_magic_id' failed. >> [sputnik3.vbench.com:28888] [[41877,0],3] orted_cmd: exit cmd, but proc >> [[41877,1],2] is alive >> [sputnik5][[41877,1],29][../../../../../opal/mca/btl/tcp/btl_tcp_endpoint.c:675:mca_btl_tcp_endpoint_complete_connect] >> connect() to 192.168.1.42 failed: Connection refused (111) >> >> salloc -N1 $OMPI_HOME/install/bin/mpirun -np 3 --map-by node -mca coll ^ml >> -display-map -mca orte_debug_daemons true --leave-session-attached >> --debug-daemons -mca pml ob1 -mca btl *sm,self* ./orte/test/mpi/simple_spawn >> >> fails with >> >> At least one pair of MPI processes are unable to reach each other for >> MPI communications. This means that no Open MPI device has indicated >> that it can be used to communicate between these processes. This is >> an error; Open MPI requires that all MPI processes be able to reach >> each other. This error can sometimes be the result of forgetting to >> specify the "self" BTL. >> >> Process 1 ([[59481,2],0]) is on host: sputnik1 >> Process 2 ([[59481,1],0]) is on host: sputnik1 >> BTLs attempted: self sm >> >> Your MPI job is now going to abort; sorry. >> -------------------------------------------------------------------------- >> [sputnik1.vbench.com:22156] [[59481,1],2] ORTE_ERROR_LOG: Unreachable in >> file ../../../../../ompi/mca/dpm/orte/dpm_orte.c at line 485 >> >> >> salloc -N1 $OMPI_HOME/install/bin/mpirun -np 3 --map-by node -mca coll ^ml >> -display-map -mca orte_debug_daemons true --leave-session-attached >> --debug-daemons -mca pml ob1 -mca btl *openib,self* >> ./orte/test/mpi/simple_spawn >> >> also doesn't work: >> At least one pair of MPI processes are unable to reach each other for >> MPI communications. This means that no Open MPI device has indicated >> that it can be used to communicate between these processes. This is >> an error; Open MPI requires that all MPI processes be able to reach >> each other. This error can sometimes be the result of forgetting to >> specify the "self" BTL. >> >> Process 1 ([[60046,1],13]) is on host: sputnik4 >> Process 2 ([[60046,2],1]) is on host: sputnik4 >> BTLs attempted: openib self >> >> Your MPI job is now going to abort; sorry. >> -------------------------------------------------------------------------- >> [sputnik4.vbench.com:25476] [[60046,1],3] ORTE_ERROR_LOG: Unreachable in >> file ../../../../../ompi/mca/dpm/orte/dpm_orte.c at line 485 >> >> >> *But* combination ^sm,openib seems to work. >> >> I tried different revisions from the beginning of October. It reproduces on >> them. >> >> Best regards, >> Elena >> >> >> >> _______________________________________________ >> devel mailing list >> de...@open-mpi.org <mailto:de...@open-mpi.org> <de...@open-mpi.org> >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >> <http://www.open-mpi.org/mailman/listinfo.cgi/devel> >> <http://www.open-mpi.org/mailman/listinfo.cgi/devel> >> Link to this post: >> http://www.open-mpi.org/community/lists/devel/2014/11/16202.php >> <http://www.open-mpi.org/community/lists/devel/2014/11/16202.php> >> <http://www.open-mpi.org/community/lists/devel/2014/11/16202.php> >> >> _______________________________________________ >> devel mailing listde...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >> Link to this post: >> http://www.open-mpi.org/community/lists/devel/2014/11/16223.php >> >> >> >> _______________________________________________ >> devel mailing listde...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >> Link to this post: >> http://www.open-mpi.org/community/lists/devel/2014/11/16225.php >> >> >> >> _______________________________________________ >> devel mailing list >> de...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >> Link to this post: >> http://www.open-mpi.org/community/lists/devel/2014/11/16226.php >> > > > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2014/11/16227.php >
