I suspect something else is wrong - the grpcomm system never has any visibility as to what data goes into the modex, or how that data is used. In other words, if the tcp btl isn't providing adequate info, then it would fail regardless of which grpcomm module was in use. So your statement about the hier module not distinguishing between peers on the same node doesn't make sense - the hier module has no idea that a tcp btl even exists, let alone have anything to do with the modex data.
You might take a look at how the tcp btl is picking its sockets. The srun direct launch method may be setting envars that confuse it, perhaps causing the procs to all pick the same socket. On Mar 16, 2011, at 12:48 PM, Damien Guinier wrote: > Hi all > > From my test, it is impossible to use "btl:tcp" with "grpcomm:hier". The > "grpcomm:hier" module is important because, "srun" launch protocol can't use > any other "grpcomm" module. > You can reproduce this bug, by using "btl:tcp" and "grpcomm:hier" , when you > create a ring(like: IMB sendrecv) > > $>salloc -N 2 -n 4 mpirun --mca grpcomm hier --mca btl self,sm,tcp ./IMB-MPI1 > Sendrecv > salloc: Granted job allocation 2979 > [cuzco95][[59536,1],2][btl_tcp_endpoint.c:486:mca_btl_tcp_endpoint_recv_connect_ack] > received unexpected process identifier [[59536,1],0] > [cuzco92][[59536,1],0][btl_tcp_endpoint.c:486:mca_btl_tcp_endpoint_recv_connect_ack] > received unexpected process identifier [[59536,1],2] > ^C > $> > > This error message show: "btl:tcp" have create a connection to a peer, but it > not the good one ( peer identity is checked with the "ack"). > To create a connection between two peers with "btl:tcp": > - Each peer broadcast theirs IP parameters with ompi_modex_send(). > - IP parameters from selected peer is received with ompi_modex_recv(). > > In fact, modex use "orte_grpcomm.set_proc_attr()" and > "orte_grpcomm.get_proc_attr()" to exchange data. The problem is > "grpcomm:hier" doesn't make difference between two peer on the same node. > From my test the IP parameters, from the fist rank on the selected node, is > always return. > > > "grpcomm:hier" is restricted to "btl:sm" and "btl:openib" ? > > > -------- > > One easy solution to fix this problem, is to add rank information in the > "name" variable on > - ompi/runtime/ompi_module_exchange.c:ompi_modex_send() > - ompi/runtime/ompi_module_exchange.c:ompi_modex_recv() > but I dislike it. > > Someone have a better solution ? > > > thanks you > Damien > > _______________________________________________ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel
