It seems the change suggested by Nysal also allows me to run into the next problem ;-)
Mark > On 09 Nov 2015, at 20:19 , George Bosilca <bosi...@icl.utk.edu> wrote: > > All 10k tests completed successfully. Nysal pinpointed the real problem > behind the deadlocks. :+1: > > George. > > > On Mon, Nov 9, 2015 at 1:13 PM, Ralph Castain <r...@open-mpi.org> wrote: > Looking at it, I think I see what was happening. The thread would start, but > then immediately see that the active flag was false and would exit. This left > the server without any listening thread - but it wouldn’t detect this had > happened. It was therefore a race between whether the thread checked the flag > before the server set it. > > Thanks Nysal - I believe this should indeed fix the problem! > > >> On Nov 9, 2015, at 9:04 AM, George Bosilca <bosi...@icl.utk.edu> wrote: >> >> Clearly Nyal got a valid point there. I launched a stress test with Nysal >> suggestion in the code, and so far it's up to few hundreds iterations >> without deadlock. I would not claim victory yet, I launched a 10k cycle to >> see where we stand (btw this never passed before). >> I'll let you know the outcome. >> >> George. >> >> >> On Mon, Nov 9, 2015 at 11:55 AM, Artem Polyakov <artpo...@gmail.com> wrote: >> >> >> 2015-11-09 22:42 GMT+06:00 Artem Polyakov <artpo...@gmail.com>: >> This is the very good point, Nysal! >> >> This is definitely a problem and I can say even more: avg. 3 from every 10 >> tasks was affected by this bug. Once the PR >> (https://github.com/pmix/master/pull/8) was applied I was able to run 100 >> testing tasks without any hangs. >> >> Here some more information on my symptoms. I was observing this without >> OMPI, just running pmix_client test binary from PMIx test suite with SLURM >> PMIx plugin. >> Periodicaly application was hanging. Investigation shows that not all >> processes are able to initialize correctly. >> Here is how such client's backtrace looks like: >> >> P.S. I think that this backtrace may be relevant to George's problem as >> well. In my case not all of the processes was hanging in the >> connect_to_server, most of them were able to move forward and reach Fence. >> George, the backtrace that you've posted was the same on both processes or >> it was the "random" one from one of them? >> >> (gdb) bt >> #0 0x00007f1448f1b7eb in recv () from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x00007f144914c191 in pmix_usock_recv_blocking (sd=9, >> data=0x7fff367f7c64 "", size=4) at src/usock/usock.c:166 >> #2 0x00007f1449152d18 in recv_connect_ack (sd=9) at >> src/client/pmix_client.c:837 >> #3 0x00007f14491546bf in usock_connect (addr=0x7fff367f7d60) at >> src/client/pmix_client.c:1103 >> #4 0x00007f144914f94c in connect_to_server (address=0x7fff367f7d60, >> cbdata=0x7fff367f7dd0) at src/client/pmix_client.c:179 >> #5 0x00007f1449150421 in PMIx_Init (proc=0x7fff367f81d0) at >> src/client/pmix_client.c:355 >> #6 0x0000000000401b97 in main (argc=9, argv=0x7fff367f83d8) at >> pmix_client.c:62 >> >> >> The server-side debug has the following lines at the end of the file: >> [cn33:00482] pmix:server register client slurm.pmix.22.0:10 >> [cn33:00482] pmix:server _register_client for nspace slurm.pmix.22.0 rank 10 >> [cn33:00482] pmix:server setup_fork for nspace slurm.pmix.22.0 rank 10 >> >> in normal operation the following lines should appear after lines above: >> .... >> [cn33:00188] listen_thread: new connection: (26, 0) >> [cn33:00188] connection_handler: new connection: 26 >> [cn33:00188] RECV CONNECT ACK FROM PEER ON SOCKET 26 >> [cn33:00188] waiting for blocking recv of 16 bytes >> [cn33:00188] blocking receive complete from remote >> .... >> >> At the client side I see the following lines >> cn33:00491] usock_peer_try_connect: attempting to connect to server >> [cn33:00491] usock_peer_try_connect: attempting to connect to server on >> socket 10 >> [cn33:00491] pmix: SEND CONNECT ACK >> [cn33:00491] sec: native create_cred >> [cn33:00491] sec: using credential 1000:1000 >> [cn33:00491] send blocking of 54 bytes to socket 10 >> [cn33:00491] blocking send complete to socket 10 >> [cn33:00491] pmix: RECV CONNECT ACK FROM SERVER >> [cn33:00491] waiting for blocking recv of 4 bytes >> [cn33:00491] blocking_recv received error 11:Resource temporarily >> unavailable from remote - cycling >> [cn33:00491] blocking_recv received error 11:Resource temporarily >> unavailable from remote - cycling >> [... repeated many times ...] >> >> With the fix for the problem highlighted by Nysal all runs cleanly. >> >> >> 2015-11-09 10:53 GMT+06:00 Nysal Jan K A <jny...@gmail.com>: >> In listen_thread(): >> 194 while (pmix_server_globals.listen_thread_active) { >> 195 FD_ZERO(&readfds); >> 196 FD_SET(pmix_server_globals.listen_socket, &readfds); >> 197 max = pmix_server_globals.listen_socket; >> >> Is it possible that pmix_server_globals.listen_thread_active can be false, >> in which case the thread just exits and will never call accept() ? >> >> In pmix_start_listening(): >> 147 /* fork off the listener thread */ >> 148 if (0 > pthread_create(&engine, NULL, listen_thread, NULL)) { >> 149 return PMIX_ERROR; >> 150 } >> 151 pmix_server_globals.listen_thread_active = true; >> >> pmix_server_globals.listen_thread_active is set to true after the thread is >> created, could this cause a race ? >> listen_thread_active might also need to be declared as volatile. >> >> Regards >> --Nysal >> >> On Sun, Nov 8, 2015 at 10:38 PM, George Bosilca <bosi...@icl.utk.edu> wrote: >> We had a power outage last week and the local disks on our cluster were >> wiped out. My tester was in there. But, I can rewrite it after SC. >> >> George. >> >> On Sat, Nov 7, 2015 at 12:04 PM, Ralph Castain <r...@open-mpi.org> wrote: >> Could you send me your stress test? I’m wondering if it is just something >> about how we set socket options >> >> >>> On Nov 7, 2015, at 8:58 AM, George Bosilca <bosi...@icl.utk.edu> wrote: >>> >>> I has to postpone this until after SC. However, I ran for 3 days a stress >>> test of UDS reproducing the opening and sending of data (what Ralph said in >>> his email) and I never could get a deadlock. >>> >>> George. >>> >>> >>> On Sat, Nov 7, 2015 at 11:26 AM, Ralph Castain <r...@open-mpi.org> wrote: >>> George was looking into it, but I don’t know if he has had time recently to >>> continue the investigation. We understand “what” is happening (accept >>> sometimes ignores the connection), but we don’t yet know “why”. I’ve done >>> some digging around the web, and found that sometimes you can try to talk >>> to a Unix Domain Socket too quickly - i.e., you open it and then send to >>> it, but the OS hasn’t yet set it up. In those cases, you can hang the >>> socket. However, I’ve tried adding some artificial delay, and while it >>> helped, it didn’t completely solve the problem. >>> >>> I have an idea for a workaround (set a timer and retry after awhile), but >>> would obviously prefer a real solution. I’m not even sure it will work as >>> it is unclear that the server (who is the one hung in accept) will break >>> free if the client closes the socket and retries. >>> >>> >>>> On Nov 6, 2015, at 10:53 PM, Artem Polyakov <artpo...@gmail.com> wrote: >>>> >>>> Hello, is there any progress on this topic? This affects our PMIx >>>> measurements. >>>> >>>> 2015-10-30 21:21 GMT+06:00 Ralph Castain <r...@open-mpi.org>: >>>> I’ve verified that the orte/util/listener thread is not being started, so >>>> I don’t think it should be involved in this problem. >>>> >>>> HTH >>>> Ralph >>>> >>>>> On Oct 30, 2015, at 8:07 AM, Ralph Castain <r...@open-mpi.org> wrote: >>>>> >>>>> Hmmm…there is a hook that would allow the PMIx server to utilize that >>>>> listener thread, but we aren’t currently using it. Each daemon plus >>>>> mpirun will call orte_start_listener, but nothing is currently >>>>> registering and so the listener in that code is supposed to just return >>>>> without starting the thread. >>>>> >>>>> So the only listener thread that should exist is the one inside the PMIx >>>>> server itself. If something else is happening, then that would be a bug. >>>>> I can look at the orte listener code to ensure that the thread isn’t >>>>> incorrectly starting. >>>>> >>>>> >>>>>> On Oct 29, 2015, at 10:03 PM, George Bosilca <bosi...@icl.utk.edu> wrote: >>>>>> >>>>>> Some progress, that puzzles me but might help you understand. Once the >>>>>> deadlock appears, if I manually kill the MPI process on the node where >>>>>> the deadlock was created, the local orte daemon doesn't notice and will >>>>>> just keep waiting. >>>>>> >>>>>> Quick question: I am under the impression that the issue is not in the >>>>>> PMIX server but somewhere around the listener_thread_fn in >>>>>> orte/util/listener.c. Possible ? >>>>>> >>>>>> George. >>>>>> >>>>>> >>>>>> On Wed, Oct 28, 2015 at 3:56 AM, Ralph Castain <r...@open-mpi.org> wrote: >>>>>> Should have also clarified: the prior fixes are indeed in the current >>>>>> master. >>>>>> >>>>>>> On Oct 28, 2015, at 12:42 AM, Ralph Castain <r...@open-mpi.org> wrote: >>>>>>> >>>>>>> Nope - I was wrong. The correction on the client side consisted of >>>>>>> attempting to timeout if the blocking recv failed. We then modified the >>>>>>> blocking send/recv so they would handle errors. >>>>>>> >>>>>>> So that problem occurred -after- the server had correctly called >>>>>>> accept. The listener code is in >>>>>>> opal/mca/pmix/pmix1xx/pmix/src/server/pmix_server_listener.c >>>>>>> >>>>>>> It looks to me like the only way we could drop the accept (assuming the >>>>>>> OS doesn’t lose it) is if the file descriptor lies outside the expected >>>>>>> range once we fall out of select: >>>>>>> >>>>>>> >>>>>>> /* Spin accepting connections until all active listen sockets >>>>>>> * do not have any incoming connections, pushing each connection >>>>>>> * onto the event queue for processing >>>>>>> */ >>>>>>> do { >>>>>>> accepted_connections = 0; >>>>>>> /* according to the man pages, select replaces the given >>>>>>> descriptor >>>>>>> * set with a subset consisting of those descriptors that >>>>>>> are ready >>>>>>> * for the specified operation - in this case, a read. So >>>>>>> we need to >>>>>>> * first check to see if this file descriptor is included >>>>>>> in the >>>>>>> * returned subset >>>>>>> */ >>>>>>> if (0 == FD_ISSET(pmix_server_globals.listen_socket, >>>>>>> &readfds)) { >>>>>>> /* this descriptor is not included */ >>>>>>> continue; >>>>>>> } >>>>>>> >>>>>>> /* this descriptor is ready to be read, which means a >>>>>>> connection >>>>>>> * request has been received - so harvest it. All we want >>>>>>> to do >>>>>>> * here is accept the connection and push the info onto the >>>>>>> event >>>>>>> * library for subsequent processing - we don't want to >>>>>>> actually >>>>>>> * process the connection here as it takes too long, and so >>>>>>> the >>>>>>> * OS might start rejecting connections due to timeout. >>>>>>> */ >>>>>>> pending_connection = PMIX_NEW(pmix_pending_connection_t); >>>>>>> event_assign(&pending_connection->ev, pmix_globals.evbase, >>>>>>> -1, >>>>>>> EV_WRITE, connection_handler, >>>>>>> pending_connection); >>>>>>> pending_connection->sd = >>>>>>> accept(pmix_server_globals.listen_socket, >>>>>>> (struct >>>>>>> sockaddr*)&(pending_connection->addr), >>>>>>> &addrlen); >>>>>>> if (pending_connection->sd < 0) { >>>>>>> PMIX_RELEASE(pending_connection); >>>>>>> if (pmix_socket_errno != EAGAIN || >>>>>>> pmix_socket_errno != EWOULDBLOCK) { >>>>>>> if (EMFILE == pmix_socket_errno) { >>>>>>> PMIX_ERROR_LOG(PMIX_ERR_OUT_OF_RESOURCE); >>>>>>> } else { >>>>>>> pmix_output(0, "listen_thread: accept() failed: >>>>>>> %s (%d).", >>>>>>> strerror(pmix_socket_errno), >>>>>>> pmix_socket_errno); >>>>>>> } >>>>>>> goto done; >>>>>>> } >>>>>>> continue; >>>>>>> } >>>>>>> >>>>>>> pmix_output_verbose(8, pmix_globals.debug_output, >>>>>>> "listen_thread: new connection: (%d, >>>>>>> %d)", >>>>>>> pending_connection->sd, >>>>>>> pmix_socket_errno); >>>>>>> /* activate the event */ >>>>>>> event_active(&pending_connection->ev, EV_WRITE, 1); >>>>>>> accepted_connections++; >>>>>>> } while (accepted_connections > 0); >>>>>>> >>>>>>> >>>>>>>> On Oct 28, 2015, at 12:25 AM, Ralph Castain <r...@open-mpi.org> wrote: >>>>>>>> >>>>>>>> Looking at the code, it appears that a fix was committed for this >>>>>>>> problem, and that we correctly resolved the issue found by Paul. The >>>>>>>> problem is that the fix didn’t get upstreamed, and so it was lost the >>>>>>>> next time we refreshed PMIx. Sigh. >>>>>>>> >>>>>>>> Let me try to recreate the fix and have you take a gander at it. >>>>>>>> >>>>>>>> >>>>>>>>> On Oct 28, 2015, at 12:22 AM, Ralph Castain <r...@open-mpi.org> wrote: >>>>>>>>> >>>>>>>>> Here is the discussion - afraid it is fairly lengthy. Ignore the >>>>>>>>> hwloc references in it as that was a separate issue: >>>>>>>>> >>>>>>>>> http://www.open-mpi.org/community/lists/devel/2015/09/18074.php >>>>>>>>> >>>>>>>>> It definitely sounds like the same issue creeping in again. I’d >>>>>>>>> appreciate any thoughts on how to correct it. If it helps, you could >>>>>>>>> look at the PMIx master - there are standalone tests in the >>>>>>>>> test/simple directory that fork/exec a child and just do the >>>>>>>>> connection. >>>>>>>>> >>>>>>>>> https://github.com/pmix/master >>>>>>>>> >>>>>>>>> The test server is simptest.c - it will spawn a single copy of >>>>>>>>> simpclient.c by default. >>>>>>>>> >>>>>>>>> >>>>>>>>>> On Oct 27, 2015, at 10:14 PM, George Bosilca <bosi...@icl.utk.edu> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Interesting. Do you have a pointer to the commit (or/and to the >>>>>>>>>> discussion)? >>>>>>>>>> >>>>>>>>>> I looked at the PMIX code, and I have identified few issues, but >>>>>>>>>> unfortunately none of them seem to fix the problem for good. >>>>>>>>>> However, now I need more than 1000 runs to get a deadlock (instead >>>>>>>>>> of few tens). >>>>>>>>>> >>>>>>>>>> Looking with "netstat -ax" at the status of the UDS while the >>>>>>>>>> processes are deadlocked, I see 2 UDS with the same name: one from >>>>>>>>>> the server which is in LISTEN state, and one for the client which is >>>>>>>>>> being in CONNECTING state (while the client already sent a message >>>>>>>>>> in the socket and is now waiting in a blocking receive). This >>>>>>>>>> somehow suggest that the server has not yet called accept on the >>>>>>>>>> UDS. Unfortunately, there are 3 threads all doing different flavors >>>>>>>>>> of even_base and select, so I have a hard time tracking the path of >>>>>>>>>> the UDS on the server side. >>>>>>>>>> >>>>>>>>>> So in order to validate my assumption I wrote a minimalistic UDS >>>>>>>>>> client and server application and tried different scenarios. The >>>>>>>>>> conclusion is that in order to see the same type of output from >>>>>>>>>> "netstat -ax" I have to call listen on the server, connect on the >>>>>>>>>> client and do not call accept on the server. >>>>>>>>>> >>>>>>>>>> With the same occasion I also confirmed that the UDS are holding the >>>>>>>>>> data sent so there is no need for further synchronization for the >>>>>>>>>> case where the data is sent first. We only need to find out how the >>>>>>>>>> server forgets to call accept. >>>>>>>>>> >>>>>>>>>> George. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Oct 27, 2015 at 7:52 PM, Ralph Castain <r...@open-mpi.org> >>>>>>>>>> wrote: >>>>>>>>>> Hmmm…this looks like it might be that problem we previously saw >>>>>>>>>> where the blocking recv hangs in a proc when the blocking send tries >>>>>>>>>> to send before the domain socket is actually ready, and so the send >>>>>>>>>> fails on the other end. As I recall, it was something to do with the >>>>>>>>>> socketoptions - and then Paul had a problem on some of his machines, >>>>>>>>>> and we backed it out? >>>>>>>>>> >>>>>>>>>> I wonder if that’s what is biting us here again, and what we need is >>>>>>>>>> to either remove the blocking send/recv’s altogether, or figure out >>>>>>>>>> a way to wait until the socket is really ready. >>>>>>>>>> >>>>>>>>>> Any thoughts? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On Oct 27, 2015, at 4:11 PM, George Bosilca <bosi...@icl.utk.edu> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> It appear the branch solve the problem at least partially. I asked >>>>>>>>>>> one of my students to hammer it pretty badly, and he reported that >>>>>>>>>>> the deadlocks still occur. He also graciously provided some >>>>>>>>>>> stacktraces: >>>>>>>>>>> >>>>>>>>>>> #0 0x00007f4bd5274aed in nanosleep () from /lib64/libc.so.6 >>>>>>>>>>> #1 0x00007f4bd52a9c94 in usleep () from /lib64/libc.so.6 >>>>>>>>>>> #2 0x00007f4bd2e42b00 in OPAL_PMIX_PMIX1XX_PMIx_Fence (procs=0x0, >>>>>>>>>>> nprocs=0, info=0x7fff3c561960, >>>>>>>>>>> ninfo=1) at src/client/pmix_client_fence.c:100 >>>>>>>>>>> #3 0x00007f4bd306e6d2 in pmix1_fence (procs=0x0, collect_data=1) >>>>>>>>>>> at pmix1_client.c:306 >>>>>>>>>>> #4 0x00007f4bd57d5cc3 in ompi_mpi_init (argc=3, >>>>>>>>>>> argv=0x7fff3c561ea8, requested=3, >>>>>>>>>>> provided=0x7fff3c561d84) at runtime/ompi_mpi_init.c:644 >>>>>>>>>>> #5 0x00007f4bd5813399 in PMPI_Init_thread (argc=0x7fff3c561d7c, >>>>>>>>>>> argv=0x7fff3c561d70, required=3, >>>>>>>>>>> provided=0x7fff3c561d84) at pinit_thread.c:69 >>>>>>>>>>> #6 0x0000000000401516 in main (argc=3, argv=0x7fff3c561ea8) at >>>>>>>>>>> osu_mbw_mr.c:86 >>>>>>>>>>> >>>>>>>>>>> And another process: >>>>>>>>>>> >>>>>>>>>>> #0 0x00007f7b9d7d8bdc in recv () from /lib64/libpthread.so.0 >>>>>>>>>>> #1 0x00007f7b9b0aa42d in >>>>>>>>>>> opal_pmix_pmix1xx_pmix_usock_recv_blocking (sd=13, >>>>>>>>>>> data=0x7ffd62139004 "", >>>>>>>>>>> size=4) at src/usock/usock.c:168 >>>>>>>>>>> #2 0x00007f7b9b0af5d9 in recv_connect_ack (sd=13) at >>>>>>>>>>> src/client/pmix_client.c:844 >>>>>>>>>>> #3 0x00007f7b9b0b085e in usock_connect (addr=0x7ffd62139330) at >>>>>>>>>>> src/client/pmix_client.c:1110 >>>>>>>>>>> #4 0x00007f7b9b0acc24 in connect_to_server >>>>>>>>>>> (address=0x7ffd62139330, cbdata=0x7ffd621390e0) >>>>>>>>>>> at src/client/pmix_client.c:181 >>>>>>>>>>> #5 0x00007f7b9b0ad569 in OPAL_PMIX_PMIX1XX_PMIx_Init >>>>>>>>>>> (proc=0x7f7b9b4e9b60) >>>>>>>>>>> at src/client/pmix_client.c:362 >>>>>>>>>>> #6 0x00007f7b9b2dbd9d in pmix1_client_init () at pmix1_client.c:99 >>>>>>>>>>> #7 0x00007f7b9b4eb95f in pmi_component_query >>>>>>>>>>> (module=0x7ffd62139490, priority=0x7ffd6213948c) >>>>>>>>>>> at ess_pmi_component.c:90 >>>>>>>>>>> #8 0x00007f7b9ce70ec5 in mca_base_select (type_name=0x7f7b9d20e059 >>>>>>>>>>> "ess", output_id=-1, >>>>>>>>>>> components_available=0x7f7b9d431eb0, >>>>>>>>>>> best_module=0x7ffd621394d0, best_component=0x7ffd621394d8, >>>>>>>>>>> priority_out=0x0) at mca_base_components_select.c:77 >>>>>>>>>>> #9 0x00007f7b9d1a956b in orte_ess_base_select () at >>>>>>>>>>> base/ess_base_select.c:40 >>>>>>>>>>> #10 0x00007f7b9d160449 in orte_init (pargc=0x0, pargv=0x0, >>>>>>>>>>> flags=32) at runtime/orte_init.c:219 >>>>>>>>>>> #11 0x00007f7b9da4377a in ompi_mpi_init (argc=3, >>>>>>>>>>> argv=0x7ffd621397f8, requested=3, >>>>>>>>>>> provided=0x7ffd621396d4) at runtime/ompi_mpi_init.c:488 >>>>>>>>>>> #12 0x00007f7b9da81399 in PMPI_Init_thread (argc=0x7ffd621396cc, >>>>>>>>>>> argv=0x7ffd621396c0, required=3, >>>>>>>>>>> provided=0x7ffd621396d4) at pinit_thread.c:69 >>>>>>>>>>> #13 0x0000000000401516 in main (argc=3, argv=0x7ffd621397f8) at >>>>>>>>>>> osu_mbw_mr.c:86 >>>>>>>>>>> >>>>>>>>>>> George. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Oct 27, 2015 at 2:36 PM, Ralph Castain <r...@open-mpi.org> >>>>>>>>>>> wrote: >>>>>>>>>>> I haven’t been able to replicate this when using the branch in this >>>>>>>>>>> PR: >>>>>>>>>>> >>>>>>>>>>> https://github.com/open-mpi/ompi/pull/1073 >>>>>>>>>>> >>>>>>>>>>> Would you mind giving it a try? It fixes some other race conditions >>>>>>>>>>> and might pick this one up too. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> On Oct 27, 2015, at 10:04 AM, Ralph Castain <r...@open-mpi.org> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Okay, I’ll take a look - I’ve been chasing a race condition that >>>>>>>>>>>> might be related >>>>>>>>>>>> >>>>>>>>>>>>> On Oct 27, 2015, at 9:54 AM, George Bosilca <bosi...@icl.utk.edu> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> No, it's using 2 nodes. >>>>>>>>>>>>> George. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Oct 27, 2015 at 12:35 PM, Ralph Castain >>>>>>>>>>>>> <r...@open-mpi.org> wrote: >>>>>>>>>>>>> Is this on a single node? >>>>>>>>>>>>> >>>>>>>>>>>>>> On Oct 27, 2015, at 9:25 AM, George Bosilca >>>>>>>>>>>>>> <bosi...@icl.utk.edu> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> I get intermittent deadlocks wit the latest trunk. The smallest >>>>>>>>>>>>>> reproducer is a shell for loop around a small (2 processes) >>>>>>>>>>>>>> short (20 seconds) MPI application. After few tens of iterations >>>>>>>>>>>>>> the MPI_Init will deadlock with the following backtrace: >>>>>>>>>>>>>> >>>>>>>>>>>>>> #0 0x00007fa94b5d9aed in nanosleep () from /lib64/libc.so.6 >>>>>>>>>>>>>> #1 0x00007fa94b60ec94 in usleep () from /lib64/libc.so.6 >>>>>>>>>>>>>> #2 0x00007fa94960ba08 in OPAL_PMIX_PMIX1XX_PMIx_Fence >>>>>>>>>>>>>> (procs=0x0, nprocs=0, info=0x7ffd7934fb90, >>>>>>>>>>>>>> ninfo=1) at src/client/pmix_client_fence.c:100 >>>>>>>>>>>>>> #3 0x00007fa9498376a2 in pmix1_fence (procs=0x0, >>>>>>>>>>>>>> collect_data=1) at pmix1_client.c:305 >>>>>>>>>>>>>> #4 0x00007fa94bb39ba4 in ompi_mpi_init (argc=3, >>>>>>>>>>>>>> argv=0x7ffd793500a8, requested=3, >>>>>>>>>>>>>> provided=0x7ffd7934ff94) at runtime/ompi_mpi_init.c:645 >>>>>>>>>>>>>> #5 0x00007fa94bb77281 in PMPI_Init_thread (argc=0x7ffd7934ff8c, >>>>>>>>>>>>>> argv=0x7ffd7934ff80, required=3, >>>>>>>>>>>>>> provided=0x7ffd7934ff94) at pinit_thread.c:69 >>>>>>>>>>>>>> #6 0x000000000040150f in main (argc=3, argv=0x7ffd793500a8) at >>>>>>>>>>>>>> osu_mbw_mr.c:86 >>>>>>>>>>>>>> >>>>>>>>>>>>>> On my machines this is reproducible at 100% after anywhere >>>>>>>>>>>>>> between 50 and 100 iterations. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> George. >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> devel mailing list >>>>>>>>>>>>>> de...@open-mpi.org >>>>>>>>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>>>>>>>>>> Link to this post: >>>>>>>>>>>>>> http://www.open-mpi.org/community/lists/devel/2015/10/18280.php >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> devel mailing list >>>>>>>>>>>>> de...@open-mpi.org >>>>>>>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>>>>>>>>> Link to this post: >>>>>>>>>>>>> http://www.open-mpi.org/community/lists/devel/2015/10/18281.php >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> devel mailing list >>>>>>>>>>>>> de...@open-mpi.org >>>>>>>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>>>>>>>>> Link to this post: >>>>>>>>>>>>> http://www.open-mpi.org/community/lists/devel/2015/10/18282.php >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> devel mailing list >>>>>>>>>>> de...@open-mpi.org >>>>>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>>>>>>> Link to this post: >>>>>>>>>>> http://www.open-mpi.org/community/lists/devel/2015/10/18284.php >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> devel mailing list >>>>>>>>>>> de...@open-mpi.org >>>>>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>>>>>>> Link to this post: >>>>>>>>>>> http://www.open-mpi.org/community/lists/devel/2015/10/18292.php >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> devel mailing list >>>>>>>>>> de...@open-mpi.org >>>>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>>>>>> Link to this post: >>>>>>>>>> http://www.open-mpi.org/community/lists/devel/2015/10/18294.php >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> devel mailing list >>>>>>>>>> de...@open-mpi.org >>>>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>>>>>> Link to this post: >>>>>>>>>> http://www.open-mpi.org/community/lists/devel/2015/10/18302.php >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> devel mailing list >>>>>> de...@open-mpi.org >>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>> Link to this post: >>>>>> http://www.open-mpi.org/community/lists/devel/2015/10/18309.php >>>>>> >>>>>> _______________________________________________ >>>>>> devel mailing list >>>>>> de...@open-mpi.org >>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>> Link to this post: >>>>>> http://www.open-mpi.org/community/lists/devel/2015/10/18320.php >>>>> >>>> >>>> >>>> _______________________________________________ >>>> devel mailing list >>>> de...@open-mpi.org >>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>> Link to this post: >>>> http://www.open-mpi.org/community/lists/devel/2015/10/18323.php >>>> >>>> >>>> >>>> -- >>>> С Уважением, Поляков Артем Юрьевич >>>> Best regards, Artem Y. Polyakov >>>> _______________________________________________ >>>> devel mailing list >>>> de...@open-mpi.org >>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>> Link to this post: >>>> http://www.open-mpi.org/community/lists/devel/2015/11/18334.php >>> >>> >>> _______________________________________________ >>> devel mailing list >>> de...@open-mpi.org >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>> Link to this post: >>> http://www.open-mpi.org/community/lists/devel/2015/11/18335.php >>> >>> _______________________________________________ >>> devel mailing list >>> de...@open-mpi.org >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>> Link to this post: >>> http://www.open-mpi.org/community/lists/devel/2015/11/18336.php >> >> >> _______________________________________________ >> devel mailing list >> de...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >> Link to this post: >> http://www.open-mpi.org/community/lists/devel/2015/11/18337.php >> >> >> _______________________________________________ >> devel mailing list >> de...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >> Link to this post: >> http://www.open-mpi.org/community/lists/devel/2015/11/18340.php >> >> >> _______________________________________________ >> devel mailing list >> de...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >> Link to this post: >> http://www.open-mpi.org/community/lists/devel/2015/11/18341.php >> >> >> >> -- >> С Уважением, Поляков Артем Юрьевич >> Best regards, Artem Y. Polyakov >> >> >> >> -- >> С Уважением, Поляков Артем Юрьевич >> Best regards, Artem Y. Polyakov >> >> _______________________________________________ >> devel mailing list >> de...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >> Link to this post: >> http://www.open-mpi.org/community/lists/devel/2015/11/18345.php >> >> _______________________________________________ >> devel mailing list >> de...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >> Link to this post: >> http://www.open-mpi.org/community/lists/devel/2015/11/18346.php > > > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2015/11/18347.php > > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2015/11/18348.php
