All 10k tests completed successfully. Nysal pinpointed the real problem behind the deadlocks. :+1:
George. On Mon, Nov 9, 2015 at 1:13 PM, Ralph Castain <r...@open-mpi.org> wrote: > Looking at it, I think I see what was happening. The thread would start, > but then immediately see that the active flag was false and would exit. > This left the server without any listening thread - but it wouldn’t detect > this had happened. It was therefore a race between whether the thread > checked the flag before the server set it. > > Thanks Nysal - I believe this should indeed fix the problem! > > > On Nov 9, 2015, at 9:04 AM, George Bosilca <bosi...@icl.utk.edu> wrote: > > Clearly Nyal got a valid point there. I launched a stress test with Nysal > suggestion in the code, and so far it's up to few hundreds iterations > without deadlock. I would not claim victory yet, I launched a 10k cycle to > see where we stand (btw this never passed before). > I'll let you know the outcome. > > George. > > > On Mon, Nov 9, 2015 at 11:55 AM, Artem Polyakov <artpo...@gmail.com> > wrote: > >> >> >> 2015-11-09 22:42 GMT+06:00 Artem Polyakov <artpo...@gmail.com>: >> >>> This is the very good point, Nysal! >>> >>> This is definitely a problem and I can say even more: avg. 3 from every >>> 10 tasks was affected by this bug. Once the PR ( >>> https://github.com/pmix/master/pull/8) was applied I was able to run >>> 100 testing tasks without any hangs. >>> >>> Here some more information on my symptoms. I was observing this without >>> OMPI, just running pmix_client test binary from PMIx test suite with SLURM >>> PMIx plugin. >>> Periodicaly application was hanging. Investigation shows that not all >>> processes are able to initialize correctly. >>> Here is how such client's backtrace looks like: >>> >> >> P.S. I think that this backtrace may be relevant to George's problem as >> well. In my case not all of the processes was hanging in the >> connect_to_server, most of them were able to move forward and reach Fence. >> George, the backtrace that you've posted was the same on both processes >> or it was the "random" one from one of them? >> >> >>> (gdb) bt >>> #0 0x00007f1448f1b7eb in recv () from >>> /lib/x86_64-linux-gnu/libpthread.so.0 >>> #1 0x00007f144914c191 in pmix_usock_recv_blocking (sd=9, >>> data=0x7fff367f7c64 "", size=4) at src/usock/usock.c:166 >>> #2 0x00007f1449152d18 in recv_connect_ack (sd=9) at >>> src/client/pmix_client.c:837 >>> #3 0x00007f14491546bf in usock_connect (addr=0x7fff367f7d60) at >>> src/client/pmix_client.c:1103 >>> #4 0x00007f144914f94c in connect_to_server (address=0x7fff367f7d60, >>> cbdata=0x7fff367f7dd0) at src/client/pmix_client.c:179 >>> #5 0x00007f1449150421 in PMIx_Init (proc=0x7fff367f81d0) at >>> src/client/pmix_client.c:355 >>> #6 0x0000000000401b97 in main (argc=9, argv=0x7fff367f83d8) at >>> pmix_client.c:62 >>> >>> >>> The server-side debug has the following lines at the end of the file: >>> [cn33:00482] pmix:server register client slurm.pmix.22.0:10 >>> [cn33:00482] pmix:server _register_client for nspace slurm.pmix.22.0 >>> rank 10 >>> [cn33:00482] pmix:server setup_fork for nspace slurm.pmix.22.0 rank 10 >>> >>> in normal operation the following lines should appear after lines above: >>> .... >>> [cn33:00188] listen_thread: new connection: (26, 0) >>> [cn33:00188] connection_handler: new connection: 26 >>> [cn33:00188] RECV CONNECT ACK FROM PEER ON SOCKET 26 >>> [cn33:00188] waiting for blocking recv of 16 bytes >>> [cn33:00188] blocking receive complete from remote >>> .... >>> >>> At the client side I see the following lines >>> cn33:00491] usock_peer_try_connect: attempting to connect to server >>> [cn33:00491] usock_peer_try_connect: attempting to connect to server on >>> socket 10 >>> [cn33:00491] pmix: SEND CONNECT ACK >>> [cn33:00491] sec: native create_cred >>> [cn33:00491] sec: using credential 1000:1000 >>> [cn33:00491] send blocking of 54 bytes to socket 10 >>> [cn33:00491] blocking send complete to socket 10 >>> [cn33:00491] pmix: RECV CONNECT ACK FROM SERVER >>> [cn33:00491] waiting for blocking recv of 4 bytes >>> [cn33:00491] blocking_recv received error 11:Resource temporarily >>> unavailable from remote - cycling >>> [cn33:00491] blocking_recv received error 11:Resource temporarily >>> unavailable from remote - cycling >>> [... repeated many times ...] >>> >>> With the fix for the problem highlighted by Nysal all runs cleanly. >>> >>> >>> 2015-11-09 10:53 GMT+06:00 Nysal Jan K A <jny...@gmail.com>: >>> >>>> In listen_thread(): >>>> 194 while (pmix_server_globals.listen_thread_active) { >>>> 195 FD_ZERO(&readfds); >>>> 196 FD_SET(pmix_server_globals.listen_socket, &readfds); >>>> 197 max = pmix_server_globals.listen_socket; >>>> >>>> Is it possible that pmix_server_globals.listen_thread_active can be >>>> false, in which case the thread just exits and will never call accept() ? >>>> >>>> In pmix_start_listening(): >>>> 147 /* fork off the listener thread */ >>>> 148 if (0 > pthread_create(&engine, NULL, listen_thread, NULL)) >>>> { >>>> 149 return PMIX_ERROR; >>>> 150 } >>>> 151 pmix_server_globals.listen_thread_active = true; >>>> >>>> pmix_server_globals.listen_thread_active is set to true after the >>>> thread is created, could this cause a race ? >>>> listen_thread_active might also need to be declared as volatile. >>>> >>>> Regards >>>> --Nysal >>>> >>>> On Sun, Nov 8, 2015 at 10:38 PM, George Bosilca <bosi...@icl.utk.edu> >>>> wrote: >>>> >>>>> We had a power outage last week and the local disks on our cluster >>>>> were wiped out. My tester was in there. But, I can rewrite it after SC. >>>>> >>>>> George. >>>>> >>>>> On Sat, Nov 7, 2015 at 12:04 PM, Ralph Castain <r...@open-mpi.org> >>>>> wrote: >>>>> >>>>>> Could you send me your stress test? I’m wondering if it is just >>>>>> something about how we set socket options >>>>>> >>>>>> >>>>>> On Nov 7, 2015, at 8:58 AM, George Bosilca <bosi...@icl.utk.edu> >>>>>> wrote: >>>>>> >>>>>> I has to postpone this until after SC. However, I ran for 3 days a >>>>>> stress test of UDS reproducing the opening and sending of data (what >>>>>> Ralph >>>>>> said in his email) and I never could get a deadlock. >>>>>> >>>>>> George. >>>>>> >>>>>> >>>>>> On Sat, Nov 7, 2015 at 11:26 AM, Ralph Castain <r...@open-mpi.org> >>>>>> wrote: >>>>>> >>>>>>> George was looking into it, but I don’t know if he has had time >>>>>>> recently to continue the investigation. We understand “what” is >>>>>>> happening >>>>>>> (accept sometimes ignores the connection), but we don’t yet know “why”. >>>>>>> I’ve done some digging around the web, and found that sometimes you can >>>>>>> try >>>>>>> to talk to a Unix Domain Socket too quickly - i.e., you open it and then >>>>>>> send to it, but the OS hasn’t yet set it up. In those cases, you can >>>>>>> hang >>>>>>> the socket. However, I’ve tried adding some artificial delay, and while >>>>>>> it >>>>>>> helped, it didn’t completely solve the problem. >>>>>>> >>>>>>> I have an idea for a workaround (set a timer and retry after >>>>>>> awhile), but would obviously prefer a real solution. I’m not even sure >>>>>>> it >>>>>>> will work as it is unclear that the server (who is the one hung in >>>>>>> accept) >>>>>>> will break free if the client closes the socket and retries. >>>>>>> >>>>>>> >>>>>>> On Nov 6, 2015, at 10:53 PM, Artem Polyakov <artpo...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>> Hello, is there any progress on this topic? This affects our PMIx >>>>>>> measurements. >>>>>>> >>>>>>> 2015-10-30 21:21 GMT+06:00 Ralph Castain <r...@open-mpi.org>: >>>>>>> >>>>>>>> I’ve verified that the orte/util/listener thread is not being >>>>>>>> started, so I don’t think it should be involved in this problem. >>>>>>>> >>>>>>>> HTH >>>>>>>> Ralph >>>>>>>> >>>>>>>> On Oct 30, 2015, at 8:07 AM, Ralph Castain <r...@open-mpi.org> >>>>>>>> wrote: >>>>>>>> >>>>>>>> Hmmm…there is a hook that would allow the PMIx server to utilize >>>>>>>> that listener thread, but we aren’t currently using it. Each daemon >>>>>>>> plus >>>>>>>> mpirun will call orte_start_listener, but nothing is currently >>>>>>>> registering >>>>>>>> and so the listener in that code is supposed to just return without >>>>>>>> starting the thread. >>>>>>>> >>>>>>>> So the only listener thread that should exist is the one inside the >>>>>>>> PMIx server itself. If something else is happening, then that would be >>>>>>>> a >>>>>>>> bug. I can look at the orte listener code to ensure that the thread >>>>>>>> isn’t >>>>>>>> incorrectly starting. >>>>>>>> >>>>>>>> >>>>>>>> On Oct 29, 2015, at 10:03 PM, George Bosilca <bosi...@icl.utk.edu> >>>>>>>> wrote: >>>>>>>> >>>>>>>> Some progress, that puzzles me but might help you understand. Once >>>>>>>> the deadlock appears, if I manually kill the MPI process on the node >>>>>>>> where >>>>>>>> the deadlock was created, the local orte daemon doesn't notice and will >>>>>>>> just keep waiting. >>>>>>>> >>>>>>>> Quick question: I am under the impression that the issue is not in >>>>>>>> the PMIX server but somewhere around the listener_thread_fn in >>>>>>>> orte/util/listener.c. Possible ? >>>>>>>> >>>>>>>> George. >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Oct 28, 2015 at 3:56 AM, Ralph Castain <r...@open-mpi.org> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Should have also clarified: the prior fixes are indeed in the >>>>>>>>> current master. >>>>>>>>> >>>>>>>>> On Oct 28, 2015, at 12:42 AM, Ralph Castain <r...@open-mpi.org> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Nope - I was wrong. The correction on the client side consisted of >>>>>>>>> attempting to timeout if the blocking recv failed. We then modified >>>>>>>>> the >>>>>>>>> blocking send/recv so they would handle errors. >>>>>>>>> >>>>>>>>> So that problem occurred -after- the server had correctly called >>>>>>>>> accept. The listener code is in >>>>>>>>> opal/mca/pmix/pmix1xx/pmix/src/server/pmix_server_listener.c >>>>>>>>> >>>>>>>>> It looks to me like the only way we could drop the accept >>>>>>>>> (assuming the OS doesn’t lose it) is if the file descriptor lies >>>>>>>>> outside >>>>>>>>> the expected range once we fall out of select: >>>>>>>>> >>>>>>>>> >>>>>>>>> /* Spin accepting connections until all active listen >>>>>>>>> sockets >>>>>>>>> * do not have any incoming connections, pushing each >>>>>>>>> connection >>>>>>>>> * onto the event queue for processing >>>>>>>>> */ >>>>>>>>> do { >>>>>>>>> accepted_connections = 0; >>>>>>>>> /* according to the man pages, select replaces the >>>>>>>>> given descriptor >>>>>>>>> * set with a subset consisting of those descriptors >>>>>>>>> that are ready >>>>>>>>> * for the specified operation - in this case, a read. >>>>>>>>> So we need to >>>>>>>>> * first check to see if this file descriptor is >>>>>>>>> included in the >>>>>>>>> * returned subset >>>>>>>>> */ >>>>>>>>> if (0 == FD_ISSET(pmix_server_globals.listen_socket, >>>>>>>>> &readfds)) { >>>>>>>>> /* this descriptor is not included */ >>>>>>>>> continue; >>>>>>>>> } >>>>>>>>> >>>>>>>>> /* this descriptor is ready to be read, which means a >>>>>>>>> connection >>>>>>>>> * request has been received - so harvest it. All we >>>>>>>>> want to do >>>>>>>>> * here is accept the connection and push the info >>>>>>>>> onto the event >>>>>>>>> * library for subsequent processing - we don't want >>>>>>>>> to actually >>>>>>>>> * process the connection here as it takes too long, >>>>>>>>> and so the >>>>>>>>> * OS might start rejecting connections due to timeout. >>>>>>>>> */ >>>>>>>>> pending_connection = >>>>>>>>> PMIX_NEW(pmix_pending_connection_t); >>>>>>>>> event_assign(&pending_connection->ev, >>>>>>>>> pmix_globals.evbase, -1, >>>>>>>>> EV_WRITE, connection_handler, >>>>>>>>> pending_connection); >>>>>>>>> pending_connection->sd = >>>>>>>>> accept(pmix_server_globals.listen_socket, >>>>>>>>> (struct >>>>>>>>> sockaddr*)&(pending_connection->addr), >>>>>>>>> &addrlen); >>>>>>>>> if (pending_connection->sd < 0) { >>>>>>>>> PMIX_RELEASE(pending_connection); >>>>>>>>> if (pmix_socket_errno != EAGAIN || >>>>>>>>> pmix_socket_errno != EWOULDBLOCK) { >>>>>>>>> if (EMFILE == pmix_socket_errno) { >>>>>>>>> PMIX_ERROR_LOG(PMIX_ERR_OUT_OF_RESOURCE); >>>>>>>>> } else { >>>>>>>>> pmix_output(0, "listen_thread: accept() >>>>>>>>> failed: %s (%d).", >>>>>>>>> strerror(pmix_socket_errno), >>>>>>>>> pmix_socket_errno); >>>>>>>>> } >>>>>>>>> goto done; >>>>>>>>> } >>>>>>>>> continue; >>>>>>>>> } >>>>>>>>> >>>>>>>>> pmix_output_verbose(8, pmix_globals.debug_output, >>>>>>>>> "listen_thread: new connection: >>>>>>>>> (%d, %d)", >>>>>>>>> pending_connection->sd, >>>>>>>>> pmix_socket_errno); >>>>>>>>> /* activate the event */ >>>>>>>>> event_active(&pending_connection->ev, EV_WRITE, 1); >>>>>>>>> accepted_connections++; >>>>>>>>> } while (accepted_connections > 0); >>>>>>>>> >>>>>>>>> >>>>>>>>> On Oct 28, 2015, at 12:25 AM, Ralph Castain <r...@open-mpi.org> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Looking at the code, it appears that a fix was committed for this >>>>>>>>> problem, and that we correctly resolved the issue found by Paul. The >>>>>>>>> problem is that the fix didn’t get upstreamed, and so it was lost the >>>>>>>>> next >>>>>>>>> time we refreshed PMIx. Sigh. >>>>>>>>> >>>>>>>>> Let me try to recreate the fix and have you take a gander at it. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Oct 28, 2015, at 12:22 AM, Ralph Castain <r...@open-mpi.org> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Here is the discussion - afraid it is fairly lengthy. Ignore the >>>>>>>>> hwloc references in it as that was a separate issue: >>>>>>>>> >>>>>>>>> http://www.open-mpi.org/community/lists/devel/2015/09/18074.php >>>>>>>>> >>>>>>>>> It definitely sounds like the same issue creeping in again. I’d >>>>>>>>> appreciate any thoughts on how to correct it. If it helps, you could >>>>>>>>> look >>>>>>>>> at the PMIx master - there are standalone tests in the test/simple >>>>>>>>> directory that fork/exec a child and just do the connection. >>>>>>>>> >>>>>>>>> https://github.com/pmix/master >>>>>>>>> >>>>>>>>> The test server is simptest.c - it will spawn a single copy of >>>>>>>>> simpclient.c by default. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Oct 27, 2015, at 10:14 PM, George Bosilca <bosi...@icl.utk.edu> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Interesting. Do you have a pointer to the commit (or/and to the >>>>>>>>> discussion)? >>>>>>>>> >>>>>>>>> I looked at the PMIX code, and I have identified few issues, but >>>>>>>>> unfortunately none of them seem to fix the problem for good. However, >>>>>>>>> now I >>>>>>>>> need more than 1000 runs to get a deadlock (instead of few tens). >>>>>>>>> >>>>>>>>> Looking with "netstat -ax" at the status of the UDS while the >>>>>>>>> processes are deadlocked, I see 2 UDS with the same name: one from the >>>>>>>>> server which is in LISTEN state, and one for the client which is >>>>>>>>> being in >>>>>>>>> CONNECTING state (while the client already sent a message in the >>>>>>>>> socket and >>>>>>>>> is now waiting in a blocking receive). This somehow suggest that the >>>>>>>>> server >>>>>>>>> has not yet called accept on the UDS. Unfortunately, there are 3 >>>>>>>>> threads >>>>>>>>> all doing different flavors of even_base and select, so I have a hard >>>>>>>>> time >>>>>>>>> tracking the path of the UDS on the server side. >>>>>>>>> >>>>>>>>> So in order to validate my assumption I wrote a minimalistic UDS >>>>>>>>> client and server application and tried different scenarios. The >>>>>>>>> conclusion >>>>>>>>> is that in order to see the same type of output from "netstat -ax" I >>>>>>>>> have >>>>>>>>> to call listen on the server, connect on the client and do not call >>>>>>>>> accept >>>>>>>>> on the server. >>>>>>>>> >>>>>>>>> With the same occasion I also confirmed that the UDS are holding >>>>>>>>> the data sent so there is no need for further synchronization for the >>>>>>>>> case >>>>>>>>> where the data is sent first. We only need to find out how the server >>>>>>>>> forgets to call accept. >>>>>>>>> >>>>>>>>> George. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Oct 27, 2015 at 7:52 PM, Ralph Castain <r...@open-mpi.org> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hmmm…this looks like it might be that problem we previously saw >>>>>>>>>> where the blocking recv hangs in a proc when the blocking send tries >>>>>>>>>> to >>>>>>>>>> send before the domain socket is actually ready, and so the send >>>>>>>>>> fails on >>>>>>>>>> the other end. As I recall, it was something to do with the >>>>>>>>>> socketoptions - >>>>>>>>>> and then Paul had a problem on some of his machines, and we backed >>>>>>>>>> it out? >>>>>>>>>> >>>>>>>>>> I wonder if that’s what is biting us here again, and what we need >>>>>>>>>> is to either remove the blocking send/recv’s altogether, or figure >>>>>>>>>> out a >>>>>>>>>> way to wait until the socket is really ready. >>>>>>>>>> >>>>>>>>>> Any thoughts? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Oct 27, 2015, at 4:11 PM, George Bosilca <bosi...@icl.utk.edu> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> It appear the branch solve the problem at least partially. I >>>>>>>>>> asked one of my students to hammer it pretty badly, and he reported >>>>>>>>>> that >>>>>>>>>> the deadlocks still occur. He also graciously provided some >>>>>>>>>> stacktraces: >>>>>>>>>> >>>>>>>>>> #0 0x00007f4bd5274aed in nanosleep () from /lib64/libc.so.6 >>>>>>>>>> #1 0x00007f4bd52a9c94 in usleep () from /lib64/libc.so.6 >>>>>>>>>> #2 0x00007f4bd2e42b00 in OPAL_PMIX_PMIX1XX_PMIx_Fence >>>>>>>>>> (procs=0x0, nprocs=0, info=0x7fff3c561960, >>>>>>>>>> ninfo=1) at src/client/pmix_client_fence.c:100 >>>>>>>>>> #3 0x00007f4bd306e6d2 in pmix1_fence (procs=0x0, collect_data=1) >>>>>>>>>> at pmix1_client.c:306 >>>>>>>>>> #4 0x00007f4bd57d5cc3 in ompi_mpi_init (argc=3, >>>>>>>>>> argv=0x7fff3c561ea8, requested=3, >>>>>>>>>> provided=0x7fff3c561d84) at runtime/ompi_mpi_init.c:644 >>>>>>>>>> #5 0x00007f4bd5813399 in PMPI_Init_thread (argc=0x7fff3c561d7c, >>>>>>>>>> argv=0x7fff3c561d70, required=3, >>>>>>>>>> provided=0x7fff3c561d84) at pinit_thread.c:69 >>>>>>>>>> #6 0x0000000000401516 in main (argc=3, argv=0x7fff3c561ea8) at >>>>>>>>>> osu_mbw_mr.c:86 >>>>>>>>>> >>>>>>>>>> And another process: >>>>>>>>>> >>>>>>>>>> #0 0x00007f7b9d7d8bdc in recv () from /lib64/libpthread.so.0 >>>>>>>>>> #1 0x00007f7b9b0aa42d in >>>>>>>>>> opal_pmix_pmix1xx_pmix_usock_recv_blocking (sd=13, >>>>>>>>>> data=0x7ffd62139004 "", >>>>>>>>>> size=4) at src/usock/usock.c:168 >>>>>>>>>> #2 0x00007f7b9b0af5d9 in recv_connect_ack (sd=13) at >>>>>>>>>> src/client/pmix_client.c:844 >>>>>>>>>> #3 0x00007f7b9b0b085e in usock_connect (addr=0x7ffd62139330) at >>>>>>>>>> src/client/pmix_client.c:1110 >>>>>>>>>> #4 0x00007f7b9b0acc24 in connect_to_server >>>>>>>>>> (address=0x7ffd62139330, cbdata=0x7ffd621390e0) >>>>>>>>>> at src/client/pmix_client.c:181 >>>>>>>>>> #5 0x00007f7b9b0ad569 in OPAL_PMIX_PMIX1XX_PMIx_Init >>>>>>>>>> (proc=0x7f7b9b4e9b60) >>>>>>>>>> at src/client/pmix_client.c:362 >>>>>>>>>> #6 0x00007f7b9b2dbd9d in pmix1_client_init () at >>>>>>>>>> pmix1_client.c:99 >>>>>>>>>> #7 0x00007f7b9b4eb95f in pmi_component_query >>>>>>>>>> (module=0x7ffd62139490, priority=0x7ffd6213948c) >>>>>>>>>> at ess_pmi_component.c:90 >>>>>>>>>> #8 0x00007f7b9ce70ec5 in mca_base_select >>>>>>>>>> (type_name=0x7f7b9d20e059 "ess", output_id=-1, >>>>>>>>>> components_available=0x7f7b9d431eb0, >>>>>>>>>> best_module=0x7ffd621394d0, best_component=0x7ffd621394d8, >>>>>>>>>> priority_out=0x0) at mca_base_components_select.c:77 >>>>>>>>>> #9 0x00007f7b9d1a956b in orte_ess_base_select () at >>>>>>>>>> base/ess_base_select.c:40 >>>>>>>>>> #10 0x00007f7b9d160449 in orte_init (pargc=0x0, pargv=0x0, >>>>>>>>>> flags=32) at runtime/orte_init.c:219 >>>>>>>>>> #11 0x00007f7b9da4377a in ompi_mpi_init (argc=3, >>>>>>>>>> argv=0x7ffd621397f8, requested=3, >>>>>>>>>> provided=0x7ffd621396d4) at runtime/ompi_mpi_init.c:488 >>>>>>>>>> #12 0x00007f7b9da81399 in PMPI_Init_thread (argc=0x7ffd621396cc, >>>>>>>>>> argv=0x7ffd621396c0, required=3, >>>>>>>>>> provided=0x7ffd621396d4) at pinit_thread.c:69 >>>>>>>>>> #13 0x0000000000401516 in main (argc=3, argv=0x7ffd621397f8) at >>>>>>>>>> osu_mbw_mr.c:86 >>>>>>>>>> >>>>>>>>>> George. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Oct 27, 2015 at 2:36 PM, Ralph Castain <r...@open-mpi.org> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> I haven’t been able to replicate this when using the branch in >>>>>>>>>>> this PR: >>>>>>>>>>> >>>>>>>>>>> https://github.com/open-mpi/ompi/pull/1073 >>>>>>>>>>> >>>>>>>>>>> Would you mind giving it a try? It fixes some other race >>>>>>>>>>> conditions and might pick this one up too. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Oct 27, 2015, at 10:04 AM, Ralph Castain <r...@open-mpi.org> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Okay, I’ll take a look - I’ve been chasing a race condition that >>>>>>>>>>> might be related >>>>>>>>>>> >>>>>>>>>>> On Oct 27, 2015, at 9:54 AM, George Bosilca <bosi...@icl.utk.edu> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> No, it's using 2 nodes. >>>>>>>>>>> George. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Oct 27, 2015 at 12:35 PM, Ralph Castain < >>>>>>>>>>> r...@open-mpi.org> wrote: >>>>>>>>>>> >>>>>>>>>>>> Is this on a single node? >>>>>>>>>>>> >>>>>>>>>>>> On Oct 27, 2015, at 9:25 AM, George Bosilca < >>>>>>>>>>>> bosi...@icl.utk.edu> wrote: >>>>>>>>>>>> >>>>>>>>>>>> I get intermittent deadlocks wit the latest trunk. The smallest >>>>>>>>>>>> reproducer is a shell for loop around a small (2 processes) short >>>>>>>>>>>> (20 >>>>>>>>>>>> seconds) MPI application. After few tens of iterations the >>>>>>>>>>>> MPI_Init will >>>>>>>>>>>> deadlock with the following backtrace: >>>>>>>>>>>> >>>>>>>>>>>> #0 0x00007fa94b5d9aed in nanosleep () from /lib64/libc.so.6 >>>>>>>>>>>> #1 0x00007fa94b60ec94 in usleep () from /lib64/libc.so.6 >>>>>>>>>>>> #2 0x00007fa94960ba08 in OPAL_PMIX_PMIX1XX_PMIx_Fence >>>>>>>>>>>> (procs=0x0, nprocs=0, info=0x7ffd7934fb90, >>>>>>>>>>>> ninfo=1) at src/client/pmix_client_fence.c:100 >>>>>>>>>>>> #3 0x00007fa9498376a2 in pmix1_fence (procs=0x0, >>>>>>>>>>>> collect_data=1) at pmix1_client.c:305 >>>>>>>>>>>> #4 0x00007fa94bb39ba4 in ompi_mpi_init (argc=3, >>>>>>>>>>>> argv=0x7ffd793500a8, requested=3, >>>>>>>>>>>> provided=0x7ffd7934ff94) at runtime/ompi_mpi_init.c:645 >>>>>>>>>>>> #5 0x00007fa94bb77281 in PMPI_Init_thread >>>>>>>>>>>> (argc=0x7ffd7934ff8c, argv=0x7ffd7934ff80, required=3, >>>>>>>>>>>> provided=0x7ffd7934ff94) at pinit_thread.c:69 >>>>>>>>>>>> #6 0x000000000040150f in main (argc=3, argv=0x7ffd793500a8) at >>>>>>>>>>>> osu_mbw_mr.c:86 >>>>>>>>>>>> >>>>>>>>>>>> On my machines this is reproducible at 100% after anywhere >>>>>>>>>>>> between 50 and 100 iterations. >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> George. >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> devel mailing list >>>>>>>>>>>> de...@open-mpi.org >>>>>>>>>>>> Subscription: >>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>>>>>>>> Link to this post: >>>>>>>>>>>> http://www.open-mpi.org/community/lists/devel/2015/10/18280.php >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> devel mailing list >>>>>>>>>>>> de...@open-mpi.org >>>>>>>>>>>> Subscription: >>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>>>>>>>> Link to this post: >>>>>>>>>>>> http://www.open-mpi.org/community/lists/devel/2015/10/18281.php >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> devel mailing list >>>>>>>>>>> de...@open-mpi.org >>>>>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>>>>>>> Link to this post: >>>>>>>>>>> http://www.open-mpi.org/community/lists/devel/2015/10/18282.php >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> devel mailing list >>>>>>>>>>> de...@open-mpi.org >>>>>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>>>>>>> Link to this post: >>>>>>>>>>> http://www.open-mpi.org/community/lists/devel/2015/10/18284.php >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> devel mailing list >>>>>>>>>> de...@open-mpi.org >>>>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>>>>>> Link to this post: >>>>>>>>>> http://www.open-mpi.org/community/lists/devel/2015/10/18292.php >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> devel mailing list >>>>>>>>>> de...@open-mpi.org >>>>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>>>>>> Link to this post: >>>>>>>>>> http://www.open-mpi.org/community/lists/devel/2015/10/18294.php >>>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> devel mailing list >>>>>>>>> de...@open-mpi.org >>>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>>>>> Link to this post: >>>>>>>>> http://www.open-mpi.org/community/lists/devel/2015/10/18302.php >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> devel mailing list >>>>>>>>> de...@open-mpi.org >>>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>>>>> Link to this post: >>>>>>>>> http://www.open-mpi.org/community/lists/devel/2015/10/18309.php >>>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> devel mailing list >>>>>>>> de...@open-mpi.org >>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>>>> Link to this post: >>>>>>>> http://www.open-mpi.org/community/lists/devel/2015/10/18320.php >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> devel mailing list >>>>>>>> de...@open-mpi.org >>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>>>> Link to this post: >>>>>>>> http://www.open-mpi.org/community/lists/devel/2015/10/18323.php >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> С Уважением, Поляков Артем Юрьевич >>>>>>> Best regards, Artem Y. Polyakov >>>>>>> _______________________________________________ >>>>>>> devel mailing list >>>>>>> de...@open-mpi.org >>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>>> Link to this post: >>>>>>> http://www.open-mpi.org/community/lists/devel/2015/11/18334.php >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> devel mailing list >>>>>>> de...@open-mpi.org >>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>>> Link to this post: >>>>>>> http://www.open-mpi.org/community/lists/devel/2015/11/18335.php >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> devel mailing list >>>>>> de...@open-mpi.org >>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>> Link to this post: >>>>>> http://www.open-mpi.org/community/lists/devel/2015/11/18336.php >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> devel mailing list >>>>>> de...@open-mpi.org >>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>> Link to this post: >>>>>> http://www.open-mpi.org/community/lists/devel/2015/11/18337.php >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> devel mailing list >>>>> de...@open-mpi.org >>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>> Link to this post: >>>>> http://www.open-mpi.org/community/lists/devel/2015/11/18340.php >>>>> >>>> >>>> >>>> _______________________________________________ >>>> devel mailing list >>>> de...@open-mpi.org >>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>> Link to this post: >>>> http://www.open-mpi.org/community/lists/devel/2015/11/18341.php >>>> >>> >>> >>> >>> -- >>> С Уважением, Поляков Артем Юрьевич >>> Best regards, Artem Y. Polyakov >>> >> >> >> >> -- >> С Уважением, Поляков Артем Юрьевич >> Best regards, Artem Y. Polyakov >> >> _______________________________________________ >> devel mailing list >> de...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >> Link to this post: >> http://www.open-mpi.org/community/lists/devel/2015/11/18345.php >> > > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2015/11/18346.php > > > > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2015/11/18347.php >
