Some progress, that puzzles me but might help you understand. Once the
deadlock appears, if I manually kill the MPI process on the node where the
deadlock was created, the local orte daemon doesn't notice and will just
keep waiting.
Quick question: I am under the impression that the issue is not in the PMIX
server but somewhere around the listener_thread_fn in orte/util/listener.c.
Possible ?
George.
On Wed, Oct 28, 2015 at 3:56 AM, Ralph Castain <r...@open-mpi.org> wrote:
> Should have also clarified: the prior fixes are indeed in the current
> master.
>
> On Oct 28, 2015, at 12:42 AM, Ralph Castain <r...@open-mpi.org> wrote:
>
> Nope - I was wrong. The correction on the client side consisted of
> attempting to timeout if the blocking recv failed. We then modified the
> blocking send/recv so they would handle errors.
>
> So that problem occurred -after- the server had correctly called accept.
> The listener code is in
> opal/mca/pmix/pmix1xx/pmix/src/server/pmix_server_listener.c
>
> It looks to me like the only way we could drop the accept (assuming the OS
> doesn’t lose it) is if the file descriptor lies outside the expected range
> once we fall out of select:
>
>
> /* Spin accepting connections until all active listen sockets
> * do not have any incoming connections, pushing each connection
> * onto the event queue for processing
> */
> do {
> accepted_connections = 0;
> /* according to the man pages, select replaces the given
> descriptor
> * set with a subset consisting of those descriptors that are
> ready
> * for the specified operation - in this case, a read. So we
> need to
> * first check to see if this file descriptor is included in
> the
> * returned subset
> */
> if (0 == FD_ISSET(pmix_server_globals.listen_socket,
> &readfds)) {
> /* this descriptor is not included */
> continue;
> }
>
> /* this descriptor is ready to be read, which means a
> connection
> * request has been received - so harvest it. All we want to do
> * here is accept the connection and push the info onto the
> event
> * library for subsequent processing - we don't want to
> actually
> * process the connection here as it takes too long, and so the
> * OS might start rejecting connections due to timeout.
> */
> pending_connection = PMIX_NEW(pmix_pending_connection_t);
> event_assign(&pending_connection->ev, pmix_globals.evbase, -1,
> EV_WRITE, connection_handler, pending_connection);
> pending_connection->sd =
> accept(pmix_server_globals.listen_socket,
> (struct
> sockaddr*)&(pending_connection->addr),
> &addrlen);
> if (pending_connection->sd < 0) {
> PMIX_RELEASE(pending_connection);
> if (pmix_socket_errno != EAGAIN ||
> pmix_socket_errno != EWOULDBLOCK) {
> if (EMFILE == pmix_socket_errno) {
> PMIX_ERROR_LOG(PMIX_ERR_OUT_OF_RESOURCE);
> } else {
> pmix_output(0, "listen_thread: accept() failed: %s
> (%d).",
> strerror(pmix_socket_errno),
> pmix_socket_errno);
> }
> goto done;
> }
> continue;
> }
>
> pmix_output_verbose(8, pmix_globals.debug_output,
> "listen_thread: new connection: (%d, %d)",
> pending_connection->sd, pmix_socket_errno);
> /* activate the event */
> event_active(&pending_connection->ev, EV_WRITE, 1);
> accepted_connections++;
> } while (accepted_connections > 0);
>
>
> On Oct 28, 2015, at 12:25 AM, Ralph Castain <r...@open-mpi.org> wrote:
>
> Looking at the code, it appears that a fix was committed for this problem,
> and that we correctly resolved the issue found by Paul. The problem is that
> the fix didn’t get upstreamed, and so it was lost the next time we
> refreshed PMIx. Sigh.
>
> Let me try to recreate the fix and have you take a gander at it.
>
>
> On Oct 28, 2015, at 12:22 AM, Ralph Castain <r...@open-mpi.org> wrote:
>
> Here is the discussion - afraid it is fairly lengthy. Ignore the hwloc
> references in it as that was a separate issue:
>
> http://www.open-mpi.org/community/lists/devel/2015/09/18074.php
>
> It definitely sounds like the same issue creeping in again. I’d appreciate
> any thoughts on how to correct it. If it helps, you could look at the PMIx
> master - there are standalone tests in the test/simple directory that
> fork/exec a child and just do the connection.
>
> https://github.com/pmix/master
>
> The test server is simptest.c - it will spawn a single copy of
> simpclient.c by default.
>
>
> On Oct 27, 2015, at 10:14 PM, George Bosilca <bosi...@icl.utk.edu> wrote:
>
> Interesting. Do you have a pointer to the commit (or/and to the
> discussion)?
>
> I looked at the PMIX code, and I have identified few issues, but
> unfortunately none of them seem to fix the problem for good. However, now I
> need more than 1000 runs to get a deadlock (instead of few tens).
>
> Looking with "netstat -ax" at the status of the UDS while the processes
> are deadlocked, I see 2 UDS with the same name: one from the server which
> is in LISTEN state, and one for the client which is being in CONNECTING
> state (while the client already sent a message in the socket and is now
> waiting in a blocking receive). This somehow suggest that the server has
> not yet called accept on the UDS. Unfortunately, there are 3 threads all
> doing different flavors of even_base and select, so I have a hard time
> tracking the path of the UDS on the server side.
>
> So in order to validate my assumption I wrote a minimalistic UDS client
> and server application and tried different scenarios. The conclusion is
> that in order to see the same type of output from "netstat -ax" I have to
> call listen on the server, connect on the client and do not call accept on
> the server.
>
> With the same occasion I also confirmed that the UDS are holding the data
> sent so there is no need for further synchronization for the case where the
> data is sent first. We only need to find out how the server forgets to call
> accept.
>
> George.
>
>
>
> On Tue, Oct 27, 2015 at 7:52 PM, Ralph Castain <r...@open-mpi.org> wrote:
>
>> Hmmm…this looks like it might be that problem we previously saw where the
>> blocking recv hangs in a proc when the blocking send tries to send before
>> the domain socket is actually ready, and so the send fails on the other
>> end. As I recall, it was something to do with the socketoptions - and then
>> Paul had a problem on some of his machines, and we backed it out?
>>
>> I wonder if that’s what is biting us here again, and what we need is to
>> either remove the blocking send/recv’s altogether, or figure out a way to
>> wait until the socket is really ready.
>>
>> Any thoughts?
>>
>>
>> On Oct 27, 2015, at 4:11 PM, George Bosilca <bosi...@icl.utk.edu> wrote:
>>
>> It appear the branch solve the problem at least partially. I asked one of
>> my students to hammer it pretty badly, and he reported that the deadlocks
>> still occur. He also graciously provided some stacktraces:
>>
>> #0 0x00007f4bd5274aed in nanosleep () from /lib64/libc.so.6
>> #1 0x00007f4bd52a9c94 in usleep () from /lib64/libc.so.6
>> #2 0x00007f4bd2e42b00 in OPAL_PMIX_PMIX1XX_PMIx_Fence (procs=0x0,
>> nprocs=0, info=0x7fff3c561960,
>> ninfo=1) at src/client/pmix_client_fence.c:100
>> #3 0x00007f4bd306e6d2 in pmix1_fence (procs=0x0, collect_data=1) at
>> pmix1_client.c:306
>> #4 0x00007f4bd57d5cc3 in ompi_mpi_init (argc=3, argv=0x7fff3c561ea8,
>> requested=3,
>> provided=0x7fff3c561d84) at runtime/ompi_mpi_init.c:644
>> #5 0x00007f4bd5813399 in PMPI_Init_thread (argc=0x7fff3c561d7c,
>> argv=0x7fff3c561d70, required=3,
>> provided=0x7fff3c561d84) at pinit_thread.c:69
>> #6 0x0000000000401516 in main (argc=3, argv=0x7fff3c561ea8) at
>> osu_mbw_mr.c:86
>>
>> And another process:
>>
>> #0 0x00007f7b9d7d8bdc in recv () from /lib64/libpthread.so.0
>> #1 0x00007f7b9b0aa42d in opal_pmix_pmix1xx_pmix_usock_recv_blocking
>> (sd=13, data=0x7ffd62139004 "",
>> size=4) at src/usock/usock.c:168
>> #2 0x00007f7b9b0af5d9 in recv_connect_ack (sd=13) at
>> src/client/pmix_client.c:844
>> #3 0x00007f7b9b0b085e in usock_connect (addr=0x7ffd62139330) at
>> src/client/pmix_client.c:1110
>> #4 0x00007f7b9b0acc24 in connect_to_server (address=0x7ffd62139330,
>> cbdata=0x7ffd621390e0)
>> at src/client/pmix_client.c:181
>> #5 0x00007f7b9b0ad569 in OPAL_PMIX_PMIX1XX_PMIx_Init
>> (proc=0x7f7b9b4e9b60)
>> at src/client/pmix_client.c:362
>> #6 0x00007f7b9b2dbd9d in pmix1_client_init () at pmix1_client.c:99
>> #7 0x00007f7b9b4eb95f in pmi_component_query (module=0x7ffd62139490,
>> priority=0x7ffd6213948c)
>> at ess_pmi_component.c:90
>> #8 0x00007f7b9ce70ec5 in mca_base_select (type_name=0x7f7b9d20e059
>> "ess", output_id=-1,
>> components_available=0x7f7b9d431eb0, best_module=0x7ffd621394d0,
>> best_component=0x7ffd621394d8,
>> priority_out=0x0) at mca_base_components_select.c:77
>> #9 0x00007f7b9d1a956b in orte_ess_base_select () at
>> base/ess_base_select.c:40
>> #10 0x00007f7b9d160449 in orte_init (pargc=0x0, pargv=0x0, flags=32) at
>> runtime/orte_init.c:219
>> #11 0x00007f7b9da4377a in ompi_mpi_init (argc=3, argv=0x7ffd621397f8,
>> requested=3,
>> provided=0x7ffd621396d4) at runtime/ompi_mpi_init.c:488
>> #12 0x00007f7b9da81399 in PMPI_Init_thread (argc=0x7ffd621396cc,
>> argv=0x7ffd621396c0, required=3,
>> provided=0x7ffd621396d4) at pinit_thread.c:69
>> #13 0x0000000000401516 in main (argc=3, argv=0x7ffd621397f8) at
>> osu_mbw_mr.c:86
>>
>> George.
>>
>>
>>
>> On Tue, Oct 27, 2015 at 2:36 PM, Ralph Castain <r...@open-mpi.org> wrote:
>>
>>> I haven’t been able to replicate this when using the branch in this PR:
>>>
>>> https://github.com/open-mpi/ompi/pull/1073
>>>
>>> Would you mind giving it a try? It fixes some other race conditions and
>>> might pick this one up too.
>>>
>>>
>>> On Oct 27, 2015, at 10:04 AM, Ralph Castain <r...@open-mpi.org> wrote:
>>>
>>> Okay, I’ll take a look - I’ve been chasing a race condition that might
>>> be related
>>>
>>> On Oct 27, 2015, at 9:54 AM, George Bosilca <bosi...@icl.utk.edu> wrote:
>>>
>>> No, it's using 2 nodes.
>>> George.
>>>
>>>
>>> On Tue, Oct 27, 2015 at 12:35 PM, Ralph Castain <r...@open-mpi.org>
>>> wrote:
>>>
>>>> Is this on a single node?
>>>>
>>>> On Oct 27, 2015, at 9:25 AM, George Bosilca <bosi...@icl.utk.edu>
>>>> wrote:
>>>>
>>>> I get intermittent deadlocks wit the latest trunk. The smallest
>>>> reproducer is a shell for loop around a small (2 processes) short (20
>>>> seconds) MPI application. After few tens of iterations the MPI_Init will
>>>> deadlock with the following backtrace:
>>>>
>>>> #0 0x00007fa94b5d9aed in nanosleep () from /lib64/libc.so.6
>>>> #1 0x00007fa94b60ec94 in usleep () from /lib64/libc.so.6
>>>> #2 0x00007fa94960ba08 in OPAL_PMIX_PMIX1XX_PMIx_Fence (procs=0x0,
>>>> nprocs=0, info=0x7ffd7934fb90,
>>>> ninfo=1) at src/client/pmix_client_fence.c:100
>>>> #3 0x00007fa9498376a2 in pmix1_fence (procs=0x0, collect_data=1) at
>>>> pmix1_client.c:305
>>>> #4 0x00007fa94bb39ba4 in ompi_mpi_init (argc=3, argv=0x7ffd793500a8,
>>>> requested=3,
>>>> provided=0x7ffd7934ff94) at runtime/ompi_mpi_init.c:645
>>>> #5 0x00007fa94bb77281 in PMPI_Init_thread (argc=0x7ffd7934ff8c,
>>>> argv=0x7ffd7934ff80, required=3,
>>>> provided=0x7ffd7934ff94) at pinit_thread.c:69
>>>> #6 0x000000000040150f in main (argc=3, argv=0x7ffd793500a8) at
>>>> osu_mbw_mr.c:86
>>>>
>>>> On my machines this is reproducible at 100% after anywhere between 50
>>>> and 100 iterations.
>>>>
>>>> Thanks,
>>>> George.
>>>>
>>>> _______________________________________________
>>>> devel mailing list
>>>> de...@open-mpi.org
>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>> Link to this post:
>>>> http://www.open-mpi.org/community/lists/devel/2015/10/18280.php
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> devel mailing list
>>>> de...@open-mpi.org
>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>> Link to this post:
>>>> http://www.open-mpi.org/community/lists/devel/2015/10/18281.php
>>>>
>>>
>>> _______________________________________________
>>> devel mailing list
>>> de...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>> Link to this post:
>>> http://www.open-mpi.org/community/lists/devel/2015/10/18282.php
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> devel mailing list
>>> de...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>> Link to this post:
>>> http://www.open-mpi.org/community/lists/devel/2015/10/18284.php
>>>
>>
>> _______________________________________________
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post:
>> http://www.open-mpi.org/community/lists/devel/2015/10/18292.php
>>
>>
>>
>> _______________________________________________
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post:
>> http://www.open-mpi.org/community/lists/devel/2015/10/18294.php
>>
>
> _______________________________________________
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2015/10/18302.php
>
>
>
>
>
>
> _______________________________________________
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2015/10/18309.php
>
