Nope - I was wrong. The correction on the client side consisted of attempting
to timeout if the blocking recv failed. We then modified the blocking send/recv
so they would handle errors.
So that problem occurred -after- the server had correctly called accept. The
listener code is in opal/mca/pmix/pmix1xx/pmix/src/server/pmix_server_listener.c
It looks to me like the only way we could drop the accept (assuming the OS
doesn’t lose it) is if the file descriptor lies outside the expected range once
we fall out of select:
/* Spin accepting connections until all active listen sockets
* do not have any incoming connections, pushing each connection
* onto the event queue for processing
*/
do {
accepted_connections = 0;
/* according to the man pages, select replaces the given descriptor
* set with a subset consisting of those descriptors that are ready
* for the specified operation - in this case, a read. So we need to
* first check to see if this file descriptor is included in the
* returned subset
*/
if (0 == FD_ISSET(pmix_server_globals.listen_socket, &readfds)) {
/* this descriptor is not included */
continue;
}
/* this descriptor is ready to be read, which means a connection
* request has been received - so harvest it. All we want to do
* here is accept the connection and push the info onto the event
* library for subsequent processing - we don't want to actually
* process the connection here as it takes too long, and so the
* OS might start rejecting connections due to timeout.
*/
pending_connection = PMIX_NEW(pmix_pending_connection_t);
event_assign(&pending_connection->ev, pmix_globals.evbase, -1,
EV_WRITE, connection_handler, pending_connection);
pending_connection->sd = accept(pmix_server_globals.listen_socket,
(struct
sockaddr*)&(pending_connection->addr),
&addrlen);
if (pending_connection->sd < 0) {
PMIX_RELEASE(pending_connection);
if (pmix_socket_errno != EAGAIN ||
pmix_socket_errno != EWOULDBLOCK) {
if (EMFILE == pmix_socket_errno) {
PMIX_ERROR_LOG(PMIX_ERR_OUT_OF_RESOURCE);
} else {
pmix_output(0, "listen_thread: accept() failed: %s
(%d).",
strerror(pmix_socket_errno),
pmix_socket_errno);
}
goto done;
}
continue;
}
pmix_output_verbose(8, pmix_globals.debug_output,
"listen_thread: new connection: (%d, %d)",
pending_connection->sd, pmix_socket_errno);
/* activate the event */
event_active(&pending_connection->ev, EV_WRITE, 1);
accepted_connections++;
} while (accepted_connections > 0);
> On Oct 28, 2015, at 12:25 AM, Ralph Castain <r...@open-mpi.org> wrote:
>
> Looking at the code, it appears that a fix was committed for this problem,
> and that we correctly resolved the issue found by Paul. The problem is that
> the fix didn’t get upstreamed, and so it was lost the next time we refreshed
> PMIx. Sigh.
>
> Let me try to recreate the fix and have you take a gander at it.
>
>
>> On Oct 28, 2015, at 12:22 AM, Ralph Castain <r...@open-mpi.org
>> <mailto:r...@open-mpi.org>> wrote:
>>
>> Here is the discussion - afraid it is fairly lengthy. Ignore the hwloc
>> references in it as that was a separate issue:
>>
>> http://www.open-mpi.org/community/lists/devel/2015/09/18074.php
>> <http://www.open-mpi.org/community/lists/devel/2015/09/18074.php>
>>
>> It definitely sounds like the same issue creeping in again. I’d appreciate
>> any thoughts on how to correct it. If it helps, you could look at the PMIx
>> master - there are standalone tests in the test/simple directory that
>> fork/exec a child and just do the connection.
>>
>> https://github.com/pmix/master <https://github.com/pmix/master>
>>
>> The test server is simptest.c - it will spawn a single copy of simpclient.c
>> by default.
>>
>>
>>> On Oct 27, 2015, at 10:14 PM, George Bosilca <bosi...@icl.utk.edu
>>> <mailto:bosi...@icl.utk.edu>> wrote:
>>>
>>> Interesting. Do you have a pointer to the commit (or/and to the discussion)?
>>>
>>> I looked at the PMIX code, and I have identified few issues, but
>>> unfortunately none of them seem to fix the problem for good. However, now I
>>> need more than 1000 runs to get a deadlock (instead of few tens).
>>>
>>> Looking with "netstat -ax" at the status of the UDS while the processes are
>>> deadlocked, I see 2 UDS with the same name: one from the server which is in
>>> LISTEN state, and one for the client which is being in CONNECTING state
>>> (while the client already sent a message in the socket and is now waiting
>>> in a blocking receive). This somehow suggest that the server has not yet
>>> called accept on the UDS. Unfortunately, there are 3 threads all doing
>>> different flavors of even_base and select, so I have a hard time tracking
>>> the path of the UDS on the server side.
>>>
>>> So in order to validate my assumption I wrote a minimalistic UDS client and
>>> server application and tried different scenarios. The conclusion is that in
>>> order to see the same type of output from "netstat -ax" I have to call
>>> listen on the server, connect on the client and do not call accept on the
>>> server.
>>>
>>> With the same occasion I also confirmed that the UDS are holding the data
>>> sent so there is no need for further synchronization for the case where the
>>> data is sent first. We only need to find out how the server forgets to call
>>> accept.
>>>
>>> George.
>>>
>>>
>>>
>>> On Tue, Oct 27, 2015 at 7:52 PM, Ralph Castain <r...@open-mpi.org
>>> <mailto:r...@open-mpi.org>> wrote:
>>> Hmmm…this looks like it might be that problem we previously saw where the
>>> blocking recv hangs in a proc when the blocking send tries to send before
>>> the domain socket is actually ready, and so the send fails on the other
>>> end. As I recall, it was something to do with the socketoptions - and then
>>> Paul had a problem on some of his machines, and we backed it out?
>>>
>>> I wonder if that’s what is biting us here again, and what we need is to
>>> either remove the blocking send/recv’s altogether, or figure out a way to
>>> wait until the socket is really ready.
>>>
>>> Any thoughts?
>>>
>>>
>>>> On Oct 27, 2015, at 4:11 PM, George Bosilca <bosi...@icl.utk.edu
>>>> <mailto:bosi...@icl.utk.edu>> wrote:
>>>>
>>>> It appear the branch solve the problem at least partially. I asked one of
>>>> my students to hammer it pretty badly, and he reported that the deadlocks
>>>> still occur. He also graciously provided some stacktraces:
>>>>
>>>> #0 0x00007f4bd5274aed in nanosleep () from /lib64/libc.so.6
>>>> #1 0x00007f4bd52a9c94 in usleep () from /lib64/libc.so.6
>>>> #2 0x00007f4bd2e42b00 in OPAL_PMIX_PMIX1XX_PMIx_Fence (procs=0x0,
>>>> nprocs=0, info=0x7fff3c561960,
>>>> ninfo=1) at src/client/pmix_client_fence.c:100
>>>> #3 0x00007f4bd306e6d2 in pmix1_fence (procs=0x0, collect_data=1) at
>>>> pmix1_client.c:306
>>>> #4 0x00007f4bd57d5cc3 in ompi_mpi_init (argc=3, argv=0x7fff3c561ea8,
>>>> requested=3,
>>>> provided=0x7fff3c561d84) at runtime/ompi_mpi_init.c:644
>>>> #5 0x00007f4bd5813399 in PMPI_Init_thread (argc=0x7fff3c561d7c,
>>>> argv=0x7fff3c561d70, required=3,
>>>> provided=0x7fff3c561d84) at pinit_thread.c:69
>>>> #6 0x0000000000401516 in main (argc=3, argv=0x7fff3c561ea8) at
>>>> osu_mbw_mr.c:86
>>>>
>>>> And another process:
>>>>
>>>> #0 0x00007f7b9d7d8bdc in recv () from /lib64/libpthread.so.0
>>>> #1 0x00007f7b9b0aa42d in opal_pmix_pmix1xx_pmix_usock_recv_blocking
>>>> (sd=13, data=0x7ffd62139004 "",
>>>> size=4) at src/usock/usock.c:168
>>>> #2 0x00007f7b9b0af5d9 in recv_connect_ack (sd=13) at
>>>> src/client/pmix_client.c:844
>>>> #3 0x00007f7b9b0b085e in usock_connect (addr=0x7ffd62139330) at
>>>> src/client/pmix_client.c:1110
>>>> #4 0x00007f7b9b0acc24 in connect_to_server (address=0x7ffd62139330,
>>>> cbdata=0x7ffd621390e0)
>>>> at src/client/pmix_client.c:181
>>>> #5 0x00007f7b9b0ad569 in OPAL_PMIX_PMIX1XX_PMIx_Init (proc=0x7f7b9b4e9b60)
>>>> at src/client/pmix_client.c:362
>>>> #6 0x00007f7b9b2dbd9d in pmix1_client_init () at pmix1_client.c:99
>>>> #7 0x00007f7b9b4eb95f in pmi_component_query (module=0x7ffd62139490,
>>>> priority=0x7ffd6213948c)
>>>> at ess_pmi_component.c:90
>>>> #8 0x00007f7b9ce70ec5 in mca_base_select (type_name=0x7f7b9d20e059 "ess",
>>>> output_id=-1,
>>>> components_available=0x7f7b9d431eb0, best_module=0x7ffd621394d0,
>>>> best_component=0x7ffd621394d8,
>>>> priority_out=0x0) at mca_base_components_select.c:77
>>>> #9 0x00007f7b9d1a956b in orte_ess_base_select () at
>>>> base/ess_base_select.c:40
>>>> #10 0x00007f7b9d160449 in orte_init (pargc=0x0, pargv=0x0, flags=32) at
>>>> runtime/orte_init.c:219
>>>> #11 0x00007f7b9da4377a in ompi_mpi_init (argc=3, argv=0x7ffd621397f8,
>>>> requested=3,
>>>> provided=0x7ffd621396d4) at runtime/ompi_mpi_init.c:488
>>>> #12 0x00007f7b9da81399 in PMPI_Init_thread (argc=0x7ffd621396cc,
>>>> argv=0x7ffd621396c0, required=3,
>>>> provided=0x7ffd621396d4) at pinit_thread.c:69
>>>> #13 0x0000000000401516 in main (argc=3, argv=0x7ffd621397f8) at
>>>> osu_mbw_mr.c:86
>>>>
>>>> George.
>>>>
>>>>
>>>>
>>>> On Tue, Oct 27, 2015 at 2:36 PM, Ralph Castain <r...@open-mpi.org
>>>> <mailto:r...@open-mpi.org>> wrote:
>>>> I haven’t been able to replicate this when using the branch in this PR:
>>>>
>>>> https://github.com/open-mpi/ompi/pull/1073
>>>> <https://github.com/open-mpi/ompi/pull/1073>
>>>>
>>>> Would you mind giving it a try? It fixes some other race conditions and
>>>> might pick this one up too.
>>>>
>>>>
>>>>> On Oct 27, 2015, at 10:04 AM, Ralph Castain <r...@open-mpi.org
>>>>> <mailto:r...@open-mpi.org>> wrote:
>>>>>
>>>>> Okay, I’ll take a look - I’ve been chasing a race condition that might be
>>>>> related
>>>>>
>>>>>> On Oct 27, 2015, at 9:54 AM, George Bosilca <bosi...@icl.utk.edu
>>>>>> <mailto:bosi...@icl.utk.edu>> wrote:
>>>>>>
>>>>>> No, it's using 2 nodes.
>>>>>> George.
>>>>>>
>>>>>>
>>>>>> On Tue, Oct 27, 2015 at 12:35 PM, Ralph Castain <r...@open-mpi.org
>>>>>> <mailto:r...@open-mpi.org>> wrote:
>>>>>> Is this on a single node?
>>>>>>
>>>>>>> On Oct 27, 2015, at 9:25 AM, George Bosilca <bosi...@icl.utk.edu
>>>>>>> <mailto:bosi...@icl.utk.edu>> wrote:
>>>>>>>
>>>>>>> I get intermittent deadlocks wit the latest trunk. The smallest
>>>>>>> reproducer is a shell for loop around a small (2 processes) short (20
>>>>>>> seconds) MPI application. After few tens of iterations the MPI_Init
>>>>>>> will deadlock with the following backtrace:
>>>>>>>
>>>>>>> #0 0x00007fa94b5d9aed in nanosleep () from /lib64/libc.so.6
>>>>>>> #1 0x00007fa94b60ec94 in usleep () from /lib64/libc.so.6
>>>>>>> #2 0x00007fa94960ba08 in OPAL_PMIX_PMIX1XX_PMIx_Fence (procs=0x0,
>>>>>>> nprocs=0, info=0x7ffd7934fb90,
>>>>>>> ninfo=1) at src/client/pmix_client_fence.c:100
>>>>>>> #3 0x00007fa9498376a2 in pmix1_fence (procs=0x0, collect_data=1) at
>>>>>>> pmix1_client.c:305
>>>>>>> #4 0x00007fa94bb39ba4 in ompi_mpi_init (argc=3, argv=0x7ffd793500a8,
>>>>>>> requested=3,
>>>>>>> provided=0x7ffd7934ff94) at runtime/ompi_mpi_init.c:645
>>>>>>> #5 0x00007fa94bb77281 in PMPI_Init_thread (argc=0x7ffd7934ff8c,
>>>>>>> argv=0x7ffd7934ff80, required=3,
>>>>>>> provided=0x7ffd7934ff94) at pinit_thread.c:69
>>>>>>> #6 0x000000000040150f in main (argc=3, argv=0x7ffd793500a8) at
>>>>>>> osu_mbw_mr.c:86
>>>>>>>
>>>>>>> On my machines this is reproducible at 100% after anywhere between 50
>>>>>>> and 100 iterations.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> George.
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> devel mailing list
>>>>>>> de...@open-mpi.org <mailto:de...@open-mpi.org>
>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>>> <http://www.open-mpi.org/mailman/listinfo.cgi/devel>
>>>>>>> Link to this post:
>>>>>>> http://www.open-mpi.org/community/lists/devel/2015/10/18280.php
>>>>>>> <http://www.open-mpi.org/community/lists/devel/2015/10/18280.php>
>>>>>>
>>>>>> _______________________________________________
>>>>>> devel mailing list
>>>>>> de...@open-mpi.org <mailto:de...@open-mpi.org>
>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>> <http://www.open-mpi.org/mailman/listinfo.cgi/devel>
>>>>>> Link to this post:
>>>>>> http://www.open-mpi.org/community/lists/devel/2015/10/18281.php
>>>>>> <http://www.open-mpi.org/community/lists/devel/2015/10/18281.php>
>>>>>>
>>>>>> _______________________________________________
>>>>>> devel mailing list
>>>>>> de...@open-mpi.org <mailto:de...@open-mpi.org>
>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>> <http://www.open-mpi.org/mailman/listinfo.cgi/devel>
>>>>>> Link to this post:
>>>>>> http://www.open-mpi.org/community/lists/devel/2015/10/18282.php
>>>>>> <http://www.open-mpi.org/community/lists/devel/2015/10/18282.php>
>>>>
>>>>
>>>> _______________________________________________
>>>> devel mailing list
>>>> de...@open-mpi.org <mailto:de...@open-mpi.org>
>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>> <http://www.open-mpi.org/mailman/listinfo.cgi/devel>
>>>> Link to this post:
>>>> http://www.open-mpi.org/community/lists/devel/2015/10/18284.php
>>>> <http://www.open-mpi.org/community/lists/devel/2015/10/18284.php>
>>>>
>>>> _______________________________________________
>>>> devel mailing list
>>>> de...@open-mpi.org <mailto:de...@open-mpi.org>
>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>> <http://www.open-mpi.org/mailman/listinfo.cgi/devel>
>>>> Link to this post:
>>>> http://www.open-mpi.org/community/lists/devel/2015/10/18292.php
>>>> <http://www.open-mpi.org/community/lists/devel/2015/10/18292.php>
>>>
>>> _______________________________________________
>>> devel mailing list
>>> de...@open-mpi.org <mailto:de...@open-mpi.org>
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>> <http://www.open-mpi.org/mailman/listinfo.cgi/devel>
>>> Link to this post:
>>> http://www.open-mpi.org/community/lists/devel/2015/10/18294.php
>>> <http://www.open-mpi.org/community/lists/devel/2015/10/18294.php>
>>>
>>> _______________________________________________
>>> devel mailing list
>>> de...@open-mpi.org <mailto:de...@open-mpi.org>
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>> <http://www.open-mpi.org/mailman/listinfo.cgi/devel>
>>> Link to this post:
>>> http://www.open-mpi.org/community/lists/devel/2015/10/18302.php
>>> <http://www.open-mpi.org/community/lists/devel/2015/10/18302.php>
>
