Could you send me your stress test? I’m wondering if it is just something about how we set socket options
> On Nov 7, 2015, at 8:58 AM, George Bosilca <bosi...@icl.utk.edu> wrote: > > I has to postpone this until after SC. However, I ran for 3 days a stress > test of UDS reproducing the opening and sending of data (what Ralph said in > his email) and I never could get a deadlock. > > George. > > > On Sat, Nov 7, 2015 at 11:26 AM, Ralph Castain <r...@open-mpi.org > <mailto:r...@open-mpi.org>> wrote: > George was looking into it, but I don’t know if he has had time recently to > continue the investigation. We understand “what” is happening (accept > sometimes ignores the connection), but we don’t yet know “why”. I’ve done > some digging around the web, and found that sometimes you can try to talk to > a Unix Domain Socket too quickly - i.e., you open it and then send to it, but > the OS hasn’t yet set it up. In those cases, you can hang the socket. > However, I’ve tried adding some artificial delay, and while it helped, it > didn’t completely solve the problem. > > I have an idea for a workaround (set a timer and retry after awhile), but > would obviously prefer a real solution. I’m not even sure it will work as it > is unclear that the server (who is the one hung in accept) will break free if > the client closes the socket and retries. > > >> On Nov 6, 2015, at 10:53 PM, Artem Polyakov <artpo...@gmail.com >> <mailto:artpo...@gmail.com>> wrote: >> >> Hello, is there any progress on this topic? This affects our PMIx >> measurements. >> >> 2015-10-30 21:21 GMT+06:00 Ralph Castain <r...@open-mpi.org >> <mailto:r...@open-mpi.org>>: >> I’ve verified that the orte/util/listener thread is not being started, so I >> don’t think it should be involved in this problem. >> >> HTH >> Ralph >> >>> On Oct 30, 2015, at 8:07 AM, Ralph Castain <r...@open-mpi.org >>> <mailto:r...@open-mpi.org>> wrote: >>> >>> Hmmm…there is a hook that would allow the PMIx server to utilize that >>> listener thread, but we aren’t currently using it. Each daemon plus mpirun >>> will call orte_start_listener, but nothing is currently registering and so >>> the listener in that code is supposed to just return without starting the >>> thread. >>> >>> So the only listener thread that should exist is the one inside the PMIx >>> server itself. If something else is happening, then that would be a bug. I >>> can look at the orte listener code to ensure that the thread isn’t >>> incorrectly starting. >>> >>> >>>> On Oct 29, 2015, at 10:03 PM, George Bosilca <bosi...@icl.utk.edu >>>> <mailto:bosi...@icl.utk.edu>> wrote: >>>> >>>> Some progress, that puzzles me but might help you understand. Once the >>>> deadlock appears, if I manually kill the MPI process on the node where the >>>> deadlock was created, the local orte daemon doesn't notice and will just >>>> keep waiting. >>>> >>>> Quick question: I am under the impression that the issue is not in the >>>> PMIX server but somewhere around the listener_thread_fn in >>>> orte/util/listener.c. Possible ? >>>> >>>> George. >>>> >>>> >>>> On Wed, Oct 28, 2015 at 3:56 AM, Ralph Castain <r...@open-mpi.org >>>> <mailto:r...@open-mpi.org>> wrote: >>>> Should have also clarified: the prior fixes are indeed in the current >>>> master. >>>> >>>>> On Oct 28, 2015, at 12:42 AM, Ralph Castain <r...@open-mpi.org >>>>> <mailto:r...@open-mpi.org>> wrote: >>>>> >>>>> Nope - I was wrong. The correction on the client side consisted of >>>>> attempting to timeout if the blocking recv failed. We then modified the >>>>> blocking send/recv so they would handle errors. >>>>> >>>>> So that problem occurred -after- the server had correctly called accept. >>>>> The listener code is in >>>>> opal/mca/pmix/pmix1xx/pmix/src/server/pmix_server_listener.c >>>>> >>>>> It looks to me like the only way we could drop the accept (assuming the >>>>> OS doesn’t lose it) is if the file descriptor lies outside the expected >>>>> range once we fall out of select: >>>>> >>>>> >>>>> /* Spin accepting connections until all active listen sockets >>>>> * do not have any incoming connections, pushing each connection >>>>> * onto the event queue for processing >>>>> */ >>>>> do { >>>>> accepted_connections = 0; >>>>> /* according to the man pages, select replaces the given >>>>> descriptor >>>>> * set with a subset consisting of those descriptors that are >>>>> ready >>>>> * for the specified operation - in this case, a read. So we >>>>> need to >>>>> * first check to see if this file descriptor is included in >>>>> the >>>>> * returned subset >>>>> */ >>>>> if (0 == FD_ISSET(pmix_server_globals.listen_socket, >>>>> &readfds)) { >>>>> /* this descriptor is not included */ >>>>> continue; >>>>> } >>>>> >>>>> /* this descriptor is ready to be read, which means a >>>>> connection >>>>> * request has been received - so harvest it. All we want to >>>>> do >>>>> * here is accept the connection and push the info onto the >>>>> event >>>>> * library for subsequent processing - we don't want to >>>>> actually >>>>> * process the connection here as it takes too long, and so >>>>> the >>>>> * OS might start rejecting connections due to timeout. >>>>> */ >>>>> pending_connection = PMIX_NEW(pmix_pending_connection_t); >>>>> event_assign(&pending_connection->ev, pmix_globals.evbase, -1, >>>>> EV_WRITE, connection_handler, >>>>> pending_connection); >>>>> pending_connection->sd = >>>>> accept(pmix_server_globals.listen_socket, >>>>> (struct >>>>> sockaddr*)&(pending_connection->addr), >>>>> &addrlen); >>>>> if (pending_connection->sd < 0) { >>>>> PMIX_RELEASE(pending_connection); >>>>> if (pmix_socket_errno != EAGAIN || >>>>> pmix_socket_errno != EWOULDBLOCK) { >>>>> if (EMFILE == pmix_socket_errno) { >>>>> PMIX_ERROR_LOG(PMIX_ERR_OUT_OF_RESOURCE); >>>>> } else { >>>>> pmix_output(0, "listen_thread: accept() failed: >>>>> %s (%d).", >>>>> strerror(pmix_socket_errno), >>>>> pmix_socket_errno); >>>>> } >>>>> goto done; >>>>> } >>>>> continue; >>>>> } >>>>> >>>>> pmix_output_verbose(8, pmix_globals.debug_output, >>>>> "listen_thread: new connection: (%d, %d)", >>>>> pending_connection->sd, >>>>> pmix_socket_errno); >>>>> /* activate the event */ >>>>> event_active(&pending_connection->ev, EV_WRITE, 1); >>>>> accepted_connections++; >>>>> } while (accepted_connections > 0); >>>>> >>>>> >>>>>> On Oct 28, 2015, at 12:25 AM, Ralph Castain <r...@open-mpi.org >>>>>> <mailto:r...@open-mpi.org>> wrote: >>>>>> >>>>>> Looking at the code, it appears that a fix was committed for this >>>>>> problem, and that we correctly resolved the issue found by Paul. The >>>>>> problem is that the fix didn’t get upstreamed, and so it was lost the >>>>>> next time we refreshed PMIx. Sigh. >>>>>> >>>>>> Let me try to recreate the fix and have you take a gander at it. >>>>>> >>>>>> >>>>>>> On Oct 28, 2015, at 12:22 AM, Ralph Castain <r...@open-mpi.org >>>>>>> <mailto:r...@open-mpi.org>> wrote: >>>>>>> >>>>>>> Here is the discussion - afraid it is fairly lengthy. Ignore the hwloc >>>>>>> references in it as that was a separate issue: >>>>>>> >>>>>>> http://www.open-mpi.org/community/lists/devel/2015/09/18074.php >>>>>>> <http://www.open-mpi.org/community/lists/devel/2015/09/18074.php> >>>>>>> >>>>>>> It definitely sounds like the same issue creeping in again. I’d >>>>>>> appreciate any thoughts on how to correct it. If it helps, you could >>>>>>> look at the PMIx master - there are standalone tests in the test/simple >>>>>>> directory that fork/exec a child and just do the connection. >>>>>>> >>>>>>> https://github.com/pmix/master <https://github.com/pmix/master> >>>>>>> >>>>>>> The test server is simptest.c - it will spawn a single copy of >>>>>>> simpclient.c by default. >>>>>>> >>>>>>> >>>>>>>> On Oct 27, 2015, at 10:14 PM, George Bosilca <bosi...@icl.utk.edu >>>>>>>> <mailto:bosi...@icl.utk.edu>> wrote: >>>>>>>> >>>>>>>> Interesting. Do you have a pointer to the commit (or/and to the >>>>>>>> discussion)? >>>>>>>> >>>>>>>> I looked at the PMIX code, and I have identified few issues, but >>>>>>>> unfortunately none of them seem to fix the problem for good. However, >>>>>>>> now I need more than 1000 runs to get a deadlock (instead of few tens). >>>>>>>> >>>>>>>> Looking with "netstat -ax" at the status of the UDS while the >>>>>>>> processes are deadlocked, I see 2 UDS with the same name: one from the >>>>>>>> server which is in LISTEN state, and one for the client which is being >>>>>>>> in CONNECTING state (while the client already sent a message in the >>>>>>>> socket and is now waiting in a blocking receive). This somehow suggest >>>>>>>> that the server has not yet called accept on the UDS. Unfortunately, >>>>>>>> there are 3 threads all doing different flavors of even_base and >>>>>>>> select, so I have a hard time tracking the path of the UDS on the >>>>>>>> server side. >>>>>>>> >>>>>>>> So in order to validate my assumption I wrote a minimalistic UDS >>>>>>>> client and server application and tried different scenarios. The >>>>>>>> conclusion is that in order to see the same type of output from >>>>>>>> "netstat -ax" I have to call listen on the server, connect on the >>>>>>>> client and do not call accept on the server. >>>>>>>> >>>>>>>> With the same occasion I also confirmed that the UDS are holding the >>>>>>>> data sent so there is no need for further synchronization for the case >>>>>>>> where the data is sent first. We only need to find out how the server >>>>>>>> forgets to call accept. >>>>>>>> >>>>>>>> George. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Oct 27, 2015 at 7:52 PM, Ralph Castain <r...@open-mpi.org >>>>>>>> <mailto:r...@open-mpi.org>> wrote: >>>>>>>> Hmmm…this looks like it might be that problem we previously saw where >>>>>>>> the blocking recv hangs in a proc when the blocking send tries to send >>>>>>>> before the domain socket is actually ready, and so the send fails on >>>>>>>> the other end. As I recall, it was something to do with the >>>>>>>> socketoptions - and then Paul had a problem on some of his machines, >>>>>>>> and we backed it out? >>>>>>>> >>>>>>>> I wonder if that’s what is biting us here again, and what we need is >>>>>>>> to either remove the blocking send/recv’s altogether, or figure out a >>>>>>>> way to wait until the socket is really ready. >>>>>>>> >>>>>>>> Any thoughts? >>>>>>>> >>>>>>>> >>>>>>>>> On Oct 27, 2015, at 4:11 PM, George Bosilca <bosi...@icl.utk.edu >>>>>>>>> <mailto:bosi...@icl.utk.edu>> wrote: >>>>>>>>> >>>>>>>>> It appear the branch solve the problem at least partially. I asked >>>>>>>>> one of my students to hammer it pretty badly, and he reported that >>>>>>>>> the deadlocks still occur. He also graciously provided some >>>>>>>>> stacktraces: >>>>>>>>> >>>>>>>>> #0 0x00007f4bd5274aed in nanosleep () from /lib64/libc.so.6 >>>>>>>>> #1 0x00007f4bd52a9c94 in usleep () from /lib64/libc.so.6 >>>>>>>>> #2 0x00007f4bd2e42b00 in OPAL_PMIX_PMIX1XX_PMIx_Fence (procs=0x0, >>>>>>>>> nprocs=0, info=0x7fff3c561960, >>>>>>>>> ninfo=1) at src/client/pmix_client_fence.c:100 >>>>>>>>> #3 0x00007f4bd306e6d2 in pmix1_fence (procs=0x0, collect_data=1) at >>>>>>>>> pmix1_client.c:306 >>>>>>>>> #4 0x00007f4bd57d5cc3 in ompi_mpi_init (argc=3, argv=0x7fff3c561ea8, >>>>>>>>> requested=3, >>>>>>>>> provided=0x7fff3c561d84) at runtime/ompi_mpi_init.c:644 >>>>>>>>> #5 0x00007f4bd5813399 in PMPI_Init_thread (argc=0x7fff3c561d7c, >>>>>>>>> argv=0x7fff3c561d70, required=3, >>>>>>>>> provided=0x7fff3c561d84) at pinit_thread.c:69 >>>>>>>>> #6 0x0000000000401516 in main (argc=3, argv=0x7fff3c561ea8) at >>>>>>>>> osu_mbw_mr.c:86 >>>>>>>>> >>>>>>>>> And another process: >>>>>>>>> >>>>>>>>> #0 0x00007f7b9d7d8bdc in recv () from /lib64/libpthread.so.0 >>>>>>>>> #1 0x00007f7b9b0aa42d in opal_pmix_pmix1xx_pmix_usock_recv_blocking >>>>>>>>> (sd=13, data=0x7ffd62139004 "", >>>>>>>>> size=4) at src/usock/usock.c:168 >>>>>>>>> #2 0x00007f7b9b0af5d9 in recv_connect_ack (sd=13) at >>>>>>>>> src/client/pmix_client.c:844 >>>>>>>>> #3 0x00007f7b9b0b085e in usock_connect (addr=0x7ffd62139330) at >>>>>>>>> src/client/pmix_client.c:1110 >>>>>>>>> #4 0x00007f7b9b0acc24 in connect_to_server (address=0x7ffd62139330, >>>>>>>>> cbdata=0x7ffd621390e0) >>>>>>>>> at src/client/pmix_client.c:181 >>>>>>>>> #5 0x00007f7b9b0ad569 in OPAL_PMIX_PMIX1XX_PMIx_Init >>>>>>>>> (proc=0x7f7b9b4e9b60) >>>>>>>>> at src/client/pmix_client.c:362 >>>>>>>>> #6 0x00007f7b9b2dbd9d in pmix1_client_init () at pmix1_client.c:99 >>>>>>>>> #7 0x00007f7b9b4eb95f in pmi_component_query (module=0x7ffd62139490, >>>>>>>>> priority=0x7ffd6213948c) >>>>>>>>> at ess_pmi_component.c:90 >>>>>>>>> #8 0x00007f7b9ce70ec5 in mca_base_select (type_name=0x7f7b9d20e059 >>>>>>>>> "ess", output_id=-1, >>>>>>>>> components_available=0x7f7b9d431eb0, best_module=0x7ffd621394d0, >>>>>>>>> best_component=0x7ffd621394d8, >>>>>>>>> priority_out=0x0) at mca_base_components_select.c:77 >>>>>>>>> #9 0x00007f7b9d1a956b in orte_ess_base_select () at >>>>>>>>> base/ess_base_select.c:40 >>>>>>>>> #10 0x00007f7b9d160449 in orte_init (pargc=0x0, pargv=0x0, flags=32) >>>>>>>>> at runtime/orte_init.c:219 >>>>>>>>> #11 0x00007f7b9da4377a in ompi_mpi_init (argc=3, argv=0x7ffd621397f8, >>>>>>>>> requested=3, >>>>>>>>> provided=0x7ffd621396d4) at runtime/ompi_mpi_init.c:488 >>>>>>>>> #12 0x00007f7b9da81399 in PMPI_Init_thread (argc=0x7ffd621396cc, >>>>>>>>> argv=0x7ffd621396c0, required=3, >>>>>>>>> provided=0x7ffd621396d4) at pinit_thread.c:69 >>>>>>>>> #13 0x0000000000401516 in main (argc=3, argv=0x7ffd621397f8) at >>>>>>>>> osu_mbw_mr.c:86 >>>>>>>>> >>>>>>>>> George. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Oct 27, 2015 at 2:36 PM, Ralph Castain <r...@open-mpi.org >>>>>>>>> <mailto:r...@open-mpi.org>> wrote: >>>>>>>>> I haven’t been able to replicate this when using the branch in this >>>>>>>>> PR: >>>>>>>>> >>>>>>>>> https://github.com/open-mpi/ompi/pull/1073 >>>>>>>>> <https://github.com/open-mpi/ompi/pull/1073> >>>>>>>>> >>>>>>>>> Would you mind giving it a try? It fixes some other race conditions >>>>>>>>> and might pick this one up too. >>>>>>>>> >>>>>>>>> >>>>>>>>>> On Oct 27, 2015, at 10:04 AM, Ralph Castain <r...@open-mpi.org >>>>>>>>>> <mailto:r...@open-mpi.org>> wrote: >>>>>>>>>> >>>>>>>>>> Okay, I’ll take a look - I’ve been chasing a race condition that >>>>>>>>>> might be related >>>>>>>>>> >>>>>>>>>>> On Oct 27, 2015, at 9:54 AM, George Bosilca <bosi...@icl.utk.edu >>>>>>>>>>> <mailto:bosi...@icl.utk.edu>> wrote: >>>>>>>>>>> >>>>>>>>>>> No, it's using 2 nodes. >>>>>>>>>>> George. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Oct 27, 2015 at 12:35 PM, Ralph Castain <r...@open-mpi.org >>>>>>>>>>> <mailto:r...@open-mpi.org>> wrote: >>>>>>>>>>> Is this on a single node? >>>>>>>>>>> >>>>>>>>>>>> On Oct 27, 2015, at 9:25 AM, George Bosilca <bosi...@icl.utk.edu >>>>>>>>>>>> <mailto:bosi...@icl.utk.edu>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> I get intermittent deadlocks wit the latest trunk. The smallest >>>>>>>>>>>> reproducer is a shell for loop around a small (2 processes) short >>>>>>>>>>>> (20 seconds) MPI application. After few tens of iterations the >>>>>>>>>>>> MPI_Init will deadlock with the following backtrace: >>>>>>>>>>>> >>>>>>>>>>>> #0 0x00007fa94b5d9aed in nanosleep () from /lib64/libc.so.6 >>>>>>>>>>>> #1 0x00007fa94b60ec94 in usleep () from /lib64/libc.so.6 >>>>>>>>>>>> #2 0x00007fa94960ba08 in OPAL_PMIX_PMIX1XX_PMIx_Fence (procs=0x0, >>>>>>>>>>>> nprocs=0, info=0x7ffd7934fb90, >>>>>>>>>>>> ninfo=1) at src/client/pmix_client_fence.c:100 >>>>>>>>>>>> #3 0x00007fa9498376a2 in pmix1_fence (procs=0x0, collect_data=1) >>>>>>>>>>>> at pmix1_client.c:305 >>>>>>>>>>>> #4 0x00007fa94bb39ba4 in ompi_mpi_init (argc=3, >>>>>>>>>>>> argv=0x7ffd793500a8, requested=3, >>>>>>>>>>>> provided=0x7ffd7934ff94) at runtime/ompi_mpi_init.c:645 >>>>>>>>>>>> #5 0x00007fa94bb77281 in PMPI_Init_thread (argc=0x7ffd7934ff8c, >>>>>>>>>>>> argv=0x7ffd7934ff80, required=3, >>>>>>>>>>>> provided=0x7ffd7934ff94) at pinit_thread.c:69 >>>>>>>>>>>> #6 0x000000000040150f in main (argc=3, argv=0x7ffd793500a8) at >>>>>>>>>>>> osu_mbw_mr.c:86 >>>>>>>>>>>> >>>>>>>>>>>> On my machines this is reproducible at 100% after anywhere between >>>>>>>>>>>> 50 and 100 iterations. >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> George. >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> devel mailing list >>>>>>>>>>>> de...@open-mpi.org <mailto:de...@open-mpi.org> >>>>>>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>>>>>>>> <http://www.open-mpi.org/mailman/listinfo.cgi/devel> >>>>>>>>>>>> Link to this post: >>>>>>>>>>>> http://www.open-mpi.org/community/lists/devel/2015/10/18280.php >>>>>>>>>>>> <http://www.open-mpi.org/community/lists/devel/2015/10/18280.php> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> devel mailing list >>>>>>>>>>> de...@open-mpi.org <mailto:de...@open-mpi.org> >>>>>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>>>>>>> <http://www.open-mpi.org/mailman/listinfo.cgi/devel> >>>>>>>>>>> Link to this post: >>>>>>>>>>> http://www.open-mpi.org/community/lists/devel/2015/10/18281.php >>>>>>>>>>> <http://www.open-mpi.org/community/lists/devel/2015/10/18281.php> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> devel mailing list >>>>>>>>>>> de...@open-mpi.org <mailto:de...@open-mpi.org> >>>>>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>>>>>>> <http://www.open-mpi.org/mailman/listinfo.cgi/devel> >>>>>>>>>>> Link to this post: >>>>>>>>>>> http://www.open-mpi.org/community/lists/devel/2015/10/18282.php >>>>>>>>>>> <http://www.open-mpi.org/community/lists/devel/2015/10/18282.php> >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> devel mailing list >>>>>>>>> de...@open-mpi.org <mailto:de...@open-mpi.org> >>>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>>>>> <http://www.open-mpi.org/mailman/listinfo.cgi/devel> >>>>>>>>> Link to this post: >>>>>>>>> http://www.open-mpi.org/community/lists/devel/2015/10/18284.php >>>>>>>>> <http://www.open-mpi.org/community/lists/devel/2015/10/18284.php> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> devel mailing list >>>>>>>>> de...@open-mpi.org <mailto:de...@open-mpi.org> >>>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>>>>> <http://www.open-mpi.org/mailman/listinfo.cgi/devel> >>>>>>>>> Link to this post: >>>>>>>>> http://www.open-mpi.org/community/lists/devel/2015/10/18292.php >>>>>>>>> <http://www.open-mpi.org/community/lists/devel/2015/10/18292.php> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> devel mailing list >>>>>>>> de...@open-mpi.org <mailto:de...@open-mpi.org> >>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>>>> <http://www.open-mpi.org/mailman/listinfo.cgi/devel> >>>>>>>> Link to this post: >>>>>>>> http://www.open-mpi.org/community/lists/devel/2015/10/18294.php >>>>>>>> <http://www.open-mpi.org/community/lists/devel/2015/10/18294.php> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> devel mailing list >>>>>>>> de...@open-mpi.org <mailto:de...@open-mpi.org> >>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>>>> <http://www.open-mpi.org/mailman/listinfo.cgi/devel> >>>>>>>> Link to this post: >>>>>>>> http://www.open-mpi.org/community/lists/devel/2015/10/18302.php >>>>>>>> <http://www.open-mpi.org/community/lists/devel/2015/10/18302.php> >>>>>> >>>>> >>>> >>>> >>>> _______________________________________________ >>>> devel mailing list >>>> de...@open-mpi.org <mailto:de...@open-mpi.org> >>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>> <http://www.open-mpi.org/mailman/listinfo.cgi/devel> >>>> Link to this post: >>>> http://www.open-mpi.org/community/lists/devel/2015/10/18309.php >>>> <http://www.open-mpi.org/community/lists/devel/2015/10/18309.php> >>>> >>>> _______________________________________________ >>>> devel mailing list >>>> de...@open-mpi.org <mailto:de...@open-mpi.org> >>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>> <http://www.open-mpi.org/mailman/listinfo.cgi/devel> >>>> Link to this post: >>>> http://www.open-mpi.org/community/lists/devel/2015/10/18320.php >>>> <http://www.open-mpi.org/community/lists/devel/2015/10/18320.php> >> >> >> _______________________________________________ >> devel mailing list >> de...@open-mpi.org <mailto:de...@open-mpi.org> >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >> <http://www.open-mpi.org/mailman/listinfo.cgi/devel> >> Link to this post: >> http://www.open-mpi.org/community/lists/devel/2015/10/18323.php >> <http://www.open-mpi.org/community/lists/devel/2015/10/18323.php> >> >> >> >> -- >> С Уважением, Поляков Артем Юрьевич >> Best regards, Artem Y. Polyakov >> _______________________________________________ >> devel mailing list >> de...@open-mpi.org <mailto:de...@open-mpi.org> >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >> <http://www.open-mpi.org/mailman/listinfo.cgi/devel> >> Link to this post: >> http://www.open-mpi.org/community/lists/devel/2015/11/18334.php >> <http://www.open-mpi.org/community/lists/devel/2015/11/18334.php> > > _______________________________________________ > devel mailing list > de...@open-mpi.org <mailto:de...@open-mpi.org> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > <http://www.open-mpi.org/mailman/listinfo.cgi/devel> > Link to this post: > http://www.open-mpi.org/community/lists/devel/2015/11/18335.php > <http://www.open-mpi.org/community/lists/devel/2015/11/18335.php> > > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2015/11/18336.php
