Are we maybe approaching this from the wrong direction? I ask because we had to
do some gyrations in the pmix framework to work around the difference in naming
schemes between OPAL and the rest of the code base, and now we have more
gyrations here.
Given that the MPI and RTE layers both rely on the structured form of the name,
what about if we just mimic that down in OPAL? I think we could perhaps do this
in a way that still allows someone to overlay it with a 64-bit unstructured
identifier if they want, but that would put the extra work on their side. In
other words, we make it easy to work with the other parts of our own code base,
acknowledging that those wanting to do something else may have to do some extra
work.
I ask because every resource manager out there assigns each process a jobid and
vpid in some form of integer format. So we have to absorb that information in
{jobid, vpid} format regardless of what we may want to do internally. What we
now have to do is immediately convert that into the unstructured form for OPAL
(where we take it in via PMI), then convert it back to structured form when
passing it up to ORTE so it can be handed to OMPI, and then convert it back to
unstructured form every time either OMPI or ORTE accesses the OPAL layer.
Seems awfully convoluted and error prone. Simplifying things for ourselves
might make more sense.
On Aug 6, 2014, at 1:21 PM, George Bosilca <bosi...@icl.utk.edu> wrote:
> Gilles,
>
> This looks right. It is really unfortunately that we have to change the
> definition of orte_process_name_t for big endian architectures, but I don't
> think there is a way around.
>
> Regarding your patch I have two comments:
> 1. There is a flagrant lack of comments ... especially on the ORTE side
> 2. at the OPAL level we are really implementing a htonll, and I really think
> we should stick to the POSIX prototype (aka. returning the changes value
> instead of doing things inplace).
>
> George.
>
>
>
> On Wed, Aug 6, 2014 at 7:02 AM, Gilles Gouaillardet
> <gilles.gouaillar...@iferc.org> wrote:
> Ralph and George,
>
> here is attached a patch that fixes the heterogeneous support without the
> abstraction violation.
>
> Cheers,
>
> Gilles
>
>
> On 2014/08/06 9:40, Gilles Gouaillardet wrote:
>> hummm
>>
>> i intentionally did not swap the two 32 bits (!)
>>
>> from the top level, what we have is :
>>
>> typedef struct {
>> union {
>> uint64_t opal;
>> struct {
>> uint32_t jobid;
>> uint32_t vpid;
>> } orte;
>> } meta_process_name_t;
>>
>> OPAL is agnostic about jobid and vpid.
>> jobid and vpid are set in ORTE/MPI and OPAL is used only
>> to transport the 64 bits
>> /* opal_process_name_t and orte_process_name_t are often casted into each
>> other */
>> at ORTE/MPI level, jobid and vpid are set individually
>> /* e.g. we do *not* do something like opal = jobid | (vpid<<32) */
>> this is why everything works fine on homogeneous clusters regardless
>> endianness.
>>
>> now in heterogeneous cluster, thing get a bit trickier ...
>>
>> i was initially unhappy with my commit and i think i found out why :
>> this is an abstraction violation !
>> the two 32 bits are not swapped by OPAL because this is what is expected by
>> the ORTE/OMPI.
>>
>> now i d like to suggest the following lightweight approach :
>>
>> at OPAL, use #if protected htonll/ntohll
>> (e.g. swap the two 32bits)
>>
>> do the trick at the ORTE level :
>>
>> simply replace
>>
>> struct orte_process_name_t {
>> orte_jobid_t jobid;
>> orte_vpid_t vpid;
>> };
>>
>> with
>>
>> #if OPAL_ENABLE_HETEROGENEOUS_SUPPORT && !defined(WORDS_BIGENDIAN)
>> struct orte_process_name_t {
>> orte_vpid_t vpid;
>> orte_jobid_t jobid;
>> };
>> #else
>> struct orte_process_name_t {
>> orte_jobid_t jobid;
>> orte_vpid_t vpid;
>> };
>> #endif
>>
>>
>> so we keep OPAL agnostic about how the uint64_t is really used at the upper
>> level.
>> an other option is to make OPAL aware of jobid and vpid but this is a bit
>> more heavyweight imho.
>>
>> i'll try this today and make sure it works.
>>
>> any thoughts ?
>>
>> Cheers,
>>
>> Gilles
>>
>>
>> On Wed, Aug 6, 2014 at 8:17 AM, Ralph Castain <r...@open-mpi.org> wrote:
>>
>>> Ah yes, so it is - sorry I missed that last test :-/
>>>
>>> On Aug 5, 2014, at 10:50 AM, George Bosilca <bosi...@icl.utk.edu> wrote:
>>>
>>> The code committed by Gilles is correctly protected for big endian (
>>> https://svn.open-mpi.org/trac/ompi/changeset/32425). I was merely
>>> pointing out that I think he should also swap the 2 32 bits in his
>>> implementation.
>>>
>>> George.
>>>
>>>
>>>
>>> On Tue, Aug 5, 2014 at 1:32 PM, Ralph Castain <r...@open-mpi.org> wrote:
>>>
>>>> On Aug 5, 2014, at 10:23 AM, George Bosilca <bosi...@icl.utk.edu> wrote:
>>>>
>>>> On Tue, Aug 5, 2014 at 1:15 PM, Ralph Castain <r...@open-mpi.org> wrote:
>>>>
>>>>> Hmmm...wouldn't that then require that you know (a) the other side is
>>>>> little endian, and (b) that you are on a big endian? Otherwise, you wind
>>>>> up
>>>>> with the same issue in reverse, yes?
>>>>>
>>>> This is similar to the 32 bits ntohl that we are using in other parts of
>>>> the project. Any little endian participant will do the conversion, while
>>>> every big endian participant will use an empty macro instead.
>>>>
>>>>
>>>>> In the ORTE methods, we explicitly set the fields (e.g., jobid =
>>>>> ntohl(remote-jobid)) to get around this problem. I missed that he did it
>>>>> by
>>>>> location instead of named fields - perhaps we should do that instead?
>>>>>
>>>> As soon as we impose the ORTE naming scheme at the OPAL level (aka. the
>>>> notion of jobid and vpid) this approach will become possible.
>>>>
>>>>
>>>> Not proposing that at all so long as the other method will work without
>>>> knowing the other side's endianness. Sounds like your approach should work
>>>> fine as long as Gilles adds a #if so big endian defines the macro away
>>>>
>>>>
>>>> George.
>>>>
>>>>
>>>>
>>>>> On Aug 5, 2014, at 10:06 AM, George Bosilca <bosi...@icl.utk.edu> wrote:
>>>>>
>>>>> Technically speaking, converting a 64 bits to a big endian
>>>>> representation requires the swap of the 2 32 bits parts. So the correct
>>>>> approach would have been:
>>>>> uint64_t htonll(uint64_t v)
>>>>> {
>>>>> return ((((uint64_t)ntohl(n)) << 32 | (uint64_t)ntohl(n >> 32));
>>>>> }
>>>>>
>>>>> George.
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Aug 5, 2014 at 5:52 AM, Ralph Castain <r...@open-mpi.org> wrote:
>>>>>
>>>>>> FWIW: that's exactly how we do it in ORTE
>>>>>>
>>>>>> On Aug 4, 2014, at 10:25 PM, Gilles Gouaillardet <
>>>>>> gilles.gouaillar...@iferc.org
>>>>>> > wrote:
>>>>>>
>>>>>> George,
>>>>>>
>>>>>> i confirm there was a problem when running on an heterogeneous cluster,
>>>>>> this is now fixed in r32425.
>>>>>>
>>>>>> i am not convinced i chose the most elegant way to achieve the desired
>>>>>> result ...
>>>>>> could you please double check this commit ?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Gilles
>>>>>>
>>>>>> On 2014/08/02 0:14, George Bosilca wrote:
>>>>>>
>>>>>> Gilles,
>>>>>>
>>>>>> The design of the BTL move was to let the opal_process_name_t be
>>>>>> agnostic to what is stored inside, and all accesses should be done
>>>>>> through the provided accessors. Thus, big endian or little endian
>>>>>> doesn’t make a difference, as long as everything goes through the
>>>>>> accessors.
>>>>>>
>>>>>> I’m skeptical about the support of heterogeneous environments in the
>>>>>> current code, so I didn’t pay much attention to handling the case in the
>>>>>> TCP BTL. But in case we do care it is enough to make the 2 macros point
>>>>>> to something meaningful instead of being empty (bswap_64 or something).
>>>>>>
>>>>>> George.
>>>>>>
>>>>>> On Aug 1, 2014, at 06:52 , Gilles Gouaillardet
>>>>>> <gilles.gouaillar...@iferc.org> <gilles.gouaillar...@iferc.org> wrote:
>>>>>>
>>>>>>
>>>>>> George and Ralph,
>>>>>>
>>>>>> i am very confused whether there is an issue or not.
>>>>>>
>>>>>>
>>>>>> anyway, today Paul and i ran basic tests on big endian machines and did
>>>>>> not face any issue related to big endianness.
>>>>>>
>>>>>> so i made my homework, digged into the code, and basically,
>>>>>> opal_process_name_t is used as an orte_process_name_t.
>>>>>> for example, in ompi_proc_init :
>>>>>>
>>>>>> OMPI_CAST_ORTE_NAME(&proc->super.proc_name)->jobid =
>>>>>> OMPI_PROC_MY_NAME->jobid;
>>>>>> OMPI_CAST_ORTE_NAME(&proc->super.proc_name)->vpid = i;
>>>>>>
>>>>>> and with
>>>>>>
>>>>>> #define OMPI_CAST_ORTE_NAME(a) ((orte_process_name_t*)(a))
>>>>>>
>>>>>> so as long as an opal_process_name_t is used as an orte_process_name_t,
>>>>>> there is no problem,
>>>>>> regardless the endianness of the homogenous cluster we are running on.
>>>>>>
>>>>>> for the sake of readability (and for being pedantic too ;-) ) in r32357,
>>>>>> &proc_temp->super.proc_name
>>>>>> could be replaced with
>>>>>> OMPI_CAST_ORTE_NAME(&proc_temp->super.proc_name)
>>>>>>
>>>>>>
>>>>>>
>>>>>> That being said, in btl/tcp, i noticed :
>>>>>>
>>>>>> in mca_btl_tcp_component_recv_handler :
>>>>>>
>>>>>> opal_process_name_t guid;
>>>>>> [...]
>>>>>> /* recv the process identifier */
>>>>>> retval = recv(sd, (char *)&guid, sizeof(guid), 0);
>>>>>> if(retval != sizeof(guid)) {
>>>>>> CLOSE_THE_SOCKET(sd);
>>>>>> return;
>>>>>> }
>>>>>> OPAL_PROCESS_NAME_NTOH(guid);
>>>>>>
>>>>>> and in mca_btl_tcp_endpoint_send_connect_ack :
>>>>>>
>>>>>> /* send process identifier to remote endpoint */
>>>>>> opal_process_name_t guid = btl_proc->proc_opal->proc_name;
>>>>>>
>>>>>> OPAL_PROCESS_NAME_HTON(guid);
>>>>>> if(mca_btl_tcp_endpoint_send_blocking(btl_endpoint, &guid,
>>>>>> sizeof(guid)) !=
>>>>>>
>>>>>> and with
>>>>>>
>>>>>> #define OPAL_PROCESS_NAME_NTOH(guid)
>>>>>> #define OPAL_PROCESS_NAME_HTON(guid)
>>>>>>
>>>>>>
>>>>>> i had no time yet to test yet, but for now, i can only suspect :
>>>>>> - there will be an issue with the tcp btl on an heterogeneous cluster
>>>>>> - for this case, the fix is to have a different version of the
>>>>>> OPAL_PROCESS_NAME_xTOy
>>>>>> on little endian arch if heterogeneous mode is supported.
>>>>>>
>>>>>>
>>>>>>
>>>>>> does that make sense ?
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Gilles
>>>>>>
>>>>>>
>>>>>> On 2014/07/31 1:29, George Bosilca wrote:
>>>>>>
>>>>>> The underlying structure changed, so a little bit of fiddling is normal.
>>>>>> Instead of using a field in the ompi_proc_t you are now using a field
>>>>>> down
>>>>>> in opal_proc_t, a field that simply cannot have the same type as before
>>>>>> (orte_process_name_t).
>>>>>>
>>>>>> George.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Jul 30, 2014 at 12:19 PM, Ralph Castain <r...@open-mpi.org>
>>>>>> <r...@open-mpi.org> wrote:
>>>>>>
>>>>>>
>>>>>> George - my point was that we regularly tested using the method in that
>>>>>> routine, and now we have to do something a little different. So it is an
>>>>>> "issue" in that we have to make changes across the code base to ensure we
>>>>>> do things the "new" way, that's all
>>>>>>
>>>>>> On Jul 30, 2014, at 9:17 AM, George Bosilca <bosi...@icl.utk.edu>
>>>>>> <bosi...@icl.utk.edu> wrote:
>>>>>>
>>>>>> No, this is not going to be an issue if the opal_identifier_t is used
>>>>>> correctly (aka only via the exposed accessors).
>>>>>>
>>>>>> George.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Jul 30, 2014 at 12:09 PM, Ralph Castain <r...@open-mpi.org>
>>>>>> <r...@open-mpi.org> wrote:
>>>>>>
>>>>>>
>>>>>> Yeah, my fix won't work for big endian machines - this is going to be an
>>>>>> issue across the code base now, so we'll have to troll and fix it. I was
>>>>>> doing the minimal change required to fix the trunk in the meantime.
>>>>>>
>>>>>> On Jul 30, 2014, at 9:06 AM, George Bosilca <bosi...@icl.utk.edu>
>>>>>> <bosi...@icl.utk.edu> wrote:
>>>>>>
>>>>>> Yes. opal_process_name_t has basically no meaning by itself, it is a 64
>>>>>> bits storage location used by the upper layer to save some local key that
>>>>>> can be later used to extract information. Calling the OPAL level compare
>>>>>> function might be a better fit there.
>>>>>>
>>>>>> George.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Jul 30, 2014 at 11:50 AM, Gilles Gouaillardet
>>>>>> <gilles.gouaillar...@gmail.com> wrote:
>>>>>>
>>>>>>
>>>>>> Ralph,
>>>>>>
>>>>>> was it really that simple ?
>>>>>>
>>>>>> proc_temp->super.proc_name has type opal_process_name_t :
>>>>>> typedef opal_identifier_t opal_process_name_t;
>>>>>> typedef uint64_t opal_identifier_t;
>>>>>>
>>>>>> *but*
>>>>>>
>>>>>> item_ptr->peer has type orte_process_name_t :
>>>>>> struct orte_process_name_t {
>>>>>> orte_jobid_t jobid;
>>>>>> orte_vpid_t vpid;
>>>>>> };
>>>>>>
>>>>>> bottom line, is r32357 still valid on a big endian arch ?
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Gilles
>>>>>>
>>>>>>
>>>>>> On Wed, Jul 30, 2014 at 11:49 PM, Ralph Castain <r...@open-mpi.org>
>>>>>> <r...@open-mpi.org>
>>>>>>
>>>>>> wrote:
>>>>>>
>>>>>>
>>>>>> I just fixed this one - all that was required was an ampersand as the
>>>>>> name was being passed into the function instead of a pointer to the name
>>>>>>
>>>>>> r32357
>>>>>>
>>>>>> On Jul 30, 2014, at 7:43 AM, Gilles GOUAILLARDET
>>>>>> <gilles.gouaillar...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Rolf,
>>>>>>
>>>>>> r32353 can be seen as a suspect...
>>>>>> Even if it is correct, it might have exposed the bug discussed in #4815
>>>>>> even more (e.g. we hit the bug 100% after the fix)
>>>>>>
>>>>>> does the attached patch to #4815 fixes the problem ?
>>>>>>
>>>>>> If yes, and if you see this issue as a showstopper, feel free to commit
>>>>>> it and drop a note to #4815
>>>>>> ( I am afk until tomorrow)
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Gilles
>>>>>>
>>>>>> Rolf vandeVaart <rvandeva...@nvidia.com> <rvandeva...@nvidia.com> wrote:
>>>>>>
>>>>>> Just an FYI that my trunk version (r32355) does not work at all anymore
>>>>>> if I do not include "--mca coll ^ml". Here is a stack trace from the
>>>>>> ibm/pt2pt/send test running on a single node.
>>>>>>
>>>>>>
>>>>>>
>>>>>> (gdb) where
>>>>>>
>>>>>> #0 0x00007f6c0d1321d0 in ?? ()
>>>>>>
>>>>>> #1 <signal handler called>
>>>>>>
>>>>>> #2 0x00007f6c183abd52 in orte_util_compare_name_fields (fields=15
>>>>>> '\017', name1=0x192350001, name2=0xbaf76c) at
>>>>>> ../../orte/util/name_fns.c:522
>>>>>>
>>>>>> #3 0x00007f6c0bea17be in bcol_basesmuma_smcm_allgather_connection
>>>>>> (sm_bcol_module=0x7f6bf3b68040, module=0xb3d200,
>>>>>> peer_list=0x7f6c0c0a6748,
>>>>>> back_files=0x7f6bf3ffd6c8,
>>>>>>
>>>>>> comm=0x6037a0, input=..., base_fname=0x7f6c0bea2606
>>>>>> "sm_payload_mem_", map_all=false) at
>>>>>> ../../../../../ompi/mca/bcol/basesmuma/bcol_basesmuma_smcm.c:237
>>>>>>
>>>>>> #4 0x00007f6c0be98307 in bcol_basesmuma_bank_init_opti
>>>>>> (payload_block=0xbc0f60, data_offset=64, bcol_module=0x7f6bf3b68040,
>>>>>> reg_data=0xba28c0)
>>>>>>
>>>>>> at
>>>>>> ../../../../../ompi/mca/bcol/basesmuma/bcol_basesmuma_buf_mgmt.c:302
>>>>>>
>>>>>> #5 0x00007f6c0cced386 in mca_coll_ml_register_bcols
>>>>>> (ml_module=0xba5c40) at
>>>>>> ../../../../../ompi/mca/coll/ml/coll_ml_module.c:510
>>>>>>
>>>>>> #6 0x00007f6c0cced68f in ml_module_memory_initialization
>>>>>> (ml_module=0xba5c40) at
>>>>>> ../../../../../ompi/mca/coll/ml/coll_ml_module.c:558
>>>>>>
>>>>>> #7 0x00007f6c0ccf06b1 in ml_discover_hierarchy (ml_module=0xba5c40) at
>>>>>> ../../../../../ompi/mca/coll/ml/coll_ml_module.c:1539
>>>>>>
>>>>>> #8 0x00007f6c0ccf4e0b in mca_coll_ml_comm_query (comm=0x6037a0,
>>>>>> priority=0x7fffe7991b58) at
>>>>>> ../../../../../ompi/mca/coll/ml/coll_ml_module.c:2963
>>>>>>
>>>>>> #9 0x00007f6c18cc5b09 in query_2_0_0 (component=0x7f6c0cf50940,
>>>>>> comm=0x6037a0, priority=0x7fffe7991b58, module=0x7fffe7991b90)
>>>>>>
>>>>>> at ../../../../ompi/mca/coll/base/coll_base_comm_select.c:372
>>>>>>
>>>>>> #10 0x00007f6c18cc5ac8 in query (component=0x7f6c0cf50940,
>>>>>> comm=0x6037a0, priority=0x7fffe7991b58, module=0x7fffe7991b90)
>>>>>>
>>>>>> at ../../../../ompi/mca/coll/base/coll_base_comm_select.c:355
>>>>>>
>>>>>> #11 0x00007f6c18cc59d2 in check_one_component (comm=0x6037a0,
>>>>>> component=0x7f6c0cf50940, module=0x7fffe7991b90)
>>>>>>
>>>>>> at ../../../../ompi/mca/coll/base/coll_base_comm_select.c:317
>>>>>>
>>>>>> #12 0x00007f6c18cc5818 in check_components (components=0x7f6c18f46ef0,
>>>>>> comm=0x6037a0) at
>>>>>> ../../../../ompi/mca/coll/base/coll_base_comm_select.c:281
>>>>>>
>>>>>> #13 0x00007f6c18cbe3c9 in mca_coll_base_comm_select (comm=0x6037a0) at
>>>>>> ../../../../ompi/mca/coll/base/coll_base_comm_select.c:117
>>>>>>
>>>>>> #14 0x00007f6c18c52301 in ompi_mpi_init (argc=1, argv=0x7fffe79924c8,
>>>>>> requested=0, provided=0x7fffe79922e8) at
>>>>>> ../../ompi/runtime/ompi_mpi_init.c:918
>>>>>>
>>>>>> #15 0x00007f6c18c86e92 in PMPI_Init (argc=0x7fffe799234c,
>>>>>> argv=0x7fffe7992340) at pinit.c:84
>>>>>>
>>>>>> #16 0x0000000000401056 in main (argc=1, argv=0x7fffe79924c8) at
>>>>>> send.c:32
>>>>>>
>>>>>> (gdb) up
>>>>>>
>>>>>> #1 <signal handler called>
>>>>>>
>>>>>> (gdb) up
>>>>>>
>>>>>> #2 0x00007f6c183abd52 in orte_util_compare_name_fields (fields=15
>>>>>> '\017', name1=0x192350001, name2=0xbaf76c) at
>>>>>> ../../orte/util/name_fns.c:522
>>>>>>
>>>>>> 522 if (name1->jobid < name2->jobid) {
>>>>>>
>>>>>> (gdb) print name1
>>>>>>
>>>>>> $1 = (const orte_process_name_t *) 0x192350001
>>>>>>
>>>>>> (gdb) print *name1
>>>>>>
>>>>>> Cannot access memory at address 0x192350001
>>>>>>
>>>>>> (gdb) print name2
>>>>>>
>>>>>> $2 = (const orte_process_name_t *) 0xbaf76c
>>>>>>
>>>>>> (gdb) print *name2
>>>>>>
>>>>>> $3 = {jobid = 2452946945, vpid = 1}
>>>>>>
>>>>>> (gdb)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: devel [mailto:devel-boun...@open-mpi.org
>>>>>> <devel-boun...@open-mpi.org>
>>>>>>
>>>>>>
>>>>>>
>>>>>> <devel-boun...@open-mpi.org> <devel-boun...@open-mpi.org>] On Behalf Of
>>>>>> Gilles
>>>>>>
>>>>>>
>>>>>> Gouaillardet
>>>>>> Sent: Wednesday, July 30, 2014 2:16 AM
>>>>>> To: Open MPI Developers
>>>>>> Subject: Re: [OMPI devel] trunk compilation errors in jenkins
>>>>>> George,
>>>>>> #4815 is indirectly related to the move :
>>>>>> in bcol/basesmuma, we used to compare ompi_process_name_t, and now
>>>>>> we (try to) compare an ompi_process_name_t and an opal_process_name_t
>>>>>> (which causes a glory SIGSEGV)
>>>>>> i proposed a temporary patch which is both broken and unelegant, could
>>>>>>
>>>>>> you
>>>>>>
>>>>>>
>>>>>> please advise a correct solution ?
>>>>>> Cheers,
>>>>>> Gilles
>>>>>> On 2014/07/27 7:37, George Bosilca wrote:
>>>>>>
>>>>>> If you have any issue with the move, I’ll be happy to help and/or
>>>>>>
>>>>>> support
>>>>>>
>>>>>>
>>>>>> you on your last move toward a completely generic BTL. To facilitate
>>>>>>
>>>>>> your
>>>>>>
>>>>>>
>>>>>> work I exposed a minimalistic set of OMPI information at the OPAL
>>>>>>
>>>>>> level. Take
>>>>>>
>>>>>>
>>>>>> a look at opal/util/proc.h for more info, but please try not to expose
>>>>>>
>>>>>> more.
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> devel mailing listde...@open-mpi.org
>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>> Link to this post: http://www.open-
>>>>>>
>>>>>> <http://www.open-mpi.org/community/lists/devel/2014/07/15348.php>
>>>>>> <http://www.open-mpi.org/community/lists/devel/2014/07/15348.php>
>>>>>>
>>>>>> mpi.org/community/lists/devel/2014/07/15348.php
>>>>>>
>>>>>> <http://www.open-mpi.org/community/lists/devel/2014/07/15348.php>
>>>>>> <http://www.open-mpi.org/community/lists/devel/2014/07/15348.php>
>>>>>>
>>>>>>
>>>>>> ------------------------------
>>>>>> This email message is for the sole use of the intended recipient(s)
>>>>>> and may contain confidential information. Any unauthorized review, use,
>>>>>> disclosure or distribution is prohibited. If you are not the intended
>>>>>> recipient, please contact the sender by reply email and destroy all
>>>>>> copies
>>>>>> of the original message.
>>>>>> ------------------------------
>>>>>> _______________________________________________
>>>>>> devel mailing listde...@open-mpi.org
>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>> Link to this
>>>>>> post:http://www.open-mpi.org/community/lists/devel/2014/07/15355.php
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> devel mailing listde...@open-mpi.org
>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>> Link to this
>>>>>> post:http://www.open-mpi.org/community/lists/devel/2014/07/15356.php
>>>>>>
>>>>>> _______________________________________________
>>>>>> devel mailing listde...@open-mpi.org
>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>> Link to this
>>>>>> post:http://www.open-mpi.org/community/lists/devel/2014/07/15363.php
>>>>>>
>>>>>> _______________________________________________
>>>>>> devel mailing listde...@open-mpi.org
>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>> Link to this
>>>>>> post:http://www.open-mpi.org/community/lists/devel/2014/07/15364.php
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> devel mailing listde...@open-mpi.org
>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>> Link to this
>>>>>> post:http://www.open-mpi.org/community/lists/devel/2014/07/15365.php
>>>>>>
>>>>>> _______________________________________________
>>>>>> devel mailing listde...@open-mpi.org
>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>> Link to this
>>>>>> post:http://www.open-mpi.org/community/lists/devel/2014/07/15366.php
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> devel mailing listde...@open-mpi.org
>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>> Link to this
>>>>>> post:http://www.open-mpi.org/community/lists/devel/2014/07/15367.php
>>>>>>
>>>>>> _______________________________________________
>>>>>> devel mailing listde...@open-mpi.org
>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>> Link to this post:
>>>>>> http://www.open-mpi.org/community/lists/devel/2014/07/15368.php
>>>>>>
>>>>>> _______________________________________________
>>>>>> devel mailing listde...@open-mpi.org
>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>> Link to this post:
>>>>>> http://www.open-mpi.org/community/lists/devel/2014/08/15446.php
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> devel mailing listde...@open-mpi.org
>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>> Link to this post:
>>>>>> http://www.open-mpi.org/community/lists/devel/2014/08/15454.php
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> devel mailing list
>>>>>> de...@open-mpi.org
>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>> Link to this post:
>>>>>> http://www.open-mpi.org/community/lists/devel/2014/08/15509.php
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> devel mailing list
>>>>>> de...@open-mpi.org
>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>> Link to this post:
>>>>>> http://www.open-mpi.org/community/lists/devel/2014/08/15514.php
>>>>>>
>>>>> _______________________________________________
>>>>> devel mailing list
>>>>> de...@open-mpi.org
>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>> Link to this post:
>>>>> http://www.open-mpi.org/community/lists/devel/2014/08/15518.php
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> devel mailing list
>>>>> de...@open-mpi.org
>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>> Link to this post:
>>>>> http://www.open-mpi.org/community/lists/devel/2014/08/15519.php
>>>>>
>>>> _______________________________________________
>>>> devel mailing list
>>>> de...@open-mpi.org
>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>> Link to this post:
>>>> http://www.open-mpi.org/community/lists/devel/2014/08/15520.php
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> devel mailing list
>>>> de...@open-mpi.org
>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>> Link to this post:
>>>> http://www.open-mpi.org/community/lists/devel/2014/08/15521.php
>>>>
>>> _______________________________________________
>>> devel mailing list
>>> de...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>> Link to this post:
>>> http://www.open-mpi.org/community/lists/devel/2014/08/15523.php
>>>
>>>
>>>
>>> _______________________________________________
>>> devel mailing list
>>> de...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>> Link to this post:
>>> http://www.open-mpi.org/community/lists/devel/2014/08/15526.php
>>>
>>
>>
>> _______________________________________________
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>
>> Link to this post:
>> http://www.open-mpi.org/community/lists/devel/2014/08/15527.php
>
>
> _______________________________________________
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/08/15529.php
>
> _______________________________________________
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/08/15530.php
