George,
i confirm there was a problem when running on an heterogeneous cluster,
this is now fixed in r32425.
i am not convinced i chose the most elegant way to achieve the desired
result ...
could you please double check this commit ?
Thanks,
Gilles
On 2014/08/02 0:14, George Bosilca wrote:
> Gilles,
>
> The design of the BTL move was to let the opal_process_name_t be agnostic to
> what is stored inside, and all accesses should be done through the provided
> accessors. Thus, big endian or little endian doesn't make a difference, as
> long as everything goes through the accessors.
>
> I'm skeptical about the support of heterogeneous environments in the current
> code, so I didn't pay much attention to handling the case in the TCP BTL. But
> in case we do care it is enough to make the 2 macros point to something
> meaningful instead of being empty (bswap_64 or something).
>
> George.
>
> On Aug 1, 2014, at 06:52 , Gilles Gouaillardet
> <gilles.gouaillar...@iferc.org> wrote:
>
>> George and Ralph,
>>
>> i am very confused whether there is an issue or not.
>>
>>
>> anyway, today Paul and i ran basic tests on big endian machines and did not
>> face any issue related to big endianness.
>>
>> so i made my homework, digged into the code, and basically,
>> opal_process_name_t is used as an orte_process_name_t.
>> for example, in ompi_proc_init :
>>
>> OMPI_CAST_ORTE_NAME(&proc->super.proc_name)->jobid =
>> OMPI_PROC_MY_NAME->jobid;
>> OMPI_CAST_ORTE_NAME(&proc->super.proc_name)->vpid = i;
>>
>> and with
>>
>> #define OMPI_CAST_ORTE_NAME(a) ((orte_process_name_t*)(a))
>>
>> so as long as an opal_process_name_t is used as an orte_process_name_t,
>> there is no problem,
>> regardless the endianness of the homogenous cluster we are running on.
>>
>> for the sake of readability (and for being pedantic too ;-) ) in r32357,
>> &proc_temp->super.proc_name
>> could be replaced with
>> OMPI_CAST_ORTE_NAME(&proc_temp->super.proc_name)
>>
>>
>>
>> That being said, in btl/tcp, i noticed :
>>
>> in mca_btl_tcp_component_recv_handler :
>>
>> opal_process_name_t guid;
>> [...]
>> /* recv the process identifier */
>> retval = recv(sd, (char *)&guid, sizeof(guid), 0);
>> if(retval != sizeof(guid)) {
>> CLOSE_THE_SOCKET(sd);
>> return;
>> }
>> OPAL_PROCESS_NAME_NTOH(guid);
>>
>> and in mca_btl_tcp_endpoint_send_connect_ack :
>>
>> /* send process identifier to remote endpoint */
>> opal_process_name_t guid = btl_proc->proc_opal->proc_name;
>>
>> OPAL_PROCESS_NAME_HTON(guid);
>> if(mca_btl_tcp_endpoint_send_blocking(btl_endpoint, &guid, sizeof(guid))
>> !=
>>
>> and with
>>
>> #define OPAL_PROCESS_NAME_NTOH(guid)
>> #define OPAL_PROCESS_NAME_HTON(guid)
>>
>>
>> i had no time yet to test yet, but for now, i can only suspect :
>> - there will be an issue with the tcp btl on an heterogeneous cluster
>> - for this case, the fix is to have a different version of the
>> OPAL_PROCESS_NAME_xTOy
>> on little endian arch if heterogeneous mode is supported.
>>
>>
>>
>> does that make sense ?
>>
>> Cheers,
>>
>> Gilles
>>
>>
>> On 2014/07/31 1:29, George Bosilca wrote:
>>> The underlying structure changed, so a little bit of fiddling is normal.
>>> Instead of using a field in the ompi_proc_t you are now using a field down
>>> in opal_proc_t, a field that simply cannot have the same type as before
>>> (orte_process_name_t).
>>>
>>> George.
>>>
>>>
>>>
>>> On Wed, Jul 30, 2014 at 12:19 PM, Ralph Castain <r...@open-mpi.org> wrote:
>>>
>>>> George - my point was that we regularly tested using the method in that
>>>> routine, and now we have to do something a little different. So it is an
>>>> "issue" in that we have to make changes across the code base to ensure we
>>>> do things the "new" way, that's all
>>>>
>>>> On Jul 30, 2014, at 9:17 AM, George Bosilca <bosi...@icl.utk.edu> wrote:
>>>>
>>>> No, this is not going to be an issue if the opal_identifier_t is used
>>>> correctly (aka only via the exposed accessors).
>>>>
>>>> George.
>>>>
>>>>
>>>>
>>>> On Wed, Jul 30, 2014 at 12:09 PM, Ralph Castain <r...@open-mpi.org> wrote:
>>>>
>>>>> Yeah, my fix won't work for big endian machines - this is going to be an
>>>>> issue across the code base now, so we'll have to troll and fix it. I was
>>>>> doing the minimal change required to fix the trunk in the meantime.
>>>>>
>>>>> On Jul 30, 2014, at 9:06 AM, George Bosilca <bosi...@icl.utk.edu> wrote:
>>>>>
>>>>> Yes. opal_process_name_t has basically no meaning by itself, it is a 64
>>>>> bits storage location used by the upper layer to save some local key that
>>>>> can be later used to extract information. Calling the OPAL level compare
>>>>> function might be a better fit there.
>>>>>
>>>>> George.
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Jul 30, 2014 at 11:50 AM, Gilles Gouaillardet <
>>>>> gilles.gouaillar...@gmail.com> wrote:
>>>>>
>>>>>> Ralph,
>>>>>>
>>>>>> was it really that simple ?
>>>>>>
>>>>>> proc_temp->super.proc_name has type opal_process_name_t :
>>>>>> typedef opal_identifier_t opal_process_name_t;
>>>>>> typedef uint64_t opal_identifier_t;
>>>>>>
>>>>>> *but*
>>>>>>
>>>>>> item_ptr->peer has type orte_process_name_t :
>>>>>> struct orte_process_name_t {
>>>>>> orte_jobid_t jobid;
>>>>>> orte_vpid_t vpid;
>>>>>> };
>>>>>>
>>>>>> bottom line, is r32357 still valid on a big endian arch ?
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Gilles
>>>>>>
>>>>>>
>>>>>> On Wed, Jul 30, 2014 at 11:49 PM, Ralph Castain <r...@open-mpi.org>
>>>>>> wrote:
>>>>>>
>>>>>>> I just fixed this one - all that was required was an ampersand as the
>>>>>>> name was being passed into the function instead of a pointer to the name
>>>>>>>
>>>>>>> r32357
>>>>>>>
>>>>>>> On Jul 30, 2014, at 7:43 AM, Gilles GOUAILLARDET <
>>>>>>> gilles.gouaillar...@gmail.com> wrote:
>>>>>>>
>>>>>>> Rolf,
>>>>>>>
>>>>>>> r32353 can be seen as a suspect...
>>>>>>> Even if it is correct, it might have exposed the bug discussed in #4815
>>>>>>> even more (e.g. we hit the bug 100% after the fix)
>>>>>>>
>>>>>>> does the attached patch to #4815 fixes the problem ?
>>>>>>>
>>>>>>> If yes, and if you see this issue as a showstopper, feel free to commit
>>>>>>> it and drop a note to #4815
>>>>>>> ( I am afk until tomorrow)
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Gilles
>>>>>>>
>>>>>>> Rolf vandeVaart <rvandeva...@nvidia.com> wrote:
>>>>>>>
>>>>>>> Just an FYI that my trunk version (r32355) does not work at all anymore
>>>>>>> if I do not include "--mca coll ^ml". Here is a stack trace from the
>>>>>>> ibm/pt2pt/send test running on a single node.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> (gdb) where
>>>>>>>
>>>>>>> #0 0x00007f6c0d1321d0 in ?? ()
>>>>>>>
>>>>>>> #1 <signal handler called>
>>>>>>>
>>>>>>> #2 0x00007f6c183abd52 in orte_util_compare_name_fields (fields=15
>>>>>>> '\017', name1=0x192350001, name2=0xbaf76c) at
>>>>>>> ../../orte/util/name_fns.c:522
>>>>>>>
>>>>>>> #3 0x00007f6c0bea17be in bcol_basesmuma_smcm_allgather_connection
>>>>>>> (sm_bcol_module=0x7f6bf3b68040, module=0xb3d200,
>>>>>>> peer_list=0x7f6c0c0a6748,
>>>>>>> back_files=0x7f6bf3ffd6c8,
>>>>>>>
>>>>>>> comm=0x6037a0, input=..., base_fname=0x7f6c0bea2606
>>>>>>> "sm_payload_mem_", map_all=false) at
>>>>>>> ../../../../../ompi/mca/bcol/basesmuma/bcol_basesmuma_smcm.c:237
>>>>>>>
>>>>>>> #4 0x00007f6c0be98307 in bcol_basesmuma_bank_init_opti
>>>>>>> (payload_block=0xbc0f60, data_offset=64, bcol_module=0x7f6bf3b68040,
>>>>>>> reg_data=0xba28c0)
>>>>>>>
>>>>>>> at
>>>>>>> ../../../../../ompi/mca/bcol/basesmuma/bcol_basesmuma_buf_mgmt.c:302
>>>>>>>
>>>>>>> #5 0x00007f6c0cced386 in mca_coll_ml_register_bcols
>>>>>>> (ml_module=0xba5c40) at
>>>>>>> ../../../../../ompi/mca/coll/ml/coll_ml_module.c:510
>>>>>>>
>>>>>>> #6 0x00007f6c0cced68f in ml_module_memory_initialization
>>>>>>> (ml_module=0xba5c40) at
>>>>>>> ../../../../../ompi/mca/coll/ml/coll_ml_module.c:558
>>>>>>>
>>>>>>> #7 0x00007f6c0ccf06b1 in ml_discover_hierarchy (ml_module=0xba5c40) at
>>>>>>> ../../../../../ompi/mca/coll/ml/coll_ml_module.c:1539
>>>>>>>
>>>>>>> #8 0x00007f6c0ccf4e0b in mca_coll_ml_comm_query (comm=0x6037a0,
>>>>>>> priority=0x7fffe7991b58) at
>>>>>>> ../../../../../ompi/mca/coll/ml/coll_ml_module.c:2963
>>>>>>>
>>>>>>> #9 0x00007f6c18cc5b09 in query_2_0_0 (component=0x7f6c0cf50940,
>>>>>>> comm=0x6037a0, priority=0x7fffe7991b58, module=0x7fffe7991b90)
>>>>>>>
>>>>>>> at ../../../../ompi/mca/coll/base/coll_base_comm_select.c:372
>>>>>>>
>>>>>>> #10 0x00007f6c18cc5ac8 in query (component=0x7f6c0cf50940,
>>>>>>> comm=0x6037a0, priority=0x7fffe7991b58, module=0x7fffe7991b90)
>>>>>>>
>>>>>>> at ../../../../ompi/mca/coll/base/coll_base_comm_select.c:355
>>>>>>>
>>>>>>> #11 0x00007f6c18cc59d2 in check_one_component (comm=0x6037a0,
>>>>>>> component=0x7f6c0cf50940, module=0x7fffe7991b90)
>>>>>>>
>>>>>>> at ../../../../ompi/mca/coll/base/coll_base_comm_select.c:317
>>>>>>>
>>>>>>> #12 0x00007f6c18cc5818 in check_components (components=0x7f6c18f46ef0,
>>>>>>> comm=0x6037a0) at
>>>>>>> ../../../../ompi/mca/coll/base/coll_base_comm_select.c:281
>>>>>>>
>>>>>>> #13 0x00007f6c18cbe3c9 in mca_coll_base_comm_select (comm=0x6037a0) at
>>>>>>> ../../../../ompi/mca/coll/base/coll_base_comm_select.c:117
>>>>>>>
>>>>>>> #14 0x00007f6c18c52301 in ompi_mpi_init (argc=1, argv=0x7fffe79924c8,
>>>>>>> requested=0, provided=0x7fffe79922e8) at
>>>>>>> ../../ompi/runtime/ompi_mpi_init.c:918
>>>>>>>
>>>>>>> #15 0x00007f6c18c86e92 in PMPI_Init (argc=0x7fffe799234c,
>>>>>>> argv=0x7fffe7992340) at pinit.c:84
>>>>>>>
>>>>>>> #16 0x0000000000401056 in main (argc=1, argv=0x7fffe79924c8) at
>>>>>>> send.c:32
>>>>>>>
>>>>>>> (gdb) up
>>>>>>>
>>>>>>> #1 <signal handler called>
>>>>>>>
>>>>>>> (gdb) up
>>>>>>>
>>>>>>> #2 0x00007f6c183abd52 in orte_util_compare_name_fields (fields=15
>>>>>>> '\017', name1=0x192350001, name2=0xbaf76c) at
>>>>>>> ../../orte/util/name_fns.c:522
>>>>>>>
>>>>>>> 522 if (name1->jobid < name2->jobid) {
>>>>>>>
>>>>>>> (gdb) print name1
>>>>>>>
>>>>>>> $1 = (const orte_process_name_t *) 0x192350001
>>>>>>>
>>>>>>> (gdb) print *name1
>>>>>>>
>>>>>>> Cannot access memory at address 0x192350001
>>>>>>>
>>>>>>> (gdb) print name2
>>>>>>>
>>>>>>> $2 = (const orte_process_name_t *) 0xbaf76c
>>>>>>>
>>>>>>> (gdb) print *name2
>>>>>>>
>>>>>>> $3 = {jobid = 2452946945, vpid = 1}
>>>>>>>
>>>>>>> (gdb)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: devel [mailto:devel-boun...@open-mpi.org
>>>>>>> <devel-boun...@open-mpi.org>] On Behalf Of Gilles
>>>>>>>
>>>>>>>> Gouaillardet
>>>>>>>> Sent: Wednesday, July 30, 2014 2:16 AM
>>>>>>>> To: Open MPI Developers
>>>>>>>> Subject: Re: [OMPI devel] trunk compilation errors in jenkins
>>>>>>>> George,
>>>>>>>> #4815 is indirectly related to the move :
>>>>>>>> in bcol/basesmuma, we used to compare ompi_process_name_t, and now
>>>>>>>> we (try to) compare an ompi_process_name_t and an opal_process_name_t
>>>>>>>> (which causes a glory SIGSEGV)
>>>>>>>> i proposed a temporary patch which is both broken and unelegant, could
>>>>>>> you
>>>>>>>
>>>>>>>> please advise a correct solution ?
>>>>>>>> Cheers,
>>>>>>>> Gilles
>>>>>>>> On 2014/07/27 7:37, George Bosilca wrote:
>>>>>>>>> If you have any issue with the move, I'll be happy to help and/or
>>>>>>> support
>>>>>>>
>>>>>>>> you on your last move toward a completely generic BTL. To facilitate
>>>>>>> your
>>>>>>>
>>>>>>>> work I exposed a minimalistic set of OMPI information at the OPAL
>>>>>>> level. Take
>>>>>>>
>>>>>>>> a look at opal/util/proc.h for more info, but please try not to expose
>>>>>>> more.
>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> devel mailing list
>>>>>>>> de...@open-mpi.org
>>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>>>> Link to this post: http://www.open-
>>>>>>> <http://www.open-mpi.org/community/lists/devel/2014/07/15348.php>
>>>>>>>
>>>>>>>> mpi.org/community/lists/devel/2014/07/15348.php
>>>>>>> <http://www.open-mpi.org/community/lists/devel/2014/07/15348.php>
>>>>>>> ------------------------------
>>>>>>> This email message is for the sole use of the intended recipient(s)
>>>>>>> and may contain confidential information. Any unauthorized review, use,
>>>>>>> disclosure or distribution is prohibited. If you are not the intended
>>>>>>> recipient, please contact the sender by reply email and destroy all
>>>>>>> copies
>>>>>>> of the original message.
>>>>>>> ------------------------------
>>>>>>> _______________________________________________
>>>>>>> devel mailing list
>>>>>>> de...@open-mpi.org
>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>>> Link to this post:
>>>>>>> http://www.open-mpi.org/community/lists/devel/2014/07/15355.php
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> devel mailing list
>>>>>>> de...@open-mpi.org
>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>>> Link to this post:
>>>>>>> http://www.open-mpi.org/community/lists/devel/2014/07/15356.php
>>>>>>>
>>>>>> _______________________________________________
>>>>>> devel mailing list
>>>>>> de...@open-mpi.org
>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>> Link to this post:
>>>>>> http://www.open-mpi.org/community/lists/devel/2014/07/15363.php
>>>>>>
>>>>> _______________________________________________
>>>>> devel mailing list
>>>>> de...@open-mpi.org
>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>> Link to this post:
>>>>> http://www.open-mpi.org/community/lists/devel/2014/07/15364.php
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> devel mailing list
>>>>> de...@open-mpi.org
>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>> Link to this post:
>>>>> http://www.open-mpi.org/community/lists/devel/2014/07/15365.php
>>>>>
>>>> _______________________________________________
>>>> devel mailing list
>>>> de...@open-mpi.org
>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>> Link to this post:
>>>> http://www.open-mpi.org/community/lists/devel/2014/07/15366.php
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> devel mailing list
>>>> de...@open-mpi.org
>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>> Link to this post:
>>>> http://www.open-mpi.org/community/lists/devel/2014/07/15367.php
>>>>
>>>
>>> _______________________________________________
>>> devel mailing list
>>> de...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>> Link to this post:
>>> http://www.open-mpi.org/community/lists/devel/2014/07/15368.php
>> _______________________________________________
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post:
>> http://www.open-mpi.org/community/lists/devel/2014/08/15446.php
>
>
>
> _______________________________________________
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/08/15454.php
