Re: [OMPI devel] Master hangs in opal_fifo test

[Gilles Gouaillardet]

Nathan,

imho, this is a compiler bug and only two versions are affected :
- intel icc 14.0.0.080 (aka 2013sp1)
- intel icc 14.0.1.106 (aka 2013sp1u1)
/* note the bug only occurs with -O1 and higher optimization levels */

here is attached a simple reproducer

a simple workaround is to configure with ac_cv_type___int128=0

Cheers,

Gilles

On 2015/02/04 4:17, Nathan Hjelm wrote:
> Thats the second report involving icc 14. I will dig into this later
> this week.
>
> -Nathan
>
> On Mon, Feb 02, 2015 at 11:03:41PM -0800, Paul Hargrove wrote:
>>I have seen opal_fifo hang on 2 distinct systems
>> + Linux/ppc32 with xlc-11.1
>> + Linux/x86-64 with icc-14.0.1.106
>>I have no explanation to offer for either hang.
>>No "weird" configure options were passed to either.
>>-Paul
>>--
>>Paul H. Hargrove  phhargr...@lbl.gov
>>Computer Languages & Systems Software (CLaSS) Group
>>Computer Science Department   Tel: +1-510-495-2352
>>Lawrence Berkeley National Laboratory Fax: +1-510-486-6900
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/devel/2015/02/16911.php
>
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2015/02/16920.php

/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil -*- */
#include 
#include 

union opal_counted_pointer_t {
struct {
uint64_t counter;
uint64_t item;
} data;
__int128_t value;
};
typedef union opal_counted_pointer_t opal_counted_pointer_t;

int main (int argc, char *argv[]) {
volatile opal_counted_pointer_t a;
opal_counted_pointer_t b;

a.data.counter = 0;
a.data.item = 0x1234567890ABCDEF;

b.data.counter = a.data.counter;
b.data.item = a.data.item;

/* bozo checks */
assert(16 == sizeof(opal_counted_pointer_t));
assert(a.data.counter == b.data.counter);
assert(a.data.item == b.data.item);
/* 
 * following assert fails on buggy compilers
 * so far, with icc -o conftest conftest.c 
 *  - intel icc 14.0.0.080 (aka 2013sp1)
 *  - intel icc 14.0.1.106 (aka 2013sp1u1)
 * older and more recents compilers work fine
 * buggy compilers work also fine but only with -O0
 */
assert(a.value == b.value);
return 0;
}

Re: [OMPI devel] Master hangs in opal_LIFO test

[Gilles Gouaillardet]

Paul,

George and i were able to reproduce this issue with icc 14.0 but not
with icc 14.3 and later

i am trying to see how the difference/bug could be automatically handled

Cheers,

Gilles

On 2015/02/03 16:18, Paul Hargrove wrote:
> CORRECTION:
>
> It is the opal_lifo (not fifo) test which hung on both systems.
>
> -Paul
>
> On Mon, Feb 2, 2015 at 11:03 PM, Paul Hargrove  wrote:
>
>> I have seen opal_fifo hang on 2 distinct systems
>>  + Linux/ppc32 with xlc-11.1
>>  + Linux/x86-64 with icc-14.0.1.106
>>
>> I have no explanation to offer for either hang.
>> No "weird" configure options were passed to either.
>>
>> -Paul
>>
>> --
>> Paul H. Hargrove  phhargr...@lbl.gov
>> Computer Languages & Systems Software (CLaSS) Group
>> Computer Science Department   Tel: +1-510-495-2352
>> Lawrence Berkeley National Laboratory Fax: +1-510-486-6900
>>
>
>
>
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2015/02/16913.php

Re: [OMPI devel] Great meeting!

[Gilles Gouaillardet]

Hi Jeff,

let me update the --with-threads configure option.

it has been removed from the master :

commit 7a55d49ca78bcc157749c04027515f12b026ec33
Author: Gilles Gouaillardet <gilles.gouaillar...@iferc.org>
List-Post: devel@lists.open-mpi.org
Date:   Tue Oct 21 19:13:19 2014 +0900

configury: per RFC, remove the --with-threads option


i did not think it was appropriate to remove this option from the v1.8
series (e.g. do not "break" configure command line), so i did not make a PR.
but if there is a consensus --with-threads can be safely removed from
the v1.8 series, i ll be happy to do the backport and make a PR.

Cheers,

Gilles

On 2015/01/30 6:20, Jeff Squyres (jsquyres) wrote:
> Thank you all who traveled and/or showed up on a webex this week for the OMPI 
> developer's meeting.  We discussed a zillion things and made much forward 
> progress.  The wiki with a summary of discussed/decided things is here:
>
> https://github.com/open-mpi/ompi/wiki/Meeting-2015-01
>
> The major agenda item for next Tuesday's call will be to go through these 
> items and answer any questions for people who weren't here.
>

Re: [OMPI devel] One sided tests

[Gilles Gouaillardet]

George,

a tentative fix is available at https://github.com/open-mpi/ompi/pull/355

i asked Nathan to review it before it lands into the master

Cheers,

Gilles

On 2015/01/22 7:08, George Bosilca wrote:
> Current trunk compiled with any compiler (gcc or icc) fails the one sided
> tests from mpi_test_suite. It deadlocks in a fetch.
>
>   George.
>
>
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2015/01/16813.php

Re: [OMPI devel] btl_openib.c:1200: mca_btl_openib_alloc: Assertion `qp != 255' failed

[Gilles Gouaillardet]

Jeff,

There are two things here :

1) ompi was crashing, and this was fixed (now use type size instead of
type extent to figure out whether
the btl should try to send the message inline or not). and yes, George
is right (e.g. use size instead of extent
or both size and extent)

2) the mpi_test_suite uses a weird type (e.g. artificially send 20k
integers to the wire when sending one
 would produce the very same result)
i briefly checked the mpi_test_suite source code, and the weird type is
send/recv with buffers whose size
is one element.
i can only guess the authors wanted to send a large message to the wire
(e.g. create traffic) without pointless
large memory allocation.
at this stage, i am tempted to conclude the authors did what they intended.

Cheers,

Gilles

On 2015/01/21 3:00, Jeff Squyres (jsquyres) wrote:
> George is right -- Gilles: was this the correct solution?
>
> Put differently: the extent of the 20K vector created below is 4 (bytes).
>
>
>
>> On Jan 19, 2015, at 2:39 AM, George Bosilca <bosi...@icl.utk.edu> wrote:
>>
>> Btw,
>>
>> MPI_Type_hvector(2, 1, 0, MPI_INT, );
>>
>> Is just a weird datatype. Because the stride is 0, this datatype a memory 
>> layout that includes 2 times the same int. I'm not sure this was indeed 
>> intended...
>>
>>   George.
>>
>>
>> On Mon, Jan 19, 2015 at 12:17 AM, Gilles Gouaillardet 
>> <gilles.gouaillar...@iferc.org> wrote:
>> Adrian,
>>
>> i just fixed this in the master
>> (https://github.com/open-mpi/ompi/commit/d14daf40d041f7a0a8e9d85b3bfd5eb570495fd2)
>>
>> the root cause is a corner case was not handled correctly :
>>
>> MPI_Type_hvector(2, 1, 0, MPI_INT, );
>>
>> type has extent = 4 *but* size = 8
>> ob1 used to test only the extent to determine whether the message should
>> be sent inlined or not
>> extent <= 256 means try to send the message inline
>> that meant a fragment of size 8 (which is greater than 65536 e.g.
>> max default size for IB) was allocated,
>> and that failed.
>>
>> now both extent and size are tested, so the message is not sent inline,
>> and it just works.
>>
>> Cheers,
>>
>> Gilles
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/devel/2015/01/16798.php
>>
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/devel/2015/01/16801.php
>

Re: [OMPI devel] Failures

[Gilles Gouaillardet]

George,

i was able to reproduce the hang with intel compiler 14.0.0
but i am still unable to reproduce it with intel compiler 14.3

i was not able to understand where the issue come from, so
i could not create an appropriate test in configure

at this stage, i can only recommend you update your compiler version


Cheers,

Gilles

On 2015/01/17 0:19, George Bosilca wrote:
> Your patch solve the issue with opal_tree. The opal_lifo remains broken.
>
>   George.
>
>
> On Fri, Jan 16, 2015 at 5:12 AM, Gilles Gouaillardet <
> gilles.gouaillar...@iferc.org> wrote:
>
>>  George,
>>
>> i pushed
>> https://github.com/open-mpi/ompi/commit/ac16970d21d21f529f1ec01ebe0520843227475b
>> in order to get the intel compiler work with ompi
>>
>> Cheers,
>>
>> Gilles
>>
>>
>> On 2015/01/16 17:29, Gilles Gouaillardet wrote:
>>
>> George,
>>
>> i was unable to reproduce the hang with icc 14.0.3.174 and greater on a
>> RHEL6 like distro.
>>
>> i was able to reproduce the opal_tree failure and found two possible
>> workarounds :
>> a) manually compile opal/class/opal_tree.lo *without* the
>> -finline-functions flag
>> b) update deserialize_add_tree_item and declare curr_delim as volatile
>> char * (see the patch below)
>>
>> this function is recursive, and the compiler could generate some
>> incorrect code.
>>
>> Cheers,
>>
>> Gilles
>>
>> diff --git a/opal/class/opal_tree.c b/opal/class/opal_tree.c
>> index e8964e0..492e8dc 100644
>> --- a/opal/class/opal_tree.c
>> +++ b/opal/class/opal_tree.c
>> @@ -465,7 +465,7 @@ int opal_tree_serialize(opal_tree_item_t
>> *start_item, opal_buffer_t *buffer)
>>  static int deserialize_add_tree_item(opal_buffer_t *data,
>>   opal_tree_item_t *parent_item,
>>   opal_tree_item_deserialize_fn_t
>> deserialize,
>> - char *curr_delim,
>> + volatile char *curr_delim,
>>   int depth)
>>  {
>>  int idx = 1, rc;
>>
>> On 2015/01/16 8:57, George Bosilca wrote:
>>
>>  Today's trunk compiled with icc fails to complete the check on 2 tests:
>> opal_lifo and opal_tree.
>>
>> For opal_tree the output is:
>> OPAL dss:unpack: got type 9 when expecting type 3
>>  Failure :  failed tree deserialization size compare
>> SUPPORT: OMPI Test failed: opal_tree_t (1 of 12 failed)
>>
>> and opal_lifo gets stuck forever in the single threaded call to thread_test
>> in a 128 bits atomic CAS. Unfortunately I lack the time to dig deep enough
>> to see what is the root cause, but a quick look at the opal_config.h file
>> indicates that our configure detects that __int128 is a supported type when
>> it should not be.
>>
>>   George
>>
>> Open MPI git d13c14e configured with --enable-debug
>> icc (ICC) 14.0.0 20130728
>>
>>
>>
>> ___
>> devel mailing listde...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/devel/2015/01/16789.php
>>
>>
>>
>> ___
>> devel mailing listde...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/devel/2015/01/16790.php
>>
>>
>>
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post:
>> http://www.open-mpi.org/community/lists/devel/2015/01/16791.php
>>
>
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2015/01/16794.php

Re: [OMPI devel] btl_openib.c:1200: mca_btl_openib_alloc: Assertion `qp != 255' failed

[Gilles Gouaillardet]

Adrian,

about the
"[n050409][[36216,1],1][btl_openib_xrc.c:58:mca_btl_openib_xrc_check_api] XRC
error: bad XRC API (require XRC from OFED pre 3.12). " message.

this means ompi was built on a system with OFED 3.12 or greater, and you
are running on a system with an earlier OFED release.

please not Jeff recently pushed a patch related to that and this message
might be a false positive.

Cheers,

Gilles

On 2015/01/19 14:17, Gilles Gouaillardet wrote:
> Adrian,
>
> i just fixed this in the master
> (https://github.com/open-mpi/ompi/commit/d14daf40d041f7a0a8e9d85b3bfd5eb570495fd2)
>
> the root cause is a corner case was not handled correctly :
>
> MPI_Type_hvector(2, 1, 0, MPI_INT, );
>
> type has extent = 4 *but* size = 8
> ob1 used to test only the extent to determine whether the message should
> be sent inlined or not
> extent <= 256 means try to send the message inline
> that meant a fragment of size 8 (which is greater than 65536 e.g.
> max default size for IB) was allocated,
> and that failed.
>
> now both extent and size are tested, so the message is not sent inline,
> and it just works.
>
> Cheers,
>
> Gilles
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2015/01/16798.php

Re: [OMPI devel] btl_openib.c:1200: mca_btl_openib_alloc: Assertion `qp != 255' failed

[Gilles Gouaillardet]

Adrian,

i just fixed this in the master
(https://github.com/open-mpi/ompi/commit/d14daf40d041f7a0a8e9d85b3bfd5eb570495fd2)

the root cause is a corner case was not handled correctly :

MPI_Type_hvector(2, 1, 0, MPI_INT, );

type has extent = 4 *but* size = 8
ob1 used to test only the extent to determine whether the message should
be sent inlined or not
extent <= 256 means try to send the message inline
that meant a fragment of size 8 (which is greater than 65536 e.g.
max default size for IB) was allocated,
and that failed.

now both extent and size are tested, so the message is not sent inline,
and it just works.

Cheers,

Gilles

Re: [OMPI devel] pthreads (was: Re: RFC: remove --disable-smp-locks)

[Gilles Gouaillardet]

Folks,

i pushed two commits in order to remove the --with-threads option and
the dead code :

commit 7a55d49ca78bcc157749c04027515f12b026ec33
Author: Gilles Gouaillardet <gilles.gouaillar...@iferc.org>
List-Post: devel@lists.open-mpi.org
Date:   Tue Oct 21 19:13:19 2014 +0900

configury: per RFC, remove the --with-threads option


commit 661c35ca677512129cef9bca1bbbf5b5b71d951b
Author: Gilles Gouaillardet <gilles.gouaillar...@iferc.org>
List-Post: devel@lists.open-mpi.org
Date:   Tue Oct 21 19:49:58 2014 +0900

cleanup dead code caused by the removal of the --with-threads
configure option


i did not remove the opal abstraction layer (e.g. opal_thread_t and
friends) since it
is not only an abstraction layer, but it brings additional features.

Cheers,

Gilles

On 2015/01/08 1:50, Ralph Castain wrote:
> See note on other thread as to why we made the pthread decision
>
>> On Jan 7, 2015, at 8:30 AM, Jeff Squyres (jsquyres) <jsquy...@cisco.com> 
>> wrote:
>>
>> Ok.  Then I'm good with Gilles' plan.
>>
>> Anyone else?
>>
>>
>> On Jan 7, 2015, at 11:29 AM, Nathan Hjelm <hje...@lanl.gov> wrote:
>>
>>> Yes, we decided some time back that pthreads is a minimum requirement
>>> for Open MPI.
>>>
>>> -Nathan
>>>
>>> On Wed, Jan 07, 2015 at 04:26:01PM +, Jeff Squyres (jsquyres) wrote:
>>>> On Jan 7, 2015, at 11:22 AM, Gilles Gouaillardet 
>>>> <gilles.gouaillar...@gmail.com> wrote:
>>>>
>>>>> Valid options are :
>>>>> --with-threads e.g. --with-threads=posix e.g. default
>>>>> And
>>>>> --with-threads=no
>>>>>
>>>>> Except configure will explicitly fail if --with-threads=no is used
>>>> Which is the moral equivalent of not having this option.  :-)  (which I 
>>>> think is your point :-) )
>>>>
>>>>> So bottom line, pthreads and pthreads only are usable
>>>> But my question remains: we all decided that OMPI will *require* pthreads, 
>>>> right?  (i.e., configure will fail if pthreads are not available)
>>>>
>>>> I am being pedantic here, I know -- but it's slightly different than what 
>>>> you're saying, and this threading stuff is already quite confusing...
>>>>
>>>>
>>>>> Cheers,
>>>>>
>>>>> Gilles 
>>>>>
>>>>> "Jeff Squyres (jsquyres)" <jsquy...@cisco.com>さんのメール:
>>>>>> On Jan 7, 2015, at 4:25 AM, Gilles Gouaillardet 
>>>>>> <gilles.gouaillar...@iferc.org> wrote:
>>>>>>
>>>>>>> Talking about thread support ...
>>>>>>>
>>>>>>> i made a RFC several monthes ago in order to remove the
>>>>>>> --with-threads option from configure
>>>>>>>
>>>>>>> /* ompi requires pthreads, no more, no less */
>>>>>> Did we decide this?  (that OMPI *requires* pthreads)
>>>>>>
>>>>>> I *think* we did.  But I just want to make sure that my (terrible) 
>>>>>> memory is correct...
>>>>>>
>>>>>>> it was accepted, but i could not find the time to implement it ...
>>>>>>>
>>>>>>> basically, i can see three steps :
>>>>>>>
>>>>>>> 1) remove the --with-threads option from configure, check for pthreads, 
>>>>>>> and set the
>>>>>>> OPAL_HAVE_POSIX_THREADS macro to 1
>>>>>> Sounds good.
>>>>>>
>>>>>>> 2) step 1) + remove #ifdef OPAL_HAVE_POSIX_THREADS and remove dead code
>>>>>>> (e.g. #ifndef OPAL_HAVE_POSIX_THREADS)
>>>>>> Also make configure fail if pthreads are not available.
>>>>>>
>>>>>>> 3) step 1) + step 2) + remove the OPAL thread abstraction layer
>>>>>>>
>>>>>>> is it a good idea to implement steps 2) and 3) ?
>>>>>>> i mean, if there is a chance we might support an other threading model 
>>>>>>> in the future,
>>>>>>> it might be better to keep some dead code for the time being.
>>>>>> I think the consensus was that pthreads are fine for the foreseeable 
>>>>>> future.  If we need to re-add the threading abstraction layer, it's 
>>>>>> annoying, but not difficult.  Might as well simplify what we have, since 
>>>>&g

Re: [OMPI devel] Failures

[Gilles Gouaillardet]

George,

i pushed
https://github.com/open-mpi/ompi/commit/ac16970d21d21f529f1ec01ebe0520843227475b
in order to get the intel compiler work with ompi

Cheers,

Gilles

On 2015/01/16 17:29, Gilles Gouaillardet wrote:
> George,
>
> i was unable to reproduce the hang with icc 14.0.3.174 and greater on a
> RHEL6 like distro.
>
> i was able to reproduce the opal_tree failure and found two possible
> workarounds :
> a) manually compile opal/class/opal_tree.lo *without* the
> -finline-functions flag
> b) update deserialize_add_tree_item and declare curr_delim as volatile
> char * (see the patch below)
>
> this function is recursive, and the compiler could generate some
> incorrect code.
>
> Cheers,
>
> Gilles
>
> diff --git a/opal/class/opal_tree.c b/opal/class/opal_tree.c
> index e8964e0..492e8dc 100644
> --- a/opal/class/opal_tree.c
> +++ b/opal/class/opal_tree.c
> @@ -465,7 +465,7 @@ int opal_tree_serialize(opal_tree_item_t
> *start_item, opal_buffer_t *buffer)
>  static int deserialize_add_tree_item(opal_buffer_t *data,
>   opal_tree_item_t *parent_item,
>   opal_tree_item_deserialize_fn_t
> deserialize,
> - char *curr_delim,
> + volatile char *curr_delim,
>   int depth)
>  {
>  int idx = 1, rc;
>
> On 2015/01/16 8:57, George Bosilca wrote:
>> Today's trunk compiled with icc fails to complete the check on 2 tests:
>> opal_lifo and opal_tree.
>>
>> For opal_tree the output is:
>> OPAL dss:unpack: got type 9 when expecting type 3
>>  Failure :  failed tree deserialization size compare
>> SUPPORT: OMPI Test failed: opal_tree_t (1 of 12 failed)
>>
>> and opal_lifo gets stuck forever in the single threaded call to thread_test
>> in a 128 bits atomic CAS. Unfortunately I lack the time to dig deep enough
>> to see what is the root cause, but a quick look at the opal_config.h file
>> indicates that our configure detects that __int128 is a supported type when
>> it should not be.
>>
>>   George
>>
>> Open MPI git d13c14e configured with --enable-debug
>> icc (ICC) 14.0.0 20130728
>>
>>
>>
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/devel/2015/01/16789.php
>
>
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2015/01/16790.php

Re: [OMPI devel] Failures

[Gilles Gouaillardet]

George,

i was unable to reproduce the hang with icc 14.0.3.174 and greater on a
RHEL6 like distro.

i was able to reproduce the opal_tree failure and found two possible
workarounds :
a) manually compile opal/class/opal_tree.lo *without* the
-finline-functions flag
b) update deserialize_add_tree_item and declare curr_delim as volatile
char * (see the patch below)

this function is recursive, and the compiler could generate some
incorrect code.

Cheers,

Gilles

diff --git a/opal/class/opal_tree.c b/opal/class/opal_tree.c
index e8964e0..492e8dc 100644
--- a/opal/class/opal_tree.c
+++ b/opal/class/opal_tree.c
@@ -465,7 +465,7 @@ int opal_tree_serialize(opal_tree_item_t
*start_item, opal_buffer_t *buffer)
 static int deserialize_add_tree_item(opal_buffer_t *data,
  opal_tree_item_t *parent_item,
  opal_tree_item_deserialize_fn_t
deserialize,
- char *curr_delim,
+ volatile char *curr_delim,
  int depth)
 {
 int idx = 1, rc;

On 2015/01/16 8:57, George Bosilca wrote:
> Today's trunk compiled with icc fails to complete the check on 2 tests:
> opal_lifo and opal_tree.
>
> For opal_tree the output is:
> OPAL dss:unpack: got type 9 when expecting type 3
>  Failure :  failed tree deserialization size compare
> SUPPORT: OMPI Test failed: opal_tree_t (1 of 12 failed)
>
> and opal_lifo gets stuck forever in the single threaded call to thread_test
> in a 128 bits atomic CAS. Unfortunately I lack the time to dig deep enough
> to see what is the root cause, but a quick look at the opal_config.h file
> indicates that our configure detects that __int128 is a supported type when
> it should not be.
>
>   George
>
> Open MPI git d13c14e configured with --enable-debug
> icc (ICC) 14.0.0 20130728
>
>
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2015/01/16789.php

Re: [OMPI devel] pthreads (was: Re: RFC: remove --disable-smp-locks)

[Gilles Gouaillardet]

Valid options are :
--with-threads e.g. --with-threads=posix e.g. default
And
--with-threads=no

Except configure will explicitly fail if --with-threads=no is used

So bottom line, pthreads and pthreads only are usable

Cheers,

Gilles 

"Jeff Squyres (jsquyres)" <jsquy...@cisco.com>さんのメール:
>On Jan 7, 2015, at 4:25 AM, Gilles Gouaillardet 
><gilles.gouaillar...@iferc.org> wrote:
>
>> Talking about thread support ...
>> 
>> i made a RFC several monthes ago in order to remove the
>> --with-threads option from configure
>> 
>> /* ompi requires pthreads, no more, no less */
>
>Did we decide this?  (that OMPI *requires* pthreads)
>
>I *think* we did.  But I just want to make sure that my (terrible) memory is 
>correct...
>
>> it was accepted, but i could not find the time to implement it ...
>> 
>> basically, i can see three steps :
>> 
>> 1) remove the --with-threads option from configure, check for pthreads, and 
>> set the
>> OPAL_HAVE_POSIX_THREADS macro to 1
>
>Sounds good.
>
>> 2) step 1) + remove #ifdef OPAL_HAVE_POSIX_THREADS and remove dead code
>> (e.g. #ifndef OPAL_HAVE_POSIX_THREADS)
>
>Also make configure fail if pthreads are not available.
>
>> 3) step 1) + step 2) + remove the OPAL thread abstraction layer
>> 
>> is it a good idea to implement steps 2) and 3) ?
>> i mean, if there is a chance we might support an other threading model in 
>> the future,
>> it might be better to keep some dead code for the time being.
>
>I think the consensus was that pthreads are fine for the foreseeable future.  
>If we need to re-add the threading abstraction layer, it's annoying, but not 
>difficult.  Might as well simplify what we have, since there's no other 
>threading system on the horizon that we need to worry about.
>
>-- 
>Jeff Squyres
>jsquy...@cisco.com
>For corporate legal information go to: 
>http://www.cisco.com/web/about/doing_business/legal/cri/
>
>___
>devel mailing list
>de...@open-mpi.org
>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>Link to this post: 
>http://www.open-mpi.org/community/lists/devel/2015/01/16750.php

Re: [OMPI devel] RFC: remove --disable-smp-locks

[Gilles Gouaillardet]

Talking about thread support ...

i made a RFC several monthes ago in order to remove the
--with-threads option from configure

/* ompi requires pthreads, no more, no less */

it was accepted, but i could not find the time to implement it ...

basically, i can see three steps :

1) remove the --with-threads option from configure, check for pthreads,
and set the
OPAL_HAVE_POSIX_THREADS macro to 1

2) step 1) + remove #ifdef OPAL_HAVE_POSIX_THREADS and remove dead code
(e.g. #ifndef OPAL_HAVE_POSIX_THREADS)

3) step 1) + step 2) + remove the OPAL thread abstraction layer

is it a good idea to implement steps 2) and 3) ?
i mean, if there is a chance we might support an other threading model
in the future,
it might be better to keep some dead code for the time being.

any thoughts ?

Cheers,

Gilles

On 2015/01/07 8:16, George Bosilca wrote:
> On Tue, Jan 6, 2015 at 4:25 PM, Jeff Squyres (jsquyres) 
> wrote:
>
>> My enthusiasm for this was primarily because I thought we had talked about
>> exactly this issue before (at the last meeting in Chicago?), and decided
>> that the option is useless -- in part, because it always *must* be enabled
>> for shared memory correctness.
>>
>> Is that incorrect?
>>
> This is correct. We need the memory fences and atomic operations for shared
> memory in all cases. When thread support is enabled we also need them in
> various other places. However, this option also turns on the lock prefix
> for the atomic operations, forcing them to always be atomic. I am not sure
> that this has no unexpected side-effects on the code.
>
>   George.
>
>
>
>>
>> On Jan 6, 2015, at 4:12 PM, George Bosilca  wrote:
>>
>>> Successive alteration of the build system made this option less relevant
>> and especially less meaningful. However, while removing it sounds like a
>> desirable cleanup, we have to keep in mind that this will enable all locks
>> and all memory barriers even in cases where they are not necessary (via
>> OPAL_WANT_SMP_LOCKS).
>>> Thus, I do not share the enthusiasm of the others. I would prefer to see
>> an evaluation of the impact on performance, a patch and a little bit more
>> than 1/2 a day to react to it (the proposed deadline seems to be today
>> January 6th) before such a drastic change.
>>>   George.
>>>
>>>
>>> On Tue, Jan 6, 2015 at 12:05 PM, Ralph Castain  wrote:
>>> +1
>>>
 On Jan 6, 2015, at 9:04 AM, Jeff Squyres (jsquyres) <
>> jsquy...@cisco.com> wrote:
 +1

 On Jan 6, 2015, at 11:55 AM, Howard Pritchard 
>> wrote:
> I agree.  Please remove this config option.
>
> 2015-01-06 9:44 GMT-07:00 Nathan Hjelm :
>
> What: Remove the --disable-smp-locks configure option from master.
>
> Why: Use of this option produces incorrect results/undefined behavior
> when any shared memory BTL is in use. Since BTL usage is enabled even
> when using cm for point-to-point this option can never be safely used.
>
> When: Thurs, Jan 6, 2015
>
> -Nathan
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
>> http://www.open-mpi.org/community/lists/devel/2015/01/16736.php
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
>> http://www.open-mpi.org/community/lists/devel/2015/01/16737.php

 --
 Jeff Squyres
 jsquy...@cisco.com
 For corporate legal information go to:
>> http://www.cisco.com/web/about/doing_business/legal/cri/
 ___
 devel mailing list
 de...@open-mpi.org
 Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
 Link to this post:
>> http://www.open-mpi.org/community/lists/devel/2015/01/16739.php
>>> ___
>>> devel mailing list
>>> de...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>> Link to this post:
>> http://www.open-mpi.org/community/lists/devel/2015/01/16740.php
>>> ___
>>> devel mailing list
>>> de...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>> Link to this post:
>> http://www.open-mpi.org/community/lists/devel/2015/01/16741.php
>>
>>
>> --
>> Jeff Squyres
>> jsquy...@cisco.com
>> For corporate legal information go to:
>> http://www.cisco.com/web/about/doing_business/legal/cri/
>>
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post:
>> http://www.open-mpi.org/community/lists/devel/2015/01/16742.php
>>
>
>
> 

Re: [OMPI devel] ompi-master build error : make can require autotools

[Gilles Gouaillardet]

Dave,

this is an automake issue indeed.

i use automake 1.12.2 (as advised at
http://www.open-mpi.org/source/building.php, and
this is the version used to build the snapshots)

automake 1.12.2 adds the following line in ompi/include/Makefile.in :

$(srcdir)/mpi.h.in:  $(am__configure_deps)
($(am__cd) $(top_srcdir) && $(AUTOHEADER))
rm -f stamp-h2
touch $@


there is no such line with automake 1.14
(it seems you are using automake 1.13.3, which does not issue such lines)

bottom line, both web site and build systems should be updated in order
to use
a more recent automake version (but which one ?)

Cheers,

Gilles
On 2015/01/07 2:02, Dave Goodell (dgoodell) wrote:
> On Jan 5, 2015, at 8:40 PM, Gilles Gouaillardet 
> <gilles.gouaillar...@iferc.org> wrote:
>
>> Dave,
>>
>> what if you do
>>
>> touch ompi/include/mpi.h.in && sleep 1 && touch 
>> config/opal_config_pthreads.m4 && ./autogen.pl && module unload 
>> cisco/autotools/ac269-am1133-lt242 && ./configure --prefix=$PWD/_prefix && 
>> make
>>
>>
>> autogen.pl nor configure does not touch ompi/include/mpi.h.in, and as a
>> consequence,
>> config/opal_config_pthreads.m4 is newer than ompi/include/mpi.h when
>> make is invoked.
>>
>> then from ompi/include/Makefile:
>>
>> $(srcdir)/mpi.h.in:  $(am__configure_deps)
>>($(am__cd) $(top_srcdir) && $(AUTOHEADER))
>>rm -f stamp-h2
>>touch $@
> I don't see that rule in my ompi/include/Makefile.  I only have a couple of 
> mentions of mpi.h.in:
>
> ✂
> DIST_COMMON = $(top_srcdir)/Makefile.ompi-rules \
> $(srcdir)/ompi/Makefile.am $(srcdir)/Makefile.in \
> $(srcdir)/Makefile.am $(srcdir)/mpi.h.in $(srcdir)/mpif.h.in \
> $(srcdir)/mpif-config.h.in $(am__include_HEADERS_DIST) \
> $(am__nobase_dist_ompi_HEADERS_DIST) $(pkginclude_HEADERS)
> [...]
> am__tagged_files = $(HEADERS) $(SOURCES) $(TAGS_FILES) $(LISP)mpi.h.in
> [...]
> stamp-h2: $(srcdir)/mpi.h.in $(top_builddir)/config.status
> @rm -f stamp-h2
> cd $(top_builddir) && $(SHELL) ./config.status ompi/include/mpi.h
> ✂
>
> Maybe the rule in your version of ompi/include/Makefile comes from an older, 
> buggy version of automake?
>
>> this means $(AUTOHEADER) is invoked, and then ompi/include/mpi.h.in is
>> touched.
> I don't see $(AUTOHEADER) being invoked by make when I run the commands you 
> listed (with "sleep 1" changed to "sleep 5" to be certain):
>
> ✂
> make[1]: Entering directory `/home/dgoodell/git/ompi-upstream/ompi'
> Making all in include
> make[2]: Entering directory `/home/dgoodell/git/ompi-upstream/ompi/include'
> make  all-am
> make[3]: Entering directory `/home/dgoodell/git/ompi-upstream/ompi/include'
>   GENERATE mpif-sizeof.h
>   LN_S mpi_portable_platform.h
> make[3]: Leaving directory `/home/dgoodell/git/ompi-upstream/ompi/include'
> make[2]: Leaving directory `/home/dgoodell/git/ompi-upstream/ompi/include'
> Making all in datatype
> ✂
>
> Here are the timestamps on the relevant files after the build completes:
>
> ✂
> -rw-rw-r-- 1 dgoodell dgoodell 33 2015-01-06 08:21:18.414503328 -0800 
> ompi/include/stamp-h2
> -rw-rw-r-- 1 dgoodell dgoodell 166283 2015-01-06 08:21:18.408502854 -0800 
> ompi/include/mpi.h
> -rwxrwxr-x 1 dgoodell dgoodell 246260 2015-01-06 08:21:09.283782006 -0800 
> config.status*
> -rw-rw-r-- 1 dgoodell dgoodell  18853 2015-01-06 08:17:07.212658002 -0800 
> config/opal_config_pthreads.m4
> -rw-rw-r-- 1 dgoodell dgoodell 165986 2015-01-06 08:17:02.209262644 -0800 
> ompi/include/mpi.h.in
> ✂
>
> -Dave
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2015/01/16738.php

Re: [OMPI devel] problem running jobs on ompi-master

[Gilles Gouaillardet]

Edgar,

First, make sure your master includes 
https://github.com/open-mpi/ompi/commit/05af80b3025dbb95bdd4280087450791291d7219


If this is not enough, try with --mca coll ^ml

Hope this helps

Gilles. 

Edgar Gabriel さんのメール:
>I have some problems running jobs with ompi-master on one of our 
>clusters (after doing a major software update). Here are scenarios that 
>work and don't work.
>
>1. Everything still seems to work with 1.8.x series without any issues
>2. With master, I can run without any issues single node, multi-process jobs
>3. With master, I can run without any issues multi node jobs, as long as 
>there is only a single MPI process per node
>
>4. I can not run multi-node jobs with multiple processes per node, ompi 
>hangs for that scenario. This is independent of whether I enforce using 
>openib or tcp, and just for the sake of simplicity I attach the output 
>for tcp (there is another openib parameter issue that still linguers, 
>but I will report that later).
>
>Here is the output that I receive if setting btl_base_verbose
>
>---snip--
>gabriel@crill:~> salloc -N 2 -n 4
>gabriel@crill:~> mpirun --mca btl tcp,self --mca btl_base_verbose 30 -np 
>  4 ./hello_world
>[crill-004:18161] mca: base: components_register: registering btl components
>[crill-004:18161] mca: base: components_register: found loaded component 
>self
>[crill-004:18161] mca: base: components_register: component self 
>register function successful
>[crill-004:18161] mca: base: components_register: found loaded component tcp
>[crill-004:18161] mca: base: components_register: component tcp register 
>function successful
>[crill-004:18161] mca: base: components_open: opening btl components
>[crill-004:18161] mca: base: components_open: found loaded component self
>[crill-004:18161] mca: base: components_open: component self open 
>function successful
>[crill-004:18161] mca: base: components_open: found loaded component tcp
>[crill-004:18161] mca: base: components_open: component tcp open 
>function successful
>[crill-004:18161] select: initializing btl component self
>[crill-004:18161] select: init of component self returned success
>[crill-004:18161] select: initializing btl component tcp
>[crill-004:18161] btl: tcp: Searching for exclude address+prefix: 
>127.0.0.1 / 8
>[crill-004:18161] btl: tcp: Found match: 127.0.0.1 (lo)
>[crill-004:18161] select: init of component tcp returned success
>[crill-003:18962] mca: base: components_register: registering btl components
>[crill-003:18962] mca: base: components_register: found loaded component 
>self
>[crill-003:18962] mca: base: components_register: component self 
>register function successful
>[crill-003:18962] mca: base: components_register: found loaded component tcp
>[crill-003:18962] mca: base: components_register: component tcp register 
>function successful
>[crill-003:18962] mca: base: components_open: opening btl components
>[crill-003:18962] mca: base: components_open: found loaded component self
>[crill-003:18962] mca: base: components_open: component self open 
>function successful
>[crill-003:18962] mca: base: components_open: found loaded component tcp
>[crill-003:18962] mca: base: components_open: component tcp open 
>function successful
>[crill-003:18963] mca: base: components_register: registering btl components
>[crill-003:18963] mca: base: components_register: found loaded component 
>self
>[crill-003:18963] mca: base: components_register: component self 
>register function successful
>[crill-003:18963] mca: base: components_register: found loaded component tcp
>[crill-003:18963] mca: base: components_register: component tcp register 
>function successful
>[crill-003:18963] mca: base: components_open: opening btl components
>[crill-003:18963] mca: base: components_open: found loaded component self
>[crill-003:18963] mca: base: components_open: component self open 
>function successful
>[crill-003:18963] mca: base: components_open: found loaded component tcp
>[crill-003:18963] mca: base: components_open: component tcp open 
>function successful
>[crill-003:18964] mca: base: components_register: registering btl components
>[crill-003:18964] mca: base: components_register: found loaded component 
>self
>[crill-003:18964] mca: base: components_register: component self 
>register function successful
>[crill-003:18964] mca: base: components_register: found loaded component tcp
>[crill-003:18964] mca: base: components_register: component tcp register 
>function successful
>[crill-003:18964] mca: base: components_open: opening btl components
>[crill-003:18964] mca: base: components_open: found loaded component self
>[crill-003:18964] mca: base: components_open: component self open 
>function successful
>[crill-003:18964] mca: base: components_open: found loaded component tcp
>[crill-003:18964] mca: base: components_open: component tcp open 
>function successful
>[crill-003:18962] select: initializing btl component self

Re: [OMPI devel] OMPI devel] [OMPI commits] Git: open-mpi/ompi branch master updated. dev-612-g05af80b

[Gilles Gouaillardet]

Ralph,

I had second thougts on what i wrote earlier, and i think the code is correct. 
e.g. reply cannot be used uninitialized.

That being said, i think reply should be initialized to null and OBJ_RELEASE'd 
if not null on exit in order to avoid a memory leak.

Sorry for the confusion,

Gilles 

Ralph Castain さんのメール:
>Hi Nadezhda
>
>I’m afraid this commit is still incorrect as it means that reply can be used 
>without ever being initialized. Somehow, you have to OBJ_NEW reply before you 
>can use it.
>
>Could you please correct this?
>
>Thanks
>Ralph
>
>> On Dec 24, 2014, at 3:30 AM, git...@crest.iu.edu wrote:
>> 
>> This is an automated email from the git hooks/post-receive script. It was
>> generated because a ref change was pushed to the repository containing
>> the project "open-mpi/ompi".
>> 
>> The branch, master has been updated
>>   via  05af80b3025dbb95bdd4280087450791291d7219 (commit)
>>  from  b9349d2eb9117c61205c98d5d2d5175d26971d23 (commit)
>> 
>> Those revisions listed above that are new to this repository have
>> not appeared on any other notification email; so we list those
>> revisions in full, below.
>> 
>> - Log -
>> https://github.com/open-mpi/ompi/commit/05af80b3025dbb95bdd4280087450791291d7219
>> 
>> commit 05af80b3025dbb95bdd4280087450791291d7219
>> Author: Nadezhda Kogteva 
>> Date:   Wed Dec 24 13:25:23 2014 +0200
>> 
>>Fix commit bffb2b7a4bb49c9188d942201b8a8f04872ff63c which broke pmix 
>> server functionality
>> 
>> diff --git a/orte/orted/pmix/pmix_server.c b/orte/orted/pmix/pmix_server.c
>> index 4f0493c..0f4c816 100644
>> --- a/orte/orted/pmix/pmix_server.c
>> +++ b/orte/orted/pmix/pmix_server.c
>> @@ -1241,9 +1241,9 @@ static void pmix_server_dmdx_resp(int status, 
>> orte_process_name_t* sender,
>> /* pass across any returned blobs */
>> opal_dss.copy_payload(reply, buffer);
>> stored = true;
>> -OBJ_RETAIN(reply);
>> -PMIX_SERVER_QUEUE_SEND(req->peer, req->tag, reply);
>> }
>> +OBJ_RETAIN(reply);
>> +PMIX_SERVER_QUEUE_SEND(req->peer, req->tag, reply);
>> } else {
>> /* If peer has an access to shared memory dstore, check
>>  * if we already stored data for the target process.
>> 
>> 
>> ---
>> 
>> Summary of changes:
>> orte/orted/pmix/pmix_server.c | 4 ++--
>> 1 file changed, 2 insertions(+), 2 deletions(-)
>> 
>> 
>> hooks/post-receive
>> -- 
>> open-mpi/ompi
>> ___
>> ompi-commits mailing list
>> ompi-comm...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/ompi-commits
>
>___
>devel mailing list
>de...@open-mpi.org
>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>Link to this post: 
>http://www.open-mpi.org/community/lists/devel/2014/12/16722.php

[OMPI devel] mpirun hang (regression in bffb2b7a4bb49c9188d942201b8a8f04872ff63c)

[Gilles Gouaillardet]

Ralph,

i tried to debug the issue reported by Siegmar at
http://www.open-mpi.org/community/lists/users/2014/12/26052.php

i have not been able to try this on an heterogeneous cluster yet, but i
could
reproduce a hang with 2 nodes and 3 tasks :

mpirun -host node0,node1 -np 3 --mca btl tcp,self --mca coll ^ml
./helloworld

git bisect pointed to commit
https://github.com/hppritcha/ompi/commit/bffb2b7a4bb49c9188d942201b8a8f04872ff63c,
and reverting a subpart of this commit "fixes" the hang
(see attached patch)

your change correctly prevents the use of uninitialized data (worst case
scenario is a crash),
but has some undesired side effects that eventually causes a hang.


could you please have a look at it ?

Cheers,

Gilles
diff --git a/orte/orted/pmix/pmix_server.c b/orte/orted/pmix/pmix_server.c
index 4f0493c..0f4c816 100644
--- a/orte/orted/pmix/pmix_server.c
+++ b/orte/orted/pmix/pmix_server.c
@@ -1241,9 +1241,9 @@ static void pmix_server_dmdx_resp(int status, orte_process_name_t* sender,
 /* pass across any returned blobs */
 opal_dss.copy_payload(reply, buffer);
 stored = true;
-OBJ_RETAIN(reply);
-PMIX_SERVER_QUEUE_SEND(req->peer, req->tag, reply);
 }
+OBJ_RETAIN(reply);
+PMIX_SERVER_QUEUE_SEND(req->peer, req->tag, reply);
 } else {
 /* If peer has an access to shared memory dstore, check
  * if we already stored data for the target process.

Re: [OMPI devel] Different behaviour with MPI_IN_PLACE in MPI_Reduce_scatter() and MPI_Ireduce_scatter()

[Gilles Gouaillardet]

Lisandro,

i fixed this in the master and made a PR for v1.8.

this is a one liner, and you can find it at
https://github.com/ggouaillardet/ompi-release/commit/0e478c1191715fff37e4deb56f8891774db62775

Cheers,

Gilles

On 2014/12/23 23:43, Lisandro Dalcin wrote:
> On 28 September 2014 at 19:13, George Bosilca  wrote:
>> Lisandro,
>>
>> Good catch. Indeed the MPI_Ireduce_scatter was not covering the case where
>> MPI_IN_PLACE was used over a communicator with a single participant. I
>> pushed a patch and schedule it for 1.8.4. Check
>> https://svn.open-mpi.org/trac/ompi/ticket/4924 for more info.
>>
> While your change fixed the issues when using MPI_IN_PLACE, now 1.8.4
> seems to fail when in-place is not used.
>
> Please try the attached example:
>
> $ mpicc -DNBCOLL=0 ireduce_scatter.c
> $ mpiexec -n 2 ./a.out
> [0] rbuf[0]= 2  expected: 2
> [0] rbuf[1]= 0  expected: 0
> [1] rbuf[0]= 2  expected: 2
> [1] rbuf[1]= 0  expected: 0
> $ mpiexec -n 1 ./a.out
> [0] rbuf[0]= 1  expected: 1
>
>
> $ mpicc -DNBCOLL=1 ireduce_scatter.c
> $ mpiexec -n 2 ./a.out
> [0] rbuf[0]= 2  expected: 2
> [0] rbuf[1]= 0  expected: 0
> [1] rbuf[0]= 2  expected: 2
> [1] rbuf[1]= 0  expected: 0
> $ mpiexec -n 1 ./a.out
> [0] rbuf[0]= 0  expected: 1
>
> The last one is wrong. Not sure what's going on. Am I missing something?
>
>
>
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2014/12/16718.php

Re: [OMPI devel] ompi-master build error : make can require autotools

[Gilles Gouaillardet]

Hi Dave,

yes, i did experience this exact behaviour.

"by accident" meant i ran autogen.pl with the required autotools versions.
then, i ran configure and make with the RHEL6 stock autotools (that are too
old for OMPI)
configure worked just fine, but make crashed because of outdated autotools

if i ran make with the latest autotools, i would probably have not noticed
the issue.

note the issue occurs only when make is invoked for the first time.
if make success, autoheader does touch mpif.h.in, so the next make do not
require autotools.

if i read between the lines, it seems autoheader is not (correctly) invoked
by autogen.pl

please let me know if you cannot reproduce this issue.
(and once again, this is a very minor annoyance, and since tarballs are
generated with make dist, tarballs are very likely unaffected, so bottom
line,
only developers that update m4 files can be affected)


Cheers,

Gilles

On Tue, Dec 23, 2014 at 2:26 AM, Dave Goodell (dgoodell) <dgood...@cisco.com
> wrote:

> On Dec 22, 2014, at 2:42 AM, Gilles Gouaillardet <
> gilles.gouaillar...@iferc.org> wrote:
>
> > Jeff and all,
> >
> > i just found "by accident" that make can require autotools.
> >
> > for example:
> >
> > from (generated) ompi/include/Makefile :
> > $(srcdir)/mpi.h.in:  $(am__configure_deps)
> >($(am__cd) $(top_srcdir) && $(AUTOHEADER))
> >rm -f stamp-h2
> >touch $@
> >
> > and $(am__configure_deps) is a bunch (all?) of .m4 files.
> >
> > from a pragmatic point of view, it means that if update a m4 file, run
> > autogen.pl and configure,
> > then, the first invokation of make will run $(AUTOHEADER)
>
> Gilles,
>
> Have you actually experienced this exact behavior?  The sequence you
> mention above shouldn't cause autoheader to be invoked by make.  Running
> autogen.pl will invoke autoheader after the m4 files were touched, so the
> mpi.h.in file will be newer than its m4 dependencies, which should mean
> that this make rule won't be executed.
>
> -Dave
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/12/16713.php
>

[OMPI devel] ompi master, libfabric and static libraries

[Gilles Gouaillardet]

Jeff,

MTT reported some errors when building some test suites :
http://mtt.open-mpi.org/index.php?do_redir=2219

the root cause was some missing flags in the wrappers.
i fixed that in 8976dcf6101412f6bd0080764d19a3e9d4edf570

there is now a second issue :
libfabric requires libnl, but the -lnl flag is not passed to the mpi
wrappers.
could you please have a look at this ?

Cheers,

Gilles

[OMPI devel] ompi-master build error : make can require autotools

[Gilles Gouaillardet]

Jeff and all,

i just found "by accident" that make can require autotools.

for example:

from (generated) ompi/include/Makefile :
$(srcdir)/mpi.h.in:  $(am__configure_deps)
($(am__cd) $(top_srcdir) && $(AUTOHEADER))
rm -f stamp-h2
touch $@

and $(am__configure_deps) is a bunch (all?) of .m4 files.

from a pragmatic point of view, it means that if update a m4 file, run
autogen.pl and configure,
then, the first invokation of make will run $(AUTOHEADER)

is this a bug (and if yes, is it OMPI related) ?
or is this just a minor annoyance that impacts only developers anyway ?

Cheers,

Gilles

Re: [OMPI devel] libfabric, config.h and hwloc

[Gilles Gouaillardet]

Jeff,

the include path is $top_srcdir/opal/mca/event/libevent2021/libevent
and the libevent config.h is in
$top_builddir/opal/mca/event/libevent2021/libevent

so if you do not use VPATH, $top_srcdir = $top_builddir and make success,
but since i use VPATH, $top_srcdir != $top_builddir and there is no
config.h in my include path,
and hence make fails.

Cheers,

Gilles



 On 2014/12/19 4:12, Jeff Squyres (jsquyres) wrote:
> On Dec 18, 2014, at 3:13 AM, Gilles Gouaillardet 
> <gilles.gouaillar...@iferc.org> wrote:
>
>> currently, ompi master cannot be built if configured with
>> --without-hwloc *and without* --without-libfabric.
>>
>> the root cause is HAVE_CONFIG_H is defined but no config.h file is found.
>>
>> i digged a bit and found that config.h is taken from a hwloc directory
>> (if the --without-hwloc option is not used),
>> so even if this "works" that is unlikely the expected behavior.
> Mmm.  I see what you're saying -- the libfabric code expects there to be a 
> config.h file; it'll basically take any config.h that's in the include path.
>
> I actually find several config.h's in the tree:
>
> -
> $ find . -name config.h
> ./opal/libltdl/config.h
> ./opal/mca/hwloc/hwloc191/hwloc/include/hwloc/autogen/config.h
> ./opal/mca/hwloc/hwloc191/hwloc/include/private/autogen/config.h
> ./opal/mca/event/libevent2021/libevent/config.h
> ./ompi/contrib/vt/vt/extlib/otf/config.h
> ./ompi/contrib/vt/vt/config.h
> -
>
> However, even if I exclude libltdl, vt, and hwloc (so that there's only a 
> single config.h left in the tree -- for libevent), everything still works:
>
> 
> $ ./configure --prefix=$bogus --disable-dlopen --disable-vt --without-hwloc
> ...etc...
> $ make
> [...succeeds...]
> $ find . -name config.h
> ./opal/mca/event/libevent2021/libevent/config.h
> -
>
> So I agree with you that it only works by chance (sorta -- libevent's 
> config.h will still have all the Right Stuff in it), I can't find a case that 
> fails.
>
> Can you detail what your specific case is that is failing?
>
> (SIDENOTE: I might not be able to find the failure because of what I mention 
> below...)
>
>> the attached patch fixes some missing #ifdef
> Good catch.  I fixed those a different way (just deleted the #includes -- 
> they weren't necessary); I committed the fix both in OMPI and upstream in 
> libfabric.
>
>> my last (cosmetic) comment is about
>> $srcdir/opal/mca/common/libfabric/Makefile.in (and several other
>> Makefile.in) :
>> [snipped]
> Good catch.  Fixed in 
> https://github.com/open-mpi/ompi/commit/be6d46490f7b80d4f5ea90c859ccbebe96bdaaba.
>   And then later fixed *that* a followup commit.  :-(
>

[OMPI devel] libfabric, config.h and hwloc

[Gilles Gouaillardet]

Jeff,

currently, ompi master cannot be built if configured with
--without-hwloc *and without* --without-libfabric.

the root cause is HAVE_CONFIG_H is defined but no config.h file is found.

i digged a bit and found that config.h is taken from a hwloc directory
(if the --without-hwloc option is not used),
so even if this "works" that is unlikely the expected behaviour.

the attached patch fixes some missing #ifdef

two workarounds are (after configure)
1) after configure, manually touch
$top_builddir/opal/mca/common/libfabric/config.h
2) manually hack $top_builddir/opal/mca/common/libfabric/Makefile and
replace
DEFS = -DHAVE_CONFIG_H
with
DEFS =


my last (cosmetic) comment is about
$srcdir/opal/mca/common/libfabric/Makefile.in (and several other
Makefile.in) :

CONFIG_HEADER = $(top_builddir)/opal/include/opal_config.h \
$(top_builddir)/ompi/include/mpi.h \
$(top_builddir)/oshmem/include/shmem.h \

$(top_builddir)/opal/mca/hwloc/hwloc191/hwloc/include/private/autogen/config.h
\

$(top_builddir)/opal/mca/hwloc/hwloc191/hwloc/include/hwloc/autogen/config.h

the last two files do not exist when ompi is configured with
--without-hwloc, is there any way
to have them excluded from Makefile by configure ?

Cheers,

Gilles

diff --git 
a/opal/mca/common/libfabric/libfabric/prov/usnic/src/usnic_direct/libnl_utils_common.c
 
b/opal/mca/common/libfabric/libfabric/prov/usnic/src/usnic_direct/libnl_utils_common.c
index c38a817..fe1e509 100644
--- 
a/opal/mca/common/libfabric/libfabric/prov/usnic/src/usnic_direct/libnl_utils_common.c
+++ 
b/opal/mca/common/libfabric/libfabric/prov/usnic/src/usnic_direct/libnl_utils_common.c
@@ -39,7 +39,9 @@
  *
  *
  */
+#ifdef HAVE_CONFIG_H
 #include "config.h"
+#endif /* HAVE_CONFIG_H */

 #include 
 #include 
diff --git 
a/opal/mca/common/libfabric/libfabric/prov/usnic/src/usnic_direct/usnic_ip_utils.c
 
b/opal/mca/common/libfabric/libfabric/prov/usnic/src/usnic_direct/usnic_ip_utils.c
index 815417f..d1727dd 100644
--- 
a/opal/mca/common/libfabric/libfabric/prov/usnic/src/usnic_direct/usnic_ip_utils.c
+++ 
b/opal/mca/common/libfabric/libfabric/prov/usnic/src/usnic_direct/usnic_ip_utils.c
@@ -39,7 +39,9 @@
  *
  *
  */
+#ifdef HAVE_CONFIG_H
 #include "config.h"
+#endif /* HAVE_CONFIG_H */

 #include 
 #include 

Re: [OMPI devel] OMPI devel] 1.8.4rc Status

[Gilles Gouaillardet]

Ralph,

You get it right.
The latest nightly tarball shoul work out of the box.
(well, -m64 must be passed manually, but this is not related whatsoever to the 
issue discussed here)

Cheers,

Gilles

"Jeff Squyres (jsquyres)" <jsquy...@cisco.com> wrote:
>Paul --
>
>The __sun macro check is now in the OMPI 1.8 tree, and is in the latest 
>nightly tarball.
>
>If I'm following this thread right -- and I might not be! -- I think Gilles is 
>saying that now that the __sun check is in, it should fix this 
>-mt/-D_REENTRANT/whatever problem.
>
>Can you confirm?
>
>
>On Dec 16, 2014, at 1:55 PM, Paul Hargrove <phhargr...@lbl.gov> wrote:
>
>> Gilles,
>> 
>> I am running mpirun on a host that ALSO will run one of the application 
>> processes.
>> Requested ifconfig and netstat outputs appear below.
>> 
>> -Paul
>> 
>> [phargrov@pcp-j-20 ~]$ ifconfig -a
>> lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 
>> index 1
>> inet 127.0.0.1 netmask ff00 
>> bge0: flags=1004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4> mtu 1500 index 
>> 2
>> inet 172.16.0.120 netmask  broadcast 172.16.255.255
>> p.ibp0: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> 
>> mtu 2044 index 3
>> inet 172.18.0.120 netmask  broadcast 172.18.255.255
>> lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 
>> index 1
>> inet6 ::1/128 
>> bge0: flags=20002004841<UP,RUNNING,MULTICAST,DHCP,IPv6> mtu 1500 index 2
>> inet6 fe80::250:45ff:fe5c:2b0/10 
>> [phargrov@pcp-j-20 ~]$ netstat -nr
>> 
>> Routing Table: IPv4
>>   Destination   Gateway   Flags  Ref Use Interface 
>>   - - -- - 
>> default  172.16.254.1 UG2 158463 bge0  
>> 127.0.0.1127.0.0.1UH5 398913 lo0   
>> 172.16.0.0   172.16.0.120 U 4  135241319 bge0  
>> 172.18.0.0   172.18.0.120 U 3 26 p.ibp0 
>> 
>> Routing Table: IPv6
>>   Destination/MaskGateway   Flags Ref   Use
>> If  
>> --- --- - --- --- 
>> - 
>> ::1 ::1 UH  2   0 
>> lo0  
>> fe80::/10   fe80::250:45ff:fe5c:2b0 U   2   0 
>> bge0 
>> 
>> On Tue, Dec 16, 2014 at 2:55 AM, Gilles Gouaillardet 
>> <gilles.gouaillar...@iferc.org> wrote:
>> Paul,
>> 
>> could you please send the output of
>> ifconfig -a
>> netstat -nr
>> 
>> on the three hosts you are using
>> (i assume you are still invoking mpirun from one node, and tasks are running 
>> on two other nodes)
>> 
>> Cheers,
>> 
>> Gilles
>> 
>> 
>> On 2014/12/16 16:00, Paul Hargrove wrote:
>>> Gilles,
>>> 
>>> I looked again carefully and I am *NOT* finding -D_REENTRANT passed to most
>>> compilations.
>>> It appears to be used for building libevent and vt, but nothing else.
>>> The output from configure contains
>>> 
>>> checking if more special flags are required for pthreads... -D_REENTRANT
>>> 
>>> only in the libevent and vt sub-configure portions.
>>> 
>>> When configured for gcc on Solaris-11 I see the following in configure
>>> 
>>> checking for C optimization flags... -m64 -D_REENTRANT -g
>>> -finline-functions -fno-strict-aliasing
>>> 
>>> but with CC=cc the equivalent line is
>>> 
>>> checking for C optimization flags... -m64 -g
>>> 
>>> In both cases the "-m64" is from the CFLAGS I have passed to configure.
>>> 
>>> However, when I use CFLAGS="-m64 -D_REENTRANT" the problem DOES NOT go away.
>>> I see
>>> 
>>> [pcp-j-20:24740] mca_oob_tcp_accept: accept() failed: Error 0 (11).
>>> 
>>> A process or daemon was unable to complete a TCP connection
>>> to another process:
>>>   Local host:pcp-j-20
>>>   Remote host:   172.18.0.120
>>> This is usually caused by a firewall on the remote host. Please
>>> check that any firewall (e.g., iptables) has been disabled and
>>> try again.
>>> -

Re: [OMPI devel] OMPI devel] 1.8.4rc Status

[Gilles Gouaillardet]

Ralph,

what goes wrong ?
(e.g. which command ?)

and which compiler (e.g. gcc < 4.9.1 ?) are you using ?

Cheers,

Gilles

On 2014/12/17 17:30, Ralph Castain wrote:
> I'm afraid I cannot generate a new rc, nor will there be a new 1.8 nightly
> tarball as (ahem) Jeff's fortran commit broke the build system. I tried to
> figure out a fix, but am too tired to get it right.
>
> So I'm afraid we are stuck for the moment until Jeff returns in the morning
> and fixes the problem. We'll have to pick this up afterwards.
>
> Sorry guys
> Ralph
>
>
> On Tue, Dec 16, 2014 at 10:59 PM, Gilles Gouaillardet <
> gilles.gouaillar...@iferc.org> wrote:
>>  Thanks Paul !
>>
>> imho the first test is useless since it does not include the commit that
>> sets the -D_REENTRANT CFLAGS on solaris/solarisstudio
>>
>> https://github.com/open-mpi/ompi-release/commit/ac8b84ce674b958dbf8c9481b300beeef0548b83
>>
>> Cheers,
>>
>> Gilles
>>
>>
>> On 2014/12/17 15:56, Paul Hargrove wrote:
>>
>> I've queued 3 tests:
>>
>> 1) openmpi-v1.8.3-272-g4e4f997
>> 2) openmpi-v1.8.4rc4 + adding -D_REENTRANT to CFLAGS and wrapper-cflags
>> 3) openmpi-v1.8.4rc4 + adding -mt to CFLAGS and wrapper-cflags
>>
>> I hope to be able to login and collect the results around noon pacific time
>> on Wed.
>>
>> -Paul
>>
>> On Tue, Dec 16, 2014 at 10:48 PM, Gilles Gouaillardet 
>> <gilles.gouaillar...@iferc.org> wrote:
>>
>>
>>  Paul,
>>
>> i understand, i will now work on a better way to figure out the required
>> flags
>>
>> the latest nightly snapshot does not include the commit i mentionned, and
>> i think
>> it is worth giving it a try (to be 100.0% sure ...)
>>
>> can you please do that tomorrow ?
>>
>> in the mean time, if we (well Ralph indeed) want to release 1.8.4, then
>> simply restore
>> the two config files i mentionned.
>>
>> Cheers,
>>
>> Gilles
>>
>>
>> On 2014/12/17 15:39, Paul Hargrove wrote:
>>
>> Gilles,
>>
>> If I have done my testing correctly (not 100% sure) then adding
>> "-D_REENTRANT" was NOT sufficient, where "-mt" was.
>>
>> I can at least test 1 tarball with one set of configure args each evening.
>> Anything more than that I cannot commit to.
>>
>> My scripts are capable of grabbing the v1.8 nightly instead of the rc if
>> that helps.
>>
>> -Paul
>>
>> On Tue, Dec 16, 2014 at 10:31 PM, Gilles Gouaillardet 
>> <gilles.gouaillar...@iferc.org> <gilles.gouaillar...@iferc.org> wrote:
>>
>>
>>  Ralph,
>>
>> i think that will not work.
>>
>> here is the full story :
>>
>> once upon a time, on solaris, we did not try to compile pthread'ed app
>> without any special parameters.
>> that was a minor annoyance on solaris 10 with old gcc : configure passed a
>> flag (-pthread if i remember correctly)
>> that was not supported by gcc (at that time) and generated tons of
>> warnings.
>> when i asked "why don't we just try no special parameter on solaris ?" i
>> was replied this is because looong time ago
>> openmpi used solaris lwp, so solaris was "special" anyway.
>> since solaris is able to build (compile+link) a pthread'ed app without any
>> flags, i removed the special case for solaris,
>> and no flag was used.
>> then i noticed that lead to bad code (errno is global instead of per
>> thread specific), so you automatically added -D_REENTRANT
>> on solaris (e.g. if the __sun__ macro is defined)
>> then i found that solarisstudio compilers do not define the __sun__macro
>> automatically (__sun and sun are defined) so i improved
>> the test (e.g. we are on solaris if __sun__ or __sun is defined)
>> this was merged (yesterday) and is not in rc4
>>
>> what we should do know is unclear for me ...
>> is -D_REENTRANT enough for gcc compilers on solaris ?
>> is -D_REENTRANT *not* enough for solarisstudio compilers on solaris ?
>> /* if -D_REENTRANT is *not* enough, then we all we have to do is use -mt
>> since that implies -D_REENTRANT */
>>
>>
>> a working solution (minus the minor annoyance i described earlier) is to
>> restore
>> config/opal_check_os_flavors.
>> m4
>> config/ompi_config_pthreads.m4
>>
>> and then i ll find a better way to correctly set the flags that must be
>> used on solaris
>>
>> that being said, and based on Paul's availability, i d rather have a new
>> tarball (

Re: [OMPI devel] OMPI devel] 1.8.4rc Status

[Gilles Gouaillardet]

Thanks Paul !

imho the first test is useless since it does not include the commit that
sets the -D_REENTRANT CFLAGS on solaris/solarisstudio
https://github.com/open-mpi/ompi-release/commit/ac8b84ce674b958dbf8c9481b300beeef0548b83

Cheers,

Gilles

On 2014/12/17 15:56, Paul Hargrove wrote:
> I've queued 3 tests:
>
> 1) openmpi-v1.8.3-272-g4e4f997
> 2) openmpi-v1.8.4rc4 + adding -D_REENTRANT to CFLAGS and wrapper-cflags
> 3) openmpi-v1.8.4rc4 + adding -mt to CFLAGS and wrapper-cflags
>
> I hope to be able to login and collect the results around noon pacific time
> on Wed.
>
> -Paul
>
> On Tue, Dec 16, 2014 at 10:48 PM, Gilles Gouaillardet <
> gilles.gouaillar...@iferc.org> wrote:
>>  Paul,
>>
>> i understand, i will now work on a better way to figure out the required
>> flags
>>
>> the latest nightly snapshot does not include the commit i mentionned, and
>> i think
>> it is worth giving it a try (to be 100.0% sure ...)
>>
>> can you please do that tomorrow ?
>>
>> in the mean time, if we (well Ralph indeed) want to release 1.8.4, then
>> simply restore
>> the two config files i mentionned.
>>
>> Cheers,
>>
>> Gilles
>>
>>
>> On 2014/12/17 15:39, Paul Hargrove wrote:
>>
>> Gilles,
>>
>> If I have done my testing correctly (not 100% sure) then adding
>> "-D_REENTRANT" was NOT sufficient, where "-mt" was.
>>
>> I can at least test 1 tarball with one set of configure args each evening.
>> Anything more than that I cannot commit to.
>>
>> My scripts are capable of grabbing the v1.8 nightly instead of the rc if
>> that helps.
>>
>> -Paul
>>
>> On Tue, Dec 16, 2014 at 10:31 PM, Gilles Gouaillardet 
>> <gilles.gouaillar...@iferc.org> wrote:
>>
>>
>>  Ralph,
>>
>> i think that will not work.
>>
>> here is the full story :
>>
>> once upon a time, on solaris, we did not try to compile pthread'ed app
>> without any special parameters.
>> that was a minor annoyance on solaris 10 with old gcc : configure passed a
>> flag (-pthread if i remember correctly)
>> that was not supported by gcc (at that time) and generated tons of
>> warnings.
>> when i asked "why don't we just try no special parameter on solaris ?" i
>> was replied this is because looong time ago
>> openmpi used solaris lwp, so solaris was "special" anyway.
>> since solaris is able to build (compile+link) a pthread'ed app without any
>> flags, i removed the special case for solaris,
>> and no flag was used.
>> then i noticed that lead to bad code (errno is global instead of per
>> thread specific), so you automatically added -D_REENTRANT
>> on solaris (e.g. if the __sun__ macro is defined)
>> then i found that solarisstudio compilers do not define the __sun__macro
>> automatically (__sun and sun are defined) so i improved
>> the test (e.g. we are on solaris if __sun__ or __sun is defined)
>> this was merged (yesterday) and is not in rc4
>>
>> what we should do know is unclear for me ...
>> is -D_REENTRANT enough for gcc compilers on solaris ?
>> is -D_REENTRANT *not* enough for solarisstudio compilers on solaris ?
>> /* if -D_REENTRANT is *not* enough, then we all we have to do is use -mt
>> since that implies -D_REENTRANT */
>>
>>
>> a working solution (minus the minor annoyance i described earlier) is to
>> restore
>> config/opal_check_os_flavors.m4
>> config/ompi_config_pthreads.m4
>>
>> and then i ll find a better way to correctly set the flags that must be
>> used on solaris
>>
>> that being said, and based on Paul's availability, i d rather have a new
>> tarball (rc5?) tested.
>> (do we *really* need -mt ? isn't -D_REENTRANT enough ?)
>> this tarball must 
>> includehttps://github.com/open-mpi/ompi-release/commit/ac8b84ce674b958dbf8c9481b300beeef0548b83
>>
>>
>> configury: test the __sun macro to detect solaris OS.
>>
>>
>> FWIW. i was unable to reproduce the problem on solaris 11 with sunstudio
>> 12.4 even if i do not use -D_REENTRANT *nor* -mt (!)
>>
>> Cheers,
>>
>> Gilles
>>
>>
>> On 2014/12/17 15:01, Ralph Castain wrote:
>>
>> Hi Paul
>>
>> Can you try the attached patch? It would require running autogen, I fear.
>> Otherwise, I can add it to the tarball.
>>
>> Ralph
>>
>>
>> On Tue, Dec 16, 2014 at 9:59 PM, Paul Hargrove <phhargr...@lbl.gov> 
>> <phhargr...@lbl.gov> <phhargr...@lbl.gov> <

Re: [OMPI devel] OMPI devel] 1.8.4rc Status

[Gilles Gouaillardet]

Ralph,

i think that will not work.

here is the full story :

once upon a time, on solaris, we did not try to compile pthread'ed app
without any special parameters.
that was a minor annoyance on solaris 10 with old gcc : configure passed
a flag (-pthread if i remember correctly)
that was not supported by gcc (at that time) and generated tons of warnings.
when i asked "why don't we just try no special parameter on solaris ?" i
was replied this is because looong time ago
openmpi used solaris lwp, so solaris was "special" anyway.
since solaris is able to build (compile+link) a pthread'ed app without
any flags, i removed the special case for solaris,
and no flag was used.
then i noticed that lead to bad code (errno is global instead of per
thread specific), so you automatically added -D_REENTRANT
on solaris (e.g. if the __sun__ macro is defined)
then i found that solarisstudio compilers do not define the __sun__macro
automatically (__sun and sun are defined) so i improved
the test (e.g. we are on solaris if __sun__ or __sun is defined)
this was merged (yesterday) and is not in rc4

what we should do know is unclear for me ...
is -D_REENTRANT enough for gcc compilers on solaris ?
is -D_REENTRANT *not* enough for solarisstudio compilers on solaris ?
/* if -D_REENTRANT is *not* enough, then we all we have to do is use -mt
since that implies -D_REENTRANT */


a working solution (minus the minor annoyance i described earlier) is to
restore
config/opal_check_os_flavors.m4
config/ompi_config_pthreads.m4

and then i ll find a better way to correctly set the flags that must be
used on solaris

that being said, and based on Paul's availability, i d rather have a new
tarball (rc5?) tested.
(do we *really* need -mt ? isn't -D_REENTRANT enough ?)
this tarball must include
https://github.com/open-mpi/ompi-release/commit/ac8b84ce674b958dbf8c9481b300beeef0548b83

configury: test the __sun macro to detect solaris OS.


FWIW. i was unable to reproduce the problem on solaris 11 with sunstudio
12.4 even if i do not use -D_REENTRANT *nor* -mt (!)

Cheers,

Gilles

On 2014/12/17 15:01, Ralph Castain wrote:
> Hi Paul
>
> Can you try the attached patch? It would require running autogen, I fear.
> Otherwise, I can add it to the tarball.
>
> Ralph
>
>
> On Tue, Dec 16, 2014 at 9:59 PM, Paul Hargrove  wrote:
>> Gilles,
>>
>> The 1.8.3 test works where the 1.8.4rc4 one fails with identical configure
>> arguments.
>>
>> While it may be overkill, I configured 1.8.4rc4 with
>>
>>CFLAGS="-m64 -mt" --with-wrapper-cflags="-m64 -mt" \
>>LDFLAGS="-mt" --with-wrapper-ldflags="-mt"
>>
>> The resulting run worked!
>>
>> So, I very strongly suspect that the problem will be resolved if one
>> restores the configure logic that my previous email shows has vanished
>> (since that would restore "-mt" to CFLAGS and wrapper cflags).
>>
>> -Paul
>>
>> On Tue, Dec 16, 2014 at 8:10 PM, Paul Hargrove  wrote:
>>> My 1.8.3 build has not completed.
>>> HOWEVER, I can already see a key difference in the configure step.
>>>
>>> In 1.8.3 "-mt" was added AUTOMATICALLY to CFLAGS by configure:
>>>
>>> checking if C compiler and POSIX threads work as is... no - Solaris, not
>>> checked
>>> checking if C++ compiler and POSIX threads work as is... no - Solaris,
>>> not checked
>>> checking if Fortran compiler and POSIX threads work as is... no -
>>> Solaris, not checked
>>> checking if C compiler and POSIX threads work with -pthread... no
>>> checking if C compiler and POSIX threads work with -pthreads... no
>>> checking if C compiler and POSIX threads work with -mt... yes
>>> checking if C++ compiler and POSIX threads work with -pthread... yes
>>> checking if Fortran compiler and POSIX threads work with -pthread... yes
>>>
>>> This is not the case in 1.8.4rc4:
>>>
>>> checking if C compiler and POSIX threads work as is... yes
>>> checking if C++ compiler and POSIX threads work as is... yes
>>> checking if Fortran compiler and POSIX threads work as is... yes
>>>
>>>
>>> So, it looks like a chunk of Solaris-specific configure logic was LOST.
>>>
>>> -Paul
>>>
>>
>> --
>> Paul H. Hargrove  phhargr...@lbl.gov
>> Computer Languages & Systems Software (CLaSS) Group
>> Computer Science Department   Tel: +1-510-495-2352
>> Lawrence Berkeley National Laboratory Fax: +1-510-486-6900
>>
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post:
>> http://www.open-mpi.org/community/lists/devel/2014/12/16625.php
>>
>
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2014/12/16626.php

Re: [OMPI devel] OMPI devel] 1.8.4rc Status

[Gilles Gouaillardet]

Paul,

i do not think -lpthread is passed automatically to LDFLAGS on Solaris,
so you might have to do it manually as well

i never used

--with-wrapper-cflags

before, so i'd rather invite you to
mpicc -show
to make sure the right flags are passed at the right place when the app
is built

Cheers,

Gilles

On 2014/12/17 12:04, Paul Hargrove wrote:
> Gilles,
>
> I am running the build of 1.8.3 first.
> As you suggest, I will only try without -m64 if 1.8.3 runs with it.
>
> Regarding "-mt" my understanding from "man cc" is that it has a DUAL
> function:
> 1) Passes -D_REENTRANT to the preprocess stage (if any)
> 2) Passes "the right flags" to the linker stage (if any)
>
> NOTE that since Solaris has historically supported a "native" (non-POSIX)
> threads library, when linking for pthreads one must pass BOTH "-mt" and
> "-lpthread", and they must be in that order.
>
> I *think* have already tried adding "-mt" to both CFLAGS and LDFLAGS, but
> am not 100% sure I've done so correctly.  I believe I need to configure with
>
>CFLAGS="-m64 -mt" --with-wrapper-cflags="-m64 -mt" \
>LDFLAGS="-mt" --with-wrapper-ldflags="-mt"
>
> if I am to be sure that orterun and the app are both compiled and linked
> with "-mt".
> Is that right?
>
> -Paul
>
> On Tue, Dec 16, 2014 at 6:25 PM, Gilles Gouaillardet <
> gilles.gouaillar...@iferc.org> wrote:
>>  Thanks Paul,
>>
>> if 1.8.3 with -m64 and the same compilers runs fine, then please do not
>> bother running 1.8.4rc4 without -m64.
>> /* i understand you are busy and i hardly believe -m64 is the root cause */
>>
>> a regression i can think of involves the flags we use for pthreads :
>> for bad reasons, we initially tested the following flags on solaris :
>> -pthread
>> -pthreads
>> -mt
>>
>> with solarisstudio 12.4, -mt was chosen
>>
>> 1.8.4rc4 has a bug (fixed in the v1.8 git): -D_REENTRANT is not
>> automatically added, so you have to do it manually.
>> i just figured out that -mt is unlikely automatically.
>> do we need this and where ?
>> CFLAGS ? (or is -D_REENTRANT enough ?)
>> LDFLAGS ? (that might be solaris and/or solarisstudio (12.4) specific and
>> i simply ignore it)
>>
>> Bottom line, i do invite you to test 1.8.4rc4 again and with
>> CFLAGS="-mt"
>> or
>> CFLAGS="-mt -m64"
>> if you previously tested 1.8.3 with -m64
>>
>> Cheers,
>>
>> Gilles
>>
>>
>>
>> On 2014/12/17 11:05, Paul Hargrove wrote:
>>
>> Gilles,
>>
>> First, please note that prior tests of 1.8.3 ran with no problems on these
>> hosts.
>> So, I *think* this problem is a regression.
>> However, I am not 100% certain that this *exact* configuration was tested.
>> So, I am RE-running a test of 1.8.3 now to be absolutely sure if this is a
>> regression.
>> I will report the outcome when I can.
>>
>> I have limited time to run the tests you are asking for.  I will do my
>> best, but am concerned that I won't be responsive enough and may hold up
>> the release.  I fully understand why you ask multiple questions in one
>> email to keep things moving.
>>
>> I am running mpirun on pcp-j-20 and "getent hosts pcp-j-20" run there yields
>>
>> $ getent hosts pcp-j-20
>> 127.0.0.1   pcp-j-20 pcp-j-20.local localhost loghost
>> 172.16.0.120pcp-j-20 pcp-j-20.local localhost loghost
>>
>> In case it matters: there is an entry for 172.18.0.0.120 in /etc/hosts as
>> pcp-j-20-ib.
>>
>> I will run a test tonight to determine if the same issue is present without
>> "-m64".
>> I will report the outcome when I can.
>>
>> Yes, I can ping and ssh to "pcp-j-{19,20}" and "172.{16,18}.0.{119,120}".
>> I see the following if run on either pcp-j-19 or pcp-j-20:
>>
>> $ for x in {pcp-j-,172.{16,18}.0.1}{19,20}; do ssh $x echo OK connecting to
>> $x; done
>> OK connecting to pcp-j-19
>> OK connecting to pcp-j-20
>> OK connecting to 172.16.0.119
>> OK connecting to 172.16.0.120
>> OK connecting to 172.18.0.119
>> OK connecting to 172.18.0.120
>>
>>
>> I will report on the 1.8.3 and the non-m64 runs when they are done.
>> Meanwhile, if you have other things you want run let me know.
>>
>> -Paul
>>
>> On Tue, Dec 16, 2014 at 5:35 PM, Gilles Gouaillardet 
>> <gilles.gouaillar...@gmail.com> wrote:
>>
>>  Thanks Paul,
>>
>> Are you i

Re: [OMPI devel] OMPI devel] 1.8.4rc Status

[Gilles Gouaillardet]

Thanks Paul,

if 1.8.3 with -m64 and the same compilers runs fine, then please do not
bother running 1.8.4rc4 without -m64.
/* i understand you are busy and i hardly believe -m64 is the root cause */

a regression i can think of involves the flags we use for pthreads :
for bad reasons, we initially tested the following flags on solaris :
-pthread
-pthreads
-mt

with solarisstudio 12.4, -mt was chosen

1.8.4rc4 has a bug (fixed in the v1.8 git): -D_REENTRANT is not
automatically added, so you have to do it manually.
i just figured out that -mt is unlikely automatically.
do we need this and where ?
CFLAGS ? (or is -D_REENTRANT enough ?)
LDFLAGS ? (that might be solaris and/or solarisstudio (12.4) specific
and i simply ignore it)

Bottom line, i do invite you to test 1.8.4rc4 again and with
CFLAGS="-mt"
or
CFLAGS="-mt -m64"
if you previously tested 1.8.3 with -m64

Cheers,

Gilles


On 2014/12/17 11:05, Paul Hargrove wrote:
> Gilles,
>
> First, please note that prior tests of 1.8.3 ran with no problems on these
> hosts.
> So, I *think* this problem is a regression.
> However, I am not 100% certain that this *exact* configuration was tested.
> So, I am RE-running a test of 1.8.3 now to be absolutely sure if this is a
> regression.
> I will report the outcome when I can.
>
> I have limited time to run the tests you are asking for.  I will do my
> best, but am concerned that I won't be responsive enough and may hold up
> the release.  I fully understand why you ask multiple questions in one
> email to keep things moving.
>
> I am running mpirun on pcp-j-20 and "getent hosts pcp-j-20" run there yields
>
> $ getent hosts pcp-j-20
> 127.0.0.1   pcp-j-20 pcp-j-20.local localhost loghost
> 172.16.0.120pcp-j-20 pcp-j-20.local localhost loghost
>
> In case it matters: there is an entry for 172.18.0.0.120 in /etc/hosts as
> pcp-j-20-ib.
>
> I will run a test tonight to determine if the same issue is present without
> "-m64".
> I will report the outcome when I can.
>
> Yes, I can ping and ssh to "pcp-j-{19,20}" and "172.{16,18}.0.{119,120}".
> I see the following if run on either pcp-j-19 or pcp-j-20:
>
> $ for x in {pcp-j-,172.{16,18}.0.1}{19,20}; do ssh $x echo OK connecting to
> $x; done
> OK connecting to pcp-j-19
> OK connecting to pcp-j-20
> OK connecting to 172.16.0.119
> OK connecting to 172.16.0.120
> OK connecting to 172.18.0.119
> OK connecting to 172.18.0.120
>
>
> I will report on the 1.8.3 and the non-m64 runs when they are done.
> Meanwhile, if you have other things you want run let me know.
>
> -Paul
>
> On Tue, Dec 16, 2014 at 5:35 PM, Gilles Gouaillardet <
> gilles.gouaillar...@gmail.com> wrote:
>> Thanks Paul,
>>
>> Are you invoking mpirun on pcp-j-20 ?
>> If yes, what does
>> getent hosts pcp-j-20
>> says ?
>>
>> BTW, did you try without -m64 ?
>>
>> Does the following work
>> ping/ssh 172.18.0.120
>>
>> Honestly, this output makes very little sense to me, so i am asking way
>> too much info hoping i can reproduce this issue or get a hint on what can
>> possibly goes wrong.
>>
>> Cheers,
>>
>> Gilles
>>
>> Paul Hargrove <phhargr...@lbl.gov> wrote:
>> Gilles,
>>
>> I am running mpirun on a host that ALSO will run one of the application
>> processes.
>> Requested ifconfig and netstat outputs appear below.
>>
>> -Paul
>>
>> [phargrov@pcp-j-20 ~]$ ifconfig -a
>> lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232
>> index 1
>> inet 127.0.0.1 netmask ff00
>> bge0: flags=1004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4> mtu 1500
>> index 2
>> inet 172.16.0.120 netmask  broadcast 172.16.255.255
>> p.ibp0: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU>
>> mtu 2044 index 3
>> inet 172.18.0.120 netmask  broadcast 172.18.255.255
>> lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252
>> index 1
>> inet6 ::1/128
>> bge0: flags=20002004841<UP,RUNNING,MULTICAST,DHCP,IPv6> mtu 1500 index 2
>> inet6 fe80::250:45ff:fe5c:2b0/10
>> [phargrov@pcp-j-20 ~]$ netstat -nr
>>
>> Routing Table: IPv4
>>   Destination   Gateway   Flags  Ref Use Interface
>>   - - -- -
>> default  172.16.254.1 UG2 158463 bge0
>> 127.0.0.1127.0.0.1UH5 398913 lo0
>> 172.16.0.0   172.16.0.120 U   

Re: [OMPI devel] OMPI devel] 1.8.4rc Status

[Gilles Gouaillardet]

Thanks Paul,

Are you invoking mpirun on pcp-j-20 ?
If yes, what does
getent hosts pcp-j-20
says ?

BTW, did you try without -m64 ?

Does the following work
ping/ssh 172.18.0.120

Honestly, this output makes very little sense to me, so i am asking way too 
much info hoping i can reproduce this issue or get a hint on what can possibly 
goes wrong.

Cheers,

Gilles

Paul Hargrove <phhargr...@lbl.gov> wrote:
>Gilles,
>
>
>I am running mpirun on a host that ALSO will run one of the application 
>processes.
>
>Requested ifconfig and netstat outputs appear below.
>
>
>-Paul
>
>
>[phargrov@pcp-j-20 ~]$ ifconfig -a
>
>lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 
>index 1
>
>        inet 127.0.0.1 netmask ff00 
>
>bge0: flags=1004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4> mtu 1500 index 2
>
>        inet 172.16.0.120 netmask  broadcast 172.16.255.255
>
>p.ibp0: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 
>2044 index 3
>
>        inet 172.18.0.120 netmask  broadcast 172.18.255.255
>
>lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 
>index 1
>
>        inet6 ::1/128 
>
>bge0: flags=20002004841<UP,RUNNING,MULTICAST,DHCP,IPv6> mtu 1500 index 2
>
>        inet6 fe80::250:45ff:fe5c:2b0/10 
>
>[phargrov@pcp-j-20 ~]$ netstat -nr
>
>
>Routing Table: IPv4
>
>  Destination           Gateway           Flags  Ref     Use     Interface 
>
>  - - -- - 
>
>default              172.16.254.1         UG        2     158463 bge0      
>
>127.0.0.1            127.0.0.1            UH        5     398913 lo0       
>
>172.16.0.0           172.16.0.120         U         4  135241319 bge0      
>
>172.18.0.0           172.18.0.120         U         3         26 p.ibp0 
>
>
>Routing Table: IPv6
>
>  Destination/Mask            Gateway                   Flags Ref   Use    If  
> 
>
>--- --- - --- --- 
>- 
>
>::1                         ::1                         UH      2       0 lo0  
> 
>
>fe80::/10                   fe80::250:45ff:fe5c:2b0     U       2       0 bge0 
>
>
>On Tue, Dec 16, 2014 at 2:55 AM, Gilles Gouaillardet 
><gilles.gouaillar...@iferc.org> wrote:
>
>Paul,
>
>could you please send the output of
>ifconfig -a
>netstat -nr
>
>on the three hosts you are using
>(i assume you are still invoking mpirun from one node, and tasks are running 
>on two other nodes)
>
>Cheers,
>
>Gilles
>
>
>
>On 2014/12/16 16:00, Paul Hargrove wrote:
>
>Gilles, I looked again carefully and I am *NOT* finding -D_REENTRANT passed to 
>most compilations. It appears to be used for building libevent and vt, but 
>nothing else. The output from configure contains checking if more special 
>flags are required for pthreads... -D_REENTRANT only in the libevent and vt 
>sub-configure portions. When configured for gcc on Solaris-11 I see the 
>following in configure checking for C optimization flags... -m64 -D_REENTRANT 
>-g -finline-functions -fno-strict-aliasing but with CC=cc the equivalent line 
>is checking for C optimization flags... -m64 -g In both cases the "-m64" is 
>from the CFLAGS I have passed to configure. However, when I use CFLAGS="-m64 
>-D_REENTRANT" the problem DOES NOT go away. I see [pcp-j-20:24740] 
>mca_oob_tcp_accept: accept() failed: Error 0 (11). 
> A process or 
>daemon was unable to complete a TCP connection to another process: Local host: 
>pcp-j-20 Remote host: 172.18.0.120 This is usually caused by a firewall on the 
>remote host. Please check that any firewall (e.g., iptables) has been disabled 
>and try again.  
>which is at least appears to have a non-zero errno. A quick grep through 
>/usr/include/sys/errno shows 11 is EAGAIN. With the oob.patch you provided the 
>failed accept goes away, BUT the connection still fails: 
> A process or 
>daemon was unable to complete a TCP connection to another process: Local host: 
>pcp-j-20 Remote host: 172.18.0.120 This is usually caused by a firewall on the 
>remote host. Please check that any firewall (e.g., iptables) has been disabled 
>and try again.  
>Use of "-mca oob_tcp_if_include bge0" to use a single interface did not fix 
>this. -Paul On Mon, Dec 15, 2014 at 7:18 PM, Paul Hargrove 
><phhar

Re: [OMPI devel] 1.8.4rc Status

[Gilles Gouaillardet]

Paul,

could you please send the output of
ifconfig -a
netstat -nr

on the three hosts you are using
(i assume you are still invoking mpirun from one node, and tasks are
running on two other nodes)

Cheers,

Gilles

On 2014/12/16 16:00, Paul Hargrove wrote:
> Gilles,
>
> I looked again carefully and I am *NOT* finding -D_REENTRANT passed to most
> compilations.
> It appears to be used for building libevent and vt, but nothing else.
> The output from configure contains
>
> checking if more special flags are required for pthreads... -D_REENTRANT
>
> only in the libevent and vt sub-configure portions.
>
> When configured for gcc on Solaris-11 I see the following in configure
>
> checking for C optimization flags... -m64 -D_REENTRANT -g
> -finline-functions -fno-strict-aliasing
>
> but with CC=cc the equivalent line is
>
> checking for C optimization flags... -m64 -g
>
> In both cases the "-m64" is from the CFLAGS I have passed to configure.
>
> However, when I use CFLAGS="-m64 -D_REENTRANT" the problem DOES NOT go away.
> I see
>
> [pcp-j-20:24740] mca_oob_tcp_accept: accept() failed: Error 0 (11).
> 
> A process or daemon was unable to complete a TCP connection
> to another process:
>   Local host:pcp-j-20
>   Remote host:   172.18.0.120
> This is usually caused by a firewall on the remote host. Please
> check that any firewall (e.g., iptables) has been disabled and
> try again.
> 
>
> which is at least appears to have a non-zero errno.
> A quick grep through /usr/include/sys/errno shows 11 is EAGAIN.
>
> With the oob.patch you provided the failed accept goes away, BUT the
> connection still fails:
>
> 
> A process or daemon was unable to complete a TCP connection
> to another process:
>   Local host:pcp-j-20
>   Remote host:   172.18.0.120
> This is usually caused by a firewall on the remote host. Please
> check that any firewall (e.g., iptables) has been disabled and
> try again.
> 
>
>
> Use of "-mca oob_tcp_if_include bge0" to use a single interface did not fix
> this.
>
>
> -Paul
>
> On Mon, Dec 15, 2014 at 7:18 PM, Paul Hargrove <phhargr...@lbl.gov> wrote:
>> Gilles,
>>
>> I am NOT seeing the problem with gcc.
>> It is only occurring with the Studio compilers.
>>
>> As I've already reported, I have tried adding either "-mt" or "-mt=yes" to
>> both LDFLAGS and --with-wrapper-ldflags.
>>
>> The "cc" manpage (on the Solaris-10 system I can get to right now) says:
>>
>>  -mt  Compile and link for multithreaded code.
>>
>>   This option passes -D_REENTRANT to the preprocessor and
>>   passes -lthread in the correct order to ld.
>>
>>   The -mt option is required if the application or
>>   libraries are multithreaded.
>>
>>   To ensure proper library linking order, you must use
>>   this option, rather than -lthread, to link with lib-
>>   thread.
>>
>>   If you are using POSIX threads, you must link with the
>>   options -mt -lpthread.  The -mt option is necessary
>>   because libC and libCrun need libthread for a mul-
>>   tithreaded application.
>>
>>   If you compile and link in separate steps and you com-
>>   pile with -mt, you might get unexpected results. If you
>>   compile one translation unit with -mt, compile all
>>   units of the program with -mt.
>>
>> I cannot connect to my Solaris-11 system right now, but I recall the text
>> to be quite similar.
>>
>> -Paul
>>
>> On Mon, Dec 15, 2014 at 7:12 PM, Gilles Gouaillardet <
>> gilles.gouaillar...@iferc.org> wrote:
>>
>>>  Paul,
>>>
>>> did you manually set -mt ?
>>>
>>> if i remember correctly, solaris 11 (at least with gcc compilers) do not
>>> need any flags
>>> (except the -D_REENTRANT that is added automatically)
>>>
>>> Cheers,
>>>
>>> Gilles
>>>
>>>
>>> On 2014/12/16 12:10, Paul Hargrove wrote:
>>>
>>> Gilles,
>>>
>>> I will try the patch when I can.
>>> However, our network is undergoing network maintenance right now, leaving
>>> me unable to reach the necessary hosts.
>>>
>>> As for -D_RE

Re: [OMPI devel] 1.8.4rc Status

[Gilles Gouaillardet]

Paul,

the root cause -D_REENTRANT is not set automatically is we test the
__sun__ macro and 12.4 compiler defines only __sun and sun

I will make a fix for that ...

Cheers,

Gilles

On 2014/12/16 16:00, Paul Hargrove wrote:
> Gilles,
>
> I looked again carefully and I am *NOT* finding -D_REENTRANT passed to most
> compilations.
> It appears to be used for building libevent and vt, but nothing else.
> The output from configure contains
>
> checking if more special flags are required for pthreads... -D_REENTRANT
>
> only in the libevent and vt sub-configure portions.
>
> When configured for gcc on Solaris-11 I see the following in configure
>
> checking for C optimization flags... -m64 -D_REENTRANT -g
> -finline-functions -fno-strict-aliasing
>
> but with CC=cc the equivalent line is
>
> checking for C optimization flags... -m64 -g
>
> In both cases the "-m64" is from the CFLAGS I have passed to configure.
>
> However, when I use CFLAGS="-m64 -D_REENTRANT" the problem DOES NOT go away.
> I see
>
> [pcp-j-20:24740] mca_oob_tcp_accept: accept() failed: Error 0 (11).
> 
> A process or daemon was unable to complete a TCP connection
> to another process:
>   Local host:pcp-j-20
>   Remote host:   172.18.0.120
> This is usually caused by a firewall on the remote host. Please
> check that any firewall (e.g., iptables) has been disabled and
> try again.
> 
>
> which is at least appears to have a non-zero errno.
> A quick grep through /usr/include/sys/errno shows 11 is EAGAIN.
>
> With the oob.patch you provided the failed accept goes away, BUT the
> connection still fails:
>
> 
> A process or daemon was unable to complete a TCP connection
> to another process:
>   Local host:pcp-j-20
>   Remote host:   172.18.0.120
> This is usually caused by a firewall on the remote host. Please
> check that any firewall (e.g., iptables) has been disabled and
> try again.
> 
>
>
> Use of "-mca oob_tcp_if_include bge0" to use a single interface did not fix
> this.
>
>
> -Paul
>
> On Mon, Dec 15, 2014 at 7:18 PM, Paul Hargrove <phhargr...@lbl.gov> wrote:
>> Gilles,
>>
>> I am NOT seeing the problem with gcc.
>> It is only occurring with the Studio compilers.
>>
>> As I've already reported, I have tried adding either "-mt" or "-mt=yes" to
>> both LDFLAGS and --with-wrapper-ldflags.
>>
>> The "cc" manpage (on the Solaris-10 system I can get to right now) says:
>>
>>  -mt  Compile and link for multithreaded code.
>>
>>   This option passes -D_REENTRANT to the preprocessor and
>>   passes -lthread in the correct order to ld.
>>
>>   The -mt option is required if the application or
>>   libraries are multithreaded.
>>
>>   To ensure proper library linking order, you must use
>>   this option, rather than -lthread, to link with lib-
>>   thread.
>>
>>   If you are using POSIX threads, you must link with the
>>   options -mt -lpthread.  The -mt option is necessary
>>   because libC and libCrun need libthread for a mul-
>>   tithreaded application.
>>
>>   If you compile and link in separate steps and you com-
>>   pile with -mt, you might get unexpected results. If you
>>   compile one translation unit with -mt, compile all
>>   units of the program with -mt.
>>
>> I cannot connect to my Solaris-11 system right now, but I recall the text
>> to be quite similar.
>>
>> -Paul
>>
>> On Mon, Dec 15, 2014 at 7:12 PM, Gilles Gouaillardet <
>> gilles.gouaillar...@iferc.org> wrote:
>>
>>>  Paul,
>>>
>>> did you manually set -mt ?
>>>
>>> if i remember correctly, solaris 11 (at least with gcc compilers) do not
>>> need any flags
>>> (except the -D_REENTRANT that is added automatically)
>>>
>>> Cheers,
>>>
>>> Gilles
>>>
>>>
>>> On 2014/12/16 12:10, Paul Hargrove wrote:
>>>
>>> Gilles,
>>>
>>> I will try the patch when I can.
>>> However, our network is undergoing network maintenance right now, leaving
>>> me unable to reach the necessary hosts.
>>>
>>> As for -D_REENTRANT, I had already reported having ve

Re: [OMPI devel] 1.8.4rc Status

[Gilles Gouaillardet]

Paul,

did you manually set -mt ?

if i remember correctly, solaris 11 (at least with gcc compilers) do not
need any flags
(except the -D_REENTRANT that is added automatically)

Cheers,

Gilles

On 2014/12/16 12:10, Paul Hargrove wrote:
> Gilles,
>
> I will try the patch when I can.
> However, our network is undergoing network maintenance right now, leaving
> me unable to reach the necessary hosts.
>
> As for -D_REENTRANT, I had already reported having verified in the "make"
> output that it had been added automatically.
>
> Additionally, the docs say that "-mt" *also* passes -D_REENTRANT to the
> preprocessor.
>
> -Paul
>
> On Mon, Dec 15, 2014 at 6:07 PM, Gilles Gouaillardet <
> gilles.gouaillar...@iferc.org> wrote:
>>  Paul,
>>
>> could you please make sure configure added  "-D_REENTRANT" to the CFLAGS ?
>> /* otherwise, errno is a global variable instead of a per thread variable,
>> which can
>> explains some weird behaviour. note this should have been already fixed */
>>
>> assuming -D_REENTRANT is set, could you please give the attached patch a
>> try ?
>>
>> i suspect the CLOSE_THE_SOCKET macro resets errno, and hence the confusing
>> error message
>> e.g. failed: Error 0 (0)
>>
>> FWIW, master is also affected.
>>
>> Cheers,
>>
>> Gilles
>>
>>
>> On 2014/12/16 10:47, Paul Hargrove wrote:
>>
>> I have tried with a oob_tcp_if_include setting so that there is now only 1
>> interface.
>> Even with just one interface and -mt=yes in both LDFLAGS and
>> wrapper-ldflags I *still* getting messages like
>>
>> [pcp-j-20:11470] mca_oob_tcp_accept: accept() failed: Error 0 (0).
>> 
>> A process or daemon was unable to complete a TCP connection
>> to another process:
>>   Local host:pcp-j-20
>>   Remote host:   172.16.0.120
>> This is usually caused by a firewall on the remote host. Please
>> check that any firewall (e.g., iptables) has been disabled and
>> try again.
>> 
>>
>>
>> I am getting less certain that my speculation about thread-safe libs is
>> correct.
>>
>> -Paul
>>
>> On Mon, Dec 15, 2014 at 1:24 PM, Paul Hargrove <phhargr...@lbl.gov> 
>> <phhargr...@lbl.gov> wrote:
>>
>>  A little more reading finds that...
>>
>> Docs says that one needs "-mt" without the "=yes".
>> That will work for both old and new compilers, where "-mt=yes" chokes
>> older ones.
>>
>> Also, man pages say "-mt" must come before "-lpthread" in the link command.
>>
>> -Paul
>>
>> On Mon, Dec 15, 2014 at 12:52 PM, Paul Hargrove <phhargr...@lbl.gov> 
>> <phhargr...@lbl.gov>
>> wrote:
>>
>>
>> On Mon, Dec 15, 2014 at 5:35 AM, Ralph Castain <r...@open-mpi.org> 
>> <r...@open-mpi.org> wrote:
>>
>>  7. Linkage issue on Solaris-11 reported by Paul Hargrove. Missing the
>> multi-threaded C libraries, apparently need "-mt=yes" in both compile and
>> link. Need someone to investigate.
>>
>>
>> The lack of multi-thread libraries is my SPECULATION.
>>
>> The fact that configuring with LDFLAGS=-mt=yes did not help may or may
>> not prove anything.
>> I didn't see them in "mpicc -show" and so maybe they needed to be in
>> wrapper-ldflags instead.
>> My time this week is quite limited, but I can "fire an forget" tests of
>> any tarballs you provide.
>>
>> -Paul
>>
>> --
>> Paul H. Hargrove  phhargr...@lbl.gov
>> Computer Languages & Systems Software (CLaSS) Group
>> Computer Science Department   Tel: +1-510-495-2352
>> Lawrence Berkeley National Laboratory Fax: +1-510-486-6900
>>
>>
>> --
>> Paul H. Hargrove  phhargr...@lbl.gov
>> Computer Languages & Systems Software (CLaSS) Group
>> Computer Science Department   Tel: +1-510-495-2352
>> Lawrence Berkeley National Laboratory Fax: +1-510-486-6900
>>
>>
>>
>> ___
>> devel mailing listde...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/devel/2014/12/16607.php
>>
>>
>>
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post:
>> http://www.open-mpi.org/community/lists/devel/2014/12/16608.php
>>
>
>
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2014/12/16610.php

Re: [OMPI devel] 1.8.4rc Status

[Gilles Gouaillardet]

Paul,

could you please make sure configure added  "-D_REENTRANT" to the CFLAGS ?
/* otherwise, errno is a global variable instead of a per thread
variable, which can
explains some weird behaviour. note this should have been already fixed */

assuming -D_REENTRANT is set, could you please give the attached patch a
try ?

i suspect the CLOSE_THE_SOCKET macro resets errno, and hence the
confusing error message
e.g. failed: Error 0 (0)

FWIW, master is also affected.

Cheers,

Gilles

On 2014/12/16 10:47, Paul Hargrove wrote:
> I have tried with a oob_tcp_if_include setting so that there is now only 1
> interface.
> Even with just one interface and -mt=yes in both LDFLAGS and
> wrapper-ldflags I *still* getting messages like
>
> [pcp-j-20:11470] mca_oob_tcp_accept: accept() failed: Error 0 (0).
> 
> A process or daemon was unable to complete a TCP connection
> to another process:
>   Local host:pcp-j-20
>   Remote host:   172.16.0.120
> This is usually caused by a firewall on the remote host. Please
> check that any firewall (e.g., iptables) has been disabled and
> try again.
> 
>
>
> I am getting less certain that my speculation about thread-safe libs is
> correct.
>
> -Paul
>
> On Mon, Dec 15, 2014 at 1:24 PM, Paul Hargrove  wrote:
>> A little more reading finds that...
>>
>> Docs says that one needs "-mt" without the "=yes".
>> That will work for both old and new compilers, where "-mt=yes" chokes
>> older ones.
>>
>> Also, man pages say "-mt" must come before "-lpthread" in the link command.
>>
>> -Paul
>>
>> On Mon, Dec 15, 2014 at 12:52 PM, Paul Hargrove 
>> wrote:
>>>
>>> On Mon, Dec 15, 2014 at 5:35 AM, Ralph Castain  wrote:
 7. Linkage issue on Solaris-11 reported by Paul Hargrove. Missing the
 multi-threaded C libraries, apparently need "-mt=yes" in both compile and
 link. Need someone to investigate.
>>>
>>> The lack of multi-thread libraries is my SPECULATION.
>>>
>>> The fact that configuring with LDFLAGS=-mt=yes did not help may or may
>>> not prove anything.
>>> I didn't see them in "mpicc -show" and so maybe they needed to be in
>>> wrapper-ldflags instead.
>>> My time this week is quite limited, but I can "fire an forget" tests of
>>> any tarballs you provide.
>>>
>>> -Paul
>>>
>>> --
>>> Paul H. Hargrove  phhargr...@lbl.gov
>>> Computer Languages & Systems Software (CLaSS) Group
>>> Computer Science Department   Tel: +1-510-495-2352
>>> Lawrence Berkeley National Laboratory Fax: +1-510-486-6900
>>>
>>
>> --
>> Paul H. Hargrove  phhargr...@lbl.gov
>> Computer Languages & Systems Software (CLaSS) Group
>> Computer Science Department   Tel: +1-510-495-2352
>> Lawrence Berkeley National Laboratory Fax: +1-510-486-6900
>>
>
>
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2014/12/16607.php

diff --git a/orte/mca/oob/tcp/oob_tcp_listener.c b/orte/mca/oob/tcp/oob_tcp_listener.c
index b6d2ad8..87ff08d 100644
--- a/orte/mca/oob/tcp/oob_tcp_listener.c
+++ b/orte/mca/oob/tcp/oob_tcp_listener.c
@@ -14,6 +14,8 @@
  * Copyright (c) 2009-2014 Cisco Systems, Inc.  All rights reserved.
  * Copyright (c) 2011  Oak Ridge National Labs.  All rights reserved.
  * Copyright (c) 2013-2014 Intel, Inc.  All rights reserved.
+ * Copyright (c) 2014  Research Organization for Information Science
+ * and Technology (RIST). All rights reserved.
  * $COPYRIGHT$
  *
  * Additional copyrights may follow
@@ -729,7 +731,6 @@ static void* listen_thread(opal_object_t *obj)
 if (pending_connection->fd < 0) {
 if (opal_socket_errno != EAGAIN || 
 opal_socket_errno != EWOULDBLOCK) {
-CLOSE_THE_SOCKET(pending_connection->fd);
 if (EMFILE == opal_socket_errno) {
 ORTE_ERROR_LOG(ORTE_ERR_SYS_LIMITS_SOCKETS);
 orte_show_help("help-orterun.txt", "orterun:sys-limit-sockets", true);
@@ -737,6 +738,7 @@ static void* listen_thread(opal_object_t *obj)
 opal_output(0, "mca_oob_tcp_accept: accept() failed: %s (%d).",
 strerror(opal_socket_errno), opal_socket_errno);
 }
+CLOSE_THE_SOCKET(pending_connection->fd);
 OBJ_RELEASE(pending_connection);
 goto done;
 }

Re: [OMPI devel] OMPI devel] [1.8.4rc2] orterun SEGVs on Solaris-10/SPARC

[Gilles Gouaillardet]

Ralph,

I cannot find a case for the %u format is guess_strlen
And since the default does not invoke va_arg()
I
it seems strlen is invoked on nnuma instead of arch

Makes sense ?

Cheers,

Gilles

Ralph Castain  wrote:
>Afraid I’m drawing a blank, Paul - I can’t see how we got to a bad address 
>down there. This is at the beginning of orte_init, so there are no threads 
>running nor has anything much happened.
>
>
>Do you have any suggestions?
>
>
>
>On Dec 12, 2014, at 9:02 AM, Paul Hargrove  wrote:
>
>
>Ralph,
>
>
>The "arch" variable looks fine:
>
>Current function is opal_hwloc_base_get_topo_signature
>
> 2134                    nnuma, nsocket, nl3, nl2, nl1, ncore, nhwt, arch);
>
>(dbx) print arch
>
>arch = 0x1001700a0 "sun4v"
>
>
>And so is "fmt":
>
>
>Current function is opal_asprintf
>
>  194       length = opal_vasprintf(ptr, fmt, ap);
>
>(dbx) print fmt
>
>fmt = 0x7eeada98 "%uN:%uS:%uL3:%uL2:%uL1:%uC:%uH:%s"
>
>
>However, things have gone bad in guess_strlen():
>
>
>Current function is guess_strlen
>
>   71                       len += (int)strlen(sarg);
>
>(dbx) print sarg
>
>sarg = 0x2 ""
>
>
>-Paul
>
>
>On Fri, Dec 12, 2014 at 2:24 AM, Ralph Castain  wrote:
>
>Hmmm….this is really odd. I actually do have a protection for that arch value 
>being NULL, and you are in the code section when it isn’t.
>
>
>Do you still have the core file around? If so, can you print out the value of 
>the “arch” variable? It would be in the opal_hwloc_base_get_topo_signature 
>level.
>
>
>I’m wondering if that value has been hosed, and the problem is memory 
>corruption somewhere.
>
>
>
>On Dec 11, 2014, at 8:56 PM, Ralph Castain  wrote:
>
>
>Thanks Paul - I will post a fix for this tomorrow. Looks like Sparc isn’t 
>returning an architecture type for some reason, and I didn’t protect against 
>it.
>
>
>
>On Dec 11, 2014, at 7:39 PM, Paul Hargrove  wrote:
>
>
>Backtrace for the Solaris-10/SPARC SEGV appears below.
>
>I've changed the subject line to distinguish this from the earlier report.
>
>
>-Paul
>
>
>program terminated by signal SEGV (no mapping at the fault address)
>
>0x7d93b634: strlen+0x0014:      lduh     [%o2], %o1
>
>Current function is guess_strlen
>
>   71                       len += (int)strlen(sarg);
>
>(dbx) where
>
>  [1] strlen(0x2, 0x7300, 0x2, 0x80808080, 0x2, 0x80808080), at 
>0x7d93b634 
>
>=>[2] guess_strlen(fmt = 0x7eeada98 
>"%uN:%uS:%uL3:%uL2:%uL1:%uC:%uH:%s", ap = 0x7058), line 71 in 
>"printf.c"
>
>  [3] opal_vasprintf(ptr = 0x70b8, fmt = 0x7eeada98 
>"%uN:%uS:%uL3:%uL2:%uL1:%uC:%uH:%s", ap = 0x7050), line 218 in 
>"printf.c"
>
>  [4] opal_asprintf(ptr = 0x70b8, fmt = 0x7eeada98 
>"%uN:%uS:%uL3:%uL2:%uL1:%uC:%uH:%s", ... = 0x807ede0103, ...), line 194 in 
>"printf.c"
>
>  [5] opal_hwloc_base_get_topo_signature(topo = 0x100128ea0), line 2134 in 
>"hwloc_base_util.c"
>
>  [6] rte_init(), line 205 in "ess_hnp_module.c"
>
>  [7] orte_init(pargc = 0x761c, pargv = 0x7610, flags 
>= 4U), line 148 in "orte_init.c"
>
>  [8] orterun(argc = 7, argv = 0x77a8), line 856 in "orterun.c"
>
>  [9] main(argc = 7, argv = 0x77a8), line 13 in "main.c"
>
>
>On Thu, Dec 11, 2014 at 7:17 PM, Ralph Castain  wrote:
>
>No, that looks different - it’s failing in mpirun itself. Can you get a line 
>number on it?
>
>
>Sorry for delay - I’m generating rc3 now
>
>
>
>On Dec 11, 2014, at 6:59 PM, Paul Hargrove  wrote:
>
>
>Don't see an rc3 yet.
>
>
>My Solaris-10/SPARC runs fail slightly differently (see below).
>
>It looks sufficiently similar that it MIGHT be the same root cause.
>
>However, lacking an rc3 to test I figured it would be better to report this 
>than to ignore it.
>
>
>The problem is present with both V8+ and V9 ABIs, and with both Gnu and Sun 
>compilers.
>
>
>-Paul
>
>
>[niagara1:29881] *** Process received signal ***
>
>[niagara1:29881] Signal: Segmentation Fault (11)
>
>[niagara1:29881] Signal code: Address not mapped (1)
>
>[niagara1:29881] Failing at address: 2
>
>/sandbox/hargrove/OMPI/openmpi-1.8.4rc2-solaris10-sparcT2-gcc346-v8plus/INST/lib/libopen-pal.so.6.2.1:opal_backtrace_print+0x24
>
>/sandbox/hargrove/OMPI/openmpi-1.8.4rc2-solaris10-sparcT2-gcc346-v8plus/INST/lib/libopen-pal.so.6.2.1:0xaa160
>
>/lib/libc.so.1:0xc5364
>
>/lib/libc.so.1:0xb9e64
>
>/lib/libc.so.1:strlen+0x14 [ Signal 11 (SEGV)]
>
>/sandbox/hargrove/OMPI/openmpi-1.8.4rc2-solaris10-sparcT2-gcc346-v8plus/INST/lib/libopen-pal.so.6.2.1:opal_vasprintf+0x20
>
>/sandbox/hargrove/OMPI/openmpi-1.8.4rc2-solaris10-sparcT2-gcc346-v8plus/INST/lib/libopen-pal.so.6.2.1:opal_asprintf+0x30
>
>/sandbox/hargrove/OMPI/openmpi-1.8.4rc2-solaris10-sparcT2-gcc346-v8plus/INST/lib/libopen-pal.so.6.2.1:opal_hwloc_base_get_topo_signature+0x24c
>

Re: [OMPI devel] OMPI devel] [OMPI users] OpenMPI 1.8.4 and hwloc in Fedora 14 using a beta gcc 5.0 compiler.

[Gilles Gouaillardet]

Ralph,

I can do that starting from monday

Cheers,

Gilles

Ralph Castain <r...@open-mpi.org> wrote:
>Thanks Brice!
>
>Our 1.8 branch probably has another 2 or so years in it, but I think we can 
>lock it down fairly soon. Since we’ve shaken a lot of the bugs out of 1.8, we 
>are now seeing the “adoption wave” that is causing bug reports. Once we get 
>thru this, I expect things will settle down again.
>
>I know Jeff is hosed, and I’m likewise next week. Can someone create a PR to 
>update 1.8 with these patches?
>
>
>> On Dec 12, 2014, at 12:32 AM, Brice Goglin <brice.gog...@inria.fr> wrote:
>> 
>> Le 12/12/2014 07:36, Gilles Gouaillardet a écrit :
>>> Brice,
>>> 
>>> ompi master is based on hwloc 1.9.1, isn't it ?
>> 
>> Yes sorry, I am often confused by all these OMPI vs hwloc branch numbers.
>> 
>>> 
>>> if some backport is required for hwloc 1.7.2 (used by ompi v1.8), then
>>> could you please update the hwloc v1.7 branch ?
>> 
>> Done. I pushed 14 commits there. This branch lags significantly behind 
>> master and v1.10 so I don't think I'll be able to maintain it much longer.
>> 
>> Brice
>> 
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/devel/2014/12/16538.php
>
>___
>devel mailing list
>de...@open-mpi.org
>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>Link to this post: 
>http://www.open-mpi.org/community/lists/devel/2014/12/16540.php

Re: [OMPI devel] [OMPI users] OpenMPI 1.8.4 and hwloc in Fedora 14 using a beta gcc 5.0 compiler.

[Gilles Gouaillardet]

Brice,

ompi master is based on hwloc 1.9.1, isn't it ?

if some backport is required for hwloc 1.7.2 (used by ompi v1.8), then
could you please update the hwloc v1.7 branch ?

Cheers,

Gilles

On 2014/12/12 15:16, Brice Goglin wrote:
> Yes.
>
> In theory, everything that's in hwloc/v1.8 should go to OMPI/master.
>
> And most of it should go to v1.8 too, but that may require some
> backporting rework. I can update hwloc/v1.7 if that helps.
>
> Brice
>
>
>
> Le 12/12/2014 03:10, Gilles Gouaillardet a écrit :
>> Brice,
>>
>> should this fix be backported to both master and v1.8 ?
>>
>> Cheers,
>>
>> Gilles
>>
>> On 2014/12/12 7:46, Brice Goglin wrote:
>>> This problem was fixed in hwloc upstream recently.
>>>
>>> https://github.com/open-mpi/hwloc/commit/790aa2e1e62be6b4f37622959de9ce3766ebc57e
>>> Brice
>>>
>>>
>>> Le 11/12/2014 23:40, Jorge D'Elia a écrit :
>>>> Dear Jeff,
>>>>
>>>> Our updates of OpenMPI to 1.8.3 (and 1.8.4) were 
>>>> all OK using Fedora >= 17 and system gcc compilers
>>>> on ia32 or ia64 machines. 
>>>>
>>>> However, the "make all" step failed using Fedora 14 
>>>> with a beta gcc 5.0 compiler on an ia32 machine 
>>>> with message like:
>>>>
>>>> Error: symbol `Lhwloc1' is already defined
>>>>
>>>> A roundabout way to solve it was perform, first, 
>>>> a separated installation of the hwloc package (we use 
>>>> Release v1.10.0 (stable)) and, second, configure 
>>>> OpenMPI using its flag: 
>>>>
>>>>   --with-hwloc=${HWLOC_HOME}
>>>>
>>>> although, in this way, the include and library path 
>>>> must be given, e.g.
>>>>
>>>>  export CFLAGS="-I/usr/beta/hwloc/include" ; echo ${CFLAGS}
>>>>  export LDFLAGS="-L/usr/beta/hwloc/lib"; echo ${LDFLAGS}
>>>>  export LIBS="-lhwloc" ; echo ${LIBS}
>>>>
>>>> In order to verify that the hwloc works OK, it would be useful 
>>>> to include in the OpenMPI distribution a simple test like
>>>>
>>>> $ gcc ${CFLAGS} ${LDFLAGS} -o hwloc-hello.exe hwloc-hello.c ${LIBS}
>>>> $ ./hwloc-hello.exe
>>>>
>>>> (we apologize to forget to use the --with-hwloc-libdir flag ...).
>>>>
>>>> With this previous step we could overcome the fatal error 
>>>> in the configuration step related to the hwloc package.
>>>>
>>>> This (fixed) trouble in the configuration step is the same 
>>>> as the reported as:
>>>>
>>>> Open MPI 1.8.1: "make all" error: symbol `Lhwloc1' is already defined
>>>>
>>>> on 2014-08-12 15:08:38
>>>>
>>>>
>>>> Regards,
>>>> Jorge.
>>>>
>>>> - Mensaje original -
>>>>> De: "Jorge D'Elia" <jde...@intec.unl.edu.ar>
>>>>> Para: "Open MPI Users" <us...@open-mpi.org>
>>>>> Enviado: Martes, 12 de Agosto 2014 16:08:38
>>>>> Asunto: Re: [OMPI users] Open MPI 1.8.1: "make all" error: symbol 
>>>>> `Lhwloc1' is already defined
>>>>>
>>>>> Dear Jeff,
>>>>>
>>>>> These new versions of the tgz files replace the previous ones:
>>>>> I had used an old outdated session environment. However, the
>>>>> configuration and installation was OK again in each case.
>>>>> Sorry for the noise caused by the previous tgz files.
>>>>>
>>>>> Regards,
>>>>> Jorge.
>>>>>
>>>>> - Mensaje original -
>>>>>> De: "Jorge D'Elia" <jde...@intec.unl.edu.ar>
>>>>>> Para: "Open MPI Users" <us...@open-mpi.org>
>>>>>> Enviados: Martes, 12 de Agosto 2014 15:16:19
>>>>>> Asunto: Re: [OMPI users] Open MPI 1.8.1: "make all" error: symbol 
>>>>>> `Lhwloc1'
>>>>>> is already defined
>>>>>>
>>>>>> Dear Jeff,
>>>>>>
>>>>>> - Mensaje original -
>>>>>>> De: "Jeff Squyres (jsquyres)" <jsquy...@cisco.com>
>>>>>>> Para: "Open MPI User's List" <us...@open-mpi.org>
>>>>>>> Enviado: Lu

Re: [OMPI devel] [1.8.4rc3] false report of no loopback interface + segv at exit

[Gilles Gouaillardet]

Ralph,

here is my understanding of what happens on Linux :

lo: 127.0.0.1/8
eth0: 192.168.122.101/24

mpirun --mca orte_oob_tcp_if_include eth0 ...

so the mpi task tries to contact orted/mpirun on 192.168.0.1/24

that works just fine if the loopback interface is active,
and that hangs if there is no loopback interface.


imho that is a linux oddity, and OMPI has nothing to do with it

Cheers,

Gilles

[root@slurm1 ~]# ping -c 3 192.168.122.101
PING 192.168.122.101 (192.168.122.101) 56(84) bytes of data.
64 bytes from 192.168.122.101: icmp_seq=1 ttl=64 time=0.013 ms
64 bytes from 192.168.122.101: icmp_seq=2 ttl=64 time=0.009 ms
64 bytes from 192.168.122.101: icmp_seq=3 ttl=64 time=0.011 ms

--- 192.168.122.101 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.009/0.011/0.013/0.001 ms



[root@slurm1 ~]# ifdown lo
[root@slurm1 ~]# ping -c 3 192.168.122.101
PING 192.168.122.101 (192.168.122.101) 56(84) bytes of data.

--- 192.168.122.101 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 11999ms


On 2014/12/12 13:54, Ralph Castain wrote:
> I honestly think it has to be a selected interface, Gilles, else we will fail 
> to connect.
>
>> On Dec 11, 2014, at 8:26 PM, Gilles Gouaillardet 
>> <gilles.gouaillar...@iferc.org> wrote:
>>
>> Paul,
>>
>> about the five warnings :
>> can you confirm you are running mpirun *not* on n15 nor n16 ?
>> if my guess is correct, then you can get up to 5 warnings : mpirun + 2 orted 
>> + 2 mpi tasks
>>
>> do you have any oob_tcp_if_include or oob_tcp_if_exclude settings in your 
>> openmpi-mca-params.conf ?
>>
>> here is attached a patch to fix this issue.
>> what we really want is test there is a loopback interface, period.
>> the current code (my bad for not having reviewed in a timely manner) seems 
>> to check
>> there is a *selected* loopback interface.
>>
>> Cheers,
>>
>> Gilles
>>
>> On 2014/12/12 13:15, Paul Hargrove wrote:
>>> Ralph,
>>>
>>> Sorry to be the bearer of more bad news.
>>> The "good" news is I've seen the new warning regarding the lack of a
>>> loopback interface.
>>> The BAD news is that it is occurring on a Linux cluster that I'ver verified
>>> DOES have 'lo' configured on the front-end and compute nodes (UP and
>>> RUNNING according to ifconfig).
>>>
>>> Though run with "-np 2" the warning appears FIVE times.
>>> ADDITIONALLY, there is a SEGV at exit!
>>>
>>> Unfortunately, despite configuring with --enable-debug, I didn't get line
>>> numbers from the core (and there was no backtrace printed).
>>>
>>> All of this appears below (and no, "-mca mtl psm" is not a typo or a joke).
>>>
>>> Let me know what tracing flags to apply to gather the info needed to debug
>>> this.
>>>
>>> -Paul
>>>
>>>
>>> $ mpirun -mca btl sm,self -np 2 -host n15,n16 -mca mtl psm examples/ring_c
>>> --
>>> WARNING: No loopback interface was found. This can cause problems
>>> when we spawn processes as they are likely to be unable to connect
>>> back to their host daemon. Sadly, it may take awhile for the connect
>>> attempt to fail, so you may experience a significant hang time.
>>>
>>> You may wish to ctrl-c out of your job and activate loopback support
>>> on at least one interface before trying again.
>>>
>>> --
>>> [... above message FOUR more times ...]
>>> Process 1 exiting
>>> Process 0 sending 10 to 1, tag 201 (2 processes in ring)
>>> Process 0 sent to 1
>>> Process 0 decremented value: 9
>>> Process 0 decremented value: 8
>>> Process 0 decremented value: 7
>>> Process 0 decremented value: 6
>>> Process 0 decremented value: 5
>>> Process 0 decremented value: 4
>>> Process 0 decremented value: 3
>>> Process 0 decremented value: 2
>>> Process 0 decremented value: 1
>>> Process 0 decremented value: 0
>>> Process 0 exiting
>>> --
>>> mpirun noticed that process rank 0 with PID 0 on node n15 exited on signal
>>> 11 (Segmentation fault).
>>> --
>>>
>>> $ /sbin/ifconfig lo
>>> loLink encap:Local Loopback
>&g

Re: [OMPI devel] [1.8.4rc3] false report of no loopback interface + segv at exit

[Gilles Gouaillardet]

Paul,

about the five warnings :
can you confirm you are running mpirun *not* on n15 nor n16 ?
if my guess is correct, then you can get up to 5 warnings : mpirun + 2
orted + 2 mpi tasks

do you have any oob_tcp_if_include or oob_tcp_if_exclude settings in
your openmpi-mca-params.conf ?

here is attached a patch to fix this issue.
what we really want is test there is a loopback interface, period.
the current code (my bad for not having reviewed in a timely manner)
seems to check
there is a *selected* loopback interface.

Cheers,

Gilles

On 2014/12/12 13:15, Paul Hargrove wrote:
> Ralph,
>
> Sorry to be the bearer of more bad news.
> The "good" news is I've seen the new warning regarding the lack of a
> loopback interface.
> The BAD news is that it is occurring on a Linux cluster that I'ver verified
> DOES have 'lo' configured on the front-end and compute nodes (UP and
> RUNNING according to ifconfig).
>
> Though run with "-np 2" the warning appears FIVE times.
> ADDITIONALLY, there is a SEGV at exit!
>
> Unfortunately, despite configuring with --enable-debug, I didn't get line
> numbers from the core (and there was no backtrace printed).
>
> All of this appears below (and no, "-mca mtl psm" is not a typo or a joke).
>
> Let me know what tracing flags to apply to gather the info needed to debug
> this.
>
> -Paul
>
>
> $ mpirun -mca btl sm,self -np 2 -host n15,n16 -mca mtl psm examples/ring_c
> --
> WARNING: No loopback interface was found. This can cause problems
> when we spawn processes as they are likely to be unable to connect
> back to their host daemon. Sadly, it may take awhile for the connect
> attempt to fail, so you may experience a significant hang time.
>
> You may wish to ctrl-c out of your job and activate loopback support
> on at least one interface before trying again.
>
> --
> [... above message FOUR more times ...]
> Process 1 exiting
> Process 0 sending 10 to 1, tag 201 (2 processes in ring)
> Process 0 sent to 1
> Process 0 decremented value: 9
> Process 0 decremented value: 8
> Process 0 decremented value: 7
> Process 0 decremented value: 6
> Process 0 decremented value: 5
> Process 0 decremented value: 4
> Process 0 decremented value: 3
> Process 0 decremented value: 2
> Process 0 decremented value: 1
> Process 0 decremented value: 0
> Process 0 exiting
> --
> mpirun noticed that process rank 0 with PID 0 on node n15 exited on signal
> 11 (Segmentation fault).
> --
>
> $ /sbin/ifconfig lo
> loLink encap:Local Loopback
>   inet addr:127.0.0.1  Mask:255.0.0.0
>   inet6 addr: ::1/128 Scope:Host
>   UP LOOPBACK RUNNING  MTU:16436  Metric:1
>   RX packets:481228 errors:0 dropped:0 overruns:0 frame:0
>   TX packets:481228 errors:0 dropped:0 overruns:0 carrier:0
>   collisions:0 txqueuelen:0
>   RX bytes:81039065 (77.2 MiB)  TX bytes:81039065 (77.2 MiB)
>
> $ ssh n15 /sbin/ifconfig lo
> loLink encap:Local Loopback
>   inet addr:127.0.0.1  Mask:255.0.0.0
>   inet6 addr: ::1/128 Scope:Host
>   UP LOOPBACK RUNNING  MTU:16436  Metric:1
>   RX packets:24885 errors:0 dropped:0 overruns:0 frame:0
>   TX packets:24885 errors:0 dropped:0 overruns:0 carrier:0
>   collisions:0 txqueuelen:0
>   RX bytes:1509940 (1.4 MiB)  TX bytes:1509940 (1.4 MiB)
>
> $ ssh n16 /sbin/ifconfig lo
> loLink encap:Local Loopback
>   inet addr:127.0.0.1  Mask:255.0.0.0
>   inet6 addr: ::1/128 Scope:Host
>   UP LOOPBACK RUNNING  MTU:16436  Metric:1
>   RX packets:24938 errors:0 dropped:0 overruns:0 frame:0
>   TX packets:24938 errors:0 dropped:0 overruns:0 carrier:0
>   collisions:0 txqueuelen:0
>   RX bytes:1543408 (1.4 MiB)  TX bytes:1543408 (1.4 MiB)
>
> $ gdb examples/ring_c core.29728
> [...]
> (gdb) where
> #0  0x002a97a19980 in ?? ()
> #1  
> #2  0x003a6d40607c in _Unwind_FindEnclosingFunction () from
> /lib64/libgcc_s.so.1
> #3  0x003a6d406b57 in _Unwind_RaiseException () from
> /lib64/libgcc_s.so.1
> #4  0x003a6d406c4c in _Unwind_ForcedUnwind () from /lib64/libgcc_s.so.1
> #5  0x003a6c30ac50 in __pthread_unwind () from
> /lib64/tls/libpthread.so.0
> #6  0x003a6c305202 in sigcancel_handler () from
> /lib64/tls/libpthread.so.0
> #7  
> #8  0x003a6b6bd9a2 in poll () from /lib64/tls/libc.so.6
> #9  0x002a978f8f7d in ?? ()
> #10 0x0021000e in ?? ()
> #11 0x in ?? ()
>
>
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> 

Re: [OMPI devel] Patch proposed: opal_set_using_threads(true) in ompi/runtime/ompi_mpi_init.c is called to late

[Gilles Gouaillardet]

George,

please allow me to jump in with naive comments ...

currently (master) both openib and usnic btl invokes opal_using_threads
in component_init() :

btl_openib_component_init(int *num_btl_modules,
  bool enable_progress_threads,
  bool enable_mpi_threads)
{
[...]
/* Currently refuse to run if MPI_THREAD_MULTIPLE is enabled */
if (opal_using_threads() && !mca_btl_base_thread_multiple_override) {
opal_output_verbose(5, opal_btl_base_framework.framework_output,
"btl:openib: MPI_THREAD_MULTIPLE not
suppported; skipping this component");
goto no_btls;
}

> The overall design in OMPI was that no OMPI module should be allowed to 
> decide if threads are on

does "OMPI module" exclude OPAL and ORTE module ?
if yes, did the btl move from OMPI down to OPAL have any impact ?

if not, then could/should opal_using_threads() abort and/or display an
error message if it is called too early
(at least in debug builds) ?

Cheers,

Gilles

On 2014/12/12 10:30, Ralph Castain wrote:
> Just to help me understand: I don't think this change actually changed any 
> behavior. However, it certainly *allows* a different behavior. Isn't that 
> true?
>
> If so, I guess the real question is for Pascal at Bull: why do you feel this 
> earlier setting is required?
>
>
>> On Dec 11, 2014, at 4:21 PM, George Bosilca  wrote:
>>
>> The overall design in OMPI was that no OMPI module should be allowed to 
>> decide if threads are on (thus it should not rely on the value returned by 
>> opal_using_threads during it's initialization stage). Instead, they should 
>> respect the level of thread support requested as an argument during the 
>> initialization step.
>>
>> And this is true even for the BTLs. The PML component init function is 
>> propagating the  enable_progress_threads and enable_mpi_threads, down to the 
>> BML, and then to the BTL. This 2 variables, enable_progress_threads and 
>> enable_mpi_threads, are exactly what the ompi_mpi_init is using to compute 
>> the the value of the opal) using_thread (and that this patch moved).
>>
>> The setting of the opal_using_threads was delayed during the initialization 
>> to ensure that it's value was not used to select a specific thread-level in 
>> any module, a behavior that is allowed now with the new setting.
>>
>> A drastic change in behavior...
>>
>>   George.
>>
>>
>> On Tue, Dec 9, 2014 at 3:33 AM, Ralph Castain > > wrote:
>> Kewl - I'll fix. Thanks!
>>
>>> On Dec 9, 2014, at 12:32 AM, Pascal Deveze >> > wrote:
>>>
>>> Hi Ralph,
>>>  
>>> This in in the trunk.
>>>  
>>> De : devel [mailto:devel-boun...@open-mpi.org 
>>> ] De la part de Ralph Castain
>>> Envoyé : mardi 9 décembre 2014 09:32
>>> À : Open MPI Developers
>>> Objet : Re: [OMPI devel] Patch proposed: opal_set_using_threads(true) in 
>>> ompi/runtime/ompi_mpi_init.c is called to late
>>>  
>>> Hi Pascal
>>>  
>>> Is this in the trunk or in the 1.8 series (or both)?
>>>  
>>>  
>>> On Dec 9, 2014, at 12:28 AM, Pascal Deveze >> > wrote:
>>>  
>>>  
>>> In case where MPI is compiled with --enable-mpi-thread-multiple, a call to 
>>> opal_using_threads() always returns 0 in the routine 
>>> btl_xxx_component_init() of the BTLs, event if the application calls 
>>> MPI_Init_thread() with MPI_THREAD_MULTIPLE.
>>>  
>>> This is because opal_set_using_threads(true) in 
>>> ompi/runtime/ompi_mpi_init.c is called to late.
>>>  
>>> I propose the following patch that solves the problem for me:
>>>  
>>> diff --git a/ompi/runtime/ompi_mpi_init.c b/ompi/runtime/ompi_mpi_init.c
>>> index 35509cf..c2370fc 100644
>>> --- a/ompi/runtime/ompi_mpi_init.c
>>> +++ b/ompi/runtime/ompi_mpi_init.c
>>> @@ -512,6 +512,13 @@ int ompi_mpi_init(int argc, char **argv, int 
>>> requested, int *provided)
>>>  }
>>> #endif
>>>  
>>> +/* If thread support was enabled, then setup OPAL to allow for
>>> +   them. */
>>> +if ((OPAL_ENABLE_PROGRESS_THREADS == 1) ||
>>> +(*provided != MPI_THREAD_SINGLE)) {
>>> +opal_set_using_threads(true);
>>> +}
>>> +
>>>  /* initialize datatypes. This step should be done early as it will
>>>   * create the local convertor and local arch used in the proc
>>>   * init.
>>> @@ -724,13 +731,6 @@ int ompi_mpi_init(int argc, char **argv, int 
>>> requested, int *provided)
>>> goto error;
>>>  }
>>>  
>>> -/* If thread support was enabled, then setup OPAL to allow for
>>> -   them. */
>>> -if ((OPAL_ENABLE_PROGRESS_THREADS == 1) ||
>>> -(*provided != MPI_THREAD_SINGLE)) {
>>> -opal_set_using_threads(true);
>>> -}
>>> -
>>>  /* start PML/BTL's */
>>>  ret = MCA_PML_CALL(enable(true));
>>>  if( OMPI_SUCCESS != ret ) 

Re: [OMPI devel] hwloc out-of-order topology discovery with SLURM 14.11.0 and openmpi 1.6

[Gilles Gouaillardet]

Ralph,

You are right,
please disregard my previous post, it was irrelevant.

i just noticed that unlike ompi v1.8 (hwloc 1.7.2 based => no warning),
master has this warning (hwloc 1.9.1)

i will build slurm vs a recent hwloc and see what happens
(FWIW RHEL6 comes with hwloc 1.5, RHEL7 comes with hwloc 1.7 and both do
*not* have this warning)

Cheers,

Gilles

On 2014/12/11 12:02, Ralph Castain wrote:
> Per his prior notes, he is using mpirun to launch his jobs. Brice has 
> confirmed that OMPI doesn't have that hwloc warning in it. So either he has 
> inadvertently linked against the Ubuntu system version of hwloc, or the 
> message must be coming from Slurm.
>
>
>> On Dec 10, 2014, at 6:14 PM, Gilles Gouaillardet 
>> <gilles.gouaillar...@iferc.org> wrote:
>>
>> Pim,
>>
>> at this stage, all i can do is acknowledge your slurm is configured to use 
>> cgroups.
>>
>> and based on your previous comment (e.g. problem only occurs with several 
>> jobs on the same node)
>> that *could* be a bug in OpenMPI (or hwloc).
>>
>> by the way, how do you start your mpi application ?
>> - do you use mpirun ?
>> - do you use srun --resv-ports ?
>>
>> i'll try to reproduce this in my test environment.
>>
>> Cheers,
>>
>> Gilles
>>
>> On 2014/12/11 2:45, Pim Schellart wrote:
>>> Dear Gilles et al.,
>>>
>>> we tested with openmpi compiled from source (version 1.8.3) both with:
>>>
>>> ./configure --prefix=/usr/local/openmpi --disable-silent-rules 
>>> --with-libltdl=external --with-devel-headers --with-slurm 
>>> --enable-heterogeneous --disable-vt --sysconfdir=/etc/openmpi
>>>
>>> and
>>>
>>> ./configure --prefix=/usr/local/openmpi --with-hwloc=/usr 
>>> --disable-silent-rules --with-libltdl=external --with-devel-headers 
>>> --with-slurm --enable-heterogeneous --disable-vt --sysconfdir=/etc/openmpi
>>>
>>> (e.g. with embedded and external hwloc) and the issue remains the same. 
>>> Meanwhile we have found another interesting detail. A job is started 
>>> consisting of four tasks split over two nodes. If this is the only job 
>>> running on those nodes the out-of-order warnings do not appear. However, if 
>>> multiple jobs are running the warnings do appear but only for the jobs that 
>>> are started later. We suspect that this is because for the first started 
>>> job the CPU cores assigned are 0 and 1 whereas they are different for the 
>>> later started jobs. I attached the output (including lstopo ---of xml 
>>> output (called for each task)) for both the working and broken case again.
>>>
>>> Kind regards,
>>>
>>> Pim Schellart
>>>
>>>
>>>
>>>
>>>> On 09 Dec 2014, at 09:38, Gilles Gouaillardet 
>>>> <gilles.gouaillar...@iferc.org> <mailto:gilles.gouaillar...@iferc.org> 
>>>> wrote:
>>>>
>>>> Pim,
>>>>
>>>> if you configure OpenMPI with --with-hwloc=external (or something like
>>>> --with-hwloc=/usr) it is very likely
>>>> OpenMPI will use the same hwloc library (e.g. the "system" library) that
>>>> is used by SLURM
>>>>
>>>> /* i do not know how Ubuntu packages OpenMPI ... */
>>>>
>>>>
>>>> The default (e.g. no --with-hwloc parameter in the configure command
>>>> line) is to use the hwloc library that is embedded within OpenMPI
>>>>
>>>> Gilles
>>>>
>>>> On 2014/12/09 17:34, Pim Schellart wrote:
>>>>> Ah, ok so that was where the confusion came from, I did see hwloc in the 
>>>>> SLURM sources but couldn't immediately figure out where exactly it was 
>>>>> used. We will try compiling openmpi with the embedded hwloc. Any 
>>>>> particular flags I should set?
>>>>>
>>>>>> On 09 Dec 2014, at 09:30, Ralph Castain <r...@open-mpi.org> 
>>>>>> <mailto:r...@open-mpi.org> wrote:
>>>>>>
>>>>>> There is no linkage between slurm and ompi when it comes to hwloc. If 
>>>>>> you directly launch your app using srun, then slurm will use its version 
>>>>>> of hwloc to do the binding. If you use mpirun to launch the app, then 
>>>>>> we'll use our internal version to do it.
>>>>>>
>>>>>> The two are completely isolated from each other.
>>>>>>
>>>>>&g

Re: [OMPI devel] hwloc out-of-order topology discovery with SLURM 14.11.0 and openmpi 1.6

[Gilles Gouaillardet]

Pim,

at this stage, all i can do is acknowledge your slurm is configured to
use cgroups.

and based on your previous comment (e.g. problem only occurs with
several jobs on the same node)
that *could* be a bug in OpenMPI (or hwloc).

by the way, how do you start your mpi application ?
- do you use mpirun ?
- do you use srun --resv-ports ?

i'll try to reproduce this in my test environment.

Cheers,

Gilles

On 2014/12/11 2:45, Pim Schellart wrote:
> Dear Gilles et al.,
>
> we tested with openmpi compiled from source (version 1.8.3) both with:
>
> ./configure --prefix=/usr/local/openmpi --disable-silent-rules 
> --with-libltdl=external --with-devel-headers --with-slurm 
> --enable-heterogeneous --disable-vt --sysconfdir=/etc/openmpi
>
> and
>
> ./configure --prefix=/usr/local/openmpi --with-hwloc=/usr 
> --disable-silent-rules --with-libltdl=external --with-devel-headers 
> --with-slurm --enable-heterogeneous --disable-vt --sysconfdir=/etc/openmpi
>
> (e.g. with embedded and external hwloc) and the issue remains the same. 
> Meanwhile we have found another interesting detail. A job is started 
> consisting of four tasks split over two nodes. If this is the only job 
> running on those nodes the out-of-order warnings do not appear. However, if 
> multiple jobs are running the warnings do appear but only for the jobs that 
> are started later. We suspect that this is because for the first started job 
> the CPU cores assigned are 0 and 1 whereas they are different for the later 
> started jobs. I attached the output (including lstopo ---of xml output 
> (called for each task)) for both the working and broken case again.
>
> Kind regards,
>
> Pim Schellart
>
>
>
>
>> On 09 Dec 2014, at 09:38, Gilles Gouaillardet 
>> <gilles.gouaillar...@iferc.org> wrote:
>>
>> Pim,
>>
>> if you configure OpenMPI with --with-hwloc=external (or something like
>> --with-hwloc=/usr) it is very likely
>> OpenMPI will use the same hwloc library (e.g. the "system" library) that
>> is used by SLURM
>>
>> /* i do not know how Ubuntu packages OpenMPI ... */
>>
>>
>> The default (e.g. no --with-hwloc parameter in the configure command
>> line) is to use the hwloc library that is embedded within OpenMPI
>>
>> Gilles
>>
>> On 2014/12/09 17:34, Pim Schellart wrote:
>>> Ah, ok so that was where the confusion came from, I did see hwloc in the 
>>> SLURM sources but couldn't immediately figure out where exactly it was 
>>> used. We will try compiling openmpi with the embedded hwloc. Any particular 
>>> flags I should set?
>>>
>>>> On 09 Dec 2014, at 09:30, Ralph Castain <r...@open-mpi.org> wrote:
>>>>
>>>> There is no linkage between slurm and ompi when it comes to hwloc. If you 
>>>> directly launch your app using srun, then slurm will use its version of 
>>>> hwloc to do the binding. If you use mpirun to launch the app, then we'll 
>>>> use our internal version to do it.
>>>>
>>>> The two are completely isolated from each other.
>>>>
>>>>
>>>>> On Dec 9, 2014, at 12:25 AM, Pim Schellart <p.schell...@gmail.com> wrote:
>>>>>
>>>>> The version that "lstopo --version" reports is the same (1.8) on all 
>>>>> nodes, but we may indeed be hitting the second issue. We can try to 
>>>>> compile a new version of openmpi, but how do we ensure that the external 
>>>>> programs (e.g. SLURM) are using the same hwloc version as the one 
>>>>> embedded in openmpi? Is it enough to just compile hwloc 1.9 separately as 
>>>>> well and link against that? Also, if this is an issue, should we file a 
>>>>> bug against hwloc or openmpi on Ubuntu for mismatching versions?
>>>>>
>>>>>> On 09 Dec 2014, at 00:50, Ralph Castain <r...@open-mpi.org> wrote:
>>>>>>
>>>>>> Hmmm...they probably linked that to the external, system hwloc version, 
>>>>>> so it sounds like one or more of your nodes has a different hwloc rpm on 
>>>>>> it.
>>>>>>
>>>>>> I couldn't leaf thru your output well enough to see all the lstopo 
>>>>>> versions, but you might check to ensure they are the same.
>>>>>>
>>>>>> Looking at the code base, you may also hit a problem here. OMPI 1.6 
>>>>>> series was based on hwloc 1.3 - the output you sent indicated you have 
>>>>>> hwloc 1.8, which is quite a big change. OMP

Re: [OMPI devel] OMPI devel] OMPI devel] openmpi and XRC API from ofed-3.12

[Gilles Gouaillardet]

Hi,

I already figured this out and did the port :-)

Cheers,

Gilles

Piotr Lesnicki <piotr.lesni...@ext.bull.net> wrote:
>Hi,
>
>Gilles, the patch I sent is vs v1.6, so I prepare a patch vs master.
>
>The patch on v1.6 can not apply on master because of changes in the
>btl openib: connecting XRC queues has changed from XOOB to UDCM.
>
>Piotr
>
>
>De : devel [devel-boun...@open-mpi.org] de la part de Gilles Gouaillardet 
>[gilles.gouaillar...@iferc.org]
>Envoyé : mercredi 10 décembre 2014 09:20
>À : Open MPI Developers
>Objet : Re: [OMPI devel] OMPI devel] openmpi and XRC API from ofed-3.12
>
>Piotr and all,
>
>i issued PR #313 (vs master) based on your patch:
>https://github.com/open-mpi/ompi/pull/313
>
>could you please have a look at it ?
>
>Cheers,
>
>Gilles
>
>On 2014/12/09 22:07, Gilles Gouaillardet wrote:
>> Thanks Piotr,
>>
>> Based on the ompi community rules, a pr should be made vs the master, so 
>> code can be reviewed and shacked a bit.
>> I already prepared such a pr based on your patch and i will push it tomorrow.
>>
>> Then the changes will be backported to the v1.8 branch, assuming this is not 
>> considered as a new feature.
>>
>> Ralph, can you please comment on that ?
>>
>> Cheers,
>>
>> Gilles
>>
>>
>> Piotr Lesnicki <piotr.lesni...@ext.bull.net>さんのメール:
>>> Hi,
>>>
>>> We indeed have a fix for XRC support on our branch at Bull and sorry I
>>> neglected to contribute it, my bad…
>>>
>>> I join here the patch on top of current v1.6.6 (should I rather
>>> submit it as a pull request ?).
>>>
>>> For v1.8+, a merge of the v1.6 code is not enough as openib connect
>>> changed from xoob to udcm. I made a version on a pre-git state, so I
>>> will update it and make a pull request.
>>>
>>> Piotr
>>>
>>>
>>>
>>>
>>> 
>>> De : devel [devel-boun...@open-mpi.org] de la part de Gilles Gouaillardet 
>>> [gilles.gouaillar...@iferc.org]
>>> Envoyé : lundi 8 décembre 2014 03:27
>>> À : Open MPI Developers
>>> Objet : Re: [OMPI devel] openmpi and XRC API from ofed-3.12
>>>
>>> Hi Piotr,
>>>
>>> this  is quite an old thread now, but i did not see any support for XRC
>>> with ofed 3.12 yet
>>> (nor in trunk nor in v1.8)
>>>
>>> my understanding is that Bull already did something similar for the v1.6
>>> series,
>>> so let me put this the other way around :
>>>
>>> does Bull have any plan to contribute this work ?
>>> (for example, publish a patch for the v1.6 series, or submit pull
>>> request(s) for master and v1.8 branch)
>>>
>>> Cheers,
>>>
>>> Gilles
>>>
>>> On 2014/04/23 21:58, Piotr Lesnicki wrote:
>>>> Hi,
>>>>
>>>> In OFED-3.12 the API for XRC has changed. I did not find
>>>> corresponding changes in Open MPI: for example the function
>>>> 'ibv_create_xrc_rcv_qp()' queried in openmpi configure script no
>>>> longer exists in ofed-3.12-rc1.
>>>>
>>>> Are there any plans to support the new XRC API ?
>>>>
>>>>
>>>> --
>>>> Piotr
>>>> ___
>>>> devel mailing list
>>>> de...@open-mpi.org
>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>> Link to this post:
>>>> http://www.open-mpi.org/community/lists/devel/2014/04/14583.php
>>> ___
>>> devel mailing list
>>> de...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>> Link to this post: 
>>> http://www.open-mpi.org/community/lists/devel/2014/12/16445.php
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/devel/2014/12/16467.php
>
>___
>devel mailing list
>de...@open-mpi.org
>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>Link to this post: 
>http://www.open-mpi.org/community/lists/devel/2014/12/16488.php
>___
>devel mailing list
>de...@open-mpi.org
>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>Link to this post: 
>http://www.open-mpi.org/community/lists/devel/2014/12/16489.php

Re: [OMPI devel] OMPI devel] openmpi and XRC API from ofed-3.12

[Gilles Gouaillardet]

Piotr and all,

i issued PR #313 (vs master) based on your patch:
https://github.com/open-mpi/ompi/pull/313

could you please have a look at it ?

Cheers,

Gilles

On 2014/12/09 22:07, Gilles Gouaillardet wrote:
> Thanks Piotr,
>
> Based on the ompi community rules, a pr should be made vs the master, so code 
> can be reviewed and shacked a bit.
> I already prepared such a pr based on your patch and i will push it tomorrow.
>
> Then the changes will be backported to the v1.8 branch, assuming this is not 
> considered as a new feature.
>
> Ralph, can you please comment on that ?
>
> Cheers,
>
> Gilles
>
>
> Piotr Lesnicki <piotr.lesni...@ext.bull.net>さんのメール:
>> Hi,
>>
>> We indeed have a fix for XRC support on our branch at Bull and sorry I
>> neglected to contribute it, my bad…
>>
>> I join here the patch on top of current v1.6.6 (should I rather
>> submit it as a pull request ?).
>>
>> For v1.8+, a merge of the v1.6 code is not enough as openib connect
>> changed from xoob to udcm. I made a version on a pre-git state, so I
>> will update it and make a pull request.
>>
>> Piotr
>>
>>
>>
>>
>> 
>> De : devel [devel-boun...@open-mpi.org] de la part de Gilles Gouaillardet 
>> [gilles.gouaillar...@iferc.org]
>> Envoyé : lundi 8 décembre 2014 03:27
>> À : Open MPI Developers
>> Objet : Re: [OMPI devel] openmpi and XRC API from ofed-3.12
>>
>> Hi Piotr,
>>
>> this  is quite an old thread now, but i did not see any support for XRC
>> with ofed 3.12 yet
>> (nor in trunk nor in v1.8)
>>
>> my understanding is that Bull already did something similar for the v1.6
>> series,
>> so let me put this the other way around :
>>
>> does Bull have any plan to contribute this work ?
>> (for example, publish a patch for the v1.6 series, or submit pull
>> request(s) for master and v1.8 branch)
>>
>> Cheers,
>>
>> Gilles
>>
>> On 2014/04/23 21:58, Piotr Lesnicki wrote:
>>> Hi,
>>>
>>> In OFED-3.12 the API for XRC has changed. I did not find
>>> corresponding changes in Open MPI: for example the function
>>> 'ibv_create_xrc_rcv_qp()' queried in openmpi configure script no
>>> longer exists in ofed-3.12-rc1.
>>>
>>> Are there any plans to support the new XRC API ?
>>>
>>>
>>> --
>>> Piotr
>>> ___
>>> devel mailing list
>>> de...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>> Link to this post:
>>> http://www.open-mpi.org/community/lists/devel/2014/04/14583.php
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/devel/2014/12/16445.php
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2014/12/16467.php

Re: [OMPI devel] hwloc out-of-order topology discovery with SLURM 14.11.0 and openmpi 1.6

[Gilles Gouaillardet]

Pim,

if you configure OpenMPI with --with-hwloc=external (or something like
--with-hwloc=/usr) it is very likely
OpenMPI will use the same hwloc library (e.g. the "system" library) that
is used by SLURM

/* i do not know how Ubuntu packages OpenMPI ... */


The default (e.g. no --with-hwloc parameter in the configure command
line) is to use the hwloc library that is embedded within OpenMPI

Gilles

On 2014/12/09 17:34, Pim Schellart wrote:
> Ah, ok so that was where the confusion came from, I did see hwloc in the 
> SLURM sources but couldn’t immediately figure out where exactly it was used. 
> We will try compiling openmpi with the embedded hwloc. Any particular flags I 
> should set?
>
>> On 09 Dec 2014, at 09:30, Ralph Castain  wrote:
>>
>> There is no linkage between slurm and ompi when it comes to hwloc. If you 
>> directly launch your app using srun, then slurm will use its version of 
>> hwloc to do the binding. If you use mpirun to launch the app, then we’ll use 
>> our internal version to do it.
>>
>> The two are completely isolated from each other.
>>
>>
>>> On Dec 9, 2014, at 12:25 AM, Pim Schellart  wrote:
>>>
>>> The version that “lstopo --version” reports is the same (1.8) on all nodes, 
>>> but we may indeed be hitting the second issue. We can try to compile a new 
>>> version of openmpi, but how do we ensure that the external programs (e.g. 
>>> SLURM) are using the same hwloc version as the one embedded in openmpi? Is 
>>> it enough to just compile hwloc 1.9 separately as well and link against 
>>> that? Also, if this is an issue, should we file a bug against hwloc or 
>>> openmpi on Ubuntu for mismatching versions?
>>>
 On 09 Dec 2014, at 00:50, Ralph Castain  wrote:

 Hmmm…they probably linked that to the external, system hwloc version, so 
 it sounds like one or more of your nodes has a different hwloc rpm on it.

 I couldn’t leaf thru your output well enough to see all the lstopo 
 versions, but you might check to ensure they are the same.

 Looking at the code base, you may also hit a problem here. OMPI 1.6 series 
 was based on hwloc 1.3 - the output you sent indicated you have hwloc 1.8, 
 which is quite a big change. OMPI 1.8 series is based on hwloc 1.9, so at 
 least that is closer (though probably still a mismatch).

 Frankly, I’d just download and install an OMPI tarball myself and avoid 
 these headaches. This mismatch in required versions is why we embed hwloc 
 as it is a critical library for OMPI, and we had to ensure that the 
 version matched our internal requirements.


> On Dec 8, 2014, at 8:50 AM, Pim Schellart  wrote:
>
> It is the default openmpi that comes with Ubuntu 14.04.
>
>> On 08 Dec 2014, at 17:17, Ralph Castain  wrote:
>>
>> Pim: is this an OMPI you built, or one you were given somehow? If you 
>> built it, how did you configure it?
>>
>>> On Dec 8, 2014, at 8:12 AM, Brice Goglin  wrote:
>>>
>>> It likely depends on how SLURM allocates the cpuset/cgroup inside the
>>> nodes. The XML warning is related to these restrictions inside the node.
>>> Anyway, my feeling is that there's a old OMPI or a old hwloc somewhere.
>>>
>>> How do we check after install whether OMPI uses the embedded or the
>>> system-wide hwloc?
>>>
>>> Brice
>>>
>>>
>>>
>>>
>>> Le 08/12/2014 17:07, Pim Schellart a écrit :
 Dear Ralph,

 the nodes are called coma## and as you can see in the logs the nodes 
 of the broken example are the same as the nodes of the working one, so 
 that doesn’t seem to be the cause. Unless (very likely) I’m missing 
 something. Anything else I can check?

 Regards,

 Pim

> On 08 Dec 2014, at 17:03, Ralph Castain  wrote:
>
> As Brice said, OMPI has its own embedded version of hwloc that we 
> use, so there is no Slurm interaction to be considered. The most 
> likely cause is that one or more of your nodes is picking up a 
> different version of OMPI. So things “work” if you happen to get 
> nodes where all the versions match, and “fail” when you get a 
> combination that includes a different version.
>
> Is there some way you can narrow down your search to find the node(s) 
> that are picking up the different version?
>
>
>> On Dec 8, 2014, at 7:48 AM, Pim Schellart  
>> wrote:
>>
>> Dear Brice,
>>
>> I am not sure why this is happening since all code seems to be using 
>> the same hwloc library version (1.8) but it does :) An MPI program 
>> is started through SLURM on two 

Re: [OMPI devel] [OMPI users] Warning about not enough registerable memory on SL6.6

[Gilles Gouaillardet]

Folks,

FWIW, i observe a similar behaviour on my system.

imho, the root cause is OFED has been upgraded from a (quite) older
version to latest 3.12 version

here is the relevant part of code (btl_openib.c from the master) :


static uint64_t calculate_max_reg (void)
{
if (0 == stat("/sys/module/mlx4_core/parameters/log_num_mtt",
)) {
} else if (0 == stat("/sys/module/ib_mthca/parameters/num_mtt",
)) {
mtts_per_seg = 1 <<
read_module_param("/sys/module/ib_mthca/parameters/log_mtts_per_seg", 1);
num_mtt =
read_module_param("/sys/module/ib_mthca/parameters/num_mtt", 1 << 20);
reserved_mtt =
read_module_param("/sys/module/ib_mthca/parameters/fmr_reserved_mtts", 0);

max_reg = (num_mtt - reserved_mtt) * opal_getpagesize () *
mtts_per_seg;
} else if (
(0 == stat("/sys/module/mlx5_core", )) ||
(0 == stat("/sys/module/mlx4_core/parameters", )) ||
(0 == stat("/sys/module/ib_mthca/parameters", ))
) {
/* mlx5 means that we have ofed 2.0 and it can always register
2xmem_total for any mlx hca */
max_reg = 2 * mem_total;
} else {
}

/* Print a warning if we can't register more than 75% of physical
   memory.  Abort if the abort_not_enough_reg_mem MCA param was
   set. */
if (max_reg < mem_total * 3 / 4) {
}
return (max_reg * 7) >> 3;
}

with OFED 3.12, the /sys/module/mlx4_core/parameters/log_num_mtt pseudo
file does *not* exist any more
/sys/module/ib_mthca/parameters/num_mtt exists so the second path is taken
and mtts_per_seg is read from
/sys/module/ib_mthca/parameters/log_mtts_per_seg

i noted that log_mtts_per_seg is also a parameter of mlx4_core :
/sys/module/mlx4_core/parameters/log_mtts_per_seg

the value is 3 in ib_mthca (and leads to a warning) but 5 in mlx4_core
(big enough, and does not lead to a warning if this value is read)


i had no time to read the latest ofed doc, so i cannot answer :
- should log_mtts_per_seg be read from mlx4_core instead ?
- is the warning a false positive ?


my only point is this warning *might* be a false positive and the root
cause *might* be calculate_max_reg logic
*could* be wrong with the latest OFED stack.

Could the Mellanox folks comment on this ?

Cheers,

Gilles




On 2014/12/09 3:18, Götz Waschk wrote:
> Hi,
>
> here's another test with openmpi 1.8.3. With 1.8.1, 32GB was detected, now
> it is just 16:
> % mpirun -np 2 /usr/lib64/openmpi-intel/bin/mpitests-osu_get_bw
> --
> WARNING: It appears that your OpenFabrics subsystem is configured to only
> allow registering part of your physical memory.  This can cause MPI jobs to
> run with erratic performance, hang, and/or crash.
>
> This may be caused by your OpenFabrics vendor limiting the amount of
> physical memory that can be registered.  You should investigate the
> relevant Linux kernel module parameters that control how much physical
> memory can be registered, and increase them to allow registering all
> physical memory on your machine.
>
> See this Open MPI FAQ item for more information on these Linux kernel module
> parameters:
>
> http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages
>
>   Local host:  pax95
>   Registerable memory: 16384 MiB
>   Total memory:49106 MiB
>
> Your MPI job will continue, but may be behave poorly and/or hang.
> --
> # OSU MPI_Get Bandwidth Test v4.3
> # Window creation: MPI_Win_allocate
> # Synchronization: MPI_Win_flush
> # Size  Bandwidth (MB/s)
> 1  28.56
> 2  58.74
>
>
> So it wasn't fixed for RHEL 6.6.
>
> Regards, Götz
>
> On Mon, Dec 8, 2014 at 4:00 PM, Götz Waschk  wrote:
>
>> Hi,
>>
>> I had tested 1.8.4rc1 and it wasn't fixed. I can try again though,
>> maybe I had made an error.
>>
>> Regards, Götz Waschk
>>
>> On Mon, Dec 8, 2014 at 3:17 PM, Joshua Ladd  wrote:
>>> Hi,
>>>
>>> This should be fixed in OMPI 1.8.3. Is it possible for you to give 1.8.3
>> a
>>> shot?
>>>
>>> Best,
>>>
>>> Josh
>>>
>>> On Mon, Dec 8, 2014 at 8:43 AM, Götz Waschk 
>> wrote:
 Dear Open-MPI experts,

 I have updated my little cluster from Scientific Linux 6.5 to 6.6,
 this included extensive changes in the Infiniband drivers and a newer
 openmpi version (1.8.1). Now I'm getting this message on all nodes
 with more than 32 GB of RAM:


 WARNING: It appears that your OpenFabrics subsystem is configured to
>> only
 allow registering part of your physical memory.  This can cause MPI jobs
 to
 run with erratic performance, hang, and/or crash.

 This may be caused by your OpenFabrics vendor limiting the amount of
 physical memory that can be registered.  You should investigate the
 relevant Linux kernel 

Re: [OMPI devel] openmpi and XRC API from ofed-3.12

[Gilles Gouaillardet]

Hi Piotr,

this  is quite an old thread now, but i did not see any support for XRC
with ofed 3.12 yet
(nor in trunk nor in v1.8)

my understanding is that Bull already did something similar for the v1.6
series,
so let me put this the other way around :

does Bull have any plan to contribute this work ?
(for example, publish a patch for the v1.6 series, or submit pull
request(s) for master and v1.8 branch)

Cheers,

Gilles

On 2014/04/23 21:58, Piotr Lesnicki wrote:
> Hi,
>
> In OFED-3.12 the API for XRC has changed. I did not find
> corresponding changes in Open MPI: for example the function
> 'ibv_create_xrc_rcv_qp()' queried in openmpi configure script no
> longer exists in ofed-3.12-rc1.
>
> Are there any plans to support the new XRC API ?
>
>
> -- 
> Piotr
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/04/14583.php

Re: [OMPI devel] OpenMPI v1.8 and v1.8.3 mpirun hangs at execution on an embedded ARM Linux kernel version 3.15.0

[Gilles Gouaillardet]

+1

if i remember correctly, all the interfaces are scanned, so there should
be some room to display
a user-friendly message (on Linux and impacted architectures) such as
"there is no loopback interface, you will likely run into some trouble"

Gilles

On 2014/12/03 13:50, Paul Hargrove wrote:
> IMHO the lack of a loopback interface should be a very uncommon occurrence.
> So, I believe that improving the error message to mention that possibility
> would help a great deal.
>
> -Paul
>
>
> On Tue, Dec 2, 2014 at 8:28 PM, Ralph Castain <r...@open-mpi.org> wrote:
>
>> We talked about this on the weekly conference call, and adding the usock
>> component to 1.8 is just not within our procedures. It would involve
>> bringing over much more of the OOB revisions (we'd have to handle the
>> transfer of messages between components, if nothing else), and that
>> involves a lot of change.
>>
>> I'll instead try to provide a faster error response so it is clearer what
>> is happening, hopefully letting the user fix the problem by turning on the
>> loopback interface.
>>
>>
>> On Nov 25, 2014, at 7:05 PM, Ralph Castain <r...@open-mpi.org> wrote:
>>
>>
>> On Nov 25, 2014, at 6:15 PM, Gilles Gouaillardet <
>> gilles.gouaillar...@iferc.org> wrote:
>>
>>  Ralph and Paul,
>>
>> On 2014/11/26 10:37, Ralph Castain wrote:
>>
>> So it looks like the issue isn't so much with our code as it is with the OS 
>> stack, yes? We aren't requiring that the loopback be "up", but the stack is 
>> in order to establish the connection, even when we are trying a non-lo 
>> interface.
>>
>>  this is correct (imho)
>>
>> I can look into generating a faster timeout on the socket creation. In the 
>> trunk, we now use unix domain sockets instead of TCP to avoid such issues, 
>> but that won't help with the 1.8 series.
>>
>>  i was about to suggest this situation could have been avoided in the
>> first place by using unix domain sockets instead of TCP sockets :-)
>>
>>
>> There were some historical reasons for not doing so - mostly because it
>> generally isn't necessary on a cluster.
>>
>>
>> is a backport (since this is already available in the trunk/master) simply
>> out of the question ?
>>
>>
>> It would be against our normal procedures, but I can raise it at next
>> week's meeting.
>>
>>
>> Cheers,
>>
>> Gilles
>>
>>   On Nov 25, 2014, at 4:50 PM, Paul Hargrove <phhargr...@lbl.gov> 
>> <phhargr...@lbl.gov> wrote:
>>
>> Ralph,
>>
>> I had a look at the problem via "mpirun -np 1 strace -o trace -ff ./hello"
>> I find that there is an attempt (by a secondary thread) to establish a TCP 
>> socket from the rank process to the eth0 address of localhost (I am guessing 
>> to reach the orted/mpirun).
>> However, when the "lo" interface is down, the Linux kernel apparently cannot 
>> establish that socket.
>>
>> In fact, if I am sufficiently patient, it turns out the "hang" is bounded, 
>> and eventually one sees:
>>
>> phargrov@blcr-armv7:~$ time mpirun -np 1 ./a.out
>> 
>> A process or daemon was unable to complete a TCP connection
>> to another process:
>>   Local host:blcr-armv7
>>   Remote host:   10.0.2.15
>> This is usually caused by a firewall on the remote host. Please
>> check that any firewall (e.g., iptables) has been disabled and
>> try again.
>> 
>>
>> real2m8.151s
>> user0m5.360s
>> sys 0m57.430s
>>
>>
>> Where blcr-armv7 and 10.0.2.15 are *both* the local (only) host.
>>
>> There is no firewall, but in case you doubt me on that, here is a 
>> demonstration using ping to show that 10.0.2.15 is only reachable when the 
>> loopback interface is enabled:
>>
>> phargrov@blcr-armv7:~$ sudo ifconfig lo up
>> phargrov@blcr-armv7:~$ ping -q -c2 10.0.2.15
>> PING 10.0.2.15 (10.0.2.15) 56(84) bytes of data.
>>
>> --- 10.0.2.15 ping statistics ---
>> 2 packets transmitted, 2 received, 0% packet loss, time 1002ms
>> rtt min/avg/max/mdev = 0.527/0.534/0.542/0.024 ms
>>
>>
>> phargrov@blcr-armv7:~$ sudo ifconfig lo down
>> phargrov@blcr-armv7:~$ ping -q -c2 10.0.2.15
>> PING 10.0.2.15 (10.0.2.15) 56(84) bytes of data.
>>
>> --- 10.0.2.15 ping statistics ---
>> 2 packets transmitted, 0 received, 1

Re: [OMPI devel] RTLD_GLOBAL question

[Gilles Gouaillardet]

$ srun --version
slurm 2.6.6-VENDOR_PROVIDED

$ srun --mpi=pmi2 -n 1 ~/hw
I am 0 / 1

$ srun -n 1 ~/hw
/csc/home1/gouaillardet/hw: symbol lookup error:
/usr/lib64/slurm/auth_munge.so: undefined symbol: slurm_verbose
srun: error: slurm_receive_msg: Zero Bytes were transmitted or received
srun: error: slurm_receive_msg[10.0.3.15]: Zero Bytes were transmitted
or received
srun: error: soleil: task 0: Exited with exit code 127

$ ldd /usr/lib64/slurm/auth_munge.so
linux-vdso.so.1 =>  (0x7fff54478000)
libmunge.so.2 => /usr/lib64/libmunge.so.2 (0x7f744760f000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x7f74473f1000)
libc.so.6 => /lib64/libc.so.6 (0x7f744705d000)
/lib64/ld-linux-x86-64.so.2 (0x003bf540)


now, if i reling auth_munge.so so it depends on libslurm :

$ srun -n 1 ~/hw
srun: symbol lookup error: /usr/lib64/slurm/auth_munge.so: undefined
symbol: slurm_auth_get_arg_desc


i can give a try to the latest slurm if needed

Cheers,

Gilles


On 2014/12/02 12:56, Ralph Castain wrote:
> Out of curiosity - how are you testing these? I have more current versions of 
> Slurm and would like to test the observations there.
>
>> On Dec 1, 2014, at 7:49 PM, Gilles Gouaillardet 
>> <gilles.gouaillar...@iferc.org> wrote:
>>
>> I d like to make a step back ...
>>
>> i previously tested with slurm 2.6.0, and it complained about the 
>> slurm_verbose symbol that is defined in libslurm.so
>> so with slurm 2.6.0, RTLD_GLOBAL or relinking is ok
>>
>> now i tested with slurm 2.6.6 and it complains about the 
>> slurm_auth_get_arg_desc symbol, and this symbol is not
>> defined in any dynamic library. it is internally defined in the static 
>> libcommon.a library, which is used to build the slurm binaries.
>>
>> as far as i understand, auth_munge.so can only be invoked from a slurm 
>> binary, which means it cannot be invoked from an mpi application
>> even if it is linked with libslurm, libpmi, ...
>>
>> that looks like a slurm design issue that the slurm folks will take care of.
>>
>> Cheers,
>>
>> Gilles
>>
>> On 2014/12/02 12:33, Ralph Castain wrote:
>>> Another option is to simply add the -lslurm -lauth flags to the pmix/s1 
>>> component as this is the only place that requires it, and it won't hurt 
>>> anything to do so.
>>>
>>>
>>>> On Dec 1, 2014, at 6:03 PM, Gilles Gouaillardet 
>>>> <gilles.gouaillar...@iferc.org> <mailto:gilles.gouaillar...@iferc.org> 
>>>> wrote:
>>>>
>>>> Jeff,
>>>>
>>>> FWIW, you can read my analysis of what is going wrong at
>>>> http://www.open-mpi.org/community/lists/pmix-devel/2014/11/0293.php 
>>>> <http://www.open-mpi.org/community/lists/pmix-devel/2014/11/0293.php> 
>>>> <http://www.open-mpi.org/community/lists/pmix-devel/2014/11/0293.php> 
>>>> <http://www.open-mpi.org/community/lists/pmix-devel/2014/11/0293.php>
>>>>
>>>> bottom line, i agree this is a slurm issue (slurm plugin should depend
>>>> on libslurm, but they do not, yet)
>>>>
>>>> a possible workaround would be to make the pmi component a "proxy" that
>>>> dlopen with RTLD_GLOBAL the "real" component in which the job is done.
>>>> that being said, the impact is quite limited (no direct launch in slurm
>>>> with pmi1, but pmi2 works fine) so it makes sense not to work around
>>>> someone else problem.
>>>> and that being said, configure could detect this broken pmi1 and not
>>>> build pmi1 support or print a user friendly error message if pmi1 is used.
>>>>
>>>> any thoughts ?
>>>>
>>>> Cheers,
>>>>
>>>> Gilles
>>>>
>>>> On 2014/12/02 7:47, Jeff Squyres (jsquyres) wrote:
>>>>> Ok, if the problem is moot, great.
>>>>>
>>>>> (sidenote: this is moot, so ignore this if you want: with this 
>>>>> explanation, I'm still not sure how RTLD_GLOBAL fixes the issue)
>>>>>
>>>>>
>>>>> On Dec 1, 2014, at 5:15 PM, Ralph Castain <r...@open-mpi.org> 
>>>>> <mailto:r...@open-mpi.org> wrote:
>>>>>
>>>>>> Easy enough to explain. We link libpmi into the pmix/s1 component. This 
>>>>>> library is missing the linkage to libslurm that contains the linkage to 
>>>>>> libauth where munge resides. So when we call a PMI function, libpmi 
>>>>>> references a call

Re: [OMPI devel] RTLD_GLOBAL question

[Gilles Gouaillardet]

I d like to make a step back ...

i previously tested with slurm 2.6.0, and it complained about the
slurm_verbose symbol that is defined in libslurm.so
so with slurm 2.6.0, RTLD_GLOBAL or relinking is ok

now i tested with slurm 2.6.6 and it complains about the
slurm_auth_get_arg_desc symbol, and this symbol is not
defined in any dynamic library. it is internally defined in the static
libcommon.a library, which is used to build the slurm binaries.

as far as i understand, auth_munge.so can only be invoked from a slurm
binary, which means it cannot be invoked from an mpi application
even if it is linked with libslurm, libpmi, ...

that looks like a slurm design issue that the slurm folks will take care of.

Cheers,

Gilles

On 2014/12/02 12:33, Ralph Castain wrote:
> Another option is to simply add the -lslurm -lauth flags to the pmix/s1 
> component as this is the only place that requires it, and it won't hurt 
> anything to do so.
>
>
>> On Dec 1, 2014, at 6:03 PM, Gilles Gouaillardet 
>> <gilles.gouaillar...@iferc.org> wrote:
>>
>> Jeff,
>>
>> FWIW, you can read my analysis of what is going wrong at
>> http://www.open-mpi.org/community/lists/pmix-devel/2014/11/0293.php 
>> <http://www.open-mpi.org/community/lists/pmix-devel/2014/11/0293.php>
>>
>> bottom line, i agree this is a slurm issue (slurm plugin should depend
>> on libslurm, but they do not, yet)
>>
>> a possible workaround would be to make the pmi component a "proxy" that
>> dlopen with RTLD_GLOBAL the "real" component in which the job is done.
>> that being said, the impact is quite limited (no direct launch in slurm
>> with pmi1, but pmi2 works fine) so it makes sense not to work around
>> someone else problem.
>> and that being said, configure could detect this broken pmi1 and not
>> build pmi1 support or print a user friendly error message if pmi1 is used.
>>
>> any thoughts ?
>>
>> Cheers,
>>
>> Gilles
>>
>> On 2014/12/02 7:47, Jeff Squyres (jsquyres) wrote:
>>> Ok, if the problem is moot, great.
>>>
>>> (sidenote: this is moot, so ignore this if you want: with this explanation, 
>>> I'm still not sure how RTLD_GLOBAL fixes the issue)
>>>
>>>
>>> On Dec 1, 2014, at 5:15 PM, Ralph Castain <r...@open-mpi.org> wrote:
>>>
>>>> Easy enough to explain. We link libpmi into the pmix/s1 component. This 
>>>> library is missing the linkage to libslurm that contains the linkage to 
>>>> libauth where munge resides. So when we call a PMI function, libpmi 
>>>> references a call to munge for authentication and hits an "unresolved 
>>>> symbol" error.
>>>>
>>>> Moe acknowledges the error is in Slurm and is fixing the linkages so this 
>>>> problem goes away
>>>>
>>>>
>>>>> On Dec 1, 2014, at 2:13 PM, Jeff Squyres (jsquyres) <jsquy...@cisco.com> 
>>>>> wrote:
>>>>>
>>>>> On Dec 1, 2014, at 5:07 PM, Ralph Castain <r...@open-mpi.org> wrote:
>>>>>
>>>>>> FWIW: It's Slurm's pmi-1 library that isn't linked correctly against its 
>>>>>> dependencies (the pmi-2 one is correct).  Moe is aware of the problem 
>>>>>> and fixing it on their side. This won't help existing installations 
>>>>>> until they upgrade, but I tend to agree with Jeff about not fixing other 
>>>>>> people's problems.
>>>>> Can you explain what is happening?
>>>>>
>>>>> I ask because I'm not sure I understand the problem such that using 
>>>>> RTLD_GLOBAL would fix it.  I.e., even if libpmi1.so isn't linked against 
>>>>> its dependencies properly, that shouldn't cause a problem if OMPI 
>>>>> components A and B are both linked against libpmi1.so, and then A is 
>>>>> loaded, and then B is loaded.
>>>>>
>>>>> ...or perhaps we can just discuss this on the call tomorrow?
>>>>>
>>>>> -- 
>>>>> Jeff Squyres
>>>>> jsquy...@cisco.com
>>>>> For corporate legal information go to: 
>>>>> http://www.cisco.com/web/about/doing_business/legal/cri/
>>>>>
>>>>> ___
>>>>> devel mailing list
>>>>> de...@open-mpi.org
>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>> Link to this post: 
>>>>> http://www.open-mpi.org/community/lists

Re: [OMPI devel] RTLD_GLOBAL question

[Gilles Gouaillardet]

Jeff,

FWIW, you can read my analysis of what is going wrong at
http://www.open-mpi.org/community/lists/pmix-devel/2014/11/0293.php

bottom line, i agree this is a slurm issue (slurm plugin should depend
on libslurm, but they do not, yet)

a possible workaround would be to make the pmi component a "proxy" that
dlopen with RTLD_GLOBAL the "real" component in which the job is done.
that being said, the impact is quite limited (no direct launch in slurm
with pmi1, but pmi2 works fine) so it makes sense not to work around
someone else problem.
and that being said, configure could detect this broken pmi1 and not
build pmi1 support or print a user friendly error message if pmi1 is used.

any thoughts ?

Cheers,

Gilles

On 2014/12/02 7:47, Jeff Squyres (jsquyres) wrote:
> Ok, if the problem is moot, great.
>
> (sidenote: this is moot, so ignore this if you want: with this explanation, 
> I'm still not sure how RTLD_GLOBAL fixes the issue)
>
>
> On Dec 1, 2014, at 5:15 PM, Ralph Castain  wrote:
>
>> Easy enough to explain. We link libpmi into the pmix/s1 component. This 
>> library is missing the linkage to libslurm that contains the linkage to 
>> libauth where munge resides. So when we call a PMI function, libpmi 
>> references a call to munge for authentication and hits an “unresolved 
>> symbol” error.
>>
>> Moe acknowledges the error is in Slurm and is fixing the linkages so this 
>> problem goes away
>>
>>
>>> On Dec 1, 2014, at 2:13 PM, Jeff Squyres (jsquyres)  
>>> wrote:
>>>
>>> On Dec 1, 2014, at 5:07 PM, Ralph Castain  wrote:
>>>
 FWIW: It’s Slurm’s pmi-1 library that isn’t linked correctly against its 
 dependencies (the pmi-2 one is correct).  Moe is aware of the problem and 
 fixing it on their side. This won’t help existing installations until they 
 upgrade, but I tend to agree with Jeff about not fixing other people’s 
 problems.
>>> Can you explain what is happening?
>>>
>>> I ask because I'm not sure I understand the problem such that using 
>>> RTLD_GLOBAL would fix it.  I.e., even if libpmi1.so isn't linked against 
>>> its dependencies properly, that shouldn't cause a problem if OMPI 
>>> components A and B are both linked against libpmi1.so, and then A is 
>>> loaded, and then B is loaded.
>>>
>>> ...or perhaps we can just discuss this on the call tomorrow?
>>>
>>> -- 
>>> Jeff Squyres
>>> jsquy...@cisco.com
>>> For corporate legal information go to: 
>>> http://www.cisco.com/web/about/doing_business/legal/cri/
>>>
>>> ___
>>> devel mailing list
>>> de...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>> Link to this post: 
>>> http://www.open-mpi.org/community/lists/devel/2014/12/16383.php
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/devel/2014/12/16384.php
>

Re: [OMPI devel] Question about tight integration with not-yet-supported queuing systems

[Gilles Gouaillardet]

Marc,

i am not aware of any mpi implementation in which mpirun does the job
allocation.

instead, mpirun gets job info from the batch manager (e.g. number of nodes)
so the job can be launched seamlessly and be properly killed in case of
a job abort
(bkill or equivalent)

Cheers,

Gilles

On 2014/12/01 17:47, marc.hoepp...@bils.se wrote:
>  
>
> HI,
>
>  sorry for the late reply - I've been traveling with limited email
> access. I think you can leave this issue be. I think I was hoping for a
> way to just launch mpirun and have it create the allocation by itself.
> It's not super important right now, more something I was wondering
> about. 
>
>  Thanks again for looking into this!
>
>  /Marc
>
> On 28.11.2014 17:10, Ralph Castain wrote: 
>
>> Hey Marc - just wanted to check to see if you felt this would indeed solve 
>> the problem for you. I'd rather not invest the time if this isn't going to 
>> meet the need, and I honestly don't know of a better solution. 
>>
>> On Nov 20, 2014, at 2:13 PM, Ralph Castain <r...@open-mpi.org> wrote: 
>>
>> Here's what I can provide: 
>>
>> * lsrun -n N bash This causes openlava to create an allocation and start you 
>> off in a bash shell (or pick your shell) 
>>
>> * mpirun . Will read the allocation and use openlava to start the 
>> daemons, and then the application, on the allocated nodes 
>>
>> You can execute as many mpirun's as you like, then release the allocation (I 
>> believe by simply exiting the shell) when done. 
>>
>> Does that match your expectations? 
>> Ralph 
>>
>> On Nov 20, 2014, at 2:03 PM, Marc Höppner <marc.hoepp...@bils.se> wrote: 
>>
>> Hi,
>>
>> yes, lsrun exists under openlava. 
>>
>> Using mpirun is fine, but openlava currently requires that to be launched 
>> through a bash script (openmpi-mpirun). Would be neater if one could do away 
>> with that. 
>>
>> Agan, thanks for looking into this!
>>
>> /Marc
>>
>> Hold on - was discussing this with a (possibly former) OpenLava developer 
>> who made some suggestions that would make this work. It all hinges on one 
>> thing. 
>>
>> Can you please check and see if you have "lsrun" on your system? If you do, 
>> then I can offer a tight integration in that we would use OpenLava to 
>> actually launch the OMPI daemons. Still not sure I could support you 
>> directly launching MPI apps without using mpirun, if that's what you are 
>> after. 
>>
>> On Nov 18, 2014, at 7:58 AM, Marc Höppner <marc.hoepp...@bils.se> wrote: 
>>
>> Hi Ralph,
>>
>> I really appreciate you guys looking into this! At least now I know that 
>> there isn't a better way to run mpi jobs. Probably worth looking into LSF 
>> again..
>>
>> Cheers,
>>
>> Marc 
>> I took a brief gander at the OpenLava source code, and a couple of things 
>> jump out. First, OpenLava is a batch scheduler and only supports batch 
>> execution - there is no interactive command for "run this job". So you would 
>> have to "bsub" mpirun regardless. 
>>
>> Once you submit the job, mpirun can certainly read the local allocation via 
>> the environment. However, we cannot use the OpenLava internal functions to 
>> launch the daemons or processes as the code is GPL2, and thus has a viral 
>> incompatible license. Ordinarily, we get around that by just executing the 
>> interactive job execution command, but OpenLava doesn't have one. 
>>
>> So we'd have no other choice but to use ssh to launch the daemons on the 
>> remote nodes. This is exactly what the provided openmpi wrapper script that 
>> comes with OpenLava already does. 
>>
>> Bottom line: I don't see a way to do any deeper integration minus the 
>> interactive execution command. If OpenLava had a way of getting an 
>> allocation and then interactively running jobs, we could support what you 
>> requested. This doesn't seem to be what they are intending, unless I'm 
>> missing something (the documentation is rather incomplete). 
>>
>> Ralph 
>>
>> On Tue, Nov 18, 2014 at 6:20 AM, Marc Höppner <marc.hoepp...@bils.se> wrote:
>>
>> Hi, 
>>
>> sure, no problem. And about the C Api, I really don't know more than what I 
>> was told in the google group post I referred to (i.e. the API is essentially 
>> identical to LSF 4-6, which should be on the web). 
>>
>> The output of env can be found here: 
>> https://dl.dropboxusercontent.com/u/1918141/env.txt [6] 
>>
>> /M 
>

Re: [OMPI devel] OMPI devel] OMPI devel] race condition in abort can cause mpirun v1.8 hang

[Gilles Gouaillardet]

Ralph,

let me correct and enhance my previous statement :

- i cannot reproduce your crash in my environment (RHEL6 like vs your
RHEL7 like)
(i configured with --enable-debug --enable-picky)

- i can reproduce the crash with
mpirun --mca mpi_param_check false

- if you configured with --without-mpi-param-check, i assume you would
get the same crash
(and if i understand correctly, there would be no way to --mca
mpi_param_check true)

here is the relevant part of my config.status :
$ grep MPI_PARAM_CHECK config.status
D["MPI_PARAM_CHECK"]=" ompi_mpi_param_check"
D["OMPI_PARAM_CHECK"]=" 1"

i will try on a centos7 box from now.
in the mean time, can you check you config.status and try again with
mpirun --mca mpi_param_check true

Cheers,

Gilles

On 2014/11/27 10:06, Gilles Gouaillardet wrote:
> I will double check this(afk right now)
> Are you running on a rhel6 like distro with gcc ?
>
> Iirc, crash vs mpi error is ruled by --with-param-check or something like 
> this...
>
> Cheers,
>
> Gilles 
>
> Ralph Castain <r...@open-mpi.org>??:
>> I tried it with both the fortran and c versions - got the same result.
>>
>>
>> This was indeed with a debug build. I wouldn't expect a segfault even with 
>> an optimized build, though - I would expect an MPI error, yes?
>>
>>
>>
>>
>> On Nov 26, 2014, at 4:26 PM, Gilles Gouaillardet 
>> <gilles.gouaillar...@gmail.com> wrote:
>>
>>
>> I will have a look
>>
>> Btw, i was running the fortran version, not the c one.
>> Did you confgure with --enable--debug ?
>> The program sends to a rank *not* in the communicator, so this behavior 
>> could make some sense on an optimized build.
>>
>> Cheers,
>>
>> Gilles
>>
>> Ralph Castain <r...@open-mpi.org>??:
>> Ick - I'm getting a segfault when trying to run that test:
>>
>>
>> MPITEST info  (0): Starting MPI_Errhandler_fatal test
>>
>> MPITEST info  (0): This test will abort after printing the results message
>>
>> MPITEST info  (0): If it does not, then a f.a.i.l.u.r.e will be noted
>>
>> [bend001:07714] *** Process received signal ***
>>
>> [bend001:07714] Signal: Segmentation fault (11)
>>
>> [bend001:07714] Signal code: Address not mapped (1)
>>
>> [bend001:07714] Failing at address: 0x50
>>
>> [bend001:07715] *** Process received signal ***
>>
>> [bend001:07715] Signal: Segmentation fault (11)
>>
>> [bend001:07715] Signal code: Address not mapped (1)
>>
>> [bend001:07715] Failing at address: 0x50
>>
>> [bend001:07714] ompi_comm_peer_lookup: invalid peer index (3)
>>
>> [bend001:07713] ompi_comm_peer_lookup: invalid peer index (3)
>>
>> [bend001:07715] ompi_comm_peer_lookup: invalid peer index (3)
>>
>> [bend001:07713] *** Process received signal ***
>>
>> [bend001:07713] Signal: Segmentation fault (11)
>>
>> [bend001:07713] Signal code: Address not mapped (1)
>>
>> [bend001:07713] Failing at address: 0x50
>>
>> [bend001:07713] [ 0] /usr/lib64/libpthread.so.0(+0xf130)[0x7f4485ecb130]
>>
>> [bend001:07713] [ 1] 
>> /home/common/openmpi/build/ompi-release/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_send+0x5d)[0x7f4480f74ca6]
>>
>> [bend001:07713] [ 2] [bend001:07714] [ 0] 
>> /usr/lib64/libpthread.so.0(+0xf130)[0x7ff457885130]
>>
>> [bend001:07714] [ 1] 
>> /home/common/openmpi/build/ompi-release/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_send+0x5d)[0x7ff44e8dbca6]
>>
>> [bend001:07714] [ 2] [bend001:07715] [ 0] 
>> /usr/lib64/libpthread.so.0(+0xf130)[0x7ffa97ff6130]
>>
>> [bend001:07715] [ 1] 
>> /home/common/openmpi/build/ompi-release/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_send+0x5d)[0x7ffa8ca6]
>>
>> [bend001:07715] [ 2] MPITEST_results: MPI_Errhandler_fatal all tests PASSED 
>> (3)
>>
>>
>>
>> This is with the head of the 1.8 branch. Any suggestions?
>>
>> Ralph
>>
>>
>>
>> On Nov 26, 2014, at 8:46 AM, Ralph Castain <r...@open-mpi.org> wrote:
>>
>>
>> Hmmmyeah, I know we saw this and resolved it in the trunk, but it looks 
>> like the fix indeed failed to come over to 1.8. I'll take a gander (pretty 
>> sure I remember how I fixed it) - thanks!
>>
>> On Nov 26, 2014, at 12:03 AM, Gilles Gouaillardet 
>> <gilles.gouaillar...@iferc.org> wrote:
>>
>> Ralph,
>>
>> i noted several hangs in mtt with the v1.8 branch.
>>
>> a simple way to reproduce it is to use the MPI_Errhandler_fatal_f test

Re: [OMPI devel] OMPI devel] OMPI devel] race condition in abort can cause mpirun v1.8 hang

[Gilles Gouaillardet]

I will double check this(afk right now)
Are you running on a rhel6 like distro with gcc ?

Iirc, crash vs mpi error is ruled by --with-param-check or something like 
this...

Cheers,

Gilles 

Ralph Castain <r...@open-mpi.org>さんのメール:
>I tried it with both the fortran and c versions - got the same result.
>
>
>This was indeed with a debug build. I wouldn’t expect a segfault even with an 
>optimized build, though - I would expect an MPI error, yes?
>
>
>
>
>On Nov 26, 2014, at 4:26 PM, Gilles Gouaillardet 
><gilles.gouaillar...@gmail.com> wrote:
>
>
>I will have a look
>
>Btw, i was running the fortran version, not the c one.
>Did you confgure with --enable--debug ?
>The program sends to a rank *not* in the communicator, so this behavior could 
>make some sense on an optimized build.
>
>Cheers,
>
>Gilles
>
>Ralph Castain <r...@open-mpi.org>さんのメール:
>Ick - I’m getting a segfault when trying to run that test:
>
>
>MPITEST info  (0): Starting MPI_Errhandler_fatal test
>
>MPITEST info  (0): This test will abort after printing the results message
>
>MPITEST info  (0): If it does not, then a f.a.i.l.u.r.e will be noted
>
>[bend001:07714] *** Process received signal ***
>
>[bend001:07714] Signal: Segmentation fault (11)
>
>[bend001:07714] Signal code: Address not mapped (1)
>
>[bend001:07714] Failing at address: 0x50
>
>[bend001:07715] *** Process received signal ***
>
>[bend001:07715] Signal: Segmentation fault (11)
>
>[bend001:07715] Signal code: Address not mapped (1)
>
>[bend001:07715] Failing at address: 0x50
>
>[bend001:07714] ompi_comm_peer_lookup: invalid peer index (3)
>
>[bend001:07713] ompi_comm_peer_lookup: invalid peer index (3)
>
>[bend001:07715] ompi_comm_peer_lookup: invalid peer index (3)
>
>[bend001:07713] *** Process received signal ***
>
>[bend001:07713] Signal: Segmentation fault (11)
>
>[bend001:07713] Signal code: Address not mapped (1)
>
>[bend001:07713] Failing at address: 0x50
>
>[bend001:07713] [ 0] /usr/lib64/libpthread.so.0(+0xf130)[0x7f4485ecb130]
>
>[bend001:07713] [ 1] 
>/home/common/openmpi/build/ompi-release/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_send+0x5d)[0x7f4480f74ca6]
>
>[bend001:07713] [ 2] [bend001:07714] [ 0] 
>/usr/lib64/libpthread.so.0(+0xf130)[0x7ff457885130]
>
>[bend001:07714] [ 1] 
>/home/common/openmpi/build/ompi-release/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_send+0x5d)[0x7ff44e8dbca6]
>
>[bend001:07714] [ 2] [bend001:07715] [ 0] 
>/usr/lib64/libpthread.so.0(+0xf130)[0x7ffa97ff6130]
>
>[bend001:07715] [ 1] 
>/home/common/openmpi/build/ompi-release/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_send+0x5d)[0x7ffa8ca6]
>
>[bend001:07715] [ 2] MPITEST_results: MPI_Errhandler_fatal all tests PASSED (3)
>
>
>
>This is with the head of the 1.8 branch. Any suggestions?
>
>Ralph
>
>
>
>On Nov 26, 2014, at 8:46 AM, Ralph Castain <r...@open-mpi.org> wrote:
>
>
>Hmmm….yeah, I know we saw this and resolved it in the trunk, but it looks like 
>the fix indeed failed to come over to 1.8. I’ll take a gander (pretty sure I 
>remember how I fixed it) - thanks!
>
>On Nov 26, 2014, at 12:03 AM, Gilles Gouaillardet 
><gilles.gouaillar...@iferc.org> wrote:
>
>Ralph,
>
>i noted several hangs in mtt with the v1.8 branch.
>
>a simple way to reproduce it is to use the MPI_Errhandler_fatal_f test
>from the intel_tests suite,
>invoke mpirun on one node and run the taks on an other node :
>
>node0$ mpirun -np 3 -host node1 --mca btl tcp,self ./MPI_Errhandler_fatal_f
>
>/* since this is a race condition, you might need to run this in a loop
>in order to hit the bug */
>
>the attached tarball contains a patch (add debug + temporary hack) and
>some log files obtained with
>--mca errmgr_base_verbose 100 --mca odls_base_verbose 100
>
>without the hack, i can reproduce the bug with -np 3 (log.ko.txt) , with
>the hack, i can still reproduce the hang (though it might
>be a different one) with -np 16 (log.ko.2.txt)
>
>i remember some similar hangs were fixed on the trunk/master a few
>monthes ago.
>i tried to backport some commits but it did not help :-(
>
>could you please have a look at this ?
>
>Cheers,
>
>Gilles
>___
>devel mailing list
>de...@open-mpi.org
>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>Link to this post: 
>http://www.open-mpi.org/community/lists/devel/2014/11/16357.php
>
>
>___
>devel mailing list
>de...@open-mpi.org
>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>Link to this post: 
>http://www.open-mpi.org/community/lists/devel/2014/11/16364.php
>
>

Re: [OMPI devel] OMPI devel] race condition in abort can cause mpirun v1.8 hang

[Gilles Gouaillardet]

I will have a look

Btw, i was running the fortran version, not the c one.
Did you confgure with --enable--debug ?
The program sends to a rank *not* in the communicator, so this behavior could 
make some sense on an optimized build.

Cheers,

Gilles

Ralph Castain <r...@open-mpi.org>さんのメール:
>Ick - I’m getting a segfault when trying to run that test:
>
>
>MPITEST info  (0): Starting MPI_Errhandler_fatal test
>
>MPITEST info  (0): This test will abort after printing the results message
>
>MPITEST info  (0): If it does not, then a f.a.i.l.u.r.e will be noted
>
>[bend001:07714] *** Process received signal ***
>
>[bend001:07714] Signal: Segmentation fault (11)
>
>[bend001:07714] Signal code: Address not mapped (1)
>
>[bend001:07714] Failing at address: 0x50
>
>[bend001:07715] *** Process received signal ***
>
>[bend001:07715] Signal: Segmentation fault (11)
>
>[bend001:07715] Signal code: Address not mapped (1)
>
>[bend001:07715] Failing at address: 0x50
>
>[bend001:07714] ompi_comm_peer_lookup: invalid peer index (3)
>
>[bend001:07713] ompi_comm_peer_lookup: invalid peer index (3)
>
>[bend001:07715] ompi_comm_peer_lookup: invalid peer index (3)
>
>[bend001:07713] *** Process received signal ***
>
>[bend001:07713] Signal: Segmentation fault (11)
>
>[bend001:07713] Signal code: Address not mapped (1)
>
>[bend001:07713] Failing at address: 0x50
>
>[bend001:07713] [ 0] /usr/lib64/libpthread.so.0(+0xf130)[0x7f4485ecb130]
>
>[bend001:07713] [ 1] 
>/home/common/openmpi/build/ompi-release/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_send+0x5d)[0x7f4480f74ca6]
>
>[bend001:07713] [ 2] [bend001:07714] [ 0] 
>/usr/lib64/libpthread.so.0(+0xf130)[0x7ff457885130]
>
>[bend001:07714] [ 1] 
>/home/common/openmpi/build/ompi-release/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_send+0x5d)[0x7ff44e8dbca6]
>
>[bend001:07714] [ 2] [bend001:07715] [ 0] 
>/usr/lib64/libpthread.so.0(+0xf130)[0x7ffa97ff6130]
>
>[bend001:07715] [ 1] 
>/home/common/openmpi/build/ompi-release/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_send+0x5d)[0x7ffa8ca6]
>
>[bend001:07715] [ 2] MPITEST_results: MPI_Errhandler_fatal all tests PASSED (3)
>
>
>
>This is with the head of the 1.8 branch. Any suggestions?
>
>Ralph
>
>
>
>On Nov 26, 2014, at 8:46 AM, Ralph Castain <r...@open-mpi.org> wrote:
>
>
>Hmmm….yeah, I know we saw this and resolved it in the trunk, but it looks like 
>the fix indeed failed to come over to 1.8. I’ll take a gander (pretty sure I 
>remember how I fixed it) - thanks!
>
>On Nov 26, 2014, at 12:03 AM, Gilles Gouaillardet 
><gilles.gouaillar...@iferc.org> wrote:
>
>Ralph,
>
>i noted several hangs in mtt with the v1.8 branch.
>
>a simple way to reproduce it is to use the MPI_Errhandler_fatal_f test
>from the intel_tests suite,
>invoke mpirun on one node and run the taks on an other node :
>
>node0$ mpirun -np 3 -host node1 --mca btl tcp,self ./MPI_Errhandler_fatal_f
>
>/* since this is a race condition, you might need to run this in a loop
>in order to hit the bug */
>
>the attached tarball contains a patch (add debug + temporary hack) and
>some log files obtained with
>--mca errmgr_base_verbose 100 --mca odls_base_verbose 100
>
>without the hack, i can reproduce the bug with -np 3 (log.ko.txt) , with
>the hack, i can still reproduce the hang (though it might
>be a different one) with -np 16 (log.ko.2.txt)
>
>i remember some similar hangs were fixed on the trunk/master a few
>monthes ago.
>i tried to backport some commits but it did not help :-(
>
>could you please have a look at this ?
>
>Cheers,
>
>Gilles
>___
>devel mailing list
>de...@open-mpi.org
>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>Link to this post: 
>http://www.open-mpi.org/community/lists/devel/2014/11/16357.php
>
>

[OMPI devel] race condition in abort can cause mpirun v1.8 hang

[Gilles Gouaillardet]

Ralph,

i noted several hangs in mtt with the v1.8 branch.

a simple way to reproduce it is to use the MPI_Errhandler_fatal_f test
from the intel_tests suite,
invoke mpirun on one node and run the taks on an other node :

node0$ mpirun -np 3 -host node1 --mca btl tcp,self ./MPI_Errhandler_fatal_f

/* since this is a race condition, you might need to run this in a loop
in order to hit the bug */

the attached tarball contains a patch (add debug + temporary hack) and
some log files obtained with
--mca errmgr_base_verbose 100 --mca odls_base_verbose 100

without the hack, i can reproduce the bug with -np 3 (log.ko.txt) , with
the hack, i can still reproduce the hang (though it might
be a different one) with -np 16 (log.ko.2.txt)

i remember some similar hangs were fixed on the trunk/master a few
monthes ago.
i tried to backport some commits but it did not help :-(

could you please have a look at this ?

Cheers,

Gilles


abort_hang.tar.gz
Description: application/gzip

Re: [OMPI devel] OpenMPI v1.8 and v1.8.3 mpirun hangs at execution on an embedded ARM Linux kernel version 3.15.0

[Gilles Gouaillardet]

Ralph and Paul,

On 2014/11/26 10:37, Ralph Castain wrote:
> So it looks like the issue isn't so much with our code as it is with the OS 
> stack, yes? We aren't requiring that the loopback be "up", but the stack is 
> in order to establish the connection, even when we are trying a non-lo 
> interface.
this is correct (imho)
> I can look into generating a faster timeout on the socket creation. In the 
> trunk, we now use unix domain sockets instead of TCP to avoid such issues, 
> but that won't help with the 1.8 series.
i was about to suggest this situation could have been avoided in the
first place by using unix domain sockets instead of TCP sockets :-)

is a backport (since this is already available in the trunk/master)
simply out of the question ?

Cheers,

Gilles

>
>> On Nov 25, 2014, at 4:50 PM, Paul Hargrove  wrote:
>>
>> Ralph,
>>
>> I had a look at the problem via "mpirun -np 1 strace -o trace -ff ./hello"
>> I find that there is an attempt (by a secondary thread) to establish a TCP 
>> socket from the rank process to the eth0 address of localhost (I am guessing 
>> to reach the orted/mpirun).
>> However, when the "lo" interface is down, the Linux kernel apparently cannot 
>> establish that socket.
>>
>> In fact, if I am sufficiently patient, it turns out the "hang" is bounded, 
>> and eventually one sees:
>>
>> phargrov@blcr-armv7:~$ time mpirun -np 1 ./a.out
>> 
>> A process or daemon was unable to complete a TCP connection
>> to another process:
>>   Local host:blcr-armv7
>>   Remote host:   10.0.2.15
>> This is usually caused by a firewall on the remote host. Please
>> check that any firewall (e.g., iptables) has been disabled and
>> try again.
>> 
>>
>> real2m8.151s
>> user0m5.360s
>> sys 0m57.430s
>>
>>
>> Where blcr-armv7 and 10.0.2.15 are *both* the local (only) host.
>>
>> There is no firewall, but in case you doubt me on that, here is a 
>> demonstration using ping to show that 10.0.2.15 is only reachable when the 
>> loopback interface is enabled:
>>
>> phargrov@blcr-armv7:~$ sudo ifconfig lo up
>> phargrov@blcr-armv7:~$ ping -q -c2 10.0.2.15
>> PING 10.0.2.15 (10.0.2.15) 56(84) bytes of data.
>>
>> --- 10.0.2.15 ping statistics ---
>> 2 packets transmitted, 2 received, 0% packet loss, time 1002ms
>> rtt min/avg/max/mdev = 0.527/0.534/0.542/0.024 ms
>>
>>
>> phargrov@blcr-armv7:~$ sudo ifconfig lo down
>> phargrov@blcr-armv7:~$ ping -q -c2 10.0.2.15
>> PING 10.0.2.15 (10.0.2.15) 56(84) bytes of data.
>>
>> --- 10.0.2.15 ping statistics ---
>> 2 packets transmitted, 0 received, 100% packet loss, time 1006ms
>>
>>
>> So, there is no "hang" -- just a 2 minute pause before the error message is 
>> generated.
>> However, it may still be possible to present a better/earlier error message 
>> when there is no loopback interface (and at least one rank process is to be 
>> launched locally).
>>
>>
>> -Paul
>>
>> On Tue, Nov 25, 2014 at 4:19 PM, Ralph Castain > > wrote:
>> I'll have to look - there isn't supposed to be such a requirement, and I 
>> certainly haven't seen it before.
>>
>>
>>> On Nov 25, 2014, at 3:26 PM, Paul Hargrove >> > wrote:
>>>
>>> Allan,
>>>
>>> I am glad things are working for you now.
>>> I can confirm (on a QEMU-emulated Versatile Express A9 board running Ubuntu 
>>> 14.04) that disabling the "lo" interface reproduces the problem.
>>> I imagine this is true on other architectures, though I did not attempt to 
>>> verify.
>>>
>>> Ralph,
>>>
>>> If oob:tcp really does need the loopback interface, shouldn't its lack be 
>>> something that could/should be detected and reported instead of hanging as 
>>> Allan saw?
>>>
>>> FWIW, neither of the following resolved the problem:
>>> -mca oob_tcp_if_exclude lo
>>> -mca oob_tcp_if_include eth0
>>>
>>>
>>> -Paul
>>>
>>> On Tue, Nov 25, 2014 at 2:58 PM, Allan Wu >> > wrote:
>>> I think I have found the problem. After inspecting the output with "-mca 
>>> state_base_verbose 10 -mca odls_base_verbose 10 -mca oob_base_verbose 100" 
>>> on both the old system and the new system, I noticed there is one line that 
>>> is different: on the old system where it works correctly, there is a line 
>>> that says: "oob:tcp:init rejecting loopback interface lo", while on the new 
>>> system there is no such line. Both system proceed to open interface eth0 
>>> afterwards. Then I checked the new system, and found out that somehow the 
>>> loopback interface is not up by default. After I opened the lo interface, 
>>> the mpirun executes normally.
>>>
>>> Does it means that OpenMPI will use lo for some initial setup? Since the 
>>> actual socket was created on eth0 I did not think of checking the lo 
>>> interface. Anyway, thanks everyone for all of 

Re: [OMPI devel] [OMPI users] MPI_Neighbor_alltoallw fails with mpi-1.8.3

[Gilles Gouaillardet]

Hi Ghislain,

that sound like a but in MPI_Dist_graph_create :-(

you can use MPI_Dist_graph_create_adjacent instead :

MPI_Dist_graph_create_adjacent(MPI_COMM_WORLD, degrees, [0],
[0],
degrees, [0], [0], info,
rankReordering, );

it does not crash and as far as i understand, it produces correct results,

according the the mpi standard (example 7.3) that should do the same
thing, that's why
i think there is a bug in MPI_Dist_graph_create

Cheers,

Gilles



On 2014/11/21 2:21, Howard Pritchard wrote:
> Hi Ghislain,
>
> I tried to run your test with mvapich 1.9 and get a "message truncated"
> failure at three ranks.
>
> Howard
>
>
> 2014-11-20 8:51 GMT-07:00 Ghislain Viguier :
>
>> Dear support,
>>
>> I'm encountering an issue with the MPI_Neighbor_alltoallw request of
>> mpi-1.8.3.
>> I have enclosed a test case with information of my workstation.
>>
>> In this test, I define a weighted topology for 5 processes, where the
>> weight represent the number of buffers to send/receive :
>> rank
>>   0 : | x |
>>   1 : | 2 | x |
>>   2 : | 1 | 1 | x |
>>   3 : | 3 | 2 | 3 | x |
>>   4 : | 5 | 2 | 2 | 2 | x |
>>
>> In this topology, the rank 1 will send/receive :
>>2 buffers to/from the rank 0,
>>1 buffer to/from the rank 2,
>>2 buffers to/from the rank 3,
>>2 buffers to/from the rank 4,
>>
>> The send buffer are defined with the MPI_Type_create_hindexed_block. This
>> allows to use a same buffer for several communications without duplicating
>> it (read only).
>> Here the rank 1 will have 2 send buffers (the max of 2, 1, 2, 2).
>> The receiver buffer is a contiguous buffer defined with
>> MPI_Type_contiguous request.
>> Here, the receiver buffer of the rank 1 is of size : 7 (2+1+2+2)
>>
>> This test case succesful for 2 or 3 processes. For 4 processes, the test
>> fails 1 times for 3 successes. For 5 processes, the test fails all the time.
>>
>> The error code is : *** MPI_ERR_IN_STATUS: error code in status
>>
>> I don't understand what I am doing wrong.
>>
>> Could you please have a look on it?
>>
>> Thank you very much.
>>
>> Best regards,
>> Ghislain Viguier
>>
>> --
>> Ghislain Viguier
>> Tél. 06 31 95 03 17
>>
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post:
>> http://www.open-mpi.org/community/lists/users/2014/11/25850.php
>>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2014/11/25852.php

diff --git a/ompi/mca/coll/basic/coll_basic_neighbor_alltoallw.c 
b/ompi/mca/coll/basic/coll_basic_neighbor_alltoallw.c
index 28ecf04..4069212 100644
--- a/ompi/mca/coll/basic/coll_basic_neighbor_alltoallw.c
+++ b/ompi/mca/coll/basic/coll_basic_neighbor_alltoallw.c
@@ -181,7 +181,7 @@ mca_coll_basic_neighbor_alltoallw_dist_graph(const void 
*sbuf, const int scounts
 /* post all receives first */
 for (neighbor = 0, reqs = basic_module->mccb_reqs ; neighbor < indegree ; 
++neighbor) {
 rc = MCA_PML_CALL(irecv((char *) rbuf + rdisps[neighbor], 
rcounts[neighbor], rdtypes[neighbor],
-inedges[neighbor], MCA_COLL_BASE_TAG_ALLTOALL, 
comm, reqs++));
+outedges[neighbor], 
MCA_COLL_BASE_TAG_ALLTOALL, comm, reqs++));
 if (OMPI_SUCCESS != rc) break;
 }

//
// Name: 027_MPI_Neighbor_alltoallw_synthetic.cpp
// Author  : 
// Version :
// Copyright   : Your copyright notice
// Description : Hello World in C++, Ansi-style
//

#include 
#include 
#include 
#include 
#include 
#include 
#include 

using namespace std;

int main(int argc, char *argv[]) {

	const int sendBufferSize = 1;

	///   MPI initialization   ///

	int ierr;
	int nbProc;
	int rank;
	ierr = MPI_Init(, );
	assert(!ierr);
	ierr = MPI_Comm_size(MPI_COMM_WORLD, );
	assert(!ierr);
	ierr = MPI_Comm_rank(MPI_COMM_WORLD, );
	assert(!ierr);

	assert(nbProc <= 5);

	///   weighted topology   ///
	//   0  | x |
	//   1  | 2 | x |
	//   2  | 1 | 1 | x |
	//   3  | 3 | 2 | 3 | x |
	//   4  | 5 | 2 | 2 | 2 | x |
	// rank   0   1   2   3   4

	int degrees = nbProc - 1;
	vector targets(4);
	vector weights(4);
	switch (rank) {

		case 0:
		targets[0] = 1; targets[1] = 2; targets[2] = 3; targets[3] = 4;
		weights[0] = 2; weights[1] = 1; weights[2] = 3; weights[3] = 5;
		break;

		case 1:
		targets[0] = 0; targets[1] = 2; targets[2] = 3; targets[3] = 4;
		weights[0] = 2; weights[1] = 1; weights[2] = 2; weights[3] = 2;
		break;

		case 2:
		targets[0] = 0; targets[1] = 1; 

Re: [OMPI devel] Question about tight integration with not-yet-supported queuing systems

[Gilles Gouaillardet]

Marc,

the reply you pointed is a bit confusing to me :

"There is a native C API which can submit/start/stop/kill/re queue jobs"
this is not what i am looking for :-(

"you need to make an appropriate call to openlava to start a remote process"
this is what i am interested in :-)
could you be more specific (e.g. point me to the functions, since the
OpenLava doc is pretty minimal ...)

the goal here is to spawn the orted daemons as part of the parallel job,
so these daemons are accounted within the parallel job.
/* if we use an API that simply spawns orted, but the orted is not
related whatsoever to the parallel job,
then we can simply use ssh */

Cheers,

Gilles


On 2014/11/18 18:24, Marc Höppner wrote:
> Hi Gilles, 
>
> thanks for the prompt reply. Yes, as far as I know there is a C API to 
> interact with jobs etc. Some mentioning here: 
> https://groups.google.com/forum/#!topic/openlava-users/w74cRUe9Y9E 
> <https://groups.google.com/forum/#!topic/openlava-users/w74cRUe9Y9E>
>
> /Marc
>
> Marc P. Hoeppner, PhD
> Team Leader
> BILS Genome Annotation Platform
> Department for Medical Biochemistry and Microbiology
> Uppsala University, Sweden
> marc.hoepp...@bils.se
>
>> On 18 Nov 2014, at 08:40, Gilles Gouaillardet 
>> <gilles.gouaillar...@iferc.org> wrote:
>>
>> Hi Marc,
>>
>> OpenLava is based on a pretty old version of LSF (4.x if i remember
>> correctly)
>> and i do not think LSF had support for parallel jobs tight integration
>> at that time.
>>
>> my understanding is that basically, there is two kind of direct
>> integration :
>> - mpirun launch: mpirun spawns orted via the API provided by the batch
>> manager
>> - direct launch: the mpi tasks are launched directly from the
>> script/command line and no mpirun/orted is involved
>>  at that time, it works with SLURM and possibly other PMI capable batch
>> manager
>>
>> i think OpenLava simply gets a list of hosts from the environment, build
>> a machinefile, pass it to mpirun that spawns orted with ssh, so this is
>> really loose integration.
>>
>> OpenMPI is based on plugins, so as long as the queing system provides an
>> API to start/stop/kill tasks, mpirun launch should not
>> be a huge effort.
>>
>> Are you aware of such an API provided by OpenLava ?
>>
>> Cheers,
>>
>> Gilles
>>
>> On 2014/11/18 16:31, Marc Höppner wrote:
>>> Hi list,
>>>
>>> I have recently started to wonder how hard it would be to add support for 
>>> queuing systems to the tight integration function of OpenMPI 
>>> (unfortunately, I am not a developer myself). Specifically, we are working 
>>> with OpenLava (www.openlava.org), which is based on an early version of 
>>> Lava/LSF and open source. It's proven quite useful in environments where 
>>> some level of LSF compatibility is needed, but without actually paying for 
>>> a (rather pricey) LSF license. 
>>>
>>> Given that openLava shares quite a bit of DNA with LSF, I was wondering how 
>>> hard it would be to add OL tight integration support to OpenMPI. Currently, 
>>> OL enables OpenMPI jobs through a wrapper script, but that's obviously not 
>>> ideal and doesn't work for some programs that have MPI support built-in 
>>> (and thus expect to be able to just execute mpirun). 
>>>
>>> Any thoughts on this would be greatly appreciated!
>>>
>>> Regards,
>>>
>>> Marc
>>>
>>> Marc P. Hoeppner, PhD
>>> Team Leader
>>> BILS Genome Annotation Platform
>>> Department for Medical Biochemistry and Microbiology
>>> Uppsala University, Sweden
>>> marc.hoepp...@bils.se
>>>
>>> ___
>>> devel mailing list
>>> de...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>> Link to this post: 
>>> http://www.open-mpi.org/community/lists/devel/2014/11/16312.php
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/devel/2014/11/16313.php
>
>
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2014/11/16314.php

Re: [OMPI devel] Question about tight integration with not-yet-supported queuing systems

[Gilles Gouaillardet]

Hi Marc,

OpenLava is based on a pretty old version of LSF (4.x if i remember
correctly)
and i do not think LSF had support for parallel jobs tight integration
at that time.

my understanding is that basically, there is two kind of direct
integration :
- mpirun launch: mpirun spawns orted via the API provided by the batch
manager
- direct launch: the mpi tasks are launched directly from the
script/command line and no mpirun/orted is involved
  at that time, it works with SLURM and possibly other PMI capable batch
manager

i think OpenLava simply gets a list of hosts from the environment, build
a machinefile, pass it to mpirun that spawns orted with ssh, so this is
really loose integration.

OpenMPI is based on plugins, so as long as the queing system provides an
API to start/stop/kill tasks, mpirun launch should not
be a huge effort.

Are you aware of such an API provided by OpenLava ?

Cheers,

Gilles

On 2014/11/18 16:31, Marc Höppner wrote:
> Hi list,
>
> I have recently started to wonder how hard it would be to add support for 
> queuing systems to the tight integration function of OpenMPI (unfortunately, 
> I am not a developer myself). Specifically, we are working with OpenLava 
> (www.openlava.org), which is based on an early version of Lava/LSF and open 
> source. It’s proven quite useful in environments where some level of LSF 
> compatibility is needed, but without actually paying for a (rather pricey) 
> LSF license. 
>
> Given that openLava shares quite a bit of DNA with LSF, I was wondering how 
> hard it would be to add OL tight integration support to OpenMPI. Currently, 
> OL enables OpenMPI jobs through a wrapper script, but that’s obviously not 
> ideal and doesn’t work for some programs that have MPI support built-in (and 
> thus expect to be able to just execute mpirun). 
>
> Any thoughts on this would be greatly appreciated!
>
> Regards,
>
> Marc
>
> Marc P. Hoeppner, PhD
> Team Leader
> BILS Genome Annotation Platform
> Department for Medical Biochemistry and Microbiology
> Uppsala University, Sweden
> marc.hoepp...@bils.se
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2014/11/16312.php

Re: [OMPI devel] Error in version 1.8.3?!

[Gilles Gouaillardet]

Harmut,

this is a known bug.

in the mean time, can you give a try to 1.8.4rc1 ?
http://www.open-mpi.org/software/ompi/v1.8/downloads/openmpi-1.8.4rc1.tar.gz

/* if i remember correctly, this is fixed already in the rc1 */

Cheers,

Gilles

On 2014/11/13 19:48, Hartmut Häfner (SCC) wrote:
> Dear developers,
> we have compiled OpenMPI 1.8.3 with Intel Compiler in version 13.1.5 (and 
> 15.0.0). The configure command is:
> ./configure CC=icc CXX=icpc FC=ifort F90=ifort CFLAGS="-O2 -fno-strict-
> aliasing" CXXFLAGS="-O2" \
> FCFLAGS="-O2" --enable-shared --enable-static --enable-mpi-
> fortran=usempif08 --with-verbs --without-psm --with-slurm --
> prefix=/software/all/openmpi
> /1.8.3_intel_13.1
> We also tried the configure command without the option --enable-mpi-fortran
> and additionally with option  -D__INTEL_COMPILER in CFLAGS, CXXFLAGS and 
> FCFLAGS.
>
> If you want to use the subroutine MPI_Sizeof within a Fortran program, you 
> always get a undefined reference.
>
> We also have tried a test program:
> program testbcast
> use mpi
> implicit none
>  
> logical :: a
> integer :: ierror, rank, size
>  
> a = .true.
> call mpi_init(ierror)
> call mpi_sizeof(a, size, ierror)
> print *,size
> call mpi_finalize(ierror)
> end program testbcast
>
> Using 
> mpif90 -o mpi_sizetest mpi_sizetest.f90
> gives
> /scratch/ifortahgFcM.o: In function `MAIN__':
> mpi_sizetest.f90:(.text+0x4c): undefined reference to `mpi_sizeof0dl_'
>
> (Environment variable OMPI_FCFLAGS is unset)
>
> If we use the GNU compiler instead of the Intel compiler, it works!( but then 
> we run into trouble with the module "mpi" using the Intel compiler for our 
> application.)
>
> We did not find any hints on this erroneous behaviour!
>
>
> Sincerly yours
>
>Hartmut Häfner
>
>
>  
> Hartmut Häfner
> Karlsruhe Institute of Technology (KIT)
> University Karlsruhe (TH)
> Steinbuch Centre for Computing (SCC)
> Scientific Computing and Simulation (SCS)
> Zirkel 2 (Campus Süd, Geb. 20.21, Raum 204)
> D-76128 Karlsruhe
>
> Fon +49(0)721 608 44869
> Fax +49(0)721 32550
> hartmut.haef...@kit.edu
>
> http://www.scc.kit.edu/personen/hartmut.haefner
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2014/11/16307.php

[OMPI devel] oshmem: put does not work with btl/vader if knem is enabled

[Gilles Gouaillardet]

Folks,

I found (at least) two issues with oshmem put if btl/vader is used with
knem enabled :

$ oshrun -np 2 --mca btl vader,self ./oshmem_max_reduction
--
SHMEM_ABORT was invoked on rank 0 (pid 11936, host=soleil) with
errorcode -1.
--
[soleil.iferc.local:11934] 1 more process has sent help message
help-shmem-api.txt / shmem-abort
[soleil.iferc.local:11934] Set MCA parameter "orte_base_help_aggregate"
to 0 to see all help / error messages


the error message is not helpful at all ...
the abort happens in the vader btl in mca_btl_vader_put_knem
   if (OPAL_UNLIKELY(0 != ioctl (mca_btl_vader.knem_fd,
KNEM_CMD_INLINE_COPY, ))) {
return OPAL_ERROR;
}
ioctl fails with EACCES

the root cause is the symmetric memory was "prepared" with
vader_prepare_src that uses
knem_cr.protection = PROT_READ;

a trivial workaround (probably not good for production) is to
knem_cr.protection = PROT_READ|PROT_WRITE;


then we run into the second issue :

in mca_btl_vader_put_knem :
icopy.remote_offset = 0;

and this is clearly not what we want ...
in my environment, we want to put to 0x0600df0, so the remote_offset
should be 0xdf0 since the symmetric memory was "prepared" starting at
0x060

i do not think the vader btl is to be blamed here ... i'd rather think
yoda way to use the btl is not correct (but only for put with vader btl
when knem is used)

i can get the test program run correctly by manually setting
icopy.remote_offset with a debugger.

please note i fixed a typo in the vader btl so make sure you update the
master.


in the mean time, what about forcing put_via_send to 1 in
mca_spml_yoda_put_internal ?
/* an other option is to unset the MCA_BTL_FLAGS_PUT flag in the vader
btl if knem is used, but i do not believe this is a vader issue */

Cheers,

Gilles

Re: [OMPI devel] OMPI devel] Jenkins vs master (and v1.8)

[Gilles Gouaillardet]

Thanks Mike,

BTW what is the distro running on your test cluster ?

Mike Dubman <mi...@dev.mellanox.co.il> wrote:
>ok, I disabled vader tests in SHMEM and it passes.
>
>it can be requested from jenkins by specifying "vader" in PR comment line.
>
>
>On Tue, Nov 11, 2014 at 11:04 AM, Gilles Gouaillardet 
><gilles.gouaillar...@iferc.org> wrote:
>
>Mike,
>
>that will remove the false positive, but also remove an important piece of 
>information :
>there is something wrong with the master.
>
>would you mind discussion this on the weekly call ?
>
>Cheers,
>
>Gilles
>
>
>
>On 2014/11/11 17:38, Mike Dubman wrote:
>
>how about if I will disable the failing test(s) and make jenkins to pass? It 
>will help us to make sure we don`t break something that did work before? On 
>Tue, Nov 11, 2014 at 7:02 AM, Gilles Gouaillardet < 
>gilles.gouaillar...@iferc.org> wrote: 
>
>Mike, Jenkins runs automated tests on each pull request, and i think this is a 
>good thing. recently, it reported a bunch of failure but i could not find 
>anything to blame in the PR itself. so i created a dummy PR 
>https://github.com/open-mpi/ompi/pull/264 with git commit --allow-empty and 
>waited for Jenkins to do its job. the test failed, which means there is an 
>issue in the master. from the master point of view, it is good to know there 
>is an issue. from the PR point of view, this is a false positive since the PR 
>does nothing wrong. i was unable to find anything on github that indicates the 
>master does not pass the automated tests. is such automated test running vs 
>the master ? if yes, where can we find the results ? in order to avoid dealing 
>with false positive, is there any possibility to disable automated tests on 
>the PR if the master does not pass the tests ? Cheers, Gilles 
>___ devel mailing list 
>de...@open-mpi.org Subscription: 
>http://www.open-mpi.org/mailman/listinfo.cgi/devel Link to this post: 
>http://www.open-mpi.org/community/lists/devel/2014/11/16283.php 
>
>
>
>___ devel mailing list 
>de...@open-mpi.org Subscription: 
>http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
>Link to this post: 
>http://www.open-mpi.org/community/lists/devel/2014/11/16284.php 
>
>
>
>___
>devel mailing list
>de...@open-mpi.org
>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>Link to this post: 
>http://www.open-mpi.org/community/lists/devel/2014/11/16285.php
>
>
>
>
>-- 
>
>
>Kind Regards,
>
>
>M.
>

Re: [OMPI devel] Jenkins vs master (and v1.8)

[Gilles Gouaillardet]

Mike,

that will remove the false positive, but also remove an important piece
of information :
there is something wrong with the master.

would you mind discussion this on the weekly call ?

Cheers,

Gilles

On 2014/11/11 17:38, Mike Dubman wrote:
> how about if I will disable the failing test(s) and make jenkins to pass?
> It will help us to make sure we don`t break something that did work before?
>
> On Tue, Nov 11, 2014 at 7:02 AM, Gilles Gouaillardet <
> gilles.gouaillar...@iferc.org> wrote:
>
>> Mike,
>>
>> Jenkins runs automated tests on each pull request, and i think this is a
>> good thing.
>>
>> recently, it reported a bunch of failure but i could not find anything
>> to blame in the PR itself.
>>
>> so i created a dummy PR https://github.com/open-mpi/ompi/pull/264 with
>> git commit --allow-empty
>> and waited for Jenkins to do its job.
>>
>> the test failed, which means there is an issue in the master.
>> from the master point of view, it is good to know there is an issue.
>> from the PR point of view, this is a false positive since the PR does
>> nothing wrong.
>>
>> i was unable to find anything on github that indicates the master does
>> not pass the automated tests.
>> is such automated test running vs the master ? if yes, where can we find
>> the results ?
>> in order to avoid dealing with false positive, is there any possibility
>> to disable automated tests on the PR
>> if the master does not pass the tests ?
>>
>> Cheers,
>>
>> Gilles
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post:
>> http://www.open-mpi.org/community/lists/devel/2014/11/16283.php
>>
>
>
>
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2014/11/16284.php

[OMPI devel] Jenkins vs master (and v1.8)

[Gilles Gouaillardet]

Mike,

Jenkins runs automated tests on each pull request, and i think this is a
good thing.

recently, it reported a bunch of failure but i could not find anything
to blame in the PR itself.

so i created a dummy PR https://github.com/open-mpi/ompi/pull/264 with
git commit --allow-empty
and waited for Jenkins to do its job.

the test failed, which means there is an issue in the master.
from the master point of view, it is good to know there is an issue.
from the PR point of view, this is a false positive since the PR does
nothing wrong.

i was unable to find anything on github that indicates the master does
not pass the automated tests.
is such automated test running vs the master ? if yes, where can we find
the results ?
in order to avoid dealing with false positive, is there any possibility
to disable automated tests on the PR
if the master does not pass the tests ?

Cheers,

Gilles

Re: [OMPI devel] OMPI devel] Pull requests on the trunk

[Gilles Gouaillardet]

My bad (mostly)

I made quite a lot of PR to get some review before commiting to the master, and 
did not follow up in a timely manner.

I closed two obsoletes PR today.

#245 should be ready for prime time.
#227 too unless George has an objection.

I asked Jeff to review #232 and #228 because they are large and/or 
objectionable changes.

I asked George to review #262 since it might require some other changes.

#261 is ready for prime time assuming this is the way we agree to go.

If you think i should post patches/links to my branches to the devel mailing 
list rather than issuing PR, or i should name my branches rfc/something, then 
just let me know.

Cheers,

Gilles


"Jeff Squyres (jsquyres)"  wrote:
>On Nov 6, 2014, at 6:21 PM, Ralph Castain  wrote:
>
>> I agree - I sent the note because I see people doing things a bit 
>> differently than expected. I have no issue with PRs for things where people 
>> want extra eyes on something before committing, or as part of an RFC. Just 
>> want to ensure folks aren’t letting them languish expecting some kind of 
>> gatekeeper to merge them…as that will never happen.
>
>+1
>
>The wiki actually specifically mentions this use case (PR's against master for 
>RFCs and extra eyes).  But it would be good to clarify that there is no 
>gatekeeper for these PRs like there is in ompi-release.
>
>-- 
>Jeff Squyres
>jsquy...@cisco.com
>For corporate legal information go to: 
>http://www.cisco.com/web/about/doing_business/legal/cri/
>
>___
>devel mailing list
>de...@open-mpi.org
>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>Link to this post: 
>http://www.open-mpi.org/community/lists/devel/2014/11/16263.php

Re: [OMPI devel] simple_spawn test fails using different set of btls.

[Gilles Gouaillardet]

Ralph,

i updated the MODEX flag to PMIX_GLOBAL
https://github.com/open-mpi/ompi/commit/d542c9ff2dc57ca5d260d0578fd5c1c556c598c7

Elena,

i was able to reproduce the issue (salloc -N 5 mpirun -np 2 is enough).
i was "lucky" to reproduce the issue : it happened because one of node
was misconfigured
with two interfaces in the same subnet (!)

could you please give a try to the attached patch ?
i did not commit it because i do not know if this is the right fix or a
simple workaround
/* for example, should the opal_proc_t be OBJ_RETAINed before invoking
add_procs and
then be OBJ_RELEASEd by the btl add_proc if it is unreachable ? */

Cheers,

Gilles

On 2014/11/06 12:46, Ralph Castain wrote:
>> On Nov 5, 2014, at 6:11 PM, Gilles Gouaillardet 
>> <gilles.gouaillar...@iferc.org> wrote:
>>
>> Elena,
>>
>> the first case (-mca btl tcp,self) crashing is a bug and i will have a look 
>> at it.
>>
>> the second case (-mca sm,self) is a feature : the sm btl cannot be used with 
>> tasks
>> having different jobids (this is the case after a spawn), and obviously, 
>> self cannot be used also,
>> so the behaviour and error message is correct.
>> /* i am not aware of any plans to make the sm btl work with tasks from 
>> different jobids */\
> That is correct - I'm also unaware of any plans to extend it at this point, 
> though IIRC Nathan at one time mentioned perhaps extending vader for that 
> purpose
>
>> the third case (-mca openib,self) is more controversial ...
>> i previously posted 
>> http://www.open-mpi.org/community/lists/devel/2014/10/16136.php 
>> <http://www.open-mpi.org/community/lists/devel/2014/10/16136.php>
>> what happens in your case (simple_spawn) is the openib modex is sent with 
>> PMIX_REMOTE,
>> that means openib btl cannot be used between tasks on the same node.
>> i am still waiting for some feedback since i cannot figure out whether this 
>> is a feature or an
>> undesired side effect / bug
> I believe it is a bug - I provided some initial values for the modex scope 
> with the expectation (and request when we committed it) that people would 
> review and modify them as appropriate. I recall setting the openib scope as 
> "remote" only because I wasn't aware of anyone using it for local comm. Since 
> Mellanox obviously is testing for that case, a scope of PMIX_GLOBAL would be 
> more appropriate
>
>> the last cast (-mca ^sm,openib) does make sense to me :
>> the tcp and self btls are used and they work just like they should.
>>
>> bottom line, i will investigate the first crash, wait for feedback about the 
>> openib btl.
>>
>> Cheers,
>>
>> Gilles
>>
>> On 2014/11/06 1:08, Elena Elkina wrote:
>>> Hi,
>>>
>>> It looks like there is a problem in trunk which reproduces with
>>> simple_spawn test (orte/test/mpi/simple_spawn.c). It seems to be a n issue
>>> with pmix. It doesn't reproduce with default set of btls. But it reproduces
>>> with several btls specified. For example,
>>>
>>> salloc -N5 $OMPI_HOME/install/bin/mpirun -np 33 --map-by node -mca coll ^ml
>>> -display-map -mca orte_debug_daemons true --leave-session-attached
>>> --debug-daemons -mca pml ob1 -mca btl *tcp,self*
>>> ./orte/test/mpi/simple_spawn
>>>
>>> gets
>>>
>>> simple_spawn: ../../ompi/group/group_init.c:215:
>>> ompi_group_increment_proc_count: Assertion `((0xdeafbeedULL << 32) +
>>> 0xdeafbeedULL) == ((opal_object_t *) (proc_pointer))->obj_magic_id' failed.
>>> [sputnik3.vbench.com:2] [[41877,0],3] orted_cmd: exit cmd, but proc
>>> [[41877,1],2] is alive
>>> [sputnik5][[41877,1],29][../../../../../opal/mca/btl/tcp/btl_tcp_endpoint.c:675:mca_btl_tcp_endpoint_complete_connect]
>>> connect() to 192.168.1.42 failed: Connection refused (111)
>>>
>>> salloc -N1 $OMPI_HOME/install/bin/mpirun -np 3 --map-by node -mca coll ^ml
>>> -display-map -mca orte_debug_daemons true --leave-session-attached
>>> --debug-daemons -mca pml ob1 -mca btl *sm,self* ./orte/test/mpi/simple_spawn
>>>
>>> fails with
>>>
>>> At least one pair of MPI processes are unable to reach each other for
>>> MPI communications.  This means that no Open MPI device has indicated
>>> that it can be used to communicate between these processes.  This is
>>> an error; Open MPI requires that all MPI processes be able to reach
>>> each other.  This error can sometimes be the result of forgetting to
>>> specify the "self" BTL.
>>>
>>>   Process 1 ([

Re: [OMPI devel] simple_spawn test fails using different set of btls.

[Gilles Gouaillardet]

Elena,

the first case (-mca btl tcp,self) crashing is a bug and i will have a
look at it.

the second case (-mca sm,self) is a feature : the sm btl cannot be used
with tasks
having different jobids (this is the case after a spawn), and obviously,
self cannot be used also,
so the behaviour and error message is correct.
/* i am not aware of any plans to make the sm btl work with tasks from
different jobids */

the third case (-mca openib,self) is more controversial ...
i previously posted
http://www.open-mpi.org/community/lists/devel/2014/10/16136.php
what happens in your case (simple_spawn) is the openib modex is sent
with PMIX_REMOTE,
that means openib btl cannot be used between tasks on the same node.
i am still waiting for some feedback since i cannot figure out whether
this is a feature or an
undesired side effect / bug

the last cast (-mca ^sm,openib) does make sense to me :
the tcp and self btls are used and they work just like they should.

bottom line, i will investigate the first crash, wait for feedback about
the openib btl.

Cheers,

Gilles

On 2014/11/06 1:08, Elena Elkina wrote:
> Hi,
>
> It looks like there is a problem in trunk which reproduces with
> simple_spawn test (orte/test/mpi/simple_spawn.c). It seems to be a n issue
> with pmix. It doesn't reproduce with default set of btls. But it reproduces
> with several btls specified. For example,
>
> salloc -N5 $OMPI_HOME/install/bin/mpirun -np 33 --map-by node -mca coll ^ml
> -display-map -mca orte_debug_daemons true --leave-session-attached
> --debug-daemons -mca pml ob1 -mca btl *tcp,self*
> ./orte/test/mpi/simple_spawn
>
> gets
>
> simple_spawn: ../../ompi/group/group_init.c:215:
> ompi_group_increment_proc_count: Assertion `((0xdeafbeedULL << 32) +
> 0xdeafbeedULL) == ((opal_object_t *) (proc_pointer))->obj_magic_id' failed.
> [sputnik3.vbench.com:2] [[41877,0],3] orted_cmd: exit cmd, but proc
> [[41877,1],2] is alive
> [sputnik5][[41877,1],29][../../../../../opal/mca/btl/tcp/btl_tcp_endpoint.c:675:mca_btl_tcp_endpoint_complete_connect]
> connect() to 192.168.1.42 failed: Connection refused (111)
>
> salloc -N1 $OMPI_HOME/install/bin/mpirun -np 3 --map-by node -mca coll ^ml
> -display-map -mca orte_debug_daemons true --leave-session-attached
> --debug-daemons -mca pml ob1 -mca btl *sm,self* ./orte/test/mpi/simple_spawn
>
> fails with
>
> At least one pair of MPI processes are unable to reach each other for
> MPI communications.  This means that no Open MPI device has indicated
> that it can be used to communicate between these processes.  This is
> an error; Open MPI requires that all MPI processes be able to reach
> each other.  This error can sometimes be the result of forgetting to
> specify the "self" BTL.
>
>   Process 1 ([[59481,2],0]) is on host: sputnik1
>   Process 2 ([[59481,1],0]) is on host: sputnik1
>   BTLs attempted: self sm
>
> Your MPI job is now going to abort; sorry.
> --
> [sputnik1.vbench.com:22156] [[59481,1],2] ORTE_ERROR_LOG: Unreachable in
> file ../../../../../ompi/mca/dpm/orte/dpm_orte.c at line 485
>
>
> salloc -N1 $OMPI_HOME/install/bin/mpirun -np 3 --map-by node -mca coll ^ml
> -display-map -mca orte_debug_daemons true --leave-session-attached
> --debug-daemons -mca pml ob1 -mca btl *openib,self*
>  ./orte/test/mpi/simple_spawn
>
> also doesn't work:
> At least one pair of MPI processes are unable to reach each other for
> MPI communications.  This means that no Open MPI device has indicated
> that it can be used to communicate between these processes.  This is
> an error; Open MPI requires that all MPI processes be able to reach
> each other.  This error can sometimes be the result of forgetting to
> specify the "self" BTL.
>
>   Process 1 ([[60046,1],13]) is on host: sputnik4
>   Process 2 ([[60046,2],1]) is on host: sputnik4
>   BTLs attempted: openib self
>
> Your MPI job is now going to abort; sorry.
> --
> [sputnik4.vbench.com:25476] [[60046,1],3] ORTE_ERROR_LOG: Unreachable in
> file ../../../../../ompi/mca/dpm/orte/dpm_orte.c at line 485
>
>
> *But* combination ^sm,openib seems to work.
>
> I tried different revisions from the beginning of October. It reproduces on
> them.
>
> Best regards,
> Elena
>
>
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2014/11/16202.php

Re: [OMPI devel] OMPI 1.8.4rc1 issues

[Gilles Gouaillardet]

Ralph,

On 2014/11/04 1:54, Ralph Castain wrote:
> Hi folks
>
> Looking at the over-the-weekend MTT reports plus at least one comment on the 
> list, we have the following issues to address:
>
> * many-to-one continues to fail. Shall I just assume this is an unfixable 
> problem or a bad test and ignore it?
my understanding is all the tasks send messages to task 0, task 0 cannot
process all of them in real time, store them and keep allocating memory
until an out of memory occurs.
this could be fixed with some kind of flow control but i cannot tell
this is an issue we want to get fixed.
/* this test is pretty extreme and unlikely to occur in the real world,
so solving it might require a lot of work for little value */
> * neighbor_allgather_self segfaults in ompi_request_null or 
> coll_base_comm_unselect or comm_activate or..., take your pick
i made PR #73 https://github.com/open-mpi/ompi-release/pull/73 to fix this.

Cheers,

Gilles

Re: [OMPI devel] [1.8.4rc1] REGRESSION on Solaris-11/x86 with two subnets

[Gilles Gouaillardet]

That works too since pthread is mandatory now
(i previously made a RFC and removing the --with-threads configure
option is in my todo list)

On 2014/11/04 14:10, Ralph Castain wrote:
> Curious - why put it under condition of pthread config? I just added it to 
> the "if solaris" section - i.e., add the flag if we are under solaris, 
> regardless of someone asking for thread support. Since we require that 
> libevent be thread-enabled, it seemed safer to always ensure those flags are 
> set.
>
>
>> On Nov 3, 2014, at 9:05 PM, Gilles Gouaillardet 
>> <gilles.gouaillar...@iferc.org> wrote:
>>
>> Ralph,
>>
>> FYI, here is attached the patch i am working on (still testing ...)
>>
>> aa207ad2f3de5b649e5439d06dca90d86f5a82c2 should be reverted then.
>>
>> Cheers,
>>
>> Gilles
>>
>>
>> On 2014/11/04 13:56, Paul Hargrove wrote:
>>> Ralph,
>>>
>>> You will see from the message I sent a moment ago that -D_REENTRANT on
>>> Solaris appears to be the problem.
>>> However, I will also try the trunk tarball as you have requested.
>>>
>>> -Paul
>>>
>>>
>>> On Mon, Nov 3, 2014 at 8:53 PM, Ralph Castain <rhc.open...@gmail.com> 
>>> <mailto:rhc.open...@gmail.com> wrote:
>>>
>>>> Hmmm...Paul, would you be able to try this with the latest trunk tarball?
>>>> This looks familiar to me, and I wonder if we are just missing a changeset
>>>> from the trunk that fixed the handshake issues we had with failing over
>>>> from one transport to another.
>>>>
>>>> Ralph
>>>>
>>>> On Nov 3, 2014, at 7:23 PM, Paul Hargrove <phhargr...@lbl.gov> 
>>>> <mailto:phhargr...@lbl.gov> wrote:
>>>>
>>>> Ralph,
>>>>
>>>> Requested output is attached.
>>>>
>>>> I have a Linux/x86 system with the same network configuration and will
>>>> soon be able to determine if the problem is specific to Solaris.
>>>>
>>>> -Paul
>>>>
>>>>
>>>> On Mon, Nov 3, 2014 at 7:11 PM, Ralph Castain <rhc.open...@gmail.com> 
>>>> <mailto:rhc.open...@gmail.com>
>>>> wrote:
>>>>
>>>>> Could you please set -mca oob_base_verbose 20? I'm not sure why the
>>>>> connection is failing.
>>>>>
>>>>> Thanks
>>>>> Ralph
>>>>>
>>>>> On Nov 3, 2014, at 5:56 PM, Paul Hargrove <phhargr...@lbl.gov> 
>>>>> <mailto:phhargr...@lbl.gov> wrote:
>>>>>
>>>>> Not clear if the following failure is Solaris-specific, but it *IS* a
>>>>> regression relative to 1.8.3.
>>>>>
>>>>> The system has 2 IPV4 interfaces:
>>>>>Ethernet on 172.16.0.119/16
>>>>>IPoIB on 172.18.0.119/16
>>>>>
>>>>> $ ifconfig bge0
>>>>> bge0: flags=1004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4> mtu 1500
>>>>> index 2
>>>>> inet 172.16.0.119 netmask  broadcast 172.16.255.255
>>>>> $ ifconfig p.ibp0
>>>>> p.ibp0:
>>>>> flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 2044
>>>>> index 3
>>>>> inet 172.18.0.119 netmask  broadcast 172.18.255.255
>>>>>
>>>>> However, I get a message from mca/oob/tcp about not being able to
>>>>> communicate between these two interfaces ON THE SAME NODE:
>>>>>
>>>>> $ /shared/OMPI/openmpi-1.8.4rc1-solaris11-x86-ib-ss12u3/INST/bin/mpirun
>>>>> -mca btl sm,self,openib -np 1 -host pcp-j-19 examples/ring_c
>>>>> [pcp-j-19:00899] mca_oob_tcp_accept: accept() failed: Error 0 (0).
>>>>> 
>>>>> A process or daemon was unable to complete a TCP connection
>>>>> to another process:
>>>>>   Local host:pcp-j-19
>>>>>   Remote host:   172.18.0.119
>>>>> This is usually caused by a firewall on the remote host. Please
>>>>> check that any firewall (e.g., iptables) has been disabled and
>>>>> try again.
>>>>> 
>>>>>
>>>>> Let me know what sort of verbose options I should use to gather any
>>>>> additional info you may

Re: [OMPI devel] [1.8.4rc1] REGRESSION on Solaris-11/x86 with two subnets

[Gilles Gouaillardet]

Ralph,

FYI, here is attached the patch i am working on (still testing ...)

aa207ad2f3de5b649e5439d06dca90d86f5a82c2 should be reverted then.

Cheers,

Gilles


On 2014/11/04 13:56, Paul Hargrove wrote:
> Ralph,
>
> You will see from the message I sent a moment ago that -D_REENTRANT on
> Solaris appears to be the problem.
> However, I will also try the trunk tarball as you have requested.
>
> -Paul
>
>
> On Mon, Nov 3, 2014 at 8:53 PM, Ralph Castain  wrote:
>
>> Hmmm...Paul, would you be able to try this with the latest trunk tarball?
>> This looks familiar to me, and I wonder if we are just missing a changeset
>> from the trunk that fixed the handshake issues we had with failing over
>> from one transport to another.
>>
>> Ralph
>>
>> On Nov 3, 2014, at 7:23 PM, Paul Hargrove  wrote:
>>
>> Ralph,
>>
>> Requested output is attached.
>>
>> I have a Linux/x86 system with the same network configuration and will
>> soon be able to determine if the problem is specific to Solaris.
>>
>> -Paul
>>
>>
>> On Mon, Nov 3, 2014 at 7:11 PM, Ralph Castain 
>> wrote:
>>
>>> Could you please set -mca oob_base_verbose 20? I'm not sure why the
>>> connection is failing.
>>>
>>> Thanks
>>> Ralph
>>>
>>> On Nov 3, 2014, at 5:56 PM, Paul Hargrove  wrote:
>>>
>>> Not clear if the following failure is Solaris-specific, but it *IS* a
>>> regression relative to 1.8.3.
>>>
>>> The system has 2 IPV4 interfaces:
>>>Ethernet on 172.16.0.119/16
>>>IPoIB on 172.18.0.119/16
>>>
>>> $ ifconfig bge0
>>> bge0: flags=1004843 mtu 1500
>>> index 2
>>> inet 172.16.0.119 netmask  broadcast 172.16.255.255
>>> $ ifconfig p.ibp0
>>> p.ibp0:
>>> flags=1001000843 mtu 2044
>>> index 3
>>> inet 172.18.0.119 netmask  broadcast 172.18.255.255
>>>
>>> However, I get a message from mca/oob/tcp about not being able to
>>> communicate between these two interfaces ON THE SAME NODE:
>>>
>>> $ /shared/OMPI/openmpi-1.8.4rc1-solaris11-x86-ib-ss12u3/INST/bin/mpirun
>>> -mca btl sm,self,openib -np 1 -host pcp-j-19 examples/ring_c
>>> [pcp-j-19:00899] mca_oob_tcp_accept: accept() failed: Error 0 (0).
>>> 
>>> A process or daemon was unable to complete a TCP connection
>>> to another process:
>>>   Local host:pcp-j-19
>>>   Remote host:   172.18.0.119
>>> This is usually caused by a firewall on the remote host. Please
>>> check that any firewall (e.g., iptables) has been disabled and
>>> try again.
>>> 
>>>
>>> Let me know what sort of verbose options I should use to gather any
>>> additional info you may need.
>>>
>>> -Paul
>>>
>>> On Fri, Oct 31, 2014 at 7:14 PM, Ralph Castain 
>>> wrote:
>>>
 Hi folks

 I know 1.8.4 isn't entirely complete just yet, but I'd like to get a
 head start on the testing so we can release by Fri Nov 7th. So please take
 a little time and test the current tarball:

 http://www.open-mpi.org/software/ompi/v1.8/

 Thanks
 Ralph


 ___
 devel mailing list
 de...@open-mpi.org
 Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
 Link to this post:
 http://www.open-mpi.org/community/lists/devel/2014/10/16138.php

>>>
>>>
>>> --
>>> Paul H. Hargrove  phhargr...@lbl.gov
>>> Future Technologies Group
>>> Computer and Data Sciences Department Tel: +1-510-495-2352
>>> Lawrence Berkeley National Laboratory Fax: +1-510-486-6900
>>>  ___
>>> devel mailing list
>>> de...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>> Link to this post:
>>> http://www.open-mpi.org/community/lists/devel/2014/11/16160.php
>>>
>>>
>>>
>>> ___
>>> devel mailing list
>>> de...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>> Link to this post:
>>> http://www.open-mpi.org/community/lists/devel/2014/11/16161.php
>>>
>>
>>
>> --
>> Paul H. Hargrove  phhargr...@lbl.gov
>> Future Technologies Group
>> Computer and Data Sciences Department Tel: +1-510-495-2352
>> Lawrence Berkeley National Laboratory Fax: +1-510-486-6900
>>  

Re: [OMPI devel] OMPI devel] [OMPI commits] Git: open-mpi/ompi branch master updated. dev-198-g68bec0a

[Gilles Gouaillardet]

Hi Dave,

I am sorry about that, the doc is not to be blamed here.
I usually do pull/commit/push in a row to avoid this kind of things but i 
screwed up this time ...
I cannot remember if i did commit/pull/push or if i simply forgot to pull

btw, is there a push option to abort if that would make github history non 
linear ?

Cheers,

Gilles

"Dave Goodell (dgoodell)" <dgood...@cisco.com> wrote:
>Hi Gilles,
>
>Please try to avoid creating merge-of-master commits like 68bec0ae ("Merge 
>branch 'master' of..."), they just clutter the history.  A rebase is almost 
>always more appropriate in this situation.
>
>https://github.com/open-mpi/ompi/wiki/GitBestPractices
>
>If you created that commit with "git pull", you can perform the rebase 
>equivalent by running "git pull --rebase".  If this doesn't make sense, just 
>let me know and I can explain further and/or improve the docs around this.
>
>Thanks,
>-Dave
>
>On Oct 31, 2014, at 2:35 AM, git...@crest.iu.edu wrote:
>
>> This is an automated email from the git hooks/post-receive script. It was
>> generated because a ref change was pushed to the repository containing
>> the project "open-mpi/ompi".
>> 
>> The branch, master has been updated
>>   via  68bec0ae1f022e095c132b3f8c7317238b318416 (commit)
>>   via  76ee98c86a7dafc922d342bc4c819ecc3ee14f52 (commit)
>>  from  672d96704cc165b91c7bcf263e97704affcd5f20 (commit)
>> 
>> Those revisions listed above that are new to this repository have
>> not appeared on any other notification email; so we list those
>> revisions in full, below.
>> 
>> - Log ---------
>> https://github.com/open-mpi/ompi/commit/68bec0ae1f022e095c132b3f8c7317238b318416
>> 
>> commit 68bec0ae1f022e095c132b3f8c7317238b318416
>> Merge: 76ee98c 672d967
>> Author: Gilles Gouaillardet <gilles.gouaillar...@iferc.org>
>> Date:   Fri Oct 31 16:34:43 2014 +0900
>> 
>>Merge branch 'master' of https://github.com/open-mpi/ompi
>> 
>> 
>> 
>> https://github.com/open-mpi/ompi/commit/76ee98c86a7dafc922d342bc4c819ecc3ee14f52
>> 
>> commit 76ee98c86a7dafc922d342bc4c819ecc3ee14f52
>> Author: Gilles Gouaillardet <gilles.gouaillar...@iferc.org>
>> Date:   Fri Oct 31 16:34:02 2014 +0900
>> 
>>btl/scif: start the listening thread once only
>> 
>> diff --git a/opal/mca/btl/scif/btl_scif.h b/opal/mca/btl/scif/btl_scif.h
>> index 741fda8..b8d9aab 100644
>> --- a/opal/mca/btl/scif/btl_scif.h
>> +++ b/opal/mca/btl/scif/btl_scif.h
>> @@ -93,6 +93,7 @@ typedef struct mca_btl_scif_module_t {
>> pthread_t listen_thread;
>> 
>> volatile bool exiting;
>> +bool listening;
>> } mca_btl_scif_module_t;
>> 
>> typedef struct mca_btl_scif_component_t {
>> diff --git a/opal/mca/btl/scif/btl_scif_add_procs.c 
>> b/opal/mca/btl/scif/btl_scif_add_procs.c
>> index 80da884..4a6d838 100644
>> --- a/opal/mca/btl/scif/btl_scif_add_procs.c
>> +++ b/opal/mca/btl/scif/btl_scif_add_procs.c
>> @@ -98,10 +98,13 @@ int mca_btl_scif_add_procs(struct mca_btl_base_module_t* 
>> btl,
>> 
>> scif_module->endpoint_count = procs_on_board;
>> 
>> -/* start listening thread */
>> -rc = pthread_create (_btl_scif_module.listen_thread, NULL, 
>> mca_btl_scif_connect_accept, NULL);
>> -if (0 > rc) {
>> -return OPAL_ERROR;
>> +if (!mca_btl_scif_module.listening) {
>> +/* start listening thread */
>> +rc = pthread_create (_btl_scif_module.listen_thread, NULL, 
>> mca_btl_scif_connect_accept, NULL);
>> +if (0 > rc) {
>> +return OPAL_ERROR;
>> +}
>> +mca_btl_scif_module.listening = true;
>> }
>> 
>> return OPAL_SUCCESS;
>> diff --git a/opal/mca/btl/scif/btl_scif_component.c 
>> b/opal/mca/btl/scif/btl_scif_component.c
>> index 7600b12..61bf2d1 100644
>> --- a/opal/mca/btl/scif/btl_scif_component.c
>> +++ b/opal/mca/btl/scif/btl_scif_component.c
>> @@ -267,6 +267,7 @@ static mca_btl_base_module_t 
>> **mca_btl_scif_component_init (int *num_btl_modules
>> 
>> base_modules[0] = _btl_scif_module.super;
>> mca_btl_scif_module.exiting = false;
>> +mca_btl_scif_module.listening = false;
>> 
>> rc = mca_btl_scif_modex_send ();
>> if (OPAL_SUCCESS != rc) {
>> diff --git a/opal/mca/btl/scif/btl_scif_module.c 
>> b/opal/mca/btl/scif/btl_scif_module.c
>> index cfb..fb36b7b 1006

[OMPI devel] btl/openib and MPI_Intercomm_create on the same host

[Gilles Gouaillardet]

Folks,

currently, the dynamic/intercomm_create fails if ran on one host with an
IB port :
mpirun -np 1 ./intercomm_create
/* misleading error message is
opal/mca/btl/openib/connect/btl_openib_connect_udcm.c:1899:udcm_process_messages]
could not find associated endpoint */

this program spawns one task and a second one, create a single
communicator and performs a barrier.

what happens here is :
- tasks 0 and 1 do not use IB as a loopback interface because 
OPAL_MODEX_RECV fails in mca_btl_openib_proc_create()
/* this is ok since the openib modex was sent with PMIX_REMOTE */

but later, task 1 will try to communicate with task 2 via the openib btl.
the reason is task 1 got the openib modex from task 2 via
ompi_comm_get_rprocs invoked by MPI_Intercomm_create

and this will cause an error with a misleading error message reported by
task 2

i wrote the attached hack to "fix" the issue.
i had to strcmp the host names since at that time, proc->proc_flags is
OPAL_PROC_NON_LOCAL

i guess several things are not being handled correctly here, could you
please advise a correct way to fix this ?

Cheers,

Gilles



diff --git a/opal/mca/btl/openib/btl_openib_proc.c 
b/opal/mca/btl/openib/btl_openib_proc.c
index 2d622fe..6e17320 100644
--- a/opal/mca/btl/openib/btl_openib_proc.c
+++ b/opal/mca/btl/openib/btl_openib_proc.c
@@ -12,6 +12,8 @@
  * Copyright (c) 2007-2008 Cisco Systems, Inc.  All rights reserved.
  * Copyright (c) 2006-2007 Voltaire All rights reserved.
  * Copyright (c) 2014  Intel, Inc. All rights reserved.
+ * Copyright (c) 2014  Research Organization for Information Science
+ * and Technology (RIST). All rights reserved.
  * $COPYRIGHT$
  *
  * Additional copyrights may follow
@@ -159,6 +161,13 @@ mca_btl_openib_proc_t* 
mca_btl_openib_proc_create(opal_proc_t* proc)
 if (0 == msg_size) {
 return NULL;
 }
+/* do NOT use ib as a loopback interface */
+if (NULL != proc->proc_hostname) {
+char * h = opal_proc_local_get()->proc_hostname;
+if (strcmp(h, proc->proc_hostname) == 0) {
+return NULL;
+}
+}

 /* Message was packed in btl_openib_component.c; the format is
listed in a comment in that file */

Re: [OMPI devel] errno and reentrance

[Gilles Gouaillardet]

Thanks Paul,

so the simplest way is to force -D_REENTRANT on Solaris, i will do that !

Cheers,

Gilles

On 2014/10/27 19:36, Paul Hargrove wrote:
> Gilles,
>
> I responded too quickly, not thinking that this test is pretty quick and
> doesn't require that I try sparc, ppc, ia64, etc.
> So my results:
>
> Solaris-{10,11}:
>   With "cc" I agree with your findings (need -D_REENTRANT for correct
> behavior).
>   With gcc either "-pthread" or "-D_REENTRANT" gave correct behavior
>
> NetBSD-5:
>   Got "KO: error 4 (0)" no matter what I tried
>
> Linux,  FreeBSD-{9,10}, NetBSD-6, OpenBSD-5:
>   Using "-pthread" or "-lpthread" was necessary to link, and sufficient for
> correct results.
>
> MacOSX-10.{5,6,7,8}:
>   No compiler options were required for 'cc' (which has been gcc, llvm-gcc
> and clang through those OS revs)
>
> Though I have access, I did not try compute nodes on BG/Q or Cray X{E,K,C}.
> Let me know if any of those are of significant concern.
>
> I no longer have AIX or IRIX access.
>
> -Paul
>
>
> On Mon, Oct 27, 2014 at 2:48 AM, Gilles Gouaillardet <
> gilles.gouaillar...@iferc.org> wrote:
>
>>  Thanks Paul !
>>
>> Gilles
>>
>> On 2014/10/27 18:47, Paul Hargrove wrote:
>>
>> On Mon, Oct 27, 2014 at 2:42 AM, Gilles Gouaillardet 
>> <gilles.gouaillar...@iferc.org> wrote:
>> [...]
>>
>>
>>  Paul, since you have access to many platforms, could you please run this
>> test with and without -D_REENTRANT / -D_THREAD_SAFE
>> and tell me where the program produces incorrect behaviour (output is
>> KO...) without the flag ?
>>
>> Thanks in advance,
>>
>> Gilles
>>
>>
>>  Gilles,
>>
>> I have a lot of things due between now and the SC14 conference.
>> I've added this test to my to-do list, but cannot be sure of how soon I'll
>> be able to get results back to you.
>>
>> Feel free to remind me off-list,
>> -Paul
>>
>>
>>
>>
>>
>>
>> ___
>> devel mailing listde...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/devel/2014/10/16095.php
>>
>>
>>
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post:
>> http://www.open-mpi.org/community/lists/devel/2014/10/16096.php
>>
>
>
>
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2014/10/16097.php

Re: [OMPI devel] errno and reentrance

[Gilles Gouaillardet]

Thanks Paul !

Gilles

On 2014/10/27 18:47, Paul Hargrove wrote:
> On Mon, Oct 27, 2014 at 2:42 AM, Gilles Gouaillardet <
> gilles.gouaillar...@iferc.org> wrote:
> [...]
>
>> Paul, since you have access to many platforms, could you please run this
>> test with and without -D_REENTRANT / -D_THREAD_SAFE
>> and tell me where the program produces incorrect behaviour (output is
>> KO...) without the flag ?
>>
>> Thanks in advance,
>>
>> Gilles
>>
> Gilles,
>
> I have a lot of things due between now and the SC14 conference.
> I've added this test to my to-do list, but cannot be sure of how soon I'll
> be able to get results back to you.
>
> Feel free to remind me off-list,
> -Paul
>
>
>
>
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2014/10/16095.php

[OMPI devel] errno and reentrance

[Gilles Gouaillardet]

Folks,

While investigating an issue started at
http://www.open-mpi.org/community/lists/users/2014/10/25562.php
i found that it is mandatory to compile with -D_REENTRANT on Solaris (10
and 11)
(otherwise errno is not per thread specific, and the pmix thread
silently misinterpret EAGAIN or EWOULDBLOCK and that
leads to random behaviour, that generally terminates the application)

This is a bug / unexpected side effect introduced by me in commit
b1c4daa9567c7647318b9b673698c2251264f22e

on a RedHat 6 like server, this is not necessary.

on aix and/or freebsd, it might be necessary to compile with
-D_THREAD_SAFE in order to get a correct behaviour.

i wrote the simple attached program in order to check the correct
behavior with/without -D_REENTRANT or -D_THREAD_SAFE.

one option is to add automatically test this in
config/opal_config_pthreads.m4,
an other option is to hardcode this for the required OS.

Paul, since you have access to many platforms, could you please run this
test with and without -D_REENTRANT / -D_THREAD_SAFE
and tell me where the program produces incorrect behaviour (output is
KO...) without the flag ?

Thanks in advance,

Gilles
#include 
#include 
#include 
#include 

static void * fn (void * arg) {
if (errno == 1) {
return (void *)-1;
}
read(0, NULL, 0);
if (errno != 0) {
return (void *)-2;
}
errno = 2;
return NULL;
}

int main (int argc, char *argv[]) {
pthread_t t;
void *s = NULL;
errno = 1;
if (pthread_create(, NULL, fn, NULL) < 0) {
perror ("pthread_create ");
return 1;
}
if (pthread_join(t, ) < 0) {
perror ("pthread_join ");
return 2;
}
if (NULL != s) {
fprintf(stderr, "KO: error 3 (%ld)\n", (long)s);
return 3;
} else if (2 == errno) {
fprintf(stderr, "KO: error 4 (%ld)\n", (long)s);
return 4;
} else {
fprintf(stderr, "OK\n");
return 0;
}
}

Re: [OMPI devel] [OMPI commits] Git: open-mpi/ompi branch master updated. dev-102-gc9c5d40

[Gilles Gouaillardet]

Ralph and George,

i just made PR 249 https://github.com/open-mpi/ompi/pull/249
this fixes heterogeneous support for the master by moving the jobid,vpid
from the ORTE down to the OPAL layer.

this required to add the OPAL_NAME dss type in order to correctly convert
an opal_process_name_t on an heterogeneous cluster.

could you please have a look at it when you get a chance ?

Cheers,

Gilles

On 2014/10/16 12:26, Gilles Gouaillardet wrote:
> OK, revert done :
>
> commit b5aea782cec116af095a7e7a7310e9e2a018
> Author: Gilles Gouaillardet <gilles.gouaillar...@iferc.org>
> Date:   Thu Oct 16 12:24:38 2014 +0900
>
> Revert "Fix heterogeneous support"
>
> Per the discussion at
> http://www.open-mpi.org/community/lists/devel/2014/10/16050.php
>
> This reverts commit c9c5d4011bf6ea1ade1a5bd9b6a77f02157dc774.
>
>
> Cheers,
>
> Gilles
>
> On 2014/10/16 12:13, Ralph Castain wrote:
>> On Oct 15, 2014, at 8:08 PM, Gilles Gouaillardet 
>> <gilles.gouaillar...@iferc.org> wrote:
>>
>>> Ralph,
>>>
>>> let me quickly reply about this one :
>>>
>>> On 2014/10/16 12:00, Ralph Castain wrote:
>>>> I also don't understand some of the changes in this commit. For example, 
>>>> why did you replace the OPAL_MODEX_SEND_STRING macro with essentially a 
>>>> hard-coded replica of that macro?
>>> OPAL_MODEX_SEND_STRING put a key of type OPAL_BYTE_OBJECT
>>>
>>> in ompi_proc_complete_init:
>>>OPAL_MODEX_RECV_VALUE(ret, OPAL_DSTORE_ARCH,
>>> (opal_proc_t*)>super,
>>>  (void**), OPAL_UINT32);
>>>
>>> a key of type OPAL_UINT32 is expected, and an key of type
>>> OPAL_BYTE_OBJECT was sent
>>>
>>> i chose to "fix" the sender (e.g. send a key of type OPAL_UINT32)
>>>
>>> should i have "fixed" the receiver instead ?
>> Hmmm...probably the receiver, but let me take a look at it. The two should 
>> have mirrored each other, which is why I couldn't understand the change. The 
>> problem may be that the recv should be recv_string, but I need to look at 
>> the two macros and see why the mirrors weren't used.
>>
>>>> Would you mind reverting this until we can better understand what is going 
>>>> on, and decide on a path forward?
>>> no problem
>>> based on my previous comment, shall i also revert the change in
>>> ompi/proc/proc.c as well ?
>> I'd revert the commit as a whole. Let's look at the hetero issue in its 
>> entirety and figure out how we want to handle it.
>>
>> Thanks
>> Ralph
>>
>>> Cheers,
>>>
>>> Gilles
>>>
>>> ___
>>> devel mailing list
>>> de...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>> Link to this post: 
>>> http://www.open-mpi.org/community/lists/devel/2014/10/16049.php
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/devel/2014/10/16050.php
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2014/10/16051.php

Re: [OMPI devel] origin/v1.8 - compilation failed

[Gilles Gouaillardet]

Mike,

the root cause is vader was not fully backported to v1.8
(two OPAL_* macros were not backported to OMPI_*)

i fixed it in https://github.com/open-mpi/ompi-release/pull/49

please note a similar warning is fixed in
https://github.com/open-mpi/ompi-release/pull/48

Cheers,

Gilles

On 2014/10/23 17:14, Mike Dubman wrote:
> *05:12:10* more information, such as the ld(1) and ld.so(8) manual pages.
>
> *05:12:10*
> --
>
> *05:12:10* make[3]: Leaving directory
> `/scrap/jenkins/workspace/hpc-ompi-shmem/label/r-vmb-centos5-u7-x86-64/ompi/mca/btl/vader'
>
> *05:12:10* make[2]: Leaving directory
> `/scrap/jenkins/workspace/hpc-ompi-shmem/label/r-vmb-centos5-u7-x86-64/ompi/mca/btl/vader'
>
> *05:12:10* Making install in mca/btl/sm
>
> *05:12:10* make[2]: Entering directory
> `/scrap/jenkins/workspace/hpc-ompi-shmem/label/r-vmb-centos5-u7-x86-64/ompi/mca/btl/sm'
>
> *05:12:10*   CC   mca_btl_sm_la-btl_sm.lo
>
> *05:12:11*   CC   mca_btl_sm_la-btl_sm_component.lo
>
> *05:12:12* btl_sm_component.c: In function 'mca_btl_sm_component_verify':
>
> *05:12:12* btl_sm_component.c:161: error: 'mca_btl_sm_dummy_get' undeclared
> (first use in this function)
>
> *05:12:12* btl_sm_component.c:161: error: (Each undeclared identifier is
> reported only once
>
> *05:12:12* btl_sm_component.c:161: error: for each function it appears in.)
>
> *05:12:12* btl_sm_component.c: In function 'mca_btl_sm_component_init':
>
> *05:12:12* btl_sm_component.c:915: error: 'opal_process_info' undeclared
> (first use in this function)
>
> *05:12:12* make[2]: *** [mca_btl_sm_la-btl_sm_component.lo] Error 1
>
> *05:12:12* make[2]: Leaving directory
> `/scrap/jenkins/workspace/hpc-ompi-shmem/label/r-vmb-centos5-u7-x86-64/ompi/mca/btl/sm'
>
> *05:12:12* make[1]: *** [install-recursive] Error 1
>
> *05:12:12* make[1]: Leaving directory
> `/scrap/jenkins/workspace/hpc-ompi-shmem/label/r-vmb-centos5-u7-x86-64/ompi'
>
> *05:12:12* make: *** [install-recursive] Error 1
>
> *05:12:12* Build step 'Execute shell' marked build as failure
>
> *05:12:12* [BFA] Scanning build for known causes...
>
>
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2014/10/16079.php

Re: [OMPI devel] OMPI BCOL hang with PMI1

[Gilles Gouaillardet]

Folks,

i commited 248acbbc3ba06c2bef04f840e07816f71f864959 in order to fix a
hang in coll/ml
when using srun (both pmi1 and pmi2)

could you please git it a try ?

Cheers,

Gilles

On 2014/10/22 23:03, Joshua Ladd wrote:
> Privet, Artem
>
> ML is the collective component that is invoking the calls into BCOL. The
> triplet basesmuma,basesmuma,ptpcoll, for example, means I want three levels
> of hierarchy - socket level, UMA level, and then network level. I am
> guessing (only a guess after a quick glance) that maybe srun is not binding
> processes which could result in the socket subgrouping code to fail (it
> should gracefully declare nothing to subgroup, but this is where the bug
> could be.) It will always come to the conclusion that processes are bound
> to the host, so the two level command line should work. Also, you need to
> look at the variable OMPI_MCA_sbgp_base_string (this defines the subgouping
> rules, and the BCOLs are the collective primitives mapped onto a particular
> communication substrate e.g. shared memory, CORE-Direct, vanilla
> point-to-point.)
>
> Can you try with:
> srun -N .. --cpu_bind=cores ...
> and see if this resolves the issue? Also, are you running on a
> hyperthreaded machine?
>
> Another experiment to try:
> I'm assuming this will hang?
> export OMPI_MCA_bcol_base_string=basesmuma,basesmuma,ptpcoll   (this says
> map shared memory collective primitives to both the group of processes
> export OMPI_MCA_sbgp_base_string=basesmsocket,basesmuma,p2p
>
> I would guess this will work
> export OMPI_MCA_bcol_base_string=basesmuma,ptpcoll   (This says only form a
> single shared memory subgroup consisting of processes on host and then a
> single point-to-point subgroup consisting of all host leaders)
> export OMPI_MCA_sbgp_base_string=basesmuma,p2p
>
> I'm speculating that this will hang because of something
> export OMPI_MCA_bcol_base_string=basesmuma,ptpcoll  (This says form groups
> consisting of all procs on the same socket and then take a local leader
> from each of these groups and form a point-to-point group)
> export OMPI_MCA_sbgp_base_string=basesmsocket,p2p
>
> In any case, Elena's suggestion to add -mca coll ^ml will silence all of
> this.
>
> Josh
>
>
>
> On Fri, Oct 17, 2014 at 11:46 AM, Artem Polyakov <artpo...@gmail.com> wrote:
>
>> Hey, Lena :).
>>
>> 2014-10-17 22:07 GMT+07:00 Elena Elkina <elena.elk...@itseez.com>:
>>
>>> Hi Artem,
>>>
>>> Actually some time ago there was a known issue with coll ml. I used to
>>> run my command lines with -mca coll ^ml to avoid these problems, so I don't
>>> know if it was fixed or not. It looks like you have the same problem.
>>>
>> but mine is with bcol, not coll framework. And as you can see modules
>> itself doesn't brake the program. Only some of their combinations. Also I
>> am curious why basesmuma module listed twice.
>>
>>
>>
>>> Best regards,
>>> Elena
>>>
>>> On Fri, Oct 17, 2014 at 7:01 PM, Artem Polyakov <artpo...@gmail.com>
>>> wrote:
>>>
>>>> Gilles,
>>>>
>>>> I checked your patch and it doesn't solve the problem I observe. I think
>>>> the reason is somewhere else.
>>>>
>>>> 2014-10-17 19:13 GMT+07:00 Gilles Gouaillardet <
>>>> gilles.gouaillar...@gmail.com>:
>>>>
>>>>> Artem,
>>>>>
>>>>> There is a known issue #235 with modex and i made PR #238 with a
>>>>> tentative fix.
>>>>>
>>>>> Could you please give it a try and reports if it solves your problem ?
>>>>>
>>>>> Cheers
>>>>>
>>>>> Gilles
>>>>>
>>>>>
>>>>> Artem Polyakov <artpo...@gmail.com> wrote:
>>>>> Hello, I have troubles with latest trunk if I use PMI1.
>>>>>
>>>>> For example, if I use 2 nodes the application hangs. See backtraces
>>>>> from both nodes below. From them I can see that second (non launching) 
>>>>> node
>>>>> hangs in bcol component selection. Here is the default setting of
>>>>> bcol_base_string parameter:
>>>>> bcol_base_string="basesmuma,basesmuma,iboffload,ptpcoll,ugni"
>>>>> according to ompi_info. I don't know if it is correct that basesmuma is
>>>>> duplicated or not.
>>>>>
>>>>> Experiments with this parameter showed that it directly influences the
>>>>> bug:
>>>>> export OMPI_MC

Re: [OMPI devel] OMPI BCOL hang with PMI1

[Gilles Gouaillardet]

Artem,

There is a known issue #235 with modex and i made PR #238 with a tentative fix.

Could you please give it a try and reports if it solves your problem ?

Cheers

Gilles

Artem Polyakov  wrote:
>Hello, I have troubles with latest trunk if I use PMI1.
>
>
>For example, if I use 2 nodes the application hangs. See backtraces from both 
>nodes below. From them I can see that second (non launching) node hangs in 
>bcol component selection. Here is the default setting of bcol_base_string 
>parameter:
>
>bcol_base_string="basesmuma,basesmuma,iboffload,ptpcoll,ugni"
>
>according to ompi_info. I don't know if it is correct that basesmuma is 
>duplicated or not.
>
>
>Experiments with this parameter showed that it directly influences the bug:
>
>export OMPI_MCA_bcol_base_string="" #  [SEGFAULT]
>
>export OMPI_MCA_bcol_base_string="ptpcoll" #  [OK]
>
>export OMPI_MCA_bcol_base_string="basesmuma,ptpcoll" #  [OK]
>
>export OMPI_MCA_bcol_base_string="basesmuma,ptpcoll,iboffload" #  [OK]
>
>export OMPI_MCA_bcol_base_string="basesmuma,ptpcoll,iboffload,ugni" #  [OK]
>
>export OMPI_MCA_bcol_base_string="basesmuma,basesmuma,ptpcoll,iboffload,ugni" 
>#  [HANG]
>
>export OMPI_MCA_bcol_base_string="basesmuma,basesmuma,iboffload,ptpcoll" # 
>[HANG]
>
>export OMPI_MCA_bcol_base_string="basesmuma,basesmuma,iboffload" # [OK]
>
>export OMPI_MCA_bcol_base_string="basesmuma,basesmuma,iboffload,ugni" # [OK]
>
>export OMPI_MCA_bcol_base_string="basesmuma,basesmuma,ptpcoll" #  [HANG]
>
>export OMPI_MCA_bcol_base_string="ptpcoll,basesmuma" #  [OK]
>
>export OMPI_MCA_bcol_base_string="ptpcoll,basesmuma,basesmuma" #  [HANG]
>
>
>I can provide other information if nessesary.
>
>
>cn1:
>
>(gdb) bt
>
>0  0x7fdebd30ac6d in poll () from /lib/x86_64-linux-gnu/libc.so.6
>
>1  0x7fdebcca64e0 in poll_dispatch (base=0x1d466b0, tv=0x7fff71aab880) at 
>poll.c:165
>
>2  0x7fdebcc9b041 in opal_libevent2021_event_base_loop (base=0x1d466b0, 
>flags=2) at event.c:1631
>
>3  0x7fdebcc35891 in opal_progress () at runtime/opal_progress.c:169
>
>4  0x7fdeb32f78cb in opal_condition_wait (c=0x7fdebdb51bc0 
>, m=0x7fdebdb51cc0 ) at 
>../../../../opal/threads/condition.h:78
>
>5  0x7fdeb32f79b8 in ompi_request_wait_completion (req=0x7fff71aab920) at 
>../../../../ompi/request/request.h:381
>
>6  0x7fdeb32f84b8 in mca_pml_ob1_recv (addr=0x7fff71aabd80, count=1, 
>datatype=0x6026c0 , src=1, tag=0, comm=0x6020a0 
>, 
>
>    status=0x7fff71aabd90) at pml_ob1_irecv.c:109
>
>7  0x7fdebd88f54d in PMPI_Recv (buf=0x7fff71aabd80, count=1, type=0x6026c0 
>, source=1, tag=0, comm=0x6020a0 , 
>
>    status=0x7fff71aabd90) at precv.c:78
>
>8  0x00400c44 in main (argc=1, argv=0x7fff71aabe98) at hellompi.c:33
>
>
>cn2:
>
>(gdb) bt
>
>0  0x7fa65aa78c6d in poll () from /lib/x86_64-linux-gnu/libc.so.6
>
>1  0x7fa65a4144e0 in poll_dispatch (base=0x20e96b0, tv=0x7fff46f44a80) at 
>poll.c:165
>
>2  0x7fa65a409041 in opal_libevent2021_event_base_loop (base=0x20e96b0, 
>flags=2) at event.c:1631
>
>3  0x7fa65a3a3891 in opal_progress () at runtime/opal_progress.c:169
>
>4  0x7fa65afbbc25 in opal_condition_wait (c=0x7fa65b2bfbc0 
>, m=0x7fa65b2bfcc0 ) at 
>../opal/threads/condition.h:78
>
>5  0x7fa65afbc1b5 in ompi_request_default_wait_all (count=2, 
>requests=0x7fff46f44c70, statuses=0x0) at request/req_wait.c:287
>
>6  0x7fa65afc7906 in comm_allgather_pml (src_buf=0x7fff46f44da0, 
>dest_buf=0x233dac0, count=288, dtype=0x7fa65b29fee0 , 
>my_rank_in_group=1, 
>
>    n_peers=2, ranks_in_comm=0x210a760, comm=0x6020a0 ) 
>at patterns/comm/allgather.c:250
>
>7  0x7fa64f14ba08 in bcol_basesmuma_smcm_allgather_connection 
>(sm_bcol_module=0x7fa64e64d010, module=0x232c800, 
>
>    peer_list=0x7fa64f3513e8 , 
>back_files=0x7fa64eae2690, comm=0x6020a0 , input=..., 
>
>    base_fname=0x7fa64f14ca8c "sm_ctl_mem_", map_all=false) at 
>bcol_basesmuma_smcm.c:205
>
>8  0x7fa64f146525 in base_bcol_basesmuma_setup_ctl 
>(sm_bcol_module=0x7fa64e64d010, cs=0x7fa64f351220 
>) at bcol_basesmuma_setup.c:344
>
>9  0x7fa64f146cbb in base_bcol_basesmuma_setup_library_buffers 
>(sm_bcol_module=0x7fa64e64d010, cs=0x7fa64f351220 
>)
>
>    at bcol_basesmuma_setup.c:550
>
>10 0x7fa64f1418d0 in mca_bcol_basesmuma_comm_query (module=0x232c800, 
>num_modules=0x232e570) at bcol_basesmuma_module.c:532
>
>11 0x7fa64fd9e5f2 in mca_coll_ml_tree_hierarchy_discovery 
>(ml_module=0x232fbe0, topo=0x232fd98, n_hierarchies=3, exclude_sbgp_name=0x0, 
>include_sbgp_name=0x0)
>
>    at coll_ml_module.c:1964
>
>12 0x7fa64fd9f3a3 in mca_coll_ml_fulltree_hierarchy_discovery 
>(ml_module=0x232fbe0, n_hierarchies=3) at coll_ml_module.c:2211
>
>13 0x7fa64fd9cbe4 in ml_discover_hierarchy (ml_module=0x232fbe0) at 
>coll_ml_module.c:1518
>
>14 0x7fa64fda164f in mca_coll_ml_comm_query (comm=0x6020a0 
>, priority=0x7fff46f45358) at coll_ml_module.c:2970
>
>15 0x7fa65b02f6aa in 

Re: [OMPI devel] Slurm direct-launch is broken on trunk

[Gilles Gouaillardet]

Ralph,

the issue occurs when "pushing" a message that is larger than 255 bytes,
and i fixed it.

/* i am not sure anyone broke this, and fwiw, git blames blamed you */

Cheers,

Gilles

$ git show 27dcca0bb20d8f42b4d048758ef4ff14ca0d79b9
commit 27dcca0bb20d8f42b4d048758ef4ff14ca0d79b9
Author: Gilles Gouaillardet <gilles.gouaillar...@iferc.org>
List-Post: devel@lists.open-mpi.org
Date:   Thu Oct 16 13:29:32 2014 +0900

pmi/s1: fix large keys

do not overwrite the PMI key when pushing a message that does
not fit within 255 bytes

diff --git a/opal/mca/pmix/base/pmix_base_fns.c
b/opal/mca/pmix/base/pmix_base_fns.c
index 56609c5..56c13ba 100644
--- a/opal/mca/pmix/base/pmix_base_fns.c
+++ b/opal/mca/pmix/base/pmix_base_fns.c
@@ -144,7 +144,7 @@ int opal_pmix_base_commit_packed( char*
buffer_to_put, int data_to_put,
 for (left = strlen (encoded_data), tmp = encoded_data ; left ; ) {
 size_t value_size = vallen > left ? left : vallen - 1;

-sprintf (tmp_key, "key%d", *pack_key);
+sprintf (tmp_key, "key%d", pkey);

 if (NULL == (pmikey = setup_key(_PROC_MY_NAME, tmp_key,
vallen))) {
 OPAL_ERROR_LOG(OPAL_ERR_BAD_PARAM);


On 2014/10/16 3:33, Ralph Castain wrote:
> When attempting to launch via srun:
>
> [bend001:03708] pmix: get all keys for proc 327680 in KVS 5.0
> [bend001:03708] GETTING KEY 327680-key0
> [bend001:03708] Read data 
> AcG1peC5obmFtZQAwMwAwMDA4AGJlbmQwMDEAcG1peC5scmFuawAwZAAwMDAycG1peC5ucmFuawAwZAAwMA
>   -
> [bend001:03708] UNSUPPORTED TYPE 0
> [bend001:03708] OPAL ERROR: Error in file pmix_s1.c at line 458
> [bend001:03709] [[5,0],2] pmix:s1 barrier complete
> [bend001:03709] pmix: get all keys for proc 327680 in KVS 5.0
> [bend001:03709] GETTING KEY 327680-key0
> [bend001:03709] Read data 
> AcG1peC5obmFtZQAwMwAwMDA4AGJlbmQwMDEAcG1peC5scmFuawAwZAAwMDAycG1peC5ucmFuawAwZAAwMA
>   -
> [bend001:03709] UNSUPPORTED TYPE 0
> [bend001:03709] OPAL ERROR: Error in file pmix_s1.c at line 458
> [bend001:03708] [[5,0],1] pmix:s1 called get for key pmix.hname
> [bend001:03708] pmix: get all keys for proc 327680 in KVS 5.0
> [bend001:03708] GETTING KEY 327680-key0
> [bend001:03708] Read data 
> AcG1peC5obmFtZQAwMwAwMDA4AGJlbmQwMDEAcG1peC5scmFuawAwZAAwMDAycG1peC5ucmFuawAwZAAwMA
>   -
> [bend001:03708] UNSUPPORTED TYPE 0
> [bend001:03708] [[5,0],1] pmix:s1 got key pmix.hname
>
>
> Looks like someone broke the common code for decoding keys. Could you please 
> fix it?
> Ralph
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2014/10/16046.php

Re: [OMPI devel] [OMPI commits] Git: open-mpi/ompi branch master updated. dev-102-gc9c5d40

[Gilles Gouaillardet]

OK, revert done :

commit b5aea782cec116af095a7e7a7310e9e2a018
Author: Gilles Gouaillardet <gilles.gouaillar...@iferc.org>
List-Post: devel@lists.open-mpi.org
Date:   Thu Oct 16 12:24:38 2014 +0900

Revert "Fix heterogeneous support"

Per the discussion at
http://www.open-mpi.org/community/lists/devel/2014/10/16050.php

This reverts commit c9c5d4011bf6ea1ade1a5bd9b6a77f02157dc774.


Cheers,

Gilles

On 2014/10/16 12:13, Ralph Castain wrote:
> On Oct 15, 2014, at 8:08 PM, Gilles Gouaillardet 
> <gilles.gouaillar...@iferc.org> wrote:
>
>> Ralph,
>>
>> let me quickly reply about this one :
>>
>> On 2014/10/16 12:00, Ralph Castain wrote:
>>> I also don't understand some of the changes in this commit. For example, 
>>> why did you replace the OPAL_MODEX_SEND_STRING macro with essentially a 
>>> hard-coded replica of that macro?
>> OPAL_MODEX_SEND_STRING put a key of type OPAL_BYTE_OBJECT
>>
>> in ompi_proc_complete_init:
>>OPAL_MODEX_RECV_VALUE(ret, OPAL_DSTORE_ARCH,
>> (opal_proc_t*)>super,
>>  (void**), OPAL_UINT32);
>>
>> a key of type OPAL_UINT32 is expected, and an key of type
>> OPAL_BYTE_OBJECT was sent
>>
>> i chose to "fix" the sender (e.g. send a key of type OPAL_UINT32)
>>
>> should i have "fixed" the receiver instead ?
> Hmmm...probably the receiver, but let me take a look at it. The two should 
> have mirrored each other, which is why I couldn't understand the change. The 
> problem may be that the recv should be recv_string, but I need to look at the 
> two macros and see why the mirrors weren't used.
>
>>> Would you mind reverting this until we can better understand what is going 
>>> on, and decide on a path forward?
>> no problem
>> based on my previous comment, shall i also revert the change in
>> ompi/proc/proc.c as well ?
> I'd revert the commit as a whole. Let's look at the hetero issue in its 
> entirety and figure out how we want to handle it.
>
> Thanks
> Ralph
>
>> Cheers,
>>
>> Gilles
>>
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/devel/2014/10/16049.php
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2014/10/16050.php

Re: [OMPI devel] [OMPI commits] Git: open-mpi/ompi branch master updated. dev-102-gc9c5d40

[Gilles Gouaillardet]

Ralph,

let me quickly reply about this one :

On 2014/10/16 12:00, Ralph Castain wrote:
> I also don't understand some of the changes in this commit. For example, why 
> did you replace the OPAL_MODEX_SEND_STRING macro with essentially a 
> hard-coded replica of that macro?
OPAL_MODEX_SEND_STRING put a key of type OPAL_BYTE_OBJECT

in ompi_proc_complete_init:
OPAL_MODEX_RECV_VALUE(ret, OPAL_DSTORE_ARCH,
(opal_proc_t*)>super,
  (void**), OPAL_UINT32);

a key of type OPAL_UINT32 is expected, and an key of type
OPAL_BYTE_OBJECT was sent

i chose to "fix" the sender (e.g. send a key of type OPAL_UINT32)

should i have "fixed" the receiver instead ?
> Would you mind reverting this until we can better understand what is going 
> on, and decide on a path forward?
no problem
based on my previous comment, shall i also revert the change in
ompi/proc/proc.c as well ?

Cheers,

Gilles

Re: [OMPI devel] Fwd: [OMPI commits] Git: open-mpi/ompi branch master updated. dev-102-gc9c5d40

[Gilles Gouaillardet]

Hi Ralph,

i am sorry i should have asked before pushing this to the master.

the master was broken in heterogeneous mode and i took the fastest path
to move it to a working state.
(please note that this commit fixes ompi/proc/proc.c and this is
independent of how opal_process_name_t vs orte_process_name_t
are handled)

the latest email i read about this was my post in the devel ML
http://www.open-mpi.org/community/lists/devel/2014/08/15532.php
if i remember correctly, this topic was also discussed in the weekly
call (and i could not attend it)

if it is finally decided to take the second option, i am afraid it could
be a bit trickier than what i anticipated :
heterogeneous.v2.patch + extra steps to introduce OPAL_PROCESS_NAME dss type
(opal_process_name_t cannot be packed/unpacked as
opal_identifier_t/OPAL_UINT64 any more)

i can make a proof of concept in a branch of my repository if this helps

Cheers,

Gilles

On 2014/10/15 23:08, Ralph Castain wrote:
> Hi Gilles
>
> I'm surprised this came into the trunk - last I saw, we hadn't fully decided 
> which approach we wanted to pursue. Did I miss some discussion?
>
> Due to some other issues, we had been leaning more towards the other 
> alternative - i.e., adding structure to the opal identifier struct. Is there 
> some reason why you chose this alternative?
>
>
> Begin forwarded message:
>
>> From: git...@crest.iu.edu
>> Subject: [OMPI commits] Git: open-mpi/ompi branch master updated. 
>> dev-102-gc9c5d40
>> Date: October 15, 2014 at 3:50:43 AM PDT
>> To: ompi-comm...@open-mpi.org
>> Reply-To: de...@open-mpi.org
>>
>> This is an automated email from the git hooks/post-receive script. It was
>> generated because a ref change was pushed to the repository containing
>> the project "open-mpi/ompi".
>>
>> The branch, master has been updated
>>   via  c9c5d4011bf6ea1ade1a5bd9b6a77f02157dc774 (commit)
>>  from  5c81658d58e260170c995030ac17e42a4032e2dd (commit)
>>
>> Those revisions listed above that are new to this repository have
>> not appeared on any other notification email; so we list those
>> revisions in full, below.
>>
>> - Log -----
>> https://github.com/open-mpi/ompi/commit/c9c5d4011bf6ea1ade1a5bd9b6a77f02157dc774
>>
>> commit c9c5d4011bf6ea1ade1a5bd9b6a77f02157dc774
>> Author: Gilles Gouaillardet <gilles.gouaillar...@iferc.org>
>> Date:   Wed Oct 15 17:19:13 2014 +0900
>>
>>Fix heterogeneous support
>>
>>* redefine orte_process_name_t so it can be converted
>>  between host and network format as an opal_identifier_t
>>  aka uint64_t by the OPAL layer.
>>* correctly send OPAL_DSTORE_ARCH key
>>
>> diff --git a/ompi/proc/proc.c b/ompi/proc/proc.c
>> index d30182f..12b781e 100644
>> --- a/ompi/proc/proc.c
>> +++ b/ompi/proc/proc.c
>> @@ -107,6 +107,7 @@ int ompi_proc_init(void)
>> OMPI_CAST_RTE_NAME(>super.proc_name)->vpid = i;
>>
>> if (i == OMPI_PROC_MY_NAME->vpid) {
>> +opal_value_t kv;
>> ompi_proc_local_proc = proc;
>> proc->super.proc_flags = OPAL_PROC_ALL_LOCAL;
>> proc->super.proc_hostname = strdup(ompi_process_info.nodename);
>> @@ -115,8 +116,13 @@ int ompi_proc_init(void)
>> opal_proc_local_set(>super);
>> #if OPAL_ENABLE_HETEROGENEOUS_SUPPORT
>> /* add our arch to the modex */
>> -OPAL_MODEX_SEND_STRING(ret, PMIX_SYNC_REQD, PMIX_REMOTE, 
>> OPAL_DSTORE_ARCH,
>> -   >super.proc_arch, OPAL_UINT32);
>> +OBJ_CONSTRUCT(, opal_value_t);
>> +kv.key = strdup(OPAL_DSTORE_ARCH);
>> +kv.type = OPAL_UINT32;
>> +kv.data.uint32 = opal_local_arch;
>> +ret = opal_pmix.put(PMIX_REMOTE, );
>> +OBJ_DESTRUCT();
>> +
>> if (OPAL_SUCCESS != ret) {
>> return ret;
>> }
>> diff --git a/opal/util/proc.h b/opal/util/proc.h
>> index 8a52a08..db5cfbc 100644
>> --- a/opal/util/proc.h
>> +++ b/opal/util/proc.h
>> @@ -23,7 +23,7 @@
>> #include "opal/dss/dss.h"
>>
>> #if OPAL_ENABLE_HETEROGENEOUS_SUPPORT
>> -#include 
>> +#include "opal/types.h"
>> #endif
>>
>> /**
>> @@ -37,22 +37,11 @@
>> typedef opal_identifier_t opal_process_name_t;
>>
>> #if OPAL_ENABLE_HETEROGENEOUS_SUPPORT && !defined(WORDS_BIGENDIAN)
>> -#define OPAL_PROCESS_NAME_NTOH(guid) opal_process_name_nto

Re: [OMPI devel] OMPI devel] OMPI devel] OMPI@GitHub: (Mostly) Open for business

[Gilles Gouaillardet]

Jeff,

that should not be an issue since github provides the infrastructure to
filter bozo requests (requests are sha1 signed with a shared secret)
https://developer.github.com/webhooks/securing/

Cheers,

Gilles

On Tue, Oct 7, 2014 at 9:46 PM, Jeff Squyres (jsquyres) <jsquy...@cisco.com>
wrote:

> On Oct 7, 2014, at 6:57 AM, Gilles Gouaillardet <
> gilles.gouaillar...@iferc.org> wrote:
>
> > so far, using webhooks looks really simple :-)
>
> Good!
>
> > a public web server (apache+php) that can
> > a) process json requests
> > b) issue curl requests
> > is required strictly speaking.
>
> My only request would be to ensure that appropriate security is put in so
> that this bot won't be spoofed by any bozo who sends a github-looking json
> request to it.
>
> > i will keep working on a proof of concept
>
> Sounds good.
>
> Note that our current public PHP web pages already issue some curl
> requests behind the scenes.  E.g., the license page (
> http://www.open-mpi.org/community/license.php) does a curl request to get
> the actual license text from our github repo and output it verbatim.  See
> the PHP source code here:
> https://github.com/open-mpi/ompi-www/blob/master/community/license.php.
>
> --
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/10/16018.php
>

Re: [OMPI devel] OMPI devel] OMPI devel] OMPI@GitHub: (Mostly) Open for business

[Gilles Gouaillardet]

Jeff,

here is a quick update :

so far, using webhooks looks really simple :-)

a public web server (apache+php) that can
a) process json requests
b) issue curl requests
is required strictly speaking.

i will keep working on a proof of concept

Cheers,

Gilles

On 2014/10/06 11:55, Gilles Gouaillardet wrote:
> Jeff,
>
> i gave it a little thoughts and that does not seem to hard to achieve.
>
> ghi https://github.com/stephencelis/ghi is a cli to manage (among other
> things) labels, milestones and assignee
>
> the elegant way would be to use webhooks and act accordingly
> (short version: github issue a json request each time an issue is
> modified to the web server of our choice)
> and the "simple" way would be to write a cron script to do this
>
> the first case means we have access to a public web server.
> i have no experience in working with the IU folks, and though this is
> the most technically elegant option,
> there are some more political implications (who
> manage/update/monitor/secure this).
>
> the second option (cron script) could be accepted more easily by IU.
>
> i will experiment on my sandbox from now.
>
> Cheers,
>
> Gilles
>
> On 2014/10/03 22:20, Gilles Gouaillardet wrote:
>> will do !
>>
>> Gilles
>>
>> "Jeff Squyres (jsquyres)" <jsquy...@cisco.com> wrote:
>>> That's a possibility.  IU could probably host this for us.
>>>
>>> Would you mind looking into how hard this would be?
>>>
>>>
>>> On Oct 3, 2014, at 8:41 AM, Gilles Gouaillardet 
>>> <gilles.gouaillar...@gmail.com> wrote:
>>>
>>>> Jeff,
>>>>
>>>> What about a bot using github's rich REST api that parses PR and 
>>>> automatically sets label/milestone/assignee when a keyword is found ?
>>>>
>>>> for example :
>>>> PR:milestone=v1.8.4:assignee=jsquyres:label=bug
>>>> or
>>>> #milestone=v1.8.4 #assignee=@jsquyres #label=bug
>>>>
>>>> i can investigate this way if needed
>>>>
>>>> Cheers,
>>>>
>>>> Gilles
>>>>
>>>> "Jeff Squyres (jsquyres)" <jsquy...@cisco.com> wrote:
>>>>> Gah.
>>>>>
>>>>> I just did some experimentation and confirmed this behavior:
>>>>>
>>>>> 1. If you do not belong to a repo, you can file issues/pull requests and 
>>>>> comment on them.
>>>>>
>>>>> 2. If you have *read* access to a repo, you can do everything from #1, 
>>>>> but you're also eligible for @mention auto-complete, and issues/pull 
>>>>> requests can be assigned to you.
>>>>>
>>>>> 3. If you have *write* access to a repo, you can do everything from #2, 
>>>>> but you can also set labels, milestones, and assignees.
>>>>>
>>>>> All the OMPI devs have *read* access to ompi-release, meaning you can 
>>>>> create/comment on issues, but not set labels/milestones/assignees.
>>>>>
>>>>> I did not expect this behavior.  Grumble.  Will have to think about it a 
>>>>> bit...
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Oct 3, 2014, at 7:07 AM, Gilles Gouaillardet 
>>>>> <gilles.gouaillar...@gmail.com> wrote:
>>>>>
>>>>>> Jeff,
>>>>>>
>>>>>> my point is that label, milestone and assignee are *not* clickable for 
>>>>>> me (see attached snapshot, "settings" icons are missing
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Gilles
>>>>>>
>>>>>> On Fri, Oct 3, 2014 at 7:36 PM, Jeff Squyres (jsquyres) 
>>>>>> <jsquy...@cisco.com> wrote:
>>>>>> You can only assign a label and milestone, and assign the PR to someone, 
>>>>>> *after* you create the PR (same with creating issues).
>>>>>>
>>>>>> See https://github.com/open-mpi/ompi/wiki/SubmittingBugs for details:
>>>>>>
>>>>>> 
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Oct 2, 2014, at 11:37 PM, Gilles Gouaillardet 
>>>>>> <gilles.gouaillar...@iferc.org> wrote:
>>>>>>
>>>>>>
>>>>>>> Jeff,
>>>>>>>
>>>

Re: [OMPI devel] OMPI devel] OMPI devel] OMPI@GitHub: (Mostly) Open for business

[Gilles Gouaillardet]

Jeff,

i gave it a little thoughts and that does not seem to hard to achieve.

ghi https://github.com/stephencelis/ghi is a cli to manage (among other
things) labels, milestones and assignee

the elegant way would be to use webhooks and act accordingly
(short version: github issue a json request each time an issue is
modified to the web server of our choice)
and the "simple" way would be to write a cron script to do this

the first case means we have access to a public web server.
i have no experience in working with the IU folks, and though this is
the most technically elegant option,
there are some more political implications (who
manage/update/monitor/secure this).

the second option (cron script) could be accepted more easily by IU.

i will experiment on my sandbox from now.

Cheers,

Gilles

On 2014/10/03 22:20, Gilles Gouaillardet wrote:
> will do !
>
> Gilles
>
> "Jeff Squyres (jsquyres)" <jsquy...@cisco.com> wrote:
>> That's a possibility.  IU could probably host this for us.
>>
>> Would you mind looking into how hard this would be?
>>
>>
>> On Oct 3, 2014, at 8:41 AM, Gilles Gouaillardet 
>> <gilles.gouaillar...@gmail.com> wrote:
>>
>>> Jeff,
>>>
>>> What about a bot using github's rich REST api that parses PR and 
>>> automatically sets label/milestone/assignee when a keyword is found ?
>>>
>>> for example :
>>> PR:milestone=v1.8.4:assignee=jsquyres:label=bug
>>> or
>>> #milestone=v1.8.4 #assignee=@jsquyres #label=bug
>>>
>>> i can investigate this way if needed
>>>
>>> Cheers,
>>>
>>> Gilles
>>>
>>> "Jeff Squyres (jsquyres)" <jsquy...@cisco.com> wrote:
>>>> Gah.
>>>>
>>>> I just did some experimentation and confirmed this behavior:
>>>>
>>>> 1. If you do not belong to a repo, you can file issues/pull requests and 
>>>> comment on them.
>>>>
>>>> 2. If you have *read* access to a repo, you can do everything from #1, but 
>>>> you're also eligible for @mention auto-complete, and issues/pull requests 
>>>> can be assigned to you.
>>>>
>>>> 3. If you have *write* access to a repo, you can do everything from #2, 
>>>> but you can also set labels, milestones, and assignees.
>>>>
>>>> All the OMPI devs have *read* access to ompi-release, meaning you can 
>>>> create/comment on issues, but not set labels/milestones/assignees.
>>>>
>>>> I did not expect this behavior.  Grumble.  Will have to think about it a 
>>>> bit...
>>>>
>>>>
>>>>
>>>>
>>>> On Oct 3, 2014, at 7:07 AM, Gilles Gouaillardet 
>>>> <gilles.gouaillar...@gmail.com> wrote:
>>>>
>>>>> Jeff,
>>>>>
>>>>> my point is that label, milestone and assignee are *not* clickable for me 
>>>>> (see attached snapshot, "settings" icons are missing
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Gilles
>>>>>
>>>>> On Fri, Oct 3, 2014 at 7:36 PM, Jeff Squyres (jsquyres) 
>>>>> <jsquy...@cisco.com> wrote:
>>>>> You can only assign a label and milestone, and assign the PR to someone, 
>>>>> *after* you create the PR (same with creating issues).
>>>>>
>>>>> See https://github.com/open-mpi/ompi/wiki/SubmittingBugs for details:
>>>>>
>>>>> 
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Oct 2, 2014, at 11:37 PM, Gilles Gouaillardet 
>>>>> <gilles.gouaillar...@iferc.org> wrote:
>>>>>
>>>>>
>>>>>> Jeff,
>>>>>>
>>>>>> i could not find how to apply a label to a PR via the web interface (and
>>>>>> i am not sure i can even do that since authority might be required)
>>>>>>
>>>>>> any idea (maybe a special keyword in the comment ...) ?
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Gilles
>>>>>>
>>>>>> On 2014/10/03 1:53, Jeff Squyres (jsquyres) wrote:
>>>>>>> On Oct 2, 2014, at 12:47 PM, Ralph Castain <r...@open-mpi.org> wrote:
>>>>>>>
>>>>>>>> How are they going to review it, given they don't have authority to do 
>>>>>>>> anythin

[OMPI devel] ompi github repository is NOT up to date

[Gilles Gouaillardet]

Folks,

currently, https://github.com/open-mpi/ompi last commit was 13 days ago
(see attached snapshot)

this is not the most up to date state !

for example, the last commit of my clone is

commit 54c839a970fc3025a08fe1c04b7d4b9767078264
Merge: dee6b63 5c5453b
Author: Gilles Gouaillardet <gilles.gouaillar...@iferc.org>
List-Post: devel@lists.open-mpi.org
Date:   Fri Oct 3 11:57:33 2014 +0900

Merge branch 'topic/fix-pmix-native-getattr'

and if i remember correctly, some stuff has been commited after that
(though this was not reflected in my clone ...)

Could someone have a look at this ?
/* i am *not* a git guru and i do not have the most up to date clone */

in the mean time, i strongly advise everybody *not* to push anything to
the ompi repository

Cheers,

Gilles

Re: [OMPI devel] OMPI devel] OMPI devel] OMPI@GitHub: (Mostly) Open for business

[Gilles Gouaillardet]

will do !

Gilles

"Jeff Squyres (jsquyres)" <jsquy...@cisco.com> wrote:
>That's a possibility.  IU could probably host this for us.
>
>Would you mind looking into how hard this would be?
>
>
>On Oct 3, 2014, at 8:41 AM, Gilles Gouaillardet 
><gilles.gouaillar...@gmail.com> wrote:
>
>> Jeff,
>> 
>> What about a bot using github's rich REST api that parses PR and 
>> automatically sets label/milestone/assignee when a keyword is found ?
>> 
>> for example :
>> PR:milestone=v1.8.4:assignee=jsquyres:label=bug
>> or
>> #milestone=v1.8.4 #assignee=@jsquyres #label=bug
>> 
>> i can investigate this way if needed
>> 
>> Cheers,
>> 
>> Gilles
>> 
>> "Jeff Squyres (jsquyres)" <jsquy...@cisco.com> wrote:
>>> Gah.
>>> 
>>> I just did some experimentation and confirmed this behavior:
>>> 
>>> 1. If you do not belong to a repo, you can file issues/pull requests and 
>>> comment on them.
>>> 
>>> 2. If you have *read* access to a repo, you can do everything from #1, but 
>>> you're also eligible for @mention auto-complete, and issues/pull requests 
>>> can be assigned to you.
>>> 
>>> 3. If you have *write* access to a repo, you can do everything from #2, but 
>>> you can also set labels, milestones, and assignees.
>>> 
>>> All the OMPI devs have *read* access to ompi-release, meaning you can 
>>> create/comment on issues, but not set labels/milestones/assignees.
>>> 
>>> I did not expect this behavior.  Grumble.  Will have to think about it a 
>>> bit...
>>> 
>>> 
>>> 
>>> 
>>> On Oct 3, 2014, at 7:07 AM, Gilles Gouaillardet 
>>> <gilles.gouaillar...@gmail.com> wrote:
>>> 
>>>> Jeff,
>>>> 
>>>> my point is that label, milestone and assignee are *not* clickable for me 
>>>> (see attached snapshot, "settings" icons are missing
>>>> 
>>>> Cheers,
>>>> 
>>>> Gilles
>>>> 
>>>> On Fri, Oct 3, 2014 at 7:36 PM, Jeff Squyres (jsquyres) 
>>>> <jsquy...@cisco.com> wrote:
>>>> You can only assign a label and milestone, and assign the PR to someone, 
>>>> *after* you create the PR (same with creating issues).
>>>> 
>>>> See https://github.com/open-mpi/ompi/wiki/SubmittingBugs for details:
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Oct 2, 2014, at 11:37 PM, Gilles Gouaillardet 
>>>> <gilles.gouaillar...@iferc.org> wrote:
>>>> 
>>>> 
>>>>> Jeff,
>>>>> 
>>>>> i could not find how to apply a label to a PR via the web interface (and
>>>>> i am not sure i can even do that since authority might be required)
>>>>> 
>>>>> any idea (maybe a special keyword in the comment ...) ?
>>>>> 
>>>>> Cheers,
>>>>> 
>>>>> Gilles
>>>>> 
>>>>> On 2014/10/03 1:53, Jeff Squyres (jsquyres) wrote:
>>>>>> On Oct 2, 2014, at 12:47 PM, Ralph Castain <r...@open-mpi.org> wrote:
>>>>>> 
>>>>>>> How are they going to review it, given they don't have authority to do 
>>>>>>> anything on that branch? Can they still comment? Can they reassign them 
>>>>>>> when done?
>>>>>> Yes, they can comment.
>>>>>> 
>>>>>> The idea was that they would apply the "reviewed" label when they have 
>>>>>> reviewed it successfully.  These "reviewed" labels will then be easily 
>>>>>> viewable in the PR listing by the RM/GK to know which ones are ready.
>>>>>> 
>>>>> 
>>>>> ___
>>>>> devel mailing list
>>>>> de...@open-mpi.org
>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>> Link to this post: 
>>>>> http://www.open-mpi.org/community/lists/devel/2014/10/15988.php
>>>> 
>>>> 
>>>> -- 
>>>> Jeff Squyres
>>>> jsquy...@cisco.com
>>>> For corporate legal information go to: 
>>>> http://www.cisco.com/web/about/doing_business/legal/cri/
>>>> 
>>>> 
>

Re: [OMPI devel] OMPI devel] OMPI@GitHub: (Mostly) Open for business

[Gilles Gouaillardet]

Jeff,

What about a bot using github's rich REST api that parses PR and automatically 
sets label/milestone/assignee when a keyword is found ?

for example :
PR:milestone=v1.8.4:assignee=jsquyres:label=bug
or
#milestone=v1.8.4 #assignee=@jsquyres #label=bug

i can investigate this way if needed

Cheers,

Gilles

"Jeff Squyres (jsquyres)" <jsquy...@cisco.com> wrote:
>Gah.
>
>I just did some experimentation and confirmed this behavior:
>
>1. If you do not belong to a repo, you can file issues/pull requests and 
>comment on them.
>
>2. If you have *read* access to a repo, you can do everything from #1, but 
>you're also eligible for @mention auto-complete, and issues/pull requests can 
>be assigned to you.
>
>3. If you have *write* access to a repo, you can do everything from #2, but 
>you can also set labels, milestones, and assignees.
>
>All the OMPI devs have *read* access to ompi-release, meaning you can 
>create/comment on issues, but not set labels/milestones/assignees.
>
>I did not expect this behavior.  Grumble.  Will have to think about it a bit...
>
>
>
>
>On Oct 3, 2014, at 7:07 AM, Gilles Gouaillardet 
><gilles.gouaillar...@gmail.com> wrote:
>
>> Jeff,
>> 
>> my point is that label, milestone and assignee are *not* clickable for me 
>> (see attached snapshot, "settings" icons are missing
>> 
>> Cheers,
>> 
>> Gilles
>> 
>> On Fri, Oct 3, 2014 at 7:36 PM, Jeff Squyres (jsquyres) <jsquy...@cisco.com> 
>> wrote:
>> You can only assign a label and milestone, and assign the PR to someone, 
>> *after* you create the PR (same with creating issues).
>> 
>> See https://github.com/open-mpi/ompi/wiki/SubmittingBugs for details:
>> 
>> 
>> 
>> 
>> 
>> 
>> On Oct 2, 2014, at 11:37 PM, Gilles Gouaillardet 
>> <gilles.gouaillar...@iferc.org> wrote:
>> 
>> 
>>> Jeff,
>>> 
>>> i could not find how to apply a label to a PR via the web interface (and
>>> i am not sure i can even do that since authority might be required)
>>> 
>>> any idea (maybe a special keyword in the comment ...) ?
>>> 
>>> Cheers,
>>> 
>>> Gilles
>>> 
>>> On 2014/10/03 1:53, Jeff Squyres (jsquyres) wrote:
>>>> On Oct 2, 2014, at 12:47 PM, Ralph Castain <r...@open-mpi.org> wrote:
>>>> 
>>>>> How are they going to review it, given they don't have authority to do 
>>>>> anything on that branch? Can they still comment? Can they reassign them 
>>>>> when done?
>>>> Yes, they can comment.
>>>> 
>>>> The idea was that they would apply the "reviewed" label when they have 
>>>> reviewed it successfully.  These "reviewed" labels will then be easily 
>>>> viewable in the PR listing by the RM/GK to know which ones are ready.
>>>> 
>>> 
>>> ___
>>> devel mailing list
>>> de...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>> Link to this post: 
>>> http://www.open-mpi.org/community/lists/devel/2014/10/15988.php
>> 
>> 
>> -- 
>> Jeff Squyres
>> jsquy...@cisco.com
>> For corporate legal information go to: 
>> http://www.cisco.com/web/about/doing_business/legal/cri/
>> 
>> 
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/devel/2014/10/15990.php
>> 
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/devel/2014/10/15992.php
>
>
>-- 
>Jeff Squyres
>jsquy...@cisco.com
>For corporate legal information go to: 
>http://www.cisco.com/web/about/doing_business/legal/cri/
>
>___
>devel mailing list
>de...@open-mpi.org
>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>Link to this post: 
>http://www.open-mpi.org/community/lists/devel/2014/10/15995.php

Re: [OMPI devel] OMPI@GitHub: (Mostly) Open for business

[Gilles Gouaillardet]

On Fri, Oct 3, 2014 at 7:29 PM, Jeff Squyres (jsquyres) <jsquy...@cisco.com>
wrote:

> On Oct 2, 2014, at 11:33 PM, Gilles Gouaillardet <
> gilles.gouaillar...@iferc.org> wrote:
>
> > the most painful part is probably to manually retrieve the git commit id
> > of a given svn commit id
> >
> > git log master, and then search r
> > /* each commit log has a line like :
> > This commit was SVN rx
> > */
>
> I think you're suggesting "git log | grep SVN rxxx", right?
>
> i do not think this returns the git commit id, does it ?

what i had in mind is :
git log master
and then /r
and manually retrieve the git commit id a few lines above

this is very likely achievable with an awk oneliner, but i do not know how
...

Cheers,

Gilles

Re: [OMPI devel] OMPI@GitHub: (Mostly) Open for business

[Gilles Gouaillardet]

Jeff,

my point is that label, milestone and assignee are *not* clickable for me
(see attached snapshot, "settings" icons are missing

Cheers,

Gilles

On Fri, Oct 3, 2014 at 7:36 PM, Jeff Squyres (jsquyres) <jsquy...@cisco.com>
wrote:

>  You can only assign a label and milestone, and assign the PR to someone,
> *after* you create the PR (same with creating issues).
>
>  See https://github.com/open-mpi/ompi/wiki/SubmittingBugs for details:
>
>
>
>
>
>
>  On Oct 2, 2014, at 11:37 PM, Gilles Gouaillardet <
> gilles.gouaillar...@iferc.org> wrote:
>
>
> Jeff,
>
> i could not find how to apply a label to a PR via the web interface (and
> i am not sure i can even do that since authority might be required)
>
> any idea (maybe a special keyword in the comment ...) ?
>
> Cheers,
>
> Gilles
>
> On 2014/10/03 1:53, Jeff Squyres (jsquyres) wrote:
>
> On Oct 2, 2014, at 12:47 PM, Ralph Castain <r...@open-mpi.org> wrote:
>
> How are they going to review it, given they don't have authority to do
> anything on that branch? Can they still comment? Can they reassign them
> when done?
>
> Yes, they can comment.
>
> The idea was that they would apply the "reviewed" label when they have
> reviewed it successfully.  These "reviewed" labels will then be easily
> viewable in the PR listing by the RM/GK to know which ones are ready.
>
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/10/15988.php
>
>
>
> --
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/10/15990.php
>

Re: [OMPI devel] OMPI@GitHub: (Mostly) Open for business

[Gilles Gouaillardet]

Jeff,

i could not find how to apply a label to a PR via the web interface (and
i am not sure i can even do that since authority might be required)

any idea (maybe a special keyword in the comment ...) ?

Cheers,

Gilles

On 2014/10/03 1:53, Jeff Squyres (jsquyres) wrote:
> On Oct 2, 2014, at 12:47 PM, Ralph Castain  wrote:
>
>> How are they going to review it, given they don't have authority to do 
>> anything on that branch? Can they still comment? Can they reassign them when 
>> done?
> Yes, they can comment.
>
> The idea was that they would apply the "reviewed" label when they have 
> reviewed it successfully.  These "reviewed" labels will then be easily 
> viewable in the PR listing by the RM/GK to know which ones are ready.
>

Re: [OMPI devel] OMPI@GitHub: (Mostly) Open for business

[Gilles Gouaillardet]

Ralph,

what about cherry-pick ?

the most painful part is probably to manually retrieve the git commit id
of a given svn commit id

git log master, and then search r
/* each commit log has a line like :
This commit was SVN rx
*/

and once you got (all) the git commits id of a given CMR, you can
cherry-pick them,
and push and issue the PR

/* i did not check whether cherry-pick automatically create/remove files */

Cheers,

Gilles

On 2014/10/03 0:51, Ralph Castain wrote:
> I agree that someone needs to do it. However, what isn't clear is *how* to do 
> it. I get the mechanics:
>
> * clone the ompi-release 1.8 branch
> * apply the required change
> * issue a pull request
>
> It's the second step that isn't clear. How the heck do I (a) find the change 
> that was originally applied, and (b) apply it to my clone of ompi-release???
>
> Do I just download the changeset(s) from Trac and patch them in? I guess that 
> will work if we don't have an alternative method.
>
>
> On Oct 2, 2014, at 3:41 AM, Jeff Squyres (jsquyres) <jsquy...@cisco.com> 
> wrote:
>
>> Fair enough.
>>
>> *Someone* needs to re-file those CMRs as pull requests; probably in some 
>> cases it's the reporter, probably in some cases it's the owner.  :-)
>>
>>
>> On Oct 2, 2014, at 6:33 AM, Gilles Gouaillardet 
>> <gilles.gouaillar...@iferc.org> wrote:
>>
>>> Hi Jeff,
>>>
>>> thumbs up for the migration !
>>>
>>> the names here are the CMR owners ('Owned by' field in TRAC)
>>> should it be the duty of the creators ('Reported by' field in TRAC) to
>>> re-create the CMR instead?
>>>
>>> /* if not, and from a git log point of view, that means the commiter
>>> will be the reviewer and not the reporter */
>>>
>>> what do you think ?
>>>
>>> Cheers,
>>>
>>> Gilles
>>>
>>> On 2014/10/02 19:24, Jeff Squyres (jsquyres) wrote:
>>>> 2. The following still-open CMRs were not brought over, and need to be 
>>>> re-filed as pull requests:
>>>>
>>>> - https://svn.open-mpi.org/trac/ompi/ticket/4216: Nathan (BLOCKER)
>>>> - https://svn.open-mpi.org/trac/ompi/ticket/4816: Ralph
>>>> - https://svn.open-mpi.org/trac/ompi/ticket/4872: Ralph
>>>> - https://svn.open-mpi.org/trac/ompi/ticket/4888: Howard
>>>> - https://svn.open-mpi.org/trac/ompi/ticket/4900: Ralph
>>>> - https://svn.open-mpi.org/trac/ompi/ticket/4905: Mike Dubman
>>>> - https://svn.open-mpi.org/trac/ompi/ticket/4917: Jeff
>>>> - https://svn.open-mpi.org/trac/ompi/ticket/4911: Jeff
>>>> - https://svn.open-mpi.org/trac/ompi/ticket/4922: Jeff
>>>> - https://svn.open-mpi.org/trac/ompi/ticket/4923: Ralph
>>>> - https://svn.open-mpi.org/trac/ompi/ticket/4925: OMPI RM 1.8
>>>> - https://svn.open-mpi.org/trac/ompi/ticket/4926: Oscar
>>>> - https://svn.open-mpi.org/trac/ompi/ticket/4927: Jeff
>>> ___
>>> devel mailing list
>>> de...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>> Link to this post: 
>>> http://www.open-mpi.org/community/lists/devel/2014/10/15966.php
>>
>> -- 
>> Jeff Squyres
>> jsquy...@cisco.com
>> For corporate legal information go to: 
>> http://www.cisco.com/web/about/doing_business/legal/cri/
>>
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/devel/2014/10/15967.php
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2014/10/15971.php

Re: [OMPI devel] OMPI@GitHub: (Mostly) Open for business

[Gilles Gouaillardet]

Hi Jeff,

thumbs up for the migration !

the names here are the CMR owners ('Owned by' field in TRAC)
should it be the duty of the creators ('Reported by' field in TRAC) to
re-create the CMR instead?

/* if not, and from a git log point of view, that means the commiter
will be the reviewer and not the reporter */

what do you think ?

Cheers,

Gilles

On 2014/10/02 19:24, Jeff Squyres (jsquyres) wrote:
> 2. The following still-open CMRs were not brought over, and need to be 
> re-filed as pull requests:
>
> - https://svn.open-mpi.org/trac/ompi/ticket/4216: Nathan (BLOCKER)
> - https://svn.open-mpi.org/trac/ompi/ticket/4816: Ralph
> - https://svn.open-mpi.org/trac/ompi/ticket/4872: Ralph
> - https://svn.open-mpi.org/trac/ompi/ticket/4888: Howard
> - https://svn.open-mpi.org/trac/ompi/ticket/4900: Ralph
> - https://svn.open-mpi.org/trac/ompi/ticket/4905: Mike Dubman
> - https://svn.open-mpi.org/trac/ompi/ticket/4917: Jeff
> - https://svn.open-mpi.org/trac/ompi/ticket/4911: Jeff
> - https://svn.open-mpi.org/trac/ompi/ticket/4922: Jeff
> - https://svn.open-mpi.org/trac/ompi/ticket/4923: Ralph
> - https://svn.open-mpi.org/trac/ompi/ticket/4925: OMPI RM 1.8
> - https://svn.open-mpi.org/trac/ompi/ticket/4926: Oscar
> - https://svn.open-mpi.org/trac/ompi/ticket/4927: Jeff

Re: [OMPI devel] Neighbor collectives with periodic Cartesian topologies of size one

[Gilles Gouaillardet]

oops, i missed graph and dist graph :-(

the attached patch fixes this
/* it was straightforward for graph and a bit more challenging for dist
graph */

i am fine for Nathan's patch for v1.8
/* v1.8 and trunk have already a different way to handle a memory leak */
/* the attached patch is vs v1.8 but it is for review only, i will port
it to the trunk */

imho, there is no ABI break with this patch.
out of curiosity, what is the ABI break that is/will be introduced with
the "real" fix ?

FYI, i also enhanced the collective/neighbor_allgather_self from the ibm
test suite in order to test
graph and dist_graph

Cheers,

Gilles


On 2014/10/01 0:58, Jeff Squyres (jsquyres) wrote:
> On the call today, we decided that Nathan's v1.8 patch is sufficient for the 
> v1.8 series, and the "real" fix will be applied to the trunk and carried 
> forward to the v1.9 series (since it introduces an ABI break for the topo 
> framework, which we try not to do in the middle of a release series).
>
> On Sep 30, 2014, at 10:40 AM, Nathan Hjelm <hje...@lanl.gov> wrote:
>
>> Not quite right. There still is no topology information at collective
>> selection time for either graph or dist graph.
>>
>> -Nathan
>>
>> On Tue, Sep 30, 2014 at 02:03:27PM +0900, Gilles Gouaillardet wrote:
>>>   Nathan,
>>>
>>>   here is a revision of the previously attached patch, and that supports
>>>   graph and dist graph.
>>>
>>>   Cheers,
>>>
>>>   Gilles
>>>
>>>   On 2014/09/30 0:05, Nathan Hjelm wrote:
>>>
>>> An equivalent change would need to be made for graph and dist graph as
>>> well. That will take a little more work. Also, I was avoiding changing
>>> anything in topo for 1.8.
>>>
>>> -Nathan
>>>
>>> On Mon, Sep 29, 2014 at 08:02:41PM +0900, Gilles Gouaillardet wrote:
>>>
>>>Nathan,
>>>
>>>why not just make the topology information available at that point as you
>>>described it ?
>>>
>>>the attached patch does this, could you please review it ?
>>>
>>>Cheers,
>>>
>>>Gilles
>>>
>>>On 2014/09/26 2:50, Nathan Hjelm wrote:
>>>
>>>  On Tue, Aug 26, 2014 at 07:03:24PM +0300, Lisandro Dalcin wrote:
>>>
>>>  I finally managed to track down some issues in mpi4py's test suite
>>>  using Open MPI 1.8+. The code below should be enough to reproduce the
>>>  problem. Run it under valgrind to make sense of my following
>>>  diagnostics.
>>>
>>>  In this code I'm creating a 2D, periodic Cartesian topology out of
>>>  COMM_SELF. In this case, the process in COMM_SELF has 4 logical in/out
>>>  links to itself. So we have size=1 but indegree=outdegree=4. However,
>>>  in ompi/mca/coll/basic/coll_basic_module.c, "size * 2" request are
>>>  being allocated to manage communication:
>>>
>>>  if (OMPI_COMM_IS_INTER(comm)) {
>>>  size = ompi_comm_remote_size(comm);
>>>  } else {
>>>  size = ompi_comm_size(comm);
>>>  }
>>>  basic_module->mccb_num_reqs = size * 2;
>>>  basic_module->mccb_reqs = (ompi_request_t**)
>>>  malloc(sizeof(ompi_request_t *) * basic_module->mccb_num_reqs);
>>>
>>>  I guess you have to also special-case for topologies and allocate
>>>  indegree+outdegree requests (not sure about this number, just
>>>  guessing).
>>>
>>>
>>>  I wish this was possible but the topology information is not available
>>>  at that point. We may be able to change that but I don't see the work
>>>  completing anytime soon. I committed an alternative fix as r32796 and
>>>  CMR'd it to 1.8.3. I can confirm that the attached reproducer no longer
>>>  produces a SEGV. Let me know if you run into any more issues.
>>>
>>>
>>>  -Nathan
>>>
>>>  ___
>>>  devel mailing list
>>>  de...@open-mpi.org
>>>  Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>  Link to this post: 
>>> http://www.open-mpi.org/community/lists/devel/2014/09/15915.php
>>>
>>> Index: ompi/mca/topo/base/topo_base_cart_create.c
>>> ===
>>> --- ompi/mca/topo/base/topo_base_cart_create.c  (revision 32807)
>>> +++ ompi/mca/topo/base/topo_base_cart_create.c  (working copy)
>>> @@ -163,10 +163,18 @@
>>>

Re: [OMPI devel] MPI_Comm_spawn crashes with the openib btl

[Gilles Gouaillardet]

Thanks Ralph !

it did fix the problem

Cheers,

Gilles

On 2014/10/01 3:04, Ralph Castain wrote:
> I fixed this in r32818 - the components shouldn't be passing back success if 
> the requested info isn't found. Hope that fixes the problem.
>
>
> On Sep 30, 2014, at 1:54 AM, Gilles Gouaillardet 
> <gilles.gouaillar...@iferc.org> wrote:
>
>> Folks,
>>
>> the dynamic/spawn test from the ibm test suite crashes if the openib btl
>> is detected
>> (the test can be ran on one node with an IB port)
>>
>> here is what happens :
>>
>> in mca_btl_openib_proc_create,
>> the macro
>>OPAL_MODEX_RECV(rc, _btl_openib_component.super.btl_version,
>>proc, , _size);
>> does not find any information *but*
>> rc is OPAL_SUCCESS
>> msg_size is not updated (e.g. left uninitialized)
>> message is not updated (e.g. left uninitialized)
>>
>> then, if msg_size is unitialized with a non zero value, and if message
>> is uninitialized with
>> a non valid address, a crash will occur when accessing message.
>>
>> /* i am not debating here the fact that there is no information returned,
>> i am simply discussing the crash */
>>
>> a simple workaround is to initialize msg_size to zero.
>>
>> that being said, is this the correct fix ?
>>
>> one possible alternate fix is to update the OPAL_MODEX_RECV_STRING macro
>> like this :
>>
>> /* from opal/mca/pmix/pmix.h */
>> #define OPAL_MODEX_RECV_STRING(r, s, p, d, sz)  \
>>do {\
>>opal_value_t *kv;   \
>>if (OPAL_SUCCESS == ((r) = opal_pmix.get(&(p)->proc_name,   \
>> (s), ))) {  \
>>if (NULL != kv)
>> {   \
>>*(d) =
>> kv->data.bo.bytes;   \
>>*(sz) =
>> kv->data.bo.size;   \
>>kv->data.bo.bytes = NULL; /* protect the data
>> */\
>>
>> OBJ_RELEASE(kv);\
>>} else {\
>>*(sz) = 0;\
>>(r) = OPAL_ERR_NOT_FOUND;
>>} \
>>}   \
>>} while(0);
>>
>> /*
>> *(sz) = 0; and (r) = OPAL_ERR_NOT_FOUND; can be seen as redundant, *(sz)
>> *or* (r) could be set
>> */
>>
>> and an other alternate fix is to update the end of the native_get
>> function like this :
>>
>> /* from opal/mca/pmix/native/pmix_native.c */
>>
>>if (found) {
>>return OPAL_SUCCESS;
>>}
>>*kv = NULL;
>>if (OPAL_SUCCESS == rc) {
>>if (OPAL_SUCCESS == ret) {
>>rc = OPAL_ERR_NOT_FOUND;
>>} else {
>>rc = ret;
>>}
>>}
>>return rc;
>>
>> Could you please advise ?
>>
>> Cheers,
>>
>> Gilles
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/devel/2014/09/15942.php
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2014/09/15950.php

[OMPI devel] MPI_Comm_spawn crashes with the openib btl

[Gilles Gouaillardet]

Folks,

the dynamic/spawn test from the ibm test suite crashes if the openib btl
is detected
(the test can be ran on one node with an IB port)

here is what happens :

in mca_btl_openib_proc_create,
the macro
OPAL_MODEX_RECV(rc, _btl_openib_component.super.btl_version,
proc, , _size);
does not find any information *but*
rc is OPAL_SUCCESS
msg_size is not updated (e.g. left uninitialized)
message is not updated (e.g. left uninitialized)

then, if msg_size is unitialized with a non zero value, and if message
is uninitialized with
a non valid address, a crash will occur when accessing message.

/* i am not debating here the fact that there is no information returned,
i am simply discussing the crash */

a simple workaround is to initialize msg_size to zero.

that being said, is this the correct fix ?

one possible alternate fix is to update the OPAL_MODEX_RECV_STRING macro
like this :

/* from opal/mca/pmix/pmix.h */
#define OPAL_MODEX_RECV_STRING(r, s, p, d, sz)  \
do {\
opal_value_t *kv;   \
if (OPAL_SUCCESS == ((r) = opal_pmix.get(&(p)->proc_name,   \
 (s), ))) {  \
if (NULL != kv)
{   \
*(d) =
kv->data.bo.bytes;   \
*(sz) =
kv->data.bo.size;   \
kv->data.bo.bytes = NULL; /* protect the data
*/\

OBJ_RELEASE(kv);\
} else {\
*(sz) = 0;\
(r) = OPAL_ERR_NOT_FOUND;
} \
}   \
} while(0);

/*
*(sz) = 0; and (r) = OPAL_ERR_NOT_FOUND; can be seen as redundant, *(sz)
*or* (r) could be set
*/

and an other alternate fix is to update the end of the native_get
function like this :

/* from opal/mca/pmix/native/pmix_native.c */

if (found) {
return OPAL_SUCCESS;
}
*kv = NULL;
if (OPAL_SUCCESS == rc) {
if (OPAL_SUCCESS == ret) {
rc = OPAL_ERR_NOT_FOUND;
} else {
rc = ret;
}
}
return rc;

Could you please advise ?

Cheers,

Gilles

Re: [OMPI devel] Neighbor collectives with periodic Cartesian topologies of size one

[Gilles Gouaillardet]

Nathan,

here is a revision of the previously attached patch, and that supports
graph and dist graph.

Cheers,

Gilles

On 2014/09/30 0:05, Nathan Hjelm wrote:
> An equivalent change would need to be made for graph and dist graph as
> well. That will take a little more work. Also, I was avoiding changing
> anything in topo for 1.8.
>
> -Nathan
>
> On Mon, Sep 29, 2014 at 08:02:41PM +0900, Gilles Gouaillardet wrote:
>>Nathan,
>>
>>why not just make the topology information available at that point as you
>>described it ?
>>
>>the attached patch does this, could you please review it ?
>>
>>Cheers,
>>
>>Gilles
>>
>>On 2014/09/26 2:50, Nathan Hjelm wrote:
>>
>>  On Tue, Aug 26, 2014 at 07:03:24PM +0300, Lisandro Dalcin wrote:
>>
>>  I finally managed to track down some issues in mpi4py's test suite
>>  using Open MPI 1.8+. The code below should be enough to reproduce the
>>  problem. Run it under valgrind to make sense of my following
>>  diagnostics.
>>
>>  In this code I'm creating a 2D, periodic Cartesian topology out of
>>  COMM_SELF. In this case, the process in COMM_SELF has 4 logical in/out
>>  links to itself. So we have size=1 but indegree=outdegree=4. However,
>>  in ompi/mca/coll/basic/coll_basic_module.c, "size * 2" request are
>>  being allocated to manage communication:
>>
>>  if (OMPI_COMM_IS_INTER(comm)) {
>>  size = ompi_comm_remote_size(comm);
>>  } else {
>>  size = ompi_comm_size(comm);
>>  }
>>  basic_module->mccb_num_reqs = size * 2;
>>  basic_module->mccb_reqs = (ompi_request_t**)
>>  malloc(sizeof(ompi_request_t *) * basic_module->mccb_num_reqs);
>>
>>  I guess you have to also special-case for topologies and allocate
>>  indegree+outdegree requests (not sure about this number, just
>>  guessing).
>>
>>
>>  I wish this was possible but the topology information is not available
>>  at that point. We may be able to change that but I don't see the work
>>  completing anytime soon. I committed an alternative fix as r32796 and
>>  CMR'd it to 1.8.3. I can confirm that the attached reproducer no longer
>>  produces a SEGV. Let me know if you run into any more issues.
>>
>>
>>  -Nathan
>>
>>  ___
>>  devel mailing list
>>  de...@open-mpi.org
>>  Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>  Link to this post: 
>> http://www.open-mpi.org/community/lists/devel/2014/09/15915.php
>> Index: ompi/mca/topo/base/topo_base_cart_create.c
>> ===
>> --- ompi/mca/topo/base/topo_base_cart_create.c   (revision 32807)
>> +++ ompi/mca/topo/base/topo_base_cart_create.c   (working copy)
>> @@ -163,10 +163,18 @@
>>  return MPI_ERR_INTERN;
>>  }
>>  
>> +assert(NULL == new_comm->c_topo);
>> +assert(!(new_comm->c_flags & OMPI_COMM_CART));
>> +new_comm->c_topo   = topo;
>> +new_comm->c_topo->mtc.cart = cart;
>> +new_comm->c_topo->reorder  = reorder;
>> +new_comm->c_flags |= OMPI_COMM_CART;
>>  ret = ompi_comm_enable(old_comm, new_comm,
>> new_rank, num_procs, topo_procs);
>>  if (OMPI_SUCCESS != ret) {
>>  /* something wrong happened during setting the communicator */
>> +new_comm->c_topo = NULL;
>> +new_comm->c_flags &= ~OMPI_COMM_CART;
>>  ompi_comm_free (_comm);
>>  free(topo_procs);
>>  if(NULL != cart->periods) free(cart->periods);
>> @@ -176,10 +184,6 @@
>>  return ret;
>>  }
>>  
>> -new_comm->c_topo   = topo;
>> -new_comm->c_topo->mtc.cart = cart;
>> -new_comm->c_topo->reorder  = reorder;
>> -new_comm->c_flags |= OMPI_COMM_CART;
>>  *comm_topo = new_comm;
>>  
>>  if( MPI_UNDEFINED == new_rank ) {
>> Index: ompi/mca/coll/basic/coll_basic_module.c
>> ===
>> --- ompi/mca/coll/basic/coll_basic_module.c  (revision 32807)
>> +++ ompi/mca/coll/basic/coll_basic_module.c  (working copy)
>> @@ -13,6 +13,8 @@
>>   * Copyright (c) 2012  Sandia National Laboratories. All rights 
>> reserved.
>>   * Copyright (c) 2013  Los Alamos National Security, LLC. All rights
>>   *  

Re: [OMPI devel] Neighbor collectives with periodic Cartesian topologies of size one

[Gilles Gouaillardet]

Nathan,

why not just make the topology information available at that point as
you described it ?

the attached patch does this, could you please review it ?

Cheers,

Gilles

On 2014/09/26 2:50, Nathan Hjelm wrote:
> On Tue, Aug 26, 2014 at 07:03:24PM +0300, Lisandro Dalcin wrote:
>> I finally managed to track down some issues in mpi4py's test suite
>> using Open MPI 1.8+. The code below should be enough to reproduce the
>> problem. Run it under valgrind to make sense of my following
>> diagnostics.
>>
>> In this code I'm creating a 2D, periodic Cartesian topology out of
>> COMM_SELF. In this case, the process in COMM_SELF has 4 logical in/out
>> links to itself. So we have size=1 but indegree=outdegree=4. However,
>> in ompi/mca/coll/basic/coll_basic_module.c, "size * 2" request are
>> being allocated to manage communication:
>>
>> if (OMPI_COMM_IS_INTER(comm)) {
>> size = ompi_comm_remote_size(comm);
>> } else {
>> size = ompi_comm_size(comm);
>> }
>> basic_module->mccb_num_reqs = size * 2;
>> basic_module->mccb_reqs = (ompi_request_t**)
>> malloc(sizeof(ompi_request_t *) * basic_module->mccb_num_reqs);
>>
>> I guess you have to also special-case for topologies and allocate
>> indegree+outdegree requests (not sure about this number, just
>> guessing).
>>
> I wish this was possible but the topology information is not available
> at that point. We may be able to change that but I don't see the work
> completing anytime soon. I committed an alternative fix as r32796 and
> CMR'd it to 1.8.3. I can confirm that the attached reproducer no longer
> produces a SEGV. Let me know if you run into any more issues.
>
>
> -Nathan
>
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2014/09/15915.php

Index: ompi/mca/topo/base/topo_base_cart_create.c
===
--- ompi/mca/topo/base/topo_base_cart_create.c  (revision 32807)
+++ ompi/mca/topo/base/topo_base_cart_create.c  (working copy)
@@ -163,10 +163,18 @@
 return MPI_ERR_INTERN;
 }

+assert(NULL == new_comm->c_topo);
+assert(!(new_comm->c_flags & OMPI_COMM_CART));
+new_comm->c_topo   = topo;
+new_comm->c_topo->mtc.cart = cart;
+new_comm->c_topo->reorder  = reorder;
+new_comm->c_flags |= OMPI_COMM_CART;
 ret = ompi_comm_enable(old_comm, new_comm,
new_rank, num_procs, topo_procs);
 if (OMPI_SUCCESS != ret) {
 /* something wrong happened during setting the communicator */
+new_comm->c_topo = NULL;
+new_comm->c_flags &= ~OMPI_COMM_CART;
 ompi_comm_free (_comm);
 free(topo_procs);
 if(NULL != cart->periods) free(cart->periods);
@@ -176,10 +184,6 @@
 return ret;
 }

-new_comm->c_topo   = topo;
-new_comm->c_topo->mtc.cart = cart;
-new_comm->c_topo->reorder  = reorder;
-new_comm->c_flags |= OMPI_COMM_CART;
 *comm_topo = new_comm;

 if( MPI_UNDEFINED == new_rank ) {
Index: ompi/mca/coll/basic/coll_basic_module.c
===
--- ompi/mca/coll/basic/coll_basic_module.c (revision 32807)
+++ ompi/mca/coll/basic/coll_basic_module.c (working copy)
@@ -13,6 +13,8 @@
  * Copyright (c) 2012  Sandia National Laboratories. All rights reserved.
  * Copyright (c) 2013  Los Alamos National Security, LLC. All rights
  * reserved.
+ * Copyright (c) 2014  Research Organization for Information Science
+ * and Technology (RIST). All rights reserved.
  * $COPYRIGHT$
  * 
  * Additional copyrights may follow
@@ -28,6 +30,7 @@
 #include "mpi.h"
 #include "ompi/mca/coll/coll.h"
 #include "ompi/mca/coll/base/base.h"
+#include "ompi/mca/topo/topo.h"
 #include "coll_basic.h"


@@ -70,6 +73,15 @@
 } else {
 size = ompi_comm_size(comm);
 }
+if (comm->c_flags & OMPI_COMM_CART) {
+int cart_size;
+assert (NULL != comm->c_topo);
+comm->c_topo->topo.cart.cartdim_get(comm, _size);
+cart_size *= 2;
+if (cart_size > size) {
+size = cart_size;
+}
+}
 basic_module->mccb_num_reqs = size * 2;
 basic_module->mccb_reqs = (ompi_request_t**) 
 malloc(sizeof(ompi_request_t *) * basic_module->mccb_num_reqs);