[OMPI devel] Continued warnings?

[Ralph H Castain]

Just curious - will this ever be fixed? From today’s head of master:

In file included from info.c:46:0:
info.c: In function 'opal_info_dup_mode':
../../opal/util/info.h:112:31: warning: '%s' directive output may be truncated 
writing up to 36 bytes into a region of size 27 [-Wformat-truncation=]
 #define OPAL_INFO_SAVE_PREFIX "_OMPI_IN_"
   ^
info.c:212:22: note: in expansion of macro 'OPAL_INFO_SAVE_PREFIX'
  OPAL_INFO_SAVE_PREFIX "%s", iterator->ie_key);
  ^
info.c:212:45: note: format string is defined here
  OPAL_INFO_SAVE_PREFIX "%s", iterator->ie_key);
 ^~
info.c:211:18: note: 'snprintf' output between 10 and 46 bytes into a 
destination of size 36
  snprintf(savedkey, OPAL_MAX_INFO_KEY,
  ^
  OPAL_INFO_SAVE_PREFIX "%s", iterator->ie_key);
  ~


___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

Re: [OMPI devel] Open MPI website borked up?

[Ralph H Castain]

I suspect this is a stale message - I’m not seeing any problem with the website


> On Aug 29, 2018, at 12:55 PM, Howard Pritchard  wrote:
> 
> Hi Folks,
> 
> Something seems to be borked up about the OMPI website.  Got to website and 
> you'll
> get some odd parsing error appearing.
> 
> Howard
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

[OMPI devel] Will info keys ever be fixed?

[Ralph H Castain]

Still seeing this in today’s head of master:

info_subscriber.c: In function 'opal_infosubscribe_change_info':
../../opal/util/info.h:112:31: warning: '%s' directive output may be truncated 
writing up to 36 bytes into a region of size 27 [-Wformat-truncation=]
 #define OPAL_INFO_SAVE_PREFIX "_OMPI_IN_"
   ^
info_subscriber.c:268:13: note: in expansion of macro 'OPAL_INFO_SAVE_PREFIX'
 OPAL_INFO_SAVE_PREFIX "%s", key);
 ^
info_subscriber.c:268:36: note: format string is defined here
 OPAL_INFO_SAVE_PREFIX "%s", key);
^~
In file included from 
/opt/local/lib/gcc7/gcc/x86_64-apple-darwin17/7.3.0/include-fixed/stdio.h:425:0,
 from ../../opal/class/opal_list.h:71,
 from ../../opal/util/info_subscriber.h:30,
 from info_subscriber.c:45:
info_subscriber.c:267:9: note: '__builtin_snprintf' output between 10 and 46 
bytes into a destination of size 36
 snprintf(modkey, OPAL_MAX_INFO_KEY,
 ^
In file included from 
/opt/local/lib/gcc7/gcc/x86_64-apple-darwin17/7.3.0/include-fixed/stdio.h:425:0,
 from ../../opal/class/opal_list.h:71,
 from ../../opal/util/info.h:30,
 from info.c:46:
info.c: In function 'opal_info_dup_mode.constprop':
../../opal/util/info.h:112:31: warning: '%s' directive output may be truncated 
writing up to 36 bytes into a region of size 28 [-Wformat-truncation=]
 #define OPAL_INFO_SAVE_PREFIX "_OMPI_IN_"
   ^
info.c:212:22: note: in expansion of macro 'OPAL_INFO_SAVE_PREFIX'
  OPAL_INFO_SAVE_PREFIX "%s", pkey);
  ^
info.c:212:45: note: format string is defined here
  OPAL_INFO_SAVE_PREFIX "%s", pkey);
 ^~
In file included from 
/opt/local/lib/gcc7/gcc/x86_64-apple-darwin17/7.3.0/include-fixed/stdio.h:425:0,
 from ../../opal/class/opal_list.h:71,
 from ../../opal/util/info.h:30,
 from info.c:46:
info.c:211:18: note: '__builtin_snprintf' output between 10 and 46 bytes into a 
destination of size 37
  snprintf(savedkey, OPAL_MAX_INFO_KEY+1,
  ^


___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

Re: [OMPI devel] mpirun error when not using span

[Ralph H Castain]

Could you please send the output from “lstopo --of xml foo.xml” (the file 
foo.xml) so I can try to replicate here?


> On Sep 4, 2018, at 12:35 PM, Shrader, David Lee  wrote:
> 
> Hello,
> 
> I have run this issue by Howard, and he asked me to forward it on to the Open 
> MPI devel mailing list. I get an error when trying to use PE=n with '--map-by 
> numa' and not using span when using more than one node:
> 
> [dshrader@ba001 openmpi-3.1.2]$ mpirun -n 16 --map-by numa:PE=4 --bind-to 
> core --report-bindings true
> --
> A request was made to bind to that would result in binding more
> processes than cpus on a resource:
> 
>Bind to: CORE
>Node:ba001
>#processes:  2
>#cpus:   1
> 
> You can override this protection by adding the "overload-allowed"
> option to your binding directive.
> --
> 
> The absolute values of the numbers passed to -n and PE don't really matter; 
> the error pops up as soon as those numbers are combined in such a way that an 
> MPI rank ends up on the second node.
> 
> If I add the "span" parameter, everything works as expected:
> 
> [dshrader@ba001 openmpi-3.1.2]$ mpirun -n 16 --map-by numa:PE=4,span 
> --bind-to core --report-bindings true
> [ba002.localdomain:58502] MCW rank 8 bound to socket 0[core 0[hwt 0]], socket 
> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], socket 0[core 3[hwt 0]]: 
> [B/B/B/B/./././././././././././././.][./././././././././././././././././.]
> [ba002.localdomain:58502] MCW rank 9 bound to socket 0[core 4[hwt 0]], socket 
> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], socket 0[core 7[hwt 0]]: 
> [././././B/B/B/B/./././././././././.][./././././././././././././././././.]
> [ba002.localdomain:58502] MCW rank 10 bound to socket 0[core 8[hwt 0]], 
> socket 0[core 9[hwt 0]], socket 0[core 10[hwt 0]], socket 0[core 11[hwt 0]]: 
> [././././././././B/B/B/B/./././././.][./././././././././././././././././.]
> [ba002.localdomain:58502] MCW rank 11 bound to socket 0[core 12[hwt 0]], 
> socket 0[core 13[hwt 0]], socket 0[core 14[hwt 0]], socket 0[core 15[hwt 0]]: 
> [././././././././././././B/B/B/B/./.][./././././././././././././././././.]
> [ba002.localdomain:58502] MCW rank 12 bound to socket 1[core 18[hwt 0]], 
> socket 1[core 19[hwt 0]], socket 1[core 20[hwt 0]], socket 1[core 21[hwt 0]]: 
> [./././././././././././././././././.][B/B/B/B/./././././././././././././.]
> [ba002.localdomain:58502] MCW rank 13 bound to socket 1[core 22[hwt 0]], 
> socket 1[core 23[hwt 0]], socket 1[core 24[hwt 0]], socket 1[core 25[hwt 0]]: 
> [./././././././././././././././././.][././././B/B/B/B/./././././././././.]
> [ba002.localdomain:58502] MCW rank 14 bound to socket 1[core 26[hwt 0]], 
> socket 1[core 27[hwt 0]], socket 1[core 28[hwt 0]], socket 1[core 29[hwt 0]]: 
> [./././././././././././././././././.][././././././././B/B/B/B/./././././.]
> [ba002.localdomain:58502] MCW rank 15 bound to socket 1[core 30[hwt 0]], 
> socket 1[core 31[hwt 0]], socket 1[core 32[hwt 0]], socket 1[core 33[hwt 0]]: 
> [./././././././././././././././././.][././././././././././././B/B/B/B/./.]
> [ba001.localdomain:11700] MCW rank 0 bound to socket 0[core 0[hwt 0]], socket 
> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], socket 0[core 3[hwt 0]]: 
> [B/B/B/B/./././././././././././././.][./././././././././././././././././.]
> [ba001.localdomain:11700] MCW rank 1 bound to socket 0[core 4[hwt 0]], socket 
> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], socket 0[core 7[hwt 0]]: 
> [././././B/B/B/B/./././././././././.][./././././././././././././././././.]
> [ba001.localdomain:11700] MCW rank 2 bound to socket 0[core 8[hwt 0]], socket 
> 0[core 9[hwt 0]], socket 0[core 10[hwt 0]], socket 0[core 11[hwt 0]]: 
> [././././././././B/B/B/B/./././././.][./././././././././././././././././.]
> [ba001.localdomain:11700] MCW rank 3 bound to socket 0[core 12[hwt 0]], 
> socket 0[core 13[hwt 0]], socket 0[core 14[hwt 0]], socket 0[core 15[hwt 0]]: 
> [././././././././././././B/B/B/B/./.][./././././././././././././././././.]
> [ba001.localdomain:11700] MCW rank 4 bound to socket 1[core 18[hwt 0]], 
> socket 1[core 19[hwt 0]], socket 1[core 20[hwt 0]], socket 1[core 21[hwt 0]]: 
> [./././././././././././././././././.][B/B/B/B/./././././././././././././.]
> [ba001.localdomain:11700] MCW rank 5 bound to socket 1[core 22[hwt 0]], 
> socket 1[core 23[hwt 0]], socket 1[core 24[hwt 0]], socket 1[core 25[hwt 0]]: 
> [./././././././././././././././././.][././././B/B/B/B/./././././././././.]
> [ba001.localdomain:11700] MCW rank 6 bound to socket 1[core 26[hwt 0]], 
> socket 1[core 27[hwt 0]], socket 1[core 28[hwt 0]], socket 1[core 29[hwt 0]]: 
> [./././././././././././././././././.][././././././././B/B/B/B/./././././.]
> [ba001.localdomain:11700] MCW rank 7 bound to socket 1[core 30[hwt 0]], 
> socket 1[core 31[hwt 0]], socket 1[core 32[hwt 0]], socket 1[core 33[hwt 0]]: 
> 

Re: [OMPI devel] Will info keys ever be fixed?

[Ralph H Castain]

On MacOS with gcc 7.3


> On Sep 11, 2018, at 3:02 PM, Jeff Squyres (jsquyres) via devel 
>  wrote:
> 
> Ralph --
> 
> What OS / compiler are you using?
> 
> I just compiled on MacOS (first time in a while) and filed a PR and a few 
> issues about the warnings I found, but I cannot replicate these warnings.  I 
> also built with gcc 7.3.0 on RHEL; couldn't replicate the warnings.
> 
> On MacOS, I'm using the default Xcode compilers:
> 
> $ gcc --version
> Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr 
> --with-gxx-include-dir=/usr/include/c++/4.2.1
> Apple LLVM version 9.1.0 (clang-902.0.39.2)
> Target: x86_64-apple-darwin17.7.0
> Thread model: posix
> InstalledDir: 
> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
> 
> 
> 
> 
> 
>> On Sep 10, 2018, at 6:57 PM, Ralph H Castain  wrote:
>> 
>> Still seeing this in today’s head of master:
>> 
>> info_subscriber.c: In function 'opal_infosubscribe_change_info':
>> ../../opal/util/info.h:112:31: warning: '%s' directive output may be 
>> truncated writing up to 36 bytes into a region of size 27 
>> [-Wformat-truncation=]
>> #define OPAL_INFO_SAVE_PREFIX "_OMPI_IN_"
>>   ^
>> info_subscriber.c:268:13: note: in expansion of macro 'OPAL_INFO_SAVE_PREFIX'
>> OPAL_INFO_SAVE_PREFIX "%s", key);
>> ^
>> info_subscriber.c:268:36: note: format string is defined here
>> OPAL_INFO_SAVE_PREFIX "%s", key);
>>^~
>> In file included from 
>> /opt/local/lib/gcc7/gcc/x86_64-apple-darwin17/7.3.0/include-fixed/stdio.h:425:0,
>> from ../../opal/class/opal_list.h:71,
>> from ../../opal/util/info_subscriber.h:30,
>> from info_subscriber.c:45:
>> info_subscriber.c:267:9: note: '__builtin_snprintf' output between 10 and 46 
>> bytes into a destination of size 36
>> snprintf(modkey, OPAL_MAX_INFO_KEY,
>> ^
>> In file included from 
>> /opt/local/lib/gcc7/gcc/x86_64-apple-darwin17/7.3.0/include-fixed/stdio.h:425:0,
>> from ../../opal/class/opal_list.h:71,
>> from ../../opal/util/info.h:30,
>> from info.c:46:
>> info.c: In function 'opal_info_dup_mode.constprop':
>> ../../opal/util/info.h:112:31: warning: '%s' directive output may be 
>> truncated writing up to 36 bytes into a region of size 28 
>> [-Wformat-truncation=]
>> #define OPAL_INFO_SAVE_PREFIX "_OMPI_IN_"
>>   ^
>> info.c:212:22: note: in expansion of macro 'OPAL_INFO_SAVE_PREFIX'
>>  OPAL_INFO_SAVE_PREFIX "%s", pkey);
>>  ^
>> info.c:212:45: note: format string is defined here
>>  OPAL_INFO_SAVE_PREFIX "%s", pkey);
>> ^~
>> In file included from 
>> /opt/local/lib/gcc7/gcc/x86_64-apple-darwin17/7.3.0/include-fixed/stdio.h:425:0,
>> from ../../opal/class/opal_list.h:71,
>> from ../../opal/util/info.h:30,
>> from info.c:46:
>> info.c:211:18: note: '__builtin_snprintf' output between 10 and 46 bytes 
>> into a destination of size 37
>>  snprintf(savedkey, OPAL_MAX_INFO_KEY+1,
>>  ^
>> 
>> 
>> ___
>> devel mailing list
>> devel@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/devel
> 
> 
> -- 
> Jeff Squyres
> jsquy...@cisco.com
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

Re: [OMPI devel] Hints for using an own pmix server

[Ralph H Castain]

I took a look at the following:

>> A remark to pmix at this point: pmix_bfrops_base_value_load() does
>> silently not handle PMIX_DATA_ARRAY type leading to not working makros
>> PMIX_VALUE_LOAD and PMIX_INFO_LOAD with that type. I think this is
>> unlucky and took me a while to figure out why it comes to a segfault
>> when pmix tried to process my PMIX_PROC_DATA infos.

It appears that this was true in the v2.x release series, but has since been 
fixed - thus, the v3.x series is okay. I’ll backport the support to the v2.x 
for their next releases.

Thanks for point it out!
Ralph

> On Oct 12, 2018, at 6:15 AM, Ralph H Castain  wrote:
> 
> Hi Stephan
> 
> 
>> On Oct 12, 2018, at 2:25 AM, Stephan Krempel > <mailto:krem...@par-tec.com>> wrote:
>> 
>> Hallo Ralph,
>> 
>>> I assume this (--with-ompi-mpix-rte) is a typo as the correct option
>>> is —with-ompi-pmix-rte?
>> 
>> You were right, this was a typo, with the correct option I now managed
>> to start an MPI helloworld program using OpenMPI and our own process
>> manager with pmix server.
> 
> Hooray! If you want me to show support for your PM on our web site, please 
> send me a little info about it. You are welcome to send it off-list if you 
> prefer.
> 
>> 
>>> It all looks okay to me for the client, but I wonder if you
>>> remembered to call register_nspace and register_client on your server
>>> prior to starting the client? If not, the connection will be dropped
>>> - you could add PMIX_MCA_ptl_base_verbose=100 to your environment to
>>> see the detailed connection handshake.
>> 
>> This has been a point that I could finally figure out from the prrte
>> code. To make it working you do not only need to call register_nspace
>> but also pass some specific information to it that OpenMPI considers to
>> be available (e.g. proc info with lrank).
> 
> My apologies - we will document this better on the PMIx web site and provide 
> some link to it on the OMPI web site. We actually do publish the info OMPI is 
> expecting, but it isn’t in an obvious enough place.
> 
>> 
>> A remark to pmix at this point: pmix_bfrops_base_value_load() does
>> silently not handle PMIX_DATA_ARRAY type leading to not working makros
>> PMIX_VALUE_LOAD and PMIX_INFO_LOAD with that type. I think this is
>> unlucky and took me a while to figure out why it comes to a segfault
>> when pmix tried to process my PMIX_PROC_DATA infos.
> 
> I’ll check that out - I don’t know why we wouldn’t handle it, so it is likely 
> just an oversight. Regardless, it should return an error if it isn’t doing it.
> 
>> 
>> So thank you again for your help so far.
>> 
>> 
>> One point that remains open and is interesting for me is if I can
>> achieve the same with the 3.1.2 release of OpenMPI. Is it somehow
>> possible to configure it as there were the "--with-ompi-pmix-rte"
>> switch from version 4.x?
> 
> I’m afraid we didn’t backport that capability to the v3.x branches. I’ll ask 
> the relevant release managers if they’d like us to do so.
> 
> Ralph
> 
>> 
>> Regards,
>> 
>> Stephan
>> 
>> 
>>> 
>>>> On Oct 9, 2018, at 3:14 PM, Stephan Krempel >>> <mailto:krem...@par-tec.com>>
>>>> wrote:
>>>> 
>>>> Hi Ralf,
>>>> 
>>>> After studying prrte a little bit, I tried something new and
>>>> followed
>>>> the description here using openmpi 4:
>>>> https://pmix.org/code/building-the-pmix-reference-server/ 
>>>> <https://pmix.org/code/building-the-pmix-reference-server/>
>>>> 
>>>> I configured openmpi 4.0.0rc3:
>>>> 
>>>> ../configure --enable-debug --prefix [...] --with-pmix=[...] \
>>>>  --with-libevent=/usr --with-ompi-mpix-rte
>>>> 
>>>> (I also tried to set --with-orte=no, but it then claims not to have
>>>> a
>>>> suitable rte and does not finish)
>>>> 
>>>> I then started my own PMIx and spawned a client compiled with mpicc
>>>> of
>>>> the new openmpi installation with this environment:
>>>> 
>>>> PMIX_NAMESPACE=namespace_3228_0_0
>>>> PMIX_RANK=0
>>>> PMIX_SERVER_URI2=pmix-server.3234;tcp4://127.0.0.1:49637
>>>> PMIX_SERVER_URI21=pmix-server.3234;tcp4://127.0.0.1:49637
>>>> PMIX_SERVER_URI=pmix-server:3234:/tmp/pmix-3234
>>>> PMIX_SERVER_URI2USOCK=pmix-server:3234:/tmp/pmix-3234
>>>> PMIX_SECURITY_MODE=native,none
&

Re: [OMPI devel] Hints for using an own pmix server

[Ralph H Castain]

> On Oct 12, 2018, at 6:15 AM, Ralph H Castain  wrote:
> 
>> One point that remains open and is interesting for me is if I can
>> achieve the same with the 3.1.2 release of OpenMPI. Is it somehow
>> possible to configure it as there were the "--with-ompi-pmix-rte"
>> switch from version 4.x?
> 
> I’m afraid we didn’t backport that capability to the v3.x branches. I’ll ask 
> the relevant release managers if they’d like us to do so.

I checked and we will not be backporting this to the v3.x series. It will begin 
with v4.x.

Ralph

___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

[OMPI devel] SC'18 PMIx BoF meeting

[Ralph H Castain]

Hello all

[I’m sharing this on the OMPI mailing lists (as well as the PMIx one) as PMIx 
has become tightly integrated to the OMPI code since v2.0 was released]

The PMIx Community will once again be hosting a Birds-of-a-Feather meeting at 
SuperComputing. This year, however, will be a little different! PMIx has come a 
long, long way over the last four years, and we are starting to see 
application-level adoption of the various APIs. Accordingly, we will be 
devoting most of this year’s meeting to a tutorial-like review of several 
use-cases, including:

* fault-tolerant OpenSHMEM implementation
* interlibrary resource coordination using OpenMP and MPI
* population modeling and swarm intelligence models running natively in an HPC 
environment
* use of the PMIx_Query interface

The meeting has been shifted to Wed night, 5:15-6:45pm, in room C144. Please 
share this with others who you feel might be interested, and do plan to attend!
Ralph

___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

Re: [OMPI devel] Hints for using an own pmix server

[Ralph H Castain]

> On Oct 17, 2018, at 3:32 AM, Stephan Krempel  wrote:
> 
> 
> Hi Ralph.
> 
 One point that remains open and is interesting for me is if I can
 achieve the same with the 3.1.2 release of OpenMPI. Is it somehow
 possible to configure it as there were the "--with-ompi-pmix-rte"
 switch from version 4.x?
>>> 
>>> I’m afraid we didn’t backport that capability to the v3.x branches.
>>> I’ll ask the relevant release managers if they’d like us to do so.
>> 
>> I checked and we will not be backporting this to the v3.x series. It
>> will begin with v4.x.
> 
> Thanks for checking out. I need to check with our users if supporting
> OpenMPI 4 will be sufficient for them, else for sure I will come back
> soon with some more questions regarding how to manage supporting
> OpenMPI 3.

If it becomes an issue, I can probably provide a patch for OMPI v3 that you 
could locally install

> 
> Thank you again for the assistance.
> 
> Best regards
> 
> Stephan
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

Re: [OMPI devel] Hints for using an own pmix server

[Ralph H Castain]

Hi Stephan


> On Oct 12, 2018, at 2:25 AM, Stephan Krempel  wrote:
> 
> Hallo Ralph,
> 
>> I assume this (--with-ompi-mpix-rte) is a typo as the correct option
>> is —with-ompi-pmix-rte?
> 
> You were right, this was a typo, with the correct option I now managed
> to start an MPI helloworld program using OpenMPI and our own process
> manager with pmix server.

Hooray! If you want me to show support for your PM on our web site, please send 
me a little info about it. You are welcome to send it off-list if you prefer.

> 
>> It all looks okay to me for the client, but I wonder if you
>> remembered to call register_nspace and register_client on your server
>> prior to starting the client? If not, the connection will be dropped
>> - you could add PMIX_MCA_ptl_base_verbose=100 to your environment to
>> see the detailed connection handshake.
> 
> This has been a point that I could finally figure out from the prrte
> code. To make it working you do not only need to call register_nspace
> but also pass some specific information to it that OpenMPI considers to
> be available (e.g. proc info with lrank).

My apologies - we will document this better on the PMIx web site and provide 
some link to it on the OMPI web site. We actually do publish the info OMPI is 
expecting, but it isn’t in an obvious enough place.

> 
> A remark to pmix at this point: pmix_bfrops_base_value_load() does
> silently not handle PMIX_DATA_ARRAY type leading to not working makros
> PMIX_VALUE_LOAD and PMIX_INFO_LOAD with that type. I think this is
> unlucky and took me a while to figure out why it comes to a segfault
> when pmix tried to process my PMIX_PROC_DATA infos.

I’ll check that out - I don’t know why we wouldn’t handle it, so it is likely 
just an oversight. Regardless, it should return an error if it isn’t doing it.

> 
> So thank you again for your help so far.
> 
> 
> One point that remains open and is interesting for me is if I can
> achieve the same with the 3.1.2 release of OpenMPI. Is it somehow
> possible to configure it as there were the "--with-ompi-pmix-rte"
> switch from version 4.x?

I’m afraid we didn’t backport that capability to the v3.x branches. I’ll ask 
the relevant release managers if they’d like us to do so.

Ralph

> 
> Regards,
> 
> Stephan
> 
> 
>> 
>>> On Oct 9, 2018, at 3:14 PM, Stephan Krempel 
>>> wrote:
>>> 
>>> Hi Ralf,
>>> 
>>> After studying prrte a little bit, I tried something new and
>>> followed
>>> the description here using openmpi 4:
>>> https://pmix.org/code/building-the-pmix-reference-server/
>>> 
>>> I configured openmpi 4.0.0rc3:
>>> 
>>> ../configure --enable-debug --prefix [...] --with-pmix=[...] \
>>>  --with-libevent=/usr --with-ompi-mpix-rte
>>> 
>>> (I also tried to set --with-orte=no, but it then claims not to have
>>> a
>>> suitable rte and does not finish)
>>> 
>>> I then started my own PMIx and spawned a client compiled with mpicc
>>> of
>>> the new openmpi installation with this environment:
>>> 
>>> PMIX_NAMESPACE=namespace_3228_0_0
>>> PMIX_RANK=0
>>> PMIX_SERVER_URI2=pmix-server.3234;tcp4://127.0.0.1:49637
>>> PMIX_SERVER_URI21=pmix-server.3234;tcp4://127.0.0.1:49637
>>> PMIX_SERVER_URI=pmix-server:3234:/tmp/pmix-3234
>>> PMIX_SERVER_URI2USOCK=pmix-server:3234:/tmp/pmix-3234
>>> PMIX_SECURITY_MODE=native,none
>>> PMIX_PTL_MODULE=tcp,usock
>>> PMIX_BFROP_BUFFER_TYPE=PMIX_BFROP_BUFFER_FULLY_DESC
>>> PMIX_GDS_MODULE=ds12,hash
>>> PMIX_DSTORE_ESH_BASE_PATH=/tmp/pmix_dstor_3234
>>> 
>>> The client is not connecting to my pmix server and it's environment
>>> after MPI_Init looks like that:
>>> 
>>> PMIX_SERVER_URI2USOCK=pmix-server:3234:/tmp/pmix-3234
>>> PMIX_RANK=0
>>> PMIX_PTL_MODULE=tcp,usock
>>> PMIX_SERVER_URI=pmix-server:3234:/tmp/pmix-3234
>>> PMIX_MCA_mca_base_component_show_load_errors=1
>>> PMIX_BFROP_BUFFER_TYPE=PMIX_BFROP_BUFFER_FULLY_DESC
>>> PMIX_DSTORE_ESH_BASE_PATH=/tmp/ompi.landroval.1001/pid.3243/pmix_ds
>>> tor_
>>> 3243
>>> PMIX_SERVER_URI2=864157696.0;tcp4://127.0.0.1:33619
>>> PMIX_SERVER_URI21=864157696.0;tcp4://127.0.0.1:33619
>>> PMIX_SECURITY_MODE=native,none
>>> PMIX_NAMESPACE=864157697
>>> PMIX_GDS_MODULE=ds12,hash
>>> ORTE_SCHIZO_DETECTION=ORTE
>>> OMPI_COMMAND=./hello_env
>>> OMPI_MCA_orte_precondition_transports=f28d6577f6b6ac08-
>>> d92c0e73869e1cfa
>>> OMPI_MCA_orte_launch=1
>>&

Re: [OMPI devel] Hints for using an own pmix server

[Ralph H Castain]

Even PRRTE won’t allow you to stop the orted from initializing its PMIx server. 
I’m not sure I really understand your objective. Remember, PMIx is just a 
library - the orted opens it and uses it to interface to its client application 
procs. It makes no sense to have some other process perform that role as it 
won’t know any job-level information.

It sounds like what you really want to do is replace the orted, and have your 
orted open your PMIx server? In other words, you want to use the PMIx reference 
library to handle all the PMIx stuff, and provide your own backend functions to 
support the PMIx server calls? If so, then your best bet would be to edit the 
PRRTE code in orte/orted/pmix and replace it with your code. You’ll have to 
deal with the ORTE data objects and PRRTE’s launch procedure, but that is 
likely easier than trying to write your own version of “orted” from scratch.

As for Slurm: it behaves the same way as PRRTE. It has a plugin that implements 
the server backend functions, and the Slurm daemons “host” the plugin. What you 
would need to do is replace that plugin with your own.
Ralph


> On Oct 8, 2018, at 5:36 PM, Gilles Gouaillardet  wrote:
> 
> Stephan,
> 
> 
> Have you already checked https://github.com/pmix/prrte ?
> 
> 
> This is the PMIx Reference RunTime Environment (PPRTE), which was built on 
> top of orted.
> 
> Long story short, it deploys the PMIx server and then you start your MPI app 
> with prun
> An example is available at 
> https://github.com/pmix/prrte/blob/master/contrib/travis/test_client.sh
> 
> 
> Cheers,
> 
> Gilles
> 
> 
> On 10/9/2018 8:45 AM, Stephan Krempel wrote:
>> Hallo everyone,
>> 
>> I am currently implementing a PMIx server and I try to use it with
>> OpenMPI. I do have an own mpiexec which starts my PMIx server and
>> launches the processes.
>> 
>> If I launch an executable linked against OpenMPI, during MPI_Init() the
>> ORTE layer starts another PMIx server and overrides my PMIX_*
>> environment so this new server is used instead of mine.
>> 
>> So I am looking for a method to prevent orte(d) from starting a PMIx
>> server.
>> 
>> I already tried to understand what the slurm support is doing, since
>> this is (at least in parts) what I think I need. Somehow when starting
>> a job with srun --mpi=pmix_v2 the ess module pmi is started, but I was
>> not able to enforce that manually by setting an MCA parameter (oss
>> should be the correct one?!?)
>> And I do not yet have a clue how the slurm support is working.
>> 
>> So does anyone has a hint for me where I can find documentation or
>> information concerning that or is there an easy way to achieve what I
>> am trying to do that I missed?
>> 
>> Thank you in advance.
>> 
>> Regards,
>> 
>> Stephan
>> ___
>> devel mailing list
>> devel@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/devel
>> 
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

Re: [OMPI devel] Hints for using an own pmix server

[Ralph H Castain]

Hi Stephan

Thanks for the clarification - that helps a great deal. You are correct that 
OMPI’s orted daemons do more than just host the PMIx server library. However, 
they are only active if you launch the OMPI processes using mpirun. This is 
probably the source of the trouble you are seeing.

Since you have a process launcher and have integrated the PMIx server support 
into your RM’s daemons, you really have no need for mpirun at all. You should 
just be able to launch the processes directly using your own launcher. The PMIx 
support will take care of the startup requirements. The application procs will 
not use the orted in such cases.

So if your system is working fine with the PMIx example programs, then just 
launch the OMPI apps the same way and it should just work.

On the Slurm side: I’m surprised that it doesn’t work without the —with-slurm 
option. An application proc doesn’t care about any of the Slurm-related code if 
PMIx is available. I might have access to a machine where I can check it…

Ralph


> On Oct 9, 2018, at 3:26 AM, Stephan Krempel  wrote:
> 
> Ralph, Gilles,
> 
> thanks for your input.
> 
> Before I answer, let me shortly explain what my general intention is.
> We do have our own resource manager and process launcher that supports
> different MPI implementations in different ways. I want to adapt it to
> PMIx to cleanly support OpenMPI and hopefully other MPI implementation
> supporting PMIx in the future, too. 
> 
>> It sounds like what you really want to do is replace the orted, and
>> have your orted open your PMIx server? In other words, you want to
>> use the PMIx reference library to handle all the PMIx stuff, and
>> provide your own backend functions to support the PMIx server calls? 
> 
> You are right, that was my original plan, and I already did it so far.
> In my environment I already can launch processes that successfully call
> PMIx client functions like put, get, fence and so on, all handled by my
> servers using the PMIx server helper library. As far as I implemented
> the server functions now, all the example programs coming with the pmix
> library are working fine.
> 
> Then I tried to use that with OpenMPI and stumbled.
> My first idea was to simply replace orted but after taking a closer
> look into OpenMPI it seems to me, that it uses/needs orted not only for
> spawning and exchange of process information, but also for its general
> communication and collectives. Am I wrong with that?
> 
> So replacing it completely is perhaps not what I want since I do not
> intent to replace OpenMPIs whole communication stuff. But perhaps I do
> mix up orte and orted here, not certain about that.
> 
>> If so, then your best bet would be to edit the PRRTE code in
>> orte/orted/pmix and replace it with your code. You’ll have to deal
>> with the ORTE data objects and PRRTE’s launch procedure, but that is
>> likely easier than trying to write your own version of “orted” from
>> scratch.
> 
> I think one problem here is, that I do not really understand which
> purposes orted fulfills overall especially beside implementing the PMIx
> server side. Can you please give me a short overview?
> 
>> As for Slurm: it behaves the same way as PRRTE. It has a plugin that
>> implements the server backend functions, and the Slurm daemons “host”
>> the plugin. What you would need to do is replace that plugin with
>> your own.
> 
> I understand that, but it also seems to need some special support by
> the several slurm modules on the OpenMPI side that I do not understand,
> yet. At least when I tried OpenMPI without slurm support and
> `srun --mpi=pmix_v2` it does not work but generates a message that
> slurm support in opemmpi is missing.
> 
> 
> 
> Stephan
> 
> 
> 
>> 
>>> On Oct 8, 2018, at 5:36 PM, Gilles Gouaillardet 
>>> wrote:
>>> 
>>> Stephan,
>>> 
>>> 
>>> Have you already checked https://github.com/pmix/prrte ?
>>> 
>>> 
>>> This is the PMIx Reference RunTime Environment (PPRTE), which was
>>> built on top of orted.
>>> 
>>> Long story short, it deploys the PMIx server and then you start
>>> your MPI app with prun
>>> An example is available at https://github.com/pmix/prrte/blob/maste
>>> r/contrib/travis/test_client.sh
>>> 
>>> 
>>> Cheers,
>>> 
>>> Gilles
>>> 
>>> 
>>> On 10/9/2018 8:45 AM, Stephan Krempel wrote:
 Hallo everyone,
 
 I am currently implementing a PMIx server and I try to use it
 with
 OpenMPI. I do have an own mpiexec which starts my PMIx server and
 launches the processes.
 
 If I launch an executable linked against OpenMPI, during
 MPI_Init() the
 ORTE layer starts another PMIx server and overrides my PMIX_*
 environment so this new server is used instead of mine.
 
 So I am looking for a method to prevent orte(d) from starting a
 PMIx
 server.
 
 I already tried to understand what the slurm support is doing,
 since
 this is (at least in parts) what I think I need. Somehow when
 

Re: [OMPI devel] Hints for using an own pmix server

[Ralph H Castain]

I assume this (--with-ompi-mpix-rte) is a typo as the correct option is 
—with-ompi-pmix-rte?

It all looks okay to me for the client, but I wonder if you remembered to call 
register_nspace and register_client on your server prior to starting the 
client? If not, the connection will be dropped - you could add 
PMIX_MCA_ptl_base_verbose=100 to your environment to see the detailed 
connection handshake.

> On Oct 9, 2018, at 3:14 PM, Stephan Krempel  wrote:
> 
> Hi Ralf,
> 
> After studying prrte a little bit, I tried something new and followed
> the description here using openmpi 4:
> https://pmix.org/code/building-the-pmix-reference-server/
> 
> I configured openmpi 4.0.0rc3:
> 
> ../configure --enable-debug --prefix [...] --with-pmix=[...] \
>  --with-libevent=/usr --with-ompi-mpix-rte
> 
> (I also tried to set --with-orte=no, but it then claims not to have a
> suitable rte and does not finish)
> 
> I then started my own PMIx and spawned a client compiled with mpicc of
> the new openmpi installation with this environment:
> 
> PMIX_NAMESPACE=namespace_3228_0_0
> PMIX_RANK=0
> PMIX_SERVER_URI2=pmix-server.3234;tcp4://127.0.0.1:49637
> PMIX_SERVER_URI21=pmix-server.3234;tcp4://127.0.0.1:49637
> PMIX_SERVER_URI=pmix-server:3234:/tmp/pmix-3234
> PMIX_SERVER_URI2USOCK=pmix-server:3234:/tmp/pmix-3234
> PMIX_SECURITY_MODE=native,none
> PMIX_PTL_MODULE=tcp,usock
> PMIX_BFROP_BUFFER_TYPE=PMIX_BFROP_BUFFER_FULLY_DESC
> PMIX_GDS_MODULE=ds12,hash
> PMIX_DSTORE_ESH_BASE_PATH=/tmp/pmix_dstor_3234
> 
> The client is not connecting to my pmix server and it's environment
> after MPI_Init looks like that:
> 
> PMIX_SERVER_URI2USOCK=pmix-server:3234:/tmp/pmix-3234
> PMIX_RANK=0
> PMIX_PTL_MODULE=tcp,usock
> PMIX_SERVER_URI=pmix-server:3234:/tmp/pmix-3234
> PMIX_MCA_mca_base_component_show_load_errors=1
> PMIX_BFROP_BUFFER_TYPE=PMIX_BFROP_BUFFER_FULLY_DESC
> PMIX_DSTORE_ESH_BASE_PATH=/tmp/ompi.landroval.1001/pid.3243/pmix_dstor_
> 3243
> PMIX_SERVER_URI2=864157696.0;tcp4://127.0.0.1:33619
> PMIX_SERVER_URI21=864157696.0;tcp4://127.0.0.1:33619
> PMIX_SECURITY_MODE=native,none
> PMIX_NAMESPACE=864157697
> PMIX_GDS_MODULE=ds12,hash
> ORTE_SCHIZO_DETECTION=ORTE
> OMPI_COMMAND=./hello_env
> OMPI_MCA_orte_precondition_transports=f28d6577f6b6ac08-d92c0e73869e1cfa
> OMPI_MCA_orte_launch=1
> OMPI_APP_CTX_NUM_PROCS=1
> OMPI_MCA_pmix=^s1,s2,cray,isolated
> OMPI_MCA_ess=singleton
> OMPI_MCA_orte_ess_num_procs=1
> 
> So something goes wrong but I do not have an idea what I am missing. Do
> you have an idea what I need to change? Do I have to set an MCA
> parameter to tell OpenMPI not to start orted, or does it need another
> hint in the client environment beside the stuff comming from the PMIx
> server helper library?
> 
> 
> Stephan
> 
> 
> On Tuesday, Oct 10 2018, 08:33 -0700 Ralph H Castain wrote:
>> Hi Stephan
>> 
>> Thanks for the clarification - that helps a great deal. You are
>> correct that OMPI’s orted daemons do more than just host the PMIx
>> server library. However, they are only active if you launch the OMPI
>> processes using mpirun. This is probably the source of the trouble
>> you are seeing.
>> 
>> Since you have a process launcher and have integrated the PMIx server
>> support into your RM’s daemons, you really have no need for mpirun at
>> all. You should just be able to launch the processes directly using
>> your own launcher. The PMIx support will take care of the startup
>> requirements. The application procs will not use the orted in such
>> cases.
>> 
>> So if your system is working fine with the PMIx example programs,
>> then just launch the OMPI apps the same way and it should just work.
>> 
>> On the Slurm side: I’m surprised that it doesn’t work without the
>> —with-slurm option. An application proc doesn’t care about any of the
>> Slurm-related code if PMIx is available. I might have access to a
>> machine where I can check it…
>> 
>> Ralph
>> 
>> 
>>> On Oct 9, 2018, at 3:26 AM, Stephan Krempel 
>>> wrote:
>>> 
>>> Ralph, Gilles,
>>> 
>>> thanks for your input.
>>> 
>>> Before I answer, let me shortly explain what my general intention
>>> is.
>>> We do have our own resource manager and process launcher that
>>> supports
>>> different MPI implementations in different ways. I want to adapt it
>>> to
>>> PMIx to cleanly support OpenMPI and hopefully other MPI
>>> implementation
>>> supporting PMIx in the future, too. 
>>> 
>>>> It sounds like what you really want to do is replace the orted

Re: [OMPI devel] btl/vader: race condition in finalize on OS X

[Ralph H Castain]

We already have the register_cleanup option in master - are you using an older 
version of PMIx that doesn’t support it?


> On Oct 2, 2018, at 4:05 AM, Jeff Squyres (jsquyres) via devel 
>  wrote:
> 
> FYI: https://github.com/open-mpi/ompi/issues/5798 brought up what may be the 
> same issue.
> 
> 
>> On Oct 2, 2018, at 3:16 AM, Gilles Gouaillardet  wrote:
>> 
>> Folks,
>> 
>> 
>> When running a simple helloworld program on OS X, we can end up with the 
>> following error message
>> 
>> 
>> A system call failed during shared memory initialization that should
>> not have.  It is likely that your MPI job will now either abort or
>> experience performance degradation.
>> 
>>  Local host:  c7.kmc.kobe.rist.or.jp
>>  System call: unlink(2) 
>> /tmp/ompi.c7.1000/pid.23376/1/vader_segment.c7.17d80001.54
>>  Error:   No such file or directory (errno 2)
>> 
>> 
>> the error does not occur on linux by default since the vader segment is in 
>> /dev/shm by default.
>> 
>> the patch below can be used to evidence the issue on linux
>> 
>> 
>> diff --git a/opal/mca/btl/vader/btl_vader_component.c 
>> b/opal/mca/btl/vader/btl_vader_component.c
>> index 115bceb..80fec05 100644
>> --- a/opal/mca/btl/vader/btl_vader_component.c
>> +++ b/opal/mca/btl/vader/btl_vader_component.c
>> @@ -204,7 +204,7 @@ static int mca_btl_vader_component_register (void)
>>OPAL_INFO_LVL_3, 
>> MCA_BASE_VAR_SCOPE_GROUP, _btl_vader_component.single_copy_mechanism);
>> OBJ_RELEASE(new_enum);
>> 
>> -if (0 == access ("/dev/shm", W_OK)) {
>> +if (0 && 0 == access ("/dev/shm", W_OK)) {
>> mca_btl_vader_component.backing_directory = "/dev/shm";
>> } else {
>> mca_btl_vader_component.backing_directory = 
>> opal_process_info.job_session_dir;
>> 
>> 
>> From my analysis, here is what happens :
>> 
>> - each rank is supposed to have its own vader_segment unlinked by btl/vader 
>> in vader_finalize().
>> 
>> - but this file might have already been destroyed by an other task in 
>> orte_ess_base_app_finalize()
>> 
>>  if (NULL == opal_pmix.register_cleanup) {
>>orte_session_dir_finalize(ORTE_PROC_MY_NAME);
>>}
>> 
>>  *all* the tasks end up removing 
>> opal_os_dirpath_destroy("/tmp/ompi.c7.1000/pid.23941/1")
>> 
>> 
>> I am not really sure about the best way to fix this.
>> 
>> - one option is to perform an intra node barrier in vader_finalize()
>> 
>> - an other option would be to implement an opal_pmix.register_cleanup
>> 
>> 
>> Any thoughts ?
>> 
>> 
>> Cheers,
>> 
>> 
>> Gilles
>> 
>> ___
>> devel mailing list
>> devel@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/devel
> 
> 
> -- 
> Jeff Squyres
> jsquy...@cisco.com
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

Re: [OMPI devel] Removing ORTE code

[Ralph H Castain]

Based on silence plus today’s telecon, the stale code has been removed: 
https://github.com/open-mpi/ompi/pull/5827


> On Sep 26, 2018, at 7:00 AM, Ralph H Castain  wrote:
> 
> We are considering a “purge” of stale ORTE code and want to know if anyone is 
> using it before proceeding. With the advent of PMIx, several ORTE features 
> are no longer required by OMPI itself. However, we acknowledge that it is 
> possible that someone out there (e.g., a researcher) is using them. The 
> specific features include:
> 
> * OOB use from within an application process. We need to retain the OOB 
> itself for daemon-to-daemon communication. However, the application processes 
> no longer open a connection to their ORTE daemon, instead relying on the PMIx 
> connection to communicate their needs.
> 
> * the DFS framework - allows an application process to access a remote file 
> via ORTE. It provided essentially a function-shipping service that was used 
> by map-reduce applications we no longer support
> 
> * the notifier framework - supported output of messages to syslog and email. 
> PMIx now provides such services if someone wants to use them
> 
> * iof/tool component - we are moving to PMIx for tool support, so there are 
> no ORTE tools using this any more
> 
> We may discover additional candidates for removal as we go forward - we’ll 
> update the list as we do. First, however, we’d really like to hear back from 
> anyone who might have a need for any of the above.
> 
> Please respond by Oct 5th
> Ralph
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

Re: [OMPI devel] Mac OS X 10.4.x users?

[Ralph H Castain]

Good lord - break away!!

> On Sep 28, 2018, at 11:11 AM, Barrett, Brian via devel 
>  wrote:
> 
> All -
> 
> In trying to clean up some warnings, I noticed one (around pack/unpack in 
> net/if.h) that is due to a workaround of a bug in MacOS X 10.4.x and earlier. 
>  The simple way to remove the warning would be to remove the workaround, 
> which would break the next major version of Open MPI on 10.4.x and earlier on 
> 64 bit systems.  10.5.x was released 11 years ago and didn’t drop support for 
> any 64 bit systems.  I posted a PR which removes support for 10.4.x and 
> earlier (through the README) and removes the warning generated workaround 
> (https://github.com/open-mpi/ompi/pull/5803).
> 
> Does anyone object to breaking 10.4.x and earlier?
> 
> Brian
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

[OMPI devel] Error in TCP BTL??

[Ralph H Castain]

I’m getting this error when trying to run a simple ring program on my Mac:

[Ralphs-iMac-2.local][[21423,14],0][btl_tcp_endpoint.c:742:mca_btl_tcp_endpoint_start_connect]
 bind() failed: Invalid argument (22)

Anyone recognize the problem? It causes the job to immediately abort. This is 
with current head of master this morning - it was working when I last used it, 
but it has been an unknown period of time.
Ralph

___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

Re: [OMPI devel] OFI issues on Open MPI v4.0.0rc1

[Ralph H Castain]

That’s why we are leaving it in master - only removing it from release branch 

Sent from my iPhone

> On Sep 20, 2018, at 7:02 PM, George Bosilca  wrote:
> 
> Why not simply ompi_ignore it ? Removing a component to bring it back later 
> would force us to lose all history. I would a rather add an .ompi_ignore and 
> give an opportunity to power users do continue playing with it.
> 
>   George.
> 
> 
>> On Thu, Sep 20, 2018 at 8:04 PM Ralph H Castain  wrote:
>> I already suggested the configure option, but it doesn’t solve the problem. 
>> I wouldn’t be terribly surprised to find that Cray also has an undetected 
>> problem given the nature of the issue - just a question of the amount of 
>> testing, variety of environments, etc.
>> 
>> Nobody has to wait for the next major release, though that isn’t so far off 
>> anyway - there has never been an issue with bringing in a new component 
>> during a release series.
>> 
>> Let’s just fix this the right way and bring it into 4.1 or 4.2. We may want 
>> to look at fixing the osc/rdma/ofi bandaid as well while we are at it.
>> 
>> Ralph
>> 
>> 
>>> On Sep 20, 2018, at 4:45 PM, Patinyasakdikul, Thananon 
>>>  wrote:
>>> 
>>> I understand and agree with your point. My initial email is just out of 
>>> curiosity.
>>> 
>>> Howard tested this BTL for Cray in the summer as well. So this seems to 
>>> only affected OPA hardware.
>>> 
>>> I just remember that in the summer, I have to make some change in libpsm2 
>>> to get this BTL to work for OPA.  Maybe this is the problem as the default 
>>> libpsm2 won't work.
>>> 
>>> So maybe we can fix this in configure step to detect version of libpsm2 and 
>>> dont build if we are not satisfied.
>>> 
>>> Another idea is maybe we dont build this BTL by default. So the user with 
>>> Cray hardware can still use it if they want. (Just rebuild with the btl)  - 
>>> We just need to verify if it still works on Cray.  This way, OFI 
>>> stakeholders does not have to wait until next major release to get this in.
>>> 
>>> 
>>> Arm
>>> 
>>> 
>>>> On Thu, Sep 20, 2018, 7:18 PM Ralph H Castain  wrote:
>>>> I suspect it is a question of what you tested and in which scenarios. 
>>>> Problem is that it can bite someone and there isn’t a clean/obvious 
>>>> solution that doesn’t require the user to do something - e.g., like having 
>>>> to know that they need to disable a BTL. Matias has proposed an mca-based 
>>>> approach, but I would much rather we just fix this correctly. Bandaids 
>>>> have a habit of becoming permanently forgotten - until someone pulls on it 
>>>> and things unravel.
>>>> 
>>>> 
>>>>> On Sep 20, 2018, at 4:14 PM, Patinyasakdikul, Thananon 
>>>>>  wrote:
>>>>> 
>>>>> In the summer, I tested this BTL with along with the MTL and able to use 
>>>>> both of them interchangeably with no problem. I dont know what changed. 
>>>>> libpsm2?
>>>>> 
>>>>> 
>>>>> Arm
>>>>> 
>>>>> 
>>>>>> On Thu, Sep 20, 2018, 7:06 PM Ralph H Castain  wrote:
>>>>>> We have too many discussion threads overlapping on the same email chain 
>>>>>> - so let’s break the discussion on the OFI problem into its own chain.
>>>>>> 
>>>>>> We have been investigating this locally and found there are a number of 
>>>>>> conflicts between the MTLs and the OFI/BTL stepping on each other. The 
>>>>>> correct solution is to move endpoint creation/reporting into a the 
>>>>>> opal/mca/common area, but that is going to take some work and will 
>>>>>> likely impact release schedules.
>>>>>> 
>>>>>> Accordingly, we propose to remove the OFI/BTL component from v4.0.0, fix 
>>>>>> the problem in master, and then consider bringing it back as a package 
>>>>>> to v4.1 or v4.2.
>>>>>> 
>>>>>> Comments? If we agree, I’ll file a PR to remove it.
>>>>>> Ralph
>>>>>> 
>>>>>> 
>>>>>>> Begin forwarded message:
>>>>>>> 
>>>>>>> From: Peter Kjellström 
>>>>>>> Subject: Re: [OMPI devel] Announcing Open MPI v4.0.0rc1
>>>>>>> Date: Septembe

[OMPI devel] OFI issues on Open MPI v4.0.0rc1

[Ralph H Castain]

We have too many discussion threads overlapping on the same email chain - so 
let’s break the discussion on the OFI problem into its own chain.

We have been investigating this locally and found there are a number of 
conflicts between the MTLs and the OFI/BTL stepping on each other. The correct 
solution is to move endpoint creation/reporting into a the opal/mca/common 
area, but that is going to take some work and will likely impact release 
schedules.

Accordingly, we propose to remove the OFI/BTL component from v4.0.0, fix the 
problem in master, and then consider bringing it back as a package to v4.1 or 
v4.2.

Comments? If we agree, I’ll file a PR to remove it.
Ralph


> Begin forwarded message:
> 
> From: Peter Kjellström 
> Subject: Re: [OMPI devel] Announcing Open MPI v4.0.0rc1
> Date: September 20, 2018 at 5:18:35 AM PDT
> To: "Gabriel, Edgar" 
> Cc: Open MPI Developers 
> Reply-To: Open MPI Developers 
> 
> On Wed, 19 Sep 2018 16:24:53 +
> "Gabriel, Edgar"  wrote:
> 
>> I performed some tests on our Omnipath cluster, and I have a mixed
>> bag of results with 4.0.0rc1
> 
> I've also tried it on our OPA cluster (skylake+centos-7+inbox) with
> very similar results.
> 
>> compute-1-1.local.4351PSM2 has not been initialized
>> compute-1-0.local.3826PSM2 has not been initialized
> 
> yup I too see these.
> 
>> mpirun detected that one or more processes exited with non-zero
>> status, thus causing the job to be terminated. The first process to
>> do so was:
>> 
>>  Process name: [[38418,1],1]
>>  Exit code:255
>>  
>> 
> 
> yup.
> 
>> 
>> 2.   The ofi mtl does not work at all on our Omnipath cluster. If
>> I try to force it using ‘mpirun –mca mtl ofi …’ I get the following
>> error message.
> 
> Yes ofi seems broken. But not even disabling it helps me completely (I
> see "mca_btl_ofi.so   [.] mca_btl_ofi_component_progress" in my
> perf top...
> 
>> 3.   The openib btl component is always getting in the way with
>> annoying warnings. It is not really used, but constantly complains:
> ...
>> [sabine.cacds.uh.edu:25996] 1 more process has sent help message
>> help-mpi-btl-openib.txt / ib port not selected
> 
> Yup.
> 
> ...
>> So bottom line, if I do
>> 
>> mpirun –mca btl^openib –mca mtl^ofi ….
>> 
>> my tests finish correctly, although mpirun will still return an error.
> 
> I get some things to work with this approach (two ranks on two nodes
> for example). But a lot of things crash rahter hard:
> 
> $ mpirun -mca btl ^openib -mca mtl
> ^ofi ./openmpi-4.0.0rc1/imb.openmpi-4.0.0rc1
> --
> PSM2 was unable to open an endpoint. Please make sure that the network
> link is active on the node and the hardware is functioning.
> 
>  Error: Failure in initializing endpoint
> --
> n909.279895hfi_userinit: assign_context command failed: Device or
> resource busy n909.279895psmi_context_open: hfi_userinit: failed,
> trying again (1/3)
> ...
>  PML add procs failed
>  --> Returned "Error" (-1) instead of "Success" (0)
> --
> [n908:298761] *** An error occurred in MPI_Init
> [n908:298761] *** reported by process [4092002305,59]
> [n908:298761] *** on a NULL communicator
> [n908:298761] *** Unknown error
> [n908:298761] *** MPI_ERRORS_ARE_FATAL (processes in this communicator
>  will now abort, [n908:298761] ***and potentially your MPI job)
> [n907:407748] 255 more processes have sent help message
>  help-mtl-psm2.txt / unable to open endpoint [n907:407748] Set MCA
>  parameter "orte_base_help_aggregate" to 0 to see all help / error
>  messages [n907:407748] 127 more processes have sent help message
>  help-mpi-runtime.txt / mpi_init:startup:internal-failure
>  [n907:407748] 56 more processes have sent help message
>  help-mpi-errors.txt / mpi_errors_are_fatal unknown handle
> 
> If I disable psm2 too I get it to run (apparantly on vader?)
> 
> /Peter K
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

Re: [OMPI devel] OFI issues on Open MPI v4.0.0rc1

[Ralph H Castain]

I already suggested the configure option, but it doesn’t solve the problem. I 
wouldn’t be terribly surprised to find that Cray also has an undetected problem 
given the nature of the issue - just a question of the amount of testing, 
variety of environments, etc.

Nobody has to wait for the next major release, though that isn’t so far off 
anyway - there has never been an issue with bringing in a new component during 
a release series.

Let’s just fix this the right way and bring it into 4.1 or 4.2. We may want to 
look at fixing the osc/rdma/ofi bandaid as well while we are at it.

Ralph


> On Sep 20, 2018, at 4:45 PM, Patinyasakdikul, Thananon 
>  wrote:
> 
> I understand and agree with your point. My initial email is just out of 
> curiosity.
> 
> Howard tested this BTL for Cray in the summer as well. So this seems to only 
> affected OPA hardware.
> 
> I just remember that in the summer, I have to make some change in libpsm2 to 
> get this BTL to work for OPA.  Maybe this is the problem as the default 
> libpsm2 won't work.
> 
> So maybe we can fix this in configure step to detect version of libpsm2 and 
> dont build if we are not satisfied.
> 
> Another idea is maybe we dont build this BTL by default. So the user with 
> Cray hardware can still use it if they want. (Just rebuild with the btl)  - 
> We just need to verify if it still works on Cray.  This way, OFI stakeholders 
> does not have to wait until next major release to get this in.
> 
> 
> Arm
> 
> 
> On Thu, Sep 20, 2018, 7:18 PM Ralph H Castain  <mailto:r...@open-mpi.org>> wrote:
> I suspect it is a question of what you tested and in which scenarios. Problem 
> is that it can bite someone and there isn’t a clean/obvious solution that 
> doesn’t require the user to do something - e.g., like having to know that 
> they need to disable a BTL. Matias has proposed an mca-based approach, but I 
> would much rather we just fix this correctly. Bandaids have a habit of 
> becoming permanently forgotten - until someone pulls on it and things unravel.
> 
> 
>> On Sep 20, 2018, at 4:14 PM, Patinyasakdikul, Thananon 
>> mailto:tpati...@vols.utk.edu>> wrote:
>> 
>> In the summer, I tested this BTL with along with the MTL and able to use 
>> both of them interchangeably with no problem. I dont know what changed. 
>> libpsm2?
>> 
>> 
>> Arm
>> 
>> 
>> On Thu, Sep 20, 2018, 7:06 PM Ralph H Castain > <mailto:r...@open-mpi.org>> wrote:
>> We have too many discussion threads overlapping on the same email chain - so 
>> let’s break the discussion on the OFI problem into its own chain.
>> 
>> We have been investigating this locally and found there are a number of 
>> conflicts between the MTLs and the OFI/BTL stepping on each other. The 
>> correct solution is to move endpoint creation/reporting into a the 
>> opal/mca/common area, but that is going to take some work and will likely 
>> impact release schedules.
>> 
>> Accordingly, we propose to remove the OFI/BTL component from v4.0.0, fix the 
>> problem in master, and then consider bringing it back as a package to v4.1 
>> or v4.2.
>> 
>> Comments? If we agree, I’ll file a PR to remove it.
>> Ralph
>> 
>> 
>>> Begin forwarded message:
>>> 
>>> From: Peter Kjellström mailto:c...@nsc.liu.se>>
>>> Subject: Re: [OMPI devel] Announcing Open MPI v4.0.0rc1
>>> Date: September 20, 2018 at 5:18:35 AM PDT
>>> To: "Gabriel, Edgar" >> <mailto:egabr...@central.uh.edu>>
>>> Cc: Open MPI Developers >> <mailto:devel@lists.open-mpi.org>>
>>> Reply-To: Open MPI Developers >> <mailto:devel@lists.open-mpi.org>>
>>> 
>>> On Wed, 19 Sep 2018 16:24:53 +
>>> "Gabriel, Edgar" mailto:egabr...@central.uh.edu>> 
>>> wrote:
>>> 
>>>> I performed some tests on our Omnipath cluster, and I have a mixed
>>>> bag of results with 4.0.0rc1
>>> 
>>> I've also tried it on our OPA cluster (skylake+centos-7+inbox) with
>>> very similar results.
>>> 
>>>> compute-1-1.local.4351PSM2 has not been initialized
>>>> compute-1-0.local.3826PSM2 has not been initialized
>>> 
>>> yup I too see these.
>>> 
>>>> mpirun detected that one or more processes exited with non-zero
>>>> status, thus causing the job to be terminated. The first process to
>>>> do so was:
>>>> 
>>>>  Process name: [[38418,1],1]
>>>>  Exit code:255
>>>>  
>>

Re: [OMPI devel] OFI issues on Open MPI v4.0.0rc1

[Ralph H Castain]

I suspect it is a question of what you tested and in which scenarios. Problem 
is that it can bite someone and there isn’t a clean/obvious solution that 
doesn’t require the user to do something - e.g., like having to know that they 
need to disable a BTL. Matias has proposed an mca-based approach, but I would 
much rather we just fix this correctly. Bandaids have a habit of becoming 
permanently forgotten - until someone pulls on it and things unravel.


> On Sep 20, 2018, at 4:14 PM, Patinyasakdikul, Thananon 
>  wrote:
> 
> In the summer, I tested this BTL with along with the MTL and able to use both 
> of them interchangeably with no problem. I dont know what changed. libpsm2?
> 
> 
> Arm
> 
> 
> On Thu, Sep 20, 2018, 7:06 PM Ralph H Castain  <mailto:r...@open-mpi.org>> wrote:
> We have too many discussion threads overlapping on the same email chain - so 
> let’s break the discussion on the OFI problem into its own chain.
> 
> We have been investigating this locally and found there are a number of 
> conflicts between the MTLs and the OFI/BTL stepping on each other. The 
> correct solution is to move endpoint creation/reporting into a the 
> opal/mca/common area, but that is going to take some work and will likely 
> impact release schedules.
> 
> Accordingly, we propose to remove the OFI/BTL component from v4.0.0, fix the 
> problem in master, and then consider bringing it back as a package to v4.1 or 
> v4.2.
> 
> Comments? If we agree, I’ll file a PR to remove it.
> Ralph
> 
> 
>> Begin forwarded message:
>> 
>> From: Peter Kjellström mailto:c...@nsc.liu.se>>
>> Subject: Re: [OMPI devel] Announcing Open MPI v4.0.0rc1
>> Date: September 20, 2018 at 5:18:35 AM PDT
>> To: "Gabriel, Edgar" > <mailto:egabr...@central.uh.edu>>
>> Cc: Open MPI Developers > <mailto:devel@lists.open-mpi.org>>
>> Reply-To: Open MPI Developers > <mailto:devel@lists.open-mpi.org>>
>> 
>> On Wed, 19 Sep 2018 16:24:53 +
>> "Gabriel, Edgar" mailto:egabr...@central.uh.edu>> 
>> wrote:
>> 
>>> I performed some tests on our Omnipath cluster, and I have a mixed
>>> bag of results with 4.0.0rc1
>> 
>> I've also tried it on our OPA cluster (skylake+centos-7+inbox) with
>> very similar results.
>> 
>>> compute-1-1.local.4351PSM2 has not been initialized
>>> compute-1-0.local.3826PSM2 has not been initialized
>> 
>> yup I too see these.
>> 
>>> mpirun detected that one or more processes exited with non-zero
>>> status, thus causing the job to be terminated. The first process to
>>> do so was:
>>> 
>>>  Process name: [[38418,1],1]
>>>  Exit code:255
>>>  
>>> 
>> 
>> yup.
>> 
>>> 
>>> 2.   The ofi mtl does not work at all on our Omnipath cluster. If
>>> I try to force it using ‘mpirun –mca mtl ofi …’ I get the following
>>> error message.
>> 
>> Yes ofi seems broken. But not even disabling it helps me completely (I
>> see "mca_btl_ofi.so   [.] mca_btl_ofi_component_progress" in my
>> perf top...
>> 
>>> 3.   The openib btl component is always getting in the way with
>>> annoying warnings. It is not really used, but constantly complains:
>> ...
>>> [sabine.cacds.uh.edu:25996 <http://sabine.cacds.uh.edu:25996/>] 1 more 
>>> process has sent help message
>>> help-mpi-btl-openib.txt / ib port not selected
>> 
>> Yup.
>> 
>> ...
>>> So bottom line, if I do
>>> 
>>> mpirun –mca btl^openib –mca mtl^ofi ….
>>> 
>>> my tests finish correctly, although mpirun will still return an error.
>> 
>> I get some things to work with this approach (two ranks on two nodes
>> for example). But a lot of things crash rahter hard:
>> 
>> $ mpirun -mca btl ^openib -mca mtl
>> ^ofi ./openmpi-4.0.0rc1/imb.openmpi-4.0.0rc1
>> --
>> PSM2 was unable to open an endpoint. Please make sure that the network
>> link is active on the node and the hardware is functioning.
>> 
>>  Error: Failure in initializing endpoint
>> --
>> n909.279895hfi_userinit: assign_context command failed: Device or
>> resource busy n909.279895psmi_context_open: hfi_userinit: failed,
>> trying again (1/3)
>> ...
>>  PML add p

[OMPI devel] Removing ORTE code

[Ralph H Castain]

We are considering a “purge” of stale ORTE code and want to know if anyone is 
using it before proceeding. With the advent of PMIx, several ORTE features are 
no longer required by OMPI itself. However, we acknowledge that it is possible 
that someone out there (e.g., a researcher) is using them. The specific 
features include:

* OOB use from within an application process. We need to retain the OOB itself 
for daemon-to-daemon communication. However, the application processes no 
longer open a connection to their ORTE daemon, instead relying on the PMIx 
connection to communicate their needs.

* the DFS framework - allows an application process to access a remote file via 
ORTE. It provided essentially a function-shipping service that was used by 
map-reduce applications we no longer support

* the notifier framework - supported output of messages to syslog and email. 
PMIx now provides such services if someone wants to use them

* iof/tool component - we are moving to PMIx for tool support, so there are no 
ORTE tools using this any more

We may discover additional candidates for removal as we go forward - we’ll 
update the list as we do. First, however, we’d really like to hear back from 
anyone who might have a need for any of the above.

Please respond by Oct 5th
Ralph

___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

Re: [OMPI devel] OMPI and PRRTE separated

[Ralph H Castain]

FYI: I have deleted all the old OMPI tags from PRRTE, so we have a clean repo 
to work with now.


> On Dec 17, 2018, at 5:58 PM, Ralph H Castain  wrote:
> 
> Hello all
> 
> For those of you working with ORTE and/or PRRTE, GitHub has severed the 
> parent/child relationship between the OMPI and PRRTE repositories. Thus, we 
> will no longer be able to directly “pull” changes made to ORTE downstream 
> into PRRTE.
> 
> This marks the end of direct support for ORTE except in release branches as 
> people have time and inclination. We invite people to instead work in PRRTE 
> on any future-facing features.
> 
> The question of what to do about OPAL remains under discussion as a 
> much-reduced copy of it currently resides in PRRTE.
> Ralph
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

[OMPI devel] OMPI and PRRTE separated

[Ralph H Castain]

Hello all

For those of you working with ORTE and/or PRRTE, GitHub has severed the 
parent/child relationship between the OMPI and PRRTE repositories. Thus, we 
will no longer be able to directly “pull” changes made to ORTE downstream into 
PRRTE.

This marks the end of direct support for ORTE except in release branches as 
people have time and inclination. We invite people to instead work in PRRTE on 
any future-facing features.

The question of what to do about OPAL remains under discussion as a 
much-reduced copy of it currently resides in PRRTE.
Ralph

___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

[OMPI devel] PMIx v3.0 Standard released

[Ralph H Castain]

The PMIx community, representing a consortium of research, academic, and 
industry partners, is pleased to announce the release of the PMIx v3.0 Standard 
document. The document can be obtained from:

* the PMIx website at 
https://pmix.org/wp-content/uploads/2018/12/pmix-standard-3.0.pdf

* the PMIx Standard repository at 
https://github.com/pmix/pmix-standard/releases/tag/v3.0

* by cloning the PMIx Standard repository and generating the document yourself. 
The source can be obtained from https://github.com/pmix/pmix-standard 
 by selecting the “v3” branch. 
Generating the document requires installation of the LaTex publishing system.

Please see below for a brief summary of the release notes. This release brings 
the Standard up to date with prior PMIx Reference Implementation (PRI) 
releases. Going forward, releases of the PRI will be timed to coincide (or 
shortly trail) releases of the corresponding Standard document. The current 
roadmap (with target schedule)  includes:

PMIx v4.0: first half of 2019
* Completion of the tool/debugger support, including a new chapter that 
specifically addresses how to create PMIx-based tools
* Description of the new PMIx Group and Process Set support
* Completion of the fabric support, including new server APIs for accessing 
fabric topology information in support of scheduling operations

PMIx v5.0: second half of 2019
* Initial work on storage integration APIs
* Introduction of Python bindings

Ralph

*
PMIx v3.0 Release Notes

Initial release of version 3.0 of the PMIx Standard. Additions/changes from 
version 2.1 include:

The following APIs were introduced in v3.0 of the PMIx Standard:
* Client APIs
* PMIx_Log , PMIx_Job_control
* PMIx_Allocation_request , PMIx_Process_monitor
* PMIx_Get_credential , PMIx_Validate_credential
 * Server APIs
* PMIx_server_IOF_deliver
* PMIx_server_collect_inventory , PMIx_server_deliver_inventory
* Tool APIs
* PMIx_IOF_pull , PMIx_IOF_push , PMIx_IOF_deregister
* PMIx_tool_connect_to_server
* Common APIs
* PMIx_IOF_channel_string

The document added a chapter on security credentials, a new section for 
Input/Output (IO) forwarding to the Process Management chapter, and a few 
blocking forms of previously-existing non-blocking APIs. Attributes supporting 
the new APIs were introduced, as well as additional attributes for a few 
existing functions.

As always, creation of this release of the Standard required a great deal of 
work on the part of a number of people. We invite you to read the 
Acknowledgement section for a list of those who contributed to the Standard in 
terms of the included definitions, functional concepts, and/or authorship. Our 
thanks go out to all.

Please provide comments on the PMIx Standard by filing issues on the document 
repository https://github.com/pmix/pmix-standard/issues 
 or by sending them to the PMIx 
Community mailing list at https://groups.google.com/forum/#!forum/pmix 
. Comments should include the 
version of the PMIx standard you are commenting about, and the page, section, 
and line numbers that you are referencing. As a reminder, please note that 
messages sent to the mailing list from an unsubscribed e-mail address will be 
ignored.

___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

Re: [OMPI devel] [OMPI users] open-mpi.org is DOWN

[Ralph H Castain]

The security scanner has apologized for a false positive and fixed their system 
- the site has been restored.

Ralph


> On Dec 22, 2018, at 12:12 PM, Ralph H Castain  wrote:
> 
> Hello all
> 
> Apologies to everyone, but I received an alert this moring that malware has 
> been detected on the www.open-mpi.org site. I have tried to contact the 
> hosting agency and the security scanners, but nobody is around on this 
> pre-holiday weekend.
> 
> Accordingly, I have taken the site OFFLINE for the indeterminate future until 
> we can get this resolved. Sadly, with the holidays upon us, I don’t know how 
> long it will take to get responses from either company. Until we do, the site 
> will remain offline for safety reasons.
> 
> Ralph
> 
> ___
> users mailing list
> us...@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

[OMPI devel] open-mpi.org is DOWN

[Ralph H Castain]

Hello all

Apologies to everyone, but I received an alert this moring that malware has 
been detected on the www.open-mpi.org site. I have tried to contact the hosting 
agency and the security scanners, but nobody is around on this pre-holiday 
weekend.

Accordingly, I have taken the site OFFLINE for the indeterminate future until 
we can get this resolved. Sadly, with the holidays upon us, I don’t know how 
long it will take to get responses from either company. Until we do, the site 
will remain offline for safety reasons.

Ralph

___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

[OMPI devel] PMIx v2.1 Standard released

[Ralph H Castain]

The PMIx community, representing a consortium of research, academic, and 
industry partners, is pleased to announce the release of the PMIx v2.1 Standard 
document. The document can be obtained from:

* the PMIx website at 
https://pmix.org/wp-content/uploads/2018/12/pmix-standard-2.1.pdf 


* the PMIx Standard repository at 
https://github.com/pmix/pmix-standard/releases/tag/v2.1 


* by cloning the PMIx Standard repository and generating the document yourself. 
The source can be obtained from https://github.com/pmix/pmix-standard 
 by selecting the “v2” branch. 
Generating the document requires installation of the LaTex publishing system.

The v2.1 update includes clarifications and corrections, plus addition of 
examples:

* Clarify description of PMIx_Connect and PMIx_Disconnect APIs.
* Explain that values for the PMIX_COLLECTIVE_ALGO are environment-dependent
* Identify the namespace/rank values required for retrieving 
attribute-associated information using the PMIx_Get API
* Provide definitions for session, job, application, and other terms used 
throughout the document
* Clarify definitions of PMIX_UNIV_SIZE versus PMIX_JOB_SIZE
* Clarify server module function return values
* Provide examples of the use of PMIx_Get for retrieval of information
* Clarify the use of PMIx_Get versus PMIx_Query_info_nb
* Clarify return values for non-blocking APIs and emphasize that callback 
functions must not be invoked prior to return from the API
* Provide detailed example for construction of the PMIx_server_register_nspace 
input information array
* Define information levels (e.g., session vs job) and associated attributes 
for both storing and retrieving values
* Clarify roles of PMIx server library and host environment for collective 
operations
* Clarify definition of PMIX_UNIV_SIZE


As always, creation of this release of the Standard required a great deal of 
work on the part of a number of people. We invite you to read the 
Acknowledgement section for a list of those who contributed to the Standard in 
terms of the included definitions, functional concepts, and/or authorship. Our 
thanks go out to all.

Please provide comments on the PMIx Standard by filing issues on the document 
repository \url{https://github.com/pmix/pmix-standard/issues 
} or by sending them to the PMIx 
Community mailing list at \url{https://groups.google.com/forum/#!forum/pmix 
}. Comments should include the 
version of the PMIx standard you are commenting about, and the page, section, 
and line numbers that you are referencing. As a reminder, please note that 
messages sent to the mailing list from an unsubscribed e-mail address will be 
ignored.

Ralph
___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

[OMPI devel] PRRTE v3.0.0rc1 available for testing

[Ralph H Castain]

Hi folks

Given a growing use of PRRTE plus OMPI’s announced plans to phase out ORTE in 
favor of PRRTE, it seems the time has come to begin generating formal releases 
of PRRTE. Accordingly, I have created a v3.0.0 release candidate for folks to 
(hopefully) test:

https://github.com/pmix/prrte/releases/tag/prrte-v3.0.0rc1

Note that the first number in the version triplet represents the PMIx level 
being supported - in this case, the release candidate supports PMIx v3 of the 
Standard. I know we haven’t yet released that formal document, but it is 
available in PR form and should be released fairly soon.

This release candidate actually includes support for some of the 
under-development PMIx v4 APIs. However, the version triplet is based on what 
we consider to be the “production” level of PRRTE, and the PMIx v4 support 
included in the current code is truly at the prototype level. Those who want to 
experiment with those interfaces are welcome to build this release against the 
PMIx master branch (which is at the cutting edge of PMIx v4 development) and 
work with them. Now that we have started formal releases, we’ll do a better job 
of branching early prior to introducing new APIs so we won’t have this mix 
again.

Note that PRRTE contains _no_ embedded code. Thus, building it requires that 
you have libevent, hwloc, and some version of PMIx installed - if not in 
standard locations, you’ll need to use the configure arguments to point to 
them. Any version of PMIx that is at least v2.0 is supported - PRRTE does _not_ 
support the PMIx v1 series.

Please give this a whirl and post any comments/bugs to the github issues.
Ralph

___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

[OMPI devel] MTT Perl client

[Ralph H Castain]

Hi folks

Per today’s telecon, I have moved the Perl MTT client into its own repository: 
https://github.com/open-mpi/mtt-legacy. All the Python client code has been 
removed from that repo.

The original MTT repo remains at https://github.com/open-mpi/mtt. I have a PR 
to remove all the Perl client code and associated libs/modules from that repo. 
We won’t commit it until people have had a chance to switch to the mtt-legacy 
repo and verify that things still work for them.

Please let us know if mtt-legacy is okay or has a problem.

Thanks
Ralph

___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

Re: [OMPI devel] MTT Perl client

[Ralph H Castain]

I very much doubt that there is a Python equivalent yet.

> On Sep 14, 2018, at 9:23 AM, Jeff Squyres (jsquyres) via devel 
>  wrote:
> 
> It's for environments where MTT is run where it can't reach the greater 
> internet (or, at least, it can't POST to the greater internet).  You run the 
> mtt-relay on a machine that is reachable by your machines running MTT, and it 
> works as a relay to mtt.open-mpi.org so that you can submit your MTT results.
> 
> It might actually be fairly protocol agnostic, IIRC (been a while since I've 
> looked at that code).
> 
> 
> 
>> On Sep 14, 2018, at 11:23 AM, Ralph H Castain  wrote:
>> 
>> Afraid I’m not familiar with that script - what does it do?
>> 
>> 
>>> On Sep 14, 2018, at 7:46 AM, Christoph Niethammer  
>>> wrote:
>>> 
>>> Works for the installation at HLRS.
>>> 
>>> Short note/question: I am using the mtt-relay script. This is written in 
>>> perl. Is there a python based replacement?
>>> 
>>> Best
>>> Christoph Niethammer
>>> 
>>> - Mensaje original -
>>> De: "Open MPI Developers" 
>>> Para: "Open MPI Developers" 
>>> CC: "Jeff Squyres" 
>>> Enviados: Martes, 11 de Septiembre 2018 20:37:40
>>> Asunto: Re: [OMPI devel] MTT Perl client
>>> 
>>> Works for me.
>>> 
>>>> On Sep 11, 2018, at 12:35 PM, Ralph H Castain  wrote:
>>>> 
>>>> Hi folks
>>>> 
>>>> Per today’s telecon, I have moved the Perl MTT client into its own 
>>>> repository: https://github.com/open-mpi/mtt-legacy. All the Python client 
>>>> code has been removed from that repo.
>>>> 
>>>> The original MTT repo remains at https://github.com/open-mpi/mtt. I have a 
>>>> PR to remove all the Perl client code and associated libs/modules from 
>>>> that repo. We won’t commit it until people have had a chance to switch to 
>>>> the mtt-legacy repo and verify that things still work for them.
>>>> 
>>>> Please let us know if mtt-legacy is okay or has a problem.
>>>> 
>>>> Thanks
>>>> Ralph
>>>> 
>>>> ___
>>>> devel mailing list
>>>> devel@lists.open-mpi.org
>>>> https://lists.open-mpi.org/mailman/listinfo/devel
>>> 
>>> 
>>> -- 
>>> Jeff Squyres
>>> jsquy...@cisco.com
>>> 
>>> ___
>>> devel mailing list
>>> devel@lists.open-mpi.org
>>> https://lists.open-mpi.org/mailman/listinfo/devel
>>> ___
>>> devel mailing list
>>> devel@lists.open-mpi.org
>>> https://lists.open-mpi.org/mailman/listinfo/devel
>> 
>> ___
>> devel mailing list
>> devel@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/devel
> 
> 
> -- 
> Jeff Squyres
> jsquy...@cisco.com
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

Re: [OMPI devel] MTT Perl client

[Ralph H Castain]

Afraid I’m not familiar with that script - what does it do?


> On Sep 14, 2018, at 7:46 AM, Christoph Niethammer  wrote:
> 
> Works for the installation at HLRS.
> 
> Short note/question: I am using the mtt-relay script. This is written in 
> perl. Is there a python based replacement?
> 
> Best
> Christoph Niethammer
> 
> - Mensaje original -
> De: "Open MPI Developers" 
> Para: "Open MPI Developers" 
> CC: "Jeff Squyres" 
> Enviados: Martes, 11 de Septiembre 2018 20:37:40
> Asunto: Re: [OMPI devel] MTT Perl client
> 
> Works for me.
> 
>> On Sep 11, 2018, at 12:35 PM, Ralph H Castain  wrote:
>> 
>> Hi folks
>> 
>> Per today’s telecon, I have moved the Perl MTT client into its own 
>> repository: https://github.com/open-mpi/mtt-legacy. All the Python client 
>> code has been removed from that repo.
>> 
>> The original MTT repo remains at https://github.com/open-mpi/mtt. I have a 
>> PR to remove all the Perl client code and associated libs/modules from that 
>> repo. We won’t commit it until people have had a chance to switch to the 
>> mtt-legacy repo and verify that things still work for them.
>> 
>> Please let us know if mtt-legacy is okay or has a problem.
>> 
>> Thanks
>> Ralph
>> 
>> ___
>> devel mailing list
>> devel@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/devel
> 
> 
> -- 
> Jeff Squyres
> jsquy...@cisco.com
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/devel
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

Re: [OMPI devel] MTT Perl client

[Ralph H Castain]

Are we good to go with this changeover? If so, I’ll delete the Perl client from 
the main MTT repo.

> On Sep 14, 2018, at 10:06 AM, Jeff Squyres (jsquyres) via devel 
>  wrote:
> 
> On Sep 14, 2018, at 12:37 PM, Gilles Gouaillardet 
>  wrote:
>> 
>> IIRC mtt-relay is not only a proxy (squid can do that too).
> 
> Probably true.  IIRC, I think mtt-relay was meant to be a 
> dirt-stupid-but-focused-to-just-one-destination relay.
> 
>> mtt results can be manually copied from a cluster behind a firewall, and 
>> then mtt-relay can “upload” these results to mtt.open-MPI.org
> 
> Yes, but then a human has to be involved, which kinda defeats at least one of 
> the goals of MTT.  Using mtt-relay allowed MTT to still function in an 
> automated fashion.
> 
> FWIW, it may not be necessary to convert mtt-relay to python (IIRC that it's 
> protocol agnostic, but like I said: it's been quite a while since I've looked 
> at that code).  It was pretty small and straightforward.  It could also just 
> stay in mtt-legacy.
> 
> -- 
> Jeff Squyres
> jsquy...@cisco.com
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

Re: [OMPI devel] mpirun error when not using span

[Ralph H Castain]

I believe the problem is actually a little different than you described. The 
issue occurs whenever the #procs combined with PE exceeds the number of cores 
on a node. It is caused by the fact that we aren’t considering the PE number 
when mapping processes - we only appear to be looking at it when binding. I’ll 
try to poke at it a bit.


> On Sep 11, 2018, at 9:17 AM, Shrader, David Lee  wrote:
> 
> Here's the xml output from lstopo. Thank you for taking a look!
> David
> 
> From: devel  on behalf of Ralph H Castain 
> 
> Sent: Monday, September 10, 2018 5:12 PM
> To: OpenMPI Devel
> Subject: Re: [OMPI devel] mpirun error when not using span
>  
> Could you please send the output from “lstopo --of xml foo.xml” (the file 
> foo.xml) so I can try to replicate here?
> 
> 
>> On Sep 4, 2018, at 12:35 PM, Shrader, David Lee > <mailto:dshra...@lanl.gov>> wrote:
>> 
>> Hello,
>> 
>> I have run this issue by Howard, and he asked me to forward it on to the 
>> Open MPI devel mailing list. I get an error when trying to use PE=n with 
>> '--map-by numa' and not using span when using more than one node:
>> 
>> [dshrader@ba001 openmpi-3.1.2]$ mpirun -n 16 --map-by numa:PE=4 --bind-to 
>> core --report-bindings true
>> --
>> A request was made to bind to that would result in binding more
>> processes than cpus on a resource:
>> 
>>Bind to: CORE
>>Node:ba001
>>#processes:  2
>>#cpus:   1
>> 
>> You can override this protection by adding the "overload-allowed"
>> option to your binding directive.
>> --
>> 
>> The absolute values of the numbers passed to -n and PE don't really matter; 
>> the error pops up as soon as those numbers are combined in such a way that 
>> an MPI rank ends up on the second node.
>> 
>> If I add the "span" parameter, everything works as expected:
>> 
>> [dshrader@ba001 openmpi-3.1.2]$ mpirun -n 16 --map-by numa:PE=4,span 
>> --bind-to core --report-bindings true
>> [ba002.localdomain:58502] MCW rank 8 bound to socket 0[core 0[hwt 0]], 
>> socket 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], socket 0[core 3[hwt 0]]: 
>> [B/B/B/B/./././././././././././././.][./././././././././././././././././.]
>> [ba002.localdomain:58502] MCW rank 9 bound to socket 0[core 4[hwt 0]], 
>> socket 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], socket 0[core 7[hwt 0]]: 
>> [././././B/B/B/B/./././././././././.][./././././././././././././././././.]
>> [ba002.localdomain:58502] MCW rank 10 bound to socket 0[core 8[hwt 0]], 
>> socket 0[core 9[hwt 0]], socket 0[core 10[hwt 0]], socket 0[core 11[hwt 0]]: 
>> [././././././././B/B/B/B/./././././.][./././././././././././././././././.]
>> [ba002.localdomain:58502] MCW rank 11 bound to socket 0[core 12[hwt 0]], 
>> socket 0[core 13[hwt 0]], socket 0[core 14[hwt 0]], socket 0[core 15[hwt 
>> 0]]: 
>> [././././././././././././B/B/B/B/./.][./././././././././././././././././.]
>> [ba002.localdomain:58502] MCW rank 12 bound to socket 1[core 18[hwt 0]], 
>> socket 1[core 19[hwt 0]], socket 1[core 20[hwt 0]], socket 1[core 21[hwt 
>> 0]]: 
>> [./././././././././././././././././.][B/B/B/B/./././././././././././././.]
>> [ba002.localdomain:58502] MCW rank 13 bound to socket 1[core 22[hwt 0]], 
>> socket 1[core 23[hwt 0]], socket 1[core 24[hwt 0]], socket 1[core 25[hwt 
>> 0]]: 
>> [./././././././././././././././././.][././././B/B/B/B/./././././././././.]
>> [ba002.localdomain:58502] MCW rank 14 bound to socket 1[core 26[hwt 0]], 
>> socket 1[core 27[hwt 0]], socket 1[core 28[hwt 0]], socket 1[core 29[hwt 
>> 0]]: 
>> [./././././././././././././././././.][././././././././B/B/B/B/./././././.]
>> [ba002.localdomain:58502] MCW rank 15 bound to socket 1[core 30[hwt 0]], 
>> socket 1[core 31[hwt 0]], socket 1[core 32[hwt 0]], socket 1[core 33[hwt 
>> 0]]: 
>> [./././././././././././././././././.][././././././././././././B/B/B/B/./.]
>> [ba001.localdomain:11700] MCW rank 0 bound to socket 0[core 0[hwt 0]], 
>> socket 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], socket 0[core 3[hwt 0]]: 
>> [B/B/B/B/./././././././././././././.][./././././././././././././././././.]
>> [ba001.localdomain:11700] MCW rank 1 bound to socket 0[core 4[hwt 0]], 
>> socket 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], socket 0[core 7[hwt 0]]: 
>> [././././B/B/B/B/./././././././././.][./././././././././././././././././.]
>> [ba001.localdomain:11700] MCW rank 2 bound to socket 0[core 8[h

Re: [OMPI devel] Gentle reminder: sign up for the face to face

[Ralph H Castain]

Done!

> On Feb 26, 2019, at 8:33 AM, Brice Goglin  wrote:
> 
> Hello Jeff
> 
> Looks like I am not allowed to modify the page but I'll be at the meeting ;)
> 
> Brice
> 
> 
> 
> Le 26/02/2019 à 17:13, Jeff Squyres (jsquyres) via devel a écrit :
>> Gentle reminder to please sign up for the face-to-face meeting and add your 
>> items to the wiki:
>> 
>>https://github.com/open-mpi/ompi/wiki/Meeting-2019-04
>> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

Re: [MTT devel] Documentation update of gh-pages failed

[Ralph H Castain]

Travis indicates they are having a “minor service outage”, so I’m guessing that 
is the cause of the problem.


> On Feb 28, 2019, at 7:47 AM, Rezanka, Deb via mtt-devel 
>  wrote:
> 
> I have a cleaned up version of gh-pages on my repo ready to go but I don’t 
> know how to do a pull request for a specific branch like that.  It has not 
> been process by travis yet. 
>  
> The problem is a regular pull request that altered the documentation and 
> should have caused travis to rebuild and deploy the changes to gh-pages. For 
> some reason the $GH_TOKEN (see scripts/deploy.sh) seems to be invalid.  This 
> is run on the Travis CI build so I don’t know how to check that. Can you help?
> deb
>  
>  
> From: mtt-devel  on behalf of Ralph H 
> Castain 
> Reply-To: Development list for the MPI Testing Tool 
> 
> Date: Thursday, February 28, 2019 at 8:42 AM
> To: Development list for the MPI Testing Tool 
> Subject: Re: [MTT devel] Documentation update of gh-pages failed
>  
> I apologize - I haven’t been closely following this. Did you do some kind of 
> “git rm -rf” on the contents of gh-pages? I don’t see anything in the commit 
> history for that branch other than a last auto-update 2 days ago.
>  
> 
> 
> On Feb 28, 2019, at 7:34 AM, Rezanka, Deb via mtt-devel 
> mailto:mtt-devel@lists.open-mpi.org>> wrote:
>  
> Hi,  
>  
> the travis deployment of updated documentation failed with:
>  
> Cloning into 'gh-pages'...
> PUSHING CHANGES
> [gh-pages 408b555] Deploy updated open-mpi/mtt to gh-pages
> 409 files changed, 991 insertions(+), 581 deletions(-)
> rewrite pages/user_guide.md (100%)
> remote: Anonymous access to open-mpi/mtt.git denied.
> fatal: Authentication failed for 'https://@github.com/open-mpi/mtt.git/ 
> <https://github.com/open-mpi/mtt.git/>'
> Script failed with status 128
>  
> Does anyone know why the Authentication would have failed? Nothing changed in 
> the .travis.yml file.
>  
> Deb Rezanka
> LANL
> ___
> mtt-devel mailing list
> mtt-devel@lists.open-mpi.org <mailto:mtt-devel@lists.open-mpi.org>
> https://lists.open-mpi.org/mailman/listinfo/mtt-devel 
> <https://lists.open-mpi.org/mailman/listinfo/mtt-devel>
>  
> ___
> mtt-devel mailing list
> mtt-devel@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/mtt-devel

___
mtt-devel mailing list
mtt-devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/mtt-devel

Re: [MTT devel] Documentation update of gh-pages failed

[Ralph H Castain]

I apologize - I haven’t been closely following this. Did you do some kind of 
“git rm -rf” on the contents of gh-pages? I don’t see anything in the commit 
history for that branch other than a last auto-update 2 days ago.


> On Feb 28, 2019, at 7:34 AM, Rezanka, Deb via mtt-devel 
>  wrote:
> 
> Hi,  
>  
> the travis deployment of updated documentation failed with:
>  
> Cloning into 'gh-pages'...
> PUSHING CHANGES
> [gh-pages 408b555] Deploy updated open-mpi/mtt to gh-pages
> 409 files changed, 991 insertions(+), 581 deletions(-)
> rewrite pages/user_guide.md (100%)
> remote: Anonymous access to open-mpi/mtt.git denied.
> fatal: Authentication failed for 'https://@github.com/open-mpi/mtt.git/ 
> '
> Script failed with status 128
>  
> Does anyone know why the Authentication would have failed? Nothing changed in 
> the .travis.yml file.
>  
> Deb Rezanka
> LANL
> ___
> mtt-devel mailing list
> mtt-devel@lists.open-mpi.org 
> https://lists.open-mpi.org/mailman/listinfo/mtt-devel 
> 
___
mtt-devel mailing list
mtt-devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/mtt-devel

Re: [OMPI devel] rml/ofi component broken in v4.0.x and v3.1.x

[Ralph H Castain]

I would recommend just removing it - frankly, I’m surprised it is in there as 
the code was deemed non-production-ready.


> On Feb 14, 2019, at 5:11 PM, Gilles Gouaillardet  wrote:
> 
> Folks,
> 
> 
> The rml/ofi component has been removed from master.
> 
> Then common/ofi was later removed from master and mtl/ofi configury component 
> was revamped not to depend on common/ofi configury stuff.
> 
> Only the latter change was backported to the release branches.
> 
> The issue is that rml/ofi is still in v4.0.x and v3.1.x (it has never been in 
> v3.0.x) and is broken since its configury still relies on
> 
> (now removed) common/ofi configury.
> 
> 
> I guess the right fix is to update the rml/ofi configury in the release 
> branches, but do we really care ?
> 
> If not, can we simply remove the rml/ofi component (e.g. cherry-pick 
> https://github.com/open-mpi/ompi/commit/8794077520b4b4ae86be3a09cfc1abbf7bcab8ad)
>  ?
> 
> 
> Cheers,
> 
> 
> Gilles
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

Re: [OMPI devel] MPI Reduce Without a Barrier

[Ralph H Castain]

There is a coll/sync component that will automatically inject those barriers 
for you so you don’t have to add them to your code. Controlled by MCA param:

coll_sync_barrier_before: Do a synchronization before each Nth collective

coll_sync_barrier_after: Do a synchronization after each Nth collective

Ralph


> On Apr 15, 2019, at 8:59 AM, Nathan Hjelm via devel 
>  wrote:
> 
> If you do that it may run out of resources and deadlock or crash. I recommend 
> either 1) adding a barrier every 100 iterations, 2) using allreduce, or 3) 
> enable coll/sync (which essentially does 1). Honestly, 2 is probably the 
> easiest option and depending on how large you run may not be any slower than 
> 1 or 3.
> 
> -Nathan
> 
>> On Apr 15, 2019, at 9:53 AM, Saliya Ekanayake  wrote:
>> 
>> Hi Devs,
>> 
>> When doing MPI_Reduce in a loop (collecting on Rank 0), is it the correct 
>> understanding that ranks other than root (0 in this case) will pass the 
>> collective as soon as their data is written to MPI buffers without waiting 
>> for all of them to be received at the root?
>> 
>> If that's the case then what would happen (semantically) if we execute 
>> MPI_Reduce in a loop without a barrier allowing non-root ranks to hit the 
>> collective multiple times while the root will be processing an earlier 
>> reduce? For example, the root can be in the first reduce invocation, while 
>> another rank is in the second the reduce invocation.
>> 
>> Thank you,
>> Saliya
>> 
>> -- 
>> Saliya Ekanayake, Ph.D
>> Postdoctoral Scholar
>> Performance and Algorithms Research (PAR) Group
>> Lawrence Berkeley National Laboratory
>> Phone: 510-486-5772
>> 
>> ___
>> devel mailing list
>> devel@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/devel
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

Re: [OMPI devel] MPI Reduce Without a Barrier

[Ralph H Castain]

Not exactly. The problem is that rank=0 initially falls behind because it is 
doing more work - i.e., it has to receive all the buffers and do something with 
them. As a result, it doesn’t get to post the next allreduce before the 
messages from the other participants arrive - which means that rank=0 has to 
stick those messages into the “unexpected message” queue. As iterations go by, 
the memory consumed by that queue gets bigger and bigger, causing rank=0 to run 
slower and slower…until you run out of memory and it aborts.

Adding the occasional barrier resolves the problem by letting rank=0 catch up 
and release the memory in the “unexpected message” queue.


> On Apr 15, 2019, at 1:33 PM, Saliya Ekanayake  wrote:
> 
> Thank you, Nathan. Could you elaborate a bit on what happens internally? From 
> your answer it seems, the program will still produce the correct output at 
> the end but it'll use more resources. 
> 
> On Mon, Apr 15, 2019 at 9:00 AM Nathan Hjelm via devel 
> mailto:devel@lists.open-mpi.org>> wrote:
> If you do that it may run out of resources and deadlock or crash. I recommend 
> either 1) adding a barrier every 100 iterations, 2) using allreduce, or 3) 
> enable coll/sync (which essentially does 1). Honestly, 2 is probably the 
> easiest option and depending on how large you run may not be any slower than 
> 1 or 3.
> 
> -Nathan
> 
> > On Apr 15, 2019, at 9:53 AM, Saliya Ekanayake  > > wrote:
> > 
> > Hi Devs,
> > 
> > When doing MPI_Reduce in a loop (collecting on Rank 0), is it the correct 
> > understanding that ranks other than root (0 in this case) will pass the 
> > collective as soon as their data is written to MPI buffers without waiting 
> > for all of them to be received at the root?
> > 
> > If that's the case then what would happen (semantically) if we execute 
> > MPI_Reduce in a loop without a barrier allowing non-root ranks to hit the 
> > collective multiple times while the root will be processing an earlier 
> > reduce? For example, the root can be in the first reduce invocation, while 
> > another rank is in the second the reduce invocation.
> > 
> > Thank you,
> > Saliya
> > 
> > -- 
> > Saliya Ekanayake, Ph.D
> > Postdoctoral Scholar
> > Performance and Algorithms Research (PAR) Group
> > Lawrence Berkeley National Laboratory
> > Phone: 510-486-5772
> > 
> > ___
> > devel mailing list
> > devel@lists.open-mpi.org 
> > https://lists.open-mpi.org/mailman/listinfo/devel 
> > 
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org 
> https://lists.open-mpi.org/mailman/listinfo/devel 
> 
> 
> 
> -- 
> Saliya Ekanayake, Ph.D
> Postdoctoral Scholar
> Performance and Algorithms Research (PAR) Group
> Lawrence Berkeley National Laboratory
> Phone: 510-486-5772
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel