Re: [OMPI devel] [Open MPI Announce] Open MPI v5.0.0rc13 is available for testing

[Christoph Niethammer via devel]

Hello Austen,

Unfortunately I could not attend the last telco and therefore I do not know if 
this was discussed.
So I'd like to bring up attention for 
https://github.com/mpi-forum/mpi-issues/issues/765
It is not voted on yet but seems to have support to go in.
As Partitioned communication is introduced in 5.0 Open MPI could do it possibly 
'right' from the start ... 

Best
Christoph

- Original Message -
From: "Austen Lauria via announce" 
To: "Open MPI Developers" , 
annou...@lists.open-mpi.org
Cc: "Austen Lauria" 
Sent: Saturday, 30 September, 2023 00:09:22
Subject: [Open MPI Announce] Open MPI v5.0.0rc13 is available for testing

Open MPI v5.0.0rc13 is now available for testing at 
https://www.open-mpi.org/software/ompi/v5.0/ .

NOTE: This is expected to be the last v5.0.0 release candidate before release 
unless any issues arise.

Please test and send feedback either via the user mailing lists or create an 
issue at https://github.com/open-mpi/ompi/issues/ .

See https://docs.open-mpi.org/en/v5.0.x/news/news-v5.0.x.html for a list of 
changes since rc12.

Thank you,
v5.0 Release Managers

___
announce mailing list
annou...@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/announce

[OMPI devel] Open MPI + UCX paramater tunging

[Christoph Niethammer via devel]

Hello,

With UCX one has to define UCX environment variables to tune the runtime 
behaviour beside the commonly used mca parameters.
This can lead - at least for me - to confusion as some of them seem to be 
"identical".
Also, to my current experience only the UCX variables seem often to have an 
effect.
Example: btl_*_eager_limit vs. UCX_RNDV_THRESH


Having now an external environment I am not totally sure, what the role of the 
mca parameters vs the UCX variables is (and the corresponding two tools 
ompi_info/ucx_info).

For the moment at least I would welcome if the Open MPI documentation could 
include some more information and maybe best practices about this.


Best
Christoph Niethammer

--

Christoph Niethammer
High Performance Computing Center Stuttgart (HLRS)
Nobelstrasse 19
70569 Stuttgart

Tel: ++49(0)711-685-87203
email: nietham...@hlrs.de
http://www.hlrs.de/people/niethammer

Re: [OMPI devel] MTT Perl client

[Christoph Niethammer]

Works for the installation at HLRS.

Short note/question: I am using the mtt-relay script. This is written in perl. 
Is there a python based replacement?

Best
Christoph Niethammer

- Mensaje original -
De: "Open MPI Developers" 
Para: "Open MPI Developers" 
CC: "Jeff Squyres" 
Enviados: Martes, 11 de Septiembre 2018 20:37:40
Asunto: Re: [OMPI devel] MTT Perl client

Works for me.

> On Sep 11, 2018, at 12:35 PM, Ralph H Castain  wrote:
> 
> Hi folks
> 
> Per today’s telecon, I have moved the Perl MTT client into its own 
> repository: https://github.com/open-mpi/mtt-legacy. All the Python client 
> code has been removed from that repo.
> 
> The original MTT repo remains at https://github.com/open-mpi/mtt. I have a PR 
> to remove all the Perl client code and associated libs/modules from that 
> repo. We won’t commit it until people have had a chance to switch to the 
> mtt-legacy repo and verify that things still work for them.
> 
> Please let us know if mtt-legacy is okay or has a problem.
> 
> Thanks
> Ralph
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/devel


-- 
Jeff Squyres
jsquy...@cisco.com

___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel
___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

Re: [OMPI devel] Upcoming nightly tarball URL changes

[Christoph Niethammer]

Hello,

After the change of the nighly tarball URL I have issues with mtt getting the 
checksum files:

...
Running command: wget --no-check-certificate -nv
   https://download.open-mpi.org/nightly/open-mpi/v2.x/md5sums.txt
md5sums.txt
https://download.open-mpi.org/nightly/open-mpi/v2.x/md5sums.txt:
   2018-03-07 23:16:19 ERROR 403: Forbidden.
...

Fetching the tarballs itself works fine.
Anything else I have to change in the setup?


Best

Christoph Niethammer


- Original Message -
From: "Open MPI Developers" <devel@lists.open-mpi.org>
To: "Open MPI Developers" <devel@lists.open-mpi.org>
Cc: "Barrett, Brian" <bbarr...@amazon.com>
Sent: Monday, February 26, 2018 11:09:42 PM
Subject: [OMPI devel] Upcoming nightly tarball URL changes

On Sunday, March 18th, the Open MPI team is going to make a change in where 
nightly tarballs are stored that will likely impact MTT test configuration.  If 
you have an automated system (including MTT) that fetches nightly tarballs, you 
will likely need to make a change in the next two weeks to avoid breakage.

Over the last year, we’ve been working behind the scenes to improve some of the 
workflows around building and releasing tarballs (both nightly and release), 
including moving them out of our web tree and into Amazon S3 (and the Amazon 
CloudFront CDN for downloads).  It’s time to make the next step, moving the 
nightly tarballs out of the web tree.

As of December, the nightly tarball builder uploads the build results to S3, 
which can be accessed from:

   https://download.open-mpi.org/nightly/open-mpi//

So to get the latest 3.0 nightly tarball version, you’d download 
https://download.open-mpi.org/nightly/open-mpi/v3.0.x/latest_snapshot.txt.  The 
build artifact tree under https://download.open-mpi.org/nightly/open-mpi/ 
matches the tree under https://www.open-mpi.org/nightly/, so scripts should 
work with only the change in root of the tree.

On Sunday, March 18th, we’ll stop mirroring tarballs to www.open-mpi.org and 
rewrite the web pages to direct users to download.open-mpi.org/ for downloading 
nightly tarballs of Open MPI.  We will add redirects from the old tarballs and 
latest_snapshot.txt files to the new location, but not all clients follow 
redirects by default.  So we’re asking everyone to proactively update their MTT 
scripts.  It should just be updating lines like:

  ompi_snapshot_url = https://www.open-mpi.org/nightly/master

to read:

  ompi_snapshot_url = https://download.open-mpi.org/nightly/open-mpi/master


Thanks,

Brian
___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel
___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

Re: [OMPI devel] orte-clean not cleaning left over temporary I/O files in /tmp

[Christoph Niethammer]

Hi Howard,

You find the pull request under https://github.com/open-mpi/ompi/pull/3739

Best
Christoph

- Original Message -
From: "Howard Pritchard" <hpprit...@gmail.com>
To: "Open MPI Developers" <devel@lists.open-mpi.org>
Sent: Thursday, June 22, 2017 4:42:14 PM
Subject: Re: [OMPI devel] orte-clean not cleaning left over temporary I/O files 
in /tmp

Hi Chris 

Please go ahead and open a PR for master and I'll open corresponding ones for 
the release branches. 

Howard 

Christoph Niethammer < [ mailto:nietham...@hlrs.de | nietham...@hlrs.de ] > 
schrieb am Do. 22. Juni 2017 um 01:10: 


Hi Howard, 

Sorry, missed the new license policy. I added a Sign-off now. 
Shall I open a pull request? 

Best 
Christoph 

- Original Message - 
From: "Howard Pritchard" < [ mailto:hpprit...@gmail.com | hpprit...@gmail.com ] 
> 
To: "Open MPI Developers" < [ mailto:devel@lists.open-mpi.org | 
devel@lists.open-mpi.org ] > 
Sent: Wednesday, June 21, 2017 5:57:05 PM 
Subject: Re: [OMPI devel] orte-clean not cleaning left over temporary I/O files 
in /tmp 

Hi Chris, 

Sorry for being a bit picky, but could you add a sign-off to the commit 
message? 
I'm not suppose to manually add it for you. 

Thanks, 

Howard 


2017-06-21 9:45 GMT-06:00 Howard Pritchard < [ mailto: [ 
mailto:hpprit...@gmail.com | hpprit...@gmail.com ] | [ 
mailto:hpprit...@gmail.com | hpprit...@gmail.com ] ] > : 



Hi Chris, 

Thanks very much for the patch! 

Howard 


2017-06-21 9:43 GMT-06:00 Christoph Niethammer < [ mailto: [ 
mailto:nietham...@hlrs.de | nietham...@hlrs.de ] | [ mailto:nietham...@hlrs.de 
| nietham...@hlrs.de ] ] > : 


Hello Ralph, 

Thanks for the update on this issue. 

I used the latest master (c38866eb3929339147259a3a46c6fc815720afdb). 

The behaviour is still the same: aborting before MPI_File_close leaves 
/tmp/OMPI_*.sm files. 
These are not removed by your updated orte-clean. 

I now seeked for the origin of these files and it seems to be in 
ompi/mca/sharedfp/sm/sharedfp_sm_file_open.c:154 
where also a left over TODO note some lines above is mentioning the need for a 
correct directory. 

I would suggest updating the path there to be under the 
 directory which is cleaned by orte-clean, 
see 

[ [ 
https://github.com/cniethammer/ompi/commit/2aedf6134813299803628e7d6856a3b781542c02
 | 
https://github.com/cniethammer/ompi/commit/2aedf6134813299803628e7d6856a3b781542c02
 ] | [ 
https://github.com/cniethammer/ompi/commit/2aedf6134813299803628e7d6856a3b781542c02
 | 
https://github.com/cniethammer/ompi/commit/2aedf6134813299803628e7d6856a3b781542c02
 ] ] 

Best 
Christoph 

- Original Message - 
From: "Ralph Castain" < [ mailto: [ mailto:r...@open-mpi.org | 
r...@open-mpi.org ] | [ mailto:r...@open-mpi.org | r...@open-mpi.org ] ] > 
To: "Open MPI Developers" < [ mailto: [ mailto:devel@lists.open-mpi.org | 
devel@lists.open-mpi.org ] | [ mailto:devel@lists.open-mpi.org | 
devel@lists.open-mpi.org ] ] > 
Sent: Wednesday, June 21, 2017 4:33:29 AM 
Subject: Re: [OMPI devel] orte-clean not cleaning left over temporary I/O files 
in /tmp 

I updated orte-clean in master, and for v3.0, so it cleans up all both current 
and legacy session directory files as well as any pmix artifacts. I don’t see 
any files named OMPI_*.sm, though that might be something from v2.x? I don’t 
recall us ever making files of that name before - anything we make should be 
under the session directory, not directly in /tmp. 

> On May 9, 2017, at 2:10 AM, Christoph Niethammer < [ mailto: [ 
> mailto:nietham...@hlrs.de | nietham...@hlrs.de ] | [ 
> mailto:nietham...@hlrs.de | nietham...@hlrs.de ] ] > wrote: 
> 
> Hi, 
> 
> I am using Open MPI 2.1.0. 
> 
> Best 
> Christoph 
> 
> - Original Message - 
> From: "Ralph Castain" < [ mailto: [ mailto:r...@open-mpi.org | 
> r...@open-mpi.org ] | [ mailto:r...@open-mpi.org | r...@open-mpi.org ] ] > 
> To: "Open MPI Developers" < [ mailto: [ mailto:devel@lists.open-mpi.org | 
> devel@lists.open-mpi.org ] | [ mailto:devel@lists.open-mpi.org | 
> devel@lists.open-mpi.org ] ] > 
> Sent: Monday, May 8, 2017 6:28:42 PM 
> Subject: Re: [OMPI devel] orte-clean not cleaning left over temporary I/O 
> files in /tmp 
> 
> What version of OMPI are you using? 
> 
>> On May 8, 2017, at 8:56 AM, Christoph Niethammer < [ mailto: [ 
>> mailto:nietham...@hlrs.de | nietham...@hlrs.de ] | [ 
>> mailto:nietham...@hlrs.de | nietham...@hlrs.de ] ] > wrote: 
>> 
>> Hello 
>> 
>> According to the manpage "...orte-clean attempts to clean up any processes 
>> and files left over from Open MPI jobs that were run in the past as well as 
>> any currently running jobs. This includes OMPI infrastructure and helpe

Re: [OMPI devel] orte-clean not cleaning left over temporary I/O files in /tmp

[Christoph Niethammer]

Hi Howard,

Sorry, missed the new license policy. I added a Sign-off now.
Shall I open a pull request?

Best
Christoph

- Original Message -
From: "Howard Pritchard" <hpprit...@gmail.com>
To: "Open MPI Developers" <devel@lists.open-mpi.org>
Sent: Wednesday, June 21, 2017 5:57:05 PM
Subject: Re: [OMPI devel] orte-clean not cleaning left over temporary I/O files 
in /tmp

Hi Chris, 

Sorry for being a bit picky, but could you add a sign-off to the commit 
message? 
I'm not suppose to manually add it for you. 

Thanks, 

Howard 


2017-06-21 9:45 GMT-06:00 Howard Pritchard < [ mailto:hpprit...@gmail.com | 
hpprit...@gmail.com ] > : 



Hi Chris, 

Thanks very much for the patch! 

Howard 


2017-06-21 9:43 GMT-06:00 Christoph Niethammer < [ mailto:nietham...@hlrs.de | 
nietham...@hlrs.de ] > : 


Hello Ralph, 

Thanks for the update on this issue. 

I used the latest master (c38866eb3929339147259a3a46c6fc815720afdb). 

The behaviour is still the same: aborting before MPI_File_close leaves 
/tmp/OMPI_*.sm files. 
These are not removed by your updated orte-clean. 

I now seeked for the origin of these files and it seems to be in 
ompi/mca/sharedfp/sm/sharedfp_sm_file_open.c:154 
where also a left over TODO note some lines above is mentioning the need for a 
correct directory. 

I would suggest updating the path there to be under the 
 directory which is cleaned by orte-clean, 
see 

[ 
https://github.com/cniethammer/ompi/commit/2aedf6134813299803628e7d6856a3b781542c02
 | 
https://github.com/cniethammer/ompi/commit/2aedf6134813299803628e7d6856a3b781542c02
 ] 

Best 
Christoph 

- Original Message - 
From: "Ralph Castain" < [ mailto:r...@open-mpi.org | r...@open-mpi.org ] > 
To: "Open MPI Developers" < [ mailto:devel@lists.open-mpi.org | 
devel@lists.open-mpi.org ] > 
Sent: Wednesday, June 21, 2017 4:33:29 AM 
Subject: Re: [OMPI devel] orte-clean not cleaning left over temporary I/O files 
in /tmp 

I updated orte-clean in master, and for v3.0, so it cleans up all both current 
and legacy session directory files as well as any pmix artifacts. I don’t see 
any files named OMPI_*.sm, though that might be something from v2.x? I don’t 
recall us ever making files of that name before - anything we make should be 
under the session directory, not directly in /tmp. 

> On May 9, 2017, at 2:10 AM, Christoph Niethammer < [ 
> mailto:nietham...@hlrs.de | nietham...@hlrs.de ] > wrote: 
> 
> Hi, 
> 
> I am using Open MPI 2.1.0. 
> 
> Best 
> Christoph 
> 
> - Original Message - 
> From: "Ralph Castain" < [ mailto:r...@open-mpi.org | r...@open-mpi.org ] > 
> To: "Open MPI Developers" < [ mailto:devel@lists.open-mpi.org | 
> devel@lists.open-mpi.org ] > 
> Sent: Monday, May 8, 2017 6:28:42 PM 
> Subject: Re: [OMPI devel] orte-clean not cleaning left over temporary I/O 
> files in /tmp 
> 
> What version of OMPI are you using? 
> 
>> On May 8, 2017, at 8:56 AM, Christoph Niethammer < [ 
>> mailto:nietham...@hlrs.de | nietham...@hlrs.de ] > wrote: 
>> 
>> Hello 
>> 
>> According to the manpage "...orte-clean attempts to clean up any processes 
>> and files left over from Open MPI jobs that were run in the past as well as 
>> any currently running jobs. This includes OMPI infrastructure and helper 
>> commands, any processes that were spawned as part of the job, and any 
>> temporary files...". 
>> 
>> If I now have a program which calls MPI_File_open, MPI_File_write and 
>> MPI_Abort() in order, I get left over files /tmp/OMPI_*.sm. 
>> Running orte-clean does not remove them. 
>> 
>> Is this a bug or a feature? 
>> 
>> Best 
>> Christoph Niethammer 
>> 
>> -- 
>> 
>> Christoph Niethammer 
>> High Performance Computing Center Stuttgart (HLRS) 
>> Nobelstrasse 19 
>> 70569 Stuttgart 
>> 
>> Tel: [ tel:%2B%2B49%280%29711-685-87203 | ++49(0)711-685-87203 ] 
>> email: [ mailto:nietham...@hlrs.de | nietham...@hlrs.de ] 
>> [ http://www.hlrs.de/people/niethammer | 
>> http://www.hlrs.de/people/niethammer ] 
>> ___ 
>> devel mailing list 
>> [ mailto:devel@lists.open-mpi.org | devel@lists.open-mpi.org ] 
>> [ https://rfd.newmexicoconsortium.org/mailman/listinfo/devel | 
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel ] 
> 
> ___ 
> devel mailing list 
> [ mailto:devel@lists.open-mpi.org | devel@lists.open-mpi.org ] 
> [ https://rfd.newmexicoconsortium.org/mailman/listinfo/devel | 
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel ] 
> ___

Re: [OMPI devel] orte-clean not cleaning left over temporary I/O files in /tmp

[Christoph Niethammer]

Hello Ralph,

Thanks for the update on this issue.

I used the latest master (c38866eb3929339147259a3a46c6fc815720afdb).

The behaviour is still the same: aborting before MPI_File_close leaves 
/tmp/OMPI_*.sm files.
These are not removed by your updated orte-clean.

I now seeked for the origin of these files and it seems to be in 
ompi/mca/sharedfp/sm/sharedfp_sm_file_open.c:154
where also a left over TODO note some lines above is mentioning the need for a 
correct directory.

I would suggest updating the path there to be under the 
 directory which is cleaned by orte-clean, 
see

 
https://github.com/cniethammer/ompi/commit/2aedf6134813299803628e7d6856a3b781542c02

Best
Christoph

- Original Message -
From: "Ralph Castain" <r...@open-mpi.org>
To: "Open MPI Developers" <devel@lists.open-mpi.org>
Sent: Wednesday, June 21, 2017 4:33:29 AM
Subject: Re: [OMPI devel] orte-clean not cleaning left over temporary I/O files 
in /tmp

I updated orte-clean in master, and for v3.0, so it cleans up all both current 
and legacy session directory files as well as any pmix artifacts. I don’t see 
any files named OMPI_*.sm, though that might be something from v2.x? I don’t 
recall us ever making files of that name before - anything we make should be 
under the session directory, not directly in /tmp.

> On May 9, 2017, at 2:10 AM, Christoph Niethammer <nietham...@hlrs.de> wrote:
> 
> Hi,
> 
> I am using Open MPI 2.1.0. 
> 
> Best
> Christoph
> 
> - Original Message -
> From: "Ralph Castain" <r...@open-mpi.org>
> To: "Open MPI Developers" <devel@lists.open-mpi.org>
> Sent: Monday, May 8, 2017 6:28:42 PM
> Subject: Re: [OMPI devel] orte-clean not cleaning left over temporary I/O 
> files in /tmp
> 
> What version of OMPI are you using?
> 
>> On May 8, 2017, at 8:56 AM, Christoph Niethammer <nietham...@hlrs.de> wrote:
>> 
>> Hello
>> 
>> According to the manpage "...orte-clean attempts to clean up any processes 
>> and files left over from Open MPI jobs that were run in the past as well as 
>> any currently running jobs. This includes OMPI infrastructure and helper 
>> commands, any processes that were spawned as part of the job, and any 
>> temporary files...".
>> 
>> If I now have a program which calls MPI_File_open, MPI_File_write and 
>> MPI_Abort() in order, I get left over files /tmp/OMPI_*.sm.
>> Running orte-clean does not remove them.
>> 
>> Is this a bug or a feature?
>> 
>> Best
>> Christoph Niethammer
>> 
>> --
>> 
>> Christoph Niethammer
>> High Performance Computing Center Stuttgart (HLRS)
>> Nobelstrasse 19
>> 70569 Stuttgart
>> 
>> Tel: ++49(0)711-685-87203
>> email: nietham...@hlrs.de
>> http://www.hlrs.de/people/niethammer
>> ___
>> devel mailing list
>> devel@lists.open-mpi.org
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
> 
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] orte-clean not cleaning left over temporary I/O files in /tmp

[Christoph Niethammer]

Hi,

I am using Open MPI 2.1.0. 

Best
Christoph

- Original Message -
From: "Ralph Castain" <r...@open-mpi.org>
To: "Open MPI Developers" <devel@lists.open-mpi.org>
Sent: Monday, May 8, 2017 6:28:42 PM
Subject: Re: [OMPI devel] orte-clean not cleaning left over temporary I/O files 
in /tmp

What version of OMPI are you using?

> On May 8, 2017, at 8:56 AM, Christoph Niethammer <nietham...@hlrs.de> wrote:
> 
> Hello
> 
> According to the manpage "...orte-clean attempts to clean up any processes 
> and files left over from Open MPI jobs that were run in the past as well as 
> any currently running jobs. This includes OMPI infrastructure and helper 
> commands, any processes that were spawned as part of the job, and any 
> temporary files...".
> 
> If I now have a program which calls MPI_File_open, MPI_File_write and 
> MPI_Abort() in order, I get left over files /tmp/OMPI_*.sm.
> Running orte-clean does not remove them.
> 
> Is this a bug or a feature?
> 
> Best
> Christoph Niethammer
> 
> --
> 
> Christoph Niethammer
> High Performance Computing Center Stuttgart (HLRS)
> Nobelstrasse 19
> 70569 Stuttgart
> 
> Tel: ++49(0)711-685-87203
> email: nietham...@hlrs.de
> http://www.hlrs.de/people/niethammer
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

[OMPI devel] orte-clean not cleaning left over temporary I/O files in /tmp

[Christoph Niethammer]

Hello

According to the manpage "...orte-clean attempts to clean up any processes and 
files left over from Open MPI jobs that were run in the past as well as any 
currently running jobs. This includes OMPI infrastructure and helper commands, 
any processes that were spawned as part of the job, and any temporary files...".

If I now have a program which calls MPI_File_open, MPI_File_write and 
MPI_Abort() in order, I get left over files /tmp/OMPI_*.sm.
Running orte-clean does not remove them.

Is this a bug or a feature?

Best
Christoph Niethammer

--

Christoph Niethammer
High Performance Computing Center Stuttgart (HLRS)
Nobelstrasse 19
70569 Stuttgart

Tel: ++49(0)711-685-87203
email: nietham...@hlrs.de
http://www.hlrs.de/people/niethammer
___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] v2.1.0rc1 has been released

[Christoph Niethammer]

Hi Gilles,

Ok. So I did a quick test.
Using romio314 I get the correct file content, using ompio the content is wrong:

$ hexdump c-xdr.dat 
000  e2ff 3040 0040    
00c

$ mpirun --mca io romio314 -np 1 mpi-io-external32
$ hexdump mpi-io-external32.dat
000  e2ff 3040 0040    
00c

$ mpirun --mca io ompio -np 1 mpi-io-external32
$ hexdump mpi-io-external32.dat
000 ffe2    4000 4030  
00c


Best
Christoph


- Original Message -
From: "Gilles Gouaillardet" <gilles.gouaillar...@gmail.com>
To: "Open MPI Developers" <devel@lists.open-mpi.org>
Sent: Tuesday, February 28, 2017 10:39:56 AM
Subject: Re: [OMPI devel] v2.1.0rc1 has been released

Hi Christoph, 

these two look verysimilar indeed ! 
did you try rc1 with your test case ? 
note you need to 
mpirun --mca io romio314 ... 

then you should definetely be credited for that too 

Cheers, 

Gilles 

On Tuesday, February 28, 2017, Christoph Niethammer < [ 
mailto:nietham...@hlrs.de | nietham...@hlrs.de ] > wrote: 


Hi Gilles, 

Is this the same issue I reported 4/29/2014: 'Wrong Endianness in Open MPI for 
external32 representation'? 
[ https://www.mail-archive.com/devel@lists.open-mpi.org/msg14698.html | 
https://www.mail-archive.com/devel@lists.open-mpi.org/msg14698.html ] 

Best 
Christoph 

- Original Message - 
From: "Gilles Gouaillardet" < [ javascript-blocked:; | gil...@rist.or.jp ] > 
To: "Open MPI Developers" < [ javascript-blocked:; | devel@lists.open-mpi.org ] 
> 
Sent: Tuesday, February 28, 2017 3:38:10 AM 
Subject: Re: [OMPI devel] v2.1.0rc1 has been released 

Jeff, 


about the external32 thing 

"Fix external32 representation in the romio314 module. 

note that for now, external32 representation is not correctly supported 
by the ompio module 

Thanks Thomas Gastine for bringing this to our attention" 


the external32 representation makes MPI-IO write data into an endian 
neutral (e.g. big endian) format 


Cheers, 


Gilles 


On 2/26/2017 11:47 PM, Jeff Squyres (jsquyres) wrote: 
> Fix external32 support 
> ^^^ JMS probably need to explain this more 
> ^^^ JMS is there a user to cite here? 

___ 
devel mailing list 
[ javascript-blocked:; | devel@lists.open-mpi.org ] 
[ https://rfd.newmexicoconsortium.org/mailman/listinfo/devel | 
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel ] 
___ 
devel mailing list 
[ javascript-blocked:; | devel@lists.open-mpi.org ] 
[ https://rfd.newmexicoconsortium.org/mailman/listinfo/devel | 
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel ] 

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] v2.1.0rc1 has been released

[Christoph Niethammer]

Hi Gilles,

Is this the same issue I reported 4/29/2014:  'Wrong Endianness in Open MPI for 
external32 representation'?
https://www.mail-archive.com/devel@lists.open-mpi.org/msg14698.html

Best
Christoph

- Original Message -
From: "Gilles Gouaillardet" 
To: "Open MPI Developers" 
Sent: Tuesday, February 28, 2017 3:38:10 AM
Subject: Re: [OMPI devel] v2.1.0rc1 has been released

Jeff,


about the external32 thing

"Fix external32 representation in the romio314 module.

note that for now, external32 representation is not correctly supported 
by the ompio module

Thanks Thomas Gastine for bringing this to our attention"


the external32 representation makes MPI-IO write data into an endian 
neutral (e.g. big endian) format


Cheers,


Gilles


On 2/26/2017 11:47 PM, Jeff Squyres (jsquyres) wrote:
> Fix external32 support
>^^^ JMS probably need to explain this more
>^^^ JMS is there a user to cite here?

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

[OMPI devel] Current progress threads status in Open MPI

[Christoph Niethammer]

Hello,

I was wondering, what is the current status of progress threads in Open MPI.
As far as I know it was on the agenda for 1.10.x to be re-enabled after it's 
removal in 1.8.x.

Now we have Open MPI 2.0.x. How to enable/disable it as the old configure 
options are not recognized any more:
  configure: WARNING: unrecognized options: --enable-progress-threads, 
--enable-multi-threads
This is wired to me, as I see a lot of [ test "$enable_progress_threads" = 
"yes" ] in the configure scripts.

My nighly 2.0.x mtt build shows ORTE progress enabled by default. But what 
about "OMPI progress"?
  ompi_info --all --all | grep -i "Thread support"
  Thread support: posix (MPI_THREAD_MULTIPLE: no, OPAL support: yes, OMPI 
progress: no, ORTE progress: yes, Event lib: yes)

Best regards
Christoph Niethammer
___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] Performance analysis proposal

[Christoph Niethammer]

Hi Artem,

Thanks for the links. I tested now with 1.10.3, 2.0.0+sm/vader performance 
regression patch under
https://github.com/hjelmn/ompi/commit/4079eec9749e47dddc6acc9c0847b3091601919f.patch
and master. I will do the 2.0.1rc in the next days as well.

Is it possible to add me to the results repository at github or should I fork 
and request you to pull?

Best
Christoph


- Original Message -
From: "Artem Polyakov" <artpo...@gmail.com>
To: "Open MPI Developers" <devel@lists.open-mpi.org>
Sent: Tuesday, August 23, 2016 5:13:30 PM
Subject: Re: [OMPI devel] Performance analysis proposal

Hi, Christoph 

Please, check https://github.com/open-mpi/ompi/wiki/Request-refactoring-test 
for the testing methodology and 
https://github.com/open-mpi/2016-summer-perf-testing 
for examples and launch scripts. 

2016-08-23 21:20 GMT+07:00 Christoph Niethammer < nietham...@hlrs.de > : 


Hello, 

I just came over this and would like to contribute some results from our 
system(s). 
Are there any specific configure options you want to see enabled beside 
--enable-mpi-thread-multiple? 
How to commit results? 

Best 
Christoph Niethammer 



- Original Message - 
From: "Arm Patinyasakdikul (apatinya)" < apati...@cisco.com > 
To: "Open MPI Developers" < devel@lists.open-mpi.org > 
Sent: Friday, July 29, 2016 8:41:06 PM 
Subject: Re: [OMPI devel] Performance analysis proposal 

Hey Artem, all, 

Thank you for the benchmark prototype. I have created the discussion page here 
: https://github.com/open-mpi/2016-summer-perf-testing/issues/1 . 


* There, I have single threaded and multithreaded performance posted. 
* The script I used is now in the repo (also in the discussion page) 
* Result with openib will come up probably next week. I have to access UTK 
machine for that. 
* I did some test and yes, I have seen some openib hang in multithreaded case. 
Thank you, 
Arm 

From: devel < devel-boun...@lists.open-mpi.org > on behalf of Artem Polyakov < 
artpo...@gmail.com > 
Reply-To: Open MPI Developers < devel@lists.open-mpi.org > 
Date: Thursday, July 28, 2016 at 10:42 PM 
To: Open MPI Developers < devel@lists.open-mpi.org > 
Subject: Re: [OMPI devel] Performance analysis proposal 

Thank you, Arm! 

Good to have vader results (I haven't tried it myself yet). Few 
comments/questions: 
1. I guess we also want to have 1-threaded performance for the "baseline" 
reference. 
2. Have you tried to run with openib, as I mentioned on the call I had some 
problems with it and I'm curious if you can reproduce in your environment. 

Github issue sounds good for me! 

2016-07-29 12:30 GMT+07:00 Arm Patinyasakdikul (apatinya) < apati...@cisco.com 
> : 


I added some result to https://github.com/open-mpi/2016-summer-perf-testing 

The result shows much better performance from 2.0.0 and master over 1.10.3 for 
vader. The test ran with Artem’s version of benchmark on OB1, single node, bind 
to socket. 

We should have a place to discuss/comment/collaborate on results. Should I open 
an issue on that repo? So we can have github style commenting/referencing. 


Arm 




On 7/28/16, 3:02 PM, "devel on behalf of Jeff Squyres (jsquyres)" < 
devel-boun...@lists.open-mpi.org on behalf of jsquy...@cisco.com > wrote: 

>On Jul 28, 2016, at 6:28 AM, Artem Polyakov < artpo...@gmail.com > wrote: 
>> 
>> Jeff and others, 
>> 
>> 1. The benchmark was updated to support shared memory case. 
>> 2. The wiki was updated with the benchmark description: 
>> https://github.com/open-mpi/ompi/wiki/Request-refactoring-test#benchmark-prototype
>>  
> 
>Sweet -- thanks! 
> 
>> Let me know if we want to put this prototype to some general place. I think 
>> it makes sense. 
> 
>I just created: 
> 
> https://github.com/open-mpi/2016-summer-perf-testing 
> 
>Want to put it there? 
> 
>Arm just ran a bunch of tests today and will be committing a bunch of results 
>in there shortly. 
> 
>-- 
>Jeff Squyres 
> jsquy...@cisco.com 
>For corporate legal information go to: 
>http://www.cisco.com/web/about/doing_business/legal/cri/ 
> 
>___ 
>devel mailing list 
> devel@lists.open-mpi.org 
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel 
___ 
devel mailing list 
devel@lists.open-mpi.org 
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel 



-- 
С Уважением, Поляков Артем Юрьевич 
Best regards, Artem Y. Polyakov 

___ 
devel mailing list 
devel@lists.open-mpi.org 
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel 
___ 
devel mailing list 
devel@lists.open-mpi.org 
https://rfd.newmexicoconsorti

Re: [OMPI devel] Performance analysis proposal

[Christoph Niethammer]

Hello,

I just came over this and would like to contribute some results from our 
system(s).
Are there any specific configure options you want to see enabled beside 
--enable-mpi-thread-multiple?
How to commit results?

Best
Christoph Niethammer



- Original Message -
From: "Arm Patinyasakdikul (apatinya)" <apati...@cisco.com>
To: "Open MPI Developers" <devel@lists.open-mpi.org>
Sent: Friday, July 29, 2016 8:41:06 PM
Subject: Re: [OMPI devel] Performance analysis proposal

Hey Artem, all, 

Thank you for the benchmark prototype. I have created the discussion page here 
: https://github.com/open-mpi/2016-summer-perf-testing/issues/1 . 


* There, I have single threaded and multithreaded performance posted. 
* The script I used is now in the repo (also in the discussion page) 
* Result with openib will come up probably next week. I have to access UTK 
machine for that. 
* I did some test and yes, I have seen some openib hang in multithreaded 
case. 
Thank you, 
Arm 

From: devel < devel-boun...@lists.open-mpi.org > on behalf of Artem Polyakov < 
artpo...@gmail.com > 
Reply-To: Open MPI Developers < devel@lists.open-mpi.org > 
Date: Thursday, July 28, 2016 at 10:42 PM 
To: Open MPI Developers < devel@lists.open-mpi.org > 
Subject: Re: [OMPI devel] Performance analysis proposal 

Thank you, Arm! 

Good to have vader results (I haven't tried it myself yet). Few 
comments/questions: 
1. I guess we also want to have 1-threaded performance for the "baseline" 
reference. 
2. Have you tried to run with openib, as I mentioned on the call I had some 
problems with it and I'm curious if you can reproduce in your environment. 

Github issue sounds good for me! 

2016-07-29 12:30 GMT+07:00 Arm Patinyasakdikul (apatinya) < apati...@cisco.com 
> : 


I added some result to https://github.com/open-mpi/2016-summer-perf-testing 

The result shows much better performance from 2.0.0 and master over 1.10.3 for 
vader. The test ran with Artem’s version of benchmark on OB1, single node, bind 
to socket. 

We should have a place to discuss/comment/collaborate on results. Should I open 
an issue on that repo? So we can have github style commenting/referencing. 


Arm 




On 7/28/16, 3:02 PM, "devel on behalf of Jeff Squyres (jsquyres)" < 
devel-boun...@lists.open-mpi.org on behalf of jsquy...@cisco.com > wrote: 

>On Jul 28, 2016, at 6:28 AM, Artem Polyakov < artpo...@gmail.com > wrote: 
>> 
>> Jeff and others, 
>> 
>> 1. The benchmark was updated to support shared memory case. 
>> 2. The wiki was updated with the benchmark description: 
>> https://github.com/open-mpi/ompi/wiki/Request-refactoring-test#benchmark-prototype
>>  
> 
>Sweet -- thanks! 
> 
>> Let me know if we want to put this prototype to some general place. I think 
>> it makes sense. 
> 
>I just created: 
> 
> https://github.com/open-mpi/2016-summer-perf-testing 
> 
>Want to put it there? 
> 
>Arm just ran a bunch of tests today and will be committing a bunch of results 
>in there shortly. 
> 
>-- 
>Jeff Squyres 
> jsquy...@cisco.com 
>For corporate legal information go to: 
>http://www.cisco.com/web/about/doing_business/legal/cri/ 
> 
>___ 
>devel mailing list 
> devel@lists.open-mpi.org 
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel 
___ 
devel mailing list 
devel@lists.open-mpi.org 
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel 



-- 
С Уважением, Поляков Артем Юрьевич 
Best regards, Artem Y. Polyakov 

___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] sm BTL performace of the openmpi-2.0.0

[Christoph Niethammer]

Hello,

I can confirm, that it works for me, too.
Thanks! 

Best
Christoph Niethammer



- Original Message -
From: tmish...@jcity.maeda.co.jp
To: "Open MPI Developers" <devel@lists.open-mpi.org>
Sent: Wednesday, August 10, 2016 5:58:50 AM
Subject: Re: [OMPI devel] sm BTL performace of the openmpi-2.0.0

Finally it worked, thanks!

[mishima@manage OMB-3.1.1-openmpi2.0.0]$ ompi_info --param btl openib
--level 5 | grep openib_flags
  MCA btl openib: parameter "btl_openib_flags" (current value:
"65847", data source: default, level: 5 tuner/det
ail, type: unsigned_int)
[mishima@manage OMB-3.1.1-openmpi2.0.0]$ mpirun -np 2 -report-bindings
osu_bw
[manage.cluster:14439] MCW rank 0 bound to socket 0[core 0[hwt 0]], socket
0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
cket 0[core 3[hwt 0]], socket 0[core 4[hwt 0]], socket 0[core 5[hwt 0]]:
[B/B/B/B/B/B][./././././.]
[manage.cluster:14439] MCW rank 1 bound to socket 0[core 0[hwt 0]], socket
0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
cket 0[core 3[hwt 0]], socket 0[core 4[hwt 0]], socket 0[core 5[hwt 0]]:
[B/B/B/B/B/B][./././././.]
# OSU MPI Bandwidth Test v3.1.1
# SizeBandwidth (MB/s)
1 1.72
2 3.52
4 7.01
814.11
16   28.17
32   55.90
64   99.83
128 159.13
256 272.98
512 476.35
1024911.49
2048   1319.96
4096   1767.78
8192   2169.53
16384  2507.96
32768  2957.28
65536  3206.90
131072 3610.33
262144 3985.18
524288 4379.47
10485764560.90
20971524661.44
41943044631.21


Tetsuya Mishima

2016/08/10 11:57:29、"devel"さんは「Re: [OMPI devel] sm BTL performace of
the openmpi-2.0.0」で書きました
> Ack, the segv is due to a typo from transcribing the patch. Fixed. Please
try the following patch and let me know if it fixes the issues.
>
>
https://github.com/hjelmn/ompi/commit/4079eec9749e47dddc6acc9c0847b3091601919f.patch

>
> -Nathan
>
> > On Aug 8, 2016, at 9:48 PM, tmish...@jcity.maeda.co.jp wrote:
> >
> > The latest patch also causes a segfault...
> >
> > By the way, I found a typo as below. _pml_ob1.use_all_rdma in the
last
> > line should be _pml_ob1.use_all_rdma:
> >
> > +mca_pml_ob1.use_all_rdma = false;
> > +(void) mca_base_component_var_register
> > (_pml_ob1_component.pmlm_version, "use_all_rdma",
> > +   "Use all available RDMA
btls
> > for the RDMA and RDMA pipeline protocols "
> > +   "(default: false)",
> > MCA_BASE_VAR_TYPE_BOOL, NULL, 0, 0,
> > +   OPAL_INFO_LVL_5,
> > MCA_BASE_VAR_SCOPE_GROUP, _pml_ob1.use_all_rdma);
> > +
> >
> > Here is the OSU_BW and gdb output:
> >
> > # OSU MPI Bandwidth Test v3.1.1
> > # SizeBandwidth (MB/s)
> > 1 2.19
> > 2 4.43
> > 4 8.98
> > 818.07
> > 16   35.58
> > 32   70.62
> > 64  108.88
> > 128 172.97
> > 256 305.73
> > 512 536.48
> > 1024957.57
> > 2048   1587.21
> > 4096   1638.81
> > 8192   2165.14
> > 16384  2482.43
> > 32768  2866.33
> > 65536  3655.33
> > 131072 4208.40
> > 262144 4596.12
> > 524288 4769.27
> > 10485764900.00
> > [manage:16596] *** Process received signal ***
> > [manage:16596] Signal: Segmentation fault (11)
> > [manage:16596] Signal code: Address not mapped (1)
> > [manage:16596] Failing at address: 0x8
> > ...
> > Core was generated by `osu_bw'.
> > Program terminated with signal 11, Segmentation fault.
> > #0  0x0031d9008806 in ?? () from /lib64/libgcc_s.so.1
> > (gdb) where
> > #0  0x0031d9008806 in ?? () from /lib64/libgcc_s.so.1
> > #1  0x0031d9008934 in _Unwind_Backtrace ()
from /lib64/libgcc_s.so.1
> > #2  0x0037ab8e5ee8 in backtrace () from /lib64/libc.so.6
> > #3  0x2b5060c14345 in opal_backtrace_print ()
> > at ./backtrace_execinfo.c:47
> > #4  0x2b5060c11180 in

Re: [OMPI devel] stdout, stderr reporting different values for isatty

[Christoph Niethammer]

Hi Gilles,

No, it is not a problem for my applications.

I would have assumed, that both are using a pipe to orted+tty or whatever 
infrastructure is used.
Different behaviour is for me often a sign that something is going wrong. So 
better ask. ;)

Regards
CHrsitoph

- Original Message -
From: "Gilles Gouaillardet" <gilles.gouaillar...@gmail.com>
To: "Open MPI Developers" <de...@open-mpi.org>
Sent: Monday, July 27, 2015 3:28:12 PM
Subject: Re: [OMPI devel] stdout,   stderr reporting different values for 
isatty

Christoph, 

that is correct 
stdout is a tty and stderr is not. 
it is a pipe to orted. 
I do not think that would be hard to change. 
is this a source of problem for your applications ? 

note this kind of behavior can be caused by the batch manager. 
if you use slurm and srun instead of mpirun, I am not even sure stdout is a 
tty. 

Cheers, 

Gilles 

On Monday, July 27, 2015, Christoph Niethammer < nietham...@hlrs.de > wrote: 


Hello, 

I know, using stdout and stderr within MPI programs is in no way good. 
Nevertheless I found that - and now wonder why - isatty inside an MPI program 
reports different values for stdout and stderr in Open MPI: 

# Running as non MPI program: 
./isatty-test 
[0/1] stdout FILENO: 1, TTY: 1 
[0/1] stderr FILENO: 2, TTY: 1 

# Running with Open MPI 1.8.7: 
mpirun -np 2 ./isatty-test 
[1/2] stdout FILENO: 1, TTY: 1 
[1/2] stderr FILENO: 2, TTY: 0 
[0/2] stdout FILENO: 1, TTY: 1 
[0/2] stderr FILENO: 2, TTY: 0 

... not sure if this is good or bad. Both are forwarded correctly to the tty as 
far as I see... 

Redirecting stdout or stderr to files does not change anything in the Open MPI 
case. 

Best regards 
Christoph Niethammer 

PS: MPICH reports in all cases 0 for isatty() on stdout and stderr. 
___ 
devel mailing list 
de...@open-mpi.org 
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel 
Link to this post: 
http://www.open-mpi.org/community/lists/devel/2015/07/17713.php 

___
devel mailing list
de...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post: 
http://www.open-mpi.org/community/lists/devel/2015/07/17714.php

[OMPI devel] C standard compatibility

[Christoph Niethammer]

Hello,

What is the C standard version to be used for the Open MPI code base?

Most seems to be < C99.
C99 features I saw so far mostuly in newer components:
* restrict keyword
* variable declaration inside for loop heads

Regards
Christoph Niethammer

--

Christoph Niethammer
High Performance Computing Center Stuttgart (HLRS)
Nobelstrasse 19
70569 Stuttgart

Tel: ++49(0)711-685-87203
email: nietham...@hlrs.de
http://www.hlrs.de/people/niethammer

[OMPI devel] stdout, stderr reporting different values for isatty

[Christoph Niethammer]

Hello,

I know, using stdout and stderr within MPI programs is in no way good.
Nevertheless I found that - and now wonder why - isatty inside an MPI program 
reports different values for stdout and stderr in Open MPI:

# Running as non MPI program:
./isatty-test 
[0/1] stdout FILENO: 1, TTY: 1
[0/1] stderr FILENO: 2, TTY: 1

# Running with Open MPI 1.8.7:
mpirun -np 2 ./isatty-test
[1/2] stdout FILENO: 1, TTY: 1
[1/2] stderr FILENO: 2, TTY: 0
[0/2] stdout FILENO: 1, TTY: 1
[0/2] stderr FILENO: 2, TTY: 0

... not sure if this is good or bad. Both are forwarded correctly to the tty as 
far as I see...

Redirecting stdout or stderr to files does not change anything in the Open MPI 
case.

Best regards
Christoph Niethammer

PS: MPICH reports in all cases 0 for isatty() on stdout and stderr. 

[OMPI devel] Missing f08 binding for Win_allocate

[Christoph Niethammer]

Hello,

The f08 binding for Win_allocate is missing in master and 1.8 series.
I fixed the problem based on master. The attached patch also works for 1.8.3.

I found some documentation in the wiki but I am not sure if this is intended 
for small fixes like this as well.
How shall I proceed to get this into master after the svn->git transition?
* Open a bug first
* fork + pull request or 
* email patch from git format-patch to devel list?

Best regards
Christoph Niethammer


--

Christoph Niethammer
High Performance Computing Center Stuttgart (HLRS)
Nobelstrasse 19
70569 Stuttgart

Tel: ++49(0)711-685-87203
email: nietham...@hlrs.de
http://www.hlrs.de/people/niethammerFrom 7b81fe5584ddc675aca7a09a7f59b5658254318f Mon Sep 17 00:00:00 2001
From: Christoph Niethammer <nietham...@hlrs.de>
Date: Wed, 15 Oct 2014 15:56:56 +0200
Subject: [PATCH] Add missing Fortran binding for Win_allocate.

---
 ompi/mpi/fortran/mpif-h/Makefile.am|   1 +
 ompi/mpi/fortran/mpif-h/profile/Makefile.am|   1 +
 ompi/mpi/fortran/mpif-h/profile/defines.h  |   2 +
 ompi/mpi/fortran/mpif-h/prototypes_mpi.h   |   2 +
 ompi/mpi/fortran/mpif-h/win_allocate_f.c   | 137 +
 ompi/mpi/fortran/use-mpi-f08/Makefile.am   |   2 +
 .../fortran/use-mpi-f08/mpi-f-interfaces-bind.h|  13 ++
 .../mpi/fortran/use-mpi-f08/mpi-f08-interfaces.F90 |  17 ++-
 .../fortran/use-mpi-f08/pmpi-f-interfaces-bind.h   |  13 ++
 .../use-mpi-f08/profile/pwin_allocate_f08.F90  |  28 +
 ompi/mpi/fortran/use-mpi-f08/win_allocate_f08.F90  |  28 +
 .../mpi-ignore-tkr-interfaces.h.in |  26 
 .../use-mpi-tkr/mpi-f90-cptr-interfaces.F90|  29 +
 13 files changed, 298 insertions(+), 1 deletion(-)
 create mode 100644 ompi/mpi/fortran/mpif-h/win_allocate_f.c
 create mode 100644 ompi/mpi/fortran/use-mpi-f08/profile/pwin_allocate_f08.F90
 create mode 100644 ompi/mpi/fortran/use-mpi-f08/win_allocate_f08.F90

diff --git a/ompi/mpi/fortran/mpif-h/Makefile.am b/ompi/mpi/fortran/mpif-h/Makefile.am
index b727329..f254975 100644
--- a/ompi/mpi/fortran/mpif-h/Makefile.am
+++ b/ompi/mpi/fortran/mpif-h/Makefile.am
@@ -386,6 +386,7 @@ libmpi_mpifh_la_SOURCES += \
 rput_f.c \
 compare_and_swap_f.c \
 fetch_and_op_f.c \
+win_allocate_f.c \
 win_allocate_shared_f.c \
 win_call_errhandler_f.c \
 win_complete_f.c \
diff --git a/ompi/mpi/fortran/mpif-h/profile/Makefile.am b/ompi/mpi/fortran/mpif-h/profile/Makefile.am
index 793c45c..66712df 100644
--- a/ompi/mpi/fortran/mpif-h/profile/Makefile.am
+++ b/ompi/mpi/fortran/mpif-h/profile/Makefile.am
@@ -314,6 +314,7 @@ linked_files = \
 prput_f.c \
 pcompare_and_swap_f.c \
 pfetch_and_op_f.c \
+pwin_allocate_f.c \
 pwin_allocate_shared_f.c \
 pwin_call_errhandler_f.c \
 pwin_complete_f.c \
diff --git a/ompi/mpi/fortran/mpif-h/profile/defines.h b/ompi/mpi/fortran/mpif-h/profile/defines.h
index 43b21b7..94fd9e4 100644
--- a/ompi/mpi/fortran/mpif-h/profile/defines.h
+++ b/ompi/mpi/fortran/mpif-h/profile/defines.h
@@ -344,6 +344,8 @@
 #define ompi_waitany_f pompi_waitany_f
 #define ompi_wait_f pompi_wait_f
 #define ompi_waitsome_f pompi_waitsome_f
+#define ompi_win_allocate_f pompi_win_allocate_f
+#define ompi_win_allocate_cptr_f pompi_win_allocate_cptr_f
 #define ompi_win_allocate_shared_f pompi_win_allocate_shared_f
 #define ompi_win_allocate_shared_cptr_f pompi_win_allocate_shared_cptr_f
 #define ompi_win_call_errhandler_f pompi_win_call_errhandler_f
diff --git a/ompi/mpi/fortran/mpif-h/prototypes_mpi.h b/ompi/mpi/fortran/mpif-h/prototypes_mpi.h
index f3115c3..b8ac7da 100644
--- a/ompi/mpi/fortran/mpif-h/prototypes_mpi.h
+++ b/ompi/mpi/fortran/mpif-h/prototypes_mpi.h
@@ -401,6 +401,8 @@ PN2(void, MPI_Waitall, mpi_waitall, MPI_WAITALL, (MPI_Fint *count, MPI_Fint *arr
 PN2(void, MPI_Waitany, mpi_waitany, MPI_WAITANY, (MPI_Fint *count, MPI_Fint *array_of_requests, MPI_Fint *index, MPI_Fint *status, MPI_Fint *ierr));
 PN2(void, MPI_Wait, mpi_wait, MPI_WAIT, (MPI_Fint *request, MPI_Fint *status, MPI_Fint *ierr));
 PN2(void, MPI_Waitsome, mpi_waitsome, MPI_WAITSOME, (MPI_Fint *incount, MPI_Fint *array_of_requests, MPI_Fint *outcount, MPI_Fint *array_of_indices, MPI_Fint *array_of_statuses, MPI_Fint *ierr));
+PN2(void, MPI_Win_allocate, mpi_win_allocate, MPI_WIN_ALLOCATE, (MPI_Aint *size, MPI_Fint *disp_unit, MPI_Fint *info, MPI_Fint *comm, char *baseptr, MPI_Fint *win, MPI_Fint *ierr));
+PN2(void, MPI_Win_allocate_cptr, mpi_win_allocate_cptr, MPI_WIN_ALLOCATE_CPTR, (MPI_Aint *size, MPI_Fint *disp_unit, MPI_Fint *info, MPI_Fint *comm, char *baseptr, MPI_Fint *win, MPI_Fint *ierr));
 PN2(void, MPI_Win_allocate_shared, mpi_win_allocate_shared, MPI_WIN_ALLOCATE_SHARED, (MPI_Aint *size, MPI_Fint *disp_unit, MPI_Fint *info, MPI_Fint *comm, char *baseptr, MPI_Fint *win, MPI_Fint *ierr));
 PN2(void, MPI_Win_allocate_sh

[OMPI devel] PML-bfo deadlocks for message size > eager limit after connection loss

[Christoph Niethammer]

Hello,

Is there anybody using/testing the bfo PML - especially with messages > eager 
limit?

Tests using messages > eager limit with the bfo PML seem to deadlock in Open 
MPI 1.6.5 as soon as one of two infiniband connections gets lost (tested by 
disconnecting wire).
I did not have an opportunity to test 1.8/trunk up to now.

Tests were executed with the following mpirun options:

mpirun -np 2 --hostfile /opt/ddt/nodes --pernode --mca pml bfo --mca 
btl_base_exclude tcp --mca pml bfo --mca btl_openib_port_error_failover 1 --mca 
btl_openib_failover_enabled 1 --mca btl_openib_port_error_failover 1 --verbose 
--mca oob_tcp_verbose 100 --mca btl_openib_verbose_failover 100 --mca 
btl_openib_verbose 100 --mca btl_base_verbose 100 --mca bml_base_verbose 100 
--mca pml_bfo_verbose 100 --mca pml_base_verbose 100 --mca opal_progress_debug 
100 --mca orte_debug_verbose 100 --mca pml_v_verbose 100 --mca 
orte_base_help_aggregate 0

Some log output is attached below.

I would appreciate any feedback concerning current status of the bfo PML as 
well as ideas how to debug and where to search for the problem inside the Open 
MPI code base.


Best regards
Christoph Niethammer

--

Christoph Niethammer
High Performance Computing Center Stuttgart (HLRS)
Nobelstrasse 19
70569 Stuttgart

Tel: ++49(0)711-685-87203
email: nietham...@hlrs.de
http://www.hlrs.de/people/niethammer






[vm2:21970] defining message event: iof_hnp_receive.c 227
[vm1:16449] Rank 0 receiving ...
[vm2:21970] [[22205,0],0] got show_help from [[22205,1],0]
--
The OpenFabrics stack has reported a network error event.  Open MPI
will try to continue, but your job may end up failing.

  Local host:vm1
  MPI process PID:   16449
  Error number:  10 (IBV_EVENT_PORT_ERR)

This error may indicate connectivity problems within the fabric;
please contact your system administrator.
--
[vm1][[22205,1],0][btl_openib.c:1350:mca_btl_openib_prepare_dst] 
frag->sg_entry.lkey = 1829372025 .addr = 1e1bee0 frag->segment.seg_key.key32[0] 
= 1829372025
[vm1][[22205,1],0][btl_openib.c:1350:mca_btl_openib_prepare_dst] 
frag->sg_entry.lkey = 1829372025 .addr = 1e28230 frag->segment.seg_key.key32[0] 
= 1829372025
[vm2:21970] defining message event: iof_hnp_receive.c 227
[vm1:16449]  Bandwidth [MB/s]: 594.353640
[vm1:16449] Rank 0: loop: 1100
[vm1:16449] Rank 0 sending ...
[vm2:21970] defining message event: iof_hnp_receive.c 227
[vm2:21970] defining message event: iof_hnp_receive.c 227
[vm1][[22205,1],0][btl_openib_failover.c:696:mca_btl_openib_endpoint_notify] 
[vm1:16449] BTL openib error: rank=0 mapping out lid=2:name=mthca0 to rank=1 on 
node=vm2 [vm1:16449] IB: Finished checking for pending_frags, total moved=0
[vm1:16449] IB: Finished checking for pending_frags, total moved=0
Error sending BROKEN CONNECTION buffer (Success)
[[22205,1],1][btl_openib_component.c:3496:handle_wc] from vm2 to: 192 error 
polling LP CQ with status RETRY EXCEEDED ERROR status number 12 for wr_id 
bdba80 opcode 1  vendor error 129 qp_idx 0
[vm2:21970] [[22205,0],0] got show_help from [[22205,1],1]
--
The InfiniBand retry count between two MPI processes has been
exceeded.  "Retry count" is defined in the InfiniBand spec 1.2
(section 12.7.38):

The total number of times that the sender wishes the receiver to
retry timeout, packet sequence, etc. errors before posting a
completion error.

This error typically means that there is something awry within the
InfiniBand fabric itself.  You should note the hosts on which this
error has occurred; it has been observed that rebooting or removing a
particular host from the job can sometimes resolve this issue.

Two MCA parameters can be used to control Open MPI's behavior with
respect to the retry count:

* btl_openib_ib_retry_count - The number of times the sender will
  attempt to retry (defaulted to 7, the maximum value).
* btl_openib_ib_timeout - The local ACK timeout parameter (defaulted
  to 20).  The actual timeout value used is calculated as:

 4.096 microseconds * (2^btl_openib_ib_timeout)

  See the InfiniBand spec 1.2 (section 12.7.34) for more details.

Below is some information about the host that raised the error and the
peer to which it was connected:

  Local host:   vm2
  Local device: mthca0
  Peer host:192

You may need to consult with your system administrator to get this
problem fixed.
--
[vm2:21982] MCA_BTL_OPENIG_FRAG=5, dropping since connection is broken 
(des=bdba80)
[[22205,1],1][btl_openib_component.c:3496:handle_wc] from vm2 to: 192 error 
polling HP CQ with status WORK REQUEST FLUSHED ERROR status number 5 for wr_id 
c56380 opcode 1  vendor error 244 qp_idx 0
[vm2:21982] MCA_BTL_O

[OMPI devel] Wrong Endianness in Open MPI for external32 representation

[Christoph Niethammer]

Hello,

It seems for me that the endianness is wrong in Open MPI's I/O for the 
external32 data representation. :O

Find attached my test programs which write the integer -30 and the double 16.25 
into a file.
Here the output on my Linux system:

mpicc c-xdr.c   -o c-xdr
mpicc mpi-io-external32.c   -o mpi-io-external32
./c-xdr
Output file: c-xdr.dat
hexdump c-xdr.dat
000  e2ff 3040 0040    
00c
./mpi-io-external32
Output file: mpi-io-external32.dat
hexdump mpi-io-external32.dat
000 ffe2    4000 4030  
00c


Best regards
Christoph Niethammer

--

Christoph Niethammer
High Performance Computing Center Stuttgart (HLRS)
Nobelstrasse 19
70569 Stuttgart

Tel: ++49(0)711-685-87203
email: nietham...@hlrs.de
http://www.hlrs.de/people/niethammer#include 
#include 

int main(int argc, char* argv[]) {
FILE *fp;
XDR xdr_output;
char filename[] = "c-xdr.dat";
printf("Output file: %s\n", filename);
int i = -30;
double d = 16.25;

fp = fopen(filename, "w+");
xdrstdio_create(_output, fp, XDR_ENCODE);
xdr_int(_output, );
xdr_double(_output, );
return 0;
}
#include 
#include 




int main(int argc, char* argv[]) {

MPI_Comm comm;
int comm_rank = 0;
int comm_size = 1;
MPI_File fh;
MPI_Status status;
MPI_Offset disp;
MPI_Datatype etype;
MPI_Datatype filetype;
char datarep[] = "external32";

int rc;
int i;
char filename[] = "mpi-io-external32.dat";
printf("Output file: %s\n", filename);
int N = -30;
double d = 16.25;

MPI_Init(, );
comm = MPI_COMM_WORLD;
MPI_Comm_rank(comm, _rank);
MPI_Comm_size(comm, _size);
//printf("Hello %d/%d\n", comm_rank, comm_size);

rc = MPI_File_open(comm, filename, MPI_MODE_CREATE | MPI_MODE_RDWR, MPI_INFO_NULL, );
if(rc != MPI_SUCCESS) {
printf("Error %d when opening file %s\n", rc, filename);
}
rc = MPI_File_set_view(fh, 0, MPI_INT, MPI_INT, datarep, MPI_INFO_NULL);
if(rc != MPI_SUCCESS) {
printf("Error %d when setting file view\n", rc);
}
if(comm_rank == 0) {
//printf("Rank %d writing data...\n", comm_rank);
MPI_File_write_shared(fh, , 1, MPI_INT, );
MPI_File_write_shared(fh, , 1, MPI_DOUBLE, );
}
MPI_Barrier(comm);


MPI_File_close();
MPI_Finalize();
return 0;
}

Re: [OMPI devel] 1-question developer poll

[Christoph Niethammer]

git (Github mirror, git-svn, git patches)

--

Christoph Niethammer
High Performance Computing Center Stuttgart (HLRS)
Nobelstrasse 19
70569 Stuttgart

Tel: ++49(0)711-685-87203
email: nietham...@hlrs.de
http://www.hlrs.de/people/niethammer



- Original Message -
From: "Jeff Squyres (jsquyres)" <jsquy...@cisco.com>
To: "Open MPI Developers List" <de...@open-mpi.org>
Sent: Wednesday, April 16, 2014 12:32:10 PM
Subject: [OMPI devel] 1-question developer poll

What source code repository technology(ies) do you use for Open MPI 
development? (indicate all that apply)

- SVN
- Mercurial
- Git

I ask this question because there's serious discussions afoot to switch OMPI's 
main SVN repo to Git, and I want to get a feel for the current landscape out 
there.

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

___
devel mailing list
de...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post: 
http://www.open-mpi.org/community/lists/devel/2014/04/14537.php

Re: [OMPI devel] Reviewing MPI_Dims_create

[Christoph Niethammer]

Hello,

Minor update as a little bug and an unused variable were left over in the patch.
I'll commit this + the parameter check change later.

Anybody volunteering for review of a cmr for 1.7.5. :)

Ah and some restults for MPI_Dims_create(100, 3, {})

original: 8.110628 sec
optimized-primes: 0.048702 sec
optimized-factorization: 0.13 sec

Regards
Christoph

--

Christoph Niethammer
High Performance Computing Center Stuttgart (HLRS)
Nobelstrasse 19
70569 Stuttgart

Tel: ++49(0)711-685-87203
email: nietham...@hlrs.de
http://www.hlrs.de/people/niethammer




- Ursprüngliche Mail -
Von: "Andreas Schäfer" <gent...@gmx.de>
An: "Open MPI Developers" <de...@open-mpi.org>
Gesendet: Dienstag, 11. Februar 2014 12:16:53
Betreff: Re: [OMPI devel] Reviewing MPI_Dims_create

Hi,

ah, that's clever indeed!

Best
-Andreas


On 12:02 Tue 11 Feb , Christoph Niethammer wrote:
> Hello,
> 
> After rethinking Jeff's comments about caching prime numbers I came to the 
> conclusion that we can omit the prime numbers at all and go directly for the 
> factorization. :D
> We then only need at most   log_2(INT_MAX) * sizeof(int) = 32 * 4 byte = 128 
> byte   of memory for the factors.
> 
> Computational costs reduce as well as the factorization itself is done by a 
> loop with at most \sqrt(num) / 2 iterations - which is the same as in the 
> original prime number detection loop.
> I think this is the cleanest way which reduces also source code size. ;)
> 
> Find attache patch against the trunk.
> 
> Best regards
> Christoph
> 
> --
> 
> Christoph Niethammer
> High Performance Computing Center Stuttgart (HLRS)
> Nobelstrasse 19
> 70569 Stuttgart
> 
> Tel: ++49(0)711-685-87203
> email: nietham...@hlrs.de
> http://www.hlrs.de/people/niethammer
> 
> 
> 
> - Ursprüngliche Mail -
> Von: "Andreas Schäfer" <gent...@gmx.de>
> An: "Open MPI Developers" <de...@open-mpi.org>
> Gesendet: Dienstag, 11. Februar 2014 06:24:56
> Betreff: Re: [OMPI devel] Reviewing MPI_Dims_create
> 
> OK, so we were thinking the same thing :-) The optimization you
> mention below is the same I've used in my updated patch.
> 
> 
> On 02:29 Tue 11 Feb , Christoph Niethammer wrote:
> > As mentioned in my former mail I did not touch the factorization
> > code.
> > But to figure out if a number n is *not* a prime number it is sufficient to 
> > check up to \sqrt(n).
> > Proof:
> > let n = p*q with q > \sqrt{n}
> > --> p < \sqrt(n)
> > So we have already found factor p before reaching \sqrt(n) and by this n is 
> > no prime any more and no need for further checks. ;)
> > 
> > 
> > The mentioned factorization may indeed include one factor which is larger 
> > than \sqrt(n). :)
> > 
> > Proof that at least one prime factor can be larger than \sqrt(n) example:
> > 6 = 2*3
> > \sqrt(6) = 2.4494897427832... < 3   Q.E.D.
> > 
> > 
> > Proof that no more than one factor can be larger than \sqrt(n):
> > let n = \prod_{i=0}^K p_i with p_i \in N  and K > 2
> > and assume w.l.o.g.  p_0 > \sqrt(n)  and  p_1 > \sqrt(n)
> > --> 1 > \prod_{i=2}^K p_i
> > which is a contradiction as all p_i \in N.  Q.E.D.
> > 
> > 
> > So your idea is still applicable with not much effort and we only need 
> > prime factors up to sqrt(n) in the factorizer code for an additional 
> > optimization. :)
> > 
> > First search all K' factors p_i < \sqrt(n). If then n \ne \prod_{i=0}^{K'} 
> > p_i we should be sure that p_{K'+1} = n / \prod_{i=0}^{K'} p_i is a prime. 
> > No complication with counts IMHO. I leave this without patch as it is 
> > already 2:30 in the morning. :P
> > 
> > Regards
> > Christoph
> > 
> > --
> > 
> > Christoph Niethammer
> > High Performance Computing Center Stuttgart (HLRS)
> > Nobelstrasse 19
> > 70569 Stuttgart
> > 
> > Tel: ++49(0)711-685-87203
> > email: nietham...@hlrs.de
> > http://www.hlrs.de/people/niethammer
> > 
> > - Ursprüngliche Mail -
> > Von: "Andreas Schäfer" <gent...@gmx.de>
> > An: "Open MPI Developers" <de...@open-mpi.org>
> > Gesendet: Montag, 10. Februar 2014 23:24:24
> > Betreff: Re: [OMPI devel] Reviewing MPI_Dims_create
> > 
> > Christoph-
> > 
> > your patch has the same problem as my original patch: indeed there may
> > be a prime factor p of n with p > sqrt(n). What's important is that
> > there may only be at most one. I've submitted an updated patch (see my
> > previous mail) which catches this special

Re: [OMPI devel] Reviewing MPI_Dims_create

[Christoph Niethammer]

Hello,

After rethinking Jeff's comments about caching prime numbers I came to the 
conclusion that we can omit the prime numbers at all and go directly for the 
factorization. :D
We then only need at most   log_2(INT_MAX) * sizeof(int) = 32 * 4 byte = 128 
byte   of memory for the factors.

Computational costs reduce as well as the factorization itself is done by a 
loop with at most \sqrt(num) / 2 iterations - which is the same as in the 
original prime number detection loop.
I think this is the cleanest way which reduces also source code size. ;)

Find attache patch against the trunk.

Best regards
Christoph

--

Christoph Niethammer
High Performance Computing Center Stuttgart (HLRS)
Nobelstrasse 19
70569 Stuttgart

Tel: ++49(0)711-685-87203
email: nietham...@hlrs.de
http://www.hlrs.de/people/niethammer



- Ursprüngliche Mail -
Von: "Andreas Schäfer" <gent...@gmx.de>
An: "Open MPI Developers" <de...@open-mpi.org>
Gesendet: Dienstag, 11. Februar 2014 06:24:56
Betreff: Re: [OMPI devel] Reviewing MPI_Dims_create

OK, so we were thinking the same thing :-) The optimization you
mention below is the same I've used in my updated patch.


On 02:29 Tue 11 Feb , Christoph Niethammer wrote:
> As mentioned in my former mail I did not touch the factorization
> code.
> But to figure out if a number n is *not* a prime number it is sufficient to 
> check up to \sqrt(n).
> Proof:
> let n = p*q with q > \sqrt{n}
> --> p < \sqrt(n)
> So we have already found factor p before reaching \sqrt(n) and by this n is 
> no prime any more and no need for further checks. ;)
> 
> 
> The mentioned factorization may indeed include one factor which is larger 
> than \sqrt(n). :)
> 
> Proof that at least one prime factor can be larger than \sqrt(n) example:
> 6 = 2*3
> \sqrt(6) = 2.4494897427832... < 3   Q.E.D.
> 
> 
> Proof that no more than one factor can be larger than \sqrt(n):
> let n = \prod_{i=0}^K p_i with p_i \in N  and K > 2
> and assume w.l.o.g.  p_0 > \sqrt(n)  and  p_1 > \sqrt(n)
> --> 1 > \prod_{i=2}^K p_i
> which is a contradiction as all p_i \in N.  Q.E.D.
> 
> 
> So your idea is still applicable with not much effort and we only need prime 
> factors up to sqrt(n) in the factorizer code for an additional optimization. 
> :)
> 
> First search all K' factors p_i < \sqrt(n). If then n \ne \prod_{i=0}^{K'} 
> p_i we should be sure that p_{K'+1} = n / \prod_{i=0}^{K'} p_i is a prime. No 
> complication with counts IMHO. I leave this without patch as it is already 
> 2:30 in the morning. :P
> 
> Regards
> Christoph
> 
> --
> 
> Christoph Niethammer
> High Performance Computing Center Stuttgart (HLRS)
> Nobelstrasse 19
> 70569 Stuttgart
> 
> Tel: ++49(0)711-685-87203
> email: nietham...@hlrs.de
> http://www.hlrs.de/people/niethammer
> 
> - Ursprüngliche Mail -
> Von: "Andreas Schäfer" <gent...@gmx.de>
> An: "Open MPI Developers" <de...@open-mpi.org>
> Gesendet: Montag, 10. Februar 2014 23:24:24
> Betreff: Re: [OMPI devel] Reviewing MPI_Dims_create
> 
> Christoph-
> 
> your patch has the same problem as my original patch: indeed there may
> be a prime factor p of n with p > sqrt(n). What's important is that
> there may only be at most one. I've submitted an updated patch (see my
> previous mail) which catches this special case.
> 
> Best
> -Andreas
> 
> 
> On 19:30 Mon 10 Feb , Christoph Niethammer wrote:
> > Hello,
> > 
> > I noticed some effort in improving the scalability of
> > MPI_Dims_create(int nnodes, int ndims, int dims[])
> > Unfortunately there were some issues with the first attempt (r30539 and 
> > r30540) which were reverted.
> > 
> > So I decided to give it a short review based on r30606
> > https://svn.open-mpi.org/trac/ompi/browser/trunk/ompi/mpi/c/dims_create.c?rev=30606
> > 
> > 
> > 1.) freeprocs is initialized to be nnodes and the subsequent divisions of 
> > freeprocs have all positive integers as divisor.
> > So IMHO it would make more sense to check if nnodes > 0 in the 
> > MPI_PARAM_CHECK section at the begin instead of the following (see patch 
> > 0001):
> > 
> > 99  if (freeprocs < 1) {
> > 100return OMPI_ERRHANDLER_INVOKE(MPI_COMM_WORLD, MPI_ERR_DIMS,
> > 101  FUNC_NAME);
> > 102 }
> > 
> > 
> > 2.) I rewrote the algorithm stopping at sqrt(n) in getprimes(int num, int 
> > *nprimes, int **pprimes)
> > which makes mathematically more sens (as the largest prime factor of any 
> > number n cannot exceed \sqrt{n}) - and should produce the right result. ;)
> > 

Re: [OMPI devel] Reviewing MPI_Dims_create

[Christoph Niethammer]

sqrt(2^31)/log(sqrt(2^31))*(1+1.2762/log(sqrt(2^31)))/1024 * 4byte = 
18,850133965051 kbyte should do it. ;)
Amazing - I think our systems are still *too small* - lets go for MPI with 
int64 types. ^^

- Ursprüngliche Mail -
Von: "Jeff Squyres (jsquyres)" <jsquy...@cisco.com>
An: "Open MPI Developers" <de...@open-mpi.org>
Gesendet: Dienstag, 11. Februar 2014 01:32:53
Betreff: Re: [OMPI devel] Reviewing MPI_Dims_create

On Feb 10, 2014, at 7:22 PM, Christoph Niethammer <nietham...@hlrs.de> wrote:

> 2.) Interesting idea: Using the approximation from the cited paper we should 
> only need around 400 MB to store all primes in the int32 range. Potential for 
> applying compression techniques still present. ^^

Per Andreas' last mail, we only need primes up to sqrt(2B) + 1 more.  That 
*has* to be less than 400MB... right?

sqrt(2B) = 46340.  So the upper limit on the size required to hold all the 
primes from 2...46340 is 46340*sizeof(int) = 185,360 bytes (plus one more, per 
Andreas, so 185,364).

This is all SWAGing, but I'm assuming the actual number must be *far* less than 
that...

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel

Re: [OMPI devel] Reviewing MPI_Dims_create

[Christoph Niethammer]

Hi Andreas,

As mentioned in my former mail I did not touch the factorization code.
But to figure out if a number n is *not* a prime number it is sufficient to 
check up to \sqrt(n).
Proof:
let n = p*q with q > \sqrt{n}
--> p < \sqrt(n)
So we have already found factor p before reaching \sqrt(n) and by this n is no 
prime any more and no need for further checks. ;)


The mentioned factorization may indeed include one factor which is larger than 
\sqrt(n). :)

Proof that at least one prime factor can be larger than \sqrt(n) example:
6 = 2*3
\sqrt(6) = 2.4494897427832... < 3   Q.E.D.


Proof that no more than one factor can be larger than \sqrt(n):
let n = \prod_{i=0}^K p_i with p_i \in N  and K > 2
and assume w.l.o.g.  p_0 > \sqrt(n)  and  p_1 > \sqrt(n)
--> 1 > \prod_{i=2}^K p_i
which is a contradiction as all p_i \in N.  Q.E.D.


So your idea is still applicable with not much effort and we only need prime 
factors up to sqrt(n) in the factorizer code for an additional optimization. :)

First search all K' factors p_i < \sqrt(n). If then n \ne \prod_{i=0}^{K'} p_i 
we should be sure that p_{K'+1} = n / \prod_{i=0}^{K'} p_i is a prime. No 
complication with counts IMHO. I leave this without patch as it is already 2:30 
in the morning. :P

Regards
Christoph

--

Christoph Niethammer
High Performance Computing Center Stuttgart (HLRS)
Nobelstrasse 19
70569 Stuttgart

Tel: ++49(0)711-685-87203
email: nietham...@hlrs.de
http://www.hlrs.de/people/niethammer

- Ursprüngliche Mail -
Von: "Andreas Schäfer" <gent...@gmx.de>
An: "Open MPI Developers" <de...@open-mpi.org>
Gesendet: Montag, 10. Februar 2014 23:24:24
Betreff: Re: [OMPI devel] Reviewing MPI_Dims_create

Christoph-

your patch has the same problem as my original patch: indeed there may
be a prime factor p of n with p > sqrt(n). What's important is that
there may only be at most one. I've submitted an updated patch (see my
previous mail) which catches this special case.

Best
-Andreas


On 19:30 Mon 10 Feb , Christoph Niethammer wrote:
> Hello,
> 
> I noticed some effort in improving the scalability of
> MPI_Dims_create(int nnodes, int ndims, int dims[])
> Unfortunately there were some issues with the first attempt (r30539 and 
> r30540) which were reverted.
> 
> So I decided to give it a short review based on r30606
> https://svn.open-mpi.org/trac/ompi/browser/trunk/ompi/mpi/c/dims_create.c?rev=30606
> 
> 
> 1.) freeprocs is initialized to be nnodes and the subsequent divisions of 
> freeprocs have all positive integers as divisor.
> So IMHO it would make more sense to check if nnodes > 0 in the 
> MPI_PARAM_CHECK section at the begin instead of the following (see patch 
> 0001):
> 
> 99if (freeprocs < 1) {
> 100  return OMPI_ERRHANDLER_INVOKE(MPI_COMM_WORLD, MPI_ERR_DIMS,
> 101FUNC_NAME);
> 102   }
> 
> 
> 2.) I rewrote the algorithm stopping at sqrt(n) in getprimes(int num, int 
> *nprimes, int **pprimes)
> which makes mathematically more sens (as the largest prime factor of any 
> number n cannot exceed \sqrt{n}) - and should produce the right result. ;)
> (see patch 0002)
> Here the improvements:
> 
> module load mpi/openmpi/trunk-gnu.4.7.3
> $ ./mpi-dims-old 100
> time used for MPI_Dims_create(100, 3, {}): 8.104007
> module swap mpi/openmpi/trunk-gnu.4.7.3 mpi/openmpi/trunk-gnu.4.7.3-testing
> $ ./mpi-dims-new 100
> time used for MPI_Dims_create(100, 3, {}): 0.060400
> 
> 
> 3.) Memory allocation for the list of prime numbers may be reduced up to a 
> factor of ~6 for 1mio nodes using the result from Dusart 1999 [1]:
> \pi(x)  < x/ln(x)(1+1.2762/ln(x))  for x > 1
> Unfortunately this saves us only 1.6 MB per process for 1mio nodes as 
> reported by tcmalloc/pprof on a test program - but it may sum up with fatter 
> nodes. :P
> 
> $ pprof --base=$PWD/primes-old.0001.heap a.out primes-new.0002.heap
> (pprof) top
> Total: -1.6 MB
>  0.3 -18.8% -18.8%  0.3 -18.8% getprimes2
>  0.0  -0.0% -18.8% -1.6 100.0% __libc_start_main
>  0.0  -0.0% -18.8% -1.6 100.0% main
> -1.9 118.8% 100.0% -1.9 118.8% getprimes
> 
> Find attached patch for it in 0003.
> 
> 
> If there are no issues I would like to commit this to trunk for further 
> testing (+cmr for 1.7.5?) end of this week.
> 
> Best regards
> Christoph
> 
> [1] 
> http://www.ams.org/journals/mcom/1999-68-225/S0025-5718-99-01037-6/home.html
> 
> 
> 
> --
> 
> Christoph Niethammer
> High Performance Computing Center Stuttgart (HLRS)
> Nobelstrasse 19
> 70569 Stuttgart
> 
> Tel: ++49(0)711-685-87203
> email: nietham...@hlrs.de
> http://www.hlrs.de/people/niethammer

> From e3292b90cac42fad8

Re: [OMPI devel] Reviewing MPI_Dims_create

[Christoph Niethammer]

Hello,

If you mean the current version in the ompi-tests/ibm svn repository I can 
confirm that it passes the topolgy/dimscreate test without errors. :)

The difference in the patches is as follows: The patch from Andreas only 
generated a table of prime numbers of up to sqrt(freeprocs) while my patch 
still produces prime numbers up to freeprocs. And for factoring we really need 
all factors up to freeprocs. The standard sqrt optimization was just introduced 
in the wrong place. :)

You are right with #3: It's a better approximation for the upper bound and the 
proof is something to be read under the Christmas tree. ;)
I just have to rethink if the ceil() is necessary in the code as I am not sure 
about rounding issues in floating point calculations here... :P

Regarding your questions:
1.) I don't think we have to cache prime numbers as MPI_Dims create will not be 
used frequently for factorization. If anybody needs faster factorization he 
would use his own - even more optimized - code. If you find some free time 
beside Open MPI go out for some harder problems at http://projecteuler.net. But 
please don't get frustrated from the assembler solutions. ;)

2.) Interesting idea: Using the approximation from the cited paper we should 
only need around 400 MB to store all primes in the int32 range. Potential for 
applying compression techniques still present. ^^

Regards
Christoph

--

Christoph Niethammer
High Performance Computing Center Stuttgart (HLRS)
Nobelstrasse 19
70569 Stuttgart

Tel: ++49(0)711-685-87203
email: nietham...@hlrs.de
http://www.hlrs.de/people/niethammer



- Ursprüngliche Mail -
Von: "Jeff Squyres (jsquyres)" <jsquy...@cisco.com>
An: "Open MPI Developers" <de...@open-mpi.org>
Gesendet: Montag, 10. Februar 2014 20:12:08
Betreff: Re: [OMPI devel] Reviewing MPI_Dims_create

Nice!  Can you verify that it passes the ibm test?  I didn't look closely, and 
to be honest, I'm not sure why the previous improvement broke the IBM test 
because it hypothetically did what you mentioned (stopped at sqrt(freenodes)).

I think patch 1 is a no-brainer.  I'm not sure about #2 because I'm not sure 
how it's different than the previous one, nor did I have time to investigate 
why the previous one broke the IBM test.  #3 seems like a good idea, too; I 
did't check the paper, but I assume it's some kind of proof about the upper 
limit on the number of primes in a given range.

Two questions:

1. Should we cache generated prime numbers?  (if so, it'll have to be done in a 
thread-safe way)

2. Should we just generate prime numbers and hard-code them into a table that 
is compiled into the code?  We would only need primes up to the sqrt of 
2billion (i.e., signed int), right?  I don't know how many that is -- if it's 
small enough, perhaps this is the easiest solution.



On Feb 10, 2014, at 1:30 PM, Christoph Niethammer <nietham...@hlrs.de> wrote:

> Hello,
> 
> I noticed some effort in improving the scalability of
> MPI_Dims_create(int nnodes, int ndims, int dims[])
> Unfortunately there were some issues with the first attempt (r30539 and 
> r30540) which were reverted.
> 
> So I decided to give it a short review based on r30606
> https://svn.open-mpi.org/trac/ompi/browser/trunk/ompi/mpi/c/dims_create.c?rev=30606
> 
> 
> 1.) freeprocs is initialized to be nnodes and the subsequent divisions of 
> freeprocs have all positive integers as divisor.
> So IMHO it would make more sense to check if nnodes > 0 in the 
> MPI_PARAM_CHECK section at the begin instead of the following (see patch 
> 0001):
> 
> 99if (freeprocs < 1) {
> 100  return OMPI_ERRHANDLER_INVOKE(MPI_COMM_WORLD, MPI_ERR_DIMS,
> 101FUNC_NAME);
> 102   }
> 
> 
> 2.) I rewrote the algorithm stopping at sqrt(n) in getprimes(int num, int 
> *nprimes, int **pprimes)
> which makes mathematically more sens (as the largest prime factor of any 
> number n cannot exceed \sqrt{n}) - and should produce the right result. ;)
> (see patch 0002)
> Here the improvements:
> 
> module load mpi/openmpi/trunk-gnu.4.7.3
> $ ./mpi-dims-old 100
> time used for MPI_Dims_create(100, 3, {}): 8.104007
> module swap mpi/openmpi/trunk-gnu.4.7.3 mpi/openmpi/trunk-gnu.4.7.3-testing
> $ ./mpi-dims-new 100
> time used for MPI_Dims_create(100, 3, {}): 0.060400
> 
> 
> 3.) Memory allocation for the list of prime numbers may be reduced up to a 
> factor of ~6 for 1mio nodes using the result from Dusart 1999 [1]:
> \pi(x)  < x/ln(x)(1+1.2762/ln(x))  for x > 1
> Unfortunately this saves us only 1.6 MB per process for 1mio nodes as 
> reported by tcmalloc/pprof on a test program - but it may sum up with fatter 
> nodes. :P
> 
> $ pprof --base=$PWD/primes-old.0001.heap a.out primes-new.0002.heap
> (pprof) top
&g

[OMPI devel] Reviewing MPI_Dims_create

[Christoph Niethammer]

Hello,

I noticed some effort in improving the scalability of
MPI_Dims_create(int nnodes, int ndims, int dims[])
Unfortunately there were some issues with the first attempt (r30539 and r30540) 
which were reverted.

So I decided to give it a short review based on r30606
https://svn.open-mpi.org/trac/ompi/browser/trunk/ompi/mpi/c/dims_create.c?rev=30606


1.) freeprocs is initialized to be nnodes and the subsequent divisions of 
freeprocs have all positive integers as divisor.
So IMHO it would make more sense to check if nnodes > 0 in the MPI_PARAM_CHECK 
section at the begin instead of the following (see patch 0001):

99  if (freeprocs < 1) {
100return OMPI_ERRHANDLER_INVOKE(MPI_COMM_WORLD, MPI_ERR_DIMS,
101  FUNC_NAME);
102 }


2.) I rewrote the algorithm stopping at sqrt(n) in getprimes(int num, int 
*nprimes, int **pprimes)
which makes mathematically more sens (as the largest prime factor of any number 
n cannot exceed \sqrt{n}) - and should produce the right result. ;)
(see patch 0002)
Here the improvements:

module load mpi/openmpi/trunk-gnu.4.7.3
$ ./mpi-dims-old 100
time used for MPI_Dims_create(100, 3, {}): 8.104007
module swap mpi/openmpi/trunk-gnu.4.7.3 mpi/openmpi/trunk-gnu.4.7.3-testing
$ ./mpi-dims-new 100
time used for MPI_Dims_create(100, 3, {}): 0.060400


3.) Memory allocation for the list of prime numbers may be reduced up to a 
factor of ~6 for 1mio nodes using the result from Dusart 1999 [1]:
\pi(x)  < x/ln(x)(1+1.2762/ln(x))  for x > 1
Unfortunately this saves us only 1.6 MB per process for 1mio nodes as reported 
by tcmalloc/pprof on a test program - but it may sum up with fatter nodes. :P

$ pprof --base=$PWD/primes-old.0001.heap a.out primes-new.0002.heap
(pprof) top
Total: -1.6 MB
 0.3 -18.8% -18.8%  0.3 -18.8% getprimes2
 0.0  -0.0% -18.8% -1.6 100.0% __libc_start_main
 0.0  -0.0% -18.8% -1.6 100.0% main
-1.9 118.8% 100.0% -1.9 118.8% getprimes

Find attached patch for it in 0003.


If there are no issues I would like to commit this to trunk for further testing 
(+cmr for 1.7.5?) end of this week.

Best regards
Christoph

[1] http://www.ams.org/journals/mcom/1999-68-225/S0025-5718-99-01037-6/home.html



--

Christoph Niethammer
High Performance Computing Center Stuttgart (HLRS)
Nobelstrasse 19
70569 Stuttgart

Tel: ++49(0)711-685-87203
email: nietham...@hlrs.de
http://www.hlrs.de/people/niethammerFrom e3292b90cac42fad80ed27a555419002ed61ab66 Mon Sep 17 00:00:00 2001
From: Christoph Niethammer <christoph.nietham...@hlrs.de>
Date: Mon, 10 Feb 2014 16:44:03 +0100
Subject: [PATCH 1/3] Move parameter check into appropriate code section at the
 begin.

---
 ompi/mpi/c/dims_create.c | 11 ++-
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/ompi/mpi/c/dims_create.c b/ompi/mpi/c/dims_create.c
index d2c3858..3d0792f 100644
--- a/ompi/mpi/c/dims_create.c
+++ b/ompi/mpi/c/dims_create.c
@@ -71,6 +71,11 @@ int MPI_Dims_create(int nnodes, int ndims, int dims[])
 return OMPI_ERRHANDLER_INVOKE (MPI_COMM_WORLD, 
MPI_ERR_DIMS, FUNC_NAME);
 }
+
+if (1 > nnodes) {
+return OMPI_ERRHANDLER_INVOKE (MPI_COMM_WORLD,
+   MPI_ERR_DIMS, FUNC_NAME);
+}
 }
 
 /* Get # of free-to-be-assigned processes and # of free dimensions */
@@ -95,11 +100,7 @@ int MPI_Dims_create(int nnodes, int ndims, int dims[])
  FUNC_NAME);
 }
 
-if (freeprocs < 1) {
-   return OMPI_ERRHANDLER_INVOKE(MPI_COMM_WORLD, MPI_ERR_DIMS,
- FUNC_NAME);
-}
-else if (freeprocs == 1) {
+if (freeprocs == 1) {
 for (i = 0; i < ndims; ++i, ++dims) {
 if (*dims == 0) {
*dims = 1;
-- 
1.8.3.2

From bc862c47ef8d581a8f6735c51983d6c9eeb95dfd Mon Sep 17 00:00:00 2001
From: Christoph Niethammer <christoph.nietham...@hlrs.de>
Date: Mon, 10 Feb 2014 18:50:51 +0100
Subject: [PATCH 2/3] Speeding up detection of prime numbers using the fact
 that the largest prime factor of any number n cannot exceed \sqrt{n}.

---
 ompi/mpi/c/dims_create.c | 9 ++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/ompi/mpi/c/dims_create.c b/ompi/mpi/c/dims_create.c
index 3d0792f..1c1c381 100644
--- a/ompi/mpi/c/dims_create.c
+++ b/ompi/mpi/c/dims_create.c
@@ -5,7 +5,7 @@
  * Copyright (c) 2004-2005 The University of Tennessee and The University
  * of Tennessee Research Foundation.  All rights
  * reserved.
- * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, 
+ * Copyright (c) 2004-2014 High Performance Computing Center Stuttgart, 
  * University of Stuttgart.  All rights reserved.
  * Copyright (c) 2004-2005 The Regent

[OMPI devel] mca_bml_r2_del_btl incorrect memory size reallocation?

[Christoph Niethammer]

Hello

I think I found a minor memory bug in the bml_r2 code in function 
mca_bml_r2_del_btl but I could not figure out when this function ever gets 
called.
How can I test this function in a proper way?

Here the diff showing the issue:

@@ -699,11 +699,11 @@ static int mca_bml_r2_del_btl(mca_btl_base_module_t* btl)
 if(!found) {
 /* doesn't even exist */
 goto CLEANUP;
 }
 /* remove from bml list */
-modules = (mca_btl_base_module_t**)malloc(sizeof(mca_btl_base_module_t*) * 
mca_bml_r2.num_btl_modules-1);
+modules = (mca_btl_base_module_t**)malloc(sizeof(mca_btl_base_module_t*) * 
(mca_bml_r2.num_btl_modules-1));
 for(i=0,m=0; i<mca_bml_r2.num_btl_modules; i++) {
 if(mca_bml_r2.btl_modules[i] != btl) {
 modules[m++] = mca_bml_r2.btl_modules[i];
 }
 }


Regards
Christoph

--

Christoph Niethammer
High Performance Computing Center Stuttgart (HLRS)
Nobelstrasse 19
70569 Stuttgart

Tel: ++49(0)711-685-87203
email: nietham...@hlrs.de
http://www.hlrs.de/people/niethammer

Re: [OMPI devel] Missing --bycore option in Open MPI 1.7.?

[Christoph Niethammer]

Hi,

Just found the following ticket which answers my question:
https://svn.open-mpi.org/trac/ompi/ticket/4044

Sorry for spam. :/

Regards
Christoph


- Ursprüngliche Mail -
Von: "Christoph Niethammer" <nietham...@hlrs.de>
An: "Open MPI Developers" <de...@open-mpi.org>
Gesendet: Mittwoch, 8. Januar 2014 14:49:54
Betreff: Missing --bycore option in Open MPI 1.7.?

Hello

Using Open MPI 1.7.3 I got the following error message when executing
mpirun -np 16 --bycore /bin/hostname

mpirun: Error: unknown option "--bycore"

The option is documented in the man pages and with Open MPI 1.6.5 everything 
works fine.
For --bysocket I get the same error but --bynode seems to be working.
Is this option on the way to be removed?

Regards
Christoph

[OMPI devel] Missing --bycore option in Open MPI 1.7.?

[Christoph Niethammer]

Hello

Using Open MPI 1.7.3 I got the following error message when executing
mpirun -np 16 --bycore /bin/hostname

mpirun: Error: unknown option "--bycore"

The option is documented in the man pages and with Open MPI 1.6.5 everything 
works fine.
For --bysocket I get the same error but --bynode seems to be working.
Is this option on the way to be removed?

Regards
Christoph

[MTT devel] mtt-relay patch - create pid file when run as daemon

[Christoph Niethammer]

Hello,

As on many systems init scripts and the handling of services is based on pid 
files I extended the mtt-relay script as follows:

If run with the --daemon option
* Create file /var/run/mtt-relay.pid  if it does not exist and write the pid of 
the background process into it.
* exit with return value 1 if /var/run/mtt-relay.pid file exists.

Patch is attached.

Best regards
Christoph Niethammer

--

Christoph Niethammer
High Performance Computing Center Stuttgart (HLRS)
Nobelstrasse 19
70569 Stuttgart

Tel: ++49(0)711-685-87203
email: nietham...@hlrs.de
http://www.hlrs.de/people/niethammer--- /home/hpcmtt/mtt-trunk/client/mtt-relay.orig2013-09-30 11:35:59.637400212 +0200
+++ /home/hpcmtt/mtt-trunk/client/mtt-relay 2013-09-30 11:45:19.496180413 +0200
@@ -93,6 +93,10 @@
 # exit;
 # }
 
+my $pidfilename = "/var/run/mtt-relay.pid";
+if (-e $pidfilename) {
+exit 1;
+}
 my $master = new HTTP::Daemon
 LocalAddr => $HOST, LocalPort => $PORT or 
 die "Problem creating an HTTP::Daemon: $!";
@@ -103,6 +107,9 @@
 my $pid = fork();
 if($pid) {
 print "# Daemon Parent exiting\n";
+open PIDFILE, ">$pidfilename";
+print PIDFILE $pid;
+close PIDFILE;
 exit 0;
 } else {
 print "# Daemon Child process continuing\n";

Re: [OMPI devel] Annual OMPI membership review: SVN accounts

[Christoph Niethammer]

You can remove Shiqing.

Rainer should be listed in future under hft-stuttgart.de. ;)

Regards
Christoph



> hlrs.de
> ===
> shiqing:  Shiqing Fan <f...@hlrs.de>
> hpcchris: Christoph Niethammer <nietham...@hlrs.de>
> rusraink: Rainer Keller <rainer.kel...@hft-stuttgart.de> **NO COMMITS IN LAST 
> YEAR**

[OMPI devel] Datasize confusion in MPI_Write can lead to data los!

[Christoph Niethammer]

Hello!

I tested openMPI at HLRS for some time without detecting new problems in the 
implementation but now I recognized some awful ones with MPI_Write which can 
lead to data los:

When creating a struct for a mixed datatype like

struct {
  short a;
  int b;
}

the C-compiler introduce a gap of 2 bytes in the data representation for this 
type due to the 4byte alignment of the integer on 32bit systems.

If I now try to use MPI_File_write to write these data to a file and use 
MPI_SHORT_INT as mpi_datatype this leads to a data los.

I located the problem at the combined use of "write" and MPI_Type_size in 
MPI_File_write.
So MPI_Type_size(MPI_SHORT_INT) returns 6 bytes where the struct uses 8 bytes 
in memory as there is a gap of 2 bytes. The write function in ad_write.c now 
leads to the los of the data because the gaps are not within the calculation 
of the complete data size to be written into the file.

This problem occures also in the other io functions.
As far as I could find out the problem seems not to be present with derived 
data types.

The question is now how to "fix":
i) Either the MPI_Standard is not clear in this point and the data types 
MPI_SHORT_INT, MPI_DOUBLE_INT, ... should be forbidden to be used with 
structs of these types,
ii) Or the implementation of the MPI_Type_size function has to be modified to 
return the value of eg. true_ub which contains the correct value
iii) Or the MPI_File_write function has not to use the write function in 
the "continues" way on the data and should take care of the gaps.

Regards 

Christoph Niethammer


signature.asc
Description: This is a digitally signed message part.

[OMPI devel] patch for btl_sm.c fixing segmentation fault

[Christoph Niethammer]

Hello,


Since some time I'm testing Open MPI at the HRLS. My main topic there is the 
thread support of Open MPI.

Some time ago I found a segmentation fault when running the svn-trunk Version. 
Thanks to the help of Sven I could locate it now to be in the shared memory 
btl. (ompi/mca/btl/sm/btl_sm.c)
There the addresses of the fifos were modified because of the different memory 
mapping for the threads. Unfortunately this modification was not applied for 
the head_locks. 

The attached patch should fix this problem.
Maybe someone could have a look on it?

Regards 

Christoph
Index: ompi/mca/btl/sm/btl_sm.c
===
--- ompi/mca/btl/sm/btl_sm.c	(Revision 15143)
+++ ompi/mca/btl/sm/btl_sm.c	(Arbeitskopie)
@@ -516,8 +516,11 @@
 /* Calculate the difference as (my_base - their_base) */
 diff = tmp_ptr[mca_btl_sm_component.my_smp_rank] - tmp_ptr[j];
 mca_btl_sm_component.fifo[j] = (ompi_fifo_t*)((char*)fifo_tmp[j]+diff);
+
+mca_btl_sm_component.fifo[j][mca_btl_sm_component.my_smp_rank].head_lock =
+  (opal_atomic_lock_t*) ((char*)mca_btl_sm_component.fifo[j][mca_btl_sm_component.my_smp_rank].head_lock + diff);
+
 mca_btl_sm_component.sm_offset[j] = diff;
-
 }
 
 for( j=mca_btl_sm_component.num_smp_procs ; j <


pgpDKJG53Fh1y.pgp
Description: PGP signature