Re: [OMPI users] MPI_IN_PLACE

[Volker Blum]

Update:

> I have yet to run a full regression test on our actual code to ensure that 
> there are no other side effects. I don’t expect any, though.

Indeed. Fixes all regressions that I had observed.

Best wishes
Volker

> On Jul 27, 2017, at 11:47 AM, Volker Blum  wrote:
> 
> Dear Gilles,
> 
> Thank you! Indeed, this appears to address the issue in my test.
> 
> I have yet to run a full regression test on our actual code to ensure that 
> there are no other side effects. I don’t expect any, though.
> 
> **
> 
> Interestingly, removing '-lmpi_usempif08 -lmpi_usempi_ignore_tkr’ actually 
> fixes a second regression that I had found only last night and not yet 
> reported.
> 
> Specifically, I have a very well tested mpi_alltoallv call (part of our 
> standard regression tests). Versions of this exact call live in multiple 
> codes, including libraries. With the unpatched OpenMPI version on the same 
> macbook and in a specific case, this suddenly resulted in:
> 
> [egr-vwb3-mbp2:3573] *** An error occurred in MPI_Alltoallv
> [egr-vwb3-mbp2:3573] *** reported by process [1710358529,0]
> [egr-vwb3-mbp2:3573] *** on communicator MPI COMMUNICATOR 7 SPLIT FROM 0
> [egr-vwb3-mbp2:3573] *** MPI_ERR_TRUNCATE: message truncated
> [egr-vwb3-mbp2:3573] *** MPI_ERRORS_ARE_FATAL (processes in this communicator 
> will now abort,
> [egr-vwb3-mbp2:3573] ***and potentially your MPI job)
> [egr-vwb3-mbp2.local:03572] 1 more process has sent help message 
> help-mpi-errors.txt / mpi_errors_are_fatal
> [egr-vwb3-mbp2.local:03572] Set MCA parameter "orte_base_help_aggregate" to 0 
> to see all help / error messages
> 
> This regression is also fixed. 
> 
> I do not have a simple test case for this behavior yet since I was still 
> investigating it. My plan was to create a test case, but extracting such a 
> case from the larger code we have is more difficult. 
> 
> I can still try to create a test case for this as well but I’ll be watching 
> with great interest if the earlier MPI_IN_PLACE regression can be nailed down 
> and fixed in a more general way. Perhaps the two are related.
> 
> Thanks again,
> Best wishes
> Volker
> 
>> On Jul 27, 2017, at 10:46 AM, Gilles Gouaillardet  wrote:
>> 
>> Volker,
>> 
>> 
>> since you are only using
>> 
>> include 'mpif.h'
>> 
>> 
>> a workaround is you edit your /.../share/openmpi/mpifort-wrapper-data.txt
>> and simply remove '-lmpi_usempif08 -lmpi_usempi_ignore_tkr'
>> 
>> 
>> Cheers,
>> 
>> Gilles
>> 
>> On 7/27/2017 3:28 PM, Volker Blum wrote:
>>> Thanks!
>>> 
>>> If you wish, please also keep me posted.
>>> 
>>> Best wishes
>>> Volker
>>> 
 On Jul 27, 2017, at 7:50 AM, Gilles Gouaillardet 
  wrote:
 
 Thanks Jeff for your offer, i will contact you off-list later
 
 
 i tried a gcc+gfortran and gcc+ifort on both linux and OS X
 so far, only gcc+ifort on OS X is failing
 i will try icc+ifort on OS X from now
 
 short story, MPI_IN_PLACE is not recognized as such by the ompi
 fortran wrapper, and i do not know why.
 
 the attached program can be used to evidence the issue.
 
 
 Cheers,
 
 Gilles
 
 On Thu, Jul 27, 2017 at 2:15 PM, Volker Blum  wrote:
> Thanks! That’s great. Sounds like the exact combination I have here.
> 
> Thanks also to George. Sorry that the test did not trigger on a more 
> standard platform - that would have simplified things.
> 
> Best wishes
> Volker
> 
>> On Jul 27, 2017, at 3:56 AM, Gilles Gouaillardet  
>> wrote:
>> 
>> Folks,
>> 
>> 
>> I am able to reproduce the issue on OS X (Sierra) with stock gcc (aka 
>> clang) and ifort 17.0.4
>> 
>> 
>> i will investigate this from now
>> 
>> 
>> Cheers,
>> 
>> Gilles
>> 
>> On 7/27/2017 9:28 AM, George Bosilca wrote:
>>> Volker,
>>> 
>>> Unfortunately, I can't replicate with icc. I tried on a x86_64 box with 
>>> Intel compiler chain 17.0.4 20170411 to no avail. I also tested the 
>>> 3.0.0-rc1 tarball and the current master, and you test completes 
>>> without errors on all cases.
>>> 
>>> Once you figure out an environment where you can consistently replicate 
>>> the issue, I would suggest to attach to the processes and:
>>> - make sure the MPI_IN_PLACE as seen through the Fortran layer matches 
>>> what the C layer expects
>>> - what is the collective algorithm used by Open MPI
>>> 
>>> I have a "Fortran 101" level question. When you pass an array a(:) as 
>>> argument, what exactly gets passed via the Fortran interface to the 
>>> corresponding C function ?
>>> 
>>> George.
>>> 
>>> On Wed, Jul 26, 2017 at 1:55 PM, Volker Blum >> > wrote:
>>> 
>>>  Thanks! Yes, trying with Intel 2017 would be very nice.
>>> 
 On Jul 26, 2017, at 6:12 PM, George Bosilca >>  > wr

Re: [OMPI users] MPI_IN_PLACE

[Volker Blum]

Dear Gilles,

Thank you! Indeed, this appears to address the issue in my test.

I have yet to run a full regression test on our actual code to ensure that 
there are no other side effects. I don’t expect any, though.

**

Interestingly, removing '-lmpi_usempif08 -lmpi_usempi_ignore_tkr’ actually 
fixes a second regression that I had found only last night and not yet reported.

Specifically, I have a very well tested mpi_alltoallv call (part of our 
standard regression tests). Versions of this exact call live in multiple codes, 
including libraries. With the unpatched OpenMPI version on the same macbook and 
in a specific case, this suddenly resulted in:

[egr-vwb3-mbp2:3573] *** An error occurred in MPI_Alltoallv
[egr-vwb3-mbp2:3573] *** reported by process [1710358529,0]
[egr-vwb3-mbp2:3573] *** on communicator MPI COMMUNICATOR 7 SPLIT FROM 0
[egr-vwb3-mbp2:3573] *** MPI_ERR_TRUNCATE: message truncated
[egr-vwb3-mbp2:3573] *** MPI_ERRORS_ARE_FATAL (processes in this communicator 
will now abort,
[egr-vwb3-mbp2:3573] ***and potentially your MPI job)
[egr-vwb3-mbp2.local:03572] 1 more process has sent help message 
help-mpi-errors.txt / mpi_errors_are_fatal
[egr-vwb3-mbp2.local:03572] Set MCA parameter "orte_base_help_aggregate" to 0 
to see all help / error messages

This regression is also fixed. 

I do not have a simple test case for this behavior yet since I was still 
investigating it. My plan was to create a test case, but extracting such a case 
from the larger code we have is more difficult. 

I can still try to create a test case for this as well but I’ll be watching 
with great interest if the earlier MPI_IN_PLACE regression can be nailed down 
and fixed in a more general way. Perhaps the two are related.

Thanks again,
Best wishes
Volker

> On Jul 27, 2017, at 10:46 AM, Gilles Gouaillardet  wrote:
> 
> Volker,
> 
> 
> since you are only using
> 
> include 'mpif.h'
> 
> 
> a workaround is you edit your /.../share/openmpi/mpifort-wrapper-data.txt
> and simply remove '-lmpi_usempif08 -lmpi_usempi_ignore_tkr'
> 
> 
> Cheers,
> 
> Gilles
> 
> On 7/27/2017 3:28 PM, Volker Blum wrote:
>> Thanks!
>> 
>> If you wish, please also keep me posted.
>> 
>> Best wishes
>> Volker
>> 
>>> On Jul 27, 2017, at 7:50 AM, Gilles Gouaillardet 
>>>  wrote:
>>> 
>>> Thanks Jeff for your offer, i will contact you off-list later
>>> 
>>> 
>>> i tried a gcc+gfortran and gcc+ifort on both linux and OS X
>>> so far, only gcc+ifort on OS X is failing
>>> i will try icc+ifort on OS X from now
>>> 
>>> short story, MPI_IN_PLACE is not recognized as such by the ompi
>>> fortran wrapper, and i do not know why.
>>> 
>>> the attached program can be used to evidence the issue.
>>> 
>>> 
>>> Cheers,
>>> 
>>> Gilles
>>> 
>>> On Thu, Jul 27, 2017 at 2:15 PM, Volker Blum  wrote:
 Thanks! That’s great. Sounds like the exact combination I have here.
 
 Thanks also to George. Sorry that the test did not trigger on a more 
 standard platform - that would have simplified things.
 
 Best wishes
 Volker
 
> On Jul 27, 2017, at 3:56 AM, Gilles Gouaillardet  
> wrote:
> 
> Folks,
> 
> 
> I am able to reproduce the issue on OS X (Sierra) with stock gcc (aka 
> clang) and ifort 17.0.4
> 
> 
> i will investigate this from now
> 
> 
> Cheers,
> 
> Gilles
> 
> On 7/27/2017 9:28 AM, George Bosilca wrote:
>> Volker,
>> 
>> Unfortunately, I can't replicate with icc. I tried on a x86_64 box with 
>> Intel compiler chain 17.0.4 20170411 to no avail. I also tested the 
>> 3.0.0-rc1 tarball and the current master, and you test completes without 
>> errors on all cases.
>> 
>> Once you figure out an environment where you can consistently replicate 
>> the issue, I would suggest to attach to the processes and:
>> - make sure the MPI_IN_PLACE as seen through the Fortran layer matches 
>> what the C layer expects
>> - what is the collective algorithm used by Open MPI
>> 
>> I have a "Fortran 101" level question. When you pass an array a(:) as 
>> argument, what exactly gets passed via the Fortran interface to the 
>> corresponding C function ?
>> 
>> George.
>> 
>> On Wed, Jul 26, 2017 at 1:55 PM, Volker Blum > > wrote:
>> 
>>   Thanks! Yes, trying with Intel 2017 would be very nice.
>> 
>>> On Jul 26, 2017, at 6:12 PM, George Bosilca >   > wrote:
>>> No, I don't have (or used where they were available) the Intel
>>   compiler. I used clang and gfortran. I can try on a Linux box with
>>   the Intel 2017 compilers.
>>>  George.
>>> 
>>> 
>>> 
>>> On Wed, Jul 26, 2017 at 11:59 AM, Volker Blum
>>   mailto:volker.b...@duke.edu>> wrote:
>>> Did you use Intel Fortran 2017 as well?
>>> 
>>> (I’m asking because I did see the same issue with a combination

Re: [OMPI users] MPI_IN_PLACE

[Gilles Gouaillardet]

Volker,


since you are only using

include 'mpif.h'


a workaround is you edit your /.../share/openmpi/mpifort-wrapper-data.txt
and simply remove '-lmpi_usempif08 -lmpi_usempi_ignore_tkr'


Cheers,

Gilles

On 7/27/2017 3:28 PM, Volker Blum wrote:

Thanks!

If you wish, please also keep me posted.

Best wishes
Volker


On Jul 27, 2017, at 7:50 AM, Gilles Gouaillardet 
 wrote:

Thanks Jeff for your offer, i will contact you off-list later


i tried a gcc+gfortran and gcc+ifort on both linux and OS X
so far, only gcc+ifort on OS X is failing
i will try icc+ifort on OS X from now

short story, MPI_IN_PLACE is not recognized as such by the ompi
fortran wrapper, and i do not know why.

the attached program can be used to evidence the issue.


Cheers,

Gilles

On Thu, Jul 27, 2017 at 2:15 PM, Volker Blum  wrote:

Thanks! That’s great. Sounds like the exact combination I have here.

Thanks also to George. Sorry that the test did not trigger on a more standard 
platform - that would have simplified things.

Best wishes
Volker


On Jul 27, 2017, at 3:56 AM, Gilles Gouaillardet  wrote:

Folks,


I am able to reproduce the issue on OS X (Sierra) with stock gcc (aka clang) 
and ifort 17.0.4


i will investigate this from now


Cheers,

Gilles

On 7/27/2017 9:28 AM, George Bosilca wrote:

Volker,

Unfortunately, I can't replicate with icc. I tried on a x86_64 box with Intel 
compiler chain 17.0.4 20170411 to no avail. I also tested the 3.0.0-rc1 tarball 
and the current master, and you test completes without errors on all cases.

Once you figure out an environment where you can consistently replicate the 
issue, I would suggest to attach to the processes and:
- make sure the MPI_IN_PLACE as seen through the Fortran layer matches what the 
C layer expects
- what is the collective algorithm used by Open MPI

I have a "Fortran 101" level question. When you pass an array a(:) as argument, 
what exactly gets passed via the Fortran interface to the corresponding C function ?

George.

On Wed, Jul 26, 2017 at 1:55 PM, Volker Blum mailto:volker.b...@duke.edu>> wrote:

   Thanks! Yes, trying with Intel 2017 would be very nice.


On Jul 26, 2017, at 6:12 PM, George Bosilca 
   > wrote:

No, I don't have (or used where they were available) the Intel

   compiler. I used clang and gfortran. I can try on a Linux box with
   the Intel 2017 compilers.

  George.



On Wed, Jul 26, 2017 at 11:59 AM, Volker Blum

   mailto:volker.b...@duke.edu>> wrote:

Did you use Intel Fortran 2017 as well?

(I’m asking because I did see the same issue with a combination

   of an earlier Intel Fortran 2017 version and OpenMPI on an
   Intel/Infiniband Linux HPC machine … but not Intel Fortran 2016 on
   the same machine. Perhaps I can revive my access to that
   combination somehow.)

Best wishes
Volker


On Jul 26, 2017, at 5:55 PM, George Bosilca

   mailto:bosi...@icl.utk.edu>> wrote:

I thought that maybe the underlying allreduce algorithm fails

   to support MPI_IN_PLACE correctly, but I can't replicate on any
   machine (including OSX) with any number of processes.

  George.



On Wed, Jul 26, 2017 at 10:59 AM, Volker Blum

   mailto:volker.b...@duke.edu>> wrote:

Thanks!

I tried ‘use mpi’, which compiles fine.

Same result as with ‘include mpif.h', in that the output is

* MPI_IN_PLACE does not appear to work as intended.
* Checking whether MPI_ALLREDUCE works at all.
* Without MPI_IN_PLACE, MPI_ALLREDUCE appears to work.

Hm. Any other thoughts?

Thanks again!
Best wishes
Volker


On Jul 26, 2017, at 4:06 PM, Gilles Gouaillardet

   mailto:gilles.gouaillar...@gmail.com>> wrote:

Volker,

With mpi_f08, you have to declare

Type(MPI_Comm) :: mpi_comm_global

(I am afk and not 100% sure of the syntax)

A simpler option is to

use mpi

Cheers,

Gilles

Volker Blum 
   > wrote:

Hi Gilles,

Thank you very much for the response!

Unfortunately, I don’t have access to a different system

   with the issue right now. As I said, it’s not new; it just keeps
   creeping up unexpectedly again on different platforms. What
   puzzles me is that I’ve encountered the same problem with low but
   reasonable frequency over a period of now over five years.

We can’t require F’08 in our application, unfortunately,

   since this standard is too new. Since we maintain a large
   application that has to run on a broad range of platforms, Fortran
   2008 would not work for many of our users. In a few years, this
   will be different, but not yet.

On gfortran: In our own tests, unfortunately, Intel Fortran

   consistently produced much faster executable code in the past. The
   latter observation may also change someday, but for us, the
   performance difference was an important constraint.

I did suspect mpif.h, too. Not sure how to best test this

   hypothesis, however.

Just replacing


include 'mpif.h'
with
use mpi_f08

did not work, for me.

This produces a number of compilation errors:

blum:/Us

Re: [OMPI users] MPI_IN_PLACE

[Volker Blum]

Thanks!

If you wish, please also keep me posted. 

Best wishes
Volker

> On Jul 27, 2017, at 7:50 AM, Gilles Gouaillardet 
>  wrote:
> 
> Thanks Jeff for your offer, i will contact you off-list later
> 
> 
> i tried a gcc+gfortran and gcc+ifort on both linux and OS X
> so far, only gcc+ifort on OS X is failing
> i will try icc+ifort on OS X from now
> 
> short story, MPI_IN_PLACE is not recognized as such by the ompi
> fortran wrapper, and i do not know why.
> 
> the attached program can be used to evidence the issue.
> 
> 
> Cheers,
> 
> Gilles
> 
> On Thu, Jul 27, 2017 at 2:15 PM, Volker Blum  wrote:
>> Thanks! That’s great. Sounds like the exact combination I have here.
>> 
>> Thanks also to George. Sorry that the test did not trigger on a more 
>> standard platform - that would have simplified things.
>> 
>> Best wishes
>> Volker
>> 
>>> On Jul 27, 2017, at 3:56 AM, Gilles Gouaillardet  wrote:
>>> 
>>> Folks,
>>> 
>>> 
>>> I am able to reproduce the issue on OS X (Sierra) with stock gcc (aka 
>>> clang) and ifort 17.0.4
>>> 
>>> 
>>> i will investigate this from now
>>> 
>>> 
>>> Cheers,
>>> 
>>> Gilles
>>> 
>>> On 7/27/2017 9:28 AM, George Bosilca wrote:
 Volker,
 
 Unfortunately, I can't replicate with icc. I tried on a x86_64 box with 
 Intel compiler chain 17.0.4 20170411 to no avail. I also tested the 
 3.0.0-rc1 tarball and the current master, and you test completes without 
 errors on all cases.
 
 Once you figure out an environment where you can consistently replicate 
 the issue, I would suggest to attach to the processes and:
 - make sure the MPI_IN_PLACE as seen through the Fortran layer matches 
 what the C layer expects
 - what is the collective algorithm used by Open MPI
 
 I have a "Fortran 101" level question. When you pass an array a(:) as 
 argument, what exactly gets passed via the Fortran interface to the 
 corresponding C function ?
 
 George.
 
 On Wed, Jul 26, 2017 at 1:55 PM, Volker Blum >>> > wrote:
 
   Thanks! Yes, trying with Intel 2017 would be very nice.
 
> On Jul 26, 2017, at 6:12 PM, George Bosilca >>>   > wrote:
> 
> No, I don't have (or used where they were available) the Intel
   compiler. I used clang and gfortran. I can try on a Linux box with
   the Intel 2017 compilers.
> 
>  George.
> 
> 
> 
> On Wed, Jul 26, 2017 at 11:59 AM, Volker Blum
   mailto:volker.b...@duke.edu>> wrote:
> Did you use Intel Fortran 2017 as well?
> 
> (I’m asking because I did see the same issue with a combination
   of an earlier Intel Fortran 2017 version and OpenMPI on an
   Intel/Infiniband Linux HPC machine … but not Intel Fortran 2016 on
   the same machine. Perhaps I can revive my access to that
   combination somehow.)
> 
> Best wishes
> Volker
> 
>> On Jul 26, 2017, at 5:55 PM, George Bosilca
   mailto:bosi...@icl.utk.edu>> wrote:
>> 
>> I thought that maybe the underlying allreduce algorithm fails
   to support MPI_IN_PLACE correctly, but I can't replicate on any
   machine (including OSX) with any number of processes.
>> 
>>  George.
>> 
>> 
>> 
>> On Wed, Jul 26, 2017 at 10:59 AM, Volker Blum
   mailto:volker.b...@duke.edu>> wrote:
>> Thanks!
>> 
>> I tried ‘use mpi’, which compiles fine.
>> 
>> Same result as with ‘include mpif.h', in that the output is
>> 
>> * MPI_IN_PLACE does not appear to work as intended.
>> * Checking whether MPI_ALLREDUCE works at all.
>> * Without MPI_IN_PLACE, MPI_ALLREDUCE appears to work.
>> 
>> Hm. Any other thoughts?
>> 
>> Thanks again!
>> Best wishes
>> Volker
>> 
>>> On Jul 26, 2017, at 4:06 PM, Gilles Gouaillardet
   >>>   > wrote:
>>> 
>>> Volker,
>>> 
>>> With mpi_f08, you have to declare
>>> 
>>> Type(MPI_Comm) :: mpi_comm_global
>>> 
>>> (I am afk and not 100% sure of the syntax)
>>> 
>>> A simpler option is to
>>> 
>>> use mpi
>>> 
>>> Cheers,
>>> 
>>> Gilles
>>> 
>>> Volker Blum >>>   > wrote:
 Hi Gilles,
 
 Thank you very much for the response!
 
 Unfortunately, I don’t have access to a different system
   with the issue right now. As I said, it’s not new; it just keeps
   creeping up unexpectedly again on different platforms. What
   puzzles me is that I’ve encountered the same problem with low but
   reasonable frequency over a period of now over five years.
 
 We can’t require F’08 in our application, unfortunately,
   since this standard is too new. Since we maintain a large
   application that has to run on a broad range of platforms, Fortran
   2008 wo

Re: [OMPI users] MPI_IN_PLACE

[Gilles Gouaillardet]

Thanks Jeff for your offer, i will contact you off-list later


i tried a gcc+gfortran and gcc+ifort on both linux and OS X
so far, only gcc+ifort on OS X is failing
i will try icc+ifort on OS X from now

short story, MPI_IN_PLACE is not recognized as such by the ompi
fortran wrapper, and i do not know why.

the attached program can be used to evidence the issue.


Cheers,

Gilles

On Thu, Jul 27, 2017 at 2:15 PM, Volker Blum  wrote:
> Thanks! That’s great. Sounds like the exact combination I have here.
>
> Thanks also to George. Sorry that the test did not trigger on a more standard 
> platform - that would have simplified things.
>
> Best wishes
> Volker
>
>> On Jul 27, 2017, at 3:56 AM, Gilles Gouaillardet  wrote:
>>
>> Folks,
>>
>>
>> I am able to reproduce the issue on OS X (Sierra) with stock gcc (aka clang) 
>> and ifort 17.0.4
>>
>>
>> i will investigate this from now
>>
>>
>> Cheers,
>>
>> Gilles
>>
>> On 7/27/2017 9:28 AM, George Bosilca wrote:
>>> Volker,
>>>
>>> Unfortunately, I can't replicate with icc. I tried on a x86_64 box with 
>>> Intel compiler chain 17.0.4 20170411 to no avail. I also tested the 
>>> 3.0.0-rc1 tarball and the current master, and you test completes without 
>>> errors on all cases.
>>>
>>> Once you figure out an environment where you can consistently replicate the 
>>> issue, I would suggest to attach to the processes and:
>>> - make sure the MPI_IN_PLACE as seen through the Fortran layer matches what 
>>> the C layer expects
>>> - what is the collective algorithm used by Open MPI
>>>
>>> I have a "Fortran 101" level question. When you pass an array a(:) as 
>>> argument, what exactly gets passed via the Fortran interface to the 
>>> corresponding C function ?
>>>
>>>  George.
>>>
>>> On Wed, Jul 26, 2017 at 1:55 PM, Volker Blum >> > wrote:
>>>
>>>Thanks! Yes, trying with Intel 2017 would be very nice.
>>>
>>>> On Jul 26, 2017, at 6:12 PM, George Bosilca >>> wrote:
>>>>
>>>> No, I don't have (or used where they were available) the Intel
>>>compiler. I used clang and gfortran. I can try on a Linux box with
>>>the Intel 2017 compilers.
>>>>
>>>>   George.
>>>>
>>>>
>>>>
>>>> On Wed, Jul 26, 2017 at 11:59 AM, Volker Blum
>>>mailto:volker.b...@duke.edu>> wrote:
>>>> Did you use Intel Fortran 2017 as well?
>>>>
>>>> (I’m asking because I did see the same issue with a combination
>>>of an earlier Intel Fortran 2017 version and OpenMPI on an
>>>Intel/Infiniband Linux HPC machine … but not Intel Fortran 2016 on
>>>the same machine. Perhaps I can revive my access to that
>>>combination somehow.)
>>>>
>>>> Best wishes
>>>> Volker
>>>>
>>>> > On Jul 26, 2017, at 5:55 PM, George Bosilca
>>>mailto:bosi...@icl.utk.edu>> wrote:
>>>> >
>>>> > I thought that maybe the underlying allreduce algorithm fails
>>>to support MPI_IN_PLACE correctly, but I can't replicate on any
>>>machine (including OSX) with any number of processes.
>>>> >
>>>> >   George.
>>>> >
>>>> >
>>>> >
>>>> > On Wed, Jul 26, 2017 at 10:59 AM, Volker Blum
>>>mailto:volker.b...@duke.edu>> wrote:
>>>> > Thanks!
>>>> >
>>>> > I tried ‘use mpi’, which compiles fine.
>>>> >
>>>> > Same result as with ‘include mpif.h', in that the output is
>>>> >
>>>> >  * MPI_IN_PLACE does not appear to work as intended.
>>>> >  * Checking whether MPI_ALLREDUCE works at all.
>>>> >  * Without MPI_IN_PLACE, MPI_ALLREDUCE appears to work.
>>>> >
>>>> > Hm. Any other thoughts?
>>>> >
>>>> > Thanks again!
>>>> > Best wishes
>>>> > Volker
>>>> >
>>>> > > On Jul 26, 2017, at 4:06 PM, Gilles Gouaillardet
>>>>>> wrote:
>>>> > >
>>>> > > Volker,
>>>> > >
>>>> > > With mpi_f08, you have to declare
>>>> > >
>>>> > > Type(MPI_Comm) :: mpi_comm_global
>>>> > >
>>>> > > (I am afk and not 100% sure of the syntax)
>>>> > >
>>>> > > A simpler option is to
>>>> > >
>>>> > > use mpi
>>>> > >
>>>> > > Cheers,
>>>> > >
>>>> > > Gilles
>>>> > >
>>>> > > Volker Blum >>> wrote:
>>>> > >> Hi Gilles,
>>>> > >>
>>>> > >> Thank you very much for the response!
>>>> > >>
>>>> > >> Unfortunately, I don’t have access to a different system
>>>with the issue right now. As I said, it’s not new; it just keeps
>>>creeping up unexpectedly again on different platforms. What
>>>puzzles me is that I’ve encountered the same problem with low but
>>>reasonable frequency over a period of now over five years.
>>>> > >>
>>>> > >> We can’t require F’08 in our application, unfortunately,
>>>since this standard is too new. Since we maintain a large
>>>application that has to run on a broad range of platforms, Fortran
>

Re: [OMPI users] MPI_IN_PLACE

[Volker Blum]

Thanks! That’s great. Sounds like the exact combination I have here.

Thanks also to George. Sorry that the test did not trigger on a more standard 
platform - that would have simplified things.

Best wishes
Volker

> On Jul 27, 2017, at 3:56 AM, Gilles Gouaillardet  wrote:
> 
> Folks,
> 
> 
> I am able to reproduce the issue on OS X (Sierra) with stock gcc (aka clang) 
> and ifort 17.0.4
> 
> 
> i will investigate this from now
> 
> 
> Cheers,
> 
> Gilles
> 
> On 7/27/2017 9:28 AM, George Bosilca wrote:
>> Volker,
>> 
>> Unfortunately, I can't replicate with icc. I tried on a x86_64 box with 
>> Intel compiler chain 17.0.4 20170411 to no avail. I also tested the 
>> 3.0.0-rc1 tarball and the current master, and you test completes without 
>> errors on all cases.
>> 
>> Once you figure out an environment where you can consistently replicate the 
>> issue, I would suggest to attach to the processes and:
>> - make sure the MPI_IN_PLACE as seen through the Fortran layer matches what 
>> the C layer expects
>> - what is the collective algorithm used by Open MPI
>> 
>> I have a "Fortran 101" level question. When you pass an array a(:) as 
>> argument, what exactly gets passed via the Fortran interface to the 
>> corresponding C function ?
>> 
>>  George.
>> 
>> On Wed, Jul 26, 2017 at 1:55 PM, Volker Blum > > wrote:
>> 
>>Thanks! Yes, trying with Intel 2017 would be very nice.
>> 
>>> On Jul 26, 2017, at 6:12 PM, George Bosilca >> wrote:
>>>
>>> No, I don't have (or used where they were available) the Intel
>>compiler. I used clang and gfortran. I can try on a Linux box with
>>the Intel 2017 compilers.
>>>
>>>   George.
>>>
>>>
>>>
>>> On Wed, Jul 26, 2017 at 11:59 AM, Volker Blum
>>mailto:volker.b...@duke.edu>> wrote:
>>> Did you use Intel Fortran 2017 as well?
>>>
>>> (I’m asking because I did see the same issue with a combination
>>of an earlier Intel Fortran 2017 version and OpenMPI on an
>>Intel/Infiniband Linux HPC machine … but not Intel Fortran 2016 on
>>the same machine. Perhaps I can revive my access to that
>>combination somehow.)
>>>
>>> Best wishes
>>> Volker
>>>
>>> > On Jul 26, 2017, at 5:55 PM, George Bosilca
>>mailto:bosi...@icl.utk.edu>> wrote:
>>> >
>>> > I thought that maybe the underlying allreduce algorithm fails
>>to support MPI_IN_PLACE correctly, but I can't replicate on any
>>machine (including OSX) with any number of processes.
>>> >
>>> >   George.
>>> >
>>> >
>>> >
>>> > On Wed, Jul 26, 2017 at 10:59 AM, Volker Blum
>>mailto:volker.b...@duke.edu>> wrote:
>>> > Thanks!
>>> >
>>> > I tried ‘use mpi’, which compiles fine.
>>> >
>>> > Same result as with ‘include mpif.h', in that the output is
>>> >
>>> >  * MPI_IN_PLACE does not appear to work as intended.
>>> >  * Checking whether MPI_ALLREDUCE works at all.
>>> >  * Without MPI_IN_PLACE, MPI_ALLREDUCE appears to work.
>>> >
>>> > Hm. Any other thoughts?
>>> >
>>> > Thanks again!
>>> > Best wishes
>>> > Volker
>>> >
>>> > > On Jul 26, 2017, at 4:06 PM, Gilles Gouaillardet
>>>> wrote:
>>> > >
>>> > > Volker,
>>> > >
>>> > > With mpi_f08, you have to declare
>>> > >
>>> > > Type(MPI_Comm) :: mpi_comm_global
>>> > >
>>> > > (I am afk and not 100% sure of the syntax)
>>> > >
>>> > > A simpler option is to
>>> > >
>>> > > use mpi
>>> > >
>>> > > Cheers,
>>> > >
>>> > > Gilles
>>> > >
>>> > > Volker Blum >> wrote:
>>> > >> Hi Gilles,
>>> > >>
>>> > >> Thank you very much for the response!
>>> > >>
>>> > >> Unfortunately, I don’t have access to a different system
>>with the issue right now. As I said, it’s not new; it just keeps
>>creeping up unexpectedly again on different platforms. What
>>puzzles me is that I’ve encountered the same problem with low but
>>reasonable frequency over a period of now over five years.
>>> > >>
>>> > >> We can’t require F’08 in our application, unfortunately,
>>since this standard is too new. Since we maintain a large
>>application that has to run on a broad range of platforms, Fortran
>>2008 would not work for many of our users. In a few years, this
>>will be different, but not yet.
>>> > >>
>>> > >> On gfortran: In our own tests, unfortunately, Intel Fortran
>>consistently produced much faster executable code in the past. The
>>latter observation may also change someday, but for us, the
>>performance difference was an important constraint.
>>> > >>
>>> > >> I did suspect mpif.h, too. Not sure how to best test this
>>hypothesis, however.
>>> > >>
>>> > >> Just replacing
>>> > >>
>>> >

Re: [OMPI users] MPI_IN_PLACE

[Jeff Hammond]

Does this happen with ifort but not other Fortran compilers? If so, write
me off-list if there's a need to report a compiler issue.

Jeff

On Wed, Jul 26, 2017 at 6:59 PM Gilles Gouaillardet 
wrote:

> Folks,
>
>
> I am able to reproduce the issue on OS X (Sierra) with stock gcc (aka
> clang) and ifort 17.0.4
>
>
> i will investigate this from now
>
>
> Cheers,
>
> Gilles
>
> On 7/27/2017 9:28 AM, George Bosilca wrote:
> > Volker,
> >
> > Unfortunately, I can't replicate with icc. I tried on a x86_64 box
> > with Intel compiler chain 17.0.4 20170411 to no avail. I also tested
> > the 3.0.0-rc1 tarball and the current master, and you test completes
> > without errors on all cases.
> >
> > Once you figure out an environment where you can consistently
> > replicate the issue, I would suggest to attach to the processes and:
> > - make sure the MPI_IN_PLACE as seen through the Fortran layer matches
> > what the C layer expects
> > - what is the collective algorithm used by Open MPI
> >
> > I have a "Fortran 101" level question. When you pass an array a(:) as
> > argument, what exactly gets passed via the Fortran interface to the
> > corresponding C function ?
> >
> >   George.
> >
> > On Wed, Jul 26, 2017 at 1:55 PM, Volker Blum  > > wrote:
> >
> > Thanks! Yes, trying with Intel 2017 would be very nice.
> >
> > > On Jul 26, 2017, at 6:12 PM, George Bosilca  > > wrote:
> > >
> > > No, I don't have (or used where they were available) the Intel
> > compiler. I used clang and gfortran. I can try on a Linux box with
> > the Intel 2017 compilers.
> > >
> > >   George.
> > >
> > >
> > >
> > > On Wed, Jul 26, 2017 at 11:59 AM, Volker Blum
> > mailto:volker.b...@duke.edu>> wrote:
> > > Did you use Intel Fortran 2017 as well?
> > >
> > > (I’m asking because I did see the same issue with a combination
> > of an earlier Intel Fortran 2017 version and OpenMPI on an
> > Intel/Infiniband Linux HPC machine … but not Intel Fortran 2016 on
> > the same machine. Perhaps I can revive my access to that
> > combination somehow.)
> > >
> > > Best wishes
> > > Volker
> > >
> > > > On Jul 26, 2017, at 5:55 PM, George Bosilca
> > mailto:bosi...@icl.utk.edu>> wrote:
> > > >
> > > > I thought that maybe the underlying allreduce algorithm fails
> > to support MPI_IN_PLACE correctly, but I can't replicate on any
> > machine (including OSX) with any number of processes.
> > > >
> > > >   George.
> > > >
> > > >
> > > >
> > > > On Wed, Jul 26, 2017 at 10:59 AM, Volker Blum
> > mailto:volker.b...@duke.edu>> wrote:
> > > > Thanks!
> > > >
> > > > I tried ‘use mpi’, which compiles fine.
> > > >
> > > > Same result as with ‘include mpif.h', in that the output is
> > > >
> > > >  * MPI_IN_PLACE does not appear to work as intended.
> > > >  * Checking whether MPI_ALLREDUCE works at all.
> > > >  * Without MPI_IN_PLACE, MPI_ALLREDUCE appears to work.
> > > >
> > > > Hm. Any other thoughts?
> > > >
> > > > Thanks again!
> > > > Best wishes
> > > > Volker
> > > >
> > > > > On Jul 26, 2017, at 4:06 PM, Gilles Gouaillardet
> >  > > wrote:
> > > > >
> > > > > Volker,
> > > > >
> > > > > With mpi_f08, you have to declare
> > > > >
> > > > > Type(MPI_Comm) :: mpi_comm_global
> > > > >
> > > > > (I am afk and not 100% sure of the syntax)
> > > > >
> > > > > A simpler option is to
> > > > >
> > > > > use mpi
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Gilles
> > > > >
> > > > > Volker Blum  > > wrote:
> > > > >> Hi Gilles,
> > > > >>
> > > > >> Thank you very much for the response!
> > > > >>
> > > > >> Unfortunately, I don’t have access to a different system
> > with the issue right now. As I said, it’s not new; it just keeps
> > creeping up unexpectedly again on different platforms. What
> > puzzles me is that I’ve encountered the same problem with low but
> > reasonable frequency over a period of now over five years.
> > > > >>
> > > > >> We can’t require F’08 in our application, unfortunately,
> > since this standard is too new. Since we maintain a large
> > application that has to run on a broad range of platforms, Fortran
> > 2008 would not work for many of our users. In a few years, this
> > will be different, but not yet.
> > > > >>
> > > > >> On gfortran: In our own tests, unfortunately, Intel Fortran
> > consistently produced much faster executable code in the past. The
> > latter observation may also change someday, but for us, the
> > performance difference was an important constraint.
> > > > >>
> > > > >> I did suspect mpif.h, too. No

Re: [OMPI users] MPI_IN_PLACE

[Gilles Gouaillardet]

Folks,


I am able to reproduce the issue on OS X (Sierra) with stock gcc (aka 
clang) and ifort 17.0.4



i will investigate this from now


Cheers,

Gilles

On 7/27/2017 9:28 AM, George Bosilca wrote:

Volker,

Unfortunately, I can't replicate with icc. I tried on a x86_64 box 
with Intel compiler chain 17.0.4 20170411 to no avail. I also tested 
the 3.0.0-rc1 tarball and the current master, and you test completes 
without errors on all cases.


Once you figure out an environment where you can consistently 
replicate the issue, I would suggest to attach to the processes and:
- make sure the MPI_IN_PLACE as seen through the Fortran layer matches 
what the C layer expects

- what is the collective algorithm used by Open MPI

I have a "Fortran 101" level question. When you pass an array a(:) as 
argument, what exactly gets passed via the Fortran interface to the 
corresponding C function ?


  George.

On Wed, Jul 26, 2017 at 1:55 PM, Volker Blum > wrote:


Thanks! Yes, trying with Intel 2017 would be very nice.

> On Jul 26, 2017, at 6:12 PM, George Bosilca mailto:bosi...@icl.utk.edu>> wrote:
>
> No, I don't have (or used where they were available) the Intel
compiler. I used clang and gfortran. I can try on a Linux box with
the Intel 2017 compilers.
>
>   George.
>
>
>
> On Wed, Jul 26, 2017 at 11:59 AM, Volker Blum
mailto:volker.b...@duke.edu>> wrote:
> Did you use Intel Fortran 2017 as well?
>
> (I’m asking because I did see the same issue with a combination
of an earlier Intel Fortran 2017 version and OpenMPI on an
Intel/Infiniband Linux HPC machine … but not Intel Fortran 2016 on
the same machine. Perhaps I can revive my access to that
combination somehow.)
>
> Best wishes
> Volker
>
> > On Jul 26, 2017, at 5:55 PM, George Bosilca
mailto:bosi...@icl.utk.edu>> wrote:
> >
> > I thought that maybe the underlying allreduce algorithm fails
to support MPI_IN_PLACE correctly, but I can't replicate on any
machine (including OSX) with any number of processes.
> >
> >   George.
> >
> >
> >
> > On Wed, Jul 26, 2017 at 10:59 AM, Volker Blum
mailto:volker.b...@duke.edu>> wrote:
> > Thanks!
> >
> > I tried ‘use mpi’, which compiles fine.
> >
> > Same result as with ‘include mpif.h', in that the output is
> >
> >  * MPI_IN_PLACE does not appear to work as intended.
> >  * Checking whether MPI_ALLREDUCE works at all.
> >  * Without MPI_IN_PLACE, MPI_ALLREDUCE appears to work.
> >
> > Hm. Any other thoughts?
> >
> > Thanks again!
> > Best wishes
> > Volker
> >
> > > On Jul 26, 2017, at 4:06 PM, Gilles Gouaillardet
mailto:gilles.gouaillar...@gmail.com>> wrote:
> > >
> > > Volker,
> > >
> > > With mpi_f08, you have to declare
> > >
> > > Type(MPI_Comm) :: mpi_comm_global
> > >
> > > (I am afk and not 100% sure of the syntax)
> > >
> > > A simpler option is to
> > >
> > > use mpi
> > >
> > > Cheers,
> > >
> > > Gilles
> > >
> > > Volker Blum mailto:volker.b...@duke.edu>> wrote:
> > >> Hi Gilles,
> > >>
> > >> Thank you very much for the response!
> > >>
> > >> Unfortunately, I don’t have access to a different system
with the issue right now. As I said, it’s not new; it just keeps
creeping up unexpectedly again on different platforms. What
puzzles me is that I’ve encountered the same problem with low but
reasonable frequency over a period of now over five years.
> > >>
> > >> We can’t require F’08 in our application, unfortunately,
since this standard is too new. Since we maintain a large
application that has to run on a broad range of platforms, Fortran
2008 would not work for many of our users. In a few years, this
will be different, but not yet.
> > >>
> > >> On gfortran: In our own tests, unfortunately, Intel Fortran
consistently produced much faster executable code in the past. The
latter observation may also change someday, but for us, the
performance difference was an important constraint.
> > >>
> > >> I did suspect mpif.h, too. Not sure how to best test this
hypothesis, however.
> > >>
> > >> Just replacing
> > >>
> > >>> include 'mpif.h'
> > >>> with
> > >>> use mpi_f08
> > >>
> > >> did not work, for me.
> > >>
> > >> This produces a number of compilation errors:
> > >>
> > >> blum:/Users/blum/codes/fhi-aims/openmpi_test> mpif90
check_mpi_in_place_08.f90 -o check_mpi_in_place_08.x
> > >> check_mpi_in_place_08.f90(55): error #6303: The assignment
operation or the binary expression operation is invalid for the
data types of the two operands.   [MPI_COMM_WORLD]
> > >>   mpi_comm_global = MPI_COMM_WORLD
> > >> --^
>

Re: [OMPI users] MPI_IN_PLACE

[George Bosilca]

Volker,

Unfortunately, I can't replicate with icc. I tried on a x86_64 box with
Intel compiler chain 17.0.4 20170411 to no avail. I also tested the
3.0.0-rc1 tarball and the current master, and you test completes without
errors on all cases.

Once you figure out an environment where you can consistently replicate the
issue, I would suggest to attach to the processes and:
- make sure the MPI_IN_PLACE as seen through the Fortran layer matches what
the C layer expects
- what is the collective algorithm used by Open MPI

I have a "Fortran 101" level question. When you pass an array a(:) as
argument, what exactly gets passed via the Fortran interface to the
corresponding C function ?

  George.

On Wed, Jul 26, 2017 at 1:55 PM, Volker Blum  wrote:

> Thanks! Yes, trying with Intel 2017 would be very nice.
>
> > On Jul 26, 2017, at 6:12 PM, George Bosilca  wrote:
> >
> > No, I don't have (or used where they were available) the Intel compiler.
> I used clang and gfortran. I can try on a Linux box with the Intel 2017
> compilers.
> >
> >   George.
> >
> >
> >
> > On Wed, Jul 26, 2017 at 11:59 AM, Volker Blum 
> wrote:
> > Did you use Intel Fortran 2017 as well?
> >
> > (I’m asking because I did see the same issue with a combination of an
> earlier Intel Fortran 2017 version and OpenMPI on an Intel/Infiniband Linux
> HPC machine … but not Intel Fortran 2016 on the same machine. Perhaps I can
> revive my access to that combination somehow.)
> >
> > Best wishes
> > Volker
> >
> > > On Jul 26, 2017, at 5:55 PM, George Bosilca 
> wrote:
> > >
> > > I thought that maybe the underlying allreduce algorithm fails to
> support MPI_IN_PLACE correctly, but I can't replicate on any machine
> (including OSX) with any number of processes.
> > >
> > >   George.
> > >
> > >
> > >
> > > On Wed, Jul 26, 2017 at 10:59 AM, Volker Blum 
> wrote:
> > > Thanks!
> > >
> > > I tried ‘use mpi’, which compiles fine.
> > >
> > > Same result as with ‘include mpif.h', in that the output is
> > >
> > >  * MPI_IN_PLACE does not appear to work as intended.
> > >  * Checking whether MPI_ALLREDUCE works at all.
> > >  * Without MPI_IN_PLACE, MPI_ALLREDUCE appears to work.
> > >
> > > Hm. Any other thoughts?
> > >
> > > Thanks again!
> > > Best wishes
> > > Volker
> > >
> > > > On Jul 26, 2017, at 4:06 PM, Gilles Gouaillardet <
> gilles.gouaillar...@gmail.com> wrote:
> > > >
> > > > Volker,
> > > >
> > > > With mpi_f08, you have to declare
> > > >
> > > > Type(MPI_Comm) :: mpi_comm_global
> > > >
> > > > (I am afk and not 100% sure of the syntax)
> > > >
> > > > A simpler option is to
> > > >
> > > > use mpi
> > > >
> > > > Cheers,
> > > >
> > > > Gilles
> > > >
> > > > Volker Blum  wrote:
> > > >> Hi Gilles,
> > > >>
> > > >> Thank you very much for the response!
> > > >>
> > > >> Unfortunately, I don’t have access to a different system with the
> issue right now. As I said, it’s not new; it just keeps creeping up
> unexpectedly again on different platforms. What puzzles me is that I’ve
> encountered the same problem with low but reasonable frequency over a
> period of now over five years.
> > > >>
> > > >> We can’t require F’08 in our application, unfortunately, since this
> standard is too new. Since we maintain a large application that has to run
> on a broad range of platforms, Fortran 2008 would not work for many of our
> users. In a few years, this will be different, but not yet.
> > > >>
> > > >> On gfortran: In our own tests, unfortunately, Intel Fortran
> consistently produced much faster executable code in the past. The latter
> observation may also change someday, but for us, the performance difference
> was an important constraint.
> > > >>
> > > >> I did suspect mpif.h, too. Not sure how to best test this
> hypothesis, however.
> > > >>
> > > >> Just replacing
> > > >>
> > > >>> include 'mpif.h'
> > > >>> with
> > > >>> use mpi_f08
> > > >>
> > > >> did not work, for me.
> > > >>
> > > >> This produces a number of compilation errors:
> > > >>
> > > >> blum:/Users/blum/codes/fhi-aims/openmpi_test> mpif90
> check_mpi_in_place_08.f90 -o check_mpi_in_place_08.x
> > > >> check_mpi_in_place_08.f90(55): error #6303: The assignment
> operation or the binary expression operation is invalid for the data types
> of the two operands.   [MPI_COMM_WORLD]
> > > >>   mpi_comm_global = MPI_COMM_WORLD
> > > >> --^
> > > >> check_mpi_in_place_08.f90(57): error #6285: There is no matching
> specific subroutine for this generic subroutine call.   [MPI_COMM_SIZE]
> > > >>   call MPI_COMM_SIZE(mpi_comm_global, n_tasks, mpierr)
> > > >> -^
> > > >> check_mpi_in_place_08.f90(58): error #6285: There is no matching
> specific subroutine for this generic subroutine call.   [MPI_COMM_RANK]
> > > >>   call MPI_COMM_RANK(mpi_comm_global, myid, mpierr)
> > > >> -^
> > > >> check_mpi_in_place_08.f90(75): error #6285: There is no matching
> specific subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
> > > >>   call MPI_AL

Re: [OMPI users] MPI_IN_PLACE

[Volker Blum]

Thanks! Yes, trying with Intel 2017 would be very nice. 

> On Jul 26, 2017, at 6:12 PM, George Bosilca  wrote:
> 
> No, I don't have (or used where they were available) the Intel compiler. I 
> used clang and gfortran. I can try on a Linux box with the Intel 2017 
> compilers.
> 
>   George.
> 
> 
> 
> On Wed, Jul 26, 2017 at 11:59 AM, Volker Blum  wrote:
> Did you use Intel Fortran 2017 as well?
> 
> (I’m asking because I did see the same issue with a combination of an earlier 
> Intel Fortran 2017 version and OpenMPI on an Intel/Infiniband Linux HPC 
> machine … but not Intel Fortran 2016 on the same machine. Perhaps I can 
> revive my access to that combination somehow.)
> 
> Best wishes
> Volker
> 
> > On Jul 26, 2017, at 5:55 PM, George Bosilca  wrote:
> >
> > I thought that maybe the underlying allreduce algorithm fails to support 
> > MPI_IN_PLACE correctly, but I can't replicate on any machine (including 
> > OSX) with any number of processes.
> >
> >   George.
> >
> >
> >
> > On Wed, Jul 26, 2017 at 10:59 AM, Volker Blum  wrote:
> > Thanks!
> >
> > I tried ‘use mpi’, which compiles fine.
> >
> > Same result as with ‘include mpif.h', in that the output is
> >
> >  * MPI_IN_PLACE does not appear to work as intended.
> >  * Checking whether MPI_ALLREDUCE works at all.
> >  * Without MPI_IN_PLACE, MPI_ALLREDUCE appears to work.
> >
> > Hm. Any other thoughts?
> >
> > Thanks again!
> > Best wishes
> > Volker
> >
> > > On Jul 26, 2017, at 4:06 PM, Gilles Gouaillardet 
> > >  wrote:
> > >
> > > Volker,
> > >
> > > With mpi_f08, you have to declare
> > >
> > > Type(MPI_Comm) :: mpi_comm_global
> > >
> > > (I am afk and not 100% sure of the syntax)
> > >
> > > A simpler option is to
> > >
> > > use mpi
> > >
> > > Cheers,
> > >
> > > Gilles
> > >
> > > Volker Blum  wrote:
> > >> Hi Gilles,
> > >>
> > >> Thank you very much for the response!
> > >>
> > >> Unfortunately, I don’t have access to a different system with the issue 
> > >> right now. As I said, it’s not new; it just keeps creeping up 
> > >> unexpectedly again on different platforms. What puzzles me is that I’ve 
> > >> encountered the same problem with low but reasonable frequency over a 
> > >> period of now over five years.
> > >>
> > >> We can’t require F’08 in our application, unfortunately, since this 
> > >> standard is too new. Since we maintain a large application that has to 
> > >> run on a broad range of platforms, Fortran 2008 would not work for many 
> > >> of our users. In a few years, this will be different, but not yet.
> > >>
> > >> On gfortran: In our own tests, unfortunately, Intel Fortran consistently 
> > >> produced much faster executable code in the past. The latter observation 
> > >> may also change someday, but for us, the performance difference was an 
> > >> important constraint.
> > >>
> > >> I did suspect mpif.h, too. Not sure how to best test this hypothesis, 
> > >> however.
> > >>
> > >> Just replacing
> > >>
> > >>> include 'mpif.h'
> > >>> with
> > >>> use mpi_f08
> > >>
> > >> did not work, for me.
> > >>
> > >> This produces a number of compilation errors:
> > >>
> > >> blum:/Users/blum/codes/fhi-aims/openmpi_test> mpif90 
> > >> check_mpi_in_place_08.f90 -o check_mpi_in_place_08.x
> > >> check_mpi_in_place_08.f90(55): error #6303: The assignment operation or 
> > >> the binary expression operation is invalid for the data types of the two 
> > >> operands.   [MPI_COMM_WORLD]
> > >>   mpi_comm_global = MPI_COMM_WORLD
> > >> --^
> > >> check_mpi_in_place_08.f90(57): error #6285: There is no matching 
> > >> specific subroutine for this generic subroutine call.   [MPI_COMM_SIZE]
> > >>   call MPI_COMM_SIZE(mpi_comm_global, n_tasks, mpierr)
> > >> -^
> > >> check_mpi_in_place_08.f90(58): error #6285: There is no matching 
> > >> specific subroutine for this generic subroutine call.   [MPI_COMM_RANK]
> > >>   call MPI_COMM_RANK(mpi_comm_global, myid, mpierr)
> > >> -^
> > >> check_mpi_in_place_08.f90(75): error #6285: There is no matching 
> > >> specific subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
> > >>   call MPI_ALLREDUCE(MPI_IN_PLACE, &
> > >> -^
> > >> check_mpi_in_place_08.f90(94): error #6285: There is no matching 
> > >> specific subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
> > >>   call MPI_ALLREDUCE(check_success, aux_check_success, 1, MPI_LOGICAL, &
> > >> -^
> > >> check_mpi_in_place_08.f90(119): error #6285: There is no matching 
> > >> specific subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
> > >>  call MPI_ALLREDUCE(test_data(:), &
> > >> ^
> > >> check_mpi_in_place_08.f90(140): error #6285: There is no matching 
> > >> specific subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
> > >>  call MPI_ALLREDUCE(check_conventional_mpi, aux_check_success, 1, 
> > >> MPI_LOGICAL, &
> > >> ^
> > >> compilation aborted for check_mpi_in_place_08.f90 (code 1)
> > 

Re: [OMPI users] MPI_IN_PLACE

[George Bosilca]

No, I don't have (or used where they were available) the Intel compiler. I
used clang and gfortran. I can try on a Linux box with the Intel 2017
compilers.

  George.



On Wed, Jul 26, 2017 at 11:59 AM, Volker Blum  wrote:

> Did you use Intel Fortran 2017 as well?
>
> (I’m asking because I did see the same issue with a combination of an
> earlier Intel Fortran 2017 version and OpenMPI on an Intel/Infiniband Linux
> HPC machine … but not Intel Fortran 2016 on the same machine. Perhaps I can
> revive my access to that combination somehow.)
>
> Best wishes
> Volker
>
> > On Jul 26, 2017, at 5:55 PM, George Bosilca  wrote:
> >
> > I thought that maybe the underlying allreduce algorithm fails to support
> MPI_IN_PLACE correctly, but I can't replicate on any machine (including
> OSX) with any number of processes.
> >
> >   George.
> >
> >
> >
> > On Wed, Jul 26, 2017 at 10:59 AM, Volker Blum 
> wrote:
> > Thanks!
> >
> > I tried ‘use mpi’, which compiles fine.
> >
> > Same result as with ‘include mpif.h', in that the output is
> >
> >  * MPI_IN_PLACE does not appear to work as intended.
> >  * Checking whether MPI_ALLREDUCE works at all.
> >  * Without MPI_IN_PLACE, MPI_ALLREDUCE appears to work.
> >
> > Hm. Any other thoughts?
> >
> > Thanks again!
> > Best wishes
> > Volker
> >
> > > On Jul 26, 2017, at 4:06 PM, Gilles Gouaillardet <
> gilles.gouaillar...@gmail.com> wrote:
> > >
> > > Volker,
> > >
> > > With mpi_f08, you have to declare
> > >
> > > Type(MPI_Comm) :: mpi_comm_global
> > >
> > > (I am afk and not 100% sure of the syntax)
> > >
> > > A simpler option is to
> > >
> > > use mpi
> > >
> > > Cheers,
> > >
> > > Gilles
> > >
> > > Volker Blum  wrote:
> > >> Hi Gilles,
> > >>
> > >> Thank you very much for the response!
> > >>
> > >> Unfortunately, I don’t have access to a different system with the
> issue right now. As I said, it’s not new; it just keeps creeping up
> unexpectedly again on different platforms. What puzzles me is that I’ve
> encountered the same problem with low but reasonable frequency over a
> period of now over five years.
> > >>
> > >> We can’t require F’08 in our application, unfortunately, since this
> standard is too new. Since we maintain a large application that has to run
> on a broad range of platforms, Fortran 2008 would not work for many of our
> users. In a few years, this will be different, but not yet.
> > >>
> > >> On gfortran: In our own tests, unfortunately, Intel Fortran
> consistently produced much faster executable code in the past. The latter
> observation may also change someday, but for us, the performance difference
> was an important constraint.
> > >>
> > >> I did suspect mpif.h, too. Not sure how to best test this hypothesis,
> however.
> > >>
> > >> Just replacing
> > >>
> > >>> include 'mpif.h'
> > >>> with
> > >>> use mpi_f08
> > >>
> > >> did not work, for me.
> > >>
> > >> This produces a number of compilation errors:
> > >>
> > >> blum:/Users/blum/codes/fhi-aims/openmpi_test> mpif90
> check_mpi_in_place_08.f90 -o check_mpi_in_place_08.x
> > >> check_mpi_in_place_08.f90(55): error #6303: The assignment operation
> or the binary expression operation is invalid for the data types of the two
> operands.   [MPI_COMM_WORLD]
> > >>   mpi_comm_global = MPI_COMM_WORLD
> > >> --^
> > >> check_mpi_in_place_08.f90(57): error #6285: There is no matching
> specific subroutine for this generic subroutine call.   [MPI_COMM_SIZE]
> > >>   call MPI_COMM_SIZE(mpi_comm_global, n_tasks, mpierr)
> > >> -^
> > >> check_mpi_in_place_08.f90(58): error #6285: There is no matching
> specific subroutine for this generic subroutine call.   [MPI_COMM_RANK]
> > >>   call MPI_COMM_RANK(mpi_comm_global, myid, mpierr)
> > >> -^
> > >> check_mpi_in_place_08.f90(75): error #6285: There is no matching
> specific subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
> > >>   call MPI_ALLREDUCE(MPI_IN_PLACE, &
> > >> -^
> > >> check_mpi_in_place_08.f90(94): error #6285: There is no matching
> specific subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
> > >>   call MPI_ALLREDUCE(check_success, aux_check_success, 1,
> MPI_LOGICAL, &
> > >> -^
> > >> check_mpi_in_place_08.f90(119): error #6285: There is no matching
> specific subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
> > >>  call MPI_ALLREDUCE(test_data(:), &
> > >> ^
> > >> check_mpi_in_place_08.f90(140): error #6285: There is no matching
> specific subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
> > >>  call MPI_ALLREDUCE(check_conventional_mpi, aux_check_success,
> 1, MPI_LOGICAL, &
> > >> ^
> > >> compilation aborted for check_mpi_in_place_08.f90 (code 1)
> > >>
> > >> This is an interesting result, however … what might I be missing?
> Another use statement?
> > >>
> > >> Best wishes
> > >> Volker
> > >>
> > >>> On Jul 26, 2017, at 2:53 PM, Gilles Gouaillardet <
> gilles.gouaillar...@gmail.com> w

Re: [OMPI users] MPI_IN_PLACE

[Volker Blum]

Did you use Intel Fortran 2017 as well?

(I’m asking because I did see the same issue with a combination of an earlier 
Intel Fortran 2017 version and OpenMPI on an Intel/Infiniband Linux HPC machine 
… but not Intel Fortran 2016 on the same machine. Perhaps I can revive my 
access to that combination somehow.)

Best wishes
Volker

> On Jul 26, 2017, at 5:55 PM, George Bosilca  wrote:
> 
> I thought that maybe the underlying allreduce algorithm fails to support 
> MPI_IN_PLACE correctly, but I can't replicate on any machine (including OSX) 
> with any number of processes.
> 
>   George.
> 
> 
> 
> On Wed, Jul 26, 2017 at 10:59 AM, Volker Blum  wrote:
> Thanks!
> 
> I tried ‘use mpi’, which compiles fine.
> 
> Same result as with ‘include mpif.h', in that the output is
> 
>  * MPI_IN_PLACE does not appear to work as intended.
>  * Checking whether MPI_ALLREDUCE works at all.
>  * Without MPI_IN_PLACE, MPI_ALLREDUCE appears to work.
> 
> Hm. Any other thoughts?
> 
> Thanks again!
> Best wishes
> Volker
> 
> > On Jul 26, 2017, at 4:06 PM, Gilles Gouaillardet 
> >  wrote:
> >
> > Volker,
> >
> > With mpi_f08, you have to declare
> >
> > Type(MPI_Comm) :: mpi_comm_global
> >
> > (I am afk and not 100% sure of the syntax)
> >
> > A simpler option is to
> >
> > use mpi
> >
> > Cheers,
> >
> > Gilles
> >
> > Volker Blum  wrote:
> >> Hi Gilles,
> >>
> >> Thank you very much for the response!
> >>
> >> Unfortunately, I don’t have access to a different system with the issue 
> >> right now. As I said, it’s not new; it just keeps creeping up unexpectedly 
> >> again on different platforms. What puzzles me is that I’ve encountered the 
> >> same problem with low but reasonable frequency over a period of now over 
> >> five years.
> >>
> >> We can’t require F’08 in our application, unfortunately, since this 
> >> standard is too new. Since we maintain a large application that has to run 
> >> on a broad range of platforms, Fortran 2008 would not work for many of our 
> >> users. In a few years, this will be different, but not yet.
> >>
> >> On gfortran: In our own tests, unfortunately, Intel Fortran consistently 
> >> produced much faster executable code in the past. The latter observation 
> >> may also change someday, but for us, the performance difference was an 
> >> important constraint.
> >>
> >> I did suspect mpif.h, too. Not sure how to best test this hypothesis, 
> >> however.
> >>
> >> Just replacing
> >>
> >>> include 'mpif.h'
> >>> with
> >>> use mpi_f08
> >>
> >> did not work, for me.
> >>
> >> This produces a number of compilation errors:
> >>
> >> blum:/Users/blum/codes/fhi-aims/openmpi_test> mpif90 
> >> check_mpi_in_place_08.f90 -o check_mpi_in_place_08.x
> >> check_mpi_in_place_08.f90(55): error #6303: The assignment operation or 
> >> the binary expression operation is invalid for the data types of the two 
> >> operands.   [MPI_COMM_WORLD]
> >>   mpi_comm_global = MPI_COMM_WORLD
> >> --^
> >> check_mpi_in_place_08.f90(57): error #6285: There is no matching specific 
> >> subroutine for this generic subroutine call.   [MPI_COMM_SIZE]
> >>   call MPI_COMM_SIZE(mpi_comm_global, n_tasks, mpierr)
> >> -^
> >> check_mpi_in_place_08.f90(58): error #6285: There is no matching specific 
> >> subroutine for this generic subroutine call.   [MPI_COMM_RANK]
> >>   call MPI_COMM_RANK(mpi_comm_global, myid, mpierr)
> >> -^
> >> check_mpi_in_place_08.f90(75): error #6285: There is no matching specific 
> >> subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
> >>   call MPI_ALLREDUCE(MPI_IN_PLACE, &
> >> -^
> >> check_mpi_in_place_08.f90(94): error #6285: There is no matching specific 
> >> subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
> >>   call MPI_ALLREDUCE(check_success, aux_check_success, 1, MPI_LOGICAL, &
> >> -^
> >> check_mpi_in_place_08.f90(119): error #6285: There is no matching specific 
> >> subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
> >>  call MPI_ALLREDUCE(test_data(:), &
> >> ^
> >> check_mpi_in_place_08.f90(140): error #6285: There is no matching specific 
> >> subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
> >>  call MPI_ALLREDUCE(check_conventional_mpi, aux_check_success, 1, 
> >> MPI_LOGICAL, &
> >> ^
> >> compilation aborted for check_mpi_in_place_08.f90 (code 1)
> >>
> >> This is an interesting result, however … what might I be missing? Another 
> >> use statement?
> >>
> >> Best wishes
> >> Volker
> >>
> >>> On Jul 26, 2017, at 2:53 PM, Gilles Gouaillardet 
> >>>  wrote:
> >>>
> >>> Volker,
> >>>
> >>> thanks, i will have a look at it
> >>>
> >>> meanwhile, if you can reproduce this issue on a more mainstream
> >>> platform (e.g. linux + gfortran) please let me know.
> >>>
> >>> since you are using ifort, Open MPI was built with Fortran 2008
> >>> bindings, so you can replace
> >>> include 'mpif.h'
> >>> with
> >>> use mpi_f08
> >>> and who knows, 

Re: [OMPI users] MPI_IN_PLACE

[George Bosilca]

I thought that maybe the underlying allreduce algorithm fails to support
MPI_IN_PLACE correctly, but I can't replicate on any machine (including
OSX) with any number of processes.

  George.



On Wed, Jul 26, 2017 at 10:59 AM, Volker Blum  wrote:

> Thanks!
>
> I tried ‘use mpi’, which compiles fine.
>
> Same result as with ‘include mpif.h', in that the output is
>
>  * MPI_IN_PLACE does not appear to work as intended.
>  * Checking whether MPI_ALLREDUCE works at all.
>  * Without MPI_IN_PLACE, MPI_ALLREDUCE appears to work.
>
> Hm. Any other thoughts?
>
> Thanks again!
> Best wishes
> Volker
>
> > On Jul 26, 2017, at 4:06 PM, Gilles Gouaillardet <
> gilles.gouaillar...@gmail.com> wrote:
> >
> > Volker,
> >
> > With mpi_f08, you have to declare
> >
> > Type(MPI_Comm) :: mpi_comm_global
> >
> > (I am afk and not 100% sure of the syntax)
> >
> > A simpler option is to
> >
> > use mpi
> >
> > Cheers,
> >
> > Gilles
> >
> > Volker Blum  wrote:
> >> Hi Gilles,
> >>
> >> Thank you very much for the response!
> >>
> >> Unfortunately, I don’t have access to a different system with the issue
> right now. As I said, it’s not new; it just keeps creeping up unexpectedly
> again on different platforms. What puzzles me is that I’ve encountered the
> same problem with low but reasonable frequency over a period of now over
> five years.
> >>
> >> We can’t require F’08 in our application, unfortunately, since this
> standard is too new. Since we maintain a large application that has to run
> on a broad range of platforms, Fortran 2008 would not work for many of our
> users. In a few years, this will be different, but not yet.
> >>
> >> On gfortran: In our own tests, unfortunately, Intel Fortran
> consistently produced much faster executable code in the past. The latter
> observation may also change someday, but for us, the performance difference
> was an important constraint.
> >>
> >> I did suspect mpif.h, too. Not sure how to best test this hypothesis,
> however.
> >>
> >> Just replacing
> >>
> >>> include 'mpif.h'
> >>> with
> >>> use mpi_f08
> >>
> >> did not work, for me.
> >>
> >> This produces a number of compilation errors:
> >>
> >> blum:/Users/blum/codes/fhi-aims/openmpi_test> mpif90
> check_mpi_in_place_08.f90 -o check_mpi_in_place_08.x
> >> check_mpi_in_place_08.f90(55): error #6303: The assignment operation or
> the binary expression operation is invalid for the data types of the two
> operands.   [MPI_COMM_WORLD]
> >>   mpi_comm_global = MPI_COMM_WORLD
> >> --^
> >> check_mpi_in_place_08.f90(57): error #6285: There is no matching
> specific subroutine for this generic subroutine call.   [MPI_COMM_SIZE]
> >>   call MPI_COMM_SIZE(mpi_comm_global, n_tasks, mpierr)
> >> -^
> >> check_mpi_in_place_08.f90(58): error #6285: There is no matching
> specific subroutine for this generic subroutine call.   [MPI_COMM_RANK]
> >>   call MPI_COMM_RANK(mpi_comm_global, myid, mpierr)
> >> -^
> >> check_mpi_in_place_08.f90(75): error #6285: There is no matching
> specific subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
> >>   call MPI_ALLREDUCE(MPI_IN_PLACE, &
> >> -^
> >> check_mpi_in_place_08.f90(94): error #6285: There is no matching
> specific subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
> >>   call MPI_ALLREDUCE(check_success, aux_check_success, 1, MPI_LOGICAL, &
> >> -^
> >> check_mpi_in_place_08.f90(119): error #6285: There is no matching
> specific subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
> >>  call MPI_ALLREDUCE(test_data(:), &
> >> ^
> >> check_mpi_in_place_08.f90(140): error #6285: There is no matching
> specific subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
> >>  call MPI_ALLREDUCE(check_conventional_mpi, aux_check_success, 1,
> MPI_LOGICAL, &
> >> ^
> >> compilation aborted for check_mpi_in_place_08.f90 (code 1)
> >>
> >> This is an interesting result, however … what might I be missing?
> Another use statement?
> >>
> >> Best wishes
> >> Volker
> >>
> >>> On Jul 26, 2017, at 2:53 PM, Gilles Gouaillardet <
> gilles.gouaillar...@gmail.com> wrote:
> >>>
> >>> Volker,
> >>>
> >>> thanks, i will have a look at it
> >>>
> >>> meanwhile, if you can reproduce this issue on a more mainstream
> >>> platform (e.g. linux + gfortran) please let me know.
> >>>
> >>> since you are using ifort, Open MPI was built with Fortran 2008
> >>> bindings, so you can replace
> >>> include 'mpif.h'
> >>> with
> >>> use mpi_f08
> >>> and who knows, that might solve your issue
> >>>
> >>>
> >>> Cheers,
> >>>
> >>> Gilles
> >>>
> >>> On Wed, Jul 26, 2017 at 5:22 PM, Volker Blum 
> wrote:
>  Dear Gilles,
> 
>  Thank you very much for the fast answer.
> 
>  Darn. I feared it might not occur on all platforms, since my former
> Macbook
>  (with an older OpenMPI version) no longer exhibited the problem, a
> different
>  Linux/Intel Machine did last December, etc.
> >>

Re: [OMPI users] MPI_IN_PLACE

[Volker Blum]

Thanks!

I tried ‘use mpi’, which compiles fine.

Same result as with ‘include mpif.h', in that the output is

 * MPI_IN_PLACE does not appear to work as intended.
 * Checking whether MPI_ALLREDUCE works at all.
 * Without MPI_IN_PLACE, MPI_ALLREDUCE appears to work.

Hm. Any other thoughts?

Thanks again!
Best wishes
Volker

> On Jul 26, 2017, at 4:06 PM, Gilles Gouaillardet 
>  wrote:
> 
> Volker,
> 
> With mpi_f08, you have to declare
> 
> Type(MPI_Comm) :: mpi_comm_global
> 
> (I am afk and not 100% sure of the syntax)
> 
> A simpler option is to
> 
> use mpi
> 
> Cheers,
> 
> Gilles
> 
> Volker Blum  wrote:
>> Hi Gilles,
>> 
>> Thank you very much for the response!
>> 
>> Unfortunately, I don’t have access to a different system with the issue 
>> right now. As I said, it’s not new; it just keeps creeping up unexpectedly 
>> again on different platforms. What puzzles me is that I’ve encountered the 
>> same problem with low but reasonable frequency over a period of now over 
>> five years.
>> 
>> We can’t require F’08 in our application, unfortunately, since this standard 
>> is too new. Since we maintain a large application that has to run on a broad 
>> range of platforms, Fortran 2008 would not work for many of our users. In a 
>> few years, this will be different, but not yet.
>> 
>> On gfortran: In our own tests, unfortunately, Intel Fortran consistently 
>> produced much faster executable code in the past. The latter observation may 
>> also change someday, but for us, the performance difference was an important 
>> constraint.
>> 
>> I did suspect mpif.h, too. Not sure how to best test this hypothesis, 
>> however. 
>> 
>> Just replacing 
>> 
>>> include 'mpif.h'
>>> with
>>> use mpi_f08
>> 
>> did not work, for me. 
>> 
>> This produces a number of compilation errors:
>> 
>> blum:/Users/blum/codes/fhi-aims/openmpi_test> mpif90 
>> check_mpi_in_place_08.f90 -o check_mpi_in_place_08.x
>> check_mpi_in_place_08.f90(55): error #6303: The assignment operation or the 
>> binary expression operation is invalid for the data types of the two 
>> operands.   [MPI_COMM_WORLD]
>>   mpi_comm_global = MPI_COMM_WORLD
>> --^
>> check_mpi_in_place_08.f90(57): error #6285: There is no matching specific 
>> subroutine for this generic subroutine call.   [MPI_COMM_SIZE]
>>   call MPI_COMM_SIZE(mpi_comm_global, n_tasks, mpierr)
>> -^
>> check_mpi_in_place_08.f90(58): error #6285: There is no matching specific 
>> subroutine for this generic subroutine call.   [MPI_COMM_RANK]
>>   call MPI_COMM_RANK(mpi_comm_global, myid, mpierr)
>> -^
>> check_mpi_in_place_08.f90(75): error #6285: There is no matching specific 
>> subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
>>   call MPI_ALLREDUCE(MPI_IN_PLACE, &
>> -^
>> check_mpi_in_place_08.f90(94): error #6285: There is no matching specific 
>> subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
>>   call MPI_ALLREDUCE(check_success, aux_check_success, 1, MPI_LOGICAL, &
>> -^
>> check_mpi_in_place_08.f90(119): error #6285: There is no matching specific 
>> subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
>>  call MPI_ALLREDUCE(test_data(:), &
>> ^
>> check_mpi_in_place_08.f90(140): error #6285: There is no matching specific 
>> subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
>>  call MPI_ALLREDUCE(check_conventional_mpi, aux_check_success, 1, 
>> MPI_LOGICAL, &
>> ^
>> compilation aborted for check_mpi_in_place_08.f90 (code 1)
>> 
>> This is an interesting result, however … what might I be missing? Another 
>> use statement?
>> 
>> Best wishes
>> Volker
>> 
>>> On Jul 26, 2017, at 2:53 PM, Gilles Gouaillardet 
>>>  wrote:
>>> 
>>> Volker,
>>> 
>>> thanks, i will have a look at it
>>> 
>>> meanwhile, if you can reproduce this issue on a more mainstream
>>> platform (e.g. linux + gfortran) please let me know.
>>> 
>>> since you are using ifort, Open MPI was built with Fortran 2008
>>> bindings, so you can replace
>>> include 'mpif.h'
>>> with
>>> use mpi_f08
>>> and who knows, that might solve your issue
>>> 
>>> 
>>> Cheers,
>>> 
>>> Gilles
>>> 
>>> On Wed, Jul 26, 2017 at 5:22 PM, Volker Blum  wrote:
 Dear Gilles,
 
 Thank you very much for the fast answer.
 
 Darn. I feared it might not occur on all platforms, since my former Macbook
 (with an older OpenMPI version) no longer exhibited the problem, a 
 different
 Linux/Intel Machine did last December, etc.
 
 On this specific machine, the configure line is
 
 ./configure CC=gcc FC=ifort F77=ifort
 
 ifort version 17.0.4
 
 blum:/Users/blum/software/openmpi-3.0.0rc1> gcc -v
 Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr
 --with-gxx-include-dir=/usr/include/c++/4.2.1
 Apple LLVM version 8.1.0 (clang-802.0.42)
 Target: x86_64-apple-darwin16.6.0
 Thread model: posix
 InstalledDi

Re: [OMPI users] MPI_IN_PLACE

[Gilles Gouaillardet]

Volker,

With mpi_f08, you have to declare

Type(MPI_Comm) :: mpi_comm_global

(I am afk and not 100% sure of the syntax)

A simpler option is to

use mpi

Cheers,

Gilles

Volker Blum  wrote:
>Hi Gilles,
>
>Thank you very much for the response!
>
>Unfortunately, I don’t have access to a different system with the issue right 
>now. As I said, it’s not new; it just keeps creeping up unexpectedly again on 
>different platforms. What puzzles me is that I’ve encountered the same problem 
>with low but reasonable frequency over a period of now over five years.
>
>We can’t require F’08 in our application, unfortunately, since this standard 
>is too new. Since we maintain a large application that has to run on a broad 
>range of platforms, Fortran 2008 would not work for many of our users. In a 
>few years, this will be different, but not yet.
>
>On gfortran: In our own tests, unfortunately, Intel Fortran consistently 
>produced much faster executable code in the past. The latter observation may 
>also change someday, but for us, the performance difference was an important 
>constraint.
>
>I did suspect mpif.h, too. Not sure how to best test this hypothesis, however. 
>
>Just replacing 
>
>> include 'mpif.h'
>> with
>> use mpi_f08
>
>did not work, for me. 
>
>This produces a number of compilation errors:
>
>blum:/Users/blum/codes/fhi-aims/openmpi_test> mpif90 check_mpi_in_place_08.f90 
>-o check_mpi_in_place_08.x
>check_mpi_in_place_08.f90(55): error #6303: The assignment operation or the 
>binary expression operation is invalid for the data types of the two operands. 
>  [MPI_COMM_WORLD]
>mpi_comm_global = MPI_COMM_WORLD
>--^
>check_mpi_in_place_08.f90(57): error #6285: There is no matching specific 
>subroutine for this generic subroutine call.   [MPI_COMM_SIZE]
>call MPI_COMM_SIZE(mpi_comm_global, n_tasks, mpierr)
>-^
>check_mpi_in_place_08.f90(58): error #6285: There is no matching specific 
>subroutine for this generic subroutine call.   [MPI_COMM_RANK]
>call MPI_COMM_RANK(mpi_comm_global, myid, mpierr)
>-^
>check_mpi_in_place_08.f90(75): error #6285: There is no matching specific 
>subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
>call MPI_ALLREDUCE(MPI_IN_PLACE, &
>-^
>check_mpi_in_place_08.f90(94): error #6285: There is no matching specific 
>subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
>call MPI_ALLREDUCE(check_success, aux_check_success, 1, MPI_LOGICAL, &
>-^
>check_mpi_in_place_08.f90(119): error #6285: There is no matching specific 
>subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
>   call MPI_ALLREDUCE(test_data(:), &
>^
>check_mpi_in_place_08.f90(140): error #6285: There is no matching specific 
>subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
>   call MPI_ALLREDUCE(check_conventional_mpi, aux_check_success, 1, 
> MPI_LOGICAL, &
>^
>compilation aborted for check_mpi_in_place_08.f90 (code 1)
>
>This is an interesting result, however … what might I be missing? Another use 
>statement?
>
>Best wishes
>Volker
>
>> On Jul 26, 2017, at 2:53 PM, Gilles Gouaillardet 
>>  wrote:
>> 
>> Volker,
>> 
>> thanks, i will have a look at it
>> 
>> meanwhile, if you can reproduce this issue on a more mainstream
>> platform (e.g. linux + gfortran) please let me know.
>> 
>> since you are using ifort, Open MPI was built with Fortran 2008
>> bindings, so you can replace
>> include 'mpif.h'
>> with
>> use mpi_f08
>> and who knows, that might solve your issue
>> 
>> 
>> Cheers,
>> 
>> Gilles
>> 
>> On Wed, Jul 26, 2017 at 5:22 PM, Volker Blum  wrote:
>>> Dear Gilles,
>>> 
>>> Thank you very much for the fast answer.
>>> 
>>> Darn. I feared it might not occur on all platforms, since my former Macbook
>>> (with an older OpenMPI version) no longer exhibited the problem, a different
>>> Linux/Intel Machine did last December, etc.
>>> 
>>> On this specific machine, the configure line is
>>> 
>>> ./configure CC=gcc FC=ifort F77=ifort
>>> 
>>> ifort version 17.0.4
>>> 
>>> blum:/Users/blum/software/openmpi-3.0.0rc1> gcc -v
>>> Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr
>>> --with-gxx-include-dir=/usr/include/c++/4.2.1
>>> Apple LLVM version 8.1.0 (clang-802.0.42)
>>> Target: x86_64-apple-darwin16.6.0
>>> Thread model: posix
>>> InstalledDir:
>>> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
>>> 
>>> The full test program is appended.
>>> 
>>> Compilation:
>>> 
>>> mpif90 check_mpi_in_place.f90
>>> 
>>> blum:/Users/blum/codes/fhi-aims/openmpi_test> which mpif90
>>> /usr/local/openmpi-3.0.0rc1/bin/mpif90
>>> 
>>> blum:/Users/blum/codes/fhi-aims/openmpi_test> which mpirun
>>> /usr/local/openmpi-3.0.0rc1/bin/mpirun
>>> 
>>> blum:/Users/blum/codes/fhi-aims/openmpi_test> mpirun -np 2 a.out
>>> * MPI_IN_PLACE does not appear to work as intended.
>>> * Checking whether MPI_ALLREDUCE works at all.
>>> * W

Re: [OMPI users] MPI_IN_PLACE

[Volker Blum]

Hi Gilles,

Thank you very much for the response!

Unfortunately, I don’t have access to a different system with the issue right 
now. As I said, it’s not new; it just keeps creeping up unexpectedly again on 
different platforms. What puzzles me is that I’ve encountered the same problem 
with low but reasonable frequency over a period of now over five years.

We can’t require F’08 in our application, unfortunately, since this standard is 
too new. Since we maintain a large application that has to run on a broad range 
of platforms, Fortran 2008 would not work for many of our users. In a few 
years, this will be different, but not yet.

On gfortran: In our own tests, unfortunately, Intel Fortran consistently 
produced much faster executable code in the past. The latter observation may 
also change someday, but for us, the performance difference was an important 
constraint.

I did suspect mpif.h, too. Not sure how to best test this hypothesis, however. 

Just replacing 

> include 'mpif.h'
> with
> use mpi_f08

did not work, for me. 

This produces a number of compilation errors:

blum:/Users/blum/codes/fhi-aims/openmpi_test> mpif90 check_mpi_in_place_08.f90 
-o check_mpi_in_place_08.x
check_mpi_in_place_08.f90(55): error #6303: The assignment operation or the 
binary expression operation is invalid for the data types of the two operands.  
 [MPI_COMM_WORLD]
mpi_comm_global = MPI_COMM_WORLD
--^
check_mpi_in_place_08.f90(57): error #6285: There is no matching specific 
subroutine for this generic subroutine call.   [MPI_COMM_SIZE]
call MPI_COMM_SIZE(mpi_comm_global, n_tasks, mpierr)
-^
check_mpi_in_place_08.f90(58): error #6285: There is no matching specific 
subroutine for this generic subroutine call.   [MPI_COMM_RANK]
call MPI_COMM_RANK(mpi_comm_global, myid, mpierr)
-^
check_mpi_in_place_08.f90(75): error #6285: There is no matching specific 
subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
call MPI_ALLREDUCE(MPI_IN_PLACE, &
-^
check_mpi_in_place_08.f90(94): error #6285: There is no matching specific 
subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
call MPI_ALLREDUCE(check_success, aux_check_success, 1, MPI_LOGICAL, &
-^
check_mpi_in_place_08.f90(119): error #6285: There is no matching specific 
subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
   call MPI_ALLREDUCE(test_data(:), &
^
check_mpi_in_place_08.f90(140): error #6285: There is no matching specific 
subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
   call MPI_ALLREDUCE(check_conventional_mpi, aux_check_success, 1, 
MPI_LOGICAL, &
^
compilation aborted for check_mpi_in_place_08.f90 (code 1)

This is an interesting result, however … what might I be missing? Another use 
statement?

Best wishes
Volker

> On Jul 26, 2017, at 2:53 PM, Gilles Gouaillardet 
>  wrote:
> 
> Volker,
> 
> thanks, i will have a look at it
> 
> meanwhile, if you can reproduce this issue on a more mainstream
> platform (e.g. linux + gfortran) please let me know.
> 
> since you are using ifort, Open MPI was built with Fortran 2008
> bindings, so you can replace
> include 'mpif.h'
> with
> use mpi_f08
> and who knows, that might solve your issue
> 
> 
> Cheers,
> 
> Gilles
> 
> On Wed, Jul 26, 2017 at 5:22 PM, Volker Blum  wrote:
>> Dear Gilles,
>> 
>> Thank you very much for the fast answer.
>> 
>> Darn. I feared it might not occur on all platforms, since my former Macbook
>> (with an older OpenMPI version) no longer exhibited the problem, a different
>> Linux/Intel Machine did last December, etc.
>> 
>> On this specific machine, the configure line is
>> 
>> ./configure CC=gcc FC=ifort F77=ifort
>> 
>> ifort version 17.0.4
>> 
>> blum:/Users/blum/software/openmpi-3.0.0rc1> gcc -v
>> Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr
>> --with-gxx-include-dir=/usr/include/c++/4.2.1
>> Apple LLVM version 8.1.0 (clang-802.0.42)
>> Target: x86_64-apple-darwin16.6.0
>> Thread model: posix
>> InstalledDir:
>> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
>> 
>> The full test program is appended.
>> 
>> Compilation:
>> 
>> mpif90 check_mpi_in_place.f90
>> 
>> blum:/Users/blum/codes/fhi-aims/openmpi_test> which mpif90
>> /usr/local/openmpi-3.0.0rc1/bin/mpif90
>> 
>> blum:/Users/blum/codes/fhi-aims/openmpi_test> which mpirun
>> /usr/local/openmpi-3.0.0rc1/bin/mpirun
>> 
>> blum:/Users/blum/codes/fhi-aims/openmpi_test> mpirun -np 2 a.out
>> * MPI_IN_PLACE does not appear to work as intended.
>> * Checking whether MPI_ALLREDUCE works at all.
>> * Without MPI_IN_PLACE, MPI_ALLREDUCE appears to work.
>> 
>> blum:/Users/blum/codes/fhi-aims/openmpi_test> mpirun -np 1 a.out
>> * MPI_IN_PLACE does not appear to work as intended.
>> * Checking whether MPI_ALLREDUCE works at all.
>> * Without MPI_IN_PLACE, MPI_ALLREDUCE appears to work.
>> 
>> Hopefully, no trivial mistakes in

Re: [OMPI users] MPI_IN_PLACE

[Gilles Gouaillardet]

Volker,

thanks, i will have a look at it

meanwhile, if you can reproduce this issue on a more mainstream
platform (e.g. linux + gfortran) please let me know.

since you are using ifort, Open MPI was built with Fortran 2008
bindings, so you can replace
include 'mpif.h'
with
use mpi_f08
and who knows, that might solve your issue


Cheers,

Gilles

On Wed, Jul 26, 2017 at 5:22 PM, Volker Blum  wrote:
> Dear Gilles,
>
> Thank you very much for the fast answer.
>
> Darn. I feared it might not occur on all platforms, since my former Macbook
> (with an older OpenMPI version) no longer exhibited the problem, a different
> Linux/Intel Machine did last December, etc.
>
> On this specific machine, the configure line is
>
> ./configure CC=gcc FC=ifort F77=ifort
>
> ifort version 17.0.4
>
> blum:/Users/blum/software/openmpi-3.0.0rc1> gcc -v
> Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr
> --with-gxx-include-dir=/usr/include/c++/4.2.1
> Apple LLVM version 8.1.0 (clang-802.0.42)
> Target: x86_64-apple-darwin16.6.0
> Thread model: posix
> InstalledDir:
> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
>
> The full test program is appended.
>
> Compilation:
>
> mpif90 check_mpi_in_place.f90
>
> blum:/Users/blum/codes/fhi-aims/openmpi_test> which mpif90
> /usr/local/openmpi-3.0.0rc1/bin/mpif90
>
> blum:/Users/blum/codes/fhi-aims/openmpi_test> which mpirun
> /usr/local/openmpi-3.0.0rc1/bin/mpirun
>
> blum:/Users/blum/codes/fhi-aims/openmpi_test> mpirun -np 2 a.out
>  * MPI_IN_PLACE does not appear to work as intended.
>  * Checking whether MPI_ALLREDUCE works at all.
>  * Without MPI_IN_PLACE, MPI_ALLREDUCE appears to work.
>
> blum:/Users/blum/codes/fhi-aims/openmpi_test> mpirun -np 1 a.out
>  * MPI_IN_PLACE does not appear to work as intended.
>  * Checking whether MPI_ALLREDUCE works at all.
>  * Without MPI_IN_PLACE, MPI_ALLREDUCE appears to work.
>
> Hopefully, no trivial mistakes in the testcase. I just spent a few days
> tracing this issue through a fairly large code, which is where the issue
> originally arose (and leads to wrong numbers).
>
> Best wishes
> Volker
>
>
>
>
>> On Jul 26, 2017, at 9:46 AM, Gilles Gouaillardet
>>  wrote:
>>
>> Volker,
>>
>> i was unable to reproduce this issue on linux
>>
>> can you please post your full configure command line, your gnu
>> compiler version and the full test program ?
>>
>> also, how many mpi tasks are you running ?
>>
>> Cheers,
>>
>> Gilles
>>
>> On Wed, Jul 26, 2017 at 4:25 PM, Volker Blum  wrote:
>>> Hi,
>>>
>>> I tried openmpi-3.0.0rc1.tar.gz using Intel Fortran 2017 and gcc on a
>>> current MacOS system. For this version, it seems to me that MPI_IN_PLACE
>>> returns incorrect results (while other MPI implementations, including some
>>> past OpenMPI versions, work fine).
>>>
>>> This can be seen with a simple Fortran example code, shown below. In the
>>> test, the values of all entries of an array “test_data” should be 1.0d0 if
>>> the behavior were as intended. However, the version of OpenMPI I have
>>> returns 0.d0 instead.
>>>
>>> I’ve seen this behavior on some other compute platforms too, in the past,
>>> so it wasn’t new to me. Still, I thought that this time, I’d ask. Any
>>> thoughts?
>>>
>>> Thank you,
>>> Best wishes
>>> Volker
>>>
>>>! size of test data array
>>>integer :: n_data
>>>
>>>! array that contains test data for MPI_IN_PLACE
>>>real*8, allocatable :: test_data(:)
>>>
>>>integer :: mpierr
>>>
>>>n_data = 10
>>>
>>>allocate(test_data(n_data),stat=mpierr)
>>>
>>>! seed test data array for allreduce call below
>>>if (myid.eq.0) then
>>>   test_data(:) = 1.d0
>>>else
>>>   test_data(:) = 0.d0
>>>end if
>>>
>>>! Sum the test_data array over all MPI tasks
>>>call MPI_ALLREDUCE(MPI_IN_PLACE, &
>>> test_data(:), &
>>> n_data, &
>>> MPI_DOUBLE_PRECISION, &
>>> MPI_SUM, &
>>> mpi_comm_global, &
>>> mpierr )
>>>
>>>! The value of all entries of test_data should now be 1.d0 on all MPI
>>> tasks.
>>>! If that is not the case, then the MPI_IN_PLACE flag may be broken.
>>>
>>>
>>>
>>>
>>>
>>>
>>> Volker Blum
>>> Associate Professor
>>> Ab Initio Materials Simulations
>>> Duke University, MEMS Department
>>> 144 Hudson Hall, Box 90300, Duke University, Durham, NC 27708, USA
>>>
>>> volker.b...@duke.edu
>>> https://aims.pratt.duke.edu
>>> +1 (919) 660 5279
>>> Twitter: Aimsduke
>>>
>>> Office:  Hudson Hall
>>>
>>>
>>>
>>>
>>> ___
>>> users mailing list
>>> users@lists.open-mpi.org
>>>
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__rfd.newmexicoconsortium.org_mailman_listinfo_users&d=DwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=I9QLPu689VeINkpRod6EQfprr_v-FLoLsAuSXIHhDsk&m=QLtQXqnGSkgQnmgI4RxZXa9R6FhMmgj2FLN452Q0Wis&s=BeracGkSHhIyI_bjKJqPHCqMuP-Se2pRmbiNfugkdK8&e=
>> __

Re: [OMPI users] MPI_IN_PLACE

[Volker Blum]

Dear Gilles,

Thank you very much for the fast answer.

Darn. I feared it might not occur on all platforms, since my former Macbook 
(with an older OpenMPI version) no longer exhibited the problem, a different 
Linux/Intel Machine did last December, etc.

On this specific machine, the configure line is

./configure CC=gcc FC=ifort F77=ifort

ifort version 17.0.4

blum:/Users/blum/software/openmpi-3.0.0rc1> gcc -v
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr 
--with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 8.1.0 (clang-802.0.42)
Target: x86_64-apple-darwin16.6.0
Thread model: posix
InstalledDir: 
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

The full test program is appended.

Compilation:

mpif90 check_mpi_in_place.f90

blum:/Users/blum/codes/fhi-aims/openmpi_test> which mpif90
/usr/local/openmpi-3.0.0rc1/bin/mpif90

blum:/Users/blum/codes/fhi-aims/openmpi_test> which mpirun
/usr/local/openmpi-3.0.0rc1/bin/mpirun

blum:/Users/blum/codes/fhi-aims/openmpi_test> mpirun -np 2 a.out
 * MPI_IN_PLACE does not appear to work as intended.
 * Checking whether MPI_ALLREDUCE works at all.
 * Without MPI_IN_PLACE, MPI_ALLREDUCE appears to work.

blum:/Users/blum/codes/fhi-aims/openmpi_test> mpirun -np 1 a.out
 * MPI_IN_PLACE does not appear to work as intended.
 * Checking whether MPI_ALLREDUCE works at all.
 * Without MPI_IN_PLACE, MPI_ALLREDUCE appears to work.

Hopefully, no trivial mistakes in the testcase. I just spent a few days tracing 
this issue through a fairly large code, which is where the issue originally 
arose (and leads to wrong numbers).

Best wishes
Volker




> On Jul 26, 2017, at 9:46 AM, Gilles Gouaillardet 
>  wrote:
>
> Volker,
>
> i was unable to reproduce this issue on linux
>
> can you please post your full configure command line, your gnu
> compiler version and the full test program ?
>
> also, how many mpi tasks are you running ?
>
> Cheers,
>
> Gilles
>
> On Wed, Jul 26, 2017 at 4:25 PM, Volker Blum  wrote:
>> Hi,
>>
>> I tried openmpi-3.0.0rc1.tar.gz using Intel Fortran 2017 and gcc on a 
>> current MacOS system. For this version, it seems to me that MPI_IN_PLACE 
>> returns incorrect results (while other MPI implementations, including some 
>> past OpenMPI versions, work fine).
>>
>> This can be seen with a simple Fortran example code, shown below. In the 
>> test, the values of all entries of an array “test_data” should be 1.0d0 if 
>> the behavior were as intended. However, the version of OpenMPI I have 
>> returns 0.d0 instead.
>>
>> I’ve seen this behavior on some other compute platforms too, in the past, so 
>> it wasn’t new to me. Still, I thought that this time, I’d ask. Any thoughts?
>>
>> Thank you,
>> Best wishes
>> Volker
>>
>>! size of test data array
>>integer :: n_data
>>
>>! array that contains test data for MPI_IN_PLACE
>>real*8, allocatable :: test_data(:)
>>
>>integer :: mpierr
>>
>>n_data = 10
>>
>>allocate(test_data(n_data),stat=mpierr)
>>
>>! seed test data array for allreduce call below
>>if (myid.eq.0) then
>>   test_data(:) = 1.d0
>>else
>>   test_data(:) = 0.d0
>>end if
>>
>>! Sum the test_data array over all MPI tasks
>>call MPI_ALLREDUCE(MPI_IN_PLACE, &
>> test_data(:), &
>> n_data, &
>> MPI_DOUBLE_PRECISION, &
>> MPI_SUM, &
>> mpi_comm_global, &
>> mpierr )
>>
>>! The value of all entries of test_data should now be 1.d0 on all MPI 
>> tasks.
>>! If that is not the case, then the MPI_IN_PLACE flag may be broken.
>>
>>
>>
>>
>>
>>
>> Volker Blum
>> Associate Professor
>> Ab Initio Materials Simulations
>> Duke University, MEMS Department
>> 144 Hudson Hall, Box 90300, Duke University, Durham, NC 27708, USA
>>
>> volker.b...@duke.edu
>> https://aims.pratt.duke.edu
>> +1 (919) 660 5279
>> Twitter: Aimsduke
>>
>> Office:  Hudson Hall
>>
>>
>>
>>
>> ___
>> users mailing list
>> users@lists.open-mpi.org
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__rfd.newmexicoconsortium.org_mailman_listinfo_users&d=DwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=I9QLPu689VeINkpRod6EQfprr_v-FLoLsAuSXIHhDsk&m=QLtQXqnGSkgQnmgI4RxZXa9R6FhMmgj2FLN452Q0Wis&s=BeracGkSHhIyI_bjKJqPHCqMuP-Se2pRmbiNfugkdK8&e=
> ___
> users mailing list
> users@lists.open-mpi.org
> https://urldefense.proofpoint.com/v2/url?u=https-3A__rfd.newmexicoconsortium.org_mailman_listinfo_users&d=DwIGaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=I9QLPu689VeINkpRod6EQfprr_v-FLoLsAuSXIHhDsk&m=QLtQXqnGSkgQnmgI4RxZXa9R6FhMmgj2FLN452Q0Wis&s=BeracGkSHhIyI_bjKJqPHCqMuP-Se2pRmbiNfugkdK8&e=

Volker Blum
Associate Professor
Ab Initio Materials Simulations
Duke University, MEMS Department
144 Hudson Hall, Box 90300, Duke University, Durham, NC 27708, USA

volker.b...@duke.edu
https://aims.pratt.duke.edu
+1 (919) 660 5

Re: [OMPI users] MPI_IN_PLACE

[Gilles Gouaillardet]

Volker,

i was unable to reproduce this issue on linux

can you please post your full configure command line, your gnu
compiler version and the full test program ?

also, how many mpi tasks are you running ?

Cheers,

Gilles

On Wed, Jul 26, 2017 at 4:25 PM, Volker Blum  wrote:
> Hi,
>
> I tried openmpi-3.0.0rc1.tar.gz using Intel Fortran 2017 and gcc on a current 
> MacOS system. For this version, it seems to me that MPI_IN_PLACE returns 
> incorrect results (while other MPI implementations, including some past 
> OpenMPI versions, work fine).
>
> This can be seen with a simple Fortran example code, shown below. In the 
> test, the values of all entries of an array “test_data” should be 1.0d0 if 
> the behavior were as intended. However, the version of OpenMPI I have returns 
> 0.d0 instead.
>
> I’ve seen this behavior on some other compute platforms too, in the past, so 
> it wasn’t new to me. Still, I thought that this time, I’d ask. Any thoughts?
>
> Thank you,
> Best wishes
> Volker
>
> ! size of test data array
> integer :: n_data
>
> ! array that contains test data for MPI_IN_PLACE
> real*8, allocatable :: test_data(:)
>
> integer :: mpierr
>
> n_data = 10
>
> allocate(test_data(n_data),stat=mpierr)
>
> ! seed test data array for allreduce call below
> if (myid.eq.0) then
>test_data(:) = 1.d0
> else
>test_data(:) = 0.d0
> end if
>
> ! Sum the test_data array over all MPI tasks
> call MPI_ALLREDUCE(MPI_IN_PLACE, &
>  test_data(:), &
>  n_data, &
>  MPI_DOUBLE_PRECISION, &
>  MPI_SUM, &
>  mpi_comm_global, &
>  mpierr )
>
> ! The value of all entries of test_data should now be 1.d0 on all MPI 
> tasks.
> ! If that is not the case, then the MPI_IN_PLACE flag may be broken.
>
>
>
>
>
>
> Volker Blum
> Associate Professor
> Ab Initio Materials Simulations
> Duke University, MEMS Department
> 144 Hudson Hall, Box 90300, Duke University, Durham, NC 27708, USA
>
> volker.b...@duke.edu
> https://aims.pratt.duke.edu
> +1 (919) 660 5279
> Twitter: Aimsduke
>
> Office:  Hudson Hall
>
>
>
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[OMPI users] MPI_IN_PLACE

[Volker Blum]

Hi,

I tried openmpi-3.0.0rc1.tar.gz using Intel Fortran 2017 and gcc on a current 
MacOS system. For this version, it seems to me that MPI_IN_PLACE returns 
incorrect results (while other MPI implementations, including some past OpenMPI 
versions, work fine).

This can be seen with a simple Fortran example code, shown below. In the test, 
the values of all entries of an array “test_data” should be 1.0d0 if the 
behavior were as intended. However, the version of OpenMPI I have returns 0.d0 
instead.

I’ve seen this behavior on some other compute platforms too, in the past, so it 
wasn’t new to me. Still, I thought that this time, I’d ask. Any thoughts?

Thank you,
Best wishes
Volker

! size of test data array
integer :: n_data
  
! array that contains test data for MPI_IN_PLACE
real*8, allocatable :: test_data(:)

integer :: mpierr

n_data = 10

allocate(test_data(n_data),stat=mpierr)

! seed test data array for allreduce call below
if (myid.eq.0) then
   test_data(:) = 1.d0
else
   test_data(:) = 0.d0
end if

! Sum the test_data array over all MPI tasks
call MPI_ALLREDUCE(MPI_IN_PLACE, &
 test_data(:), &
 n_data, &
 MPI_DOUBLE_PRECISION, &
 MPI_SUM, &
 mpi_comm_global, &
 mpierr )

! The value of all entries of test_data should now be 1.d0 on all MPI tasks.
! If that is not the case, then the MPI_IN_PLACE flag may be broken.






Volker Blum
Associate Professor
Ab Initio Materials Simulations
Duke University, MEMS Department 
144 Hudson Hall, Box 90300, Duke University, Durham, NC 27708, USA

volker.b...@duke.edu
https://aims.pratt.duke.edu
+1 (919) 660 5279
Twitter: Aimsduke

Office:  Hudson Hall




___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] MPI_IN_PLACE with GATHERV, AGATHERV, and SCATERV

[Gerlach, Charles A.]

The IN_PLACE alterations to my code encompassed GATHERV as well, but as I 
continued to debug, it appeared more and more as though SCATTERV was the only 
problem case. 
So, I do not forsee any GATHER reproducers, but I'll certainly send 'em if I 
find 'em. 

I followed the link to the bug-diff, and I can confirm that my scatter_f.c and 
scatterv_f.c are wrong in my 1.6.5. Haven't re-compiled yet to make sure 
everything else goes.

-Charles

-Original Message-
From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Nathan Hjelm
Sent: Tuesday, October 22, 2013 2:50 PM
To: Open MPI Users
Subject: Re: [OMPI users] MPI_IN_PLACE with GATHERV, AGATHERV, and SCATERV

Ok, I think we have this resolved in trunk and the fix will go into 1.7.4. The 
check for MPI_IN_PLACE was wrong in the mpif-h bindings. The fix was tested 
with your reproducer. Both MPI_SCATTER and MPI_SCATTERV had this bug. The bug 
does not exist in 1.6.x though so I don't know why it was failing there.

I don't see a problem with MPI_GATHER or MPI_GATHERV though. Can you send a 
reproducer for those?

-Nathan Hjelm
HPC-3, LANL

On Tue, Oct 22, 2013 at 02:28:38PM +, Gerlach, Charles A. wrote:
> My reproducer is below (SCATTERV only). It needs to be compiled with 64-bit 
> default reals, and I'm running on four cores of a single linux86-64 box 
> running SLED 12.3 (except where noted). 
> 
> Using Open-MPI with different compilers:
> 
> With g95: The non-root procs print the correct values, but the root process 
> seg faults somewhere inside the SCATTERV call.
> With portland: I get: -1614907703: __hpf_esend: not implemented
>  (All procs print out the correct values.) With 
> Intel (on a Mac Pro): Complains about a null communicator in MPI_FINALIZE and 
> crashes. All procs print out the correct values.
> 
> With all three of these compilers, if I comment out the entire IF (MYPN.EQ.0) 
> code so that all procs pass RARR1 into both the send and recv buffers, I get 
> no errors.
> 
> With gfortran: This works either way (with IN_PLACE or without).
> 
> Other MPI implementations:
> 
> With MPICH2 (any compiler) and Intel Visual Fortran on Windows, the IN_PLACE 
> code works. 
> They specifically prohibit passing RARR1 into both the send and recv buffers 
> on the root proc. 
> 
> Reproducer:
> 
> PROGRAM MAIN
> 
>   IMPLICIT NONE
> 
>   REAL, DIMENSION(1200)  :: RARR1
>   INTEGER, DIMENSION(4) :: SEND_NUM, SEND_OFF
>   INTEGER :: RECV_NUM, MYPN, NPES, IERR
> 
>   INTEGER :: I, J
> 
>   INCLUDE 'mpif.h'
> 
>   SEND_NUM = (/ 300, 300, 300, 300 /)
>   SEND_OFF = (/ 0, 300, 600, 900 /)
>   RECV_NUM = 300
> 
>   CALL MPI_INIT(IERR)
> 
>   CALL MPI_COMM_SIZE(MPI_COMM_WORLD, NPES, IERR)
>   CALL MPI_COMM_RANK(MPI_COMM_WORLD, MYPN, IERR)
> 
>   IF (MYPN.EQ.0) THEN
>  DO I = 1,1200
> RARR1(I) = 0.001*I
>  ENDDO
>   ELSE
>  RARR1 = 0.0
>   ENDIF
> 
>   IF (MYPN.EQ.0) THEN
>  CALL MPI_SCATTERV(RARR1,SEND_NUM,SEND_OFF,MPI_DOUBLE_PRECISION, &
>   MPI_IN_PLACE,RECV_NUM,MPI_DOUBLE_PRECISION,0,MPI_COMM_WORLD,IERR)
>   ELSE
>  CALL MPI_SCATTERV(RARR1,SEND_NUM,SEND_OFF,MPI_DOUBLE_PRECISION, &
>   RARR1,RECV_NUM,MPI_DOUBLE_PRECISION,0,MPI_COMM_WORLD,IERR)
>   ENDIF
> 
>   OPEN(71+MYPN,FORM='FORMATTED',POSITION='APPEND')
>   WRITE(71+MYPN,'(3E15.7)') RARR1(1:300)
>   CLOSE(71+MYPN)
> 
>   CALL MPI_FINALIZE(IERR)
> 
> END PROGRAM MAIN
> 
> 
> 
> From: users [users-boun...@open-mpi.org] on behalf of Nathan Hjelm 
> [hje...@lanl.gov]
> Sent: Wednesday, October 09, 2013 12:37 PM
> To: Open MPI Users
> Subject: Re: [OMPI users] MPI_IN_PLACE with GATHERV, AGATHERV, and 
> SCATERV
> 
> These functions are tested nightly and there has been no indication 
> any of these functions fail with MPI_IN_PLACE. Can you provide a reproducer?
> 
> -Nathan
> HPC-3, LANL
> 
> On Tue, Oct 08, 2013 at 07:40:50PM +, Gerlach, Charles A. wrote:
> >I have an MPI code that was developed using MPICH1 and OpenMPI before the
> >MPI2 standards became commonplace (before MPI_IN_PLACE was an option).
> >
> >
> >
> >So, my code has many examples of GATHERV, AGATHERV and SCATTERV, where I
> >pass the same array in as the SEND_BUF and the RECV_BUF, and this has
> >worked fine for many years.
> >
> >
> >
> >Intel MPI and MPICH2 explicitly disallow this behavior according to the
> >MPI2 standard. So, I have gone through and used MPI_IN_PLACE for all the
> >GATHERV/SCATTERVs that used to pass the same array twice. Thi

Re: [OMPI users] MPI_IN_PLACE with GATHERV, AGATHERV, and SCATERV

[Nathan Hjelm]

Ok, I think we have this resolved in trunk and the fix will go into 1.7.4. The
check for MPI_IN_PLACE was wrong in the mpif-h bindings. The fix was tested with
your reproducer. Both MPI_SCATTER and MPI_SCATTERV had this bug. The bug does 
not
exist in 1.6.x though so I don't know why it was failing there.

I don't see a problem with MPI_GATHER or MPI_GATHERV though. Can you send a
reproducer for those?

-Nathan Hjelm
HPC-3, LANL

On Tue, Oct 22, 2013 at 02:28:38PM +, Gerlach, Charles A. wrote:
> My reproducer is below (SCATTERV only). It needs to be compiled with 64-bit 
> default reals, and I'm running on four cores of a single linux86-64 box 
> running SLED 12.3 (except where noted). 
> 
> Using Open-MPI with different compilers:
> 
> With g95: The non-root procs print the correct values, but the root process 
> seg faults somewhere inside the SCATTERV call.
> With portland: I get: -1614907703: __hpf_esend: not implemented
>  (All procs print out the correct values.)
> With Intel (on a Mac Pro): Complains about a null communicator in 
> MPI_FINALIZE and crashes. All procs print out the correct values.
> 
> With all three of these compilers, if I comment out the entire IF (MYPN.EQ.0) 
> code so that all procs pass RARR1 into both the send and recv buffers, I get 
> no errors.
> 
> With gfortran: This works either way (with IN_PLACE or without).
> 
> Other MPI implementations:
> 
> With MPICH2 (any compiler) and Intel Visual Fortran on Windows, the IN_PLACE 
> code works. 
> They specifically prohibit passing RARR1 into both the send and recv buffers 
> on the root proc. 
> 
> Reproducer:
> 
> PROGRAM MAIN
> 
>   IMPLICIT NONE
> 
>   REAL, DIMENSION(1200)  :: RARR1
>   INTEGER, DIMENSION(4) :: SEND_NUM, SEND_OFF
>   INTEGER :: RECV_NUM, MYPN, NPES, IERR
> 
>   INTEGER :: I, J
> 
>   INCLUDE 'mpif.h'
> 
>   SEND_NUM = (/ 300, 300, 300, 300 /)
>   SEND_OFF = (/ 0, 300, 600, 900 /)
>   RECV_NUM = 300
> 
>   CALL MPI_INIT(IERR)
> 
>   CALL MPI_COMM_SIZE(MPI_COMM_WORLD, NPES, IERR)
>   CALL MPI_COMM_RANK(MPI_COMM_WORLD, MYPN, IERR)
> 
>   IF (MYPN.EQ.0) THEN
>  DO I = 1,1200
> RARR1(I) = 0.001*I
>  ENDDO
>   ELSE
>  RARR1 = 0.0
>   ENDIF
> 
>   IF (MYPN.EQ.0) THEN
>  CALL MPI_SCATTERV(RARR1,SEND_NUM,SEND_OFF,MPI_DOUBLE_PRECISION, &
>   MPI_IN_PLACE,RECV_NUM,MPI_DOUBLE_PRECISION,0,MPI_COMM_WORLD,IERR)
>   ELSE
>  CALL MPI_SCATTERV(RARR1,SEND_NUM,SEND_OFF,MPI_DOUBLE_PRECISION, &
>   RARR1,RECV_NUM,MPI_DOUBLE_PRECISION,0,MPI_COMM_WORLD,IERR)
>   ENDIF
> 
>   OPEN(71+MYPN,FORM='FORMATTED',POSITION='APPEND')
>   WRITE(71+MYPN,'(3E15.7)') RARR1(1:300)
>   CLOSE(71+MYPN)
> 
>   CALL MPI_FINALIZE(IERR)
> 
> END PROGRAM MAIN
> 
> 
> 
> From: users [users-boun...@open-mpi.org] on behalf of Nathan Hjelm 
> [hje...@lanl.gov]
> Sent: Wednesday, October 09, 2013 12:37 PM
> To: Open MPI Users
> Subject: Re: [OMPI users] MPI_IN_PLACE with GATHERV, AGATHERV, and SCATERV
> 
> These functions are tested nightly and there has been no indication any of 
> these
> functions fail with MPI_IN_PLACE. Can you provide a reproducer?
> 
> -Nathan
> HPC-3, LANL
> 
> On Tue, Oct 08, 2013 at 07:40:50PM +, Gerlach, Charles A. wrote:
> >I have an MPI code that was developed using MPICH1 and OpenMPI before the
> >MPI2 standards became commonplace (before MPI_IN_PLACE was an option).
> >
> >
> >
> >So, my code has many examples of GATHERV, AGATHERV and SCATTERV, where I
> >pass the same array in as the SEND_BUF and the RECV_BUF, and this has
> >worked fine for many years.
> >
> >
> >
> >Intel MPI and MPICH2 explicitly disallow this behavior according to the
> >MPI2 standard. So, I have gone through and used MPI_IN_PLACE for all the
> >GATHERV/SCATTERVs that used to pass the same array twice. This code now
> >works with MPICH2 and Intel_MPI, but fails with OpenMPI-1.6.5 on multiple
> >platforms and compilers.
> >
> >
> >
> >PLATFORM  COMPILERSUCCESS? (For at least one
> >simple example)
> >
> >
> >
> >SLED 12.3 (x86-64) - Portland group  - fails
> >
> >SLED 12.3 (x86-64) - g95 - fails
> >
> >SLED 12.3 (x86-64) - gfortran   - works
> >
> >
> >
> >OS X 10.8 -- intel-fails
> >
> &g

Re: [OMPI users] MPI_IN_PLACE with GATHERV, AGATHERV, and SCATERV

[Gerlach, Charles A.]

My reproducer is below (SCATTERV only). It needs to be compiled with 64-bit 
default reals, and I'm running on four cores of a single linux86-64 box running 
SLED 12.3 (except where noted). 

Using Open-MPI with different compilers:

With g95: The non-root procs print the correct values, but the root process seg 
faults somewhere inside the SCATTERV call.
With portland: I get: -1614907703: __hpf_esend: not implemented
 (All procs print out the correct values.)
With Intel (on a Mac Pro): Complains about a null communicator in MPI_FINALIZE 
and crashes. All procs print out the correct values.

With all three of these compilers, if I comment out the entire IF (MYPN.EQ.0) 
code so that all procs pass RARR1 into both the send and recv buffers, I get no 
errors.

With gfortran: This works either way (with IN_PLACE or without).

Other MPI implementations:

With MPICH2 (any compiler) and Intel Visual Fortran on Windows, the IN_PLACE 
code works. 
They specifically prohibit passing RARR1 into both the send and recv buffers on 
the root proc. 

Reproducer:

PROGRAM MAIN

  IMPLICIT NONE

  REAL, DIMENSION(1200)  :: RARR1
  INTEGER, DIMENSION(4) :: SEND_NUM, SEND_OFF
  INTEGER :: RECV_NUM, MYPN, NPES, IERR

  INTEGER :: I, J

  INCLUDE 'mpif.h'

  SEND_NUM = (/ 300, 300, 300, 300 /)
  SEND_OFF = (/ 0, 300, 600, 900 /)
  RECV_NUM = 300

  CALL MPI_INIT(IERR)

  CALL MPI_COMM_SIZE(MPI_COMM_WORLD, NPES, IERR)
  CALL MPI_COMM_RANK(MPI_COMM_WORLD, MYPN, IERR)

  IF (MYPN.EQ.0) THEN
 DO I = 1,1200
RARR1(I) = 0.001*I
 ENDDO
  ELSE
 RARR1 = 0.0
  ENDIF

  IF (MYPN.EQ.0) THEN
 CALL MPI_SCATTERV(RARR1,SEND_NUM,SEND_OFF,MPI_DOUBLE_PRECISION, &
  MPI_IN_PLACE,RECV_NUM,MPI_DOUBLE_PRECISION,0,MPI_COMM_WORLD,IERR)
  ELSE
 CALL MPI_SCATTERV(RARR1,SEND_NUM,SEND_OFF,MPI_DOUBLE_PRECISION, &
  RARR1,RECV_NUM,MPI_DOUBLE_PRECISION,0,MPI_COMM_WORLD,IERR)
  ENDIF

  OPEN(71+MYPN,FORM='FORMATTED',POSITION='APPEND')
  WRITE(71+MYPN,'(3E15.7)') RARR1(1:300)
  CLOSE(71+MYPN)

  CALL MPI_FINALIZE(IERR)

END PROGRAM MAIN



From: users [users-boun...@open-mpi.org] on behalf of Nathan Hjelm 
[hje...@lanl.gov]
Sent: Wednesday, October 09, 2013 12:37 PM
To: Open MPI Users
Subject: Re: [OMPI users] MPI_IN_PLACE with GATHERV, AGATHERV, and SCATERV

These functions are tested nightly and there has been no indication any of these
functions fail with MPI_IN_PLACE. Can you provide a reproducer?

-Nathan
HPC-3, LANL

On Tue, Oct 08, 2013 at 07:40:50PM +, Gerlach, Charles A. wrote:
>I have an MPI code that was developed using MPICH1 and OpenMPI before the
>MPI2 standards became commonplace (before MPI_IN_PLACE was an option).
>
>
>
>So, my code has many examples of GATHERV, AGATHERV and SCATTERV, where I
>pass the same array in as the SEND_BUF and the RECV_BUF, and this has
>worked fine for many years.
>
>
>
>Intel MPI and MPICH2 explicitly disallow this behavior according to the
>MPI2 standard. So, I have gone through and used MPI_IN_PLACE for all the
>GATHERV/SCATTERVs that used to pass the same array twice. This code now
>works with MPICH2 and Intel_MPI, but fails with OpenMPI-1.6.5 on multiple
>platforms and compilers.
>
>
>
>PLATFORM  COMPILERSUCCESS? (For at least one
>simple example)
>
>
>
>SLED 12.3 (x86-64) - Portland group  - fails
>
>SLED 12.3 (x86-64) - g95 - fails
>
>SLED 12.3 (x86-64) - gfortran   - works
>
>
>
>OS X 10.8 -- intel-fails
>
>
>
>
>
>In every case where OpenMPI fails with the MPI_IN_PLACE code, I can go
>back to the original code that passes the same array twice instead of
>using MPI_IN_PLACE, and it is fine.
>
>
>
>I have made a test case doing an individual GATHERV with MPI_IN_PLACE, and
>it works with OpenMPI.  So it looks like there is some interaction with my
>code that is causing the problem. I have no idea how to go about trying to
>debug it.
>
>
>
>
>
>In summary:
>
>
>
>OpenMPI-1.6.5 crashes my code when I use GATHERV, AGATHERV, and SCATTERV
>with MPI_IN_PLACE.
>
>Intel MPI and MPICH2 work with my code when I use GATHERV, AGATHERV, and
>SCATTERV with MPI_IN_PLACE.
>
>
>
>OpenMPI-1.6.5 works with my code when I pass the same array to SEND_BUF
>and RECV_BUF instead of using MPI_IN_PLACE for those same GATHERV,
>AGATHERV, and SCATTERVs.
>
>
>
>
>
>-Charles

> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] MPI_IN_PLACE with GATHERV, AGATHERV, and SCATERV

[Nathan Hjelm]

These functions are tested nightly and there has been no indication any of these
functions fail with MPI_IN_PLACE. Can you provide a reproducer?

-Nathan
HPC-3, LANL

On Tue, Oct 08, 2013 at 07:40:50PM +, Gerlach, Charles A. wrote:
>I have an MPI code that was developed using MPICH1 and OpenMPI before the
>MPI2 standards became commonplace (before MPI_IN_PLACE was an option).
> 
> 
> 
>So, my code has many examples of GATHERV, AGATHERV and SCATTERV, where I
>pass the same array in as the SEND_BUF and the RECV_BUF, and this has
>worked fine for many years.
> 
> 
> 
>Intel MPI and MPICH2 explicitly disallow this behavior according to the
>MPI2 standard. So, I have gone through and used MPI_IN_PLACE for all the
>GATHERV/SCATTERVs that used to pass the same array twice. This code now
>works with MPICH2 and Intel_MPI, but fails with OpenMPI-1.6.5 on multiple
>platforms and compilers.
> 
> 
> 
>PLATFORM  COMPILERSUCCESS? (For at least one
>simple example)
> 
>
> 
>SLED 12.3 (x86-64) - Portland group  - fails
> 
>SLED 12.3 (x86-64) - g95 - fails
> 
>SLED 12.3 (x86-64) - gfortran   - works
> 
> 
> 
>OS X 10.8 -- intel-fails
> 
> 
> 
> 
> 
>In every case where OpenMPI fails with the MPI_IN_PLACE code, I can go
>back to the original code that passes the same array twice instead of
>using MPI_IN_PLACE, and it is fine.
> 
> 
> 
>I have made a test case doing an individual GATHERV with MPI_IN_PLACE, and
>it works with OpenMPI.  So it looks like there is some interaction with my
>code that is causing the problem. I have no idea how to go about trying to
>debug it.
> 
> 
> 
> 
> 
>In summary:
> 
> 
> 
>OpenMPI-1.6.5 crashes my code when I use GATHERV, AGATHERV, and SCATTERV
>with MPI_IN_PLACE.
> 
>Intel MPI and MPICH2 work with my code when I use GATHERV, AGATHERV, and
>SCATTERV with MPI_IN_PLACE.
> 
> 
> 
>OpenMPI-1.6.5 works with my code when I pass the same array to SEND_BUF
>and RECV_BUF instead of using MPI_IN_PLACE for those same GATHERV,
>AGATHERV, and SCATTERVs.  
> 
> 
> 
> 
> 
>-Charles

> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] MPI_IN_PLACE with GATHERV, AGATHERV, and SCATERV

[Jeff Hammond]

"I have made a test case..." means there is little reason not to
attach said test case to the email for verification :-)

The following is in mpi.h.in in the OpenMPI trunk.

=
/*
 * Just in case you need it.  :-)
 */
#define OPEN_MPI 1

/*
 * MPI version
 */
#define MPI_VERSION 2
#define MPI_SUBVERSION 2
=

Two things can be said from this:

(1) You can workaround this non-portable awfulness with the C
preprocess by testing for the OPEN_MPI symbol.

(2) OpenMPI claims to be compliant with the MPI 2.2 standard, hence
any failures to adhere to the behavior specified in that document for
MPI_IN_PLACE is erroneous.

Best,

Jeff

On Tue, Oct 8, 2013 at 2:40 PM, Gerlach, Charles A.
 wrote:
> I have an MPI code that was developed using MPICH1 and OpenMPI before the
> MPI2 standards became commonplace (before MPI_IN_PLACE was an option).
>
>
>
> So, my code has many examples of GATHERV, AGATHERV and SCATTERV, where I
> pass the same array in as the SEND_BUF and the RECV_BUF, and this has worked
> fine for many years.
>
>
>
> Intel MPI and MPICH2 explicitly disallow this behavior according to the MPI2
> standard. So, I have gone through and used MPI_IN_PLACE for all the
> GATHERV/SCATTERVs that used to pass the same array twice. This code now
> works with MPICH2 and Intel_MPI, but fails with OpenMPI-1.6.5 on multiple
> platforms and compilers.
>
>
>
> PLATFORM  COMPILERSUCCESS? (For at least one
> simple example)
>
> 
>
> SLED 12.3 (x86-64) – Portland group  - fails
>
> SLED 12.3 (x86-64) – g95 - fails
>
> SLED 12.3 (x86-64) – gfortran   - works
>
>
>
> OS X 10.8 -- intel-fails
>
>
>
>
>
> In every case where OpenMPI fails with the MPI_IN_PLACE code, I can go back
> to the original code that passes the same array twice instead of using
> MPI_IN_PLACE, and it is fine.
>
>
>
> I have made a test case doing an individual GATHERV with MPI_IN_PLACE, and
> it works with OpenMPI.  So it looks like there is some interaction with my
> code that is causing the problem. I have no idea how to go about trying to
> debug it.
>
>
>
>
>
> In summary:
>
>
>
> OpenMPI-1.6.5 crashes my code when I use GATHERV, AGATHERV, and SCATTERV
> with MPI_IN_PLACE.
>
> Intel MPI and MPICH2 work with my code when I use GATHERV, AGATHERV, and
> SCATTERV with MPI_IN_PLACE.
>
>
>
> OpenMPI-1.6.5 works with my code when I pass the same array to SEND_BUF and
> RECV_BUF instead of using MPI_IN_PLACE for those same GATHERV, AGATHERV, and
> SCATTERVs.
>
>
>
>
>
> -Charles
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users



-- 
Jeff Hammond
jeff.scie...@gmail.com

[OMPI users] MPI_IN_PLACE with GATHERV, AGATHERV, and SCATERV

[Gerlach, Charles A.]

I have an MPI code that was developed using MPICH1 and OpenMPI before the MPI2 
standards became commonplace (before MPI_IN_PLACE was an option).

So, my code has many examples of GATHERV, AGATHERV and SCATTERV, where I pass 
the same array in as the SEND_BUF and the RECV_BUF, and this has worked fine 
for many years.

Intel MPI and MPICH2 explicitly disallow this behavior according to the MPI2 
standard. So, I have gone through and used MPI_IN_PLACE for all the 
GATHERV/SCATTERVs that used to pass the same array twice. This code now works 
with MPICH2 and Intel_MPI, but fails with OpenMPI-1.6.5 on multiple platforms 
and compilers.

PLATFORM  COMPILERSUCCESS? (For at least one simple 
example)

SLED 12.3 (x86-64) - Portland group  - fails
SLED 12.3 (x86-64) - g95 - fails
SLED 12.3 (x86-64) - gfortran   - works

OS X 10.8 -- intel-fails


In every case where OpenMPI fails with the MPI_IN_PLACE code, I can go back to 
the original code that passes the same array twice instead of using 
MPI_IN_PLACE, and it is fine.

I have made a test case doing an individual GATHERV with MPI_IN_PLACE, and it 
works with OpenMPI.  So it looks like there is some interaction with my code 
that is causing the problem. I have no idea how to go about trying to debug it.


In summary:

OpenMPI-1.6.5 crashes my code when I use GATHERV, AGATHERV, and SCATTERV with 
MPI_IN_PLACE.
Intel MPI and MPICH2 work with my code when I use GATHERV, AGATHERV, and 
SCATTERV with MPI_IN_PLACE.

OpenMPI-1.6.5 works with my code when I pass the same array to SEND_BUF and 
RECV_BUF instead of using MPI_IN_PLACE for those same GATHERV, AGATHERV, and 
SCATTERVs.


-Charles

Re: [OMPI users] MPI_IN_PLACE in a call to MPI_Allreduce in Fortran

[Hugo Gagnon]

On Wed, Sep 11, 2013, at 13:24, Jeff Squyres (jsquyres) wrote:
> On Sep 11, 2013, at 7:22 PM, Hugo Gagnon
>  wrote:
> 
> >> This is definitely a puzzle, because I just installed gcc 4.8.1 on my
> >> 10.8.4 OS X MBP,
> > 
> > I also just recompiled gcc 4.8.1_3 from MacPorts, and will recompile
> > openmpi 1.6.5 myself rather than using MacPorts' version.  May I ask
> > what are the exact options you passed to ./configure?
> 
> Here's what I used:
> 
> ./configure --prefix=/Users/jsquyres/bogus --disable-vt
> --enable-mpirun-prefix-by-default --enable-mpi-f77 --enable-mpi-f90
> CC=/opt/local/bin/gcc-mp-4.8 F77=/opt/local/bin/gfortran-mp-4.8
> FC=/opt/local/bin/gfortran-mp-4.8
> 
> I don't think I realized that you used ompi from MacPorts.  I'll try that
> one and see what happens.

It works now!!  There must be something wrong with the MacPorts build...

> 
> -- 
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] MPI_IN_PLACE in a call to MPI_Allreduce in Fortran

[Jeff Squyres (jsquyres)]

On Sep 11, 2013, at 7:22 PM, Hugo Gagnon  
wrote:

>> This is definitely a puzzle, because I just installed gcc 4.8.1 on my
>> 10.8.4 OS X MBP,
> 
> I also just recompiled gcc 4.8.1_3 from MacPorts, and will recompile
> openmpi 1.6.5 myself rather than using MacPorts' version.  May I ask
> what are the exact options you passed to ./configure?

Here's what I used:

./configure --prefix=/Users/jsquyres/bogus --disable-vt 
--enable-mpirun-prefix-by-default --enable-mpi-f77 --enable-mpi-f90 
CC=/opt/local/bin/gcc-mp-4.8 F77=/opt/local/bin/gfortran-mp-4.8 
FC=/opt/local/bin/gfortran-mp-4.8

I don't think I realized that you used ompi from MacPorts.  I'll try that one 
and see what happens.

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] MPI_IN_PLACE in a call to MPI_Allreduce in Fortran

[Hugo Gagnon]

On Wed, Sep 11, 2013, at 12:26, Jeff Squyres (jsquyres) wrote:
> On Sep 10, 2013, at 2:33 PM, Hugo Gagnon
>  wrote:
> 
> > I only get the correct output when I use the more "conventional" syntax:
> > 
> > ...
> > call MPI_Allreduce(a_loc,a,2,MPI_INTEGER,MPI_SUM,MPI_COMM_WORLD,ierr)
> > ...
> 
> What is a_loc?  I'm assuming you know it can't be the same buffer as a.
> 
> > However, I get the wrong output when I use MPI_IN_PLACE:
> > 
> > ...
> > MPI_Allreduce(MPI_IN_PLACE,a,2,MPI_INTEGER,MPI_SUM,MPI_COMM_WORLD,ierr)
> > ...
> 
> This is definitely a puzzle, because I just installed gcc 4.8.1 on my
> 10.8.4 OS X MBP,

I also just recompiled gcc 4.8.1_3 from MacPorts, and will recompile
openmpi 1.6.5 myself rather than using MacPorts' version.  May I ask
what are the exact options you passed to ./configure?

I'll let you know if that fixes the problem.  Thank you.

 compiled OMPI 1.6.5 from the open-mpi.org web site, and
> ran the test program, and I get the correct answers:
> 
> -
> [6:23] jsquyres-mac:~/mpi ❯❯❯ mpif90 in_place.f90
> [6:23] jsquyres-mac:~/mpi ❯❯❯ mpirun -np 2 ./a.out
>0   4   6
>1   4   6
> [6:24] jsquyres-mac:~/mpi ❯❯❯ mpirun -np 3 ./a.out
>0   7  10
>1   7  10
>2   7  10
> [6:24] jsquyres-mac:~/mpi ❯❯❯ mpirun -np 4 ./a.out
>0  10  14
>1  10  14
>2  10  14
>3  10  14
> [6:24] jsquyres-mac:~/mpi ❯❯❯ mpicc --version
> gcc-mp-4.8 (MacPorts gcc48 4.8.1_3) 4.8.1
> Copyright (C) 2013 Free Software Foundation, Inc.
> This is free software; see the source for copying conditions.  There is
> NO
> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
> PURPOSE.
> 
> [6:24] jsquyres-mac:~/mpi ❯❯❯ mpif90 --version
> GNU Fortran (MacPorts gcc48 4.8.1_3) 4.8.1
> Copyright (C) 2013 Free Software Foundation, Inc.
> 
> GNU Fortran comes with NO WARRANTY, to the extent permitted by law.
> You may redistribute copies of GNU Fortran
> under the terms of the GNU General Public License.
> For more information about these matters, see the file named COPYING
> 
> [6:24] jsquyres-mac:~/mpi ❯❯❯ 
> -
> 
> Since I'm unable to replicate the problem, can you dig into the OMPI
> internals and see what's going wrong (e.g., via gdb or some other
> debugger)?
> 
> -- 
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] MPI_IN_PLACE in a call to MPI_Allreduce in Fortran

[Jeff Squyres (jsquyres)]

On Sep 10, 2013, at 2:33 PM, Hugo Gagnon  
wrote:

> I only get the correct output when I use the more "conventional" syntax:
> 
> ...
> call MPI_Allreduce(a_loc,a,2,MPI_INTEGER,MPI_SUM,MPI_COMM_WORLD,ierr)
> ...

What is a_loc?  I'm assuming you know it can't be the same buffer as a.

> However, I get the wrong output when I use MPI_IN_PLACE:
> 
> ...
> MPI_Allreduce(MPI_IN_PLACE,a,2,MPI_INTEGER,MPI_SUM,MPI_COMM_WORLD,ierr)
> ...

This is definitely a puzzle, because I just installed gcc 4.8.1 on my 10.8.4 OS 
X MBP, compiled OMPI 1.6.5 from the open-mpi.org web site, and ran the test 
program, and I get the correct answers:

-
[6:23] jsquyres-mac:~/mpi ❯❯❯ mpif90 in_place.f90
[6:23] jsquyres-mac:~/mpi ❯❯❯ mpirun -np 2 ./a.out
   0   4   6
   1   4   6
[6:24] jsquyres-mac:~/mpi ❯❯❯ mpirun -np 3 ./a.out
   0   7  10
   1   7  10
   2   7  10
[6:24] jsquyres-mac:~/mpi ❯❯❯ mpirun -np 4 ./a.out
   0  10  14
   1  10  14
   2  10  14
   3  10  14
[6:24] jsquyres-mac:~/mpi ❯❯❯ mpicc --version
gcc-mp-4.8 (MacPorts gcc48 4.8.1_3) 4.8.1
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[6:24] jsquyres-mac:~/mpi ❯❯❯ mpif90 --version
GNU Fortran (MacPorts gcc48 4.8.1_3) 4.8.1
Copyright (C) 2013 Free Software Foundation, Inc.

GNU Fortran comes with NO WARRANTY, to the extent permitted by law.
You may redistribute copies of GNU Fortran
under the terms of the GNU General Public License.
For more information about these matters, see the file named COPYING

[6:24] jsquyres-mac:~/mpi ❯❯❯ 
-

Since I'm unable to replicate the problem, can you dig into the OMPI internals 
and see what's going wrong (e.g., via gdb or some other debugger)?

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] MPI_IN_PLACE in a call to MPI_Allreduce in Fortran

[Hugo Gagnon]

I only get the correct output when I use the more "conventional" syntax:

...
call MPI_Allreduce(a_loc,a,2,MPI_INTEGER,MPI_SUM,MPI_COMM_WORLD,ierr)
...

However, I get the wrong output when I use MPI_IN_PLACE:

...
MPI_Allreduce(MPI_IN_PLACE,a,2,MPI_INTEGER,MPI_SUM,MPI_COMM_WORLD,ierr)
...

hence my question to this forum in the first place.

I also tried the code snippet at

https://svn.open-mpi.org/trac/ompi/ticket/1982

and that doesn't work for me either, i.e. all I get is zeros.

-- 
  Hugo Gagnon

On Tue, Sep 10, 2013, at 5:58, Jeff Squyres (jsquyres) wrote:
> On Sep 7, 2013, at 5:14 AM, Hugo Gagnon
>  wrote:
> 
> > $ openmpif90 test.f90
> > $ openmpirun -np 2 a.out
> >   0   4   6
> >   1   4   6
> > 
> > Now I'd be curious to know why your OpenMPI implementation handles
> > MPI_IN_PLACE correctly and not mine!
> 
> I don't understand -- this looks like the correct output to me.
> 
> Are you seeing some other problem?
> 
> -- 
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] MPI_IN_PLACE in a call to MPI_Allreduce in Fortran

[Jeff Squyres (jsquyres)]

On Sep 7, 2013, at 5:14 AM, Hugo Gagnon  
wrote:

> $ openmpif90 test.f90
> $ openmpirun -np 2 a.out
>   0   4   6
>   1   4   6
> 
> Now I'd be curious to know why your OpenMPI implementation handles
> MPI_IN_PLACE correctly and not mine!

I don't understand -- this looks like the correct output to me.

Are you seeing some other problem?

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] MPI_IN_PLACE in a call to MPI_Allreduce in Fortran

[Tom Rosmond]

What Fortran compiler is your OpenMPI build with?  Some fortran's don't
understand MPI_IN_PLACE.  Do a 'fortran MPI_IN_PLACE' search to see
several instances.

T. Rosmond



On Sat, 2013-09-07 at 10:16 -0400, Hugo Gagnon wrote:
> Nope, no luck.  My environment is:
> 
> OpenMPI 1.6.5
> gcc 4.8.1
> Mac OS 10.8
> 
> I found a ticket reporting a similar problem on OS X:
> 
> https://svn.open-mpi.org/trac/ompi/ticket/1982
> 
> It said to make sure $prefix/share/ompi/mpif90-wrapper-data.txt had the
> following line:
> 
> compiler_flags=-Wl,-commons,use_dylibs
> 
> I checked mine and it does (I even tried to include it explicitly on the
> command line but without success), what should I do next?
> 

Re: [OMPI users] MPI_IN_PLACE in a call to MPI_Allreduce in Fortran

[Hugo Gagnon]

Nope, no luck.  My environment is:

OpenMPI 1.6.5
gcc 4.8.1
Mac OS 10.8

I found a ticket reporting a similar problem on OS X:

https://svn.open-mpi.org/trac/ompi/ticket/1982

It said to make sure $prefix/share/ompi/mpif90-wrapper-data.txt had the
following line:

compiler_flags=-Wl,-commons,use_dylibs

I checked mine and it does (I even tried to include it explicitly on the
command line but without success), what should I do next?

-- 
  Hugo Gagnon

On Sat, Sep 7, 2013, at 0:39, Tom Rosmond wrote:
> Just as an experiment, try replacing
> 
> use mpi
> 
>   with
> 
> include 'mpif.h'
> 
> If that fixes the problem, you can confront the  OpenMPI experts
> 
> T. Rosmond
> 
> 
> 
> On Fri, 2013-09-06 at 23:14 -0400, Hugo Gagnon wrote:
> > Thanks for the input but it still doesn't work for me...  Here's the
> > version without MPI_IN_PLACE that does work:
> > 
> > program test
> > use mpi
> > integer :: ierr, myrank, a(2), a_loc(2) = 0
> > call MPI_Init(ierr)
> > call MPI_Comm_rank(MPI_COMM_WORLD,myrank,ierr)
> > if (myrank == 0) then
> >   a_loc(1) = 1
> >   a_loc(2) = 2
> > else
> >   a_loc(1) = 3
> >   a_loc(2) = 4
> > endif
> > call MPI_Allreduce(a_loc,a,2,MPI_INTEGER,MPI_SUM,MPI_COMM_WORLD,ierr)
> > write(*,*) myrank, a(:)
> > call MPI_Finalize(ierr)
> > end program test
> > 
> > $ openmpif90 test.f90
> > $ openmpirun -np 2 a.out
> >0   4   6
> >1   4   6
> > 
> > Now I'd be curious to know why your OpenMPI implementation handles
> > MPI_IN_PLACE correctly and not mine!
> > 
> 
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] MPI_IN_PLACE in a call to MPI_Allreduce in Fortran

[Tom Rosmond]

Just as an experiment, try replacing

use mpi

  with

include 'mpif.h'

If that fixes the problem, you can confront the  OpenMPI experts

T. Rosmond



On Fri, 2013-09-06 at 23:14 -0400, Hugo Gagnon wrote:
> Thanks for the input but it still doesn't work for me...  Here's the
> version without MPI_IN_PLACE that does work:
> 
> program test
> use mpi
> integer :: ierr, myrank, a(2), a_loc(2) = 0
> call MPI_Init(ierr)
> call MPI_Comm_rank(MPI_COMM_WORLD,myrank,ierr)
> if (myrank == 0) then
>   a_loc(1) = 1
>   a_loc(2) = 2
> else
>   a_loc(1) = 3
>   a_loc(2) = 4
> endif
> call MPI_Allreduce(a_loc,a,2,MPI_INTEGER,MPI_SUM,MPI_COMM_WORLD,ierr)
> write(*,*) myrank, a(:)
> call MPI_Finalize(ierr)
> end program test
> 
> $ openmpif90 test.f90
> $ openmpirun -np 2 a.out
>0   4   6
>1   4   6
> 
> Now I'd be curious to know why your OpenMPI implementation handles
> MPI_IN_PLACE correctly and not mine!
> 

Re: [OMPI users] MPI_IN_PLACE in a call to MPI_Allreduce in Fortran

[Tom Rosmond]

I'm afraid I can't answer that.   Here's my environment:

 OpenMPI 1.6.1
 IFORT 12.0.3.174
 Scientific Linux 6.4

 What fortran compiler are you using?  

T. Rosmond



On Fri, 2013-09-06 at 23:14 -0400, Hugo Gagnon wrote:
> Thanks for the input but it still doesn't work for me...  Here's the
> version without MPI_IN_PLACE that does work:
> 
> program test
> use mpi
> integer :: ierr, myrank, a(2), a_loc(2) = 0
> call MPI_Init(ierr)
> call MPI_Comm_rank(MPI_COMM_WORLD,myrank,ierr)
> if (myrank == 0) then
>   a_loc(1) = 1
>   a_loc(2) = 2
> else
>   a_loc(1) = 3
>   a_loc(2) = 4
> endif
> call MPI_Allreduce(a_loc,a,2,MPI_INTEGER,MPI_SUM,MPI_COMM_WORLD,ierr)
> write(*,*) myrank, a(:)
> call MPI_Finalize(ierr)
> end program test
> 
> $ openmpif90 test.f90
> $ openmpirun -np 2 a.out
>0   4   6
>1   4   6
> 
> Now I'd be curious to know why your OpenMPI implementation handles
> MPI_IN_PLACE correctly and not mine!
> 

Re: [OMPI users] MPI_IN_PLACE in a call to MPI_Allreduce in Fortran

[Hugo Gagnon]

Thanks for the input but it still doesn't work for me...  Here's the
version without MPI_IN_PLACE that does work:

program test
use mpi
integer :: ierr, myrank, a(2), a_loc(2) = 0
call MPI_Init(ierr)
call MPI_Comm_rank(MPI_COMM_WORLD,myrank,ierr)
if (myrank == 0) then
  a_loc(1) = 1
  a_loc(2) = 2
else
  a_loc(1) = 3
  a_loc(2) = 4
endif
call MPI_Allreduce(a_loc,a,2,MPI_INTEGER,MPI_SUM,MPI_COMM_WORLD,ierr)
write(*,*) myrank, a(:)
call MPI_Finalize(ierr)
end program test

$ openmpif90 test.f90
$ openmpirun -np 2 a.out
   0   4   6
   1   4   6

Now I'd be curious to know why your OpenMPI implementation handles
MPI_IN_PLACE correctly and not mine!

-- 
  Hugo Gagnon

On Fri, Sep 6, 2013, at 22:37, Tom Rosmond wrote:
> Hello,
> 
> Your syntax defining 'a' is not correct.  This code works correctly.
> 
> program test
> use mpi
> integer :: ierr, myrank, a(2) = 0
> call MPI_Init(ierr)
> call MPI_Comm_rank(MPI_COMM_WORLD,myrank,ierr)
> if (myrank == 0) then
>  a(1) = 1
>  a(2) = 2
> else
>  a(1) = 3
>  a(2) = 4
> endif
> call
> MPI_Allreduce(MPI_IN_PLACE,a,2,MPI_INTEGER,MPI_SUM,MPI_COMM_WORLD,ierr)
> write(*,*) myrank, a(:)
> call MPI_Finalize(ierr)
> end program test
> 
> mpif90 test.f90
> mpirun -np 2 a.out  
>  0   4   6
>  1   4   6
> 
> T. Rosmond
> 
> 
> 
> 
> 
> On Fri, 2013-09-06 at 21:27 -0400, Hugo Gagnon wrote:
> > Hello,
> > 
> > I'm trying to run this bit of code:
> > 
> > program test
> > use mpi
> > integer :: ierr, myrank, a(2) = 0
> > call MPI_Init(ierr)
> > call MPI_Comm_rank(MPI_COMM_WORLD,myrank,ierr)
> > if (myrank == 0) a(1) = 1; a(2) = 2
> > if (myrank == 1) a(1) = 3; a(2) = 4
> > call
> > MPI_Allreduce(MPI_IN_PLACE,a,2,MPI_INTEGER,MPI_SUM,MPI_COMM_WORLD,ierr)
> > write(*,*) myrank, a(:)
> > call MPI_Finalize(ierr)
> > end program test
> > 
> > but the output is not what I'd expect:
> > 
> > $ openmpif90 test.f90
> > $ openmpirun -np 2 a.out
> >0   0   0
> >1   0   0
> > 
> > I tried a version without MPI_IN_PLACE and the call to MPI_Allreduce
> > works fine in that case.  Am I doing something wrong with MPI_IN_PLACE? 
> > I'm using OpenMPI 1.6.5 compiled with gcc 4.8.1 on Mac OS 10.8.
> > 
> > Thanks,
> 
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] MPI_IN_PLACE in a call to MPI_Allreduce in Fortran

[Tom Rosmond]

Hello,

Your syntax defining 'a' is not correct.  This code works correctly.

program test
use mpi
integer :: ierr, myrank, a(2) = 0
call MPI_Init(ierr)
call MPI_Comm_rank(MPI_COMM_WORLD,myrank,ierr)
if (myrank == 0) then
 a(1) = 1
 a(2) = 2
else
 a(1) = 3
 a(2) = 4
endif
call
MPI_Allreduce(MPI_IN_PLACE,a,2,MPI_INTEGER,MPI_SUM,MPI_COMM_WORLD,ierr)
write(*,*) myrank, a(:)
call MPI_Finalize(ierr)
end program test

mpif90 test.f90
mpirun -np 2 a.out  
 0   4   6
 1   4   6

T. Rosmond





On Fri, 2013-09-06 at 21:27 -0400, Hugo Gagnon wrote:
> Hello,
> 
> I'm trying to run this bit of code:
> 
> program test
> use mpi
> integer :: ierr, myrank, a(2) = 0
> call MPI_Init(ierr)
> call MPI_Comm_rank(MPI_COMM_WORLD,myrank,ierr)
> if (myrank == 0) a(1) = 1; a(2) = 2
> if (myrank == 1) a(1) = 3; a(2) = 4
> call
> MPI_Allreduce(MPI_IN_PLACE,a,2,MPI_INTEGER,MPI_SUM,MPI_COMM_WORLD,ierr)
> write(*,*) myrank, a(:)
> call MPI_Finalize(ierr)
> end program test
> 
> but the output is not what I'd expect:
> 
> $ openmpif90 test.f90
> $ openmpirun -np 2 a.out
>0   0   0
>1   0   0
> 
> I tried a version without MPI_IN_PLACE and the call to MPI_Allreduce
> works fine in that case.  Am I doing something wrong with MPI_IN_PLACE? 
> I'm using OpenMPI 1.6.5 compiled with gcc 4.8.1 on Mac OS 10.8.
> 
> Thanks,

[OMPI users] MPI_IN_PLACE in a call to MPI_Allreduce in Fortran

[Hugo Gagnon]

Hello,

I'm trying to run this bit of code:

program test
use mpi
integer :: ierr, myrank, a(2) = 0
call MPI_Init(ierr)
call MPI_Comm_rank(MPI_COMM_WORLD,myrank,ierr)
if (myrank == 0) a(1) = 1; a(2) = 2
if (myrank == 1) a(1) = 3; a(2) = 4
call
MPI_Allreduce(MPI_IN_PLACE,a,2,MPI_INTEGER,MPI_SUM,MPI_COMM_WORLD,ierr)
write(*,*) myrank, a(:)
call MPI_Finalize(ierr)
end program test

but the output is not what I'd expect:

$ openmpif90 test.f90
$ openmpirun -np 2 a.out
   0   0   0
   1   0   0

I tried a version without MPI_IN_PLACE and the call to MPI_Allreduce
works fine in that case.  Am I doing something wrong with MPI_IN_PLACE? 
I'm using OpenMPI 1.6.5 compiled with gcc 4.8.1 on Mac OS 10.8.

Thanks,
-- 
  Hugo Gagnon

Re: [OMPI users] MPI_IN_PLACE not working for Fortran-compiled code linked with mpicc on Mac OS X

[Dave Goodell]

On Jan 4, 2013, at 2:55 AM CST, Torbjörn Björkman wrote:

> It seems that a very old bug (svn.open-mpi.org/trac/ompi/ticket/1982) is 
> playing up when linking fortran code with mpicc on Mac OS X 10.6 and the 
> Macports distribution openmpi @1.6.3_0+gcc44. I got it working by reading up 
> on this discussion thread:
> http://www.open-mpi.org/community/lists/users/2011/11/17862.php
> and applying the fix given there, add '-Wl,-commons,use_dylibs', to the c 
> compiler flags solves the problem. 

I'm not an Open MPI developer (or user, really), but in MPICH we also had to 
ensure that we passed both "-Wl,-commons,use_dylibs" *and* 
"-Wl,-flat_namespace" in the end.  For MPI users that do not use Fortran (and 
therefore don't need common blocks to work correctly between the app and the 
library), we provide a "--enable-two-level-namespace" configure option to allow 
users to generate two-level namespace dylibs instead.  Some combinations of 
third-party dylibs will require two-level namespaced MPI dylibs.

I don't know if Open MPI is using "-Wl,-flat_namespace" or not, but this is 
something else that any investigation should probably check.

For reference on the later MPICH discoveries about dynamically linking common 
symbols on Darwin: http://trac.mpich.org/projects/mpich/ticket/1590

-Dave

[OMPI users] MPI_IN_PLACE not working for Fortran-compiled code linked with mpicc on Mac OS X

[Torbjörn Björkman]

It seems that a very old bug (svn.open-mpi.org/trac/ompi/ticket/1982) is
playing up when linking fortran code with mpicc on Mac OS X 10.6 and the
Macports distribution openmpi @1.6.3_0+gcc44. I got it working by reading
up on this discussion thread:
http://www.open-mpi.org/community/lists/users/2011/11/17862.php
and applying the fix given there, add '-Wl,-commons,use_dylibs', to the c
compiler flags solves the problem.

In addition to that discussion should be mentioned that it is in fact a
link flag, not a compiler issue, so the important thing is to supply the
flag to the linker, not the fortran compiler. The fix would be to add
-Wl,-commons,use_dylibs to the flags supplied by mpicc, but I guess that
there could be reasons for not doing that, but that is for you experts to
say.

I'm posting to the users' list, because I'm unsure of whether this
qualifies as a bug propre, but at least it falls in the category
"undocumented behaviour that totally baffled a fairly experienced user used
to solving his own problems and who is generally not a whiner".

Cheers,
Torbjörn

-- 
==
Torbjörn Björkman

[OMPI users] mpi_in_place not working in mpi_allreduce

[David Zhang]

Dear all:

I ran this simple fortran code and got unexpected result:

!
program reduce
implicit none

include 'mpif.h'

integer :: ierr, rank
real*8 :: send(5)

call mpi_init(ierr)
call mpi_comm_rank(mpi_comm_world,rank,ierr)

send = real(rank)

print *, rank,':',send
call
mpi_allreduce(MPI_IN_PLACE,send,size(send),mpi_real8,mpi_sum,mpi_comm_world,ierr)
print *, rank,'#',send

call mpi_finalize(ierr)

end program reduce

!

When running with 3 processes

mpirun -np 3 reduce

The results I'm expecting is the sum of all 3 vectors, but I got the
unexpected result:

  0 :   0.0.
0.0.0.
   2 :   2.2.
2.2.2.
   1 :   1.1.
1.1.1.
   0 #   0.0.
0.0.0.
   1 #   0.0.
0.0.0.
   2 #   0.0.
0.0.0.


During compilation and running there were no errors or warnings.  I install
openMPI via fink.  I believe somehow fink messed up during installation.
Instead of installing MPI from source (which takes hours on my machine), I
would like to know if there is a better than to find out what the problem
is, so that I could fix my current installation rather than reinstall MPI
from scratch.

-- 
David Zhang
University of California, San Diego

[OMPI users] MPI_IN_PLACE variant of MPI_Reduce

[Martin Siegert]

Hi,

I am confused about the syntax of the "in place" variant of
MPI_Reduce, in particular about the significance of the recvbuf
for the non-root processes. I.e., is the following legal?

buf = (double *)malloc(l*sizeof(double);
random(buf, l); /* fill buf with something */
if (myid == 0) {
   MPI_Reduce(MPI_IN_PLACE, buf, l, MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD);
} else {
   MPI_Reduce(buf, NULL, l, MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD);
}

This appears to violate the standard:
http://www.mpi-forum.org/docs/mpi22-report/node103.htm#Node103
"The routine is called by all group members using the same arguments for
 count, datatype, op, root and  comm. Thus, all processes provide input
 buffers and output buffers of the same length, with elements of the
 same type."

However, this "same length" requirement is already violated at the root
process where I can specify MPI_IN_PLACE. Unfortunately, the standard
says nothing about the other processes in this case. Do I need a valid
receive buffer there?

Cheers,
Martin

-- 
Martin Siegert
Head, Research Computing
WestGrid Site Lead
IT Servicesphone: 778 782-4691
Simon Fraser Universityfax:   778 782-4242
Burnaby, British Columbia  email: sieg...@sfu.ca
Canada  V5A 1S6

Re: [OMPI users] OMPI users] MPI_IN_PLACE in FortranwithMPI_REDUCE / MPI_ALLREDUCE

[Jeff Squyres]

093950 S _MPI_FORTRAN_IN_PLACE
00093954 S _mpi_fortran_in_place
00093958 S _mpi_fortran_in_place_
0009395c S _mpi_fortran_in_place__
00093950 S _MPI_FORTRAN_IN_PLACE
00093954 S _mpi_fortran_in_place
00093958 S _mpi_fortran_in_place_
0009395c S _mpi_fortran_in_place__
e00c D __ZN3MPI8IN_PLACEE
e00c D __ZN3MPI8IN_PLACEE
 U _mpi_fortran_in_place__
 U _mpi_fortran_in_place__

So the __ZN3MPI8IN_PLACEE symbol, that I guess refers to the Fortran  
MPI_IN_PLACE constant is being defined incorrectly in the 1.3.3  
version as a S (symbol in a section other than those above), while  
it should be defined as a D (data section  symbol) as part of an  
"external" common block, as it happens in 1.2.9. So when linking the  
1.3.3 version the MPI_IN_PLACE constant will never have the same  
address as any of the mpi_fortran_in_place variables, but rather its  
own address.


Thanks again for your help,
Ricardo

---
Prof. Ricardo Fonseca

GoLP - Grupo de Lasers e Plasmas
Instituto de Plasmas e Fusão Nuclear
Instituto Superior Técnico
Av. Rovisco Pais
1049-001 Lisboa
Portugal

tel: +351 21 8419202
fax: +351 21 8464455
web: http://cfp.ist.utl.pt/golp/

On Aug 1, 2009, at 17:00 , users-requ...@open-mpi.org wrote:


Message: 2
Date: Sat, 1 Aug 2009 07:44:47 -0400
From: Jeff Squyres 
Subject: Re: [OMPI users] OMPI users] MPI_IN_PLACE in Fortran
withMPI_REDUCE  / MPI_ALLREDUCE
To: Open MPI Users 
Message-ID: 
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes

Hmm.  FWIW, I'm unable to replicate your error.  I tried with the  
OMPI
SVN trunk and a build of the OMPI 1.3.3 tarball using the GNU  
compiler

suite on RHEL4U5.

I've even compiled your sample code with "mpif90" using the "use mpi"
statement -- I did not get an unclassifiable statement.  What version
of Open MPI are you using?  Please sent the info listed here:

http://www.open-mpi.org/community/help/

Can you confirm that you're not accidentally mixing and matching
multiple versions of Open MPI?


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users



--
Jeff Squyres
jsquy...@cisco.com

Re: [OMPI users] OMPI users] MPI_IN_PLACE in Fortran withMPI_REDUCE / MPI_ALLREDUCE

[Ricardo Fonseca]

Hi Jeff

This is a Mac OS X (10.5.7) specific issue, that occurs for all  
versions > 1.2.9 that I've tested (1.3.0 through the 1.4 nightly),  
regardless of what fortran compiler you use (ifort / g95 / gfortran).  
I've been able to replicate this issue on other OS X machines, and I  
am sure that I am using the correct headers / libraries. Version 1.2.9  
is working correctly. Here are some system details:


$ uname -a
Darwin zamblap.epp.ist.utl.pt 9.7.0 Darwin Kernel Version 9.7.0: Tue  
Mar 31 22:52:17 PDT 2009; root:xnu-1228.12.14~1/RELEASE_I386 i386


$ gcc --version
i686-apple-darwin9-gcc-4.0.1 (GCC) 4.0.1 (Apple Inc. build 5493)

$ ld -v
@(#)PROGRAM:ld  PROJECT:ld64-85.2.1

This might be a (again, Mac OS X specific) libtool issue. If you look  
at the name list of the generated .dylib libraries for 1.3.3 you get:


$ nm /opt/openmpi/1.3.3-g95-32/lib/*.dylib | grep -i in_place
000a4d30 S _MPI_FORTRAN_IN_PLACE
000a4d34 S _mpi_fortran_in_place
000a4d38 S _mpi_fortran_in_place_
000a4d3c S _mpi_fortran_in_place__
000a4d30 S _MPI_FORTRAN_IN_PLACE
000a4d34 S _mpi_fortran_in_place
000a4d38 S _mpi_fortran_in_place_
000a4d3c S _mpi_fortran_in_place__
7328 S __ZN3MPI8IN_PLACEE
7328 S __ZN3MPI8IN_PLACEE
 U _mpi_fortran_in_place__
 U _mpi_fortran_in_place__
00036eea D _orte_snapc_base_store_in_place
00036eea D _orte_snapc_base_store_in_place

But for 1.2.9 you get:

$ nm /opt/openmpi/1.2.9-g95-32/lib/*.dylib | grep -i in_place
00093950 S _MPI_FORTRAN_IN_PLACE
00093954 S _mpi_fortran_in_place
00093958 S _mpi_fortran_in_place_
0009395c S _mpi_fortran_in_place__
00093950 S _MPI_FORTRAN_IN_PLACE
00093954 S _mpi_fortran_in_place
00093958 S _mpi_fortran_in_place_
0009395c S _mpi_fortran_in_place__
e00c D __ZN3MPI8IN_PLACEE
e00c D __ZN3MPI8IN_PLACEE
 U _mpi_fortran_in_place__
 U _mpi_fortran_in_place__

So the __ZN3MPI8IN_PLACEE symbol, that I guess refers to the Fortran  
MPI_IN_PLACE constant is being defined incorrectly in the 1.3.3  
version as a S (symbol in a section other than those above), while it  
should be defined as a D (data section  symbol) as part of an  
"external" common block, as it happens in 1.2.9. So when linking the  
1.3.3 version the MPI_IN_PLACE constant will never have the same  
address as any of the mpi_fortran_in_place variables, but rather its  
own address.


Thanks again for your help,
Ricardo

---
Prof. Ricardo Fonseca

GoLP - Grupo de Lasers e Plasmas
Instituto de Plasmas e Fusão Nuclear
Instituto Superior Técnico
Av. Rovisco Pais
1049-001 Lisboa
Portugal

tel: +351 21 8419202
fax: +351 21 8464455
web: http://cfp.ist.utl.pt/golp/

On Aug 1, 2009, at 17:00 , users-requ...@open-mpi.org wrote:


Message: 2
Date: Sat, 1 Aug 2009 07:44:47 -0400
From: Jeff Squyres 
Subject: Re: [OMPI users] OMPI users] MPI_IN_PLACE in Fortran
withMPI_REDUCE  / MPI_ALLREDUCE
To: Open MPI Users 
Message-ID: 
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes

Hmm.  FWIW, I'm unable to replicate your error.  I tried with the OMPI
SVN trunk and a build of the OMPI 1.3.3 tarball using the GNU compiler
suite on RHEL4U5.

I've even compiled your sample code with "mpif90" using the "use mpi"
statement -- I did not get an unclassifiable statement.  What version
of Open MPI are you using?  Please sent the info listed here:

http://www.open-mpi.org/community/help/

Can you confirm that you're not accidentally mixing and matching
multiple versions of Open MPI?

Re: [OMPI users] OMPI users] MPI_IN_PLACE in Fortran withMPI_REDUCE / MPI_ALLREDUCE

[Jeff Squyres]

Hmm.  FWIW, I'm unable to replicate your error.  I tried with the OMPI  
SVN trunk and a build of the OMPI 1.3.3 tarball using the GNU compiler  
suite on RHEL4U5.


I've even compiled your sample code with "mpif90" using the "use mpi"  
statement -- I did not get an unclassifiable statement.  What version  
of Open MPI are you using?  Please sent the info listed here:


http://www.open-mpi.org/community/help/

Can you confirm that you're not accidentally mixing and matching  
multiple versions of Open MPI?




On Jul 30, 2009, at 10:41 AM, Ricardo Fonseca wrote:


(I just realized I had the wrong subject line, here it goes again)

Hi Jeff

Yes, I am using the right one. I've installed the freshly compiled  
openmpi into /opt/openmpi/1.3.3-g95-32. If I edit the mpif.h file by  
hand and put "error!" in the first line I get:


zamblap:sandbox zamb$ edit /opt/openmpi/1.3.3-g95-32/include/mpif.h

zamblap:sandbox zamb$ mpif77 inplace_test.f90

In file mpif.h:1

   Included at inplace_test.f90:7

error!

1

Error: Unclassifiable statement at (1)

(btw, if I use the F90 bindings instead I get a similar problem,  
except the address for the MPI_IN_PLACE fortran constant is slightly  
different from the F77 binding, i.e. instead of 0x50920 I get 0x508e0)


Thanks for your help,

Ricardo


On Jul 29, 2009, at 17:00 , users-requ...@open-mpi.org wrote:


Message: 2
Date: Wed, 29 Jul 2009 07:54:38 -0500
From: Jeff Squyres 
Subject: Re: [OMPI users] OMPI users] MPI_IN_PLACE in Fortran
withMPI_REDUCE  / MPI_ALLREDUCE
To: "Open MPI Users" 
Message-ID: <986510b6-7103-4d7b-b7d6-9d8afdc19...@cisco.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed;  
delsp=yes


Can you confirm that you're using the right mpif.h?

Keep in mind that each MPI implementation's mpif.h is different --
it's a common mistake to assume that the mpif.h from one MPI
implementation should work with another implementation (e.g., someone
copied mpif.h from one MPI to your software's source tree, so the
compiler always finds that one instead of the MPI-implementation-
provided mpif.h.).


On Jul 28, 2009, at 1:17 PM, Ricardo Fonseca wrote:


Hi George

I did some extra digging and found that (for some reason) the
MPI_IN_PLACE parameter is not being recognized as such by
mpi_reduce_f (reduce_f.c:61). I added a couple of printfs:

   printf(" sendbuf = %p \n", sendbuf );

   printf(" MPI_FORTRAN_IN_PLACE = %p \n", &MPI_FORTRAN_IN_PLACE );
   printf(" mpi_fortran_in_place = %p \n", &mpi_fortran_in_place );
   printf(" mpi_fortran_in_place_ = %p \n",  
&mpi_fortran_in_place_ );

   printf(" mpi_fortran_in_place__ = %p \n",
&mpi_fortran_in_place__ );

And this is what I get on node 0:

sendbuf = 0x50920
MPI_FORTRAN_IN_PLACE = 0x17cd30
mpi_fortran_in_place = 0x17cd34
mpi_fortran_in_place_ = 0x17cd38
mpi_fortran_in_place__ = 0x17cd3c

This makes OMPI_F2C_IN_PLACE(sendbuf) fail. If I replace the line:

sendbuf = OMPI_F2C_IN_PLACE(sendbuf);

with:

   if ( sendbuf == 0x50920 ) {
 printf("sendbuf is MPI_IN_PLACE!\n");
 sendbuf = MPI_IN_PLACE;
   }

Then the code works and gives the correct result:

sendbuf is MPI_IN_PLACE!
Result:
3. 3. 3. 3.

So my guess is that somehow the MPI_IN_PLACE constant for fortran is
getting the wrong address. Could this be related to the fortran
compilers I'm using (ifort / g95)?

Ricardo




___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users



--
Jeff Squyres
jsquy...@cisco.com

Re: [OMPI users] OMPI users] MPI_IN_PLACE in Fortran withMPI_REDUCE / MPI_ALLREDUCE

[Ricardo Fonseca]

(I just realized I had the wrong subject line, here it goes again)

Hi Jeff

Yes, I am using the right one. I've installed the freshly compiled  
openmpi into /opt/openmpi/1.3.3-g95-32. If I edit the mpif.h file by  
hand and put "error!" in the first line I get:


zamblap:sandbox zamb$ edit /opt/openmpi/1.3.3-g95-32/include/mpif.h

zamblap:sandbox zamb$ mpif77 inplace_test.f90

In file mpif.h:1

   Included at inplace_test.f90:7

error!

1

Error: Unclassifiable statement at (1)

(btw, if I use the F90 bindings instead I get a similar problem,  
except the address for the MPI_IN_PLACE fortran constant is slightly  
different from the F77 binding, i.e. instead of 0x50920 I get 0x508e0)


Thanks for your help,

Ricardo


On Jul 29, 2009, at 17:00 , users-requ...@open-mpi.org wrote:


Message: 2
Date: Wed, 29 Jul 2009 07:54:38 -0500
From: Jeff Squyres 
Subject: Re: [OMPI users] OMPI users] MPI_IN_PLACE in Fortran
withMPI_REDUCE  / MPI_ALLREDUCE
To: "Open MPI Users" 
Message-ID: <986510b6-7103-4d7b-b7d6-9d8afdc19...@cisco.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed; delsp=yes

Can you confirm that you're using the right mpif.h?

Keep in mind that each MPI implementation's mpif.h is different --
it's a common mistake to assume that the mpif.h from one MPI
implementation should work with another implementation (e.g., someone
copied mpif.h from one MPI to your software's source tree, so the
compiler always finds that one instead of the MPI-implementation-
provided mpif.h.).


On Jul 28, 2009, at 1:17 PM, Ricardo Fonseca wrote:


Hi George

I did some extra digging and found that (for some reason) the
MPI_IN_PLACE parameter is not being recognized as such by
mpi_reduce_f (reduce_f.c:61). I added a couple of printfs:

   printf(" sendbuf = %p \n", sendbuf );

   printf(" MPI_FORTRAN_IN_PLACE = %p \n", &MPI_FORTRAN_IN_PLACE );
   printf(" mpi_fortran_in_place = %p \n", &mpi_fortran_in_place );
   printf(" mpi_fortran_in_place_ = %p \n", &mpi_fortran_in_place_ );
   printf(" mpi_fortran_in_place__ = %p \n",
&mpi_fortran_in_place__ );

And this is what I get on node 0:

sendbuf = 0x50920
MPI_FORTRAN_IN_PLACE = 0x17cd30
mpi_fortran_in_place = 0x17cd34
mpi_fortran_in_place_ = 0x17cd38
mpi_fortran_in_place__ = 0x17cd3c

This makes OMPI_F2C_IN_PLACE(sendbuf) fail. If I replace the line:

sendbuf = OMPI_F2C_IN_PLACE(sendbuf);

with:

   if ( sendbuf == 0x50920 ) {
 printf("sendbuf is MPI_IN_PLACE!\n");
 sendbuf = MPI_IN_PLACE;
   }

Then the code works and gives the correct result:

sendbuf is MPI_IN_PLACE!
Result:
3. 3. 3. 3.

So my guess is that somehow the MPI_IN_PLACE constant for fortran is
getting the wrong address. Could this be related to the fortran
compilers I'm using (ifort / g95)?

Ricardo

Re: [OMPI users] OMPI users] MPI_IN_PLACE in Fortran withMPI_REDUCE / MPI_ALLREDUCE

[Jeff Squyres]

Can you confirm that you're using the right mpif.h?

Keep in mind that each MPI implementation's mpif.h is different --  
it's a common mistake to assume that the mpif.h from one MPI  
implementation should work with another implementation (e.g., someone  
copied mpif.h from one MPI to your software's source tree, so the  
compiler always finds that one instead of the MPI-implementation- 
provided mpif.h.).



On Jul 28, 2009, at 1:17 PM, Ricardo Fonseca wrote:


Hi George

I did some extra digging and found that (for some reason) the  
MPI_IN_PLACE parameter is not being recognized as such by  
mpi_reduce_f (reduce_f.c:61). I added a couple of printfs:


printf(" sendbuf = %p \n", sendbuf );

printf(" MPI_FORTRAN_IN_PLACE = %p \n", &MPI_FORTRAN_IN_PLACE );
printf(" mpi_fortran_in_place = %p \n", &mpi_fortran_in_place );
printf(" mpi_fortran_in_place_ = %p \n", &mpi_fortran_in_place_ );
printf(" mpi_fortran_in_place__ = %p \n",  
&mpi_fortran_in_place__ );


And this is what I get on node 0:

 sendbuf = 0x50920
 MPI_FORTRAN_IN_PLACE = 0x17cd30
 mpi_fortran_in_place = 0x17cd34
 mpi_fortran_in_place_ = 0x17cd38
 mpi_fortran_in_place__ = 0x17cd3c

This makes OMPI_F2C_IN_PLACE(sendbuf) fail. If I replace the line:

sendbuf = OMPI_F2C_IN_PLACE(sendbuf);

with:

if ( sendbuf == 0x50920 ) {
  printf("sendbuf is MPI_IN_PLACE!\n");
  sendbuf = MPI_IN_PLACE;
}

Then the code works and gives the correct result:

sendbuf is MPI_IN_PLACE!
 Result:
 3. 3. 3. 3.

So my guess is that somehow the MPI_IN_PLACE constant for fortran is  
getting the wrong address. Could this be related to the fortran  
compilers I'm using (ifort / g95)?


Ricardo

---
Prof. Ricardo Fonseca

GoLP - Grupo de Lasers e Plasmas
Instituto de Plasmas e Fusão Nuclear
Instituto Superior Técnico
Av. Rovisco Pais
1049-001 Lisboa
Portugal

tel: +351 21 8419202
fax: +351 21 8464455
web: http://cfp.ist.utl.pt/golp/

On Jul 28, 2009, at 17:00 , users-requ...@open-mpi.org wrote:


Message: 1
Date: Tue, 28 Jul 2009 11:16:34 -0400
From: George Bosilca 
Subject: Re: [OMPI users] OMPI users] MPI_IN_PLACE in Fortran with
MPI_REDUCE / MPI_ALLREDUCE
To: Open MPI Users 
Message-ID: 
Content-Type: text/plain; charset=ISO-8859-1; format=flowed;  
delsp=yes


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users



--
Jeff Squyres
jsquy...@cisco.com

Re: [OMPI users] OMPI users] MPI_IN_PLACE in Fortran with MPI_REDUCE / MPI_ALLREDUCE

[Ricardo Fonseca]

Hi George

I did some extra digging and found that (for some reason) the  
MPI_IN_PLACE parameter is not being recognized as such by mpi_reduce_f  
(reduce_f.c:61). I added a couple of printfs:


printf(" sendbuf = %p \n", sendbuf );

printf(" MPI_FORTRAN_IN_PLACE = %p \n", &MPI_FORTRAN_IN_PLACE );
printf(" mpi_fortran_in_place = %p \n", &mpi_fortran_in_place );
printf(" mpi_fortran_in_place_ = %p \n", &mpi_fortran_in_place_ );
printf(" mpi_fortran_in_place__ = %p \n",  
&mpi_fortran_in_place__ );


And this is what I get on node 0:

 sendbuf = 0x50920
 MPI_FORTRAN_IN_PLACE = 0x17cd30
 mpi_fortran_in_place = 0x17cd34
 mpi_fortran_in_place_ = 0x17cd38
 mpi_fortran_in_place__ = 0x17cd3c

This makes OMPI_F2C_IN_PLACE(sendbuf) fail. If I replace the line:

sendbuf = OMPI_F2C_IN_PLACE(sendbuf);

with:

if ( sendbuf == 0x50920 ) {
  printf("sendbuf is MPI_IN_PLACE!\n");
  sendbuf = MPI_IN_PLACE;
}

Then the code works and gives the correct result:

sendbuf is MPI_IN_PLACE!
 Result:
 3. 3. 3. 3.

So my guess is that somehow the MPI_IN_PLACE constant for fortran is  
getting the wrong address. Could this be related to the fortran  
compilers I'm using (ifort / g95)?


Ricardo

---
Prof. Ricardo Fonseca

GoLP - Grupo de Lasers e Plasmas
Instituto de Plasmas e Fusão Nuclear
Instituto Superior Técnico
Av. Rovisco Pais
1049-001 Lisboa
Portugal

tel: +351 21 8419202
fax: +351 21 8464455
web: http://cfp.ist.utl.pt/golp/

On Jul 28, 2009, at 17:00 , users-requ...@open-mpi.org wrote:


Message: 1
Date: Tue, 28 Jul 2009 11:16:34 -0400
From: George Bosilca 
Subject: Re: [OMPI users] OMPI users] MPI_IN_PLACE in Fortran with
MPI_REDUCE / MPI_ALLREDUCE
To: Open MPI Users 
Message-ID: 
Content-Type: text/plain; charset=ISO-8859-1; format=flowed; delsp=yes

Re: [OMPI users] OMPI users] MPI_IN_PLACE in Fortran with MPI_REDUCE / MPI_ALLREDUCE

[Ricardo Fonseca]

Hi George

I don't think this is a library mismatch. I just followed your  
instructions and got:


$ otool -L a.out
a.out:
	/opt/openmpi/1.3.3-g95-32/lib/libmpi_f77.0.dylib (compatibility  
version 1.0.0, current version 1.0.0)
	/opt/openmpi/1.3.3-g95-32/lib/libmpi.0.dylib (compatibility version  
1.0.0, current version 1.0.0)
	/opt/openmpi/1.3.3-g95-32/lib/libopen-rte.0.dylib (compatibility  
version 1.0.0, current version 1.0.0)
	/opt/openmpi/1.3.3-g95-32/lib/libopen-pal.0.dylib (compatibility  
version 1.0.0, current version 1.0.0)
	/usr/lib/libutil.dylib (compatibility version 1.0.0, current version  
1.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current  
version 111.1.4)
	/usr/lib/libmx.A.dylib (compatibility version 1.0.0, current version  
292.4.0)


My configure line is just:

./configure --prefix=/opt/openmpi/1.3.3-g95-32 \
   --enable-mpirun-prefix-by-default \
F77=g95 FC=g95

Ricardo


On Jul 28, 2009, at 17:00 , users-requ...@open-mpi.org wrote:


Ricardo,

I checked on Linux and on Mac OS X 10.5.7 with the fortran compilers
from (hpc.sourceforge.net) and I get the correct answer. As you only
report problems on Mac OS X I wonder if the real source of the problem
is not coming from a library mismatch. As you know, Open MPI is
bundled in Leopard. We had problems in the past if the user install
their own version, if the paths are not correctly set.

Let's try to understand what the problem is on your system. First
please compile your version of Open MPI by adding --enable-mpirun-
prefix-by-default to the configure line. Then once everything is
installed, compile a simple application (inplace.F is a good example),
do a "otool -L a.out" and send out the output.

  Thanks,
george.

Re: [OMPI users] OMPI users] MPI_IN_PLACE in Fortran with MPI_REDUCE / MPI_ALLREDUCE

[George Bosilca]

Ricardo,

I checked on Linux and on Mac OS X 10.5.7 with the fortran compilers  
from (hpc.sourceforge.net) and I get the correct answer. As you only  
report problems on Mac OS X I wonder if the real source of the problem  
is not coming from a library mismatch. As you know, Open MPI is  
bundled in Leopard. We had problems in the past if the user install  
their own version, if the paths are not correctly set.


Let's try to understand what the problem is on your system. First  
please compile your version of Open MPI by adding --enable-mpirun- 
prefix-by-default to the configure line. Then once everything is  
installed, compile a simple application (inplace.F is a good example),  
do a "otool -L a.out" and send out the output.


  Thanks,
george.

On Jul 28, 2009, at 10:15 , Ricardo Fonseca wrote:

Hi George

Thanks for the input. This might be an OS specific problem: I'm  
running Mac OS X 10.5.7, and this problem appears in openmpi  
versions 1.3.2, 1.3.3 and 1.4a1r21734, using Intel Ifort Compiler  
11.0 and 11.1 (and also g95 + 1.3.2). I haven't tried older  
versions. Also, I'm running on a single machine:


zamb$ mpif90 inplace_test.f90
zamb$ mpirun -np 2 ./a.out
 Result:
   2.00   2.00   2.00   2.00

I've tryed the same code under Linux (openmpi-1.3.3 + gfortran) and  
it works (and also other platforms / MPIs ).


Can you think of some --mca options I should try? (or any other  
ideas...)


Cheers,
Ricardo
---
Prof. Ricardo Fonseca

GoLP - Grupo de Lasers e Plasmas
Instituto de Plasmas e Fusão Nuclear
Instituto Superior Técnico
Av. Rovisco Pais
1049-001 Lisboa
Portugal

tel: +351 21 8419202
fax: +351 21 8464455
web: http://cfp.ist.utl.pt/golp/

On Jul 28, 2009, at 4:24 , users-requ...@open-mpi.org wrote:


Message: 4
Date: Mon, 27 Jul 2009 17:13:23 -0400
From: George Bosilca 
Subject: Re: [OMPI users] MPI_IN_PLACE in Fortran with MPI_REDUCE /
MPI_ALLREDUCE
To: Open MPI Users 
Message-ID: <966a51c3-a15f-425b-a6b0-81221033c...@eecs.utk.edu>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes

Ricardo,

I can't reproduce your problem with the latest version (trunk  
r21734).
If I run the provided program on two nodes I get the following  
answer.

[***]$ mpif77 inplace.f -o inplace -g
[***]$ mpirun -bynode -np 2 ./inplace
 Result:
   3.000   3.000   3.000   3.000

This seems correct and in sync with the C answer.

  george.


On Jul 27, 2009, at 09:42 , Ricardo Fonseca wrote:


program inplace

 use mpi

 implicit none

 integer :: ierr, rank, rsize, bsize
 real, dimension( 2, 2 ) :: buffer, out
 integer :: rc

 call MPI_INIT(ierr)
 call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierr)
 call MPI_COMM_SIZE(MPI_COMM_WORLD, rsize, ierr)

 buffer = rank + 1
 bsize = size(buffer,1) * size(buffer,2)

 if ( rank == 0 ) then
   call mpi_reduce( MPI_IN_PLACE, buffer, bsize, MPI_REAL, MPI_SUM,
0, MPI_COMM_WORLD, ierr )
 else
   call mpi_reduce( buffer,   0,  bsize, MPI_REAL, MPI_SUM,
0, MPI_COMM_WORLD, ierr )
 endif

 ! use allreduce instead
 ! call mpi_allreduce( MPI_IN_PLACE, buffer, bsize, MPI_REAL,
MPI_SUM, MPI_COMM_WORLD, ierr )

 if ( rank == 0 ) then
   print *, 'Result:'
   print *, buffer
 endif

 rc = 0
 call mpi_finalize( rc )

end program





___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] OMPI users] MPI_IN_PLACE in Fortran with MPI_REDUCE / MPI_ALLREDUCE

[Ricardo Fonseca]

Hi George

Thanks for the input. This might be an OS specific problem: I'm  
running Mac OS X 10.5.7, and this problem appears in openmpi versions  
1.3.2, 1.3.3 and 1.4a1r21734, using Intel Ifort Compiler 11.0 and 11.1  
(and also g95 + 1.3.2). I haven't tried older versions. Also, I'm  
running on a single machine:


zamb$ mpif90 inplace_test.f90
zamb$ mpirun -np 2 ./a.out
 Result:
   2.00   2.00   2.00   2.00

I've tryed the same code under Linux (openmpi-1.3.3 + gfortran) and it  
works (and also other platforms / MPIs ).


Can you think of some --mca options I should try? (or any other  
ideas...)


Cheers,
Ricardo
---
Prof. Ricardo Fonseca

GoLP - Grupo de Lasers e Plasmas
Instituto de Plasmas e Fusão Nuclear
Instituto Superior Técnico
Av. Rovisco Pais
1049-001 Lisboa
Portugal

tel: +351 21 8419202
fax: +351 21 8464455
web: http://cfp.ist.utl.pt/golp/

On Jul 28, 2009, at 4:24 , users-requ...@open-mpi.org wrote:


Message: 4
Date: Mon, 27 Jul 2009 17:13:23 -0400
From: George Bosilca 
Subject: Re: [OMPI users] MPI_IN_PLACE in Fortran with MPI_REDUCE /
MPI_ALLREDUCE
To: Open MPI Users 
Message-ID: <966a51c3-a15f-425b-a6b0-81221033c...@eecs.utk.edu>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes

Ricardo,

I can't reproduce your problem with the latest version (trunk r21734).
If I run the provided program on two nodes I get the following answer.
[***]$ mpif77 inplace.f -o inplace -g
[***]$ mpirun -bynode -np 2 ./inplace
 Result:
   3.000   3.000   3.000   3.000

This seems correct and in sync with the C answer.

  george.


On Jul 27, 2009, at 09:42 , Ricardo Fonseca wrote:


program inplace

 use mpi

 implicit none

 integer :: ierr, rank, rsize, bsize
 real, dimension( 2, 2 ) :: buffer, out
 integer :: rc

 call MPI_INIT(ierr)
 call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierr)
 call MPI_COMM_SIZE(MPI_COMM_WORLD, rsize, ierr)

 buffer = rank + 1
 bsize = size(buffer,1) * size(buffer,2)

 if ( rank == 0 ) then
   call mpi_reduce( MPI_IN_PLACE, buffer, bsize, MPI_REAL, MPI_SUM,
0, MPI_COMM_WORLD, ierr )
 else
   call mpi_reduce( buffer,   0,  bsize, MPI_REAL, MPI_SUM,
0, MPI_COMM_WORLD, ierr )
 endif

 ! use allreduce instead
 ! call mpi_allreduce( MPI_IN_PLACE, buffer, bsize, MPI_REAL,
MPI_SUM, MPI_COMM_WORLD, ierr )

 if ( rank == 0 ) then
   print *, 'Result:'
   print *, buffer
 endif

 rc = 0
 call mpi_finalize( rc )

end program

Re: [OMPI users] MPI_IN_PLACE in Fortran with MPI_REDUCE / MPI_ALLREDUCE

[George Bosilca]

Ricardo,

I can't reproduce your problem with the latest version (trunk r21734).  
If I run the provided program on two nodes I get the following answer.

[***]$ mpif77 inplace.f -o inplace -g
[***]$ mpirun -bynode -np 2 ./inplace
 Result:
   3.000   3.000   3.000   3.000

This seems correct and in sync with the C answer.

  george.


On Jul 27, 2009, at 09:42 , Ricardo Fonseca wrote:


program inplace

  use mpi

  implicit none

  integer :: ierr, rank, rsize, bsize
  real, dimension( 2, 2 ) :: buffer, out
  integer :: rc

  call MPI_INIT(ierr)
  call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierr)
  call MPI_COMM_SIZE(MPI_COMM_WORLD, rsize, ierr)

  buffer = rank + 1
  bsize = size(buffer,1) * size(buffer,2)

  if ( rank == 0 ) then
call mpi_reduce( MPI_IN_PLACE, buffer, bsize, MPI_REAL, MPI_SUM,  
0, MPI_COMM_WORLD, ierr )

  else
call mpi_reduce( buffer,   0,  bsize, MPI_REAL, MPI_SUM,  
0, MPI_COMM_WORLD, ierr )

  endif

  ! use allreduce instead
  ! call mpi_allreduce( MPI_IN_PLACE, buffer, bsize, MPI_REAL,  
MPI_SUM, MPI_COMM_WORLD, ierr )


  if ( rank == 0 ) then
print *, 'Result:'
print *, buffer
  endif

  rc = 0
  call mpi_finalize( rc )

end program

[OMPI users] MPI_IN_PLACE in Fortran with MPI_REDUCE / MPI_ALLREDUCE

[Ricardo Fonseca]

Hi guys

I'm having a little trouble using MPI_IN_PLACE with  MPI_REDUCE /  
MPI_ALLREDUCE in Fortran. If I try to MPI_IN_PLACE with C bindings it  
works fine running on 2 nodes:


Result:
3.00 3.00 3.00 3.00

Regardless of using MPI_Reduce or MPI_Allreduce. However, this fails  
for the fortran version:


 Result:
   2.00   2.00   2.00   2.00

Apparently, MPI is ignoring values from the root node. Here's the  
source for the Fortran code:


---

program inplace

  use mpi

  implicit none

  integer :: ierr, rank, rsize, bsize
  real, dimension( 2, 2 ) :: buffer, out
  integer :: rc

  call MPI_INIT(ierr)
  call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierr)
  call MPI_COMM_SIZE(MPI_COMM_WORLD, rsize, ierr)

  buffer = rank + 1
  bsize = size(buffer,1) * size(buffer,2)

  if ( rank == 0 ) then
call mpi_reduce( MPI_IN_PLACE, buffer, bsize, MPI_REAL, MPI_SUM,  
0, MPI_COMM_WORLD, ierr )

  else
call mpi_reduce( buffer,   0,  bsize, MPI_REAL, MPI_SUM,  
0, MPI_COMM_WORLD, ierr )

  endif

  ! use allreduce instead
  ! call mpi_allreduce( MPI_IN_PLACE, buffer, bsize, MPI_REAL,  
MPI_SUM, MPI_COMM_WORLD, ierr )


  if ( rank == 0 ) then
print *, 'Result:'
print *, buffer
  endif

  rc = 0
  call mpi_finalize( rc )

end program

---

Any ideas?

Cheers,
Ricardo




---
Prof. Ricardo Fonseca

GoLP - Grupo de Lasers e Plasmas
Instituto de Plasmas e Fusão Nuclear
Instituto Superior Técnico
Av. Rovisco Pais
1049-001 Lisboa
Portugal

tel: +351 21 8419202
fax: +351 21 8464455
web: http://cfp.ist.utl.pt/golp/

Re: [OMPI users] MPI_IN_PLACE

[Graham E Fagg]

Hi David
 yep they do (reduce the values to a single location) and in a tree 
topology it would look something like the following:



proc   3  4  5   6
local values   30 40 50  60
partial sums   -  -  -   -


proc1  2
local values10 20
partial sums10+30+40 (80)  20+50+60 (130)


proc 0
local values 0
partial sums 0+80+130 = 210

result at root (0)   210

For in place the value (0) at the root would be in its Send buffer

The MPI_IN_PLACE option is more important for allreduce as it saves lots 
of potential local data movement.


I suggest that you look on the web for a MPI primer or tutorial to gain 
more understanding.


G.


On Mon, 6 Mar 2006, Xiaoning (David) Yang wrote:


I'm not quite sure how collective computation calls work. For example, for
an MPI_REDUCE with MPI_SUM, do all the processes collect values from all the
processes and calculate the sum and put result in recvbuf on root? Sounds
strange.

David

* Correspondence *




From: Jeff Squyres 
Reply-To: Open MPI Users 
Date: Mon, 6 Mar 2006 13:22:23 -0500
To: Open MPI Users 
Subject: Re: [OMPI users] MPI_IN_PLACE

Generally, yes.  There are some corner cases where we have to
allocate additional buffers, but that's the main/easiest benefit to
describe.  :-)


On Mar 6, 2006, at 11:21 AM, Xiaoning (David) Yang wrote:


Jeff,

Thank you for the reply. In other words, MPI_IN_PLACE only
eliminates data
movement on root, right?

David

* Correspondence *




From: Jeff Squyres 
Reply-To: Open MPI Users 
Date: Fri, 3 Mar 2006 19:18:52 -0500
To: Open MPI Users 
Subject: Re: [OMPI users] MPI_IN_PLACE

On Mar 3, 2006, at 6:42 PM, Xiaoning (David) Yang wrote:


  call MPI_REDUCE(mypi,pi,1,MPI_DOUBLE_PRECISION,MPI_SUM,0,
 &  MPI_COMM_WORLD,ierr)

Can I use MPI_IN_PLACE in the MPI_REDUCE call? If I can, how?
Thanks for any help!


MPI_IN_PLACE is an MPI-2 construct, and is defined in the MPI-2
standard.  Its use in MPI_REDUCE is defined in section 7.3.3:

http://www.mpi-forum.org/docs/mpi-20-html/node150.htm#Node150

It says:

"The ``in place'' option for intracommunicators is specified by
passing the value MPI_IN_PLACE to the argument sendbuf at the root.
In such case, the input data is taken at the root from the receive
buffer, where it will be replaced by the output data."

In the simple pi example program, it doesn't make much sense to use
MPI_IN_PLACE except as an example to see how it is used (i.e., it
won't gain much in terms of efficiency because you're only dealing
with a single MPI_DOUBLE_PRECISION).  But you would want to put an
"if" statement around the call to MPI_REDUCE and pass MPI_IN_PLACE as
the first argument, and mypi as the second argument for the root.
For all other processes, use the same MPI_REDUCE that you're using
now.

--
{+} Jeff Squyres
{+} The Open MPI Project
{+} http://www.open-mpi.org/


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users



___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users



--
{+} Jeff Squyres
{+} The Open MPI Project
{+} http://www.open-mpi.org/


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users



___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users




Thanks,
Graham.
--
Dr Graham E. Fagg   | Distributed, Parallel and Meta-Computing
Innovative Computing Lab. PVM3.4, HARNESS, FT-MPI, SNIPE & Open MPI
Computer Science Dept   | Suite 203, 1122 Volunteer Blvd,
University of Tennessee | Knoxville, Tennessee, USA. TN 37996-3450
Email: f...@cs.utk.edu  | Phone:+1(865)974-5790 | Fax:+1(865)974-8296
Broken complex systems are always derived from working simple systems
--

Re: [OMPI users] MPI_IN_PLACE

[Jeff Squyres]

On Mar 6, 2006, at 3:38 PM, Xiaoning (David) Yang wrote:

I'm not quite sure how collective computation calls work. For  
example, for an MPI_REDUCE with MPI_SUM, do all the processes  
collect values from all the processes and calculate the sum and put  
result in recvbuf on root? Sounds strange.


The implementation of how MPI_REDUCE works is not specified by the  
standard.  Only the semantics are specified (when MPI_REDUCE with  
MPI_SUM returns, the root's recvbuf has the sum of all the data from  
the non-root processes).  As such, an MPI implementation is free to  
implement it however it wishes.


There has been a considerable amount of research on how to optimize  
collective algorithm implementations in MPI over the past ~5 years  
(and outside of MPI for 20+ years before that).


--
{+} Jeff Squyres
{+} The Open MPI Project
{+} http://www.open-mpi.org/

Re: [OMPI users] MPI_IN_PLACE

[Xiaoning (David) Yang]

I'm not quite sure how collective computation calls work. For example, for
an MPI_REDUCE with MPI_SUM, do all the processes collect values from all the
processes and calculate the sum and put result in recvbuf on root? Sounds
strange.

David

* Correspondence *



> From: Jeff Squyres 
> Reply-To: Open MPI Users 
> Date: Mon, 6 Mar 2006 13:22:23 -0500
> To: Open MPI Users 
> Subject: Re: [OMPI users] MPI_IN_PLACE
> 
> Generally, yes.  There are some corner cases where we have to
> allocate additional buffers, but that's the main/easiest benefit to
> describe.  :-)
> 
> 
> On Mar 6, 2006, at 11:21 AM, Xiaoning (David) Yang wrote:
> 
>> Jeff,
>> 
>> Thank you for the reply. In other words, MPI_IN_PLACE only
>> eliminates data
>> movement on root, right?
>> 
>> David
>> 
>> * Correspondence *
>> 
>> 
>> 
>>> From: Jeff Squyres 
>>> Reply-To: Open MPI Users 
>>> Date: Fri, 3 Mar 2006 19:18:52 -0500
>>> To: Open MPI Users 
>>> Subject: Re: [OMPI users] MPI_IN_PLACE
>>> 
>>> On Mar 3, 2006, at 6:42 PM, Xiaoning (David) Yang wrote:
>>> 
>>>>   call MPI_REDUCE(mypi,pi,1,MPI_DOUBLE_PRECISION,MPI_SUM,0,
>>>>  &  MPI_COMM_WORLD,ierr)
>>>> 
>>>> Can I use MPI_IN_PLACE in the MPI_REDUCE call? If I can, how?
>>>> Thanks for any help!
>>> 
>>> MPI_IN_PLACE is an MPI-2 construct, and is defined in the MPI-2
>>> standard.  Its use in MPI_REDUCE is defined in section 7.3.3:
>>> 
>>> http://www.mpi-forum.org/docs/mpi-20-html/node150.htm#Node150
>>> 
>>> It says:
>>> 
>>> "The ``in place'' option for intracommunicators is specified by
>>> passing the value MPI_IN_PLACE to the argument sendbuf at the root.
>>> In such case, the input data is taken at the root from the receive
>>> buffer, where it will be replaced by the output data."
>>> 
>>> In the simple pi example program, it doesn't make much sense to use
>>> MPI_IN_PLACE except as an example to see how it is used (i.e., it
>>> won't gain much in terms of efficiency because you're only dealing
>>> with a single MPI_DOUBLE_PRECISION).  But you would want to put an
>>> "if" statement around the call to MPI_REDUCE and pass MPI_IN_PLACE as
>>> the first argument, and mypi as the second argument for the root.
>>> For all other processes, use the same MPI_REDUCE that you're using
>>> now.
>>> 
>>> -- 
>>> {+} Jeff Squyres
>>> {+} The Open MPI Project
>>> {+} http://www.open-mpi.org/
>>> 
>>> 
>>> ___
>>> users mailing list
>>> us...@open-mpi.org
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>> 
>> 
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 
> 
> -- 
> {+} Jeff Squyres
> {+} The Open MPI Project
> {+} http://www.open-mpi.org/
> 
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] MPI_IN_PLACE

[Jeff Squyres]

Generally, yes.  There are some corner cases where we have to  
allocate additional buffers, but that's the main/easiest benefit to  
describe.  :-)



On Mar 6, 2006, at 11:21 AM, Xiaoning (David) Yang wrote:


Jeff,

Thank you for the reply. In other words, MPI_IN_PLACE only  
eliminates data

movement on root, right?

David

* Correspondence *




From: Jeff Squyres 
Reply-To: Open MPI Users 
Date: Fri, 3 Mar 2006 19:18:52 -0500
To: Open MPI Users 
Subject: Re: [OMPI users] MPI_IN_PLACE

On Mar 3, 2006, at 6:42 PM, Xiaoning (David) Yang wrote:


  call MPI_REDUCE(mypi,pi,1,MPI_DOUBLE_PRECISION,MPI_SUM,0,
 &  MPI_COMM_WORLD,ierr)

Can I use MPI_IN_PLACE in the MPI_REDUCE call? If I can, how?
Thanks for any help!


MPI_IN_PLACE is an MPI-2 construct, and is defined in the MPI-2
standard.  Its use in MPI_REDUCE is defined in section 7.3.3:

http://www.mpi-forum.org/docs/mpi-20-html/node150.htm#Node150

It says:

"The ``in place'' option for intracommunicators is specified by
passing the value MPI_IN_PLACE to the argument sendbuf at the root.
In such case, the input data is taken at the root from the receive
buffer, where it will be replaced by the output data."

In the simple pi example program, it doesn't make much sense to use
MPI_IN_PLACE except as an example to see how it is used (i.e., it
won't gain much in terms of efficiency because you're only dealing
with a single MPI_DOUBLE_PRECISION).  But you would want to put an
"if" statement around the call to MPI_REDUCE and pass MPI_IN_PLACE as
the first argument, and mypi as the second argument for the root.
For all other processes, use the same MPI_REDUCE that you're using  
now.


--
{+} Jeff Squyres
{+} The Open MPI Project
{+} http://www.open-mpi.org/


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users



___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users



--
{+} Jeff Squyres
{+} The Open MPI Project
{+} http://www.open-mpi.org/

Re: [OMPI users] MPI_IN_PLACE

[Xiaoning (David) Yang]

Jeff,

Thank you for the reply. In other words, MPI_IN_PLACE only eliminates data
movement on root, right?

David

* Correspondence *



> From: Jeff Squyres 
> Reply-To: Open MPI Users 
> Date: Fri, 3 Mar 2006 19:18:52 -0500
> To: Open MPI Users 
> Subject: Re: [OMPI users] MPI_IN_PLACE
> 
> On Mar 3, 2006, at 6:42 PM, Xiaoning (David) Yang wrote:
> 
>>   call MPI_REDUCE(mypi,pi,1,MPI_DOUBLE_PRECISION,MPI_SUM,0,
>>  &  MPI_COMM_WORLD,ierr)
>> 
>> Can I use MPI_IN_PLACE in the MPI_REDUCE call? If I can, how?
>> Thanks for any help!
> 
> MPI_IN_PLACE is an MPI-2 construct, and is defined in the MPI-2
> standard.  Its use in MPI_REDUCE is defined in section 7.3.3:
> 
> http://www.mpi-forum.org/docs/mpi-20-html/node150.htm#Node150
> 
> It says:
> 
> "The ``in place'' option for intracommunicators is specified by
> passing the value MPI_IN_PLACE to the argument sendbuf at the root.
> In such case, the input data is taken at the root from the receive
> buffer, where it will be replaced by the output data."
> 
> In the simple pi example program, it doesn't make much sense to use
> MPI_IN_PLACE except as an example to see how it is used (i.e., it
> won't gain much in terms of efficiency because you're only dealing
> with a single MPI_DOUBLE_PRECISION).  But you would want to put an
> "if" statement around the call to MPI_REDUCE and pass MPI_IN_PLACE as
> the first argument, and mypi as the second argument for the root.
> For all other processes, use the same MPI_REDUCE that you're using now.
> 
> -- 
> {+} Jeff Squyres
> {+} The Open MPI Project
> {+} http://www.open-mpi.org/
> 
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] MPI_IN_PLACE

[Jeff Squyres]

On Mar 3, 2006, at 6:42 PM, Xiaoning (David) Yang wrote:


  call MPI_REDUCE(mypi,pi,1,MPI_DOUBLE_PRECISION,MPI_SUM,0,
 &  MPI_COMM_WORLD,ierr)

Can I use MPI_IN_PLACE in the MPI_REDUCE call? If I can, how?  
Thanks for any help!


MPI_IN_PLACE is an MPI-2 construct, and is defined in the MPI-2  
standard.  Its use in MPI_REDUCE is defined in section 7.3.3:


http://www.mpi-forum.org/docs/mpi-20-html/node150.htm#Node150

It says:

"The ``in place'' option for intracommunicators is specified by  
passing the value MPI_IN_PLACE to the argument sendbuf at the root.  
In such case, the input data is taken at the root from the receive  
buffer, where it will be replaced by the output data."


In the simple pi example program, it doesn't make much sense to use  
MPI_IN_PLACE except as an example to see how it is used (i.e., it  
won't gain much in terms of efficiency because you're only dealing  
with a single MPI_DOUBLE_PRECISION).  But you would want to put an  
"if" statement around the call to MPI_REDUCE and pass MPI_IN_PLACE as  
the first argument, and mypi as the second argument for the root.   
For all other processes, use the same MPI_REDUCE that you're using now.


--
{+} Jeff Squyres
{+} The Open MPI Project
{+} http://www.open-mpi.org/

Re: [OMPI users] MPI_IN_PLACE

[Xiaoning (David) Yang]

Jeff,

Thanks.

Here is a simple code from the book "Using MPI" that I want to modify to use
MPI_IN_PLACE.

  program main
  include "mpif.h"
  double precision  PI25DT
  parameter(PI25DT = 3.141592653589793238462643d0)
  double precision  mypi, pi, h, sum, x, f, a
  double precision starttime, endtime
  integer n, myid, numprocs, i, ierr
c function to integrate
  f(a) = 4.d0 / (1.d0 + a*a)

  call MPI_INIT(ierr)
  call MPI_COMM_RANK(MPI_COMM_WORLD, myid, ierr)
  call MPI_COMM_SIZE(MPI_COMM_WORLD, numprocs, ierr)

 10   if ( myid .eq. 0 ) then
 print *, 'Enter the number of intervals: (0 quits) '
 read(*,*) n
  endif
  starttime = MPI_WTIME()
c broadcast n
  call MPI_BCAST(n,1,MPI_INTEGER,0,MPI_COMM_WORLD,ierr)
c check for quit signal
  if ( n .le. 0 ) goto 30
c calculate the interval size
  h = 1.0d0/n
  sum  = 0.0d0
  do 20 i = myid+1, n, numprocs
 x = h * (dble(i) - 0.5d0)
 sum = sum + f(x)
 20   continue
  mypi = h * sum
c collect all the partial sums
  call MPI_REDUCE(mypi,pi,1,MPI_DOUBLE_PRECISION,MPI_SUM,0,
 &  MPI_COMM_WORLD,ierr)
c node 0 prints the answer.
  endtime = MPI_WTIME()
  if (myid .eq. 0) then
 print *, 'pi is ', pi, ' Error is', abs(pi - PI25DT)
 print *, 'time is ', endtime-starttime, ' seconds'
  endif
  goto 10
 30   call MPI_FINALIZE(ierr)
  stop
  end


Can I use MPI_IN_PLACE in the MPI_REDUCE call? If I can, how? Thanks for any
help!

David

* Correspondence *



> From: Jeff Squyres 
> Reply-To: Open MPI Users 
> Date: Fri, 3 Mar 2006 18:04:28 -0500
> To: Open MPI Users 
> Subject: Re: [OMPI users] MPI_IN_PLACE
> 
> On Mar 3, 2006, at 4:40 PM, Xiaoning (David) Yang wrote:
> 
>> Does Open MPI supports MPI_IN_PLACE? Thanks.
> 
> Yes.
> 
> -- 
> {+} Jeff Squyres
> {+} The Open MPI Project
> {+} http://www.open-mpi.org/
> 
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] MPI_IN_PLACE

[Jeff Squyres]

On Mar 3, 2006, at 4:40 PM, Xiaoning (David) Yang wrote:


Does Open MPI supports MPI_IN_PLACE? Thanks.


Yes.

--
{+} Jeff Squyres
{+} The Open MPI Project
{+} http://www.open-mpi.org/

[OMPI users] MPI_IN_PLACE

[Xiaoning (David) Yang]

Does Open MPI supports MPI_IN_PLACE? Thanks.

David

* Correspondence *