Re: [OMPI users] Is building with "--enable-mpi-thread-multiple" recommended?

[Paul Kapinos]

Hi,

On 03/03/17 12:41, Mark Dixon wrote:

Your 20% memory bandwidth performance hit on 2.x and the OPA problem are
concerning - will look at that. Are there tickets open for them?


OPA performance issue on CP2K (15x slowdown) :
https://www.mail-archive.com/users@lists.open-mpi.org//msg30593.html
(cf. the thread) workaround is to disable IB failback on OPA,
> --mca btl ^tcp,openib
With this  tweak on OPA, OpenMPI's CP2K is less than 10% slower than Intel MPI's 
(the same result as on InfiniBand) - which is much much better that 1500%, huh. 
However Open MPI's CP2K still stays slower than Intel MPI's due to worse 
MPI_Alltoallv, as far as I understood the profiles.

I will mail to CP2K developers soon...



20% bandwidth with Open MPI 2.x: cf.
https://www.mail-archive.com/devel@lists.open-mpi.org/msg00043.html
- Nathan Hjelm mean the hooks are removed by intention. We have a (nasty) 
workaround, cf.

https://www.mail-archive.com/devel@lists.open-mpi.org/msg00052.html
As far as I can see this issue is on InfiniBand only.


Best

Paul

--
Dipl.-Inform. Paul Kapinos   -   High Performance Computing,
RWTH Aachen University, IT Center
Seffenter Weg 23,  D 52074  Aachen (Germany)
Tel: +49 241/80-24915



smime.p7s
Description: S/MIME Cryptographic Signature
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Is building with "--enable-mpi-thread-multiple" recommended?

[Mark Dixon]

On Fri, 3 Mar 2017, Paul Kapinos wrote:
...
Note that on 1.10.x series (even on 1.10.6), enabling of 
MPI_THREAD_MULTIPLE in lead to (silent) shutdown of the InfiniBand 
fabric for that application => SLOW!


2.x versions (tested: 2.0.1) handle MPI_THREAD_MULTIPLE on InfiniBand 
the right way up, however due to absence of memory hooks (= nut aligned 
memory allocation) we get 20% less bandwidth on IB with 2.x versions 
compared to 1.10.x versions of Open MPI (regardless with or without 
support of MPI_THREAD_MULTIPLE).


On Intel OmniPath network both above issues seem to be not present, but 
due to a performance bug in MPI_Free_mem your application can be 
horribly slow (seen: CP2K) if the InfiniBand failback of OPA not 
disabled manually, see 
https://www.mail-archive.com/users@lists.open-mpi.org//msg30593.html

...

Hi Paul,

All very useful - thanks :)

Our (limited) testing seems to show no difference on 2.x with 
MPI_THREAD_MULTIPLE enabled vs. disabled as well, which is good news. Glad 
to hear another opinion.


Your 20% memory bandwidth performance hit on 2.x and the OPA problem are 
concerning - will look at that. Are there tickets open for them?


Cheers,

Mark
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Is building with "--enable-mpi-thread-multiple" recommended?

[Paul Kapinos]

Hi Mark,


On 02/18/17 09:14, Mark Dixon wrote:

On Fri, 17 Feb 2017, r...@open-mpi.org wrote:


Depends on the version, but if you are using something in the v2.x range, you
should be okay with just one installed version




How good is MPI_THREAD_MULTIPLE support these days and how far up the wishlist
is it, please?


Note that on 1.10.x series (even on 1.10.6), enabling of MPI_THREAD_MULTIPLE in 
lead to (silent) shutdown of the InfiniBand fabric for that application => SLOW!


2.x versions (tested: 2.0.1) handle MPI_THREAD_MULTIPLE on InfiniBand the right 
way up, however due to absence of memory hooks (= nut aligned memory allocation) 
we get 20% less bandwidth on IB with 2.x versions compared to 1.10.x versions of 
Open MPI (regardless with or without support of MPI_THREAD_MULTIPLE).


On Intel OmniPath network both above issues seem to be not present, but due to a 
performance bug in MPI_Free_mem your application can be horribly slow (seen: 
CP2K) if the InfiniBand failback of OPA not disabled manually, see

https://www.mail-archive.com/users@lists.open-mpi.org//msg30593.html

Best,

Paul Kapinos



--
Dipl.-Inform. Paul Kapinos   -   High Performance Computing,
RWTH Aachen University, IT Center
Seffenter Weg 23,  D 52074  Aachen (Germany)
Tel: +49 241/80-24915



smime.p7s
Description: S/MIME Cryptographic Signature
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Is building with "--enable-mpi-thread-multiple" recommended?

[r...@open-mpi.org]

FWIW: have you taken a look at the event notification mechanisms in PMIx yet? 
The intent there, among other features, is to provide async notification of 
events generated either by the system (e.g., node failures and/or congestion) 
or other application processes.

https://pmix.github.io/master 

OMPI includes PMIx support beginning with OMPI v2.0, and various RMs are 
releasing their integrated support as well.
Ralph

> On Feb 18, 2017, at 10:07 AM, Michel Lesoinne  wrote:
> 
> I am also a proponent of the multiple thread support. For many reasons:
>  - code simplification
>  - easier support of computation/communication overlap with fewer 
> synchronization points
>  - possibility of creating exception aware MPI Code (I think the MPI standard 
> cruelly lacks constructs for a natural clean handling of application 
> exceptions across processes)
> 
> So it is good to hear there is progress.
> 
> On Feb 18, 2017 7:43 AM, "r...@open-mpi.org " 
> > wrote:
> We have been making a concerted effort to resolve outstanding issues as the 
> interest in threaded applications has grown. It should be pretty good now, 
> but we do see occasional bug reports, so it isn’t perfect.
> 
> > On Feb 18, 2017, at 12:14 AM, Mark Dixon  > > wrote:
> >
> > On Fri, 17 Feb 2017, r...@open-mpi.org  wrote:
> >
> >> Depends on the version, but if you are using something in the v2.x range, 
> >> you should be okay with just one installed version
> >
> > Thanks Ralph.
> >
> > How good is MPI_THREAD_MULTIPLE support these days and how far up the 
> > wishlist is it, please?
> >
> > We don't get many openmpi-specific queries from users but, other than core 
> > binding, it seems to be the thing we get asked about the most (I normally 
> > point those people at mvapich2 or intelmpi instead).
> >
> > Cheers,
> >
> > Mark
> > ___
> > users mailing list
> > users@lists.open-mpi.org 
> > https://rfd.newmexicoconsortium.org/mailman/listinfo/users 
> > 
> 
> ___
> users mailing list
> users@lists.open-mpi.org 
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Is building with "--enable-mpi-thread-multiple" recommended?

[Michel Lesoinne]

I am also a proponent of the multiple thread support. For many reasons:
 - code simplification
 - easier support of computation/communication overlap with fewer
synchronization points
 - possibility of creating exception aware MPI Code (I think the MPI
standard cruelly lacks constructs for a natural clean handling of
application exceptions across processes)

So it is good to hear there is progress.

On Feb 18, 2017 7:43 AM, "r...@open-mpi.org"  wrote:

> We have been making a concerted effort to resolve outstanding issues as
> the interest in threaded applications has grown. It should be pretty good
> now, but we do see occasional bug reports, so it isn’t perfect.
>
> > On Feb 18, 2017, at 12:14 AM, Mark Dixon  wrote:
> >
> > On Fri, 17 Feb 2017, r...@open-mpi.org wrote:
> >
> >> Depends on the version, but if you are using something in the v2.x
> range, you should be okay with just one installed version
> >
> > Thanks Ralph.
> >
> > How good is MPI_THREAD_MULTIPLE support these days and how far up the
> wishlist is it, please?
> >
> > We don't get many openmpi-specific queries from users but, other than
> core binding, it seems to be the thing we get asked about the most (I
> normally point those people at mvapich2 or intelmpi instead).
> >
> > Cheers,
> >
> > Mark
> > ___
> > users mailing list
> > users@lists.open-mpi.org
> > https://rfd.newmexicoconsortium.org/mailman/listinfo/users
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Is building with "--enable-mpi-thread-multiple" recommended?

[r...@open-mpi.org]

We have been making a concerted effort to resolve outstanding issues as the 
interest in threaded applications has grown. It should be pretty good now, but 
we do see occasional bug reports, so it isn’t perfect.

> On Feb 18, 2017, at 12:14 AM, Mark Dixon  wrote:
> 
> On Fri, 17 Feb 2017, r...@open-mpi.org wrote:
> 
>> Depends on the version, but if you are using something in the v2.x range, 
>> you should be okay with just one installed version
> 
> Thanks Ralph.
> 
> How good is MPI_THREAD_MULTIPLE support these days and how far up the 
> wishlist is it, please?
> 
> We don't get many openmpi-specific queries from users but, other than core 
> binding, it seems to be the thing we get asked about the most (I normally 
> point those people at mvapich2 or intelmpi instead).
> 
> Cheers,
> 
> Mark
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Is building with "--enable-mpi-thread-multiple" recommended?

[Mark Dixon]

On Fri, 17 Feb 2017, r...@open-mpi.org wrote:

Depends on the version, but if you are using something in the v2.x 
range, you should be okay with just one installed version


Thanks Ralph.

How good is MPI_THREAD_MULTIPLE support these days and how far up the 
wishlist is it, please?


We don't get many openmpi-specific queries from users but, other than core 
binding, it seems to be the thing we get asked about the most (I normally 
point those people at mvapich2 or intelmpi instead).


Cheers,

Mark
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Is building with "--enable-mpi-thread-multiple" recommended?

[r...@open-mpi.org]

Depends on the version, but if you are using something in the v2.x range, you 
should be okay with just one installed version

> On Feb 17, 2017, at 4:41 AM, Mark Dixon  wrote:
> 
> Hi,
> 
> We have some users who would like to try out openmpi MPI_THREAD_MULTIPLE 
> support on our InfiniBand cluster. I am wondering if we should enable it on 
> our production cluster-wide version, or install it as a separate "here be 
> dragons" copy.
> 
> I seem to recall openmpi folk cautioning that MPI_THREAD_MULTIPLE support was 
> pretty crazy and that enabling it could have problems for 
> non-MPI_THREAD_MULTIPLE codes (never mind codes that explicitly used it), so 
> such an install shouldn't be used unless for codes that actually need it.
> 
> Is that still the case, please?
> 
> Thanks,
> 
> Mark
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[OMPI users] Is building with "--enable-mpi-thread-multiple" recommended?

[Mark Dixon]

Hi,

We have some users who would like to try out openmpi MPI_THREAD_MULTIPLE 
support on our InfiniBand cluster. I am wondering if we should enable it 
on our production cluster-wide version, or install it as a separate "here 
be dragons" copy.


I seem to recall openmpi folk cautioning that MPI_THREAD_MULTIPLE support 
was pretty crazy and that enabling it could have problems for 
non-MPI_THREAD_MULTIPLE codes (never mind codes that explicitly used it), 
so such an install shouldn't be used unless for codes that actually need 
it.


Is that still the case, please?

Thanks,

Mark
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users