Re: [OMPI users] Help on getting CMA works

[Nathan Hjelm]

I don't know the reasoning for requiring --with-cma to enable CMA but I
am looking at auto-detecting CMA instead of requiring Open MPI to be
configured with --with-cma. This will likely go into the 1.9 release
series and not 1.8.

-Nathan

On Thu, Feb 19, 2015 at 09:31:43PM -0500, Eric Chamberland wrote:
> Maybe it is a stupid question, but... why it is not tested and enabled by
> default at configure time since it is part of the kernel?
> 
> Eric
> 
> 
> On 02/19/2015 03:53 PM, Nathan Hjelm wrote:
> >Great! I will add an MCA variable to force CMA and also enable it if 1)
> >no yama and 2) no PR_SET_PTRACER.
> >
> >You might also look at using xpmem. You can find a version that supports
> >3.x @ https://github.com/hjelmn/xpmem . It is a kernel module +
> >userspace library that can be used by vader as a single-copy mechanism.
> >
> >In benchmarks it performs better than CMA but it may or may not perform
> >better with a real application.
> >
> >See:
> >
> >http://blogs.cisco.com/performance/the-vader-shared-memory-transport-in-open-mpi-now-featuring-3-flavors-of-zero-copy
> >
> >-Nathan
> >
> >On Thu, Feb 19, 2015 at 03:32:43PM -0500, Eric Chamberland wrote:
> >>On 02/19/2015 02:58 PM, Nathan Hjelm wrote:
> >>>On Thu, Feb 19, 2015 at 12:16:49PM -0500, Eric Chamberland wrote:
> On 02/19/2015 11:56 AM, Nathan Hjelm wrote:
> >If you have yama installed you can try:
> Nope, I do not have it installed... is it absolutely necessary? (and would
> it change something when it fails when I am root?)
> 
> Other question: In addition to "--with-cma" configure flag, do we have to
> pass any options to "mpicc" when compiling/linking an mpi application to 
> use
> cma?
> >>>No. CMA should work out of the box. You appear to have a setup I haven't
> >>>yet tested. It doesn't have yama nor does it have the PR_SET_PTRACER
> >>>prctl. Its quite possible there are no restriction on ptrace in this
> >>>setup. Can you try changing the following line at
> >>>opal/mca/btl/vader/btl_vader_component.c:370 from:
> >>>
> >>>bool cma_happy = false;
> >>>
> >>>to
> >>>
> >>>bool cma_happy = true;
> >>>
> >>ok! (as of the officiel release, this is line 386.)
> >>
> >>>and let me know if that works. If it does I will update vader to allow
> >>>CMA in this configuration.
> >>Yep!  It now works perfectly.  Testing with
> >>https://computing.llnl.gov/tutorials/mpi/samples/C/mpi_bandwidth.c, on my
> >>own computer (dual Xeon), I have this:
> >>
> >>Without CMA:
> >>
> >>***Message size:  100 *** best  /  avg  / worst (MB/sec)
> >>task pair:0 -1:8363.52 / 7946.77 / 5391.14
> >>
> >>with CMA:
> >>task pair:0 -1:9137.92 / 8955.98 / 7489.83
> >>
> >>Great!
> >>
> >>Now I have to bench my real application... ;-)
> >>
> >>Thanks!
> >>
> >>Eric
> >>
> >>___
> >>users mailing list
> >>us...@open-mpi.org
> >>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>Link to this post: 
> >>http://www.open-mpi.org/community/lists/users/2015/02/26355.php
> 
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/02/26362.php


pgpKWremq_SLC.pgp
Description: PGP signature

Re: [OMPI users] Help on getting CMA works

[Eric Chamberland]

Maybe it is a stupid question, but... why it is not tested and enabled 
by default at configure time since it is part of the kernel?


Eric


On 02/19/2015 03:53 PM, Nathan Hjelm wrote:

Great! I will add an MCA variable to force CMA and also enable it if 1)
no yama and 2) no PR_SET_PTRACER.

You might also look at using xpmem. You can find a version that supports
3.x @ https://github.com/hjelmn/xpmem . It is a kernel module +
userspace library that can be used by vader as a single-copy mechanism.

In benchmarks it performs better than CMA but it may or may not perform
better with a real application.

See:

http://blogs.cisco.com/performance/the-vader-shared-memory-transport-in-open-mpi-now-featuring-3-flavors-of-zero-copy

-Nathan

On Thu, Feb 19, 2015 at 03:32:43PM -0500, Eric Chamberland wrote:

On 02/19/2015 02:58 PM, Nathan Hjelm wrote:

On Thu, Feb 19, 2015 at 12:16:49PM -0500, Eric Chamberland wrote:

On 02/19/2015 11:56 AM, Nathan Hjelm wrote:

If you have yama installed you can try:

Nope, I do not have it installed... is it absolutely necessary? (and would
it change something when it fails when I am root?)

Other question: In addition to "--with-cma" configure flag, do we have to
pass any options to "mpicc" when compiling/linking an mpi application to use
cma?

No. CMA should work out of the box. You appear to have a setup I haven't
yet tested. It doesn't have yama nor does it have the PR_SET_PTRACER
prctl. Its quite possible there are no restriction on ptrace in this
setup. Can you try changing the following line at
opal/mca/btl/vader/btl_vader_component.c:370 from:

bool cma_happy = false;

to

bool cma_happy = true;


ok! (as of the officiel release, this is line 386.)


and let me know if that works. If it does I will update vader to allow
CMA in this configuration.

Yep!  It now works perfectly.  Testing with
https://computing.llnl.gov/tutorials/mpi/samples/C/mpi_bandwidth.c, on my
own computer (dual Xeon), I have this:

Without CMA:

***Message size:  100 *** best  /  avg  / worst (MB/sec)
task pair:0 -1:8363.52 / 7946.77 / 5391.14

with CMA:
task pair:0 -1:9137.92 / 8955.98 / 7489.83

Great!

Now I have to bench my real application... ;-)

Thanks!

Eric

___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2015/02/26355.php

Re: [OMPI users] Help on getting CMA works

[Nathan Hjelm]

Aurélien, I should also point out your fix has already been applied to
the 1.8 branch and will be included in 1.8.5.

-Nathan

On Thu, Feb 19, 2015 at 02:57:38PM -0700, Nathan Hjelm wrote:
> 
> Hmm, wait. Yes. Your change went in after 1.8.4 and has the same
> effect. If yama ins't installed it is safe to assume that the ptrace
> scope is effectively 0. So, your patch does fix the issue.
> 
> -Nathan
> 
> On Thu, Feb 19, 2015 at 02:53:47PM -0700, Nathan Hjelm wrote:
> > 
> > I don't think that will fix this issue. In this case yama is not
> > installed and it appears PR_SET_PTRACER is not available. This forces
> > vader to assume that CMA can not be used when that isn't always the
> > case. I think it might be safe to assume that CMA is unrestricted here.
> > 
> > -Nathan
> > 
> > On Thu, Feb 19, 2015 at 04:35:00PM -0500, Aurélien Bouteiller wrote:
> > > Nathan, 
> > > 
> > > I think I already pushed a patch for this particular issue last month. I 
> > > do not know if it has been back ported to release yet. 
> > > 
> > > See 
> > > here:https://github.com/open-mpi/ompi/commit/ee3b0903164898750137d3b71a8f067e16521102
> > > 
> > > Aurelien 
> > > 
> > > --
> > >   ~~~ Aurélien Bouteiller, Ph.D. ~~~
> > >  ~ Research Scientist @ ICL ~
> > > The University of Tennessee, Innovative Computing Laboratory
> > > 1122 Volunteer Blvd, suite 309, Knoxville, TN 37996
> > > tel: +1 (865) 974-9375   fax: +1 (865) 974-8296
> > > https://icl.cs.utk.edu/~bouteill/
> > > 
> > > 
> > > 
> > > 
> > > > Le 19 févr. 2015 à 15:53, Nathan Hjelm  a écrit :
> > > > 
> > > > 
> > > > Great! I will add an MCA variable to force CMA and also enable it if 1)
> > > > no yama and 2) no PR_SET_PTRACER.
> > > > 
> > > > You might also look at using xpmem. You can find a version that supports
> > > > 3.x @ https://github.com/hjelmn/xpmem . It is a kernel module +
> > > > userspace library that can be used by vader as a single-copy mechanism.
> > > > 
> > > > In benchmarks it performs better than CMA but it may or may not perform
> > > > better with a real application.
> > > > 
> > > > See:
> > > > 
> > > > http://blogs.cisco.com/performance/the-vader-shared-memory-transport-in-open-mpi-now-featuring-3-flavors-of-zero-copy
> > > > 
> > > > -Nathan
> > > > 
> > > > On Thu, Feb 19, 2015 at 03:32:43PM -0500, Eric Chamberland wrote:
> > > >> On 02/19/2015 02:58 PM, Nathan Hjelm wrote:
> > > >>> On Thu, Feb 19, 2015 at 12:16:49PM -0500, Eric Chamberland wrote:
> > >  
> > >  On 02/19/2015 11:56 AM, Nathan Hjelm wrote:
> > > > 
> > > > If you have yama installed you can try:
> > >  
> > >  Nope, I do not have it installed... is it absolutely necessary? (and 
> > >  would
> > >  it change something when it fails when I am root?)
> > >  
> > >  Other question: In addition to "--with-cma" configure flag, do we 
> > >  have to
> > >  pass any options to "mpicc" when compiling/linking an mpi 
> > >  application to use
> > >  cma?
> > > >>> 
> > > >>> No. CMA should work out of the box. You appear to have a setup I 
> > > >>> haven't
> > > >>> yet tested. It doesn't have yama nor does it have the PR_SET_PTRACER
> > > >>> prctl. Its quite possible there are no restriction on ptrace in this
> > > >>> setup. Can you try changing the following line at
> > > >>> opal/mca/btl/vader/btl_vader_component.c:370 from:
> > > >>> 
> > > >>> bool cma_happy = false;
> > > >>> 
> > > >>> to
> > > >>> 
> > > >>> bool cma_happy = true;
> > > >>> 
> > > >> 
> > > >> ok! (as of the officiel release, this is line 386.)
> > > >> 
> > > >>> and let me know if that works. If it does I will update vader to allow
> > > >>> CMA in this configuration.
> > > >> 
> > > >> Yep!  It now works perfectly.  Testing with
> > > >> https://computing.llnl.gov/tutorials/mpi/samples/C/mpi_bandwidth.c, on 
> > > >> my
> > > >> own computer (dual Xeon), I have this:
> > > >> 
> > > >> Without CMA:
> > > >> 
> > > >> ***Message size:  100 *** best  /  avg  / worst (MB/sec)
> > > >>   task pair:0 -1:8363.52 / 7946.77 / 5391.14
> > > >> 
> > > >> with CMA:
> > > >>   task pair:0 -1:9137.92 / 8955.98 / 7489.83
> > > >> 
> > > >> Great!
> > > >> 
> > > >> Now I have to bench my real application... ;-)
> > > >> 
> > > >> Thanks!
> > > >> 
> > > >> Eric
> > > >> 
> > > >> ___
> > > >> users mailing list
> > > >> us...@open-mpi.org
> > > >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> > > >> Link to this post: 
> > > >> http://www.open-mpi.org/community/lists/users/2015/02/26355.php
> > > > ___
> > > > users mailing list
> > > > us...@open-mpi.org
> > > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> > > > Link to this post: 
> > > > http://www.open-mpi.org/community/lists/users/2015/02/26356.php
> > > 
> > > 

Re: [OMPI users] Help on getting CMA works

[Nathan Hjelm]

Hmm, wait. Yes. Your change went in after 1.8.4 and has the same
effect. If yama ins't installed it is safe to assume that the ptrace
scope is effectively 0. So, your patch does fix the issue.

-Nathan

On Thu, Feb 19, 2015 at 02:53:47PM -0700, Nathan Hjelm wrote:
> 
> I don't think that will fix this issue. In this case yama is not
> installed and it appears PR_SET_PTRACER is not available. This forces
> vader to assume that CMA can not be used when that isn't always the
> case. I think it might be safe to assume that CMA is unrestricted here.
> 
> -Nathan
> 
> On Thu, Feb 19, 2015 at 04:35:00PM -0500, Aurélien Bouteiller wrote:
> > Nathan, 
> > 
> > I think I already pushed a patch for this particular issue last month. I do 
> > not know if it has been back ported to release yet. 
> > 
> > See 
> > here:https://github.com/open-mpi/ompi/commit/ee3b0903164898750137d3b71a8f067e16521102
> > 
> > Aurelien 
> > 
> > --
> >   ~~~ Aurélien Bouteiller, Ph.D. ~~~
> >  ~ Research Scientist @ ICL ~
> > The University of Tennessee, Innovative Computing Laboratory
> > 1122 Volunteer Blvd, suite 309, Knoxville, TN 37996
> > tel: +1 (865) 974-9375   fax: +1 (865) 974-8296
> > https://icl.cs.utk.edu/~bouteill/
> > 
> > 
> > 
> > 
> > > Le 19 févr. 2015 à 15:53, Nathan Hjelm  a écrit :
> > > 
> > > 
> > > Great! I will add an MCA variable to force CMA and also enable it if 1)
> > > no yama and 2) no PR_SET_PTRACER.
> > > 
> > > You might also look at using xpmem. You can find a version that supports
> > > 3.x @ https://github.com/hjelmn/xpmem . It is a kernel module +
> > > userspace library that can be used by vader as a single-copy mechanism.
> > > 
> > > In benchmarks it performs better than CMA but it may or may not perform
> > > better with a real application.
> > > 
> > > See:
> > > 
> > > http://blogs.cisco.com/performance/the-vader-shared-memory-transport-in-open-mpi-now-featuring-3-flavors-of-zero-copy
> > > 
> > > -Nathan
> > > 
> > > On Thu, Feb 19, 2015 at 03:32:43PM -0500, Eric Chamberland wrote:
> > >> On 02/19/2015 02:58 PM, Nathan Hjelm wrote:
> > >>> On Thu, Feb 19, 2015 at 12:16:49PM -0500, Eric Chamberland wrote:
> >  
> >  On 02/19/2015 11:56 AM, Nathan Hjelm wrote:
> > > 
> > > If you have yama installed you can try:
> >  
> >  Nope, I do not have it installed... is it absolutely necessary? (and 
> >  would
> >  it change something when it fails when I am root?)
> >  
> >  Other question: In addition to "--with-cma" configure flag, do we have 
> >  to
> >  pass any options to "mpicc" when compiling/linking an mpi application 
> >  to use
> >  cma?
> > >>> 
> > >>> No. CMA should work out of the box. You appear to have a setup I haven't
> > >>> yet tested. It doesn't have yama nor does it have the PR_SET_PTRACER
> > >>> prctl. Its quite possible there are no restriction on ptrace in this
> > >>> setup. Can you try changing the following line at
> > >>> opal/mca/btl/vader/btl_vader_component.c:370 from:
> > >>> 
> > >>> bool cma_happy = false;
> > >>> 
> > >>> to
> > >>> 
> > >>> bool cma_happy = true;
> > >>> 
> > >> 
> > >> ok! (as of the officiel release, this is line 386.)
> > >> 
> > >>> and let me know if that works. If it does I will update vader to allow
> > >>> CMA in this configuration.
> > >> 
> > >> Yep!  It now works perfectly.  Testing with
> > >> https://computing.llnl.gov/tutorials/mpi/samples/C/mpi_bandwidth.c, on my
> > >> own computer (dual Xeon), I have this:
> > >> 
> > >> Without CMA:
> > >> 
> > >> ***Message size:  100 *** best  /  avg  / worst (MB/sec)
> > >>   task pair:0 -1:8363.52 / 7946.77 / 5391.14
> > >> 
> > >> with CMA:
> > >>   task pair:0 -1:9137.92 / 8955.98 / 7489.83
> > >> 
> > >> Great!
> > >> 
> > >> Now I have to bench my real application... ;-)
> > >> 
> > >> Thanks!
> > >> 
> > >> Eric
> > >> 
> > >> ___
> > >> users mailing list
> > >> us...@open-mpi.org
> > >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> > >> Link to this post: 
> > >> http://www.open-mpi.org/community/lists/users/2015/02/26355.php
> > > ___
> > > users mailing list
> > > us...@open-mpi.org
> > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> > > Link to this post: 
> > > http://www.open-mpi.org/community/lists/users/2015/02/26356.php
> > 
> > ___
> > users mailing list
> > us...@open-mpi.org
> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> > Link to this post: 
> > http://www.open-mpi.org/community/lists/users/2015/02/26358.php



> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/02/26359.php

Re: [OMPI users] Help on getting CMA works

[Nathan Hjelm]

I don't think that will fix this issue. In this case yama is not
installed and it appears PR_SET_PTRACER is not available. This forces
vader to assume that CMA can not be used when that isn't always the
case. I think it might be safe to assume that CMA is unrestricted here.

-Nathan

On Thu, Feb 19, 2015 at 04:35:00PM -0500, Aurélien Bouteiller wrote:
> Nathan, 
> 
> I think I already pushed a patch for this particular issue last month. I do 
> not know if it has been back ported to release yet. 
> 
> See 
> here:https://github.com/open-mpi/ompi/commit/ee3b0903164898750137d3b71a8f067e16521102
> 
> Aurelien 
> 
> --
>   ~~~ Aurélien Bouteiller, Ph.D. ~~~
>  ~ Research Scientist @ ICL ~
> The University of Tennessee, Innovative Computing Laboratory
> 1122 Volunteer Blvd, suite 309, Knoxville, TN 37996
> tel: +1 (865) 974-9375   fax: +1 (865) 974-8296
> https://icl.cs.utk.edu/~bouteill/
> 
> 
> 
> 
> > Le 19 févr. 2015 à 15:53, Nathan Hjelm  a écrit :
> > 
> > 
> > Great! I will add an MCA variable to force CMA and also enable it if 1)
> > no yama and 2) no PR_SET_PTRACER.
> > 
> > You might also look at using xpmem. You can find a version that supports
> > 3.x @ https://github.com/hjelmn/xpmem . It is a kernel module +
> > userspace library that can be used by vader as a single-copy mechanism.
> > 
> > In benchmarks it performs better than CMA but it may or may not perform
> > better with a real application.
> > 
> > See:
> > 
> > http://blogs.cisco.com/performance/the-vader-shared-memory-transport-in-open-mpi-now-featuring-3-flavors-of-zero-copy
> > 
> > -Nathan
> > 
> > On Thu, Feb 19, 2015 at 03:32:43PM -0500, Eric Chamberland wrote:
> >> On 02/19/2015 02:58 PM, Nathan Hjelm wrote:
> >>> On Thu, Feb 19, 2015 at 12:16:49PM -0500, Eric Chamberland wrote:
>  
>  On 02/19/2015 11:56 AM, Nathan Hjelm wrote:
> > 
> > If you have yama installed you can try:
>  
>  Nope, I do not have it installed... is it absolutely necessary? (and 
>  would
>  it change something when it fails when I am root?)
>  
>  Other question: In addition to "--with-cma" configure flag, do we have to
>  pass any options to "mpicc" when compiling/linking an mpi application to 
>  use
>  cma?
> >>> 
> >>> No. CMA should work out of the box. You appear to have a setup I haven't
> >>> yet tested. It doesn't have yama nor does it have the PR_SET_PTRACER
> >>> prctl. Its quite possible there are no restriction on ptrace in this
> >>> setup. Can you try changing the following line at
> >>> opal/mca/btl/vader/btl_vader_component.c:370 from:
> >>> 
> >>> bool cma_happy = false;
> >>> 
> >>> to
> >>> 
> >>> bool cma_happy = true;
> >>> 
> >> 
> >> ok! (as of the officiel release, this is line 386.)
> >> 
> >>> and let me know if that works. If it does I will update vader to allow
> >>> CMA in this configuration.
> >> 
> >> Yep!  It now works perfectly.  Testing with
> >> https://computing.llnl.gov/tutorials/mpi/samples/C/mpi_bandwidth.c, on my
> >> own computer (dual Xeon), I have this:
> >> 
> >> Without CMA:
> >> 
> >> ***Message size:  100 *** best  /  avg  / worst (MB/sec)
> >>   task pair:0 -1:8363.52 / 7946.77 / 5391.14
> >> 
> >> with CMA:
> >>   task pair:0 -1:9137.92 / 8955.98 / 7489.83
> >> 
> >> Great!
> >> 
> >> Now I have to bench my real application... ;-)
> >> 
> >> Thanks!
> >> 
> >> Eric
> >> 
> >> ___
> >> users mailing list
> >> us...@open-mpi.org
> >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> >> Link to this post: 
> >> http://www.open-mpi.org/community/lists/users/2015/02/26355.php
> > ___
> > users mailing list
> > us...@open-mpi.org
> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> > Link to this post: 
> > http://www.open-mpi.org/community/lists/users/2015/02/26356.php
> 
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/02/26358.php


pgp7NUNlOKKzV.pgp
Description: PGP signature

Re: [OMPI users] Help on getting CMA works

[Aurélien Bouteiller]

Nathan, 

I think I already pushed a patch for this particular issue last month. I do not 
know if it has been back ported to release yet. 

See 
here:https://github.com/open-mpi/ompi/commit/ee3b0903164898750137d3b71a8f067e16521102

Aurelien 

--
  ~~~ Aurélien Bouteiller, Ph.D. ~~~
 ~ Research Scientist @ ICL ~
The University of Tennessee, Innovative Computing Laboratory
1122 Volunteer Blvd, suite 309, Knoxville, TN 37996
tel: +1 (865) 974-9375   fax: +1 (865) 974-8296
https://icl.cs.utk.edu/~bouteill/




> Le 19 févr. 2015 à 15:53, Nathan Hjelm  a écrit :
> 
> 
> Great! I will add an MCA variable to force CMA and also enable it if 1)
> no yama and 2) no PR_SET_PTRACER.
> 
> You might also look at using xpmem. You can find a version that supports
> 3.x @ https://github.com/hjelmn/xpmem . It is a kernel module +
> userspace library that can be used by vader as a single-copy mechanism.
> 
> In benchmarks it performs better than CMA but it may or may not perform
> better with a real application.
> 
> See:
> 
> http://blogs.cisco.com/performance/the-vader-shared-memory-transport-in-open-mpi-now-featuring-3-flavors-of-zero-copy
> 
> -Nathan
> 
> On Thu, Feb 19, 2015 at 03:32:43PM -0500, Eric Chamberland wrote:
>> On 02/19/2015 02:58 PM, Nathan Hjelm wrote:
>>> On Thu, Feb 19, 2015 at 12:16:49PM -0500, Eric Chamberland wrote:
 
 On 02/19/2015 11:56 AM, Nathan Hjelm wrote:
> 
> If you have yama installed you can try:
 
 Nope, I do not have it installed... is it absolutely necessary? (and would
 it change something when it fails when I am root?)
 
 Other question: In addition to "--with-cma" configure flag, do we have to
 pass any options to "mpicc" when compiling/linking an mpi application to 
 use
 cma?
>>> 
>>> No. CMA should work out of the box. You appear to have a setup I haven't
>>> yet tested. It doesn't have yama nor does it have the PR_SET_PTRACER
>>> prctl. Its quite possible there are no restriction on ptrace in this
>>> setup. Can you try changing the following line at
>>> opal/mca/btl/vader/btl_vader_component.c:370 from:
>>> 
>>> bool cma_happy = false;
>>> 
>>> to
>>> 
>>> bool cma_happy = true;
>>> 
>> 
>> ok! (as of the officiel release, this is line 386.)
>> 
>>> and let me know if that works. If it does I will update vader to allow
>>> CMA in this configuration.
>> 
>> Yep!  It now works perfectly.  Testing with
>> https://computing.llnl.gov/tutorials/mpi/samples/C/mpi_bandwidth.c, on my
>> own computer (dual Xeon), I have this:
>> 
>> Without CMA:
>> 
>> ***Message size:  100 *** best  /  avg  / worst (MB/sec)
>>   task pair:0 -1:8363.52 / 7946.77 / 5391.14
>> 
>> with CMA:
>>   task pair:0 -1:9137.92 / 8955.98 / 7489.83
>> 
>> Great!
>> 
>> Now I have to bench my real application... ;-)
>> 
>> Thanks!
>> 
>> Eric
>> 
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/users/2015/02/26355.php
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/02/26356.php

Re: [OMPI users] Help on getting CMA works

[Eric Chamberland]

On 02/19/2015 03:53 PM, Nathan Hjelm wrote:


Great! I will add an MCA variable to force CMA and also enable it if 1)
no yama and 2) no PR_SET_PTRACER.


cool, thanks again!



You might also look at using xpmem. You can find a version that supports
3.x @ https://github.com/hjelmn/xpmem . It is a kernel module +
userspace library that can be used by vader as a single-copy mechanism.

In benchmarks it performs better than CMA but it may or may not perform
better with a real application.

See:

http://blogs.cisco.com/performance/the-vader-shared-memory-transport-in-open-mpi-now-featuring-3-flavors-of-zero-copy


ok, I will look (and relay the information to colleagues).

Thanks,

Eric

Re: [OMPI users] Help on getting CMA works

[Nathan Hjelm]

Great! I will add an MCA variable to force CMA and also enable it if 1)
no yama and 2) no PR_SET_PTRACER.

You might also look at using xpmem. You can find a version that supports
3.x @ https://github.com/hjelmn/xpmem . It is a kernel module +
userspace library that can be used by vader as a single-copy mechanism.

In benchmarks it performs better than CMA but it may or may not perform
better with a real application.

See:

http://blogs.cisco.com/performance/the-vader-shared-memory-transport-in-open-mpi-now-featuring-3-flavors-of-zero-copy

-Nathan

On Thu, Feb 19, 2015 at 03:32:43PM -0500, Eric Chamberland wrote:
> On 02/19/2015 02:58 PM, Nathan Hjelm wrote:
> >On Thu, Feb 19, 2015 at 12:16:49PM -0500, Eric Chamberland wrote:
> >>
> >>On 02/19/2015 11:56 AM, Nathan Hjelm wrote:
> >>>
> >>>If you have yama installed you can try:
> >>
> >>Nope, I do not have it installed... is it absolutely necessary? (and would
> >>it change something when it fails when I am root?)
> >>
> >>Other question: In addition to "--with-cma" configure flag, do we have to
> >>pass any options to "mpicc" when compiling/linking an mpi application to use
> >>cma?
> >
> >No. CMA should work out of the box. You appear to have a setup I haven't
> >yet tested. It doesn't have yama nor does it have the PR_SET_PTRACER
> >prctl. Its quite possible there are no restriction on ptrace in this
> >setup. Can you try changing the following line at
> >opal/mca/btl/vader/btl_vader_component.c:370 from:
> >
> >bool cma_happy = false;
> >
> >to
> >
> >bool cma_happy = true;
> >
> 
> ok! (as of the officiel release, this is line 386.)
> 
> >and let me know if that works. If it does I will update vader to allow
> >CMA in this configuration.
> 
> Yep!  It now works perfectly.  Testing with
> https://computing.llnl.gov/tutorials/mpi/samples/C/mpi_bandwidth.c, on my
> own computer (dual Xeon), I have this:
> 
> Without CMA:
> 
> ***Message size:  100 *** best  /  avg  / worst (MB/sec)
>task pair:0 -1:8363.52 / 7946.77 / 5391.14
> 
> with CMA:
>task pair:0 -1:9137.92 / 8955.98 / 7489.83
> 
> Great!
> 
> Now I have to bench my real application... ;-)
> 
> Thanks!
> 
> Eric
> 
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/02/26355.php


pgp4qiwgVsc3t.pgp
Description: PGP signature

Re: [OMPI users] Help on getting CMA works

[Eric Chamberland]

On 02/19/2015 02:58 PM, Nathan Hjelm wrote:

On Thu, Feb 19, 2015 at 12:16:49PM -0500, Eric Chamberland wrote:


On 02/19/2015 11:56 AM, Nathan Hjelm wrote:


If you have yama installed you can try:


Nope, I do not have it installed... is it absolutely necessary? (and would
it change something when it fails when I am root?)

Other question: In addition to "--with-cma" configure flag, do we have to
pass any options to "mpicc" when compiling/linking an mpi application to use
cma?


No. CMA should work out of the box. You appear to have a setup I haven't
yet tested. It doesn't have yama nor does it have the PR_SET_PTRACER
prctl. Its quite possible there are no restriction on ptrace in this
setup. Can you try changing the following line at
opal/mca/btl/vader/btl_vader_component.c:370 from:

bool cma_happy = false;

to

bool cma_happy = true;



ok! (as of the officiel release, this is line 386.)


and let me know if that works. If it does I will update vader to allow
CMA in this configuration.


Yep!  It now works perfectly.  Testing with 
https://computing.llnl.gov/tutorials/mpi/samples/C/mpi_bandwidth.c, on 
my own computer (dual Xeon), I have this:


Without CMA:

***Message size:  100 *** best  /  avg  / worst (MB/sec)
   task pair:0 -1:8363.52 / 7946.77 / 5391.14

with CMA:
   task pair:0 -1:9137.92 / 8955.98 / 7489.83

Great!

Now I have to bench my real application... ;-)

Thanks!

Eric

Re: [OMPI users] Help on getting CMA works

[Nathan Hjelm]

On Thu, Feb 19, 2015 at 12:16:49PM -0500, Eric Chamberland wrote:
> 
> On 02/19/2015 11:56 AM, Nathan Hjelm wrote:
> >
> >If you have yama installed you can try:
> 
> Nope, I do not have it installed... is it absolutely necessary? (and would
> it change something when it fails when I am root?)
> 
> Other question: In addition to "--with-cma" configure flag, do we have to
> pass any options to "mpicc" when compiling/linking an mpi application to use
> cma?

No. CMA should work out of the box. You appear to have a setup I haven't
yet tested. It doesn't have yama nor does it have the PR_SET_PTRACER
prctl. Its quite possible there are no restriction on ptrace in this
setup. Can you try changing the following line at
opal/mca/btl/vader/btl_vader_component.c:370 from:

bool cma_happy = false;

to

bool cma_happy = true;

and let me know if that works. If it does I will update vader to allow
CMA in this configuration.

-Nathan


pgp8k5J9uaK7b.pgp
Description: PGP signature

Re: [OMPI users] Help on getting CMA works

[Eric Chamberland]

On 02/19/2015 11:56 AM, Nathan Hjelm wrote:


If you have yama installed you can try:


Nope, I do not have it installed... is it absolutely necessary? (and 
would it change something when it fails when I am root?)


Other question: In addition to "--with-cma" configure flag, do we have 
to pass any options to "mpicc" when compiling/linking an mpi application 
to use cma?


Thanks,

Eric



echo 1 > /proc/sys/kernel/yama/ptrace_scope

as root.

-Nathan

On Thu, Feb 19, 2015 at 11:06:09AM -0500, Eric Chamberland wrote:

By the way,

I have tried two others things:

#1- I launched it as root:

mpiexec --mca mca_btl_vader_single_copy_mechanism cma --allow-run-as-root
-np 2 ./hw

#2- Found this 
(http://askubuntu.com/questions/146160/what-is-the-ptrace-scope-workaround-for-wine-programs-and-are-there-any-risks)
and tried this:

sudo setcap cap_sys_ptrace=eip /tmp/hw

On both RedHat 6.5 and OpenSuse 12.3 and still get the same error message!!!
:-/

Sorry, I am not a kernel expert...

What's wrong?

Thanks,

Eric

On 02/18/2015 04:48 PM, Éric Chamberland wrote:


Le 2015-02-18 15:14, Nathan Hjelm a écrit :

I recommend using vader for CMA. It has code to get around the ptrace
setting. Run with mca_btl_vader_single_copy_mechanism cma (should be the
default).

Ok, I tried it, but it gives exactly the same error message!

Eric


-Nathan

On Wed, Feb 18, 2015 at 02:56:01PM -0500, Eric Chamberland wrote:

Hi,

I have configured with "--with-cma" on 2 differents OS (RedHat 6.6 and
OpenSuse 12.3), but in both case, I have the following error when
launching
a simple mpi_hello_world.c example:

/opt/openmpi-1.8.4_cma/bin/mpiexec --mca btl_sm_use_cma 1 -np 2 /tmp/hw
--

WARNING: Linux kernel CMA support was requested via the
btl_vader_single_copy_mechanism MCA variable, but CMA support is
not available due to restrictive ptrace settings.

The vader shared memory BTL will fall back on another single-copy
mechanism if one is available. This may result in lower performance.

   Local host: compile
--

Hello world from process 0 of 2
Hello world from process 1 of 2
[compile:23874] 1 more process has sent help message
help-btl-vader.txt /
cma-permission-denied
[compile:23874] Set MCA parameter "orte_base_help_aggregate" to 0 to
see all
help / error messages

After I googled the subject, it seems there is a kernel parameter to
modify,
but I can't find it for OpenSuse 12.3 or RedHat 6.6...

Here is the "config.log" issued from RedHat 6.6...

http://www.giref.ulaval.ca/~ericc/ompi_bug/config.184_cma.gz

Thanks,

Eric
___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2015/02/26339.php


___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2015/02/26342.php


___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2015/02/26351.php

Re: [OMPI users] Help on getting CMA works

[Nathan Hjelm]

If you have yama installed you can try:

echo 1 > /proc/sys/kernel/yama/ptrace_scope

as root.

-Nathan

On Thu, Feb 19, 2015 at 11:06:09AM -0500, Eric Chamberland wrote:
> By the way,
> 
> I have tried two others things:
> 
> #1- I launched it as root:
> 
> mpiexec --mca mca_btl_vader_single_copy_mechanism cma --allow-run-as-root
> -np 2 ./hw
> 
> #2- Found this 
> (http://askubuntu.com/questions/146160/what-is-the-ptrace-scope-workaround-for-wine-programs-and-are-there-any-risks)
> and tried this:
> 
> sudo setcap cap_sys_ptrace=eip /tmp/hw
> 
> On both RedHat 6.5 and OpenSuse 12.3 and still get the same error message!!!
> :-/
> 
> Sorry, I am not a kernel expert...
> 
> What's wrong?
> 
> Thanks,
> 
> Eric
> 
> On 02/18/2015 04:48 PM, Éric Chamberland wrote:
> >
> >Le 2015-02-18 15:14, Nathan Hjelm a écrit :
> >>I recommend using vader for CMA. It has code to get around the ptrace
> >>setting. Run with mca_btl_vader_single_copy_mechanism cma (should be the
> >>default).
> >Ok, I tried it, but it gives exactly the same error message!
> >
> >Eric
> >
> >>-Nathan
> >>
> >>On Wed, Feb 18, 2015 at 02:56:01PM -0500, Eric Chamberland wrote:
> >>>Hi,
> >>>
> >>>I have configured with "--with-cma" on 2 differents OS (RedHat 6.6 and
> >>>OpenSuse 12.3), but in both case, I have the following error when
> >>>launching
> >>>a simple mpi_hello_world.c example:
> >>>
> >>>/opt/openmpi-1.8.4_cma/bin/mpiexec --mca btl_sm_use_cma 1 -np 2 /tmp/hw
> >>>--
> >>>
> >>>WARNING: Linux kernel CMA support was requested via the
> >>>btl_vader_single_copy_mechanism MCA variable, but CMA support is
> >>>not available due to restrictive ptrace settings.
> >>>
> >>>The vader shared memory BTL will fall back on another single-copy
> >>>mechanism if one is available. This may result in lower performance.
> >>>
> >>>   Local host: compile
> >>>--
> >>>
> >>>Hello world from process 0 of 2
> >>>Hello world from process 1 of 2
> >>>[compile:23874] 1 more process has sent help message
> >>>help-btl-vader.txt /
> >>>cma-permission-denied
> >>>[compile:23874] Set MCA parameter "orte_base_help_aggregate" to 0 to
> >>>see all
> >>>help / error messages
> >>>
> >>>After I googled the subject, it seems there is a kernel parameter to
> >>>modify,
> >>>but I can't find it for OpenSuse 12.3 or RedHat 6.6...
> >>>
> >>>Here is the "config.log" issued from RedHat 6.6...
> >>>
> >>>http://www.giref.ulaval.ca/~ericc/ompi_bug/config.184_cma.gz
> >>>
> >>>Thanks,
> >>>
> >>>Eric
> >>>___
> >>>users mailing list
> >>>us...@open-mpi.org
> >>>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>>Link to this post:
> >>>http://www.open-mpi.org/community/lists/users/2015/02/26339.php
> >
> >___
> >users mailing list
> >us...@open-mpi.org
> >Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> >Link to this post:
> >http://www.open-mpi.org/community/lists/users/2015/02/26342.php
> 
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/02/26351.php


pgpqgDxaFnCcb.pgp
Description: PGP signature

Re: [OMPI users] Help on getting CMA works

[Eric Chamberland]

By the way,

I have tried two others things:

#1- I launched it as root:

mpiexec --mca mca_btl_vader_single_copy_mechanism cma 
--allow-run-as-root -np 2 ./hw


#2- Found this 
(http://askubuntu.com/questions/146160/what-is-the-ptrace-scope-workaround-for-wine-programs-and-are-there-any-risks) 
and tried this:


sudo setcap cap_sys_ptrace=eip /tmp/hw

On both RedHat 6.5 and OpenSuse 12.3 and still get the same error 
message!!! :-/


Sorry, I am not a kernel expert...

What's wrong?

Thanks,

Eric

On 02/18/2015 04:48 PM, Éric Chamberland wrote:


Le 2015-02-18 15:14, Nathan Hjelm a écrit :

I recommend using vader for CMA. It has code to get around the ptrace
setting. Run with mca_btl_vader_single_copy_mechanism cma (should be the
default).

Ok, I tried it, but it gives exactly the same error message!

Eric


-Nathan

On Wed, Feb 18, 2015 at 02:56:01PM -0500, Eric Chamberland wrote:

Hi,

I have configured with "--with-cma" on 2 differents OS (RedHat 6.6 and
OpenSuse 12.3), but in both case, I have the following error when
launching
a simple mpi_hello_world.c example:

/opt/openmpi-1.8.4_cma/bin/mpiexec --mca btl_sm_use_cma 1 -np 2 /tmp/hw
--

WARNING: Linux kernel CMA support was requested via the
btl_vader_single_copy_mechanism MCA variable, but CMA support is
not available due to restrictive ptrace settings.

The vader shared memory BTL will fall back on another single-copy
mechanism if one is available. This may result in lower performance.

   Local host: compile
--

Hello world from process 0 of 2
Hello world from process 1 of 2
[compile:23874] 1 more process has sent help message
help-btl-vader.txt /
cma-permission-denied
[compile:23874] Set MCA parameter "orte_base_help_aggregate" to 0 to
see all
help / error messages

After I googled the subject, it seems there is a kernel parameter to
modify,
but I can't find it for OpenSuse 12.3 or RedHat 6.6...

Here is the "config.log" issued from RedHat 6.6...

http://www.giref.ulaval.ca/~ericc/ompi_bug/config.184_cma.gz

Thanks,

Eric
___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2015/02/26339.php


___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2015/02/26342.php

Re: [OMPI users] Help on getting CMA works

[Éric Chamberland]

Le 2015-02-18 15:14, Nathan Hjelm a écrit :

I recommend using vader for CMA. It has code to get around the ptrace
setting. Run with mca_btl_vader_single_copy_mechanism cma (should be the
default).

Ok, I tried it, but it gives exactly the same error message!

Eric


-Nathan

On Wed, Feb 18, 2015 at 02:56:01PM -0500, Eric Chamberland wrote:

Hi,

I have configured with "--with-cma" on 2 differents OS (RedHat 6.6 and
OpenSuse 12.3), but in both case, I have the following error when launching
a simple mpi_hello_world.c example:

/opt/openmpi-1.8.4_cma/bin/mpiexec --mca btl_sm_use_cma 1 -np 2 /tmp/hw
--
WARNING: Linux kernel CMA support was requested via the
btl_vader_single_copy_mechanism MCA variable, but CMA support is
not available due to restrictive ptrace settings.

The vader shared memory BTL will fall back on another single-copy
mechanism if one is available. This may result in lower performance.

   Local host: compile
--
Hello world from process 0 of 2
Hello world from process 1 of 2
[compile:23874] 1 more process has sent help message help-btl-vader.txt /
cma-permission-denied
[compile:23874] Set MCA parameter "orte_base_help_aggregate" to 0 to see all
help / error messages

After I googled the subject, it seems there is a kernel parameter to modify,
but I can't find it for OpenSuse 12.3 or RedHat 6.6...

Here is the "config.log" issued from RedHat 6.6...

http://www.giref.ulaval.ca/~ericc/ompi_bug/config.184_cma.gz

Thanks,

Eric
___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2015/02/26339.php

Re: [OMPI users] Help on getting CMA works

[Nathan Hjelm]

I recommend using vader for CMA. It has code to get around the ptrace
setting. Run with mca_btl_vader_single_copy_mechanism cma (should be the
default).

-Nathan

On Wed, Feb 18, 2015 at 02:56:01PM -0500, Eric Chamberland wrote:
> Hi,
> 
> I have configured with "--with-cma" on 2 differents OS (RedHat 6.6 and
> OpenSuse 12.3), but in both case, I have the following error when launching
> a simple mpi_hello_world.c example:
> 
> /opt/openmpi-1.8.4_cma/bin/mpiexec --mca btl_sm_use_cma 1 -np 2 /tmp/hw
> --
> WARNING: Linux kernel CMA support was requested via the
> btl_vader_single_copy_mechanism MCA variable, but CMA support is
> not available due to restrictive ptrace settings.
> 
> The vader shared memory BTL will fall back on another single-copy
> mechanism if one is available. This may result in lower performance.
> 
>   Local host: compile
> --
> Hello world from process 0 of 2
> Hello world from process 1 of 2
> [compile:23874] 1 more process has sent help message help-btl-vader.txt /
> cma-permission-denied
> [compile:23874] Set MCA parameter "orte_base_help_aggregate" to 0 to see all
> help / error messages
> 
> After I googled the subject, it seems there is a kernel parameter to modify,
> but I can't find it for OpenSuse 12.3 or RedHat 6.6...
> 
> Here is the "config.log" issued from RedHat 6.6...
> 
> http://www.giref.ulaval.ca/~ericc/ompi_bug/config.184_cma.gz
> 
> Thanks,
> 
> Eric
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/02/26339.php


pgpIGBg9JM3Ge.pgp
Description: PGP signature

[OMPI users] Help on getting CMA works

[Eric Chamberland]

Hi,

I have configured with "--with-cma" on 2 differents OS (RedHat 6.6 and 
OpenSuse 12.3), but in both case, I have the following error when 
launching a simple mpi_hello_world.c example:


/opt/openmpi-1.8.4_cma/bin/mpiexec --mca btl_sm_use_cma 1 -np 2 /tmp/hw
--
WARNING: Linux kernel CMA support was requested via the
btl_vader_single_copy_mechanism MCA variable, but CMA support is
not available due to restrictive ptrace settings.

The vader shared memory BTL will fall back on another single-copy
mechanism if one is available. This may result in lower performance.

  Local host: compile
--
Hello world from process 0 of 2
Hello world from process 1 of 2
[compile:23874] 1 more process has sent help message help-btl-vader.txt 
/ cma-permission-denied
[compile:23874] Set MCA parameter "orte_base_help_aggregate" to 0 to see 
all help / error messages


After I googled the subject, it seems there is a kernel parameter to 
modify, but I can't find it for OpenSuse 12.3 or RedHat 6.6...


Here is the "config.log" issued from RedHat 6.6...

http://www.giref.ulaval.ca/~ericc/ompi_bug/config.184_cma.gz

Thanks,

Eric