Great! I will add an MCA variable to force CMA and also enable it if 1) no yama and 2) no PR_SET_PTRACER.
You might also look at using xpmem. You can find a version that supports 3.x @ https://github.com/hjelmn/xpmem . It is a kernel module + userspace library that can be used by vader as a single-copy mechanism. In benchmarks it performs better than CMA but it may or may not perform better with a real application. See: http://blogs.cisco.com/performance/the-vader-shared-memory-transport-in-open-mpi-now-featuring-3-flavors-of-zero-copy -Nathan On Thu, Feb 19, 2015 at 03:32:43PM -0500, Eric Chamberland wrote: > On 02/19/2015 02:58 PM, Nathan Hjelm wrote: > >On Thu, Feb 19, 2015 at 12:16:49PM -0500, Eric Chamberland wrote: > >> > >>On 02/19/2015 11:56 AM, Nathan Hjelm wrote: > >>> > >>>If you have yama installed you can try: > >> > >>Nope, I do not have it installed... is it absolutely necessary? (and would > >>it change something when it fails when I am root?) > >> > >>Other question: In addition to "--with-cma" configure flag, do we have to > >>pass any options to "mpicc" when compiling/linking an mpi application to use > >>cma? > > > >No. CMA should work out of the box. You appear to have a setup I haven't > >yet tested. It doesn't have yama nor does it have the PR_SET_PTRACER > >prctl. Its quite possible there are no restriction on ptrace in this > >setup. Can you try changing the following line at > >opal/mca/btl/vader/btl_vader_component.c:370 from: > > > >bool cma_happy = false; > > > >to > > > >bool cma_happy = true; > > > > ok! (as of the officiel release, this is line 386.) > > >and let me know if that works. If it does I will update vader to allow > >CMA in this configuration. > > Yep! It now works perfectly. Testing with > https://computing.llnl.gov/tutorials/mpi/samples/C/mpi_bandwidth.c, on my > own computer (dual Xeon), I have this: > > Without CMA: > > ***Message size: 1000000 *** best / avg / worst (MB/sec) > task pair: 0 - 1: 8363.52 / 7946.77 / 5391.14 > > with CMA: > task pair: 0 - 1: 9137.92 / 8955.98 / 7489.83 > > Great! > > Now I have to bench my real application... ;-) > > Thanks! > > Eric > > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2015/02/26355.php
pgp4qiwgVsc3t.pgp
Description: PGP signature
