Nathan, I think I already pushed a patch for this particular issue last month. I do not know if it has been back ported to release yet.
See here:https://github.com/open-mpi/ompi/commit/ee3b0903164898750137d3b71a8f067e16521102 Aurelien -- ~~~ Aurélien Bouteiller, Ph.D. ~~~ ~ Research Scientist @ ICL ~ The University of Tennessee, Innovative Computing Laboratory 1122 Volunteer Blvd, suite 309, Knoxville, TN 37996 tel: +1 (865) 974-9375 fax: +1 (865) 974-8296 https://icl.cs.utk.edu/~bouteill/ > Le 19 févr. 2015 à 15:53, Nathan Hjelm <hje...@lanl.gov> a écrit : > > > Great! I will add an MCA variable to force CMA and also enable it if 1) > no yama and 2) no PR_SET_PTRACER. > > You might also look at using xpmem. You can find a version that supports > 3.x @ https://github.com/hjelmn/xpmem . It is a kernel module + > userspace library that can be used by vader as a single-copy mechanism. > > In benchmarks it performs better than CMA but it may or may not perform > better with a real application. > > See: > > http://blogs.cisco.com/performance/the-vader-shared-memory-transport-in-open-mpi-now-featuring-3-flavors-of-zero-copy > > -Nathan > > On Thu, Feb 19, 2015 at 03:32:43PM -0500, Eric Chamberland wrote: >> On 02/19/2015 02:58 PM, Nathan Hjelm wrote: >>> On Thu, Feb 19, 2015 at 12:16:49PM -0500, Eric Chamberland wrote: >>>> >>>> On 02/19/2015 11:56 AM, Nathan Hjelm wrote: >>>>> >>>>> If you have yama installed you can try: >>>> >>>> Nope, I do not have it installed... is it absolutely necessary? (and would >>>> it change something when it fails when I am root?) >>>> >>>> Other question: In addition to "--with-cma" configure flag, do we have to >>>> pass any options to "mpicc" when compiling/linking an mpi application to >>>> use >>>> cma? >>> >>> No. CMA should work out of the box. You appear to have a setup I haven't >>> yet tested. It doesn't have yama nor does it have the PR_SET_PTRACER >>> prctl. Its quite possible there are no restriction on ptrace in this >>> setup. Can you try changing the following line at >>> opal/mca/btl/vader/btl_vader_component.c:370 from: >>> >>> bool cma_happy = false; >>> >>> to >>> >>> bool cma_happy = true; >>> >> >> ok! (as of the officiel release, this is line 386.) >> >>> and let me know if that works. If it does I will update vader to allow >>> CMA in this configuration. >> >> Yep! It now works perfectly. Testing with >> https://computing.llnl.gov/tutorials/mpi/samples/C/mpi_bandwidth.c, on my >> own computer (dual Xeon), I have this: >> >> Without CMA: >> >> ***Message size: 1000000 *** best / avg / worst (MB/sec) >> task pair: 0 - 1: 8363.52 / 7946.77 / 5391.14 >> >> with CMA: >> task pair: 0 - 1: 9137.92 / 8955.98 / 7489.83 >> >> Great! >> >> Now I have to bench my real application... ;-) >> >> Thanks! >> >> Eric >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2015/02/26355.php > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2015/02/26356.php
