The system OpenMPI version runs fine without a link to psm2. Gunnar
2017-02-01 11:30 GMT+01:00 Ole Holm Nielsen <[email protected]>: > Hi Gunnar, > > For OmniPath networks you *must* install and use the PSM2 library. The > old PSM library is used for the previous product PathScale. The vanilla EB > files do not contain the PSM2 configurations, so you just have to add a > line or two. > > I already gave you information about how to build OpenMPI with EB by > adding configopts for PSM2 in the OpenMPI EB file. You need to follow > these instructions carefully. The OmniPath software is downloaded from > https://downloadcenter.intel.com/search?keyword=Omni-Path > > Best regards, > Ole > > > On 02/01/2017 11:23 AM, Gunnar Sauer wrote: > >> Thanks for your suggestions, Jure and Ole. I had checked that the >> OpenMPI version built by EasyBuild links to the same ibverbs and psm >> (libpsm_infinipath.so) versions as the system OpenMPI. The only >> difference is that the EasyBuild version links to hwloc, which the >> system version doesn't. And psm2 appears not to be available on the >> system. >> >> I understand that it is nearly impossible to figure out why the >> EasyBuild version has such a bad MPI latency performance, because I >> don't have the details of the system I am running on. For this reason I >> am giving up further EasyBuild testing until I have access to a >> different cluster with more available hardware+software information. >> >> If anybody involved in EasyBuild documentation is reading this, I'd >> appreciate if it were documented, which prerequisites the EasyBuild >> toolchains have with respect to the Infiniband installation and which >> configure options require checking prior to running the toolchain >> installation. I find it very confusing that different people either say >> that the foss toolchain takes care of an optimized OpenMPI installation, >> others say that I need to set up correct psm and ibverbs linking, and in >> the end it looks like the necessary options are included in the foss >> toolchain, but they don't give the expected performance. I feel kind of >> lost and give up for the moment. I'll return to testing EasyBuild in a >> few weeks or after my internship. >> >> Thank you >> Gunnar >> >> 2017-01-31 8:51 GMT+01:00 Ole Holm Nielsen <[email protected] >> <mailto:[email protected]>>: >> >> >> Hi, >> >> We use EasyBuild to provide OpenMPI in the "foss" toolchain. We use >> the Intel OmniPath fabric 100 Gbit/s. To build OpenMPI with >> OmniPath (and Slurm) support I simply add these files to the EB file: >> >> # Support of Slurm >> configopts += '--with-slurm --with-pmi=/usr/include/slurm >> --with-pmi-libdir=/usr ' >> # Support of OmniPath PSM, >> https://www.open-mpi.org/faq/?category=building#build-p2p >> <https://www.open-mpi.org/faq/?category=building#build-p2p> >> # Requires: yum install libpsm2 libpsm2-devel >> # configopts += '--with-psm --with-psm2 ' >> configopts += '--with-psm2 ' >> >> With this OpenMPI build I obtain a bandwidth of about 99 Gbit/s and >> a latency of 1.2 usec, using the OSU micro benchmarks. >> >> I chose to ignore the Intel-supplied RPM for openmpi that comes with >> their OmniPath support >> >> I have further details in my Wiki page: >> https://wiki.fysik.dtu.dk/niflheim/OmniPath#openmpi-configuration >> <https://wiki.fysik.dtu.dk/niflheim/OmniPath#openmpi-configuration> >> >> Best regards, >> Ole >> >> >> >> On 01/29/2017 02:51 PM, Gunnar Sauer wrote: >> >> Hello Kenneth, >> thanks for coming back to my question. I am sorry to say than I >> cannot >> follow the EasyBuild route anymore for the purpose of my >> internship, but >> I am definitely interested to solve the problems (described >> below) for >> myself and for my future career. (I'll have to buy access to >> some public >> cluster like Sabalcore or see whether I can work on our university >> cluster without a specific project. So it may take 1-2 weeks >> until I can >> proceed.) >> >> I have run the HPCC benchmark (version 1.5.0 from >> http://icl.cs.utk.edu/hpcc/software/index.html >> <http://icl.cs.utk.edu/hpcc/software/index.html>) once with the >> >> preinstalled gcc/openmpi/openblas of the company's Xeon cluster, >> and >> secondly with the foss/2016b toolchain built previously on the >> same >> cluster. This was meant as a quick check whether the forum users >> were >> right, saying that it doesn't matter for the MPI performance >> whether you >> use and optimized OpenMPI version or the generic EasyBuild >> OpenMPI built >> from source - or whether our engineers were right, saying that >> you have >> to use the system tools including an OpenMPI that has been set >> up for >> the Infiniband hardware if you want any decent MPI performance. >> >> When I presented the numbers below, showing ping pong latencies >> of 10000 >> us (EasyBuild) compared to 2 us (system tools), we had a quick >> discussion, and my task is now to write a build script >> independent from >> EasyBuild, respecting the existing tools. Here are the results >> of the >> ping pong test, first for the system tools (see also attached >> hpccoutf.system for the complete HPCC ouput), second for the >> foss/2016b >> toolchain (see also attached hpccoutf.eb): >> >> System compiler, openmpi, openblas: >> >> Major Benchmark results: >> ------------------------ >> >> Max Ping Pong Latency: 0.002115 msecs >> Randomly Ordered Ring Latency: 0.001384 msecs >> Min Ping Pong Bandwidth: 2699.150014 >> <tel:2699.150014> MB/s >> >> Naturally Ordered Ring Bandwidth: 549.443306 MB/s >> Randomly Ordered Ring Bandwidth: 508.267423 MB/s >> >> EasyBuild foss/2016b toolchain: >> >> Major Benchmark results: >> ------------------------ >> >> Max Ping Pong Latency: 10.000019 msecs >> Randomly Ordered Ring Latency: 4.251704 msecs >> Min Ping Pong Bandwidth: 62.532243 MB/s >> Naturally Ordered Ring Bandwidth: 134.390539 MB/s >> Randomly Ordered Ring Bandwidth: 144.071750 MB/s >> >> I'd appreciate if somebody could analyze the attached full >> outputs to >> suggest what I have done wrong. >> >> These results are along the lines of previous tests of mine that >> showed >> that EasyBuilds toolchain is not practical to solve large linear >> equation systems that involve more than one node. As soon as >> inter-node >> communication is involved, performance drops from 10-100 Gflop/s >> to >> 0.1-0.01 Gflop/s and gets worse the more nodes are involved even >> if the >> system is scaled to fill always 80% of the nodes' memory. >> >> These are just my initial discouraging attempts with EasyBuild. >> I'll be >> happy to find out that the performance problem is due to my >> mistake, >> because I might not have found the relevant documentation, >> forgotten to >> set some compiler flag or something else. >> >> Thank you >> Gunnar >> >> >> 2017-01-28 17:38 GMT+01:00 Kenneth Hoste <[email protected] >> <mailto:[email protected]> >> <mailto:[email protected] <mailto:[email protected]>>>: >> >> Hi Gunnar, >> >> On 25/01/2017 19:08, Gunnar Sauer wrote: >> >> Hello Jens, >> >> 2017-01-25 13:03 GMT+01:00 Jens Timmerman >> <[email protected] <mailto:[email protected]> >> <mailto:[email protected] >> >> <mailto:[email protected]>>>: >> >> >> Hello Gunnar, >> >> >> On 24/01/2017 19:54, Gunnar Sauer wrote: >> > Hello EasyBuild experts, >> > >> >> > But which toolchain do I choose on the Xeon >> cluster, which >> provides >> > all those optimized tools through already existing >> modules? >> Can I >> > tweak the goolf toolchain to use the existing >> system modules? >> Yes, you could create your own toolchain to use the >> already >> existing >> modules, this is exactly how the Cray toolchain >> works, see >> >> http://easybuild.readthedocs.io/en/latest/Using_external_mod >> ules.html >> <http://easybuild.readthedocs.io/en/latest/Using_external_mo >> dules.html> >> >> <http://easybuild.readthedocs.io/en/latest/Using_external_mo >> dules.html >> <http://easybuild.readthedocs.io/en/latest/Using_external_mo >> dules.html>> >> for more information on how to create your own >> toolchain from >> existing >> compilers and libraries. >> >> >> Ok, I'll try to understand the details how to set up a new >> toolchain and go this path. I have found the GCC-system, >> which >> seems to lead in the right direction. Would it be >> feasible to >> extend GCC-system to include OpenMPI-system and >> OpenBLAS-system in >> a similar fashion? >> >> >> The GCC-system easyconfig file leverages the SystemCompiler >> easyblock. >> >> To also support OpenMPI-system and OpenBLAS-system, a similar >> SystemLibrary easyblock should be created that forces you to >> specify >> the required information about the system library you would >> like to use. >> >> Alan's suggestion of just grabbing the module file that is >> generated >> using "--module-only --force" and adjusting it as needed is >> a good >> one though, it may take you a long way... >> >> >> >> And yes, these toolchains have infiniband support. >> >> So, it would be very nice to know what optimizations >> are being >> done at >> your company that make the internal toolchain even >> better >> optimized, so >> all EasyBuild >> users could all benefit from this knowledge and >> potentially >> millions of >> CPU hours could be saved. >> >> >> I will see, whether they share the details with me, or >> if they >> even have the details. As I understood, the cluster has >> been set >> up and is maintained by an external company. When we >> discussed >> today using the foss stack, I only got very discouraging >> answers: >> infiniband couldn't be configured correctly using a >> generic MPI >> installation procedure, BLAS would be an order of >> magnitude slower >> unless you put in the correct parameters for the specific >> architecture, etc. >> Nevertheless, I am currently trying to set up the HPL >> benchmark, >> and I will compare the results with easybuild's foss >> toolchain and >> with the cluster's 'builtin' toolchain. >> >> >> I'd very interested in hearing more about this, i.e. how the >> benchmark results turned out, how the existing toolchains were >> configured compared to how we tackle things in EasyBuild, etc. >> >> It's certainly possible that there was some heavy tuning >> done w.r.t. >> configuration parameters (in particular for the MPI); the >> downside >> of the easyconfigs we include in EasyBuild is that we need >> to keep >> them generic enough so that they'll work out of the box. >> For OpenMPI specifically, it makes a lot of sense to tweak the >> corresponding easyconfig file with additional/different >> system-specific configure options. >> >> >> I'm really serious here, if you can share this >> information, we >> would >> love to hear it so we can incorporate, but I do >> understand >> that this >> might be proprietary information. >> >> TL;DR: >> If you can share your highly optimized toolchains >> with us we >> will be >> pleased to support them in EasyBuild if they can help >> us >> getting faster >> software runtimes! >> >> >> Also thanks for the other replies! I need to gain some >> more >> experience with EasyBuild before I can make use of all >> your >> suggestions. >> >> >> Don't hesitate to let us know if you have any questions! >> >> >> >> regards, >> >> Kenneth >> >> >> >> -- >> Ole Holm Nielsen >> PhD, Manager of IT services >> Department of Physics, Technical University of Denmark >> >> >> >
