Thanks for your suggestions, Jure and Ole. I had checked that the OpenMPI version built by EasyBuild links to the same ibverbs and psm (libpsm_infinipath.so) versions as the system OpenMPI. The only difference is that the EasyBuild version links to hwloc, which the system version doesn't. And psm2 appears not to be available on the system.
I understand that it is nearly impossible to figure out why the EasyBuild version has such a bad MPI latency performance, because I don't have the details of the system I am running on. For this reason I am giving up further EasyBuild testing until I have access to a different cluster with more available hardware+software information. If anybody involved in EasyBuild documentation is reading this, I'd appreciate if it were documented, which prerequisites the EasyBuild toolchains have with respect to the Infiniband installation and which configure options require checking prior to running the toolchain installation. I find it very confusing that different people either say that the foss toolchain takes care of an optimized OpenMPI installation, others say that I need to set up correct psm and ibverbs linking, and in the end it looks like the necessary options are included in the foss toolchain, but they don't give the expected performance. I feel kind of lost and give up for the moment. I'll return to testing EasyBuild in a few weeks or after my internship. Thank you Gunnar 2017-01-31 8:51 GMT+01:00 Ole Holm Nielsen <[email protected]>: > Hi, > > We use EasyBuild to provide OpenMPI in the "foss" toolchain. We use the > Intel OmniPath fabric 100 Gbit/s. To build OpenMPI with OmniPath (and > Slurm) support I simply add these files to the EB file: > > # Support of Slurm > configopts += '--with-slurm --with-pmi=/usr/include/slurm > --with-pmi-libdir=/usr ' > # Support of OmniPath PSM, https://www.open-mpi.org/faq/? > category=building#build-p2p > # Requires: yum install libpsm2 libpsm2-devel > # configopts += '--with-psm --with-psm2 ' > configopts += '--with-psm2 ' > > With this OpenMPI build I obtain a bandwidth of about 99 Gbit/s and a > latency of 1.2 usec, using the OSU micro benchmarks. > > I chose to ignore the Intel-supplied RPM for openmpi that comes with their > OmniPath support > > I have further details in my Wiki page: > https://wiki.fysik.dtu.dk/niflheim/OmniPath#openmpi-configuration > > Best regards, > Ole > > > > On 01/29/2017 02:51 PM, Gunnar Sauer wrote: > >> Hello Kenneth, >> thanks for coming back to my question. I am sorry to say than I cannot >> follow the EasyBuild route anymore for the purpose of my internship, but >> I am definitely interested to solve the problems (described below) for >> myself and for my future career. (I'll have to buy access to some public >> cluster like Sabalcore or see whether I can work on our university >> cluster without a specific project. So it may take 1-2 weeks until I can >> proceed.) >> >> I have run the HPCC benchmark (version 1.5.0 from >> http://icl.cs.utk.edu/hpcc/software/index.html) once with the >> preinstalled gcc/openmpi/openblas of the company's Xeon cluster, and >> secondly with the foss/2016b toolchain built previously on the same >> cluster. This was meant as a quick check whether the forum users were >> right, saying that it doesn't matter for the MPI performance whether you >> use and optimized OpenMPI version or the generic EasyBuild OpenMPI built >> from source - or whether our engineers were right, saying that you have >> to use the system tools including an OpenMPI that has been set up for >> the Infiniband hardware if you want any decent MPI performance. >> >> When I presented the numbers below, showing ping pong latencies of 10000 >> us (EasyBuild) compared to 2 us (system tools), we had a quick >> discussion, and my task is now to write a build script independent from >> EasyBuild, respecting the existing tools. Here are the results of the >> ping pong test, first for the system tools (see also attached >> hpccoutf.system for the complete HPCC ouput), second for the foss/2016b >> toolchain (see also attached hpccoutf.eb): >> >> System compiler, openmpi, openblas: >> >> Major Benchmark results: >> ------------------------ >> >> Max Ping Pong Latency: 0.002115 msecs >> Randomly Ordered Ring Latency: 0.001384 msecs >> Min Ping Pong Bandwidth: 2699.150014 MB/s >> Naturally Ordered Ring Bandwidth: 549.443306 MB/s >> Randomly Ordered Ring Bandwidth: 508.267423 MB/s >> >> EasyBuild foss/2016b toolchain: >> >> Major Benchmark results: >> ------------------------ >> >> Max Ping Pong Latency: 10.000019 msecs >> Randomly Ordered Ring Latency: 4.251704 msecs >> Min Ping Pong Bandwidth: 62.532243 MB/s >> Naturally Ordered Ring Bandwidth: 134.390539 MB/s >> Randomly Ordered Ring Bandwidth: 144.071750 MB/s >> >> I'd appreciate if somebody could analyze the attached full outputs to >> suggest what I have done wrong. >> >> These results are along the lines of previous tests of mine that showed >> that EasyBuilds toolchain is not practical to solve large linear >> equation systems that involve more than one node. As soon as inter-node >> communication is involved, performance drops from 10-100 Gflop/s to >> 0.1-0.01 Gflop/s and gets worse the more nodes are involved even if the >> system is scaled to fill always 80% of the nodes' memory. >> >> These are just my initial discouraging attempts with EasyBuild. I'll be >> happy to find out that the performance problem is due to my mistake, >> because I might not have found the relevant documentation, forgotten to >> set some compiler flag or something else. >> >> Thank you >> Gunnar >> >> >> 2017-01-28 17:38 GMT+01:00 Kenneth Hoste <[email protected] >> <mailto:[email protected]>>: >> >> Hi Gunnar, >> >> On 25/01/2017 19:08, Gunnar Sauer wrote: >> >>> Hello Jens, >>> >>> 2017-01-25 13:03 GMT+01:00 Jens Timmerman <[email protected] >>> <mailto:[email protected]>>: >>> >>> >>> Hello Gunnar, >>> >>> >>> On 24/01/2017 19:54, Gunnar Sauer wrote: >>> > Hello EasyBuild experts, >>> > >>> >>> > But which toolchain do I choose on the Xeon cluster, which >>> provides >>> > all those optimized tools through already existing modules? >>> Can I >>> > tweak the goolf toolchain to use the existing system modules? >>> Yes, you could create your own toolchain to use the already >>> existing >>> modules, this is exactly how the Cray toolchain works, see >>> http://easybuild.readthedocs.io/en/latest/Using_external_mod >>> ules.html >>> <http://easybuild.readthedocs.io/en/latest/Using_external_mo >>> dules.html> >>> for more information on how to create your own toolchain from >>> existing >>> compilers and libraries. >>> >>> >>> Ok, I'll try to understand the details how to set up a new >>> toolchain and go this path. I have found the GCC-system, which >>> seems to lead in the right direction. Would it be feasible to >>> extend GCC-system to include OpenMPI-system and OpenBLAS-system in >>> a similar fashion? >>> >> >> The GCC-system easyconfig file leverages the SystemCompiler easyblock. >> >> To also support OpenMPI-system and OpenBLAS-system, a similar >> SystemLibrary easyblock should be created that forces you to specify >> the required information about the system library you would like to >> use. >> >> Alan's suggestion of just grabbing the module file that is generated >> using "--module-only --force" and adjusting it as needed is a good >> one though, it may take you a long way... >> >> >> >>> And yes, these toolchains have infiniband support. >>> >>> So, it would be very nice to know what optimizations are being >>> done at >>> your company that make the internal toolchain even better >>> optimized, so >>> all EasyBuild >>> users could all benefit from this knowledge and potentially >>> millions of >>> CPU hours could be saved. >>> >>> >>> I will see, whether they share the details with me, or if they >>> even have the details. As I understood, the cluster has been set >>> up and is maintained by an external company. When we discussed >>> today using the foss stack, I only got very discouraging answers: >>> infiniband couldn't be configured correctly using a generic MPI >>> installation procedure, BLAS would be an order of magnitude slower >>> unless you put in the correct parameters for the specific >>> architecture, etc. >>> Nevertheless, I am currently trying to set up the HPL benchmark, >>> and I will compare the results with easybuild's foss toolchain and >>> with the cluster's 'builtin' toolchain. >>> >> >> I'd very interested in hearing more about this, i.e. how the >> benchmark results turned out, how the existing toolchains were >> configured compared to how we tackle things in EasyBuild, etc. >> >> It's certainly possible that there was some heavy tuning done w.r.t. >> configuration parameters (in particular for the MPI); the downside >> of the easyconfigs we include in EasyBuild is that we need to keep >> them generic enough so that they'll work out of the box. >> For OpenMPI specifically, it makes a lot of sense to tweak the >> corresponding easyconfig file with additional/different >> system-specific configure options. >> >> >>> I'm really serious here, if you can share this information, we >>> would >>> love to hear it so we can incorporate, but I do understand >>> that this >>> might be proprietary information. >>> >>> TL;DR: >>> If you can share your highly optimized toolchains with us we >>> will be >>> pleased to support them in EasyBuild if they can help us >>> getting faster >>> software runtimes! >>> >>> >>> Also thanks for the other replies! I need to gain some more >>> experience with EasyBuild before I can make use of all your >>> suggestions. >>> >> >> Don't hesitate to let us know if you have any questions! >> >> >> >> regards, >> >> Kenneth >> >> >> > -- > Ole Holm Nielsen > PhD, Manager of IT services > Department of Physics, Technical University of Denmark >
