Thanks for your suggestions, Jure and Ole. I had checked that the
OpenMPI version built by EasyBuild links to the same ibverbs and psm
(libpsm_infinipath.so) versions as the system OpenMPI. The only
difference is that the EasyBuild version links to hwloc, which the
system version doesn't. And psm2 appears not to be available on the system.
I understand that it is nearly impossible to figure out why the
EasyBuild version has such a bad MPI latency performance, because I
don't have the details of the system I am running on. For this reason I
am giving up further EasyBuild testing until I have access to a
different cluster with more available hardware+software information.
If anybody involved in EasyBuild documentation is reading this, I'd
appreciate if it were documented, which prerequisites the EasyBuild
toolchains have with respect to the Infiniband installation and which
configure options require checking prior to running the toolchain
installation. I find it very confusing that different people either say
that the foss toolchain takes care of an optimized OpenMPI installation,
others say that I need to set up correct psm and ibverbs linking, and in
the end it looks like the necessary options are included in the foss
toolchain, but they don't give the expected performance. I feel kind of
lost and give up for the moment. I'll return to testing EasyBuild in a
few weeks or after my internship.
Thank you
Gunnar
2017-01-31 8:51 GMT+01:00 Ole Holm Nielsen <ole.h.niel...@fysik.dtu.dk
<mailto:ole.h.niel...@fysik.dtu.dk>>:
Hi,
We use EasyBuild to provide OpenMPI in the "foss" toolchain. We use
the Intel OmniPath fabric 100 Gbit/s. To build OpenMPI with
OmniPath (and Slurm) support I simply add these files to the EB file:
# Support of Slurm
configopts += '--with-slurm --with-pmi=/usr/include/slurm
--with-pmi-libdir=/usr '
# Support of OmniPath PSM,
https://www.open-mpi.org/faq/?category=building#build-p2p
<https://www.open-mpi.org/faq/?category=building#build-p2p>
# Requires: yum install libpsm2 libpsm2-devel
# configopts += '--with-psm --with-psm2 '
configopts += '--with-psm2 '
With this OpenMPI build I obtain a bandwidth of about 99 Gbit/s and
a latency of 1.2 usec, using the OSU micro benchmarks.
I chose to ignore the Intel-supplied RPM for openmpi that comes with
their OmniPath support
I have further details in my Wiki page:
https://wiki.fysik.dtu.dk/niflheim/OmniPath#openmpi-configuration
<https://wiki.fysik.dtu.dk/niflheim/OmniPath#openmpi-configuration>
Best regards,
Ole
On 01/29/2017 02:51 PM, Gunnar Sauer wrote:
Hello Kenneth,
thanks for coming back to my question. I am sorry to say than I
cannot
follow the EasyBuild route anymore for the purpose of my
internship, but
I am definitely interested to solve the problems (described
below) for
myself and for my future career. (I'll have to buy access to
some public
cluster like Sabalcore or see whether I can work on our university
cluster without a specific project. So it may take 1-2 weeks
until I can
proceed.)
I have run the HPCC benchmark (version 1.5.0 from
http://icl.cs.utk.edu/hpcc/software/index.html
<http://icl.cs.utk.edu/hpcc/software/index.html>) once with the
preinstalled gcc/openmpi/openblas of the company's Xeon cluster, and
secondly with the foss/2016b toolchain built previously on the same
cluster. This was meant as a quick check whether the forum users
were
right, saying that it doesn't matter for the MPI performance
whether you
use and optimized OpenMPI version or the generic EasyBuild
OpenMPI built
from source - or whether our engineers were right, saying that
you have
to use the system tools including an OpenMPI that has been set
up for
the Infiniband hardware if you want any decent MPI performance.
When I presented the numbers below, showing ping pong latencies
of 10000
us (EasyBuild) compared to 2 us (system tools), we had a quick
discussion, and my task is now to write a build script
independent from
EasyBuild, respecting the existing tools. Here are the results
of the
ping pong test, first for the system tools (see also attached
hpccoutf.system for the complete HPCC ouput), second for the
foss/2016b
toolchain (see also attached hpccoutf.eb):
System compiler, openmpi, openblas:
Major Benchmark results:
------------------------
Max Ping Pong Latency: 0.002115 msecs
Randomly Ordered Ring Latency: 0.001384 msecs
Min Ping Pong Bandwidth: 2699.150014
<tel:2699.150014> MB/s
Naturally Ordered Ring Bandwidth: 549.443306 MB/s
Randomly Ordered Ring Bandwidth: 508.267423 MB/s
EasyBuild foss/2016b toolchain:
Major Benchmark results:
------------------------
Max Ping Pong Latency: 10.000019 msecs
Randomly Ordered Ring Latency: 4.251704 msecs
Min Ping Pong Bandwidth: 62.532243 MB/s
Naturally Ordered Ring Bandwidth: 134.390539 MB/s
Randomly Ordered Ring Bandwidth: 144.071750 MB/s
I'd appreciate if somebody could analyze the attached full
outputs to
suggest what I have done wrong.
These results are along the lines of previous tests of mine that
showed
that EasyBuilds toolchain is not practical to solve large linear
equation systems that involve more than one node. As soon as
inter-node
communication is involved, performance drops from 10-100 Gflop/s to
0.1-0.01 Gflop/s and gets worse the more nodes are involved even
if the
system is scaled to fill always 80% of the nodes' memory.
These are just my initial discouraging attempts with EasyBuild.
I'll be
happy to find out that the performance problem is due to my mistake,
because I might not have found the relevant documentation,
forgotten to
set some compiler flag or something else.
Thank you
Gunnar
2017-01-28 17:38 GMT+01:00 Kenneth Hoste <kenneth.ho...@ugent.be
<mailto:kenneth.ho...@ugent.be>
<mailto:kenneth.ho...@ugent.be <mailto:kenneth.ho...@ugent.be>>>:
Hi Gunnar,
On 25/01/2017 19:08, Gunnar Sauer wrote:
Hello Jens,
2017-01-25 13:03 GMT+01:00 Jens Timmerman
<jens.timmer...@ugent.be <mailto:jens.timmer...@ugent.be>
<mailto:jens.timmer...@ugent.be
<mailto:jens.timmer...@ugent.be>>>:
Hello Gunnar,
On 24/01/2017 19:54, Gunnar Sauer wrote:
> Hello EasyBuild experts,
>
> But which toolchain do I choose on the Xeon
cluster, which
provides
> all those optimized tools through already existing
modules?
Can I
> tweak the goolf toolchain to use the existing
system modules?
Yes, you could create your own toolchain to use the
already
existing
modules, this is exactly how the Cray toolchain
works, see
http://easybuild.readthedocs.io/en/latest/Using_external_modules.html
<http://easybuild.readthedocs.io/en/latest/Using_external_modules.html>
<http://easybuild.readthedocs.io/en/latest/Using_external_modules.html
<http://easybuild.readthedocs.io/en/latest/Using_external_modules.html>>
for more information on how to create your own
toolchain from
existing
compilers and libraries.
Ok, I'll try to understand the details how to set up a new
toolchain and go this path. I have found the GCC-system,
which
seems to lead in the right direction. Would it be
feasible to
extend GCC-system to include OpenMPI-system and
OpenBLAS-system in
a similar fashion?
The GCC-system easyconfig file leverages the SystemCompiler
easyblock.
To also support OpenMPI-system and OpenBLAS-system, a similar
SystemLibrary easyblock should be created that forces you to
specify
the required information about the system library you would
like to use.
Alan's suggestion of just grabbing the module file that is
generated
using "--module-only --force" and adjusting it as needed is
a good
one though, it may take you a long way...
And yes, these toolchains have infiniband support.
So, it would be very nice to know what optimizations
are being
done at
your company that make the internal toolchain even
better
optimized, so
all EasyBuild
users could all benefit from this knowledge and
potentially
millions of
CPU hours could be saved.
I will see, whether they share the details with me, or
if they
even have the details. As I understood, the cluster has
been set
up and is maintained by an external company. When we
discussed
today using the foss stack, I only got very discouraging
answers:
infiniband couldn't be configured correctly using a
generic MPI
installation procedure, BLAS would be an order of
magnitude slower
unless you put in the correct parameters for the specific
architecture, etc.
Nevertheless, I am currently trying to set up the HPL
benchmark,
and I will compare the results with easybuild's foss
toolchain and
with the cluster's 'builtin' toolchain.
I'd very interested in hearing more about this, i.e. how the
benchmark results turned out, how the existing toolchains were
configured compared to how we tackle things in EasyBuild, etc.
It's certainly possible that there was some heavy tuning
done w.r.t.
configuration parameters (in particular for the MPI); the
downside
of the easyconfigs we include in EasyBuild is that we need
to keep
them generic enough so that they'll work out of the box.
For OpenMPI specifically, it makes a lot of sense to tweak the
corresponding easyconfig file with additional/different
system-specific configure options.
I'm really serious here, if you can share this
information, we
would
love to hear it so we can incorporate, but I do
understand
that this
might be proprietary information.
TL;DR:
If you can share your highly optimized toolchains
with us we
will be
pleased to support them in EasyBuild if they can help us
getting faster
software runtimes!
Also thanks for the other replies! I need to gain some more
experience with EasyBuild before I can make use of all your
suggestions.
Don't hesitate to let us know if you have any questions!
regards,
Kenneth
--
Ole Holm Nielsen
PhD, Manager of IT services
Department of Physics, Technical University of Denmark
