Hello Kenneth,
thanks for coming back to my question. I am sorry to say than I cannot
follow the EasyBuild route anymore for the purpose of my internship, but
I am definitely interested to solve the problems (described below) for
myself and for my future career. (I'll have to buy access to some public
cluster like Sabalcore or see whether I can work on our university
cluster without a specific project. So it may take 1-2 weeks until I can
proceed.)
I have run the HPCC benchmark (version 1.5.0 from
http://icl.cs.utk.edu/hpcc/software/index.html) once with the
preinstalled gcc/openmpi/openblas of the company's Xeon cluster, and
secondly with the foss/2016b toolchain built previously on the same
cluster. This was meant as a quick check whether the forum users were
right, saying that it doesn't matter for the MPI performance whether you
use and optimized OpenMPI version or the generic EasyBuild OpenMPI built
from source - or whether our engineers were right, saying that you have
to use the system tools including an OpenMPI that has been set up for
the Infiniband hardware if you want any decent MPI performance.
When I presented the numbers below, showing ping pong latencies of 10000
us (EasyBuild) compared to 2 us (system tools), we had a quick
discussion, and my task is now to write a build script independent from
EasyBuild, respecting the existing tools. Here are the results of the
ping pong test, first for the system tools (see also attached
hpccoutf.system for the complete HPCC ouput), second for the foss/2016b
toolchain (see also attached hpccoutf.eb):
System compiler, openmpi, openblas:
Major Benchmark results:
------------------------
Max Ping Pong Latency: 0.002115 msecs
Randomly Ordered Ring Latency: 0.001384 msecs
Min Ping Pong Bandwidth: 2699.150014 MB/s
Naturally Ordered Ring Bandwidth: 549.443306 MB/s
Randomly Ordered Ring Bandwidth: 508.267423 MB/s
EasyBuild foss/2016b toolchain:
Major Benchmark results:
------------------------
Max Ping Pong Latency: 10.000019 msecs
Randomly Ordered Ring Latency: 4.251704 msecs
Min Ping Pong Bandwidth: 62.532243 MB/s
Naturally Ordered Ring Bandwidth: 134.390539 MB/s
Randomly Ordered Ring Bandwidth: 144.071750 MB/s
I'd appreciate if somebody could analyze the attached full outputs to
suggest what I have done wrong.
These results are along the lines of previous tests of mine that showed
that EasyBuilds toolchain is not practical to solve large linear
equation systems that involve more than one node. As soon as inter-node
communication is involved, performance drops from 10-100 Gflop/s to
0.1-0.01 Gflop/s and gets worse the more nodes are involved even if the
system is scaled to fill always 80% of the nodes' memory.
These are just my initial discouraging attempts with EasyBuild. I'll be
happy to find out that the performance problem is due to my mistake,
because I might not have found the relevant documentation, forgotten to
set some compiler flag or something else.
Thank you
Gunnar
2017-01-28 17:38 GMT+01:00 Kenneth Hoste <[email protected]
<mailto:[email protected]>>:
Hi Gunnar,
On 25/01/2017 19:08, Gunnar Sauer wrote:
Hello Jens,
2017-01-25 13:03 GMT+01:00 Jens Timmerman <[email protected]
<mailto:[email protected]>>:
Hello Gunnar,
On 24/01/2017 19:54, Gunnar Sauer wrote:
> Hello EasyBuild experts,
>
> But which toolchain do I choose on the Xeon cluster, which
provides
> all those optimized tools through already existing modules?
Can I
> tweak the goolf toolchain to use the existing system modules?
Yes, you could create your own toolchain to use the already
existing
modules, this is exactly how the Cray toolchain works, see
http://easybuild.readthedocs.io/en/latest/Using_external_modules.html
<http://easybuild.readthedocs.io/en/latest/Using_external_modules.html>
for more information on how to create your own toolchain from
existing
compilers and libraries.
Ok, I'll try to understand the details how to set up a new
toolchain and go this path. I have found the GCC-system, which
seems to lead in the right direction. Would it be feasible to
extend GCC-system to include OpenMPI-system and OpenBLAS-system in
a similar fashion?
The GCC-system easyconfig file leverages the SystemCompiler easyblock.
To also support OpenMPI-system and OpenBLAS-system, a similar
SystemLibrary easyblock should be created that forces you to specify
the required information about the system library you would like to use.
Alan's suggestion of just grabbing the module file that is generated
using "--module-only --force" and adjusting it as needed is a good
one though, it may take you a long way...
And yes, these toolchains have infiniband support.
So, it would be very nice to know what optimizations are being
done at
your company that make the internal toolchain even better
optimized, so
all EasyBuild
users could all benefit from this knowledge and potentially
millions of
CPU hours could be saved.
I will see, whether they share the details with me, or if they
even have the details. As I understood, the cluster has been set
up and is maintained by an external company. When we discussed
today using the foss stack, I only got very discouraging answers:
infiniband couldn't be configured correctly using a generic MPI
installation procedure, BLAS would be an order of magnitude slower
unless you put in the correct parameters for the specific
architecture, etc.
Nevertheless, I am currently trying to set up the HPL benchmark,
and I will compare the results with easybuild's foss toolchain and
with the cluster's 'builtin' toolchain.
I'd very interested in hearing more about this, i.e. how the
benchmark results turned out, how the existing toolchains were
configured compared to how we tackle things in EasyBuild, etc.
It's certainly possible that there was some heavy tuning done w.r.t.
configuration parameters (in particular for the MPI); the downside
of the easyconfigs we include in EasyBuild is that we need to keep
them generic enough so that they'll work out of the box.
For OpenMPI specifically, it makes a lot of sense to tweak the
corresponding easyconfig file with additional/different
system-specific configure options.
I'm really serious here, if you can share this information, we
would
love to hear it so we can incorporate, but I do understand
that this
might be proprietary information.
TL;DR:
If you can share your highly optimized toolchains with us we
will be
pleased to support them in EasyBuild if they can help us
getting faster
software runtimes!
Also thanks for the other replies! I need to gain some more
experience with EasyBuild before I can make use of all your
suggestions.
Don't hesitate to let us know if you have any questions!
regards,
Kenneth
