Howdy,
On our mixed-node x86_64 cluster, we have Ivy Bridge
for the login and most (1000+) compute nodes and Westmere
for the 10 or so large memory nodes.
Dummying down all builds for the Westmere (with no AVX)
just didn't seem like a good idea.
So, on that system (ada), we usually set EASYBUILD_PREFIX
to /software/easybuild so that Ivy Bridge software/modules
end up in that directory. We also have some Westmere modules
(for development and users) that sets EASYBUILD_INSTALLDIR
to /software/easybuild/Westmere (Note: I'm using a similar approach
on our BG/Q to distinguish between CNK/bgsys (compute nodes) and
non-CNK/ppc64 (login nodes) builds.
For our users we have:
Westmere/1 - simply adds (using Lmod's path priority feature)
/software/easybuild/Westmere/modules/all
to the MODULEPATH i.e.
prepend_path{"MODULEPATH","/software/easybuild/Westmere/modules/all",priority=100}
For our developers, we currently have the following modules
EasyBuild/2.1.1 # in /software/easybuild - stock EB install (built by EB)
EasyBuild/2.1.1-tamusc # in /sofware/easybuild - stock EB install (built
by EB)
with some files (framework and easyblocks) overridden with
symlinks to our devel tree
EasyBuild-ada/2.1.1 # in /software/tamusc.. loads 2.1.1. above and sets
up environment (stock EB)
Easybuild-tamusc-ada/2.1.1 # in /software/tamus - loada 2.1.1-tamusc
above and sets up env (modified EB)
EasyBuild-ada-Westmere # loads EasyBuild-tamusc-ada and Westmere above
and sets
EASYBUILD_INSTALLPATH to /software/easybuild/Westmere
At some point, I was working on some module code so that
you couldn't load the last of the above while on an AVX-enabled
(e.g. Ivy Bridge) node, thus preventing accidentally polluting
the Westmere tree. But, for now, we just have to remember to
login to one of the Westmere nodes when building for that model.
Then I just use EB as is without having to tailor each individual
easyconfig (or easyblock) for the CPU model in question. Ivy Bridge
builds are optimized for Ivy Bridge and Westmere builds are optimized
for Westmere.
It may be a little more work at the outset, and does require duplicate
builds, but its a lot easier once set up and requires much less effort
(than what you are describing) to build.
FWIW/YMMV... just another $.02 tossed in the air in case anyone wants a
penny or two...
Jack Perdue
Lead Systems Administrator
High Performance Research Computing
TAMU Division of Research
[email protected] http://sc.tamu.edu
SC Helpdesk: [email protected]
On 07/15/2015 03:37 AM, Martin wrote:
Hi,
thanks for the information. At the moment I would like to avoid
building different stacks. It seems my colleagues are opposed to that
options as they feel it introduces uneeded complexity.
Personally I plan to revisit that option later when there is more
confidence that this does actually work. Please if you can post the
slides somewhere, references like these always help then I can say:
"Look guys, these people have been doing it for years and it works.
Why not just try it?" :)
/Martin
On Tue, Jul 14, 2015 at 6:05 PM Pablo Escobar Lopez
<[email protected] <mailto:[email protected]>>
wrote:
Hi Martin,
openblas will ignore the optarch option in easybuild but you can
use something like this in the OpenBLAS easyconfig
buildopts = 'TARGET=SANDYBRIDGE BINARY=64 ' + threading + '
CC="$CC" FC="$F77"'
here you have the list of supported targets in openblas
https://github.com/xianyi/OpenBLAS/blob/develop/TargetList.txt
If you want to share the same openblas installation across all
your different cpu types you will need to use the target which
match your oldest cpu. Another approach is to maintain a different
software stack by cpu type as described in the slides I attach
(page 9)
regards,
Pablo.
2015-07-14 13:21 GMT+02:00 Martin <[email protected]
<mailto:[email protected]>>:
Hi,
I have a couple of different CPUs and in my situation EB is
not so much about a HPC site or repeatable build but rather
about managing the stuff. So here's the thing:
When using R/3.2.0-foss-2015a-bare (just a simply
--try-software-version easyconfig) and subsequently installing
a few R packages will work on one host but fail on another.
The error is a classical illegal instruction in the underlying
OpenBLAS library (I could reproduce this specific problem
The problem is that OpenBLAS/0.2.13-GCC-4.9.2-LAPACK-3.5.0
seems to using some magic so that is is processor dependent.
The processors involved are:
* AMD Opteron(TM) Processor 6238 and
* Six-Core AMD Opteron(tm) Processor 8431
There are also a few Intel processors and a few other AMD CPUs
here. What I'm looking for is a setting for easybuild that
optimizes for portability between compatible CPU architectures.
Does anyone have suggestions how to achieve the most
compatibility and care less about byte-level reproducability
or performance? The target platform I have in mind is "64bit
x86 compatible".
I have optarch="" in my config.cfg where I thought that would
already take care of it.
Any thoughts would be appreciated.
/Martin
--
--
http://www.xing.com/profile/Martin_Marcher
http://www.linkedin.com/in/martinmarcher
Mobil: +43 / 660 / 62 45 103
<tel:%2B43%20%2F%20660%20%2F%C2%A062%2045%20103>
UID: ATU68801424
--
Pablo Escobar López
HPC systems engineer
Biozentrum, University of Basel
Swiss Institute of Bioinformatics SIB
Email: [email protected]
<mailto:[email protected]>
Phone: +41 61 267 21 80
http://www.biozentrum.unibas.ch
--
--
http://www.xing.com/profile/Martin_Marcher
http://www.linkedin.com/in/martinmarcher
Mobil: +43 / 660 / 62 45 103
UID: ATU68801424
