For what it's worth...
On 15/07/15 16:15, Jack Perdue wrote:
Howdy,
On our mixed-node x86_64 cluster, we have Ivy Bridge
for the login and most (1000+) compute nodes and Westmere
for the 10 or so large memory nodes.
Dummying down all builds for the Westmere (with no AVX)
just didn't seem like a good idea.
So, on that system (ada), we usually set EASYBUILD_PREFIX
to /software/easybuild so that Ivy Bridge software/modules
end up in that directory. We also have some Westmere modules
(for development and users) that sets EASYBUILD_INSTALLDIR
to /software/easybuild/Westmere (Note: I'm using a similar approach
on our BG/Q to distinguish between CNK/bgsys (compute nodes) and
non-CNK/ppc64 (login nodes) builds.
For our users we have:
Westmere/1 - simply adds (using Lmod's path priority feature)
/software/easybuild/Westmere/modules/all
to the MODULEPATH i.e.
prepend_path{"MODULEPATH","/software/easybuild/Westmere/modules/all",priority=100}
For our developers, we currently have the following modules
EasyBuild/2.1.1 # in /software/easybuild - stock EB install (built by EB)
EasyBuild/2.1.1-tamusc # in /sofware/easybuild - stock EB install
(built by EB)
with some files (framework and easyblocks) overridden with
symlinks to our devel tree
EasyBuild-ada/2.1.1 # in /software/tamusc.. loads 2.1.1. above and
sets up environment (stock EB)
Easybuild-tamusc-ada/2.1.1 # in /software/tamus - loada 2.1.1-tamusc
above and sets up env (modified EB)
EasyBuild-ada-Westmere # loads EasyBuild-tamusc-ada and Westmere above
and sets
EASYBUILD_INSTALLPATH to /software/easybuild/Westmere
At some point, I was working on some module code so that
you couldn't load the last of the above while on an AVX-enabled
(e.g. Ivy Bridge) node, thus preventing accidentally polluting
the Westmere tree. But, for now, we just have to remember to
login to one of the Westmere nodes when building for that model.
Then I just use EB as is without having to tailor each individual
easyconfig (or easyblock) for the CPU model in question. Ivy Bridge
builds are optimized for Ivy Bridge and Westmere builds are optimized
for Westmere.
It may be a little more work at the outset, and does require duplicate
builds, but its a lot easier once set up and requires much less effort
(than what you are describing) to build.
FWIW/YMMV... just another $.02 tossed in the air in case anyone wants
a penny or two...
Our setup is very similar (other than the different variants of EasyBuild).
Basically, our 'eb' command is aliased to something like 'eb
--installpath=/apps/gent/$OS_LOCAL/$ARCH_LOCAL' in the account we use to
install software with EasyBuild.
$OS_LOCAL and $ARCH_LOCAL are defined automagically via a script in
/etc/profile.d.
That way, we can simply submit jobs using "eb --job" to different
clusters (which got a major update in EB v2.2.0 btw), and be sure that
the installations end up where they need to be, and are specific to the
architecture/OS of the system you're on.
K.
