Hi Douglas, A little time ago, I had to consolidate to a "common experience" around a dozen of HPC clusters, spread globally. Inevitably, it included the complete spectrum of x86 CPUs released over a span of several years.
The long and short of it is: * Expert users will have a demand to provide their python/R/other custom (f.i. C++) codes, having some level of choice on their builds (think: some provide RPMs externally). * Any node before AVX microarchitecture either got phased out or setup in a special queue (i.e. avoid them) * Any node above it, was free to be used at will for runs, but builds were done in a well planned manner * The assumptions above worked well, because the majority of workloads have been I/O bound; rest handcrafted * By default, builds would go on AVX microarchitecture (f.i. ivybridge happened to be the common denominator) * Builds were done on dedicated queues and the supported scheme involved using SGE's features, as needed, to steer the builds where they should go, per microarchitecture. * On that last one, nodes have been tagged, each with a corresponding `cpuflags` tag. f.i. `:avx:avx2:avx512cd:avx512f:f16c:fma:lm:sse:sse4_2:` - if you have SGE, or not, copy from this, and it will allow you to do reliable substring search: https://github.com/easybuilders/easybuild/wiki/Conference-call-notes-20171011 ``` CPU_FLAGS_LIST="lm sse sse4_2 f16c avx avx2 avx512 avx512f avx512dq avx512cd avx512bw avx512vl eagerfpu epb ida epb fma" cpuflags="$(grep ^flags /proc/cpuinfo |uniq|cut -d: -f2-|xargs -n1|sort|comm -1 -2 - <(echo $CPU_FLAGS_LIST|xargs -n1|sort)|xargs|sed 's/ /:/g;s/$/:/g;s/^/:/g')" ``` That's all folks! it basically worked well, with only very minor maintenance and without annoying surprises. And the name of the game is to watch out your allocation of CPU horsepower and check if there is some user who needs further help to optimise code. enjoy - I hope this helps, F. -- echo "sysadmin know better bash than english" | sed s/min/mins/ \ | sed 's/better bash/bash better/' # Yelling in a CERN forum ________________________________________ From: [email protected] [[email protected]] on behalf of Douglas Scofield [[email protected]] Sent: Thursday, November 07, 2019 12:57 To: [email protected] Subject: [easybuild] EasyBuild in a heterogeneous HPC centre Hej, We are curious how to maintain architecture-specific EasyBuild trees. We are new to EasyBuild and already have many, many (mostly bioinformatics) tools installed in hand-maintained module and software trees. In our centre, we have clusters running Sandy Bridge EP, Haswell EP, and Broadwell EP. Most of our users are on Broadwell, so we if we compile with -march=native etc., we compile for Broadwell and for Sandy Bridge, which covers it and Haswell. In our standard module tree we manage architecture automatically, keying off a $Cluster variable set by the module system. We handle architectures as if we had modules versioned Tool/Version/Architecture, with the last bit hidden from the user. We have also decided to (mostly) hide our EasyBuild tree from the user, and instead provide access to EasyBuild-built tools using what we are calling alias modules, which we place in our standard module tree. An alias module performs a 'module use' of the easybuild tree and then loads the appropriate EasyBuild-built modules. The large majority of our users do not care about toolchains, etc. Those that do, we will have docs they can consult for working with EasyBuild modules directly. The very large majority of our installed tools do not currently have easyconfigs. We are guessing that our architecture solution with EasyBuild will end up being completely separate EasyBuild trees, accessed using distinct 'module use' paths. The EasyBuild docs point to a 2015 presentation by Pablo Escobar describing an automounter solution which we are definitely not going to implement, but this suggests completely separate trees as well. How is this typically handled by other centres ? Thanks in advance, Douglas — Douglas G. Scofield Evolutionary Biology Centre, Uppsala University [email protected] [email protected] När du har kontakt med oss på Uppsala universitet med e-post så innebär det att vi behandlar dina personuppgifter. För att läsa mer om hur vi gör det kan du läsa här: http://www.uu.se/om-uu/dataskydd-personuppgifter/ E-mailing Uppsala University means that we will process your personal data. For more information on how this is performed, please read here: http://www.uu.se/en/about-uu/data-protection-policy
