Hi Bart, On Dec 20, 2013, at 1:43 AM, Bart Verleye wrote: > /share/easybuild/installation: with the release version of EB installed. > /share/easybuild/"OS"/"Hardware": for the software and the module files
I was following the same scheme in the beginning and quickly realized that as time progresses there was no "one-true-buildset" and merged the above pieces, favoring multiple buildsets. (see below the output of ls) Partly, it was that one of our clusters was significantly heterogeneous and partly it was that EB releases turned old buildsets quickly obsolete. (btw. libreadline linked against ncurses was very educating experience last summer) Driven by the need to both fix bugs but not take the carpet under users' feet, we have ended up in a scheme of each EB merely becoming part of ~monthly buildsets: (`module load HPCBIOS/XYZ` will switch MODULEPATH and bring up the relevant buildset) Generally speaking, each buildset has a "preferred" EB version corresponding to it (and that should theoretically turn a buildset self-standing and, hence, reproducible) ``` $ ls -l /opt/apps/ total 16 lrwxrwxrwx 1 sw clusterusers 16 Aug 14 15:22 HPCBIOS -> HPCBIOS.20130715 drwxr-xr-x 5 sw clusterusers 54 Mar 1 2013 HPCBIOS.20130228 drwxr-xr-x 8 sw clusterusers 102 Mar 7 2013 HPCBIOS.20130301 drwxr-xr-x 6 sw clusterusers 68 Apr 25 2013 HPCBIOS.20130401 drwxr-xr-x 5 sw clusterusers 54 May 17 2013 HPCBIOS.20130501 drwxr-xr-x 5 sw clusterusers 54 Jun 1 2013 HPCBIOS.20130517 drwxr-xr-x 5 sw clusterusers 68 Jun 3 2013 HPCBIOS.20130601d drwxr-xr-x 5 sw clusterusers 68 Jun 3 2013 HPCBIOS.20130601e drwxr-xr-x 5 sw clusterusers 54 Jul 16 16:13 HPCBIOS.20130601h drwxr-xr-x 5 sw clusterusers 68 Aug 9 09:25 HPCBIOS.20130601s drwxr-xr-x 5 sw clusterusers 68 Aug 12 21:31 HPCBIOS.20130715 drwxr-xr-x 5 sw clusterusers 68 Oct 7 10:40 HPCBIOS.20131004 drwxr-xr-x 4 sw clusterusers 49 Dec 3 02:44 HPCBIOS.20131117 [...] drwxr-xr-x 27 sw clusterusers 8192 Aug 12 18:42 sources ``` Each directory above includes a "modules" & "software" subdir. The above scheme allows to cleanly jump from one set to another without surprising the users (normally, this happens at scheduled maintenance, by simply switching the symlink - our MODULEPATH is a constant over time!) N.B. "d", "e", "h", "s" corresponded to nodeset names and have a relationship to your current design about supporting multiple node architectures. I have dropped the scheme 'cause I realized that imkl really saved the day: it auto-tunes per node type and didn't want to waste more time on that. > /share/easybuild/src: where the source files will be kept. yes, sharing them in a parent dir as we also do above makes much sense. > /share/easybuild/git: a local respo of the develop (not an installation). I rather consider that business as software development, therefor a single repo wouldn't be my kind of answer. In fact, our "sw" user (others call it "build*") has a checkout of easyconfigs repo, which is used in read-only mode (ie. ALL the development activity and new features really happen in a full-fledged git style model elsewhere; and wherever we don't do it, we simply need to improve upon our processes ;-) In that respect, Xavier's work I find more suitable answer for development efforts in user space (or even for one-off experiments): https://github.com/hpcugent/easybuild-framework/blob/master/easybuild/scripts/install-EasyBuild-develop.sh > When an admin wants to install software, (s)he goes to the correct build > node, and sources the attached init file. This will set the environment. Intellectual question to you (and perhaps @.at fellows): * Have you considered turning most of init.sh into a *modulefile* instead? (some assumptions about shell initialization may be needed for that) > When local changes are made, e.g. paths to licences, this is done in the git > directory, and pushed to a branch 'Pan', never to be PR'd to the main repo. I think we do something similar here, we have the git branch uni.lu-customizations or so. > When new easyconfig/blocks have to be created, that is done in a fork of the > repo in the home directory of the admin. By contributing that work to the > develop, it will also be available in /share/easybuild/git and eventually in > the release version of EB. Eventual merge of the work towards upstream, makes sense, that's what we do. > Once the problem with group write settings will be solved, I think this > approach should work fine. > > There is also documentation is progress for this: > https://wiki.auckland.ac.nz/display/CER/Copy+of+Installing+software+with+EB+on+Pan We've noticed your worthy documentation efforts over there. Back in summer'12 I've shown to UGent fellows some similar idea on documentation per-application that could be as extensive as: http://eniac.cyi.ac.cy/display/UserDoc/GROMACS # eb deprecates "Compilation Instructions" :-) We have all come to agree that it would be great to share such effort and even given a name to that vaporware project: "EasyDoc". Not much has changed since then, but the need is still lurking around. I like Confluence's rendering but such effort would require a more open markup (git-compatible, too) which Confluence ditched after v3... to be continued. cheers, Fotis -- echo "sysadmin know better bash than english" | sed s/min/mins/ \ | sed 's/better bash/bash better/' # Yelling in a CERN forum
