Hi Olaf, some extra bits from this side, too:
> There is no reason at all to not use multiple toolchains. Trying toolchains is cheap with easybuild, so it's recommended to check them out. If you don't care about Intel/Clang ones, you'll settle somewhere among goolf/goalf. >> (2) what if there is no eb for my toolchain? You just make one; if you have many packages to use over it, ok, that's a bit more work: it may be the case that you need to invent a for-loop that will rebuild eg. all "bio" packages over it. >> I noticed that for some programs, there are no goalf-1.1.0-no-OFED >> easybuilds. Amoung these, there are essentials like Perl. >> I understand that there will be conflicts if users try load a goalf-based >> module (say, Perl) and a goolf-based module (say Python) at the same time >> (zlib). Would you recommend to build Perl with goolf-1.4.10, or to create a >> Perl-5.16.3-goalf-1.1.0-no-OFED.eb myself? Maybe there is a reason that >> there is no such easyconfig? Should I trust the --try-options? Try --try :) > The goalf toolchain is more like a toy toolchain that should work on a wide > variety of platforms. N.B. goalf contains ATLAS, which is known to give trouble under certain situations (AVX cpus, VMs etc). If you can go with goolf as a first-timer, it may be preferred. > If there is config for a particular toolchain, you can either try using > --try-toolchain=<toolchain>,<version> or you copy the configs and adjust the > toolchain. Please not that dependencies are not automatically build with the > toolchain and as such there might be some effort to build a dependency graph > and rebuilt every package in there with --try-toolchain. I think we all resort to some scripting in such situation; easybuild does the heavy work, so you need to provide just some glue code on top. >> (3) OS Independence >> Besides the fact that easybuild bootstraps its own compiler, it still uses >> some OS ressources. For example, if I use GCC/4.6.3 to compile helloworld.c >> on a compute node (with no development tools installed), ld complains about >> crt1.o not being there. So I am wondering: To what extend are the resulting >> packages OS-independent? If I build on RHEL 6.4, will they run on RHEL 6.3, >> 6.5, RHEL 5 (or even Ubuntu)? > > Depends(™). For pure python packages this is for sure different than it is > for C++ where no ABI standard exists and as such people depend particular > compiler versions and the compiler implementation of name mangling etc. > In general EL6 is a rolling release and the GCC version is kept at the same > major release. Same goes for SLES. It also depends on how far you take it > with eb, there is actually no reason not to roll libc and base libraries as > part of dependencies if you want to take it that far. But a great thing about > EB is that you can abstract most details in the config or in the block - > things like what needs to be present as either system package dependency or > EB dependency - and simply recompile! > > We have module/software trees for the following distros > > *EL6-based (CentOS6.x, RHEL 6.x - were we have 6.2/6.3/6.4 in use) > *Ubuntu 12.04 > *SLES11 (in use SP1,SP2,SP3) N.B. RH gives guarantees for forward ABI compatibility with a major version (eg. RHEL6.x) and major-v backward compatibility, in my experience, is pretty fine most of the time. Everything else is a gamble, big or small depending on the context. >> (4) Best practice for new releases >> How would you deploy new releases of software, for example when a new >> release of easybuild comes out? I decided that a new release would go into a >> new EASYBUILD_INSTALLPATH, and I inform users to update their .modulerc to >> contain the new modulepath. Is this what you would recommend? > > We handle that with environment variables as well, so we basically have a > "production" directory and this is what our EASYBUILD_INSTALLPATH points to, > but this is just a symlink to a directory that just contains the date - like > 24-04-2013. > > -> production > -> 29-04-2013 > -> 10-09-2013 > -> 02-12-2013 > > > And then we just let production point to any one of them. We have a very similar scheme, the rules of the game are described below: ``` fgeorgatos@gaia-12:~: $ ls -als /opt/apps/ total 20 4 drwxr-xr-x 17 sw clusterusers 4096 Nov 17 23:15 . 4 drwxr-xr-x 6 root root 4096 Aug 2 18:00 .. 0 lrwxrwxrwx 1 sw clusterusers 16 Sep 28 10:30 HPCBIOS -> HPCBIOS.20130902 0 drwxr-xr-x 5 sw clusterusers 54 Mar 1 2013 HPCBIOS.20130226 0 drwxr-xr-x 6 sw clusterusers 68 May 8 2013 HPCBIOS.20130301 0 drwxr-xr-x 6 sw clusterusers 68 Mar 2 2013 HPCBIOS.20130302 0 drwxr-xr-x 6 sw clusterusers 68 Apr 1 2013 HPCBIOS.20130401 0 drwxr-xr-x 5 sw clusterusers 54 Jun 1 2013 HPCBIOS.20130501 0 drwxr-xr-x 5 sw clusterusers 54 Jul 15 20:23 HPCBIOS.20130601 0 drwxr-xr-x 6 sw clusterusers 110 Sep 28 10:29 HPCBIOS.20130715 0 drwxr-xr-x 5 sw clusterusers 68 Sep 26 18:31 HPCBIOS.20130902 0 drwxr-xr-x 5 sw clusterusers 68 Oct 6 01:04 HPCBIOS.20131004 0 drwxr-xr-x 6 sw clusterusers 68 Nov 17 23:23 HPCBIOS.20131117 0 drwxr-xr-x 5 sw clusterusers 72 Aug 9 09:09 OLD 0 drwxr-xr-x 4 sw clusterusers 35 Jun 6 2012 default 0 drwxr-xr-x 4 sw clusterusers 35 Mar 3 2013 experimental 12 drwxr-xr-x 27 sw clusterusers 8192 Aug 12 18:42 sources 0 drwxr-xr-x 7 sw clusterusers 80 Mar 3 2013 testing ``` * At each maintenance window, the tip (top symlink) is promoted to latest buildset * Buildsets (HPCBIOS.YYYYMMDD) are now roughly corresponding to EasyBuild releases * Buildsets are ~immutable (only modified if a hard bug turns them unusable) * Each buildset is nothing else than a workdir of EB, defined via EASYBUILD_PREFIX * New users see whichever most recent default tuning, during last maintenance * Buildsets are kept -nominally- for 12 months, then deprecated * users may jump in future or past (think `module load HPCBIOS/20131117`) * users needing continuity >12 months MUST provide their own user/group buildsets * some extra modulefiles/builds under "default", help the magic of all this. Logic of organization of modules (ie. MODULEPATH) is the classic unix style: * user/group/world What you read above was the "world" definition, user/groups are encouraged to deviate at will. Quite a few of our bioinformatics users ride on the idea and use modules/eb pretty well. We are still missing the "group" automation to be `module use`-defined, now it's manual. The model on our side will be fully implemented when we switch to Lmod, because then buildsets, say "HPCBIOS/20131117" would be preloaded/replaceable with "HPCBIOS/20131004" etc. (think of "bootstrapping MODULEPATH") notification to the rest in the list: Also, Lmod's sticky modules may allow to handle MUCH better the tuning of system-wide variables, esp. the ones that relate to PRACE or HPCBIOS business; not there yet, but looking forward to it! (think of http://hpcbios.readthedocs.org/en/latest/HPCBIOS_2012-98.html and similar cases) If all these come in place, it will be possible to provide the max of flexibility and ease-of-use. And still have your system tunable for many different contexts, in a way that does not suck! I think Peter & Aaron have already walked much of this road, AFAI remember. It makes sense. :) enjoy, Fotis -- echo "sysadmin know better bash than english" | sed s/min/mins/ \ | sed 's/better bash/bash better/' # Yelling in a CERN forum
