Hi Trey,
On 22/08/14 20:47, Trey Dockendorf wrote:
I am giving EB a go as a fallback to my currently "beta" Lmod+manual install of
a couple dozen apps. I currently have the following
Do you have a full list of application, and which
compilers/MPI/BLAS/LAPACK you're aiming for?
MODULEPATH_ROOT=/apps/modulefiles # These are all .lua files
LMOD_PACKAGE_PATH=${MODULEPATH_ROOT}/Site # Contains SitePackage.lua
LMOD_PKG_ROOT=/apps
Example of some of the apps installed:
Core:
/apps/gcc/4.8.2
Compiler:
/apps/gcc-4.8.2/openmpi/1.8.1
/apps/gcc-4.8.2/mvapich2/2.0
One remark here: we generally strictly separate software install
directories from module install directories, i.e. I'd use
"/apps/software" and "/apps/modulefiles".
Not that I can think of a very good reason for that, however. ;-)
etc...
My Lua files make heavy use of helper functions in SitePackage.lua so that they
only contain the following plus whatever environment variables are needed:
Are you using helper function to avoid having to copy-paste-adjust
largely similar blocks across Lua module files, or because some
functionality is missing in Lmod that you need?
What I'm really asking is: which helper functions would you still need
if the (Lua) module files are generated automagically for you (e.g. by
EasyBuild)?
Compiler/gcc/4.8.2/mvapich2/2.0.lua:
local pkg = loadPkgDefaults(1)
setPkgInfo(pkg)
family("mpi")
<pushenv, setenv and prepend_path lines>
# pkg properties generated based on lua file location
prependModulePath(pathJoin("MPI", pkg.compiler, pkg.id))
My workflow has been something like the following:
1. Build app as user
2. Use 'sudo -E' to install
EasyBuild won't allow you to install stuff as root, since it's generally
a bad idea (who knows what those crazy scientists have come up with in
their code ;-)).
3. Add .lua file to my local modulefiles ($HOME/modulefiles) which is in Git.
3a. The lua files contain comment at the top with steps to build - not best way
to reproduce builds but better than no documentation
+1 on the latter
4. Push updated modulefiles repo to /home/admin/modulefiles (bare repo)
5. Become root and pull repo from /home/admin/modulefiles to /apps/modulefiles
The git workflow could be improved greatly using post-receive hooks. I hope to
add some post-receive hooks that update the Lmod spider cache and also write an
updated XML file that contains the module hierarchy and dependencies that is
ingested by our website.
I now want to evaluate EB 1.14.0 to see if I can completely replace my workflow
with EB. I'm concerned with how well I can merge my existed workflow with the
EB design. For example, I know Lua files are not yet supported but am aware
it's on ROADMAP, which is excellent.
The Lua module files are a blocker for now, yes, but we're on that, as
I've already mentioned to you. I'll bug Petar some more to get our hands
on his work w.r.t. adding support for generating Lua module files...
Depending on how heavily you would still need your custom helper
functions, we may need to provide hooks that you can tap into to get
your site-specific tweaks included as well, but that shouldn't be too
hard (we already support stuff like --modules-footer that 'modtclfooter'
easyconfig parameter).
Is it totally crazy or unwise to install software using EB then "convert" the
EB generated modulefiles into Lua and have users load modules that are not generated by
EB?
That seems rather involved, yes. Getting all the details right would be
tricky.
Would it then be necessary to build apps using those same Lua files or would EB
need to have the EB generated modulefiles in my MODULEPATH?
EasyBuild would need the module files it generates itself. Although
we're working towards removing that 'restriction' (part of
https://github.com/hpcugent/easybuild-framework/pull/1017 for example).
I have successfully built GCC/4.8.3, MVAPICH2/2.0 and OpenMPI/1.8.1 using EB into
subdirectories under my existing LMOD_PKG_ROOT. I had to make the following ENV changes
so that "module" commands would only show what EB has built.
module use /apps/easybuild/modules/all/Core
export MODULEPATH_ROOT=/apps/easybuild/modules/all
export LMOD_DEFAULT_MODULEPATH=/apps/easybuild/modules/all/Core
unset LMOD_PACKAGE_PATH
export MODULEPATH=/apps/easybuild/modules/all/Core
Is there a more sane way to use EB with Lmod but referencing a totally
different set of modules?
Generating one set of module files with EB, and then "translating" those
to the ones you expose to users isn't something we had to deal with
before, so it won't work.
I also think it's not a very good idea to do so (especially in the long
run), you'll run into all kinds of surprises...
Below is my current EB config. Right now only me and one other person would be building
software for use by our users. So I want to also make sure the permissions are such that
we can "destroy" each other's installs (hence the umask=002 in config below).
For a lack of proper documentation, see
https://github.com/hpcugent/easybuild-framework/pull/902, i.e. the PR in
which control over umask and friends were added.
So far everything installed is 2755/0644 except for the "easybuild" directories. I'm
wondering if other sites use a dedicated "easybuild" user for building software?
We've been using a dedicated user for building/installing software,
which we share with multiple people. Which is a bad idea, for practical
purposes. :-)
The proper way to do it is what you're suggesting: give users the
required permissions, and maybe also make them act under a common
'easybuild' (POSIX) group.
I'm hoping that I'm starting off in the right direction, and want to be sure
that what I'm doing is within scope of what EB expects and will be something I
can live with for next few years before we switch our entire cluster over to
this new setup (a couple weeks and counting :-/).
Thanks
- Trey
$ egrep -v '^$|^#' ~/.easybuild/config.cfg
[MAIN]
debug= True
[basic]
robot=/home/treydock/easybuild,/usr/easybuild/easyconfigs
[config]
buildpath=/opt/easybuild/treydock
Consider using `/dev/shm` here.
installpath=/apps/easybuild
module-naming-scheme=HierarchicalMNS
There be dragons here with EasyBuild v1.14.0 (see
https://github.com/hpcugent/easybuild-framework/issues/1011). The open
issues should hopefully be fixed by EasyBuild v1.15.0 (it's looking
promising, PRs are open for most of the issues, especially the most
important/tricky ones).
modules-tool=Lmod
sourcepath=/home/admin/src/el6
subdir-modules=modules
subdir-software=software
[informative]
[override]
umask=002
Check out --set-gid-bit, --sticky-bit and --group too.
[regtest]
[software]
[unittest]
=============================
Trey Dockendorf
Systems Analyst I
Texas A&M University
Academy for Advanced Telecommunications and Learning Technologies
Phone: (979)458-2396
Email: [email protected]
Jabber: [email protected]
regards,
Kenneth
