On 23/11/2018 10:57, Ole Holm Nielsen wrote:
I completely agree with Loris! I have been wanting to hide irrelevant
modules from the listing of "module avail" for a very long time (the
curious users can always use "module spider"). The "module list" should
list all loaded modules, hidden or not.
At the recent Supercomputing 2018 conference I asked Davide Vanzo
(EasyBuild) and Robert McLay (Lmod) exactly this question, and Davide's
answer is that you must rebuild all of the module .lua files with
--module-only and --hide-deps. You *do not* have to rebuild the
software itself!
It's also possible to hide modules by configuring Lmod, see the
isVisibleHook function in
https://lmod.readthedocs.io/en/latest/170_hooks.html
I think Davide's solution of rebuilding the module files is the simplest
way forward. However, it isn't trivial to do correctly on a running
system, because you will be messing with the locations of modules in the
.lua files which could break user jobs. It is strongly recommended to
make a copy of the module tree and work on this without impacting the
production system. When completed, you can point users to the location
of the new module tree (on a shared file system, I presume).
This leads us to an interesting project for interested EasyBuild sites.
Can we design some tools to perform:
1. Create a configuration file with "system" module names (not end-user
applications) that we want to hide. My favorites include all module
names beginning with:
Autoconf, Automake, Autotools, Bison, CMake, LibTIFF, LibUUID, M4, Szip,
Tcl, Tk, Tkinter, XML-Parser, XZ, bzip2, binutils, cURL, expat, flex,
fontconfig, freetype, gettext, help2man, hwloc, lib*, ncurses, numactl,
pkg-config, tmux, util-linux, zlib
2. Write a script that traverses all directories in
$EASYBUILD_PREFIX/modules/all and rebuilds all the .lua files using "eb
--module-only --hide-deps". Maybe the above "system" module names must
be rebuilt with --hidden, whereas end-user applications must be rebuilt
with a --hide-deps list equal to the desired hidden modules. I'm unsure
of the correct logic here, but maybe EasyBuild experts can comment on this?
The "eb --help" explains how to hide modules:
--hidden Install 'hidden' module file(s) by prefixing their
version with '.' (def False)
--hide-deps=HIDE-DEPS Comma separated list of dependencies that you
want automatically hidden, (e.g. --hide-deps=zlib,ncurses)
(type comma-separated list)
Does anyone have the time and courage to work on a setup with such a
functionality?
Just to add my 2 cents: the support in EasyBuild to hide
dependencies/installations via --hide-deps / --hidden dates from before
Lmod had support for hiding modules via a .modulerc file.
Hiding modules via Lmod is a lot better than letting EasyBuild hide
modules (by letting the filename of the module file start with a '.', to
make it a classic hidden file in Linux, there's nothing more to it), for
a number of reasons:
* You can do it for existing software installations.
* It doesn't affect users in any (negative) way: the module names stay
exactly the same (i.e. no '.' is injected), so existing "module load"
commands will still work.
* You can roll back if you want to, for example if users start
complaining that modules that were available before have suddenly gone
missing (not too many people are aware of 'ml --show-hidden avail').
We should probably update the EasyBuild documentation on --hide-deps to
make that clear...
Anyone up for taking a stab at that?
regards,
Kenneth
Best regards,
Ole
On 11/23/2018 10:17 AM, Loris Bennett wrote:
Hi Sam,
Thanks for the info about rebuilding the module files.
Regarding the visibility of modules, I just want to exclude some from
the results of 'module av'. I assume 'module list' will still show all
the modules which are loaded, hidden or not. That doesn't bother me,
since, as far as I know, very few of our users look at the output of
'module list' anyway.
Cheers,
Loris
Sam Moors <sammo...@gmail.com> writes:
Hi Loris,
You can instruct EB to only rebuild the modules with the
'--module-only' option.
I don't know about your second question though.
That said, are you sure you want to hide the dependencies?
I think it is useful for the users to know which deps are loaded, so
they can avoid conflicts with other modules, even if it may look
daunting to see 50+ modules loaded in some cases.
Cheers,
Sam
On Fri, Nov 23, 2018 at 8:25 AM Loris Bennett
<loris.benn...@fu-berlin.de> wrote:
Hi,
Now that I have installed a couple of packages with easybuild I have a
surprising number of modules and realise that I should have been using
--hide-deps. I have two questions about this:
1. Is there any way to retrospectively hide modules, other than
tweaking
the module files by hand, say, rebuilding the modules?
2. Can a set of modules be defined which will be hidden by default?
Cheers,
Loris
--
Dr. Loris Bennett (Mr.)
ZEDAT, Freie Universität Berlin Email loris.benn...@fu-berlin.de
