Re: [easybuild] Retrospectively hiding modules / hiding by default

[Kenneth Hoste]

On 23/11/2018 10:57, Ole Holm Nielsen wrote:
I completely agree with Loris!  I have been wanting to hide irrelevant 
modules from the listing of "module avail" for a very long time (the 
curious users can always use "module spider").  The "module list" should 
list all loaded modules, hidden or not.


At the recent Supercomputing 2018 conference I asked Davide Vanzo 
(EasyBuild) and Robert McLay (Lmod) exactly this question, and Davide's 
answer is that you must rebuild all of the module .lua files with 
--module-only and --hide-deps.  You *do not* have to rebuild the 
software itself!


It's also possible to hide modules by configuring Lmod, see the 
isVisibleHook function in 
https://lmod.readthedocs.io/en/latest/170_hooks.html


I think Davide's solution of rebuilding the module files is the simplest 
way forward.  However, it isn't trivial to do correctly on a running 
system, because you will be messing with the locations of modules in the 
.lua files which could break user jobs.  It is strongly recommended to 
make a copy of the module tree and work on this without impacting the 
production system.  When completed, you can point users to the location 
of the new module tree (on a shared file system, I presume).


This leads us to an interesting project for interested EasyBuild sites. 
Can we design some tools to perform:


1. Create a configuration file with "system" module names (not end-user 
applications) that we want to hide.  My favorites include all module 
names beginning with:
Autoconf, Automake, Autotools, Bison, CMake, LibTIFF, LibUUID, M4, Szip, 
Tcl, Tk, Tkinter, XML-Parser, XZ, bzip2, binutils, cURL, expat, flex, 
fontconfig, freetype, gettext, help2man, hwloc, lib*, ncurses, numactl, 
pkg-config, tmux, util-linux, zlib


2. Write a script that traverses all directories in 
$EASYBUILD_PREFIX/modules/all and rebuilds all the .lua files using "eb 
--module-only --hide-deps".  Maybe the above "system" module names must 
be rebuilt with --hidden, whereas end-user applications must be rebuilt 
with a --hide-deps list equal to the desired hidden modules.  I'm unsure 
of the correct logic here, but maybe EasyBuild experts can comment on this?


The "eb --help" explains how to hide modules:

  --hidden  Install 'hidden' module file(s) by prefixing their 
     version with '.' (def False)
  --hide-deps=HIDE-DEPS  Comma separated list of dependencies that you 
want automatically hidden, (e.g. --hide-deps=zlib,ncurses)  
(type comma-separated list)


Does anyone have the time and courage to work on a setup with such a 
functionality?


Just to add my 2 cents: the support in EasyBuild to hide 
dependencies/installations via --hide-deps / --hidden dates from before 
Lmod had support for hiding modules via a .modulerc file.


Hiding modules via Lmod is a lot better than letting EasyBuild hide 
modules (by letting the filename of the module file start with a '.', to 
make it a classic hidden file in Linux, there's nothing more to it), for 
a number of reasons:


* You can do it for existing software installations.

* It doesn't affect users in any (negative) way: the module names stay 
exactly the same (i.e. no '.' is injected), so existing "module load" 
commands will still work.


* You can roll back if you want to, for example if users start 
complaining that modules that were available before have suddenly gone 
missing (not too many people are aware of 'ml --show-hidden avail').



We should probably update the EasyBuild documentation on --hide-deps to 
make that clear...


Anyone up for taking a stab at that?


regards,

Kenneth




Best regards,
Ole


On 11/23/2018 10:17 AM, Loris Bennett wrote:

Hi Sam,

Thanks for the info about rebuilding the module files.

Regarding the visibility of modules, I just want to exclude some from
the results of 'module av'.  I assume 'module list' will still show all
the modules which are loaded, hidden or not.  That doesn't bother me,
since, as far as I know, very few of our users look at the output of
'module list' anyway.

Cheers,

Loris

Sam Moors  writes:


Hi Loris,

You can instruct EB to only rebuild the modules with the 
'--module-only' option.

I don't know about your second question though.

That said, are you sure you want to hide the dependencies?



I think it is useful for the users to know which deps are loaded, so
they can avoid conflicts with other modules, even if it may look
daunting to see 50+ modules loaded in some cases.

Cheers,
Sam

On Fri, Nov 23, 2018 at 8:25 AM Loris Bennett 
 wrote:


  Hi,

  Now that I have installed a couple of packages with easybuild I have a
  surprising number of modules and realise that I should have been using
  --hide-deps. I have two questions about this:

  1. Is there any way to retrospectively hide modules, other than 
tweaking

  the module files by hand, say, rebuilding the modules?
  2. Can a set of modules be defined which will be hidden by default?

  Cheers,

  Loris

  --
  Dr. 

Re: [easybuild] Retrospectively hiding modules / hiding by default

[Ole Holm Nielsen]

(Resending because it seems that my reply got trapped by our spam-filter).
On 11/23/2018 10:53 AM, Markus Geimer wrote:
> You don't even need an Lmod hook.  Just put something like


hide-version foo/42.0

in a global rc file evaluated by Lmod (I believe it has to be TCL).
This can easily be done after the fact.  I'm using this for quite a
while now and it works like a charm.  See also

https://lmod.readthedocs.io/en/latest/040_FAQ.html?highlight=hide-version


Thanks for suggesting the use of Lmod "hide-version" for hiding 
unwanted/obsolete/system modules from the "module avail" command!  There 
are also some useful hints in 
https://github.com/TACC/Lmod/blob/master/Transition_to_Lmod7.txt.


IMHO, this is the best available solution in stead of rebuilding your 
entire module tree from scratch (which may still be what we want to do 
in the long term).


I've written the attached script for conveniently generating a 
~/.modulerc file from a list of module name patterns which you want to 
hide.  On a CentOS 7 system with the Lmod RPM from EPEL, you can copy 
the .modulerc file to the Lmod system default file /usr/share/lmod/etc/rc.


For the record, I've also documented this procedure and the script in my 
EasyBuild Wiki page at 
https://wiki.fysik.dtu.dk/niflheim/EasyBuild_modules#hiding-modules-with-lmod


/Ole

#!/bin/sh

# Create ~/.modulerc file with module hide-version information
# The hidden modules will not be shown by "module avail".
# List system-wide modulerc file by:
# $ module --config 2>&1 | grep MODULERCFILE
# MODULERCFILE /usr/share/lmod/etc/rc
MODULERC=~/.modulerc

# List available modules
TEMP=/tmp/modulerc.$$
rm -f $TEMP
module --terse --show-hidden avail > $TEMP 2>&1

# Generate a hide list

cat < $MODULERC
#%Module
# Documentation of hide-version:
# https://lmod.readthedocs.io/en/latest/040_FAQ.html?highlight=hide-version
# and https://github.com/TACC/Lmod/blob/master/Transition_to_Lmod7.txt
global env
if { [info exists env(LMOD_VERSION_MAJOR)]} {
EOF

# Define patterns for which modules to hide

cat <> $MODULERC
GCCcore-5.4.0
GCCcore-6.1.0
GCCcore/5.4.0
GCCcore/6.1.0
GCC-5.4.0-2.26
GCC-6.3.0-2.27
foss/2016a
foss-2016a
foss/2016b
foss-2016b
Autoconf
Automake
Autotools
Bison
CMake
LibTIFF
LibUUID
M4
Szip
Tcl/
Tk/
Tkinter
XML-Parser
XZ
bzip2
binutils
cURL
expat
flex
fontconfig
freetype
gettext
gompi-2016b
gompi/2016b
gperf
help2man
hwloc
libevent
libffi
libjpeg-turbo
libpng
libreadline
LibTIFF
libtool
LibUUID
libxml2
ncurses
numactl
pkg-config
tmux
util-linux
zlib
EOF

# Terminating bracket
echo "}" >> $MODULERC

echo File $MODULERC has been created

rm -f $TEMP

Re: [easybuild] Retrospectively hiding modules / hiding by default

[Ole Holm Nielsen]

On 11/23/2018 10:53 AM, Markus Geimer wrote:> You don't even need an 
Lmod hook.  Just put something like


hide-version foo/42.0

in a global rc file evaluated by Lmod (I believe it has to be TCL).
This can easily be done after the fact.  I'm using this for quite a
while now and it works like a charm.  See also

https://lmod.readthedocs.io/en/latest/040_FAQ.html?highlight=hide-version


Thanks for suggesting the use of Lmod "hide-version" for hiding 
unwanted/obsolete/system modules from the "module avail" command!  There 
are also some useful hints in 
https://github.com/TACC/Lmod/blob/master/Transition_to_Lmod7.txt.


IMHO, this is the best available solution in stead of rebuilding your 
entire module tree from scratch (which may still be what we want to do 
in the long term).


I've written the attached script for conveniently generating a 
~/.modulerc file from a list of module name patterns which you want to 
hide.  On a CentOS 7 system with the Lmod RPM from EPEL, you can copy 
the .modulerc file to the Lmod system default file /usr/share/lmod/etc/rc.


For the record, I've also documented this procedure and the script in my 
EasyBuild Wiki page at 
https://wiki.fysik.dtu.dk/niflheim/EasyBuild_modules#hiding-modules-with-lmod


/Ole
#!/bin/sh

# Create ~/.modulerc file with module hide-version information
# The hidden modules will not be shown by "module avail".
# List system-wide modulerc file by:
# $ module --config 2>&1 | grep MODULERCFILE
# MODULERCFILE /usr/share/lmod/etc/rc
MODULERC=~/.modulerc

# List available modules
TEMP=/tmp/modulerc.$$
rm -f $TEMP
module --terse --show-hidden avail > $TEMP 2>&1

# Generate a hide list

cat < $MODULERC
#%Module
# Documentation of hide-version:
# https://lmod.readthedocs.io/en/latest/040_FAQ.html?highlight=hide-version
# and https://github.com/TACC/Lmod/blob/master/Transition_to_Lmod7.txt
global env
if { [info exists env(LMOD_VERSION_MAJOR)]} {
EOF

# Define patterns for which modules to hide

cat <> $MODULERC
GCCcore-5.4.0
GCCcore-6.1.0
GCCcore/5.4.0
GCCcore/6.1.0
GCC-5.4.0-2.26
GCC-6.3.0-2.27
foss/2016a
foss-2016a
foss/2016b
foss-2016b
Autoconf
Automake
Autotools
Bison
CMake
LibTIFF
LibUUID
M4
Szip
Tcl/
Tk/
Tkinter
XML-Parser
XZ
bzip2
binutils
cURL
expat
flex
fontconfig
freetype
gettext
gompi-2016b
gompi/2016b
gperf
help2man
hwloc
libevent
libffi
libjpeg-turbo
libpng
libreadline
LibTIFF
libtool
LibUUID
libxml2
ncurses
numactl
pkg-config
tmux
util-linux
zlib
EOF

# Terminating bracket
echo "}" >> $MODULERC

echo File $MODULERC has been created

rm -f $TEMP

Re: [easybuild] Retrospectively hiding modules / hiding by default

[Ward Poelmans]

On 23/11/18 10:57, Ole Holm Nielsen wrote:

> 
> It's also possible to hide modules by configuring Lmod, see the
> isVisibleHook function in
> https://lmod.readthedocs.io/en/latest/170_hooks.html

We hide modules from older toolchain with the hooks:
https://github.com/sisc-hpc/Lmod-UGent/blob/master/SitePackage.lua#L214

You can use regex, Lmod properties, etc to hide modules.

Ward

Re: [easybuild] Retrospectively hiding modules / hiding by default

[Fotis Georgatos]

Sure, I think you are on the same road I was trying to describe in my very last 
email... good luck ;)

F.

> On 23 Nov 2018, at 12:24 PM, Jon Roddom - UKRI STFC 
>  wrote:
> 
> It may be better to develop an entirely new EasyBuild 
> module stack from scratch.  Then one can have two disjoint module stacks 
> "old" and "current" available for the users, and users should be 
> migrated slowly to the "current" stack until the "old" stack can be 
> removed after a year or two.  Does this sound like a good approach?

Re: [easybuild] Retrospectively hiding modules / hiding by default

[Fotis Georgatos]

I think there is a cheap and safe path to overcome the below (manual) process 
and the associated risk, by changing the delivery directory for the new 
modulefiles and refresh all.

In fact, this would be an easy way to test things before you change MODULEPATH 
for everybody else.

Hoping this helps!
F.

> On 23 Nov 2018, at 11:45 AM, Alvarez, Damian  wrote:
> 
> -You will need to manually delete the old non-hidden modules carefully.

RE: [easybuild] Retrospectively hiding modules / hiding by default

[Jon Roddom - UKRI STFC]

We configure EB using env vars so I have EASYBUILD_HIDE_DEPS defined as so:

DEPS_LIST="Autoconf,Automake,Autotools,binutils,Bison,flex,gettext,help2man,hwloc,libpciaccess,libtool,libxml2,M4,ncurses,numactl,xorg-macros,XZ,zlib,libpng,ncurses,libsodium,pkg-config,Tcl,bzip2,OpenPGM,libreadline,util-linux,freetype,SQLite,libffi,gettext,expat,gperf,libxml2,fontconfig,gettext,GMP,ZeroMQ,XML-Parser,intltool,X11,Tk,Tkinter,SQLite,Doxygen,GSL,Guile,Libint,Szip,gc,libmatheval,libunistring,libxc,libxsmm,pkgconfig"

export EASYBUILD_HIDE_DEPS=$(echo $DEPS_LIST | sed -e $'s:,:\\\n:g' | sort -n | 
paste -sd ',' - | sed 's:,:,:g')

I'm in the habit of doing a dry run every time I install something now so I can 
have a quick scan of what EB is going to do and I add anything that I think the 
users won’t be too interested in. They can always see everything with ' ml av 
--show-hidden' (using Lmod).

Cheers,

Jon


Jon Roddom
SCARF Systems Administrator
Scientific Computing Department
STFC Rutherford Appleton Laboratory



-Original Message-
From: easybuild-requ...@lists.ugent.be 
[mailto:easybuild-requ...@lists.ugent.be] On Behalf Of Ole Holm Nielsen
Sent: 23 November 2018 11:10
To: easybuild@lists.ugent.be
Subject: Re: [easybuild] Retrospectively hiding modules / hiding by default

On 11/23/2018 11:44 AM, Alvarez, Damian wrote:
> A few notes of caution when rebuilding the whole stack:
> 
> -Doing --module-only is not guaranteed to work in all cases. Sometimes 
> variables are computed during the build, and might have an effect on the 
> module created. EB will print a diff if the module already exist, pay 
> attention to it.
> -However, if you are reinstalling a newly hidden module, EB won't find the 
> old one (since it will look for module/version, instead of module/.version) 
> and so the diff won't be printed and you can potentially miss something.
> -You will need to manually delete the old non-hidden modules carefully.
> -You have to be careful and filter out the hidden dependencies, you can't say 
> "eb $MY_EB_REPO/*eb --module-only", or you'll be recreating already existing 
> and visible modules.

Thanks very much for your important words of caution!  Dragons are 
obviously lurking in this field!

In this case, hiding of existing modules seems to be an error prone or 
impossible task.  It may be better to develop an entirely new EasyBuild 
module stack from scratch.  Then one can have two disjoint module stacks 
"old" and "current" available for the users, and users should be 
migrated slowly to the "current" stack until the "old" stack can be 
removed after a year or two.  Does this sound like a good approach?

We still need documentation of EASYBUILD_HIDE_DEPS and examples of usage 
for building a new stack.

> In my opinion, for a full stack makes more sense to use the rc files from 
> lmod, as Markus said.

Thanks.  Robert McLay (the author of Lmod) did recommend the 
isVisibleHook though when I asked him at the Supercomputing 2018 conference.

Thanks,
Ole

Re: [easybuild] Retrospectively hiding modules / hiding by default

[Ole Holm Nielsen]

On 11/23/2018 11:44 AM, Alvarez, Damian wrote:

A few notes of caution when rebuilding the whole stack:

-Doing --module-only is not guaranteed to work in all cases. Sometimes 
variables are computed during the build, and might have an effect on the module 
created. EB will print a diff if the module already exist, pay attention to it.
-However, if you are reinstalling a newly hidden module, EB won't find the old 
one (since it will look for module/version, instead of module/.version) and so 
the diff won't be printed and you can potentially miss something.
-You will need to manually delete the old non-hidden modules carefully.
-You have to be careful and filter out the hidden dependencies, you can't say "eb 
$MY_EB_REPO/*eb --module-only", or you'll be recreating already existing and visible 
modules.


Thanks very much for your important words of caution!  Dragons are 
obviously lurking in this field!


In this case, hiding of existing modules seems to be an error prone or 
impossible task.  It may be better to develop an entirely new EasyBuild 
module stack from scratch.  Then one can have two disjoint module stacks 
"old" and "current" available for the users, and users should be 
migrated slowly to the "current" stack until the "old" stack can be 
removed after a year or two.  Does this sound like a good approach?


We still need documentation of EASYBUILD_HIDE_DEPS and examples of usage 
for building a new stack.



In my opinion, for a full stack makes more sense to use the rc files from lmod, 
as Markus said.


Thanks.  Robert McLay (the author of Lmod) did recommend the 
isVisibleHook though when I asked him at the Supercomputing 2018 conference.


Thanks,
Ole

Re: [easybuild] Retrospectively hiding modules / hiding by default

[Alan O'Cais]

Hi Ole,
On Fri, 23 Nov 2018 at 11:23, Ole Holm Nielsen 
mailto:ole.h.niel...@fysik.dtu.dk>> wrote:

Can you kindly point to any documentation of EASYBUILD_HIDE_DEPS and any
examples of usage?  I haven't been able to find it with Google or in
https://easybuild.readthedocs.io/en/latest/

See 
https://easybuild.readthedocs.io/en/latest/Manipulating_dependencies.html#installing-dependencies-as-hidden-modules-using-hide-deps


When you define a default set of hidden modules, this only applies to
any new modules which you build, right?  All previous unhidden modules
will still appear in the "module avail" list, right?

Correct

Would you agree that the module stack has to be rebuilt in order to hide
already existing modules?  There are no magic tricks?
The modules (not the software) have to be rebuilt unless you use Lmod to hide 
them. No magic tricks (without using the Lmod feature), at the end of the day 
if there is a dependency then it appears as 'dependency/version' in the module 
file and needs to be rewritten to 'dependency/.version' somehow.

 Alan


--
Dr. Alan O'Cais
E-CAM Software Manager
Juelich Supercomputing Centre
Forschungszentrum Juelich GmbH
52425 Juelich, Germany

Phone: +49 2461 61 5213
Fax: +49 2461 61 6656
E-mail: a.oc...@fz-juelich.de
WWW:http://www.fz-juelich.de/ias/jsc/EN




Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt

Re: [easybuild] Retrospectively hiding modules / hiding by default

[Alvarez, Damian]

A few notes of caution when rebuilding the whole stack:

-Doing --module-only is not guaranteed to work in all cases. Sometimes 
variables are computed during the build, and might have an effect on the module 
created. EB will print a diff if the module already exist, pay attention to it.
-However, if you are reinstalling a newly hidden module, EB won't find the old 
one (since it will look for module/version, instead of module/.version) and so 
the diff won't be printed and you can potentially miss something.
-You will need to manually delete the old non-hidden modules carefully.
-You have to be careful and filter out the hidden dependencies, you can't say 
"eb $MY_EB_REPO/*eb --module-only", or you'll be recreating already existing 
and visible modules.

In my opinion, for a full stack makes more sense to use the rc files from lmod, 
as Markus said.

Cheers,
Damian

On 23.11.18, 11:25, "easybuild-requ...@lists.ugent.be on behalf of Ole Holm 
Nielsen"  wrote:

On 11/23/2018 10:44 AM, Alan O'Cais wrote:> Installing hidden modules
after the fact won't help you here unless you
> rebuild your entire stack of modules (since all modules will need to be
> rewritten to include the now-hidden dependencies). We use a config
> option to define a default set of hidden modules:
>
> 
EASYBUILD_HIDE_DEPS=ANTLR,APR,APR-util,AT-SPI2-ATK,AT-SPI2-core,ATK,Autoconf,Automake,adwaita-icon-theme,ant,assimp,Bison,babl,binutils,byacc,bzip2,CUSP,Coreutils,cairo,cling,configurable-http-proxy,DB,DBus,DocBook-XML,Dyninst,dbus-glib,damageproto,ETSF_IO,Exiv2,eudev,expat,FFmpeg,FLTK,FTGL,FoX,fixesproto,fontsproto,fontconfig,freeglut,freetype,GCCcore,GDAL,GEGL,GL2PS,GLEW,GLM,GLib,GLPK,GPC,GObject-Introspection,GTI,GTK+,GTS,Gdk-Pixbuf,Ghostscript,GraphicsMagick,GtkSourceView,g2clib,g2lib,gc,gexiv2,gflags,glog,glproto,googletest,gperf,guile,grib_api,gsettings-desktop-schemas,gettext,gzip,HarfBuzz,icc,ifort,inputproto,intltool,itstool,JUnit,JSON-C,JSON-GLib,JasPer,jhbuild,kbproto,LMDB,LZO,LevelDB,LibTIFF,LibUUID,Libint,LittleCMS,libGLU,libICE,libSM,libX11,libXau,libXaw,libXcursor,libXdamage,libXdmcp,libXext,libXfixes,libXfont,libXft,libXi,libXinerama,libXmu,libXp,libXpm,libXrandr,libXrender,libXt,libXtst,libcerf,libcroco,libctl,libdap,libdrm,libdwarf,libelf,libepoxy,libevent,libffi,libfontenc,libgd,libgeotiff,libglade,libidn,libjpeg-turbo,libmatheval,libmypaint,libpng,libpciaccess,libpthread-stubs,libreadline,librsvg,libsndfile,libspatialindex,libtool,libunistring,libunwind,libyaml,libxcb,libxkbcommon,libxml2,libxslt,libyuv,M4,MATIO,Mesa,makedepend,motif,msgpack-c,NASM,NLopt,ncurses,nettle,nodejs,nvenc_sdk,nvidia,OPARI2,OTF2,PCRE,PDT,PROJ,Pango,Pmw,PnMPI,PyCairo,PyGObject,Python-Xpra,patchelf,pixman,pkg-config,pkgconfig,popt,printproto,protobuf,pscom,pybind11,Qhull,Qt,Qt5,qrupdate,randrproto,recordproto,renderproto,S-Lang,SCons,SIP,SQLite,SWIG,Serf,Szip,scrollkeeper,snappy,Tk,texinfo,UDUNITS,util-linux,vpx,wxPropertyGrid,wxWidgets,XML-Parser,XZ,XKeyboardConfig,x264,x265,xbitmaps,xcb-proto,xcb-util,xcb-util-image,xcb-util-keysyms,xcb-util-renderutil,xcb-util-wm,xextproto,xineramaproto,xorg-macros,xprop,xproto,xtrans,Yasm,zlib


Can you kindly point to any documentation of EASYBUILD_HIDE_DEPS and any
examples of usage?  I haven't been able to find it with Google or in
https://easybuild.readthedocs.io/en/latest/

When you define a default set of hidden modules, this only applies to
any new modules which you build, right?  All previous unhidden modules
will still appear in the "module avail" list, right?

Would you agree that the module stack has to be rebuilt in order to hide
already existing modules?  There are no magic tricks?

Thanks,
Ole






Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt

Re: [easybuild] Retrospectively hiding modules / hiding by default

[Ole Holm Nielsen]

On 11/23/2018 10:44 AM, Alan O'Cais wrote:> Installing hidden modules 
after the fact won't help you here unless you
rebuild your entire stack of modules (since all modules will need to be 
rewritten to include the now-hidden dependencies). We use a config 
option to define a default set of hidden modules:


EASYBUILD_HIDE_DEPS=ANTLR,APR,APR-util,AT-SPI2-ATK,AT-SPI2-core,ATK,Autoconf,Automake,adwaita-icon-theme,ant,assimp,Bison,babl,binutils,byacc,bzip2,CUSP,Coreutils,cairo,cling,configurable-http-proxy,DB,DBus,DocBook-XML,Dyninst,dbus-glib,damageproto,ETSF_IO,Exiv2,eudev,expat,FFmpeg,FLTK,FTGL,FoX,fixesproto,fontsproto,fontconfig,freeglut,freetype,GCCcore,GDAL,GEGL,GL2PS,GLEW,GLM,GLib,GLPK,GPC,GObject-Introspection,GTI,GTK+,GTS,Gdk-Pixbuf,Ghostscript,GraphicsMagick,GtkSourceView,g2clib,g2lib,gc,gexiv2,gflags,glog,glproto,googletest,gperf,guile,grib_api,gsettings-desktop-schemas,gettext,gzip,HarfBuzz,icc,ifort,inputproto,intltool,itstool,JUnit,JSON-C,JSON-GLib,JasPer,jhbuild,kbproto,LMDB,LZO,LevelDB,LibTIFF,LibUUID,Libint,LittleCMS,libGLU,libICE,libSM,libX11,libXau,libXaw,libXcursor,libXdamage,libXdmcp,libXext,libXfixes,libXfont,libXft,libXi,libXinerama,libXmu,libXp,libXpm,libXrandr,libXrender,libXt,libXtst,libcerf,libcroco,libctl,libdap,libdrm,libdwarf,libelf,libepoxy,libevent,libffi,libfontenc,libgd,libgeotiff,libglade,libidn,libjpeg-turbo,libmatheval,libmypaint,libpng,libpciaccess,libpthread-stubs,libreadline,librsvg,libsndfile,libspatialindex,libtool,libunistring,libunwind,libyaml,libxcb,libxkbcommon,libxml2,libxslt,libyuv,M4,MATIO,Mesa,makedepend,motif,msgpack-c,NASM,NLopt,ncurses,nettle,nodejs,nvenc_sdk,nvidia,OPARI2,OTF2,PCRE,PDT,PROJ,Pango,Pmw,PnMPI,PyCairo,PyGObject,Python-Xpra,patchelf,pixman,pkg-config,pkgconfig,popt,printproto,protobuf,pscom,pybind11,Qhull,Qt,Qt5,qrupdate,randrproto,recordproto,renderproto,S-Lang,SCons,SIP,SQLite,SWIG,Serf,Szip,scrollkeeper,snappy,Tk,texinfo,UDUNITS,util-linux,vpx,wxPropertyGrid,wxWidgets,XML-Parser,XZ,XKeyboardConfig,x264,x265,xbitmaps,xcb-proto,xcb-util,xcb-util-image,xcb-util-keysyms,xcb-util-renderutil,xcb-util-wm,xextproto,xineramaproto,xorg-macros,xprop,xproto,xtrans,Yasm,zlib



Can you kindly point to any documentation of EASYBUILD_HIDE_DEPS and any 
examples of usage?  I haven't been able to find it with Google or in 
https://easybuild.readthedocs.io/en/latest/


When you define a default set of hidden modules, this only applies to 
any new modules which you build, right?  All previous unhidden modules 
will still appear in the "module avail" list, right?


Would you agree that the module stack has to be rebuilt in order to hide 
already existing modules?  There are no magic tricks?


Thanks,
Ole

Re: [easybuild] Retrospectively hiding modules / hiding by default

[Ole Holm Nielsen]

I completely agree with Loris!  I have been wanting to hide irrelevant 
modules from the listing of "module avail" for a very long time (the 
curious users can always use "module spider").  The "module list" should 
list all loaded modules, hidden or not.


At the recent Supercomputing 2018 conference I asked Davide Vanzo 
(EasyBuild) and Robert McLay (Lmod) exactly this question, and Davide's 
answer is that you must rebuild all of the module .lua files with 
--module-only and --hide-deps.  You *do not* have to rebuild the 
software itself!


It's also possible to hide modules by configuring Lmod, see the 
isVisibleHook function in 
https://lmod.readthedocs.io/en/latest/170_hooks.html


I think Davide's solution of rebuilding the module files is the simplest 
way forward.  However, it isn't trivial to do correctly on a running 
system, because you will be messing with the locations of modules in the 
.lua files which could break user jobs.  It is strongly recommended to 
make a copy of the module tree and work on this without impacting the 
production system.  When completed, you can point users to the location 
of the new module tree (on a shared file system, I presume).


This leads us to an interesting project for interested EasyBuild sites. 
Can we design some tools to perform:


1. Create a configuration file with "system" module names (not end-user 
applications) that we want to hide.  My favorites include all module 
names beginning with:
Autoconf, Automake, Autotools, Bison, CMake, LibTIFF, LibUUID, M4, Szip, 
Tcl, Tk, Tkinter, XML-Parser, XZ, bzip2, binutils, cURL, expat, flex, 
fontconfig, freetype, gettext, help2man, hwloc, lib*, ncurses, numactl, 
pkg-config, tmux, util-linux, zlib


2. Write a script that traverses all directories in 
$EASYBUILD_PREFIX/modules/all and rebuilds all the .lua files using "eb 
--module-only --hide-deps".  Maybe the above "system" module names must 
be rebuilt with --hidden, whereas end-user applications must be rebuilt 
with a --hide-deps list equal to the desired hidden modules.  I'm unsure 
of the correct logic here, but maybe EasyBuild experts can comment on this?


The "eb --help" explains how to hide modules:

 --hidden  Install 'hidden' module file(s) by prefixing their 
version with '.' (def False)
 --hide-deps=HIDE-DEPS  Comma separated list of dependencies that you 
want automatically hidden, (e.g. --hide-deps=zlib,ncurses) 
 (type comma-separated list)


Does anyone have the time and courage to work on a setup with such a 
functionality?


Best regards,
Ole


On 11/23/2018 10:17 AM, Loris Bennett wrote:

Hi Sam,

Thanks for the info about rebuilding the module files.

Regarding the visibility of modules, I just want to exclude some from
the results of 'module av'.  I assume 'module list' will still show all
the modules which are loaded, hidden or not.  That doesn't bother me,
since, as far as I know, very few of our users look at the output of
'module list' anyway.

Cheers,

Loris

Sam Moors  writes:


Hi Loris,

You can instruct EB to only rebuild the modules with the '--module-only' option.
I don't know about your second question though.

That said, are you sure you want to hide the dependencies?



I think it is useful for the users to know which deps are loaded, so
they can avoid conflicts with other modules, even if it may look
daunting to see 50+ modules loaded in some cases.

Cheers,
Sam

On Fri, Nov 23, 2018 at 8:25 AM Loris Bennett  
wrote:

  Hi,

  Now that I have installed a couple of packages with easybuild I have a
  surprising number of modules and realise that I should have been using
  --hide-deps. I have two questions about this:

  1. Is there any way to retrospectively hide modules, other than tweaking
  the module files by hand, say, rebuilding the modules?
  2. Can a set of modules be defined which will be hidden by default?

  Cheers,

  Loris

  --
  Dr. Loris Bennett (Mr.)
  ZEDAT, Freie Universität Berlin Email loris.benn...@fu-berlin.de




--
Ole Holm Nielsen
PhD, Senior HPC Officer
Department of Physics, Technical University of Denmark,
Building 307, DK-2800 Kongens Lyngby, Denmark
E-mail: ole.h.niel...@fysik.dtu.dk
Homepage: http://dcwww.fysik.dtu.dk/~ohnielse/
Tel: (+45) 4525 3187 / Mobile (+45) 5180 1620

Re: [easybuild] Retrospectively hiding modules / hiding by default

[Markus Geimer]

All,

You don't even need an Lmod hook.  Just put something like

hide-version foo/42.0

in a global rc file evaluated by Lmod (I believe it has to be TCL).
This can easily be done after the fact.  I'm using this for quite a
while now and it works like a charm.  See also

https://lmod.readthedocs.io/en/latest/040_FAQ.html?highlight=hide-version

Hth,
Markus

On 11/23/2018 10:44 AM, Alan O'Cais wrote:
> Hi Loris,
> 
> Installing hidden modules after the fact won't help you here unless you
> rebuild your entire stack of modules (since all modules will need to be
> rewritten to include the now-hidden dependencies). We use a config
> option to define a default set of hidden modules:
> 
> EASYBUILD_HIDE_DEPS=ANTLR,APR,APR-util,AT-SPI2-ATK,AT-SPI2-core,ATK,Autoconf,Automake,adwaita-icon-theme,ant,assimp,Bison,babl,binutils,byacc,bzip2,CUSP,Coreutils,cairo,cling,configurable-http-proxy,DB,DBus,DocBook-XML,Dyninst,dbus-glib,damageproto,ETSF_IO,Exiv2,eudev,expat,FFmpeg,FLTK,FTGL,FoX,fixesproto,fontsproto,fontconfig,freeglut,freetype,GCCcore,GDAL,GEGL,GL2PS,GLEW,GLM,GLib,GLPK,GPC,GObject-Introspection,GTI,GTK+,GTS,Gdk-Pixbuf,Ghostscript,GraphicsMagick,GtkSourceView,g2clib,g2lib,gc,gexiv2,gflags,glog,glproto,googletest,gperf,guile,grib_api,gsettings-desktop-schemas,gettext,gzip,HarfBuzz,icc,ifort,inputproto,intltool,itstool,JUnit,JSON-C,JSON-GLib,JasPer,jhbuild,kbproto,LMDB,LZO,LevelDB,LibTIFF,LibUUID,Libint,LittleCMS,libGLU,libICE,libSM,libX11,libXau,libXaw,libXcursor,libXdamage,libXdmcp,libXext,libXfixes,libXfont,libXft,libXi,libXinerama,libXmu,libXp,libXpm,libXrandr,libXrender,libXt,libXtst,libcerf,libcroco,libctl,libdap,libdrm,libdwarf,libelf,libepoxy,libevent,libffi,libfontenc,libgd,libgeotiff,libglade,libidn,libjpeg-turbo,libmatheval,libmypaint,libpng,libpciaccess,libpthread-stubs,libreadline,librsvg,libsndfile,libspatialindex,libtool,libunistring,libunwind,libyaml,libxcb,libxkbcommon,libxml2,libxslt,libyuv,M4,MATIO,Mesa,makedepend,motif,msgpack-c,NASM,NLopt,ncurses,nettle,nodejs,nvenc_sdk,nvidia,OPARI2,OTF2,PCRE,PDT,PROJ,Pango,Pmw,PnMPI,PyCairo,PyGObject,Python-Xpra,patchelf,pixman,pkg-config,pkgconfig,popt,printproto,protobuf,pscom,pybind11,Qhull,Qt,Qt5,qrupdate,randrproto,recordproto,renderproto,S-Lang,SCons,SIP,SQLite,SWIG,Serf,Szip,scrollkeeper,snappy,Tk,texinfo,UDUNITS,util-linux,vpx,wxPropertyGrid,wxWidgets,XML-Parser,XZ,XKeyboardConfig,x264,x265,xbitmaps,xcb-proto,xcb-util,xcb-util-image,xcb-util-keysyms,xcb-util-renderutil,xcb-util-wm,xextproto,xineramaproto,xorg-macros,xprop,xproto,xtrans,Yasm,zlib
> 
> You can also tell Lmod to hide certain modules (which is something you
> can do after the fact) via a hook (but I haven't done
> this): https://lmod.readthedocs.io/en/latest/170_hooks.html?highlight=hidden
> 
> Alan
> 
> On Fri, 23 Nov 2018 at 10:17, Loris Bennett  > wrote:
> 
> Hi Sam,
> 
> Thanks for the info about rebuilding the module files.
> 
> Regarding the visibility of modules, I just want to exclude some from
> the results of 'module av'.  I assume 'module list' will still show all
> the modules which are loaded, hidden or not.  That doesn't bother me,
> since, as far as I know, very few of our users look at the output of
> 'module list' anyway.
> 
> Cheers,
> 
> Loris
> 
> Sam Moors mailto:sammo...@gmail.com>> writes:
> 
> > Hi Loris,
> >
> > You can instruct EB to only rebuild the modules with the
> '--module-only' option.
> > I don't know about your second question though.
> >
> > That said, are you sure you want to hide the dependencies?
> 
> > I think it is useful for the users to know which deps are loaded, so
> > they can avoid conflicts with other modules, even if it may look
> > daunting to see 50+ modules loaded in some cases.
> >
> > Cheers,
> > Sam
> >
> > On Fri, Nov 23, 2018 at 8:25 AM Loris Bennett
> mailto:loris.benn...@fu-berlin.de>> wrote:
> >
> >  Hi,
> >
> >  Now that I have installed a couple of packages with easybuild I
> have a
> >  surprising number of modules and realise that I should have been
> using
> >  --hide-deps. I have two questions about this:
> >
> >  1. Is there any way to retrospectively hide modules, other than
> tweaking
> >  the module files by hand, say, rebuilding the modules?
> >  2. Can a set of modules be defined which will be hidden by default?
> >
> >  Cheers,
> >
> >  Loris
> >
> >  --
> >  Dr. Loris Bennett (Mr.)
> >  ZEDAT, Freie Universität Berlin Email loris.benn...@fu-berlin.de
> 
> >
> >
> -- 
> Dr. Loris Bennett (Mr.)
> ZEDAT, Freie Universität Berlin         Email
> loris.benn...@fu-berlin.de 
> 
> 
> 
> -- 
> Dr. Alan O'Cais
> E-CAM Software Manager
> Juelich Supercomputing Centre
> Forschungszentrum Juelich GmbH

Re: [easybuild] Retrospectively hiding modules / hiding by default

[Alan O'Cais]

Hi Loris,

Installing hidden modules after the fact won't help you here unless you rebuild 
your entire stack of modules (since all modules will need to be rewritten to 
include the now-hidden dependencies). We use a config option to define a 
default set of hidden modules:

EASYBUILD_HIDE_DEPS=ANTLR,APR,APR-util,AT-SPI2-ATK,AT-SPI2-core,ATK,Autoconf,Automake,adwaita-icon-theme,ant,assimp,Bison,babl,binutils,byacc,bzip2,CUSP,Coreutils,cairo,cling,configurable-http-proxy,DB,DBus,DocBook-XML,Dyninst,dbus-glib,damageproto,ETSF_IO,Exiv2,eudev,expat,FFmpeg,FLTK,FTGL,FoX,fixesproto,fontsproto,fontconfig,freeglut,freetype,GCCcore,GDAL,GEGL,GL2PS,GLEW,GLM,GLib,GLPK,GPC,GObject-Introspection,GTI,GTK+,GTS,Gdk-Pixbuf,Ghostscript,GraphicsMagick,GtkSourceView,g2clib,g2lib,gc,gexiv2,gflags,glog,glproto,googletest,gperf,guile,grib_api,gsettings-desktop-schemas,gettext,gzip,HarfBuzz,icc,ifort,inputproto,intltool,itstool,JUnit,JSON-C,JSON-GLib,JasPer,jhbuild,kbproto,LMDB,LZO,LevelDB,LibTIFF,LibUUID,Libint,LittleCMS,libGLU,libICE,libSM,libX11,libXau,libXaw,libXcursor,libXdamage,libXdmcp,libXext,libXfixes,libXfont,libXft,libXi,libXinerama,libXmu,libXp,libXpm,libXrandr,libXrender,libXt,libXtst,libcerf,libcroco,libctl,libdap,libdrm,libdwarf,libelf,libepoxy,libevent,libffi,libfontenc,libgd,libgeotiff,libglade,libidn,libjpeg-turbo,libmatheval,libmypaint,libpng,libpciaccess,libpthread-stubs,libreadline,librsvg,libsndfile,libspatialindex,libtool,libunistring,libunwind,libyaml,libxcb,libxkbcommon,libxml2,libxslt,libyuv,M4,MATIO,Mesa,makedepend,motif,msgpack-c,NASM,NLopt,ncurses,nettle,nodejs,nvenc_sdk,nvidia,OPARI2,OTF2,PCRE,PDT,PROJ,Pango,Pmw,PnMPI,PyCairo,PyGObject,Python-Xpra,patchelf,pixman,pkg-config,pkgconfig,popt,printproto,protobuf,pscom,pybind11,Qhull,Qt,Qt5,qrupdate,randrproto,recordproto,renderproto,S-Lang,SCons,SIP,SQLite,SWIG,Serf,Szip,scrollkeeper,snappy,Tk,texinfo,UDUNITS,util-linux,vpx,wxPropertyGrid,wxWidgets,XML-Parser,XZ,XKeyboardConfig,x264,x265,xbitmaps,xcb-proto,xcb-util,xcb-util-image,xcb-util-keysyms,xcb-util-renderutil,xcb-util-wm,xextproto,xineramaproto,xorg-macros,xprop,xproto,xtrans,Yasm,zlib

You can also tell Lmod to hide certain modules (which is something you can do 
after the fact) via a hook (but I haven't done this): 
https://lmod.readthedocs.io/en/latest/170_hooks.html?highlight=hidden

Alan

On Fri, 23 Nov 2018 at 10:17, Loris Bennett 
mailto:loris.benn...@fu-berlin.de>> wrote:
Hi Sam,

Thanks for the info about rebuilding the module files.

Regarding the visibility of modules, I just want to exclude some from
the results of 'module av'.  I assume 'module list' will still show all
the modules which are loaded, hidden or not.  That doesn't bother me,
since, as far as I know, very few of our users look at the output of
'module list' anyway.

Cheers,

Loris

Sam Moors mailto:sammo...@gmail.com>> writes:

> Hi Loris,
>
> You can instruct EB to only rebuild the modules with the '--module-only' 
> option.
> I don't know about your second question though.
>
> That said, are you sure you want to hide the dependencies?

> I think it is useful for the users to know which deps are loaded, so
> they can avoid conflicts with other modules, even if it may look
> daunting to see 50+ modules loaded in some cases.
>
> Cheers,
> Sam
>
> On Fri, Nov 23, 2018 at 8:25 AM Loris Bennett 
> mailto:loris.benn...@fu-berlin.de>> wrote:
>
>  Hi,
>
>  Now that I have installed a couple of packages with easybuild I have a
>  surprising number of modules and realise that I should have been using
>  --hide-deps. I have two questions about this:
>
>  1. Is there any way to retrospectively hide modules, other than tweaking
>  the module files by hand, say, rebuilding the modules?
>  2. Can a set of modules be defined which will be hidden by default?
>
>  Cheers,
>
>  Loris
>
>  --
>  Dr. Loris Bennett (Mr.)
>  ZEDAT, Freie Universität Berlin Email 
> loris.benn...@fu-berlin.de
>
>
--
Dr. Loris Bennett (Mr.)
ZEDAT, Freie Universität Berlin Email 
loris.benn...@fu-berlin.de


--
Dr. Alan O'Cais
E-CAM Software Manager
Juelich Supercomputing Centre
Forschungszentrum Juelich GmbH
52425 Juelich, Germany

Phone: +49 2461 61 5213
Fax: +49 2461 61 6656
E-mail: a.oc...@fz-juelich.de
WWW:http://www.fz-juelich.de/ias/jsc/EN




Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt

Re: [easybuild] Retrospectively hiding modules / hiding by default

[Loris Bennett]

Hi Sam,

Thanks for the info about rebuilding the module files.

Regarding the visibility of modules, I just want to exclude some from
the results of 'module av'.  I assume 'module list' will still show all
the modules which are loaded, hidden or not.  That doesn't bother me,
since, as far as I know, very few of our users look at the output of
'module list' anyway.

Cheers,

Loris

Sam Moors  writes:

> Hi Loris,
>
> You can instruct EB to only rebuild the modules with the '--module-only' 
> option.
> I don't know about your second question though.
>
> That said, are you sure you want to hide the dependencies?

> I think it is useful for the users to know which deps are loaded, so
> they can avoid conflicts with other modules, even if it may look
> daunting to see 50+ modules loaded in some cases.
>
> Cheers,
> Sam
>
> On Fri, Nov 23, 2018 at 8:25 AM Loris Bennett  
> wrote:
>
>  Hi,
>
>  Now that I have installed a couple of packages with easybuild I have a
>  surprising number of modules and realise that I should have been using
>  --hide-deps. I have two questions about this:
>
>  1. Is there any way to retrospectively hide modules, other than tweaking
>  the module files by hand, say, rebuilding the modules?
>  2. Can a set of modules be defined which will be hidden by default?
>
>  Cheers,
>
>  Loris
>
>  -- 
>  Dr. Loris Bennett (Mr.)
>  ZEDAT, Freie Universität Berlin Email loris.benn...@fu-berlin.de
>
>
-- 
Dr. Loris Bennett (Mr.)
ZEDAT, Freie Universität Berlin Email loris.benn...@fu-berlin.de

Re: [easybuild] Retrospectively hiding modules / hiding by default

[Sam Moors]

Hi Loris,

You can instruct EB to only rebuild the modules with the '--module-only'
option.
I don't know about your second question though.

That said, are you sure you want to hide the dependencies?
I think it is useful for the users to know which deps are loaded, so they
can avoid conflicts with other modules, even if it may look daunting to see
50+ modules loaded in some cases.

Cheers,
Sam

On Fri, Nov 23, 2018 at 8:25 AM Loris Bennett 
wrote:

> Hi,
>
> Now that I have installed a couple of packages with easybuild I have a
> surprising number of modules and realise that I should have been using
> --hide-deps.  I have two questions about this:
>
> 1. Is there any way to retrospectively hide modules, other than tweaking
>the module files by hand, say, rebuilding the modules?
> 2. Can a set of modules be defined which will be hidden by default?
>
> Cheers,
>
> Loris
>
> --
> Dr. Loris Bennett (Mr.)
> ZEDAT, Freie Universität Berlin Email loris.benn...@fu-berlin.de
>

[easybuild] Retrospectively hiding modules / hiding by default

[Loris Bennett]

Hi,

Now that I have installed a couple of packages with easybuild I have a
surprising number of modules and realise that I should have been using
--hide-deps.  I have two questions about this:

1. Is there any way to retrospectively hide modules, other than tweaking
   the module files by hand, say, rebuilding the modules?
2. Can a set of modules be defined which will be hidden by default?

Cheers,

Loris

-- 
Dr. Loris Bennett (Mr.)
ZEDAT, Freie Universität Berlin Email loris.benn...@fu-berlin.de