Re: [easybuild] How to setup EB in the global /etc/profile.d/ ?
On 10/14/2016 10:08 AM, Kenneth Hoste wrote: Following the Lmod documentation in http://lmod.readthedocs.io/en/latest/070_standard_modules.html I've come up with the following shell initialization file /etc/profile.d/z01_EasyBuild.sh which gets called after Lmod has been initialized (by z00_lmod.sh): if [ -z "$__Init_Default_Modules" ]; then export __Init_Default_Modules=1 export EASYBUILD_MODULES_TOOL=Lmod export EASYBUILD_PREFIX=/home/modules module use $EASYBUILD_PREFIX/modules/all else module refresh fi (obviously, the EASYBUILD_PREFIX is site-dependent). Please note that I refrain from calling the recommended "module --initial_load restore". I believe this is unwarranted, unless the sysadmin actually wants users to always load some specific set of system default modules - which I don't want to do. I also don't load the EasyBuild module, because only module builders need to do that, but not the normal users. It seems to me that the above script would be a suitable minimum setup for automatic initialization of EB modules. Question to the experts: Is this approach sound, or do you see any problems? Defining $EASYBUILD_PREFIX by default is probably not something you want to do for everyone, since that instructs EasyBuild to install software in this location, and I expect that most users won't have write permissions there? That's a good point! Can you enlighten me on how to provide globally available modules as I've done, while at the same time allowing normal users to build their own private modules in their $HOME? I guess that end users could define in their .bashrc, for example, export EASYBUILD_PREFIX=$HOME/modules When they do module use $EASYBUILD_PREFIX/modules/all that should simply prepend their personal module path to the system-wide one I define above - or maybe I'm totally mistaken here? I did a short test of this as a normal user, and it seems to work as expected. I'm happy to include this in the EasyBuild documentation somewhere, any suggestions for a particular location? Should it be a new page? Are you up for making a pull request to the EasyBuild documentation for this (see the 'docs' subdir in https://github.com/hpcugent/easybuild)? I'd be happy to start working with Github, but it'll take some time for me to familiarize myself with this ecosystem. But I don't see a natural place in the current EB documentation to make contributions. Perhaps you could consider the possibility of making a new section in https://easybuild.readthedocs.io/en/latest/, for example, "EasyBuild for End Users" or similar. I'm thinking that end users should know how to use EB modules, and sysadmins should think of ways to provide modules to end users. If you provide a section according to your wishes, perhaps you could include my above script as an example? Thanks, Ole -- Ole Holm Nielsen PhD, Manager of IT services Department of Physics, Technical University of Denmark, Building 307, DK-2800 Kongens Lyngby, Denmark E-mail: ole.h.niel...@fysik.dtu.dk Homepage: http://dcwww.fysik.dtu.dk/~ohnielse/ Tel: (+45) 4525 3187 / Mobile (+45) 5180 1620
Re: [easybuild] How to setup EB in the global /etc/profile.d/ ?
On 14/10/16 09:45, Ole Holm Nielsen wrote: Hi Kenneth, Cc: Robert, I've been trying to set up a default user environment with modules provided by EasyBuild (EB) and using Lmod. We run CentOS 7.2. The EB discussion is in this thread: https://lists.ugent.be/wws/arc/easybuild/2016-10/msg00052.html My goal is to automatically provide modules built by EB to every normal user, but not actually loading any modules, since users have different needs. Following the Lmod documentation in http://lmod.readthedocs.io/en/latest/070_standard_modules.html I've come up with the following shell initialization file /etc/profile.d/z01_EasyBuild.sh which gets called after Lmod has been initialized (by z00_lmod.sh): if [ -z "$__Init_Default_Modules" ]; then export __Init_Default_Modules=1 export EASYBUILD_MODULES_TOOL=Lmod export EASYBUILD_PREFIX=/home/modules module use $EASYBUILD_PREFIX/modules/all else module refresh fi (obviously, the EASYBUILD_PREFIX is site-dependent). Please note that I refrain from calling the recommended "module --initial_load restore". I believe this is unwarranted, unless the sysadmin actually wants users to always load some specific set of system default modules - which I don't want to do. I also don't load the EasyBuild module, because only module builders need to do that, but not the normal users. It seems to me that the above script would be a suitable minimum setup for automatic initialization of EB modules. Question to the experts: Is this approach sound, or do you see any problems? Defining $EASYBUILD_PREFIX by default is probably not something you want to do for everyone, since that instructs EasyBuild to install software in this location, and I expect that most users won't have write permissions there? Other than that, this looks OK to me. If it seems OK, could you perhaps consider adding the above example to the mentioned Lmod page as a convenient way to get started (at least when using EB). I don't know if this usage example is appropriate for inclusion in the EB documentation, but as a newcomer to EB I certainly felt the need for a good way to provide EB modules to normal users, and I had to work hard to solve this problem. In my Wiki page about Lmod and EasyBuild I've documented the above: https://wiki.fysik.dtu.dk/niflheim/EasyBuild_modules#global-setup-of-modules-for-all-users I'm happy to include this in the EasyBuild documentation somewhere, any suggestions for a particular location? Should it be a new page? Are you up for making a pull request to the EasyBuild documentation for this (see the 'docs' subdir in https://github.com/hpcugent/easybuild)? regards, Kenneth
Re: [easybuild] How to setup EB in the global /etc/profile.d/ ?
Hi Kenneth, Cc: Robert, I've been trying to set up a default user environment with modules provided by EasyBuild (EB) and using Lmod. We run CentOS 7.2. The EB discussion is in this thread: https://lists.ugent.be/wws/arc/easybuild/2016-10/msg00052.html My goal is to automatically provide modules built by EB to every normal user, but not actually loading any modules, since users have different needs. Following the Lmod documentation in http://lmod.readthedocs.io/en/latest/070_standard_modules.html I've come up with the following shell initialization file /etc/profile.d/z01_EasyBuild.sh which gets called after Lmod has been initialized (by z00_lmod.sh): if [ -z "$__Init_Default_Modules" ]; then export __Init_Default_Modules=1 export EASYBUILD_MODULES_TOOL=Lmod export EASYBUILD_PREFIX=/home/modules module use $EASYBUILD_PREFIX/modules/all else module refresh fi (obviously, the EASYBUILD_PREFIX is site-dependent). Please note that I refrain from calling the recommended "module --initial_load restore". I believe this is unwarranted, unless the sysadmin actually wants users to always load some specific set of system default modules - which I don't want to do. I also don't load the EasyBuild module, because only module builders need to do that, but not the normal users. It seems to me that the above script would be a suitable minimum setup for automatic initialization of EB modules. Question to the experts: Is this approach sound, or do you see any problems? If it seems OK, could you perhaps consider adding the above example to the mentioned Lmod page as a convenient way to get started (at least when using EB). I don't know if this usage example is appropriate for inclusion in the EB documentation, but as a newcomer to EB I certainly felt the need for a good way to provide EB modules to normal users, and I had to work hard to solve this problem. In my Wiki page about Lmod and EasyBuild I've documented the above: https://wiki.fysik.dtu.dk/niflheim/EasyBuild_modules#global-setup-of-modules-for-all-users Thanks, Ole -- Ole Holm Nielsen PhD, Manager of IT services Department of Physics, Technical University of Denmark, Building 307, DK-2800 Kongens Lyngby, Denmark
Re: [easybuild] How to setup EB in the global /etc/profile.d/ ?
On 10/13/2016 05:15 PM, Kenneth Hoste wrote: You need to move the 'module --initial_load restore' after the 'module use', otherwise the EasyBuild module is indeed not available. No Catch-22 imho... And then you shouldn't need to load the EasyBuild module again, it should be loaded via the restore... Confirmed! This initialization script works correctly: if [ -z "$__Init_Default_Modules" ]; then export __Init_Default_Modules=1 export EASYBUILD_MODULES_TOOL=Lmod export EASYBUILD_PREFIX=/home/modules module use $EASYBUILD_PREFIX/modules/all export LMOD_SYSTEM_DEFAULT_MODULES=EasyBuild module --initial_load restore ... Thanks, Ole
Re: [easybuild] How to setup EB in the global /etc/profile.d/ ?
On 10/13/2016 04:27 PM, Kenneth Hoste wrote: I'm afraid I don't have one. We don't load Easybuild by default for our users. But you should take a look at: https://lmod.readthedocs.io/en/latest/070_standard_modules.html Thanks, this page is actually useful and specific! So I've created a new file /etc/profile.d/z01_EasyBuild.sh (to be loaded *after* Lmod's z00_* scripts) with this content: if [ -z "$__Init_Default_Modules" ]; then export __Init_Default_Modules=1; # module --initial_load restore export EASYBUILD_MODULES_TOOL=Lmod export EASYBUILD_PREFIX=/home/modules module use $EASYBUILD_PREFIX/modules/all Is this right? Are the modules really in /home/modules/modules/all? Yes, I NFS-mount /home/modules, and there are 5 EB directories there: # ls -la /home/modules total 16 drwxrwxr-x. 7 modules modules 80 Oct 13 12:17 . drwxr-xr-x. 3 rootroot 0 Oct 13 16:34 .. drwxrwxr-x. 2 modules modules6 Oct 13 14:21 build drwxrwxr-x. 35 modules modules 4096 Oct 13 14:21 ebfiles_repo drwxrwxr-x. 15 modules modules 4096 Oct 13 13:55 modules drwxrwxr-x. 36 modules modules 4096 Oct 13 14:21 software drwxrwxr-x. 16 modules modules 4096 Sep 29 15:28 sources Are you suggesting that only the "modules" subdirectory will actually be needed by users? Anyway, the EB module building requires access to all subdirectories, so it seems convenient to have all subdirectories exposed this way. Am I making any mistakes here? The recommended line "module --initial_load restore" generates an unwarranted message: The system default contains no modules (env var: LMOD_SYSTEM_DEFAULT_MODULES is empty) No changes in loaded modules so I got rid of it by commenting this line out. Well, you need to define $LMOD_SYSTEM_DEFAULT_MODULES :) Do I really *need* to? What's the point, and what should I do then? Question: Does anyone see potential problems or bad side effects with such a global setup of EB for all users? Having a proper default setup in place makes sense, but users can easily override it with their own config file or via environment variables. That sounds like a "Good Thing" for advanced users, right? The only thing we do is load Lmod itself (a symlink to init/bash in /etc/profile.d). Question: How do your users initialize the usage of EB? They can define a config file, or define environment variables (typically via .bashrc), up to them... Ah, so it's DIY for your users! I'm trying to make it easy for new users so that they don't need to manipulate .bashrc after reading some web documentation... I guess that's a matter of taste. /Ole
Re: [easybuild] How to setup EB in the global /etc/profile.d/ ?
On 13/10/16 16:11, Ole Holm Nielsen wrote: On 10/13/2016 03:28 PM, Ward Poelmans wrote: For loading Easybuild by default, I would use a Lmod default collection. Could you possibly offer a specific example of what you mean? Like a complete script to put into /etc/profile.d/ ? I'm afraid I don't have one. We don't load Easybuild by default for our users. But you should take a look at: https://lmod.readthedocs.io/en/latest/070_standard_modules.html Thanks, this page is actually useful and specific! So I've created a new file /etc/profile.d/z01_EasyBuild.sh (to be loaded *after* Lmod's z00_* scripts) with this content: if [ -z "$__Init_Default_Modules" ]; then export __Init_Default_Modules=1; # module --initial_load restore export EASYBUILD_MODULES_TOOL=Lmod export EASYBUILD_PREFIX=/home/modules module use $EASYBUILD_PREFIX/modules/all Is this right? Are the modules really in /home/modules/modules/all? module load EasyBuild else module refresh fi This seems to do what I want: Initialize EB with the specific usage of Lmod and our EASYBUILD_PREFIX. N.B.: This is on CentOS 7.2. The recommended line "module --initial_load restore" generates an unwarranted message: The system default contains no modules (env var: LMOD_SYSTEM_DEFAULT_MODULES is empty) No changes in loaded modules so I got rid of it by commenting this line out. Well, you need to define $LMOD_SYSTEM_DEFAULT_MODULES :) Question: Does anyone see potential problems or bad side effects with such a global setup of EB for all users? Having a proper default setup in place makes sense, but users can easily override it with their own config file or via environment variables. The only thing we do is load Lmod itself (a symlink to init/bash in /etc/profile.d). Question: How do your users initialize the usage of EB? They can define a config file, or define environment variables (typically via .bashrc), up to them... K.
Re: [easybuild] How to setup EB in the global /etc/profile.d/ ?
On 10/13/2016 03:28 PM, Ward Poelmans wrote: For loading Easybuild by default, I would use a Lmod default collection. Could you possibly offer a specific example of what you mean? Like a complete script to put into /etc/profile.d/ ? I'm afraid I don't have one. We don't load Easybuild by default for our users. But you should take a look at: https://lmod.readthedocs.io/en/latest/070_standard_modules.html Thanks, this page is actually useful and specific! So I've created a new file /etc/profile.d/z01_EasyBuild.sh (to be loaded *after* Lmod's z00_* scripts) with this content: if [ -z "$__Init_Default_Modules" ]; then export __Init_Default_Modules=1; # module --initial_load restore export EASYBUILD_MODULES_TOOL=Lmod export EASYBUILD_PREFIX=/home/modules module use $EASYBUILD_PREFIX/modules/all module load EasyBuild else module refresh fi This seems to do what I want: Initialize EB with the specific usage of Lmod and our EASYBUILD_PREFIX. N.B.: This is on CentOS 7.2. The recommended line "module --initial_load restore" generates an unwarranted message: The system default contains no modules (env var: LMOD_SYSTEM_DEFAULT_MODULES is empty) No changes in loaded modules so I got rid of it by commenting this line out. Question: Does anyone see potential problems or bad side effects with such a global setup of EB for all users? The only thing we do is load Lmod itself (a symlink to init/bash in /etc/profile.d). Question: How do your users initialize the usage of EB? /Ole
Re: [easybuild] How to setup EB in the global /etc/profile.d/ ?
On 13-10-16 15:20, Ole Holm Nielsen wrote: > On 10/13/2016 01:54 PM, Ward Poelmans wrote: >> On 13-10-16 13:48, Ole Holm Nielsen wrote: >> >> For loading Easybuild by default, I would use a Lmod default collection. > > Could you possibly offer a specific example of what you mean? Like a > complete script to put into /etc/profile.d/ ? I'm afraid I don't have one. We don't load Easybuild by default for our users. But you should take a look at: https://lmod.readthedocs.io/en/latest/070_standard_modules.html The only thing we do is load Lmod itself (a symlink to init/bash in /etc/profile.d). Ward
Re: [easybuild] How to setup EB in the global /etc/profile.d/ ?
On 10/13/2016 01:54 PM, Ward Poelmans wrote: On 13-10-16 13:48, Ole Holm Nielsen wrote: I would like to enable the EB/Lmod modules to all users in a global bash/tcsh setup script. On CentOS 7.2 initialization is done using scripts in /etc/profile.d/ Basically I want every user shell to execute these commands: export EASYBUILD_MODULES_TOOL=Lmod export EASYBUILD_PREFIX=/home/modules module use $EASYBUILD_PREFIX/modules/all module load EasyBuild The settings you can put in a global config file? Much safer then environment variables. For loading Easybuild by default, I would use a Lmod default collection. Could you possibly offer a specific example of what you mean? Like a complete script to put into /etc/profile.d/ ? Thanks, Ole
Re: [easybuild] How to setup EB in the global /etc/profile.d/ ?
Hi André, On 13/10/16 14:39, André Gemünd wrote: Hi Ward, sorry for interfering with this thread, but *is* there actually a global config file? I asked this some time ago on the ml, and as far as I remember there are only local config files ($XDG_...), no global config e.g. under the installation prefix. See http://easybuild.readthedocs.io/en/latest/Configuration.html#default-configuration-files (and the demo available at http://easybuild.readthedocs.io/en/latest/demos/configuring.html). By default, EasyBuild will consider /etc/easybuild.d/*.cfg, but it's true you can override this via $XDG_CONFIG_DIRS . If that's an issue, we can certainly consider having a hardcoded location in there that can not be tweaked, if this is desirable... regards, Kenneth Cheers Andre - Am 13. Okt 2016 um 13:54 schrieb Ward Poelmans ward.poelm...@ugent.be: On 13-10-16 13:48, Ole Holm Nielsen wrote: I would like to enable the EB/Lmod modules to all users in a global bash/tcsh setup script. On CentOS 7.2 initialization is done using scripts in /etc/profile.d/ Basically I want every user shell to execute these commands: export EASYBUILD_MODULES_TOOL=Lmod export EASYBUILD_PREFIX=/home/modules module use $EASYBUILD_PREFIX/modules/all module load EasyBuild The settings you can put in a global config file? Much safer then environment variables. For loading Easybuild by default, I would use a Lmod default collection. Ward
Re: [easybuild] How to setup EB in the global /etc/profile.d/ ?
Hi Ward, sorry for interfering with this thread, but *is* there actually a global config file? I asked this some time ago on the ml, and as far as I remember there are only local config files ($XDG_...), no global config e.g. under the installation prefix. Cheers Andre - Am 13. Okt 2016 um 13:54 schrieb Ward Poelmans ward.poelm...@ugent.be: > On 13-10-16 13:48, Ole Holm Nielsen wrote: >> I would like to enable the EB/Lmod modules to all users in a global >> bash/tcsh setup script. On CentOS 7.2 initialization is done using >> scripts in /etc/profile.d/ >> >> Basically I want every user shell to execute these commands: >> >> export EASYBUILD_MODULES_TOOL=Lmod >> export EASYBUILD_PREFIX=/home/modules >> module use $EASYBUILD_PREFIX/modules/all >> module load EasyBuild > > The settings you can put in a global config file? Much safer then > environment variables. > > For loading Easybuild by default, I would use a Lmod default collection. > > Ward
Re: [easybuild] How to setup EB in the global /etc/profile.d/ ?
On 13-10-16 13:48, Ole Holm Nielsen wrote: > I would like to enable the EB/Lmod modules to all users in a global > bash/tcsh setup script. On CentOS 7.2 initialization is done using > scripts in /etc/profile.d/ > > Basically I want every user shell to execute these commands: > > export EASYBUILD_MODULES_TOOL=Lmod > export EASYBUILD_PREFIX=/home/modules > module use $EASYBUILD_PREFIX/modules/all > module load EasyBuild The settings you can put in a global config file? Much safer then environment variables. For loading Easybuild by default, I would use a Lmod default collection. Ward
