Re: [easybuild] eb in singularity

[CEDRIC CLERGET]

Hello Shahzeb, 

If you want to load /etc/profile each time you launch image, you could add -l 
to shebang (#!/bin/sh -l) in /exec /run /shell files located in container, it 
will simulate a shell login and will load /etc/profile 

Best regards, 
Cédric Clerget 


De: "Siddiqui, Shahzeb"  
À: easybuild@lists.ugent.be 
Cc: "Gregory M. Kurtzer"  
Envoyé: Jeudi 4 Mai 2017 18:53:35 
Objet: [easybuild] eb in singularity 



Hello folks, 



I would like to find out how Easybuild and Singularity are going to work 
together. I am trying to design an eb environment in Singularity as a container 
solution to host all of the eb apps in a prod environment. Is there anyone in 
the HPC community that is working on this? 



Currently, I have setup a container environment that can build packages in 
container and also install RPMs from easybuild via Artifactory that is done 
through the bootstrap process. 



One of things that puzzles me is how to setup an isolated container environment 
that runs /etc/profile for container. I’ve noticed I need to do this inorder to 
get module command to work in container. Currently, I have to do this manually 
after shelling in. 





-bash-4.2$ singularity shell /nfs/grid/software/testing.img 

Singularity: Invoking an interactive shell within container... 



Singularity.testing.img> module --version 

sh: module: command not found 

Singularity.testing.img> env | grep MODULEPATH 

MODULEPATH_ROOT=/usr/share/modulefiles 

MODULEPATH=/modulefiles/Core/:/nfs/grid/software/RHEL7-BUILD/easybuild/modules/all/:/nfs/grid/software/RHEL7/non-easybuild/modules/all/:/nfs/grid/software/RHEL7/easybuild/modules/all/Core:/nfs/grid/software/moduledomains:/etc/modulefiles:/usr/share/modulefiles:/usr/share/modulefiles/Linux:/usr/share/modulefiles/Core:/usr/share/lmod/lmod/modulefiles/Core
 

Singularity.testing.img> unset MODULEPATH 

Singularity.testing.img> . /etc/profile 

Singularity.testing.img> . /nfs/grid/software/ 

module-setup.sh RHEL7/ RHEL7-BUILD/ 

Singularity.testing.img> . /nfs/grid/software/module-setup.sh 

Singularity.testing.img> ml av 



--- 
/usr/share/lmod/lmod/modulefiles/Core 
 

lmod/6.5.1 settarg/6.5.1 



 
/nfs/grid/software/RHEL7-BUILD/easybuild/modules/all 
 

EasyBuild/3.1.2 



 
/nfs/grid/software/RHEL7/easybuild/modules/all/Core 
- 

Advisor/2017_update1 IGV/2.3.80-Java-1.8.0_92 Java/1.8.0_92 gaussian/16-AVX 
iompi/2017.01 tbb/2017.2.132 

GCC/5.4.0-2.27 Inspector/2017_update1 VTune/2017_update1 gaussian/16-SSE2 (D) 
ipp/2017.1.132 

GCC/6.2.0-2.27 (D) IntelClusterChecker/2017.1.016 daal/2017.1.132 intel/2017.01 
itac/2017.1.024 



Where: 

D: Default Module 



Use "module spider" to find all possible modules. 

Use "module keyword key1 key2 ..." to search for all possible modules matching 
any of the "keys". 



Singularity.testing.img> ml EasyBuild 

Singularity.testing.img> eb --version 

This is EasyBuild 3.1.2 (framework: 3.1.2, easyblocks: 3.1.2) on host 
amrndhl1157.pfizer.com. 

Singularity.testing.img> eb --show-config 

# 

# Current EasyBuild configuration 

# (C: command line argument, D: default value, E: environment variable, F: 
configuration file) 

# 

buildpath (D) = /home/siddis14/.local/easybuild/build 

installpath (D) = /home/siddis14/.local/easybuild 

repositorypath (D) = /home/siddis14/.local/easybuild/ebfiles_repo 

robot-paths (D) = 
/nfs/grid/software/RHEL7-BUILD/easybuild/software/EasyBuild/3.1.2/lib/python2.7/site-packages/easybuild_easyconfigs-3.1.2-py2.7.egg/easybuild/easyconfigs
 

sourcepath (D) = /home/siddis14/.local/easybuild/sources 

Singularity.testing.img> eb zlib-1.2.8.eb 

== temporary log file in case of crash /tmp/eb-BfnLvm/easybuild-ybJc4_.log 

== zlib/1.2.8 is already installed (module found), skipping 

== No easyconfigs left to be built. 

== Build succeeded for 0 out of 0 

== Temporary log file(s) /tmp/eb-BfnLvm/easybuild-ybJc4_.log* have been 
removed. 

== Temporary directory /tmp/eb-BfnLvm has been removed. 

Singularity.testing.img> eb zlib-1.2.8.eb --rebuild 

== temporary log file in case of crash /tmp/eb-IMN3vl/easybuild-lWCjI1.log 

== processing EasyBuild easyconfig 
/nfs/grid/software/RHEL7-BUILD/easybuild/software/EasyBuild/3.1.2/lib/python2.7/site-packages/easybuild_easyconfigs-3.1.2-py2.7.egg/easybuild/easyconfigs/z/zlib/zlib-1.2.8.eb
 

== building and installing zlib/1.2.8... 

== fetching files... 

== creating build dir, resetting environment... 

== unpacking... 

== patching... 

== preparing... 

== configuring... 

== building... 

== testing... 

== installing... 

== taking care of extensions... 

== postprocessing... 

[easybuild] eb in singularity

[Siddiqui, Shahzeb]

Hello folks,

I would like to find out how Easybuild and Singularity are going to work 
together. I am trying to design an eb environment in Singularity as a container 
solution to host all of the eb apps in a prod environment. Is there anyone in 
the HPC community that is working on this?

Currently, I have setup a container environment that can build packages in 
container and also install RPMs from easybuild via Artifactory that is done 
through the bootstrap process.

One of things that puzzles me is how to setup an isolated container environment 
that runs /etc/profile for container. I've noticed I need to do this inorder to 
get module command to work in container. Currently, I have to do this manually 
after shelling in.


-bash-4.2$ singularity shell /nfs/grid/software/testing.img
Singularity: Invoking an interactive shell within container...

Singularity.testing.img> module --version
sh: module: command not found
Singularity.testing.img> env | grep MODULEPATH
MODULEPATH_ROOT=/usr/share/modulefiles
MODULEPATH=/modulefiles/Core/:/nfs/grid/software/RHEL7-BUILD/easybuild/modules/all/:/nfs/grid/software/RHEL7/non-easybuild/modules/all/:/nfs/grid/software/RHEL7/easybuild/modules/all/Core:/nfs/grid/software/moduledomains:/etc/modulefiles:/usr/share/modulefiles:/usr/share/modulefiles/Linux:/usr/share/modulefiles/Core:/usr/share/lmod/lmod/modulefiles/Core
Singularity.testing.img> unset MODULEPATH
Singularity.testing.img> . /etc/profile
Singularity.testing.img> . /nfs/grid/software/
module-setup.sh  RHEL7/   RHEL7-BUILD/
Singularity.testing.img> . /nfs/grid/software/module-setup.sh
Singularity.testing.img> ml av

--- 
/usr/share/lmod/lmod/modulefiles/Core 

   lmod/6.5.1settarg/6.5.1

 
/nfs/grid/software/RHEL7-BUILD/easybuild/modules/all 

   EasyBuild/3.1.2

 
/nfs/grid/software/RHEL7/easybuild/modules/all/Core 
-
   Advisor/2017_update1IGV/2.3.80-Java-1.8.0_92  Java/1.8.0_92  
   gaussian/16-AVX iompi/2017.01  tbb/2017.2.132
   GCC/5.4.0-2.27  Inspector/2017_update1
VTune/2017_update1gaussian/16-SSE2 (D)ipp/2017.1.132
   GCC/6.2.0-2.27   (D)IntelClusterChecker/2017.1.016
daal/2017.1.132   intel/2017.01   itac/2017.1.024

  Where:
   D:  Default Module

Use "module spider" to find all possible modules.
Use "module keyword key1 key2 ..." to search for all possible modules matching 
any of the "keys".

Singularity.testing.img> ml EasyBuild
Singularity.testing.img> eb --version
This is EasyBuild 3.1.2 (framework: 3.1.2, easyblocks: 3.1.2) on host 
amrndhl1157.pfizer.com.
Singularity.testing.img> eb --show-config
#
# Current EasyBuild configuration
# (C: command line argument, D: default value, E: environment variable, F: 
configuration file)
#
buildpath  (D) = /home/siddis14/.local/easybuild/build
installpath(D) = /home/siddis14/.local/easybuild
repositorypath (D) = /home/siddis14/.local/easybuild/ebfiles_repo
robot-paths(D) = 
/nfs/grid/software/RHEL7-BUILD/easybuild/software/EasyBuild/3.1.2/lib/python2.7/site-packages/easybuild_easyconfigs-3.1.2-py2.7.egg/easybuild/easyconfigs
sourcepath (D) = /home/siddis14/.local/easybuild/sources
Singularity.testing.img> eb zlib-1.2.8.eb
== temporary log file in case of crash /tmp/eb-BfnLvm/easybuild-ybJc4_.log
== zlib/1.2.8 is already installed (module found), skipping
== No easyconfigs left to be built.
== Build succeeded for 0 out of 0
== Temporary log file(s) /tmp/eb-BfnLvm/easybuild-ybJc4_.log* have been removed.
== Temporary directory /tmp/eb-BfnLvm has been removed.
Singularity.testing.img> eb zlib-1.2.8.eb --rebuild
== temporary log file in case of crash /tmp/eb-IMN3vl/easybuild-lWCjI1.log
== processing EasyBuild easyconfig 
/nfs/grid/software/RHEL7-BUILD/easybuild/software/EasyBuild/3.1.2/lib/python2.7/site-packages/easybuild_easyconfigs-3.1.2-py2.7.egg/easybuild/easyconfigs/z/zlib/zlib-1.2.8.eb
== building and installing zlib/1.2.8...
== fetching files...
== creating build dir, resetting environment...
== unpacking...
== patching...
== preparing...
== configuring...
== building...
== testing...
== installing...
== taking care of extensions...
== postprocessing...
== sanity checking...
== cleaning up...
== creating module...
== permissions...
== packaging...
== COMPLETED: Installation ended successfully
== Results of the build can be found in the log file(s) 
/home/siddis14/.local/easybuild/software/zlib/1.2.8/easybuild/easybuild-zlib-1.2.8-20170504.163248.log
== Build succeeded for 1 out of 1
== Temporary log file(s) /tmp/eb-IMN3vl/easybuild-lWCjI1.log* have been removed.
== Temporary directory /tmp/eb-IMN3vl has been removed.

Re: [easybuild] how to specify choices for dependencies?

[Andreas Hilboll]

Thanks, Alan and Gizo,

I'll take a look into this.

Would it be welcome to include this functionality in an upcoming PR to
include ecCodes and EMOS?  Or rather not, since this apparently depends
on lmod being the module system of choice?

Cheers,
  Andreas



Alan O'Cais writes:

> This sounds like a reasonable approach to me. You should probably specify one 
> python as one of your `builddependencies` and then create a modluafooter that 
> ensures you have a Python loaded, something like
> if not isloaded("Python") then
>   load("Python")
> end
> That should still allow you to keep the python bindings as part of your 
> sanity check too.
>
> Alan
>
> On 2 May 2017 at 21:12, 
> > wrote:
>
> Hi,
>
> you could implemet this logic as some lua function defined in SitePackage.lua 
> and then use the
> option modluafooter in easyconfig of emos. Not very nice, but should work.
>
> http://lmod.readthedocs.io/en/latest/050_lua_modulefiles.html
>
> Best,
> Gizo
>
> Quoting Andreas Hilboll >:
>
> Hi all,
>
> I want to create easyconfigs for ecCodes[1] and EMOS[2].  ecCodes comes
> with Python bindings, so I'm setting a
>
> versionsuffix = '-Python-%(pyver)s'
>
> EMOS depends on ecCodes, but does not depend on Python at all.
>
> I have built two ecCodes modules, one for Python-2.7.11 and one for
> Python-3.5.1.  I'm wondering how to specify the dependencies in the EMOS
> easyconfig, without having to provide a Python-dependent easyconfig for
> EMOS.  Ideally, when loading EMOS, lmod should pick the ecCodes module
> which causes the least conflict:
>
> - if no other Python is loaded, chose any default
> - if either Python/2.7.11 or Python/3.5.1 are already loaded, chose the
>   matching ecCodes module
>
> Is this somehow possible?
>
> Cheers,
>   Andreas.
>
> [1] https://software.ecmwf.int/wiki/display/ECC
> [2] https://software.ecmwf.int/wiki/display/EMOS
>
> --
> Dr. Andreas Hilboll
>
> Center for Marine Environmental Sciences (MARUM)
> - AND -
> Institute of Environmental Physics (IUP)
>
> University of Bremen
>
> NW1 / S3132
> Otto-Hahn-Allee 1
> D-28359 Bremen
> Germany
>
> +49(0)421 218 62133  (phone)
> +49(0)421 218 98 62133 (fax)
> http://www.iup.uni-bremen.de/~hilboll
>
>
> --
> Dr. Gizo Nanava
> Leibniz Universitaet IT Services
> Leibniz Universitaet Hannover
> Schlosswender Str. 5
> D-30159 Hannover
> Tel +49 511 762 7919085
> http://www.luis.uni-hannover.de


-- 
Dr. Andreas Hilboll

Center for Marine Environmental Sciences (MARUM)
- AND -
Institute of Environmental Physics (IUP)

University of Bremen

NW1 / S3132
Otto-Hahn-Allee 1
D-28359 Bremen
Germany

+49(0)421 218 62133  (phone)
+49(0)421 218 98 62133 (fax)
http://www.iup.uni-bremen.de/~hilboll

Re: [easybuild] Software/easyconfigs update avail check

[Ward Poelmans]

On 04-05-17 10:39, Jens Timmerman wrote:
> 
> On 04/05/2017 08:31, Henkel, Andreas wrote:
>> Hi, 
>>
>> Recently we started using eb for our new cluster. Yesterday, in our group 
>> meeting a question was raised concerning updates and security patches for 
>> installed software similar to apt update/upgrade, yum update,...?
>> Or is there a routine to check for newer releases of installed easyconfigs? 
> Short answer: No
> 
> There are no such systems in place, EasyBuild does not have a security
> team, so there are no systems in place for installing security updates.

There is currently one exception: openssl. We have an easyconfig for it
but we put it as a osdependency in all cases. So whatever security
updates happen to the OS provided version of openssl, Easybuild uses it.

Ward

Re: [easybuild] Software/easyconfigs update avail check

[Alvarez, Damian]

Just to add on that, you could create drop-in replacements for selected 
packages. You could create MariaDB-10.X-intel-2016b.eb (for instance), update 
the easyconfig file every time there is a new release in the 10.X series, and 
reinstall it on top of the old one, so bug fixes and security updates could be 
applied transparently* to the users and to other packages. That’s normally 
something you don’t want to do for reproducibility reasons, but it makes sense 
in certain cases.

Damian

*Static libraries might still be an issue though.

On 04/05/17 10:39, "easybuild-requ...@lists.ugent.be on behalf of Jens 
Timmerman"  wrote:

I do realize there exist a few bad scenarios, e.g. software that opens
up sockets to listen to incomming connections such as MariaDB and
mongoDB. If these packages have security issues (or are badly
configured) your users risk loss of confidentiality on their data or
even arbitrary code executing as their user (not root, not a sudo user).
Or a software package that has issues and opens up a users files to the
world.
This would require you to either educate your users not to use the bad
software in an unsafe way, or manually remove it and install a patched
version and tell them to use that version. If you are a non admin user
using EasyBuild to install your own software on a system, you will need
to follow up on this as there is currently  nobody paid to do this
centrally for all EB easyconfigs.





Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt

Re: [easybuild] Software/easyconfigs update avail check

[Jens Timmerman]

Hi Andreas


On 04/05/2017 08:31, Henkel, Andreas wrote:
> Hi, 
>
> Recently we started using eb for our new cluster. Yesterday, in our group 
> meeting a question was raised concerning updates and security patches for 
> installed software similar to apt update/upgrade, yum update,...?
> Or is there a routine to check for newer releases of installed easyconfigs? 
Short answer: No

There are no such systems in place, EasyBuild does not have a security
team, so there are no systems in place for installing security updates.

The main point of EasyBuild is to get user software installed on the
system, usable by users, for users. You would not run EasyBuild as root
(this was even explicitly very hard to do until as of very recent, now
it has a very clear config option informing you you should not use that
option, but some people have their own reasons to do things as root) And
you would not install software with setuid, and you would not run
EasyBuild installed software as root (or more a privileged sudo capable
user) but only as a non privileged user.
As such from the point of a system administrator the security of your
system would not be impacted by software installed by EasyBuild, at
least, not any more then any software any user can scp to their home dir
and run there.
(At least, unless you mount home and scratch as no exec)
Since we allow our users to write and compile their own software,
EasyBuild was developed to provide centrally installed (and optimized)
software as a convenience to our user, it has never been in our scope to
have EasyBuild installed software to be the only allowable software to run.

Furthermore, EasyBuild doesn't 'update' installed software. An installed
software package will always stay installed, newer versions will be
installed alongside it, not replace it, this is one of the main reasons
to use EasyBuild, so you can easily use different versions of the same
package on the same system.

And lastly, the easyconfig files provided by EasyBuild should be seen as
only 'examples'. People can privately have 1000's of applications that
are not known and never published to the EasyBuild, so no central
EasyBuild security team could ever notify you of updates for these packages.

I do realize there exist a few bad scenarios, e.g. software that opens
up sockets to listen to incomming connections such as MariaDB and
mongoDB. If these packages have security issues (or are badly
configured) your users risk loss of confidentiality on their data or
even arbitrary code executing as their user (not root, not a sudo user).
Or a software package that has issues and opens up a users files to the
world.
This would require you to either educate your users not to use the bad
software in an unsafe way, or manually remove it and install a patched
version and tell them to use that version. If you are a non admin user
using EasyBuild to install your own software on a system, you will need
to follow up on this as there is currently  nobody paid to do this
centrally for all EB easyconfigs.

There are tools out there (like the fedora packagers tools) that will
scan upstream links to automatically inform packagers of new installs.
But as far as I know nobody has looked at this yet EasyBuild.
Some work was started in to making it easy for you to bump all the
dependencies of a given package to the latest version, [0] but this
still relies in someone figuring out a new version is available, and
adding it to the central (or local) Easyconfigs repository.

I hope this clears up a few things for you.

Regards,
Jens Timmerman
> Best, 
> Andreas Henkel
[0] https://github.com/hpcugent/easybuild-framework/pull/2136

[easybuild] Software/easyconfigs update avail check

[Henkel, Andreas]

Hi, 

Recently we started using eb for our new cluster. Yesterday, in our group 
meeting a question was raised concerning updates and security patches for 
installed software similar to apt update/upgrade, yum update,...?
Or is there a routine to check for newer releases of installed easyconfigs? 

Best, 
Andreas Henkel