Hello Di,
On 08/05/2017 19:43, Di Pe wrote:
Interesting discussion about Singularity, there was an older thread here
https://mail.google.com/mail/u/0/#search/singularity/159045c3913ad572
and it would be great to learn what EB leadership thinks about
integration of containers in EB?
To look a little bit into our requirements we would like to make it
easy to share the software stacks we build using EB with
1) other researchers outside our organization who are also running
things on HPC systems
2) other software platforms in the same organization (outside HPC
clusters)
In both cases we want to support a certain version of a containerized
software stack for a certain period of time. Take for example a data
science stack. We would bundle a certain version of R with a certain
version of Python and some other ancillary tools. We would build a
container that has one version of Python2 and one version of R and
another container that has Python3 with R.
I would not need or even want any Lmod environment inside the
container because we may need to tie R and python together. For
example R argparse is linked to a specific version of Python and
Python rpy2 is linked to a specific version or R. This cannot be
handled well with lmod.
As Jens already asked, can you clarify the problem here with modules?
Am I right that this is a chicken-egg situation (Python depends on R, R
depends on Python)?
You could probably break this by installing rpy2 as a separate module?
Another issue that plays into this is size of the container. We want
to keep the container relatively small to facilitate sharing but the
easybuild environment is actually pretty large with sources and
everything. if there was a good way to remove all the unnecessary
things including compiler after the container is created that would be
great.
Sources is not a problem, those are kept in a different location
(configurable via --sourcepath).
Cleaning up build modules that only serve as build dependencies for
other things shouldn't be too hard either.
Getting rid of the compilers as a whole is a different matter though,
there are runtime dependencies on libraries provided by the compiler...
Yet another issue is that non-HPC people are not used to lmod. So if I
have hadoop or devops folks they need to know a little bit of lmod to
be able to troubleshoot the software stack and because that is not the
case (at least here) they are building stuff by hand or are using
crappy Ubuntu debs even through the HPC person one floor down has
already done the (better) job using EB. In that sense it would be
great if we could use EB to create a container stack that support only
one version of each software and wherethe software inside the
container simply installs to /usr/local to they can use it as a base
and then fuzz with it further if they need to.
I you want to have all the software that is included in the container
available out-of-the-box, you could avoid using modules.
EasyBuild currently doesn't have support for this, but you could just
capture the changes that should be made to the environment and use the
commands that make those changes a part of the container startup script
(or whatever sets up the environment).
Modules are mostly useful to let users juggle with different
versions/builds of software, and make them pick which tools they'd like
to use together.
Another interesting discussion was Singularity vs Docker. After
reading this
https://www.nextplatform.com/2017/04/10/singularity-containers-hpc-reproducibility-mobility/
and trying out singularity I think the answer is that we want to use
both. Docker plain will probably never be usable on HPC systems (See
the story about rejected pull requests) so you have to take a detour
and use Shifter (Shifter needs to be integrated by a HPC sysadmin
requiring root access and a test plan. It is optimized towards and
tested with Slurm.) Singularity works out of the box on HPC and
standalone systems. The benefit of Shifter seems to be that it can use
Docker images natively while singularity requires you to either build
singularity images or import docker images. In my ad-hoc testing I was
having much more problems with building singularity images (e.g FS
corruption ) than importing docker images with singularity. The latter
worked really well and fast.
As far as I know, Shifter is specific to Cray systems. And it also needs
quite a bit of infrastructure to work well, as opposed to Singularity
that doesn't really need anything special.
So perhaps it makes sense to focus on integrating Docker builds into
EB as one could deliver a more comprehensive solution to bigdata,
devops and HPC people.
Interested to hear some thoughts
We're certainly interested to look into integration between EasyBuild
and Singularity, but just don't have the time to dive into it...
A couple of use cases I had in mind:
* use EasyBuild to populate Singularity images; this could be done via
the definition file that specifies how to construct the container, or
maybe as an alternative packaging target to RPM (so letting EasyBuild
construct Singularity images directly)
* use a Singularity container to run EasyBuild in a jailed environment,
shielded from the OS; this would be mainly useful for testing EasyBuild
on different operating systems (regardless of the host OS), and to
ensure that we catch all required dependencies (by using a bare container)
As Jens already mentioned, this would require that you instruct
EasyBuild to build software generically, so that it works on different
processor architectures.
I'm happy to support any efforts on this (answering questions, reviewing
PRs, etc.).
regards,
Kenneth
DP
On Thu, May 4, 2017 at 9:53 AM, Siddiqui, Shahzeb
<[email protected] <mailto:[email protected]>> wrote:
Hello folks,
I would like to find out how Easybuild and Singularity are going
to work together. I am trying to design an eb environment in
Singularity as a container solution to host all of the eb apps in
a prod environment. Is there anyone in the HPC community that is
working on this?
Currently, I have setup a container environment that can build
packages in container and also install RPMs from easybuild via
Artifactory that is done through the bootstrap process.
One of things that puzzles me is how to setup an isolated
container environment that runs /etc/profile for container. I’ve
noticed I need to do this inorder to get module command to work in
container. Currently, I have to do this manually after shelling in.
-bash-4.2$ singularity shell /nfs/grid/software/testing.img
Singularity: Invoking an interactive shell within container...
Singularity.testing.img> module --version
sh: module: command not found
Singularity.testing.img> env | grep MODULEPATH
MODULEPATH_ROOT=/usr/share/modulefiles
MODULEPATH=/modulefiles/Core/:/nfs/grid/software/RHEL7-BUILD/easybuild/modules/all/:/nfs/grid/software/RHEL7/non-easybuild/modules/all/:/nfs/grid/software/RHEL7/easybuild/modules/all/Core:/nfs/grid/software/moduledomains:/etc/modulefiles:/usr/share/modulefiles:/usr/share/modulefiles/Linux:/usr/share/modulefiles/Core:/usr/share/lmod/lmod/modulefiles/Core
Singularity.testing.img> unset MODULEPATH
Singularity.testing.img> . /etc/profile
Singularity.testing.img> . /nfs/grid/software/
module-setup.sh RHEL7/ RHEL7-BUILD/
Singularity.testing.img> . /nfs/grid/software/module-setup.sh
Singularity.testing.img> ml av
-----------------------------------------------------------
/usr/share/lmod/lmod/modulefiles/Core
------------------------------------------------------------
lmod/6.5.1 settarg/6.5.1
----------------------------------------------------
/nfs/grid/software/RHEL7-BUILD/easybuild/modules/all
----------------------------------------------------
EasyBuild/3.1.2
----------------------------------------------------
/nfs/grid/software/RHEL7/easybuild/modules/all/Core
-----------------------------------------------------
Advisor/2017_update1 IGV/2.3.80-Java-1.8.0_92
Java/1.8.0_92 gaussian/16-AVX iompi/2017.01
tbb/2017.2.132
GCC/5.4.0-2.27 Inspector/2017_update1 VTune/2017_update1
gaussian/16-SSE2 (D) ipp/2017.1.132
GCC/6.2.0-2.27 (D) IntelClusterChecker/2017.1.016
daal/2017.1.132 intel/2017.01 itac/2017.1.024
Where:
D: Default Module
Use "module spider" to find all possible modules.
Use "module keyword key1 key2 ..." to search for all possible
modules matching any of the "keys".
Singularity.testing.img> ml EasyBuild
Singularity.testing.img> eb --version
This is EasyBuild 3.1.2 (framework: 3.1.2, easyblocks: 3.1.2) on
host amrndhl1157.pfizer.com <http://amrndhl1157.pfizer.com>.
Singularity.testing.img> eb --show-config
#
# Current EasyBuild configuration
# (C: command line argument, D: default value, E: environment
variable, F: configuration file)
#
buildpath (D) = /home/siddis14/.local/easybuild/build
installpath (D) = /home/siddis14/.local/easybuild
repositorypath (D) = /home/siddis14/.local/easybuild/ebfiles_repo
robot-paths (D) =
/nfs/grid/software/RHEL7-BUILD/easybuild/software/EasyBuild/3.1.2/lib/python2.7/site-packages/easybuild_easyconfigs-3.1.2-py2.7.egg/easybuild/easyconfigs
sourcepath (D) = /home/siddis14/.local/easybuild/sources
Singularity.testing.img> eb zlib-1.2.8.eb
== temporary log file in case of crash
/tmp/eb-BfnLvm/easybuild-ybJc4_.log
== zlib/1.2.8 is already installed (module found), skipping
== No easyconfigs left to be built.
== Build succeeded for 0 out of 0
== Temporary log file(s) /tmp/eb-BfnLvm/easybuild-ybJc4_.log* have
been removed.
== Temporary directory /tmp/eb-BfnLvm has been removed.
Singularity.testing.img> eb zlib-1.2.8.eb --rebuild
== temporary log file in case of crash
/tmp/eb-IMN3vl/easybuild-lWCjI1.log
== processing EasyBuild easyconfig
/nfs/grid/software/RHEL7-BUILD/easybuild/software/EasyBuild/3.1.2/lib/python2.7/site-packages/easybuild_easyconfigs-3.1.2-py2.7.egg/easybuild/easyconfigs/z/zlib/zlib-1.2.8.eb
== building and installing zlib/1.2.8. <http://1.2.8.>..
== fetching files...
== creating build dir, resetting environment...
== unpacking...
== patching...
== preparing...
== configuring...
== building...
== testing...
== installing...
== taking care of extensions...
== postprocessing...
== sanity checking...
== cleaning up...
== creating module...
== permissions...
== packaging...
== COMPLETED: Installation ended successfully
== Results of the build can be found in the log file(s)
/home/siddis14/.local/easybuild/software/zlib/1.2.8/easybuild/easybuild-zlib-1.2.8-20170504.163248.log
== Build succeeded for 1 out of 1
== Temporary log file(s) /tmp/eb-IMN3vl/easybuild-lWCjI1.log* have
been removed.
== Temporary directory /tmp/eb-IMN3vl has been removed.
Singularity.testing.img> module use $HOME/.local/easybuild/modules/all
Due to MODULEPATH changes the following have been reloaded:
1) EasyBuild/3.1.2
Singularity.testing.img> module av
--------------------------------------------------------
/home/siddis14/.local/easybuild/modules/all
---------------------------------------------------------
EasyBuild/3.1.2 (L,D) M4/1.4.17 (D) zlib/1.2.8 (D)
-----------------------------------------------------------
/usr/share/lmod/lmod/modulefiles/Core
------------------------------------------------------------
lmod/6.5.1 settarg/6.5.1
----------------------------------------------------
/nfs/grid/software/RHEL7-BUILD/easybuild/modules/all
----------------------------------------------------
EasyBuild/3.1.2
----------------------------------------------------
/nfs/grid/software/RHEL7/easybuild/modules/all/Core
-----------------------------------------------------
Advisor/2017_update1 IGV/2.3.80-Java-1.8.0_92
Java/1.8.0_92 gaussian/16-AVX iompi/2017.01
tbb/2017.2.132
GCC/5.4.0-2.27 Inspector/2017_update1
VTune/2017_update1 gaussian/16-SSE2 (D) ipp/2017.1.132
GCC/6.2.0-2.27 (D) IntelClusterChecker/2017.1.016
daal/2017.1.132 intel/2017.01 itac/2017.1.024
Where:
L: Module is loaded
D: Default Module
Use "module spider" to find all possible modules.
Use "module keyword key1 key2 ..." to search for all possible
modules matching any of the "keys".
Shahzeb Siddiqui
HPC Linux Engineer
B2220-447.2
Groton, CT
