Hi,
On 08/05/17 19:43, Di Pe wrote:
Interesting discussion about Singularity, there was an older thread here
https://mail.google.com/mail/u/0/#search/singularity/159045c3913ad572
This link doesn't work on my system.
and it would be great to learn what EB leadership thinks about
integration of containers in EB?
To look a little bit into our requirements we would like to make it easy
to share the software stacks we build using EB with
1) other researchers outside our organization who are also running
things on HPC systems
2) other software platforms in the same organization (outside HPC
clusters)
My answer here would be: share your EasyConfig files. If everything is
setup correctly you will have an archive of installed EaysConfig files.
You can share these with other people for use on their system. If you
create 1 meta easyconfig that depends on all the software you need they
can reproduce your environment with a single eb metaeasyconfig.eb --robot
In both cases we want to support a certain version of a containerized
software stack for a certain period of time.
What does a containerized software stack mean? Do you mean it is
shielded form the other software on the system?
Take for example a data
science stack. We would bundle a certain version of R with a certain
version of Python and some other ancillary tools. We would build a
container that has one version of Python2 and one version of R and
another container that has Python3 with R.
You can create 2 meta modules for these.
myfirstsmodule:
- R 3.0
- R 3.0 dependencies
- Python 2.7
- Python 2.7 dependencies
mysecondmodule:
- R 3.1.0
- R 3.1.0 dependencies
- Python3
- Python3 dependencies
I would not need or even want any Lmod environment inside the container
because we may need to tie R and python together. For example R argparse
is linked to a specific version of Python and Python rpy2 is linked to a
specific version or R. This cannot be handled well with lmod.
Can you please explain a bit more what environment modules can't handle
well here?
A case where 2 python modules would be loaded at the same time?
So perhaps it makes sense to focus on integrating Docker builds into EB
as one could deliver a more comprehensive solution to bigdata, devops
and HPC people.
Interested to hear some thoughts
The above being said, we do use singularity on our systems, and use
Easybuild created software inside the containers. However, that would
indeed require you to install lmod (or environment-modules) in your
container.
We include in our /etc/singularity.conf
bind path = /apps
bind path = /etc/modulefiles # (Default location for environment-modules)
So all our EasyBuild built software is available inside the containers
(and works if they have the correct glibc)
But this is probably the reverse of what you want to accomplish.
So for me the answer is:
If you want to distribute your software, distribute your Easyconfig
files, and show the people how easily they can reproduce your build,
should only be one command, if something goes wrong this should be
considered a bug in EasyBuild (or a quirky thing in the softwares build
procedure, which a clever person could fix in an EasyBlock)
EasyBuild will by default try to optimize for the hardware environment
it was built on anyway, that is kind of the point of it, so creating a
container and then sharing it and expecting the software to work on a
different cpu is not something that's currently on our radar.
The entire point of EB was to have reproducible builds that non the less
take advantage of your cpu architecture.
(these are my views at the moment, if someone wants to put a lot of work
into extending EasyBuild to do different things than what was it's
original point, I guess we would probably still merge it in, if it works
and is useful to other people)
@Siddiqui, internally we have a wrapper around singularity that sets the
environment and creates a list of commands that basically first invoke
singularity and then run the profile and set other environment
variables. It was in part (almost fully) written by Kenenth, so it
integrates well with EasyBuild, I should inquire with Kenneth if we can
make this public, or at least document what it does so you can benefit
from his work :)
Regards,
Jens Timmerman
DP
On Thu, May 4, 2017 at 9:53 AM, Siddiqui, Shahzeb
<[email protected] <mailto:[email protected]>> wrote:
Hello folks,____
__ __
I would like to find out how Easybuild and Singularity are going to
work together. I am trying to design an eb environment in
Singularity as a container solution to host all of the eb apps in a
prod environment. Is there anyone in the HPC community that is
working on this?____
__ __
Currently, I have setup a container environment that can build
packages in container and also install RPMs from easybuild via
Artifactory that is done through the bootstrap process.____
__ __
One of things that puzzles me is how to setup an isolated container
environment that runs /etc/profile for container. I’ve noticed I
need to do this inorder to get module command to work in container.
Currently, I have to do this manually after shelling in.____
__ __
__ __
-bash-4.2$ singularity shell /nfs/grid/software/testing.img __ __
Singularity: Invoking an interactive shell within container...____
__ __
Singularity.testing.img> module --version____
sh: module: command not found____
Singularity.testing.img> env | grep MODULEPATH____
MODULEPATH_ROOT=/usr/share/modulefiles____
MODULEPATH=/modulefiles/Core/:/nfs/grid/software/RHEL7-BUILD/easybuild/modules/all/:/nfs/grid/software/RHEL7/non-easybuild/modules/all/:/nfs/grid/software/RHEL7/easybuild/modules/all/Core:/nfs/grid/software/moduledomains:/etc/modulefiles:/usr/share/modulefiles:/usr/share/modulefiles/Linux:/usr/share/modulefiles/Core:/usr/share/lmod/lmod/modulefiles/Core____
Singularity.testing.img> unset MODULEPATH____
Singularity.testing.img> . /etc/profile____
Singularity.testing.img> . /nfs/grid/software/____
module-setup.sh RHEL7/ RHEL7-BUILD/ ____
Singularity.testing.img> . /nfs/grid/software/module-setup.sh ____
Singularity.testing.img> ml av____
__ __
-----------------------------------------------------------
/usr/share/lmod/lmod/modulefiles/Core
------------------------------------------------------------____
lmod/6.5.1 settarg/6.5.1____
__ __
----------------------------------------------------
/nfs/grid/software/RHEL7-BUILD/easybuild/modules/all
----------------------------------------------------____
EasyBuild/3.1.2____
__ __
----------------------------------------------------
/nfs/grid/software/RHEL7/easybuild/modules/all/Core
-----------------------------------------------------____
Advisor/2017_update1 IGV/2.3.80-Java-1.8.0_92
Java/1.8.0_92 gaussian/16-AVX iompi/2017.01
tbb/2017.2.132____
GCC/5.4.0-2.27 Inspector/2017_update1
VTune/2017_update1 gaussian/16-SSE2 (D) ipp/2017.1.132____
GCC/6.2.0-2.27 (D) IntelClusterChecker/2017.1.016
daal/2017.1.132 intel/2017.01 itac/2017.1.024____
__ __
Where:____
D: Default Module____
__ __
Use "module spider" to find all possible modules.____
Use "module keyword key1 key2 ..." to search for all possible
modules matching any of the "keys".____
__ __
Singularity.testing.img> ml EasyBuild____
Singularity.testing.img> eb --version____
This is EasyBuild 3.1.2 (framework: 3.1.2, easyblocks: 3.1.2) on
host amrndhl1157.pfizer.com <http://amrndhl1157.pfizer.com>.____
Singularity.testing.img> eb --show-config____
#____
# Current EasyBuild configuration____
# (C: command line argument, D: default value, E: environment
variable, F: configuration file)____
#____
buildpath (D) = /home/siddis14/.local/easybuild/build____
installpath (D) = /home/siddis14/.local/easybuild____
repositorypath (D) = /home/siddis14/.local/easybuild/ebfiles_repo____
robot-paths (D) =
/nfs/grid/software/RHEL7-BUILD/easybuild/software/EasyBuild/3.1.2/lib/python2.7/site-packages/easybuild_easyconfigs-3.1.2-py2.7.egg/easybuild/easyconfigs____
sourcepath (D) = /home/siddis14/.local/easybuild/sources____
Singularity.testing.img> eb zlib-1.2.8.eb____
== temporary log file in case of crash
/tmp/eb-BfnLvm/easybuild-ybJc4_.log____
== zlib/1.2.8 is already installed (module found), skipping____
== No easyconfigs left to be built.____
== Build succeeded for 0 out of 0____
== Temporary log file(s) /tmp/eb-BfnLvm/easybuild-ybJc4_.log* have
been removed.____
== Temporary directory /tmp/eb-BfnLvm has been removed.____
Singularity.testing.img> eb zlib-1.2.8.eb --rebuild____
== temporary log file in case of crash
/tmp/eb-IMN3vl/easybuild-lWCjI1.log____
== processing EasyBuild easyconfig
/nfs/grid/software/RHEL7-BUILD/easybuild/software/EasyBuild/3.1.2/lib/python2.7/site-packages/easybuild_easyconfigs-3.1.2-py2.7.egg/easybuild/easyconfigs/z/zlib/zlib-1.2.8.eb____
== building and installing zlib/1.2.8. <http://1.2.8.>..____
== fetching files...____
== creating build dir, resetting environment...____
== unpacking...____
== patching...____
== preparing...____
== configuring...____
== building...____
== testing...____
== installing...____
== taking care of extensions...____
== postprocessing...____
== sanity checking...____
== cleaning up...____
== creating module...____
== permissions...____
== packaging...____
== COMPLETED: Installation ended successfully____
== Results of the build can be found in the log file(s)
/home/siddis14/.local/easybuild/software/zlib/1.2.8/easybuild/easybuild-zlib-1.2.8-20170504.163248.log____
== Build succeeded for 1 out of 1____
== Temporary log file(s) /tmp/eb-IMN3vl/easybuild-lWCjI1.log* have
been removed.____
== Temporary directory /tmp/eb-IMN3vl has been removed.____
__ __
Singularity.testing.img> module use
$HOME/.local/easybuild/modules/all____
__ __
Due to MODULEPATH changes the following have been reloaded:____
1) EasyBuild/3.1.2____
__ __
Singularity.testing.img> module av____
__ __
--------------------------------------------------------
/home/siddis14/.local/easybuild/modules/all
---------------------------------------------------------____
EasyBuild/3.1.2 (L,D) M4/1.4.17 (D) zlib/1.2.8 (D)____
__ __
-----------------------------------------------------------
/usr/share/lmod/lmod/modulefiles/Core
------------------------------------------------------------____
lmod/6.5.1 settarg/6.5.1____
__ __
----------------------------------------------------
/nfs/grid/software/RHEL7-BUILD/easybuild/modules/all
----------------------------------------------------____
EasyBuild/3.1.2____
__ __
----------------------------------------------------
/nfs/grid/software/RHEL7/easybuild/modules/all/Core
-----------------------------------------------------____
Advisor/2017_update1 IGV/2.3.80-Java-1.8.0_92
Java/1.8.0_92 gaussian/16-AVX iompi/2017.01
tbb/2017.2.132____
GCC/5.4.0-2.27 Inspector/2017_update1
VTune/2017_update1 gaussian/16-SSE2 (D) ipp/2017.1.132____
GCC/6.2.0-2.27 (D) IntelClusterChecker/2017.1.016
daal/2017.1.132 intel/2017.01 itac/2017.1.024____
__ __
Where:____
L: Module is loaded____
D: Default Module____
__ __
Use "module spider" to find all possible modules.____
Use "module keyword key1 key2 ..." to search for all possible
modules matching any of the "keys".____
__ __
__ __
__ __
Shahzeb Siddiqui____
HPC Linux Engineer____
B2220-447.2____
Groton, CT____
__ __
