Here is conversation between Greg and I, I have not got any response from Kenneth but would like to know if EB will build singularity containers or will we need to create singularity definition files and run eb inside the definition file.
If each application is a container, we need a orchestration tool to resolve dependencies. A per app container model is difficult and will take some time, whereas a software collection model in a container is straight forward. Since I build my packages as RPMs, I can install the packages as RPMs and also build eb packages as RPMs. This way if I need to recreate the container or create custom container environment I can do this via yum groups. eb needs a similar feature like FPM to generate RPM but now write out a singularity definition file that is then used to bootstrap the image that can then install the eb package as RPM or do a eb build inside container. On Tue, Apr 11, 2017 at 1:04 PM, Siddiqui, Shahzeb <[email protected]<mailto:[email protected]>> wrote: Hello Gregory, It was easy for me to create a single singularity container that has all my collection of software packages, but it was difficult for me to figure out how to create individual containers that can interact together. Ahh, and there-in lies orchestration of containers. Initially I would try to setup Lmod in container, I was able to figure this out and then I was able to build EB packages. My concern is a single collection scheme seems to be limited in the sense that software packages are not truly independent. I would expect that I can install Singularity containers like rpm which have all the dependencies that pick up other singularity containers. Initially I can start out by installing dependency for GCC as separate containers but the one thing that caused issues was setting up Lmod on each container to allow EB to build properly. In a 1-1 app to container scenario how will I implement the module behavior to pick up paths installed in different containers. Again, as soon as you start talking about multiple containers interacting with each other, and dependencies, we end up with a layer of orchestration. This is both a good and bad thing. Thinking in terms of reproducibility in it's simplest form, we have a single Singularity image that has within it all of the "bits" necessary for a given software workflow. The moment we start requiring other containers, we do not have a single point of reproducibility anymore. Not that this is a bad thing, but it compounds the problem of complexity exponentially. Assuming EB installs in some path /usr/local/easybuild and there is a module file for GCC. Users can execute commands against the container: singularity exec GCC-5.4.0-2.27.img gcc –V provided that GCC module is loaded. I don’t think this would still work since dependent modules need to exist in same container. Single container solution would be simple but I only see this useful for managing a group of software in a container. There would be no way to interact between two containers. Perhaps a standardized method to interacting with an orchestration system like Kubernetes (not HPC ready, but Slurm doesn't have the necessary bits to deal with this easily). Ken, is there anyone from EB working on this, I would like to get started on this approach. Looking forward to hearing back from both of you. Thoughts? Regards, From: Gregory M. Kurtzer [mailto:[email protected]<mailto:[email protected]>] Sent: Tuesday, April 11, 2017 1:25 PM To: Siddiqui, Shahzeb Cc: Kenneth Hoste Subject: Re: Singularity + EasyBuild Hello Shahzeb, I think this is a very interesting idea, but I will have to defer to Kenneth to comment as it pertains to EB. My initial feeling is that this should be possible, and I would love to see EB generate the build commands for the package and the dependencies such that it can be transposed into a Singularity build definition file. That way we can bring the deffile and build it anywhere (e.g. Singularity-hub) and create packages from that. Thoughts? Greg Thoughts? On Mon, Apr 10, 2017 at 8:00 AM, Siddiqui, Shahzeb <[email protected]<mailto:[email protected]>> wrote: Hello Greg/Ken, My name is Shahzeb Siddiqui, I am a HPC Engineer at Pfizer and I manage our software applications. I am currently using EasyBuild to build out our software collection, and I have briefly played out with Singularity to build out Easybuild packages. I have already build my EB pkgs with RPM and in the next few months I would like to get EB pkgs in Singularity. Before I start on this journey, I would like to know if you guys have started work on this. If so, I would like to know what is the current progress and potentially find command grounds for collaboration. Do you have a timeframe when EB + Singularity will work together. I see Singularity as an alternate to RPM packages which can ease deployment of packages without relying on FPM. At Pfizer, we retain software packages for 10 years that were built on older architecture and OS. I think Singularity would be able to address some of these issues, by allowing EB to build Singularity containers for any OS. Before we introduce Singularity at our HPC site, we would like to have a support model, potentially a consultant that can help architect this for us. Do you provide commercial support for this tool, we would need to address some of the security concerns to comply with Pfizer policies. I look forward to hearing back from you. Regards, Shahzeb Siddiqui HPC Linux Engineer B2220-447.2 Groton, CT From: [email protected] [mailto:[email protected]] On Behalf Of Di Pe Sent: Monday, May 8, 2017 1:43 PM To: [email protected] Subject: Re: [easybuild] eb in singularity Interesting discussion about Singularity, there was an older thread here https://mail.google.com/mail/u/0/#search/singularity/159045c3913ad572<https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.google.com_mail_u_0_-23search_singularity_159045c3913ad572&d=DwMFaQ&c=UE1eNsedaKncO0Yl_u8bfw&r=RMJdCm7m5fiPWhajwKUnEW5yn4eK2YdUWW-MLVShghg&m=U1Ye_dBtCfKKvzdeYTPD9IUfYLhVCCQYcTW04W3jMFI&s=5Y8Aj2dkooBU_izF82b21XY7ckGP_7CRyAjXFR29TUA&e=> and it would be great to learn what EB leadership thinks about integration of containers in EB? To look a little bit into our requirements we would like to make it easy to share the software stacks we build using EB with 1) other researchers outside our organization who are also running things on HPC systems 2) other software platforms in the same organization (outside HPC clusters) In both cases we want to support a certain version of a containerized software stack for a certain period of time. Take for example a data science stack. We would bundle a certain version of R with a certain version of Python and some other ancillary tools. We would build a container that has one version of Python2 and one version of R and another container that has Python3 with R. I would not need or even want any Lmod environment inside the container because we may need to tie R and python together. For example R argparse is linked to a specific version of Python and Python rpy2 is linked to a specific version or R. This cannot be handled well with lmod. Another issue that plays into this is size of the container. We want to keep the container relatively small to facilitate sharing but the easybuild environment is actually pretty large with sources and everything. if there was a good way to remove all the unnecessary things including compiler after the container is created that would be great. Yet another issue is that non-HPC people are not used to lmod. So if I have hadoop or devops folks they need to know a little bit of lmod to be able to troubleshoot the software stack and because that is not the case (at least here) they are building stuff by hand or are using crappy Ubuntu debs even through the HPC person one floor down has already done the (better) job using EB. In that sense it would be great if we could use EB to create a container stack that support only one version of each software and wherethe software inside the container simply installs to /usr/local to they can use it as a base and then fuzz with it further if they need to. Another interesting discussion was Singularity vs Docker. After reading this https://www.nextplatform.com/2017/04/10/singularity-containers-hpc-reproducibility-mobility/<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.nextplatform.com_2017_04_10_singularity-2Dcontainers-2Dhpc-2Dreproducibility-2Dmobility_&d=DwMFaQ&c=UE1eNsedaKncO0Yl_u8bfw&r=RMJdCm7m5fiPWhajwKUnEW5yn4eK2YdUWW-MLVShghg&m=U1Ye_dBtCfKKvzdeYTPD9IUfYLhVCCQYcTW04W3jMFI&s=vQBRkyNKZ0AuhN1k27RTgwrznsvOcwJfGydKEpPx5y4&e=> and trying out singularity I think the answer is that we want to use both. Docker plain will probably never be usable on HPC systems (See the story about rejected pull requests) so you have to take a detour and use Shifter (Shifter needs to be integrated by a HPC sysadmin requiring root access and a test plan. It is optimized towards and tested with Slurm.) Singularity works out of the box on HPC and standalone systems. The benefit of Shifter seems to be that it can use Docker images natively while singularity requires you to either build singularity images or import docker images. In my ad-hoc testing I was having much more problems with building singularity images (e.g FS corruption ) than importing docker images with singularity. The latter worked really well and fast. So perhaps it makes sense to focus on integrating Docker builds into EB as one could deliver a more comprehensive solution to bigdata, devops and HPC people. Interested to hear some thoughts DP On Thu, May 4, 2017 at 9:53 AM, Siddiqui, Shahzeb <[email protected]<mailto:[email protected]>> wrote: Hello folks, I would like to find out how Easybuild and Singularity are going to work together. I am trying to design an eb environment in Singularity as a container solution to host all of the eb apps in a prod environment. Is there anyone in the HPC community that is working on this? Currently, I have setup a container environment that can build packages in container and also install RPMs from easybuild via Artifactory that is done through the bootstrap process. One of things that puzzles me is how to setup an isolated container environment that runs /etc/profile for container. I’ve noticed I need to do this inorder to get module command to work in container. Currently, I have to do this manually after shelling in. -bash-4.2$ singularity shell /nfs/grid/software/testing.img Singularity: Invoking an interactive shell within container... Singularity.testing.img> module --version sh: module: command not found Singularity.testing.img> env | grep MODULEPATH MODULEPATH_ROOT=/usr/share/modulefiles MODULEPATH=/modulefiles/Core/:/nfs/grid/software/RHEL7-BUILD/easybuild/modules/all/:/nfs/grid/software/RHEL7/non-easybuild/modules/all/:/nfs/grid/software/RHEL7/easybuild/modules/all/Core:/nfs/grid/software/moduledomains:/etc/modulefiles:/usr/share/modulefiles:/usr/share/modulefiles/Linux:/usr/share/modulefiles/Core:/usr/share/lmod/lmod/modulefiles/Core Singularity.testing.img> unset MODULEPATH Singularity.testing.img> . /etc/profile Singularity.testing.img> . /nfs/grid/software/ module-setup.sh RHEL7/ RHEL7-BUILD/ Singularity.testing.img> . /nfs/grid/software/module-setup.sh Singularity.testing.img> ml av ----------------------------------------------------------- /usr/share/lmod/lmod/modulefiles/Core ------------------------------------------------------------ lmod/6.5.1 settarg/6.5.1 ---------------------------------------------------- /nfs/grid/software/RHEL7-BUILD/easybuild/modules/all ---------------------------------------------------- EasyBuild/3.1.2 ---------------------------------------------------- /nfs/grid/software/RHEL7/easybuild/modules/all/Core ----------------------------------------------------- Advisor/2017_update1 IGV/2.3.80-Java-1.8.0_92 Java/1.8.0_92 gaussian/16-AVX iompi/2017.01 tbb/2017.2.132 GCC/5.4.0-2.27 Inspector/2017_update1 VTune/2017_update1 gaussian/16-SSE2 (D) ipp/2017.1.132 GCC/6.2.0-2.27 (D) IntelClusterChecker/2017.1.016 daal/2017.1.132 intel/2017.01 itac/2017.1.024 Where: D: Default Module Use "module spider" to find all possible modules. Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys". Singularity.testing.img> ml EasyBuild Singularity.testing.img> eb --version This is EasyBuild 3.1.2 (framework: 3.1.2, easyblocks: 3.1.2) on host amrndhl1157.pfizer.com<http://amrndhl1157.pfizer.com>. Singularity.testing.img> eb --show-config # # Current EasyBuild configuration # (C: command line argument, D: default value, E: environment variable, F: configuration file) # buildpath (D) = /home/siddis14/.local/easybuild/build installpath (D) = /home/siddis14/.local/easybuild repositorypath (D) = /home/siddis14/.local/easybuild/ebfiles_repo robot-paths (D) = /nfs/grid/software/RHEL7-BUILD/easybuild/software/EasyBuild/3.1.2/lib/python2.7/site-packages/easybuild_easyconfigs-3.1.2-py2.7.egg/easybuild/easyconfigs sourcepath (D) = /home/siddis14/.local/easybuild/sources Singularity.testing.img> eb zlib-1.2.8.eb == temporary log file in case of crash /tmp/eb-BfnLvm/easybuild-ybJc4_.log == zlib/1.2.8 is already installed (module found), skipping == No easyconfigs left to be built. == Build succeeded for 0 out of 0 == Temporary log file(s) /tmp/eb-BfnLvm/easybuild-ybJc4_.log* have been removed. == Temporary directory /tmp/eb-BfnLvm has been removed. Singularity.testing.img> eb zlib-1.2.8.eb --rebuild == temporary log file in case of crash /tmp/eb-IMN3vl/easybuild-lWCjI1.log == processing EasyBuild easyconfig /nfs/grid/software/RHEL7-BUILD/easybuild/software/EasyBuild/3.1.2/lib/python2.7/site-packages/easybuild_easyconfigs-3.1.2-py2.7.egg/easybuild/easyconfigs/z/zlib/zlib-1.2.8.eb == building and installing zlib/1.2.8.<https://urldefense.proofpoint.com/v2/url?u=http-3A__1.2.8.&d=DwMFaQ&c=UE1eNsedaKncO0Yl_u8bfw&r=RMJdCm7m5fiPWhajwKUnEW5yn4eK2YdUWW-MLVShghg&m=U1Ye_dBtCfKKvzdeYTPD9IUfYLhVCCQYcTW04W3jMFI&s=2DbV7b8CLu85Uado7mlKmdBIcJvdmaZr0IIo3nO0XtU&e=>.. == fetching files... == creating build dir, resetting environment... == unpacking... == patching... == preparing... == configuring... == building... == testing... == installing... == taking care of extensions... == postprocessing... == sanity checking... == cleaning up... == creating module... == permissions... == packaging... == COMPLETED: Installation ended successfully == Results of the build can be found in the log file(s) /home/siddis14/.local/easybuild/software/zlib/1.2.8/easybuild/easybuild-zlib-1.2.8-20170504.163248.log == Build succeeded for 1 out of 1 == Temporary log file(s) /tmp/eb-IMN3vl/easybuild-lWCjI1.log* have been removed. == Temporary directory /tmp/eb-IMN3vl has been removed. Singularity.testing.img> module use $HOME/.local/easybuild/modules/all Due to MODULEPATH changes the following have been reloaded: 1) EasyBuild/3.1.2 Singularity.testing.img> module av -------------------------------------------------------- /home/siddis14/.local/easybuild/modules/all --------------------------------------------------------- EasyBuild/3.1.2 (L,D) M4/1.4.17 (D) zlib/1.2.8 (D) ----------------------------------------------------------- /usr/share/lmod/lmod/modulefiles/Core ------------------------------------------------------------ lmod/6.5.1 settarg/6.5.1 ---------------------------------------------------- /nfs/grid/software/RHEL7-BUILD/easybuild/modules/all ---------------------------------------------------- EasyBuild/3.1.2 ---------------------------------------------------- /nfs/grid/software/RHEL7/easybuild/modules/all/Core ----------------------------------------------------- Advisor/2017_update1 IGV/2.3.80-Java-1.8.0_92 Java/1.8.0_92 gaussian/16-AVX iompi/2017.01 tbb/2017.2.132 GCC/5.4.0-2.27 Inspector/2017_update1 VTune/2017_update1 gaussian/16-SSE2 (D) ipp/2017.1.132 GCC/6.2.0-2.27 (D) IntelClusterChecker/2017.1.016 daal/2017.1.132 intel/2017.01 itac/2017.1.024 Where: L: Module is loaded D: Default Module Use "module spider" to find all possible modules. Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys". Shahzeb Siddiqui HPC Linux Engineer B2220-447.2 Groton, CT
