Regarding pulling in OS packages, we've had some improvement on this by building EasyBuild modules _inside_ of a minimal OS Singularity container. Specifically, this one, which is a proxy for the RedHat distribution we run in production.
https://github.com/pescobar/singularity-easybuild/blob/master/centos-7-easybuild-3.5.1.bootstrap So far, the resulting modules seem to work well. This at least limits the number of things that can be accidentally picked up during compilation. One does tend to stumble across a number of bugs and undeclared osdependencies, though. Also, beware that loading a library module doesn't necessarily mean that it will be used in preference to the OS version. (I was quite shocked to learn this.) See this issue. https://github.com/easybuilders/easybuild-easyconfigs/issues/5776 Cheers, Mike Michael Coleman Computational Scientist, Research Advanced Computing Services mcole...@uoregon.edu University of Oregon 141 Deschutes Hall 1477 East 13th Ave. Eugene, Oregon 97403 +1 541 346 5172 -----Original Message----- From: easybuild-requ...@lists.ugent.be [mailto:easybuild-requ...@lists.ugent.be] On Behalf Of McGough, Benjamin E Sent: Monday, February 12, 2018 11:41 AM To: easybuild <easybuild@lists.ugent.be> Cc: John <j...@fuzzdog.com> Subject: Re: [easybuild] Toolchains [I am working with John on this...] We can definitely open an issue, thanks! My gut feeling is that CMake should not be included in the toolchain; there is a precedent of depending it into packages that need it, and it looks like version conflicts do happen. To contrast, I have _never_ in my life run into a build that required a specific version of 'make.' Another package for possible toolchain inclusion would be pkg-config. Ubuntu does not include it in default server installs, unlike CentOS. We have 3 problems we would like to solve with EasyBuild. *OS Packages* OS packages are hurting us. Autoconf is extremely greedy and seems to grab any library available when configuring. We have a long list of OS libraries that have been unintentionally included in many of our EasyBuild packages. We also have a small community of users who compile their own code. OS migrations often mean they have to re-build everything, and we have found scientists are not as responsible about documenting their build procedures as, say, software engineers. ;) If we could isolate them from OS tools by loading an EasyBuild toolchain as their build environment, their binaries would run any/every-where they can load that toolchain, even if they do not create an easyconfig. Solution: Modify EasyBuild toolchains to include all dependencies and deploy a minimal OS. *Containers* We have begun to use containers for cloud computing - primarily AWS Batch, but also Azure and locally with Singularity. We must provide our users with identical software packages in and out of our containers in order to help them migrate their jobs from our slurm cluster to other places. EasyBuild is perfect for this! Solution: Use modified EasyBuild toolchains to build in a minimal OS container, then re-build into our traditional NFS "apps" folder using that same container. [This is actually working pretty well, however our OS is not yet minimal. See https://github.com/FredHutch/ls2 - we are close to releasing full R+Bioconductor and Python containers with hundreds of installed language packages. I think I could supply a list of missing packages pretty easily, and should be able to adopt our Dockerfile to CentOS and other distributions.] Both of these solutions involve moving all dependencies into easyconfigs. Including an osdependencies entry for the one package we cannot eliminate: libc headers (libc6-dev in our case). Finally, *Support libraries* We have users who want to load all their environment modules at once at the beginning of their scripts/jobs. They use R and Python together with other packages. In order to avoid conflicts, we try to keep "robot" package versions consistent across all easyconfigs, but this is difficult, time consuming, and error prone as I know of no automated way to do this. It also means we cannot always use unmodified public easyconfigs due to dependent library versions and off-the-shelf EasyBuild is a big feature! Solution: Create a middle-ware library easyconfig bundle For example: common-libs-1.0-foss-2016b with X, Tcl/Tk, curl, git?, and libs: image/graphics, terminal, data formats, data access, utility. I really do not like this solution as it would remove dependency information from the easyconfig for a package using the bundle, but the maintenance nightmare of all our library versions is worth it right now. I hope there is a better solution. Automated easyconfig creation? A database of preferred package versions? ? Thank you for all the work you have done on EasyBuild! We would not be where we are today without it. Ben McGough System Administrator / Center IT - Scientific Computing / 206.667.7818 SciComp Office Hours: Wed. 10-noon M2-B102 SciComp Cloud Consulting: Tue. 3pm Thomas D3-106 bmcgo...@fredhutch.org / Fred Hutch / Cures Start Here ----- Original Message ----- From: "Kenneth Hoste" <kenneth.ho...@ugent.be> To: "easybuild" <easybuild@lists.ugent.be>, "John" <j...@fuzzdog.com> Sent: Sunday, February 11, 2018 1:04:40 AM Subject: Re: [easybuild] Toolchains Hi John, On 09/02/2018 23:19, John Dey wrote: > *I would like to propose additional packages for foss toolchains. > Toolchains should contain enough tools to build packages independent > of the OS dev tools. I propose that foss should contain: > * > *make,**bz2, and xzutils. > * > * > Why? > *Adding this small set of tools would allow us to build everything > without having to add the OS dev packages.* We are targeting our EB > environment to support minimally configured cluster nodes and for > Docker containers. Starting with a base Ubuntu container we do not > want to have to add the compilers that are part of the dev packages. > Containers will be used to build packages and also for our computing > environment. > * This proposal has come up before, and it makes sense, but we should do it in an organized fashion and only for future toolchains probably... There are some other tools that we could/should include: Autotools (Autoconf + Automake + libtool), patch, zlib, CMake, zip, etc. The best way would probably to define a 'builddeps' bundle? Note that this wouldn't help with totally avoiding some OS packages, since you'll still need several of these tools to build GCC(core) + binutils. But we could of course include the required build dependencies there individually... The best way to follow up on this is probably to open an issue in the easyconfigs repository? regards, Kenneth > * > > * > > *John Dey* > > *HPC Operations* > > SciComp Computing > *O* _206.667.4308_ > *M* _360.649.2731_ > > jf...@fredhutch.org<mailto:jf...@fredhutch.org> > > > Fred Hutchinson Cancer Research Center > 1100 Fairview Ave. N., Mail Stop J3-516 > Seattle, WA 98109 > *fredhutch.org*<http://www.fredhutch.org/> > > -- > John Dey > j...@fuzzdog.com <mailto:j...@fuzzdog.com>
