Regarding pulling in OS packages, we've had some improvement on this by 
building EasyBuild modules _inside_ of a minimal OS Singularity container.  
Specifically, this one, which is a proxy for the RedHat distribution we run in 

So far, the resulting modules seem to work well.  This at least limits the 
number of things that can be accidentally picked up during compilation.  One 
does tend to stumble across a number of bugs and undeclared osdependencies, 

Also, beware that loading a library module doesn't necessarily mean that it 
will be used in preference to the OS version.  (I was quite shocked to learn 
this.)  See this issue.


Michael Coleman
Computational Scientist, Research Advanced Computing Services
University of Oregon
141 Deschutes Hall 
1477 East 13th Ave.
Eugene, Oregon  97403
+1 541 346 5172

-----Original Message-----
[] On Behalf Of McGough, Benjamin E
Sent: Monday, February 12, 2018 11:41 AM
To: easybuild <>
Cc: John <>
Subject: Re: [easybuild] Toolchains

[I am working with John on this...]

We can definitely open an issue, thanks! My gut feeling is that CMake should 
not be included in the toolchain; there is a precedent of depending it into 
packages that need it, and it looks like version conflicts do happen. To 
contrast, I have _never_ in my life run into a build that required a specific 
version of 'make.' Another package for possible toolchain inclusion would be 
pkg-config. Ubuntu does not include it in default server installs, unlike 

We have 3 problems we would like to solve with EasyBuild.

*OS Packages*

OS packages are hurting us. Autoconf is extremely greedy and seems to grab any 
library available when configuring. We have a long list of OS libraries that 
have been unintentionally included in many of our EasyBuild packages.

We also have a small community of users who compile their own code. OS 
migrations often mean they have to re-build everything, and we have found 
scientists are not as responsible about documenting their build procedures as, 
say, software engineers. ;)  If we could isolate them from OS tools by loading 
an EasyBuild toolchain as their build environment, their binaries would run 
any/every-where they can load that toolchain, even if they do not create an 

Solution: Modify EasyBuild toolchains to include all dependencies and deploy a 
minimal OS.


We have begun to use containers for cloud computing - primarily AWS Batch, but 
also Azure and locally with Singularity. We must provide our users with 
identical software packages in and out of our containers in order to help them 
migrate their jobs from our slurm cluster to other places. EasyBuild is perfect 
for this!

Solution: Use modified EasyBuild toolchains to build in a minimal OS container, 
then re-build into our traditional NFS "apps" folder using that same container.

[This is actually working pretty well, however our OS is not yet minimal. See - we are close to releasing full 
R+Bioconductor and Python containers with hundreds of installed language 
packages. I think I could supply a list of missing packages pretty easily, and 
should be able to adopt our Dockerfile to CentOS and other distributions.]

Both of these solutions involve moving all dependencies into easyconfigs. 
Including an osdependencies entry for the one package we cannot eliminate: libc 
headers (libc6-dev in our case).


*Support libraries*

We have users who want to load all their environment modules at once at the 
beginning of their scripts/jobs. They use R and Python together with other 
packages. In order to avoid conflicts, we try to keep "robot" package versions 
consistent across all easyconfigs, but this is difficult, time consuming, and 
error prone as I know of no automated way to do this. It also means we cannot 
always use unmodified public easyconfigs due to dependent library versions and 
off-the-shelf EasyBuild is a big feature!

Solution: Create a middle-ware library easyconfig bundle

For example: common-libs-1.0-foss-2016b with X, Tcl/Tk, curl, git?, and libs: 
image/graphics, terminal, data formats, data access, utility.

I really do not like this solution as it would remove dependency information 
from the easyconfig for a package using the bundle, but the maintenance 
nightmare of all our library versions is worth it right now. I hope there is a 
better solution. Automated easyconfig creation? A database of preferred package 
versions? ?

Thank you for all the work you have done on EasyBuild! We would not be where we 
are today without it.

Ben McGough
System Administrator / Center IT - Scientific Computing / 206.667.7818
SciComp Office Hours: Wed. 10-noon M2-B102
SciComp Cloud Consulting: Tue. 3pm Thomas D3-106 / Fred Hutch / Cures Start Here

----- Original Message -----
From: "Kenneth Hoste" <>
To: "easybuild" <>, "John" <>
Sent: Sunday, February 11, 2018 1:04:40 AM
Subject: Re: [easybuild] Toolchains

Hi John,

On 09/02/2018 23:19, John Dey wrote:
> *I would like to propose additional packages for foss toolchains. 
> Toolchains should contain enough tools to build packages independent 
> of the OS dev tools.   I propose that foss should contain:
> *
> *make,**bz2, and xzutils.
> *
> *
> Why?
> *Adding this small set of tools would allow us to build everything 
> without having to add the OS dev packages.* We are targeting our EB 
> environment to support minimally configured cluster nodes and for 
> Docker containers. Starting with a base Ubuntu container we do not 
> want to have to add the compilers that are part of the dev packages. 
> Containers will be used to build packages and also for our computing 
> environment.
> *

This proposal has come up before, and it makes sense, but we should do 
it in an organized fashion and only for future toolchains probably...

There are some other tools that we could/should include: Autotools 
(Autoconf + Automake + libtool), patch, zlib, CMake, zip, etc.

The best way would probably to define a 'builddeps' bundle?

Note that this wouldn't help with totally avoiding some OS packages, 
since you'll still need several of these tools to build GCC(core) + 
But we could of course include the required build dependencies there 

The best way to follow up on this is probably to open an issue in the 
easyconfigs repository?



> *
> *
> *John Dey*
> *HPC Operations*
> SciComp Computing
> *O* _206.667.4308_
> *M* _360.649.2731_
> Fred Hutchinson Cancer Research Center
> 1100 Fairview Ave. N., Mail Stop J3-516
> Seattle, WA 98109
> **<>
> -- 
> John Dey
> <>

Reply via email to