Re: [easybuild] Toolchains

[McGough, Benjamin E]

[I am working with John on this...]

We can definitely open an issue, thanks! My gut feeling is that CMake should 
not be included in the toolchain; there is a precedent of depending it into 
packages that need it, and it looks like version conflicts do happen. To 
contrast, I have _never_ in my life run into a build that required a specific 
version of 'make.' Another package for possible toolchain inclusion would be 
pkg-config. Ubuntu does not include it in default server installs, unlike 
CentOS.

We have 3 problems we would like to solve with EasyBuild.

*OS Packages*

OS packages are hurting us. Autoconf is extremely greedy and seems to grab any 
library available when configuring. We have a long list of OS libraries that 
have been unintentionally included in many of our EasyBuild packages.

We also have a small community of users who compile their own code. OS 
migrations often mean they have to re-build everything, and we have found 
scientists are not as responsible about documenting their build procedures as, 
say, software engineers. ;)  If we could isolate them from OS tools by loading 
an EasyBuild toolchain as their build environment, their binaries would run 
any/every-where they can load that toolchain, even if they do not create an 
easyconfig.

Solution: Modify EasyBuild toolchains to include all dependencies and deploy a 
minimal OS.

*Containers*

We have begun to use containers for cloud computing - primarily AWS Batch, but 
also Azure and locally with Singularity. We must provide our users with 
identical software packages in and out of our containers in order to help them 
migrate their jobs from our slurm cluster to other places. EasyBuild is perfect 
for this!

Solution: Use modified EasyBuild toolchains to build in a minimal OS container, 
then re-build into our traditional NFS "apps" folder using that same container.

[This is actually working pretty well, however our OS is not yet minimal. See 
https://github.com/FredHutch/ls2 - we are close to releasing full 
R+Bioconductor and Python containers with hundreds of installed language 
packages. I think I could supply a list of missing packages pretty easily, and 
should be able to adopt our Dockerfile to CentOS and other distributions.]

Both of these solutions involve moving all dependencies into easyconfigs. 
Including an osdependencies entry for the one package we cannot eliminate: libc 
headers (libc6-dev in our case).

Finally,

*Support libraries*

We have users who want to load all their environment modules at once at the 
beginning of their scripts/jobs. They use R and Python together with other 
packages. In order to avoid conflicts, we try to keep "robot" package versions 
consistent across all easyconfigs, but this is difficult, time consuming, and 
error prone as I know of no automated way to do this. It also means we cannot 
always use unmodified public easyconfigs due to dependent library versions and 
off-the-shelf EasyBuild is a big feature!

Solution: Create a middle-ware library easyconfig bundle

For example: common-libs-1.0-foss-2016b with X, Tcl/Tk, curl, git?, and libs: 
image/graphics, terminal, data formats, data access, utility.

I really do not like this solution as it would remove dependency information 
from the easyconfig for a package using the bundle, but the maintenance 
nightmare of all our library versions is worth it right now. I hope there is a 
better solution. Automated easyconfig creation? A database of preferred package 
versions? ?

Thank you for all the work you have done on EasyBuild! We would not be where we 
are today without it.

Ben McGough
System Administrator / Center IT - Scientific Computing / 206.667.7818
SciComp Office Hours: Wed. 10-noon M2-B102
SciComp Cloud Consulting: Tue. 3pm Thomas D3-106
bmcgo...@fredhutch.org / Fred Hutch / Cures Start Here

- Original Message -
From: "Kenneth Hoste" <kenneth.ho...@ugent.be>
To: "easybuild" <easybuild@lists.ugent.be>, "John" <j...@fuzzdog.com>
Sent: Sunday, February 11, 2018 1:04:40 AM
Subject: Re: [easybuild] Toolchains

Hi John,

On 09/02/2018 23:19, John Dey wrote:
> *I would like to propose additional packages for foss toolchains. 
> Toolchains should contain enough tools to build packages independent 
> of the OS dev tools.   I propose that foss should contain:
> *
> *make,**bz2, and xzutils.
> *
> *
> Why?
> *Adding this small set of tools would allow us to build everything 
> without having to add the OS dev packages.* We are targeting our EB 
> environment to support minimally configured cluster nodes and for 
> Docker containers. Starting with a base Ubuntu container we do not 
> want to have to add the compilers that are part of the dev packages. 
> Containers will be used to build packages and also for our computing 
> environment.
> *

This proposal has come up before, and it makes sense, but we should

Re: [easybuild] Toolchains

[Kenneth Hoste]

Hi John,

On 09/02/2018 23:19, John Dey wrote:
*I would like to propose additional packages for foss toolchains. 
Toolchains should contain enough tools to build packages independent 
of the OS dev tools.   I propose that foss should contain:

*
*make,**bz2, and xzutils.
*
*
Why?
*Adding this small set of tools would allow us to build everything 
without having to add the OS dev packages.* We are targeting our EB 
environment to support minimally configured cluster nodes and for 
Docker containers. Starting with a base Ubuntu container we do not 
want to have to add the compilers that are part of the dev packages. 
Containers will be used to build packages and also for our computing 
environment.

*


This proposal has come up before, and it makes sense, but we should do 
it in an organized fashion and only for future toolchains probably...


There are some other tools that we could/should include: Autotools 
(Autoconf + Automake + libtool), patch, zlib, CMake, zip, etc.


The best way would probably to define a 'builddeps' bundle?

Note that this wouldn't help with totally avoiding some OS packages, 
since you'll still need several of these tools to build GCC(core) + 
binutils.
But we could of course include the required build dependencies there 
individually...


The best way to follow up on this is probably to open an issue in the 
easyconfigs repository?



regards,

Kenneth


*

*

*John Dey*

*HPC Operations*

SciComp Computing
*O* _206.667.4308_
*M* _360.649.2731_

jf...@fredhutch.org


Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N., Mail Stop J3-516
Seattle, WA 98109
*fredhutch.org*

--
John Dey
j...@fuzzdog.com 

[easybuild] Toolchains

[John Dey]

*I would like to propose additional packages for foss toolchains.
Toolchains should contain enough tools to build packages independent of the
OS dev tools.   I propose that foss should contain:*
*make,*
* bz2, and xzutils.  *



*Why?Adding this small set of tools would allow us to build everything
without having to add the OS dev packages. We are targeting our EB
environment to support minimally configured cluster nodes and for Docker
containers. Starting with a base Ubuntu container we do not want to have to
add the compilers that are part of the dev packages. Containers will be
used to build packages and also for our computing environment.*


*John Dey*

*HPC Operations*

SciComp Computing
*O* *206.667.4308*
*M* *360.649.2731*

jf...@fredhutch.org






Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N., Mail Stop J3-516
Seattle, WA 98109
*fredhutch.org* 
-- 
John Dey
j...@fuzzdog.com

Re: [easybuild] Toolchains

[Pablo Escobar Lopez]

Hi Keith

toolchains are not installed by default, you need to build/install one
toolchain and then use it to continue building more apps. You can list all
the toolchain that easybuild supports by default with the command " eb
--list-toolchains" to decide which one you want to use.

I recommend you to start testing with goolf-1.4.10. To install this
toolchain this command should work (it will take a while...)

eb goolf-1.4.10.eb -r


2014-03-06 22:43 GMT+01:00 Keith Hostetler <keith.r.hostetler...@nd.edu>:

> The thing I can not wrap my head around is, like it says on your link,
> "supported toolchains". From that page: "...the first thing you'll need to
> do is either pick a supported compiler toolchain, or construct your own and
> make EasyBuild support it". I would think that supported means that I can
> use them right after installation, but now I am being told that I need to
> compile the GCC toolchain myself. Isn't that then making my own toolchain,
> how else can you possibly "pick a supported compiler toolchain"?
> 
> From: easybuild-requ...@lists.ugent.be [easybuild-requ...@lists.ugent.be]
> On Behalf Of Kenneth Hoste [kenneth.ho...@ugent.be]
> Sent: Thursday, March 06, 2014 4:29 PM
> To: easybuild@lists.ugent.be
> Subject: Re: [easybuild] Toolchains
>
> Hi Keith,
>
> On 06/03/14 22:09, Keith Hostetler wrote:
> > I see. Do you need a tar file to create every toolchain? If not, what
> else do you use as the source?
>
> Usually, a toolchain involves a compiler and a bunch of libraries (most
> commonly MPI, BLAS, LAPACK, FFT).
>
> See https://github.com/hpcugent/easybuild/wiki/Compiler-toolchains for
> more details.
>
> So, most commonly toolchains are basically bundles of a handful of
> tools, for which EasyBuild knows how to prepare.
> You'll need source tarballs or installation files for each of the
> toolchain elements, but the toolchain itself doesn't have any sources of
> its own.
>
>
> regards,
>
> Kenneth
>
> > 
> > From: wpoel...@gmail.com [wpoel...@gmail.com] On Behalf Of Ward
> Poelmans [ward.poelm...@ugent.be]
> > Sent: Wednesday, March 05, 2014 9:48 AM
> > To: easybuild@lists.ugent.be
> > Subject: Re: [easybuild] Toolchains
> >
> > On Tue, Mar 4, 2014 at 9:27 PM, Keith Hostetler
> > <keith.r.hostetler...@nd.edu> wrote:
> >> You say that it is normal for the dependencies to fail, but if they
> fail then it won't use them and thus won't create the software. As I said,
> I used GCC as my toolchain, but when I do, it fails due to unresolved
> dependencies and so I don't get a software compilation. Do I need to build
> GCC's toolchain too?
> > Yes, you need to build a GCC toolchain with easybuild. I suppose that
> > until now, you've used a system gcc?
> >
> > Easybuild is build aroud the idea that it is in full control: the
> > entire (well, almost entire), toolchain is build by easybuild.
> >
> > Ward
>
>


-- 
Pablo Escobar López
HPC systems engineer
Biozentrum, University of Basel
Swiss Institute of Bioinformatics SIB
Email: pablo.escobarlo...@unibas.ch
Phone: +41 61 267 15 82
http://www.biozentrum.unibas.ch

RE: [easybuild] Toolchains

[Keith Hostetler]

I see. Do you need a tar file to create every toolchain? If not, what else do 
you use as the source? 

From: wpoel...@gmail.com [wpoel...@gmail.com] On Behalf Of Ward Poelmans 
[ward.poelm...@ugent.be]
Sent: Wednesday, March 05, 2014 9:48 AM
To: easybuild@lists.ugent.be
Subject: Re: [easybuild] Toolchains

On Tue, Mar 4, 2014 at 9:27 PM, Keith Hostetler
<keith.r.hostetler...@nd.edu> wrote:
> You say that it is normal for the dependencies to fail, but if they fail then 
> it won't use them and thus won't create the software. As I said, I used GCC 
> as my toolchain, but when I do, it fails due to unresolved dependencies and 
> so I don't get a software compilation. Do I need to build GCC's toolchain too?

Yes, you need to build a GCC toolchain with easybuild. I suppose that
until now, you've used a system gcc?

Easybuild is build aroud the idea that it is in full control: the
entire (well, almost entire), toolchain is build by easybuild.

Ward

RE: [easybuild] Toolchains

[Keith Hostetler]

You say that it is normal for the dependencies to fail, but if they fail then 
it won't use them and thus won't create the software. As I said, I used GCC as 
my toolchain, but when I do, it fails due to unresolved dependencies and so I 
don't get a software compilation. Do I need to build GCC's toolchain too?

From: wpoel...@gmail.com [wpoel...@gmail.com] On Behalf Of Ward Poelmans 
[ward.poelm...@ugent.be]
Sent: Monday, March 03, 2014 12:11 PM
To: easybuild@lists.ugent.be
Subject: Re: [easybuild] Toolchains

On Thu, Feb 27, 2014 at 10:34 PM,  <khost...@nd.edu> wrote:
> So say I am building openmpi and I use dummy and dummy as name and version,
> then what toolchain is it defaulting to? I would have originally thought it
> was gcc, but when I try to manually use GCC as the name and dummy as the
> version, it fails saying "Dependencies not met. Cannot resolve ["GCC/dummy..."
> and this is while using --robot.

You're not supposed to use the dummy toolchain. The dummy toolchain
defaults to nothing. It's there to build other toolchains.

The dependencies that fail are perfectly normal. You need to use the
GCC (or any other) toolchain for all the software you're compiling.

Ward

[easybuild] Toolchains

[khostetl]

So say I am building openmpi and I use dummy and dummy as name and version,
then what toolchain is it defaulting to? I would have originally thought it
was gcc, but when I try to manually use GCC as the name and dummy as the
version, it fails saying "Dependencies not met. Cannot resolve ["GCC/dummy..."
and this is while using --robot.