Re: Should we skip foss/2018a (Re: [easybuild] 2018a toolchains)

[Ole Holm Nielsen]

On 12/08/2017 12:39 PM, Alvarez, Damian wrote:

+1 for going for 18.1. But the glibc issues (not kernel issues) have been 
solved already (https://access.redhat.com/errata/RHBA-2017:3296). This package 
should find its way into CentOS 7.4 real quick (maybe it is already there, I am 
not sure).


The CentOS 7.4 package glibc-2.17-196.el7_4.2.x86_64 dated December 5 
contains the packages mentioned in the RedHat erratum.


/Ole

Re: Should we skip foss/2018a (Re: [easybuild] 2018a toolchains)

[Alvarez, Damian]

+1 for going for 18.1. But the glibc issues (not kernel issues) have been 
solved already (https://access.redhat.com/errata/RHBA-2017:3296). This package 
should find its way into CentOS 7.4 real quick (maybe it is already there, I am 
not sure).

Damian

On 08/12/17 10:35, "easybuild-requ...@lists.ugent.be on behalf of Joachim Hein" 
 wrote:

Hi,

Reading through Kenneth’ reply, I am honestly wondering whether we should 
skip a foss/2018a.  The below does not suggest any big upgrade.  Considering 
the human cost of moving all "helper” packages (such as HDF5, netcdf, Python 
and Pythonwrapers, Boost) that our users require, I am close to thinking, this 
is not worth it.  In the current HPC environment, a tool chain is way more than 
compiler, mpi, blas, fft library.  It is a full blown “ecosystem”.  In addition 
we see increasing demands on interoperability of packages.

We are still building software with foss/2017a, because updates for 
foss/2017b are still in the pipeline (e.g. not contributed yet, pull request 
under development, in the git development branch).  For various reasons, it 
seems to take the EasyBuild community now about 1/2 year to release (inclusion 
in EB release) most of the things required for a toolchain.  Looking into the 
pull requests, I see a bigger emphasis to get packages into EB that weren’t 
supported before, which is a good thing in my view.  Skipping toss/2018a would 
allow us to spend our time on new packages since we are not so absorbed in 
porting, reviewing, merging updates all the old stuff.

Intel 2018a is a different matter, because of the kernel issues with RH 
7.4.  I am under the impression that we should go for intel 18.1.  Not having a 
foss/2018a would reduce the timing constraints when we release an intel/2018a.

Just a few thoughts for discussion.

Best wishes
   Joachim



> On 6 Dec 2017, at 09:27, Kenneth Hoste  wrote:
>
> Dear Franky,
>
> On 28/11/2017 11:43, Backeljauw Franky wrote:
>> Hi all,
>>
>> I’m wondering whether there are any suggestions already on what would be 
put in the 2018a toolchains?
>
> I just took a quick look at what's currently available, and what we can 
consider for the 2018a toolchains.
> Mind you, we will not set these in stone before Jan 1st 2018 (and 
hopefully not long afterwards)...
>
> First of all: for GCC(core), we should stick to 6.4.0 for the time being 
(latest 6.x release), since the latest version of the Intel compilers (2018 
update 1) does not support GCC 7.x officially yet. I checked with Intel 
support, supporting GCC 7.x is only planned for the 2019 versions to be 
released early 2018 (sigh).
>
> If we'll stick to GCC 6.4.0, it may make sense to also stick to binutils 
2.28, i.e. the base for both foss/2017b and intel/2017b, for two reasons:
>
> * based on the release notes for binutils 2.29, there's nothing really 
shocking included in terms of new features
> * that allows us to reuse the easyconfig files that currently use the 
GCCcore 6.4.0 toolchain + binutils 2.28 as build dep
>
> On the other hand, we do miss any bug fixes included with the more recent 
binutils versions...
>
> Updating to the latest binutils 2.29.1 is worth considering, but may pose 
a problem for easyconfigs using the GCCcore/6.4.0 toolchain in combination with 
foss/2018a (since they'd be using a slightly different version of binutils). I 
may be OK in practice though...
>
> For Open MPI, the most obvious thing to do is to update to v2.1.2 
(compared to v2.1.1 included in the 2017b toolchains).
> There's a more recent Open MPI v3.0.1 available already (Oct'17), but 
we're usually careful with picking up new major versions;
> the release notes [1] also indicate potential large changes w.r.t. 
backward compatibility.
>
> If somebody has (good or bad) experiences with Open MPI v3.0.1 w.r.t. 
stability or known problems, please let us know!
>
> For OpenBLAS, there (still) is no new release, so we can stick to the 
patched v0.2.20 we're using in foss/2017b.
> At some point, we may need to start looking for alternatives, since there 
seems to be a problem with getting new OpenBLAS releases out...
>
> For FFTW, a minor version bump to 3.3.7 makes sense (vs 3.3.6 in 2017b). 
ScaLAPACK stays at 2.0.2.
>
> For intel/2018a, it makes sense to go with the 2018 update 1 release I 
think, especially taking into account the glibc-related problems in the latest 
2017 update.
> This basically boils down to promoting the intel/2018.01 that was defined 
in [2] to intel/2018a (assuming we stick to GCCcore 6.4.0 and binutils 2.28).
>
> I'm happy to hear some feedback on this, maybe we should also briefly 
discuss this during the EasyBuild conf call this afternoon (I'll add it to the 

Should we skip foss/2018a (Re: [easybuild] 2018a toolchains)

[Joachim Hein]

Hi,

Reading through Kenneth’ reply, I am honestly wondering whether we should skip 
a foss/2018a.  The below does not suggest any big upgrade.  Considering the 
human cost of moving all "helper” packages (such as HDF5, netcdf, Python and 
Pythonwrapers, Boost) that our users require, I am close to thinking, this is 
not worth it.  In the current HPC environment, a tool chain is way more than 
compiler, mpi, blas, fft library.  It is a full blown “ecosystem”.  In addition 
we see increasing demands on interoperability of packages.  

We are still building software with foss/2017a, because updates for foss/2017b 
are still in the pipeline (e.g. not contributed yet, pull request under 
development, in the git development branch).  For various reasons, it seems to 
take the EasyBuild community now about 1/2 year to release (inclusion in EB 
release) most of the things required for a toolchain.  Looking into the pull 
requests, I see a bigger emphasis to get packages into EB that weren’t 
supported before, which is a good thing in my view.  Skipping toss/2018a would 
allow us to spend our time on new packages since we are not so absorbed in 
porting, reviewing, merging updates all the old stuff. 

Intel 2018a is a different matter, because of the kernel issues with RH 7.4.  I 
am under the impression that we should go for intel 18.1.  Not having a 
foss/2018a would reduce the timing constraints when we release an intel/2018a.

Just a few thoughts for discussion.

Best wishes
   Joachim



> On 6 Dec 2017, at 09:27, Kenneth Hoste  wrote:
> 
> Dear Franky,
> 
> On 28/11/2017 11:43, Backeljauw Franky wrote:
>> Hi all,
>> 
>> I’m wondering whether there are any suggestions already on what would be put 
>> in the 2018a toolchains?
> 
> I just took a quick look at what's currently available, and what we can 
> consider for the 2018a toolchains.
> Mind you, we will not set these in stone before Jan 1st 2018 (and hopefully 
> not long afterwards)...
> 
> First of all: for GCC(core), we should stick to 6.4.0 for the time being 
> (latest 6.x release), since the latest version of the Intel compilers (2018 
> update 1) does not support GCC 7.x officially yet. I checked with Intel 
> support, supporting GCC 7.x is only planned for the 2019 versions to be 
> released early 2018 (sigh).
> 
> If we'll stick to GCC 6.4.0, it may make sense to also stick to binutils 
> 2.28, i.e. the base for both foss/2017b and intel/2017b, for two reasons:
> 
> * based on the release notes for binutils 2.29, there's nothing really 
> shocking included in terms of new features
> * that allows us to reuse the easyconfig files that currently use the GCCcore 
> 6.4.0 toolchain + binutils 2.28 as build dep
> 
> On the other hand, we do miss any bug fixes included with the more recent 
> binutils versions...
> 
> Updating to the latest binutils 2.29.1 is worth considering, but may pose a 
> problem for easyconfigs using the GCCcore/6.4.0 toolchain in combination with 
> foss/2018a (since they'd be using a slightly different version of binutils). 
> I may be OK in practice though...
> 
> For Open MPI, the most obvious thing to do is to update to v2.1.2 (compared 
> to v2.1.1 included in the 2017b toolchains).
> There's a more recent Open MPI v3.0.1 available already (Oct'17), but we're 
> usually careful with picking up new major versions;
> the release notes [1] also indicate potential large changes w.r.t. backward 
> compatibility.
> 
> If somebody has (good or bad) experiences with Open MPI v3.0.1 w.r.t. 
> stability or known problems, please let us know!
> 
> For OpenBLAS, there (still) is no new release, so we can stick to the patched 
> v0.2.20 we're using in foss/2017b.
> At some point, we may need to start looking for alternatives, since there 
> seems to be a problem with getting new OpenBLAS releases out...
> 
> For FFTW, a minor version bump to 3.3.7 makes sense (vs 3.3.6 in 2017b). 
> ScaLAPACK stays at 2.0.2.
> 
> For intel/2018a, it makes sense to go with the 2018 update 1 release I think, 
> especially taking into account the glibc-related problems in the latest 2017 
> update.
> This basically boils down to promoting the intel/2018.01 that was defined in 
> [2] to intel/2018a (assuming we stick to GCCcore 6.4.0 and binutils 2.28).
> 
> I'm happy to hear some feedback on this, maybe we should also briefly discuss 
> this during the EasyBuild conf call this afternoon (I'll add it to the 
> agenda).
> 
> 
> regards,
> 
> Kenneth
> 
> 
> [1] https://raw.githubusercontent.com/open-mpi/ompi/v3.0.x/NEWS
> [2] https://github.com/easybuilders/easybuild-easyconfigs/pull/5345
> 
> 
> 
>> 
>> — Regards,
>> 
>> Franky
>> 
>