Re: [easybuild] resume build for software with extensions

[Kenneth Hoste]

Dear Yann,

Sorry for not getting back to you on this...

(more below your gist link to the debug log)

On 12/03/2019 11:50, Yann Sagon wrote:

Dear Kennet, thanks for your answer.

On Fri, Mar 8, 2019 at 11:21 AM Kenneth Hoste > wrote:


Dear Yann,

On 07/03/2019 14:06, Yann Sagon wrote:
 > Dear list,
 >
 > sometime ago, I asked this question on the same list. I think I
tried
 > the procedure but I'm unable to  make it work again.
 >
 > My issue: I try to install a new version of R. This easyconfig
contains
 > more than 500 extensions, and some of them are failling during the
 > build. I then comment them out and relaunch the build.
Unfortunately,
 > the build skip only the R build, not the already built extensions.
 >
 > I have modified the R-3.5.1-foss-2018b.eb file to remove every
extension.
 > I did the install using:
 >
 > eb --module-only R-3.5.1-foss-2018b-without-ext.eb
 > Then I added every extension again and I did:
 > eb --skip --rebuild  ./R-3.5.1-foss-2018b.eb
 > And it fails on a given extenion.
 > I comment out this extension and re do the command:
 > eb --skip --rebuild  ./R-3.5.1-foss-2018b.eb
 > But it tries to rebuild every extension already built in the
previsous step.
 >
 > Any clue?

That's not how it's supposed to work...


I guessed so:)


Is it actually rebuilding all extensions?


It seems, but it's not clear to me.

Please check the output of one build

https://gist.github.com/ysagon/09a8a41ff15a242be140cd4654ba2201

And the output of a second attempt right after the previous attempt:

https://gist.github.com/ysagon/adf2053406f026a138d2d190cc3ae528

You can see that the "installing extension " lines are for the same 
extension, but the total number of extension remaining is not the same?!



Keep in mind that EasyBuild will first check which extensions are
already installed (by trying to 'load' them in R), before it will
continue and install what is missing.

Do you see the same behavior if you use --force rather than --rebuild?


I was using --force before but with the same result.


Can you supply a debug log where you see this?


Here it is (the one from the second build)

https://gist.github.com/ysagon/a6d45b6e52e3a452886770351e6c4c59



In order to know which extensions are already installed and thus can be 
skipped, EasyBuild tries to load each library.


For this, it passes for example "library(mime)" via stdin to an R command.

Here's the result of that for the 'mime' extension, from your debug log:

== 2019-03-12 11:27:20,687 run.py:192 INFO running cmd: R -q --no-save
== 2019-03-12 11:27:21,211 easyblock.py:1411 INFO exts_filter result > 
library(mime)

Error: package or namespace load failed for ‘mime’:
 package ‘mime’ was installed by an R version with different internals; 
it needs to be reinstalled for use with this R version

Execution halted
 1
== 2019-03-12 11:27:21,212 easyblock.py:1413 INFO Not skipping mime


That explains why mime is not being skipped, since it can't be loaded.

It looks like maybe mime is being picked up from somewhere else, not 
from the R installation?

Anything special in your environment that may be causing this?
Perhaps you have $R_LIBS set to a location where stuff was installed for 
an older version of R?


Note that even if you would be able to complete the whole installation, 
the sanity check would fail for these extensions...



regards,

Kenneth



Best



--
Logo UNIGE  Yann Sagon
Référent HPC

Division du système et des technologies de l'information et de la 
communication

Université de Genève | 24 rue Général-Dufour
Tél 022 379 77 37 | Bureau 151

www.unige.ch/stic 

Re: [easybuild] resume build for software with extensions

[Yann Sagon]

ping!


-- 
[image: Logo UNIGE] Yann Sagon
Référent HPC

Division du système et des technologies de l'information et de la
communication
Université de Genève | 24 rue Général-Dufour
Tél 022 379 77 37 | Bureau 151

www.unige.ch/stic

Re: [easybuild] resume build for software with extensions

[Yann Sagon]

Dear Kennet, thanks for your answer.

On Fri, Mar 8, 2019 at 11:21 AM Kenneth Hoste 
wrote:

> Dear Yann,
>
> On 07/03/2019 14:06, Yann Sagon wrote:
> > Dear list,
> >
> > sometime ago, I asked this question on the same list. I think I tried
> > the procedure but I'm unable to  make it work again.
> >
> > My issue: I try to install a new version of R. This easyconfig contains
> > more than 500 extensions, and some of them are failling during the
> > build. I then comment them out and relaunch the build. Unfortunately,
> > the build skip only the R build, not the already built extensions.
> >
> > I have modified the R-3.5.1-foss-2018b.eb file to remove every extension.
> > I did the install using:
> >
> > eb --module-only R-3.5.1-foss-2018b-without-ext.eb
> > Then I added every extension again and I did:
> > eb --skip --rebuild  ./R-3.5.1-foss-2018b.eb
> > And it fails on a given extenion.
> > I comment out this extension and re do the command:
> > eb --skip --rebuild  ./R-3.5.1-foss-2018b.eb
> > But it tries to rebuild every extension already built in the previsous
> step.
> >
> > Any clue?
>
> That's not how it's supposed to work...
>

I guessed so:)


> Is it actually rebuilding all extensions?
>

It seems, but it's not clear to me.

Please check the output of one build

https://gist.github.com/ysagon/09a8a41ff15a242be140cd4654ba2201

And the output of a second attempt right after the previous attempt:

https://gist.github.com/ysagon/adf2053406f026a138d2d190cc3ae528

You can see that the "installing extension " lines are for the same
extension, but the total number of extension remaining is not the same?!


Keep in mind that EasyBuild will first check which extensions are
> already installed (by trying to 'load' them in R), before it will
> continue and install what is missing.
>
> Do you see the same behavior if you use --force rather than --rebuild?
>

I was using --force before but with the same result.


>
> Can you supply a debug log where you see this?
>

Here it is (the one from the second build)

https://gist.github.com/ysagon/a6d45b6e52e3a452886770351e6c4c59

Best



-- 
[image: Logo UNIGE] Yann Sagon
Référent HPC

Division du système et des technologies de l'information et de la
communication
Université de Genève | 24 rue Général-Dufour
Tél 022 379 77 37 | Bureau 151

www.unige.ch/stic

Re: [easybuild] resume build for software with extensions

[Kenneth Hoste]

Dear Yann,

On 07/03/2019 14:06, Yann Sagon wrote:

Dear list,

sometime ago, I asked this question on the same list. I think I tried 
the procedure but I'm unable to  make it work again.


My issue: I try to install a new version of R. This easyconfig contains 
more than 500 extensions, and some of them are failling during the 
build. I then comment them out and relaunch the build. Unfortunately, 
the build skip only the R build, not the already built extensions.


I have modified the R-3.5.1-foss-2018b.eb file to remove every extension.
I did the install using:

eb --module-only R-3.5.1-foss-2018b-without-ext.eb
Then I added every extension again and I did:
eb --skip --rebuild  ./R-3.5.1-foss-2018b.eb
And it fails on a given extenion.
I comment out this extension and re do the command:
eb --skip --rebuild  ./R-3.5.1-foss-2018b.eb
But it tries to rebuild every extension already built in the previsous step.

Any clue?


That's not how it's supposed to work...

Is it actually rebuilding all extensions?
Keep in mind that EasyBuild will first check which extensions are 
already installed (by trying to 'load' them in R), before it will 
continue and install what is missing.


Do you see the same behavior if you use --force rather than --rebuild?

Can you supply a debug log where you see this?


regards,

Kenneth



Best

--
Logo UNIGE  Yann Sagon
Référent HPC

Division du système et des technologies de l'information et de la 
communication

Université de Genève | 24 rue Général-Dufour
Tél 022 379 77 37 | Bureau 151

www.unige.ch/stic