Re: [easybuild] next EasyBuild conf call: Wed Dec 11th 2019 (*today*), 5pm CET

[Kenneth Hoste]

Notes on today's conf call are now available at 
https://github.com/easybuilders/easybuild/wiki/Conference-call-notes-20191211 



There will not be a conf call on Christmas Day (exactly 2 weeks from 
now), so the next conf call is planned for Wed Jan 8th 2020.



regards,

Kenneth

On 11/12/2019 08:49, Kenneth Hoste wrote:

Dear EasyBuilders,

The next EasyBuild conf call is planned for Wed Dec 11th 2019 (today!), 
at 5pm CET.


You can join the conf call via https://tiny.cc/eb_conf_call .

Current agenda:

* update on recent developments

* installing CUDA software: picking list of CUDA compute capabilities

* Q

Suggestions for additional topics are welcome (please let me know if you 
have any).



More information on the EasyBuild conf calls is available at 
https://github.com/easybuilders/easybuild/wiki/Conference-calls .



regards,

Kenneth

Re: [easybuild] CP2K/6.1-foss-2019a stalls

[Marcelo Carignano]

Hello again Luca,
following your advise, I am trying with the CSCS recipes.
The build goes ok for CrayGNU, M4 and Bison, but fails at flex:

== preparing...

== configuring...

== building...

== FAILED: Installation ended unsuccessfully (build directory:
/cray_home/cari/.local/easybuild/build/flex/2.6.4/CrayGNU-19.03): build
failed (first 300 chars): cmd " make -j 36 " exited with exit code 2 and
output:

Making all in src

make[1]: Entering directory
'/cray_home/cari/.local/easybuild/build/flex/2.6.4/CrayGNU-19.03/flex-2.6.4/src'

make  all-am

make[2]: Entering directory
'/cray_home/cari/.local/easybuild/build/flex/2.6.4/CrayGNU-19.03/flex-2.6.4/s
(took 16 sec)

== Results of the build can be found in the log file(s)
/tmp/eb-3aAQri/easybuild-flex-2.6.4-20191211.205055.PxPKd.log

ERROR: Build of
/cray_home/cari/.local/easybuild/software/EasyBuild/4.0.1/easybuild/easyconfigs/f/flex/flex-2.6.4-CrayGNU-19.03.eb
failed (err: 'build failed (first 300 chars): cmd " make -j 36 " exited
with exit code 2 and output:\nMaking all in src\nmake[1]: Entering
directory
\'/cray_home/cari/.local/easybuild/build/flex/2.6.4/CrayGNU-19.03/flex-2.6.4/src\'\nmake
all-am\nmake[2]: Entering directory
\'/cray_home/cari/.local/easybuild/build/flex/2.6.4/CrayGNU-19.03/flex-2.6.4/s')

cari@xc50-1:~/Builds/EasyBuild>

The log file shows:

./stage1flex   -o stage1scan.c ./scan.l

Makefile:1696: recipe for target 'stage1scan.c' failed

make[2]: *** [stage1scan.c] Illegal instruction (core dumped)

make[2]: Leaving directory
'/cray_home/cari/.local/easybuild/build/flex/2.6.4/CrayGNU-19.03/flex-2.6.4/src'

Makefile:546: recipe for target 'all' failed

make[1]: *** [all] Error 2

make[1]: Leaving directory
'/cray_home/cari/.local/easybuild/build/flex/2.6.4/CrayGNU-19.03/flex-2.6.4/src'

Makefile:533: recipe for target 'all-recursive' failed

make: *** [all-recursive] Error 1

 (at
easybuild/software/EasyBuild/4.0.1/lib/python2.7/site-packages/easybuild/tools/run.py:529
in parse_cmd_output)

== 2019-12-11 20:51:12,760 easyblock.py:3082 WARNING build failed (first
300 chars): cmd " make -j 36 " exited with exit code 2 and output:

Making all in src

make[1]: Entering directory
'/cray_home/cari/.local/easybuild/build/flex/2.6.4/CrayGNU-19.03/flex-2.6.4/src'

make  all-am

make[2]: Entering directory
'/cray_home/cari/.local/easybuild/build/flex/2.6.4/CrayGNU-19.03/flex-2.6.4/s

== 2019-12-11 20:51:12,760 easyblock.py:294 INFO Closing log for
application name flex version 2.6.4

cari@xc50-1:~/Builds/EasyBuild>

The command I am using is:

eb  flex-2.6.4-CrayGNU-19.03.eb --robot --modules-tool EnvironmentModulesC
--module-syntax Tcl --optarch=x86-skylake

This stage went just fine while installing CP2K/6.1-foss-2019a
Not sure what could be wrong
I am using 19.03 since this is the one installed right now.

Any advice?

Thanks,

Marcelo.



On Wed, Dec 11, 2019 at 5:04 PM Marsella Luca  wrote:

> Hi Marcelo,
>
>
>
> The issue that you encounter might also be unrelated to the EasyBuild
> recipe that you use. Would you like to try a different recipe for testing?
>
> For instance, we deploy CP2K on the Cray XC50 at CSCS with custom
> EasyBuild recipes, that you can find in the mirror below:
>
>
> https://github.com/easybuilders/CSCS/tree/master/easybuild/easyconfigs/c/CP2K
>
> The CP2K recipes that we use rely on the EasyBuild toolchain “CrayGNU”,
> based on the MPI library “cray-mpich” (default in the Cray XC50).
>
>
>
> Best regards,
>
> Luca
>
>
>
> --
>
> Luca Marsella, PhD
>
> CSCS Swiss National Supercomputing Centre
>
> Via Trevano 131
>
> CH-6900 Lugano
>
> Switzerland
>
>
>
> *From: * on behalf of Marcelo Carignano
> 
> *Reply-To: *"easybuild@lists.ugent.be" 
> *Date: *Wednesday, 11 December 2019 at 14:28
> *To: *"easybuild@lists.ugent.be" 
> *Subject: *[easybuild] CP2K/6.1-foss-2019a stalls
>
>
>
> Hello,
>
> I have installed CP2K/6.1-foss-2019a on a Cray XC50.
>
> No errors reported.
>
> Then I proceed to test it by submitting one of my jobs.
>
> The run starts with no problem, but after a time (that depends on each
> test) the program stalls. It doesn't get killed, it just stops writing the
> output.
>
> I use the slurm scheduler installed in the computer, and in the script I
> load the corresponding module.
>
>
>
> Is anyone familiar with this type of problems?
>
> Maybe there is something wrong that I am missing, but cannot imagine what,
> especially because the programs actually starts running. The job is a
> molecular dynamics, it goes 10, or up to 50 MD steps before it stalls.
>
>
>
> Thanks,
>
> Marcelo
>

[easybuild] help needed

[Ernesto Eduardo Diaz Conde]

Hello good, I'm almost a month trying to install Pytorch, I've tried
several variants even changing the toolchain but I always get error in the
testing process before installing.

 File "test/run_test.py", line 400, in main
raise RuntimeError(message)
RuntimeError: test_torch failed! Received signal: SIGILL
 (at
easybuild/software/EasyBuild/4.0.1/lib/python2.7/site-packages/easybuild/tools/run.py:529
in parse_cmd_output)
== 2019-12-11 11:34:06,221 easyblock.py:3082 WARNING build failed (first
300 chars): cmd "  export
PYTHONPATH=/opt/apps/easybuild/build/PyTorch/1.2.0/foss-2019b-Python-3.7.2/pytorch-1.2.0/build/lib.linux-x86_64-3.7:$PYTHONPATH
&& export
LD_LIBRARY_PATH=/opt/apps/easybuild/build/PyTorch/1.2.0/foss-2019b-Python-3.7.2/pytorch-1.2.0/torch/lib64:$LD_LIBRARY_PATH
&& python test/run_test.p
== 2019-12-11 11:34:06,226 easyblock.py:294 INFO Closing log for
application name PyTorch version 1.2.0

Re: [easybuild] generate a Dockerfile with easybuild

[Juan Cabrera]

Thanks Alan,

I can build a docker image now. I will check if it works and easybuild
is inside.

Juan

On 11/12/19 15:41, Alan O'Cais wrote:
> Ok, from the test suite I was able to figure out that this works:
> eb -d --experimental -C --container-config=centos:7
> --container-type=docker M4-1.4.18.eb
>
> On Wed, 11 Dec 2019 at 15:35, Alan O'Cais  > wrote:
>
> I can't figure out what should go there either! I think we need an
> example for each case, the docker one is missing.
>
> Alan
>
> On Wed, 11 Dec 2019 at 14:20, Juan Cabrera  > wrote:
>
> Hi,
>
> I would like to have a centos7 docker with eb to build modules.
>
> I try the experimental procedure  in the documentation to get
> a Dockerfile
> https://easybuild.readthedocs.io/en/latest/Containers.html
>
> But the command
>
>  eb --experimental -C --container-config
> 'bootstrap=yum,from=centos:centos7,osversion=7'
> --container-type docker $CFGS1/m/M4/M4-1.4.18.eb
>
> Gives this error:
>
> == temporary log file in case of crash
> /tmp/eb-yNoEvs/easybuild-IbzBNd.log
> ERROR: Unsupported container config
> 'bootstrap=yum,from=centos:centos7,osversion=7'
>
> To generate a syngularity file, the command
>
> eb --experimental -C --container-config
> 'bootstrap=yum,from=centos:centos7,osversion=7' 
> $CFGS1/m/M4/M4-1.4.18.eb
>
> seems to work. I get a
> .local/easybuild/containers/Singularity.M4-1.4.18 file.
>
> Perhaps I misunderstood the --container-config flag,
>
> Can somebody help me?
>
> Regards
>
> Juan
> -- 
>
> Juan CABRERA
> Correspondant informatique
> Département de Mathématiques
>
> T. 081724919
> juan.cabr...@unamur.be 
> http://staff.unamur.be/jbcabrer
>
> Université de Namur ASBL
> Rue de Bruxelles 61 - 5000 Namur
> Belgique
>
> Let’s respect the environment together.
> Only print this message if necessary!
>
>
>
> -- 
> Dr. Alan O'Cais
> E-CAM Software Manager
> Juelich Supercomputing Centre
> Forschungszentrum Juelich GmbH
> 52425 Juelich, Germany
>
> Phone: +49 2461 61 5213
> Fax: +49 2461 61 6656
> E-mail: a.oc...@fz-juelich.de 
> WWW:    http://www.fz-juelich.de/ias/jsc/EN
>
>
>
> -- 
> Dr. Alan O'Cais
> E-CAM Software Manager
> Juelich Supercomputing Centre
> Forschungszentrum Juelich GmbH
> 52425 Juelich, Germany
>
> Phone: +49 2461 61 5213
> Fax: +49 2461 61 6656
> E-mail: a.oc...@fz-juelich.de 
> WWW:    http://www.fz-juelich.de/ias/jsc/EN
>
>
> 
> 
> Forschungszentrum Juelich GmbH
> 52425 Juelich
> Sitz der Gesellschaft: Juelich
> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
> Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
> Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
> Prof. Dr. Sebastian M. Schmidt
> 
> 
>
-- 

Juan CABRERA
Correspondant informatique
Département de Mathématiques

T. 081724919
juan.cabr...@unamur.be 
http://staff.unamur.be/jbcabrer

Université de Namur ASBL
Rue de Bruxelles 61 - 5000 Namur
Belgique

Let’s respect the environment together.
Only print this message if necessary!

Re: [easybuild] generate a Dockerfile with easybuild

[Alan O'Cais]

Ok, from the test suite I was able to figure out that this works:
eb -d --experimental -C --container-config=centos:7 --container-type=docker 
M4-1.4.18.eb

On Wed, 11 Dec 2019 at 15:35, Alan O'Cais 
mailto:a.oc...@fz-juelich.de>> wrote:
I can't figure out what should go there either! I think we need an example for 
each case, the docker one is missing.

Alan

On Wed, 11 Dec 2019 at 14:20, Juan Cabrera 
mailto:juan.cabr...@unamur.be>> wrote:
Hi,

I would like to have a centos7 docker with eb to build modules.

I try the experimental procedure  in the documentation to get a Dockerfile 
https://easybuild.readthedocs.io/en/latest/Containers.html

But the command

 eb --experimental -C --container-config 
'bootstrap=yum,from=centos:centos7,osversion=7' --container-type docker 
$CFGS1/m/M4/M4-1.4.18.eb

Gives this error:

== temporary log file in case of crash /tmp/eb-yNoEvs/easybuild-IbzBNd.log
ERROR: Unsupported container config 
'bootstrap=yum,from=centos:centos7,osversion=7'

To generate a syngularity file, the command

eb --experimental -C --container-config 
'bootstrap=yum,from=centos:centos7,osversion=7'  $CFGS1/m/M4/M4-1.4.18.eb

seems to work. I get a .local/easybuild/containers/Singularity.M4-1.4.18 file.

Perhaps I misunderstood the --container-config flag,

Can somebody help me?

Regards

Juan
--
[cid:16ef56289978e723c3b1]

Juan CABRERA
Correspondant informatique
Département de Mathématiques

T. 081724919
juan.cabr...@unamur.be
http://staff.unamur.be/jbcabrer

Université de Namur ASBL
Rue de Bruxelles 61 - 5000 Namur
Belgique

Let’s respect the environment together.
Only print this message if necessary!


--
Dr. Alan O'Cais
E-CAM Software Manager
Juelich Supercomputing Centre
Forschungszentrum Juelich GmbH
52425 Juelich, Germany

Phone: +49 2461 61 5213
Fax: +49 2461 61 6656
E-mail: a.oc...@fz-juelich.de
WWW:http://www.fz-juelich.de/ias/jsc/EN


--
Dr. Alan O'Cais
E-CAM Software Manager
Juelich Supercomputing Centre
Forschungszentrum Juelich GmbH
52425 Juelich, Germany

Phone: +49 2461 61 5213
Fax: +49 2461 61 6656
E-mail: a.oc...@fz-juelich.de
WWW:http://www.fz-juelich.de/ias/jsc/EN




Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt

Re: [easybuild] generate a Dockerfile with easybuild

[Alan O'Cais]

I can't figure out what should go there either! I think we need an example for 
each case, the docker one is missing.

Alan

On Wed, 11 Dec 2019 at 14:20, Juan Cabrera 
mailto:juan.cabr...@unamur.be>> wrote:
Hi,

I would like to have a centos7 docker with eb to build modules.

I try the experimental procedure  in the documentation to get a Dockerfile 
https://easybuild.readthedocs.io/en/latest/Containers.html

But the command

 eb --experimental -C --container-config 
'bootstrap=yum,from=centos:centos7,osversion=7' --container-type docker 
$CFGS1/m/M4/M4-1.4.18.eb

Gives this error:

== temporary log file in case of crash /tmp/eb-yNoEvs/easybuild-IbzBNd.log
ERROR: Unsupported container config 
'bootstrap=yum,from=centos:centos7,osversion=7'

To generate a syngularity file, the command

eb --experimental -C --container-config 
'bootstrap=yum,from=centos:centos7,osversion=7'  $CFGS1/m/M4/M4-1.4.18.eb

seems to work. I get a .local/easybuild/containers/Singularity.M4-1.4.18 file.

Perhaps I misunderstood the --container-config flag,

Can somebody help me?

Regards

Juan
--
[cid:16ef56289978e723c3b1]

Juan CABRERA
Correspondant informatique
Département de Mathématiques

T. 081724919
juan.cabr...@unamur.be
http://staff.unamur.be/jbcabrer

Université de Namur ASBL
Rue de Bruxelles 61 - 5000 Namur
Belgique

Let’s respect the environment together.
Only print this message if necessary!


--
Dr. Alan O'Cais
E-CAM Software Manager
Juelich Supercomputing Centre
Forschungszentrum Juelich GmbH
52425 Juelich, Germany

Phone: +49 2461 61 5213
Fax: +49 2461 61 6656
E-mail: a.oc...@fz-juelich.de
WWW:http://www.fz-juelich.de/ias/jsc/EN




Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt

Re: [easybuild] CP2K/6.1-foss-2019a stalls

[Marcelo Carignano]

Thank you Luca, I will be testing that right now.
Grazie mille,
Marcelo.

On Wed, Dec 11, 2019 at 5:04 PM Marsella Luca  wrote:

> Hi Marcelo,
>
>
>
> The issue that you encounter might also be unrelated to the EasyBuild
> recipe that you use. Would you like to try a different recipe for testing?
>
> For instance, we deploy CP2K on the Cray XC50 at CSCS with custom
> EasyBuild recipes, that you can find in the mirror below:
>
>
> https://github.com/easybuilders/CSCS/tree/master/easybuild/easyconfigs/c/CP2K
>
> The CP2K recipes that we use rely on the EasyBuild toolchain “CrayGNU”,
> based on the MPI library “cray-mpich” (default in the Cray XC50).
>
>
>
> Best regards,
>
> Luca
>
>
>
> --
>
> Luca Marsella, PhD
>
> CSCS Swiss National Supercomputing Centre
>
> Via Trevano 131
>
> CH-6900 Lugano
>
> Switzerland
>
>
>
> *From: * on behalf of Marcelo Carignano
> 
> *Reply-To: *"easybuild@lists.ugent.be" 
> *Date: *Wednesday, 11 December 2019 at 14:28
> *To: *"easybuild@lists.ugent.be" 
> *Subject: *[easybuild] CP2K/6.1-foss-2019a stalls
>
>
>
> Hello,
>
> I have installed CP2K/6.1-foss-2019a on a Cray XC50.
>
> No errors reported.
>
> Then I proceed to test it by submitting one of my jobs.
>
> The run starts with no problem, but after a time (that depends on each
> test) the program stalls. It doesn't get killed, it just stops writing the
> output.
>
> I use the slurm scheduler installed in the computer, and in the script I
> load the corresponding module.
>
>
>
> Is anyone familiar with this type of problems?
>
> Maybe there is something wrong that I am missing, but cannot imagine what,
> especially because the programs actually starts running. The job is a
> molecular dynamics, it goes 10, or up to 50 MD steps before it stalls.
>
>
>
> Thanks,
>
> Marcelo
>

[easybuild] CP2K/6.1-foss-2019a stalls

[Marcelo Carignano]

Hello,
I have installed CP2K/6.1-foss-2019a on a Cray XC50.
No errors reported.
Then I proceed to test it by submitting one of my jobs.
The run starts with no problem, but after a time (that depends on each
test) the program stalls. It doesn't get killed, it just stops writing the
output.
I use the slurm scheduler installed in the computer, and in the script I
load the corresponding module.

Is anyone familiar with this type of problems?
Maybe there is something wrong that I am missing, but cannot imagine what,
especially because the programs actually starts running. The job is a
molecular dynamics, it goes 10, or up to 50 MD steps before it stalls.

Thanks,

Marcelo.

[easybuild] generate a Dockerfile with easybuild

[Juan Cabrera]

Hi,

I would like to have a centos7 docker with eb to build modules.

I try the experimental procedure  in the documentation to get a
Dockerfile https://easybuild.readthedocs.io/en/latest/Containers.html

But the command

 eb --experimental -C --container-config
'bootstrap=yum,from=centos:centos7,osversion=7' --container-type docker
$CFGS1/m/M4/M4-1.4.18.eb

Gives this error:

== temporary log file in case of crash /tmp/eb-yNoEvs/easybuild-IbzBNd.log
ERROR: Unsupported container config
'bootstrap=yum,from=centos:centos7,osversion=7'

To generate a syngularity file, the command

eb --experimental -C --container-config
'bootstrap=yum,from=centos:centos7,osversion=7'  $CFGS1/m/M4/M4-1.4.18.eb

seems to work. I get a .local/easybuild/containers/Singularity.M4-1.4.18
file.

Perhaps I misunderstood the --container-config flag,

Can somebody help me?

Regards

Juan
-- 

Juan CABRERA
Correspondant informatique
Département de Mathématiques

T. 081724919
juan.cabr...@unamur.be 
http://staff.unamur.be/jbcabrer

Université de Namur ASBL
Rue de Bruxelles 61 - 5000 Namur
Belgique

Let’s respect the environment together.
Only print this message if necessary!

Re: [easybuild] easyconfig path for existing module?

[Yann Sagon]

Dear Kenneth,

Le 10.12.2019 à 14:11, Kenneth Hoste a écrit :


Enabling or disabling fixed-installdir-naming-scheme only affects the 
name of the installation directory (and whether or not you can use "eb 
--module-only" to install an additional module file with a different 
module naming scheme for an existing installation, see also 
https://easybuild.readthedocs.io/en/latest/EasyBuild4-overview-of-changes.html#fixed-installdir-naming-scheme-enabled-by-default 
.


The problem you're hitting is that when you're using HierarchicalMNS, 
the easyconfig file for each of the dependencies must be available.
EasyBuild relies on same stuff (like 'moduleclass') which is only 
available in the easyconfig file to determine the module name in a 
module hierarchy (which is step 0 in determining whether a module is 
already avaiable for that dependency).


Check whether "eb --search pixman-0.38.0" yields any results for the 
pixman easyconfig file.

indeed it doesn't return the eb file.


The solution to your problem is most likely to pass the location of 
the pixman easyconfig file to either --robot or --robot-paths.
A common way to avoid this issue in general is to add the location of 
the easyconfigs archive (see 'repositorypath' in the output of "eb 
--show-config") to --robot-paths in your EasyBuild configuration; see 
also 
https://easybuild.readthedocs.io/en/latest/Using_the_EasyBuild_command_line.html#controlling-the-robot-search-path.


In fact this is what I was already doing, but when I built pixman, the 
rights were wrong on repositorypath directory and the eb wasn't written! 
My bad, problem solved!


Thanks

Best

--

Logo UNIGE  Yann Sagon
Référent HPC

Division du système et des technologies de l'information et de la 
communication

Université de Genève | 24 rue Général-Dufour
Tél 022 379 77 37 | Bureau 151

www.unige.ch/stic 

Re: [easybuild] Upgrade 4.0.0 to 4.1.0

[Kenneth Hoste]

Dear William,

On 11/12/2019 10:59, William Brown wrote:
<< commenting out the "sys.exit(1)" in the install_fake_vsc function in 
easybuild/tools/filetools.py>>


That worked, thanks very much.  I now have EasyBuild 4.1.0 on both clusters.


Great, thanks for confirming.

We'll look into relaxing the vsc import check a bit to ignore imports 
triggered from pkgutil.py, see 
https://github.com/easybuilders/easybuild-framework/pull/3120 .



regards,

Kenneth



It is entirely possible that the system that fails has a system-wide 
vsc.base installed, as it is a national cluster that I don't manage - 
while the one where it worked I built myself and there are no extra 
Python packages in the system Python - after all I add those with 
EasyBuild like BioPython etc.


William


*William Brown*
Systems Administrator

*RCSI* Information Technology
Royal College of Surgeons in Ireland
Building 121 St Stephens Green, IT dept
*T: *015720012
*E: *williambr...@rcsi.ie *W: *www.rcsi.com 

/Transforming Healthcare Education, Research and Service: RCSI Strategic 
Plan 2018-2022 /






-Original Message-
From: easybuild-requ...@lists.ugent.be 
 On Behalf Of Kenneth Hoste

Sent: 10 December 2019 19:03
To: easybuild@lists.ugent.be
Subject: Re: [easybuild] Upgrade 4.0.0 to 4.1.0

Dear William,

On 10/12/2019 19:15, William Brown wrote:
 > I suspect that I am missing something here..
 >
 > I wanted to upgrade from 4.0.0 to 4.1.0 so I tried first
 >
 > $ eb --install-latest-eb-release
 >
 > ..which failed with:
 >
 > == postprocessing...
 >
 > == sanity checking...
 >
 > ERROR: Detected import from 'vsc' namespace in
 > /usr/lib64/python2.7/pkgutil.py (line 246)
 >
 > vsc-base & vsc-install were ingested into the EasyBuild framework in
 > EasyBuild v4.0
 >
 > The functionality you need may be available in the 'easybuild.base.*'
 > namespace.


It seems like you have vsc-base installed system-wide, and EasyBuild is 
tripping over that during the sanity check...


The vsc import check is done to ensure you're not still using "import 
vsc" in any of the additional Python modules you're using with EasyBuild 
(via --include-* or via --hooks, for example).


EasyBuild 4.x catches any imports from the 'vsc' namespace, and produces 
the error you've hit.
In this case though, it seems like the import of the vsc namespace is 
done by Python itself, so that's backfiring now...


One thing you could try is to get rid of the system-wide installation of 
vsc-base, if that's an option (it's useless anyway for EasyBuild 4.x).


I'll look into making the vsc import check a bit less strict, but that 
won't do you much good now (since the problem is in the EasyBuild v4.0.0 
installation you're using to install EasyBuild v4.1.0).


A workaround you could try is to use EasyBuild v3.9.4 to run "eb 
--install-latest-eb-release", if you have that installed (older 3.x 
versions won't work to install EasyBuild 4.x though).


If you don't have EasyBuild v3.9.4 installed, you could temporarily 
install it using "pip install --prefix /tmp/eb394 easybuild==3.9.4", so 
you can use that to install EasyBuild v4.1.0.


An alternative (perhaps more desperate/ugly) workaround is to
(temporarily) tweak your EasyBuild v4.0.0 installation, by commenting 
out the "sys.exit(1)" in the install_fake_vsc function in 
easybuild/tools/filetools.py.
If you then try "eb --install-latest-eb-release", you'll still see the 
errors being printed, but they won't be fatal, and the v4.1.0 
installation should hopefully complete.


Do let me know if that helps at all...


 > So I decided to go back to bootstrapping EasyBuild, although using the
 > same EASYBUILDPREFIX path as my exiting installation (as I don't
 > really want to have to migrate the contents).

That's fine, you can use the bootstrap procedure in an existing software 
stack, all it really does is install EasyBuild in a temporary location, 
and then use that EasyBuild installation to install EasyBuild in the 
specified location. It doesn't care at all if other modules are already 
installed there (and it won't touch them either).



 >
 > That also fails.
 >
 > [[INFO]] running post install command 'easy_install --upgrade
 > --prefix=/tmp/tmpOqoOOs/eb_stage1 vsc-base<2.9.0'
 >
 > Traceback (most recent call last):
 >
 >    File "bootstrap_eb.py", line 1124, in 
 >
 >      main()
 >
 >    File "bootstrap_eb.py", line 916, in main
 >
 >      templates = stage1(tmpdir, sourcepath, distribute_egg_dir,
 > forcedversion)
 >
 >    File "bootstrap_eb.py", line 635, in stage1
 >
 >      import vsc.utils.fancylogger
 >
 > ImportError: No module named utils.fancylogger
 >
 > Looking in the bootstrap_eb.py file it does indeed contain 'import
 > vsc.utils.fancylogger'.  Reading back in older discussions I get the
 > impression that vsc.utils was being replaced.

Indeed, we (still) haven't cleaned up the bootstrap script to get rid of 
the stuff it's using from 

RE: [easybuild] Upgrade 4.0.0 to 4.1.0

[William Brown]

<< commenting out the "sys.exit(1)" in the install_fake_vsc function in easybuild/tools/filetools.py>>

That worked, thanks very much.  I now have EasyBuild 4.1.0 on both clusters.

It is entirely possible that the system that fails has a system-wide vsc.base installed, as it is a national cluster that I don't manage - while the one where it worked I built myself and there are no extra Python packages in the system Python - after all I add those with EasyBuild like BioPython etc.

William

William Brown 
Systems Administrator

RCSI 
Information Technology Royal College of Surgeons in Ireland Building 121 St Stephens Green, IT dept   T: 015720012 
E: williambr...@rcsi.ie W: www.rcsi.com
Transforming 
Healthcare Education, Research and Service: RCSI Strategic Plan 
2018-2022
-Original Message-
From: easybuild-requ...@lists.ugent.be  On Behalf Of Kenneth Hoste
Sent: 10 December 2019 19:03
To: easybuild@lists.ugent.be
Subject: Re: [easybuild] Upgrade 4.0.0 to 4.1.0

Dear William,

On 10/12/2019 19:15, William Brown wrote:
> I suspect that I am missing something here..
> 
> I wanted to upgrade from 4.0.0 to 4.1.0 so I tried first
> 
> $ eb --install-latest-eb-release
> 
> ..which failed with:
> 
> == postprocessing...
> 
> == sanity checking...
> 
> ERROR: Detected import from 'vsc' namespace in 
> /usr/lib64/python2.7/pkgutil.py (line 246)
> 
> vsc-base & vsc-install were ingested into the EasyBuild framework in 
> EasyBuild v4.0
> 
> The functionality you need may be available in the 'easybuild.base.*' 
> namespace.


It seems like you have vsc-base installed system-wide, and EasyBuild is tripping over that during the sanity check...

The vsc import check is done to ensure you're not still using "import vsc" in any of the additional Python modules you're using with EasyBuild (via --include-* or via --hooks, for example).

EasyBuild 4.x catches any imports from the 'vsc' namespace, and produces the error you've hit.
In this case though, it seems like the import of the vsc namespace is done by Python itself, so that's backfiring now...

One thing you could try is to get rid of the system-wide installation of vsc-base, if that's an option (it's useless anyway for EasyBuild 4.x).

I'll look into making the vsc import check a bit less strict, but that won't do you much good now (since the problem is in the EasyBuild v4.0.0 installation you're using to install EasyBuild v4.1.0).

A workaround you could try is to use EasyBuild v3.9.4 to run "eb --install-latest-eb-release", if you have that installed (older 3.x versions won't work to install EasyBuild 4.x though).

If you don't have EasyBuild v3.9.4 installed, you could temporarily install it using "pip install --prefix /tmp/eb394 easybuild==3.9.4", so you can use that to install EasyBuild v4.1.0.

An alternative (perhaps more desperate/ugly) workaround is to
(temporarily) tweak your EasyBuild v4.0.0 installation, by commenting out the "sys.exit(1)" in the install_fake_vsc function in easybuild/tools/filetools.py.
If you then try "eb --install-latest-eb-release", you'll still see the errors being printed, but they won't be fatal, and the v4.1.0 installation should hopefully complete.

Do let me know if that helps at all...


> So I decided to go back to bootstrapping EasyBuild, although using the 
> same EASYBUILDPREFIX path as my exiting installation (as I don't 
> really want to have to migrate the contents).

That's fine, you can use the bootstrap procedure in an existing software stack, all it really does is install EasyBuild in a temporary location, and then use that EasyBuild installation to install EasyBuild in the specified location. It doesn't care at all if other modules are already installed there (and it won't touch them either).


> 
> That also fails.
> 
> [[INFO]] running post install command 'easy_install --upgrade
> --prefix=/tmp/tmpOqoOOs/eb_stage1 vsc-base<2.9.0'
> 
> Traceback (most recent call last):
> 
>    File "bootstrap_eb.py", line 1124, in 
> 
>      main()
> 
>    File "bootstrap_eb.py", line 916, in main
> 
>      templates = stage1(tmpdir, sourcepath, distribute_egg_dir,
> forcedversion)
> 
>    File "bootstrap_eb.py", line 635, in stage1
> 
>      import vsc.utils.fancylogger
> 
> ImportError: No module named utils.fancylogger
> 
> Looking in the bootstrap_eb.py file it does indeed contain 'import 
> vsc.utils.fancylogger'.  Reading back in older discussions I get the 
> impression that vsc.utils was being replaced.

Indeed, we (still) haven't cleaned up the bootstrap script to get rid of the stuff it's using from vsc-base. That's just one of the issues with it, there's others...

This issue is a bit puzzling though. Why is the import from the 'vsc' 
namespace failing, while the bootstrap script still installs vsc-base?

Do you have a broken system-wide vsc-base installation perhaps?

This could also explain while you're having the issues with updating to
v4.1.0 using v4.0.0 while others don't.


> I must have missed