date:20191118

Re: [easybuild] TensorFlow/2.0.0 for foss-2019a-Python-3.7.2 fail with ssl error.

2019-11-18 Thread Yann Sagon


Dear Kenneth, dear Jack

Le 12.11.2019 à 17:00, Kenneth Hoste a écrit :

Hi Yann,

I'm no expert in issues like this, but I suspect it has nothing to do 
with the Python installation itself.


Which Linux distro is this?


It's CentOS7 but it WAS CentOS6 and we didn't recompiled everything.

The Python-3.7.2 for foss-2019a was linked against an EB ssl instead of 
the one from the system. And this wasn't working for this eb file in 
particular because the ssl was too recent!


I rebuilt Python using system SSL and it's working!

Best

--

Logo UNIGE  Yann Sagon
Référent HPC

Division du système et des technologies de l'information et de la 
communication

Université de Genève | 24 rue Général-Dufour
Tél 022 379 77 37 | Bureau 151

www.unige.ch/stic

Re: [easybuild] Can't get local install to work

2019-11-18 Thread Kenneth Hoste


Dear Niek,

Sorry that you're hitting these issues, let's try and get them sorted...

On 18/11/2019 20:00, Niek de Klein wrote:

Hi all,

I am trying to have my own installation of EasyBuild, but I have 3 problems:

1. When I try to run bootstrap.py it gives an error that md5sums are 
missing, but I'm not sure where to add them as I thought that should 
then have been included in the EasyBuild-4.0.1.eb file:


     easybuild.tools.build_log.EasyBuildError: "Build of 
/tmp/tmpfcUSAH/EasyBuild-4.0.1.eb failed (err: 'build failed (first 300 
chars): Missing checksum for easybuild-framework-4.0.1.tar.gz')"


Sorry for not responding to the GitHub issue [1] you opened on this 
(will do so now with the answer below).


We still need to clean up the bootstrap script a bit after the EasyBuild 
v4.0 release, it's still doing a bunch of stuff it shouldn't anymore, 
and it's not compatible yet with Python 3 (see also 
https://github.com/easybuilders/easybuild-framework/pull/3028).



Your active EasyBuild configuration has --enforce-checksums enabled (via 
$EASYBUILD_ENFORCE_CHECKSUMS), which explains the problem you're seeing.


The bootstrap script auto-generates an easyconfig file, but it doesn't 
include checksums (it could I guess, with a bit more work).


You can dance around this though, by temporarily disabling the checksums 
requirement, only for the bootstrap:


EASYBUILD_DISABLE_ENFORCE_CHECKSUMS=1 python bootstrap 


If you want to, you can bootstrap EasyBuild to a temporary location (for 
example /tmp/$USER), and then use that installation to pull in an 
easyconfig file for EasyBuild v4.0.1 that does have checksums, using "eb 
--from-pr 9149 --installpath " (which pulls in the 
easyconfig from 
https://github.com/easybuilders/easybuild-easyconfigs/pull/9149/files) 
for the actual installation you intend to use.





2. I changed the robo and installation path to be different, but I keep 
getting "Failed to determine minimal toolchain for dep ..." errors. The 
eb --show-config output is at the end of this e-mail, and shows that my 
robot-paths is 
/groups/umcg-biogen/tmp01/apps/easybuild-easyconfigs/easybuild/easyconfigs/, 
which is where I cloned the easyconfigs repository.


What happens if you use "eb --search" with this setup, can 'eb' find 
easyconfig files?


Try something like "eb --search tensorflow-1.14" for example.




3. Even if the robo path wasn't working, tools that are already in my 
module avail are not being picked up. For example, EasyBuild that is not 
working uses "toolchain = {'name': 'foss', 'version': '2018b'}" and has 
as one of the dependencies "('ncurses', '6.0'),". When I try to install 
it I get


ERROR: Failed to process easyconfig 
/groups/umcg-biogen/tmp01/tools/brain_eQTL/EasyBuild/easyconfigs/SAMtools-1.5-foss-2018b.eb: 
Failed to determine minimal toolchain for dep ncurses 6.0


but when I do ml avail ncur I get:

-- 
/groups/umcg-biogen/tmp01/apps/modules/devel 
---

    ncurses/6.0-foss-2018b

--- 
/apps/modules/devel 
---

    ncurses/6.0    ncurses/6.1-foss-2018b    ncurses/6.1-GCCcore-7.3.0 (D)

So the ncurses module is already available


Am I missing something in my configuration?


This is a consequence of having --minimal-toolchains enabled, which 
requires that EasyBuild is able to find the easyconfig file for the 
toolchain (foss-2018b.eb in this case).


It needs to figure out what the (versions of) the subtoolchains are for 
foss/2018b, so it can check for ncurses installed with GCCcore first, 
before gompi, before foss.


So this is basically the same issue as with problem 2, I bet "eb 
--search ^foss-2018b.eb" doesn't give you any result?



Are you 100% sure the path you specified for --robot-paths is correct?

Note that you can also prepend or append to the robot search path 
(rather than override the default), see 
https://easybuild.readthedocs.io/en/latest/Using_the_EasyBuild_command_line.html#controlling-robot-search-path 
.


I hope this helps...


regards,

Kenneth


[1] https://github.com/easybuilders/easybuild-framework/issues/3079



Thanks for your help!
Niek de Klein


#
# Current EasyBuild configuration
# (C: command line argument, D: default value, E: environment variable, 
F: configuration file)

#
buildpath                   (E) = 
/groups/umcg-biogen/tmp01/apps//.tmp/easybuild/builds/
containerpath               (D) = 
/home/umcg-ndeklein/.local/easybuild/containers

enforce-checksums           (E) = True
installpath                 (E) = /groups/umcg-biogen/tmp01/apps/
minimal-toolchains          (E) = True
repositorypath              (D) = 
/home/umcg-ndeklein/.local/easybuild/ebfiles_repo
robot-paths                 (E) = 
/groups/umcg-biogen/tmp01/apps/easybuild-easyconfigs/easybuild/easyconfigs/

Re: [easybuild] Can't get local install to work

2019-11-18 Thread Niek de Klein

Hi Kenneth,

I spoke too soon, some of my installations work but not all. When I try to
isntall Autoconf-2.69-foss-2018.eb I get

ERROR: Failed to process easyconfig
/groups/umcg-biogen/tmp01/apps/easybuild-easyconfigs/easybuild/easyconfigs/a/Autoconf/Autoconf-2.69-foss-2018b.eb:
Failed to determine minimal toolchain for dep M4 1.4.17

with eb --search I get:

eb --search M4-1.4.17-foss-2018b
 *
/groups/umcg-biogen/tmp01/apps/easybuild-easyconfigs/easybuild/easyconfigs/m/M4/M4-1.4.17-foss-2018b.eb

eb --search ^foss-2018b.eb
 *
/groups/umcg-biogen/tmp01/apps/easybuild-easyconfigs/easybuild/easyconfigs/f/foss/foss-2018b.eb

So it can find both the M4-1.4.17-foss-2018b.eb file and the foss-2018.eb
file.

Kind regards,
Niek





On Mon, Nov 18, 2019 at 3:21 PM Niek de Klein  wrote:

> Hi Kenneth,
>
> Thanks for your help!
>
> 1. is solved now with your suggestion of
> EASYBUILD_DISABLE_ENFORCE_CHECKSUMS=1
>
> Now that the bootstrap.py ran correctly it can also find the easybuild
> files in the location that I have configured, so this solves 1 and 2 as
> well.
>
> Thanks,
> Niek
>
> On Mon, Nov 18, 2019 at 3:01 PM Kenneth Hoste 
> wrote:
>
>> Dear Niek,
>>
>> Sorry that you're hitting these issues, let's try and get them sorted...
>>
>> On 18/11/2019 20:00, Niek de Klein wrote:
>> > Hi all,
>> >
>> > I am trying to have my own installation of EasyBuild, but I have 3
>> problems:
>> >
>> > 1. When I try to run bootstrap.py it gives an error that md5sums are
>> > missing, but I'm not sure where to add them as I thought that should
>> > then have been included in the EasyBuild-4.0.1.eb file:
>> >
>> >  easybuild.tools.build_log.EasyBuildError: "Build of
>> > /tmp/tmpfcUSAH/EasyBuild-4.0.1.eb failed (err: 'build failed (first 300
>> > chars): Missing checksum for easybuild-framework-4.0.1.tar.gz')"
>>
>> Sorry for not responding to the GitHub issue [1] you opened on this
>> (will do so now with the answer below).
>>
>> We still need to clean up the bootstrap script a bit after the EasyBuild
>> v4.0 release, it's still doing a bunch of stuff it shouldn't anymore,
>> and it's not compatible yet with Python 3 (see also
>> https://github.com/easybuilders/easybuild-framework/pull/3028).
>>
>>
>> Your active EasyBuild configuration has --enforce-checksums enabled (via
>> $EASYBUILD_ENFORCE_CHECKSUMS), which explains the problem you're seeing.
>>
>> The bootstrap script auto-generates an easyconfig file, but it doesn't
>> include checksums (it could I guess, with a bit more work).
>>
>> You can dance around this though, by temporarily disabling the checksums
>> requirement, only for the bootstrap:
>>
>> EASYBUILD_DISABLE_ENFORCE_CHECKSUMS=1 python bootstrap 
>>
>>
>> If you want to, you can bootstrap EasyBuild to a temporary location (for
>> example /tmp/$USER), and then use that installation to pull in an
>> easyconfig file for EasyBuild v4.0.1 that does have checksums, using "eb
>> --from-pr 9149 --installpath " (which pulls in the
>> easyconfig from
>> https://github.com/easybuilders/easybuild-easyconfigs/pull/9149/files)
>> for the actual installation you intend to use.
>>
>> >
>> >
>> > 2. I changed the robo and installation path to be different, but I keep
>> > getting "Failed to determine minimal toolchain for dep ..." errors. The
>> > eb --show-config output is at the end of this e-mail, and shows that my
>> > robot-paths is
>> >
>> /groups/umcg-biogen/tmp01/apps/easybuild-easyconfigs/easybuild/easyconfigs/,
>>
>> > which is where I cloned the easyconfigs repository.
>>
>> What happens if you use "eb --search" with this setup, can 'eb' find
>> easyconfig files?
>>
>> Try something like "eb --search tensorflow-1.14" for example.
>>
>>
>> >
>> > 3. Even if the robo path wasn't working, tools that are already in my
>> > module avail are not being picked up. For example, EasyBuild that is
>> not
>> > working uses "toolchain = {'name': 'foss', 'version': '2018b'}" and has
>> > as one of the dependencies "('ncurses', '6.0'),". When I try to install
>> > it I get
>> >
>> > ERROR: Failed to process easyconfig
>> >
>> /groups/umcg-biogen/tmp01/tools/brain_eQTL/EasyBuild/easyconfigs/SAMtools-1.5-foss-2018b.eb:
>>
>> > Failed to determine minimal toolchain for dep ncurses 6.0
>> >
>> > but when I do ml avail ncur I get:
>> >
>> > --
>> > /groups/umcg-biogen/tmp01/apps/modules/devel
>> > ---
>> > ncurses/6.0-foss-2018b
>> >
>> > ---
>> > /apps/modules/devel
>> > ---
>> > ncurses/6.0ncurses/6.1-foss-2018bncurses/6.1-GCCcore-7.3.0
>> (D)
>> >
>> > So the ncurses module is already available
>> >
>> >
>> > Am I missing something in my configuration?
>>
>> This is a consequence of having --minimal-toolchains enabled, which
>> requires that EasyBuild is able to find the easyconfig

Re: [easybuild] Can't get local install to work

2019-11-18 Thread Kenneth Hoste

On 18/11/2019 21:50, Niek de Klein wrote:

Hi Kenneth,

I spoke too soon, some of my installations work but not all. When I try 
to isntall Autoconf-2.69-foss-2018.eb I get

ERROR: Failed to process easyconfig 
/groups/umcg-biogen/tmp01/apps/easybuild-easyconfigs/easybuild/easyconfigs/a/Autoconf/Autoconf-2.69-foss-2018b.eb: 
Failed to determine minimal toolchain for dep M4 1.4.17

with eb --search I get:

eb --search M4-1.4.17-foss-2018b
  * 
/groups/umcg-biogen/tmp01/apps/easybuild-easyconfigs/easybuild/easyconfigs/m/M4/M4-1.4.17-foss-2018b.eb

eb --search ^foss-2018b.eb
  * 
/groups/umcg-biogen/tmp01/apps/easybuild-easyconfigs/easybuild/easyconfigs/f/foss/foss-2018b.eb

So it can find both the M4-1.4.17-foss-2018b.eb file and the 
foss-2018.eb file.

Can you try "eb M4-1.4.17-foss-2018b.eb -D"?

Kind regards,
Niek

On Mon, Nov 18, 2019 at 3:21 PM Niek de Klein > wrote:

Hi Kenneth,

Thanks for your help!

1. is solved now with your suggestion of
EASYBUILD_DISABLE_ENFORCE_CHECKSUMS=1

Now that the bootstrap.py ran correctly it can also find the
easybuild files in the location that I have configured, so this
solves 1 and 2 as well.

Thanks,
Niek

On Mon, Nov 18, 2019 at 3:01 PM Kenneth Hoste
mailto:kenneth.ho...@ugent.be>> wrote:

Dear Niek,

Sorry that you're hitting these issues, let's try and get them
sorted...

On 18/11/2019 20:00, Niek de Klein wrote:
 > Hi all,
 >
 > I am trying to have my own installation of EasyBuild, but I
have 3 problems:
 >
 > 1. When I try to run bootstrap.py it gives an error that
md5sums are
 > missing, but I'm not sure where to add them as I thought that
should
 > then have been included in the EasyBuild-4.0.1.eb file:
 >
 >      easybuild.tools.build_log.EasyBuildError: "Build of
 > /tmp/tmpfcUSAH/EasyBuild-4.0.1.eb failed (err: 'build failed
(first 300
 > chars): Missing checksum for easybuild-framework-4.0.1.tar.gz')"

Sorry for not responding to the GitHub issue [1] you opened on this
(will do so now with the answer below).

We still need to clean up the bootstrap script a bit after the
EasyBuild
v4.0 release, it's still doing a bunch of stuff it shouldn't
anymore,
and it's not compatible yet with Python 3 (see also
https://github.com/easybuilders/easybuild-framework/pull/3028).

Your active EasyBuild configuration has --enforce-checksums
enabled (via
$EASYBUILD_ENFORCE_CHECKSUMS), which explains the problem you're
seeing.

The bootstrap script auto-generates an easyconfig file, but it
doesn't
include checksums (it could I guess, with a bit more work).

You can dance around this though, by temporarily disabling the
checksums
requirement, only for the bootstrap:

EASYBUILD_DISABLE_ENFORCE_CHECKSUMS=1 python bootstrap 

If you want to, you can bootstrap EasyBuild to a temporary
location (for
example /tmp/$USER), and then use that installation to pull in an
easyconfig file for EasyBuild v4.0.1 that does have checksums,
using "eb
--from-pr 9149 --installpath " (which pulls in the
easyconfig from
https://github.com/easybuilders/easybuild-easyconfigs/pull/9149/files)

for the actual installation you intend to use.

 >
 >
 > 2. I changed the robo and installation path to be different,
but I keep
 > getting "Failed to determine minimal toolchain for dep ..."
errors. The
 > eb --show-config output is at the end of this e-mail, and
shows that my
 > robot-paths is
 >

/groups/umcg-biogen/tmp01/apps/easybuild-easyconfigs/easybuild/easyconfigs/,

 > which is where I cloned the easyconfigs repository.

What happens if you use "eb --search" with this setup, can 'eb'
find
easyconfig files?

Try something like "eb --search tensorflow-1.14" for example.

 >
 > 3. Even if the robo path wasn't working, tools that are
already in my
 > module avail are not being picked up. For example, EasyBuild
that is not
 > working uses "toolchain = {'name': 'foss', 'version':
'2018b'}" and has
 > as one of the dependencies "('ncurses', '6.0'),". When I try
to install
 > it I get
 >
 > ERROR: Failed to process easyconfig
 >

/groups/umcg-biogen/tmp01/tools/brain_eQTL/EasyBuild/easyconfigs/SAMtools-1.5-foss-2018b.eb:

 > Failed to determine minimal toolchain for dep ncurses 6.0
 >
 > but when I do ml avail ncur I get:
 >
 >

Re: [easybuild] Can't get local install to work

2019-11-18 Thread Niek de Klein

Hi Kenneth,

Thanks for your help!

1. is solved now with your suggestion of
EASYBUILD_DISABLE_ENFORCE_CHECKSUMS=1

Now that the bootstrap.py ran correctly it can also find the easybuild
files in the location that I have configured, so this solves 1 and 2 as
well.

Thanks,
Niek

On Mon, Nov 18, 2019 at 3:01 PM Kenneth Hoste 
wrote:

> Dear Niek,
>
> Sorry that you're hitting these issues, let's try and get them sorted...
>
> On 18/11/2019 20:00, Niek de Klein wrote:
> > Hi all,
> >
> > I am trying to have my own installation of EasyBuild, but I have 3
> problems:
> >
> > 1. When I try to run bootstrap.py it gives an error that md5sums are
> > missing, but I'm not sure where to add them as I thought that should
> > then have been included in the EasyBuild-4.0.1.eb file:
> >
> >  easybuild.tools.build_log.EasyBuildError: "Build of
> > /tmp/tmpfcUSAH/EasyBuild-4.0.1.eb failed (err: 'build failed (first 300
> > chars): Missing checksum for easybuild-framework-4.0.1.tar.gz')"
>
> Sorry for not responding to the GitHub issue [1] you opened on this
> (will do so now with the answer below).
>
> We still need to clean up the bootstrap script a bit after the EasyBuild
> v4.0 release, it's still doing a bunch of stuff it shouldn't anymore,
> and it's not compatible yet with Python 3 (see also
> https://github.com/easybuilders/easybuild-framework/pull/3028).
>
>
> Your active EasyBuild configuration has --enforce-checksums enabled (via
> $EASYBUILD_ENFORCE_CHECKSUMS), which explains the problem you're seeing.
>
> The bootstrap script auto-generates an easyconfig file, but it doesn't
> include checksums (it could I guess, with a bit more work).
>
> You can dance around this though, by temporarily disabling the checksums
> requirement, only for the bootstrap:
>
> EASYBUILD_DISABLE_ENFORCE_CHECKSUMS=1 python bootstrap 
>
>
> If you want to, you can bootstrap EasyBuild to a temporary location (for
> example /tmp/$USER), and then use that installation to pull in an
> easyconfig file for EasyBuild v4.0.1 that does have checksums, using "eb
> --from-pr 9149 --installpath " (which pulls in the
> easyconfig from
> https://github.com/easybuilders/easybuild-easyconfigs/pull/9149/files)
> for the actual installation you intend to use.
>
> >
> >
> > 2. I changed the robo and installation path to be different, but I keep
> > getting "Failed to determine minimal toolchain for dep ..." errors. The
> > eb --show-config output is at the end of this e-mail, and shows that my
> > robot-paths is
> >
> /groups/umcg-biogen/tmp01/apps/easybuild-easyconfigs/easybuild/easyconfigs/,
>
> > which is where I cloned the easyconfigs repository.
>
> What happens if you use "eb --search" with this setup, can 'eb' find
> easyconfig files?
>
> Try something like "eb --search tensorflow-1.14" for example.
>
>
> >
> > 3. Even if the robo path wasn't working, tools that are already in my
> > module avail are not being picked up. For example, EasyBuild that is not
> > working uses "toolchain = {'name': 'foss', 'version': '2018b'}" and has
> > as one of the dependencies "('ncurses', '6.0'),". When I try to install
> > it I get
> >
> > ERROR: Failed to process easyconfig
> >
> /groups/umcg-biogen/tmp01/tools/brain_eQTL/EasyBuild/easyconfigs/SAMtools-1.5-foss-2018b.eb:
>
> > Failed to determine minimal toolchain for dep ncurses 6.0
> >
> > but when I do ml avail ncur I get:
> >
> > --
> > /groups/umcg-biogen/tmp01/apps/modules/devel
> > ---
> > ncurses/6.0-foss-2018b
> >
> > ---
> > /apps/modules/devel
> > ---
> > ncurses/6.0ncurses/6.1-foss-2018bncurses/6.1-GCCcore-7.3.0
> (D)
> >
> > So the ncurses module is already available
> >
> >
> > Am I missing something in my configuration?
>
> This is a consequence of having --minimal-toolchains enabled, which
> requires that EasyBuild is able to find the easyconfig file for the
> toolchain (foss-2018b.eb in this case).
>
> It needs to figure out what the (versions of) the subtoolchains are for
> foss/2018b, so it can check for ncurses installed with GCCcore first,
> before gompi, before foss.
>
> So this is basically the same issue as with problem 2, I bet "eb
> --search ^foss-2018b.eb" doesn't give you any result?
>
>
> Are you 100% sure the path you specified for --robot-paths is correct?
>
> Note that you can also prepend or append to the robot search path
> (rather than override the default), see
>
> https://easybuild.readthedocs.io/en/latest/Using_the_EasyBuild_command_line.html#controlling-robot-search-path
> .
>
> I hope this helps...
>
>
> regards,
>
> Kenneth
>
>
> [1] https://github.com/easybuilders/easybuild-framework/issues/3079
>
> >
> > Thanks for your help!
> > Niek de Klein
> >
> >
> > #
> > # Current EasyBuild configuration
> > # (C: command line

Re: [easybuild] TensorFlow/2.0.0 for foss-2019a-Python-3.7.2 fail with ssl error.

Re: [easybuild] Can't get local install to work

Re: [easybuild] Can't get local install to work

Re: [easybuild] Can't get local install to work

Re: [easybuild] Can't get local install to work

5 matches

Site Navigation

Mail list logo

Footer information