Re: [easybuild] Bazel-0.26.1-GCCcore-8.3.0.eb fails - zip missing
Dear Kenneth,
Done.
Regards
Loris
Kenneth Hoste writes:
> Dear Loris,
>
> Please open an issue on this, since we can actually avoid that you need to
> have
> 'zip' installed in the OS.
>
> We can include 'zip' as a build dependency for Bazel, after creating an
> easyconfig file for Zip (see http://infozip.sourceforge.net/Zip.html).
> We already have easyconfigs for UnZip (which is apparently a separate
> project...).
>
>
> regards,
>
> Kenneth
>
> On 26/02/2020 09:59, Loris Bennett wrote:
>> Hi Terje,
>>
>> Yes, thank you, 'osdependencies' was what I was thinking of. Strangely,
>> searching for 'dependencies' on https://easybuild.readthedocs.io/
>> doesn't return any results for 'osdependencies'.
>>
>> Cheers,
>>
>> Loris
>>
>> Terje Kvernes writes:
>>
>>> Hi Loris,
>>>
>>> In easyconfigs, “osdependencies” is used to designate dependencies that the
>>> OS has to provide. This is typically seen used something like this:
>>>
>>> # OpenSSL is required
>>> osdependencies = ['openssl', ('openssl-devel', 'libssl-dev',
>>> 'libopenssl-devel’)]
>>>
>>> These are package names that need to be installed on the system, openssl
>>> and one
>>> of the devel packages listed — as their name varies depending on the
>>> flavour of
>>> your linux. This can be overridden with the command line parameter
>>> --ignore-osdeps.
>>>
>>> I am not aware of a way to specify a required binary though, but specifying
>>> a package providing “zip” would probably be a good idea.
>>>
On 26 Feb 2020, at 09:14, Loris Bennett wrote:
Hi,
So the admin node on which I normally build software does have 'zip',
but as I was building TensorFlow with GPU support, I was doing this on a
GPU node, which didn't have 'zip' installed :-/
I had a vague idea that external dependencies could be specified in an
easyconfig, but I could only find some thing on external modules. Maybe
I was thinking of '--filter-deps', but that's sort of the opposite of
what one would need.
Thanks for the pointer,
Cheers,
Loris
Lars Viklund writes:
> For reference, EB documents the shell tools (including `unzip`) that may
> be
> occasionally needed by easyconfigs:
>
> https://easybuild.readthedocs.io/en/latest/Installation.html#required-shell-tools
>
> Sounds like Bazel goes a bit beyond this and also assumes that the
> environment
> has the ability to zip things. Lovely :D
>
> // Lars
>
> On 2/25/20 5:06 PM, Terje Kvernes wrote:
>> Hi Loris,
>>
>> From the output it would seem that zip is an implied system dependency
>> in
>> Bazel. Depending on the underlying distribution, the easiest solution is
>> probably to install zip as a system package via yum/dnf/apt-get/etc.
>>
>> I took a quick look at the easyconfig tree and didn’t find a suitable
>> zip to
>> add as a dependency, so I’m guessing this has gone under the radar as
>> “everyone” already had zip installed.
>>
>> I’m not sure what the long-term solution is here, as I’m not sure one
>> wants to
>> move zip from a system dependency to a toolchain one, but opening an
>> issue on
>> https://github.com/easybuilders/easybuild-easyconfigs/issues is probably
>> a
>> good idea. :-)
>>
>>> On 25 Feb 2020, at 16:53, Loris Bennett
>>> wrote:
>>>
>>> Hi,
>>>
>>> Bazel-0.26.1-GCCcore-8.3.0.eb is failing with the following error:
>>>
>>> ^MESC[1AESC[KESC[31mESC[1mERROR:
>>> ESC[0m/trinity/shared/easybuild/build/Bazel/0.26.1/GCCcore-8.3.0/src/BUILD:339:2:
>>> Executing genrule //src:package-zip_nojdk failed (Exit 127): bash
>>> failed: error executing command
>>> (cd /tmp/eb-0CJXT7/bazel_M82PHrmR/out/execroot/io_bazel && \
>>> exec env - \
>>>
>>> LD_LIBRARY_PATH=/trinity/shared/easybuild/software/Java/1.8.0_192/lib:/trinity/shared/easybuild/software/Python/3.7.4-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/libffi/3.2.1-GCCcore-8.3.0/lib64:/trinity/shared/easybuild/software/libffi/3.2.1-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/GMP/6.1.2-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/XZ/5.2.4-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/SQLite/3.29.0-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/Tcl/8.6.9-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/libreadline/8.0-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/ncurses/6.1-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/bzip2/1.0.8-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/binutils/2.32-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/zlib/1.2.11-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/GCCcore/8.3.0/lib/gcc/x86_64-pc-linux-gnu/8.3.0:/trinity/shared/easybuild/software/GCCcore/8.3.0/lib64:/trinity/shared/easybuild/software/GCCcore/8.3.0/lib
>>> \
>>>
Re: [easybuild] Bazel-0.26.1-GCCcore-8.3.0.eb fails - zip missing
Dear Loris,
Please open an issue on this, since we can actually avoid that you need
to have 'zip' installed in the OS.
We can include 'zip' as a build dependency for Bazel, after creating an
easyconfig file for Zip (see http://infozip.sourceforge.net/Zip.html).
We already have easyconfigs for UnZip (which is apparently a separate
project...).
regards,
Kenneth
On 26/02/2020 09:59, Loris Bennett wrote:
Hi Terje,
Yes, thank you, 'osdependencies' was what I was thinking of. Strangely,
searching for 'dependencies' on https://easybuild.readthedocs.io/
doesn't return any results for 'osdependencies'.
Cheers,
Loris
Terje Kvernes writes:
Hi Loris,
In easyconfigs, “osdependencies” is used to designate dependencies that the OS
has to provide. This is typically seen used something like this:
# OpenSSL is required
osdependencies = ['openssl', ('openssl-devel', 'libssl-dev',
'libopenssl-devel’)]
These are package names that need to be installed on the system, openssl and one
of the devel packages listed — as their name varies depending on the flavour of
your linux. This can be overridden with the command line parameter
--ignore-osdeps.
I am not aware of a way to specify a required binary though, but specifying a
package providing “zip” would probably be a good idea.
On 26 Feb 2020, at 09:14, Loris Bennett wrote:
Hi,
So the admin node on which I normally build software does have 'zip',
but as I was building TensorFlow with GPU support, I was doing this on a
GPU node, which didn't have 'zip' installed :-/
I had a vague idea that external dependencies could be specified in an
easyconfig, but I could only find some thing on external modules. Maybe
I was thinking of '--filter-deps', but that's sort of the opposite of
what one would need.
Thanks for the pointer,
Cheers,
Loris
Lars Viklund writes:
For reference, EB documents the shell tools (including `unzip`) that may be
occasionally needed by easyconfigs:
https://easybuild.readthedocs.io/en/latest/Installation.html#required-shell-tools
Sounds like Bazel goes a bit beyond this and also assumes that the environment
has the ability to zip things. Lovely :D
// Lars
On 2/25/20 5:06 PM, Terje Kvernes wrote:
Hi Loris,
From the output it would seem that zip is an implied system dependency in
Bazel. Depending on the underlying distribution, the easiest solution is
probably to install zip as a system package via yum/dnf/apt-get/etc.
I took a quick look at the easyconfig tree and didn’t find a suitable zip to
add as a dependency, so I’m guessing this has gone under the radar as
“everyone” already had zip installed.
I’m not sure what the long-term solution is here, as I’m not sure one wants to
move zip from a system dependency to a toolchain one, but opening an issue on
https://github.com/easybuilders/easybuild-easyconfigs/issues is probably a
good idea. :-)
On 25 Feb 2020, at 16:53, Loris Bennett wrote:
Hi,
Bazel-0.26.1-GCCcore-8.3.0.eb is failing with the following error:
^MESC[1AESC[KESC[31mESC[1mERROR:
ESC[0m/trinity/shared/easybuild/build/Bazel/0.26.1/GCCcore-8.3.0/src/BUILD:339:2:
Executing genrule //src:package-zip_nojdk failed (Exit 127): bash failed:
error executing command
(cd /tmp/eb-0CJXT7/bazel_M82PHrmR/out/execroot/io_bazel && \
exec env - \
LD_LIBRARY_PATH=/trinity/shared/easybuild/software/Java/1.8.0_192/lib:/trinity/shared/easybuild/software/Python/3.7.4-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/libffi/3.2.1-GCCcore-8.3.0/lib64:/trinity/shared/easybuild/software/libffi/3.2.1-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/GMP/6.1.2-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/XZ/5.2.4-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/SQLite/3.29.0-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/Tcl/8.6.9-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/libreadline/8.0-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/ncurses/6.1-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/bzip2/1.0.8-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/binutils/2.32-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/zlib/1.2.11-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/GCCcore/8.3.0/lib/gcc/x86_64-pc-linux-gnu/8.3.0:/trinity/shared/easybuild/software/GCCcore/8.3.0/lib64:/trinity/shared/easybuild/software/GCCcore/8.3.0/lib
\
Re: [easybuild] Bazel-0.26.1-GCCcore-8.3.0.eb fails - zip missing
Hi Terje,
Yes, thank you, 'osdependencies' was what I was thinking of. Strangely,
searching for 'dependencies' on https://easybuild.readthedocs.io/
doesn't return any results for 'osdependencies'.
Cheers,
Loris
Terje Kvernes writes:
> Hi Loris,
>
> In easyconfigs, “osdependencies” is used to designate dependencies that the
> OS has to provide. This is typically seen used something like this:
>
> # OpenSSL is required
> osdependencies = ['openssl', ('openssl-devel', 'libssl-dev',
> 'libopenssl-devel’)]
>
> These are package names that need to be installed on the system, openssl and
> one
> of the devel packages listed — as their name varies depending on the flavour
> of
> your linux. This can be overridden with the command line parameter
> --ignore-osdeps.
>
> I am not aware of a way to specify a required binary though, but specifying a
> package providing “zip” would probably be a good idea.
>
>> On 26 Feb 2020, at 09:14, Loris Bennett wrote:
>>
>> Hi,
>>
>> So the admin node on which I normally build software does have 'zip',
>> but as I was building TensorFlow with GPU support, I was doing this on a
>> GPU node, which didn't have 'zip' installed :-/
>>
>> I had a vague idea that external dependencies could be specified in an
>> easyconfig, but I could only find some thing on external modules. Maybe
>> I was thinking of '--filter-deps', but that's sort of the opposite of
>> what one would need.
>>
>> Thanks for the pointer,
>>
>> Cheers,
>>
>> Loris
>>
>> Lars Viklund writes:
>>
>>> For reference, EB documents the shell tools (including `unzip`) that may be
>>> occasionally needed by easyconfigs:
>>>
>>> https://easybuild.readthedocs.io/en/latest/Installation.html#required-shell-tools
>>>
>>> Sounds like Bazel goes a bit beyond this and also assumes that the
>>> environment
>>> has the ability to zip things. Lovely :D
>>>
>>> // Lars
>>>
>>> On 2/25/20 5:06 PM, Terje Kvernes wrote:
Hi Loris,
From the output it would seem that zip is an implied system dependency in
Bazel. Depending on the underlying distribution, the easiest solution is
probably to install zip as a system package via yum/dnf/apt-get/etc.
I took a quick look at the easyconfig tree and didn’t find a suitable zip
to
add as a dependency, so I’m guessing this has gone under the radar as
“everyone” already had zip installed.
I’m not sure what the long-term solution is here, as I’m not sure one
wants to
move zip from a system dependency to a toolchain one, but opening an issue
on
https://github.com/easybuilders/easybuild-easyconfigs/issues is probably a
good idea. :-)
> On 25 Feb 2020, at 16:53, Loris Bennett
> wrote:
>
> Hi,
>
> Bazel-0.26.1-GCCcore-8.3.0.eb is failing with the following error:
>
> ^MESC[1AESC[KESC[31mESC[1mERROR:
> ESC[0m/trinity/shared/easybuild/build/Bazel/0.26.1/GCCcore-8.3.0/src/BUILD:339:2:
> Executing genrule //src:package-zip_nojdk failed (Exit 127): bash
> failed: error executing command
>(cd /tmp/eb-0CJXT7/bazel_M82PHrmR/out/execroot/io_bazel && \
>exec env - \
>
> LD_LIBRARY_PATH=/trinity/shared/easybuild/software/Java/1.8.0_192/lib:/trinity/shared/easybuild/software/Python/3.7.4-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/libffi/3.2.1-GCCcore-8.3.0/lib64:/trinity/shared/easybuild/software/libffi/3.2.1-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/GMP/6.1.2-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/XZ/5.2.4-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/SQLite/3.29.0-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/Tcl/8.6.9-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/libreadline/8.0-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/ncurses/6.1-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/bzip2/1.0.8-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/binutils/2.32-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/zlib/1.2.11-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/GCCcore/8.3.0/lib/gcc/x86_64-pc-linux-gnu/8.3.0:/trinity/shared/easybuild/software/GCCcore/8.3.0/lib64:/trinity/shared/easybuild/software/GCCcore/8.3.0/lib
> \
>
>
Re: [easybuild] Bazel-0.26.1-GCCcore-8.3.0.eb fails - zip missing
Hi Loris,
In easyconfigs, “osdependencies” is used to designate dependencies that the OS
has to provide. This is typically seen used something like this:
# OpenSSL is required
osdependencies = ['openssl', ('openssl-devel', 'libssl-dev',
'libopenssl-devel’)]
These are package names that need to be installed on the system, openssl and
one of the devel packages listed — as their name varies depending on the
flavour of your linux. This can be overridden with the command line parameter
--ignore-osdeps.
I am not aware of a way to specify a required binary though, but specifying a
package providing “zip” would probably be a good idea.
> On 26 Feb 2020, at 09:14, Loris Bennett wrote:
>
> Hi,
>
> So the admin node on which I normally build software does have 'zip',
> but as I was building TensorFlow with GPU support, I was doing this on a
> GPU node, which didn't have 'zip' installed :-/
>
> I had a vague idea that external dependencies could be specified in an
> easyconfig, but I could only find some thing on external modules. Maybe
> I was thinking of '--filter-deps', but that's sort of the opposite of
> what one would need.
>
> Thanks for the pointer,
>
> Cheers,
>
> Loris
>
> Lars Viklund writes:
>
>> For reference, EB documents the shell tools (including `unzip`) that may be
>> occasionally needed by easyconfigs:
>>
>> https://easybuild.readthedocs.io/en/latest/Installation.html#required-shell-tools
>>
>> Sounds like Bazel goes a bit beyond this and also assumes that the
>> environment
>> has the ability to zip things. Lovely :D
>>
>> // Lars
>>
>> On 2/25/20 5:06 PM, Terje Kvernes wrote:
>>> Hi Loris,
>>>
>>> From the output it would seem that zip is an implied system dependency in
>>> Bazel. Depending on the underlying distribution, the easiest solution is
>>> probably to install zip as a system package via yum/dnf/apt-get/etc.
>>>
>>> I took a quick look at the easyconfig tree and didn’t find a suitable zip to
>>> add as a dependency, so I’m guessing this has gone under the radar as
>>> “everyone” already had zip installed.
>>>
>>> I’m not sure what the long-term solution is here, as I’m not sure one wants
>>> to
>>> move zip from a system dependency to a toolchain one, but opening an issue
>>> on
>>> https://github.com/easybuilders/easybuild-easyconfigs/issues is probably a
>>> good idea. :-)
>>>
On 25 Feb 2020, at 16:53, Loris Bennett wrote:
Hi,
Bazel-0.26.1-GCCcore-8.3.0.eb is failing with the following error:
^MESC[1AESC[KESC[31mESC[1mERROR:
ESC[0m/trinity/shared/easybuild/build/Bazel/0.26.1/GCCcore-8.3.0/src/BUILD:339:2:
Executing genrule //src:package-zip_nojdk failed (Exit 127): bash failed:
error executing command
(cd /tmp/eb-0CJXT7/bazel_M82PHrmR/out/execroot/io_bazel && \
exec env - \
LD_LIBRARY_PATH=/trinity/shared/easybuild/software/Java/1.8.0_192/lib:/trinity/shared/easybuild/software/Python/3.7.4-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/libffi/3.2.1-GCCcore-8.3.0/lib64:/trinity/shared/easybuild/software/libffi/3.2.1-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/GMP/6.1.2-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/XZ/5.2.4-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/SQLite/3.29.0-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/Tcl/8.6.9-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/libreadline/8.0-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/ncurses/6.1-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/bzip2/1.0.8-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/binutils/2.32-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/zlib/1.2.11-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/GCCcore/8.3.0/lib/gcc/x86_64-pc-linux-gnu/8.3.0:/trinity/shared/easybuild/software/GCCcore/8.3.0/lib64:/trinity/shared/easybuild/software/GCCcore/8.3.0/lib
\
PATH=/trinity/shared/easybuild/software/Java/1.8.0_192:/trinity/shared/easybuild/software/Java/1.8.0_192/bin:/trinity/shared/easybuild/software/Python/3.7.4-GCCcore-8.3.0/bin:/trinity/shared/easybuild/software/XZ/5.2.4-GCCcore-8.3.0/bin:/trinity/shared/easybuild/software/SQLite/3.29.0-GCCcore-8.3.0/bin:/trinity/shared/easybuild/software/Tcl/8.6.9-GCCcore-8.3.0/bin:/trinity/shared/easybuild/software/ncurses/6.1-GCCcore-8.3.0/bin:/trinity/shared/easybuild/software/bzip2/1.0.8-GCCcore-8.3.0/bin:/trinity/shared/easybuild/software/binutils/2.32-GCCcore-8.3.0/bin:/trinity/shared/easybuild/software/GCCcore/8.3.0/bin:/trinity/shared/easybuild/software/EasyBuild/4.1.1/bin:/trinity/shared/zedat/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/bin/syscfg:/trinity/home/build/.local/bin:/trinity/home/build/bin
\
/bin/bash -c 'source
external/bazel_tools/tools/genrule/genrule-setup.sh; src/package-bazel.sh
bazel-out/k8-opt/bin/src/package_nojdk.zip
bazel-out/k8-opt/bin/src/embedded_tools_nojdk.zip
Re: [easybuild] Bazel-0.26.1-GCCcore-8.3.0.eb fails - zip missing
Hi, So the admin node on which I normally build software does have 'zip', but as I was building TensorFlow with GPU support, I was doing this on a GPU node, which didn't have 'zip' installed :-/ I had a vague idea that external dependencies could be specified in an easyconfig, but I could only find some thing on external modules. Maybe I was thinking of '--filter-deps', but that's sort of the opposite of what one would need. Thanks for the pointer, Cheers, Loris Lars Viklund writes: > For reference, EB documents the shell tools (including `unzip`) that may be > occasionally needed by easyconfigs: > > https://easybuild.readthedocs.io/en/latest/Installation.html#required-shell-tools > > Sounds like Bazel goes a bit beyond this and also assumes that the environment > has the ability to zip things. Lovely :D > > // Lars > > On 2/25/20 5:06 PM, Terje Kvernes wrote: >> Hi Loris, >> >> From the output it would seem that zip is an implied system dependency in >> Bazel. Depending on the underlying distribution, the easiest solution is >> probably to install zip as a system package via yum/dnf/apt-get/etc. >> >> I took a quick look at the easyconfig tree and didn’t find a suitable zip to >> add as a dependency, so I’m guessing this has gone under the radar as >> “everyone” already had zip installed. >> >> I’m not sure what the long-term solution is here, as I’m not sure one wants >> to >> move zip from a system dependency to a toolchain one, but opening an issue on >> https://github.com/easybuilders/easybuild-easyconfigs/issues is probably a >> good idea. :-) >> >>> On 25 Feb 2020, at 16:53, Loris Bennett wrote: >>> >>> Hi, >>> >>> Bazel-0.26.1-GCCcore-8.3.0.eb is failing with the following error: >>> >>> ^MESC[1AESC[KESC[31mESC[1mERROR: >>> ESC[0m/trinity/shared/easybuild/build/Bazel/0.26.1/GCCcore-8.3.0/src/BUILD:339:2: >>> Executing genrule //src:package-zip_nojdk failed (Exit 127): bash failed: >>> error executing command >>> (cd /tmp/eb-0CJXT7/bazel_M82PHrmR/out/execroot/io_bazel && \ >>> exec env - \ >>> >>> LD_LIBRARY_PATH=/trinity/shared/easybuild/software/Java/1.8.0_192/lib:/trinity/shared/easybuild/software/Python/3.7.4-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/libffi/3.2.1-GCCcore-8.3.0/lib64:/trinity/shared/easybuild/software/libffi/3.2.1-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/GMP/6.1.2-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/XZ/5.2.4-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/SQLite/3.29.0-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/Tcl/8.6.9-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/libreadline/8.0-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/ncurses/6.1-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/bzip2/1.0.8-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/binutils/2.32-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/zlib/1.2.11-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/GCCcore/8.3.0/lib/gcc/x86_64-pc-linux-gnu/8.3.0:/trinity/shared/easybuild/software/GCCcore/8.3.0/lib64:/trinity/shared/easybuild/software/GCCcore/8.3.0/lib >>> \ >>> >>> PATH=/trinity/shared/easybuild/software/Java/1.8.0_192:/trinity/shared/easybuild/software/Java/1.8.0_192/bin:/trinity/shared/easybuild/software/Python/3.7.4-GCCcore-8.3.0/bin:/trinity/shared/easybuild/software/XZ/5.2.4-GCCcore-8.3.0/bin:/trinity/shared/easybuild/software/SQLite/3.29.0-GCCcore-8.3.0/bin:/trinity/shared/easybuild/software/Tcl/8.6.9-GCCcore-8.3.0/bin:/trinity/shared/easybuild/software/ncurses/6.1-GCCcore-8.3.0/bin:/trinity/shared/easybuild/software/bzip2/1.0.8-GCCcore-8.3.0/bin:/trinity/shared/easybuild/software/binutils/2.32-GCCcore-8.3.0/bin:/trinity/shared/easybuild/software/GCCcore/8.3.0/bin:/trinity/shared/easybuild/software/EasyBuild/4.1.1/bin:/trinity/shared/zedat/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/bin/syscfg:/trinity/home/build/.local/bin:/trinity/home/build/bin >>> \ >>> /bin/bash -c 'source >>> external/bazel_tools/tools/genrule/genrule-setup.sh; src/package-bazel.sh >>> bazel-out/k8-opt/bin/src/package_nojdk.zip >>> bazel-out/k8-opt/bin/src/embedded_tools_nojdk.zip >>> bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar >>> bazel-out/k8-opt/bin/src/install_base_key_nojdk >>> bazel-out/k8-opt/bin/src/main/native/libunix.so >>> bazel-out/k8-opt/bin/src/main/tools/build-runfiles >>> bazel-out/k8-opt/bin/src/main/tools/process-wrapper >>> src/main/tools/jdk.BUILD bazel-out/k8-opt/bin/src/main/tools/linux-sandbox >>> bazel-out/k8-opt/bin/tools/osx/xcode-locator >>> bazel-out/k8-opt/bin/src/main/tools/daemonize') >>> Execution platform: @bazel_tools//platforms:host_platform >>> ESC[32m[1,793 / 1,794]ESC[0m checking cached actions >>> ^MESC[1AESC[Ksrc/package-bazel.sh: line 42: zip: command not found >>> >>> Anyone know what sort of zip package is missing here? >>> >>> Cheers, >>> >>> Loris >>> >>> -- >>> Dr. Loris
Re: [easybuild] Bazel-0.26.1-GCCcore-8.3.0.eb fails - zip missing
For reference, EB documents the shell tools (including `unzip`) that may be occasionally needed by easyconfigs: https://easybuild.readthedocs.io/en/latest/Installation.html#required-shell-tools Sounds like Bazel goes a bit beyond this and also assumes that the environment has the ability to zip things. Lovely :D // Lars On 2/25/20 5:06 PM, Terje Kvernes wrote: Hi Loris, From the output it would seem that zip is an implied system dependency in Bazel. Depending on the underlying distribution, the easiest solution is probably to install zip as a system package via yum/dnf/apt-get/etc. I took a quick look at the easyconfig tree and didn’t find a suitable zip to add as a dependency, so I’m guessing this has gone under the radar as “everyone” already had zip installed. I’m not sure what the long-term solution is here, as I’m not sure one wants to move zip from a system dependency to a toolchain one, but opening an issue on https://github.com/easybuilders/easybuild-easyconfigs/issues is probably a good idea. :-) On 25 Feb 2020, at 16:53, Loris Bennett wrote: Hi, Bazel-0.26.1-GCCcore-8.3.0.eb is failing with the following error: ^MESC[1AESC[KESC[31mESC[1mERROR: ESC[0m/trinity/shared/easybuild/build/Bazel/0.26.1/GCCcore-8.3.0/src/BUILD:339:2: Executing genrule //src:package-zip_nojdk failed (Exit 127): bash failed: error executing command (cd /tmp/eb-0CJXT7/bazel_M82PHrmR/out/execroot/io_bazel && \ exec env - \ LD_LIBRARY_PATH=/trinity/shared/easybuild/software/Java/1.8.0_192/lib:/trinity/shared/easybuild/software/Python/3.7.4-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/libffi/3.2.1-GCCcore-8.3.0/lib64:/trinity/shared/easybuild/software/libffi/3.2.1-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/GMP/6.1.2-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/XZ/5.2.4-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/SQLite/3.29.0-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/Tcl/8.6.9-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/libreadline/8.0-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/ncurses/6.1-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/bzip2/1.0.8-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/binutils/2.32-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/zlib/1.2.11-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/GCCcore/8.3.0/lib/gcc/x86_64-pc-linux-gnu/8.3.0:/trinity/shared/easybuild/software/GCCcore/8.3.0/lib64:/trinity/shared/easybuild/software/GCCcore/8.3.0/lib \ PATH=/trinity/shared/easybuild/software/Java/1.8.0_192:/trinity/shared/easybuild/software/Java/1.8.0_192/bin:/trinity/shared/easybuild/software/Python/3.7.4-GCCcore-8.3.0/bin:/trinity/shared/easybuild/software/XZ/5.2.4-GCCcore-8.3.0/bin:/trinity/shared/easybuild/software/SQLite/3.29.0-GCCcore-8.3.0/bin:/trinity/shared/easybuild/software/Tcl/8.6.9-GCCcore-8.3.0/bin:/trinity/shared/easybuild/software/ncurses/6.1-GCCcore-8.3.0/bin:/trinity/shared/easybuild/software/bzip2/1.0.8-GCCcore-8.3.0/bin:/trinity/shared/easybuild/software/binutils/2.32-GCCcore-8.3.0/bin:/trinity/shared/easybuild/software/GCCcore/8.3.0/bin:/trinity/shared/easybuild/software/EasyBuild/4.1.1/bin:/trinity/shared/zedat/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/bin/syscfg:/trinity/home/build/.local/bin:/trinity/home/build/bin \ /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; src/package-bazel.sh bazel-out/k8-opt/bin/src/package_nojdk.zip bazel-out/k8-opt/bin/src/embedded_tools_nojdk.zip bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar bazel-out/k8-opt/bin/src/install_base_key_nojdk bazel-out/k8-opt/bin/src/main/native/libunix.so bazel-out/k8-opt/bin/src/main/tools/build-runfiles bazel-out/k8-opt/bin/src/main/tools/process-wrapper src/main/tools/jdk.BUILD bazel-out/k8-opt/bin/src/main/tools/linux-sandbox bazel-out/k8-opt/bin/tools/osx/xcode-locator bazel-out/k8-opt/bin/src/main/tools/daemonize') Execution platform: @bazel_tools//platforms:host_platform ESC[32m[1,793 / 1,794]ESC[0m checking cached actions ^MESC[1AESC[Ksrc/package-bazel.sh: line 42: zip: command not found Anyone know what sort of zip package is missing here? Cheers, Loris -- Dr. Loris Bennett (Mr.) ZEDAT, Freie Universität Berlin Email loris.benn...@fu-berlin.de
Re: [easybuild] Bazel-0.26.1-GCCcore-8.3.0.eb fails - zip missing
Hi Loris, From the output it would seem that zip is an implied system dependency in Bazel. Depending on the underlying distribution, the easiest solution is probably to install zip as a system package via yum/dnf/apt-get/etc. I took a quick look at the easyconfig tree and didn’t find a suitable zip to add as a dependency, so I’m guessing this has gone under the radar as “everyone” already had zip installed. I’m not sure what the long-term solution is here, as I’m not sure one wants to move zip from a system dependency to a toolchain one, but opening an issue on https://github.com/easybuilders/easybuild-easyconfigs/issues is probably a good idea. :-) > On 25 Feb 2020, at 16:53, Loris Bennett wrote: > > Hi, > > Bazel-0.26.1-GCCcore-8.3.0.eb is failing with the following error: > > ^MESC[1AESC[KESC[31mESC[1mERROR: > ESC[0m/trinity/shared/easybuild/build/Bazel/0.26.1/GCCcore-8.3.0/src/BUILD:339:2: > Executing genrule //src:package-zip_nojdk failed (Exit 127): bash failed: > error executing command >(cd /tmp/eb-0CJXT7/bazel_M82PHrmR/out/execroot/io_bazel && \ >exec env - \ > > LD_LIBRARY_PATH=/trinity/shared/easybuild/software/Java/1.8.0_192/lib:/trinity/shared/easybuild/software/Python/3.7.4-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/libffi/3.2.1-GCCcore-8.3.0/lib64:/trinity/shared/easybuild/software/libffi/3.2.1-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/GMP/6.1.2-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/XZ/5.2.4-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/SQLite/3.29.0-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/Tcl/8.6.9-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/libreadline/8.0-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/ncurses/6.1-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/bzip2/1.0.8-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/binutils/2.32-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/zlib/1.2.11-GCCcore-8.3.0/lib:/trinity/shared/easybuild/software/GCCcore/8.3.0/lib/gcc/x86_64-pc-linux-gnu/8.3.0:/trinity/shared/easybuild/software/GCCcore/8.3.0/lib64:/trinity/shared/easybuild/software/GCCcore/8.3.0/lib > \ > > PATH=/trinity/shared/easybuild/software/Java/1.8.0_192:/trinity/shared/easybuild/software/Java/1.8.0_192/bin:/trinity/shared/easybuild/software/Python/3.7.4-GCCcore-8.3.0/bin:/trinity/shared/easybuild/software/XZ/5.2.4-GCCcore-8.3.0/bin:/trinity/shared/easybuild/software/SQLite/3.29.0-GCCcore-8.3.0/bin:/trinity/shared/easybuild/software/Tcl/8.6.9-GCCcore-8.3.0/bin:/trinity/shared/easybuild/software/ncurses/6.1-GCCcore-8.3.0/bin:/trinity/shared/easybuild/software/bzip2/1.0.8-GCCcore-8.3.0/bin:/trinity/shared/easybuild/software/binutils/2.32-GCCcore-8.3.0/bin:/trinity/shared/easybuild/software/GCCcore/8.3.0/bin:/trinity/shared/easybuild/software/EasyBuild/4.1.1/bin:/trinity/shared/zedat/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/bin/syscfg:/trinity/home/build/.local/bin:/trinity/home/build/bin > \ >/bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; > src/package-bazel.sh bazel-out/k8-opt/bin/src/package_nojdk.zip > bazel-out/k8-opt/bin/src/embedded_tools_nojdk.zip > bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar > bazel-out/k8-opt/bin/src/install_base_key_nojdk > bazel-out/k8-opt/bin/src/main/native/libunix.so > bazel-out/k8-opt/bin/src/main/tools/build-runfiles > bazel-out/k8-opt/bin/src/main/tools/process-wrapper src/main/tools/jdk.BUILD > bazel-out/k8-opt/bin/src/main/tools/linux-sandbox > bazel-out/k8-opt/bin/tools/osx/xcode-locator > bazel-out/k8-opt/bin/src/main/tools/daemonize') > Execution platform: @bazel_tools//platforms:host_platform > ESC[32m[1,793 / 1,794]ESC[0m checking cached actions > ^MESC[1AESC[Ksrc/package-bazel.sh: line 42: zip: command not found > > Anyone know what sort of zip package is missing here? > > Cheers, > > Loris > > -- > Dr. Loris Bennett (Mr.) > ZEDAT, Freie Universität Berlin Email loris.benn...@fu-berlin.de -- Terje Kvernes Head sysadmin Department of Mathematics University of Oslo
Re: [easybuild] Bazel-0.12.0-GCCcore-6.4.0 build issues
Hi Kenneth, Thanks, I overlooked the part that it would be not completely clear *which* EBROOT you should check... For the dummy toolchain, I wonder if there would even be a problem. I believe my problem comes from the fact that Bazel does some part of its installation with my GCCcore-6.4.0 libstdc++, then does a reset of the environment halfway (or at least I read something like this?) and then continues with my system libstdc++. Since my system libstdc++ only has older GLIBCXX versions, that fails. But, if you'd build with the dummy toolchain, wouldn't you in both cases you the same libstdc++ anyway - namely the system version? In that case, I'd not expect any problems to begin with, so do you even need to do those regex replacements for the dummy toolchain or can you just use the Bazel installscript 'as is'? Anyway, I have put this in as an issue for the EasyBlock and we can take it from there to look for a proper solution (https://github.com/easybuilders/easybuild-easyblocks/issues/1491). Cheers, Caspar - Original Message - From: "Kenneth Hoste" To: "easybuild" , "Caspar van Leeuwen" Sent: Friday, 31 August, 2018 08:41:59 Subject: Re: [easybuild] Bazel-0.12.0-GCCcore-6.4.0 build issues Hi Caspar, On 30/08/2018 21:31, Caspar van Leeuwen wrote: > Dear all, > > I get an error when building Bazel-0.12.0: > > bazel-out/host/bin/external/io_bazel/third_party/protobuf/3.4.0/js_embed: > /usr/lib64/libstdc++.so.6: > version `GLIBCXX_3.4.21' not found (required by > bazel-out/host/bin/external/io_bazel/third_party/protobuf/3.4.0/js_embed) > > As you can see, it picks up the system libstdc++, rather than the > GCCcore-6.4.0 one. > > The Bazel EasyBlock seems to search for the location of five tools: ar, > cpp, dwp, gcc and ld. > > == 2018-08-30 13:28:35,003 easyblock.py:2536 INFO Running method > configure_step part of step configure > == 2018-08-30 13:28:35,023 filetools.py:343 INFO Command ar found at > /hpc/sw_test/EB_regression/software/binutils/2.28-GCCcore-6.4.0/bin/ar > == 2018-08-30 13:28:35,025 filetools.py:343 INFO Command cpp found at > /hpc/sw_test/EB_regression/software/GCCcore/6.4.0/bin/cpp > == 2018-08-30 13:28:35,027 filetools.py:343 INFO Command dwp found at > /hpc/sw_test/EB_regression/software/binutils/2.28-GCCcore-6.4.0/bin/dwp > == 2018-08-30 13:28:35,028 filetools.py:343 INFO Command gcc found at > /scratch-shared/jenkins/regression/tmp/eb-JpFDuQ/tmpzLIai8/rpath_wrappers/gcc_wrapper/gcc > == 2018-08-30 13:28:35,029 filetools.py:343 INFO Command ld found at > /scratch-shared/jenkins/regression/tmp/eb-JpFDuQ/tmpzLIai8/rpath_wrappers/ld_wrapper/ld > > From what I understand from the EasyBlock, this is used to overwrite > some info in Bazel's installscripts. However, it picks up on our > compiler wrappers, rather than the actual gcc. > > My assumption is that somehow, the location of those tools is used to > infer the location of the GCC that use used, and hence inject the > correct lib-path for thát GCC, so that Bazel picks up on e.g. the > correct libstc++. Is there anyone who has experience with the Bazel > EasyBlock that can confirm that theory? And how should I solve this? I'm > a bit surprised the EasyBlock relies on the location of gcc, even though > in EasyBuild we have stuff like EBROOTGCCCORE that could have been used. > Any reason why that approach was not taken? Well, the short answer is: it's easier to check where those binaries are via 'which', because they could be either in $EBROOTGCC, $EBROOTGCCCORE or $EBROOTBINUTILS... In addition, we have Bazel easyconfigs that use the dummy toolchain, so we still need a fallback to using 'which' anyway. However, your example shows that this approach is not reliable, since we may pick up wrappers rather than the actual binaries, this was overlooked. So, we should fix this in the Bazel easyblock. So, either we check first for $EBROOT* and fall back to 'which' (but this will not fix your problem when Bazel is being built with the 'dummy' toolchain), or we come up with a reliable way to check whether we're picking up the actual binary or a wrapper (one way that comes to mind is checking the output of "file /path/to/gcc"). regards, Kenneth
Re: [easybuild] Bazel-0.12.0-GCCcore-6.4.0 build issues
Hi Caspar, On 30/08/2018 21:31, Caspar van Leeuwen wrote: Dear all, I get an error when building Bazel-0.12.0: bazel-out/host/bin/external/io_bazel/third_party/protobuf/3.4.0/js_embed: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by bazel-out/host/bin/external/io_bazel/third_party/protobuf/3.4.0/js_embed) As you can see, it picks up the system libstdc++, rather than the GCCcore-6.4.0 one. The Bazel EasyBlock seems to search for the location of five tools: ar, cpp, dwp, gcc and ld. == 2018-08-30 13:28:35,003 easyblock.py:2536 INFO Running method configure_step part of step configure == 2018-08-30 13:28:35,023 filetools.py:343 INFO Command ar found at /hpc/sw_test/EB_regression/software/binutils/2.28-GCCcore-6.4.0/bin/ar == 2018-08-30 13:28:35,025 filetools.py:343 INFO Command cpp found at /hpc/sw_test/EB_regression/software/GCCcore/6.4.0/bin/cpp == 2018-08-30 13:28:35,027 filetools.py:343 INFO Command dwp found at /hpc/sw_test/EB_regression/software/binutils/2.28-GCCcore-6.4.0/bin/dwp == 2018-08-30 13:28:35,028 filetools.py:343 INFO Command gcc found at /scratch-shared/jenkins/regression/tmp/eb-JpFDuQ/tmpzLIai8/rpath_wrappers/gcc_wrapper/gcc == 2018-08-30 13:28:35,029 filetools.py:343 INFO Command ld found at /scratch-shared/jenkins/regression/tmp/eb-JpFDuQ/tmpzLIai8/rpath_wrappers/ld_wrapper/ld From what I understand from the EasyBlock, this is used to overwrite some info in Bazel's installscripts. However, it picks up on our compiler wrappers, rather than the actual gcc. My assumption is that somehow, the location of those tools is used to infer the location of the GCC that use used, and hence inject the correct lib-path for thát GCC, so that Bazel picks up on e.g. the correct libstc++. Is there anyone who has experience with the Bazel EasyBlock that can confirm that theory? And how should I solve this? I'm a bit surprised the EasyBlock relies on the location of gcc, even though in EasyBuild we have stuff like EBROOTGCCCORE that could have been used. Any reason why that approach was not taken? Well, the short answer is: it's easier to check where those binaries are via 'which', because they could be either in $EBROOTGCC, $EBROOTGCCCORE or $EBROOTBINUTILS... In addition, we have Bazel easyconfigs that use the dummy toolchain, so we still need a fallback to using 'which' anyway. However, your example shows that this approach is not reliable, since we may pick up wrappers rather than the actual binaries, this was overlooked. So, we should fix this in the Bazel easyblock. So, either we check first for $EBROOT* and fall back to 'which' (but this will not fix your problem when Bazel is being built with the 'dummy' toolchain), or we come up with a reliable way to check whether we're picking up the actual binary or a wrapper (one way that comes to mind is checking the output of "file /path/to/gcc"). regards, Kenneth
Re: [easybuild] bazel
Hello Yann, On 16/03/2017 10:11, Yann Sagon wrote: Hello, I'm trying to build Bazel 0.4.4 using module load foss/2016b (I have tried with other versions as well, seme issue) on Centos 6.8. I'm unable to compile it. It seems people are having success using devtoolset-4-toolchain but if I can do it with easybuild I prefer. Does someone know what could be the difference between devtoolset and foss/x and why it's not working? Here is the issue that I have opened on github for Bazel: https://github.com/bazelbuild/bazel/issues/2534 As mentioned in the Bazel issue, this looks like a bug in either GCC or binutils... See also Which foss versions have you tried? regards, Kenneth
