Re: [gentoo-dev] Bazel Build eclass
On Sun, Nov 18, 2018 at 09:01:00AM +0100, Michał Górny wrote:
> On Sun, 2018-11-18 at 15:37 +0800, Jason Zaman wrote:
> > On Sat, Nov 17, 2018 at 11:54:24PM +0100, Michał Górny wrote:
> > > On Sun, 2018-11-18 at 03:37 +0800, Jason Zaman wrote:
> > > > Hey all,
> > > >
> > > > I've been using Bazel (https://bazel.build/) to build TensorFlow for a
> > > > while now. Here is a bazel.eclass I'd like to commit to make it easier
> > > > for packages that use it to build. It's basically bits that I've
> > > > refactored out of the TensorFlow ebuild that would be useful to other
> > > > packages as well. I have a bump to sci-libs/tensorflow-1.12.0 prepared
> > > > that uses this eclass and have tested a full install.
> > > >
> > > > -- Jason
Here is a v2 of the eclass for review:
Major changes:
- removed MULTIBUILD_VARIANT in favour of BUILD_DIR
- changed bazel_load_distfiles() to tokenize and skip items instead of a
giant sed command.
# Copyright 1999-2018 Jason Zaman
# Distributed under the terms of the GNU General Public License v2
# @ECLASS: bazel.eclass
# @MAINTAINER:
# Jason Zaman
# @AUTHOR:
# Jason Zaman
# @BLURB: Utility functions for packages using Bazel Build
# @DESCRIPTION:
# A utility eclass providing functions to run the Bazel Build system.
#
# This eclass does not export any phase functions.
case "${EAPI:-0}" in
0|1|2|3|4|5|6)
die "Unsupported EAPI=${EAPI:-0} (too old) for ${ECLASS}"
;;
7)
;;
*)
die "Unsupported EAPI=${EAPI} (unknown) for ${ECLASS}"
;;
esac
if [[ ! ${_BAZEL_ECLASS} ]]; then
inherit multiprocessing toolchain-funcs
BDEPEND=">=dev-util/bazel-0.19"
# @FUNCTION: bazel_get_flags
# @DESCRIPTION:
# Obtain and print the bazel flags for target and host *FLAGS.
#
# To add more flags to this, append the flags to the
# appropriate variable before calling this function
bazel_get_flags() {
local i fs=()
for i in ${CFLAGS}; do
fs+=( "--conlyopt=${i}" )
done
for i in ${BUILD_CFLAGS}; do
fs+=( "--host_conlyopt=${i}" )
done
for i in ${CXXFLAGS}; do
fs+=( "--cxxopt=${i}" )
done
for i in ${BUILD_CXXFLAGS}; do
fs+=( "--host_cxxopt=${i}" )
done
for i in ${CPPFLAGS}; do
fs+=( "--conlyopt=${i}" "--cxxopt=${i}" )
done
for i in ${BUILD_CPPFLAGS}; do
fs+=( "--host_conlyopt=${i}" "--host_cxxopt=${i}" )
done
for i in ${LDFLAGS}; do
fs+=( "--linkopt=${i}" )
done
for i in ${BUILD_LDFLAGS}; do
fs+=( "--host_linkopt=${i}" )
done
echo "${fs[*]}"
}
# @FUNCTION: bazel_setup_bazelrc
# @DESCRIPTION:
# Creates the bazelrc with common options that will be passed
# to bazel. This will be called by ebazel automatically so
# does not need to be called from the ebuild.
bazel_setup_bazelrc() {
if [[ -f "${T}/bazelrc" ]]; then
return
fi
# F: fopen_wr
# P: /proc/self/setgroups
# Even with standalone enabled, the Bazel sandbox binary is run for
feature test:
#
https://github.com/bazelbuild/bazel/blob/7b091c1397a82258e26ab5336df6c8dae1d97384/src/main/java/com/google/devtools/build/lib/sandbox/LinuxSandboxedSpawnRunner.java#L61
#
https://github.com/bazelbuild/bazel/blob/76555482873ffcf1d32fb40106f89231b37f850a/src/main/tools/linux-sandbox-pid1.cc#L113
addpredict /proc
mkdir -p "${T}/bazel-cache" || die
mkdir -p "${T}/bazel-distdir" || die
cat > "${T}/bazelrc" <<-EOF || die
startup --batch
# dont strip HOME, portage sets a temp per-package dir
build --action_env HOME
# make bazel respect MAKEOPTS
build --jobs=$(makeopts_jobs)
build --compilation_mode=opt --host_compilation_mode=opt
# FLAGS
build $(bazel_get_flags)
# Use standalone strategy to deactivate the bazel sandbox,
since it
# conflicts with FEATURES=sandbox.
build --spawn_strategy=standalone --genrule_strategy=standalone
test --spawn_strategy=standalone --genrule_strategy=standalone
build --strip=never
build --verbose_failures --noshow_loading_progress
test --verbose_test_summary --verbose_failures
--noshow_loading_progress
# make bazel only fetch distfiles from the cache
fetch --repository_cache="${T}/bazel-cache/"
--distdir="${T}/bazel-distdir/"
build --repository_cache="${T}/bazel-cache/"
--distdir="${T}/bazel-distdir/"
build --define=PREFIX=${EPREFIX%/}/usr
build --define=LIBDIR=\$(PREFIX)/$(get_libdir)
EOF
if
Re: [gentoo-dev] Bazel Build eclass
On Sun, 2018-11-18 at 15:37 +0800, Jason Zaman wrote:
> On Sat, Nov 17, 2018 at 11:54:24PM +0100, Michał Górny wrote:
> > On Sun, 2018-11-18 at 03:37 +0800, Jason Zaman wrote:
> > > Hey all,
> > >
> > > I've been using Bazel (https://bazel.build/) to build TensorFlow for a
> > > while now. Here is a bazel.eclass I'd like to commit to make it easier
> > > for packages that use it to build. It's basically bits that I've
> > > refactored out of the TensorFlow ebuild that would be useful to other
> > > packages as well. I have a bump to sci-libs/tensorflow-1.12.0 prepared
> > > that uses this eclass and have tested a full install.
> > >
> > > -- Jason
> > >
> > > # Copyright 1999-2018 Jason Zaman
> > > # Distributed under the terms of the GNU General Public License v2
> > >
> > > # @ECLASS: bazel.eclass
> > > # @MAINTAINER:
> > > # Jason Zaman
> > > # @AUTHOR:
> > > # Jason Zaman
> > > # @BLURB: Utility functions for packages using Bazel Build
> > > # @DESCRIPTION:
> > > # A utility eclass providing functions to run the Bazel Build system.
> > > #
> > > # This eclass does not export any phase functions.
> > >
> > > case "${EAPI:-0}" in
> > > 0|1|2|3|4|5|6)
> > > die "Unsupported EAPI=${EAPI:-0} (too old) for ${ECLASS}"
> > > ;;
> > > 7)
> > > ;;
> > > *)
> > > die "Unsupported EAPI=${EAPI} (unknown) for ${ECLASS}"
> > > ;;
> > > esac
> > >
> > > if [[ ! ${_BAZEL_ECLASS} ]]; then
> > >
> > > inherit multiprocessing toolchain-funcs
> > >
> > > BDEPEND=">=dev-util/bazel-0.19"
> > >
> > > # @FUNCTION: bazel_get_flags
> > > # @DESCRIPTION:
> > > # Obtain and print the bazel flags for target and host *FLAGS.
> > > #
> > > # To add more flags to this, append the flags to the
> > > # appropriate variable before calling this function
> > > bazel_get_flags() {
> > > local i fs=()
> > > for i in ${CFLAGS}; do
> > > fs+=( "--conlyopt=${i}" )
> > > done
> > > for i in ${BUILD_CFLAGS}; do
> > > fs+=( "--host_conlyopt=${i}" )
> > > done
> > > for i in ${CXXFLAGS}; do
> > > fs+=( "--cxxopt=${i}" )
> > > done
> > > for i in ${BUILD_CXXFLAGS}; do
> > > fs+=( "--host_cxxopt=${i}" )
> > > done
> > > for i in ${CPPFLAGS}; do
> > > fs+=( "--conlyopt=${i}" "--cxxopt=${i}" )
> > > done
> > > for i in ${BUILD_CPPFLAGS}; do
> > > fs+=( "--host_conlyopt=${i}" "--host_cxxopt=${i}" )
> > > done
> > > for i in ${LDFLAGS}; do
> > > fs+=( "--linkopt=${i}" )
> > > done
> > > for i in ${BUILD_LDFLAGS}; do
> > > fs+=( "--host_linkopt=${i}" )
> > > done
> > > echo "${fs[*]}"
> > > }
> > >
> > > # @FUNCTION: bazel_setup_bazelrc
> > > # @DESCRIPTION:
> > > # Creates the bazelrc with common options that will be passed
> > > # to bazel. This will be called by ebazel automatically so
> > > # does not need to be called from the ebuild.
> > > bazel_setup_bazelrc() {
> > > if [[ -f "${T}/bazelrc" ]]; then
> > > return
> > > fi
> > >
> > > # F: fopen_wr
> > > # P: /proc/self/setgroups
> > > # Even with standalone enabled, the Bazel sandbox binary is run for
> > > feature test:
> > > #
> > > https://github.com/bazelbuild/bazel/blob/7b091c1397a82258e26ab5336df6c8dae1d97384/src/main/java/com/google/devtools/build/lib/sandbox/LinuxSandboxedSpawnRunner.java#L61
> > > #
> > > https://github.com/bazelbuild/bazel/blob/76555482873ffcf1d32fb40106f89231b37f850a/src/main/tools/linux-sandbox-pid1.cc#L113
> > > addpredict /proc
> > >
> > > mkdir -p "${T}/bazel-cache" || die
> > > mkdir -p "${T}/bazel-distdir" || die
> > >
> > > cat > "${T}/bazelrc" <<-EOF || die
> > > startup --batch
> >
> > Maybe indent this stuff to make it stand out from ebuild code.
> >
> > >
> > > # dont strip HOME, portage sets a temp per-package dir
> > > build --action_env HOME
> > >
> > > # make bazel respect MAKEOPTS
> > > build --jobs=$(makeopts_jobs)
> > > build --compilation_mode=opt --host_compilation_mode=opt
> > >
> > > # FLAGS
> > > build $(bazel_get_flags)
> > >
> > > # Use standalone strategy to deactivate the bazel sandbox, since it
> > > # conflicts with FEATURES=sandbox.
> > > build --spawn_strategy=standalone --genrule_strategy=standalone
> > > test --spawn_strategy=standalone --genrule_strategy=standalone
> > >
> > > build --strip=never
> > > build --verbose_failures --noshow_loading_progress
> > > test --verbose_test_summary --verbose_failures --noshow_loading_progress
> > >
> > > # make bazel only fetch distfiles from the cache
> > > fetch --repository_cache="${T}/bazel-cache/"
> > > --distdir="${T}/bazel-distdir/"
> > > build --repository_cache="${T}/bazel-cache/"
> > > --distdir="${T}/bazel-distdir/"
> > >
> > > build --define=PREFIX=${EPREFIX%/}/usr
> > > build --define=LIBDIR=\$(PREFIX)/$(get_libdir)
> > >
> > > EOF
> > >
> > > tc-is-cross-compiler || \
> > > echo "build
Re: [gentoo-dev] Bazel Build eclass
On Sat, Nov 17, 2018 at 11:54:24PM +0100, Michał Górny wrote:
> On Sun, 2018-11-18 at 03:37 +0800, Jason Zaman wrote:
> > Hey all,
> >
> > I've been using Bazel (https://bazel.build/) to build TensorFlow for a
> > while now. Here is a bazel.eclass I'd like to commit to make it easier
> > for packages that use it to build. It's basically bits that I've
> > refactored out of the TensorFlow ebuild that would be useful to other
> > packages as well. I have a bump to sci-libs/tensorflow-1.12.0 prepared
> > that uses this eclass and have tested a full install.
> >
> > -- Jason
> >
> > # Copyright 1999-2018 Jason Zaman
> > # Distributed under the terms of the GNU General Public License v2
> >
> > # @ECLASS: bazel.eclass
> > # @MAINTAINER:
> > # Jason Zaman
> > # @AUTHOR:
> > # Jason Zaman
> > # @BLURB: Utility functions for packages using Bazel Build
> > # @DESCRIPTION:
> > # A utility eclass providing functions to run the Bazel Build system.
> > #
> > # This eclass does not export any phase functions.
> >
> > case "${EAPI:-0}" in
> > 0|1|2|3|4|5|6)
> > die "Unsupported EAPI=${EAPI:-0} (too old) for ${ECLASS}"
> > ;;
> > 7)
> > ;;
> > *)
> > die "Unsupported EAPI=${EAPI} (unknown) for ${ECLASS}"
> > ;;
> > esac
> >
> > if [[ ! ${_BAZEL_ECLASS} ]]; then
> >
> > inherit multiprocessing toolchain-funcs
> >
> > BDEPEND=">=dev-util/bazel-0.19"
> >
> > # @FUNCTION: bazel_get_flags
> > # @DESCRIPTION:
> > # Obtain and print the bazel flags for target and host *FLAGS.
> > #
> > # To add more flags to this, append the flags to the
> > # appropriate variable before calling this function
> > bazel_get_flags() {
> > local i fs=()
> > for i in ${CFLAGS}; do
> > fs+=( "--conlyopt=${i}" )
> > done
> > for i in ${BUILD_CFLAGS}; do
> > fs+=( "--host_conlyopt=${i}" )
> > done
> > for i in ${CXXFLAGS}; do
> > fs+=( "--cxxopt=${i}" )
> > done
> > for i in ${BUILD_CXXFLAGS}; do
> > fs+=( "--host_cxxopt=${i}" )
> > done
> > for i in ${CPPFLAGS}; do
> > fs+=( "--conlyopt=${i}" "--cxxopt=${i}" )
> > done
> > for i in ${BUILD_CPPFLAGS}; do
> > fs+=( "--host_conlyopt=${i}" "--host_cxxopt=${i}" )
> > done
> > for i in ${LDFLAGS}; do
> > fs+=( "--linkopt=${i}" )
> > done
> > for i in ${BUILD_LDFLAGS}; do
> > fs+=( "--host_linkopt=${i}" )
> > done
> > echo "${fs[*]}"
> > }
> >
> > # @FUNCTION: bazel_setup_bazelrc
> > # @DESCRIPTION:
> > # Creates the bazelrc with common options that will be passed
> > # to bazel. This will be called by ebazel automatically so
> > # does not need to be called from the ebuild.
> > bazel_setup_bazelrc() {
> > if [[ -f "${T}/bazelrc" ]]; then
> > return
> > fi
> >
> > # F: fopen_wr
> > # P: /proc/self/setgroups
> > # Even with standalone enabled, the Bazel sandbox binary is run for
> > feature test:
> > #
> > https://github.com/bazelbuild/bazel/blob/7b091c1397a82258e26ab5336df6c8dae1d97384/src/main/java/com/google/devtools/build/lib/sandbox/LinuxSandboxedSpawnRunner.java#L61
> > #
> > https://github.com/bazelbuild/bazel/blob/76555482873ffcf1d32fb40106f89231b37f850a/src/main/tools/linux-sandbox-pid1.cc#L113
> > addpredict /proc
> >
> > mkdir -p "${T}/bazel-cache" || die
> > mkdir -p "${T}/bazel-distdir" || die
> >
> > cat > "${T}/bazelrc" <<-EOF || die
> > startup --batch
>
> Maybe indent this stuff to make it stand out from ebuild code.
>
> >
> > # dont strip HOME, portage sets a temp per-package dir
> > build --action_env HOME
> >
> > # make bazel respect MAKEOPTS
> > build --jobs=$(makeopts_jobs)
> > build --compilation_mode=opt --host_compilation_mode=opt
> >
> > # FLAGS
> > build $(bazel_get_flags)
> >
> > # Use standalone strategy to deactivate the bazel sandbox, since it
> > # conflicts with FEATURES=sandbox.
> > build --spawn_strategy=standalone --genrule_strategy=standalone
> > test --spawn_strategy=standalone --genrule_strategy=standalone
> >
> > build --strip=never
> > build --verbose_failures --noshow_loading_progress
> > test --verbose_test_summary --verbose_failures --noshow_loading_progress
> >
> > # make bazel only fetch distfiles from the cache
> > fetch --repository_cache="${T}/bazel-cache/"
> > --distdir="${T}/bazel-distdir/"
> > build --repository_cache="${T}/bazel-cache/"
> > --distdir="${T}/bazel-distdir/"
> >
> > build --define=PREFIX=${EPREFIX%/}/usr
> > build --define=LIBDIR=\$(PREFIX)/$(get_libdir)
> >
> > EOF
> >
> > tc-is-cross-compiler || \
> > echo "build --nodistinct_host_configuration" >> "${T}/bazelrc"
> > || die
>
> Don't do || chains, they are unreadable.
ok
> > }
> >
> > # @FUNCTION: ebazel
> > # @USAGE: [...]
> > # @DESCRIPTION:
> > # Run bazel with the
Re: [gentoo-dev] Bazel Build eclass
On Sun, 2018-11-18 at 03:37 +0800, Jason Zaman wrote:
> Hey all,
>
> I've been using Bazel (https://bazel.build/) to build TensorFlow for a
> while now. Here is a bazel.eclass I'd like to commit to make it easier
> for packages that use it to build. It's basically bits that I've
> refactored out of the TensorFlow ebuild that would be useful to other
> packages as well. I have a bump to sci-libs/tensorflow-1.12.0 prepared
> that uses this eclass and have tested a full install.
>
> -- Jason
>
> # Copyright 1999-2018 Jason Zaman
> # Distributed under the terms of the GNU General Public License v2
>
> # @ECLASS: bazel.eclass
> # @MAINTAINER:
> # Jason Zaman
> # @AUTHOR:
> # Jason Zaman
> # @BLURB: Utility functions for packages using Bazel Build
> # @DESCRIPTION:
> # A utility eclass providing functions to run the Bazel Build system.
> #
> # This eclass does not export any phase functions.
>
> case "${EAPI:-0}" in
> 0|1|2|3|4|5|6)
> die "Unsupported EAPI=${EAPI:-0} (too old) for ${ECLASS}"
> ;;
> 7)
> ;;
> *)
> die "Unsupported EAPI=${EAPI} (unknown) for ${ECLASS}"
> ;;
> esac
>
> if [[ ! ${_BAZEL_ECLASS} ]]; then
>
> inherit multiprocessing toolchain-funcs
>
> BDEPEND=">=dev-util/bazel-0.19"
>
> # @FUNCTION: bazel_get_flags
> # @DESCRIPTION:
> # Obtain and print the bazel flags for target and host *FLAGS.
> #
> # To add more flags to this, append the flags to the
> # appropriate variable before calling this function
> bazel_get_flags() {
> local i fs=()
> for i in ${CFLAGS}; do
> fs+=( "--conlyopt=${i}" )
> done
> for i in ${BUILD_CFLAGS}; do
> fs+=( "--host_conlyopt=${i}" )
> done
> for i in ${CXXFLAGS}; do
> fs+=( "--cxxopt=${i}" )
> done
> for i in ${BUILD_CXXFLAGS}; do
> fs+=( "--host_cxxopt=${i}" )
> done
> for i in ${CPPFLAGS}; do
> fs+=( "--conlyopt=${i}" "--cxxopt=${i}" )
> done
> for i in ${BUILD_CPPFLAGS}; do
> fs+=( "--host_conlyopt=${i}" "--host_cxxopt=${i}" )
> done
> for i in ${LDFLAGS}; do
> fs+=( "--linkopt=${i}" )
> done
> for i in ${BUILD_LDFLAGS}; do
> fs+=( "--host_linkopt=${i}" )
> done
> echo "${fs[*]}"
> }
>
> # @FUNCTION: bazel_setup_bazelrc
> # @DESCRIPTION:
> # Creates the bazelrc with common options that will be passed
> # to bazel. This will be called by ebazel automatically so
> # does not need to be called from the ebuild.
> bazel_setup_bazelrc() {
> if [[ -f "${T}/bazelrc" ]]; then
> return
> fi
>
> # F: fopen_wr
> # P: /proc/self/setgroups
> # Even with standalone enabled, the Bazel sandbox binary is run for
> feature test:
> #
> https://github.com/bazelbuild/bazel/blob/7b091c1397a82258e26ab5336df6c8dae1d97384/src/main/java/com/google/devtools/build/lib/sandbox/LinuxSandboxedSpawnRunner.java#L61
> #
> https://github.com/bazelbuild/bazel/blob/76555482873ffcf1d32fb40106f89231b37f850a/src/main/tools/linux-sandbox-pid1.cc#L113
> addpredict /proc
>
> mkdir -p "${T}/bazel-cache" || die
> mkdir -p "${T}/bazel-distdir" || die
>
> cat > "${T}/bazelrc" <<-EOF || die
> startup --batch
Maybe indent this stuff to make it stand out from ebuild code.
>
> # dont strip HOME, portage sets a temp per-package dir
> build --action_env HOME
>
> # make bazel respect MAKEOPTS
> build --jobs=$(makeopts_jobs)
> build --compilation_mode=opt --host_compilation_mode=opt
>
> # FLAGS
> build $(bazel_get_flags)
>
> # Use standalone strategy to deactivate the bazel sandbox, since it
> # conflicts with FEATURES=sandbox.
> build --spawn_strategy=standalone --genrule_strategy=standalone
> test --spawn_strategy=standalone --genrule_strategy=standalone
>
> build --strip=never
> build --verbose_failures --noshow_loading_progress
> test --verbose_test_summary --verbose_failures --noshow_loading_progress
>
> # make bazel only fetch distfiles from the cache
> fetch --repository_cache="${T}/bazel-cache/"
> --distdir="${T}/bazel-distdir/"
> build --repository_cache="${T}/bazel-cache/"
> --distdir="${T}/bazel-distdir/"
>
> build --define=PREFIX=${EPREFIX%/}/usr
> build --define=LIBDIR=\$(PREFIX)/$(get_libdir)
>
> EOF
>
> tc-is-cross-compiler || \
> echo "build --nodistinct_host_configuration" >> "${T}/bazelrc"
> || die
Don't do || chains, they are unreadable.
> }
>
> # @FUNCTION: ebazel
> # @USAGE: [...]
> # @DESCRIPTION:
> # Run bazel with the bazelrc and output_base.
> #
> # If $MULTIBUILD_VARIANT is set, this will make an output_base
> # specific to that variant.
> # bazel_setup_bazelrc will be called and the created bazelrc
> # will be passed to bazel.
> #
>
