Re: [gentoo-dev] Bazel Build eclass
On Sat, Nov 17, 2018 at 11:54:24PM +0100, Michał Górny wrote:
> On Sun, 2018-11-18 at 03:37 +0800, Jason Zaman wrote:
> > Hey all,
> >
> > I've been using Bazel (https://bazel.build/) to build TensorFlow for a
> > while now. Here is a bazel.eclass I'd like to commit to make it easier
> > for packages that use it to build. It's basically bits that I've
> > refactored out of the TensorFlow ebuild that would be useful to other
> > packages as well. I have a bump to sci-libs/tensorflow-1.12.0 prepared
> > that uses this eclass and have tested a full install.
> >
> > -- Jason
> >
> > # Copyright 1999-2018 Jason Zaman
> > # Distributed under the terms of the GNU General Public License v2
> >
> > # @ECLASS: bazel.eclass
> > # @MAINTAINER:
> > # Jason Zaman
> > # @AUTHOR:
> > # Jason Zaman
> > # @BLURB: Utility functions for packages using Bazel Build
> > # @DESCRIPTION:
> > # A utility eclass providing functions to run the Bazel Build system.
> > #
> > # This eclass does not export any phase functions.
> >
> > case "${EAPI:-0}" in
> > 0|1|2|3|4|5|6)
> > die "Unsupported EAPI=${EAPI:-0} (too old) for ${ECLASS}"
> > ;;
> > 7)
> > ;;
> > *)
> > die "Unsupported EAPI=${EAPI} (unknown) for ${ECLASS}"
> > ;;
> > esac
> >
> > if [[ ! ${_BAZEL_ECLASS} ]]; then
> >
> > inherit multiprocessing toolchain-funcs
> >
> > BDEPEND=">=dev-util/bazel-0.19"
> >
> > # @FUNCTION: bazel_get_flags
> > # @DESCRIPTION:
> > # Obtain and print the bazel flags for target and host *FLAGS.
> > #
> > # To add more flags to this, append the flags to the
> > # appropriate variable before calling this function
> > bazel_get_flags() {
> > local i fs=()
> > for i in ${CFLAGS}; do
> > fs+=( "--conlyopt=${i}" )
> > done
> > for i in ${BUILD_CFLAGS}; do
> > fs+=( "--host_conlyopt=${i}" )
> > done
> > for i in ${CXXFLAGS}; do
> > fs+=( "--cxxopt=${i}" )
> > done
> > for i in ${BUILD_CXXFLAGS}; do
> > fs+=( "--host_cxxopt=${i}" )
> > done
> > for i in ${CPPFLAGS}; do
> > fs+=( "--conlyopt=${i}" "--cxxopt=${i}" )
> > done
> > for i in ${BUILD_CPPFLAGS}; do
> > fs+=( "--host_conlyopt=${i}" "--host_cxxopt=${i}" )
> > done
> > for i in ${LDFLAGS}; do
> > fs+=( "--linkopt=${i}" )
> > done
> > for i in ${BUILD_LDFLAGS}; do
> > fs+=( "--host_linkopt=${i}" )
> > done
> > echo "${fs[*]}"
> > }
> >
> > # @FUNCTION: bazel_setup_bazelrc
> > # @DESCRIPTION:
> > # Creates the bazelrc with common options that will be passed
> > # to bazel. This will be called by ebazel automatically so
> > # does not need to be called from the ebuild.
> > bazel_setup_bazelrc() {
> > if [[ -f "${T}/bazelrc" ]]; then
> > return
> > fi
> >
> > # F: fopen_wr
> > # P: /proc/self/setgroups
> > # Even with standalone enabled, the Bazel sandbox binary is run for
> > feature test:
> > #
> > https://github.com/bazelbuild/bazel/blob/7b091c1397a82258e26ab5336df6c8dae1d97384/src/main/java/com/google/devtools/build/lib/sandbox/LinuxSandboxedSpawnRunner.java#L61
> > #
> > https://github.com/bazelbuild/bazel/blob/76555482873ffcf1d32fb40106f89231b37f850a/src/main/tools/linux-sandbox-pid1.cc#L113
> > addpredict /proc
> >
> > mkdir -p "${T}/bazel-cache" || die
> > mkdir -p "${T}/bazel-distdir" || die
> >
> > cat > "${T}/bazelrc" <<-EOF || die
> > startup --batch
>
> Maybe indent this stuff to make it stand out from ebuild code.
>
> >
> > # dont strip HOME, portage sets a temp per-package dir
> > build --action_env HOME
> >
> > # make bazel respect MAKEOPTS
> > build --jobs=$(makeopts_jobs)
> > build --compilation_mode=opt --host_compilation_mode=opt
> >
> > # FLAGS
> > build $(bazel_get_flags)
> >
> > # Use standalone strategy to deactivate the bazel sandbox, since it
> > # conflicts with FEATURES=sandbox.
> > build --spawn_strategy=standalone --genrule_strategy=standalone
> > test --spawn_strategy=standalone --genrule_strategy=standalone
> >
> > build --strip=never
> > build --verbose_failures --noshow_loading_progress
> > test --verbose_test_summary --verbose_failures --noshow_loading_progress
> >
> > # make bazel only fetch distfiles from the cache
> > fetch --repository_cache="${T}/bazel-cache/"
> > --distdir="${T}/bazel-distdir/"
> > build --repository_cache="${T}/bazel-cache/"
> > --distdir="${T}/bazel-distdir/"
> >
> > build --define=PREFIX=${EPREFIX%/}/usr
> > build --define=LIBDIR=\$(PREFIX)/$(get_libdir)
> >
> > EOF
> >
> > tc-is-cross-compiler || \
> > echo "build --nodistinct_host_configuration" >> "${T}/bazelrc"
> > || die
>
> Don't do || chains, they are unreadable.
ok
> > }
> >
> > # @FUNCTION: ebazel
> > # @USAGE: [...]
> > # @DESCRIPTION:
> > # Run bazel with the
Re: [gentoo-portage-dev] [PATCH] ecompress-file: Support decompressing .lz
On 11/17/18 12:14 AM, Michał Górny wrote:
> Signed-off-by: Michał Górny
> ---
> bin/ecompress-file | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/bin/ecompress-file b/bin/ecompress-file
> index bc8fe5451..ccc2701c3 100755
> --- a/bin/ecompress-file
> +++ b/bin/ecompress-file
> @@ -29,6 +29,9 @@ compress_file() {
> *.lzma|*.xz)
> unxz -f "${x}" || __helpers_die "unxz failed"
> x=${x%.*};;
> + *.lz)
> + lzip -df "${x}" || __helpers_die "lzip -d
> failed"
> + x=${x%.lz};;
> esac
>
> filtered_args+=( "$x" )
>
Looks good, please merge.
--
Thanks,
Zac
signature.asc
Description: OpenPGP digital signature
Re: [gentoo-portage-dev] [PATCH 2/3] Support FEATURES=pid-sandbox
On 11/14/18 12:02 AM, Michał Górny wrote:
> @@ -531,6 +543,15 @@ def _exec(binary, mycommand, opt_name, fd_pipes, env,
> gid, groups, uid, umask,
>
> errno.errorcode.get(ctypes.get_errno(), '?')),
> noiselevel=-1)
> else:
> + if unshare_pid:
> + # pid namespace
> requires us to become init
> + # TODO: do init-ty stuff
> + # therefore, fork() ASAP
> + fork_ret = os.fork()
> + if fork_ret != 0:
> + pid, status =
> os.waitpid(fork_ret, 0)
> + assert pid ==
> fork_ret
> + os._exit(status)
> if unshare_mount:
> # mark the whole
> filesystem as private to avoid
> # mounts escaping the
> namespace
> @@ -541,6 +562,18 @@ def _exec(binary, mycommand, opt_name, fd_pipes, env,
> gid, groups, uid, umask,
> # TODO: should
> it be fatal maybe?
>
> writemsg("Unable to mark mounts private: %d\n" % (mount_ret,),
>
> noiselevel=-1)
> + if unshare_pid:
> + if mount_ret != 0:
> + # can't proceed
> without private mounts
> + os._exit(1)
For the benefit of anyone not watching
https://github.com/gentoo/portage/pull/379, the mount_ret is expected
to be non-zero inside a chroot where `mount --make-rprivate /` would
fail because / is not a mountpoint, and this is an extremely valuable
use case for tools like catalyst that perform builds inside a chroot.
Based on /proc/[pid]/mountinfo documentation in the proc(5) manpage,
and empirical analysis of /proc/self/mountinfo in various states,
it should be reasonable to assume that the current propagation flags
are suitable if there is not a shared / or /proc mountpoint found in
/proc/self/mountinfo. We can use a function like this to check if the
`mount --make-rprivate /` call is needed:
def want_make_rprivate():
try:
with open('/proc/self/mountinfo', 'rb') as f:
if re.match(rb'^\S+ \S+ \S+ \S+ (?P/|/proc) \S+
(?:\S+:\S+ )*(?Pshared:\S+ )', f.read(), re.MULTILINE) is None:
return False
except EnvironmentError:
pass
return True
--
Thanks,
Zac
signature.asc
Description: OpenPGP digital signature
Re: [gentoo-dev] Bazel Build eclass
On Sun, 2018-11-18 at 03:37 +0800, Jason Zaman wrote:
> Hey all,
>
> I've been using Bazel (https://bazel.build/) to build TensorFlow for a
> while now. Here is a bazel.eclass I'd like to commit to make it easier
> for packages that use it to build. It's basically bits that I've
> refactored out of the TensorFlow ebuild that would be useful to other
> packages as well. I have a bump to sci-libs/tensorflow-1.12.0 prepared
> that uses this eclass and have tested a full install.
>
> -- Jason
>
> # Copyright 1999-2018 Jason Zaman
> # Distributed under the terms of the GNU General Public License v2
>
> # @ECLASS: bazel.eclass
> # @MAINTAINER:
> # Jason Zaman
> # @AUTHOR:
> # Jason Zaman
> # @BLURB: Utility functions for packages using Bazel Build
> # @DESCRIPTION:
> # A utility eclass providing functions to run the Bazel Build system.
> #
> # This eclass does not export any phase functions.
>
> case "${EAPI:-0}" in
> 0|1|2|3|4|5|6)
> die "Unsupported EAPI=${EAPI:-0} (too old) for ${ECLASS}"
> ;;
> 7)
> ;;
> *)
> die "Unsupported EAPI=${EAPI} (unknown) for ${ECLASS}"
> ;;
> esac
>
> if [[ ! ${_BAZEL_ECLASS} ]]; then
>
> inherit multiprocessing toolchain-funcs
>
> BDEPEND=">=dev-util/bazel-0.19"
>
> # @FUNCTION: bazel_get_flags
> # @DESCRIPTION:
> # Obtain and print the bazel flags for target and host *FLAGS.
> #
> # To add more flags to this, append the flags to the
> # appropriate variable before calling this function
> bazel_get_flags() {
> local i fs=()
> for i in ${CFLAGS}; do
> fs+=( "--conlyopt=${i}" )
> done
> for i in ${BUILD_CFLAGS}; do
> fs+=( "--host_conlyopt=${i}" )
> done
> for i in ${CXXFLAGS}; do
> fs+=( "--cxxopt=${i}" )
> done
> for i in ${BUILD_CXXFLAGS}; do
> fs+=( "--host_cxxopt=${i}" )
> done
> for i in ${CPPFLAGS}; do
> fs+=( "--conlyopt=${i}" "--cxxopt=${i}" )
> done
> for i in ${BUILD_CPPFLAGS}; do
> fs+=( "--host_conlyopt=${i}" "--host_cxxopt=${i}" )
> done
> for i in ${LDFLAGS}; do
> fs+=( "--linkopt=${i}" )
> done
> for i in ${BUILD_LDFLAGS}; do
> fs+=( "--host_linkopt=${i}" )
> done
> echo "${fs[*]}"
> }
>
> # @FUNCTION: bazel_setup_bazelrc
> # @DESCRIPTION:
> # Creates the bazelrc with common options that will be passed
> # to bazel. This will be called by ebazel automatically so
> # does not need to be called from the ebuild.
> bazel_setup_bazelrc() {
> if [[ -f "${T}/bazelrc" ]]; then
> return
> fi
>
> # F: fopen_wr
> # P: /proc/self/setgroups
> # Even with standalone enabled, the Bazel sandbox binary is run for
> feature test:
> #
> https://github.com/bazelbuild/bazel/blob/7b091c1397a82258e26ab5336df6c8dae1d97384/src/main/java/com/google/devtools/build/lib/sandbox/LinuxSandboxedSpawnRunner.java#L61
> #
> https://github.com/bazelbuild/bazel/blob/76555482873ffcf1d32fb40106f89231b37f850a/src/main/tools/linux-sandbox-pid1.cc#L113
> addpredict /proc
>
> mkdir -p "${T}/bazel-cache" || die
> mkdir -p "${T}/bazel-distdir" || die
>
> cat > "${T}/bazelrc" <<-EOF || die
> startup --batch
Maybe indent this stuff to make it stand out from ebuild code.
>
> # dont strip HOME, portage sets a temp per-package dir
> build --action_env HOME
>
> # make bazel respect MAKEOPTS
> build --jobs=$(makeopts_jobs)
> build --compilation_mode=opt --host_compilation_mode=opt
>
> # FLAGS
> build $(bazel_get_flags)
>
> # Use standalone strategy to deactivate the bazel sandbox, since it
> # conflicts with FEATURES=sandbox.
> build --spawn_strategy=standalone --genrule_strategy=standalone
> test --spawn_strategy=standalone --genrule_strategy=standalone
>
> build --strip=never
> build --verbose_failures --noshow_loading_progress
> test --verbose_test_summary --verbose_failures --noshow_loading_progress
>
> # make bazel only fetch distfiles from the cache
> fetch --repository_cache="${T}/bazel-cache/"
> --distdir="${T}/bazel-distdir/"
> build --repository_cache="${T}/bazel-cache/"
> --distdir="${T}/bazel-distdir/"
>
> build --define=PREFIX=${EPREFIX%/}/usr
> build --define=LIBDIR=\$(PREFIX)/$(get_libdir)
>
> EOF
>
> tc-is-cross-compiler || \
> echo "build --nodistinct_host_configuration" >> "${T}/bazelrc"
> || die
Don't do || chains, they are unreadable.
> }
>
> # @FUNCTION: ebazel
> # @USAGE: [...]
> # @DESCRIPTION:
> # Run bazel with the bazelrc and output_base.
> #
> # If $MULTIBUILD_VARIANT is set, this will make an output_base
> # specific to that variant.
> # bazel_setup_bazelrc will be called and the created bazelrc
> # will be passed to bazel.
> #
>
Re: [gentoo-dev] [pre-GLEP] Gentoo binary package container format
On Sat, 2018-11-17 at 14:05 +, Roy Bamford wrote: > On 2018.11.17 11:21, Michał Górny wrote: > > Hi, > > > > Here's a pre-GLEP draft based on the earlier discussion on gentoo- > > portage-dev mailing list. The specification uses GLEP form as it > > provides for cleanly specifying the motivation and rationale. > > > > [snip glep proposal] > > -- > > Best regards, > > Michał Górny > > > > Team, > > One of the attractions of the existing format is that > tar xf /path/to/tarball -C /mnt/gentoo > works to fix things like glibc being removed and other > missing essential portage components. > > In effect, each binary package can be treated as a > single package stage3 when a user needs a get out of jail > free card. > > Does this proposal allow for installing the payload without > the use of the Gentoo package manager from some random > distro being used as a rescue media? Yes, and it can also be done via one-liner, though it's going to be more complex than before, e.g.: tar -xOf mypackage-1.gpkg.tar mypackage-1/image.tar.lz | tar --lzip -x -C /mnt/gentoo --strip-components 1 Though I wouldn't recommend using it but instead unpacking it normally and inspecting the contents first. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
[gentoo-dev] [PATCH] eclass/webapp.eclass: fix ROOT path with EAPI=7
From: Conrad Kostecki
Closes: https://bugs.gentoo.org/671258
Signed-off-by: Conrad Kostecki
---
eclass/webapp.eclass | 14 +++---
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/eclass/webapp.eclass b/eclass/webapp.eclass
index 8983af334ab..e11835735ca 100644
--- a/eclass/webapp.eclass
+++ b/eclass/webapp.eclass
@@ -42,9 +42,9 @@ IS_REPLACE=0
INSTALL_CHECK_FILE="installed_by_webapp_eclass"
SETUP_CHECK_FILE="setup_by_webapp_eclass"
-ETC_CONFIG="${ROOT}etc/vhosts/webapp-config"
-WEBAPP_CONFIG="${ROOT}usr/sbin/webapp-config"
-WEBAPP_CLEANER="${ROOT}usr/sbin/webapp-cleaner"
+ETC_CONFIG="${ROOT%/}/etc/vhosts/webapp-config"
+WEBAPP_CONFIG="${ROOT%/}/usr/sbin/webapp-config"
+WEBAPP_CLEANER="${ROOT%/}/usr/sbin/webapp-cleaner"
#
==
# INTERNAL FUNCTIONS
@@ -365,7 +365,7 @@ webapp_src_preinst() {
# @DESCRIPTION:
# The default pkg_setup() for this eclass. This will gather required variables
# from webapp-config and check if there is an application installed to
-# `${ROOT}/var/www/localhost/htdocs/${PN}/' if USE=vhosts is not set.
+# `${ROOT%/}/var/www/localhost/htdocs/${PN}/' if USE=vhosts is not set.
#
# You need to call this function BEFORE anything else has run in your custom
# pkg_setup().
@@ -389,7 +389,7 @@ webapp_pkg_setup() {
G_HOSTNAME="localhost"
webapp_read_config
- local my_dir="${ROOT}${VHOST_ROOT}/${MY_HTDOCSBASE}/${PN}"
+ local my_dir="${ROOT%/}/${VHOST_ROOT}/${MY_HTDOCSBASE}/${PN}"
# if USE=vhosts is enabled OR no application is installed we're done
here
if ! has vhosts ${IUSE} || use vhosts || [[ ! -d "${my_dir}" ]]; then
@@ -453,7 +453,7 @@ webapp_src_install() {
# @FUNCTION: webapp_pkg_postinst
# @DESCRIPTION:
# The default pkg_postinst() for this eclass. This installs the web
application to
-# `${ROOT}/var/www/localhost/htdocs/${PN}/' if USE=vhosts is not set. Otherwise
+# `${ROOT%/}/var/www/localhost/htdocs/${PN}/' if USE=vhosts is not set.
Otherwise
# display a short notice how to install this application with webapp-config.
#
# You need to call this function AFTER everything else has run in your custom
@@ -464,7 +464,7 @@ webapp_pkg_postinst() {
webapp_read_config
# sanity checks, to catch bugs in the ebuild
- if [[ ! -f "${ROOT}${MY_APPDIR}/${INSTALL_CHECK_FILE}" ]]; then
+ if [[ ! -f "${ROOT%/}/${MY_APPDIR}/${INSTALL_CHECK_FILE}" ]]; then
eerror
eerror "This ebuild did not call webapp_src_install() at the
end"
eerror "of the src_install() function"
--
2.19.1
[gentoo-dev] Bazel Build eclass
Hey all,
I've been using Bazel (https://bazel.build/) to build TensorFlow for a
while now. Here is a bazel.eclass I'd like to commit to make it easier
for packages that use it to build. It's basically bits that I've
refactored out of the TensorFlow ebuild that would be useful to other
packages as well. I have a bump to sci-libs/tensorflow-1.12.0 prepared
that uses this eclass and have tested a full install.
-- Jason
# Copyright 1999-2018 Jason Zaman
# Distributed under the terms of the GNU General Public License v2
# @ECLASS: bazel.eclass
# @MAINTAINER:
# Jason Zaman
# @AUTHOR:
# Jason Zaman
# @BLURB: Utility functions for packages using Bazel Build
# @DESCRIPTION:
# A utility eclass providing functions to run the Bazel Build system.
#
# This eclass does not export any phase functions.
case "${EAPI:-0}" in
0|1|2|3|4|5|6)
die "Unsupported EAPI=${EAPI:-0} (too old) for ${ECLASS}"
;;
7)
;;
*)
die "Unsupported EAPI=${EAPI} (unknown) for ${ECLASS}"
;;
esac
if [[ ! ${_BAZEL_ECLASS} ]]; then
inherit multiprocessing toolchain-funcs
BDEPEND=">=dev-util/bazel-0.19"
# @FUNCTION: bazel_get_flags
# @DESCRIPTION:
# Obtain and print the bazel flags for target and host *FLAGS.
#
# To add more flags to this, append the flags to the
# appropriate variable before calling this function
bazel_get_flags() {
local i fs=()
for i in ${CFLAGS}; do
fs+=( "--conlyopt=${i}" )
done
for i in ${BUILD_CFLAGS}; do
fs+=( "--host_conlyopt=${i}" )
done
for i in ${CXXFLAGS}; do
fs+=( "--cxxopt=${i}" )
done
for i in ${BUILD_CXXFLAGS}; do
fs+=( "--host_cxxopt=${i}" )
done
for i in ${CPPFLAGS}; do
fs+=( "--conlyopt=${i}" "--cxxopt=${i}" )
done
for i in ${BUILD_CPPFLAGS}; do
fs+=( "--host_conlyopt=${i}" "--host_cxxopt=${i}" )
done
for i in ${LDFLAGS}; do
fs+=( "--linkopt=${i}" )
done
for i in ${BUILD_LDFLAGS}; do
fs+=( "--host_linkopt=${i}" )
done
echo "${fs[*]}"
}
# @FUNCTION: bazel_setup_bazelrc
# @DESCRIPTION:
# Creates the bazelrc with common options that will be passed
# to bazel. This will be called by ebazel automatically so
# does not need to be called from the ebuild.
bazel_setup_bazelrc() {
if [[ -f "${T}/bazelrc" ]]; then
return
fi
# F: fopen_wr
# P: /proc/self/setgroups
# Even with standalone enabled, the Bazel sandbox binary is run for
feature test:
#
https://github.com/bazelbuild/bazel/blob/7b091c1397a82258e26ab5336df6c8dae1d97384/src/main/java/com/google/devtools/build/lib/sandbox/LinuxSandboxedSpawnRunner.java#L61
#
https://github.com/bazelbuild/bazel/blob/76555482873ffcf1d32fb40106f89231b37f850a/src/main/tools/linux-sandbox-pid1.cc#L113
addpredict /proc
mkdir -p "${T}/bazel-cache" || die
mkdir -p "${T}/bazel-distdir" || die
cat > "${T}/bazelrc" <<-EOF || die
startup --batch
# dont strip HOME, portage sets a temp per-package dir
build --action_env HOME
# make bazel respect MAKEOPTS
build --jobs=$(makeopts_jobs)
build --compilation_mode=opt --host_compilation_mode=opt
# FLAGS
build $(bazel_get_flags)
# Use standalone strategy to deactivate the bazel sandbox, since it
# conflicts with FEATURES=sandbox.
build --spawn_strategy=standalone --genrule_strategy=standalone
test --spawn_strategy=standalone --genrule_strategy=standalone
build --strip=never
build --verbose_failures --noshow_loading_progress
test --verbose_test_summary --verbose_failures --noshow_loading_progress
# make bazel only fetch distfiles from the cache
fetch --repository_cache="${T}/bazel-cache/"
--distdir="${T}/bazel-distdir/"
build --repository_cache="${T}/bazel-cache/"
--distdir="${T}/bazel-distdir/"
build --define=PREFIX=${EPREFIX%/}/usr
build --define=LIBDIR=\$(PREFIX)/$(get_libdir)
EOF
tc-is-cross-compiler || \
echo "build --nodistinct_host_configuration" >> "${T}/bazelrc"
|| die
}
# @FUNCTION: ebazel
# @USAGE: [...]
# @DESCRIPTION:
# Run bazel with the bazelrc and output_base.
#
# If $MULTIBUILD_VARIANT is set, this will make an output_base
# specific to that variant.
# bazel_setup_bazelrc will be called and the created bazelrc
# will be passed to bazel.
#
# Will automatically die if bazel does not exit cleanly.
ebazel() {
bazel_setup_bazelrc
# Use different build folders for each multibuild variant.
local base_suffix="${MULTIBUILD_VARIANT+-}${MULTIBUILD_VARIANT}"
local output_base="${WORKDIR}/bazel-base${base_suffix}"
mkdir -p
Re: [gentoo-dev] [pre-GLEP] Gentoo binary package container format
On Sat, Nov 17, 2018 at 9:05 AM Roy Bamford wrote: > > Does this proposal allow for installing the payload without > the use of the Gentoo package manager from some random > distro being used as a rescue media? > Yes, it is a tarball of tarballs. There would be an extra step, but a vanilla tarball containing the files to be extracted could be extracted as long as you have tar and the appropriate decompressor (not specified and could change, but I imagine it will remain bzip2 for now). -- Rich
Re: [gentoo-dev] [pre-GLEP] Gentoo binary package container format
On 2018.11.17 11:21, Michał Górny wrote: > Hi, > > Here's a pre-GLEP draft based on the earlier discussion on gentoo- > portage-dev mailing list. The specification uses GLEP form as it > provides for cleanly specifying the motivation and rationale. > >[snip glep proposal] > -- > Best regards, > Michał Górny > Team, One of the attractions of the existing format is that tar xf /path/to/tarball -C /mnt/gentoo works to fix things like glibc being removed and other missing essential portage components. In effect, each binary package can be treated as a single package stage3 when a user needs a get out of jail free card. Does this proposal allow for installing the payload without the use of the Gentoo package manager from some random distro being used as a rescue media? -- Regards, Roy Bamford (Neddyseagoon) a member of elections gentoo-ops forum-mods pgpGykl9iWp_g.pgp Description: PGP signature
Re: [gentoo-dev] [PATCH v3 2/2] fortran-2.eclass: support EAPI 7
On Mon, 5 Nov 2018 18:37:55 +0300 Andrew Savchenko wrote: > Hi all! > > Here follow updated patches for fortran-2.eclass EAPI 7 update. > > Patch 2 contains only code cleanup and fixes unrelated to EAPI 7 > update: With no comments for ~12 days both patches are applied now. Best regards, Andrew Savchenko pgpZF42GEkxsK.pgp Description: PGP signature
[gentoo-dev] [pre-GLEP] Gentoo binary package container format
Hi, Here's a pre-GLEP draft based on the earlier discussion on gentoo- portage-dev mailing list. The specification uses GLEP form as it provides for cleanly specifying the motivation and rationale. (Note: the number assignment is not official, just took the next number to satisfy the glep converter script) Also available via HTTPS: rst: https://dev.gentoo.org/~mgorny/tmp/glep-0078.rst html: https://dev.gentoo.org/~mgorny/tmp/glep-0078.html --- GLEP: 78 Title: Gentoo binary package container format Author: Michał Górny Type: Standards Track Status: Draft Version: 1 Created: 2018-11-15 Last-Modified: 2018-11-16 Post-History: 2018-11-17 Content-Type: text/x-rst --- Abstract This GLEP proposes a new binary package container format for Gentoo. The current tbz2/XPAK format is shortly described, and its deficiences are listed. Accordingly, the requirements for a new format are set and a gpkg format satisfying them is proposed. The rationale for various design decisions is provided. Motivation == The current Portage binary package format - The historical ``.tbz2`` binary package format used by Portage is a concatenation of two distinct formats: header-oriented compressed .tar format (used to hold package files) and trailer-oriented custom XPAK format (used to hold metadata) [#MAN-XPAK]_. The format has already been extended incompatibly twice. The first time, support for storing multiple successive builds of binary package for a single ebuild version has been added. This feature relies on appending additional hyphen, followed by an integer to the package filename. It is disabled by default (preserving backwards compatibility) and controlled by ``binpkg-multi-instance`` feature. The second time, support for additional compression formats has been added. When format other than bzip2 is used, the ``.tbz2`` suffix is replaced by ``.xpak`` and Portage relies on magic bytes to detect compression used. For backwards compatibility, Portage still defaults to using bzip2; compression program can be switched using ``BINPKG_COMPRESS`` configuration variable. Additionally, there have been minor changes to the stored metadata and file storage policies. In particular, behavior regarding ``INSTALL_MASK``, controllable file compression and stripping has changed over time. Problems with the current binary package format --- The following problems were identified with the package format currently in use: 1. **The packages rely on custom binary archive format to store metadata.** It is entirely Gentoo invented, and requires dedicated tooling to work with it. In fact, the reference implementation in Portage does not even include a CLI tool to work with tbz2 packages; an unofficial implementation is provided as part of portage-utils toolkit [#PORTAGE-UTILS]_. 2. **The format relies on obscure compressor feature of ignoring trailing garbage**. While this behavior is traditionally implemented by many compressors, the original reasons for it have become long irrelevant and it is not surprising that new compressors do not support it. In particular, Portage already hit this problem twice: once when users replaced bzip2 with parallel-capable pbzip2 implementation [#PBZIP2]_, and the second time when support for zstd compressor was added [#ZSTD]_. 3. **Placing metadata at the end of file makes partial fetches complex.** While it is technically possible to obtain package metadata remotely without fetching the whole package, it usually requires e.g. 2-3 HTTP requests with rather complex driver. For comparison, if metadata was placed at the beginning of the file, early-terminated pipeline with a single fetch request would suffice. 4. **Extending the format with OpenPGP signatures is non-trivial.** Depending on the implementation details, it either requires fetching additional detached signature, breaking backwards compatibility or introducing more custom logic to reassemble OpenPGP packets. 5. **Metadata is not compressed.** This is not a significant problem, it is just listed for completeness. Goals for a new container format The following goals have been set for a replacement format: 1. **The packages must remain contained in a single file.** As a matter of user convenience, it should be possible to transfer binary packages without having to use multiple files, and to install them from any location. 2. **The file format must be entirely based on common file formats, respecting best practices, with as little customization as necessary to satisfy the requirements.** In particular, it is unacceptable to create new binary formats. 3. **The file format should provide for partial fetching of binary packages.** It should be possible to easily fetch and read the package metadata
Re: [gentoo-portage-dev] [PATCH] ecompress-file: Support decompressing .lz
On Sat, 2018-11-17 at 09:22 +0100, Ulrich Mueller wrote:
> > > > > > On Sat, 17 Nov 2018, Michał Górny wrote:
> > Signed-off-by: Michał Górny
> > ---
> > bin/ecompress-file | 3 +++
> > 1 file changed, 3 insertions(+)
> > diff --git a/bin/ecompress-file b/bin/ecompress-file
> > index bc8fe5451..ccc2701c3 100755
> > --- a/bin/ecompress-file
> > +++ b/bin/ecompress-file
> > @@ -29,6 +29,9 @@ compress_file() {
> > *.lzma|*.xz)
> > unxz -f "${x}" || __helpers_die "unxz failed"
> > x=${x%.*};;
> > + *.lz)
> > + lzip -df "${x}" || __helpers_die "lzip -d
> > failed"
> > + x=${x%.lz};;
> > esac
>
>
> > filtered_args+=( "$x" )
> > --
> > 2.19.1
>
> Does that mean that portage will gain a dependency on lzip now?
>
No, I don't think that's necessary. That code is a cheap unsupported
hack that throws QA warnings. I need it to support corner case of
PORTAGE_COMPRESS=lzip FEATURES=binpkg-docompress old binpackages after
switching to FEATURES=-binpkg-docompress.
--
Best regards,
Michał Górny
signature.asc
Description: This is a digitally signed message part
Re: [gentoo-portage-dev] [PATCH] ecompress-file: Support decompressing .lz
> On Sat, 17 Nov 2018, Michał Górny wrote:
> Signed-off-by: Michał Górny
> ---
> bin/ecompress-file | 3 +++
> 1 file changed, 3 insertions(+)
> diff --git a/bin/ecompress-file b/bin/ecompress-file
> index bc8fe5451..ccc2701c3 100755
> --- a/bin/ecompress-file
> +++ b/bin/ecompress-file
> @@ -29,6 +29,9 @@ compress_file() {
> *.lzma|*.xz)
> unxz -f "${x}" || __helpers_die "unxz failed"
> x=${x%.*};;
> + *.lz)
> + lzip -df "${x}" || __helpers_die "lzip -d
> failed"
> + x=${x%.lz};;
> esac
> filtered_args+=( "$x" )
> --
> 2.19.1
Does that mean that portage will gain a dependency on lzip now?
Ulrich
signature.asc
Description: PGP signature
[gentoo-portage-dev] [PATCH] ecompress-file: Support decompressing .lz
Signed-off-by: Michał Górny
---
bin/ecompress-file | 3 +++
1 file changed, 3 insertions(+)
diff --git a/bin/ecompress-file b/bin/ecompress-file
index bc8fe5451..ccc2701c3 100755
--- a/bin/ecompress-file
+++ b/bin/ecompress-file
@@ -29,6 +29,9 @@ compress_file() {
*.lzma|*.xz)
unxz -f "${x}" || __helpers_die "unxz failed"
x=${x%.*};;
+ *.lz)
+ lzip -df "${x}" || __helpers_die "lzip -d
failed"
+ x=${x%.lz};;
esac
filtered_args+=( "$x" )
--
2.19.1
