Re: [gentoo-portage-dev] [PATCH] Support different (de)compressors for binary packages
On Wed, Jul 5, 2017 at 1:03 AM, Michał Górny wrote:
> On wto, 2017-07-04 at 23:33 +0200, Manuel Rüger wrote:
>> On 04.07.2017 21:29, Zac Medico wrote:
>> > On Fri, Jun 30, 2017 at 2:49 AM, Manuel Rüger wrote:
>> > > +
>> > > COMPRESSION_COMMAND=$(PYTHONPATH=${PORTAGE_PYTHONPATH:-${PORTAGE_PYM_PATH}}
>> > > \
>> > > + "${PORTAGE_PYTHON:-/usr/bin/python}"
>> > > "$PORTAGE_BIN_PATH"/binpkg-helper.py \
>> > > + compressioncmd ${CATEGORY}/${P})
>> > > + [ -z "${COMPRESSION_COMMAND}" ] && \
>> > > + die "Failed to get COMPRESSION_COMMAND"
>> > > tar $tar_options -cf - $PORTAGE_BINPKG_TAR_OPTS -C "${PROOT}" .
>> > > | \
>> > > - $PORTAGE_BZIP2_COMMAND -c > "$PORTAGE_BINPKG_TMPFILE"
>> > > + $COMPRESSION_COMMAND -c > "$PORTAGE_BINPKG_TMPFILE"
>> > > assert "failed to pack binary package: '$PORTAGE_BINPKG_TMPFILE'"
>> > > PYTHONPATH=${PORTAGE_PYTHONPATH:-${PORTAGE_PYM_PATH}} \
>> > > "${PORTAGE_PYTHON:-/usr/bin/python}"
>> >
>> > If all that we really need is COMPRESSION_COMMAND, then the helper
>> > script is overkill. We should just pass a variable from
>> > doebuild_environment function. The variable name must be prefixed with
>> > PORTAGE_.
>> >
>> > Also, note that your mail client wrapped lines in this patch.
>> >
>>
>> Thanks for the review Zac!
>>
>> We'd need a bit more as the COMPRESSION_COMMAND depends on
>> ${CATEGORY}/${P} here in order to avoid a catch22 when using a
>> decompressor that is set to something a standard install doesn't include.
>>
>> Assume all binpkgs are set to be compressed with zstd, the patch makes
>> sure an app-arch/zstd binpkg will still be compressed with bzip2.
>
> And app-arch/bzip2 binpackage?
>
> In other words, that's the wrong approach. If I set binpackages to use
> X, I want them to use X, not 'use X if someone thought it correct to use
> X'. Just do what the user says to do. If you try to outsmart him, you're
> building another Windows.
Yeah, people can use package.env if they want per-package compression settings.
--
Thanks,
Zac
Re: [gentoo-portage-dev] [PATCH] Support different (de)compressors for binary packages
On wto, 2017-07-04 at 23:33 +0200, Manuel Rüger wrote:
> On 04.07.2017 21:29, Zac Medico wrote:
> > On Fri, Jun 30, 2017 at 2:49 AM, Manuel Rüger wrote:
> > > +
> > > COMPRESSION_COMMAND=$(PYTHONPATH=${PORTAGE_PYTHONPATH:-${PORTAGE_PYM_PATH}}
> > > \
> > > + "${PORTAGE_PYTHON:-/usr/bin/python}"
> > > "$PORTAGE_BIN_PATH"/binpkg-helper.py \
> > > + compressioncmd ${CATEGORY}/${P})
> > > + [ -z "${COMPRESSION_COMMAND}" ] && \
> > > + die "Failed to get COMPRESSION_COMMAND"
> > > tar $tar_options -cf - $PORTAGE_BINPKG_TAR_OPTS -C "${PROOT}" . |
> > > \
> > > - $PORTAGE_BZIP2_COMMAND -c > "$PORTAGE_BINPKG_TMPFILE"
> > > + $COMPRESSION_COMMAND -c > "$PORTAGE_BINPKG_TMPFILE"
> > > assert "failed to pack binary package: '$PORTAGE_BINPKG_TMPFILE'"
> > > PYTHONPATH=${PORTAGE_PYTHONPATH:-${PORTAGE_PYM_PATH}} \
> > > "${PORTAGE_PYTHON:-/usr/bin/python}"
> >
> > If all that we really need is COMPRESSION_COMMAND, then the helper
> > script is overkill. We should just pass a variable from
> > doebuild_environment function. The variable name must be prefixed with
> > PORTAGE_.
> >
> > Also, note that your mail client wrapped lines in this patch.
> >
>
> Thanks for the review Zac!
>
> We'd need a bit more as the COMPRESSION_COMMAND depends on
> ${CATEGORY}/${P} here in order to avoid a catch22 when using a
> decompressor that is set to something a standard install doesn't include.
>
> Assume all binpkgs are set to be compressed with zstd, the patch makes
> sure an app-arch/zstd binpkg will still be compressed with bzip2.
And app-arch/bzip2 binpackage?
In other words, that's the wrong approach. If I set binpackages to use
X, I want them to use X, not 'use X if someone thought it correct to use
X'. Just do what the user says to do. If you try to outsmart him, you're
building another Windows.
--
Best regards,
Michał Górny
signature.asc
Description: This is a digitally signed message part
Re: [gentoo-portage-dev] [PATCH] Support different (de)compressors for binary packages
On Tue, Jul 4, 2017 at 2:33 PM, Manuel Rüger wrote:
> On 04.07.2017 21:29, Zac Medico wrote:
>> On Fri, Jun 30, 2017 at 2:49 AM, Manuel Rüger wrote:
>>> +
>>> COMPRESSION_COMMAND=$(PYTHONPATH=${PORTAGE_PYTHONPATH:-${PORTAGE_PYM_PATH}}
>>> \
>>> + "${PORTAGE_PYTHON:-/usr/bin/python}"
>>> "$PORTAGE_BIN_PATH"/binpkg-helper.py \
>>> + compressioncmd ${CATEGORY}/${P})
>>> + [ -z "${COMPRESSION_COMMAND}" ] && \
>>> + die "Failed to get COMPRESSION_COMMAND"
>>> tar $tar_options -cf - $PORTAGE_BINPKG_TAR_OPTS -C "${PROOT}" . | \
>>> - $PORTAGE_BZIP2_COMMAND -c > "$PORTAGE_BINPKG_TMPFILE"
>>> + $COMPRESSION_COMMAND -c > "$PORTAGE_BINPKG_TMPFILE"
>>> assert "failed to pack binary package: '$PORTAGE_BINPKG_TMPFILE'"
>>> PYTHONPATH=${PORTAGE_PYTHONPATH:-${PORTAGE_PYM_PATH}} \
>>> "${PORTAGE_PYTHON:-/usr/bin/python}"
>>
>> If all that we really need is COMPRESSION_COMMAND, then the helper
>> script is overkill. We should just pass a variable from
>> doebuild_environment function. The variable name must be prefixed with
>> PORTAGE_.
>>
>> Also, note that your mail client wrapped lines in this patch.
>>
>
> Thanks for the review Zac!
>
> We'd need a bit more as the COMPRESSION_COMMAND depends on
> ${CATEGORY}/${P} here in order to avoid a catch22 when using a
> decompressor that is set to something a standard install doesn't include.
>
> Assume all binpkgs are set to be compressed with zstd, the patch makes
> sure an app-arch/zstd binpkg will still be compressed with bzip2.
Inside the doebuild_environment function, there's a mycpv, mypv, and
cat variables available to use, and it's also possible to access
relevant package.env variables via mysettings.
> Mails were sent with Thunderbird, I'll set up git send-mail next time.
Okay, thanks.
--
Thanks,
Zac
Re: [gentoo-portage-dev] [PATCH] Support different (de)compressors for binary packages
On 04.07.2017 21:29, Zac Medico wrote:
> On Fri, Jun 30, 2017 at 2:49 AM, Manuel Rüger wrote:
>> +
>> COMPRESSION_COMMAND=$(PYTHONPATH=${PORTAGE_PYTHONPATH:-${PORTAGE_PYM_PATH}}
>> \
>> + "${PORTAGE_PYTHON:-/usr/bin/python}"
>> "$PORTAGE_BIN_PATH"/binpkg-helper.py \
>> + compressioncmd ${CATEGORY}/${P})
>> + [ -z "${COMPRESSION_COMMAND}" ] && \
>> + die "Failed to get COMPRESSION_COMMAND"
>> tar $tar_options -cf - $PORTAGE_BINPKG_TAR_OPTS -C "${PROOT}" . | \
>> - $PORTAGE_BZIP2_COMMAND -c > "$PORTAGE_BINPKG_TMPFILE"
>> + $COMPRESSION_COMMAND -c > "$PORTAGE_BINPKG_TMPFILE"
>> assert "failed to pack binary package: '$PORTAGE_BINPKG_TMPFILE'"
>> PYTHONPATH=${PORTAGE_PYTHONPATH:-${PORTAGE_PYM_PATH}} \
>> "${PORTAGE_PYTHON:-/usr/bin/python}"
>
> If all that we really need is COMPRESSION_COMMAND, then the helper
> script is overkill. We should just pass a variable from
> doebuild_environment function. The variable name must be prefixed with
> PORTAGE_.
>
> Also, note that your mail client wrapped lines in this patch.
>
Thanks for the review Zac!
We'd need a bit more as the COMPRESSION_COMMAND depends on
${CATEGORY}/${P} here in order to avoid a catch22 when using a
decompressor that is set to something a standard install doesn't include.
Assume all binpkgs are set to be compressed with zstd, the patch makes
sure an app-arch/zstd binpkg will still be compressed with bzip2.
Mails were sent with Thunderbird, I'll set up git send-mail next time.
signature.asc
Description: OpenPGP digital signature
Re: [gentoo-portage-dev] [PATCH] Support different (de)compressors for binary packages
On Fri, Jun 30, 2017 at 2:49 AM, Manuel Rüger wrote:
> +
> COMPRESSION_COMMAND=$(PYTHONPATH=${PORTAGE_PYTHONPATH:-${PORTAGE_PYM_PATH}}
> \
> + "${PORTAGE_PYTHON:-/usr/bin/python}"
> "$PORTAGE_BIN_PATH"/binpkg-helper.py \
> + compressioncmd ${CATEGORY}/${P})
> + [ -z "${COMPRESSION_COMMAND}" ] && \
> + die "Failed to get COMPRESSION_COMMAND"
> tar $tar_options -cf - $PORTAGE_BINPKG_TAR_OPTS -C "${PROOT}" . | \
> - $PORTAGE_BZIP2_COMMAND -c > "$PORTAGE_BINPKG_TMPFILE"
> + $COMPRESSION_COMMAND -c > "$PORTAGE_BINPKG_TMPFILE"
> assert "failed to pack binary package: '$PORTAGE_BINPKG_TMPFILE'"
> PYTHONPATH=${PORTAGE_PYTHONPATH:-${PORTAGE_PYM_PATH}} \
> "${PORTAGE_PYTHON:-/usr/bin/python}"
If all that we really need is COMPRESSION_COMMAND, then the helper
script is overkill. We should just pass a variable from
doebuild_environment function. The variable name must be prefixed with
PORTAGE_.
Also, note that your mail client wrapped lines in this patch.
--
Thanks,
Zac
[gentoo-portage-dev] [PATCH] Support different (de)compressors for binary packages
This patch allows to set the compressor for binary packages via a
BINPKG_COMPRESSION variable. BINPKG_COMPRESSION_ARGS allows to specify
command-line arguments for that compressor.
---
bin/binpkg-helper.py | 91
+++
bin/misc-functions.sh | 9 +++-
bin/quickpkg | 67 +++---
man/make.conf.5 | 25 ++
pym/_emerge/BinpkgExtractorAsync.py | 43 +++--
pym/portage/dbapi/bintree.py | 8 +--
pym/portage/util/compression_probe.py | 45 ++---
7 files changed, 253 insertions(+), 35 deletions(-)
create mode 100755 bin/binpkg-helper.py
diff --git a/bin/binpkg-helper.py b/bin/binpkg-helper.py
new file mode 100755
index 0..b603747cf
--- /dev/null
+++ b/bin/binpkg-helper.py
@@ -0,0 +1,91 @@
+#!/usr/bin/python -b
+# Copyright 2009-2017 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+import argparse
+import sys
+
+import portage
+portage._internal_caller = True
+from portage import os
+from portage import cpv_getkey
+from portage.process import find_binary
+from portage.util import (
+ shlex_split,
+ varexpand,
+ )
+from portage.util.compression_probe import _compressors
+
+def command_compressioncmd(args):
+ settings = portage.settings
+ usage = "usage: compressioncmd ${CATEGORY}/${P}\n"
+
+ if len(args) != 1:
+ sys.stderr.write(usage)
+ sys.stderr.write("One argument is required, got %s\n" %
len(args))
+ return 1
+
+ cpv = args[0]
+ binpkg_compression = settings.get("BINPKG_COMPRESSION", "bzip2")
+ try:
+ compression = _compressors[binpkg_compression]
+ except KeyError as e:
+ sys.stderr.write("Invalid or unsupported compression method:
%s" %
e.args[0])
+ return 1
+ # Fallback to bzip2 for the package providing the decompressor
+ # This solves bootstrapping a client without the decompressor
+
+ if cpv_getkey(cpv) is None:
+ sys.stderr.write("The argument must be in the
${CATEGORY}/${PN}-${PV}
format. Provided argument was: %s\n" % cpv)
+ return 1
+
+ if cpv_getkey(cpv) == compression["package"]:
+ compression = _compressors["bzip2"]
+ try:
+ compression_binary =
shlex_split(varexpand(compression["compress"],
mydict=settings))[0]
+ except IndexError as e:
+ sys.stderr.write("Invalid or unsupported compression method:
%s" %
e.args[0])
+ return 1
+ if find_binary(compression_binary) is None:
+ missing_package = compression["package"]
+ sys.stderr.write("File compression unsupported %s. Missing
package:
%s" % (binpkg_compression, missing_package))
+ return 1
+ cmd = [varexpand(x, mydict=settings) for x in
shlex_split(compression["compress"])]
+ # Filter empty elements
+ cmd = [x for x in cmd if x != ""]
+
+ print(' '.join(cmd))
+ return os.EX_OK
+
+def main(argv):
+
+ if argv and isinstance(argv[0], bytes):
+ for i, x in enumerate(argv):
+ argv[i] = portage._unicode_decode(x, errors='strict')
+
+ valid_commands = ('compressioncmd',)
+ description = "Returns the compression command"
+ usage = "usage: %s COMMAND [args]" % \
+ os.path.basename(argv[0])
+
+ parser = argparse.ArgumentParser(description=description, usage=usage)
+ options, args = parser.parse_known_args(argv[1:])
+
+ if not args:
+ parser.error("missing command argument")
+
+ command = args[0]
+
+ if command not in valid_commands:
+ parser.error("invalid command: '%s'" % command)
+
+ if command == 'compressioncmd':
+ rval = command_compressioncmd(args[1:])
+ else:
+ raise AssertionError("invalid command: '%s'" % command)
+
+ return rval
+
+if __name__ == "__main__":
+ rval = main(sys.argv[:])
+ sys.exit(rval)
diff --git a/bin/misc-functions.sh b/bin/misc-functions.sh
index 58755a1e1..0ba0db226 100755
--- a/bin/misc-functions.sh
+++ b/bin/misc-functions.sh
@@ -453,7 +453,7 @@ __dyn_package() {
# Make sure $PWD is not ${D} so that we don't leave gmon.out files
# in there in case any tools were built with -pg in CFLAGS.
- cd "${T}"
+ cd "${T}" || die
if [[ -n ${PKG_INSTALL_MASK} ]] ; then
PROOT=${T}/packaging/
@@ -478,8 +478,13 @@ __dyn_package() {
[ -z "${PORTAGE_BINPKG_TMPFILE}" ] && \
die "PORTAGE_BINPKG_TMPFILE is unset"
mkdir -p "${PORTAGE_BINPKG_TMPFILE%/*}" || die "mkdir failed"
+
COMPRESSION_COMMAND=$(PYTHONPATH=${PORTAGE_PYTHONPATH:-${PORTAGE_PYM_PATH}}
\
+ "${PORTAGE_PYTHON:-/usr/bin/python}"
