Re: [gentoo-portage-dev] [PATCH] Default BINPKG_COMPRESSION to zstd (bug 715108)
On 5/12/20 1:28 AM, Francesco Riosa wrote: > > Il 11/05/20 22:21, Brian Dolbec ha scritto: >> On Sun, 10 May 2020 19:29:34 -0700 >> Zac Medico wrote: >> >>> This includes a _compat_upgrade.binpkg_compression script that the >>> ebuild can call in pkg_preinst in order to maintain a >>> backward-compatible bzip2 default when appropriate, ensuring that >>> binary package consumers are not caught off guard. > > [snip] > > For your interest, the binpkg archive for a small LAMP container goes > from 659 to 540 MB, the "--rm" flag isn't really needed for binpkgs but > it help reminder that zstd is the only one that does _not_ remove the > original file after compression > > BINPKG_COMPRESS="zstd" > BINPKG_COMPRESS_FLAGS="--rm --long --threads=1 --adapt=min=7,max=19" I'm using those flag now. Thanks! -- Thanks, Zac signature.asc Description: OpenPGP digital signature
Re: [gentoo-portage-dev] [PATCH] Default BINPKG_COMPRESSION to zstd (bug 715108)
On 5/11/20 1:21 PM, Brian Dolbec wrote: >> diff --git a/man/make.conf.5 b/man/make.conf.5 >> index f82fed65a..a3bd662ae 100644 >> --- a/man/make.conf.5 >> +++ b/man/make.conf.5 >> @@ -1,4 +1,4 @@ >> -.TH "MAKE.CONF" "5" "Nov 2019" "Portage VERSION" "Portage" >> +.TH "MAKE.CONF" "5" "May 2020" "Portage VERSION" "Portage" >> .SH "NAME" >> make.conf \- custom settings for Portage >> .SH "SYNOPSIS" >> @@ -115,7 +115,7 @@ This variable is used to determine the >> compression used for \fIbinary packages\fR. Supported settings and >> compression algorithms are: bzip2, gzip, lz4, lzip, lzop, xz, zstd. >> .br >> -Defaults to "bzip2". >> +Defaults to "zstd". >> .br >> .I Example: >> .nf > looks good, I've not tested it, but changes are minor > I've merged this and released it in portage-2.3.100: https://gitweb.gentoo.org/proj/portage.git/commit/?id=230595cf600cae6beb6ebf6f817d08ace433c3ea -- Thanks, Zac signature.asc Description: OpenPGP digital signature
Re: [gentoo-portage-dev] [PATCH] Default BINPKG_COMPRESSION to zstd (bug 715108)
Il 11/05/20 22:21, Brian Dolbec ha scritto: On Sun, 10 May 2020 19:29:34 -0700 Zac Medico wrote: This includes a _compat_upgrade.binpkg_compression script that the ebuild can call in pkg_preinst in order to maintain a backward-compatible bzip2 default when appropriate, ensuring that binary package consumers are not caught off guard. [snip] For your interest, the binpkg archive for a small LAMP container goes from 659 to 540 MB, the "--rm" flag isn't really needed for binpkgs but it help reminder that zstd is the only one that does _not_ remove the original file after compression BINPKG_COMPRESS="zstd" BINPKG_COMPRESS_FLAGS="--rm --long --threads=1 --adapt=min=7,max=19" tails of the binpkgdir list: Before 659M total 3411658 Feb 13 16:03 media-fonts/corefonts-1-r7.tbz2 3421845 May 5 19:09 dev-python/virtualenv-16.7.9-r1.tbz2 3450323 May 3 08:54 dev-libs/nss-3.52.tbz2 3475698 May 5 19:09 app-portage/gentoolkit-0.4.8.tbz2 3545935 May 8 07:48 app-text/poppler-data-0.4.9.tbz2 3582965 May 10 23:08 dev-python/cython-0.29.15.tbz2 3870491 May 5 18:59 dev-libs/glib-2.62.6.tbz2 3889030 May 9 07:49 sys-apps/groff-1.22.3.tbz2 4225576 Apr 22 17:07 dev-libs/openssl-1.1.1g.tbz2 4374231 May 5 20:21 sys-libs/db-5.3.28-r2.tbz2 4889813 May 5 19:13 sys-devel/gdb-9.1.tbz2 6418288 Apr 3 11:13 app-editors/vim-core-8.2.0360.tbz2 7971823 May 5 18:49 dev-scheme/guile-2.2.4.tbz2 8841520 Apr 30 10:08 sys-devel/binutils-2.33.1-r1.tbz2 9076674 May 5 19:10 media-gfx/imagemagick-7.0.10.7-r1.tbz2 9201769 May 11 13:25 net-print/cups-2.3.3-r1.tbz2 9297311 May 1 08:13 dev-db/mysql-connector-c-8.0.20.tbz2 12229701 Apr 30 17:10 dev-util/cmake-3.16.5.tbz2 12977553 Apr 8 01:44 dev-libs/boost-1.72.0-r1.tbz2 13479702 May 5 18:55 dev-libs/icu-65.1-r1.tbz2 13788355 May 11 13:27 app-text/ghostscript-gpl-9.50.tbz2 14028322 Feb 27 01:55 dev-lang/perl-5.30.1.tbz2 14131663 May 5 19:11 net-dns/bind-tools-9.14.8.tbz2 14664302 May 5 19:05 sys-libs/glibc-2.30-r8.tbz2 15235721 Apr 22 17:09 dev-vcs/git-2.26.2.tbz2 25024737 May 5 18:52 dev-lang/python-3.7.7-r2.tbz2 26011846 May 10 23:09 app-admin/ansible-2.9.7.tbz2 26213157 May 5 18:53 dev-lang/python-3.8.2-r2.tbz2 34949529 May 5 19:15 dev-lang/php-7.3.17.tbz2 35364894 May 5 18:59 dev-lang/php-7.4.5.tbz2 48130470 May 5 19:09 dev-db/mariadb-10.4.12.tbz2 123472207 Apr 18 19:32 sys-devel/gcc-9.3.0.tbz2 After 540M total 3000375 May 12 00:44 media-fonts/corefonts-1-r7.tbz2 3010790 May 12 00:53 net-analyzer/net-snmp-5.8-r5.tbz2 3130214 May 12 00:38 dev-libs/nss-3.52.tbz2 3221436 May 12 00:23 media-fonts/urw-fonts-2.4.9.tbz2 3386623 May 12 01:04 app-portage/gentoolkit-0.4.8.tbz2 3541169 May 12 00:33 sys-apps/groff-1.22.3.tbz2 3675561 May 12 01:06 dev-libs/glib-2.62.6.tbz2 4305825 May 12 00:49 dev-libs/openssl-1.1.1g.tbz2 4828074 May 12 00:49 sys-devel/gdb-9.1.tbz2 6392717 May 12 01:04 app-editors/vim-core-8.2.0360.tbz2 6883802 May 12 00:54 net-dns/bind-tools-9.14.8.tbz2 7704310 May 12 00:16 sys-devel/binutils-2.33.1-r1.tbz2 8044941 May 12 00:43 dev-scheme/guile-2.2.4.tbz2 8442280 May 12 00:57 dev-vcs/git-2.26.2.tbz2 8945372 May 12 02:22 media-gfx/imagemagick-7.0.10.7-r1.tbz2 8989043 May 12 02:05 net-print/cups-2.3.3-r1.tbz2 10140244 May 12 02:08 dev-util/cmake-3.16.5.tbz2 12680888 May 12 02:03 sys-libs/glibc-2.30-r8.tbz2 12991659 May 12 00:56 dev-libs/icu-65.1-r1.tbz2 13634394 May 12 02:10 dev-libs/boost-1.72.0-r1.tbz2 13861694 May 12 00:27 dev-lang/perl-5.30.1.tbz2 13905742 May 12 02:12 app-text/ghostscript-gpl-9.50.tbz2 16061883 May 12 01:03 app-admin/ansible-2.9.7.tbz2 18244923 May 12 00:50 dev-lang/python-3.7.7-r2.tbz2 19061497 May 12 02:15 dev-db/mariadb-10.4.12.tbz2 19284388 May 12 00:52 dev-lang/python-3.8.2-r2.tbz2 30342169 May 12 02:22 dev-lang/php-7.3.17.tbz2 30767719 May 12 02:21 dev-lang/php-7.4.5.tbz2 100930187 May 12 02:01 sys-devel/gcc-9.3.0.tbz2
Re: [gentoo-portage-dev] [PATCH] Default BINPKG_COMPRESSION to zstd (bug 715108)
On Sun, 10 May 2020 19:29:34 -0700
Zac Medico wrote:
> This includes a _compat_upgrade.binpkg_compression script that the
> ebuild can call in pkg_preinst in order to maintain a
> backward-compatible bzip2 default when appropriate, ensuring that
> binary package consumers are not caught off guard.
>
> Bug: https://bugs.gentoo.org/715108
> Signed-off-by: Zac Medico
> ---
> .travis.yml | 4 ++
> cnf/make.globals | 5 ++-
> .../_compat_upgrade/binpkg_compression.py | 40
> +++ .../tests/resolver/ResolverPlayground.py |
> 1 + man/make.conf.5 | 4 +-
> 5 files changed, 50 insertions(+), 4 deletions(-)
> create mode 100644 lib/portage/_compat_upgrade/binpkg_compression.py
>
> diff --git a/.travis.yml b/.travis.yml
> index 9269d4034..2132c8c87 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -1,3 +1,4 @@
> +dist: bionic
> language: python
> python:
> - 2.7
> @@ -8,6 +9,9 @@ python:
> - pypy3
>
> # command to install dependencies
> +before_install:
> +# Use "dist: bionic" to get a zstd with --long support.
> +- sudo apt-get -y install zstd
> install:
> - pip install tox
>
> diff --git a/cnf/make.globals b/cnf/make.globals
> index 4a59dbe3c..dd3f28f70 100644
> --- a/cnf/make.globals
> +++ b/cnf/make.globals
> @@ -34,8 +34,9 @@ RPMDIR="/var/cache/rpm"
> # Temporary build directory
> PORTAGE_TMPDIR="/var/tmp"
>
> -# The compression used for binary packages. Defaults to zstd when
> USE=zstd is enabled. -BINPKG_COMPRESS="bzip2"
> +# The compression used for binary packages. Defaults to zstd except
> for +# existing installs where bzip2 is used for backward
> compatibility. +BINPKG_COMPRESS="zstd"
>
> # Fetching command (3 tries, passive ftp for firewall compatibility)
> FETCHCOMMAND="wget -t 3 -T 60 --passive-ftp -O
> \"\${DISTDIR}/\${FILE}\" \"\${URI}\"" diff --git
> a/lib/portage/_compat_upgrade/binpkg_compression.py
> b/lib/portage/_compat_upgrade/binpkg_compression.py new file mode
> 100644 index 0..0f5704733 --- /dev/null
> +++ b/lib/portage/_compat_upgrade/binpkg_compression.py
> @@ -0,0 +1,40 @@
> +# Copyright 2020 Gentoo Authors
> +# Distributed under the terms of the GNU General Public License v2
> +
> +import re
> +
> +import portage
> +from portage import os
> +from portage.const import GLOBAL_CONFIG_PATH
> +
> +COMPAT_BINPKG_COMPRESS = 'bzip2'
> +
> +
> +def main():
> + """
> + If the current installation is still configured to use the
> old
> + default BINPKG_COMPRESS=bzip2 setting, then patch
> make.globals
> + inside ${ED} to maintain backward compatibility, ensuring
> that
> + binary package consumers are not caught off guard. This is
> + intended to be called from the ebuild as follows:
> +
> + pkg_preinst() {
> + python_setup
> + env -u BINPKG_COMPRESS
> +
> PYTHONPATH="${D%/}$(python_get_sitedir)${PYTHONPATH:+:${PYTHONPATH}}"
> \
> + "${PYTHON}" -m
> portage._compat_upgrade.binpkg_compression || die
> + }
> + """
> + if portage.settings.get('BINPKG_COMPRESS',
> COMPAT_BINPKG_COMPRESS) == COMPAT_BINPKG_COMPRESS:
> + config_path = os.path.join(os.environ['ED'],
> GLOBAL_CONFIG_PATH.lstrip(os.sep), 'make.globals')
> + with open(config_path) as f:
> + content = f.read()
> + compat_setting =
> 'BINPKG_COMPRESS="{}"'.format(COMPAT_BINPKG_COMPRESS)
> + portage.output.EOutput().einfo('Setting
> make.globals default {} for backward
> compatibility'.format(compat_setting))
> + content = re.sub('^BINPKG_COMPRESS=.*$',
> compat_setting, content, flags=re.MULTILINE)
> + with open(config_path, 'wt') as f:
> + f.write(content)
> +
> +
> +if __name__ == '__main__':
> + main()
> diff --git a/lib/portage/tests/resolver/ResolverPlayground.py
> b/lib/portage/tests/resolver/ResolverPlayground.py index
> 98831e000..de80a0cc1 100644 ---
> a/lib/portage/tests/resolver/ResolverPlayground.py +++
> b/lib/portage/tests/resolver/ResolverPlayground.py @@ -112,6 +112,7
> @@ class ResolverPlayground(object): "uname",
> "uniq",
> "xargs",
> + "zstd",
> )
> # Exclude internal wrappers from PATH lookup.
> orig_path = os.environ['PATH']
> diff --git a/man/make.conf.5 b/man/make.conf.5
> index f82fed65a..a3bd662ae 100644
> --- a/man/make.conf.5
> +++ b/man/make.conf.5
> @@ -1,4 +1,4 @@
> -.TH "MAKE.CONF" "5" "Nov 2019" "Portage VERSION" "Portage"
> +.TH "MAKE.CONF" "5" "May 2020" "Portage VERSION" "Portage"
> .SH "NAME"
> make.conf \- custom settings for Portage
> .SH "SYNOPSIS"
> @@ -115,7 +115,7 @@ This variable is used to determine the
> compression used for \fIbinary packages\fR.
[gentoo-portage-dev] [PATCH] Default BINPKG_COMPRESSION to zstd (bug 715108)
This includes a _compat_upgrade.binpkg_compression script that the
ebuild can call in pkg_preinst in order to maintain a backward-compatible
bzip2 default when appropriate, ensuring that binary package consumers
are not caught off guard.
Bug: https://bugs.gentoo.org/715108
Signed-off-by: Zac Medico
---
.travis.yml | 4 ++
cnf/make.globals | 5 ++-
.../_compat_upgrade/binpkg_compression.py | 40 +++
.../tests/resolver/ResolverPlayground.py | 1 +
man/make.conf.5 | 4 +-
5 files changed, 50 insertions(+), 4 deletions(-)
create mode 100644 lib/portage/_compat_upgrade/binpkg_compression.py
diff --git a/.travis.yml b/.travis.yml
index 9269d4034..2132c8c87 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -1,3 +1,4 @@
+dist: bionic
language: python
python:
- 2.7
@@ -8,6 +9,9 @@ python:
- pypy3
# command to install dependencies
+before_install:
+# Use "dist: bionic" to get a zstd with --long support.
+- sudo apt-get -y install zstd
install:
- pip install tox
diff --git a/cnf/make.globals b/cnf/make.globals
index 4a59dbe3c..dd3f28f70 100644
--- a/cnf/make.globals
+++ b/cnf/make.globals
@@ -34,8 +34,9 @@ RPMDIR="/var/cache/rpm"
# Temporary build directory
PORTAGE_TMPDIR="/var/tmp"
-# The compression used for binary packages. Defaults to zstd when USE=zstd is
enabled.
-BINPKG_COMPRESS="bzip2"
+# The compression used for binary packages. Defaults to zstd except for
+# existing installs where bzip2 is used for backward compatibility.
+BINPKG_COMPRESS="zstd"
# Fetching command (3 tries, passive ftp for firewall compatibility)
FETCHCOMMAND="wget -t 3 -T 60 --passive-ftp -O \"\${DISTDIR}/\${FILE}\"
\"\${URI}\""
diff --git a/lib/portage/_compat_upgrade/binpkg_compression.py
b/lib/portage/_compat_upgrade/binpkg_compression.py
new file mode 100644
index 0..0f5704733
--- /dev/null
+++ b/lib/portage/_compat_upgrade/binpkg_compression.py
@@ -0,0 +1,40 @@
+# Copyright 2020 Gentoo Authors
+# Distributed under the terms of the GNU General Public License v2
+
+import re
+
+import portage
+from portage import os
+from portage.const import GLOBAL_CONFIG_PATH
+
+COMPAT_BINPKG_COMPRESS = 'bzip2'
+
+
+def main():
+ """
+ If the current installation is still configured to use the old
+ default BINPKG_COMPRESS=bzip2 setting, then patch make.globals
+ inside ${ED} to maintain backward compatibility, ensuring that
+ binary package consumers are not caught off guard. This is
+ intended to be called from the ebuild as follows:
+
+ pkg_preinst() {
+ python_setup
+ env -u BINPKG_COMPRESS
+
PYTHONPATH="${D%/}$(python_get_sitedir)${PYTHONPATH:+:${PYTHONPATH}}" \
+ "${PYTHON}" -m
portage._compat_upgrade.binpkg_compression || die
+ }
+ """
+ if portage.settings.get('BINPKG_COMPRESS', COMPAT_BINPKG_COMPRESS) ==
COMPAT_BINPKG_COMPRESS:
+ config_path = os.path.join(os.environ['ED'],
GLOBAL_CONFIG_PATH.lstrip(os.sep), 'make.globals')
+ with open(config_path) as f:
+ content = f.read()
+ compat_setting =
'BINPKG_COMPRESS="{}"'.format(COMPAT_BINPKG_COMPRESS)
+ portage.output.EOutput().einfo('Setting make.globals
default {} for backward compatibility'.format(compat_setting))
+ content = re.sub('^BINPKG_COMPRESS=.*$',
compat_setting, content, flags=re.MULTILINE)
+ with open(config_path, 'wt') as f:
+ f.write(content)
+
+
+if __name__ == '__main__':
+ main()
diff --git a/lib/portage/tests/resolver/ResolverPlayground.py
b/lib/portage/tests/resolver/ResolverPlayground.py
index 98831e000..de80a0cc1 100644
--- a/lib/portage/tests/resolver/ResolverPlayground.py
+++ b/lib/portage/tests/resolver/ResolverPlayground.py
@@ -112,6 +112,7 @@ class ResolverPlayground(object):
"uname",
"uniq",
"xargs",
+ "zstd",
)
# Exclude internal wrappers from PATH lookup.
orig_path = os.environ['PATH']
diff --git a/man/make.conf.5 b/man/make.conf.5
index f82fed65a..a3bd662ae 100644
--- a/man/make.conf.5
+++ b/man/make.conf.5
@@ -1,4 +1,4 @@
-.TH "MAKE.CONF" "5" "Nov 2019" "Portage VERSION" "Portage"
+.TH "MAKE.CONF" "5" "May 2020" "Portage VERSION" "Portage"
.SH "NAME"
make.conf \- custom settings for Portage
.SH "SYNOPSIS"
@@ -115,7 +115,7 @@ This variable is used to determine the compression used for
\fIbinary
packages\fR. Supported settings and compression algorithms are: bzip2, gzip,
lz4, lzip, lzop, xz, zstd.
.br
-Defaults to "bzip2".
+Defaults to "zstd".
.br
.I Example:
.nf
--
2.25.3
