Re: [gentoo-portage-dev] [PATCH] Default BINPKG_COMPRESSION to zstd (bug 715108)

[Zac Medico]

On 5/12/20 1:28 AM, Francesco Riosa wrote:
> 
> Il 11/05/20 22:21, Brian Dolbec ha scritto:
>> On Sun, 10 May 2020 19:29:34 -0700
>> Zac Medico  wrote:
>>
>>> This includes a _compat_upgrade.binpkg_compression script that the
>>> ebuild can call in pkg_preinst in order to maintain a
>>> backward-compatible bzip2 default when appropriate, ensuring that
>>> binary package consumers are not caught off guard.
> 
> [snip]
> 
> For your interest, the binpkg archive for a small LAMP container goes
> from 659 to 540 MB, the "--rm" flag isn't really needed for binpkgs but
> it help reminder that zstd is the only one that does _not_ remove the
> original file after compression
> 
> BINPKG_COMPRESS="zstd"
> BINPKG_COMPRESS_FLAGS="--rm --long --threads=1 --adapt=min=7,max=19"

I'm using those flag now. Thanks!
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-portage-dev] [PATCH] Default BINPKG_COMPRESSION to zstd (bug 715108)

[Zac Medico]

On 5/11/20 1:21 PM, Brian Dolbec wrote:
>> diff --git a/man/make.conf.5 b/man/make.conf.5
>> index f82fed65a..a3bd662ae 100644
>> --- a/man/make.conf.5
>> +++ b/man/make.conf.5
>> @@ -1,4 +1,4 @@
>> -.TH "MAKE.CONF" "5" "Nov 2019" "Portage VERSION" "Portage"
>> +.TH "MAKE.CONF" "5" "May 2020" "Portage VERSION" "Portage"
>>  .SH "NAME"
>>  make.conf \- custom settings for Portage
>>  .SH "SYNOPSIS"
>> @@ -115,7 +115,7 @@ This variable is used to determine the
>> compression used for \fIbinary packages\fR. Supported settings and
>> compression algorithms are: bzip2, gzip, lz4, lzip, lzop, xz, zstd.
>>  .br
>> -Defaults to "bzip2".
>> +Defaults to "zstd".
>>  .br
>>  .I Example:
>>  .nf
> looks good, I've not tested it, but changes are minor
> 

I've merged this and released it in portage-2.3.100:

https://gitweb.gentoo.org/proj/portage.git/commit/?id=230595cf600cae6beb6ebf6f817d08ace433c3ea
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-portage-dev] [PATCH] Default BINPKG_COMPRESSION to zstd (bug 715108)

[Francesco Riosa]

Il 11/05/20 22:21, Brian Dolbec ha scritto:

On Sun, 10 May 2020 19:29:34 -0700
Zac Medico  wrote:


This includes a _compat_upgrade.binpkg_compression script that the
ebuild can call in pkg_preinst in order to maintain a
backward-compatible bzip2 default when appropriate, ensuring that
binary package consumers are not caught off guard.


[snip]

For your interest, the binpkg archive for a small LAMP container goes 
from 659 to 540 MB, the "--rm" flag isn't really needed for binpkgs but 
it help reminder that zstd is the only one that does _not_ remove the 
original file after compression


BINPKG_COMPRESS="zstd"
BINPKG_COMPRESS_FLAGS="--rm --long --threads=1 --adapt=min=7,max=19"

tails of the binpkgdir list:

Before 659M total
  3411658 Feb 13 16:03 media-fonts/corefonts-1-r7.tbz2
  3421845 May  5 19:09 dev-python/virtualenv-16.7.9-r1.tbz2
  3450323 May  3 08:54 dev-libs/nss-3.52.tbz2
  3475698 May  5 19:09 app-portage/gentoolkit-0.4.8.tbz2
  3545935 May  8 07:48 app-text/poppler-data-0.4.9.tbz2
  3582965 May 10 23:08 dev-python/cython-0.29.15.tbz2
  3870491 May  5 18:59 dev-libs/glib-2.62.6.tbz2
  3889030 May  9 07:49 sys-apps/groff-1.22.3.tbz2
  4225576 Apr 22 17:07 dev-libs/openssl-1.1.1g.tbz2
  4374231 May  5 20:21 sys-libs/db-5.3.28-r2.tbz2
  4889813 May  5 19:13 sys-devel/gdb-9.1.tbz2
  6418288 Apr  3 11:13 app-editors/vim-core-8.2.0360.tbz2
  7971823 May  5 18:49 dev-scheme/guile-2.2.4.tbz2
  8841520 Apr 30 10:08 sys-devel/binutils-2.33.1-r1.tbz2
  9076674 May  5 19:10 media-gfx/imagemagick-7.0.10.7-r1.tbz2
  9201769 May 11 13:25 net-print/cups-2.3.3-r1.tbz2
  9297311 May  1 08:13 dev-db/mysql-connector-c-8.0.20.tbz2
 12229701 Apr 30 17:10 dev-util/cmake-3.16.5.tbz2
 12977553 Apr  8 01:44 dev-libs/boost-1.72.0-r1.tbz2
 13479702 May  5 18:55 dev-libs/icu-65.1-r1.tbz2
 13788355 May 11 13:27 app-text/ghostscript-gpl-9.50.tbz2
 14028322 Feb 27 01:55 dev-lang/perl-5.30.1.tbz2
 14131663 May  5 19:11 net-dns/bind-tools-9.14.8.tbz2
 14664302 May  5 19:05 sys-libs/glibc-2.30-r8.tbz2
 15235721 Apr 22 17:09 dev-vcs/git-2.26.2.tbz2
 25024737 May  5 18:52 dev-lang/python-3.7.7-r2.tbz2
 26011846 May 10 23:09 app-admin/ansible-2.9.7.tbz2
 26213157 May  5 18:53 dev-lang/python-3.8.2-r2.tbz2
 34949529 May  5 19:15 dev-lang/php-7.3.17.tbz2
 35364894 May  5 18:59 dev-lang/php-7.4.5.tbz2
 48130470 May  5 19:09 dev-db/mariadb-10.4.12.tbz2
123472207 Apr 18 19:32 sys-devel/gcc-9.3.0.tbz2

After 540M total
  3000375 May 12 00:44 media-fonts/corefonts-1-r7.tbz2
  3010790 May 12 00:53 net-analyzer/net-snmp-5.8-r5.tbz2
  3130214 May 12 00:38 dev-libs/nss-3.52.tbz2
  3221436 May 12 00:23 media-fonts/urw-fonts-2.4.9.tbz2
  3386623 May 12 01:04 app-portage/gentoolkit-0.4.8.tbz2
  3541169 May 12 00:33 sys-apps/groff-1.22.3.tbz2
  3675561 May 12 01:06 dev-libs/glib-2.62.6.tbz2
  4305825 May 12 00:49 dev-libs/openssl-1.1.1g.tbz2
  4828074 May 12 00:49 sys-devel/gdb-9.1.tbz2
  6392717 May 12 01:04 app-editors/vim-core-8.2.0360.tbz2
  6883802 May 12 00:54 net-dns/bind-tools-9.14.8.tbz2
  7704310 May 12 00:16 sys-devel/binutils-2.33.1-r1.tbz2
  8044941 May 12 00:43 dev-scheme/guile-2.2.4.tbz2
  8442280 May 12 00:57 dev-vcs/git-2.26.2.tbz2
  8945372 May 12 02:22 media-gfx/imagemagick-7.0.10.7-r1.tbz2
  8989043 May 12 02:05 net-print/cups-2.3.3-r1.tbz2
 10140244 May 12 02:08 dev-util/cmake-3.16.5.tbz2
 12680888 May 12 02:03 sys-libs/glibc-2.30-r8.tbz2
 12991659 May 12 00:56 dev-libs/icu-65.1-r1.tbz2
 13634394 May 12 02:10 dev-libs/boost-1.72.0-r1.tbz2
 13861694 May 12 00:27 dev-lang/perl-5.30.1.tbz2
 13905742 May 12 02:12 app-text/ghostscript-gpl-9.50.tbz2
 16061883 May 12 01:03 app-admin/ansible-2.9.7.tbz2
 18244923 May 12 00:50 dev-lang/python-3.7.7-r2.tbz2
 19061497 May 12 02:15 dev-db/mariadb-10.4.12.tbz2
 19284388 May 12 00:52 dev-lang/python-3.8.2-r2.tbz2
 30342169 May 12 02:22 dev-lang/php-7.3.17.tbz2
 30767719 May 12 02:21 dev-lang/php-7.4.5.tbz2
100930187 May 12 02:01 sys-devel/gcc-9.3.0.tbz2

Re: [gentoo-portage-dev] [PATCH] Default BINPKG_COMPRESSION to zstd (bug 715108)

[Brian Dolbec]

On Sun, 10 May 2020 19:29:34 -0700
Zac Medico  wrote:

> This includes a _compat_upgrade.binpkg_compression script that the
> ebuild can call in pkg_preinst in order to maintain a
> backward-compatible bzip2 default when appropriate, ensuring that
> binary package consumers are not caught off guard.
> 
> Bug: https://bugs.gentoo.org/715108
> Signed-off-by: Zac Medico 
> ---
>  .travis.yml   |  4 ++
>  cnf/make.globals  |  5 ++-
>  .../_compat_upgrade/binpkg_compression.py | 40
> +++ .../tests/resolver/ResolverPlayground.py  |
> 1 + man/make.conf.5   |  4 +-
>  5 files changed, 50 insertions(+), 4 deletions(-)
>  create mode 100644 lib/portage/_compat_upgrade/binpkg_compression.py
> 
> diff --git a/.travis.yml b/.travis.yml
> index 9269d4034..2132c8c87 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -1,3 +1,4 @@
> +dist: bionic
>  language: python
>  python:
>  - 2.7
> @@ -8,6 +9,9 @@ python:
>  - pypy3
>  
>  # command to install dependencies
> +before_install:
> +# Use "dist: bionic" to get a zstd with --long support.
> +- sudo apt-get -y install zstd
>  install:
>  - pip install tox
>  
> diff --git a/cnf/make.globals b/cnf/make.globals
> index 4a59dbe3c..dd3f28f70 100644
> --- a/cnf/make.globals
> +++ b/cnf/make.globals
> @@ -34,8 +34,9 @@ RPMDIR="/var/cache/rpm"
>  # Temporary build directory
>  PORTAGE_TMPDIR="/var/tmp"
>  
> -# The compression used for binary packages. Defaults to zstd when
> USE=zstd is enabled. -BINPKG_COMPRESS="bzip2"
> +# The compression used for binary packages. Defaults to zstd except
> for +# existing installs where bzip2 is used for backward
> compatibility. +BINPKG_COMPRESS="zstd"
>  
>  # Fetching command (3 tries, passive ftp for firewall compatibility)
>  FETCHCOMMAND="wget -t 3 -T 60 --passive-ftp -O
> \"\${DISTDIR}/\${FILE}\" \"\${URI}\"" diff --git
> a/lib/portage/_compat_upgrade/binpkg_compression.py
> b/lib/portage/_compat_upgrade/binpkg_compression.py new file mode
> 100644 index 0..0f5704733 --- /dev/null
> +++ b/lib/portage/_compat_upgrade/binpkg_compression.py
> @@ -0,0 +1,40 @@
> +# Copyright 2020 Gentoo Authors
> +# Distributed under the terms of the GNU General Public License v2
> +
> +import re
> +
> +import portage
> +from portage import os
> +from portage.const import GLOBAL_CONFIG_PATH
> +
> +COMPAT_BINPKG_COMPRESS = 'bzip2'
> +
> +
> +def main():
> + """
> + If the current installation is still configured to use the
> old
> + default BINPKG_COMPRESS=bzip2 setting, then patch
> make.globals
> + inside ${ED} to maintain backward compatibility, ensuring
> that
> + binary package consumers are not caught off guard. This is
> + intended to be called from the ebuild as follows:
> +
> + pkg_preinst() {
> + python_setup
> + env -u BINPKG_COMPRESS
> +
> PYTHONPATH="${D%/}$(python_get_sitedir)${PYTHONPATH:+:${PYTHONPATH}}"
> \
> + "${PYTHON}" -m
> portage._compat_upgrade.binpkg_compression || die
> + }
> + """
> + if portage.settings.get('BINPKG_COMPRESS',
> COMPAT_BINPKG_COMPRESS) == COMPAT_BINPKG_COMPRESS:
> + config_path = os.path.join(os.environ['ED'],
> GLOBAL_CONFIG_PATH.lstrip(os.sep), 'make.globals')
> + with open(config_path) as f:
> + content = f.read()
> + compat_setting =
> 'BINPKG_COMPRESS="{}"'.format(COMPAT_BINPKG_COMPRESS)
> + portage.output.EOutput().einfo('Setting
> make.globals default {} for backward
> compatibility'.format(compat_setting))
> + content = re.sub('^BINPKG_COMPRESS=.*$',
> compat_setting, content, flags=re.MULTILINE)
> + with open(config_path, 'wt') as f:
> + f.write(content)
> +
> +
> +if __name__ == '__main__':
> + main()
> diff --git a/lib/portage/tests/resolver/ResolverPlayground.py
> b/lib/portage/tests/resolver/ResolverPlayground.py index
> 98831e000..de80a0cc1 100644 ---
> a/lib/portage/tests/resolver/ResolverPlayground.py +++
> b/lib/portage/tests/resolver/ResolverPlayground.py @@ -112,6 +112,7
> @@ class ResolverPlayground(object): "uname",
>   "uniq",
>   "xargs",
> + "zstd",
>   )
>   # Exclude internal wrappers from PATH lookup.
>   orig_path = os.environ['PATH']
> diff --git a/man/make.conf.5 b/man/make.conf.5
> index f82fed65a..a3bd662ae 100644
> --- a/man/make.conf.5
> +++ b/man/make.conf.5
> @@ -1,4 +1,4 @@
> -.TH "MAKE.CONF" "5" "Nov 2019" "Portage VERSION" "Portage"
> +.TH "MAKE.CONF" "5" "May 2020" "Portage VERSION" "Portage"
>  .SH "NAME"
>  make.conf \- custom settings for Portage
>  .SH "SYNOPSIS"
> @@ -115,7 +115,7 @@ This variable is used to determine the
> compression used for \fIbinary packages\fR.