Re: [gentoo-portage-dev] [PATCH] portage.getpid: call os.getpid() lazily
On Sat, 30 Jan 2021 04:59:32 -0800 Zac Medico wrote: > Call os.getpid() lazily, which eliminates getpid calls when possible > after os.fork() in the portage.process module. > > Bug: https://bugs.gentoo.org/767913 > Signed-off-by: Zac Medico > --- > lib/portage/__init__.py | 4 +++- > 1 file changed, 3 insertions(+), 1 deletion(-) > > diff --git a/lib/portage/__init__.py b/lib/portage/__init__.py > index 3c9f78497..24c9d8b89 100644 > --- a/lib/portage/__init__.py > +++ b/lib/portage/__init__.py > @@ -375,7 +375,7 @@ _sync_mode = False > class _ForkWatcher: > @staticmethod > def hook(_ForkWatcher): > - _ForkWatcher.current_pid = _os.getpid() > + _ForkWatcher.current_pid = None > # Force instantiation of a new event loop policy as > a workaround # for https://bugs.python.org/issue22087. > asyncio.set_event_loop_policy(None) > @@ -388,6 +388,8 @@ def getpid(): > """ > Cached version of os.getpid(). ForkProcess updates the cache. > """ > + if _ForkWatcher.current_pid is None: > + _ForkWatcher.current_pid = _os.getpid() > return _ForkWatcher.current_pid > > def _get_stdin(): looks good
Re: [gentoo-portage-dev] [PATCH] emerge: disable --autounmask-license by default
On Sat, 30 Jan 2021 03:49:41 -0800
Zac Medico wrote:
> Disable --autounmask-license by default, in order to limit user
> exposure to risks associated with package.license changes.
> The changes that this option suggests are only intended to be
> accepted when a user has made a conscious decision to accept
> the corresponding license(s). Creation of package.license
> changes introduces a risk that users may erroneously accept the
> changes due to some kind of accident or misunderstanding,
> rather than due to conscious decisions about licenses.
> These risks provide motivation to disable --autounmask-license
> by default. The --autounmask-use option will remain as the
> only autounmask option that is still enabled by default.
>
> The unit tests demonstrate interactions between --autounmask
> and --autounmask-license options. The --autounmask option
> enables --autounmask-license unless --autounmask-license=n
> has been specified. If --autounmask=n is used to disable
> autounmask, then --autounmask-license=y has no effect.
>
> Bug: https://bugs.gentoo.org/766773
> Signed-off-by: Zac Medico
> ---
makes sense, code looks good
> lib/_emerge/create_depgraph_params.py | 8 +++---
> lib/portage/tests/resolver/test_autounmask.py | 25
> +-- man/emerge.1 |
> 11 +++- 3 files changed, 31 insertions(+), 13 deletions(-)
>
> diff --git a/lib/_emerge/create_depgraph_params.py
> b/lib/_emerge/create_depgraph_params.py index 0d0e07b9c..25dd2a1b4
> 100644 --- a/lib/_emerge/create_depgraph_params.py
> +++ b/lib/_emerge/create_depgraph_params.py
> @@ -1,4 +1,4 @@
> -# Copyright 1999-2018 Gentoo Foundation
> +# Copyright 1999-2021 Gentoo Authors
> # Distributed under the terms of the GNU General Public License v2
>
> import logging
> @@ -45,7 +45,7 @@ def create_depgraph_params(myopts, myaction):
> autounmask_keep_masks = myopts.get("--autounmask-keep-masks")
>
> autounmask = myopts.get("--autounmask")
> - autounmask_license = myopts.get('--autounmask-license')
> + autounmask_license = myopts.get('--autounmask-license', 'y'
> if autounmask is True else 'n') autounmask_use =
> myopts.get('--autounmask-use') if autounmask == 'n':
> autounmask = False
> @@ -53,7 +53,7 @@ def create_depgraph_params(myopts, myaction):
> if autounmask is None:
> if autounmask_use in (None, 'y'):
> autounmask = True
> - elif autounmask_license in (None, 'y'):
> + if autounmask_license in ('y',):
> autounmask = True
>
> # Do not enable package.accept_keywords or
> package.mask @@ -67,7 +67,7 @@ def create_depgraph_params(myopts,
> myaction):
> myparams['autounmask'] = autounmask
> myparams['autounmask_keep_use'] = True if autounmask_use ==
> 'n' else False
> - myparams['autounmask_keep_license'] = True if
> autounmask_license == 'n' else False
> + myparams['autounmask_keep_license'] = False if
> autounmask_license == 'y' else True
> myparams['autounmask_keep_keywords'] = False if
> autounmask_keep_keywords in (None, 'n') else True
> myparams['autounmask_keep_masks'] = False if autounmask_keep_masks in
> (None, 'n') else True diff --git
> a/lib/portage/tests/resolver/test_autounmask.py
> b/lib/portage/tests/resolver/test_autounmask.py index
> a3bf0ff94..86ae4bbf6 100644 ---
> a/lib/portage/tests/resolver/test_autounmask.py +++
> b/lib/portage/tests/resolver/test_autounmask.py @@ -1,4 +1,4 @@ -#
> Copyright 2010-2019 Gentoo Authors +# Copyright 2010-2021 Gentoo
> Authors # Distributed under the terms of the GNU General Public
> License v2 from portage.tests import TestCase
> @@ -440,13 +440,34 @@ class AutounmaskTestCase(TestCase):
> mergelist=["dev-libs/A-1"],
> license_changes={
> "dev-libs/A-1": set(["TEST"]) }),
> - # Test default --autounmask-license
> + # Test that --autounmask enables
> --autounmask-license ResolverPlaygroundTestCase(
> ["=dev-libs/A-1"],
> + options={"--autounmask":
> True}, success=False,
> mergelist=["dev-libs/A-1"],
> license_changes={
> "dev-libs/A-1": set(["TEST"]) }),
> + # Test that --autounmask-license is
> not enabled by default
> + ResolverPlaygroundTestCase(
> + ["=dev-libs/A-1"],
> + success=False,
> + ),
> +
> + # Test that --autounmask does not
> override --autounmask-license=n
> + ResolverPlaygroundTestCase(
> +
Re: [gentoo-portage-dev] [PATCH] emaint --fix merges: add -y, --yes option
On Sat, 23 Jan 2021 15:38:49 -0800
Zac Medico wrote:
> Since the emaint --fix merges uses emerge --ask, add
> a -y, --yes option to use --ask=n instead.
Looks good to me
>
> Bug: https://bugs.gentoo.org/766767
> Signed-off-by: Zac Medico
> ---
> lib/portage/emaint/modules/merges/__init__.py | 14 --
> lib/portage/emaint/modules/merges/merges.py | 11 +++
> man/emaint.1 | 6 +-
> 3 files changed, 24 insertions(+), 7 deletions(-)
>
> diff --git a/lib/portage/emaint/modules/merges/__init__.py
> b/lib/portage/emaint/modules/merges/__init__.py index
> 89aa758a0..449f39dce 100644 ---
> a/lib/portage/emaint/modules/merges/__init__.py +++
> b/lib/portage/emaint/modules/merges/__init__.py @@ -1,4 +1,4 @@
> -# Copyright 2005-2014 Gentoo Foundation
> +# Copyright 2005-2021 Gentoo Authors
> # Distributed under the terms of the GNU General Public License v2
>
> doc = """Scan for failed merges and fix them."""
> @@ -26,7 +26,17 @@ module_spec = {
> 'action': 'store_true',
> 'func': 'purge'
> }
> - }
> + },
> + 'opt_desc': {
> + 'yes': {
> + "short": "-y",
> + "long": "--yes",
> + "help": ("(merges submodule
> only): Do not prompt for "
> + "emerge
> invocations"),
> + "action": "store_true",
> + "dest": "yes",
> + }
> + },
> }
> }
> }
> diff --git a/lib/portage/emaint/modules/merges/merges.py
> b/lib/portage/emaint/modules/merges/merges.py index
> 775dc59d2..d60916f1e 100644 ---
> a/lib/portage/emaint/modules/merges/merges.py +++
> b/lib/portage/emaint/modules/merges/merges.py @@ -1,4 +1,4 @@
> -# Copyright 2005-2020 Gentoo Authors
> +# Copyright 2005-2021 Gentoo Authors
> # Distributed under the terms of the GNU General Public License v2
>
> import portage
> @@ -186,7 +186,7 @@ class MergesHandler:
> pkg_atoms.add(pkg_atom)
>
>
> - def _emerge_pkg_atoms(self, module_output, pkg_atoms):
> + def _emerge_pkg_atoms(self, module_output, pkg_atoms,
> yes=False): """
> Emerge the specified packages atoms.
>
> @@ -194,6 +194,8 @@ class MergesHandler:
> @type module_output: Class
> @param pkg_atoms: packages atoms to emerge
> @type pkg_atoms: set
> + @param yes: do not prompt for emerge invocations
> + @type yes: bool
> @rtype: list
> @return: List of results
> """
> @@ -206,7 +208,7 @@ class MergesHandler:
> portage._python_interpreter,
> '-b',
> os.path.join(EPREFIX or '/', 'usr', 'bin',
> 'emerge'),
> - '--ask',
> + '--ask=n' if yes else '--ask',
> '--quiet',
> '--oneshot',
> '--complete-graph=y'
> @@ -265,7 +267,8 @@ class MergesHandler:
> errors.append(', '.join(sorted(failed_pkgs)))
> return (False, errors)
> self._remove_failed_dirs(failed_pkgs)
> - results = self._emerge_pkg_atoms(module_output,
> pkg_atoms)
> + results = self._emerge_pkg_atoms(module_output,
> pkg_atoms,
> + yes=kwargs.get('options', {}).get("yes",
> False)) # list any new failed merges
> for pkg in sorted(self._scan()):
> results.append("'%s' still found as a failed
> merge." % pkg) diff --git a/man/emaint.1 b/man/emaint.1
> index d244756e9..c9f8ab939 100644
> --- a/man/emaint.1
> +++ b/man/emaint.1
> @@ -1,4 +1,4 @@
> -.TH "EMAINT" "1" "Jan 2017" "Portage VERSION" "Portage"
> +.TH "EMAINT" "1" "Jan 2021" "Portage VERSION" "Portage"
> .SH NAME
> emaint \- performs package management related system health checks
> and maintenance .SH SYNOPSIS
> @@ -82,6 +82,10 @@ OPTION
> .BR \-t \ \fINUM\fR,\ \-\-time \ \fINUM\fR
> Changes the minimum age \fINUM\fR (in days) of the logs to be listed
> or deleted.
> +.SH OPTIONS merges command only
> +.TP
> +.BR \-y ", " \-\-yes
> +Do not prompt for emerge invocations.
> .SH OPTIONS sync command only
> .TP
> .BR \-a ", " \-\-auto
Re: [gentoo-portage-dev] [PATCH] repoman: Add --jobs and --load-average options (bug 448462)
On Sun, 16 Aug 2020 20:26:56 -0700
Zac Medico wrote:
> Add --jobs and --load-average options which allow dependency checks
> for multiple profiles to run in parallel. The increase in performance
> is non-linear for the number of jobs, but it can be worthwhile
> (I measured a 35% decrease in time when running 'repoman -j8 full'
> on sys-apps/portage). For the -j1 case (default), all dependency
> checks run in the main process as usual, so there is no significant
> performance penalty for the default case.
>
> Bug: https://bugs.gentoo.org/448462
> Signed-off-by: Zac Medico
> ---
> repoman/lib/repoman/argparser.py | 9 ++
> .../repoman/modules/scan/depend/profile.py| 117
> +++--- repoman/man/repoman.1 |
> 9 +- 3 files changed, 116 insertions(+), 19 deletions(-)
>
> diff --git a/repoman/lib/repoman/argparser.py
> b/repoman/lib/repoman/argparser.py index 670a0e91d..6d545ccca 100644
> --- a/repoman/lib/repoman/argparser.py
> +++ b/repoman/lib/repoman/argparser.py
> @@ -199,6 +199,15 @@ def parse_args(argv, repoman_default_opts):
> '--output-style', dest='output_style',
> choices=output_keys, help='select output type', default='default')
>
> + parser.add_argument(
> + '-j', '--jobs', dest='jobs', action='store',
> type=int, default=1,
> + help='Specifies the number of jobs (processes) to
> run simultaneously.') +
> + parser.add_argument(
> + '-l', '--load-average', dest='load_average',
> action='store', type=float, default=None,
> + help='Specifies that no new jobs (processes) should
> be started if there are others '
> + 'jobs running and the load average is at
> least load (a floating-point number).') +
> parser.add_argument(
> '--mode', dest='mode', choices=mode_keys,
> help='specify which mode repoman will run in
> (default=full)') diff --git
> a/repoman/lib/repoman/modules/scan/depend/profile.py
> b/repoman/lib/repoman/modules/scan/depend/profile.py index
> 39d8b550c..1eb69422a 100644 ---
> a/repoman/lib/repoman/modules/scan/depend/profile.py +++
> b/repoman/lib/repoman/modules/scan/depend/profile.py @@ -2,7 +2,9 @@
>
> import copy
> +import functools
> import os
> +import types
> from pprint import pformat
>
> from _emerge.Package import Package
> @@ -15,6 +17,10 @@ from repoman.modules.scan.depend._gen_arches
> import _gen_arches from portage.dep import Atom
> from portage.package.ebuild.profile_iuse import iter_iuse_vars
> from portage.util import getconfig
> +from portage.util.futures import asyncio
> +from portage.util.futures.compat_coroutine import coroutine,
> coroutine_return +from portage.util.futures.executor.fork import
> ForkExecutor +from portage.util.futures.iter_completed import
> async_iter_completed
>
> def sort_key(item):
> @@ -58,16 +64,14 @@ class ProfileDependsChecks(ScanBase):
> def check(self, **kwargs):
> '''Perform profile dependant dependency checks
>
> - @param arches:
> @param pkg: Package in which we check (object).
> @param ebuild: Ebuild which we check (object).
> - @param baddepsyntax: boolean
> - @param unknown_pkgs: set of tuples (type,
> atom.unevaluated_atom) @returns: dictionary
> '''
> ebuild = kwargs.get('ebuild').get()
> pkg = kwargs.get('pkg').get()
> - unknown_pkgs, baddepsyntax = _depend_checks(
> +
> + ebuild.unknown_pkgs, ebuild.baddepsyntax =
> _depend_checks( ebuild, pkg, self.portdb, self.qatracker,
> self.repo_metadata, self.repo_settings.qadata)
>
> @@ -90,8 +94,64 @@ class ProfileDependsChecks(ScanBase):
> relevant_profiles.append((keyword,
> groups, prof))
> relevant_profiles.sort(key=sort_key)
> + ebuild.relevant_profiles = relevant_profiles
> +
> + if self.options.jobs <= 1:
> + for task in self._iter_tasks(None, None,
> ebuild, pkg):
> + task, results = task
> + for result in results:
> + self._check_result(task,
> result) +
> + loop = asyncio._wrap_loop()
> + loop.run_until_complete(self._async_check(loop=loop,
> **kwargs)) +
> + return False
> +
> + @coroutine
> + def _async_check(self, loop=None, **kwargs):
> + '''Perform async profile dependant dependency checks
> +
> + @param arches:
> + @param pkg: Package in which we check (object).
> + @param ebuild: Ebuild which we check (object).
> + @param baddepsyntax: boolean
> + @param unknown_pkgs: set of tuples (type,
> atom.unevaluated_atom)
> + @returns: dictionary
> + '''
> + loop = asyncio._wrap_loop(loop)
> +
Re: [gentoo-portage-dev] [PATCH 0/2] emerge: Add short -l option for --load-average (bug 699256)
On Sat, 15 Aug 2020 12:26:42 -0700 Zac Medico wrote: > Add a short -l option for --load-average just like make(1) has. > > Bug: https://bugs.gentoo.org/699256 > Signed-off-by: Zac Medico > > Zac Medico (2): > emerge: Remove deprecated --changelog option (bug 699256) > emerge: Add short -l option for --load-average (bug 699256) > > lib/_emerge/actions.py | 5 -- > lib/_emerge/main.py| 5 +- > lib/_emerge/resolver/output.py | 32 +--- > lib/_emerge/resolver/output_helpers.py | 106 > - man/emerge.1 | > 8 +- 5 files changed, 5 insertions(+), 151 deletions(-) > looks good :)
Re: [gentoo-portage-dev] [PATCH 1/2] lib/portage/repository/config.py: fix trailing comma
On Tue, 4 Aug 2020 21:56:08 -0400 Aaron Bauman wrote: > On Tue, Aug 04, 2020 at 09:43:03PM -0400, Aaron Bauman wrote: > > * This fixes the one instance of a trailing comma in the repo which > > actually causes a tuple to be created. > > > > Signed-off-by: Aaron Bauman > > --- > > lib/portage/repository/config.py | 2 +- > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > diff --git a/lib/portage/repository/config.py > > b/lib/portage/repository/config.py index 0f3e582f8..f4dd834b5 100644 > > --- a/lib/portage/repository/config.py > > +++ b/lib/portage/repository/config.py > > @@ -871,7 +871,7 @@ class RepoConfigLoader: > > continue > > if repo.masters is None: > > if self.mainRepo() and repo_name > > != self.mainRepo().name: > > - repo.masters = > > self.mainRepo(), > > + repo.masters = > > self.mainRepo() else: > > repo.masters = () > > else: > > -- > > 2.28.0 > > > > > > Please disregard this patch series. > Please don't enable this check. Pylint is stupid to enforce this one. When adding another item to the list or dictionary, it is better to have the tailing comma already in place. Git blame doesn't work if you have to add the comma to extend the list, etc.. It also adds unnecessary churn for something that does not affect operation in the slightest. Is one of the bad ones pylint wants to enforce.
Re: [gentoo-portage-dev] [PATCH] cnf/repo.postsync.d/example: add egencache --update-pkg-desc-index (bug 735626)
On Sun, 2 Aug 2020 16:38:06 -0700
Zac Medico wrote:
> Add an egencache --update-pkg-desc-index example for users
> of app-portage/esearch to migrate to.
>
> Bug: https://bugs.gentoo.org/735626
> Signed-off-by: Zac Medico
> ---
> cnf/repo.postsync.d/example | 15 ++-
> 1 file changed, 14 insertions(+), 1 deletion(-)
>
> diff --git a/cnf/repo.postsync.d/example b/cnf/repo.postsync.d/example
> index 533bf719c..6c349ab3c 100644
> --- a/cnf/repo.postsync.d/example
> +++ b/cnf/repo.postsync.d/example
> @@ -1,4 +1,4 @@
> -#!/bin/sh
> +#!/bin/bash
> # Example /etc/portage/repo.postsync.d script. Make it executable
> (chmod +x) for # Portage to process it.
> #
> @@ -45,6 +45,19 @@ if [ -n "${repository_name}" ]; then
> ret=1
> fi
> fi
> +
> + # Regenerate the metadata/pkg_desc_index file if needed.
> It's not
> + # needed for https://gitweb.gentoo.org/repo/sync/gentoo.git
> which
> + # provides a freshly generated copy.
> + if [[ ! -e ${repository_path}/metadata/pkg_desc_index || (
> + -d ${repository_path}/metadata/md5-cache &&
> + -n "$(find "${repository_path}/metadata/md5-cache"
> -type f -newer "${repository_path}/metadata/pkg_desc_index" -print
> -quit)" ) ]]; then
> + if ! egencache --update-pkg-desc-index
> --repo="${repository_name}"
> + then
> + echo "!!! egencache failed!"
> + ret=1
> + fi
> + fi
> fi
>
> # Return explicit status.
is good with me, but my bash skills suck... so if there is a mistake I
wouldn't know ;)
Re: [gentoo-portage-dev] [PATCH] travis.yml: drop python 2.7 (bug 731114)
On Mon, 6 Jul 2020 11:42:06 -0700
Zac Medico wrote:
> It should be pretty safe to drop support for python2.7 at this point.
>
> Bug: https://bugs.gentoo.org/731114
> Signed-off-by: Zac Medico
> ---
> .travis.yml | 1 -
> tox.ini | 6 ++
> 2 files changed, 2 insertions(+), 5 deletions(-)
>
> diff --git a/.travis.yml b/.travis.yml
> index 2132c8c87..d2935fdab 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -1,7 +1,6 @@
> dist: bionic
> language: python
> python:
> -- 2.7
> - 3.6
> - 3.7
> - 3.8
> diff --git a/tox.ini b/tox.ini
> index 79b5b45cb..050a2c455 100644
> --- a/tox.ini
> +++ b/tox.ini
> @@ -1,14 +1,12 @@
> [tox]
> -envlist = py27,py36,py37,py38,py39,pypy3
> +envlist = py36,py37,py38,py39,pypy3
> skipsdist = True
>
> [testenv]
> deps =
> pygost
> pyyaml
> - py27,py36,py37,py38,py39,pypy3: lxml!=4.2.0
> - py27: pyblake2
> - py27: pysha3
> + py36,py37,py38,py39,pypy3: lxml!=4.2.0
> setenv =
> PYTHONPATH={toxinidir}/lib
> commands =
Go for it!
Re: [gentoo-portage-dev] [PATCH 2/2] Support PORTAGE_LOG_FILTER_FILE_CMD (bug 709746)
On Fri, 19 Jun 2020 13:39:19 -0700
Zac Medico wrote:
> This variable specifies a command that filters build log output to a
> log file. The plan is to extend this to support a separate filter for
> tty output in the future.
>
> In order to enable the EbuildPhase class to write elog messages to
> the build log with PORTAGE_LOG_FILTER_FILE_CMD support, convert its
> _elog method to a coroutine, and add a SchedulerInterface async_output
> method for it to use.
>
> Use a new BuildLogger class to manage log output (with or without a
> filter command), with compression support provided by PipeLogger.
> BuildLogger has a stdin property which provides access to a writable
> binary file stream (refers to a pipe) that log content is written to.
>
> Bug: https://bugs.gentoo.org/709746
> Signed-off-by: Zac Medico
> ---
> lib/_emerge/AbstractEbuildProcess.py | 3 +-
> lib/_emerge/BinpkgFetcher.py | 3 +-
> lib/_emerge/EbuildFetcher.py | 3 +-
> lib/_emerge/EbuildPhase.py| 47 ++--
> lib/_emerge/SpawnProcess.py | 58 +++---
> lib/portage/dbapi/_MergeProcess.py| 3 +-
> .../ebuild/_config/special_env_vars.py| 8 +-
> lib/portage/util/_async/BuildLogger.py| 109
> ++ lib/portage/util/_async/SchedulerInterface.py |
> 32 - man/make.conf.5 | 7 +-
> 10 files changed, 243 insertions(+), 30 deletions(-)
> create mode 100644 lib/portage/util/_async/BuildLogger.py
>
> diff --git a/lib/_emerge/AbstractEbuildProcess.py
> b/lib/_emerge/AbstractEbuildProcess.py index 1c1955cfe..ae1aae55f
> 100644 --- a/lib/_emerge/AbstractEbuildProcess.py
> +++ b/lib/_emerge/AbstractEbuildProcess.py
> @@ -1,4 +1,4 @@
> -# Copyright 1999-2019 Gentoo Foundation
> +# Copyright 1999-2020 Gentoo Authors
> # Distributed under the terms of the GNU General Public License v2
>
> import errno
> @@ -196,6 +196,7 @@ class AbstractEbuildProcess(SpawnProcess):
> null_fd = os.open('/dev/null', os.O_RDONLY)
> self.fd_pipes[0] = null_fd
>
> + self.log_filter_file =
> self.settings.get('PORTAGE_LOG_FILTER_FILE_CMD') try:
> SpawnProcess._start(self)
> finally:
> diff --git a/lib/_emerge/BinpkgFetcher.py
> b/lib/_emerge/BinpkgFetcher.py index 36d027de3..2e5861cc1 100644
> --- a/lib/_emerge/BinpkgFetcher.py
> +++ b/lib/_emerge/BinpkgFetcher.py
> @@ -1,4 +1,4 @@
> -# Copyright 1999-2018 Gentoo Foundation
> +# Copyright 1999-2020 Gentoo Authors
> # Distributed under the terms of the GNU General Public License v2
>
> import functools
> @@ -158,6 +158,7 @@ class _BinpkgFetcherProcess(SpawnProcess):
> self.env = fetch_env
> if settings.selinux_enabled():
> self._selinux_type =
> settings["PORTAGE_FETCH_T"]
> + self.log_filter_file =
> settings.get('PORTAGE_LOG_FILTER_FILE_CMD') SpawnProcess._start(self)
>
> def _pipe(self, fd_pipes):
> diff --git a/lib/_emerge/EbuildFetcher.py
> b/lib/_emerge/EbuildFetcher.py index 1e40994fb..55349c33c 100644
> --- a/lib/_emerge/EbuildFetcher.py
> +++ b/lib/_emerge/EbuildFetcher.py
> @@ -1,4 +1,4 @@
> -# Copyright 1999-2018 Gentoo Foundation
> +# Copyright 1999-2020 Gentoo Authors
> # Distributed under the terms of the GNU General Public License v2
>
> import copy
> @@ -225,6 +225,7 @@ class _EbuildFetcherProcess(ForkProcess):
> settings["NOCOLOR"] = nocolor
>
> self._settings = settings
> + self.log_filter_file =
> settings.get('PORTAGE_LOG_FILTER_FILE_CMD') ForkProcess._start(self)
>
> # Free settings now since it's no longer needed in
> diff --git a/lib/_emerge/EbuildPhase.py b/lib/_emerge/EbuildPhase.py
> index 477e0ba97..ddb3dc719 100644
> --- a/lib/_emerge/EbuildPhase.py
> +++ b/lib/_emerge/EbuildPhase.py
> @@ -26,6 +26,8 @@ from portage.package.ebuild.prepare_build_dirs
> import (_prepare_workdir, from portage.util.futures.compat_coroutine
> import coroutine from portage.util import writemsg
> from portage.util._async.AsyncTaskFuture import AsyncTaskFuture
> +from portage.util._async.BuildLogger import BuildLogger
> +from portage.util.futures import asyncio
> from portage.util.futures.executor.fork import ForkExecutor
>
> try:
> @@ -69,6 +71,11 @@ class EbuildPhase(CompositeTask):
> _locked_phases = ("setup", "preinst", "postinst", "prerm",
> "postrm")
> def _start(self):
> + future = asyncio.ensure_future(self._async_start(),
> loop=self.scheduler)
> + self._start_task(AsyncTaskFuture(future=future),
> self._async_start_exit) +
> + @coroutine
> + def _async_start(self):
>
> need_builddir = self.phase not in
> EbuildProcess._phases_without_builddir
> @@ -126,7 +133,7 @@ class EbuildPhase(CompositeTask):
> # Force
Re: [gentoo-portage-dev] [PATCH 1/2] PipeLogger: non-blocking write to pipe (bug 709746)
On Fri, 19 Jun 2020 13:39:18 -0700
Zac Medico wrote:
> Add support to write to a non-blocking pipe instead of a
> log file. This is needed for the purposes of bug 709746,
> where PipeLogger will write to a pipe that is drained
> by anoher PipeLogger instance which is running in the same
> process.
>
> Bug: https://bugs.gentoo.org/709746
> Signed-off-by: Zac Medico
> ---
> lib/portage/tests/process/test_PipeLogger.py | 58
> lib/portage/util/_async/PipeLogger.py| 73
> +++- 2 files changed, 115 insertions(+), 16
> deletions(-) create mode 100644
> lib/portage/tests/process/test_PipeLogger.py
>
> diff --git a/lib/portage/tests/process/test_PipeLogger.py
> b/lib/portage/tests/process/test_PipeLogger.py new file mode 100644
> index 0..2bd94cf39
> --- /dev/null
> +++ b/lib/portage/tests/process/test_PipeLogger.py
> @@ -0,0 +1,58 @@
> +# Copyright 2020 Gentoo Authors
> +# Distributed under the terms of the GNU General Public License v2
> +
> +from portage import os
> +from portage.tests import TestCase
> +from portage.util._async.PipeLogger import PipeLogger
> +from portage.util.futures import asyncio
> +from portage.util.futures._asyncio.streams import _reader, _writer
> +from portage.util.futures.compat_coroutine import coroutine,
> coroutine_return +from portage.util.futures.unix_events import
> _set_nonblocking +
> +
> +class PipeLoggerTestCase(TestCase):
> +
> + @coroutine
> + def _testPipeLoggerToPipe(self, test_string, loop=None):
> + """
> + Test PipeLogger writing to a pipe connected to a
> PipeReader.
> + This verifies that PipeLogger does not deadlock when
> writing
> + to a pipe that's drained by a PipeReader running in
> the same
> + process (requires non-blocking write).
> + """
> +
> + input_fd, writer_pipe = os.pipe()
> + _set_nonblocking(writer_pipe)
> + writer_pipe = os.fdopen(writer_pipe, 'wb', 0)
> + writer = asyncio.ensure_future(_writer(writer_pipe,
> test_string.encode('ascii'), loop=loop), loop=loop)
> + writer.add_done_callback(lambda writer:
> writer_pipe.close()) +
> + pr, pw = os.pipe()
> +
> + consumer = PipeLogger(background=True,
> + input_fd=input_fd,
> + log_file_path=os.fdopen(pw, 'wb', 0),
> + scheduler=loop)
> + consumer.start()
> +
> + # Before starting the reader, wait here for a
> moment, in order
> + # to exercise PipeLogger's handling of EAGAIN during
> write.
> + yield asyncio.wait([writer], timeout=0.01)
> +
> + reader = _reader(pr, loop=loop)
> + yield writer
> + content = yield reader
> + yield consumer.async_wait()
> +
> + self.assertEqual(consumer.returncode, os.EX_OK)
> +
> + coroutine_return(content.decode('ascii', 'replace'))
> +
> + def testPipeLogger(self):
> + loop = asyncio._wrap_loop()
> +
> + for x in (1, 2, 5, 6, 7, 8, 2**5, 2**10, 2**12,
> 2**13, 2**14, 2**17, 2**17 + 1):
> + test_string = x * "a"
> + output =
> loop.run_until_complete(self._testPipeLoggerToPipe(test_string,
> loop=loop))
> + self.assertEqual(test_string, output,
> + "x = %s, len(output) = %s" % (x,
> len(output))) diff --git a/lib/portage/util/_async/PipeLogger.py
> b/lib/portage/util/_async/PipeLogger.py index a4258f350..ce8afb846
> 100644 --- a/lib/portage/util/_async/PipeLogger.py
> +++ b/lib/portage/util/_async/PipeLogger.py
> @@ -8,6 +8,10 @@ import sys
>
> import portage
> from portage import os, _encodings, _unicode_encode
> +from portage.util.futures import asyncio
> +from portage.util.futures._asyncio.streams import _writer
> +from portage.util.futures.compat_coroutine import coroutine
> +from portage.util.futures.unix_events import _set_nonblocking
> from _emerge.AbstractPollTask import AbstractPollTask
>
> class PipeLogger(AbstractPollTask):
> @@ -21,13 +25,16 @@ class PipeLogger(AbstractPollTask):
> """
>
> __slots__ = ("input_fd", "log_file_path", "stdout_fd") + \
> - ("_log_file", "_log_file_real")
> + ("_io_loop_task", "_log_file", "_log_file_nb",
> "_log_file_real")
> def _start(self):
>
> log_file_path = self.log_file_path
> - if log_file_path is not None:
> -
> + if hasattr(log_file_path, 'write'):
> + self._log_file_nb = True
> + self._log_file = log_file_path
> + _set_nonblocking(self._log_file.fileno())
> + elif log_file_path is not None:
> self._log_file =
> open(_unicode_encode(log_file_path, encoding=_encodings['fs'],
> errors='strict'), mode='ab') if
Re: [gentoo-portage-dev] [PATCH] _writer: fix unsafe finally clause (bug 728580)
On Thu, 18 Jun 2020 00:35:48 -0700
Zac Medico wrote:
> In the coroutine finally clause, do not call remove_writer in cases
> where fd has been closed and then re-allocated to a concurrent
> coroutine as in bug 716636.
>
> Also, assume that the caller will put the file in non-blocking mode
> and close the file when done, so that this function is suitable for
> use within a loop.
>
> Bug: https://bugs.gentoo.org/728580
> Signed-off-by: Zac Medico
> ---
> lib/portage/util/futures/_asyncio/process.py | 11 -
> lib/portage/util/futures/_asyncio/streams.py | 50
> ++-- 2 files changed, 33 insertions(+), 28
> deletions(-)
>
> diff --git a/lib/portage/util/futures/_asyncio/process.py
> b/lib/portage/util/futures/_asyncio/process.py index
> 020164c9b..2d3e9b0fd 100644 ---
> a/lib/portage/util/futures/_asyncio/process.py +++
> b/lib/portage/util/futures/_asyncio/process.py @@ -1,9 +1,12 @@
> -# Copyright 2018 Gentoo Foundation
> +# Copyright 2018-2020 Gentoo Authors
> # Distributed under the terms of the GNU General Public License v2
>
> +import os
> +
> import portage
> portage.proxy.lazyimport.lazyimport(globals(),
> 'portage.util.futures:asyncio',
> + 'portage.util.futures.unix_events:_set_nonblocking',
> )
> from portage.util.futures._asyncio.streams import _reader, _writer
> from portage.util.futures.compat_coroutine import coroutine,
> coroutine_return @@ -59,7 +62,11 @@ class _Process(object):
> if input is not None:
> if self._proc.stdin is None:
> raise TypeError('communicate:
> expected file or int, got {}'.format(type(self._proc.stdin)))
> - writer =
> asyncio.ensure_future(_writer(self._proc.stdin, input),
> loop=self._loop)
> + stdin = self._proc.stdin
> + stdin = os.fdopen(stdin, 'wb', 0) if
> isinstance(stdin, int) else stdin
> + _set_nonblocking(stdin.fileno())
> + writer =
> asyncio.ensure_future(_writer(stdin, input, loop=self._loop),
> loop=self._loop)
> + writer.add_done_callback(lambda writer:
> stdin.close())
> try:
> yield asyncio.wait(futures + [self.wait()],
> loop=self._loop) diff --git
> a/lib/portage/util/futures/_asyncio/streams.py
> b/lib/portage/util/futures/_asyncio/streams.py index
> 650a16491..870307e1e 100644 ---
> a/lib/portage/util/futures/_asyncio/streams.py +++
> b/lib/portage/util/futures/_asyncio/streams.py @@ -1,4 +1,4 @@ -#
> Copyright 2018 Gentoo Foundation +# Copyright 2018-2020 Gentoo Authors
> # Distributed under the terms of the GNU General Public License v2
>
> import errno
> @@ -8,7 +8,6 @@ import portage
> portage.proxy.lazyimport.lazyimport(globals(),
> '_emerge.PipeReader:PipeReader',
> 'portage.util.futures:asyncio',
> - 'portage.util.futures.unix_events:_set_nonblocking',
> )
> from portage.util.futures.compat_coroutine import coroutine
>
> @@ -59,38 +58,37 @@ class _Reader(object):
> @coroutine
> def _writer(output_file, content, loop=None):
> """
> - Asynchronously write bytes to output file, and close it when
> - done. If an EnvironmentError other than EAGAIN is
> encountered,
> - which typically indicates that the other end of the pipe has
> - close, the error is raised. This function is a coroutine.
> + Asynchronously write bytes to output file. The output file is
> + assumed to be in non-blocking mode. If an EnvironmentError
> + other than EAGAIN is encountered, which typically indicates
> that
> + the other end of the pipe has closed, the error is raised.
> + This function is a coroutine.
>
> - @param output_file: output file descriptor
> - @type output_file: file or int
> + @param output_file: output file
> + @type output_file: file object
> @param content: content to write
> @type content: bytes
> @param loop: asyncio.AbstractEventLoop (or compatible)
> @type loop: event loop
> """
> - fd = output_file if isinstance(output_file, int) else
> output_file.fileno()
> - _set_nonblocking(fd)
> loop = asyncio._wrap_loop(loop)
> - try:
> - while content:
> + fd = output_file.fileno()
> + while content:
> + try:
> + content = content[os.write(fd, content):]
> + except EnvironmentError as e:
> + if e.errno != errno.EAGAIN:
> + raise
> waiter = loop.create_future()
> - loop.add_writer(fd, lambda:
> waiter.set_result(None))
> + loop.add_writer(fd, lambda: waiter.done() or
> waiter.set_result(None)) try:
> yield waiter
> - while content:
> - try:
> - content =
>
Re: [gentoo-portage-dev] erroneous behavior in 2-style USE dependencies?
On Tue, 16 Jun 2020 20:43:44 +0200 (CEST) "michael.lienhardt" wrote: > Dear all, > My bad for not noticing it sooner, but when there is a dependency > like "=sys-fs/udev-208-r1:0/0[static-libs?]" (that occurs in > virtual/libgudev-215-r3), since 'static-libs' is not a use flags of > sys-fs/udev-242, it is silently not considered during dependency > solving by emerge. However, the PMS states: - it is an > error for a use dependency to be applied to an ebuild which does not > have the flag in question in class="ectt-1000">IUSE_REFERENCEABLE - For EAPIs > listed in table href="https://dev.gentoo.org/~ulm/pms/head/pms.html#x1-56001r4;>5.4 > as not supporting profile defined IUSE > injection, IUSE_REFERENCEABLE > is equal to the calculated IUSE > value. For EAPIs where profile defined class="ectt-1000">IUSE injection is supported, class="ectt-1000">IUSE_REFERENCEABLE is equal to class="ectt-1000">IUSE_EFFECTIVE And 'static-libs' is not in the > IUSE_EFFECTIVE of sys-fs/udev-242 (that ebuild has > EAPI=6). So it seems to me that this current behavior of emerge > should be considered an error, no? Or the PMS should be updated? > Best, Michael Please do NOT send html emails. text only please
Re: [gentoo-portage-dev] [PATCH] repos.conf: add bool sync-openpgp-key-refresh option (bug 661518)
On Fri, 12 Jun 2020 16:51:51 -0700
Zac Medico wrote:
> Add a sync-openpgp-key-refresh option that makes it possible to
> disable key refresh, which may be useful in cases when it is not
> possible to refresh keys.
>
> Key refresh is enabled by default, and if it is disabled then
> the SyncBase._refresh_keys method will output an ewarn message
> like this when the --quiet option is not enabled:
>
> * Key refresh is disabled via a repos.conf sync-openpgp-key-refresh
> * setting, and this is a security vulnerability because it prevents
> * detection of revoked keys!
>
> Bug: https://bugs.gentoo.org/661518
> Signed-off-by: Zac Medico
> ---
> lib/portage/repository/config.py | 10 +-
> lib/portage/sync/syncbase.py | 9 -
> man/portage.5| 9 -
> 3 files changed, 25 insertions(+), 3 deletions(-)
>
> diff --git a/lib/portage/repository/config.py
> b/lib/portage/repository/config.py index 50ab18026..6155c130a 100644
> --- a/lib/portage/repository/config.py
> +++ b/lib/portage/repository/config.py
> @@ -1,4 +1,4 @@
> -# Copyright 2010-2019 Gentoo Authors
> +# Copyright 2010-2020 Gentoo Authors
> # Distributed under the terms of the GNU General Public License v2
>
> from __future__ import unicode_literals
> @@ -113,6 +113,7 @@ class RepoConfig(object):
> 'sync_hooks_only_on_change',
> 'sync_openpgp_keyserver',
> 'sync_openpgp_key_path',
> + 'sync_openpgp_key_refresh',
> 'sync_openpgp_key_refresh_retry_count',
> 'sync_openpgp_key_refresh_retry_delay_exp_base',
> 'sync_openpgp_key_refresh_retry_delay_max',
> @@ -233,6 +234,9 @@ class RepoConfig(object):
> self.sync_openpgp_key_path = repo_opts.get(
> 'sync-openpgp-key-path', None)
>
> + self.sync_openpgp_key_refresh = repo_opts.get(
> + 'sync-openpgp-key-refresh', 'true').lower()
> in ('true', 'yes') +
> for k in ('sync_openpgp_key_refresh_retry_count',
> 'sync_openpgp_key_refresh_retry_delay_exp_base',
> 'sync_openpgp_key_refresh_retry_delay_max',
> @@ -497,6 +501,8 @@ class RepoConfig(object):
> repo_msg.append(indent + "location: " +
> self.location) if not self.strict_misc_digests:
> repo_msg.append(indent +
> "strict-misc-digests: false")
> + if not self.sync_openpgp_key_refresh:
> + repo_msg.append(indent +
> "sync-openpgp-key-refresh: no") if self.sync_type:
> repo_msg.append(indent + "sync-type: " +
> self.sync_type) if self.sync_umask:
> @@ -609,6 +615,7 @@ class RepoConfigLoader(object):
>
> 'sync_hooks_only_on_change',
>
> 'sync_openpgp_keyserver',
> 'sync_openpgp_key_path',
> +
> 'sync_openpgp_key_refresh', 'sync_openpgp_key_refresh_retry_count',
>
> 'sync_openpgp_key_refresh_retry_delay_exp_base',
>
> 'sync_openpgp_key_refresh_retry_delay_max',
> @@ -1047,6 +1054,7 @@ class RepoConfigLoader(object):
> bool_keys = (
> "strict_misc_digests",
> "sync_allow_hardlinks",
> + "sync_openpgp_key_refresh",
> "sync_rcu",
> )
> str_or_int_keys = (
> diff --git a/lib/portage/sync/syncbase.py
> b/lib/portage/sync/syncbase.py index 46644d68e..74818a420 100644
> --- a/lib/portage/sync/syncbase.py
> +++ b/lib/portage/sync/syncbase.py
> @@ -1,4 +1,4 @@
> -# Copyright 2014-2018 Gentoo Foundation
> +# Copyright 2014-2020 Gentoo Authors
> # Distributed under the terms of the GNU General Public License v2
>
> '''
> @@ -252,6 +252,13 @@ class SyncBase(object):
> @type openpgp_env: gemato.openpgp.OpenPGPEnvironment
> """
> out = portage.output.EOutput(quiet=('--quiet' in
> self.options['emerge_config'].opts)) +
> + if not self.repo.sync_openpgp_key_refresh:
> + out.ewarn('Key refresh is disabled via a
> repos.conf sync-openpgp-key-refresh')
> + out.ewarn('setting, and this is a security
> vulnerability because it prevents')
> + out.ewarn('detection of revoked keys!')
> + return
> +
> out.ebegin('Refreshing keys via WKD')
> if openpgp_env.refresh_keys_wkd():
> out.eend(0)
> diff --git a/man/portage.5 b/man/portage.5
> index 36c871123..136ebaafe 100644
> --- a/man/portage.5
> +++ b/man/portage.5
> @@ -1,4 +1,4 @@
> -.TH "PORTAGE" "5" "Apr 2019" "Portage VERSION" "Portage"
> +.TH "PORTAGE" "5" "Jun 2020" "Portage VERSION"
Re: [gentoo-portage-dev] [PATCH] Default BINPKG_COMPRESSION to zstd (bug 715108)
On Sun, 10 May 2020 19:29:34 -0700
Zac Medico wrote:
> This includes a _compat_upgrade.binpkg_compression script that the
> ebuild can call in pkg_preinst in order to maintain a
> backward-compatible bzip2 default when appropriate, ensuring that
> binary package consumers are not caught off guard.
>
> Bug: https://bugs.gentoo.org/715108
> Signed-off-by: Zac Medico
> ---
> .travis.yml | 4 ++
> cnf/make.globals | 5 ++-
> .../_compat_upgrade/binpkg_compression.py | 40
> +++ .../tests/resolver/ResolverPlayground.py |
> 1 + man/make.conf.5 | 4 +-
> 5 files changed, 50 insertions(+), 4 deletions(-)
> create mode 100644 lib/portage/_compat_upgrade/binpkg_compression.py
>
> diff --git a/.travis.yml b/.travis.yml
> index 9269d4034..2132c8c87 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -1,3 +1,4 @@
> +dist: bionic
> language: python
> python:
> - 2.7
> @@ -8,6 +9,9 @@ python:
> - pypy3
>
> # command to install dependencies
> +before_install:
> +# Use "dist: bionic" to get a zstd with --long support.
> +- sudo apt-get -y install zstd
> install:
> - pip install tox
>
> diff --git a/cnf/make.globals b/cnf/make.globals
> index 4a59dbe3c..dd3f28f70 100644
> --- a/cnf/make.globals
> +++ b/cnf/make.globals
> @@ -34,8 +34,9 @@ RPMDIR="/var/cache/rpm"
> # Temporary build directory
> PORTAGE_TMPDIR="/var/tmp"
>
> -# The compression used for binary packages. Defaults to zstd when
> USE=zstd is enabled. -BINPKG_COMPRESS="bzip2"
> +# The compression used for binary packages. Defaults to zstd except
> for +# existing installs where bzip2 is used for backward
> compatibility. +BINPKG_COMPRESS="zstd"
>
> # Fetching command (3 tries, passive ftp for firewall compatibility)
> FETCHCOMMAND="wget -t 3 -T 60 --passive-ftp -O
> \"\${DISTDIR}/\${FILE}\" \"\${URI}\"" diff --git
> a/lib/portage/_compat_upgrade/binpkg_compression.py
> b/lib/portage/_compat_upgrade/binpkg_compression.py new file mode
> 100644 index 0..0f5704733 --- /dev/null
> +++ b/lib/portage/_compat_upgrade/binpkg_compression.py
> @@ -0,0 +1,40 @@
> +# Copyright 2020 Gentoo Authors
> +# Distributed under the terms of the GNU General Public License v2
> +
> +import re
> +
> +import portage
> +from portage import os
> +from portage.const import GLOBAL_CONFIG_PATH
> +
> +COMPAT_BINPKG_COMPRESS = 'bzip2'
> +
> +
> +def main():
> + """
> + If the current installation is still configured to use the
> old
> + default BINPKG_COMPRESS=bzip2 setting, then patch
> make.globals
> + inside ${ED} to maintain backward compatibility, ensuring
> that
> + binary package consumers are not caught off guard. This is
> + intended to be called from the ebuild as follows:
> +
> + pkg_preinst() {
> + python_setup
> + env -u BINPKG_COMPRESS
> +
> PYTHONPATH="${D%/}$(python_get_sitedir)${PYTHONPATH:+:${PYTHONPATH}}"
> \
> + "${PYTHON}" -m
> portage._compat_upgrade.binpkg_compression || die
> + }
> + """
> + if portage.settings.get('BINPKG_COMPRESS',
> COMPAT_BINPKG_COMPRESS) == COMPAT_BINPKG_COMPRESS:
> + config_path = os.path.join(os.environ['ED'],
> GLOBAL_CONFIG_PATH.lstrip(os.sep), 'make.globals')
> + with open(config_path) as f:
> + content = f.read()
> + compat_setting =
> 'BINPKG_COMPRESS="{}"'.format(COMPAT_BINPKG_COMPRESS)
> + portage.output.EOutput().einfo('Setting
> make.globals default {} for backward
> compatibility'.format(compat_setting))
> + content = re.sub('^BINPKG_COMPRESS=.*$',
> compat_setting, content, flags=re.MULTILINE)
> + with open(config_path, 'wt') as f:
> + f.write(content)
> +
> +
> +if __name__ == '__main__':
> + main()
> diff --git a/lib/portage/tests/resolver/ResolverPlayground.py
> b/lib/portage/tests/resolver/ResolverPlayground.py index
> 98831e000..de80a0cc1 100644 ---
> a/lib/portage/tests/resolver/ResolverPlayground.py +++
> b/lib/portage/tests/resolver/ResolverPlayground.py @@ -112,6 +112,7
> @@ class ResolverPlayground(object): "uname",
> "uniq",
> "xargs",
> + "zstd",
> )
> # Exclude internal wrappers from PATH lookup.
> orig_path = os.environ['PATH']
> diff --git a/man/make.conf.5 b/man/make.conf.5
> index f82fed65a..a3bd662ae 100644
> --- a/man/make.conf.5
> +++ b/man/make.conf.5
> @@ -1,4 +1,4 @@
> -.TH "MAKE.CONF" "5" "Nov 2019" "Portage VERSION" "Portage"
> +.TH "MAKE.CONF" "5" "May 2020" "Portage VERSION" "Portage"
> .SH "NAME"
> make.conf \- custom settings for Portage
> .SH "SYNOPSIS"
> @@ -115,7 +115,7 @@ This variable is used to determine the
> compression used for \fIbinary packages\fR.
Re: [gentoo-portage-dev] [PATCH] Enable FEATURES=parallel-install by default (bug 715110)
On Sun, 10 May 2020 21:32:25 -0700
Zac Medico wrote:
> The feature enables finer grained locks for install operations, and
> everyone agrees that it's safe to enable by default.
>
> Bug: https://bugs.gentoo.org/715110
> Signed-off-by: Zac Medico
> ---
> cnf/make.globals | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/cnf/make.globals b/cnf/make.globals
> index 4a59dbe3c..5ba1ac6fa 100644
> --- a/cnf/make.globals
> +++ b/cnf/make.globals
> @@ -55,7 +55,7 @@ FETCHCOMMAND_SFTP="bash -c \"x=\\\${2#sftp://} ;
> host=\\\${x%%/*} ; port=\\\${ho FEATURES="assume-digests
> binpkg-docompress binpkg-dostrip binpkg-logs
> config-protect-if-modified distlocks ebuild-locks fixlafiles
> ipc-sandbox merge-sync multilib-strict
> - network-sandbox news parallel-fetch pid-sandbox
> + network-sandbox news parallel-fetch parallel-install
> pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps
>sandbox sfperms strict
>unknown-features-warn unmerge-logs unmerge-orphans
> userfetch
works for me :)
Re: [gentoo-portage-dev] [PATCH] make.globals: add default BINPKG_COMPRESS setting (bug 715108)
On Sun, 26 Apr 2020 14:57:32 -0700
Zac Medico wrote:
> The ebuild will have a default enabled USE=zstd which changes the
> default to zstd here.
>
> Bug: https://bugs.gentoo.org/715108
> Bug: https://bugs.gentoo.org/719456
> Signed-off-by: Zac Medico
> ---
> cnf/make.globals | 5 -
> 1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/cnf/make.globals b/cnf/make.globals
> index 139e1ce97..4a59dbe3c 100644
> --- a/cnf/make.globals
> +++ b/cnf/make.globals
> @@ -1,4 +1,4 @@
> -# Copyright 1999-2019 Gentoo Authors
> +# Copyright 1999-2020 Gentoo Authors
> # Distributed under the terms of the GNU General Public License v2
> # System-wide defaults for the Portage system
>
> @@ -34,6 +34,9 @@ RPMDIR="/var/cache/rpm"
> # Temporary build directory
> PORTAGE_TMPDIR="/var/tmp"
>
> +# The compression used for binary packages. Defaults to zstd when
> USE=zstd is enabled. +BINPKG_COMPRESS="bzip2"
> +
> # Fetching command (3 tries, passive ftp for firewall compatibility)
> FETCHCOMMAND="wget -t 3 -T 60 --passive-ftp -O
> \"\${DISTDIR}/\${FILE}\" \"\${URI}\"" RESUMECOMMAND="wget -c -t 3 -T
> 60 --passive-ftp -O \"\${DISTDIR}/\${FILE}\" \"\${URI}\""
works for me :)
Re: [gentoo-portage-dev] [PATCH] depgraph: respect
On Sat, 11 Apr 2020 17:39:53 -0700
Zac Medico wrote:
> When searching for slot operator rebuilds, respect non slot-operator
> components of parent dependencies, so that a like the not be completely ignored. This will prevent erroneous attempts to
> trigger slot operator rebuilds for upgrades that would break
> when backtracking tried to resolve the breakage).
>
> Bug: https://bugs.gentoo.org/717140
> Signed-off-by: Zac Medico
> ---
> lib/_emerge/depgraph.py | 12
> +--- .../resolver/test_slot_operator_reverse_deps.py |
> 1 - 2 files changed, 9 insertions(+), 4 deletions(-)
>
> diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
> index 6d1f62178..ec90e59df 100644
> --- a/lib/_emerge/depgraph.py
> +++ b/lib/_emerge/depgraph.py
> @@ -2068,9 +2068,15 @@ class depgraph(object):
> for parent, atom in
> self._dynamic_config._parent_atoms.get(existing_pkg, []): if
> isinstance(parent, Package): if parent in built_slot_operator_parents:
> - # This parent may
> need to be rebuilt, so its
> - # dependencies
> aren't necessarily relevant.
> - continue
> + # This parent may
> need to be rebuilt, therefore
> + # discard its soname
> and built slot operator
> + # dependency
> components which are not necessarily
> + # relevant.
> + if atom.soname:
> + continue
> + elif atom.package
> and atom.slot_operator_built:
> + # This
> discards the slot/subslot component.
> + atom =
> atom.with_slot("=")
> if replacement_parent is not
> None and \ (replacement_parent.slot_atom == parent.slot_atom
> diff --git
> a/lib/portage/tests/resolver/test_slot_operator_reverse_deps.py
> b/lib/portage/tests/resolver/test_slot_operator_reverse_deps.py index
> 5c5295510..e0e738d6b 100644 ---
> a/lib/portage/tests/resolver/test_slot_operator_reverse_deps.py +++
> b/lib/portage/tests/resolver/test_slot_operator_reverse_deps.py @@
> -122,7 +122,6 @@ class
> SlotOperatorReverseDepsLibGit2TestCase(TestCase): trigger an upgrade
> to dev-libs/libgit2-1.0.0-r1, ultimately resulting in an undesirable
> downgrade to dev-libs/libgit2-0.28.4-r1. """
> - self.todo = True
>
> ebuilds = {
>
looks fine to me :)
Re: [gentoo-portage-dev] Portage project lead election
On Tue, 16 Apr 2019 13:05:40 -0700 Zac Medico wrote: > Hi all, > > We're due for a Portage project lead election [1] which is required by > GLEP 39 [2]. Please reply with nominations. Thanks! > > [1] https://bugs.gentoo.org/683510 > [2] https://www.gentoo.org/glep/glep-0039.html Well, as Zac is the only real active dev in the team, I nominate him. I have been debating withdrawing from the team as I am quite busy with other things atm. I am also looking to establish a new source of income. But I may have to retire... I will be going to the east coast for some months to work on some things that should help minimize my finacial load requirements. So, I will be quite inactive for some time while I establish myself there. Thanks Zac...
Re: [gentoo-portage-dev] [PATCH] locks: handle sshfs hardlink inode numbers (bug 678218)
On Sun, 17 Feb 2019 15:04:29 -0800 Zac Medico wrote: > Since hardlinks on sshfs do not have matching inode numbers, detect > this behavior and use a simple stat call to detect if lock_path has > been removed. > > Bug: https://bugs.gentoo.org/678218 > Signed-off-by: Zac Medico > --- > lib/portage/locks.py | 27 +++ > 1 file changed, 27 insertions(+) > > diff --git a/lib/portage/locks.py b/lib/portage/locks.py > index 74c2c086a..510925da0 100644 > --- a/lib/portage/locks.py > +++ b/lib/portage/locks.py > @@ -340,6 +340,33 @@ def _lockfile_was_removed(lock_fd, lock_path): > > hardlink_stat = os.stat(hardlink_path) > if hardlink_stat.st_ino != fstat_st.st_ino or > hardlink_stat.st_dev != fstat_st.st_dev: > + # Create another hardlink in order to detect > whether or not > + # hardlink inode numbers are expected to > match. For example, > + # inode numbers are not expected to match > for sshfs. > + inode_test = hardlink_path + '-inode-test' > + try: > + os.unlink(inode_test) > + except OSError as e: > + if e.errno not in (errno.ENOENT, > errno.ESTALE): > + _raise_exc(e) > + try: > + os.link(hardlink_path, inode_test) > + except OSError as e: > + if e.errno not in (errno.ENOENT, > errno.ESTALE): > + _raise_exc(e) > + return True > + else: > + if not > os.path.samefile(hardlink_path, inode_test): > + # This implies that inode > numbers are not expected > + # to match for this file > system, so use a simple > + # stat call to detect if > lock_path has been removed. > + return not > os.path.exists(lock_path) > + finally: > + try: > + os.unlink(inode_test) > + except OSError as e: > + if e.errno not in > (errno.ENOENT, errno.ESTALE): > + _raise_exc(e) > return True > finally: > try: Looks fine to me
Re: [gentoo-portage-dev] [PATCH] pid-sandbox: execute pid-ns-init as pid 1 (bug 675312)
On Sun, 13 Jan 2019 16:27:21 -0800
Zac Medico wrote:
> Execute pid-ns-init as the first fork after unshare, as
> required for it to have pid 1 and become the default reaper
> of orphaned descendant processes. In _exec, exec a separate
> pid-ns-init process to behave as a supervisor which will
> forward signals to init and forward exit status to the parent
> process.
>
> Fixes: a75d5546e3a4 ("Introduce a tiny init replacement for inside
> pid namespace") Bug: https://bugs.gentoo.org/670484
> ---
> bin/pid-ns-init| 44
> ++ lib/portage/process.py |
> 26 ++--- 2 files changed, 59 insertions(+), 11
> deletions(-)
>
> diff --git a/bin/pid-ns-init b/bin/pid-ns-init
> index 843257b70..3792eeaa4 100644
> --- a/bin/pid-ns-init
> +++ b/bin/pid-ns-init
> @@ -1,23 +1,59 @@
> #!/usr/bin/env python
> -# Copyright 2018 Gentoo Authors
> +# Copyright 2018-2019 Gentoo Authors
> # Distributed under the terms of the GNU General Public License v2
>
> +import functools
> import os
> +import signal
> import sys
>
>
> +KILL_SIGNALS = (
> + signal.SIGINT,
> + signal.SIGTERM,
> + signal.SIGHUP,
> +)
> +
> +def forward_kill_signal(main_child_pid, signum, frame):
> + os.kill(main_child_pid, signum)
> +
> +
> def main(argv):
> if len(argv) < 2:
> - return 'Usage: {} '.format(argv[0])
> - main_child_pid = int(argv[1])
> + return 'Usage: {} or
> argv0..'.format(argv[0]) +
> + if len(argv) == 2:
> + # The child process is init (pid 1) in a child pid
> namespace, and
> + # the current process supervises from within the
> global pid namespace
> + # (forwarding signals to init and forwarding exit
> status to the parent
> + # process).
> + main_child_pid = int(argv[1])
> + else:
> + # The current process is init (pid 1) in a child pid
> namespace.
> + binary = argv[1]
> + args = argv[2:]
> +
> + main_child_pid = os.fork()
> + if main_child_pid == 0:
> + os.execv(binary, args)
> +
> + sig_handler = functools.partial(forward_kill_signal,
> main_child_pid)
> + for signum in KILL_SIGNALS:
> + signal.signal(signum, sig_handler)
>
> # wait for child processes
> while True:
> - pid, status = os.wait()
> + try:
> + pid, status = os.wait()
> + except OSError as e:
> + if e.errno == errno.EINTR:
> + continue
> + raise
> if pid == main_child_pid:
> if os.WIFEXITED(status):
> return os.WEXITSTATUS(status)
> elif os.WIFSIGNALED(status):
> + signal.signal(os.WTERMSIG(status),
> signal.SIG_DFL) os.kill(os.getpid(), os.WTERMSIG(status))
> # go to the unreachable place
> break
> diff --git a/lib/portage/process.py b/lib/portage/process.py
> index 7103b6b31..3e07f806c 100644
> --- a/lib/portage/process.py
> +++ b/lib/portage/process.py
> @@ -564,15 +564,27 @@ def _exec(binary, mycommand, opt_name, fd_pipes,
> noiselevel=-1)
> else:
> if unshare_pid:
> - # pid
> namespace requires us to become init
> - fork_ret =
> os.fork()
> - if
> fork_ret != 0:
> -
> os.execv(portage._python_interpreter, [
> +
> main_child_pid = os.fork()
> + if
> main_child_pid == 0:
> + #
> pid namespace requires us to become init
> +
> binary, myargs = portage._python_interpreter, [
> +
> portage._python_interpreter,
> +
> os.path.join(portage._bin_path,
> +
> 'pid-ns-init')]
> + [binary] + myargs
> + else:
> + #
> Execute a supervisor process which will forward
> + #
> signals to init and forward exit status to the
> + #
> parent process. The supervisor process runs in
> + #
> the global pid namespace, so skip /proc remount
> + #
> and other setup that's intended only for the
> + #
> init process.
> +
>
Re: [gentoo-portage-dev] [PATCH] portage.process.spawn: inherit env by default (bug 672440)
On Sun, 2 Dec 2018 23:55:23 -0800
Zac Medico wrote:
> Make child processes inherit the current process's environment by
> default, so that behavior is equivalent to the standard library's
> subprocess module.
>
> Bug: https://bugs.gentoo.org/672440
> Signed-off-by: Zac Medico
> ---
> lib/portage/process.py | 10 +++---
> 1 file changed, 7 insertions(+), 3 deletions(-)
>
> diff --git a/lib/portage/process.py b/lib/portage/process.py
> index ed1a49247..ce3e42a8f 100644
> --- a/lib/portage/process.py
> +++ b/lib/portage/process.py
> @@ -219,7 +219,7 @@ spawned_pids = _dummy_list()
> def cleanup():
> pass
>
> -def spawn(mycommand, env={}, opt_name=None, fd_pipes=None,
> returnpid=False, +def spawn(mycommand, env=None, opt_name=None,
> fd_pipes=None, returnpid=False, uid=None, gid=None, groups=None,
> umask=None, cwd=None, logfile=None, path_lookup=True, pre_exec=None,
>close_fds=(sys.version_info < (3, 4)), unshare_net=False,
> @@ -230,8 +230,10 @@ def spawn(mycommand, env={}, opt_name=None,
> fd_pipes=None, returnpid=False,
> @param mycommand: the command to execute
> @type mycommand: String or List (Popen style list)
> - @param env: A dict of Key=Value pairs for env variables
> - @type env: Dictionary
> + @param env: If env is not None, it must be a mapping that
> defines the environment
> + variables for the new process; these are used
> instead of the default behavior
> + of inheriting the current process's environment.
> + @type env: None or Mapping
> @param opt_name: an optional name for the spawn'd process
> (defaults to the binary name) @type opt_name: String
> @param fd_pipes: A dict of mapping for pipes, { '0': stdin,
> '1': stdout } for example @@ -281,6 +283,8 @@ def spawn(mycommand,
> env={}, opt_name=None, fd_pipes=None, returnpid=False, if
> isinstance(mycommand, basestring): mycommand = mycommand.split()
>
> + env = os.environ if env is None else env
> +
> if sys.hexversion < 0x300:
> # Avoid a potential UnicodeEncodeError from
> os.execve(). env_bytes = {}
LGTM
Re: [gentoo-portage-dev] [PATCH 0/2] git: drop privileges for gc and merge (bug 669496)
On Fri, 23 Nov 2018 16:15:23 -0800
Zac Medico wrote:
> Use portage.process.spawn (with new cwd parameter) and
> self.spawn_kwargs to drop privileges for git gc and merge commands.
>
> Fixes: 3cd8cf93abb6 ("GitSync: abort checkout for signature problem
> (bug 660372)") Fixes: 903c4b1a6768 ("GitSync: support sync-depth (bug
> 552814)") Bug: https://bugs.gentoo.org/669496
> Signed-off-by: Zac Medico
>
> Zac Medico (2):
> portage.process.spawn: add cwd parameter
> git: drop privileges for gc and merge (bug 669496)
>
> lib/portage/process.py| 13 ++---
> lib/portage/sync/modules/git/git.py | 10 ++
> lib/portage/tests/sync/test_sync_local.py | 22 --
> 3 files changed, 36 insertions(+), 9 deletions(-)
>
This looks good to me :)
Re: [gentoo-portage-dev] [PATCH] phase-helpers.sh: has/best_version -b in any phase (bug 665038)
On Sun, 2 Sep 2018 15:03:28 -0700
Zac Medico wrote:
> Since BROOT is only defined in src_* phases, make has/best_version -b
> use the equivalent /${PORTAGE_OVERRIDE_EPREFIX#/} value that is
> defined in all phases. This makes has/best_version -b in EAPI 7
> behave exactly the same as --host-root behaves in EAPI 6, allowing
> python ebuilds to call python_setup in any ebuild phase.
>
> Bug: https://bugs.gentoo.org/665038
> ---
> bin/phase-helpers.sh | 7 ++-
> 1 file changed, 6 insertions(+), 1 deletion(-)
>
> diff --git a/bin/phase-helpers.sh b/bin/phase-helpers.sh
> index 5c9f957e9..ba3f27930 100644
> --- a/bin/phase-helpers.sh
> +++ b/bin/phase-helpers.sh
> @@ -916,7 +916,12 @@ ___best_version_and_has_version_common() {
> case ${root_arg} in
> -r)
> root=${ROOT%/}/${EPREFIX#/} ;; -d) root=${ESYSROOT} ;;
> - -b) root=${BROOT:-/} ;;
> + -b)
> + #
> Use /${PORTAGE_OVERRIDE_EPREFIX#/} which is equivalent
> + # to BROOT, except
> BROOT is only defined in src_* phases.
> +
> root=/${PORTAGE_OVERRIDE_EPREFIX#/}
> + cmd+=(env
> EPREFIX="${PORTAGE_OVERRIDE_EPREFIX}")
> + ;;
> esac
> else
> case ${root_arg} in
> --
> 2.16.4
>
>
LGTM
Re: [gentoo-portage-dev] [PATCH 0/4] Add sync-rcu support for rsync (bug 662070)
On Mon, 6 Aug 2018 00:40:29 -0700 Zac Medico wrote: > Add a boolean sync-rcu repos.conf setting that behaves as follows: > > sync-rcu = yes|no > > Enable read-copy-update (RCU) behavior for sync operations. The > current latest immutable version of a repository will be > referenced by a symlink found where the repository would normally be > located (see the location setting). Repository consumers should > resolve the cannonical path of this symlink before attempt to access > the repository, and all operations should be read-only, since > the repository is considered immutable. Updates occur by atomic > replacement of the symlink, which causes new consumers to use the > new immutable version, while any earlier consumers continue to > use the cannonical path that was resolved earlier. This option > requires sync-allow-hardlinks and sync-rcu-store-dir options to > be enabled, and currently also requires that sync-type is set > to rsync. This option is disabled by default, since the symlink > usage would require special handling for scenarios involving bind > mounts and chroots. > > sync-rcu-store-dir > > Directory path reserved for sync-rcu storage. This directory must > have a unique value for each repository (do not set it in the > DEFAULT section). This directory must not contain any other files > or directories aside from those that are created automatically > when sync-rcu is enabled. > > sync-rcu-spare-snapshots = 1 > > Number of spare snapshots for sync-rcu to retain with expired > ttl. This protects the previous latest snapshot from being removed > immediately after a new version becomes available, since it might > still be used by running processes. > > sync-rcu-ttl-days = 7 > > Number of days for sync-rcu to retain previous immutable snapshots > of a repository. After the ttl of a particular snapshot has > expired, it will be remove automatically (the latest snapshot > is exempt, and sync-rcu-spare-snapshots configures the number of > previous snapshots that are exempt). If the ttl is set too low, > then a snapshot could expire while it is in use by a running > process. > > Zac Medico (4): > Implement asyncio.iscoroutinefunction for compat_coroutine > Add _sync_decorator module > rsync: split out repo storage framework > Add sync-rcu support for rsync (bug 662070) > > lib/portage/repository/config.py | 36 ++- > lib/portage/repository/storage/__init__.py | 0 > .../repository/storage/hardlink_quarantine.py | 95 > lib/portage/repository/storage/hardlink_rcu.py | 251 > + > lib/portage/repository/storage/inplace.py | 49 > lib/portage/repository/storage/interface.py| 87 +++ > lib/portage/sync/controller.py | 1 + > lib/portage/sync/modules/rsync/rsync.py| 85 ++- > lib/portage/sync/syncbase.py | 33 > +++ .../tests/util/futures/test_compat_coroutine.py| 14 ++ > lib/portage/util/futures/_asyncio/__init__.py | 14 ++ > lib/portage/util/futures/_sync_decorator.py| 54 + > lib/portage/util/futures/compat_coroutine.py | 12 + > man/portage.5 | 35 +++ 14 files > changed, 700 insertions(+), 66 deletions(-) create mode 100644 > lib/portage/repository/storage/__init__.py create mode 100644 > lib/portage/repository/storage/hardlink_quarantine.py create mode > 100644 lib/portage/repository/storage/hardlink_rcu.py create mode > 100644 lib/portage/repository/storage/inplace.py create mode 100644 > lib/portage/repository/storage/interface.py create mode 100644 > lib/portage/util/futures/_sync_decorator.py > series looks good, just the one typo
Re: [gentoo-portage-dev] [PATCH 3/4] rsync: split out repo storage framework
On Mon, 6 Aug 2018 00:40:32 -0700 Zac Medico wrote: > Since there aremany ways to manage repository storage, split out a > repo storage framework. The HardlinkQuarantineRepoStorage class > implements the existing default behavior, and the InplaceRepoStorage > class implements the legacy behavior (when sync-allow-hardlinks is > disabled in repos.conf). > > Each class implements RepoStorageInterface, which uses coroutine > methods since coroutines are well-suited to the I/O bound tasks that > these methods perform. The _sync_decorator is used to convert > coroutine methods to synchronous methods, for smooth integration into > the surrounding synchronous code. > > Bug: https://bugs.gentoo.org/662070 > missing space in first line of commit message s/aremany/are many ---
Re: [gentoo-portage-dev] [PATCH] XARGS: use gxargs for USERLAND=BSD (bug 663256)
On Thu, 9 Aug 2018 16:04:42 -0700
Zac Medico wrote:
> For USERLAND=BSD, set XARGS="gxargs -r" if gxargs is available,
> so the code from bug 630292 works for USERLAND=BSD.
>
> Fixes: 50283f1abb77 (install-qa-check.d/60pngfix: parallel support
> (bug 630292)) Reported-by: Michał Górny
> Bug: https://bugs.gentoo.org/663256
> ---
> bin/isolated-functions.sh | 6 +-
> 1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/bin/isolated-functions.sh b/bin/isolated-functions.sh
> index 28ca94532..cac42a4c5 100644
> --- a/bin/isolated-functions.sh
> +++ b/bin/isolated-functions.sh
> @@ -448,7 +448,11 @@ fi
> if [[ -z ${XARGS} ]] ; then
> case ${USERLAND} in
> BSD)
> - export XARGS="xargs"
> + if type -P gxargs > /dev/null; then
> + export XARGS="gxargs -r"
> + else
> + export XARGS="xargs"
> + fi
> ;;
> *)
> export XARGS="xargs -r"
LGTM
Re: [gentoo-portage-dev] [PATCH 2/2 v2] Update /usr/portage references (bug 378603)
On Sun, 5 Aug 2018 22:32:34 -0700
Zac Medico wrote:
> Update all relevant references in docs, messages, and comments
> to refer to /var/db/repos/gentoo instead of /usr/portage. Also
> update DISTDIR and PKGDIR references to refer to the new
> /var/cache/{distfiles,binpkgs} locations.
>
> Bug: https://bugs.gentoo.org/378603
> ---
> [PATCH 2/2 v2] fixes DISTDIR and PKGDIR references to rever to the
> new /var/cache/{distfiles,binpkgs} locations
>
> cnf/make.conf.example| 12 +--
> lib/portage/__init__.py | 2 +-
> lib/portage/cache/flat_hash.py | 2 +-
> lib/portage/tests/news/test_NewsItem.py | 2 +-
> lib/portage/tests/resolver/ResolverPlayground.py | 2 +-
> lib/portage/xml/metadata.py | 4 ++--
> man/ebuild.5 | 4 ++--
> man/emerge.1 | 6 +++---
> man/make.conf.5 | 20
> +- man/portage.5|
> 26
> man/quickpkg.1 | 2 +-
> repoman/lib/repoman/__init__.py | 2 +-
> repoman/lib/repoman/checks/herds/herdbase.py | 2 +- 13 files
> changed, 43 insertions(+), 43 deletions(-)
>
> diff --git a/cnf/make.conf.example b/cnf/make.conf.example
> index 04f3a0274..c16f2afbd 100644
> --- a/cnf/make.conf.example
> +++ b/cnf/make.conf.example
> @@ -14,7 +14,7 @@
> # https://wiki.gentoo.org/wiki/Handbook:X86/Working/USE
> #
> # The available list of use flags with descriptions is in your
> portage tree. -# Use 'less' to view them: -->
> less /usr/portage/profiles/use.desc <-- +# Use 'less' to view them:
> --> less /var/db/repos/gentoo/profiles/use.desc <-- #
> # 'ufed' is an ncurses/dialog interface available in portage to make
> handling # useflags for you. 'emerge app-portage/ufed'
> @@ -111,7 +111,7 @@
> # will protect the default locations of DISTDIR and PKGDIR, but
> users are # warned that any other locations inside PORTDIR are
> not necessarily safe # for data storage.
> -#PORTDIR=/usr/portage
> +#PORTDIR=/var/db/repos/gentoo
> #
> # DISTDIR is where all of the source code tarballs will be placed for
> # emerges. After packages are built, it is safe to remove any and
> @@ -119,16 +119,16 @@
> # fetched on demand for a given build. If you would like to
> # selectively prune obsolete files from this directory, see
> # eclean from the gentoolkit package. Note that locations under
> -# /usr/portage are not necessarily safe for data storage. See the
> +# /var/db/repos/gentoo are not necessarily safe for data
> storage. See the # PORTDIR documentation for more information.
> -#DISTDIR=/usr/portage/distfiles
> +#DISTDIR=/var/cache/distfiles
> #
> # PKGDIR is the location of binary packages that you can have created
> # with '--buildpkg' or '-b' while emerging a package. This can
> get # up to several hundred megs, or even a few gigs. Note that
> -# locations under /usr/portage are not necessarily safe for data
> +# locations under /var/db/repos/gentoo are not necessarily safe
> for data # storage. See the PORTDIR documentation for more
> information. -#PKGDIR=/usr/portage/packages
> +#PKGDIR=/var/cache/binpkgs
> #
> # PORT_LOGDIR is the location where portage will store all the logs
> it # creates from each individual merge. They are stored as
> diff --git a/lib/portage/__init__.py b/lib/portage/__init__.py
> index 166bfc700..61a240100 100644
> --- a/lib/portage/__init__.py
> +++ b/lib/portage/__init__.py
> @@ -133,7 +133,7 @@ except ImportError as e:
> sys.stderr.write("!!! Failed to complete portage imports.
> There are internal modules for\n") sys.stderr.write("!!! portage and
> failure here indicates that you have a problem with your\n")
> sys.stderr.write("!!! installation of portage. Please try a rescue
> portage located in the\n")
> - sys.stderr.write("!!! portage tree under
> '/usr/portage/sys-apps/portage/files/' (default).\n")
> + sys.stderr.write("!!! portage tree under
> '/var/db/repos/gentoo/sys-apps/portage/files/' (default).\n")
This should also get updated replacing "portage tree" with "gentoo
tree"
...
> The packages, after being created, will be placed in \fBPKGDIR\fR.
> This variable is defined in \fBmake.conf\fR(5) and defaults to
> -/usr/portage/packages.
> +/var/cache/binpkgs.
> .SH OPTIONS
> .TP
> .B
> diff --git a/repoman/lib/repoman/__init__.py
> b/repoman/lib/repoman/__init__.py index 89779b95c..d1312a267 100644
> --- a/repoman/lib/repoman/__init__.py
> +++ b/repoman/lib/repoman/__init__.py
> @@ -14,7 +14,7 @@ except ImportError as e:
> sys.stderr.write("!!! Failed to complete portage imports.
> There are internal modules for\n") sys.stderr.write("!!! portage and
> failure here indicates that you have a problem with your\n")
>
Re: [gentoo-portage-dev] [PATCH 2/2] Update /usr/portage references (bug 378603)
On Sun, 5 Aug 2018 22:46:58 -0700
Zac Medico wrote:
> On 08/05/2018 09:59 PM, Ulrich Mueller wrote:
> >> On Sun, 5 Aug 2018, Zac Medico wrote:
> >
> >> --- a/cnf/make.conf.example
> >> +++ b/cnf/make.conf.example
> >> [...]
> >
> >> @@ -119,16 +119,16 @@
> >> # fetched on demand for a given build. If you would like to
> >> # selectively prune obsolete files from this directory, see
> >> # eclean from the gentoolkit package. Note that locations
> >> under -# /usr/portage are not necessarily safe for data
> >> storage. See the +# /var/db/repos/gentoo are not necessarily
> >> safe for data storage. See the # PORTDIR documentation for
> >> more information. -#DISTDIR=/usr/portage/distfiles
> >> +#DISTDIR=/var/db/repos/gentoo/distfiles
> >
> > Shouldn't this be /var/cache/distfiles ...
> >
> >> #
> >> # PKGDIR is the location of binary packages that you can have
> >> created # with '--buildpkg' or '-b' while emerging a package.
> >> This can get # up to several hundred megs, or even a few gigs.
> >> Note that -# locations under /usr/portage are not necessarily
> >> safe for data +# locations under /var/db/repos/gentoo are not
> >> necessarily safe for data # storage. See the PORTDIR
> >> documentation for more information. -#PKGDIR=/usr/portage/packages
> >> +#PKGDIR=/var/db/repos/gentoo/packages
> >
> > ... and /var/cache/binpkgs?
>
> Thanks, I've fixed the /var/cache/{distfiles,binpkgs} locations in v2.
>
> >> --- a/lib/portage/cache/flat_hash.py
> >> +++ b/lib/portage/cache/flat_hash.py
> >> @@ -144,7 +144,7 @@ class database(fs_template.FsBased):
> >># Only recurse 1 deep, in
> >> order to avoid iteration over # entries from another nested cache
> >> instance. This can # happen if the user nests an overlay inside
> >> - # /usr/portage/local as
> >> in bug #302764.
> >> +
> >> # /var/db/repos/gentoo/local as in bug #302764.
> >
> > Shouldn't a local overlay be in /var/db/repos/local, but never
> > inside of the gentoo repo?
>
> Yes, but for a long time /usr/local/portage was somewhat standard, in
> fact it's still mentioned here:
>
> https://wiki.gentoo.org/wiki/Custom_repository
>
> Nowadays, repository verification will prevent that from working...
>
But that too can be changed along with all the user install
documentation which will need to be updated as well. The new
recomended location should be /var/db/repos/local. I will be updating
layman for /var/db/repos/... as well. That is the intention of the
"repos/" subdir.
All these changes as well as the catlayst changes need to be
co-ordinated so that snapshots and portage and stages don't precede the
docs changes.
> >> --- a/man/make.conf.5
> >> +++ b/man/make.conf.5
> >> @@ -219,10 +219,10 @@ Use the \fBPORTAGE_RO_DISTDIRS\fR variable
> >> to specify one or more read-only directories containing distfiles.
> >>
> >> Note
> >> -that locations under /usr/portage are not necessarily safe for
> >> data storage. +that locations under /var/db/repos/gentoo are not
> >> necessarily safe for data storage.
> >
> > IMHO this statement should be made much stronger.
>
> Yes, we could also say something about repository verification here.
> The old defaults set a really bad example, so we needed statements
> like this to steer people in the right direction.
> --
> Thanks,
> Zac
>
pgpT1yfKTqhPZ.pgp
Description: OpenPGP digital signature
Re: [gentoo-portage-dev] [PATCH] repoman: populate implicit IUSE for empty profile (bug 660982)
On Sun, 15 Jul 2018 16:02:03 -0700 Zac Medico wrote: > For the empty profile that's used to check dependencies of > ebuilds that have empty KEYWORDS, populate implicit IUSE > from all of the make.defaults files found in the relevant > repositories (this should take less than 1 second on most > hardware). Since the IUSE.missing check cannot be performed > without implicit IUSE settings, this makes the IUSE.missing > check work for ebuilds with empty KEYWORDS. > > Bug: https://bugs.gentoo.org/660982 > --- > pym/portage/dbapi/__init__.py | 10 +++- > repoman/pym/repoman/modules/scan/depend/profile.py | 30 > +- 2 files changed, 32 insertions(+), 8 > deletions(-) > > diff --git a/pym/portage/dbapi/__init__.py > b/pym/portage/dbapi/__init__.py index 61d301839..6fca6090c 100644 > --- a/pym/portage/dbapi/__init__.py > +++ b/pym/portage/dbapi/__init__.py > @@ -219,17 +219,13 @@ class dbapi(object): > def _repoman_iuse_implicit_cnstr(self, pkg, metadata): > """ > In repoman's version of _iuse_implicit_cnstr, > account for modifications > - of the self.settings reference between calls, and > treat all flags as > - valid for the empty profile because it does not have > any implicit IUSE > - settings. See bug 660982. > + of the self.settings reference between calls. > """ > eapi_attrs = _get_eapi_attrs(metadata["EAPI"]) > if eapi_attrs.iuse_effective: > - iuse_implicit_match = lambda flag: (True if > not self.settings.profile_path > - else > self.settings._iuse_effective_match(flag)) > + iuse_implicit_match = lambda flag: > self.settings._iuse_effective_match(flag) else: > - iuse_implicit_match = lambda flag: (True if > not self.settings.profile_path > - else > self.settings._iuse_implicit_match(flag)) > + iuse_implicit_match = lambda flag: > self.settings._iuse_implicit_match(flag) return iuse_implicit_match > > def _iuse_implicit_cnstr(self, pkg, metadata): > diff --git a/repoman/pym/repoman/modules/scan/depend/profile.py > b/repoman/pym/repoman/modules/scan/depend/profile.py index > 8e0a22f9c..233ed8e4b 100644 --- > a/repoman/pym/repoman/modules/scan/depend/profile.py +++ > b/repoman/pym/repoman/modules/scan/depend/profile.py @@ -2,6 +2,7 @@ > > > import copy > +import os > from pprint import pformat > > from _emerge.Package import Package > @@ -12,7 +13,8 @@ from repoman.modules.scan.scanbase import ScanBase > from repoman.modules.scan.depend._depend_checks import _depend_checks > from repoman.modules.scan.depend._gen_arches import _gen_arches > from portage.dep import Atom > - > +from portage.package.ebuild.profile_iuse import iter_iuse_vars > +from portage.util import getconfig > > def sort_key(item): > return item[2].sub_path > @@ -102,6 +104,10 @@ class ProfileDependsChecks(ScanBase): > local_config=False, > > _unmatched_removal=self.options.unmatched_removal, > env=self.env, > repositories=self.repo_settings.repoman_settings.repositories) + > + if not prof.abs_path: > + > self._populate_implicit_iuse(dep_settings) + > dep_settings.categories = > self.repo_settings.repoman_settings.categories if > self.options.without_mask: dep_settings._mask_manager_obj = \ > @@ -257,3 +263,25 @@ class ProfileDependsChecks(ScanBase): > def runInEbuilds(self): > '''Ebuild level scans''' > return (True, [self.check]) > + > + @staticmethod > + def _populate_implicit_iuse(config): > + """ > + Populate implicit IUSE for the empty profile, see > bug 660982. > + """ > + dest = config.configdict['defaults'] > + for repo in config.repositories: > + for parent_dir, dirs, files in > os.walk(os.path.join(repo.location, 'profiles')): > + src = > getconfig(os.path.join(parent_dir, 'make.defaults')) > + if not src: > + continue > + for k, v in iter_iuse_vars(src): > + v_before = dest.get(k) > + if v_before is not None: > + merged_values = > set(v_before.split()) > + > merged_values.update(v.split()) > + v = ' > '.join(sorted(merged_values)) > + dest[k] = v > + > + config.regenerate() > + config._init_iuse() looks good
Re: [gentoo-portage-dev] [PATCH] RepoConfig: fix make.conf PORTDIR override (bug 661276)
On Sun, 15 Jul 2018 14:09:24 -0700 Zac Medico wrote: > Pass the main-repo name from repos.conf as the repository name > for the PORTDIR RepoConfig constructor, so that it can override > the main repo location even if the profiles/repo_name file does > not exist yet (like in a stage3 tarball). > > Bug: https://bugs.gentoo.org/661276 > Reported-by: Jorge Manuel B. S. Vicetto > --- > pym/portage/repository/config.py | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/pym/portage/repository/config.py > b/pym/portage/repository/config.py index bf2b6dd03..e45e67dec 100644 > --- a/pym/portage/repository/config.py > +++ b/pym/portage/repository/config.py > @@ -524,7 +524,8 @@ class RepoConfigLoader(object): > (base_priority == 0 and ov > is portdir): repo_opts = default_repo_opts.copy() > repo_opts['location'] = ov > - repo = RepoConfig(None, > repo_opts, local_config=local_config) > + name = > prepos['DEFAULT'].main_repo if ov is portdir else None > + repo = RepoConfig(name, > repo_opts, local_config=local_config) # repos_conf_opts contains > options from repos.conf repos_conf_opts = repos_conf.get(repo.name) > if repos_conf_opts is not > None: looks good
Re: [gentoo-portage-dev] [PATCH 0/5] webrsync: support sync-openpgp-key-path (bug 661838)
On Fri, 27 Jul 2018 00:56:40 -0700 Zac Medico wrote: > Add repos.conf settings that enable sync-openpgp-key-path support for > both emerge-webrsync and emerge-delta-webrsync: > > sync-webrsync-delta = true|false > Use app-portage/emerge-delta-webrsync to minimize > bandwidth. Defaults to false. > > sync-webrsync-keep-snapshots = true|false > Keep snapshots in DISTDIR (do not delete). Defaults to false. > > sync-webrsync-verify-signature = true|false > Require the detached tarball signature to contain a good > OpenPGP signature. This uses the OpenPGP key(ring) specified by the > sync-openpgp-key-path setting. Defaults to false. > > Zac Medico (5): > emerge-webrsync: exit early for signature problem (bug 661838) > webrsync: support sync-openpgp-key-path (bug 661838) > webrsync: support sync-webrsync-keep-snapshots > emerge-delta-webrsync: exit early for signature problem (bug 661838) > webrsync: support emerge-delta-webrsync (bug 661838) > > bin/emerge-webrsync | 22 ++- > lib/portage/sync/modules/webrsync/__init__.py | 6 +- > lib/portage/sync/modules/webrsync/webrsync.py | 84 > --- > man/portage.5 | 12 > misc/emerge-delta-webrsync| 23 +++- 5 files > changed, 135 insertions(+), 12 deletions(-) > this series looks good. I'm also surprised to see nearly all the changes were on the python side. Barely any changes to the bash scripts.
Re: [gentoo-portage-dev] Objections to renaming pym directory to lib?
On Tue, 17 Jul 2018 13:28:05 -0700 Zac Medico wrote: > Are there any objections to renaming the pym directory to lib [1]? > Note that the git log --follow option makes this kind of rename > fairly painless. > > [1] https://github.com/gentoo/portage/pull/343 is fine with me pgpewJwnz8KUC.pgp Description: OpenPGP digital signature
Re: [gentoo-portage-dev] [PATCH] config: cache profile.bashrc stat results (bug 649806)
On Sat, 14 Jul 2018 22:05:55 -0700
Zac Medico wrote:
> Optimize config to stat profile.bashrc files once in the constructor,
> in order to avoid repeated stat calls in the setcpv method.
>
> Bug: https://bugs.gentoo.org/649806
> ---
> pym/portage/package/ebuild/config.py | 13 -
> 1 file changed, 8 insertions(+), 5 deletions(-)
>
> diff --git a/pym/portage/package/ebuild/config.py
> b/pym/portage/package/ebuild/config.py index 88acac5cc..320d9f6c0
> 100644 --- a/pym/portage/package/ebuild/config.py
> +++ b/pym/portage/package/ebuild/config.py
> @@ -274,6 +274,7 @@ class config(object):
> self.mycpv = clone.mycpv
> self._setcpv_args_hash =
> clone._setcpv_args_hash self._soname_provided = clone._soname_provided
> + self._profile_bashrc = clone._profile_bashrc
>
> # immutable attributes (internal policy
> ensures lack of mutation) self._locations_manager =
> clone._locations_manager @@ -725,6 +726,10 @@ class config(object):
> self._license_manager.extract_global_changes( \
>
> self.configdict["conf"].get("ACCEPT_LICENSE",
> ""))
> + # profile.bashrc
> + self._profile_bashrc =
> tuple(os.path.isfile(os.path.join(profile.location, 'profile.bashrc'))
> + for profile in profiles_complex)
> +
> if local_config:
> #package.properties
> propdict =
> grabdict_package(os.path.join( @@ -1596,11 +1601,9 @@ class
> config(object):
> bashrc_files = []
>
> - for profile in
> self._locations_manager.profiles_complex:
> - profile_bashrc =
> os.path.join(profile.location,
> - 'profile.bashrc')
> - if os.path.exists(profile_bashrc):
> - bashrc_files.append(profile_bashrc)
> + for profile, profile_bashrc in
> zip(self._locations_manager.profiles_complex, self._profile_bashrc):
> + if profile_bashrc:
> +
> bashrc_files.append(os.path.join(profile.location, 'profile.bashrc'))
> if profile in self._pbashrcdict: cpdict =
> self._pbashrcdict[profile].get(cp) if cpdict:
looks good
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] portdbapi.cp_list: cache repo associations (bug 650814)
# assume it's iterable
> - mytrees = mytree
> + repos =
> [self.repositories.get_repo_for_location(location)
> + for location in mytree]
> elif self._better_cache is None:
> - mytrees = self.porttrees
> + repos = list(self.repositories)
> else:
> - mytrees = [repo.location for repo in
> self._better_cache[mycp]]
> - for oroot in mytrees:
> + repos = reversed(self._better_cache[mycp])
> + mylist = []
> + for repo in repos:
> + oroot = repo.location
> try:
> file_list =
> os.listdir(os.path.join(oroot, mycp)) except OSError:
> @@ -986,16 +1002,17 @@ class portdbapi(dbapi):
> writemsg(_("\nInvalid
> ebuild version: %s\n") % \ os.path.join(oroot, mycp, x),
> noiselevel=-1) continue
> -
> d[_pkg_str(mysplit[0]+"/"+pf, db=self)] = None
> - if invalid_category and d:
> +
> mylist.append(_pkg_str(mysplit[0]+"/"+pf, db=self, repo=repo.name))
> + if invalid_category and mylist:
> writemsg(_("\n!!! '%s' has a category that
> is not listed in " \ "%setc/portage/categories\n") % \
> (mycp,
> self.settings["PORTAGE_CONFIGROOT"]), noiselevel=-1) mylist = []
> - else:
> - mylist = list(d)
> - # Always sort in ascending order here since it's
> handy
> - # and the result can be easily cached and reused.
> + # Always sort in ascending order here since it's
> handy and
> + # the result can be easily cached and reused. Since
> mylist
> + # is initially in ascending order by repo.priority,
> stable
> + # sort by version produces results in ascending
> order by
> + # (pkg.version, repo.priority).
> self._cpv_sort_ascending(mylist)
> if self.frozen and mytree is None:
> cachelist = mylist[:]
looks fine
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] _unmerge_dirs: revisit parents of removed symlinks (bug 640058)
On Thu, 12 Jul 2018 21:09:46 -0700
Zac Medico wrote:
> When removal of a symlink is triggered by removal of the directory
> that it points to, revisit the parent directories of the symlink.
>
> Bug: https://bugs.gentoo.org/640058
> ---
> pym/portage/dbapi/vartree.py | 23 +--
> 1 file changed, 21 insertions(+), 2 deletions(-)
>
> diff --git a/pym/portage/dbapi/vartree.py
> b/pym/portage/dbapi/vartree.py index 1a86940f1..43e3c4f1a 100644
> --- a/pym/portage/dbapi/vartree.py
> +++ b/pym/portage/dbapi/vartree.py
> @@ -2753,9 +2753,13 @@ class dblink(object):
> real_root = self.settings['ROOT']
>
> dirs = sorted(dirs)
> - dirs.reverse()
> + revisit = {}
>
> - for obj, inode_key in dirs:
> + while True:
> + try:
> + obj, inode_key = dirs.pop()
> + except IndexError:
> + break
> # Treat any directory named "info" as a
> candidate here, # since it might have been in INFOPATH previously even
> # though it may not be there now.
> @@ -2818,6 +2822,7 @@ class dblink(object):
> raise
> if e.errno != errno.ENOENT:
> show_unmerge("---",
> unmerge_desc["!empty"], "dir", obj)
> + revisit[obj] = inode_key
>
> # Since we didn't remove this
> directory, record the directory # itself for use in syncfs calls, if
> we have removed another @@ -2838,6 +2843,7 @@ class dblink(object):
> # no need to protect symlinks that
> point to it. unmerge_syms = protected_symlinks.pop(inode_key, None)
> if unmerge_syms is not None:
> + parents = []
> for relative_path in
> unmerge_syms: obj = os.path.join(real_root,
>
> relative_path.lstrip(os.sep))
> @@ -2849,6 +2855,19 @@ class dblink(object):
> raise
> del e
> show_unmerge("!!!",
> "", "sym", obj)
> + else:
> +
> parents.append(os.path.dirname(obj)) +
> + if parents:
> + # Revisit parents
> recursively (bug 640058).
> + recursive_parents =
> []
> + for parent in
> set(parents):
> + while parent
> in revisit:
> +
> recursive_parents.append(parent)
> +
> parent = os.path.dirname(parent) +
> + for parent in
> sorted(set(recursive_parents)):
> +
> dirs.append((parent, revisit.pop(parent)))
> def isowner(self, filename, destroot=None):
> """
LGTM
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] dbapi: fix repoman implicit IUSE (bug 660982)
On Thu, 12 Jul 2018 02:59:03 -0700 Zac Medico wrote: > Account for repoman modifications of the portdbapi self.settings > reference, and treat all flags as valid for the empty profile > because it does not have any implicit IUSE settings. > > Bug: https://bugs.gentoo.org/660982 > --- > pym/_emerge/Package.py| 5 - > pym/portage/dbapi/__init__.py | 16 > 2 files changed, 20 insertions(+), 1 deletion(-) > > diff --git a/pym/_emerge/Package.py b/pym/_emerge/Package.py > index a7ce00bc9..5f34f3d27 100644 > --- a/pym/_emerge/Package.py > +++ b/pym/_emerge/Package.py > @@ -93,7 +93,10 @@ class Package(Task): > # sync metadata with validated repo (may be > UNKNOWN_REPO) self._metadata['repository'] = self.cpv.repo > > - implicit_match = db._iuse_implicit_cnstr(self.cpv, > self._metadata) > + if self.root_config.settings.local_config: > + implicit_match = > db._iuse_implicit_cnstr(self.cpv, self._metadata) > + else: > + implicit_match = > db._repoman_iuse_implicit_cnstr(self.cpv, self._metadata) usealiases > = self.root_config.settings._use_manager.getUseAliases(self) > self.iuse = self._iuse(self, self._metadata["IUSE"].split(), > implicit_match, usealiases, self.eapi) diff --git > a/pym/portage/dbapi/__init__.py b/pym/portage/dbapi/__init__.py index > d320cc75f..61d301839 100644 --- a/pym/portage/dbapi/__init__.py > +++ b/pym/portage/dbapi/__init__.py > @@ -216,6 +216,22 @@ class dbapi(object): > > yield cpv > > + def _repoman_iuse_implicit_cnstr(self, pkg, metadata): > + """ > + In repoman's version of _iuse_implicit_cnstr, > account for modifications > + of the self.settings reference between calls, and > treat all flags as > + valid for the empty profile because it does not have > any implicit IUSE > + settings. See bug 660982. > + """ > + eapi_attrs = _get_eapi_attrs(metadata["EAPI"]) > + if eapi_attrs.iuse_effective: > + iuse_implicit_match = lambda flag: (True if > not self.settings.profile_path > + else > self.settings._iuse_effective_match(flag)) > + else: > + iuse_implicit_match = lambda flag: (True if > not self.settings.profile_path > + else > self.settings._iuse_implicit_match(flag)) > + return iuse_implicit_match > + > def _iuse_implicit_cnstr(self, pkg, metadata): > """ > Construct a callable that checks if a given USE flag > should looks good thanks. Please add the test case ebuild that was supplied to the repoman gen-b0rk repo "not-broken" category. https://gitweb.gentoo.org/repo/proj/gen-b0rk.git/ -- Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] global_event_loop: use asyncio event loop (bug 654390)
On Mon, 7 May 2018 03:18:32 -0700 Zac Medico <zmed...@gentoo.org> wrote: > + # This attribute it used by _wrap_loop to detect if > the typo, s/it/is this asynio stuff is mostly beyond my familiarity... but all looks ok, and I know you ran the tests on it... should be good to merge unless anyone else objects... -- Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] config.environ: always strip slash from SYSROOT (bug 654600)
On Wed, 2 May 2018 18:03:38 -0700
Zac Medico <zmed...@gentoo.org> wrote:
> Since SYSROOT=/ interacts badly with autotools.eclass (bug 654600),
> and no EAPI expects SYSROOT to have a trailing slash, always strip
> the trailing slash from SYSROOT.
>
> Bug: https://bugs.gentoo.org/654600
> Fixes: a41dacf7926c ("Export SYSROOT and ESYSROOT in ebuild env in
> EAPI 7") ---
> pym/portage/package/ebuild/config.py | 9 +++--
> 1 file changed, 7 insertions(+), 2 deletions(-)
>
> diff --git a/pym/portage/package/ebuild/config.py
> b/pym/portage/package/ebuild/config.py index f9b257b86..96b2ebb71
> 100644 --- a/pym/portage/package/ebuild/config.py
> +++ b/pym/portage/package/ebuild/config.py
> @@ -2813,11 +2813,16 @@ class config(object):
> mydict.pop("ECLASSDIR", None)
>
> if not
> eapi_attrs.path_variables_end_with_trailing_slash:
> - for v in ("D", "ED", "ROOT", "EROOT",
> "SYSROOT", "ESYSROOT",
> - "BROOT"):
> + for v in ("D", "ED", "ROOT", "EROOT",
> "ESYSROOT", "BROOT"): if v in mydict:
> mydict[v] =
> mydict[v].rstrip(os.path.sep)
> + # Since SYSROOT=/ interacts badly with
> autotools.eclass (bug 654600),
> + # and no EAPI expects SYSROOT to have a trailing
> slash, always strip
> + # the trailing slash from SYSROOT.
> + if 'SYSROOT' in mydict:
> + mydict['SYSROOT'] =
> mydict['SYSROOT'].rstrip(os.path.sep) +
> try:
> builddir = mydict["PORTAGE_BUILDDIR"]
> distdir = mydict["DISTDIR"]
looks good
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH 0/5] EbuildFetcher._get_uri_map(): fix event loop recursion (bug 653810)
On Sun, 22 Apr 2018 15:30:09 -0700 Zac Medico <zmed...@gentoo.org> wrote: > Bug: https://bugs.gentoo.org/653810 > > Zac Medico (5): > portdbapi: add async_fetch_map method (bug 653810) > EbuildFetcher: inherit CompositeTask (bug 653810) > EbuildFetcher: add _async_uri_map method (bug 653810) > EbuildFetcher: use _async_uri_map in _start (bug 653810) > EbuildFetcher: add async_already_fetched method (bug 653810) > > pym/_emerge/EbuildBuild.py| 8 +++- > pym/_emerge/EbuildFetcher.py | 105 > -- > pym/portage/dbapi/porttree.py | 75 +++--- 3 > files changed, 146 insertions(+), 42 deletions(-) > I didn't see any errors glaring at me... :) looks like it should be good -- Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] BinpkgFetcher: use async lock (bug 614110)
On Sat, 21 Apr 2018 12:27:28 -0700
Zac Medico <zmed...@gentoo.org> wrote:
> In order to avoid event loop recursion, convert the
> _BinpkgFetcherProcess.lock() method to an async_lock
> method for use by BinpkgFetcher.
>
> Bug: https://bugs.gentoo.org/614110
> ---
> pym/_emerge/BinpkgFetcher.py | 53
> +++- 1 file changed, 33
> insertions(+), 20 deletions(-)
>
> diff --git a/pym/_emerge/BinpkgFetcher.py
> b/pym/_emerge/BinpkgFetcher.py index 5ca7a45cf..2bbc0a26f 100644
> --- a/pym/_emerge/BinpkgFetcher.py
> +++ b/pym/_emerge/BinpkgFetcher.py
> @@ -32,11 +32,24 @@ class BinpkgFetcher(CompositeTask):
> pkg.cpv) + ".partial"
>
> def _start(self):
> - self._start_task(
> -
> _BinpkgFetcherProcess(background=self.background,
> - logfile=self.logfile, pkg=self.pkg,
> pkg_path=self.pkg_path,
> - pretend=self.pretend,
> scheduler=self.scheduler),
> - self._fetcher_exit)
> + fetcher =
> _BinpkgFetcherProcess(background=self.background,
> + logfile=self.logfile, pkg=self.pkg,
> pkg_path=self.pkg_path,
> + pretend=self.pretend,
> scheduler=self.scheduler) +
> + if not self.pretend:
> +
> portage.util.ensure_dirs(os.path.dirname(self.pkg_path))
> + if "distlocks" in
> self.pkg.root_config.settings.features:
> + self._start_task(
> +
> AsyncTaskFuture(future=fetcher.async_lock()),
> +
> functools.partial(self._start_locked, fetcher))
> + return
> +
> + self._start_task(fetcher, self._fetcher_exit)
> +
> + def _start_locked(self, fetcher, lock_task):
> + self._assert_current(lock_task)
> + lock_task.future.result()
> + self._start_task(fetcher, self._fetcher_exit)
>
> def _fetcher_exit(self, fetcher):
> self._assert_current(fetcher)
> @@ -68,13 +81,8 @@ class _BinpkgFetcherProcess(SpawnProcess):
> pretend = self.pretend
> bintree = pkg.root_config.trees["bintree"]
> settings = bintree.settings
> - use_locks = "distlocks" in settings.features
> pkg_path = self.pkg_path
>
> - if not pretend:
> -
> portage.util.ensure_dirs(os.path.dirname(pkg_path))
> - if use_locks:
> - self.lock()
> exists = os.path.exists(pkg_path)
> resume = exists and os.path.basename(pkg_path) in
> bintree.invalids if not (pretend or resume):
> @@ -184,7 +192,7 @@ class _BinpkgFetcherProcess(SpawnProcess):
> except
> OSError: pass
>
> - def lock(self):
> + def async_lock(self):
> """
> This raises an AlreadyLocked exception if lock() is
> called while a lock is already held. In order to avoid this, call
> @@ -194,17 +202,22 @@ class _BinpkgFetcherProcess(SpawnProcess):
> if self._lock_obj is not None:
> raise self.AlreadyLocked((self._lock_obj,))
>
> - async_lock = AsynchronousLock(path=self.pkg_path,
> - scheduler=self.scheduler)
> - async_lock.start()
> + result = self.scheduler.create_future()
>
> - if async_lock.wait() != os.EX_OK:
> - # TODO: Use CompositeTask for better
> handling, like in EbuildPhase.
> - raise AssertionError("AsynchronousLock
> failed with returncode %s" \
> - % (async_lock.returncode,))
> + def acquired_lock(async_lock):
> + if async_lock.wait() == os.EX_OK:
> + self.locked = True
> + result.set_result(None)
> + else:
> + result.set_exception(AssertionError(
> + "AsynchronousLock failed
> with returncode %s"
> + % (async_lock.returncode,)))
>
> - self._lock_obj = async_lock
> - self.locked = True
> + self._lock_obj = AsynchronousLock(path=self.pkg_path,
> + scheduler=self.scheduler)
> + self._lock_obj.addExitListener(acquired_lock)
> + self._lock_obj.start()
> + return result
>
> class AlreadyLocked(portage.exception.PortageException):
> pass
Looks fine to me :)
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH v3 1/3] repoman: Add commit message verification
On Mon, 19 Feb 2018 16:00:55 +0100 Michał Górny <mgo...@gentoo.org> wrote: > Add a check for common mistakes in commit messages. For now, it is > pretty rough and works only for -m/-F. It will be extended to work > in the interactive mode in the future. > --- This version series looks good, thank you :) -- Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] emerge --search: use slash to auto-detect category (bug 647940)
On Sun, 18 Feb 2018 13:27:31 -0800
Zac Medico <zmed...@gentoo.org> wrote:
> Since search strings containing a slash do no work unless category
> match mode is enabled, use slash to auto-detect category match mode,
> so that users do not have to prefix the search string with the special
> @ symbol.
>
> Bug: https://bugs.gentoo.org/647940
> ---
> pym/_emerge/search.py | 4
> 1 file changed, 4 insertions(+)
>
> diff --git a/pym/_emerge/search.py b/pym/_emerge/search.py
> index 36897fb15..eb52b2ca3 100644
> --- a/pym/_emerge/search.py
> +++ b/pym/_emerge/search.py
> @@ -256,6 +256,10 @@ class search(object):
> if self.searchkey.startswith('@'):
> match_category = 1
> self.searchkey = self.searchkey[1:]
> + # Auto-detect category match mode (@ symbol can be
> deprecated
> + # after this is available in a stable version of
> portage).
> + if '/' in self.searchkey:
> + match_category = 1
> fuzzy = False
> if regexsearch:
> self.searchre=re.compile(self.searchkey,re.I)
looks fine
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH v2 1/3] repoman: Add commit message verification
On Sat, 17 Feb 2018 13:56:46 +0100 Michał Górny <mgo...@gentoo.org> wrote: > + > + footer_re = re.compile(r'^[\w-]+:') > + > + @classmethod > + def verify_commit_message(cls, msg): ... > + if all(cls.footer_re.match(l) for l in lines if l.strip()): Why declare the footer_re in the class space? It is only used in this new function. Then the function could be @staticmethod instead. I don't see the advantage of it this way. If it is for re-use in other possible check functions, then perhaps it would be best to split this out to it's own class/file that can be added to easily. And just run it from the Actions class as M.J. Everitt suggested. -- Brian Dolbec
Re: [gentoo-portage-dev] [PATCH v2 1/3] repoman: Add commit message verification
> > +
> > bug_closes_without_url = True
> > + else:
> > + for l in lines:
> > + # we recommend wrapping at
> > 72 but accept up to 80;
> > + # do not complain if we
> > have single word (usually
> > + # it means very long URL)
> > + if len(l.strip()) > 80 and
> > len(l.split()) > 1:
> > + body_too_long =
> > True +
> > + if multiple_footers:
> > + problems.append('multiple footer-style
> > blocks found, please combine them into one')
> > + if gentoo_bug_used:
> > + problems.append('please replace Gentoo-Bug
> > with GLEP 66-compliant Bug/Closes')
> > + if bug_closes_without_url:
> > + problems.append('Bug/Closes footers
> > require full URL')
> > + if body_too_long:
> > + problems.append('body lines should be
> > wrapped at 72 (max 80) characters') +
> > + if problems:
> > + return (False, '\n'.join('- %s' % x for x
> > in problems))
> > + return (True, None)
> > diff --git a/repoman/pym/repoman/tests/commit/__init__.py
> > b/repoman/pym/repoman/tests/commit/__init__.py new file mode 100644
> > index 0..d74fd94a7
> > --- /dev/null
> > +++ b/repoman/pym/repoman/tests/commit/__init__.py
> > @@ -0,0 +1,2 @@
> > +# Copyright 2011-2018 Gentoo Foundation
> > +# Distributed under the terms of the GNU General Public License v2
> > diff --git a/repoman/pym/repoman/tests/commit/__test__.py
> > b/repoman/pym/repoman/tests/commit/__test__.py new file mode 100644
> > index 0..8b1378917
> > --- /dev/null
> > +++ b/repoman/pym/repoman/tests/commit/__test__.py
> > @@ -0,0 +1 @@
> > +
> > diff --git a/repoman/pym/repoman/tests/simple/test_simple.py
> > b/repoman/pym/repoman/tests/simple/test_simple.py index
> > a24e0d5a3..3d7a70ad0 100644 ---
> > a/repoman/pym/repoman/tests/simple/test_simple.py +++
> > b/repoman/pym/repoman/tests/simple/test_simple.py @@ -1,4 +1,4 @@
> > -# Copyright 2011-2015 Gentoo Foundation
> > +# Copyright 2011-2018 Gentoo Foundation
> > # Distributed under the terms of the GNU General Public License v2
> >
> > import subprocess
> > @@ -194,13 +194,13 @@ class SimpleRepomanTestCase(TestCase):
> > ("", repoman_cmd + ("full", "-d")),
> > ("", cp_cmd + (test_ebuild,
> > test_ebuild[:-8] + "2.ebuild")), ("", git_cmd + ("add",
> > test_ebuild[:-8] + "2.ebuild")),
> > - ("", repoman_cmd + ("commit", "-m", "bump
> > to version 2")),
> > + ("", repoman_cmd + ("commit", "-m",
> > "cat/pkg: bump to version 2")), ("", cp_cmd + (test_ebuild,
> > test_ebuild[:-8] + "3.ebuild")), ("", git_cmd + ("add",
> > test_ebuild[:-8] + "3.ebuild")),
> > - ("dev-libs", repoman_cmd + ("commit",
> > "-m", "bump to version 3")),
> > + ("dev-libs", repoman_cmd + ("commit",
> > "-m", "cat/pkg: bump to version 3")), ("", cp_cmd + (test_ebuild,
> > test_ebuild[:-8] + "4.ebuild")), ("", git_cmd + ("add",
> > test_ebuild[:-8] + "4.ebuild")),
> > - ("dev-libs/A", repoman_cmd + ("commit",
> > "-m", "bump to version 4")),
> > + ("dev-libs/A", repoman_cmd + ("commit",
> > "-m", "cat/pkg: bump to version 4")), )
> >
> > env = {
> Might I suggest breaking out checks into a separate module? I think
> that hard-coding it all is likely to become a pain as time goes on,
> more checks get added, etc ...
>
So, you would like the commit message checks to be a plug-in system
like the ebuild checks are now.
Yes that is possible, that would also allow overlays and sub-distros
to customize it to their liking. But you do realize it would be a
module system with only the one plug-in. It would add a slight amount
of overhead and subsequently a tiny bit more time. Although it is not
much as I remember when I first developed the plug-in system for emaint.
I will consider that for the stage3 work in progress, but for the
master branch of repoman, the hard-coded version above would be fine.
--
Brian Dolbec
pgprVkLpLeUb3.pgp
Description: OpenPGP digital signature
Re: [gentoo-portage-dev] [PATCH] emerge: disable --changed-deps-report by default (bug 645780)
On Fri, 2 Feb 2018 18:52:35 -0800
Zac Medico <zmed...@gentoo.org> wrote:
> This option is too noisy to enable by default, since it reports
> hundreds of packages for most users.
>
> Suggested-by: Michał Górny <mgo...@gentoo.org>
> Bug: https://bugs.gentoo.org/645780
> ---
> man/emerge.1 | 7 +--
> pym/_emerge/create_depgraph_params.py | 6 ++
> 2 files changed, 3 insertions(+), 10 deletions(-)
>
> diff --git a/man/emerge.1 b/man/emerge.1
> index 189e6f879..810336fa4 100644
> --- a/man/emerge.1
> +++ b/man/emerge.1
> @@ -469,12 +469,7 @@ respect to changed build\-time dependencies is
> controlled by the Tells emerge to report ebuilds for which the ebuild
> dependencies have changed since the installed instance was built.
> Behavior with respect to changed build\-time dependencies is
> controlled by the -\fB\-\-with\-bdeps\fR option. If the
> \fB\-\-update\fR and \fB\-\-deep\fR -options are enabled then this
> option is enabled automatically for a -dependency calculation if the
> cost of report generation is relatively -insignificant (any
> calculation exclusively involving binary packages is -exempt). The
> \fIEMERGE_DEFAULT_OPTS\fR variable may be used to disable -this by
> default. +\fB\-\-with\-bdeps\fR option.
> .TP
> .BR \-\-changed\-use ", " \-U
> Tells emerge to include installed packages where USE flags have
> diff --git a/pym/_emerge/create_depgraph_params.py
> b/pym/_emerge/create_depgraph_params.py index ecd65335c..6964bc65b
> 100644 --- a/pym/_emerge/create_depgraph_params.py
> +++ b/pym/_emerge/create_depgraph_params.py
> @@ -127,10 +127,8 @@ def create_depgraph_params(myopts, myaction):
> if changed_deps is not None:
> myparams['changed_deps'] = changed_deps
>
> - changed_deps_report = myopts.get('--changed-deps-report')
> - if (changed_deps_report != 'n' and
> - not (myaction == 'remove' or '--usepkgonly' in
> myopts) and
> - deep is True and '--update' in myopts):
> + changed_deps_report = myopts.get('--changed-deps-report',
> 'n') == 'y'
> + if changed_deps_report:
> myparams['changed_deps_report'] = True
>
> if myopts.get("--selective") == "n":
looks good, if not already merged.
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] repoman: Add commit message verification
On Sun, 04 Feb 2018 18:58:21 +0100 Michał Górny <mgo...@gentoo.org> wrote: > W dniu nie, 04.02.2018 o godzinie 09∶51 -0800, użytkownik Brian Dolbec > napisał: > > > > I know there are not a lot of repoman unit tests, but, can you > > please create some to test this? That way it doesn't get any worse. > > > > In the PR, you mentioned adding in an editor call to re-edit the > > message. This would be a good thing and I know is not that > > difficult to accomplish. Is this coming in another patch? > > > > I'm still wondering how to implement it best. In particular, how to > handle the non-EDITOR (stdin) branch? > hmm, yeah, that was not an option for the work tool I wrote and added the re-edit to. It had 2 files it needed to edit or re-edit. In the case of passing in a pre-saved message, that should be left for the user to fix. Perhaps in the case of a stdin -m "foo" message we could just let the user rely on bash their history to bring up that previous command, to edit before hitting enter again. -- Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] repoman: Add commit message verification
On Fri, 2 Feb 2018 23:53:05 +0100
Michał Górny <mgo...@gentoo.org> wrote:
> Add a check for common mistakes in commit messages. For now, it is
> pretty rough and exits immediately but it should be integrated with
> the editor in the future.
> ---
> repoman/pym/repoman/actions.py | 68
> ++ 1 file changed, 68
> insertions(+)
>
> diff --git a/repoman/pym/repoman/actions.py
> b/repoman/pym/repoman/actions.py index b76a6e466..97a71d0f5 100644
> --- a/repoman/pym/repoman/actions.py
> +++ b/repoman/pym/repoman/actions.py
> @@ -124,6 +124,15 @@ class Actions(object):
>
> commitmessage = commitmessage.rstrip()
>
> + res, expl = self.verify_commit_message(commitmessage)
> + if not res:
> + print(bad("RepoMan does not like your commit
> message:"))
> + print(expl)
> + if self.options.force:
> + print('(but proceeding due to
> --force)')
> + else:
> + sys.exit(1)
> +
> # Update copyright for new and changed files
> year = time.strftime('%Y', time.gmtime())
> for fn in chain(mynew, mychanged):
> @@ -585,3 +594,62 @@ class Actions(object):
> print("* no commit message? aborting
> commit.") sys.exit(1)
> return commitmessage
> +
> + footer_re = re.compile(r'^[\w-]+:')
> +
> + def verify_commit_message(self, msg):
> + """
> + Check whether the message roughly complies with
> GLEP66. Returns
> + (True, None) if it does, (False, ) if
> it does not.
> + """
> +
> + problems = []
> + paras = msg.strip().split('\n\n')
> + summary = paras.pop(0)
> +
> + if '\n' in summary.strip():
> + problems.append('commit message must start
> with a *single* line of summary, followed by empty line')
> + if ':' not in summary:
> + problems.append('summary line must start
> with a logical unit name, e.g. "cat/pkg:"')
> + # accept 69 overall or unit+50, in case of very long
> package names
> + if len(summary.strip()) > 69 and
> len(summary.split(':', 1)[-1]) > 50:
> + problems.append('summary line is too long
> (max 69 characters)') +
> + multiple_footers = False
> + gentoo_bug_used = False
> + bug_closes_without_url = False
> + body_too_long = False
> +
> + found_footer = False
> + for p in paras:
> + lines = p.splitlines()
> + # if all lines look like footer material, we
> assume it's footer
> + # else, we assume it's body text
> + if all(self.footer_re.match(l) for l in
> lines if l.strip()):
> + # multiple footer-like blocks?
> + if found_footer:
> + multiple_footers = True
> + found_footer = True
> + for l in lines:
> + if
> l.startswith('Gentoo-Bug'):
> + gentoo_bug_used =
> True
> + elif l.startswith('Bug:') or
> l.startswith('Closes:'):
> + if 'http://' not in
> l and 'https://' not in l:
> +
> bug_closes_without_url = True
> + else:
> + for l in lines:
> + if len(l.strip()) > 72:
> + body_too_long = True
> +
> + if multiple_footers:
> + problems.append('multiple footer-style
> blocks found, please combine them into one')
> + if gentoo_bug_used:
> + problems.append('please replace Gentoo-Bug
> with GLEP 66-compliant Bug/Closes')
> + if bug_closes_without_url:
> + problems.append('Bug/Closes footers require
> full URL')
> + if body_too_long:
> + problems.append('body lines should be
> wrapped at 72 characters') +
> + if problems:
> + return (False, '\n'.join('- %s' % x for x in
> problems))
> + return (True, None)
I know there are not a lot of repoman unit tests, but, can you please
create some to test this? That way it doesn't get any worse.
In the PR, you mentioned adding in an editor call to re-edit the
message. This would be a good thing and I know is not that difficult
to accomplish. Is this coming in another patch?
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH v2] [checksum] Disable pygcrypt backend due to breakage
On Tue, 5 Dec 2017 18:08:47 +0100
Michał Górny <mgo...@gentoo.org> wrote:
> Closes: https://bugs.gentoo.org/615620
> ---
> pym/portage/checksum.py | 6 +-
> 1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/pym/portage/checksum.py b/pym/portage/checksum.py
> index 5424ce56b..9e7bffea9 100644
> --- a/pym/portage/checksum.py
> +++ b/pym/portage/checksum.py
> @@ -150,7 +150,11 @@ if "SHA3_256" not in hashfunc_map or "SHA3_512"
> not in hashfunc_map: # (GnuPG).
> gcrypt_algos = frozenset(('RMD160', 'WHIRLPOOL', 'SHA3_256',
> 'SHA3_512', 'STREEBOG256', 'STREEBOG512'))
> -if gcrypt_algos.difference(hashfunc_map):
> +# Note: currently disabled due to resource exhaustion bugs in
> pygcrypt. +# Please do not reenable until upstream has a fix.
> +# https://bugs.gentoo.org/615620
> +if False:
> +#if gcrypt_algos.difference(hashfunc_map):
> try:
> import binascii
> import pygcrypt.hashcontext
looks good thanks, merge please
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] [checksum] Disable pygcrypt backend due to breakage
On Tue, 5 Dec 2017 08:42:43 -0800
Brian Dolbec <dol...@gentoo.org> wrote:
> On Tue, 5 Dec 2017 17:34:23 +0100
> Michał Górny <mgo...@gentoo.org> wrote:
>
> > Closes: https://bugs.gentoo.org/615620
> > ---
> > pym/portage/checksum.py | 5 -
> > 1 file changed, 4 insertions(+), 1 deletion(-)
> >
> > diff --git a/pym/portage/checksum.py b/pym/portage/checksum.py
> > index 5424ce56b..0841ab231 100644
> > --- a/pym/portage/checksum.py
> > +++ b/pym/portage/checksum.py
> > @@ -150,7 +150,10 @@ if "SHA3_256" not in hashfunc_map or "SHA3_512"
> > not in hashfunc_map: # (GnuPG).
> > gcrypt_algos = frozenset(('RMD160', 'WHIRLPOOL', 'SHA3_256',
> > 'SHA3_512', 'STREEBOG256', 'STREEBOG512'))
> > -if gcrypt_algos.difference(hashfunc_map):
> > +# Note: currently disabled due to resource exhaustion bugs in
> > pygcrypt. +# Please do not reenable until upstream has a fix.
> > +# https://bugs.gentoo.org/615620
> > +if False and gcrypt_algos.difference(hashfunc_map):
> > try:
> > import binascii
> > import pygcrypt.hashcontext
>
>
>
> It would be better to just comment out the original if, then add a new
> line to replace it with just if False. it would be clearer what the
> original code should be. Of course with the reason comments...
>
sorry, brain is still struggling with this damn headache...
It would be clearer later when re-enabling if it takes a long time
for pygrcrypt to get fixed. I know it would be for me, my memory
isn't as good as it used to be. Also if it ends up someone else
looks to modify it later.
+# Note: currently disabled due to resource exhaustion bugs in pygcrypt.
+# Please do not reenable until upstream has a fix.
+# https://bugs.gentoo.org/615620
-if gcrypt_algos.difference(hashfunc_map):
+#if gcrypt_algos.difference(hashfunc_map):
+if False
try:
import binascii
import pygcrypt.hashcontext
I approve the above form to merge... Thanks
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] depgraph: eval disjunctive build deps earlier (bug 639346)
639346, where a redundant jre
> implementation
> + # was pulled in because DEPEND was evaluated
> after
> + # RDEPEND.
> + ResolverPlaygroundTestCase(
> + ['dev-db/hsqldb'],
> + success=True,
> + mergelist=[
> + 'dev-java/icedtea-3.6',
> + 'virtual/jdk-1.8',
> + 'virtual/jre-1.8',
> + 'dev-db/hsqldb-1.8',
> + ],
> + ),
> +
> + # The jdk is not needed with --usepkg, so
> the jre should
> + # be preferred in this case.
> + ResolverPlaygroundTestCase(
> + ['dev-db/hsqldb'],
> + options = {
> + '--usepkg': True
> + },
> + success=True,
> + mergelist=[
> + 'dev-java/oracle-jre-bin-1.8',
> + 'virtual/jre-1.8',
> + '[binary]dev-db/hsqldb-1.8',
> + ],
> + ),
> + )
> +
> + playground = ResolverPlayground(debug=False,
> + binpkgs=binpkgs, ebuilds=ebuilds)
> +
> + try:
> + for test_case in test_cases:
> + playground.run_TestCase(test_case)
> +
> self.assertEqual(test_case.test_success, True,
> + test_case.fail_msg)
> + finally:
> + playground.debug = False
> + playground.cleanup()
> diff --git a/pym/portage/tests/resolver/test_onlydeps_minimal.py
> b/pym/portage/tests/resolver/test_onlydeps_minimal.py index
> 13c79ed55..efda02c59 100644 ---
> a/pym/portage/tests/resolver/test_onlydeps_minimal.py +++
> b/pym/portage/tests/resolver/test_onlydeps_minimal.py @@ -25,9 +25,10
> @@ class OnlydepsMinimalTestCase(TestCase): success = True,
> options = { "--onlydeps": True,
> "--onlydeps-with-rdeps":
> "y" },
> - mergelist = ["dev-libs/B-1",
> + ambiguous_merge_order = True,
> + mergelist = [("dev-libs/B-1",
>"dev-libs/C-1",
> - "dev-libs/D-1"]),
> + "dev-libs/D-1")]),
> ResolverPlaygroundTestCase(
> ["dev-libs/A"],
> all_permutations = True,
looks good
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] dep_zapdeps: install new package, avoid downgrade (bug 635540)
es = (
> + # Test bug 635540, where we need to install
> libtirpc
> + # rather than downgrade glibc.
> + ResolverPlaygroundTestCase(
> + ['net-misc/foo'],
> + success=True,
> + mergelist=[
> + 'net-libs/libtirpc-1',
> + 'net-misc/foo-1',
> + ],
> + ),
> + )
> +
> + playground = ResolverPlayground(debug=False,
> + ebuilds=ebuilds, installed=installed,
> world=world) +
> + try:
> + for test_case in test_cases:
> + playground.run_TestCase(test_case)
> +
> self.assertEqual(test_case.test_success, True,
> + test_case.fail_msg)
> + finally:
> + playground.debug = False
> + playground.cleanup()
> +
> + # In some cases it's necessary to downgrade due to
> + # the installed package being masked (glibc is a
> + # not an ideal example because it's usually not
> + # practical to downgrade it).
> + user_config = {
> + "package.mask" : (
> + ">=sys-libs/glibc-2.26",
> + ),
> + }
> +
> + test_cases = (
> + ResolverPlaygroundTestCase(
> + ['net-misc/foo'],
> + success=True,
> + mergelist=[
> + 'sys-libs/glibc-2.24',
> + 'net-misc/foo-1',
> + ],
> + ),
> + )
> +
> + playground = ResolverPlayground(debug=False,
> + ebuilds=ebuilds, installed=installed,
> world=world,
> + user_config=user_config)
> +
> + try:
> + for test_case in test_cases:
> + playground.run_TestCase(test_case)
> +
> self.assertEqual(test_case.test_success, True,
> + test_case.fail_msg)
> + finally:
> + playground.debug = False
> + playground.cleanup()
looks good
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH v5] file_copy: use sendfile return value to measure bytes copied (bug 635126)
blocks when appropriate).
> Effects
> + * on file offsets are undefined.
> + *
> * Return: On success, the number of bytes to copy before the next
> hole,
> * and -1 on failure (errno is set appropriately). Returns 0 when
> fd_in
> * reaches EOF.
> @@ -250,7 +289,7 @@ _reflink_linux_file_copy(PyObject *self, PyObject
> *args)
> * syscall is not available (less than Linux
> 4.5). */
> error = 0;
> -copyfunc = sendfile;
> +copyfunc = sf_wrapper;
> copyfunc_ret = copyfunc(fd_out,
> fd_in,
> _out,
> @@ -284,27 +323,18 @@ _reflink_linux_file_copy(PyObject *self,
> PyObject *args) } else {
> stat_in_acquired = 1;
>
> -/* For the sendfile call, the fd_in file offset must
> be
> - * exactly equal to offset_out. Use lseek to ensure
> - * correct state, in case an EINTR retry caused it to
> - * get out of sync somewhow.
> - */
> -if (lseek(fd_in, offset_out, SEEK_SET) < 0) {
> -error = errno;
> -} else {
> -while (offset_out < stat_in.st_size) {
> -copyfunc_ret = sendfile(fd_out,
> -fd_in,
> -_out,
> -stat_in.st_size -
> offset_out);
> +while (offset_out < stat_in.st_size) {
> +copyfunc_ret = sf_wrapper(fd_out,
> + fd_in,
> + _out,
> + stat_in.st_size -
> offset_out);
> -if (copyfunc_ret < 0) {
> -error = errno;
> -if (errno == EINVAL && !offset_out) {
> -sendfile_works = 0;
> -}
> -break;
> +if (copyfunc_ret < 0) {
> +error = errno;
> +if (errno == EINVAL && !offset_out) {
> +sendfile_works = 0;
> }
> +break;
> }
> }
> }
Looks good, has been checked by chutzpah
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] install-qa-check.d/10ignored-flags: fix LDFLAGS check (bug 455232)
On Sun, 15 Oct 2017 20:59:55 -0700
Zac Medico <zmed...@gentoo.org> wrote:
> From: Felix Janda <felix.ja...@posteo.de>
>
> Since binutils-2.23.51.0.9+ defaults to hash-style=gnu, use
> a __gentoo_check_ldflags__ symbol instead. The check is now
> enabled by adding "-Wl,--defsym=__gentoo_check_ldflags__=0"
> to LDFLAGS. The symbol will be automatically removed when
> the binary is stripped.
>
> Bug: https://bugs.gentoo.org/455232
> Closes: https://github.com/gentoo/portage/pull/220
> ---
> bin/install-qa-check.d/10ignored-flags | 5 +++--
> 1 file changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/bin/install-qa-check.d/10ignored-flags
> b/bin/install-qa-check.d/10ignored-flags index 7aa9eb695..28aec6787
> 100644 --- a/bin/install-qa-check.d/10ignored-flags
> +++ b/bin/install-qa-check.d/10ignored-flags
> @@ -64,9 +64,10 @@ ignored_flag_check() {
> fi
>
> # Check for files built without respecting LDFLAGS
> - if [[ "${LDFLAGS}" == *,--hash-style=gnu* ]] && \
> + if [[ "${LDFLAGS}" == *,--defsym=__gentoo_check_ldflags__*
> ]] && \ ! has binchecks ${RESTRICT} ; then
> - f=$(scanelf -qyRF '#k%p' -k .hash "${ED}")
> + f=$(LC_ALL=C comm -3 <(scanelf -qyRF '#k%p'
> -k .dynsym "${ED}" | LC_ALL=C sort) \
> + <(scanelf -qyRF '#s%p' -s
> __gentoo_check_ldflags__ "${ED}" | LC_ALL=C sort)) if [[ -n ${f} ]] ;
> then echo "${f}" > "${T}"/scanelf-ignored-LDFLAGS.log
> if [ "${QA_STRICT_FLAGS_IGNORED-unset}" =
> unset ] ; then
looks good
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] prepstrip: use debugedit from rpm if necessary (bug 634378)
On Mon, 16 Oct 2017 01:17:23 -0700
Zac Medico <zmed...@gentoo.org> wrote:
> If debugedit is not found in PATH, then use debugedit
> installed by app-arch/rpm (if available).
>
> Suggested-by: Francesco Riosa <viv...@gmail.com>
> Bug: https://bugs.gentoo.org/634378
> ---
> bin/ebuild-helpers/prepstrip | 20
> 1 file changed, 16 insertions(+), 4 deletions(-)
>
> diff --git a/bin/ebuild-helpers/prepstrip
> b/bin/ebuild-helpers/prepstrip index 43caa29c7..ea5c0dc09 100755
> --- a/bin/ebuild-helpers/prepstrip
> +++ b/bin/ebuild-helpers/prepstrip
> @@ -84,7 +84,19 @@ esac
>
> prepstrip_sources_dir=${EPREFIX}/usr/src/debug/${CATEGORY}/${PF}
>
> -type -P debugedit >/dev/null && debugedit_found=true ||
> debugedit_found=false +debugedit=$(type -P debugedit)
> +if [[ -z ${debugedit} ]]; then
> + debugedit_paths=(
> + "${EPREFIX}/usr/libexec/rpm/debugedit"
> + )
> + for x in "${debugedit_paths[@]}"; do
> + if [[ -x ${x} ]]; then
> + debugedit=${x}
> + break
> + fi
> + done
> +fi
> +[[ ${debugedit} ]] && debugedit_found=true || debugedit_found=false
> debugedit_warned=false
>
> __multijob_init
> @@ -101,8 +113,8 @@ save_elf_sources() {
> if ! ${debugedit_found} ; then
> if ! ${debugedit_warned} ; then
> debugedit_warned=true
> - ewarn "FEATURES=installsources is enabled
> but the debugedit binary could not"
> - ewarn "be found. This feature will not work
> unless debugedit is installed!"
> + ewarn "FEATURES=installsources is enabled
> but the debugedit binary could not be"
> + ewarn "found. This feature will not work
> unless debugedit or rpm is installed!" fi
> return 0
> fi
> @@ -112,7 +124,7 @@ save_elf_sources() {
> # since we're editing the ELF here, we should recompute the
> build-id # (the -i flag below). save that output so we don't need to
> recompute # it later on in the save_elf_debug step.
> - buildid=$(debugedit -i \
> + buildid=$("${debugedit}" -i \
> -b "${WORKDIR}" \
> -d "${prepstrip_sources_dir}" \
> -l
> "${tmpdir}/sources/${x##*/}.${BASHPID:-$(__bashpid)}" \
Looks fine to me
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] want_restart_for_use_change: backtrack only if needed (bug 632598)
'--autounmask-backtrack':
> 'y',
> + '--backtrack': 2,
> + },
> + success=False,
> + ambiguous_merge_order=True,
> + mergelist=[
> + ('dev-libs/C-1',
> 'dev-libs/A-2'),
> + 'dev-libs/D-1',
> + ],
> + use_changes={'dev-libs/C-1': {'y':
> True, 'x': True}},
> + ),
> + )
> +
> + playground = ResolverPlayground(
> + ebuilds=ebuilds, installed=installed,
> world=world) +
> + try:
> + for test_case in test_cases:
> + playground.run_TestCase(test_case)
> +
> self.assertEqual(test_case.test_success, True,
> + test_case.fail_msg)
> + finally:
> + playground.cleanup()
Looks fine
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] _solve_..slot_conflicts: handle forced reinstall (bug 632202)
On Fri, 29 Sep 2017 01:15:59 -0700 Zac Medico <zmed...@gentoo.org> wrote: > Fix the slot conflict solver to use the _want_installed_pkg > method to identify installed packages that are intended to > be reinstalled for some reason (such as @preserved-rebuild). > > X-Gentoo-bug: 632202 > X-Gentoo-bug-url: https://bugs.gentoo.org/632202 > --- > pym/_emerge/depgraph.py | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py > index 3b81c5c76..b0149c4dc 100644 > --- a/pym/_emerge/depgraph.py > +++ b/pym/_emerge/depgraph.py > @@ -1337,7 +1337,8 @@ class depgraph(object): > > self._dynamic_config._parent_atoms.get(pkg, > [])) > for parent, atom in all_parent_atoms: > - is_arg_parent = isinstance(parent, > AtomArg) > + is_arg_parent = (inst_pkg is not > None and > + not > self._want_installed_pkg(inst_pkg)) is_non_conflict_parent = parent > not in conflict_pkgs and \ parent not in indirect_conflict_pkgs > looks good -- Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] _solve_..slot_conflicts: make "forced" set recursive (bug 632210)
On Fri, 29 Sep 2017 00:26:01 -0700
Zac Medico <zmed...@gentoo.org> wrote:
> When the slot conflict solver decides that it is "forced"
> to choose a particular package, recursively force the
> dependencies as well. Prior to this fix, substitution of
> @world in the arguments for SlotConflictMaskUpdateTestCase
> caused the test to fail because the solver removed
> boost-build-1.53.0 from the graph event though it had
> added the parent boost-1.53.0 package to the "forced"
> set.
>
> X-Gentoo-bug: 632210
> X-Gentoo-bug-url: https://bugs.gentoo.org/632210
> ---
> pym/_emerge/depgraph.py | 13
> + pym/portage/tests/resolver/test_slot_conflict_update.py
> | 2 +- 2 files changed, 14 insertions(+), 1 deletion(-)
>
> diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py
> index 785c036b8..3b81c5c76 100644
> --- a/pym/_emerge/depgraph.py
> +++ b/pym/_emerge/depgraph.py
> @@ -1457,6 +1457,19 @@ class depgraph(object):
>
> # Remove 'non_conflict_node' and or_tuples from
> 'forced'. forced = set(pkg for pkg in forced if isinstance(pkg,
> Package)) +
> + # Add dependendencies of forced packages.
> + stack = list(forced)
> + traversed = set()
> + while stack:
> + pkg = stack.pop()
> + traversed.add(pkg)
> + for child in conflict_graph.child_nodes(pkg):
> + if (isinstance(child, Package) and
> + child not in traversed):
> + forced.add(child)
> + stack.append(child)
> +
> non_forced = set(pkg for pkg in conflict_pkgs if pkg
> not in forced)
> if debug:
> diff --git a/pym/portage/tests/resolver/test_slot_conflict_update.py
> b/pym/portage/tests/resolver/test_slot_conflict_update.py index
> 331e5788b..f251d01f1 100644 ---
> a/pym/portage/tests/resolver/test_slot_conflict_update.py +++
> b/pym/portage/tests/resolver/test_slot_conflict_update.py @@ -80,7
> +80,7 @@ class SlotConflictUpdateTestCase(TestCase): # this behavior
> makes SlotConflictMaskUpdateTestCase # fail.
> ResolverPlaygroundTestCase(
> - world,
> + ['@world'],
> all_permutations = True,
> options = {"--update": True,
> "--deep": True}, success = True,
looks good
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] depgraph: make _minimize_children deterministic (bug 631894)
On Sun, 24 Sep 2017 11:57:19 -0700 Zac Medico <zmed...@gentoo.org> wrote: > In order for the eliminate_pkg loop to produce deterministic results, > the order of the pkgs list must not be random. Prefer to eliminate > installed packages first, in case rebuilds are needed, and also sort > in ascending order so that older versions are eliminated first. > > X-Gentoo-bug: 631894 > X-Gentoo-bug-url: https://bugs.gentoo.org/631894 > --- > pym/_emerge/depgraph.py | 9 + > 1 file changed, 9 insertions(+) > > diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py > index b4fc5f297..785c036b8 100644 > --- a/pym/_emerge/depgraph.py > +++ b/pym/_emerge/depgraph.py > @@ -3584,6 +3584,15 @@ class depgraph(object): > if > atom_set.findAtomForPackage(pkg2, > modified_use=self._pkg_use_enabled(pkg2)): atom_pkg_graph.add(pkg2, > atom) > + # In order for the following eliminate_pkg > loop to produce > + # deterministic results, the order of the > pkgs list must > + # not be random (bug 631894). Prefer to > eliminate installed > + # packages first, in case rebuilds are > needed, and also sort > + # in ascending order so that older versions > are eliminated > + # first. > + pkgs = (sorted(pkg for pkg in pkgs if > pkg.installed) + > + sorted(pkg for pkg in pkgs if not > pkg.installed)) + > for pkg in pkgs: > eliminate_pkg = True > for atom in > atom_pkg_graph.parent_nodes(pkg): Looks good -- Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] gnome2-utils postinst: Restrict file types to fix false positives
On Tue, 29 Aug 2017 10:20:40 +0200
Michał Górny <mgo...@gentoo.org> wrote:
> Restrict the file types checked by the gtk-icon-cache-update check
> to match the one used inside the tool. This ensures that we do not
> report any false positives for files that are intentionally skipped.
>
> Bug: https://bugs.gentoo.org/629148
> ---
> bin/postinst-qa-check.d/50gnome2-utils | 7 ++-
> 1 file changed, 6 insertions(+), 1 deletion(-)
>
> diff --git a/bin/postinst-qa-check.d/50gnome2-utils
> b/bin/postinst-qa-check.d/50gnome2-utils index d0cbb4037..84025ab83
> 100644 --- a/bin/postinst-qa-check.d/50gnome2-utils
> +++ b/bin/postinst-qa-check.d/50gnome2-utils
> @@ -6,7 +6,12 @@ gnome2_icon_cache_check() {
> # gnome2_icon_cache_update updates only themes with
> an index [[ -f ${d}/index.theme ]] || continue
>
> - find_args=()
> + find_args=(
> + # gtk-update-icon-cache supports only
> specific file
> + # suffixes; match that to avoid false
> positives
> + '(' -name '*.png' -o -name '*.svg'
> + -o -name '*.xpm' -o -name '*.icon'
> ')'
> + )
> # if the cache does not exist at all, we complain
> for any file # otherwise, we look for files newer than the cache
> [[ -f ${d}/icon-theme.cache ]] &&
Works for me, since Gilles approved the change :)
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] Rewrite doins in python (bug 624526)
On Mon, 14 Aug 2017 13:31:53 -0700 Zac Medico <zmed...@gentoo.org> wrote: > On 08/14/2017 12:49 PM, M. J. Everitt wrote: > > On 14/08/17 08:39, Zac Medico wrote: > >> From: Hidehiko Abe <hideh...@chromium.org> > >> > >> doins is written in bash. However, specifically in case that > >> too many files are installed, it is very slow. > >> This CL rewrites the script in python for performance. > >> > >> BUG=chromium:712659 > >> TEST=time (./setup_board --forace && \ > >> ./build_package --withdev && \ > >> ./build_image --noenable_rootfs_verification test) > >> ===Before=== > >> real21m35.445s > >> user93m40.588s > >> sys 21m31.224s > >> > >> ===After=== > >> real17m30.106s > >> user94m1.812s > >> sys 20m13.468s > >> > > I know I'm gonna get chewed out on this one, but here goes > > anyway ... > > > > Surely for a package like chromium, who's build time is already in > > the 'hours' range anyway, surely a couple of minutes gain for the > > install phase is neither here nor there?! If there were some > > genuine filesystem iop gains/etc for this change, I think I'd > > likely support it further .. > > It's going to reduce time, power consumption, and heat generation for > all portage users. Also, we can use portage.util.file_copy to optimize > it further with zero-copy, reflink, and sparse file support. > > > On this basis, what do the performance differences look like on an > > 'average' package > > Well, it's very inefficient to fork/exec the install command for many > files as the existing bash implementation does. The performance > difference is related to the number of files. > > > .. and are there any regressions in this regard?! > > It's supposed to fallback to calling the install command if there are > any unrecognized options, so the intention is for 100% compatibility. > > > I take issue with the copyright assignment, as I believe the > > legal .. err .. IANAL devs are campaigning for full rights to be > > owned and enforced by Gentoo Inc LLC in the US .. even if they have > > no idea what that means or does .. :] > > Since it's a BSD-style license, we can copy the code into our project > as long as we retain the copyright notice. Patches look fine to me, glad for the speedup :) -- Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] emerge --getbinpkg: https support for If-Modified-Since
On Mon, 31 Jul 2017 00:11:09 -0700
Zac Medico <zmed...@gentoo.org> wrote:
> When https certificate and hostname verification is enabled for
> stdlib http clients (PEP 476), use python for If-Modified-Since
> header support. When python lacks PEP 476 support, continue to
> use FETCHCOMMAND for https certificate and hostname verification
> (see security bug 469888).
>
> X-Gentoo-bug: 625246
> X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=625246
> ---
> pym/portage/dbapi/bintree.py | 10 ++
> pym/portage/util/_urlopen.py | 12
> 2 files changed, 18 insertions(+), 4 deletions(-)
>
> diff --git a/pym/portage/dbapi/bintree.py
> b/pym/portage/dbapi/bintree.py index c833968c2..95bd5dbf8 100644
> --- a/pym/portage/dbapi/bintree.py
> +++ b/pym/portage/dbapi/bintree.py
> @@ -18,7 +18,7 @@ portage.proxy.lazyimport.lazyimport(globals(),
> 'portage.util:atomic_ofstream,ensure_dirs,normalize_path,' +
> \ 'writemsg,writemsg_stdout',
> 'portage.util.path:first_existing',
> - 'portage.util._urlopen:urlopen@_urlopen',
> + 'portage.util._urlopen:urlopen@_urlopen,have_pep_476@_have_pep_476',
> 'portage.versions:best,catpkgsplit,catsplit,_pkg_str',
> )
>
> @@ -851,9 +851,9 @@ class binarytree(object):
> download_timestamp +
> ttl > time.time(): raise UseCachedCopyOfRemoteIndex()
>
> - # Don't use urlopen for https, since
> it doesn't support
> - # certificate/hostname verification
> (bug #469888).
> - if parsed_url.scheme not in
> ('https',):
> + # Don't use urlopen for https, unless
> + # PEP 476 is supported (bug #469888).
> + if parsed_url.scheme not in
> ('https',) or _have_pep_476(): try:
> f = _urlopen(url,
> if_modified_since=local_timestamp) if hasattr(f, 'headers') and
> f.headers.get('timestamp', ''): @@ -965,6 +965,8 @@ class
> binarytree(object): "\n")
> rmt_idx = pkgindex
> except EnvironmentError as e:
> + # This includes URLError which is
> raised for SSL
> + # certificate errors when PEP 476 is
> supported. writemsg(_("\n\n!!! Error fetching binhost package" \
> " info from '%s'\n") %
> _hide_url_passwd(base_url)) # With Python 2, the EnvironmentError
> message may diff --git a/pym/portage/util/_urlopen.py
> b/pym/portage/util/_urlopen.py index 4cfe183b1..fc9db74a0 100644
> --- a/pym/portage/util/_urlopen.py
> +++ b/pym/portage/util/_urlopen.py
> @@ -26,6 +26,18 @@ if sys.hexversion >= 0x300:
> # and the file-'mtime'
> TIMESTAMP_TOLERANCE = 5
>
> +
> +def have_pep_476():
> + """
> + Test whether ssl certificate verification is enabled by
> default for
> + stdlib http clients (PEP 476).
> +
> + @returns: bool, True if ssl certificate verification is
> enabled by
> + default
> + """
> + return hasattr(__import__('ssl'),
> '_create_unverified_context') +
> +
> def urlopen(url, if_modified_since=None):
> parse_result = urllib_parse.urlparse(url)
> if parse_result.scheme not in ("http", "https"):
looks fine
--
Brian Dolbec
Re: [gentoo-portage-dev] Leader election
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On Sun, 2 Jul 2017 11:28:09 -0700 Brian Dolbec <dol...@gentoo.org> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA512 > > On Sat, 24 Jun 2017 20:20:16 -0700 > Brian Dolbec <dol...@gentoo.org> wrote: > > > On Sun, 18 Jun 2017 15:48:44 +0200 > > Alexander Berntsen <berna...@gentoo.org> wrote: > > > > > Friends, > > > > > > It's that time of year. We're having a leader election again, as > > > well as a general development meeting. The agenda will be updated > > > in more detail at: > > > https://wiki.gentoo.org/wiki/Project:Portage/Meetings > > > > > > Please schedule a time at: http://whenisgood.net/portage > > > > > > Thanks. > > > - -- > > > Alexander > > > berna...@gentoo.org > > > https://secure.plaimi.net/~alexander > > > > > > > I've picked Sunday, July 2 at 4:00 PM UTC > > > > We've decided (the members in attendance), to do the lead election via > email. > > So, nominations are open from now to July 5, 2017. > > Voting will be closed July 10, 2017, results posted here again. > OK, looks the all cast votes were for Zac. So, by unanimous decision, I hereby declare Zac as our new project lead. Congrats Zac. - -- Brian Dolbec -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQKTBAEBCgB9FiEEpdfHTggcxw20pKr1+70IcnWCDtgFAllk+z1fFIAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEE1 RDdDNzRFMDgxQ0M3MERCNEE0QUFGNUZCQkQwODcyNzU4MjBFRDgACgkQ+70IcnWC Dtj0Ew//Xl+GzPKM3+0EKKMJn+Q6G2/7z5byPFl6bPevYFTPlmmN7a+AxULw+NfB 7+EwqYdVzLk51GLzBk4Y3Pgt7gFEPbigBB/w2Fubb24QvNzwUiZ3O6N8F4o7BbVh 8T0yuRYpFn79mWIOACudEgCt9HfSSAYcCREjIWyqrXSxuqCs4SKGRhReKx8S2XGt eUjD67HCCUKGmnVIZJrOFiQomDegLxl/gUqB2kHvzZWnpRVfYGHJbLXGCQtR8fro Qm2qvnnIbyxtYf3xaCKLPTCMqYwOtQuEL2xwuJ/KVvIcRErxJIYgutksjJIodEWj vE18hE7RzVsIyiIB/wvlJk2+ViTRSwlkRY2af6Sq0QakPlI+OtrKgqxPNQE041hl kMWogpawZm3z5hCwh9TuGW9ZCbI3ego6woTYp9DE9B953UiYwg0MbjBnh4AFeE0O Qy23zybjrD6ANzc1328W3hZpU5jjWnWEAvOlaH/pGH7vYBgdMfPcs0vyijkKE6gg JOwrFGpIAqcFBkIfa6cUgc2OQX3yBKZJgrqfFHniRo0KGXt3Jl9e4+eG0EK6CcqZ eVSAjflo+5mCVjaVGugXrcTd3O6/SVzsUH+/woazyLKogc1x+4rzk0dVmfDBPuow zFEbdcPzlJW6wxpPhCjHz1GFD/MzZsEvAxGW+q+GF71GLlJXeKI= =hDeZ -END PGP SIGNATURE-
Re: [gentoo-portage-dev] Re: [gentoo-dev] taking a break from arches stabilization
On Mon, 10 Jul 2017 20:20:12 +0200 Alexander Berntsen <berna...@gentoo.org> wrote: > On 10/07/17 20:11, Jonas Stein wrote: > > It would be so motivating to see that many user are glad about a > > special package. One gets rarely feedback. > > Interesting idea. We could have some (separate) portage-y helper tool > send a standardised email that could easily be filtered based on > sender and/or topic. 'emerge foo --thank' just thanks people, 'emerge > foo --thank="msg"' thanks people with a msg. > > It's gimmicky and could be abused. But then again perhaps it wouldn't > be abused, and it would just be wholesome fun. Maybe someone would > meet their future lover via the thank-parametre. > > What do you think? We had a past GSOC project that ended up with a decent start to a stats system. I know I had hoped for (and suggested) a survey system for it. A maintainer could put up a stabilization survey, or some other query about a pkg/version. The people interested could fill it out, the maintainer could get feedback. It would also give estimates of the number of people using the pkg/version (dpends on the number of people with stats enabled). There was even a dev that recently tried to get it deployed (again). I volunteered to help. But it never got the vm from it from infra. Infra is not set up to allow root access to a system for only certain systems/devs. So in order to maintain a system, you primarily have to be in the infra team. So, as a result, progress stalls, dies out. -- Brian Dolbec pgpVROTFPVD5Q.pgp Description: OpenPGP digital signature
Re: [gentoo-portage-dev] [PATCH] fuzzy search: weigh category similarity independently (bug 623648)
On Sat, 8 Jul 2017 13:03:58 -0700 Zac Medico <zmed...@gentoo.org> wrote: > Weigh the similarity of category and package names independently, > in order to avoid matching lots of irrelevant packages in the same > category when the package name is much shorter than the category > name. > > X-Gentoo-bug: 623648 > X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=623648 > --- > pym/_emerge/search.py | 24 +--- > 1 file changed, 21 insertions(+), 3 deletions(-) > > diff --git a/pym/_emerge/search.py b/pym/_emerge/search.py > index 20a0c026e..dc91ad315 100644 > --- a/pym/_emerge/search.py > +++ b/pym/_emerge/search.py > @@ -264,15 +264,33 @@ class search(object): > if self.fuzzy: > fuzzy = True > cutoff = > float(self.search_similarity) / 100 > - seq_match = difflib.SequenceMatcher() > - > seq_match.set_seq2(self.searchkey.lower()) > + if match_category: > + # Weigh the similarity of > category and package > + # names independently, in > order to avoid matching > + # lots of irrelevant > packages in the same category > + # when the package name is > much shorter than the > + # category name. > + part_split = portage.catsplit > + else: > + part_split = lambda > match_string: (match_string,) > - def fuzzy_search(match_string): > + part_matchers = [] > + for part in > part_split(self.searchkey): > + seq_match = > difflib.SequenceMatcher() > + > seq_match.set_seq2(part.lower()) > + > part_matchers.append(seq_match) + > + def fuzzy_search_part(seq_match, > match_string): seq_match.set_seq1(match_string.lower()) > return > (seq_match.real_quick_ratio() >= cutoff and seq_match.quick_ratio() > >= cutoff and seq_match.ratio() >= cutoff) > > + def fuzzy_search(match_string): > + return > all(fuzzy_search_part(seq_match, part) > + for seq_match, part > in zip( > + part_matchers, > part_split(match_string))) + > for package in self._cp_all(): > self._spinner_update() > looks fine -- Brian Dolbec
Re: [gentoo-portage-dev] Leader election
On Thu, 6 Jul 2017 20:12:46 +0200 Alexander Berntsen <berna...@gentoo.org> wrote: > On 06/07/17 19:25, Brian Dolbec wrote: > > You can't, I declined this time around > You didn't reply here. So technically you just didn't accept. ;) > > I vote for Zac then. Crap, I didn't notice that it wasn't sent to the list for my reply. I forwarded it. -- Brian Dolbec pgpBAbic0A84k.pgp Description: OpenPGP digital signature
Fw: [gentoo-portage-dev] Leader election
Begin forwarded message: Date: Thu, 6 Jul 2017 08:40:45 -0700 From: Brian Dolbec <dol...@gentoo.org> To: Zac Medico <zmed...@gentoo.org> Subject: Re: [gentoo-portage-dev] Leader election On Tue, 4 Jul 2017 13:02:35 -0700 Zac Medico <zmed...@gentoo.org> wrote: > On 07/02/2017 12:02 PM, Brian Dolbec wrote: > > On Sun, 2 Jul 2017 11:28:09 -0700 > > Brian Dolbec <dol...@gentoo.org> wrote: > > > >> On Sat, 24 Jun 2017 20:20:16 -0700 > >> Brian Dolbec <dol...@gentoo.org> wrote: > > > >>> On Sun, 18 Jun 2017 15:48:44 +0200 > >>> Alexander Berntsen <berna...@gentoo.org> wrote: > >>> > >>>> Friends, > >>>> > >>>> It's that time of year. We're having a leader election again, as > >>>> well as a general development meeting. The agenda will be updated > >>>> in more detail at: > >>>> https://wiki.gentoo.org/wiki/Project:Portage/Meetings > >>>> > >>>> Please schedule a time at: http://whenisgood.net/portage > >>>> > >>>> Thanks. > >>>> - -- > >>>> Alexander > >>>> berna...@gentoo.org > >>>> https://secure.plaimi.net/~alexander > >>>> > >>> > >>> I've picked Sunday, July 2 at 4:00 PM UTC > >>> > > > >> We've decided (the members in attendance), to do the lead election > >> via email. > > > >> So, nominations are open from now to July 5, 2017. > > > >> Voting will be closed July 10, 2017, results posted here again. > > > >> - -- > >> Brian Dolbec > > > > > > I nominate Zac > > Thank you, I accept. > > I also nominate Brian Dolbec and Alexander Berntsen. Thanks Zac, but, I decline :( I have not been a very attentive lead this past year. I also have a lot of catch-up work to do in other projects as well as portage. So, I would like to withdraw from the lead role, hopefully I'll be able to get caught up in those projects. -- Brian Dolbec -- Brian Dolbec pgp5KiXdzOqzq.pgp Description: OpenPGP digital signature
Re: [gentoo-portage-dev] Leader election
On Thu, 6 Jul 2017 17:45:33 +0200 Alexander Berntsen <berna...@gentoo.org> wrote: > I vote for Brian. You can't, I declined this time around -- Brian Dolbec pgp3t5YNJcNZh.pgp Description: OpenPGP digital signature
Re: [gentoo-portage-dev] Leader election
On Thu, 6 Jul 2017 08:35:13 -0700 Brian Dolbec <dol...@gentoo.org> wrote: > On Thu, 6 Jul 2017 11:40:45 +0200 > Alexander Berntsen <berna...@gentoo.org> wrote: > > > On 04/07/17 22:02, Zac Medico wrote: > > > Thank you, I accept. > > > > > > I also nominate Brian Dolbec and Alexander Berntsen. > > Thank you, I accept. > > > > > > Do we vote here in the same thread? > > yeah > With the voting now open, I cast my vote for Zac. -- Brian Dolbec pgpWdMDpnSJJn.pgp Description: OpenPGP digital signature
Re: [gentoo-portage-dev] Leader election
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On Sun, 2 Jul 2017 11:28:09 -0700 Brian Dolbec <dol...@gentoo.org> wrote: > On Sat, 24 Jun 2017 20:20:16 -0700 > Brian Dolbec <dol...@gentoo.org> wrote: > > > On Sun, 18 Jun 2017 15:48:44 +0200 > > Alexander Berntsen <berna...@gentoo.org> wrote: > > > > > Friends, > > > > > > It's that time of year. We're having a leader election again, as > > > well as a general development meeting. The agenda will be updated > > > in more detail at: > > > https://wiki.gentoo.org/wiki/Project:Portage/Meetings > > > > > > Please schedule a time at: http://whenisgood.net/portage > > > > > > Thanks. > > > - -- > > > Alexander > > > berna...@gentoo.org > > > https://secure.plaimi.net/~alexander > > > > > > > I've picked Sunday, July 2 at 4:00 PM UTC > > > > We've decided (the members in attendance), to do the lead election via > email. > > So, nominations are open from now to July 5, 2017. > > Voting will be closed July 10, 2017, results posted here again. > > - -- > Brian Dolbec > I nominate Zac - -- Brian Dolbec -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQKTBAEBCgB9FiEEpdfHTggcxw20pKr1+70IcnWCDtgFAllZQ1RfFIAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEE1 RDdDNzRFMDgxQ0M3MERCNEE0QUFGNUZCQkQwODcyNzU4MjBFRDgACgkQ+70IcnWC DthGIQ/9FRnG7Y+FpZ5RO5WeUuXSOOgD0RyrpfHU29K1HYn1PMibP/gYMv5TVlEC uJzXUqoGj5MUVMefVtddUwtRM+Xmqml4NWMSpH1f6Pzwb4jyx2siWtA3Y54xF6Hn 3cfj0XI0yJUQDUEFteAD3OgAHHPDDQtSSf2JhiAO0ddErkHAKaggLkKLBwxbhR6L JE9Ei8+txvCLCpi+ojeQp7oy9jfRTyiPj7Dl1ognUv6jA1ob6ED71vJiaPrPC8qT uEHhK3mxp0yLWE0tEUYFS4ULJMVcLMzCmZo11fkTCqNgJjF/oTDF1wjBDHiuxqza 5yiU1f8r/BN9WvksJ2TTZVKAVo//6/Lq2p3TvqiERlbR67g5WbgOnjE4+TiSlCG4 PhN1oSZIyp9fn2eU3OxTvXwHga6PFlPfo++l7GzJgM/RRiCyD699tyxnDzQ4papd 2Tr4PziPLLs4S79pfncM07knFwiEwGtAy6eT2GRTOOlZ+vb6HwNEdUwDoXKSvw6j GcOCYk9GBcJNTuK/aGYp0RbVrh6s6JKU1y+TiB7f4v/7Idxfm3eH8i5XuCnQjT2J oHAH+SJs3PqGSh+/jeGlOkiDh7oPvgdBYisUTtwzJ1r+snHKgfwDSrZrQ6MO3WcR URHl7Pzy2nRmBLhjlZJwhNPtFHBsSkF7PR+KuxJjIkYLZNFrO6Q= =dvs9 -END PGP SIGNATURE-
Re: [gentoo-portage-dev] [PATCH] doman: QA-complain on compressed files
On Sun, 2 Jul 2017 19:57:57 +0200
Michał Górny <mgo...@gentoo.org> wrote:
> Passing compressed files to doman is not permitted by the PMS.
> However, Portage permitting it results in many developers repeatedly
> violating this and arguing that it's permitted because nobody
> explicitly forbids it. Add an explicit QA warning to Portage to cease
> it. ---
> bin/ebuild-helpers/doman | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/bin/ebuild-helpers/doman b/bin/ebuild-helpers/doman
> index da66ac335..43d29f22e 100755
> --- a/bin/ebuild-helpers/doman
> +++ b/bin/ebuild-helpers/doman
> @@ -30,6 +30,7 @@ for x in "$@" ; do
>
> # These will be automatically decompressed by ecompressdir.
> if has ${suffix} Z gz bz2 ; then
> + eqawarn "QA Notice: doman argument '${x}' is
> compressed, this is not portable" realname=${x%.*}
> suffix=${realname##*.}
> fi
sounds good
--
Brian Dolbec
Re: [gentoo-portage-dev] Leader election
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On Sat, 24 Jun 2017 20:20:16 -0700 Brian Dolbec <dol...@gentoo.org> wrote: > On Sun, 18 Jun 2017 15:48:44 +0200 > Alexander Berntsen <berna...@gentoo.org> wrote: > > > Friends, > > > > It's that time of year. We're having a leader election again, as > > well as a general development meeting. The agenda will be updated > > in more detail at: > > https://wiki.gentoo.org/wiki/Project:Portage/Meetings > > > > Please schedule a time at: http://whenisgood.net/portage > > > > Thanks. > > - -- > > Alexander > > berna...@gentoo.org > > https://secure.plaimi.net/~alexander > > > > I've picked Sunday, July 2 at 4:00 PM UTC > We've decided (the members in attendance), to do the lead election via email. So, nominations are open from now to July 5, 2017. Voting will be closed July 10, 2017, results posted here again. - -- Brian Dolbec -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQKTBAEBCgB9FiEEpdfHTggcxw20pKr1+70IcnWCDtgFAllZOzlfFIAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEE1 RDdDNzRFMDgxQ0M3MERCNEE0QUFGNUZCQkQwODcyNzU4MjBFRDgACgkQ+70IcnWC DtgpHA//SMhHQyMV+d99LCv9CxEISrCyy+5f1pOZFzkuow1L8XriahG8D0Yc67Gh qTOvHfuktQpZiCB8y2heZ6UnPJFSVCVjbRdPIfpWooG6Yv3fHGK8JZcEPB5X/zi3 pDTuJE1zrvlVmqOiramd4WvuCuhEi/811QGB87/Z1HjaoRi/i20TeeeYrwlqw7+f ioVqRhAgJkVFmWmoWrGM2rS4NaI2SjBhItlwv4Ty4MxdLNMAhG1t0BvnIP1uLGyj 2+X95Ih4rhhhyXLifwMD2HqzkBUIe9Zp+6PtR66ulhzgh5yBv+Om+ATtG3S2ICTz Fwh79MQKWC9UtugdL/c05eMuzmaSrDBOo8fC6WWRJ78FikCe2mTQGMz1mBTuR7l8 QZgnZexOauGUA+RWvK8PMNlQfBkj9LPGe4BsxA9StkH33o12aST8Fl7IGNRA+ajq TpsETss61/xull1Pizg1oTMsi1OIuQoyP2pYPDijHNRVD+2Ol2V86cVZPkLi1nZu Y7EDeLQxFZ5h+4406/vcRtSENugpV0WD0UtIcz3/h9nOEwKNiai2UODmtwV4bG44 Ua1BCahRla4RVQPIDSVav9E6HJY7oM1hcXY1yCh/DsqxMgfXeDlAJ0n0MbhRdEjc DbtF1LbsAJ/90/E3/lwhNSsDA3sm6YdBGJWKTh4JC+8SDRiAdSA= =FF3B -END PGP SIGNATURE-
Re: [gentoo-portage-dev] [PATCH v2] checks: Add more deprecated eclasses
On Mon, 19 Jun 2017 15:35:49 +0200
Michał Górny <mgo...@gentoo.org> wrote:
> Changes in v2:
> * multilib-minimal is suggested replacement of autotools-multilib.
>
> ---
> repoman/pym/repoman/modules/scan/ebuild/checks.py | 5 +
> 1 file changed, 5 insertions(+)
>
> diff --git a/repoman/pym/repoman/modules/scan/ebuild/checks.py
> b/repoman/pym/repoman/modules/scan/ebuild/checks.py index
> e6e5d78ba..de03bedd2 100644 ---
> a/repoman/pym/repoman/modules/scan/ebuild/checks.py +++
> b/repoman/pym/repoman/modules/scan/ebuild/checks.py @@ -416,14
> +416,18 @@ class InheritDeprecated(LineCheck):
> # deprecated eclass : new eclass (False if no new eclass)
> deprecated_eclasses = {
> + "autotools-multilib": "multilib-minimal",
> + "autotools-utils": False,
> "base": False,
> "bash-completion": "bash-completion-r1",
> "boost-utils": False,
> "clutter": "gnome2",
> "confutils": False,
> "distutils": "distutils-r1",
> + "fdo-mime": "xdg-utils",
> "games": False,
> "gems": "ruby-fakegem",
> + "git-2": "git-r3",
> "gpe": False,
> "gst-plugins-bad": "gstreamer",
> "gst-plugins-base": "gstreamer",
> @@ -434,6 +438,7 @@ class InheritDeprecated(LineCheck):
> "python": "python-r1 / python-single-r1 /
> python-any-r1", "ruby": "ruby-ng",
> "x-modular": "xorg-2",
> + "xfconf": False,
> }
>
> _inherit_re = re.compile(r'^\s*inherit\s(.*)$')
looks fine
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] GitSync: Support setting environment variables for git
shlexed_env =
> shlex_split(self.repo.module_specific_options['sync-git-env'])
> + env = dict((k, v) for k, _, v in
> (assignment.partition('=') for assignment in shlexed_env) if k)
> + self.spawn_kwargs['env'].update(env)
> +
> + if
> self.repo.module_specific_options.get('sync-git-clone-env'):
> + shlexed_env =
> shlex_split(self.repo.module_specific_options['sync-git-clone-env'])
> + clone_env = dict((k, v) for k, _, v in
> (assignment.partition('=') for assignment in shlexed_env) if k)
> + self.spawn_kwargs['env'].update(clone_env)
> +
> if self.settings.get("PORTAGE_QUIET") == "1":
> git_cmd_opts += " --quiet"
> if self.repo.clone_depth is not None:
> @@ -86,6 +96,16 @@ class GitSync(NewBase):
> '''
>
> git_cmd_opts = ""
> + if
> self.repo.module_specific_options.get('sync-git-env'):
> + shlexed_env =
> shlex_split(self.repo.module_specific_options['sync-git-env'])
> + env = dict((k, v) for k, _, v in
> (assignment.partition('=') for assignment in shlexed_env) if k)
> + self.spawn_kwargs['env'].update(env)
> +
> + if
> self.repo.module_specific_options.get('sync-git-pull-env'):
> + shlexed_env =
> shlex_split(self.repo.module_specific_options['sync-git-pull-env'])
> + pull_env = dict((k, v) for k, _, v in
> (assignment.partition('=') for assignment in shlexed_env) if k)
> + self.spawn_kwargs['env'].update(pull_env)
> +
> if self.settings.get("PORTAGE_QUIET") == "1":
> git_cmd_opts += " --quiet"
> if
> self.repo.module_specific_options.get('sync-git-pull-extra-opts'):
looks good, Thanks Manuel, Zac
--
Brian Dolbec
Re: [gentoo-portage-dev] Re: [PATCH v2] depgraph: prune unnecessary rebuilds for --autounmask-continue (bug 619626)
On Thu, 1 Jun 2017 09:11:55 -0700 Zac Medico <zmed...@gentoo.org> wrote: > On Thu, Jun 1, 2017 at 8:50 AM, Zac Medico <zmed...@gentoo.org> wrote: > > > > @@ -6224,10 +6283,10 @@ class depgraph(object): > > iuses = > > pkg.iuse.all > > old_use = > > self._pkg_use_enabled(pkg) > > if myeb: > > - > > pkgsettings.setcpv(myeb) > > + > > now_use = self._pkg_use_enabled(myeb) > > else: > > > > pkgsettings.setcpv(pkg) > > - now_use = > > pkgsettings["PORTAGE_USE"].split() > > + > > now_use = pkgsettings["PORTAGE_USE"].split() > > forced_flags > > = set() > > > > forced_flags.update(pkgsettings.useforce) > > > > forced_flags.update(pkgsettings.usemask) > > > > > I've updated the above code to generate forced_flags correctly: > > if myeb: > now_use = self._pkg_use_enabled(myeb) > forced_flags = set(chain( > myeb.use.force, myeb.use.mask)) > else: > pkgsettings.setcpv(pkg) > now_use = pkgsettings["PORTAGE_USE"].split() > forced_flags = set(chain( > pkgsettings.useforce, pkgsettings.usemask)) > OK, LGTM -- Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] depgraph: account for binpkg-multi-instance in unused warning (bug 619620)
On Mon, 29 May 2017 15:11:41 -0700 Zac Medico <zmed...@gentoo.org> wrote: > On Mon, May 29, 2017 at 3:03 PM, Brian Dolbec <dol...@gentoo.org> > wrote: > > > On Sun, 28 May 2017 03:07:50 -0700 > > Zac Medico <zmed...@gentoo.org> wrote: > > > > > With FEATURES=binpkg-multi-instance, it is normal to have some > > > unused binary packages, so don't warn if the selected package > > > is the latest version and the most recent build. > > > > >^^^ > > is or isn't the latest version? > > > is the latest version (in this case we "don't warn"). OK, WFM -- Brian Dolbec
Re: [gentoo-portage-dev] [PATCH v2] man/portage.5: document -* in profile "packages" files (bug 610670)
On Sun, 28 May 2017 13:15:12 -0700 Zac Medico <zmed...@gentoo.org> wrote: > The -* wildcard has been supported since portage-2.3.4, but it was > not explicitly documented. > > X-Gentoo-Bug: 610670 > X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=610670 > --- > [PATCH v2] clarifies that -* discards @system and @profile > packages as suggested by Martin Vaeth. > > man/portage.5 | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/man/portage.5 b/man/portage.5 > index 82e979ecf..366a1fa85 100644 > --- a/man/portage.5 > +++ b/man/portage.5 > @@ -369,7 +369,8 @@ hurt \fBemerge\fR's ability to parallelize. > .I Note: > In a cascading profile setup, you can remove packages in children > profiles which were added by parent profiles by prefixing the atom > with -a '\-'. > +a '\-'. The '\-*' wildcard discards all @system and @profile packages > +added by parent profiles. > > .I Example: > .nf LGTM -- Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] depgraph: account for binpkg-multi-instance in unused warning (bug 619620)
On Sun, 28 May 2017 03:07:50 -0700 Zac Medico <zmed...@gentoo.org> wrote: > With FEATURES=binpkg-multi-instance, it is normal to have some > unused binary packages, so don't warn if the selected package > is the latest version and the most recent build. > ^^^ is or isn't the latest version? > X-Gentoo-bug: 619620 > X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=619620 > --- > pym/_emerge/depgraph.py | 9 + > 1 file changed, 9 insertions(+) > > diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py > index 726835dd4..2dc432431 100644 > --- a/pym/_emerge/depgraph.py > +++ b/pym/_emerge/depgraph.py > @@ -876,6 +876,15 @@ class depgraph(object): > > self._dynamic_config.ignored_binaries.pop(pkg) > break > > + # NOTE: The Package.__ge__ > implementation accounts for > + # differences in build_time, so the > warning about "ignored" > + # packages will be triggered if both > packages are the same > + # version and selected_pkg is not > the most recent build. > + if (selected_pkg.type_name == > "binary" and > + selected_pkg >= pkg): > + > self._dynamic_config.ignored_binaries.pop(pkg) > + break > + > if selected_pkg.installed and \ > selected_pkg.cpv == pkg.cpv > and \ selected_pkg.build_time == pkg.build_time: -- Brian Dolbec
Re: [gentoo-portage-dev] [PATCH 1/2] emerge: terminate backtracking early for autounmask changes (bug 615680)
uired by app-misc/B (argument)
> + #=app-misc/D-0 foo
> +
> + # NOTE: The --autounmask-backtrack=n output
> is preferable here,
> + # because it highlights the unsolvable
> dependency conflict.
> + # It would be better if it eliminated the
> autounmask suggestion,
> + # since that suggestion won't solve the
> conflict. )
>
> playground = ResolverPlayground(ebuilds=ebuilds,
> debug=False) diff --git
> a/pym/portage/tests/resolver/test_slot_conflict_unsatisfied_deep_deps.py
> b/pym/portage/tests/resolver/test_slot_conflict_unsatisfied_deep_deps.py
> index 13f7e67..846ba0e 100644 ---
> a/pym/portage/tests/resolver/test_slot_conflict_unsatisfied_deep_deps.py
> +++
> b/pym/portage/tests/resolver/test_slot_conflict_unsatisfied_deep_deps.py
> @@ -79,6 +79,7 @@ class
> SlotConflictUnsatisfiedDeepDepsTestCase(TestCase): ["@world"],
> options={ "--autounmask": "y",
> + "--autounmask-backtrack":
> "y", "--complete-graph": True,
> "--selective": True,
> "--deep": 1
> @@ -89,11 +90,63 @@ class
> SlotConflictUnsatisfiedDeepDepsTestCase(TestCase):
> unsatisfied_deps=["dev-libs/initially-unsatisfied"], success=False),
>
> + # With --autounmask-backtrack=y:
> + #[ebuild N~] dev-libs/A-2
> + #[ebuild N ] dev-libs/C-1
> + #[ebuild N ] dev-libs/D-1
> + #[ebuild N ] dev-libs/B-1
> + #
> + #The following keyword changes are necessary
> to proceed:
> + # (see "package.accept_keywords" in the
> portage(5) man page for more details)
> + ## required by dev-libs/C-1::test_repo
> + ## required by @selected
> + ## required by @world (argument)
> + #=dev-libs/A-2 ~x86
> + #
> + #!!! Problems have been detected with your
> world file
> + #!!! Please run emaint --check world
> + #
> + #
> + #!!! Ebuilds for the following packages are
> either all
> + #!!! masked or don't exist:
> + #dev-libs/broken
> + #
> + #emerge: there are no ebuilds to satisfy
> "dev-libs/initially-unsatisfied".
> + #(dependency required by
> "dev-libs/broken-1::test_repo" [installed])
> + #(dependency required by "@selected" [set])
> + #(dependency required by "@world" [argument])
> +
> + # Without --autounmask-backtrack=y:
> + #!!! Multiple package instances within a
> single package slot have been pulled
> + #!!! into the dependency graph, resulting in
> a slot conflict:
> + #
> + #dev-libs/A:0
> + #
> + # (dev-libs/A-1:0/0::test_repo, ebuild
> scheduled for merge) pulled in by
> + #(no parents that aren't satisfied by
> other packages in this slot)
> + #
> + # (dev-libs/A-2:0/0::test_repo, ebuild
> scheduled for merge) pulled in by
> + #>=dev-libs/A-2 required by
> (dev-libs/C-1:0/0::test_repo, ebuild scheduled for merge)
> + #^^ ^
> + #
> + #The following keyword changes are necessary
> to proceed:
> + # (see "package.accept_keywords" in the
> portage(5) man page for more details)
> + ## required by dev-libs/C-1::test_repo
> + ## required by @selected
> + ## required by @world (argument)
> + #=dev-libs/A-2 ~x86
> + #
> + #emerge: there are no ebuilds to satisfy
> "dev-libs/initially-unsatisfied".
> + #(dependency required by
> "dev-libs/broken-1::test_repo" [installed])
> + #(dependency required by "@selected" [set])
> + #(dependency required by "@world" [argument])
> +
> # Test --deep = True
> ResolverPlaygroundTestCase(
> ["@world"],
> options={
> "--autounmask": "y",
> + "--autounmask-backtrack":
> "y", "--complete-graph": True,
> "--selective": True,
> "--deep": True
> @@ -103,6 +156,14 @@ class
> SlotConflictUnsatisfiedDeepDepsTestCase(TestCase):
> unstable_keywords=["dev-libs/A-2"],
> unsatisfied_deps=["dev-libs/initially-unsatisfied"], success=False),
> +
> + # The effects of --autounmask-backtrack are
> the same as the previous test case.
> + # Both test cases can randomly succeed with
> --autounmask-backtrack=n, when
> + # "backtracking due to unsatisfied dep"
> randomly occurs before the autounmask
> + # unstable keyword change. It would be
> possible to eliminate backtracking here
> + # by recognizing that there are no
> alternatives to satisfy the dev-libs/broken
> + # atom in the world file. Then the test
> cases will consistently succeed with
> + # --autounmask-backtrack=n.
> )
>
> playground = ResolverPlayground(ebuilds=ebuilds,
> installed=installed,
Looks good, thank you :)
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH 2/2] emerge: default --backtrack=10 (bug 540562)
On Tue, 9 May 2017 20:52:35 -0700
Zac Medico <zmed...@gentoo.org> wrote:
> Since the default behavior is now for emerge to terminate early for
> autounmask changes (unless either --autounmask-backtrack=y or
> --autounmask-continue is enabled), it is much less likely that time
> will be wasted by fruitless backtracking. Therefore, raise the default
> backtrack value from 3 to 10, restoring it to the value it had prior
> to commit 1891388ea0ae0dd58903a71a3adc779731523601 (see bug 536926).
> This will allow many users to avoid having to manually raise the
> --backtrack value.
>
> X-Gentoo-bug: 540562
> X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=540562
> ---
> man/emerge.1| 2 +-
> pym/_emerge/depgraph.py | 2 +-
> 2 files changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/man/emerge.1 b/man/emerge.1
> index 94edc90..240d4de 100644
> --- a/man/emerge.1
> +++ b/man/emerge.1
> @@ -412,7 +412,7 @@ precedence over existing changes. This option is
> automatically enabled with .BR \-\-backtrack=COUNT
> Specifies an integer number of times to backtrack if
> dependency calculation fails due to a conflict or an
> -unsatisfied dependency (default: \'3\').
> +unsatisfied dependency (default: \'10\').
> .TP
> .BR "\-\-binpkg\-changed\-deps [ y | n ]"
> Tells emerge to ignore binary packages for which the corresponding
> diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py
> index 53910dd..cda497b 100644
> --- a/pym/_emerge/depgraph.py
> +++ b/pym/_emerge/depgraph.py
> @@ -9460,7 +9460,7 @@ def _backtrack_depgraph(settings, trees,
> myopts, myparams, myaction, myfiles, sp
> debug = "--debug" in myopts
> mydepgraph = None
> - max_retries = myopts.get('--backtrack', 3)
> + max_retries = myopts.get('--backtrack', 10)
> max_depth = max(1, (max_retries + 1) // 2)
> allow_backtracking = max_retries > 0
> backtracker = Backtracker(max_depth)
yeah, is good :)
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] file_copy: fix lseek offset after EINTR (bug 618086)
On Tue, 9 May 2017 23:59:34 -0700
Zac Medico <zmed...@gentoo.org> wrote:
> Fix the lseek offset for the plain read/write loop to account
> for buffered data that has not been written to to the output
> file yet (due to previous interruption by EINTR). This code
> only affects Linux 2.6.32 and earlier (newer kernels use
> copy_file_range or sendfile).
>
> X-Gentoo-bug: 618086
> X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=618086
> ---
> src/portage_util_file_copy_reflink_linux.c | 13 -
> 1 file changed, 8 insertions(+), 5 deletions(-)
>
> diff --git a/src/portage_util_file_copy_reflink_linux.c
> b/src/portage_util_file_copy_reflink_linux.c index 2fb17a0..4be9e05
> 100644 --- a/src/portage_util_file_copy_reflink_linux.c
> +++ b/src/portage_util_file_copy_reflink_linux.c
> @@ -323,12 +323,14 @@ _reflink_linux_file_copy(PyObject *self,
> PyObject *args) if (buf == NULL) {
> error = errno;
>
> -/* For the read call, the fd_in file offset must be
> - * exactly equal to offset_out. Use lseek to ensure
> - * correct state, in case an EINTR retry caused it to
> - * get out of sync somewhow.
> +/* For the read call, the fd_in file offset must be
> exactly
> + * equal to offset_out + buf_bytes, where buf_bytes
> is the
> + * amount of buffered data that has not been written
> to
> + * to the output file yet. Use lseek to ensure
> correct state,
> + * in case an EINTR retry caused it to get out of
> sync
> + * somewhow.
> */
> -} else if (lseek(fd_in, offset_out, SEEK_SET) < 0) {
> +} else if (lseek(fd_in, offset_out + buf_bytes,
> SEEK_SET) < 0) { error = errno;
> } else {
> while (1) {
> @@ -345,6 +347,7 @@ _reflink_linux_file_copy(PyObject *self, PyObject
> *args)
> } else if (buf_bytes < 0) {
> error = errno;
> + buf_bytes = 0;
> break;
> }
> }
looks fine :)
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] emerge: add --onlydeps-with-rdeps=<y|n> option (bug 294719)
bs/D-1": { },
> + }
> + installed = {
> + }
> +
> + test_cases = (
> + ResolverPlaygroundTestCase(
> + ["dev-libs/A"],
> + all_permutations = True,
> + success = True,
> + options = { "--onlydeps": True,
> + "--onlydeps-with-rdeps":
> "y" },
> + mergelist = ["dev-libs/B-1",
> + "dev-libs/C-1",
> + "dev-libs/D-1"]),
> + ResolverPlaygroundTestCase(
> + ["dev-libs/A"],
> + all_permutations = True,
> + success = True,
> + options = { "--onlydeps": True,
> + "--onlydeps-with-rdeps":
> "n" },
> + mergelist = ["dev-libs/B-1"]),
> + )
> +
> + playground = ResolverPlayground(ebuilds=ebuilds,
> + installed=installed, debug=False)
> + try:
> + for test_case in test_cases:
> + playground.run_TestCase(test_case)
> +
> self.assertEqual(test_case.test_success, True, test_case.fail_msg)
> + finally:
> + playground.cleanup()
looks fine
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] file_copy: replace loff_t with off_t for portability (bug 617778)
On Sun, 7 May 2017 16:50:40 -0700
Zac Medico <zmed...@gentoo.org> wrote:
> The loff_t type is a GNU extension, so use the portable off_t
> type instead. Also, enable Large File Support macros in setup.py,
> for 64-bit offsets.
>
> Reported-by: Patrick Steinhardt <p...@pks.im>
> X-Gentoo-bug: 617778
> X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=617778
> ---
> setup.py | 5 -
> src/portage_util_file_copy_reflink_linux.c | 6 +++---
> 2 files changed, 7 insertions(+), 4 deletions(-)
>
> diff --git a/setup.py b/setup.py
> index e993177..1ba6f87 100755
> --- a/setup.py
> +++ b/setup.py
> @@ -676,7 +676,10 @@ setup(
> ['$sysconfdir/portage/repo.postsync.d',
> ['cnf/repo.postsync.d/example']], ],
>
> - ext_modules = [Extension(name=n, sources=m) for n, m in
> x_c_helpers.items()],
> + ext_modules = [Extension(name=n, sources=m,
> + extra_compile_args=['-D_FILE_OFFSET_BITS=64',
> + '-D_LARGEFILE_SOURCE', '-D_LARGEFILE64_SOURCE'])
> + for n, m in x_c_helpers.items()],
>
> cmdclass = {
> 'build': x_build,
> diff --git a/src/portage_util_file_copy_reflink_linux.c
> b/src/portage_util_file_copy_reflink_linux.c index b031d96..2fb17a0
> 100644 --- a/src/portage_util_file_copy_reflink_linux.c
> +++ b/src/portage_util_file_copy_reflink_linux.c
> @@ -66,7 +66,7 @@ initreflink_linux(void)
> * (errno is set appropriately).
> */
> static ssize_t
> -cfr_wrapper(int fd_out, int fd_in, loff_t *off_out, size_t len)
> +cfr_wrapper(int fd_out, int fd_in, off_t *off_out, size_t len)
> {
> #ifdef __NR_copy_file_range
> return syscall(__NR_copy_file_range, fd_in, NULL, fd_out,
> @@ -96,7 +96,7 @@ cfr_wrapper(int fd_out, int fd_in, loff_t *off_out,
> size_t len)
> * reaches EOF.
> */
> static off_t
> -do_lseek_data(int fd_out, int fd_in, loff_t *off_out) {
> +do_lseek_data(int fd_out, int fd_in, off_t *off_out) {
> #ifdef SEEK_DATA
> /* Use lseek SEEK_DATA/SEEK_HOLE for sparse file support,
> * as suggested in the copy_file_range man page.
> @@ -189,7 +189,7 @@ _reflink_linux_file_copy(PyObject *self, PyObject
> *args) ssize_t buf_bytes, buf_offset, copyfunc_ret;
> struct stat stat_in, stat_out;
> char* buf;
> -ssize_t (*copyfunc)(int, int, loff_t *, size_t);
> + ssize_t (*copyfunc)(int, int, off_t *, size_t);
>
> if (!PyArg_ParseTuple(args, "ii", _in, _out))
> return NULL;
looks fine and is Anthony approved ;)
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] Eventloop: fix deadlock involving idle_add/call_soon (bug 617550)
On Fri, 5 May 2017 02:12:53 -0700
Zac Medico <zmed...@gentoo.org> wrote:
> Guarantee that newly added idle_add/call_soon callbacks have an
> opportunity to execute before the event loop decides to wait on
> self._thread_condition without a timeout. This fixes a case where
> the event loop would wait on self._thread_condition indefinitely,
> even though a callback scheduled by the AsynchronousTask._async_wait
> method needed to be executed first.
>
> X-Gentoo-bug: 617550
> X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=617550
> ---
> pym/portage/util/_eventloop/EventLoop.py | 18 --
> 1 file changed, 16 insertions(+), 2 deletions(-)
>
> diff --git a/pym/portage/util/_eventloop/EventLoop.py
> b/pym/portage/util/_eventloop/EventLoop.py index 712838e..b7c8762
> 100644 --- a/pym/portage/util/_eventloop/EventLoop.py
> +++ b/pym/portage/util/_eventloop/EventLoop.py
> @@ -108,6 +108,15 @@ class EventLoop(object):
> self._poll_event_handler_ids = {}
> # Increment id for each new handler.
> self._event_handler_id = 0
> + # New call_soon callbacks must have an opportunity to
> + # execute before it's safe to wait on
> self._thread_condition
> + # without a timeout, since delaying its execution
> indefinitely
> + # could lead to a deadlock. The following attribute
> stores the
> + # event handler id of the most recently added
> call_soon callback.
> + # If this attribute has changed since the last time
> that the
> + # call_soon callbacks have been called, then it's
> not safe to
> + # wait on self._thread_condition without a timeout.
> + self._call_soon_id = 0
> # Use OrderedDict in order to emulate the FIFO queue
> behavior # of the AbstractEventLoop.call_soon method.
> self._idle_callbacks = OrderedDict()
> @@ -250,10 +259,15 @@ class EventLoop(object):
>
> if not event_handlers:
> with self._thread_condition:
> + prev_call_soon_id =
> self._call_soon_id if self._run_timeouts():
> events_handled += 1
> timeouts_checked = True
> - if not event_handlers and not
> events_handled and may_block: +
> + call_soon =
> bool(prev_call_soon_id != self._call_soon_id) +
> + if (not call_soon and not
> event_handlers
> + and not events_handled and
> may_block): # Block so that we don't waste cpu time by looping too
> # quickly. This makes
> EventLoop useful for code that needs # to wait for timeout callbacks
> regardless of whether or @@ -457,7 +471,7 @@ class EventLoop(object):
> @return: an integer ID
> """
> with self._thread_condition:
> - source_id = self._new_source_id()
> + source_id = self._call_soon_id =
> self._new_source_id() self._idle_callbacks[source_id] =
> self._idle_callback_class( args=args, callback=callback,
> source_id=source_id) self._thread_condition.notify()
looks good
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] repoman: Add a check for relative dosym candidates
On Sat, 15 Apr 2017 13:04:56 +0200
Michał Górny <mgo...@gentoo.org> wrote:
> Add a check for dosym with target path matching absolute paths
> controlled by the package manager, e.g. /bin, /etc...
>
> Example output:
>
> ebuild.absdosym 5
>app-editors/nano/nano-2.5.3.ebuild: dosym '/bin/nano'... could use
> relative path on line: 81 ---
> repoman/man/repoman.1 | 4
> repoman/pym/repoman/modules/scan/ebuild/checks.py | 2 +-
> repoman/pym/repoman/qa_data.py| 4
> 3 files changed, 9 insertions(+), 1 deletion(-)
>
> diff --git a/repoman/man/repoman.1 b/repoman/man/repoman.1
> index 9b106906f..78e4b7275 100644
> --- a/repoman/man/repoman.1
> +++ b/repoman/man/repoman.1
> @@ -292,6 +292,10 @@ Some files listed in SRC_URI aren't referenced
> in the Manifest .B digest.unused
> Some files listed in the Manifest aren't referenced in SRC_URI
> .TP
> +.B ebuild.absdosym
> +Ebuild uses 'dosym' with explicit absolute path where relative path
> +could be used
> +.TP
> .B ebuild.badheader
> This ebuild has a malformed header
> .TP
> diff --git a/repoman/pym/repoman/modules/scan/ebuild/checks.py
> b/repoman/pym/repoman/modules/scan/ebuild/checks.py index
> 57c8b10a5..e6e5d78ba 100644 ---
> a/repoman/pym/repoman/modules/scan/ebuild/checks.py +++
> b/repoman/pym/repoman/modules/scan/ebuild/checks.py @@ -921,7 +921,7
> @@ class EbuildNonRelativeDosym(LineCheck): def check(self, num,
> line): match = self.regex.match(line)
> if match:
> - return "dosym '%s' could use relative path"
> % (match.group(1), ) + " on line: %d"
> + return "dosym '%s'... could use relative
> path" % (match.group(1), ) + " on line: %d"
>
> _base_check_classes = (InheritEclass, LineCheck, PhaseCheck)
> diff --git a/repoman/pym/repoman/qa_data.py
> b/repoman/pym/repoman/qa_data.py index 132a55be3..a59fed778 100644
> --- a/repoman/pym/repoman/qa_data.py
> +++ b/repoman/pym/repoman/qa_data.py
> @@ -195,6 +195,9 @@ qahelp = {
> "Some files listed in SRC_URI aren't referenced in
> the Manifest"), "digest.unused": (
> "Some files listed in the Manifest aren't referenced
> in SRC_URI"),
> + "ebuild.absdosym": (
> + "This ebuild uses absolute target to dosym where
> relative symlink"
> + " could be used instead"),
> "ebuild.majorsyn": (
> "This ebuild has a major syntax error"
> " that may cause the ebuild to fail partially or
> fully"), @@ -262,6 +265,7 @@ qawarnings = set((
> "RDEPEND.suspect",
> "virtual.suspect",
> "RESTRICT.invalid",
> + "ebuild.absdosym",
> "ebuild.minorsyn",
> "ebuild.badheader",
> "ebuild.patches",
looks good. merge approved :)
thank you
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH 2/2] EbuildBuild: async spawn_nofetch in _fetchonly_exit (bug 614116)
eturn os.EX_OK
> -
> - # We must create our private PORTAGE_TMPDIR before calling
> - # doebuild_environment(), since lots of variables such
> - # as PORTAGE_BUILDDIR refer to paths inside PORTAGE_TMPDIR.
> - portage_tmpdir = settings.get('PORTAGE_TMPDIR')
> - if not portage_tmpdir or not os.access(portage_tmpdir,
> os.W_OK):
> - portage_tmpdir = None
> - private_tmpdir = tempfile.mkdtemp(dir=portage_tmpdir)
> - settings['PORTAGE_TMPDIR'] = private_tmpdir
> - settings.backup_changes('PORTAGE_TMPDIR')
> - # private temp dir was just created, so it's not locked yet
> - settings.pop('PORTAGE_BUILDDIR_LOCKED', None)
> -
> - try:
> - doebuild_environment(ebuild_path, 'nofetch',
> - settings=settings, db=portdb)
> + # Prevent temporary config changes from interfering
> + # with config instances that are reused.
> + settings = self.settings = config(clone=settings)
> +
> + # We must create our private PORTAGE_TMPDIR before
> calling
> + # doebuild_environment(), since lots of variables
> such
> + # as PORTAGE_BUILDDIR refer to paths inside
> PORTAGE_TMPDIR.
> + portage_tmpdir = settings.get('PORTAGE_TMPDIR')
> + if not portage_tmpdir or not
> os.access(portage_tmpdir, os.W_OK):
> + portage_tmpdir = None
> + private_tmpdir = self._private_tmpdir =
> tempfile.mkdtemp(
> + dir=portage_tmpdir)
> + settings['PORTAGE_TMPDIR'] = private_tmpdir
> + settings.backup_changes('PORTAGE_TMPDIR')
> + # private temp dir was just created, so it's not
> locked yet
> + settings.pop('PORTAGE_BUILDDIR_LOCKED', None)
> +
> + doebuild_environment(self.ebuild_path, 'nofetch',
> + settings=settings, db=self.portdb)
> restrict = settings['PORTAGE_RESTRICT'].split()
> defined_phases = settings['DEFINED_PHASES'].split()
> if not defined_phases:
> @@ -76,18 +88,38 @@ def spawn_nofetch(portdb, ebuild_path,
> settings=None, fd_pipes=None):
> if 'fetch' not in restrict and \
> 'nofetch' not in defined_phases:
> - return os.EX_OK
> + self.returncode = os.EX_OK
> + self._async_wait()
> + return
>
> prepare_build_dirs(settings=settings)
> - ebuild_phase = EbuildPhase(background=False,
> +
> + ebuild_phase =
> EbuildPhase(background=self.background, phase='nofetch',
> -
> scheduler=SchedulerInterface(portage._internal_caller and
> + scheduler=self.scheduler,
> + fd_pipes=self.fd_pipes, settings=settings)
> +
> + self._start_task(ebuild_phase, self._nofetch_exit)
> +
> + def _nofetch_exit(self, ebuild_phase):
> + self._final_exit(ebuild_phase)
> + elog_process(self.settings.mycpv, self.settings)
> + shutil.rmtree(self._private_tmpdir)
> + self._async_wait()
> +
> +
> +def spawn_nofetch(portdb, ebuild_path, settings=None, fd_pipes=None):
> + """
> + Create a NofetchPrivateTmpdir instance, and execute it
> synchronously.
> + This function must not be called from asynchronous code,
> since it will
> + trigger event loop recursion which is incompatible with
> asyncio.
> + """
> + nofetch = SpawnNofetchWithoutBuilddir(background=False,
> + portdb=portdb,
> + ebuild_path=ebuild_path,
> +
> scheduler=SchedulerInterface(portage._internal_caller and
> global_event_loop() or EventLoop(main=False)),
> - fd_pipes=fd_pipes, settings=settings)
> - ebuild_phase.start()
> - ebuild_phase.wait()
> - elog_process(settings.mycpv, settings)
> - finally:
> - shutil.rmtree(private_tmpdir)
> -
> - return ebuild_phase.returncode
> + fd_pipes=fd_pipes, settings=settings)
> +
> + nofetch.start()
> + return nofetch.wait()
looks good
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH 1/2] EbuildBuild: eliminate call to digestgen (bug 614116)
On Sun, 2 Apr 2017 19:36:53 -0700 Zac Medico <zmed...@gentoo.org> wrote: > Eliminate the call to digestgen in EbuildBuild._fetchonly_exit, > and make Scheduler._generate_digests call it earlier when > --fetchonly is enabled. This avoids event loop recursion which > is not compatible with asyncio (digestgen makes many calls that > can trigger event loop recursion). > > X-Gentoo-bug: 614116 > X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=614116 > --- > pym/_emerge/EbuildBuild.py | 5 - > pym/_emerge/Scheduler.py | 3 --- > 2 files changed, 8 deletions(-) > > diff --git a/pym/_emerge/EbuildBuild.py b/pym/_emerge/EbuildBuild.py > index 001f55f..11eb1c9 100644 > --- a/pym/_emerge/EbuildBuild.py > +++ b/pym/_emerge/EbuildBuild.py > @@ -21,7 +21,6 @@ from _emerge.TaskSequence import TaskSequence > import portage > from portage import _encodings, _unicode_decode, _unicode_encode, os > from portage.package.ebuild.digestcheck import digestcheck > -from portage.package.ebuild.digestgen import digestgen > from portage.package.ebuild.doebuild import _check_temp_dir > from portage.package.ebuild._spawn_nofetch import spawn_nofetch > > @@ -168,10 +167,6 @@ class EbuildBuild(CompositeTask): > if self.returncode != os.EX_OK: > portdb = > self.pkg.root_config.trees[self._tree].dbapi spawn_nofetch(portdb, > self._ebuild_path, settings=self.settings) > - elif 'digest' in self.settings.features: > - if not digestgen(mysettings=self.settings, > - > myportdb=self.pkg.root_config.trees[self._tree].dbapi): > - self.returncode = 1 > self.wait() > > def _pre_clean_exit(self, pre_clean_phase): > diff --git a/pym/_emerge/Scheduler.py b/pym/_emerge/Scheduler.py > index 58ff971..079fac7 100644 > --- a/pym/_emerge/Scheduler.py > +++ b/pym/_emerge/Scheduler.py > @@ -616,9 +616,6 @@ class Scheduler(PollScheduler): > tasks are started. > """ > > - if '--fetchonly' in self.myopts: > - return os.EX_OK > - > digest = '--digest' in self.myopts > if not digest: > for pkgsettings in self.pkgsettings.values(): looks good -- Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] AsynchronousLock: add async_unlock method (bug 614108)
self):
> self._registered = False
>
> @@ -156,7 +205,8 @@ class _LockProcess(AbstractPollTask):
> """
>
> __slots__ = ('path',) + \
> - ('_acquired', '_kill_test', '_proc', '_files',
> '_reg_id', '_unlocked')
> + ('_acquired', '_kill_test', '_proc', '_files',
> + '_reg_id','_unlock_future')
>
> def _start(self):
> in_pr, in_pw = os.pipe()
> @@ -223,13 +273,16 @@ class _LockProcess(AbstractPollTask):
> return
>
> if not self.cancelled and \
> - not self._unlocked:
> + self._unlock_future is None:
> # We don't want lost locks going
> unnoticed, so it's # only safe to ignore if either the cancel() or
> # unlock() methods have been
> previously called. raise AssertionError("lock process failed with
> returncode %s" \ % (proc.returncode,))
>
> + if self._unlock_future is not None:
> + self._unlock_future.set_result(None)
> +
> def _cancel(self):
> if self._proc is not None:
> self._proc.cancel()
> @@ -271,16 +324,36 @@ class _LockProcess(AbstractPollTask):
> os.close(pipe_in)
>
> def unlock(self):
> + """
> + This method is deprecated in favor of async_unlock,
> since waiting
> + for the child process to respond can trigger event
> loop recursion
> + which is incompatible with asyncio.
> + """
> + self._unlock()
> + self._proc.wait()
> + self._proc = None
> +
> + def _unlock(self):
> if self._proc is None:
> raise AssertionError('not locked')
> - if self.returncode is None:
> + if not self._acquired:
> raise AssertionError('lock not acquired yet')
> if self.returncode != os.EX_OK:
> raise AssertionError("lock process failed
> with returncode %s" \ % (self.returncode,))
> - self._unlocked = True
> + if self._unlock_future is not None:
> + raise AssertionError("already unlocked")
> + self._unlock_future = self.scheduler.create_future()
> os.write(self._files['pipe_out'], b'\0')
> os.close(self._files['pipe_out'])
> self._files = None
> - self._proc.wait()
> - self._proc = None
> +
> + def async_unlock(self):
> + """
> + Release the lock asynchronously. Release
> notification is available
> + via the add_done_callback method of the returned
> Future instance. +
> + @returns: Future, result is None
> + """
> + self._unlock()
> + return self._unlock_future
> diff --git a/pym/portage/tests/locks/test_asynchronous_lock.py
> b/pym/portage/tests/locks/test_asynchronous_lock.py index
> 3a2ccfb..ab67242 100644 ---
> a/pym/portage/tests/locks/test_asynchronous_lock.py +++
> b/pym/portage/tests/locks/test_asynchronous_lock.py @@ -1,6 +1,7 @@
> # Copyright 2010-2011 Gentoo Foundation
> # Distributed under the terms of the GNU General Public License v2
>
> +import itertools
> import signal
> import tempfile
>
> @@ -17,7 +18,8 @@ class AsynchronousLockTestCase(TestCase):
> tempdir = tempfile.mkdtemp()
> try:
> path = os.path.join(tempdir, 'lock_me')
> - for force_async in (True, False):
> + for force_async, async_unlock in
> itertools.product(
> + (True, False), repeat=2):
> for force_dummy in (True, False):
> async_lock =
> AsynchronousLock(path=path, scheduler=scheduler,
> _force_async=force_async, @@ -26,7 +28,10 @@ class
> AsynchronousLockTestCase(TestCase): async_lock.start()
> self.assertEqual(async_lock.wait(),
> os.EX_OK) self.assertEqual(async_lock.returncode, os.EX_OK)
> - async_lock.unlock()
> + if async_unlock:
> +
> scheduler.run_until_complete(async_lock.async_unlock())
> + else:
> + async_lock.unlock()
>
> async_lock =
> AsynchronousLock(path=path, scheduler=scheduler,
> _force_async=force_async, @@ -34,8 +39,10 @@ class
> AsynchronousLockTestCase(TestCase): async_lock.start()
> self.assertEqual(async_lock.wait(),
> os.EX_OK) self.assertEqual(async_lock.returncode, os.EX_OK)
> - async_lock.unlock()
> -
> + if async_unlock:
> +
> scheduler.run_until_complete(async_lock.async_unlock())
> + else:
> + async_lock.unlock()
> finally:
> shutil.rmtree(tempdir)
>
looks fine :)
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] emerge: fix --autounmask-continue to work with --getbinpkg (bug 614474)
On Sat, 1 Apr 2017 18:02:03 -0700
Zac Medico <zmed...@gentoo.org> wrote:
> Fix action_build to populate binarytree instances with remote package
> metadata following config reload, using a new getbinpkg_refresh=False
> parameter to prevent redundant fetching of remote package metadata.
>
> X-Gentoo-bug: 614474
> X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=614474
> Fixes: e2d88ef3ff59 ("Add emerge --autounmask-continue option (bug
> 582624)") ---
> pym/_emerge/actions.py | 15 +++
> pym/portage/dbapi/bintree.py | 19 +++
> 2 files changed, 30 insertions(+), 4 deletions(-)
>
> diff --git a/pym/_emerge/actions.py b/pym/_emerge/actions.py
> index cc0269d..818fab9 100644
> --- a/pym/_emerge/actions.py
> +++ b/pym/_emerge/actions.py
> @@ -347,6 +347,21 @@ def action_build(emerge_config,
> trees=DeprecationWarning, adjust_configs(emerge_config.opts,
> emerge_config.trees) settings, trees, mtimedb = emerge_config
>
> + # After config reload, the freshly
> instantiated binarytree
> + # instances need to load remote metadata if
> --getbinpkg
> + # is enabled. Use getbinpkg_refresh=False to
> use cached
> + # metadata, since the cache is already fresh.
> + if "--getbinpkg" in emerge_config.opts:
> + for root_trees in
> emerge_config.trees.values():
> + try:
> +
> root_trees["bintree"].populate(
> +
> getbinpkgs=True,
> +
> getbinpkg_refresh=False)
> + except ParseError as e:
> +
> writemsg("\n\n!!!%s.\nSee make.conf(5) for more info.\n"
> + %
> e, noiselevel=-1)
> + return 1
> +
> if "--autounmask-only" in myopts:
> mydepgraph.display_problems()
> return 0
> diff --git a/pym/portage/dbapi/bintree.py
> b/pym/portage/dbapi/bintree.py index 12c3d3e..ca90ba8 100644
> --- a/pym/portage/dbapi/bintree.py
> +++ b/pym/portage/dbapi/bintree.py
> @@ -510,8 +510,16 @@ class binarytree(object):
> except PortageException:
> pass
>
> - def populate(self, getbinpkgs=0):
> - "populates the binarytree"
> + def populate(self, getbinpkgs=False, getbinpkg_refresh=True):
> + """
> + Populates the binarytree with package metadata.
> +
> + @param getbinpkgs: include remote packages
> + @type getbinpkgs: bool
> + @param getbinpkg_refresh: attempt to refresh the
> cache
> + of remote package metadata if getbinpkgs is
> also True
> + @type getbinpkg_refresh: bool
> + """
>
> if self._populating:
> return
> @@ -522,13 +530,13 @@ class binarytree(object):
> pkgindex_lock =
> lockfile(self._pkgindex_file, wantnewlockfile=1)
> self._populating = True
> - self._populate(getbinpkgs)
> + self._populate(getbinpkgs,
> getbinpkg_refresh=getbinpkg_refresh) finally:
> if pkgindex_lock:
> unlockfile(pkgindex_lock)
> self._populating = False
>
> - def _populate(self, getbinpkgs=0):
> + def _populate(self, getbinpkgs=False,
> getbinpkg_refresh=True): if (not os.path.isdir(self.pkgdir) and not
> getbinpkgs): return 0
>
> @@ -832,6 +840,9 @@ class binarytree(object):
> url = base_url.rstrip("/") +
> "/Packages" f = None
>
> + if not getbinpkg_refresh and
> local_timestamp:
> + raise
> UseCachedCopyOfRemoteIndex() +
> try:
> ttl =
> float(pkgindex.header.get("TTL", 0)) except ValueError:
looks good. Thanks for migrating that 0 for False
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] depgraph: trigger slot operator rebuilds via _complete_graph (bug 614390)
"RDEPEND": "dev-libs/foo:=
> app-misc/B",
> + },
> +
> + "app-misc/C-2" : {
> + "EAPI": "6",
> + "DEPEND": "dev-libs/foo:=
> app-misc/B",
> + "RDEPEND": "dev-libs/foo:=
> app-misc/B",
> + },
> +
> + "app-misc/D-1" : {
> + "EAPI": "6",
> + "DEPEND": "dev-libs/foo:=",
> + "RDEPEND": "dev-libs/foo:=",
> + },
> +
> + "app-misc/D-2" : {
> + "EAPI": "6",
> + "DEPEND": "dev-libs/foo:=",
> + "RDEPEND": "dev-libs/foo:=",
> + },
> +
> + "dev-libs/foo-1" : {
> + "EAPI": "6",
> + "SLOT": "0/1",
> + },
> +
> + "dev-libs/foo-2" : {
> + "EAPI": "6",
> + "SLOT": "0/2",
> + },
> + }
> +
> + installed = {
> + "app-misc/meta-pkg-1" : {
> + "EAPI": "6",
> + "DEPEND": "=app-misc/B-1
> =app-misc/C-1 =app-misc/D-1 =dev-libs/foo-1",
> + "RDEPEND": "=app-misc/B-1
> =app-misc/C-1 =app-misc/D-1 =dev-libs/foo-1",
> + },
> +
> + "app-misc/B-1" : {
> + "EAPI": "6",
> + "DEPEND": "dev-libs/foo:0/1=",
> + "RDEPEND": "dev-libs/foo:0/1=",
> + },
> +
> + "app-misc/C-1" : {
> + "EAPI": "6",
> + "DEPEND": "dev-libs/foo:0/1=
> app-misc/B",
> + "RDEPEND": "dev-libs/foo:0/1=
> app-misc/B",
> + },
> +
> + "app-misc/D-1" : {
> + "EAPI": "6",
> + "DEPEND": "dev-libs/foo:0/1=",
> + "RDEPEND": "dev-libs/foo:0/1=",
> + },
> +
> + "dev-libs/foo-1" : {
> + "EAPI": "6",
> + "SLOT": "0/1",
> + },
> + }
> +
> + world = (
> + "app-misc/meta-pkg",
> + )
> +
> + test_cases = (
> + # Test bug 614390, where the
> depgraph._complete_graph
> + # method pulled in an installed package that
> had been
> + # scheduled for rebuild by the previous
> calculation,
> + # triggering an unsolved slot conflict and
> preventing
> + # slot operator rebuilds.
> + ResolverPlaygroundTestCase(
> + ["=app-misc/meta-pkg-2",
> "app-misc/C"],
> + options = {
> + "--backtrack": 5,
> + },
> + success = True,
> + ambiguous_merge_order = True,
> + mergelist = [
> + 'dev-libs/foo-2',
> + ('app-misc/D-1',
> 'app-misc/C-1', 'app-misc/B-2'),
> + 'app-misc/meta-pkg-2',
> + ]
> + ),
> + )
> +
> + playground = ResolverPlayground(debug=False,
> + ebuilds=ebuilds, installed=installed,
> + world=world)
> + try:
> + for test_case in test_cases:
> + playground.run_TestCase(test_case)
> +
> self.assertEqual(test_case.test_success, True,
> + test_case.fail_msg)
> + finally:
> + # Disable debug so that cleanup works.
> + playground.debug = False
> + playground.cleanup()
looks good. I look forward to getting less conflicts with this...
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] pkg_use_display: show masked/forced state of USE_EXPAND flags (bug 490562)
On Wed, 22 Mar 2017 09:32:36 -0700
Zac Medico <zmed...@gentoo.org> wrote:
> Fix pkg_use_display to test if the prefixed flag is in use.force or
> use.mask, rather than the unprefixed flag.
>
> X-Gentoo-bug: 490562
> X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=490562
> ---
> pym/_emerge/UseFlagDisplay.py | 19 +--
> 1 file changed, 13 insertions(+), 6 deletions(-)
>
> diff --git a/pym/_emerge/UseFlagDisplay.py
> b/pym/_emerge/UseFlagDisplay.py index f460474..12820e9 100644
> --- a/pym/_emerge/UseFlagDisplay.py
> +++ b/pym/_emerge/UseFlagDisplay.py
> @@ -3,6 +3,7 @@
>
> from __future__ import unicode_literals
>
> +import collections
> from itertools import chain
> import sys
>
> @@ -60,6 +61,10 @@ class UseFlagDisplay(object):
> sort_separated = cmp_sort_key(_cmp_separated)
> del _cmp_separated
>
> +
> +_flag_info = collections.namedtuple('_flag_info', ('flag',
> 'display')) +
> +
> def pkg_use_display(pkg, opts, modified_use=None):
> settings = pkg.root_config.settings
> use_expand = pkg.use.expand
> @@ -81,27 +86,29 @@ def pkg_use_display(pkg, opts, modified_use=None):
> if f.startswith(flag_prefix):
> use_expand_flags.add(f)
> use_enabled.setdefault(
> - varname.upper(),
> []).append(f[len(flag_prefix):])
> + varname.upper(), []).append(
> + _flag_info(f,
> f[len(flag_prefix):]))
> for f in pkg.iuse.all:
> if f.startswith(flag_prefix):
> use_expand_flags.add(f)
> if f not in use:
> use_disabled.setdefault(
> - varname.upper(),
> []).append(f[len(flag_prefix):])
> + varname.upper(),
> []).append(
> +
> _flag_info(f, f[len(flag_prefix):]))
> var_order = set(use_enabled)
> var_order.update(use_disabled)
> var_order = sorted(var_order)
> var_order.insert(0, 'USE')
> use.difference_update(use_expand_flags)
> - use_enabled['USE'] = list(use)
> + use_enabled['USE'] = list(_flag_info(f, f) for f in use)
> use_disabled['USE'] = []
>
> for f in pkg.iuse.all:
> if f not in use and \
> f not in use_expand_flags:
> - use_disabled['USE'].append(f)
> + use_disabled['USE'].append(_flag_info(f, f))
>
> flag_displays = []
> for varname in var_order:
> @@ -109,9 +116,9 @@ def pkg_use_display(pkg, opts, modified_use=None):
> continue
> flags = []
> for f in use_enabled.get(varname, []):
> - flags.append(UseFlagDisplay(f, True, f in
> forced_flags))
> + flags.append(UseFlagDisplay(f.display, True,
> f.flag in forced_flags)) for f in use_disabled.get(varname, []):
> - flags.append(UseFlagDisplay(f, False, f in
> forced_flags))
> + flags.append(UseFlagDisplay(f.display,
> False, f.flag in forced_flags)) if alphabetical_use:
> flags.sort(key=UseFlagDisplay.sort_combined)
> else:
Sorry, Zac, I somehow missed this one... too busy with my head burried
in twisted test fixes and new deps...
Looks fine :)
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] SpawnProcess: fix event loop recursion in _pipe_logger_exit (bug 613990)
On Sun, 26 Mar 2017 23:54:37 -0700
Zac Medico <zmed...@gentoo.org> wrote:
> Fix SpawnProcess._pipe_logger_exit to wait for process exit status
> asynchronously, in order to avoid event loop recursion. This is
> required for asyncio compatibility, and also protects emerge from
> exceeding the maximum recursion depth limit like in bug 402335.
>
> X-Gentoo-bug: 613990
> X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=613990
> ---
> pym/_emerge/SpawnProcess.py | 3 +--
> pym/_emerge/SubProcess.py | 23 ++-
> 2 files changed, 23 insertions(+), 3 deletions(-)
>
> diff --git a/pym/_emerge/SpawnProcess.py b/pym/_emerge/SpawnProcess.py
> index e046640..7326254 100644
> --- a/pym/_emerge/SpawnProcess.py
> +++ b/pym/_emerge/SpawnProcess.py
> @@ -170,8 +170,7 @@ class SpawnProcess(SubProcess):
>
> def _pipe_logger_exit(self, pipe_logger):
> self._pipe_logger = None
> - self._unregister()
> - self.wait()
> + self._async_waitpid()
>
> def _waitpid_loop(self):
> SubProcess._waitpid_loop(self)
> diff --git a/pym/_emerge/SubProcess.py b/pym/_emerge/SubProcess.py
> index 13d9382..b81cfd5 100644
> --- a/pym/_emerge/SubProcess.py
> +++ b/pym/_emerge/SubProcess.py
> @@ -12,7 +12,7 @@ import errno
> class SubProcess(AbstractPollTask):
>
> __slots__ = ("pid",) + \
> - ("_dummy_pipe_fd", "_files", "_reg_id")
> + ("_dummy_pipe_fd", "_files", "_reg_id",
> "_waitpid_id")
> # This is how much time we allow for waitpid to succeed after
> # we've sent a kill signal to our subprocess.
> @@ -101,6 +101,23 @@ class SubProcess(AbstractPollTask):
>
> return self.returncode
>
> + def _async_waitpid(self):
> + """
> + Wait for exit status of self.pid asynchronously, and
> then
> + set the returncode and notify exit listeners. This is
> + prefered over _waitpid_loop, since the synchronous
> nature
> + of _waitpid_loop can cause event loop recursion.
> + """
> + if self._waitpid_id is None:
> + self._waitpid_id =
> self.scheduler.child_watch_add(
> + self.pid, self._async_waitpid_cb)
> +
> + def _async_waitpid_cb(self, pid, condition, user_data=None):
> + if pid != self.pid:
> + raise AssertionError("expected pid %s, got
> %s" % (self.pid, pid))
> + self._set_returncode((pid, condition))
> + self.wait()
> +
> def _waitpid_loop(self):
> source_id = self.scheduler.child_watch_add(
> self.pid, self._waitpid_cb)
> @@ -129,6 +146,10 @@ class SubProcess(AbstractPollTask):
> self.scheduler.source_remove(self._reg_id)
> self._reg_id = None
>
> + if self._waitpid_id is not None:
> +
> self.scheduler.source_remove(self._waitpid_id)
> + self._waitpid_id = None
> +
> if self._files is not None:
> for f in self._files.values():
> if isinstance(f, int):
looks fine
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] emerge: use asyncio interfaces for spinner during owner lookup (bug 591760)
On Sun, 26 Mar 2017 18:01:01 -0700 Zac Medico <zmed...@gentoo.org> wrote: > X-Gentoo-bug: 591760 > X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=591760 > --- > pym/_emerge/depgraph.py | 13 - > pym/portage/dbapi/vartree.py | 22 +- > 2 files changed, 17 insertions(+), 18 deletions(-) > > diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py > index 7c9130b..04e724d 100644 > --- a/pym/_emerge/depgraph.py > +++ b/pym/_emerge/depgraph.py > @@ -3668,17 +3668,20 @@ class depgraph(object): > def select_files(self, args): > # Use the global event loop for spinner progress > # indication during file owner lookups (bug #461412). > - spinner_id = None > + def spinner_cb(): > + self._frozen_config.spinner.update() > + spinner_cb.handle = > self._event_loop.call_soon(spinner_cb) + > + spinner_cb.handle = None > try: > spinner = self._frozen_config.spinner > if spinner is not None and \ > spinner.update is not > spinner.update_quiet: > - spinner_id = > self._event_loop.idle_add( > - > self._frozen_config.spinner.update) > + spinner_cb.handle = > self._event_loop.call_soon(spinner_cb) return self._select_files(args) > finally: > - if spinner_id is not None: > - > self._event_loop.source_remove(spinner_id) > + if spinner_cb.handle is not None: > + spinner_cb.handle.cancel() > > def _select_files(self, myfiles): > """Given a list of .tbz2s, .ebuilds sets, and deps, > populate diff --git a/pym/portage/dbapi/vartree.py > b/pym/portage/dbapi/vartree.py index c421dc5..7c8f150 100644 > --- a/pym/portage/dbapi/vartree.py > +++ b/pym/portage/dbapi/vartree.py > @@ -1369,32 +1369,28 @@ class vardbapi(dbapi): > global_event_loop() or > EventLoop(main=False)) root = self._vardb._eroot > > - def search_pkg(cpv): > + def search_pkg(cpv, search_future): > dblnk = self._vardb._dblink(cpv) > + results = [] > for path, name, is_basename in > path_info_list: if is_basename: > for p in > dblnk._contents.keys(): if os.path.basename(p) == name: > - > search_pkg.results.append((dblnk, > + > results.append((dblnk, dblnk._contents.unmap_key( > > p)[len(root):])) > else: > key = > dblnk._match_contents(path) if key is not False: > - > search_pkg.results.append( > + > results.append( (dblnk, key[len(root):])) > - search_pkg.complete = True > - return False > - > - search_pkg.results = [] > + search_future.set_result(results) > > for cpv in self._vardb.cpv_all(): > - del search_pkg.results[:] > - search_pkg.complete = False > - event_loop.idle_add(search_pkg, cpv) > - while not search_pkg.complete: > - event_loop.iteration() > - for result in search_pkg.results: > + search_future = > event_loop.create_future() > + event_loop.call_soon(search_pkg, > cpv, search_future) > + > event_loop.run_until_complete(search_future) > + for result in search_future.result(): > yield result > > class vartree(object): looks fine. Question, will this help make the spinner more consistent with spin? -- Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] Future: implement add_done_callback for asyncio compat (bug 591760)
# Class variables serving as defaults for instance
> variables. _state = _PENDING
> _result = None
> _exception = None
> + _loop = None
> +
> + def __init__(self, loop=None):
> + """Initialize the future.
> +
> + The optional loop argument allows explicitly
> setting the event
> + loop object used by the future. If it's not
> provided, the future uses
> + the default event loop.
> + """
> + if loop is None:
> + self._loop = global_event_loop()
> + else:
> + self._loop = loop
> + self._callbacks = []
>
> def cancel(self):
> """Cancel the future and schedule callbacks.
> @@ -54,8 +78,27 @@ except ImportError:
> if self._state != _PENDING:
> return False
> self._state = _CANCELLED
> + self._schedule_callbacks()
> return True
>
> + def _schedule_callbacks(self):
> + """Internal: Ask the event loop to call all
> callbacks. +
> + The callbacks are scheduled to be called as
> soon as possible. Also
> + clears the callback list.
> + """
> + callbacks = self._callbacks[:]
> + if not callbacks:
> + return
> +
> + self._callbacks[:] = []
> + for callback in callbacks:
> + self._loop.call_soon(callback, self)
> +
> + def cancelled(self):
> + """Return True if the future was
> cancelled."""
> + return self._state == _CANCELLED
> +
> def done(self):
> """Return True if the future is done.
>
> @@ -93,6 +136,29 @@ except ImportError:
> raise InvalidStateError('Exception
> is not set.') return self._exception
>
> + def add_done_callback(self, fn):
> + """Add a callback to be run when the future
> becomes done. +
> + The callback is called with a single
> argument - the future object. If
> + the future is already done when this is
> called, the callback is
> + scheduled with call_soon.
> + """
> + if self._state != _PENDING:
> + self._loop.call_soon(fn, self)
> + else:
> + self._callbacks.append(fn)
> +
> + def remove_done_callback(self, fn):
> + """Remove all instances of a callback from
> the "call when done" list. +
> + Returns the number of callbacks removed.
> + """
> + filtered_callbacks = [f for f in
> self._callbacks if f != fn]
> + removed_count = len(self._callbacks) -
> len(filtered_callbacks)
> + if removed_count:
> + self._callbacks[:] =
> filtered_callbacks
> + return removed_count
> +
> def set_result(self, result):
> """Mark the future done and set its result.
>
> @@ -103,6 +169,7 @@ except ImportError:
> raise InvalidStateError('{}:
> {!r}'.format(self._state, self)) self._result = result
> self._state = _FINISHED
> + self._schedule_callbacks()
>
> def set_exception(self, exception):
> """Mark the future done and set an exception.
> @@ -116,3 +183,8 @@ except ImportError:
> exception = exception()
> self._exception = exception
> self._state = _FINISHED
> + self._schedule_callbacks()
> +
> +
> +if Future is None:
> + Future = _EventLoopFuture
looks fine... /me ignoring the lack of parameters descriptions in the
docstrings
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH 5/5] FetchIterator: add terminate method
On Thu, 23 Mar 2017 19:55:00 -0700
Zac Medico <zmed...@gentoo.org> wrote:
> Add a terminate method to FetchIterator so that it doesn't
> delay termination of emirrordist via SIGINT. Otherwise it
> it is possible for the __iter__ method to loop for a very
> long time after SIGINT has been delivered. For example,
> this could happen if there are many ebuilds with stale
> cache and RESTRICT=mirror. This issue was discovered
> during testing of changes to the MirrorDistTask.terminate
> implementation.
> ---
> pym/portage/_emirrordist/FetchIterator.py | 21 +
> pym/portage/_emirrordist/MirrorDistTask.py | 6 +-
> 2 files changed, 26 insertions(+), 1 deletion(-)
>
> diff --git a/pym/portage/_emirrordist/FetchIterator.py
> b/pym/portage/_emirrordist/FetchIterator.py index 16a0b04..3841979
> 100644 --- a/pym/portage/_emirrordist/FetchIterator.py
> +++ b/pym/portage/_emirrordist/FetchIterator.py
> @@ -1,6 +1,8 @@
> # Copyright 2013 Gentoo Foundation
> # Distributed under the terms of the GNU General Public License v2
>
> +import threading
> +
> from portage import os
> from portage.checksum import (_apply_hash_filter,
> _filter_unaccelarated_hashes, _hash_filter)
> @@ -13,6 +15,19 @@ class FetchIterator(object):
> def __init__(self, config):
> self._config = config
> self._log_failure = config.log_failure
> + self._terminated = threading.Event()
> +
> + def terminate(self):
> + """
> + Schedules early termination of the __iter__ method,
> which is
> + useful because under some conditions it's possible
> for __iter__
> + to loop for a long time without yielding to the
> caller. For
> + example, it's useful when there are many ebuilds
> with stale
> + cache and RESTRICT=mirror.
> +
> + This method is thread-safe (and safe for signal
> handlers).
> + """
> + self._terminated.set()
>
> def _iter_every_cp(self):
> # List categories individually, in order to start
> yielding quicker, @@ -37,6 +52,9 @@ class FetchIterator(object):
>
> for cp in self._iter_every_cp():
>
> + if self._terminated.is_set():
> + return
> +
> for tree in portdb.porttrees:
>
> # Reset state so the Manifest is
> pulled once @@ -46,6 +64,9 @@ class FetchIterator(object):
>
> for cpv in portdb.cp_list(cp,
> mytree=tree):
> + if self._terminated.is_set():
> + return
> +
> try:
> restrict, =
> portdb.aux_get(cpv, ("RESTRICT",), mytree=tree)
> diff --git a/pym/portage/_emirrordist/MirrorDistTask.py
> b/pym/portage/_emirrordist/MirrorDistTask.py index 0702eb1..8da9f06
> 100644 --- a/pym/portage/_emirrordist/MirrorDistTask.py
> +++ b/pym/portage/_emirrordist/MirrorDistTask.py
> @@ -32,9 +32,11 @@ class MirrorDistTask(CompositeTask):
> self._config = config
> self._term_rlock = threading.RLock()
> self._term_callback_handle = None
> + self._fetch_iterator = None
>
> def _start(self):
> - fetch =
> TaskScheduler(iter(FetchIterator(self._config)),
> + self._fetch_iterator = FetchIterator(self._config)
> + fetch = TaskScheduler(iter(self._fetch_iterator),
> max_jobs=self._config.options.jobs,
> max_load=self._config.options.load_average,
> event_loop=self._config.event_loop)
> @@ -226,6 +228,8 @@ class MirrorDistTask(CompositeTask):
> self._term_callback)
>
> def _term_callback(self):
> + if self._fetch_iterator is not None:
> + self._fetch_iterator.terminate()
> self.cancel()
> self.wait()
>
Not that I know enough about all this that I could say "your doing it
wrong"
But the series looks OK, I didn't see any obvious goofs ;)
Thanks
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH v2] emerge: fix --usepkg when ebuild is not available (bug 613360)
On Mon, 20 Mar 2017 18:22:40 -0700
Zac Medico <zmed...@gentoo.org> wrote:
> Fix emerge --usepkg to use a binary package when the corresponding
> ebuild is not available (and --use-ebuild-visibility is not enabled),
> in cases when no other package is available to satisfy the dependency.
> This reverts an unintended behavior change from commit
> e309323f156528a8a79a1f755e1326e8880346b7.
>
> Fixes: e309323f1565 ("emerge: fix --use-ebuild-visibility to reject
> binary packages (bug 612960)") X-Gentoo-bug: 613360
> X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=613360
> ---
> [PATCH v2] moves the conditional logic earier, for more backward
> compatiblity
>
> pym/_emerge/depgraph.py| 3 ++-
> .../resolver/test_binary_pkg_ebuild_visibility.py | 26
> ++ 2 files changed, 28 insertions(+), 1
> deletion(-)
>
> diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py
> index 543f4dc..7c9130b 100644
> --- a/pym/_emerge/depgraph.py
> +++ b/pym/_emerge/depgraph.py
> @@ -6062,7 +6062,8 @@ class depgraph(object):
>
> identical_binary
> = True break
>
> - if not
> identical_binary and pkg.built:
> + if (not
> identical_binary and pkg.built and
> +
> (use_ebuild_visibility or matched_packages)): # If the ebuild no
> longer exists or it's # keywords have been dropped, reject built
> #
> instances (installed or binary). diff --git
> a/pym/portage/tests/resolver/test_binary_pkg_ebuild_visibility.py
> b/pym/portage/tests/resolver/test_binary_pkg_ebuild_visibility.py
> index ea65abd..0d01d06 100644 ---
> a/pym/portage/tests/resolver/test_binary_pkg_ebuild_visibility.py +++
> b/pym/portage/tests/resolver/test_binary_pkg_ebuild_visibility.py @@
> -104,6 +104,32 @@ class BinaryPkgEbuildVisibilityTestCase(TestCase):
> '[binary]app-misc/foo-3', ], ),
> +
> + # The default behavior is to enforce ebuild
> visibility as
> + # long as a visible package is available to
> satisfy the
> + # current atom. In the following test case,
> ebuild visibility
> + # is ignored in order to satisfy the
> =app-misc/foo-3 atom.
> + ResolverPlaygroundTestCase(
> + ["=app-misc/foo-3"],
> + options = {
> + "--usepkg": True,
> + },
> + success = True,
> + mergelist = [
> + '[binary]app-misc/foo-3',
> + ],
> + ),
> +
> + # Verify that --use-ebuild-visibility works
> with --usepkg
> + # when no other visible package is available.
> + ResolverPlaygroundTestCase(
> + ["=app-misc/foo-3"],
> + options = {
> + "--use-ebuild-visibility":
> "y",
> + "--usepkg": True,
> + },
> + success = False,
> + ),
> )
>
> playground = ResolverPlayground(binpkgs=binpkgs,
> ebuilds=ebuilds,
looks good
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] emerge: fix --use-ebuild-visibility to reject binary packages (bug 612960)
On Fri, 17 Mar 2017 20:48:22 -0700 Zac Medico <zmed...@gentoo.org> wrote: > Fix the --use-ebuild-visibility option to reject binary packages for > which ebuilds are either masked or unavailable. The included test case > fails with out this fix. > > X-Gentoo-bug: 612960 > X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=612960 > --- > pym/_emerge/depgraph.py| 32 +++- > .../resolver/test_binary_pkg_ebuild_visibility.py | 94 > ++ 2 files changed, 123 insertions(+), 3 > deletions(-) create mode 100644 Looks fine -- Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] depgraph: avoid missed update with slot operator and circ dep (bug 612874)
# not have a dependency on clang, the
> upgrade from llvm:0
> + # to llvm:4 makes the installed
> sys-devel/clang-3.9.1-r100
> + # instance eligible for removal by emerge
> --depclean, which
> + # explains why clang does not appear in the
> mergelist.
> + ResolverPlaygroundTestCase(
> + ["@world"],
> + options = {"--update": True,
> "--deep": True},
> + success = True,
> + ambiguous_merge_order = True,
> + mergelist = [
> + 'sys-devel/llvm-4.0.0',
> + (
> + 'media-libs/mesa-17.0.1',
> +
> '[uninstall]sys-devel/llvm-3.9.1',
> + '!sys-devel/llvm:0',
> + )
> + ],
> + ),
> +
> + )
> +
> + playground = ResolverPlayground(ebuilds=ebuilds,
> + installed=installed, world=world)
> + try:
> + for test_case in test_cases:
> + playground.run_TestCase(test_case)
> +
> self.assertEqual(test_case.test_success, True,
> + test_case.fail_msg)
> + finally:
> + playground.cleanup()
> diff --git a/pym/portage/util/digraph.py b/pym/portage/util/digraph.py
> index 4a9cb43..b6be0c9 100644
> --- a/pym/portage/util/digraph.py
> +++ b/pym/portage/util/digraph.py
> @@ -93,6 +93,12 @@ class digraph(object):
> del self.nodes[node]
> self.order = order
>
> + def has_edge(self, child, parent):
> + """
> + Return True if the given edge exists.
> + """
> + return child in self.nodes[parent][0]
> +
> def remove_edge(self, child, parent):
> """
> Remove edge in the direction from child to parent.
> Note that it is
Looks good
--
Brian Dolbec
Re: [gentoo-portage-dev] OT: Screen bragging. Was: [PROPOSAL] Don't split user visible messages across multiple lines
On Fri, 17 Mar 2017 06:58:23 + (UTC) Duncan <1i5t5.dun...@cox.net> wrote: > Brian Dolbec posted on Thu, 16 Mar 2017 01:08:30 -0700 as excerpted: > > >> > We could also increase the max. line length to something like > >> > 120 or 130. > >> > >> I think more people should chime in on that. I use vertical splits > >> for the screen when coding, and 120 characters is too long for me, > >> but if the preferred width ends up changing to 120 or 130 I can > >> work with it. > >> > >> > > You need to get some large 4K monitors... love them :D I treated > > myself to two 28 inch ones during boxing week sales. > > My aging eyes love them :) They are so much better than my old 24 > > inch 1080p monitors. Those were getting tired/starting to loos > > clarity along with my eyes working at them all week long. I now > > work with larger fonts which are still physically smaller than my > > old monitors, but so much clearer. My eyes don't get nearly so > > tired as they did with my other monitors. > > > > ;) > > Posting resistance failing... > > Try a 65-inch 4K with a 48-inch 1080p (now the older monitor, often > running youtube full-screen) off to the side. =:^) > > (They're actually TVs used as monitors via the HDMI input, no actual > TV hooked up. Above about 32-inch, TVs tend to be cheaper than > stand-alone monitors and of course they're the same 4K high or > full-hd lower resolution these days.) > > Six 1280x1080 working windows three wide by two stacked on the 65" > 4k, still set for 96 dpi standard, FreeMono Bold 9.0 in my konsole > windows, yields: > > $ echo $COLUMNS x $LINES > 179 x 78 > > And that's six of those on the 65" 4K, PLUS the full-screen 1080p > youtube or whatever window on the 48". =:^) > > This is the first time I can honestly say I have enough screen space > that most of the time I'm not actively using it all. =:^) > Yeah, I've seen lots of those types of setups. Just a few years ago when I was an A/C mechanic for my day job, I serviced A/C systems at numerous business's trading floors, where many stations ran 2 or 3 pc's mostly to house/power the video cards to drive 4,6, sometimes 8 monitors in a dual high setups, then numerous big screens overhead in various places around the floor... I don't think I could handle the large ones like you have, my eyes haven't got the focal range for that anymore. As it is, I recently had to pull my monitors closer about 8 inches because my eyes were getting too tired by the end of the week. They were just too close to the limits of my eyes and glasses. I do need to get a decent video card to drive these new 4K's, then maybe I'll think about re-connecting the old 24 inch 1080p's back up to the old card too ;) But if I really needed more screen space, I'd pick up 2 more of these 4k's, 2 video cards and spin them vertical ;) That could make for a great driving/flight simulator :D -- Brian Dolbec
Re: [gentoo-portage-dev] [PROPOSAL] Don't split user visible messages across multiple lines
On Thu, 16 Mar 2017 09:43:08 +0200
Alexandru Elisei <alexandru.eli...@gmail.com> wrote:
> On Thu, Mar 16, 2017 at 12:32 AM, Brian Dolbec <dol...@gentoo.org>
> wrote:
> > That could be pretty hard to do for all messages.
> > Especially messages with embedded data
> >
> > eg: "%s is missing %s required use flag..." % ('sys-apps/foo',
> > 'bar')
>
> For that case we could use:
>
> "%s is missing %s required use flag..." % \
> % ('sys-apps/foo', 'bar')
Yes, but that is not what I meant. When you are searching the code for
a message.
the message in your terminal would read
"sys-apps/foo is missing bar required use flag..."
It may not be obvious to people to break up the text to search into 2
strings. "is missing" and "required use flag..." trying one, failing
that try the other to find the code location. Then there is the
problem of translations moving/re-arranging the text to suit the
language (minor I know, but still a factor to consider).
> > I know I don't always enforce the line length for a few characters,
> > also when clarity is more important than line length.
>
> I totally agree with that.
>
> > We could also increase the max. line length to something like 120
> > or 130.
>
> I think more people should chime in on that. I use vertical splits for
> the screen when coding, and 120 characters is too long for me, but if
> the preferred width ends up changing to 120 or 130 I can work with it.
>
You need to get some large 4K monitors... love them :D
I treated myself to two 28 inch ones during boxing week sales.
My aging eyes love them :) They are so much better than my old 24 inch
1080p monitors. Those were getting tired/starting to loos clarity along
with my eyes working at them all week long. I now work with larger
fonts which are still physically smaller than my old monitors, but
so much clearer. My eyes don't get nearly so tired as they did
with my other monitors.
;)
My work has a 130 col limit. With an editor at that,
plus a wide, open files pane on the left of the editor window, I still
have another large terminal window open to it's right with some bare
screen real-estate patches and borders... ;) Not to mention nearly 100
lines of code view-able in the editor.
--
Brian Dolbec
Re: [gentoo-portage-dev] [PROPOSAL] Don't split user visible messages across multiple lines
On Mon, 27 Feb 2017 20:33:28 +0200
Alexandru Elisei <alexandru.eli...@gmail.com> wrote:
> I was working on emerge --sync and my test repo configuration
> generated several warning messages. Finding the exact location where
> the warnings was generated was surprisingly difficult because the
> message string was split across three lines due to the 80 character
> line limit and I had to grep for different patterns until I found one
> that matched part of a line.
>
> I propose that messages that are visible to the user never be split
> across multiple lines. This is also the coding style convention used
> by the linux kernel:
> https://www.kernel.org/doc/html/latest/process/coding-style.html#breaking-long-lines-and-strings
>
> If accepted, this could go in the DEVELOPING file.
>
That could be pretty hard to do for all messages.
Especially messages with embedded data
eg: "%s is missing %s required use flag..." % ('sys-apps/foo', 'bar')
I know I don't always enforce the line length for a few characters,
also when clarity is more important than line length.
We could also increase the max. line length to something like 120 or 130.
--
Brian Dolbec
