Re: [gentoo-dev] Deprecating repoman
On Fri, 11 Mar 2022 10:25:19 -0800 Alec Warner wrote: > On Fri, Mar 11, 2022 at 9:14 AM Peter Stuge wrote: > > > > Matt Turner wrote: > > > repoman is inferior to other tooling mentioned. The other tooling > > > is actually run in CI. > > > > The problem seems to be that CI is running something other than > > developers run, not the other way around. > > > > > > > Developers should get the same warnings locally as in CI. > > > > I agree. And developers and their tools should not have to bend to > > whatever CI does, I think the other way around makes more sense. > > > > > > CI doesn't use repoman because of performance issues. > > I think the problem is that making committers use the CI tool is > technically a fairly straightforward change; while everything you > discuss further in the thread is not; because it implies people will > work on improving tools that have shown little or no progress on > improving in the 15 years I've been in Gentoo. > Somewhat incorrect here. I did the majority of the re-write to make the code maintainable, extensible not that long ago. It even has the ability to be repository configurable and include the ability for custom repository plugin checks to be added and run. The CI system was being developed at the same time using pkgcheck. See my other reply to WilliamH for more detail about that and fundamental speed differences. > > > > What if pkgcore evolves to provide a portage-compatible API? > > No API is planned and no one is working on it. > I and a few others did some work to ensure pkgcore had some usable api's from early in it's development. But pkgcore is so different from portage code, it was difficult for early to mediocre python devs to wrap their head around. There was even work to make porthole be able to use pkgcore as it's backend package manager, but it was never fully utilized due to more work needed on pkgcore for some functionality. > > > > Then CI could use repoman without performance problems, and > > developers would also enjoy improved performance, without spending > > time on replacing tooling which already works for them. > No, pkgcore and it's QA tool pkgcheck are designed to work together. Repoman is designed to work with portage base code. Checks can be designed to work on either, but not both. The missing checks that the CI does can be added to repoman, but the primary dev(s) creating them on the CI don't recreate them for repoman. They want to work on pkgcheck. Repoman code will NEVER be as fast as pkgcheck due to the legacy portage code structure it relies on. Pkgcore was a re-design from the ground up to to improve upon the shortcomings of the original portage structure. This resulted in vast speed improvements and reduction in lines of code to do the same functionality. Portage (due to Zac) has made vast improvements in structure, but still requires a ton more changes to get to where it should be. > The benefit of getting people to change their behavior is that the > benefits (less breakage, better tooling) are achieved now; as opposed > to in some hypothetical future where repoman is improved. > Note that we have been waiting for 'improved repoman' for about 8 > years; so..I'm not willing to bet on it when we have better tooling > that works today and the primary complaint is that...what exactly? > > The new workflow with pkgcheck was announced at the end of 2019: > https://blogs.gentoo.org/mgorny/2019/12/12/a-better-ebuild-workflow-with-pure-git-and-pkgcheck > > It's been 2 years, I think we can bring everyone into the fold here. > > > > > Looking into the future then maybe portage could even come to use > > pkgcore for the low-level things that pkgcore does, then even users > > could enjoy improved performance. > > Many things could happen but again, if you talk to people who work on > pkgcore; there is no concrete plan for this. So I don't see why we are > betting on it happening. > > If you read radhermit's blog: > https://pkgcraft.github.io/posts/timesink-chronicles/ you can get one > take on the project and why it struggled. > > -A > > > > > > > Right? > > > > //Peter > > >
Re: [gentoo-dev] Deprecating repoman
On Thu, 10 Mar 2022 12:07:40 -0600 William Hubbs wrote: > On Thu, Mar 10, 2022 at 09:29:59AM -0800, Matt Turner wrote: > > On Wed, Mar 9, 2022 at 11:09 PM Joonas Niilola > > wrote: > > > > > > On 9.3.2022 23.00, Matt Turner wrote: > > > > I'd like to deprecate and ultimately remove repoman. I believe > > > > that dev-util/pkgcheck and pkgcommit (from > > > > app-portage/mgorny-dev-scripts) are far superior replacements, > > > > and it makes sense to have people using the same tool and > > > > seeing the same warnings as in the CI. > > > > > > > > Are there any useful checks or behaviors of repoman that are > > > > missing from pkgcheck and pkgcommit? > > > > > > > > Thanks, > > > > Matt > > > > > > > > > > I still fail to see the "why" in here. Repoman is better than > > > pure 'git commit' that I know some people still like to use, and > > > as long as it's kept maintained. > > > > repoman is inferior to other tooling mentioned. The other tooling is > > actually run in CI. Developers should get the same warnings locally > > as in CI. Restated another way: I'm tired of telling people to stop > > using repoman or "pkgcheck would have caught that". > > I am going to nit-pick here, but pkgcheck pulls in pkgcore still. As > far asI know, pkgcore was meant to be a portage-like package manager, > but it isn't maintained. So, can we break that dependency before we > make pkgcheck the official tool for qa checks? I would rather not have > pkgcore landing on everyone's systems unless it is usable. If I am > wrong about pkgcore, please correct me and I'll be quiet, but if not > let's make pkgcheck independent from it before we deprecate repoman. > > Thanks, > > William William. pkgcheck uses pkgcore base code as it's library for accessing and evaluating repositories and ebuilds just like repoman uses portage api's to do the same. This is why pkgcheck is primarily faster than repoman. pkgcore was a re-design from the ground up of how to handle repositories and ebuilds within them, with efficiency and speed in mind. Unfortunately, pkgcore never gained enough developer support to to keep up with EAPI changes and the miriad of portage options that kept expanding. So pkgcore never ended up replacing portage as our main package manager. This is one reason why Brian eventually gave up on it and subsequently left gentoo. While I put a lot of effort into the re-write of repoman so that the code is maintainable, more easily extendable. It will never be as fast as pkgcheck due to the legacy portage code it is based on. Yes, repoman could be updated to do more checks and catch everything that pkgcheck does. It can be made to do many of the checks in parrallel, but it will never be as efficient as pkgcheck due to the underlying portage code it uses. I largely stopped working on repoman/gentoo due to the near constant bitching about nearly everything I did from another (ahem) gentoo developer which shall remain un-named. My only concern about the possible deprecation of repoman, is the fact that is is based on the current official and primary package manager code used in gentoo. While pkgcheck is not. For that reason, changes made to the core code is mostly automatically carried over to repoman without need to change repoman as well. Ideally, development should be moved to pkgcore and portage be put into maintenance mode only. With pkgcore taking over as the primary package manager for gentoo. But Zac is just too damn efficient at updating/fixing portage bugs! As well as adding features and options... (too damn many to remember half of them now)
Re: [gentoo-portage-dev] [PATCH] portage.getpid: call os.getpid() lazily
On Sat, 30 Jan 2021 04:59:32 -0800 Zac Medico wrote: > Call os.getpid() lazily, which eliminates getpid calls when possible > after os.fork() in the portage.process module. > > Bug: https://bugs.gentoo.org/767913 > Signed-off-by: Zac Medico > --- > lib/portage/__init__.py | 4 +++- > 1 file changed, 3 insertions(+), 1 deletion(-) > > diff --git a/lib/portage/__init__.py b/lib/portage/__init__.py > index 3c9f78497..24c9d8b89 100644 > --- a/lib/portage/__init__.py > +++ b/lib/portage/__init__.py > @@ -375,7 +375,7 @@ _sync_mode = False > class _ForkWatcher: > @staticmethod > def hook(_ForkWatcher): > - _ForkWatcher.current_pid = _os.getpid() > + _ForkWatcher.current_pid = None > # Force instantiation of a new event loop policy as > a workaround # for https://bugs.python.org/issue22087. > asyncio.set_event_loop_policy(None) > @@ -388,6 +388,8 @@ def getpid(): > """ > Cached version of os.getpid(). ForkProcess updates the cache. > """ > + if _ForkWatcher.current_pid is None: > + _ForkWatcher.current_pid = _os.getpid() > return _ForkWatcher.current_pid > > def _get_stdin(): looks good
Re: [gentoo-portage-dev] [PATCH] emerge: disable --autounmask-license by default
On Sat, 30 Jan 2021 03:49:41 -0800
Zac Medico wrote:
> Disable --autounmask-license by default, in order to limit user
> exposure to risks associated with package.license changes.
> The changes that this option suggests are only intended to be
> accepted when a user has made a conscious decision to accept
> the corresponding license(s). Creation of package.license
> changes introduces a risk that users may erroneously accept the
> changes due to some kind of accident or misunderstanding,
> rather than due to conscious decisions about licenses.
> These risks provide motivation to disable --autounmask-license
> by default. The --autounmask-use option will remain as the
> only autounmask option that is still enabled by default.
>
> The unit tests demonstrate interactions between --autounmask
> and --autounmask-license options. The --autounmask option
> enables --autounmask-license unless --autounmask-license=n
> has been specified. If --autounmask=n is used to disable
> autounmask, then --autounmask-license=y has no effect.
>
> Bug: https://bugs.gentoo.org/766773
> Signed-off-by: Zac Medico
> ---
makes sense, code looks good
> lib/_emerge/create_depgraph_params.py | 8 +++---
> lib/portage/tests/resolver/test_autounmask.py | 25
> +-- man/emerge.1 |
> 11 +++- 3 files changed, 31 insertions(+), 13 deletions(-)
>
> diff --git a/lib/_emerge/create_depgraph_params.py
> b/lib/_emerge/create_depgraph_params.py index 0d0e07b9c..25dd2a1b4
> 100644 --- a/lib/_emerge/create_depgraph_params.py
> +++ b/lib/_emerge/create_depgraph_params.py
> @@ -1,4 +1,4 @@
> -# Copyright 1999-2018 Gentoo Foundation
> +# Copyright 1999-2021 Gentoo Authors
> # Distributed under the terms of the GNU General Public License v2
>
> import logging
> @@ -45,7 +45,7 @@ def create_depgraph_params(myopts, myaction):
> autounmask_keep_masks = myopts.get("--autounmask-keep-masks")
>
> autounmask = myopts.get("--autounmask")
> - autounmask_license = myopts.get('--autounmask-license')
> + autounmask_license = myopts.get('--autounmask-license', 'y'
> if autounmask is True else 'n') autounmask_use =
> myopts.get('--autounmask-use') if autounmask == 'n':
> autounmask = False
> @@ -53,7 +53,7 @@ def create_depgraph_params(myopts, myaction):
> if autounmask is None:
> if autounmask_use in (None, 'y'):
> autounmask = True
> - elif autounmask_license in (None, 'y'):
> + if autounmask_license in ('y',):
> autounmask = True
>
> # Do not enable package.accept_keywords or
> package.mask @@ -67,7 +67,7 @@ def create_depgraph_params(myopts,
> myaction):
> myparams['autounmask'] = autounmask
> myparams['autounmask_keep_use'] = True if autounmask_use ==
> 'n' else False
> - myparams['autounmask_keep_license'] = True if
> autounmask_license == 'n' else False
> + myparams['autounmask_keep_license'] = False if
> autounmask_license == 'y' else True
> myparams['autounmask_keep_keywords'] = False if
> autounmask_keep_keywords in (None, 'n') else True
> myparams['autounmask_keep_masks'] = False if autounmask_keep_masks in
> (None, 'n') else True diff --git
> a/lib/portage/tests/resolver/test_autounmask.py
> b/lib/portage/tests/resolver/test_autounmask.py index
> a3bf0ff94..86ae4bbf6 100644 ---
> a/lib/portage/tests/resolver/test_autounmask.py +++
> b/lib/portage/tests/resolver/test_autounmask.py @@ -1,4 +1,4 @@ -#
> Copyright 2010-2019 Gentoo Authors +# Copyright 2010-2021 Gentoo
> Authors # Distributed under the terms of the GNU General Public
> License v2 from portage.tests import TestCase
> @@ -440,13 +440,34 @@ class AutounmaskTestCase(TestCase):
> mergelist=["dev-libs/A-1"],
> license_changes={
> "dev-libs/A-1": set(["TEST"]) }),
> - # Test default --autounmask-license
> + # Test that --autounmask enables
> --autounmask-license ResolverPlaygroundTestCase(
> ["=dev-libs/A-1"],
> + options={"--autounmask":
> True}, success=False,
> mergelist=["dev-libs/A-1"],
> license_changes={
> "dev-libs/A-1": set(["TEST"]) }),
> + # Test that --autounmask-license is
> not enabled by default
> + ResolverPlaygroundTestCase(
> + ["=dev-libs/A-1"],
> + success=False,
> + ),
> +
> + # Test that --autounmask does not
> override --autounmask-license=n
> + ResolverPlaygroundTestCase(
> +
Re: [gentoo-portage-dev] [PATCH] emaint --fix merges: add -y, --yes option
On Sat, 23 Jan 2021 15:38:49 -0800
Zac Medico wrote:
> Since the emaint --fix merges uses emerge --ask, add
> a -y, --yes option to use --ask=n instead.
Looks good to me
>
> Bug: https://bugs.gentoo.org/766767
> Signed-off-by: Zac Medico
> ---
> lib/portage/emaint/modules/merges/__init__.py | 14 --
> lib/portage/emaint/modules/merges/merges.py | 11 +++
> man/emaint.1 | 6 +-
> 3 files changed, 24 insertions(+), 7 deletions(-)
>
> diff --git a/lib/portage/emaint/modules/merges/__init__.py
> b/lib/portage/emaint/modules/merges/__init__.py index
> 89aa758a0..449f39dce 100644 ---
> a/lib/portage/emaint/modules/merges/__init__.py +++
> b/lib/portage/emaint/modules/merges/__init__.py @@ -1,4 +1,4 @@
> -# Copyright 2005-2014 Gentoo Foundation
> +# Copyright 2005-2021 Gentoo Authors
> # Distributed under the terms of the GNU General Public License v2
>
> doc = """Scan for failed merges and fix them."""
> @@ -26,7 +26,17 @@ module_spec = {
> 'action': 'store_true',
> 'func': 'purge'
> }
> - }
> + },
> + 'opt_desc': {
> + 'yes': {
> + "short": "-y",
> + "long": "--yes",
> + "help": ("(merges submodule
> only): Do not prompt for "
> + "emerge
> invocations"),
> + "action": "store_true",
> + "dest": "yes",
> + }
> + },
> }
> }
> }
> diff --git a/lib/portage/emaint/modules/merges/merges.py
> b/lib/portage/emaint/modules/merges/merges.py index
> 775dc59d2..d60916f1e 100644 ---
> a/lib/portage/emaint/modules/merges/merges.py +++
> b/lib/portage/emaint/modules/merges/merges.py @@ -1,4 +1,4 @@
> -# Copyright 2005-2020 Gentoo Authors
> +# Copyright 2005-2021 Gentoo Authors
> # Distributed under the terms of the GNU General Public License v2
>
> import portage
> @@ -186,7 +186,7 @@ class MergesHandler:
> pkg_atoms.add(pkg_atom)
>
>
> - def _emerge_pkg_atoms(self, module_output, pkg_atoms):
> + def _emerge_pkg_atoms(self, module_output, pkg_atoms,
> yes=False): """
> Emerge the specified packages atoms.
>
> @@ -194,6 +194,8 @@ class MergesHandler:
> @type module_output: Class
> @param pkg_atoms: packages atoms to emerge
> @type pkg_atoms: set
> + @param yes: do not prompt for emerge invocations
> + @type yes: bool
> @rtype: list
> @return: List of results
> """
> @@ -206,7 +208,7 @@ class MergesHandler:
> portage._python_interpreter,
> '-b',
> os.path.join(EPREFIX or '/', 'usr', 'bin',
> 'emerge'),
> - '--ask',
> + '--ask=n' if yes else '--ask',
> '--quiet',
> '--oneshot',
> '--complete-graph=y'
> @@ -265,7 +267,8 @@ class MergesHandler:
> errors.append(', '.join(sorted(failed_pkgs)))
> return (False, errors)
> self._remove_failed_dirs(failed_pkgs)
> - results = self._emerge_pkg_atoms(module_output,
> pkg_atoms)
> + results = self._emerge_pkg_atoms(module_output,
> pkg_atoms,
> + yes=kwargs.get('options', {}).get("yes",
> False)) # list any new failed merges
> for pkg in sorted(self._scan()):
> results.append("'%s' still found as a failed
> merge." % pkg) diff --git a/man/emaint.1 b/man/emaint.1
> index d244756e9..c9f8ab939 100644
> --- a/man/emaint.1
> +++ b/man/emaint.1
> @@ -1,4 +1,4 @@
> -.TH "EMAINT" "1" "Jan 2017" "Portage VERSION" "Portage"
> +.TH "EMAINT" "1" "Jan 2021" "Portage VERSION" "Portage"
> .SH NAME
> emaint \- performs package management related system health checks
> and maintenance .SH SYNOPSIS
> @@ -82,6 +82,10 @@ OPTION
> .BR \-t \ \fINUM\fR,\ \-\-time \ \fINUM\fR
> Changes the minimum age \fINUM\fR (in days) of the logs to be listed
> or deleted.
> +.SH OPTIONS merges command only
> +.TP
> +.BR \-y ", " \-\-yes
> +Do not prompt for emerge invocations.
> .SH OPTIONS sync command only
> .TP
> .BR \-a ", " \-\-auto
Re: [gentoo-portage-dev] [PATCH] repoman: Add --jobs and --load-average options (bug 448462)
On Sun, 16 Aug 2020 20:26:56 -0700
Zac Medico wrote:
> Add --jobs and --load-average options which allow dependency checks
> for multiple profiles to run in parallel. The increase in performance
> is non-linear for the number of jobs, but it can be worthwhile
> (I measured a 35% decrease in time when running 'repoman -j8 full'
> on sys-apps/portage). For the -j1 case (default), all dependency
> checks run in the main process as usual, so there is no significant
> performance penalty for the default case.
>
> Bug: https://bugs.gentoo.org/448462
> Signed-off-by: Zac Medico
> ---
> repoman/lib/repoman/argparser.py | 9 ++
> .../repoman/modules/scan/depend/profile.py| 117
> +++--- repoman/man/repoman.1 |
> 9 +- 3 files changed, 116 insertions(+), 19 deletions(-)
>
> diff --git a/repoman/lib/repoman/argparser.py
> b/repoman/lib/repoman/argparser.py index 670a0e91d..6d545ccca 100644
> --- a/repoman/lib/repoman/argparser.py
> +++ b/repoman/lib/repoman/argparser.py
> @@ -199,6 +199,15 @@ def parse_args(argv, repoman_default_opts):
> '--output-style', dest='output_style',
> choices=output_keys, help='select output type', default='default')
>
> + parser.add_argument(
> + '-j', '--jobs', dest='jobs', action='store',
> type=int, default=1,
> + help='Specifies the number of jobs (processes) to
> run simultaneously.') +
> + parser.add_argument(
> + '-l', '--load-average', dest='load_average',
> action='store', type=float, default=None,
> + help='Specifies that no new jobs (processes) should
> be started if there are others '
> + 'jobs running and the load average is at
> least load (a floating-point number).') +
> parser.add_argument(
> '--mode', dest='mode', choices=mode_keys,
> help='specify which mode repoman will run in
> (default=full)') diff --git
> a/repoman/lib/repoman/modules/scan/depend/profile.py
> b/repoman/lib/repoman/modules/scan/depend/profile.py index
> 39d8b550c..1eb69422a 100644 ---
> a/repoman/lib/repoman/modules/scan/depend/profile.py +++
> b/repoman/lib/repoman/modules/scan/depend/profile.py @@ -2,7 +2,9 @@
>
> import copy
> +import functools
> import os
> +import types
> from pprint import pformat
>
> from _emerge.Package import Package
> @@ -15,6 +17,10 @@ from repoman.modules.scan.depend._gen_arches
> import _gen_arches from portage.dep import Atom
> from portage.package.ebuild.profile_iuse import iter_iuse_vars
> from portage.util import getconfig
> +from portage.util.futures import asyncio
> +from portage.util.futures.compat_coroutine import coroutine,
> coroutine_return +from portage.util.futures.executor.fork import
> ForkExecutor +from portage.util.futures.iter_completed import
> async_iter_completed
>
> def sort_key(item):
> @@ -58,16 +64,14 @@ class ProfileDependsChecks(ScanBase):
> def check(self, **kwargs):
> '''Perform profile dependant dependency checks
>
> - @param arches:
> @param pkg: Package in which we check (object).
> @param ebuild: Ebuild which we check (object).
> - @param baddepsyntax: boolean
> - @param unknown_pkgs: set of tuples (type,
> atom.unevaluated_atom) @returns: dictionary
> '''
> ebuild = kwargs.get('ebuild').get()
> pkg = kwargs.get('pkg').get()
> - unknown_pkgs, baddepsyntax = _depend_checks(
> +
> + ebuild.unknown_pkgs, ebuild.baddepsyntax =
> _depend_checks( ebuild, pkg, self.portdb, self.qatracker,
> self.repo_metadata, self.repo_settings.qadata)
>
> @@ -90,8 +94,64 @@ class ProfileDependsChecks(ScanBase):
> relevant_profiles.append((keyword,
> groups, prof))
> relevant_profiles.sort(key=sort_key)
> + ebuild.relevant_profiles = relevant_profiles
> +
> + if self.options.jobs <= 1:
> + for task in self._iter_tasks(None, None,
> ebuild, pkg):
> + task, results = task
> + for result in results:
> + self._check_result(task,
> result) +
> + loop = asyncio._wrap_loop()
> + loop.run_until_complete(self._async_check(loop=loop,
> **kwargs)) +
> + return False
> +
> + @coroutine
> + def _async_check(self, loop=None, **kwargs):
> + '''Perform async profile dependant dependency checks
> +
> + @param arches:
> + @param pkg: Package in which we check (object).
> + @param ebuild: Ebuild which we check (object).
> + @param baddepsyntax: boolean
> + @param unknown_pkgs: set of tuples (type,
> atom.unevaluated_atom)
> + @returns: dictionary
> + '''
> + loop = asyncio._wrap_loop(loop)
> +
Re: [gentoo-portage-dev] [PATCH 0/2] emerge: Add short -l option for --load-average (bug 699256)
On Sat, 15 Aug 2020 12:26:42 -0700 Zac Medico wrote: > Add a short -l option for --load-average just like make(1) has. > > Bug: https://bugs.gentoo.org/699256 > Signed-off-by: Zac Medico > > Zac Medico (2): > emerge: Remove deprecated --changelog option (bug 699256) > emerge: Add short -l option for --load-average (bug 699256) > > lib/_emerge/actions.py | 5 -- > lib/_emerge/main.py| 5 +- > lib/_emerge/resolver/output.py | 32 +--- > lib/_emerge/resolver/output_helpers.py | 106 > - man/emerge.1 | > 8 +- 5 files changed, 5 insertions(+), 151 deletions(-) > looks good :)
Re: [gentoo-dev] Re: Python 2.7 cleanup: plan B
On Tue, 11 Aug 2020 23:41:33 + (UTC)
"Thomas Mueller" wrote:
> > Hi, everyone.
>
> > TL;DR: we might keep Python 2.7 supported as a build-time dependency
> > of a few packages as necessary, while removing the eclass support
> > for installing packages for py2.7.
>
>
> > As I've mentioned earlier, the plan is to get rid of Python 2.7
> > target support at the beginning of 2021. The plan was to last rite
> > all remaining packages failing to support Python 3 at 2021-01-01,
> > and remove the eclass support on 2021-02-15. At the same time, the
> > Python interpreter was going to stay around for as long as
> > necessary.
>
> > I've also mentioned that there is a high risk that this will not be
> > possible because of a few large entities ignoring the problem
> > and failing to port their build system scripts away from Python 2.
> > We can't really last rite all major web browsers, and postponing
> > the deadline indefinitely is not a good solution either.
>
> > Therefore, I advise the following plan B: if it is impossible to
> > remove Python 2.7 support from packages entirely, the support for
> > installing Python packages for Python 2.7 will be removed.
> > However, there will be exemptions granted for build-time
> > dependencies on the Python interpreter to keep things working, for
> > as long as the interpreter itself is going to stay.
>
> > The candidates for exemptions are pypy/pypy3 (CPython 2.7 is needed
> > for bootstrap on new platforms), Mozilla products, WebKit and
> > WebKit-based browsers.
>
> > Best regards,
> > Micha=C5=82 G=C3=B3rny
>
> I believe net-print/hplip also depends on Python 2.7. This is
> print/hplip in NetBSD pkgsrc and FreeBSD ports.
>
> Or did they fix that recently?
>
> Possibly hplip is mainly for older HP printers, not sure about what's
> going on with HP laser printers more recently.
>
> Maybe that's why there has been reluctance to fix the Python 2.7
> dependency?
>
> I have such an HP printer (LaserJet M1212nf MFP), and my experience
> dissuades me from ordering anything further from HP.
>
> I think most everybody involved with open-source would agree that
> proprietary binary plugins suck.
>
> Tom
>
>
latest hplip has: PYTHON_COMPAT=( python3_{6,7,8} )
and older one was just missing py3.8
Re: [gentoo-portage-dev] [PATCH 1/2] lib/portage/repository/config.py: fix trailing comma
On Tue, 4 Aug 2020 21:56:08 -0400 Aaron Bauman wrote: > On Tue, Aug 04, 2020 at 09:43:03PM -0400, Aaron Bauman wrote: > > * This fixes the one instance of a trailing comma in the repo which > > actually causes a tuple to be created. > > > > Signed-off-by: Aaron Bauman > > --- > > lib/portage/repository/config.py | 2 +- > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > diff --git a/lib/portage/repository/config.py > > b/lib/portage/repository/config.py index 0f3e582f8..f4dd834b5 100644 > > --- a/lib/portage/repository/config.py > > +++ b/lib/portage/repository/config.py > > @@ -871,7 +871,7 @@ class RepoConfigLoader: > > continue > > if repo.masters is None: > > if self.mainRepo() and repo_name > > != self.mainRepo().name: > > - repo.masters = > > self.mainRepo(), > > + repo.masters = > > self.mainRepo() else: > > repo.masters = () > > else: > > -- > > 2.28.0 > > > > > > Please disregard this patch series. > Please don't enable this check. Pylint is stupid to enforce this one. When adding another item to the list or dictionary, it is better to have the tailing comma already in place. Git blame doesn't work if you have to add the comma to extend the list, etc.. It also adds unnecessary churn for something that does not affect operation in the slightest. Is one of the bad ones pylint wants to enforce.
Re: [gentoo-portage-dev] [PATCH] cnf/repo.postsync.d/example: add egencache --update-pkg-desc-index (bug 735626)
On Sun, 2 Aug 2020 16:38:06 -0700
Zac Medico wrote:
> Add an egencache --update-pkg-desc-index example for users
> of app-portage/esearch to migrate to.
>
> Bug: https://bugs.gentoo.org/735626
> Signed-off-by: Zac Medico
> ---
> cnf/repo.postsync.d/example | 15 ++-
> 1 file changed, 14 insertions(+), 1 deletion(-)
>
> diff --git a/cnf/repo.postsync.d/example b/cnf/repo.postsync.d/example
> index 533bf719c..6c349ab3c 100644
> --- a/cnf/repo.postsync.d/example
> +++ b/cnf/repo.postsync.d/example
> @@ -1,4 +1,4 @@
> -#!/bin/sh
> +#!/bin/bash
> # Example /etc/portage/repo.postsync.d script. Make it executable
> (chmod +x) for # Portage to process it.
> #
> @@ -45,6 +45,19 @@ if [ -n "${repository_name}" ]; then
> ret=1
> fi
> fi
> +
> + # Regenerate the metadata/pkg_desc_index file if needed.
> It's not
> + # needed for https://gitweb.gentoo.org/repo/sync/gentoo.git
> which
> + # provides a freshly generated copy.
> + if [[ ! -e ${repository_path}/metadata/pkg_desc_index || (
> + -d ${repository_path}/metadata/md5-cache &&
> + -n "$(find "${repository_path}/metadata/md5-cache"
> -type f -newer "${repository_path}/metadata/pkg_desc_index" -print
> -quit)" ) ]]; then
> + if ! egencache --update-pkg-desc-index
> --repo="${repository_name}"
> + then
> + echo "!!! egencache failed!"
> + ret=1
> + fi
> + fi
> fi
>
> # Return explicit status.
is good with me, but my bash skills suck... so if there is a mistake I
wouldn't know ;)
Re: [gentoo-portage-dev] [PATCH] travis.yml: drop python 2.7 (bug 731114)
On Mon, 6 Jul 2020 11:42:06 -0700
Zac Medico wrote:
> It should be pretty safe to drop support for python2.7 at this point.
>
> Bug: https://bugs.gentoo.org/731114
> Signed-off-by: Zac Medico
> ---
> .travis.yml | 1 -
> tox.ini | 6 ++
> 2 files changed, 2 insertions(+), 5 deletions(-)
>
> diff --git a/.travis.yml b/.travis.yml
> index 2132c8c87..d2935fdab 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -1,7 +1,6 @@
> dist: bionic
> language: python
> python:
> -- 2.7
> - 3.6
> - 3.7
> - 3.8
> diff --git a/tox.ini b/tox.ini
> index 79b5b45cb..050a2c455 100644
> --- a/tox.ini
> +++ b/tox.ini
> @@ -1,14 +1,12 @@
> [tox]
> -envlist = py27,py36,py37,py38,py39,pypy3
> +envlist = py36,py37,py38,py39,pypy3
> skipsdist = True
>
> [testenv]
> deps =
> pygost
> pyyaml
> - py27,py36,py37,py38,py39,pypy3: lxml!=4.2.0
> - py27: pyblake2
> - py27: pysha3
> + py36,py37,py38,py39,pypy3: lxml!=4.2.0
> setenv =
> PYTHONPATH={toxinidir}/lib
> commands =
Go for it!
Re: [gentoo-dev] dev-python/rstcheck-3.3.1: Add rstcheck python package (#16399)
On Thu, 25 Jun 2020 23:23:29 +0100 Samuel Bernardo wrote: > Hi Brian > > On 6/25/20 11:18 PM, Brian Dolbec wrote: > > You add yourself as primary maintainer. The proxy maintainers will > > add themselves for the merge to the repo after all review is done. > > This will mean that you will need to maintain the pkg, do the > > version bumps, etc.. The proxy team will help merge the changes to > > the ebuild tree. > > That means that I can submit the ebuild metadata without maintainer? > > Thanks > > yes, you do not need to add a Gentoo maintainer unless asked to.
Re: [gentoo-dev] dev-python/rstcheck-3.3.1: Add rstcheck python package (#16399)
On Thu, 25 Jun 2020 23:11:29 +0100 Samuel Bernardo wrote: > Hi, > > I send this email to ask for your help on selecting the project > maintainer for a new ebuild. > > I created a pull request for the ebuild in subject[1] and the QA > reports complaints about missing project maintainer[2]. What should I > do? > > Thanks > > [1] https://github.com/gentoo/gentoo/pull/16399 > > [2] > https://qa-reports.gentoo.org/output/gentoo-ci/2e4d12bbfa/output.html#dev-python/rstcheck > > You add yourself as primary maintainer. The proxy maintainers will add themselves for the merge to the repo after all review is done. This will mean that you will need to maintain the pkg, do the version bumps, etc.. The proxy team will help merge the changes to the ebuild tree.
Re: [gentoo-portage-dev] [PATCH 2/2] Support PORTAGE_LOG_FILTER_FILE_CMD (bug 709746)
On Fri, 19 Jun 2020 13:39:19 -0700
Zac Medico wrote:
> This variable specifies a command that filters build log output to a
> log file. The plan is to extend this to support a separate filter for
> tty output in the future.
>
> In order to enable the EbuildPhase class to write elog messages to
> the build log with PORTAGE_LOG_FILTER_FILE_CMD support, convert its
> _elog method to a coroutine, and add a SchedulerInterface async_output
> method for it to use.
>
> Use a new BuildLogger class to manage log output (with or without a
> filter command), with compression support provided by PipeLogger.
> BuildLogger has a stdin property which provides access to a writable
> binary file stream (refers to a pipe) that log content is written to.
>
> Bug: https://bugs.gentoo.org/709746
> Signed-off-by: Zac Medico
> ---
> lib/_emerge/AbstractEbuildProcess.py | 3 +-
> lib/_emerge/BinpkgFetcher.py | 3 +-
> lib/_emerge/EbuildFetcher.py | 3 +-
> lib/_emerge/EbuildPhase.py| 47 ++--
> lib/_emerge/SpawnProcess.py | 58 +++---
> lib/portage/dbapi/_MergeProcess.py| 3 +-
> .../ebuild/_config/special_env_vars.py| 8 +-
> lib/portage/util/_async/BuildLogger.py| 109
> ++ lib/portage/util/_async/SchedulerInterface.py |
> 32 - man/make.conf.5 | 7 +-
> 10 files changed, 243 insertions(+), 30 deletions(-)
> create mode 100644 lib/portage/util/_async/BuildLogger.py
>
> diff --git a/lib/_emerge/AbstractEbuildProcess.py
> b/lib/_emerge/AbstractEbuildProcess.py index 1c1955cfe..ae1aae55f
> 100644 --- a/lib/_emerge/AbstractEbuildProcess.py
> +++ b/lib/_emerge/AbstractEbuildProcess.py
> @@ -1,4 +1,4 @@
> -# Copyright 1999-2019 Gentoo Foundation
> +# Copyright 1999-2020 Gentoo Authors
> # Distributed under the terms of the GNU General Public License v2
>
> import errno
> @@ -196,6 +196,7 @@ class AbstractEbuildProcess(SpawnProcess):
> null_fd = os.open('/dev/null', os.O_RDONLY)
> self.fd_pipes[0] = null_fd
>
> + self.log_filter_file =
> self.settings.get('PORTAGE_LOG_FILTER_FILE_CMD') try:
> SpawnProcess._start(self)
> finally:
> diff --git a/lib/_emerge/BinpkgFetcher.py
> b/lib/_emerge/BinpkgFetcher.py index 36d027de3..2e5861cc1 100644
> --- a/lib/_emerge/BinpkgFetcher.py
> +++ b/lib/_emerge/BinpkgFetcher.py
> @@ -1,4 +1,4 @@
> -# Copyright 1999-2018 Gentoo Foundation
> +# Copyright 1999-2020 Gentoo Authors
> # Distributed under the terms of the GNU General Public License v2
>
> import functools
> @@ -158,6 +158,7 @@ class _BinpkgFetcherProcess(SpawnProcess):
> self.env = fetch_env
> if settings.selinux_enabled():
> self._selinux_type =
> settings["PORTAGE_FETCH_T"]
> + self.log_filter_file =
> settings.get('PORTAGE_LOG_FILTER_FILE_CMD') SpawnProcess._start(self)
>
> def _pipe(self, fd_pipes):
> diff --git a/lib/_emerge/EbuildFetcher.py
> b/lib/_emerge/EbuildFetcher.py index 1e40994fb..55349c33c 100644
> --- a/lib/_emerge/EbuildFetcher.py
> +++ b/lib/_emerge/EbuildFetcher.py
> @@ -1,4 +1,4 @@
> -# Copyright 1999-2018 Gentoo Foundation
> +# Copyright 1999-2020 Gentoo Authors
> # Distributed under the terms of the GNU General Public License v2
>
> import copy
> @@ -225,6 +225,7 @@ class _EbuildFetcherProcess(ForkProcess):
> settings["NOCOLOR"] = nocolor
>
> self._settings = settings
> + self.log_filter_file =
> settings.get('PORTAGE_LOG_FILTER_FILE_CMD') ForkProcess._start(self)
>
> # Free settings now since it's no longer needed in
> diff --git a/lib/_emerge/EbuildPhase.py b/lib/_emerge/EbuildPhase.py
> index 477e0ba97..ddb3dc719 100644
> --- a/lib/_emerge/EbuildPhase.py
> +++ b/lib/_emerge/EbuildPhase.py
> @@ -26,6 +26,8 @@ from portage.package.ebuild.prepare_build_dirs
> import (_prepare_workdir, from portage.util.futures.compat_coroutine
> import coroutine from portage.util import writemsg
> from portage.util._async.AsyncTaskFuture import AsyncTaskFuture
> +from portage.util._async.BuildLogger import BuildLogger
> +from portage.util.futures import asyncio
> from portage.util.futures.executor.fork import ForkExecutor
>
> try:
> @@ -69,6 +71,11 @@ class EbuildPhase(CompositeTask):
> _locked_phases = ("setup", "preinst", "postinst", "prerm",
> "postrm")
> def _start(self):
> + future = asyncio.ensure_future(self._async_start(),
> loop=self.scheduler)
> + self._start_task(AsyncTaskFuture(future=future),
> self._async_start_exit) +
> + @coroutine
> + def _async_start(self):
>
> need_builddir = self.phase not in
> EbuildProcess._phases_without_builddir
> @@ -126,7 +133,7 @@ class EbuildPhase(CompositeTask):
> # Force
Re: [gentoo-portage-dev] [PATCH 1/2] PipeLogger: non-blocking write to pipe (bug 709746)
On Fri, 19 Jun 2020 13:39:18 -0700
Zac Medico wrote:
> Add support to write to a non-blocking pipe instead of a
> log file. This is needed for the purposes of bug 709746,
> where PipeLogger will write to a pipe that is drained
> by anoher PipeLogger instance which is running in the same
> process.
>
> Bug: https://bugs.gentoo.org/709746
> Signed-off-by: Zac Medico
> ---
> lib/portage/tests/process/test_PipeLogger.py | 58
> lib/portage/util/_async/PipeLogger.py| 73
> +++- 2 files changed, 115 insertions(+), 16
> deletions(-) create mode 100644
> lib/portage/tests/process/test_PipeLogger.py
>
> diff --git a/lib/portage/tests/process/test_PipeLogger.py
> b/lib/portage/tests/process/test_PipeLogger.py new file mode 100644
> index 0..2bd94cf39
> --- /dev/null
> +++ b/lib/portage/tests/process/test_PipeLogger.py
> @@ -0,0 +1,58 @@
> +# Copyright 2020 Gentoo Authors
> +# Distributed under the terms of the GNU General Public License v2
> +
> +from portage import os
> +from portage.tests import TestCase
> +from portage.util._async.PipeLogger import PipeLogger
> +from portage.util.futures import asyncio
> +from portage.util.futures._asyncio.streams import _reader, _writer
> +from portage.util.futures.compat_coroutine import coroutine,
> coroutine_return +from portage.util.futures.unix_events import
> _set_nonblocking +
> +
> +class PipeLoggerTestCase(TestCase):
> +
> + @coroutine
> + def _testPipeLoggerToPipe(self, test_string, loop=None):
> + """
> + Test PipeLogger writing to a pipe connected to a
> PipeReader.
> + This verifies that PipeLogger does not deadlock when
> writing
> + to a pipe that's drained by a PipeReader running in
> the same
> + process (requires non-blocking write).
> + """
> +
> + input_fd, writer_pipe = os.pipe()
> + _set_nonblocking(writer_pipe)
> + writer_pipe = os.fdopen(writer_pipe, 'wb', 0)
> + writer = asyncio.ensure_future(_writer(writer_pipe,
> test_string.encode('ascii'), loop=loop), loop=loop)
> + writer.add_done_callback(lambda writer:
> writer_pipe.close()) +
> + pr, pw = os.pipe()
> +
> + consumer = PipeLogger(background=True,
> + input_fd=input_fd,
> + log_file_path=os.fdopen(pw, 'wb', 0),
> + scheduler=loop)
> + consumer.start()
> +
> + # Before starting the reader, wait here for a
> moment, in order
> + # to exercise PipeLogger's handling of EAGAIN during
> write.
> + yield asyncio.wait([writer], timeout=0.01)
> +
> + reader = _reader(pr, loop=loop)
> + yield writer
> + content = yield reader
> + yield consumer.async_wait()
> +
> + self.assertEqual(consumer.returncode, os.EX_OK)
> +
> + coroutine_return(content.decode('ascii', 'replace'))
> +
> + def testPipeLogger(self):
> + loop = asyncio._wrap_loop()
> +
> + for x in (1, 2, 5, 6, 7, 8, 2**5, 2**10, 2**12,
> 2**13, 2**14, 2**17, 2**17 + 1):
> + test_string = x * "a"
> + output =
> loop.run_until_complete(self._testPipeLoggerToPipe(test_string,
> loop=loop))
> + self.assertEqual(test_string, output,
> + "x = %s, len(output) = %s" % (x,
> len(output))) diff --git a/lib/portage/util/_async/PipeLogger.py
> b/lib/portage/util/_async/PipeLogger.py index a4258f350..ce8afb846
> 100644 --- a/lib/portage/util/_async/PipeLogger.py
> +++ b/lib/portage/util/_async/PipeLogger.py
> @@ -8,6 +8,10 @@ import sys
>
> import portage
> from portage import os, _encodings, _unicode_encode
> +from portage.util.futures import asyncio
> +from portage.util.futures._asyncio.streams import _writer
> +from portage.util.futures.compat_coroutine import coroutine
> +from portage.util.futures.unix_events import _set_nonblocking
> from _emerge.AbstractPollTask import AbstractPollTask
>
> class PipeLogger(AbstractPollTask):
> @@ -21,13 +25,16 @@ class PipeLogger(AbstractPollTask):
> """
>
> __slots__ = ("input_fd", "log_file_path", "stdout_fd") + \
> - ("_log_file", "_log_file_real")
> + ("_io_loop_task", "_log_file", "_log_file_nb",
> "_log_file_real")
> def _start(self):
>
> log_file_path = self.log_file_path
> - if log_file_path is not None:
> -
> + if hasattr(log_file_path, 'write'):
> + self._log_file_nb = True
> + self._log_file = log_file_path
> + _set_nonblocking(self._log_file.fileno())
> + elif log_file_path is not None:
> self._log_file =
> open(_unicode_encode(log_file_path, encoding=_encodings['fs'],
> errors='strict'), mode='ab') if
Re: [gentoo-portage-dev] [PATCH] _writer: fix unsafe finally clause (bug 728580)
On Thu, 18 Jun 2020 00:35:48 -0700
Zac Medico wrote:
> In the coroutine finally clause, do not call remove_writer in cases
> where fd has been closed and then re-allocated to a concurrent
> coroutine as in bug 716636.
>
> Also, assume that the caller will put the file in non-blocking mode
> and close the file when done, so that this function is suitable for
> use within a loop.
>
> Bug: https://bugs.gentoo.org/728580
> Signed-off-by: Zac Medico
> ---
> lib/portage/util/futures/_asyncio/process.py | 11 -
> lib/portage/util/futures/_asyncio/streams.py | 50
> ++-- 2 files changed, 33 insertions(+), 28
> deletions(-)
>
> diff --git a/lib/portage/util/futures/_asyncio/process.py
> b/lib/portage/util/futures/_asyncio/process.py index
> 020164c9b..2d3e9b0fd 100644 ---
> a/lib/portage/util/futures/_asyncio/process.py +++
> b/lib/portage/util/futures/_asyncio/process.py @@ -1,9 +1,12 @@
> -# Copyright 2018 Gentoo Foundation
> +# Copyright 2018-2020 Gentoo Authors
> # Distributed under the terms of the GNU General Public License v2
>
> +import os
> +
> import portage
> portage.proxy.lazyimport.lazyimport(globals(),
> 'portage.util.futures:asyncio',
> + 'portage.util.futures.unix_events:_set_nonblocking',
> )
> from portage.util.futures._asyncio.streams import _reader, _writer
> from portage.util.futures.compat_coroutine import coroutine,
> coroutine_return @@ -59,7 +62,11 @@ class _Process(object):
> if input is not None:
> if self._proc.stdin is None:
> raise TypeError('communicate:
> expected file or int, got {}'.format(type(self._proc.stdin)))
> - writer =
> asyncio.ensure_future(_writer(self._proc.stdin, input),
> loop=self._loop)
> + stdin = self._proc.stdin
> + stdin = os.fdopen(stdin, 'wb', 0) if
> isinstance(stdin, int) else stdin
> + _set_nonblocking(stdin.fileno())
> + writer =
> asyncio.ensure_future(_writer(stdin, input, loop=self._loop),
> loop=self._loop)
> + writer.add_done_callback(lambda writer:
> stdin.close())
> try:
> yield asyncio.wait(futures + [self.wait()],
> loop=self._loop) diff --git
> a/lib/portage/util/futures/_asyncio/streams.py
> b/lib/portage/util/futures/_asyncio/streams.py index
> 650a16491..870307e1e 100644 ---
> a/lib/portage/util/futures/_asyncio/streams.py +++
> b/lib/portage/util/futures/_asyncio/streams.py @@ -1,4 +1,4 @@ -#
> Copyright 2018 Gentoo Foundation +# Copyright 2018-2020 Gentoo Authors
> # Distributed under the terms of the GNU General Public License v2
>
> import errno
> @@ -8,7 +8,6 @@ import portage
> portage.proxy.lazyimport.lazyimport(globals(),
> '_emerge.PipeReader:PipeReader',
> 'portage.util.futures:asyncio',
> - 'portage.util.futures.unix_events:_set_nonblocking',
> )
> from portage.util.futures.compat_coroutine import coroutine
>
> @@ -59,38 +58,37 @@ class _Reader(object):
> @coroutine
> def _writer(output_file, content, loop=None):
> """
> - Asynchronously write bytes to output file, and close it when
> - done. If an EnvironmentError other than EAGAIN is
> encountered,
> - which typically indicates that the other end of the pipe has
> - close, the error is raised. This function is a coroutine.
> + Asynchronously write bytes to output file. The output file is
> + assumed to be in non-blocking mode. If an EnvironmentError
> + other than EAGAIN is encountered, which typically indicates
> that
> + the other end of the pipe has closed, the error is raised.
> + This function is a coroutine.
>
> - @param output_file: output file descriptor
> - @type output_file: file or int
> + @param output_file: output file
> + @type output_file: file object
> @param content: content to write
> @type content: bytes
> @param loop: asyncio.AbstractEventLoop (or compatible)
> @type loop: event loop
> """
> - fd = output_file if isinstance(output_file, int) else
> output_file.fileno()
> - _set_nonblocking(fd)
> loop = asyncio._wrap_loop(loop)
> - try:
> - while content:
> + fd = output_file.fileno()
> + while content:
> + try:
> + content = content[os.write(fd, content):]
> + except EnvironmentError as e:
> + if e.errno != errno.EAGAIN:
> + raise
> waiter = loop.create_future()
> - loop.add_writer(fd, lambda:
> waiter.set_result(None))
> + loop.add_writer(fd, lambda: waiter.done() or
> waiter.set_result(None)) try:
> yield waiter
> - while content:
> - try:
> - content =
>
Re: [gentoo-portage-dev] erroneous behavior in 2-style USE dependencies?
On Tue, 16 Jun 2020 20:43:44 +0200 (CEST) "michael.lienhardt" wrote: > Dear all, > My bad for not noticing it sooner, but when there is a dependency > like "=sys-fs/udev-208-r1:0/0[static-libs?]" (that occurs in > virtual/libgudev-215-r3), since 'static-libs' is not a use flags of > sys-fs/udev-242, it is silently not considered during dependency > solving by emerge. However, the PMS states: - it is an > error for a use dependency to be applied to an ebuild which does not > have the flag in question in class="ectt-1000">IUSE_REFERENCEABLE - For EAPIs > listed in table href="https://dev.gentoo.org/~ulm/pms/head/pms.html#x1-56001r4;>5.4 > as not supporting profile defined IUSE > injection, IUSE_REFERENCEABLE > is equal to the calculated IUSE > value. For EAPIs where profile defined class="ectt-1000">IUSE injection is supported, class="ectt-1000">IUSE_REFERENCEABLE is equal to class="ectt-1000">IUSE_EFFECTIVE And 'static-libs' is not in the > IUSE_EFFECTIVE of sys-fs/udev-242 (that ebuild has > EAPI=6). So it seems to me that this current behavior of emerge > should be considered an error, no? Or the PMS should be updated? > Best, Michael Please do NOT send html emails. text only please
Re: [gentoo-portage-dev] [PATCH] repos.conf: add bool sync-openpgp-key-refresh option (bug 661518)
On Fri, 12 Jun 2020 16:51:51 -0700
Zac Medico wrote:
> Add a sync-openpgp-key-refresh option that makes it possible to
> disable key refresh, which may be useful in cases when it is not
> possible to refresh keys.
>
> Key refresh is enabled by default, and if it is disabled then
> the SyncBase._refresh_keys method will output an ewarn message
> like this when the --quiet option is not enabled:
>
> * Key refresh is disabled via a repos.conf sync-openpgp-key-refresh
> * setting, and this is a security vulnerability because it prevents
> * detection of revoked keys!
>
> Bug: https://bugs.gentoo.org/661518
> Signed-off-by: Zac Medico
> ---
> lib/portage/repository/config.py | 10 +-
> lib/portage/sync/syncbase.py | 9 -
> man/portage.5| 9 -
> 3 files changed, 25 insertions(+), 3 deletions(-)
>
> diff --git a/lib/portage/repository/config.py
> b/lib/portage/repository/config.py index 50ab18026..6155c130a 100644
> --- a/lib/portage/repository/config.py
> +++ b/lib/portage/repository/config.py
> @@ -1,4 +1,4 @@
> -# Copyright 2010-2019 Gentoo Authors
> +# Copyright 2010-2020 Gentoo Authors
> # Distributed under the terms of the GNU General Public License v2
>
> from __future__ import unicode_literals
> @@ -113,6 +113,7 @@ class RepoConfig(object):
> 'sync_hooks_only_on_change',
> 'sync_openpgp_keyserver',
> 'sync_openpgp_key_path',
> + 'sync_openpgp_key_refresh',
> 'sync_openpgp_key_refresh_retry_count',
> 'sync_openpgp_key_refresh_retry_delay_exp_base',
> 'sync_openpgp_key_refresh_retry_delay_max',
> @@ -233,6 +234,9 @@ class RepoConfig(object):
> self.sync_openpgp_key_path = repo_opts.get(
> 'sync-openpgp-key-path', None)
>
> + self.sync_openpgp_key_refresh = repo_opts.get(
> + 'sync-openpgp-key-refresh', 'true').lower()
> in ('true', 'yes') +
> for k in ('sync_openpgp_key_refresh_retry_count',
> 'sync_openpgp_key_refresh_retry_delay_exp_base',
> 'sync_openpgp_key_refresh_retry_delay_max',
> @@ -497,6 +501,8 @@ class RepoConfig(object):
> repo_msg.append(indent + "location: " +
> self.location) if not self.strict_misc_digests:
> repo_msg.append(indent +
> "strict-misc-digests: false")
> + if not self.sync_openpgp_key_refresh:
> + repo_msg.append(indent +
> "sync-openpgp-key-refresh: no") if self.sync_type:
> repo_msg.append(indent + "sync-type: " +
> self.sync_type) if self.sync_umask:
> @@ -609,6 +615,7 @@ class RepoConfigLoader(object):
>
> 'sync_hooks_only_on_change',
>
> 'sync_openpgp_keyserver',
> 'sync_openpgp_key_path',
> +
> 'sync_openpgp_key_refresh', 'sync_openpgp_key_refresh_retry_count',
>
> 'sync_openpgp_key_refresh_retry_delay_exp_base',
>
> 'sync_openpgp_key_refresh_retry_delay_max',
> @@ -1047,6 +1054,7 @@ class RepoConfigLoader(object):
> bool_keys = (
> "strict_misc_digests",
> "sync_allow_hardlinks",
> + "sync_openpgp_key_refresh",
> "sync_rcu",
> )
> str_or_int_keys = (
> diff --git a/lib/portage/sync/syncbase.py
> b/lib/portage/sync/syncbase.py index 46644d68e..74818a420 100644
> --- a/lib/portage/sync/syncbase.py
> +++ b/lib/portage/sync/syncbase.py
> @@ -1,4 +1,4 @@
> -# Copyright 2014-2018 Gentoo Foundation
> +# Copyright 2014-2020 Gentoo Authors
> # Distributed under the terms of the GNU General Public License v2
>
> '''
> @@ -252,6 +252,13 @@ class SyncBase(object):
> @type openpgp_env: gemato.openpgp.OpenPGPEnvironment
> """
> out = portage.output.EOutput(quiet=('--quiet' in
> self.options['emerge_config'].opts)) +
> + if not self.repo.sync_openpgp_key_refresh:
> + out.ewarn('Key refresh is disabled via a
> repos.conf sync-openpgp-key-refresh')
> + out.ewarn('setting, and this is a security
> vulnerability because it prevents')
> + out.ewarn('detection of revoked keys!')
> + return
> +
> out.ebegin('Refreshing keys via WKD')
> if openpgp_env.refresh_keys_wkd():
> out.eend(0)
> diff --git a/man/portage.5 b/man/portage.5
> index 36c871123..136ebaafe 100644
> --- a/man/portage.5
> +++ b/man/portage.5
> @@ -1,4 +1,4 @@
> -.TH "PORTAGE" "5" "Apr 2019" "Portage VERSION" "Portage"
> +.TH "PORTAGE" "5" "Jun 2020" "Portage VERSION"
Re: [gentoo-dev] Last rites: app-cdr/sync2d
On Fri, 5 Jun 2020 18:27:51 -0700 Christopher Head wrote: > On Fri, 5 Jun 2020 12:40:17 -0700 > Matt Turner wrote: > > > With that in mind, I don't expect it to gain Python 3 support, nor > > do I expect an additional 15 days of waiting time to change that > > fact. 15 vs 30 days doesn't seem worth squabbling over. > > Not that I care about this specific case, but isn’t the 30-day time > period also meant as a nice long warning time for people *using* the > package to give them time to migrate to something else before it > starts to be unsupported, potentially break the depgraph, no longer be > installable on additional systems, and so on? The fact that the tree is git based now. It is easy to checkout the tree and checkout the tree at a state where the pkg is still there. You can then copy it to an overlay to keep around for hownever long. The 30 days were more for when the tree was in CVS. There it was much more complicated to get the pkg from the attic. Also, if it is installed, there is a copy of the ebuild in the installed db: /var/db/pkg/cat/pkg-version The only other potential caveat is that the source tarball must still be available from upstream. Our mirrors will eventually drop the source since the pkg/version is no longer in the ebuild tree.
Re: [gentoo-dev] Value of Continuous integration vs Code Review / Pull Requests
On Wed, 27 May 2020 01:28:02 -0700 Alec Warner wrote: > On Wed, May 27, 2020 at 1:09 AM Brian Dolbec > wrote: > > > On Tue, 26 May 2020 20:24:56 -0700 > > Alec Warner wrote: > > > > > The TL;DR is that a crack team of infra-folks[0] have been putting > > > together demos of CI services and things like gitlab / gitea / > > > gerrit and so on. > > > > > > Some of these come in combined (e.g. gitlab offers repo hosting, > > > code review / pull reqs, CI services, and deploy services.) Some > > > of these are piecemeal (e.g. gerrit has code review, zuul has CI) > > > and gitea offers repo-hosting but CI is separate (e.g. drone.) > > > > > > On the infra-side, I think we are pretty happy with repo-hosting > > > (gitolite) and repo-serving (gitweb). We are missing a CI piece > > > and a pull-request piece. Most of the users using PRs use either > > > a gitlab or github mirror. > > > > > > I think the value of CI is pretty obvious to me (and I see tons > > > of use cases in Infra.) We could easily build CI into our current > > > repository solution (e.g. gitolite.) However gitolite doesn't > > > really support PRs in a uniform way and so CI is mostly for > > > submitted code; similar to the existing ::gentoo repo CI offered > > > by mgorny. > > > > > > If we build a code review solution (like gitea / gerrit) would > > > people use it? Would you use it if you couldn't merge (because > > > the code review solution can't gpg sign your commits or merges) > > > so a tool like the existing pram tool would be needed to merge? > > > > > > -A > > > > > > [0] Mostly arzano, if I'm honest. I am just the point-y haired > > > manager in this effort. > > > > There are several CI systems that could be used. As for CI, my > > experience has been with buildbot. It would be fairly > > straightforward to integrate with the current gitolite/gitweb > > hosting. It is also extremely flexible in its capabilities, > > although can be difficult to master. It has a good web interface, > > but it can even be run via it's API's using curses, python, > > bash, There is a buildbot-travis plugin which is capable of > > running existing .travis file configurations. It also extends its > > capabilities to include custom buildbot step definitions. I have > > not packaged it yet for gentoo, but it is on my todo list. One of > > buildbot's capabilities is the ability to run tests/builds on > > multiple workers (different arches or whatever) both in parallel or > > series. It could be made to integrate with our bugzilla using the > > python client (pybugs was it?). > > My understanding is that CI systems are all quite similar. We have > configured gitlab-ci, drone, and zuul as tests and I am familiar with > travis-ci and buildbot from other projects. I'm less concerned about > this aspect tbh. I'm also not super concerned about packaging. E.g. > for gitlab-runner (the worker portion of gitlab-ci) we just deploy > upstream containers and don't package them in ::gentoo at all. Our > onprem gitlab is a container solution; gitea is a container solution; > our SSO IDP is a container solution; gerrit is currently a container > solution. These don't bother me (too much, ugh zuul uses zookeeper? > ugh.) > > The major differentiators for CI appear to be: > - SCM support (we currently only really care about git compatible > systems, but some contenders only support some providers.) > - builders / workers (how do they deploy). For example gitlab has a > container based work executor while zuul uses ansible; I suspect > - Authentication (ideally should support SAML or openID so we can > integrate with our alpha sso solution for Gentoo.) > - The webUI; e.g is the solution horrible to use / interact with. > Hard to say without a trial, IMHO. > > -A > > > > > > But that still leaves a PR/code review option. > > > > I have another buildbot version bump to 2.8.0 to do. While reading over the changes. There are changes to the gerrit integration (GerritEventLogPoller) which I did not know it had or used. There is also a change in the 2.7.0 buildbot-worker: * Command buildbot-worker create-worker now supports ipv6 address for buildmaster connection. I know that buildbot have been creating buildbot containers since the 1.x series, but I've not worked with them. I am planning to learn how to use buildbot with containers and kubernetes. I will certainly check out how that worker command works. whether it is a docker or kubernetes command or either... So, buildbot may be a good option since it can integrate with gerrit and (I think) dynamically deploy workers in containers. So be able to integrate with our existing git/gitweb/bugzilla as well as github/gitlab
Re: [gentoo-dev] Packages up for grabs
On Wed, 27 May 2020 17:39:21 +0200 Piotr Karbowski wrote: > Hi, > > On 27/05/2020 01.31, Brian Dolbec wrote: > > * dev-python/boto3 > > * dev-python/botocore > > Do you mind if I join you on those? I use them a lot, and I planned to > comaintain awscli since Patrick is the only one current maintainer of > those, boto3 is vital part of it too. > > -- Piotr. > Yes, by all means join in :)
Re: [gentoo-dev] Value of Continuous integration vs Code Review / Pull Requests
On Tue, 26 May 2020 20:24:56 -0700 Alec Warner wrote: > The TL;DR is that a crack team of infra-folks[0] have been putting > together demos of CI services and things like gitlab / gitea / gerrit > and so on. > > Some of these come in combined (e.g. gitlab offers repo hosting, code > review / pull reqs, CI services, and deploy services.) Some of these > are piecemeal (e.g. gerrit has code review, zuul has CI) and gitea > offers repo-hosting but CI is separate (e.g. drone.) > > On the infra-side, I think we are pretty happy with repo-hosting > (gitolite) and repo-serving (gitweb). We are missing a CI piece and a > pull-request piece. Most of the users using PRs use either a gitlab > or github mirror. > > I think the value of CI is pretty obvious to me (and I see tons of use > cases in Infra.) We could easily build CI into our current repository > solution (e.g. gitolite.) However gitolite doesn't really support PRs > in a uniform way and so CI is mostly for submitted code; similar to > the existing ::gentoo repo CI offered by mgorny. > > If we build a code review solution (like gitea / gerrit) would people > use it? Would you use it if you couldn't merge (because the code > review solution can't gpg sign your commits or merges) so a tool like > the existing pram tool would be needed to merge? > > -A > > [0] Mostly arzano, if I'm honest. I am just the point-y haired > manager in this effort. There are several CI systems that could be used. As for CI, my experience has been with buildbot. It would be fairly straightforward to integrate with the current gitolite/gitweb hosting. It is also extremely flexible in its capabilities, although can be difficult to master. It has a good web interface, but it can even be run via it's API's using curses, python, bash, There is a buildbot-travis plugin which is capable of running existing .travis file configurations. It also extends its capabilities to include custom buildbot step definitions. I have not packaged it yet for gentoo, but it is on my todo list. One of buildbot's capabilities is the ability to run tests/builds on multiple workers (different arches or whatever) both in parallel or series. It could be made to integrate with our bugzilla using the python client (pybugs was it?). But that still leaves a PR/code review option.
Re: [gentoo-dev] Packages up for grabs
On Tue, 26 May 2020 23:12:06 +0100 Andrey Utkin wrote: > I have transitioned to "away" state as I have to reclaim my time for > other uses. Here I am trying to reduce the scope of my Gentoo > responsibilities to make potential return to activity less dreadful > and overwhelming. > > Call for successors > === > > The following are the packages I do not currently use myself so have > the least motivation about. Dropping me from maintainers list is > encouraged. > > WIFI access point service: > > * net-wireless/hostapd > (Co-maintained by zerochaos, still additional attention is > encouraged.) > > Python API for AWS: > > * dev-python/s3transfer > * dev-python/boto3 > * dev-python/botocore I can take boto3, botocore since they are needed for buildbot
Re: [gentoo-dev] [RFC] Anti-spam for goose
On Sat, 23 May 2020 07:20:22 +1200 Kent Fredric wrote: > On Thu, 21 May 2020 10:47:07 +0200 > Michał Górny wrote: > > > Other ideas > > === > > Do you have any other ideas on how we could resolve this? > > And a question I'd like to revisit, because nobody responded to it: > > - What are the incentives a would-be spammer has to spam this service. > > Services that see spam *typically* have a definable objective. > > *Typically* it revolves around the ability to submit /arbitrary text/, > which allows them to hawk something, and this becomes a profit motive. > > If we implement data validation so that there's no way for them to > profit off what they spam, seems likely they'll be less motivated to > develop the necessary circumvention tools. ( as in, we shouldn't > accept arbitrary CAT/PN pairs as being valid until something can > confirm those pairs exist in reality ) > > There may be people trying to jack the data up, but ... it seems a > less worthy target. > > So it seems the largest risk isn't so much "spam", but "denial of > service", or "data pollution". > > Of course, we should still mitigate, but /how/ we mitigate seems to > pivot around this somewhat. We cannot exclude overlays which will have cat/pkg not in the main gentoo repo. So, we should not excludea submission that includes a few of these. They would just become irrelevant outliers to our processesing of the data. In fact some of these outlier pkgs could be relevant to our including that pkg into the main repo. But, like you I agree that purely spam submissions would be few, if any.
Re: [gentoo-portage-dev] [PATCH] Default BINPKG_COMPRESSION to zstd (bug 715108)
On Sun, 10 May 2020 19:29:34 -0700
Zac Medico wrote:
> This includes a _compat_upgrade.binpkg_compression script that the
> ebuild can call in pkg_preinst in order to maintain a
> backward-compatible bzip2 default when appropriate, ensuring that
> binary package consumers are not caught off guard.
>
> Bug: https://bugs.gentoo.org/715108
> Signed-off-by: Zac Medico
> ---
> .travis.yml | 4 ++
> cnf/make.globals | 5 ++-
> .../_compat_upgrade/binpkg_compression.py | 40
> +++ .../tests/resolver/ResolverPlayground.py |
> 1 + man/make.conf.5 | 4 +-
> 5 files changed, 50 insertions(+), 4 deletions(-)
> create mode 100644 lib/portage/_compat_upgrade/binpkg_compression.py
>
> diff --git a/.travis.yml b/.travis.yml
> index 9269d4034..2132c8c87 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -1,3 +1,4 @@
> +dist: bionic
> language: python
> python:
> - 2.7
> @@ -8,6 +9,9 @@ python:
> - pypy3
>
> # command to install dependencies
> +before_install:
> +# Use "dist: bionic" to get a zstd with --long support.
> +- sudo apt-get -y install zstd
> install:
> - pip install tox
>
> diff --git a/cnf/make.globals b/cnf/make.globals
> index 4a59dbe3c..dd3f28f70 100644
> --- a/cnf/make.globals
> +++ b/cnf/make.globals
> @@ -34,8 +34,9 @@ RPMDIR="/var/cache/rpm"
> # Temporary build directory
> PORTAGE_TMPDIR="/var/tmp"
>
> -# The compression used for binary packages. Defaults to zstd when
> USE=zstd is enabled. -BINPKG_COMPRESS="bzip2"
> +# The compression used for binary packages. Defaults to zstd except
> for +# existing installs where bzip2 is used for backward
> compatibility. +BINPKG_COMPRESS="zstd"
>
> # Fetching command (3 tries, passive ftp for firewall compatibility)
> FETCHCOMMAND="wget -t 3 -T 60 --passive-ftp -O
> \"\${DISTDIR}/\${FILE}\" \"\${URI}\"" diff --git
> a/lib/portage/_compat_upgrade/binpkg_compression.py
> b/lib/portage/_compat_upgrade/binpkg_compression.py new file mode
> 100644 index 0..0f5704733 --- /dev/null
> +++ b/lib/portage/_compat_upgrade/binpkg_compression.py
> @@ -0,0 +1,40 @@
> +# Copyright 2020 Gentoo Authors
> +# Distributed under the terms of the GNU General Public License v2
> +
> +import re
> +
> +import portage
> +from portage import os
> +from portage.const import GLOBAL_CONFIG_PATH
> +
> +COMPAT_BINPKG_COMPRESS = 'bzip2'
> +
> +
> +def main():
> + """
> + If the current installation is still configured to use the
> old
> + default BINPKG_COMPRESS=bzip2 setting, then patch
> make.globals
> + inside ${ED} to maintain backward compatibility, ensuring
> that
> + binary package consumers are not caught off guard. This is
> + intended to be called from the ebuild as follows:
> +
> + pkg_preinst() {
> + python_setup
> + env -u BINPKG_COMPRESS
> +
> PYTHONPATH="${D%/}$(python_get_sitedir)${PYTHONPATH:+:${PYTHONPATH}}"
> \
> + "${PYTHON}" -m
> portage._compat_upgrade.binpkg_compression || die
> + }
> + """
> + if portage.settings.get('BINPKG_COMPRESS',
> COMPAT_BINPKG_COMPRESS) == COMPAT_BINPKG_COMPRESS:
> + config_path = os.path.join(os.environ['ED'],
> GLOBAL_CONFIG_PATH.lstrip(os.sep), 'make.globals')
> + with open(config_path) as f:
> + content = f.read()
> + compat_setting =
> 'BINPKG_COMPRESS="{}"'.format(COMPAT_BINPKG_COMPRESS)
> + portage.output.EOutput().einfo('Setting
> make.globals default {} for backward
> compatibility'.format(compat_setting))
> + content = re.sub('^BINPKG_COMPRESS=.*$',
> compat_setting, content, flags=re.MULTILINE)
> + with open(config_path, 'wt') as f:
> + f.write(content)
> +
> +
> +if __name__ == '__main__':
> + main()
> diff --git a/lib/portage/tests/resolver/ResolverPlayground.py
> b/lib/portage/tests/resolver/ResolverPlayground.py index
> 98831e000..de80a0cc1 100644 ---
> a/lib/portage/tests/resolver/ResolverPlayground.py +++
> b/lib/portage/tests/resolver/ResolverPlayground.py @@ -112,6 +112,7
> @@ class ResolverPlayground(object): "uname",
> "uniq",
> "xargs",
> + "zstd",
> )
> # Exclude internal wrappers from PATH lookup.
> orig_path = os.environ['PATH']
> diff --git a/man/make.conf.5 b/man/make.conf.5
> index f82fed65a..a3bd662ae 100644
> --- a/man/make.conf.5
> +++ b/man/make.conf.5
> @@ -1,4 +1,4 @@
> -.TH "MAKE.CONF" "5" "Nov 2019" "Portage VERSION" "Portage"
> +.TH "MAKE.CONF" "5" "May 2020" "Portage VERSION" "Portage"
> .SH "NAME"
> make.conf \- custom settings for Portage
> .SH "SYNOPSIS"
> @@ -115,7 +115,7 @@ This variable is used to determine the
> compression used for \fIbinary packages\fR.
Re: [gentoo-portage-dev] [PATCH] Enable FEATURES=parallel-install by default (bug 715110)
On Sun, 10 May 2020 21:32:25 -0700
Zac Medico wrote:
> The feature enables finer grained locks for install operations, and
> everyone agrees that it's safe to enable by default.
>
> Bug: https://bugs.gentoo.org/715110
> Signed-off-by: Zac Medico
> ---
> cnf/make.globals | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/cnf/make.globals b/cnf/make.globals
> index 4a59dbe3c..5ba1ac6fa 100644
> --- a/cnf/make.globals
> +++ b/cnf/make.globals
> @@ -55,7 +55,7 @@ FETCHCOMMAND_SFTP="bash -c \"x=\\\${2#sftp://} ;
> host=\\\${x%%/*} ; port=\\\${ho FEATURES="assume-digests
> binpkg-docompress binpkg-dostrip binpkg-logs
> config-protect-if-modified distlocks ebuild-locks fixlafiles
> ipc-sandbox merge-sync multilib-strict
> - network-sandbox news parallel-fetch pid-sandbox
> + network-sandbox news parallel-fetch parallel-install
> pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps
>sandbox sfperms strict
>unknown-features-warn unmerge-logs unmerge-orphans
> userfetch
works for me :)
Re: [gentoo-dev] rfc: "emerge --sync" vs "emaint sync"
On Wed, 6 May 2020 17:02:42 -0500
William Hubbs wrote:
> All,
>
> I know that most of our documentation tells people to use "emerge
> --sync"; however, today I heard about "emaint sync" for the first
> time. ;-)
>
> Which one should we use? Will there be a phase-out for "emerge
> --sync" or "emaint sync"? Are the plans to keep both available?
>
> Thanks,
>
> William
>
Hi William. emaint --sync is not going to replace emerge --sync.
They both use the same plugin sync modules.
The difference between them is that emerge --sync is generally done to
sync all repositories defined which have [auto-sync] = yes. Zac
extended it to do individual repositories as well if you specify the
repo name.
ie: emerge --sync foo
emaint --sync offers fine grained syncing of individual repositories
via several options. With emaint you can set a repo's [auto-sync] = no
and still manually sync it on demand from emaint. It offers some
additional capabilities that some power users/devs may need/enjoy
depending on their work flow.
Here are the emaint sync options:
-r REPO, --repo REPO (sync module only): -r, --repo Sync the specified repo
-A, --allrepos(sync module only): -A, --allrepos Sync all repos that
have a sync-url defined
-a, --auto(sync module only): -a, --auto Sync auto-sync enabled
repos only
--sync-submodule {glsa,news,profiles}
(sync module only): Restrict sync to the specified
submodule(s)
ie: emaint sync -r foo -r bar
The --sync-submodule option was added for developers working from the
git tree with the other git tree submodules added the the base gentoo
git ebuild tree. In our case the profiles did end up as part of the
main ebuild tree. But you can add the glsa and news git repos to the
git ebuild tree in order to make a complete repository.
NOTE: the above pertains to the developer git tree, not the anonymous
git tree. That git repo has had the glsa and news repos added to it
for general consumption.
Re: [gentoo-dev] Last standing Python 2.7 dependency
On Sun, 3 May 2020 08:37:27 -0400 Brian Evans wrote: > On 5/3/20 2:58 AM, Fabian Groffen wrote: > > On 02-05-2020 23:24:42 -0700, Brian Dolbec wrote: > >> On Sun, 3 May 2020 07:28:50 +0200 > >> Viktar Patotski wrote: > >> > >>> Hi all, > >>> > >>> I'd also like to clean my system and have it Python 2.7 free. Are > >>> there any guidelines to check which packages are still using > >>> pyton2_7 in my system? > >>> > >>> Thanks, > >>> Viktar > >>> > >> > >> There are both equery and enalyze commands in gentoolkit that can > >> give you reports about what pkgs are installed. > >> > >> equery hasuse > >> enalyze analyze [use|pkguse] > >> > >> for help on them: > >> equery -h > >> equery hasuse -h > >> enalyze -h > >> enalyze a -h > > > > In addition to these great tools, portage-utils' quse might also be > > useful: > > > > % quse python2_7 > > ... > > > > > > Thanks, > > Fabian > > > > All of the mentioned tools will show if packages have the flag but not > necessarily have it active. Not True: enalyze does strictly installed pkgs analysis, plus has the ability to rebuild package.accept_keywords and package.use files after profile changes or disaster (file loss, etc) > > eix has an option to search the active flag: > > eix --installed-with-use > > However, this still skips build-time dependencies that may keep python > 2.7 around. > > The most accurate way to see what's tied to python 2.7 is to pretend > to remove it: > emerge -pvc dev-lang/python:2.7 > > Brian >
Re: [gentoo-dev] Last standing Python 2.7 dependency
On Sun, 3 May 2020 07:28:50 +0200 Viktar Patotski wrote: > Hi all, > > I'd also like to clean my system and have it Python 2.7 free. Are > there any guidelines to check which packages are still using pyton2_7 > in my system? > > Thanks, > Viktar > There are both equery and enalyze commands in gentoolkit that can give you reports about what pkgs are installed. equery hasuse enalyze analyze [use|pkguse] for help on them: equery -h equery hasuse -h enalyze -h enalyze a -h
Re: [gentoo-portage-dev] [PATCH] make.globals: add default BINPKG_COMPRESS setting (bug 715108)
On Sun, 26 Apr 2020 14:57:32 -0700
Zac Medico wrote:
> The ebuild will have a default enabled USE=zstd which changes the
> default to zstd here.
>
> Bug: https://bugs.gentoo.org/715108
> Bug: https://bugs.gentoo.org/719456
> Signed-off-by: Zac Medico
> ---
> cnf/make.globals | 5 -
> 1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/cnf/make.globals b/cnf/make.globals
> index 139e1ce97..4a59dbe3c 100644
> --- a/cnf/make.globals
> +++ b/cnf/make.globals
> @@ -1,4 +1,4 @@
> -# Copyright 1999-2019 Gentoo Authors
> +# Copyright 1999-2020 Gentoo Authors
> # Distributed under the terms of the GNU General Public License v2
> # System-wide defaults for the Portage system
>
> @@ -34,6 +34,9 @@ RPMDIR="/var/cache/rpm"
> # Temporary build directory
> PORTAGE_TMPDIR="/var/tmp"
>
> +# The compression used for binary packages. Defaults to zstd when
> USE=zstd is enabled. +BINPKG_COMPRESS="bzip2"
> +
> # Fetching command (3 tries, passive ftp for firewall compatibility)
> FETCHCOMMAND="wget -t 3 -T 60 --passive-ftp -O
> \"\${DISTDIR}/\${FILE}\" \"\${URI}\"" RESUMECOMMAND="wget -c -t 3 -T
> 60 --passive-ftp -O \"\${DISTDIR}/\${FILE}\" \"\${URI}\""
works for me :)
Re: [gentoo-dev] Packages up for grabs: dev-python/pythondialog /ramlfications /repoze-sphinx-autointerface /robotframework /sleekxmpp /sphinxcontrib-issuetracker /toro /tx*
On Sun, 26 Apr 2020 17:43:16 +0200 Michał Górny wrote: > The Python team will no longer maintain the following packages. They > all need porting to Python 3.7 and newer, and have some obstacle > in doing so. They have only few revdeps which we do not maintain. > > dev-python/pythondialog > dev-python/ramlfications > dev-python/repoze-sphinx-autointerface > dev-python/robotframework > dev-python/sleekxmpp > dev-python/sphinxcontrib-issuetracker > dev-python/toro > dev-python/txaio > dev-python/txAMQP > dev-python/txgithub > dev-python/txrequests > > > pythondialog is used by net-analyzer/gvm-tools (jonas.licht@) > and net-vpn/protonvpn-cli (ilmostro7@). It is missing tests. > > ramlfications are used by dev-util/buildbot (dolsen@ indicate he'll > revive it). The tests fail already and it's on 2016 snapshot. > It seems pretty much dead. > > repoze-sphinx-autointerface is used by dev-python/txtorcon (just taken > by dolsen@). It is missing tests. > > robotframework is needed only for dev-python/robotbackgroundlogger > (zmedico@). It is missing tests. > > sleekxmpp is used by net-im/err (pinkbyte@) and net-im/spectrum2 > (conikost@, andrey_utkin@). It is broken with py3.7 and upstream > has archived the repository. > > sphinxcontrib-issuetracker is used only by dev-python/pydocstyle > (python@, monsieurp@, andrewammerlaan@). Either the tests or the > whole package is broken with Sphinx 3 (and possibly older). > > toro is needed only by dev-python/thriftpy (sci@). Some of the tests > are failing (already) and no upstream activity since 2016. > > txaio is only used by dev-util/buildbot (dolsen@) and sci-libs/vtk > (sci@). All tests fail with a test suite error. > > txAMQP by dev-python/carbon (patrick@, grobian@). Tests require > a system instance of rabbitmq server. > > txgithub is the only package that I managed to test up to Python 3.8 > (though either test coverage or API coverage is near zero). It is > used by dev-util/buildbot (dolsen@). > > txrequests is used by dev-util/buildbot (dolsen@). It is missing > tests. > Yes, I'll take the buildbot ones again. ramlfications had been dropped by buildbot. It is only for buildbot-0.9.7 which I'll be cleaning out, so we can mark it for cleaning.
Re: [gentoo-dev] Last rites: dev-python/txtorcon
On Sun, 26 Apr 2020 14:49:25 +0200 Michał Górny wrote: > # Michał Górny (2020-04-26) > # Unmaintained. Stuck on Python 3.6. No revdeps. > # Removal in 30 days. Bug #719604. > dev-python/txtorcon > I'm taking maintainence back, is a dependency for crossbar which I am bringing back too.
Re: [gentoo-dev] [RFC] Ideas for gentoostats implementation
On Sun, 26 Apr 2020 11:32:06 +0200 Toralf Förster wrote: > On 4/26/20 11:09 AM, Ulrich Mueller wrote: > > Instead of using the IP address, you could generate a UUID when > > installing the tool. > > like the pfl tool did ? > Like the last gentoostats gsoc project did. As for enterprise/school/multiple clone deployments. Those are generated by one person/team, then deployed. We would need that person/team to only enable their test system for gentoostats/disabled for deployments. Repeated failure to do that could result in that uuid being blacklisted. Part of the initial profile details for that vm/image would be some details about approx numbers of deployments (yes, subject to change. But useful to know whether it is 10-15 or 100-500. type of deployment ie: vm/docker/kubernetes/desktop/server...
Re: [gentoo-dev] Packages up for grabs: dev-python/{autobahn,cbor,girder-client,kazoo,libcloud,logbook}
On Fri, 24 Apr 2020 08:22:47 +0200 Michał Górny wrote: > Hi, > > The following packages will no longer be maintained by the Python > team: > > dev-python/autobahn > dev-python/cbor I'll maintain these again along with buildbot now that I am back. > dev-python/girder-client > dev-python/kazoo > dev-python/libcloud > dev-python/logbook > > All of them are stuck on py3.6 and have 1-2 revdeps, and aren't very > useful to the general Python community. All of them have some > problems that prevent us from testing them, and generally would > qualify for last rites if they were not blocked by the revdeps. > > autobahn is only used by vpython (sci@), vtk (sci@) and buildbot (last > rited). It is multiple versions behind upstream. > > cbor is needed for autobahn and ros. It hasn't been touched since > 2016. The test suite fails, and is pretty unreadable. It looks like > a new serialization format that didn't work out. Hopefully newer buildbot versions won't need as many deps and we can clean some of the old ones... > > girder-client is only sed by yt (sci-astronomy@). The ebuild doesn't > declare tests. > > kazoo has no real revdeps (only zake that are 'testing utilities' for > kazoo). Testing requires a running Zookeeper instance. > > libcloud is used by salt only (chutzpah@). It has all tests > restricted because of 'known test failures'. Someone needs to put an > effort to make the tests pass or at least filter out these that don't > pass and run the rest. > > logbook is used only by pyfa and nikola. I've tried bumping it > but a lot of tests still fail with the newest version and then one > seems to hang. > > Expect more as I proceed with py3.7 testing. >
Re: [gentoo-portage-dev] [PATCH] depgraph: respect
On Sat, 11 Apr 2020 17:39:53 -0700
Zac Medico wrote:
> When searching for slot operator rebuilds, respect non slot-operator
> components of parent dependencies, so that a like the not be completely ignored. This will prevent erroneous attempts to
> trigger slot operator rebuilds for upgrades that would break
> when backtracking tried to resolve the breakage).
>
> Bug: https://bugs.gentoo.org/717140
> Signed-off-by: Zac Medico
> ---
> lib/_emerge/depgraph.py | 12
> +--- .../resolver/test_slot_operator_reverse_deps.py |
> 1 - 2 files changed, 9 insertions(+), 4 deletions(-)
>
> diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
> index 6d1f62178..ec90e59df 100644
> --- a/lib/_emerge/depgraph.py
> +++ b/lib/_emerge/depgraph.py
> @@ -2068,9 +2068,15 @@ class depgraph(object):
> for parent, atom in
> self._dynamic_config._parent_atoms.get(existing_pkg, []): if
> isinstance(parent, Package): if parent in built_slot_operator_parents:
> - # This parent may
> need to be rebuilt, so its
> - # dependencies
> aren't necessarily relevant.
> - continue
> + # This parent may
> need to be rebuilt, therefore
> + # discard its soname
> and built slot operator
> + # dependency
> components which are not necessarily
> + # relevant.
> + if atom.soname:
> + continue
> + elif atom.package
> and atom.slot_operator_built:
> + # This
> discards the slot/subslot component.
> + atom =
> atom.with_slot("=")
> if replacement_parent is not
> None and \ (replacement_parent.slot_atom == parent.slot_atom
> diff --git
> a/lib/portage/tests/resolver/test_slot_operator_reverse_deps.py
> b/lib/portage/tests/resolver/test_slot_operator_reverse_deps.py index
> 5c5295510..e0e738d6b 100644 ---
> a/lib/portage/tests/resolver/test_slot_operator_reverse_deps.py +++
> b/lib/portage/tests/resolver/test_slot_operator_reverse_deps.py @@
> -122,7 +122,6 @@ class
> SlotOperatorReverseDepsLibGit2TestCase(TestCase): trigger an upgrade
> to dev-libs/libgit2-1.0.0-r1, ultimately resulting in an undesirable
> downgrade to dev-libs/libgit2-0.28.4-r1. """
> - self.todo = True
>
> ebuilds = {
>
looks fine to me :)
Re: [gentoo-portage-dev] Portage project lead election
On Tue, 16 Apr 2019 13:05:40 -0700 Zac Medico wrote: > Hi all, > > We're due for a Portage project lead election [1] which is required by > GLEP 39 [2]. Please reply with nominations. Thanks! > > [1] https://bugs.gentoo.org/683510 > [2] https://www.gentoo.org/glep/glep-0039.html Well, as Zac is the only real active dev in the team, I nominate him. I have been debating withdrawing from the team as I am quite busy with other things atm. I am also looking to establish a new source of income. But I may have to retire... I will be going to the east coast for some months to work on some things that should help minimize my finacial load requirements. So, I will be quite inactive for some time while I establish myself there. Thanks Zac...
Re: [gentoo-dev] New Python application with multiple new dependencies
On Sat, 13 Apr 2019 23:03:53 +0200 Michał Górny wrote: > On Sat, 2019-04-13 at 22:27 +0200, Ralph Seichter wrote: > > Hi folks, > > > > imagine a Python application "monty" with the following requirements > > listed in setup.py: > > > > install_requires=[ > > 'ham>=1.0', > > 'spam>=2.0', > > 'eggs>=1.5' > > ] > > > > If I want to add "monty" as a new Gentoo package, and if none of > > "ham", "spam" and "eggs" are so far available as Gentoo packages, > > what is the recommended way of handling this situation? > > I presume you want to proxy-maintain all of those packages. > > > > > 1. Create one pull request that contains separate commits for adding > > each of the four Python packages? > > This one. We generally find it easier when we see the big picture > in one place. > > By the way, there's gentoo-proxy-maint@ ml for this kind of questions. > Also, it is best to state that these new pkgs are a depencency of "monty" in their commit messages. It can be clearer as to why these are in the tree if "monty" drops them as a dependency later and there are no other reverse deps for them in the tree. ie: app-foo/ham: New pkg, depencency of app-foo/monty-1.2.3
Re: [gentoo-dev] adding app-crypt/gentoo-keys to @system
On Tue, 19 Feb 2019 23:03:51 -0600 Matthew Thode wrote: > On 19-02-20 00:00:04, Michael Orlitzky wrote: > > On 2/19/19 11:21 PM, Matthew Thode wrote: > > >> > > >> What problem would this solve? (Is adding gentoo-keys to @system > > >> the least bad way to solve it?) > > >> > > > > > > It'd allow the stage tarballs (3,4) to use webrsync-gpg to verify > > > portage tarballs. This is useful for the initial sync (as called > > > out in our manual). Otherwise using emerge-webrsync could be > > > mitm'd or otherwise messed with. > > > > Ok, then I agree with the goal if not the solution. This is a > > portage-specific thing, namely > > > > FEATURES=webrsync-gpg > > > > that should be enabled by default on a stage3. (Making new users go > > out of their way to add basic security is daft.) Portage already has > > USE=rsync-verify, and I think we could either > > > > a) expand the meaning of that flag to include enabling > > webrsync-gpg by default, and to pull in gentoo-keys; or > > > > b) add another (default-on) flag like USE=webrsync-verify to do it > > > > That flag would be enabled by default, so gentoo-keys would be > > pulled in as part of @system without actually being *in* the > > @system. Something along those lines would achieve the same goal in > > a cleaner way. > > > > > > This worksforme (optional, default enabled dep of portage with a > default feature flag change). > > > > As far how we treat deps of @system packages, since this does not > > > have any deps that should help check that box for anyone > > > worried. > > > > I meant the other way around. Once gentoo-keys is in @system, > > packages will (inconsistently) omit gentoo-keys from (R)DEPEND. > > There's no real policy or consensus on the matter, and it makes it > > a real PITA if we ever want to remove things from @system, because > > lots of packages will break in unpredictable ways. > > > > Ah, ya, that makes sense. > One of the things that releng has bantered about the last few years is making a stage4 with these extra non @system pkgs. The stage4 would allow all the extra pkgs needed for new installs without adding to @system. The system set could possibly be trimmed a little more then too. Then knowledgeable users could work with minimal stage3's when it suits their purpose while new users doing installs get the advantage of the additional pre-installed pkgs.
Re: [gentoo-portage-dev] [PATCH] locks: handle sshfs hardlink inode numbers (bug 678218)
On Sun, 17 Feb 2019 15:04:29 -0800 Zac Medico wrote: > Since hardlinks on sshfs do not have matching inode numbers, detect > this behavior and use a simple stat call to detect if lock_path has > been removed. > > Bug: https://bugs.gentoo.org/678218 > Signed-off-by: Zac Medico > --- > lib/portage/locks.py | 27 +++ > 1 file changed, 27 insertions(+) > > diff --git a/lib/portage/locks.py b/lib/portage/locks.py > index 74c2c086a..510925da0 100644 > --- a/lib/portage/locks.py > +++ b/lib/portage/locks.py > @@ -340,6 +340,33 @@ def _lockfile_was_removed(lock_fd, lock_path): > > hardlink_stat = os.stat(hardlink_path) > if hardlink_stat.st_ino != fstat_st.st_ino or > hardlink_stat.st_dev != fstat_st.st_dev: > + # Create another hardlink in order to detect > whether or not > + # hardlink inode numbers are expected to > match. For example, > + # inode numbers are not expected to match > for sshfs. > + inode_test = hardlink_path + '-inode-test' > + try: > + os.unlink(inode_test) > + except OSError as e: > + if e.errno not in (errno.ENOENT, > errno.ESTALE): > + _raise_exc(e) > + try: > + os.link(hardlink_path, inode_test) > + except OSError as e: > + if e.errno not in (errno.ENOENT, > errno.ESTALE): > + _raise_exc(e) > + return True > + else: > + if not > os.path.samefile(hardlink_path, inode_test): > + # This implies that inode > numbers are not expected > + # to match for this file > system, so use a simple > + # stat call to detect if > lock_path has been removed. > + return not > os.path.exists(lock_path) > + finally: > + try: > + os.unlink(inode_test) > + except OSError as e: > + if e.errno not in > (errno.ENOENT, errno.ESTALE): > + _raise_exc(e) > return True > finally: > try: Looks fine to me
Re: [gentoo-portage-dev] [PATCH] pid-sandbox: execute pid-ns-init as pid 1 (bug 675312)
On Sun, 13 Jan 2019 16:27:21 -0800
Zac Medico wrote:
> Execute pid-ns-init as the first fork after unshare, as
> required for it to have pid 1 and become the default reaper
> of orphaned descendant processes. In _exec, exec a separate
> pid-ns-init process to behave as a supervisor which will
> forward signals to init and forward exit status to the parent
> process.
>
> Fixes: a75d5546e3a4 ("Introduce a tiny init replacement for inside
> pid namespace") Bug: https://bugs.gentoo.org/670484
> ---
> bin/pid-ns-init| 44
> ++ lib/portage/process.py |
> 26 ++--- 2 files changed, 59 insertions(+), 11
> deletions(-)
>
> diff --git a/bin/pid-ns-init b/bin/pid-ns-init
> index 843257b70..3792eeaa4 100644
> --- a/bin/pid-ns-init
> +++ b/bin/pid-ns-init
> @@ -1,23 +1,59 @@
> #!/usr/bin/env python
> -# Copyright 2018 Gentoo Authors
> +# Copyright 2018-2019 Gentoo Authors
> # Distributed under the terms of the GNU General Public License v2
>
> +import functools
> import os
> +import signal
> import sys
>
>
> +KILL_SIGNALS = (
> + signal.SIGINT,
> + signal.SIGTERM,
> + signal.SIGHUP,
> +)
> +
> +def forward_kill_signal(main_child_pid, signum, frame):
> + os.kill(main_child_pid, signum)
> +
> +
> def main(argv):
> if len(argv) < 2:
> - return 'Usage: {} '.format(argv[0])
> - main_child_pid = int(argv[1])
> + return 'Usage: {} or
> argv0..'.format(argv[0]) +
> + if len(argv) == 2:
> + # The child process is init (pid 1) in a child pid
> namespace, and
> + # the current process supervises from within the
> global pid namespace
> + # (forwarding signals to init and forwarding exit
> status to the parent
> + # process).
> + main_child_pid = int(argv[1])
> + else:
> + # The current process is init (pid 1) in a child pid
> namespace.
> + binary = argv[1]
> + args = argv[2:]
> +
> + main_child_pid = os.fork()
> + if main_child_pid == 0:
> + os.execv(binary, args)
> +
> + sig_handler = functools.partial(forward_kill_signal,
> main_child_pid)
> + for signum in KILL_SIGNALS:
> + signal.signal(signum, sig_handler)
>
> # wait for child processes
> while True:
> - pid, status = os.wait()
> + try:
> + pid, status = os.wait()
> + except OSError as e:
> + if e.errno == errno.EINTR:
> + continue
> + raise
> if pid == main_child_pid:
> if os.WIFEXITED(status):
> return os.WEXITSTATUS(status)
> elif os.WIFSIGNALED(status):
> + signal.signal(os.WTERMSIG(status),
> signal.SIG_DFL) os.kill(os.getpid(), os.WTERMSIG(status))
> # go to the unreachable place
> break
> diff --git a/lib/portage/process.py b/lib/portage/process.py
> index 7103b6b31..3e07f806c 100644
> --- a/lib/portage/process.py
> +++ b/lib/portage/process.py
> @@ -564,15 +564,27 @@ def _exec(binary, mycommand, opt_name, fd_pipes,
> noiselevel=-1)
> else:
> if unshare_pid:
> - # pid
> namespace requires us to become init
> - fork_ret =
> os.fork()
> - if
> fork_ret != 0:
> -
> os.execv(portage._python_interpreter, [
> +
> main_child_pid = os.fork()
> + if
> main_child_pid == 0:
> + #
> pid namespace requires us to become init
> +
> binary, myargs = portage._python_interpreter, [
> +
> portage._python_interpreter,
> +
> os.path.join(portage._bin_path,
> +
> 'pid-ns-init')]
> + [binary] + myargs
> + else:
> + #
> Execute a supervisor process which will forward
> + #
> signals to init and forward exit status to the
> + #
> parent process. The supervisor process runs in
> + #
> the global pid namespace, so skip /proc remount
> + #
> and other setup that's intended only for the
> + #
> init process.
> +
>
Re: [gentoo-portage-dev] [PATCH] portage.process.spawn: inherit env by default (bug 672440)
On Sun, 2 Dec 2018 23:55:23 -0800
Zac Medico wrote:
> Make child processes inherit the current process's environment by
> default, so that behavior is equivalent to the standard library's
> subprocess module.
>
> Bug: https://bugs.gentoo.org/672440
> Signed-off-by: Zac Medico
> ---
> lib/portage/process.py | 10 +++---
> 1 file changed, 7 insertions(+), 3 deletions(-)
>
> diff --git a/lib/portage/process.py b/lib/portage/process.py
> index ed1a49247..ce3e42a8f 100644
> --- a/lib/portage/process.py
> +++ b/lib/portage/process.py
> @@ -219,7 +219,7 @@ spawned_pids = _dummy_list()
> def cleanup():
> pass
>
> -def spawn(mycommand, env={}, opt_name=None, fd_pipes=None,
> returnpid=False, +def spawn(mycommand, env=None, opt_name=None,
> fd_pipes=None, returnpid=False, uid=None, gid=None, groups=None,
> umask=None, cwd=None, logfile=None, path_lookup=True, pre_exec=None,
>close_fds=(sys.version_info < (3, 4)), unshare_net=False,
> @@ -230,8 +230,10 @@ def spawn(mycommand, env={}, opt_name=None,
> fd_pipes=None, returnpid=False,
> @param mycommand: the command to execute
> @type mycommand: String or List (Popen style list)
> - @param env: A dict of Key=Value pairs for env variables
> - @type env: Dictionary
> + @param env: If env is not None, it must be a mapping that
> defines the environment
> + variables for the new process; these are used
> instead of the default behavior
> + of inheriting the current process's environment.
> + @type env: None or Mapping
> @param opt_name: an optional name for the spawn'd process
> (defaults to the binary name) @type opt_name: String
> @param fd_pipes: A dict of mapping for pipes, { '0': stdin,
> '1': stdout } for example @@ -281,6 +283,8 @@ def spawn(mycommand,
> env={}, opt_name=None, fd_pipes=None, returnpid=False, if
> isinstance(mycommand, basestring): mycommand = mycommand.split()
>
> + env = os.environ if env is None else env
> +
> if sys.hexversion < 0x300:
> # Avoid a potential UnicodeEncodeError from
> os.execve(). env_bytes = {}
LGTM
Re: [gentoo-portage-dev] [PATCH 0/2] git: drop privileges for gc and merge (bug 669496)
On Fri, 23 Nov 2018 16:15:23 -0800
Zac Medico wrote:
> Use portage.process.spawn (with new cwd parameter) and
> self.spawn_kwargs to drop privileges for git gc and merge commands.
>
> Fixes: 3cd8cf93abb6 ("GitSync: abort checkout for signature problem
> (bug 660372)") Fixes: 903c4b1a6768 ("GitSync: support sync-depth (bug
> 552814)") Bug: https://bugs.gentoo.org/669496
> Signed-off-by: Zac Medico
>
> Zac Medico (2):
> portage.process.spawn: add cwd parameter
> git: drop privileges for gc and merge (bug 669496)
>
> lib/portage/process.py| 13 ++---
> lib/portage/sync/modules/git/git.py | 10 ++
> lib/portage/tests/sync/test_sync_local.py | 22 --
> 3 files changed, 36 insertions(+), 9 deletions(-)
>
This looks good to me :)
Re: [gentoo-portage-dev] [PATCH] phase-helpers.sh: has/best_version -b in any phase (bug 665038)
On Sun, 2 Sep 2018 15:03:28 -0700
Zac Medico wrote:
> Since BROOT is only defined in src_* phases, make has/best_version -b
> use the equivalent /${PORTAGE_OVERRIDE_EPREFIX#/} value that is
> defined in all phases. This makes has/best_version -b in EAPI 7
> behave exactly the same as --host-root behaves in EAPI 6, allowing
> python ebuilds to call python_setup in any ebuild phase.
>
> Bug: https://bugs.gentoo.org/665038
> ---
> bin/phase-helpers.sh | 7 ++-
> 1 file changed, 6 insertions(+), 1 deletion(-)
>
> diff --git a/bin/phase-helpers.sh b/bin/phase-helpers.sh
> index 5c9f957e9..ba3f27930 100644
> --- a/bin/phase-helpers.sh
> +++ b/bin/phase-helpers.sh
> @@ -916,7 +916,12 @@ ___best_version_and_has_version_common() {
> case ${root_arg} in
> -r)
> root=${ROOT%/}/${EPREFIX#/} ;; -d) root=${ESYSROOT} ;;
> - -b) root=${BROOT:-/} ;;
> + -b)
> + #
> Use /${PORTAGE_OVERRIDE_EPREFIX#/} which is equivalent
> + # to BROOT, except
> BROOT is only defined in src_* phases.
> +
> root=/${PORTAGE_OVERRIDE_EPREFIX#/}
> + cmd+=(env
> EPREFIX="${PORTAGE_OVERRIDE_EPREFIX}")
> + ;;
> esac
> else
> case ${root_arg} in
> --
> 2.16.4
>
>
LGTM
Re: [gentoo-portage-dev] [PATCH 0/4] Add sync-rcu support for rsync (bug 662070)
On Mon, 6 Aug 2018 00:40:29 -0700 Zac Medico wrote: > Add a boolean sync-rcu repos.conf setting that behaves as follows: > > sync-rcu = yes|no > > Enable read-copy-update (RCU) behavior for sync operations. The > current latest immutable version of a repository will be > referenced by a symlink found where the repository would normally be > located (see the location setting). Repository consumers should > resolve the cannonical path of this symlink before attempt to access > the repository, and all operations should be read-only, since > the repository is considered immutable. Updates occur by atomic > replacement of the symlink, which causes new consumers to use the > new immutable version, while any earlier consumers continue to > use the cannonical path that was resolved earlier. This option > requires sync-allow-hardlinks and sync-rcu-store-dir options to > be enabled, and currently also requires that sync-type is set > to rsync. This option is disabled by default, since the symlink > usage would require special handling for scenarios involving bind > mounts and chroots. > > sync-rcu-store-dir > > Directory path reserved for sync-rcu storage. This directory must > have a unique value for each repository (do not set it in the > DEFAULT section). This directory must not contain any other files > or directories aside from those that are created automatically > when sync-rcu is enabled. > > sync-rcu-spare-snapshots = 1 > > Number of spare snapshots for sync-rcu to retain with expired > ttl. This protects the previous latest snapshot from being removed > immediately after a new version becomes available, since it might > still be used by running processes. > > sync-rcu-ttl-days = 7 > > Number of days for sync-rcu to retain previous immutable snapshots > of a repository. After the ttl of a particular snapshot has > expired, it will be remove automatically (the latest snapshot > is exempt, and sync-rcu-spare-snapshots configures the number of > previous snapshots that are exempt). If the ttl is set too low, > then a snapshot could expire while it is in use by a running > process. > > Zac Medico (4): > Implement asyncio.iscoroutinefunction for compat_coroutine > Add _sync_decorator module > rsync: split out repo storage framework > Add sync-rcu support for rsync (bug 662070) > > lib/portage/repository/config.py | 36 ++- > lib/portage/repository/storage/__init__.py | 0 > .../repository/storage/hardlink_quarantine.py | 95 > lib/portage/repository/storage/hardlink_rcu.py | 251 > + > lib/portage/repository/storage/inplace.py | 49 > lib/portage/repository/storage/interface.py| 87 +++ > lib/portage/sync/controller.py | 1 + > lib/portage/sync/modules/rsync/rsync.py| 85 ++- > lib/portage/sync/syncbase.py | 33 > +++ .../tests/util/futures/test_compat_coroutine.py| 14 ++ > lib/portage/util/futures/_asyncio/__init__.py | 14 ++ > lib/portage/util/futures/_sync_decorator.py| 54 + > lib/portage/util/futures/compat_coroutine.py | 12 + > man/portage.5 | 35 +++ 14 files > changed, 700 insertions(+), 66 deletions(-) create mode 100644 > lib/portage/repository/storage/__init__.py create mode 100644 > lib/portage/repository/storage/hardlink_quarantine.py create mode > 100644 lib/portage/repository/storage/hardlink_rcu.py create mode > 100644 lib/portage/repository/storage/inplace.py create mode 100644 > lib/portage/repository/storage/interface.py create mode 100644 > lib/portage/util/futures/_sync_decorator.py > series looks good, just the one typo
Re: [gentoo-portage-dev] [PATCH 3/4] rsync: split out repo storage framework
On Mon, 6 Aug 2018 00:40:32 -0700 Zac Medico wrote: > Since there aremany ways to manage repository storage, split out a > repo storage framework. The HardlinkQuarantineRepoStorage class > implements the existing default behavior, and the InplaceRepoStorage > class implements the legacy behavior (when sync-allow-hardlinks is > disabled in repos.conf). > > Each class implements RepoStorageInterface, which uses coroutine > methods since coroutines are well-suited to the I/O bound tasks that > these methods perform. The _sync_decorator is used to convert > coroutine methods to synchronous methods, for smooth integration into > the surrounding synchronous code. > > Bug: https://bugs.gentoo.org/662070 > missing space in first line of commit message s/aremany/are many ---
Re: [gentoo-portage-dev] [PATCH] XARGS: use gxargs for USERLAND=BSD (bug 663256)
On Thu, 9 Aug 2018 16:04:42 -0700
Zac Medico wrote:
> For USERLAND=BSD, set XARGS="gxargs -r" if gxargs is available,
> so the code from bug 630292 works for USERLAND=BSD.
>
> Fixes: 50283f1abb77 (install-qa-check.d/60pngfix: parallel support
> (bug 630292)) Reported-by: Michał Górny
> Bug: https://bugs.gentoo.org/663256
> ---
> bin/isolated-functions.sh | 6 +-
> 1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/bin/isolated-functions.sh b/bin/isolated-functions.sh
> index 28ca94532..cac42a4c5 100644
> --- a/bin/isolated-functions.sh
> +++ b/bin/isolated-functions.sh
> @@ -448,7 +448,11 @@ fi
> if [[ -z ${XARGS} ]] ; then
> case ${USERLAND} in
> BSD)
> - export XARGS="xargs"
> + if type -P gxargs > /dev/null; then
> + export XARGS="gxargs -r"
> + else
> + export XARGS="xargs"
> + fi
> ;;
> *)
> export XARGS="xargs -r"
LGTM
Re: [gentoo-portage-dev] [PATCH 2/2 v2] Update /usr/portage references (bug 378603)
On Sun, 5 Aug 2018 22:32:34 -0700
Zac Medico wrote:
> Update all relevant references in docs, messages, and comments
> to refer to /var/db/repos/gentoo instead of /usr/portage. Also
> update DISTDIR and PKGDIR references to refer to the new
> /var/cache/{distfiles,binpkgs} locations.
>
> Bug: https://bugs.gentoo.org/378603
> ---
> [PATCH 2/2 v2] fixes DISTDIR and PKGDIR references to rever to the
> new /var/cache/{distfiles,binpkgs} locations
>
> cnf/make.conf.example| 12 +--
> lib/portage/__init__.py | 2 +-
> lib/portage/cache/flat_hash.py | 2 +-
> lib/portage/tests/news/test_NewsItem.py | 2 +-
> lib/portage/tests/resolver/ResolverPlayground.py | 2 +-
> lib/portage/xml/metadata.py | 4 ++--
> man/ebuild.5 | 4 ++--
> man/emerge.1 | 6 +++---
> man/make.conf.5 | 20
> +- man/portage.5|
> 26
> man/quickpkg.1 | 2 +-
> repoman/lib/repoman/__init__.py | 2 +-
> repoman/lib/repoman/checks/herds/herdbase.py | 2 +- 13 files
> changed, 43 insertions(+), 43 deletions(-)
>
> diff --git a/cnf/make.conf.example b/cnf/make.conf.example
> index 04f3a0274..c16f2afbd 100644
> --- a/cnf/make.conf.example
> +++ b/cnf/make.conf.example
> @@ -14,7 +14,7 @@
> # https://wiki.gentoo.org/wiki/Handbook:X86/Working/USE
> #
> # The available list of use flags with descriptions is in your
> portage tree. -# Use 'less' to view them: -->
> less /usr/portage/profiles/use.desc <-- +# Use 'less' to view them:
> --> less /var/db/repos/gentoo/profiles/use.desc <-- #
> # 'ufed' is an ncurses/dialog interface available in portage to make
> handling # useflags for you. 'emerge app-portage/ufed'
> @@ -111,7 +111,7 @@
> # will protect the default locations of DISTDIR and PKGDIR, but
> users are # warned that any other locations inside PORTDIR are
> not necessarily safe # for data storage.
> -#PORTDIR=/usr/portage
> +#PORTDIR=/var/db/repos/gentoo
> #
> # DISTDIR is where all of the source code tarballs will be placed for
> # emerges. After packages are built, it is safe to remove any and
> @@ -119,16 +119,16 @@
> # fetched on demand for a given build. If you would like to
> # selectively prune obsolete files from this directory, see
> # eclean from the gentoolkit package. Note that locations under
> -# /usr/portage are not necessarily safe for data storage. See the
> +# /var/db/repos/gentoo are not necessarily safe for data
> storage. See the # PORTDIR documentation for more information.
> -#DISTDIR=/usr/portage/distfiles
> +#DISTDIR=/var/cache/distfiles
> #
> # PKGDIR is the location of binary packages that you can have created
> # with '--buildpkg' or '-b' while emerging a package. This can
> get # up to several hundred megs, or even a few gigs. Note that
> -# locations under /usr/portage are not necessarily safe for data
> +# locations under /var/db/repos/gentoo are not necessarily safe
> for data # storage. See the PORTDIR documentation for more
> information. -#PKGDIR=/usr/portage/packages
> +#PKGDIR=/var/cache/binpkgs
> #
> # PORT_LOGDIR is the location where portage will store all the logs
> it # creates from each individual merge. They are stored as
> diff --git a/lib/portage/__init__.py b/lib/portage/__init__.py
> index 166bfc700..61a240100 100644
> --- a/lib/portage/__init__.py
> +++ b/lib/portage/__init__.py
> @@ -133,7 +133,7 @@ except ImportError as e:
> sys.stderr.write("!!! Failed to complete portage imports.
> There are internal modules for\n") sys.stderr.write("!!! portage and
> failure here indicates that you have a problem with your\n")
> sys.stderr.write("!!! installation of portage. Please try a rescue
> portage located in the\n")
> - sys.stderr.write("!!! portage tree under
> '/usr/portage/sys-apps/portage/files/' (default).\n")
> + sys.stderr.write("!!! portage tree under
> '/var/db/repos/gentoo/sys-apps/portage/files/' (default).\n")
This should also get updated replacing "portage tree" with "gentoo
tree"
...
> The packages, after being created, will be placed in \fBPKGDIR\fR.
> This variable is defined in \fBmake.conf\fR(5) and defaults to
> -/usr/portage/packages.
> +/var/cache/binpkgs.
> .SH OPTIONS
> .TP
> .B
> diff --git a/repoman/lib/repoman/__init__.py
> b/repoman/lib/repoman/__init__.py index 89779b95c..d1312a267 100644
> --- a/repoman/lib/repoman/__init__.py
> +++ b/repoman/lib/repoman/__init__.py
> @@ -14,7 +14,7 @@ except ImportError as e:
> sys.stderr.write("!!! Failed to complete portage imports.
> There are internal modules for\n") sys.stderr.write("!!! portage and
> failure here indicates that you have a problem with your\n")
>
Re: [gentoo-portage-dev] [PATCH 2/2] Update /usr/portage references (bug 378603)
On Sun, 5 Aug 2018 22:46:58 -0700
Zac Medico wrote:
> On 08/05/2018 09:59 PM, Ulrich Mueller wrote:
> >> On Sun, 5 Aug 2018, Zac Medico wrote:
> >
> >> --- a/cnf/make.conf.example
> >> +++ b/cnf/make.conf.example
> >> [...]
> >
> >> @@ -119,16 +119,16 @@
> >> # fetched on demand for a given build. If you would like to
> >> # selectively prune obsolete files from this directory, see
> >> # eclean from the gentoolkit package. Note that locations
> >> under -# /usr/portage are not necessarily safe for data
> >> storage. See the +# /var/db/repos/gentoo are not necessarily
> >> safe for data storage. See the # PORTDIR documentation for
> >> more information. -#DISTDIR=/usr/portage/distfiles
> >> +#DISTDIR=/var/db/repos/gentoo/distfiles
> >
> > Shouldn't this be /var/cache/distfiles ...
> >
> >> #
> >> # PKGDIR is the location of binary packages that you can have
> >> created # with '--buildpkg' or '-b' while emerging a package.
> >> This can get # up to several hundred megs, or even a few gigs.
> >> Note that -# locations under /usr/portage are not necessarily
> >> safe for data +# locations under /var/db/repos/gentoo are not
> >> necessarily safe for data # storage. See the PORTDIR
> >> documentation for more information. -#PKGDIR=/usr/portage/packages
> >> +#PKGDIR=/var/db/repos/gentoo/packages
> >
> > ... and /var/cache/binpkgs?
>
> Thanks, I've fixed the /var/cache/{distfiles,binpkgs} locations in v2.
>
> >> --- a/lib/portage/cache/flat_hash.py
> >> +++ b/lib/portage/cache/flat_hash.py
> >> @@ -144,7 +144,7 @@ class database(fs_template.FsBased):
> >># Only recurse 1 deep, in
> >> order to avoid iteration over # entries from another nested cache
> >> instance. This can # happen if the user nests an overlay inside
> >> - # /usr/portage/local as
> >> in bug #302764.
> >> +
> >> # /var/db/repos/gentoo/local as in bug #302764.
> >
> > Shouldn't a local overlay be in /var/db/repos/local, but never
> > inside of the gentoo repo?
>
> Yes, but for a long time /usr/local/portage was somewhat standard, in
> fact it's still mentioned here:
>
> https://wiki.gentoo.org/wiki/Custom_repository
>
> Nowadays, repository verification will prevent that from working...
>
But that too can be changed along with all the user install
documentation which will need to be updated as well. The new
recomended location should be /var/db/repos/local. I will be updating
layman for /var/db/repos/... as well. That is the intention of the
"repos/" subdir.
All these changes as well as the catlayst changes need to be
co-ordinated so that snapshots and portage and stages don't precede the
docs changes.
> >> --- a/man/make.conf.5
> >> +++ b/man/make.conf.5
> >> @@ -219,10 +219,10 @@ Use the \fBPORTAGE_RO_DISTDIRS\fR variable
> >> to specify one or more read-only directories containing distfiles.
> >>
> >> Note
> >> -that locations under /usr/portage are not necessarily safe for
> >> data storage. +that locations under /var/db/repos/gentoo are not
> >> necessarily safe for data storage.
> >
> > IMHO this statement should be made much stronger.
>
> Yes, we could also say something about repository verification here.
> The old defaults set a really bad example, so we needed statements
> like this to steer people in the right direction.
> --
> Thanks,
> Zac
>
pgpT1yfKTqhPZ.pgp
Description: OpenPGP digital signature
Re: [gentoo-portage-dev] [PATCH] repoman: populate implicit IUSE for empty profile (bug 660982)
On Sun, 15 Jul 2018 16:02:03 -0700 Zac Medico wrote: > For the empty profile that's used to check dependencies of > ebuilds that have empty KEYWORDS, populate implicit IUSE > from all of the make.defaults files found in the relevant > repositories (this should take less than 1 second on most > hardware). Since the IUSE.missing check cannot be performed > without implicit IUSE settings, this makes the IUSE.missing > check work for ebuilds with empty KEYWORDS. > > Bug: https://bugs.gentoo.org/660982 > --- > pym/portage/dbapi/__init__.py | 10 +++- > repoman/pym/repoman/modules/scan/depend/profile.py | 30 > +- 2 files changed, 32 insertions(+), 8 > deletions(-) > > diff --git a/pym/portage/dbapi/__init__.py > b/pym/portage/dbapi/__init__.py index 61d301839..6fca6090c 100644 > --- a/pym/portage/dbapi/__init__.py > +++ b/pym/portage/dbapi/__init__.py > @@ -219,17 +219,13 @@ class dbapi(object): > def _repoman_iuse_implicit_cnstr(self, pkg, metadata): > """ > In repoman's version of _iuse_implicit_cnstr, > account for modifications > - of the self.settings reference between calls, and > treat all flags as > - valid for the empty profile because it does not have > any implicit IUSE > - settings. See bug 660982. > + of the self.settings reference between calls. > """ > eapi_attrs = _get_eapi_attrs(metadata["EAPI"]) > if eapi_attrs.iuse_effective: > - iuse_implicit_match = lambda flag: (True if > not self.settings.profile_path > - else > self.settings._iuse_effective_match(flag)) > + iuse_implicit_match = lambda flag: > self.settings._iuse_effective_match(flag) else: > - iuse_implicit_match = lambda flag: (True if > not self.settings.profile_path > - else > self.settings._iuse_implicit_match(flag)) > + iuse_implicit_match = lambda flag: > self.settings._iuse_implicit_match(flag) return iuse_implicit_match > > def _iuse_implicit_cnstr(self, pkg, metadata): > diff --git a/repoman/pym/repoman/modules/scan/depend/profile.py > b/repoman/pym/repoman/modules/scan/depend/profile.py index > 8e0a22f9c..233ed8e4b 100644 --- > a/repoman/pym/repoman/modules/scan/depend/profile.py +++ > b/repoman/pym/repoman/modules/scan/depend/profile.py @@ -2,6 +2,7 @@ > > > import copy > +import os > from pprint import pformat > > from _emerge.Package import Package > @@ -12,7 +13,8 @@ from repoman.modules.scan.scanbase import ScanBase > from repoman.modules.scan.depend._depend_checks import _depend_checks > from repoman.modules.scan.depend._gen_arches import _gen_arches > from portage.dep import Atom > - > +from portage.package.ebuild.profile_iuse import iter_iuse_vars > +from portage.util import getconfig > > def sort_key(item): > return item[2].sub_path > @@ -102,6 +104,10 @@ class ProfileDependsChecks(ScanBase): > local_config=False, > > _unmatched_removal=self.options.unmatched_removal, > env=self.env, > repositories=self.repo_settings.repoman_settings.repositories) + > + if not prof.abs_path: > + > self._populate_implicit_iuse(dep_settings) + > dep_settings.categories = > self.repo_settings.repoman_settings.categories if > self.options.without_mask: dep_settings._mask_manager_obj = \ > @@ -257,3 +263,25 @@ class ProfileDependsChecks(ScanBase): > def runInEbuilds(self): > '''Ebuild level scans''' > return (True, [self.check]) > + > + @staticmethod > + def _populate_implicit_iuse(config): > + """ > + Populate implicit IUSE for the empty profile, see > bug 660982. > + """ > + dest = config.configdict['defaults'] > + for repo in config.repositories: > + for parent_dir, dirs, files in > os.walk(os.path.join(repo.location, 'profiles')): > + src = > getconfig(os.path.join(parent_dir, 'make.defaults')) > + if not src: > + continue > + for k, v in iter_iuse_vars(src): > + v_before = dest.get(k) > + if v_before is not None: > + merged_values = > set(v_before.split()) > + > merged_values.update(v.split()) > + v = ' > '.join(sorted(merged_values)) > + dest[k] = v > + > + config.regenerate() > + config._init_iuse() looks good
Re: [gentoo-portage-dev] [PATCH] RepoConfig: fix make.conf PORTDIR override (bug 661276)
On Sun, 15 Jul 2018 14:09:24 -0700 Zac Medico wrote: > Pass the main-repo name from repos.conf as the repository name > for the PORTDIR RepoConfig constructor, so that it can override > the main repo location even if the profiles/repo_name file does > not exist yet (like in a stage3 tarball). > > Bug: https://bugs.gentoo.org/661276 > Reported-by: Jorge Manuel B. S. Vicetto > --- > pym/portage/repository/config.py | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/pym/portage/repository/config.py > b/pym/portage/repository/config.py index bf2b6dd03..e45e67dec 100644 > --- a/pym/portage/repository/config.py > +++ b/pym/portage/repository/config.py > @@ -524,7 +524,8 @@ class RepoConfigLoader(object): > (base_priority == 0 and ov > is portdir): repo_opts = default_repo_opts.copy() > repo_opts['location'] = ov > - repo = RepoConfig(None, > repo_opts, local_config=local_config) > + name = > prepos['DEFAULT'].main_repo if ov is portdir else None > + repo = RepoConfig(name, > repo_opts, local_config=local_config) # repos_conf_opts contains > options from repos.conf repos_conf_opts = repos_conf.get(repo.name) > if repos_conf_opts is not > None: looks good
Re: [gentoo-portage-dev] [PATCH 0/5] webrsync: support sync-openpgp-key-path (bug 661838)
On Fri, 27 Jul 2018 00:56:40 -0700 Zac Medico wrote: > Add repos.conf settings that enable sync-openpgp-key-path support for > both emerge-webrsync and emerge-delta-webrsync: > > sync-webrsync-delta = true|false > Use app-portage/emerge-delta-webrsync to minimize > bandwidth. Defaults to false. > > sync-webrsync-keep-snapshots = true|false > Keep snapshots in DISTDIR (do not delete). Defaults to false. > > sync-webrsync-verify-signature = true|false > Require the detached tarball signature to contain a good > OpenPGP signature. This uses the OpenPGP key(ring) specified by the > sync-openpgp-key-path setting. Defaults to false. > > Zac Medico (5): > emerge-webrsync: exit early for signature problem (bug 661838) > webrsync: support sync-openpgp-key-path (bug 661838) > webrsync: support sync-webrsync-keep-snapshots > emerge-delta-webrsync: exit early for signature problem (bug 661838) > webrsync: support emerge-delta-webrsync (bug 661838) > > bin/emerge-webrsync | 22 ++- > lib/portage/sync/modules/webrsync/__init__.py | 6 +- > lib/portage/sync/modules/webrsync/webrsync.py | 84 > --- > man/portage.5 | 12 > misc/emerge-delta-webrsync| 23 +++- 5 files > changed, 135 insertions(+), 12 deletions(-) > this series looks good. I'm also surprised to see nearly all the changes were on the python side. Barely any changes to the bash scripts.
Re: [gentoo-dev] rfc: moving default location of portage tree (was: [gentoo-project] Call for agenda items - Council meeting 2018-07-29)
On Fri, 27 Jul 2018 16:31:15 +0200 Ulrich Mueller wrote: > > On Fri, 27 Jul 2018, Corentin “Nado” Pazdera wrote: > > > July 27, 2018 4:07 PM, "William Hubbs" > > wrote: > > >> Section 5.5.2 describes the directory structure of /var/cache. > >> These paths are all optional [1]. > >> > >> /var/cache/fonts > >> /var/cache/man > >> /var/cache/www > >> /var/cache/ > >> > >> Gentoo isn't a package, so I don't think /var/cache/gentoo/* is > >> appropriate. Here is my proposal: > >> > >> /usr/portage -> /var/db/repos/gentoo > >> /usr/portage/distfiles -> /var/cache/portage/distfiles > >> /usr/portage/packages -> /var/cache/portage/binpkgs > >> > >> I'm not 100% comfortable with /var/db, but I don't have any better > >> suggestion either. > >> > >> [1] > >> http://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.html#varcacheApplicationCacheData > >> > > > From the same source > > "No other requirements are made on the data format of the cache > > directories." > > And as you have quoted it, everything under /var/cache is > > optional. > > > So anything which doesn't conflict with another package seems fine > > according to FHS. > > That's how I would read it, too. We could of course invent a package > name (like "package-manager" for virtual/package-manager) but it seems > cumbersome, and I don't see any benefit of it. > > There also is /var/cache/fonts, so the FHS itself lists an example of > a directory that's not named after a specific package. > > Ulrich /var/db/repos/gentoo /var/cache/distfiles /var/cache/binpkgs Works for me, just please keep "portage" out of it, after all distfiles are not restricted to portage use only, and neither are binpkgs. There is alternate binpkg installers. pgpNgTs22e92A.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] Packages up for grabs: app-backup/fsarchiver, app-cdr/daa2iso...
On Thu, 19 Jul 2018 10:10:11 +0200 Michał Górny wrote: > Hello, > > Due to Markos Chandras' prolonged absence, the following packages are > up for grabs now: > > dev-util/buildbot-slave I've taken over all buildbot maintenance. This one will be tree-cleaned soon. The old buildbot-0.8 releases are long since dead upstream. I will begin that process once I stabilize one of the newer releases. pgpETlmIeVsOT.pgp Description: OpenPGP digital signature
Re: [gentoo-portage-dev] Objections to renaming pym directory to lib?
On Tue, 17 Jul 2018 13:28:05 -0700 Zac Medico wrote: > Are there any objections to renaming the pym directory to lib [1]? > Note that the git log --follow option makes this kind of rename > fairly painless. > > [1] https://github.com/gentoo/portage/pull/343 is fine with me pgpewJwnz8KUC.pgp Description: OpenPGP digital signature
Re: [gentoo-portage-dev] [PATCH] config: cache profile.bashrc stat results (bug 649806)
On Sat, 14 Jul 2018 22:05:55 -0700
Zac Medico wrote:
> Optimize config to stat profile.bashrc files once in the constructor,
> in order to avoid repeated stat calls in the setcpv method.
>
> Bug: https://bugs.gentoo.org/649806
> ---
> pym/portage/package/ebuild/config.py | 13 -
> 1 file changed, 8 insertions(+), 5 deletions(-)
>
> diff --git a/pym/portage/package/ebuild/config.py
> b/pym/portage/package/ebuild/config.py index 88acac5cc..320d9f6c0
> 100644 --- a/pym/portage/package/ebuild/config.py
> +++ b/pym/portage/package/ebuild/config.py
> @@ -274,6 +274,7 @@ class config(object):
> self.mycpv = clone.mycpv
> self._setcpv_args_hash =
> clone._setcpv_args_hash self._soname_provided = clone._soname_provided
> + self._profile_bashrc = clone._profile_bashrc
>
> # immutable attributes (internal policy
> ensures lack of mutation) self._locations_manager =
> clone._locations_manager @@ -725,6 +726,10 @@ class config(object):
> self._license_manager.extract_global_changes( \
>
> self.configdict["conf"].get("ACCEPT_LICENSE",
> ""))
> + # profile.bashrc
> + self._profile_bashrc =
> tuple(os.path.isfile(os.path.join(profile.location, 'profile.bashrc'))
> + for profile in profiles_complex)
> +
> if local_config:
> #package.properties
> propdict =
> grabdict_package(os.path.join( @@ -1596,11 +1601,9 @@ class
> config(object):
> bashrc_files = []
>
> - for profile in
> self._locations_manager.profiles_complex:
> - profile_bashrc =
> os.path.join(profile.location,
> - 'profile.bashrc')
> - if os.path.exists(profile_bashrc):
> - bashrc_files.append(profile_bashrc)
> + for profile, profile_bashrc in
> zip(self._locations_manager.profiles_complex, self._profile_bashrc):
> + if profile_bashrc:
> +
> bashrc_files.append(os.path.join(profile.location, 'profile.bashrc'))
> if profile in self._pbashrcdict: cpdict =
> self._pbashrcdict[profile].get(cp) if cpdict:
looks good
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] portdbapi.cp_list: cache repo associations (bug 650814)
# assume it's iterable
> - mytrees = mytree
> + repos =
> [self.repositories.get_repo_for_location(location)
> + for location in mytree]
> elif self._better_cache is None:
> - mytrees = self.porttrees
> + repos = list(self.repositories)
> else:
> - mytrees = [repo.location for repo in
> self._better_cache[mycp]]
> - for oroot in mytrees:
> + repos = reversed(self._better_cache[mycp])
> + mylist = []
> + for repo in repos:
> + oroot = repo.location
> try:
> file_list =
> os.listdir(os.path.join(oroot, mycp)) except OSError:
> @@ -986,16 +1002,17 @@ class portdbapi(dbapi):
> writemsg(_("\nInvalid
> ebuild version: %s\n") % \ os.path.join(oroot, mycp, x),
> noiselevel=-1) continue
> -
> d[_pkg_str(mysplit[0]+"/"+pf, db=self)] = None
> - if invalid_category and d:
> +
> mylist.append(_pkg_str(mysplit[0]+"/"+pf, db=self, repo=repo.name))
> + if invalid_category and mylist:
> writemsg(_("\n!!! '%s' has a category that
> is not listed in " \ "%setc/portage/categories\n") % \
> (mycp,
> self.settings["PORTAGE_CONFIGROOT"]), noiselevel=-1) mylist = []
> - else:
> - mylist = list(d)
> - # Always sort in ascending order here since it's
> handy
> - # and the result can be easily cached and reused.
> + # Always sort in ascending order here since it's
> handy and
> + # the result can be easily cached and reused. Since
> mylist
> + # is initially in ascending order by repo.priority,
> stable
> + # sort by version produces results in ascending
> order by
> + # (pkg.version, repo.priority).
> self._cpv_sort_ascending(mylist)
> if self.frozen and mytree is None:
> cachelist = mylist[:]
looks fine
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] _unmerge_dirs: revisit parents of removed symlinks (bug 640058)
On Thu, 12 Jul 2018 21:09:46 -0700
Zac Medico wrote:
> When removal of a symlink is triggered by removal of the directory
> that it points to, revisit the parent directories of the symlink.
>
> Bug: https://bugs.gentoo.org/640058
> ---
> pym/portage/dbapi/vartree.py | 23 +--
> 1 file changed, 21 insertions(+), 2 deletions(-)
>
> diff --git a/pym/portage/dbapi/vartree.py
> b/pym/portage/dbapi/vartree.py index 1a86940f1..43e3c4f1a 100644
> --- a/pym/portage/dbapi/vartree.py
> +++ b/pym/portage/dbapi/vartree.py
> @@ -2753,9 +2753,13 @@ class dblink(object):
> real_root = self.settings['ROOT']
>
> dirs = sorted(dirs)
> - dirs.reverse()
> + revisit = {}
>
> - for obj, inode_key in dirs:
> + while True:
> + try:
> + obj, inode_key = dirs.pop()
> + except IndexError:
> + break
> # Treat any directory named "info" as a
> candidate here, # since it might have been in INFOPATH previously even
> # though it may not be there now.
> @@ -2818,6 +2822,7 @@ class dblink(object):
> raise
> if e.errno != errno.ENOENT:
> show_unmerge("---",
> unmerge_desc["!empty"], "dir", obj)
> + revisit[obj] = inode_key
>
> # Since we didn't remove this
> directory, record the directory # itself for use in syncfs calls, if
> we have removed another @@ -2838,6 +2843,7 @@ class dblink(object):
> # no need to protect symlinks that
> point to it. unmerge_syms = protected_symlinks.pop(inode_key, None)
> if unmerge_syms is not None:
> + parents = []
> for relative_path in
> unmerge_syms: obj = os.path.join(real_root,
>
> relative_path.lstrip(os.sep))
> @@ -2849,6 +2855,19 @@ class dblink(object):
> raise
> del e
> show_unmerge("!!!",
> "", "sym", obj)
> + else:
> +
> parents.append(os.path.dirname(obj)) +
> + if parents:
> + # Revisit parents
> recursively (bug 640058).
> + recursive_parents =
> []
> + for parent in
> set(parents):
> + while parent
> in revisit:
> +
> recursive_parents.append(parent)
> +
> parent = os.path.dirname(parent) +
> + for parent in
> sorted(set(recursive_parents)):
> +
> dirs.append((parent, revisit.pop(parent)))
> def isowner(self, filename, destroot=None):
> """
LGTM
--
Brian Dolbec
Re: [gentoo-dev] rfc: moving default location of portage tree
On Thu, 12 Jul 2018 17:35:41 -0700 Raymond Jennings wrote: > On Thu, Jul 12, 2018 at 12:47 PM Brian Dolbec > wrote: > > > > On Thu, 12 Jul 2018 11:49:37 -0700 > > Raymond Jennings wrote: > > > > > In that case, I vote for /var/cache/portage, since that's > > > literally what purpose it serves. Namely, the cache of the > > > gentoo infra's current copy of teh portage tree. > > > > > > On Thu, Jul 12, 2018 at 11:00 AM Alec Warner > > > wrote: > > > > > > > > On Thu, Jul 12, 2018 at 1:43 PM, Raymond Jennings > > > > wrote: > > > >> > > > >> Just for the record, but would putting a setting inside > > > >> /etc/portage/make.conf be the appropriate way to handle this? > > > >> > > > > > > > > The settings already exist (and have existed for 10 years.) This > > > > bikeshed discussion is literally trying to decide what the > > > > default should be. > > > > > > > > -A > > > > > > > > > > > This is not a personal attack against you. Just picked this one to > > say something again... > > > > > > PLEASE, PLEASE stop calling it the "portage" tree. The repo name is > > "gentoo". "portage is the default package manager, but not the only > > choice. Far too often, it takes awhile to figure out what someone > > is trying to say because of that confusion between the tree and the > > package manager. > > Point of order: > > http://distfiles.gentoo.org/snapshots and numerous pieces of > documentation call it "portage" > > The confusion is ingrained by documentation. > Yes, it is, and well we can't very well change the documentation until we can get an end to this new default path bikeshed. Council will need to make the decision (soon I hope)... Then we make all the changes necessary. -- Brian Dolbec
Re: [gentoo-dev] rfc: moving default location of portage tree
On Thu, 12 Jul 2018 11:49:37 -0700 Raymond Jennings wrote: > In that case, I vote for /var/cache/portage, since that's literally > what purpose it serves. Namely, the cache of the gentoo infra's > current copy of teh portage tree. > > On Thu, Jul 12, 2018 at 11:00 AM Alec Warner > wrote: > > > > On Thu, Jul 12, 2018 at 1:43 PM, Raymond Jennings > > wrote: > >> > >> Just for the record, but would putting a setting inside > >> /etc/portage/make.conf be the appropriate way to handle this? > >> > > > > The settings already exist (and have existed for 10 years.) This > > bikeshed discussion is literally trying to decide what the default > > should be. > > > > -A > > > This is not a personal attack against you. Just picked this one to say something again... PLEASE, PLEASE stop calling it the "portage" tree. The repo name is "gentoo". "portage is the default package manager, but not the only choice. Far too often, it takes awhile to figure out what someone is trying to say because of that confusion between the tree and the package manager. PLUS, it has been decided already long ago that the directory name should reflect the repository name. We have been enforcing that rule for overlays for a long time. It has just been taking a long time to get our tooling in order so that we can change our own to follow that rule. So, "portage" should not be a directory name in the new default path. -- Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] dbapi: fix repoman implicit IUSE (bug 660982)
On Thu, 12 Jul 2018 02:59:03 -0700 Zac Medico wrote: > Account for repoman modifications of the portdbapi self.settings > reference, and treat all flags as valid for the empty profile > because it does not have any implicit IUSE settings. > > Bug: https://bugs.gentoo.org/660982 > --- > pym/_emerge/Package.py| 5 - > pym/portage/dbapi/__init__.py | 16 > 2 files changed, 20 insertions(+), 1 deletion(-) > > diff --git a/pym/_emerge/Package.py b/pym/_emerge/Package.py > index a7ce00bc9..5f34f3d27 100644 > --- a/pym/_emerge/Package.py > +++ b/pym/_emerge/Package.py > @@ -93,7 +93,10 @@ class Package(Task): > # sync metadata with validated repo (may be > UNKNOWN_REPO) self._metadata['repository'] = self.cpv.repo > > - implicit_match = db._iuse_implicit_cnstr(self.cpv, > self._metadata) > + if self.root_config.settings.local_config: > + implicit_match = > db._iuse_implicit_cnstr(self.cpv, self._metadata) > + else: > + implicit_match = > db._repoman_iuse_implicit_cnstr(self.cpv, self._metadata) usealiases > = self.root_config.settings._use_manager.getUseAliases(self) > self.iuse = self._iuse(self, self._metadata["IUSE"].split(), > implicit_match, usealiases, self.eapi) diff --git > a/pym/portage/dbapi/__init__.py b/pym/portage/dbapi/__init__.py index > d320cc75f..61d301839 100644 --- a/pym/portage/dbapi/__init__.py > +++ b/pym/portage/dbapi/__init__.py > @@ -216,6 +216,22 @@ class dbapi(object): > > yield cpv > > + def _repoman_iuse_implicit_cnstr(self, pkg, metadata): > + """ > + In repoman's version of _iuse_implicit_cnstr, > account for modifications > + of the self.settings reference between calls, and > treat all flags as > + valid for the empty profile because it does not have > any implicit IUSE > + settings. See bug 660982. > + """ > + eapi_attrs = _get_eapi_attrs(metadata["EAPI"]) > + if eapi_attrs.iuse_effective: > + iuse_implicit_match = lambda flag: (True if > not self.settings.profile_path > + else > self.settings._iuse_effective_match(flag)) > + else: > + iuse_implicit_match = lambda flag: (True if > not self.settings.profile_path > + else > self.settings._iuse_implicit_match(flag)) > + return iuse_implicit_match > + > def _iuse_implicit_cnstr(self, pkg, metadata): > """ > Construct a callable that checks if a given USE flag > should looks good thanks. Please add the test case ebuild that was supplied to the repoman gen-b0rk repo "not-broken" category. https://gitweb.gentoo.org/repo/proj/gen-b0rk.git/ -- Brian Dolbec
Re: [gentoo-dev] rfc: moving default location of portage tree
On Mon, 9 Jul 2018 12:21:36 -0500 William Hubbs wrote: > All, > > is there a tracker for when the portage tree can be moved out of > /usr/portage by default? > > If not, what is the status of us being able to do this? > > Thanks, > > William > I don't recall a tracker bug ever being created. It required the stages be generated by catalyst-3 which can be configured to any location for the tree defaults. catalyst-2 had paths hard-coded all over the place, plus used those paths as keys in python dictionaries... I believe all stages are now built with catalyst-3 for all arches, but I don't know about some of the lesser used arches as some of those are older dates. That and a portage release with the new default location set in it's backup configs. So, it should be ready to convert if the minor arches stage are being generated with catalyst-3 -- Brian Dolbec pgpJWo1O2DkxL.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] [PATCH v3 10/12] glep-0063: Make 2-yearly expiration term mandatory
On Fri, 06 Jul 2018 08:51:12 +0200 Michał Górny wrote: > W dniu pią, 06.07.2018 o godzinie 06∶28 +, użytkownik Robin H. > Johnson napisał: > > On Fri, Jul 06, 2018 at 08:18:32AM +0200, Michał Górny wrote: > > > > option a) > > > > 2 years + N: > > > > 2 weeks <= N <= 3 months. > > > > > > > > option b) > > > > Change the wording to be 'at most 2 years' instead of 'exactly > > > > 2 years'. > > > > > > That *is* the wording. > > > > I apologize. I took ulm's post as canonical and didn't confirm in > > the original GLEP text. > > > > Further change to follow in response to the original text. > > > > > > Separately: > > > > Is two weeks enough time for a new key distribution to users? > > > > > > I originally wanted to specify one month but k_f insisted on > > > something shorter. 2 weeks were the compromise we agreed on. > > > That said, I'd say weekly 'gpg --refresh' is what we should > > > recommend as the bare minimum. > > > > > > That said, the point of two weeks is mostly to give us time to > > > remind developers that their key is expiring and to give them > > > time to actually read their mail and do it before it actually > > > expires. > > > > Please let's start reminding them BEFORE that. I have seen a lot of > > .away files over the last decade, and taking a 2-week offline > > vacation does happen. > > The problem is, Gentoo developers are really hostile people. If you > remind them *before* the term, they are not very nice because how does > someone dare remind very important developer who was planning to do it > week before expiration, and now he needed to waste his precious time > reading your mail. > I never experienced a single developer show or say anything like you are suggesting. Most, thanked me for the reminder whether it was in IRC or email. And since I always cc'd the gkeys alias (most needed changes to meet the spec too), I'm sure Kristian will confirm this. -- Brian Dolbec pgp5aZWnCcqDB.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] [PATCH v3 10/12] glep-0063: Make 2-yearly expiration term mandatory
On Fri, 06 Jul 2018 08:18:32 +0200 Michał Górny wrote: > W dniu pią, 06.07.2018 o godzinie 06∶08 +, użytkownik Robin H. > Johnson napisał: > > On Fri, Jul 06, 2018 at 07:43:56AM +0200, Ulrich Mueller wrote: > > > > > > > > On Thu, 5 Jul 2018, Michał Górny wrote: > > > > Replace the disjoint 'minimum' and 'recommendation' for > > > > expiration with a single requirement. Make it 2 years. Also, > > > > remove disjoint expiration recommendation for the primary key > > > > and subkeys since many developers fail at implementing that > > > > anyway. > > > > > > Still NACK. If expiration is exactly 2 years and renewal must > > > happen 2 weeks before the expiry date, then it is not possible to > > > keep the same date. > > > > > > Example: The key will expire at 2018-12-31, so it must be renewed > > > at 2018-12-17 or earlier. This will make it impossible to keep > > > the same month and day (unless one would reset it to 2019-12-31, > > > which is only one year though). > > > > > > So please, make it something like 2 years + 3 months. > > > > option a) > > 2 years + N: > > 2 weeks <= N <= 3 months. > > > > option b) > > Change the wording to be 'at most 2 years' instead of 'exactly 2 > > years'. > > That *is* the wording. > > > Separately: > > Is two weeks enough time for a new key distribution to users? > > I originally wanted to specify one month but k_f insisted on something > shorter. 2 weeks were the compromise we agreed on. That said, I'd > say weekly 'gpg --refresh' is what we should recommend as the bare > minimum. > > That said, the point of two weeks is mostly to give us time to remind > developers that their key is expiring and to give them time to > actually read their mail and do it before it actually expires. > I have gkeys spec-check start warning at 30 days, and it has been my experience that often it only gets renewed last minute (depends on how active the developer is. As it is one of those things that gets put off thinking there is still lots of time... But also, many of those had keys that did not meet the spec requirements. -- Brian Dolbec pgphYyrmtOdFC.pgp Description: OpenPGP digital signature
[gentoo-dev] Lastrites: app-crypt/gkeys-gen
# Brian Dolbec (4 July 2018) # No longer needed, newer app-crypt/gnupg-2 versions have simple cli options # for generating new keys and doing key maintenance. Removal in a month app-crypt/gkeys-gen -- Brian Dolbec
Re: [gentoo-dev] rfc: killing mediawiki
On Wed, 4 Jul 2018 19:21:53 +1200 Kent Fredric wrote: > On Tue, 3 Jul 2018 12:39:43 -0500 > William Hubbs wrote: > > > I don't care that we have a wiki, but can we please look into > > killing mediawiki and look at something with a git backend? It > > would be very nice to be able to edit wiki pages in markdown or > > another similar format and use git to control the changes instead > > of editing in a browser. > > 1. Who are the primary beneficiaries of this suggestion?: > > a. People with expert knowledge of development systems and people > with Gentoo Privileges > > b. End users who may not be experts, in all things development, but > may be able to contribute and consume content. > > 2. What compromises in flexibility does this create? Eg: Do suggested >replacements have capacity to have arbitrary HTML and templating? > Or are they restricted to the terribly narrow featureset of Markdown? > > > If you're optimising for 1-a and your choice of compromise results in > a reduction in functionality with regards to clear, flexible, and > expressive content, it will be hard to sell me on the idea. > 3) The ability of someone with a sight disability, the actual ability to contribute to the project. We as a project should try make our documentation accessible to as broad an audience as possible. I know from personal experience and having worked with such a person, to make it possible to work with cli tools to be able to interact with normally browser only visual interfaces which are extremely poor to work with using a screen reader. At my work, I personally made a github python library and cli interface for github, and worked to create templates for a go-jira command line interface so a visually impaired co-worker could interact with those systems. -- Brian Dolbec pgp2ADbD_1YTy.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] rfc: why are we still distributing the portage tree via rsync?
On Tue, 3 Jul 2018 10:22:35 -0500 William Hubbs wrote: > All, > > Mostly because of the recent "trustless infrastructure" thread, I am > wondering why we are still distributing the portage tree primarily > via rsync instead of git? > > Can someone educate me on that, and is it worth considering moving > away from rsync distribution? > > Thanks, > > William > because: 1) it is still the most bandwidth economical means of distributing the tree 2) we have a large infrastructure of rsync mirrors, which we do not for git. 3) see #1 -- Brian Dolbec pgpjViOjx5GaR.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] Trustless Infrastructure
On Mon, 2 Jul 2018 11:01:58 -0500 R0b0t1 wrote: > On Mon, Jul 2, 2018 at 10:36 AM, Jason A. Donenfeld > wrote: > > Hey guys, > > > > While our infrastructure team has some nice technical competence, > > the recent disaster and ongoing embarrassing aftermath has made > > ever more urgent the need to have end-to-end signatures between > > developers and users. While the infrastructure team seems fairly > > impressive at deploying services and keeping the house running > > smoothly, I'd rather we don't place additional burden on them to do > > everything they're doing securely. Specifically, I'd like to ensure > > that 100% of Gentoo's infrastructure can be hacked, yet not > > backdoor a single witting user of the portage tree. Right now, as > > it stands, rsync distributes signatures to users that are derived > > from some infrastructure-controlled keys, not from the developers > > themselves. > > > > Proposal: > > - Sign every file in the portage tree so that it has a corresponding > > .asc. Repoman will need support for this. > > Signed hashes should be faster, no? Each directory with files could > have a manifest. > > > - Ensure the naming scheme of portage files is sufficiently strict, > > so that renaming or re-parenting signed files doesn't result in > > RCE. [*] > > - Distribute said .asc files with rsync per usual. > > Rsync would work with this setup, but there is also webrsync-gpg in > Portage right now. This covers the vast majority of usecases right > now. There is often no need to sync more than once per day. > > Speaking of, the keys for that have lapsed. Will they be updated? > > Cheers, > R0b0t1 > app-crypt/gentoo-keys has been updated yesterday. with renewed interest in gpg signatures, I will endeavor to keep it updated until is fully automated. -- Brian Dolbec
Re: [gentoo-dev] Suggestions for simplifying VIDEO_CARDS situation
On Sat, 16 Jun 2018 21:40:10 -0700 Matt Turner wrote: > Hello, > > VIDEO_CARDS is an annoying mess. We used to have radeon, intel, and > some others in media-libs/mesa's VIDEO_CARDS. radeon and intel > corresponded to disparate sets of drivers -- VIDEO_CARDS=radeon has > meant classic r100, r200, r300, and r600 drivers and gallium r600 and > radeonsi drivers. VIDEO_CARDS=intel has meant classic i915 and i965 > drivers as well as gallium i915. > > I added more-specific VIDEO_CARDS for those separate drivers a few > years ago, so that users could set VIDEO_CARDS="radeon radeonsi" and > only get the one radeonsi driver they actually wanted while still > enabling support for x11-libs/libdrm's radeon support code which is > used by most of those radeon drivers. Of course some users want this > control and others don't care at all. > > The confusion comes in with "classic" DRI drivers vs Gallium drivers. > The Gallium abstraction layer allows a hardware driver to handle > multiple APIs -- OpenGL, D3D9, OpenCL, video decode APIs, etc. For > instance, users try to build the classic i965 driver (there is no > Gallium driver for this hardware) with USE=opencl or USE=vaapi and > don't understand why they didn't get what they wanted (or REQUIRED_USE > prevents them from doing so). > > Should of Mesa's USE flags, d3d9, llvm, lm_sensors, opencl, openmax, > unwind, vaapi, vdpau, xa, and xvmc are Gallium-only. Should I make a > USE_EXPAND for Gallium-only options to attempt to avoid confusion? > Another point of confusion: not all Gallium drivers support all of > these features. For instance only the r600 and radeonsi drivers > support OpenCL. How to best handle this? > > It seems like at one extreme you build an extensive set of > REQUIRED_USE conditions that force users to micromanage their USE > flags, or you let them enable all sorts of impossible combinations and > deal with the confused bug reports. > > I would like to somehow get rid of the 'classic' and 'gallium' USE > flags entirely, but I'm not totally sure how. Maybe I can enable them > dependent on VIDEO_CARDS... > > Suggestions welcome. > What about creating a tiny pkg that has all the combinations in a dictionary and can set the USE and VIDEO_CARDS flags according to the video card(s) you have. This would be along the same lines as the app-portage/cpuid2cpuflags pkg. Then the pkg is updated as new drivers and combinations are changed. Perhaps have it run in pkg_postisnt to print any irregularities it finds and ewarn they need fixing. -- Brian Dolbec
Re: [gentoo-dev] Openstack Summit MeetUp: dinner planning for Wednesday?
On Mon, 14 May 2018 22:49:26 + "Robin H. Johnson" <robb...@gentoo.org> wrote: > On Sun, May 13, 2018 at 01:11:30PM -0700, Jack Morgan wrote: > > Gentoo, > > > > I will be attending the Openstack Summit in Vancouver, BC. The > > conference is May 21-24th. I would like to organize a Gentoo meetup > > for those attending or living in the area. I personally will be > > there from May 20th - 26th. > > > > Please reply if you are interested in meeting and which > > day(s)/time(s) you are available. I'm looking forward to it! > I'm Vancouver-local, and will ALSO be attending the summit. > > Available for events during the daytime during the conference, and > also evenings of Monday & Wednesday. I am NOT available on the > evenings. > > As a note, Monday is a (provincial) public holiday in Vancouver, and > you may find a lot of restaurants closed (more than the ones that are > usually closed Mondays). > > Unless there are specific objections, let's try to plan the meetup for > Wednesday evening based on the above? > > I can help to arrange a dinner outing Wednesday evening. If you're > interested, privately email me. Please include a mention of specific > dietary requirements if any. > I am also local to Vancouver, but won't be at the event. Wednesday works better for me too. My daughter has ball games Tuesday and Thursday. -- Brian Dolbec pgphPm7lUDY8d.pgp Description: OpenPGP digital signature
Re: [gentoo-portage-dev] [PATCH] global_event_loop: use asyncio event loop (bug 654390)
On Mon, 7 May 2018 03:18:32 -0700 Zac Medico <zmed...@gentoo.org> wrote: > + # This attribute it used by _wrap_loop to detect if > the typo, s/it/is this asynio stuff is mostly beyond my familiarity... but all looks ok, and I know you ran the tests on it... should be good to merge unless anyone else objects... -- Brian Dolbec
Re: [gentoo-dev] Monthly x11@ project status for May 2018
On Mon, 7 May 2018 13:38:47 -0700 Matt Turner <matts...@gentoo.org> wrote: > > If there's a way to have repoman alert developers to deprecated > dependencies in the same way we handle deprecated eclasses, I'd like > to know about it. > > Currently there is not. Thinking out loud... It would mean parsing package.mask to generate the list filtering out those with "masked for removal", from other general mask reasons, but even that is not consistent. from another email in today's batch... (not to pick on this one, was just a lucky coincidence) eg: Subject: [gentoo-dev] Last rites: old, ruby23-only, slots of various ruby packages # Hans de Graaff <gra...@gentoo.org> (7 May 2018) # Old slots that are ruby23-only and no longer maintained # upstream, and that do not have packages depend on them. # Migrate to newer slot of the same package. Perhaps we would need to add a separate last-rites.mask list that the package manager merges internally as part of the .mask stack. That would make just one file to load without a need for filtering. A separate list might also be beneficial for the undertakers and any tooling for its automation. (I am not familiar with any of that tooling) I would then suggest the dependency depth check to default to one (or two) so as to not slow things down drastically. Perhaps a Q/A check report to scan the whole tree on a weekly basis. But that would also require PMS to be updated for the new file, which means a council vote... -- Brian Dolbec pgpDUQqsEMJ4M.pgp Description: OpenPGP digital signature
Re: [gentoo-portage-dev] [PATCH] config.environ: always strip slash from SYSROOT (bug 654600)
On Wed, 2 May 2018 18:03:38 -0700
Zac Medico <zmed...@gentoo.org> wrote:
> Since SYSROOT=/ interacts badly with autotools.eclass (bug 654600),
> and no EAPI expects SYSROOT to have a trailing slash, always strip
> the trailing slash from SYSROOT.
>
> Bug: https://bugs.gentoo.org/654600
> Fixes: a41dacf7926c ("Export SYSROOT and ESYSROOT in ebuild env in
> EAPI 7") ---
> pym/portage/package/ebuild/config.py | 9 +++--
> 1 file changed, 7 insertions(+), 2 deletions(-)
>
> diff --git a/pym/portage/package/ebuild/config.py
> b/pym/portage/package/ebuild/config.py index f9b257b86..96b2ebb71
> 100644 --- a/pym/portage/package/ebuild/config.py
> +++ b/pym/portage/package/ebuild/config.py
> @@ -2813,11 +2813,16 @@ class config(object):
> mydict.pop("ECLASSDIR", None)
>
> if not
> eapi_attrs.path_variables_end_with_trailing_slash:
> - for v in ("D", "ED", "ROOT", "EROOT",
> "SYSROOT", "ESYSROOT",
> - "BROOT"):
> + for v in ("D", "ED", "ROOT", "EROOT",
> "ESYSROOT", "BROOT"): if v in mydict:
> mydict[v] =
> mydict[v].rstrip(os.path.sep)
> + # Since SYSROOT=/ interacts badly with
> autotools.eclass (bug 654600),
> + # and no EAPI expects SYSROOT to have a trailing
> slash, always strip
> + # the trailing slash from SYSROOT.
> + if 'SYSROOT' in mydict:
> + mydict['SYSROOT'] =
> mydict['SYSROOT'].rstrip(os.path.sep) +
> try:
> builddir = mydict["PORTAGE_BUILDDIR"]
> distdir = mydict["DISTDIR"]
looks good
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH 0/5] EbuildFetcher._get_uri_map(): fix event loop recursion (bug 653810)
On Sun, 22 Apr 2018 15:30:09 -0700 Zac Medico <zmed...@gentoo.org> wrote: > Bug: https://bugs.gentoo.org/653810 > > Zac Medico (5): > portdbapi: add async_fetch_map method (bug 653810) > EbuildFetcher: inherit CompositeTask (bug 653810) > EbuildFetcher: add _async_uri_map method (bug 653810) > EbuildFetcher: use _async_uri_map in _start (bug 653810) > EbuildFetcher: add async_already_fetched method (bug 653810) > > pym/_emerge/EbuildBuild.py| 8 +++- > pym/_emerge/EbuildFetcher.py | 105 > -- > pym/portage/dbapi/porttree.py | 75 +++--- 3 > files changed, 146 insertions(+), 42 deletions(-) > I didn't see any errors glaring at me... :) looks like it should be good -- Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] BinpkgFetcher: use async lock (bug 614110)
On Sat, 21 Apr 2018 12:27:28 -0700
Zac Medico <zmed...@gentoo.org> wrote:
> In order to avoid event loop recursion, convert the
> _BinpkgFetcherProcess.lock() method to an async_lock
> method for use by BinpkgFetcher.
>
> Bug: https://bugs.gentoo.org/614110
> ---
> pym/_emerge/BinpkgFetcher.py | 53
> +++- 1 file changed, 33
> insertions(+), 20 deletions(-)
>
> diff --git a/pym/_emerge/BinpkgFetcher.py
> b/pym/_emerge/BinpkgFetcher.py index 5ca7a45cf..2bbc0a26f 100644
> --- a/pym/_emerge/BinpkgFetcher.py
> +++ b/pym/_emerge/BinpkgFetcher.py
> @@ -32,11 +32,24 @@ class BinpkgFetcher(CompositeTask):
> pkg.cpv) + ".partial"
>
> def _start(self):
> - self._start_task(
> -
> _BinpkgFetcherProcess(background=self.background,
> - logfile=self.logfile, pkg=self.pkg,
> pkg_path=self.pkg_path,
> - pretend=self.pretend,
> scheduler=self.scheduler),
> - self._fetcher_exit)
> + fetcher =
> _BinpkgFetcherProcess(background=self.background,
> + logfile=self.logfile, pkg=self.pkg,
> pkg_path=self.pkg_path,
> + pretend=self.pretend,
> scheduler=self.scheduler) +
> + if not self.pretend:
> +
> portage.util.ensure_dirs(os.path.dirname(self.pkg_path))
> + if "distlocks" in
> self.pkg.root_config.settings.features:
> + self._start_task(
> +
> AsyncTaskFuture(future=fetcher.async_lock()),
> +
> functools.partial(self._start_locked, fetcher))
> + return
> +
> + self._start_task(fetcher, self._fetcher_exit)
> +
> + def _start_locked(self, fetcher, lock_task):
> + self._assert_current(lock_task)
> + lock_task.future.result()
> + self._start_task(fetcher, self._fetcher_exit)
>
> def _fetcher_exit(self, fetcher):
> self._assert_current(fetcher)
> @@ -68,13 +81,8 @@ class _BinpkgFetcherProcess(SpawnProcess):
> pretend = self.pretend
> bintree = pkg.root_config.trees["bintree"]
> settings = bintree.settings
> - use_locks = "distlocks" in settings.features
> pkg_path = self.pkg_path
>
> - if not pretend:
> -
> portage.util.ensure_dirs(os.path.dirname(pkg_path))
> - if use_locks:
> - self.lock()
> exists = os.path.exists(pkg_path)
> resume = exists and os.path.basename(pkg_path) in
> bintree.invalids if not (pretend or resume):
> @@ -184,7 +192,7 @@ class _BinpkgFetcherProcess(SpawnProcess):
> except
> OSError: pass
>
> - def lock(self):
> + def async_lock(self):
> """
> This raises an AlreadyLocked exception if lock() is
> called while a lock is already held. In order to avoid this, call
> @@ -194,17 +202,22 @@ class _BinpkgFetcherProcess(SpawnProcess):
> if self._lock_obj is not None:
> raise self.AlreadyLocked((self._lock_obj,))
>
> - async_lock = AsynchronousLock(path=self.pkg_path,
> - scheduler=self.scheduler)
> - async_lock.start()
> + result = self.scheduler.create_future()
>
> - if async_lock.wait() != os.EX_OK:
> - # TODO: Use CompositeTask for better
> handling, like in EbuildPhase.
> - raise AssertionError("AsynchronousLock
> failed with returncode %s" \
> - % (async_lock.returncode,))
> + def acquired_lock(async_lock):
> + if async_lock.wait() == os.EX_OK:
> + self.locked = True
> + result.set_result(None)
> + else:
> + result.set_exception(AssertionError(
> + "AsynchronousLock failed
> with returncode %s"
> + % (async_lock.returncode,)))
>
> - self._lock_obj = async_lock
> - self.locked = True
> + self._lock_obj = AsynchronousLock(path=self.pkg_path,
> + scheduler=self.scheduler)
> + self._lock_obj.addExitListener(acquired_lock)
> + self._lock_obj.start()
> + return result
>
> class AlreadyLocked(portage.exception.PortageException):
> pass
Looks fine to me :)
--
Brian Dolbec
Re: [gentoo-dev] Questions on overlays, repositories and PMS
On Fri, 23 Feb 2018 18:36:07 +0100 Michael Lienhardt <michael.lienha...@laposte.net> wrote: > I started refactoring my solver to make it more modular, to fix some > details w.r.t. the PMS and to manage different repositories. I thus > have several questions on how multiple repositories work in portage. > > 1. My understanding was that /etc/portage/repos.conf replaced the > PORTDIR_OVERLAY variable, however this variable is still documented > (e.g. in > https://devmanual.gentoo.org/general-concepts/overlay/index.html). > Was my intuition right? Or in other word, it is enough to only look > at /etc/portage/repos.conf? In general, an overlay is a repository, > i.e., a valid tree layout for the PMS, right (as stated in > https://devmanual.gentoo.org/general-concepts/overlay/index.html)? > Yes, you should not have to look at the old make.conf variables, yes, they do work, but mostly for backward compatibility with some old third party tools. Those really need to be updated. I forget now how many there are, but if I remember correctly only 2 or 3 are left. And yes, we have been slow to get them updated or replaced. > 2. the PMS states that any valid repository has a profiles folder > which can contain profiles and a package.mask file. > - can the profiles in a repository different from DEFAULT be > selected? > - is the package.mask file apply only on the packages of that > repository, or on every packages of every repositories listed > in /etc/portage/repos.conf? > > 3. many repositories do not have an eclass folder, and miss many > (optional) configuration files in the profiles folder (like > arch.list, categories): > - is such information implicitly inherited from the DEFAULT > repository (even though > https://wiki.gentoo.org/wiki//etc/portage/repos.conf states that it > is not)? the brother overlay > (https://github.com/stefan-langenmaier/brother-overlay) does not > specify any masters > - when the eclass folder, profiles/arch.list and such are present, > is the data from the DEFAULT repository still implicitly inherited? > - when the eclass folder, profiles/arch.list and such are present, > are they visible globally (i.e., a package from another repository > can use a keyword of the arch.list and inherit from one of the > eclass)? > > 4. is the "masters" attribute in /etc/portage/repos.conf make the > repository inherit other data than the eclasses? > > 5. since every repos can have a profiles/categories file, is the > file /etc/portage/categories obsolete (or should it be)? > > > Best Regards, > Michael Lienhardt > > -- Brian Dolbec
Re: [gentoo-portage-dev] [PATCH v3 1/3] repoman: Add commit message verification
On Mon, 19 Feb 2018 16:00:55 +0100 Michał Górny <mgo...@gentoo.org> wrote: > Add a check for common mistakes in commit messages. For now, it is > pretty rough and works only for -m/-F. It will be extended to work > in the interactive mode in the future. > --- This version series looks good, thank you :) -- Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] emerge --search: use slash to auto-detect category (bug 647940)
On Sun, 18 Feb 2018 13:27:31 -0800
Zac Medico <zmed...@gentoo.org> wrote:
> Since search strings containing a slash do no work unless category
> match mode is enabled, use slash to auto-detect category match mode,
> so that users do not have to prefix the search string with the special
> @ symbol.
>
> Bug: https://bugs.gentoo.org/647940
> ---
> pym/_emerge/search.py | 4
> 1 file changed, 4 insertions(+)
>
> diff --git a/pym/_emerge/search.py b/pym/_emerge/search.py
> index 36897fb15..eb52b2ca3 100644
> --- a/pym/_emerge/search.py
> +++ b/pym/_emerge/search.py
> @@ -256,6 +256,10 @@ class search(object):
> if self.searchkey.startswith('@'):
> match_category = 1
> self.searchkey = self.searchkey[1:]
> + # Auto-detect category match mode (@ symbol can be
> deprecated
> + # after this is available in a stable version of
> portage).
> + if '/' in self.searchkey:
> + match_category = 1
> fuzzy = False
> if regexsearch:
> self.searchre=re.compile(self.searchkey,re.I)
looks fine
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH v2 1/3] repoman: Add commit message verification
On Sat, 17 Feb 2018 13:56:46 +0100 Michał Górny <mgo...@gentoo.org> wrote: > + > + footer_re = re.compile(r'^[\w-]+:') > + > + @classmethod > + def verify_commit_message(cls, msg): ... > + if all(cls.footer_re.match(l) for l in lines if l.strip()): Why declare the footer_re in the class space? It is only used in this new function. Then the function could be @staticmethod instead. I don't see the advantage of it this way. If it is for re-use in other possible check functions, then perhaps it would be best to split this out to it's own class/file that can be added to easily. And just run it from the Actions class as M.J. Everitt suggested. -- Brian Dolbec
Re: [gentoo-portage-dev] [PATCH v2 1/3] repoman: Add commit message verification
> > +
> > bug_closes_without_url = True
> > + else:
> > + for l in lines:
> > + # we recommend wrapping at
> > 72 but accept up to 80;
> > + # do not complain if we
> > have single word (usually
> > + # it means very long URL)
> > + if len(l.strip()) > 80 and
> > len(l.split()) > 1:
> > + body_too_long =
> > True +
> > + if multiple_footers:
> > + problems.append('multiple footer-style
> > blocks found, please combine them into one')
> > + if gentoo_bug_used:
> > + problems.append('please replace Gentoo-Bug
> > with GLEP 66-compliant Bug/Closes')
> > + if bug_closes_without_url:
> > + problems.append('Bug/Closes footers
> > require full URL')
> > + if body_too_long:
> > + problems.append('body lines should be
> > wrapped at 72 (max 80) characters') +
> > + if problems:
> > + return (False, '\n'.join('- %s' % x for x
> > in problems))
> > + return (True, None)
> > diff --git a/repoman/pym/repoman/tests/commit/__init__.py
> > b/repoman/pym/repoman/tests/commit/__init__.py new file mode 100644
> > index 0..d74fd94a7
> > --- /dev/null
> > +++ b/repoman/pym/repoman/tests/commit/__init__.py
> > @@ -0,0 +1,2 @@
> > +# Copyright 2011-2018 Gentoo Foundation
> > +# Distributed under the terms of the GNU General Public License v2
> > diff --git a/repoman/pym/repoman/tests/commit/__test__.py
> > b/repoman/pym/repoman/tests/commit/__test__.py new file mode 100644
> > index 0..8b1378917
> > --- /dev/null
> > +++ b/repoman/pym/repoman/tests/commit/__test__.py
> > @@ -0,0 +1 @@
> > +
> > diff --git a/repoman/pym/repoman/tests/simple/test_simple.py
> > b/repoman/pym/repoman/tests/simple/test_simple.py index
> > a24e0d5a3..3d7a70ad0 100644 ---
> > a/repoman/pym/repoman/tests/simple/test_simple.py +++
> > b/repoman/pym/repoman/tests/simple/test_simple.py @@ -1,4 +1,4 @@
> > -# Copyright 2011-2015 Gentoo Foundation
> > +# Copyright 2011-2018 Gentoo Foundation
> > # Distributed under the terms of the GNU General Public License v2
> >
> > import subprocess
> > @@ -194,13 +194,13 @@ class SimpleRepomanTestCase(TestCase):
> > ("", repoman_cmd + ("full", "-d")),
> > ("", cp_cmd + (test_ebuild,
> > test_ebuild[:-8] + "2.ebuild")), ("", git_cmd + ("add",
> > test_ebuild[:-8] + "2.ebuild")),
> > - ("", repoman_cmd + ("commit", "-m", "bump
> > to version 2")),
> > + ("", repoman_cmd + ("commit", "-m",
> > "cat/pkg: bump to version 2")), ("", cp_cmd + (test_ebuild,
> > test_ebuild[:-8] + "3.ebuild")), ("", git_cmd + ("add",
> > test_ebuild[:-8] + "3.ebuild")),
> > - ("dev-libs", repoman_cmd + ("commit",
> > "-m", "bump to version 3")),
> > + ("dev-libs", repoman_cmd + ("commit",
> > "-m", "cat/pkg: bump to version 3")), ("", cp_cmd + (test_ebuild,
> > test_ebuild[:-8] + "4.ebuild")), ("", git_cmd + ("add",
> > test_ebuild[:-8] + "4.ebuild")),
> > - ("dev-libs/A", repoman_cmd + ("commit",
> > "-m", "bump to version 4")),
> > + ("dev-libs/A", repoman_cmd + ("commit",
> > "-m", "cat/pkg: bump to version 4")), )
> >
> > env = {
> Might I suggest breaking out checks into a separate module? I think
> that hard-coding it all is likely to become a pain as time goes on,
> more checks get added, etc ...
>
So, you would like the commit message checks to be a plug-in system
like the ebuild checks are now.
Yes that is possible, that would also allow overlays and sub-distros
to customize it to their liking. But you do realize it would be a
module system with only the one plug-in. It would add a slight amount
of overhead and subsequently a tiny bit more time. Although it is not
much as I remember when I first developed the plug-in system for emaint.
I will consider that for the stage3 work in progress, but for the
master branch of repoman, the hard-coded version above would be fine.
--
Brian Dolbec
pgprVkLpLeUb3.pgp
Description: OpenPGP digital signature
Re: [gentoo-portage-dev] [PATCH] emerge: disable --changed-deps-report by default (bug 645780)
On Fri, 2 Feb 2018 18:52:35 -0800
Zac Medico <zmed...@gentoo.org> wrote:
> This option is too noisy to enable by default, since it reports
> hundreds of packages for most users.
>
> Suggested-by: Michał Górny <mgo...@gentoo.org>
> Bug: https://bugs.gentoo.org/645780
> ---
> man/emerge.1 | 7 +--
> pym/_emerge/create_depgraph_params.py | 6 ++
> 2 files changed, 3 insertions(+), 10 deletions(-)
>
> diff --git a/man/emerge.1 b/man/emerge.1
> index 189e6f879..810336fa4 100644
> --- a/man/emerge.1
> +++ b/man/emerge.1
> @@ -469,12 +469,7 @@ respect to changed build\-time dependencies is
> controlled by the Tells emerge to report ebuilds for which the ebuild
> dependencies have changed since the installed instance was built.
> Behavior with respect to changed build\-time dependencies is
> controlled by the -\fB\-\-with\-bdeps\fR option. If the
> \fB\-\-update\fR and \fB\-\-deep\fR -options are enabled then this
> option is enabled automatically for a -dependency calculation if the
> cost of report generation is relatively -insignificant (any
> calculation exclusively involving binary packages is -exempt). The
> \fIEMERGE_DEFAULT_OPTS\fR variable may be used to disable -this by
> default. +\fB\-\-with\-bdeps\fR option.
> .TP
> .BR \-\-changed\-use ", " \-U
> Tells emerge to include installed packages where USE flags have
> diff --git a/pym/_emerge/create_depgraph_params.py
> b/pym/_emerge/create_depgraph_params.py index ecd65335c..6964bc65b
> 100644 --- a/pym/_emerge/create_depgraph_params.py
> +++ b/pym/_emerge/create_depgraph_params.py
> @@ -127,10 +127,8 @@ def create_depgraph_params(myopts, myaction):
> if changed_deps is not None:
> myparams['changed_deps'] = changed_deps
>
> - changed_deps_report = myopts.get('--changed-deps-report')
> - if (changed_deps_report != 'n' and
> - not (myaction == 'remove' or '--usepkgonly' in
> myopts) and
> - deep is True and '--update' in myopts):
> + changed_deps_report = myopts.get('--changed-deps-report',
> 'n') == 'y'
> + if changed_deps_report:
> myparams['changed_deps_report'] = True
>
> if myopts.get("--selective") == "n":
looks good, if not already merged.
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] repoman: Add commit message verification
On Sun, 04 Feb 2018 18:58:21 +0100 Michał Górny <mgo...@gentoo.org> wrote: > W dniu nie, 04.02.2018 o godzinie 09∶51 -0800, użytkownik Brian Dolbec > napisał: > > > > I know there are not a lot of repoman unit tests, but, can you > > please create some to test this? That way it doesn't get any worse. > > > > In the PR, you mentioned adding in an editor call to re-edit the > > message. This would be a good thing and I know is not that > > difficult to accomplish. Is this coming in another patch? > > > > I'm still wondering how to implement it best. In particular, how to > handle the non-EDITOR (stdin) branch? > hmm, yeah, that was not an option for the work tool I wrote and added the re-edit to. It had 2 files it needed to edit or re-edit. In the case of passing in a pre-saved message, that should be left for the user to fix. Perhaps in the case of a stdin -m "foo" message we could just let the user rely on bash their history to bring up that previous command, to edit before hitting enter again. -- Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] repoman: Add commit message verification
On Fri, 2 Feb 2018 23:53:05 +0100
Michał Górny <mgo...@gentoo.org> wrote:
> Add a check for common mistakes in commit messages. For now, it is
> pretty rough and exits immediately but it should be integrated with
> the editor in the future.
> ---
> repoman/pym/repoman/actions.py | 68
> ++ 1 file changed, 68
> insertions(+)
>
> diff --git a/repoman/pym/repoman/actions.py
> b/repoman/pym/repoman/actions.py index b76a6e466..97a71d0f5 100644
> --- a/repoman/pym/repoman/actions.py
> +++ b/repoman/pym/repoman/actions.py
> @@ -124,6 +124,15 @@ class Actions(object):
>
> commitmessage = commitmessage.rstrip()
>
> + res, expl = self.verify_commit_message(commitmessage)
> + if not res:
> + print(bad("RepoMan does not like your commit
> message:"))
> + print(expl)
> + if self.options.force:
> + print('(but proceeding due to
> --force)')
> + else:
> + sys.exit(1)
> +
> # Update copyright for new and changed files
> year = time.strftime('%Y', time.gmtime())
> for fn in chain(mynew, mychanged):
> @@ -585,3 +594,62 @@ class Actions(object):
> print("* no commit message? aborting
> commit.") sys.exit(1)
> return commitmessage
> +
> + footer_re = re.compile(r'^[\w-]+:')
> +
> + def verify_commit_message(self, msg):
> + """
> + Check whether the message roughly complies with
> GLEP66. Returns
> + (True, None) if it does, (False, ) if
> it does not.
> + """
> +
> + problems = []
> + paras = msg.strip().split('\n\n')
> + summary = paras.pop(0)
> +
> + if '\n' in summary.strip():
> + problems.append('commit message must start
> with a *single* line of summary, followed by empty line')
> + if ':' not in summary:
> + problems.append('summary line must start
> with a logical unit name, e.g. "cat/pkg:"')
> + # accept 69 overall or unit+50, in case of very long
> package names
> + if len(summary.strip()) > 69 and
> len(summary.split(':', 1)[-1]) > 50:
> + problems.append('summary line is too long
> (max 69 characters)') +
> + multiple_footers = False
> + gentoo_bug_used = False
> + bug_closes_without_url = False
> + body_too_long = False
> +
> + found_footer = False
> + for p in paras:
> + lines = p.splitlines()
> + # if all lines look like footer material, we
> assume it's footer
> + # else, we assume it's body text
> + if all(self.footer_re.match(l) for l in
> lines if l.strip()):
> + # multiple footer-like blocks?
> + if found_footer:
> + multiple_footers = True
> + found_footer = True
> + for l in lines:
> + if
> l.startswith('Gentoo-Bug'):
> + gentoo_bug_used =
> True
> + elif l.startswith('Bug:') or
> l.startswith('Closes:'):
> + if 'http://' not in
> l and 'https://' not in l:
> +
> bug_closes_without_url = True
> + else:
> + for l in lines:
> + if len(l.strip()) > 72:
> + body_too_long = True
> +
> + if multiple_footers:
> + problems.append('multiple footer-style
> blocks found, please combine them into one')
> + if gentoo_bug_used:
> + problems.append('please replace Gentoo-Bug
> with GLEP 66-compliant Bug/Closes')
> + if bug_closes_without_url:
> + problems.append('Bug/Closes footers require
> full URL')
> + if body_too_long:
> + problems.append('body lines should be
> wrapped at 72 characters') +
> + if problems:
> + return (False, '\n'.join('- %s' % x for x in
> problems))
> + return (True, None)
I know there are not a lot of repoman unit tests, but, can you please
create some to test this? That way it doesn't get any worse.
In the PR, you mentioned adding in an editor call to re-edit the
message. This would be a good thing and I know is not that difficult
to accomplish. Is this coming in another patch?
--
Brian Dolbec
Re: [gentoo-dev] AMD64 Arch Testers needed urgently
On Wed, 13 Dec 2017 15:55:59 -0500 Lucas Ramage <ramage.luca...@gmail.com> wrote: > I see, well I can setup buildbot to do that. Is there some place in > particular that I should send my test results? > > On Wed, Dec 13, 2017 at 1:28 PM, Aaron W. Swenson > <titanof...@gentoo.org> wrote: > > > On 2017-12-13 13:20, Lucas Ramage wrote: > > > > In my discussions with other developers, I've found that this > > > > is the > > > > > > > biggest concern. Most devs are runnning ~amd64, so they don't > > > feel that > > > > they can mark things stable. > > > > > > W > > > hat about running a stable chroot? Are there any tools that can > > > be used to automate this process? > > > > Yes, a stable chroot is okay, and there is app-portage/tatt that > > can run through the various USE flag combinations for you. > > > > It’s output isn’t terribly helpful, so it takes a little while to > > understand what it’s trying to tell you. > > > > > Lucas, ping me on irc/email me directly. I have been maintaining buildbot in gentoo for awhile and my day job is working on/developing buildbot scripts. I am planning out some pkg bumping scripts to help with regular pkg maintenance. I hope to have some initial code running during the holiday break. There should be a bunch of shared factory code possible as there will be some overlap in their needs. -- Brian Dolbec
Re: [gentoo-portage-dev] [PATCH v2] [checksum] Disable pygcrypt backend due to breakage
On Tue, 5 Dec 2017 18:08:47 +0100
Michał Górny <mgo...@gentoo.org> wrote:
> Closes: https://bugs.gentoo.org/615620
> ---
> pym/portage/checksum.py | 6 +-
> 1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/pym/portage/checksum.py b/pym/portage/checksum.py
> index 5424ce56b..9e7bffea9 100644
> --- a/pym/portage/checksum.py
> +++ b/pym/portage/checksum.py
> @@ -150,7 +150,11 @@ if "SHA3_256" not in hashfunc_map or "SHA3_512"
> not in hashfunc_map: # (GnuPG).
> gcrypt_algos = frozenset(('RMD160', 'WHIRLPOOL', 'SHA3_256',
> 'SHA3_512', 'STREEBOG256', 'STREEBOG512'))
> -if gcrypt_algos.difference(hashfunc_map):
> +# Note: currently disabled due to resource exhaustion bugs in
> pygcrypt. +# Please do not reenable until upstream has a fix.
> +# https://bugs.gentoo.org/615620
> +if False:
> +#if gcrypt_algos.difference(hashfunc_map):
> try:
> import binascii
> import pygcrypt.hashcontext
looks good thanks, merge please
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] [checksum] Disable pygcrypt backend due to breakage
On Tue, 5 Dec 2017 08:42:43 -0800
Brian Dolbec <dol...@gentoo.org> wrote:
> On Tue, 5 Dec 2017 17:34:23 +0100
> Michał Górny <mgo...@gentoo.org> wrote:
>
> > Closes: https://bugs.gentoo.org/615620
> > ---
> > pym/portage/checksum.py | 5 -
> > 1 file changed, 4 insertions(+), 1 deletion(-)
> >
> > diff --git a/pym/portage/checksum.py b/pym/portage/checksum.py
> > index 5424ce56b..0841ab231 100644
> > --- a/pym/portage/checksum.py
> > +++ b/pym/portage/checksum.py
> > @@ -150,7 +150,10 @@ if "SHA3_256" not in hashfunc_map or "SHA3_512"
> > not in hashfunc_map: # (GnuPG).
> > gcrypt_algos = frozenset(('RMD160', 'WHIRLPOOL', 'SHA3_256',
> > 'SHA3_512', 'STREEBOG256', 'STREEBOG512'))
> > -if gcrypt_algos.difference(hashfunc_map):
> > +# Note: currently disabled due to resource exhaustion bugs in
> > pygcrypt. +# Please do not reenable until upstream has a fix.
> > +# https://bugs.gentoo.org/615620
> > +if False and gcrypt_algos.difference(hashfunc_map):
> > try:
> > import binascii
> > import pygcrypt.hashcontext
>
>
>
> It would be better to just comment out the original if, then add a new
> line to replace it with just if False. it would be clearer what the
> original code should be. Of course with the reason comments...
>
sorry, brain is still struggling with this damn headache...
It would be clearer later when re-enabling if it takes a long time
for pygrcrypt to get fixed. I know it would be for me, my memory
isn't as good as it used to be. Also if it ends up someone else
looks to modify it later.
+# Note: currently disabled due to resource exhaustion bugs in pygcrypt.
+# Please do not reenable until upstream has a fix.
+# https://bugs.gentoo.org/615620
-if gcrypt_algos.difference(hashfunc_map):
+#if gcrypt_algos.difference(hashfunc_map):
+if False
try:
import binascii
import pygcrypt.hashcontext
I approve the above form to merge... Thanks
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] depgraph: eval disjunctive build deps earlier (bug 639346)
639346, where a redundant jre
> implementation
> + # was pulled in because DEPEND was evaluated
> after
> + # RDEPEND.
> + ResolverPlaygroundTestCase(
> + ['dev-db/hsqldb'],
> + success=True,
> + mergelist=[
> + 'dev-java/icedtea-3.6',
> + 'virtual/jdk-1.8',
> + 'virtual/jre-1.8',
> + 'dev-db/hsqldb-1.8',
> + ],
> + ),
> +
> + # The jdk is not needed with --usepkg, so
> the jre should
> + # be preferred in this case.
> + ResolverPlaygroundTestCase(
> + ['dev-db/hsqldb'],
> + options = {
> + '--usepkg': True
> + },
> + success=True,
> + mergelist=[
> + 'dev-java/oracle-jre-bin-1.8',
> + 'virtual/jre-1.8',
> + '[binary]dev-db/hsqldb-1.8',
> + ],
> + ),
> + )
> +
> + playground = ResolverPlayground(debug=False,
> + binpkgs=binpkgs, ebuilds=ebuilds)
> +
> + try:
> + for test_case in test_cases:
> + playground.run_TestCase(test_case)
> +
> self.assertEqual(test_case.test_success, True,
> + test_case.fail_msg)
> + finally:
> + playground.debug = False
> + playground.cleanup()
> diff --git a/pym/portage/tests/resolver/test_onlydeps_minimal.py
> b/pym/portage/tests/resolver/test_onlydeps_minimal.py index
> 13c79ed55..efda02c59 100644 ---
> a/pym/portage/tests/resolver/test_onlydeps_minimal.py +++
> b/pym/portage/tests/resolver/test_onlydeps_minimal.py @@ -25,9 +25,10
> @@ class OnlydepsMinimalTestCase(TestCase): success = True,
> options = { "--onlydeps": True,
> "--onlydeps-with-rdeps":
> "y" },
> - mergelist = ["dev-libs/B-1",
> + ambiguous_merge_order = True,
> + mergelist = [("dev-libs/B-1",
>"dev-libs/C-1",
> - "dev-libs/D-1"]),
> + "dev-libs/D-1")]),
> ResolverPlaygroundTestCase(
> ["dev-libs/A"],
> all_permutations = True,
looks good
--
Brian Dolbec
Re: [gentoo-dev] Help testing ebuilds? golang/Fabio load balancer
On Sat, 11 Nov 2017 12:31:15 -0500 Michael Orlitzky <m...@gentoo.org> wrote: > Essentially,we have two commands to create a directory, "dodir" and > "do-empty-dir" (which we call "keepdir"). The latter is only necessary > due to an implementation detail, so it doesn't belong in the user > interface -- the PM should figure out what to do. > > As far as the actual implementation goes, I'm not sure that > automatically-generated ".keep" files are better than having the > package manager maintain its own database. The latter would be more > complex, but would avoid littering everyone's filesystems with > ".keep" files. > Well, for: 1) using .keepdir files makes this package manager non-specific, ie: using a different PM would mean it too sees that it is suppose to be kept. 2) Most ebuilds don't need/or use .keepdir, so I doubt there are that many littering up your filesystem. 3) There already is a database of the files installed by a package. But, how does it know which other packages might need to install to that directory. So, what does it do on unmerge of the package. Typically, the package manager will refuse to remove a non-empty directory after the files it is removing are removed. 4) Creating a single database for these will make the system more vulnerable to corruption and data loss. It would save on disk usage and number of inodes used. But there is a tradeoff between space and robustness. If one .keepdir is lost, it may only affect a few packages. If the database becomes corrupted. It could potentially mean the loss of much more of your installed pkg data. Spanning a great deal of the installed pkgs. In this case simplicity of .keepdir is in my opinion, much better than a single db. -- Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] dep_zapdeps: install new package, avoid downgrade (bug 635540)
es = (
> + # Test bug 635540, where we need to install
> libtirpc
> + # rather than downgrade glibc.
> + ResolverPlaygroundTestCase(
> + ['net-misc/foo'],
> + success=True,
> + mergelist=[
> + 'net-libs/libtirpc-1',
> + 'net-misc/foo-1',
> + ],
> + ),
> + )
> +
> + playground = ResolverPlayground(debug=False,
> + ebuilds=ebuilds, installed=installed,
> world=world) +
> + try:
> + for test_case in test_cases:
> + playground.run_TestCase(test_case)
> +
> self.assertEqual(test_case.test_success, True,
> + test_case.fail_msg)
> + finally:
> + playground.debug = False
> + playground.cleanup()
> +
> + # In some cases it's necessary to downgrade due to
> + # the installed package being masked (glibc is a
> + # not an ideal example because it's usually not
> + # practical to downgrade it).
> + user_config = {
> + "package.mask" : (
> + ">=sys-libs/glibc-2.26",
> + ),
> + }
> +
> + test_cases = (
> + ResolverPlaygroundTestCase(
> + ['net-misc/foo'],
> + success=True,
> + mergelist=[
> + 'sys-libs/glibc-2.24',
> + 'net-misc/foo-1',
> + ],
> + ),
> + )
> +
> + playground = ResolverPlayground(debug=False,
> + ebuilds=ebuilds, installed=installed,
> world=world,
> + user_config=user_config)
> +
> + try:
> + for test_case in test_cases:
> + playground.run_TestCase(test_case)
> +
> self.assertEqual(test_case.test_success, True,
> + test_case.fail_msg)
> + finally:
> + playground.debug = False
> + playground.cleanup()
looks good
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH v5] file_copy: use sendfile return value to measure bytes copied (bug 635126)
blocks when appropriate).
> Effects
> + * on file offsets are undefined.
> + *
> * Return: On success, the number of bytes to copy before the next
> hole,
> * and -1 on failure (errno is set appropriately). Returns 0 when
> fd_in
> * reaches EOF.
> @@ -250,7 +289,7 @@ _reflink_linux_file_copy(PyObject *self, PyObject
> *args)
> * syscall is not available (less than Linux
> 4.5). */
> error = 0;
> -copyfunc = sendfile;
> +copyfunc = sf_wrapper;
> copyfunc_ret = copyfunc(fd_out,
> fd_in,
> _out,
> @@ -284,27 +323,18 @@ _reflink_linux_file_copy(PyObject *self,
> PyObject *args) } else {
> stat_in_acquired = 1;
>
> -/* For the sendfile call, the fd_in file offset must
> be
> - * exactly equal to offset_out. Use lseek to ensure
> - * correct state, in case an EINTR retry caused it to
> - * get out of sync somewhow.
> - */
> -if (lseek(fd_in, offset_out, SEEK_SET) < 0) {
> -error = errno;
> -} else {
> -while (offset_out < stat_in.st_size) {
> -copyfunc_ret = sendfile(fd_out,
> -fd_in,
> -_out,
> -stat_in.st_size -
> offset_out);
> +while (offset_out < stat_in.st_size) {
> +copyfunc_ret = sf_wrapper(fd_out,
> + fd_in,
> + _out,
> + stat_in.st_size -
> offset_out);
> -if (copyfunc_ret < 0) {
> -error = errno;
> -if (errno == EINVAL && !offset_out) {
> -sendfile_works = 0;
> -}
> -break;
> +if (copyfunc_ret < 0) {
> +error = errno;
> +if (errno == EINVAL && !offset_out) {
> +sendfile_works = 0;
> }
> +break;
> }
> }
> }
Looks good, has been checked by chutzpah
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] install-qa-check.d/10ignored-flags: fix LDFLAGS check (bug 455232)
On Sun, 15 Oct 2017 20:59:55 -0700
Zac Medico <zmed...@gentoo.org> wrote:
> From: Felix Janda <felix.ja...@posteo.de>
>
> Since binutils-2.23.51.0.9+ defaults to hash-style=gnu, use
> a __gentoo_check_ldflags__ symbol instead. The check is now
> enabled by adding "-Wl,--defsym=__gentoo_check_ldflags__=0"
> to LDFLAGS. The symbol will be automatically removed when
> the binary is stripped.
>
> Bug: https://bugs.gentoo.org/455232
> Closes: https://github.com/gentoo/portage/pull/220
> ---
> bin/install-qa-check.d/10ignored-flags | 5 +++--
> 1 file changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/bin/install-qa-check.d/10ignored-flags
> b/bin/install-qa-check.d/10ignored-flags index 7aa9eb695..28aec6787
> 100644 --- a/bin/install-qa-check.d/10ignored-flags
> +++ b/bin/install-qa-check.d/10ignored-flags
> @@ -64,9 +64,10 @@ ignored_flag_check() {
> fi
>
> # Check for files built without respecting LDFLAGS
> - if [[ "${LDFLAGS}" == *,--hash-style=gnu* ]] && \
> + if [[ "${LDFLAGS}" == *,--defsym=__gentoo_check_ldflags__*
> ]] && \ ! has binchecks ${RESTRICT} ; then
> - f=$(scanelf -qyRF '#k%p' -k .hash "${ED}")
> + f=$(LC_ALL=C comm -3 <(scanelf -qyRF '#k%p'
> -k .dynsym "${ED}" | LC_ALL=C sort) \
> + <(scanelf -qyRF '#s%p' -s
> __gentoo_check_ldflags__ "${ED}" | LC_ALL=C sort)) if [[ -n ${f} ]] ;
> then echo "${f}" > "${T}"/scanelf-ignored-LDFLAGS.log
> if [ "${QA_STRICT_FLAGS_IGNORED-unset}" =
> unset ] ; then
looks good
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] prepstrip: use debugedit from rpm if necessary (bug 634378)
On Mon, 16 Oct 2017 01:17:23 -0700
Zac Medico <zmed...@gentoo.org> wrote:
> If debugedit is not found in PATH, then use debugedit
> installed by app-arch/rpm (if available).
>
> Suggested-by: Francesco Riosa <viv...@gmail.com>
> Bug: https://bugs.gentoo.org/634378
> ---
> bin/ebuild-helpers/prepstrip | 20
> 1 file changed, 16 insertions(+), 4 deletions(-)
>
> diff --git a/bin/ebuild-helpers/prepstrip
> b/bin/ebuild-helpers/prepstrip index 43caa29c7..ea5c0dc09 100755
> --- a/bin/ebuild-helpers/prepstrip
> +++ b/bin/ebuild-helpers/prepstrip
> @@ -84,7 +84,19 @@ esac
>
> prepstrip_sources_dir=${EPREFIX}/usr/src/debug/${CATEGORY}/${PF}
>
> -type -P debugedit >/dev/null && debugedit_found=true ||
> debugedit_found=false +debugedit=$(type -P debugedit)
> +if [[ -z ${debugedit} ]]; then
> + debugedit_paths=(
> + "${EPREFIX}/usr/libexec/rpm/debugedit"
> + )
> + for x in "${debugedit_paths[@]}"; do
> + if [[ -x ${x} ]]; then
> + debugedit=${x}
> + break
> + fi
> + done
> +fi
> +[[ ${debugedit} ]] && debugedit_found=true || debugedit_found=false
> debugedit_warned=false
>
> __multijob_init
> @@ -101,8 +113,8 @@ save_elf_sources() {
> if ! ${debugedit_found} ; then
> if ! ${debugedit_warned} ; then
> debugedit_warned=true
> - ewarn "FEATURES=installsources is enabled
> but the debugedit binary could not"
> - ewarn "be found. This feature will not work
> unless debugedit is installed!"
> + ewarn "FEATURES=installsources is enabled
> but the debugedit binary could not be"
> + ewarn "found. This feature will not work
> unless debugedit or rpm is installed!" fi
> return 0
> fi
> @@ -112,7 +124,7 @@ save_elf_sources() {
> # since we're editing the ELF here, we should recompute the
> build-id # (the -i flag below). save that output so we don't need to
> recompute # it later on in the save_elf_debug step.
> - buildid=$(debugedit -i \
> + buildid=$("${debugedit}" -i \
> -b "${WORKDIR}" \
> -d "${prepstrip_sources_dir}" \
> -l
> "${tmpdir}/sources/${x##*/}.${BASHPID:-$(__bashpid)}" \
Looks fine to me
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] want_restart_for_use_change: backtrack only if needed (bug 632598)
'--autounmask-backtrack':
> 'y',
> + '--backtrack': 2,
> + },
> + success=False,
> + ambiguous_merge_order=True,
> + mergelist=[
> + ('dev-libs/C-1',
> 'dev-libs/A-2'),
> + 'dev-libs/D-1',
> + ],
> + use_changes={'dev-libs/C-1': {'y':
> True, 'x': True}},
> + ),
> + )
> +
> + playground = ResolverPlayground(
> + ebuilds=ebuilds, installed=installed,
> world=world) +
> + try:
> + for test_case in test_cases:
> + playground.run_TestCase(test_case)
> +
> self.assertEqual(test_case.test_success, True,
> + test_case.fail_msg)
> + finally:
> + playground.cleanup()
Looks fine
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] _solve_..slot_conflicts: handle forced reinstall (bug 632202)
On Fri, 29 Sep 2017 01:15:59 -0700 Zac Medico <zmed...@gentoo.org> wrote: > Fix the slot conflict solver to use the _want_installed_pkg > method to identify installed packages that are intended to > be reinstalled for some reason (such as @preserved-rebuild). > > X-Gentoo-bug: 632202 > X-Gentoo-bug-url: https://bugs.gentoo.org/632202 > --- > pym/_emerge/depgraph.py | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py > index 3b81c5c76..b0149c4dc 100644 > --- a/pym/_emerge/depgraph.py > +++ b/pym/_emerge/depgraph.py > @@ -1337,7 +1337,8 @@ class depgraph(object): > > self._dynamic_config._parent_atoms.get(pkg, > [])) > for parent, atom in all_parent_atoms: > - is_arg_parent = isinstance(parent, > AtomArg) > + is_arg_parent = (inst_pkg is not > None and > + not > self._want_installed_pkg(inst_pkg)) is_non_conflict_parent = parent > not in conflict_pkgs and \ parent not in indirect_conflict_pkgs > looks good -- Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] _solve_..slot_conflicts: make "forced" set recursive (bug 632210)
On Fri, 29 Sep 2017 00:26:01 -0700
Zac Medico <zmed...@gentoo.org> wrote:
> When the slot conflict solver decides that it is "forced"
> to choose a particular package, recursively force the
> dependencies as well. Prior to this fix, substitution of
> @world in the arguments for SlotConflictMaskUpdateTestCase
> caused the test to fail because the solver removed
> boost-build-1.53.0 from the graph event though it had
> added the parent boost-1.53.0 package to the "forced"
> set.
>
> X-Gentoo-bug: 632210
> X-Gentoo-bug-url: https://bugs.gentoo.org/632210
> ---
> pym/_emerge/depgraph.py | 13
> + pym/portage/tests/resolver/test_slot_conflict_update.py
> | 2 +- 2 files changed, 14 insertions(+), 1 deletion(-)
>
> diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py
> index 785c036b8..3b81c5c76 100644
> --- a/pym/_emerge/depgraph.py
> +++ b/pym/_emerge/depgraph.py
> @@ -1457,6 +1457,19 @@ class depgraph(object):
>
> # Remove 'non_conflict_node' and or_tuples from
> 'forced'. forced = set(pkg for pkg in forced if isinstance(pkg,
> Package)) +
> + # Add dependendencies of forced packages.
> + stack = list(forced)
> + traversed = set()
> + while stack:
> + pkg = stack.pop()
> + traversed.add(pkg)
> + for child in conflict_graph.child_nodes(pkg):
> + if (isinstance(child, Package) and
> + child not in traversed):
> + forced.add(child)
> + stack.append(child)
> +
> non_forced = set(pkg for pkg in conflict_pkgs if pkg
> not in forced)
> if debug:
> diff --git a/pym/portage/tests/resolver/test_slot_conflict_update.py
> b/pym/portage/tests/resolver/test_slot_conflict_update.py index
> 331e5788b..f251d01f1 100644 ---
> a/pym/portage/tests/resolver/test_slot_conflict_update.py +++
> b/pym/portage/tests/resolver/test_slot_conflict_update.py @@ -80,7
> +80,7 @@ class SlotConflictUpdateTestCase(TestCase): # this behavior
> makes SlotConflictMaskUpdateTestCase # fail.
> ResolverPlaygroundTestCase(
> - world,
> + ['@world'],
> all_permutations = True,
> options = {"--update": True,
> "--deep": True}, success = True,
looks good
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] depgraph: make _minimize_children deterministic (bug 631894)
On Sun, 24 Sep 2017 11:57:19 -0700 Zac Medico <zmed...@gentoo.org> wrote: > In order for the eliminate_pkg loop to produce deterministic results, > the order of the pkgs list must not be random. Prefer to eliminate > installed packages first, in case rebuilds are needed, and also sort > in ascending order so that older versions are eliminated first. > > X-Gentoo-bug: 631894 > X-Gentoo-bug-url: https://bugs.gentoo.org/631894 > --- > pym/_emerge/depgraph.py | 9 + > 1 file changed, 9 insertions(+) > > diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py > index b4fc5f297..785c036b8 100644 > --- a/pym/_emerge/depgraph.py > +++ b/pym/_emerge/depgraph.py > @@ -3584,6 +3584,15 @@ class depgraph(object): > if > atom_set.findAtomForPackage(pkg2, > modified_use=self._pkg_use_enabled(pkg2)): atom_pkg_graph.add(pkg2, > atom) > + # In order for the following eliminate_pkg > loop to produce > + # deterministic results, the order of the > pkgs list must > + # not be random (bug 631894). Prefer to > eliminate installed > + # packages first, in case rebuilds are > needed, and also sort > + # in ascending order so that older versions > are eliminated > + # first. > + pkgs = (sorted(pkg for pkg in pkgs if > pkg.installed) + > + sorted(pkg for pkg in pkgs if not > pkg.installed)) + > for pkg in pkgs: > eliminate_pkg = True > for atom in > atom_pkg_graph.parent_nodes(pkg): Looks good -- Brian Dolbec
Re: [gentoo-dev] Reviving the Sandbox project
On Fri, 22 Sep 2017 15:06:49 + James McMechan <james_mcmec...@hotmail.com> wrote: > On Fri, Sep 22, 2017 at 5:27 AM, Rich Freeman <ri...@gentoo.org> > wrote: > >On Fri, Sep 22, 2017 at 7:38 AM, Sergei Trofimovich > ><sly...@gentoo.org> wrote: > >> > >> Some other distros try harder to isolate build environment either > >> through chroot and/or private mount/user/network namespace that > >> contains only explicitly specified files in build environment. > >> > >> That would require more cooperation from package manager to fetch > >> list of all visible depends. > >> > > > >I definitely think this is something that would be VERY useful to > >have, because it makes build-time dependency issues almost > >impossible. > > > >However, you don't need that complete solution just to have a > >sandbox. You could simply create a container with the entire > >contents of the main filesystem, but read-only, with the exception > >of the build area. That would replicate the functionality of the > >current sandbox and would be easier than building out just the files > >specified in the dependencies. > > > >The main issue I see with making it dependency aware is performance. > >You would need to walk all the dependencies and their run-time > >dependencies, and the system set, and then individually bind-mount > >the files that belong to them read-only into the container. That > >isn't necessarily difficult, but I imagine that it would be slow. > >Walking the dependencies obviously already happens during resolution > >so that could probably be cached. You could also cache the list of > >files for the system set, and you could even maintain a snapshot of > >these that is used as the base of the container (somebody would need > >to work out the savings of doing this vs the cost of updating it > >every time a system set package changes). > > > >However, the main thing I wanted to point out is that you don't need > >a dependency-aware solution to just replace the existing sandbox. > > > >> I like clear sandbox error reporting. > > > >As far as error reporting goes, you would get kernel-level errors > >like attempting to write to a read-only bind mount, or trying to > >read a file that doesn't exist. To a portage dev it should be > >pretty obvious what is going on. You could add canned text to the > >portage error output at the end. I'm not sure how visible the > >problems would be to portage except to the degree that the build > >system makes them visible > >- it would just see make terminate with a non-zero return. > > > >I would think that containers would be almost completely bug-free > >though. The only thing I could see as an issue is some build system > >that relies on IPC with the host, network access, etc. Namespaces > >themselves are very robust, and the kernel already looks at every > >process as belonging to a set of namespaces even in the default case > >where you only have one set of namespaces on the system, so they > >don't involve different code paths/etc. > > > >-- > >Rich > > Another possibility, could be to have the sandbox be an overlayfs > not of the build directory but of the entire system starting at "/" > and stick that into the container, only CLONE_NEWNS looks to be > needed. > > A container with all writes going to the upper layer of the overlay > could be safe against even #RM -RF / and other disasters, while still > tracking what is happening during the build. > > Should performance be too much of a problem having a bind mount of > the build directory on top of the overlay should give normal > performance to everything that is obeying good practice. > > After the build completes the directory that was mounted as upper > could be scanned to find any wayward writes that had occurred... > > This would not require LD_PRELOAD or ptrace eliminating most of the > "high magic" currently used. > > Just yesterday I had a lot of ptrace failures while trying something > odd with ROOT= and custom USE > > Jim McMechan I kinda like this idea, It looks to me to have pretty much all the benefits of a sandbox and nearly none of the drawbacks. Sounds like it should get more research into this idea, figure out it's limitations and if there are any caveats. -- Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] gnome2-utils postinst: Restrict file types to fix false positives
On Tue, 29 Aug 2017 10:20:40 +0200
Michał Górny <mgo...@gentoo.org> wrote:
> Restrict the file types checked by the gtk-icon-cache-update check
> to match the one used inside the tool. This ensures that we do not
> report any false positives for files that are intentionally skipped.
>
> Bug: https://bugs.gentoo.org/629148
> ---
> bin/postinst-qa-check.d/50gnome2-utils | 7 ++-
> 1 file changed, 6 insertions(+), 1 deletion(-)
>
> diff --git a/bin/postinst-qa-check.d/50gnome2-utils
> b/bin/postinst-qa-check.d/50gnome2-utils index d0cbb4037..84025ab83
> 100644 --- a/bin/postinst-qa-check.d/50gnome2-utils
> +++ b/bin/postinst-qa-check.d/50gnome2-utils
> @@ -6,7 +6,12 @@ gnome2_icon_cache_check() {
> # gnome2_icon_cache_update updates only themes with
> an index [[ -f ${d}/index.theme ]] || continue
>
> - find_args=()
> + find_args=(
> + # gtk-update-icon-cache supports only
> specific file
> + # suffixes; match that to avoid false
> positives
> + '(' -name '*.png' -o -name '*.svg'
> + -o -name '*.xpm' -o -name '*.icon'
> ')'
> + )
> # if the cache does not exist at all, we complain
> for any file # otherwise, we look for files newer than the cache
> [[ -f ${d}/icon-theme.cache ]] &&
Works for me, since Gilles approved the change :)
--
Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] Rewrite doins in python (bug 624526)
On Mon, 14 Aug 2017 13:31:53 -0700 Zac Medico <zmed...@gentoo.org> wrote: > On 08/14/2017 12:49 PM, M. J. Everitt wrote: > > On 14/08/17 08:39, Zac Medico wrote: > >> From: Hidehiko Abe <hideh...@chromium.org> > >> > >> doins is written in bash. However, specifically in case that > >> too many files are installed, it is very slow. > >> This CL rewrites the script in python for performance. > >> > >> BUG=chromium:712659 > >> TEST=time (./setup_board --forace && \ > >> ./build_package --withdev && \ > >> ./build_image --noenable_rootfs_verification test) > >> ===Before=== > >> real21m35.445s > >> user93m40.588s > >> sys 21m31.224s > >> > >> ===After=== > >> real17m30.106s > >> user94m1.812s > >> sys 20m13.468s > >> > > I know I'm gonna get chewed out on this one, but here goes > > anyway ... > > > > Surely for a package like chromium, who's build time is already in > > the 'hours' range anyway, surely a couple of minutes gain for the > > install phase is neither here nor there?! If there were some > > genuine filesystem iop gains/etc for this change, I think I'd > > likely support it further .. > > It's going to reduce time, power consumption, and heat generation for > all portage users. Also, we can use portage.util.file_copy to optimize > it further with zero-copy, reflink, and sparse file support. > > > On this basis, what do the performance differences look like on an > > 'average' package > > Well, it's very inefficient to fork/exec the install command for many > files as the existing bash implementation does. The performance > difference is related to the number of files. > > > .. and are there any regressions in this regard?! > > It's supposed to fallback to calling the install command if there are > any unrecognized options, so the intention is for 100% compatibility. > > > I take issue with the copyright assignment, as I believe the > > legal .. err .. IANAL devs are campaigning for full rights to be > > owned and enforced by Gentoo Inc LLC in the US .. even if they have > > no idea what that means or does .. :] > > Since it's a BSD-style license, we can copy the code into our project > as long as we retain the copyright notice. Patches look fine to me, glad for the speedup :) -- Brian Dolbec
Re: [gentoo-portage-dev] [PATCH] emerge --getbinpkg: https support for If-Modified-Since
On Mon, 31 Jul 2017 00:11:09 -0700
Zac Medico <zmed...@gentoo.org> wrote:
> When https certificate and hostname verification is enabled for
> stdlib http clients (PEP 476), use python for If-Modified-Since
> header support. When python lacks PEP 476 support, continue to
> use FETCHCOMMAND for https certificate and hostname verification
> (see security bug 469888).
>
> X-Gentoo-bug: 625246
> X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=625246
> ---
> pym/portage/dbapi/bintree.py | 10 ++
> pym/portage/util/_urlopen.py | 12
> 2 files changed, 18 insertions(+), 4 deletions(-)
>
> diff --git a/pym/portage/dbapi/bintree.py
> b/pym/portage/dbapi/bintree.py index c833968c2..95bd5dbf8 100644
> --- a/pym/portage/dbapi/bintree.py
> +++ b/pym/portage/dbapi/bintree.py
> @@ -18,7 +18,7 @@ portage.proxy.lazyimport.lazyimport(globals(),
> 'portage.util:atomic_ofstream,ensure_dirs,normalize_path,' +
> \ 'writemsg,writemsg_stdout',
> 'portage.util.path:first_existing',
> - 'portage.util._urlopen:urlopen@_urlopen',
> + 'portage.util._urlopen:urlopen@_urlopen,have_pep_476@_have_pep_476',
> 'portage.versions:best,catpkgsplit,catsplit,_pkg_str',
> )
>
> @@ -851,9 +851,9 @@ class binarytree(object):
> download_timestamp +
> ttl > time.time(): raise UseCachedCopyOfRemoteIndex()
>
> - # Don't use urlopen for https, since
> it doesn't support
> - # certificate/hostname verification
> (bug #469888).
> - if parsed_url.scheme not in
> ('https',):
> + # Don't use urlopen for https, unless
> + # PEP 476 is supported (bug #469888).
> + if parsed_url.scheme not in
> ('https',) or _have_pep_476(): try:
> f = _urlopen(url,
> if_modified_since=local_timestamp) if hasattr(f, 'headers') and
> f.headers.get('timestamp', ''): @@ -965,6 +965,8 @@ class
> binarytree(object): "\n")
> rmt_idx = pkgindex
> except EnvironmentError as e:
> + # This includes URLError which is
> raised for SSL
> + # certificate errors when PEP 476 is
> supported. writemsg(_("\n\n!!! Error fetching binhost package" \
> " info from '%s'\n") %
> _hide_url_passwd(base_url)) # With Python 2, the EnvironmentError
> message may diff --git a/pym/portage/util/_urlopen.py
> b/pym/portage/util/_urlopen.py index 4cfe183b1..fc9db74a0 100644
> --- a/pym/portage/util/_urlopen.py
> +++ b/pym/portage/util/_urlopen.py
> @@ -26,6 +26,18 @@ if sys.hexversion >= 0x300:
> # and the file-'mtime'
> TIMESTAMP_TOLERANCE = 5
>
> +
> +def have_pep_476():
> + """
> + Test whether ssl certificate verification is enabled by
> default for
> + stdlib http clients (PEP 476).
> +
> + @returns: bool, True if ssl certificate verification is
> enabled by
> + default
> + """
> + return hasattr(__import__('ssl'),
> '_create_unverified_context') +
> +
> def urlopen(url, if_modified_since=None):
> parse_result = urllib_parse.urlparse(url)
> if parse_result.scheme not in ("http", "https"):
looks fine
--
Brian Dolbec
Re: [gentoo-portage-dev] Leader election
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On Sun, 2 Jul 2017 11:28:09 -0700 Brian Dolbec <dol...@gentoo.org> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA512 > > On Sat, 24 Jun 2017 20:20:16 -0700 > Brian Dolbec <dol...@gentoo.org> wrote: > > > On Sun, 18 Jun 2017 15:48:44 +0200 > > Alexander Berntsen <berna...@gentoo.org> wrote: > > > > > Friends, > > > > > > It's that time of year. We're having a leader election again, as > > > well as a general development meeting. The agenda will be updated > > > in more detail at: > > > https://wiki.gentoo.org/wiki/Project:Portage/Meetings > > > > > > Please schedule a time at: http://whenisgood.net/portage > > > > > > Thanks. > > > - -- > > > Alexander > > > berna...@gentoo.org > > > https://secure.plaimi.net/~alexander > > > > > > > I've picked Sunday, July 2 at 4:00 PM UTC > > > > We've decided (the members in attendance), to do the lead election via > email. > > So, nominations are open from now to July 5, 2017. > > Voting will be closed July 10, 2017, results posted here again. > OK, looks the all cast votes were for Zac. So, by unanimous decision, I hereby declare Zac as our new project lead. Congrats Zac. - -- Brian Dolbec -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQKTBAEBCgB9FiEEpdfHTggcxw20pKr1+70IcnWCDtgFAllk+z1fFIAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEE1 RDdDNzRFMDgxQ0M3MERCNEE0QUFGNUZCQkQwODcyNzU4MjBFRDgACgkQ+70IcnWC Dtj0Ew//Xl+GzPKM3+0EKKMJn+Q6G2/7z5byPFl6bPevYFTPlmmN7a+AxULw+NfB 7+EwqYdVzLk51GLzBk4Y3Pgt7gFEPbigBB/w2Fubb24QvNzwUiZ3O6N8F4o7BbVh 8T0yuRYpFn79mWIOACudEgCt9HfSSAYcCREjIWyqrXSxuqCs4SKGRhReKx8S2XGt eUjD67HCCUKGmnVIZJrOFiQomDegLxl/gUqB2kHvzZWnpRVfYGHJbLXGCQtR8fro Qm2qvnnIbyxtYf3xaCKLPTCMqYwOtQuEL2xwuJ/KVvIcRErxJIYgutksjJIodEWj vE18hE7RzVsIyiIB/wvlJk2+ViTRSwlkRY2af6Sq0QakPlI+OtrKgqxPNQE041hl kMWogpawZm3z5hCwh9TuGW9ZCbI3ego6woTYp9DE9B953UiYwg0MbjBnh4AFeE0O Qy23zybjrD6ANzc1328W3hZpU5jjWnWEAvOlaH/pGH7vYBgdMfPcs0vyijkKE6gg JOwrFGpIAqcFBkIfa6cUgc2OQX3yBKZJgrqfFHniRo0KGXt3Jl9e4+eG0EK6CcqZ eVSAjflo+5mCVjaVGugXrcTd3O6/SVzsUH+/woazyLKogc1x+4rzk0dVmfDBPuow zFEbdcPzlJW6wxpPhCjHz1GFD/MzZsEvAxGW+q+GF71GLlJXeKI= =hDeZ -END PGP SIGNATURE-
