[gentoo-portage-dev] [PATCH v2] Stop implicitly manipulating `NO_COLOR`/`NOCOLOR`
Stop implicitly forcing `NO_COLOR` and `NOCOLOR` in ebuild environment.
This is undesired for several reasons:
1. It makes it impossible to control color for emerge output
independently of command output, e.g. when one goes to a pty while
the other goes to logs.
2. It makes it impossible to get color output in logs when running
`emerge --jobs ...`.
3. Forcing `NO_COLOR=1` turns out to cause random test failures,
and while fixing them is commendable, it is a pain for arch testing
and it is currently blocking stabilization requests.
With the new approach, the color output in programs is consistent
between using ``--jobs`` or ``--quiet-build``, and not. Therefore,
both cases generate uniform logs. In order to obtain logs free of color
codes, one can either filter them via `ansifilter(1)` (as the manpages
already recommend) or explicitly set `NO_COLOR`.
Furthermore, one can combine color-free build output (for clean logs)
with colorful emerge output by using:
NO_COLOR=true emerge --color y ...
Bug: https://bugs.gentoo.org/918515
Signed-off-by: Michał Górny
---
NEWS | 5 +
bin/ebuild | 4
lib/_emerge/AbstractEbuildProcess.py | 6 --
lib/_emerge/actions.py | 5 -
lib/portage/tests/resolver/ResolverPlayground.py | 3 ---
5 files changed, 5 insertions(+), 18 deletions(-)
Changes in v2: add a NEWS entry
diff --git a/NEWS b/NEWS
index f1305744f..eb5eb4ab0 100644
--- a/NEWS
+++ b/NEWS
@@ -9,6 +9,11 @@ Release notes take the form of the following optional
categories:
portage-3.0.56 (UNRELEASED)
--
+Breaking changes:
+* ebuild: no longer implicitly set NOCOLOR and NO_COLOR when color output
+ is disabled for emerge itself. In order to obtain color-free logs, please
+ either explicitly set NO_COLOR environment variable or use e.g. ansifilter.
+
Features:
* sync: git: Add sync-git-verify-max-age-days option.
diff --git a/bin/ebuild b/bin/ebuild
index cbefb5816..69db474c0 100755
--- a/bin/ebuild
+++ b/bin/ebuild
@@ -132,10 +132,6 @@ def main():
or not sys.stdout.isatty()
):
portage.output.nocolor()
-portage.settings.unlock()
-portage.settings["NO_COLOR"] = "true"
-portage.settings.backup_changes("NO_COLOR")
-portage.settings.lock()
apply_priorities(portage.settings)
diff --git a/lib/_emerge/AbstractEbuildProcess.py
b/lib/_emerge/AbstractEbuildProcess.py
index 96d91b5da..3a8a18b5f 100644
--- a/lib/_emerge/AbstractEbuildProcess.py
+++ b/lib/_emerge/AbstractEbuildProcess.py
@@ -80,12 +80,6 @@ class AbstractEbuildProcess(SpawnProcess):
self._async_wait()
return
-if self.background:
-# Automatically prevent color codes from showing up in logs,
-# since we're not displaying to a terminal anyway.
-self.settings["NOCOLOR"] = "true"
-self.settings["NO_COLOR"] = "true"
-
start_ipc_daemon = False
if self._enable_ipc_daemon:
self.settings.pop("PORTAGE_EBUILD_EXIT_FILE", None)
diff --git a/lib/_emerge/actions.py b/lib/_emerge/actions.py
index 37264114e..dbd9707a8 100644
--- a/lib/_emerge/actions.py
+++ b/lib/_emerge/actions.py
@@ -2749,15 +2749,10 @@ def adjust_config(myopts, settings):
if "--color" in myopts:
if "y" == myopts["--color"]:
portage.output.havecolor = 1
-settings["NO_COLOR"] = ""
else:
portage.output.havecolor = 0
-settings["NO_COLOR"] = "true"
-settings.backup_changes("NO_COLOR")
elif settings.get("TERM") == "dumb" or not sys.stdout.isatty():
portage.output.havecolor = 0
-settings["NO_COLOR"] = "true"
-settings.backup_changes("NO_COLOR")
if "--pkg-format" in myopts:
settings["PORTAGE_BINPKG_FORMAT"] = myopts["--pkg-format"]
diff --git a/lib/portage/tests/resolver/ResolverPlayground.py
b/lib/portage/tests/resolver/ResolverPlayground.py
index 475c4aaac..115800a60 100644
--- a/lib/portage/tests/resolver/ResolverPlayground.py
+++ b/lib/portage/tests/resolver/ResolverPlayground.py
@@ -591,9 +591,6 @@ class ResolverPlayground:
"PORTAGE_TMPDIR": os.path.join(self.eroot, "var/tmp"),
}
-if portage.util.no_color(os.environ):
-make_conf["NO_COLOR"] = os.environ["NO_COLOR"]
-
# Pass along PORTAGE_USERNAME and PORTAGE_GRPNAME since they
# need to be inherited by ebuild subprocesses.
if "PORTAGE_USERNAME" in os.environ:
--
2.43.0
Re: [gentoo-portage-dev] [PATCH] Stop implicitly manipulating `NO_COLOR`/`NOCOLOR`
On Sun, 2023-11-26 at 18:33 +0100, Ulrich Mueller wrote: > > > > > > On Sun, 26 Nov 2023, Michał Górny wrote: > > > 3. Forcing `NO_COLOR=1` turns out to cause random test failures, > > and while fixing them is commendable, it is a pain for arch testing > > and it is currently blocking stabilization requests. > > I'd argue that test suites that fail because of NO_COLOR are broken. > IIUC, these failures will show up for users who set NO_COLOR=1 in their > make.conf or in the environment? Sure, they are broken. However, as I said I'd rather fix them in my spare time than have to fix them to unblock a security stablereq because implicit Portage logic blocks all AT work on a minor issue that's non- trivial to fix. > > > With the new approach, the color output in programs is consistent > > between using ``--jobs`` or ``--quiet-build``, and not. Therefore, > > both cases generate uniform logs. In order to obtain logs free of color > > codes, one can either filter them via `ansifilter(1)` (as the manpages > > already recommend) or explicitly set `NO_COLOR`. > > Will this still guarantee that NO_COLOR=1 from the environment is always > passed on to the build system? > > Otherwise, it would break ebuild-mode in some configurations. > Yes, I've tested it and it worked. Both for passing from external env and via make.conf. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
[gentoo-portage-dev] [PATCH] Stop implicitly manipulating `NO_COLOR`/`NOCOLOR`
Stop implicitly forcing `NO_COLOR` and `NOCOLOR` in ebuild environment.
This is undesired for several reasons:
1. It makes it impossible to control color for emerge output
independently of command output, e.g. when one goes to a pty while
the other goes to logs.
2. It makes it impossible to get color output in logs when running
`emerge --jobs ...`.
3. Forcing `NO_COLOR=1` turns out to cause random test failures,
and while fixing them is commendable, it is a pain for arch testing
and it is currently blocking stabilization requests.
With the new approach, the color output in programs is consistent
between using ``--jobs`` or ``--quiet-build``, and not. Therefore,
both cases generate uniform logs. In order to obtain logs free of color
codes, one can either filter them via `ansifilter(1)` (as the manpages
already recommend) or explicitly set `NO_COLOR`.
Furthermore, one can combine color-free build output (for clean logs)
with colorful emerge output by using:
NO_COLOR=true emerge --color y ...
Bug: https://bugs.gentoo.org/918515
---
bin/ebuild | 4
lib/_emerge/AbstractEbuildProcess.py | 6 --
lib/_emerge/actions.py | 5 -
lib/portage/tests/resolver/ResolverPlayground.py | 3 ---
4 files changed, 18 deletions(-)
diff --git a/bin/ebuild b/bin/ebuild
index cbefb5816..69db474c0 100755
--- a/bin/ebuild
+++ b/bin/ebuild
@@ -132,10 +132,6 @@ def main():
or not sys.stdout.isatty()
):
portage.output.nocolor()
-portage.settings.unlock()
-portage.settings["NO_COLOR"] = "true"
-portage.settings.backup_changes("NO_COLOR")
-portage.settings.lock()
apply_priorities(portage.settings)
diff --git a/lib/_emerge/AbstractEbuildProcess.py
b/lib/_emerge/AbstractEbuildProcess.py
index 96d91b5da..3a8a18b5f 100644
--- a/lib/_emerge/AbstractEbuildProcess.py
+++ b/lib/_emerge/AbstractEbuildProcess.py
@@ -80,12 +80,6 @@ class AbstractEbuildProcess(SpawnProcess):
self._async_wait()
return
-if self.background:
-# Automatically prevent color codes from showing up in logs,
-# since we're not displaying to a terminal anyway.
-self.settings["NOCOLOR"] = "true"
-self.settings["NO_COLOR"] = "true"
-
start_ipc_daemon = False
if self._enable_ipc_daemon:
self.settings.pop("PORTAGE_EBUILD_EXIT_FILE", None)
diff --git a/lib/_emerge/actions.py b/lib/_emerge/actions.py
index 37264114e..dbd9707a8 100644
--- a/lib/_emerge/actions.py
+++ b/lib/_emerge/actions.py
@@ -2749,15 +2749,10 @@ def adjust_config(myopts, settings):
if "--color" in myopts:
if "y" == myopts["--color"]:
portage.output.havecolor = 1
-settings["NO_COLOR"] = ""
else:
portage.output.havecolor = 0
-settings["NO_COLOR"] = "true"
-settings.backup_changes("NO_COLOR")
elif settings.get("TERM") == "dumb" or not sys.stdout.isatty():
portage.output.havecolor = 0
-settings["NO_COLOR"] = "true"
-settings.backup_changes("NO_COLOR")
if "--pkg-format" in myopts:
settings["PORTAGE_BINPKG_FORMAT"] = myopts["--pkg-format"]
diff --git a/lib/portage/tests/resolver/ResolverPlayground.py
b/lib/portage/tests/resolver/ResolverPlayground.py
index 475c4aaac..115800a60 100644
--- a/lib/portage/tests/resolver/ResolverPlayground.py
+++ b/lib/portage/tests/resolver/ResolverPlayground.py
@@ -591,9 +591,6 @@ class ResolverPlayground:
"PORTAGE_TMPDIR": os.path.join(self.eroot, "var/tmp"),
}
-if portage.util.no_color(os.environ):
-make_conf["NO_COLOR"] = os.environ["NO_COLOR"]
-
# Pass along PORTAGE_USERNAME and PORTAGE_GRPNAME since they
# need to be inherited by ebuild subprocesses.
if "PORTAGE_USERNAME" in os.environ:
--
2.43.0
Re: [gentoo-portage-dev] Fixing EAPI 8 --enable-static once and for all
On Sat, 2022-11-19 at 09:33 +0100, Ulrich Mueller wrote: > > > > > > On Sat, 19 Nov 2022, David Seifert wrote: > > > With this extensive analysis, I believe this patch to be safe. > > Still looks like an incompatible change, so it will need an EAPI bump. > > EAPI 9 feature bug is here: https://bugs.gentoo.org/815169 I dare say we've done worse retroactive changes. -- Best regards, Michał Górny
Re: [gentoo-portage-dev] Normaliser function for distfiles
On Mon, 2022-05-16 at 19:37 +0200, Markus Walter wrote: > Hello all, > > is it possible to do the following: after fetching a distfile portage runs > an external normaliser program specified in an ebuild before checking the > hash? > > My use case is the following: I would like to improve the gs-elpa program > and provide a precomputed overlay for melpa. However the melpa distfiles are > rebuilt everyday and cause checksum failures. However the only thing > changing are the timestamps. Hence if a normaliser program could simply set > all timestamps to some predefined value (say 1.1.1970) then this problem > should vanish. > This would require a new EAPI. We don't really want more Portage- specific hacks that are going to break for everyone not using Portage or the very specific Portage version. I'm not saying that it's not doable but I'm not convinced the added complexity is really worth the effort, especially given that this looks like a very special corner case. In the end, fixing Melpa is the right thing to do. For a start, you'd have to ensure that the "normalizer script" (or its dependencies, if you put it in the repo) is available at the time of fetching. This pretty much goes back to the problem of "fetch dependencies", and requires a major design change in Portage that reduces separation between fetching and installing that we have now. I mean, right now Portage pretty much assumes that you can do a `--fetchonly` with no extra packages necessary. The "normalizer" wouldn't be trivial either. In the end, we're talking about getting 100% consistent results on all platforms, over a reasonably long timeframe. -- Best regards, Michał Górny
Re: [gentoo-portage-dev] The build system (and install layout) of Portage
On Thu, 2022-03-17 at 20:57 +, James Le Cuirot wrote: > On Thu, 2022-03-17 at 18:22 +0100, Michał Górny wrote: > > An alternative is to go back to using (at least partially) Makefiles or > > Meson. However, that would have the important drawback that we'd lose > > the ability to install Portage as a regular Python package (e.g. inside > > a virtualenv). > > What is the use case for doing that? I thought maybe testing, but then you can > run Portage and its unit tests in-place without installing it at all right > now. Using Portage as a library for package inspection, etc. -- i.e. mostly CI purposes. -- Best regards, Michał Górny
[gentoo-portage-dev] The build system (and install layout) of Portage
Hi, everyone. You've probably had the opportunity to hear that a lot has changed in Python packaging since Portage's setup.py was written in 2014. There were some minor changes to keep it working since but it's time to reconsider. Long story short, distutils is strongly deprecated, setuptools deprecated most of the customizations (and we're relying heavily on customizations), PEP 517 doesn't cover our use cases exactly... and it's quite likely that sooner or later our build system will fall apart. On top of that, setuptools is going through a stage of "let's vendor a new dependency every week", so it doesn't look like a feasible long-term solution. A large part of the problem is that Portage is heavily relying on non-Pythonic idioms for installing stuff. At this point, I'd love to use flit for Portage, as that's a nice build system with upstream that understands packagers. I mean, if someone tells me they want to avoid vendoring stuff *and* creating cyclic dependencies (all other Python build systems vendor tons of stuff right now), then I can't but appreciate that very much. That said, I don't think flit is powerful enough to handle all the intricacies Portage requires, starting with the fancy way it's installing scripts to many different directories. An alternative is to go back to using (at least partially) Makefiles or Meson. However, that would have the important drawback that we'd lose the ability to install Portage as a regular Python package (e.g. inside a virtualenv). At this point, I've pretty much lost all motivation to work on it. Nevertheless, it's something that needs to be done eventually. Does anyone have some idea, motivation and will to transition Portage to another build system? -- Best regards, Michał Górny
Re: [gentoo-portage-dev] [PATCH] portage.output: Replace darkblue colors with teal
On Sat, 2021-12-04 at 10:15 +0100, Fabian Groffen wrote: > On 04-12-2021 10:13:09 +0100, Michał Górny wrote: > > On Sat, 2021-12-04 at 09:56 +0100, Fabian Groffen wrote: > > > Why don't you change your color.map instead of changing the default for > > > everyone? > > > > Why should we keep a stupid default? Should we optimize Gentoo for > > people who don't want to be able to read Portage's output? > > You're assuming everyone uses the terminal in the way you do. I simply > don't think that's how the world looks like. On the other hand, you're assuming that everyone uses the terminal in the way you do. > No need for calling things stupid, IMO. Using dark blue on black background is stupid. -- Best regards, Michał Górny
Re: [gentoo-portage-dev] [PATCH] portage.output: Replace darkblue colors with teal
On Sat, 2021-12-04 at 09:56 +0100, Fabian Groffen wrote: > Why don't you change your color.map instead of changing the default for > everyone? Why should we keep a stupid default? Should we optimize Gentoo for people who don't want to be able to read Portage's output? -- Best regards, Michał Górny
[gentoo-portage-dev] [PATCH] portage.output: Replace darkblue colors with teal
The "darkblue" color is often barely visible on dark terminals which
makes reading emerge output really hard (I basically have to copy-paste
it a lot in order to be able to read it at all). Replace it with teal
that does not seem to have any significant use in the output.
Signed-off-by: Michał Górny
---
lib/portage/output.py | 6 +++---
man/color.map.5 | 6 +++---
2 files changed, 6 insertions(+), 6 deletions(-)
The before-after screenshot can be found on the PR:
https://github.com/gentoo/portage/pull/776
diff --git a/lib/portage/output.py b/lib/portage/output.py
index 42f487f8a..77375b012 100644
--- a/lib/portage/output.py
+++ b/lib/portage/output.py
@@ -157,7 +157,7 @@ _styles["UNMERGE_WARN"] = ("red",)
_styles["SECURITY_WARN"] = ("red",)
_styles["MERGE_LIST_PROGRESS"] = ("yellow",)
_styles["PKG_BLOCKER"] = ("red",)
-_styles["PKG_BLOCKER_SATISFIED"] = ("darkblue",)
+_styles["PKG_BLOCKER_SATISFIED"] = ("teal",)
_styles["PKG_MERGE"] = ("darkgreen",)
_styles["PKG_MERGE_SYSTEM"] = ("darkgreen",)
_styles["PKG_MERGE_WORLD"] = ("green",)
@@ -165,8 +165,8 @@ _styles["PKG_BINARY_MERGE"] = ("purple",)
_styles["PKG_BINARY_MERGE_SYSTEM"] = ("purple",)
_styles["PKG_BINARY_MERGE_WORLD"] = ("fuchsia",)
_styles["PKG_UNINSTALL"] = ("red",)
-_styles["PKG_NOMERGE"] = ("darkblue",)
-_styles["PKG_NOMERGE_SYSTEM"] = ("darkblue",)
+_styles["PKG_NOMERGE"] = ("teal",)
+_styles["PKG_NOMERGE_SYSTEM"] = ("teal",)
_styles["PKG_NOMERGE_WORLD"] = ("blue",)
_styles["PROMPT_CHOICE_DEFAULT"] = ("green",)
_styles["PROMPT_CHOICE_OTHER"] = ("red",)
diff --git a/man/color.map.5 b/man/color.map.5
index 288bf7fbd..92a1baa91 100644
--- a/man/color.map.5
+++ b/man/color.map.5
@@ -40,7 +40,7 @@ Defines color used for numbers indicating merge progress.
\fBPKG_BLOCKER\fR = \fI"red"\fR
Defines color used for unsatisfied blockers.
.TP
-\fBPKG_BLOCKER_SATISFIED\fR = \fI"darkblue"\fR
+\fBPKG_BLOCKER_SATISFIED\fR = \fI"teal"\fR
Defines color used for satisfied blockers.
.TP
\fBPKG_MERGE\fR = \fI"darkgreen"\fR
@@ -63,10 +63,10 @@ package.
Defines color used for world packages planned to be merged using a binary
package.
.TP
-\fBPKG_NOMERGE\fR = \fI"darkblue"\fR
+\fBPKG_NOMERGE\fR = \fI"teal"\fR
Defines color used for packages not planned to be merged.
.TP
-\fBPKG_NOMERGE_SYSTEM\fR = \fI"darkblue"\fR
+\fBPKG_NOMERGE_SYSTEM\fR = \fI"teal"\fR
Defines color used for system packages not planned to be merged.
.TP
\fBPKG_NOMERGE_WORLD\fR = \fI"blue"\fR
--
2.34.1
[gentoo-portage-dev] [PATCH] emerge: Do not resolve ambiguous package names via installed pkgs
When dealing with ambiguous package names on command-line, do not use installed packages to resolve them. This is counterintuitive in the best case, and probably the wrong answer most of the time. After all, if a user calls emerge with a specific package name, it is quite likely that he intends to install a missing package rather than rebuild or upgrade an individual package (though the latter can also happen). Rather than making assumptions that can go wrong, just display the ambiguity message as usual. Bug: https://bugs.gentoo.org/828059 Signed-off-by: Michał Górny --- lib/_emerge/depgraph.py | 15 --- 1 file changed, 15 deletions(-) diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py index 07431c8ee..f6549eba6 100644 --- a/lib/_emerge/depgraph.py +++ b/lib/_emerge/depgraph.py @@ -4806,21 +4806,6 @@ class depgraph: if len(non_virtual_cps) == 1: installed_cp_set = non_virtual_cps -if len(expanded_atoms) > 1 and len(installed_cp_set) == 1: -installed_cp = next(iter(installed_cp_set)) -for atom in expanded_atoms: -if atom.cp == installed_cp: -available = False -for pkg in self._iter_match_pkgs_any( -root_config, atom.without_use, onlydeps=onlydeps -): -if not pkg.installed: -available = True -break -if available: -expanded_atoms = [atom] -break - # If a non-virtual package and one or more virtual packages # are in expanded_atoms, use the non-virtual package. if len(expanded_atoms) > 1: -- 2.34.1
Re: [gentoo-portage-dev] [RFC] LTS branch of Portage
On Tue, 2021-10-05 at 13:16 -0400, Michael Orlitzky wrote: > On Tue, 2021-10-05 at 17:13 +0200, Michał Górny wrote: > > > > > 2. What happens to the LTS branch when the next EAPI is released? > > > > > > > I haven't really thought about it. Are you suggesting that we could > > bump 'master' Portage to newer EAPI earlier or...? > > > > I just mean that, a priori, the LTS branch won't be able to install > ebuilds that use the next EAPI. Backporting an entire EAPI doesn't seem > appropriate for a stable series; but maintaining an increasingly- > obsolete branch at that point doesn't make much sense either. Ah, that makes sense. Indeed, I suppose a new EAPI will be a good point of switching master to LTS. -- Best regards, Michał Górny
Re: [gentoo-portage-dev] [RFC] LTS branch of Portage
On Tue, 2021-10-05 at 10:15 -0400, Michael Orlitzky wrote: > On Tue, 2021-10-05 at 10:31 +0200, Michał Górny wrote: > > Hi, everyone. > > > > I've been thinking about this for some time already, and the recent > > FILESDIR mess seems to confirm it: I'd like to start a more stable LTS > > branch of Portage. > > > > I think this is healthy for most software projects, but two things > stand out to me for portage specifically: > > 1. Most portage users can fake this already by telling portage to > accept only stable keywords for sys-apps/portage. I know it's not > exactly the same, but it provides many of the same benefits. That's not going to be possible long-term when I start making more changes ;-). > 2. What happens to the LTS branch when the next EAPI is released? > I haven't really thought about it. Are you suggesting that we could bump 'master' Portage to newer EAPI earlier or...? -- Best regards, Michał Górny
[gentoo-portage-dev] [RFC] LTS branch of Portage
Hi, everyone. I've been thinking about this for some time already, and the recent FILESDIR mess seems to confirm it: I'd like to start a more stable LTS branch of Portage. Roughly, the idea is that: - master becomes 3.1.x, and primary development happens there - 3.0.x becomes the LTS branch and only major bugfixes are backported there As things settle down in the future, master would become 3.2.x, 3.1.x would become LTS, 3.0.x will be discontinued and so on. WDYT? -- Best regards, Michał Górny
Re: [gentoo-portage-dev] [PATCH 0/4] Output rewrite for better clarify and greppability
On Tue, 2021-09-28 at 16:20 +0200, Michał Górny wrote: > Hi, > > Ok, so it's more major than I originally intended but I think it's > a good direction (once you get used to it). Roughly: > I'm retracting this in favor of whatever gentoo-dev participants come up with. -- Best regards, Michał Górny
Re: [gentoo-portage-dev] [PATCH 0/4] Output rewrite for better clarify and greppability
On Tue, 2021-09-28 at 11:28 -0400, Mike Gilbert wrote: > On Tue, Sep 28, 2021 at 10:20 AM Michał Górny wrote: > > > > Hi, > > > > Ok, so it's more major than I originally intended but I think it's > > a good direction (once you get used to it). Roughly: > > > > 1. All bash color vars are now prefixed with `PORTAGE_COLOR_` to avoid > >accidental collisions with ebuild vars (e.g. ebuild setting > >`GOOD=foo` broke `elog` before). > > 2. There are specific color vars for all kinds of output functions, > >and now `einfo` messages use distinct color (dark green) from `elog`, > >and `eqawarn` (brown) from `ewarn`. > > 3. Messages are now prefixed by their kind, making it possible to > >distinguish them without colors and grep for specific kind of logs: > >- `[--]` for einfo & ebegin > >- `[II]` for elog > >- `[WW]` for ewarn > >- `[QA]` for eqawarn > >- `[EE]` for eerror > > 4. Finally, I've replaced most of `>>>` and `!!!` in Portage output with > >four `>>>>` and `` to align the output again. > > I like it. > > Maybe add a reference to this bug? https://bugs.gentoo.org/728046 > Oh right, I also need to work on removing redundant 'QA Notice:' thingies. -- Best regards, Michał Górny
[gentoo-portage-dev] [PATCH 3/4] Use verbose prefixes for output messages
Use verbose prefixes for output functions einfo, elog... in order to
make the message types distinguishable on non-color terminals,
and to enable the possibility of grepping them. The prefixes are
roughly inspired by Xorg logs, and are:
- `[--]` for einfo (and ebegin)
- `[II]` for elog
- `[WW]` for ewarn
- `[QA]` for eqawarn
- `[EE]` for eerror
Signed-off-by: Michał Górny
---
bin/isolated-functions.sh | 12 ++--
lib/portage/output.py | 12 ++--
2 files changed, 12 insertions(+), 12 deletions(-)
diff --git a/bin/isolated-functions.sh b/bin/isolated-functions.sh
index df806d403..c7fda6784 100644
--- a/bin/isolated-functions.sh
+++ b/bin/isolated-functions.sh
@@ -270,7 +270,7 @@ eqawarn() {
__elog_base QA "$*"
[[ ${RC_ENDCOL} != "yes" && ${LAST_E_CMD} == "ebegin" ]] && echo >&2
echo -e "$@" | while read -r ; do
- echo " ${PORTAGE_COLOR_QAWARN}*${PORTAGE_COLOR_NORMAL}
${REPLY}" >&2
+ echo "${PORTAGE_COLOR_QAWARN}[QA]${PORTAGE_COLOR_NORMAL}
${REPLY}" >&2
done
LAST_E_CMD="eqawarn"
return 0
@@ -280,7 +280,7 @@ elog() {
__elog_base LOG "$*"
[[ ${RC_ENDCOL} != "yes" && ${LAST_E_CMD} == "ebegin" ]] && echo >&2
echo -e "$@" | while read -r ; do
- echo " ${PORTAGE_COLOR_LOG}*${PORTAGE_COLOR_NORMAL} ${REPLY}"
>&2
+ echo "${PORTAGE_COLOR_LOG}[II]${PORTAGE_COLOR_NORMAL} ${REPLY}"
>&2
done
LAST_E_CMD="elog"
return 0
@@ -290,7 +290,7 @@ einfo() {
__elog_base INFO "$*"
[[ ${RC_ENDCOL} != "yes" && ${LAST_E_CMD} == "ebegin" ]] && echo >&2
echo -e "$@" | while read -r ; do
- echo " ${PORTAGE_COLOR_INFO}*${PORTAGE_COLOR_NORMAL} ${REPLY}"
>&2
+ echo "${PORTAGE_COLOR_INFO}[--]${PORTAGE_COLOR_NORMAL}
${REPLY}" >&2
done
LAST_E_CMD="einfo"
return 0
@@ -299,7 +299,7 @@ einfo() {
einfon() {
__elog_base INFO "$*"
[[ ${RC_ENDCOL} != "yes" && ${LAST_E_CMD} == "ebegin" ]] && echo >&2
- echo -ne " ${PORTAGE_COLOR_INFO}*${PORTAGE_COLOR_NORMAL} $*" >&2
+ echo -ne "${PORTAGE_COLOR_INFO}[--]${PORTAGE_COLOR_NORMAL} $*" >&2
LAST_E_CMD="einfon"
return 0
}
@@ -308,7 +308,7 @@ ewarn() {
__elog_base WARN "$*"
[[ ${RC_ENDCOL} != "yes" && ${LAST_E_CMD} == "ebegin" ]] && echo >&2
echo -e "$@" | while read -r ; do
- echo " ${PORTAGE_COLOR_WARN}*${PORTAGE_COLOR_NORMAL}
${RC_INDENTATION}${REPLY}" >&2
+ echo "${PORTAGE_COLOR_WARN}[WW]${PORTAGE_COLOR_NORMAL}
${RC_INDENTATION}${REPLY}" >&2
done
LAST_E_CMD="ewarn"
return 0
@@ -318,7 +318,7 @@ eerror() {
__elog_base ERROR "$*"
[[ ${RC_ENDCOL} != "yes" && ${LAST_E_CMD} == "ebegin" ]] && echo >&2
echo -e "$@" | while read -r ; do
- echo " ${PORTAGE_COLOR_ERR}*${PORTAGE_COLOR_NORMAL}
${RC_INDENTATION}${REPLY}" >&2
+ echo "${PORTAGE_COLOR_ERR}[EE]${PORTAGE_COLOR_NORMAL}
${RC_INDENTATION}${REPLY}" >&2
done
LAST_E_CMD="eerror"
return 0
diff --git a/lib/portage/output.py b/lib/portage/output.py
index 42f487f8a..deae207fc 100644
--- a/lib/portage/output.py
+++ b/lib/portage/output.py
@@ -680,7 +680,7 @@ class EOutput:
if not self.quiet:
if self.__last_e_cmd == "ebegin":
self._write(out, "\n")
-self._write(out, colorize("ERR", " * ") + msg + "\n")
+self._write(out, colorize("ERR", "[EE] ") + msg + "\n")
self.__last_e_cmd = "eerror"
def einfo(self, msg):
@@ -694,7 +694,7 @@ class EOutput:
if not self.quiet:
if self.__last_e_cmd == "ebegin":
self._write(out, "\n")
-self._write(out, colorize("INFO", " * ") + msg + "\n")
+self._write(out, colorize("INFO", "[--] ") + msg + "\n")
self.__last_e_cmd = "einfo"
def einfon(self, msg):
@@ -708,7 +708,7 @@ class EOutput:
if not self.quiet:
if self.__last_e_cmd == "ebegin":
self._write(out, "\n")
-self
[gentoo-portage-dev] [PATCH 2/4] Use distinct colors for output msg types
Introduce distinct colors per output function. For elog and eerror
the colors remain the same but the names change. For einfo and eqawarn,
use darker colors to distinguish them.
Signed-off-by: Michał Górny
---
bin/isolated-functions.sh| 30 +++---
bin/save-ebuild-env.sh | 6 ++--
lib/portage/elog/messages.py | 10 +++---
lib/portage/elog/mod_echo.py | 4 +--
lib/portage/output.py| 59 +++-
5 files changed, 81 insertions(+), 28 deletions(-)
diff --git a/bin/isolated-functions.sh b/bin/isolated-functions.sh
index 3e79ea6e3..df806d403 100644
--- a/bin/isolated-functions.sh
+++ b/bin/isolated-functions.sh
@@ -270,7 +270,7 @@ eqawarn() {
__elog_base QA "$*"
[[ ${RC_ENDCOL} != "yes" && ${LAST_E_CMD} == "ebegin" ]] && echo >&2
echo -e "$@" | while read -r ; do
- echo " ${PORTAGE_COLOR_WARN}*${PORTAGE_COLOR_NORMAL} ${REPLY}"
>&2
+ echo " ${PORTAGE_COLOR_QAWARN}*${PORTAGE_COLOR_NORMAL}
${REPLY}" >&2
done
LAST_E_CMD="eqawarn"
return 0
@@ -280,7 +280,7 @@ elog() {
__elog_base LOG "$*"
[[ ${RC_ENDCOL} != "yes" && ${LAST_E_CMD} == "ebegin" ]] && echo >&2
echo -e "$@" | while read -r ; do
- echo " ${PORTAGE_COLOR_GOOD}*${PORTAGE_COLOR_NORMAL} ${REPLY}"
>&2
+ echo " ${PORTAGE_COLOR_LOG}*${PORTAGE_COLOR_NORMAL} ${REPLY}"
>&2
done
LAST_E_CMD="elog"
return 0
@@ -290,7 +290,7 @@ einfo() {
__elog_base INFO "$*"
[[ ${RC_ENDCOL} != "yes" && ${LAST_E_CMD} == "ebegin" ]] && echo >&2
echo -e "$@" | while read -r ; do
- echo " ${PORTAGE_COLOR_GOOD}*${PORTAGE_COLOR_NORMAL} ${REPLY}"
>&2
+ echo " ${PORTAGE_COLOR_INFO}*${PORTAGE_COLOR_NORMAL} ${REPLY}"
>&2
done
LAST_E_CMD="einfo"
return 0
@@ -299,7 +299,7 @@ einfo() {
einfon() {
__elog_base INFO "$*"
[[ ${RC_ENDCOL} != "yes" && ${LAST_E_CMD} == "ebegin" ]] && echo >&2
- echo -ne " ${PORTAGE_COLOR_GOOD}*${PORTAGE_COLOR_NORMAL} $*" >&2
+ echo -ne " ${PORTAGE_COLOR_INFO}*${PORTAGE_COLOR_NORMAL} $*" >&2
LAST_E_CMD="einfon"
return 0
}
@@ -318,7 +318,7 @@ eerror() {
__elog_base ERROR "$*"
[[ ${RC_ENDCOL} != "yes" && ${LAST_E_CMD} == "ebegin" ]] && echo >&2
echo -e "$@" | while read -r ; do
- echo " ${PORTAGE_COLOR_BAD}*${PORTAGE_COLOR_NORMAL}
${RC_INDENTATION}${REPLY}" >&2
+ echo " ${PORTAGE_COLOR_ERR}*${PORTAGE_COLOR_NORMAL}
${RC_INDENTATION}${REPLY}" >&2
done
LAST_E_CMD="eerror"
return 0
@@ -378,12 +378,16 @@ __unset_colors() {
COLS=80
ENDCOL=
- PORTAGE_COLOR_GOOD=
- PORTAGE_COLOR_WARN=
PORTAGE_COLOR_BAD=
+ PORTAGE_COLOR_BRACKET=
+ PORTAGE_COLOR_ERR=
+ PORTAGE_COLOR_GOOD=
PORTAGE_COLOR_HILITE=
+ PORTAGE_COLOR_INFO=
+ PORTAGE_COLOR_LOG=
PORTAGE_COLOR_NORMAL=
- PORTAGE_COLOR_BRACKET=
+ PORTAGE_COLOR_QAWARN=
+ PORTAGE_COLOR_WARN=
}
__set_colors() {
@@ -403,12 +407,16 @@ __set_colors() {
if [ -n "${PORTAGE_COLORMAP}" ] ; then
eval ${PORTAGE_COLORMAP}
else
- PORTAGE_COLOR_GOOD=$'\e[32;01m'
- PORTAGE_COLOR_WARN=$'\e[33;01m'
PORTAGE_COLOR_BAD=$'\e[31;01m'
- PORTAGE_COLOR_HILITE=$'\e[36;01m'
PORTAGE_COLOR_BRACKET=$'\e[34;01m'
+ PORTAGE_COLOR_ERR=$'\e[31;01m'
+ PORTAGE_COLOR_GOOD=$'\e[32;01m'
+ PORTAGE_COLOR_HILITE=$'\e[36;01m'
+ PORTAGE_COLOR_INFO=$'\e[32m'
+ PORTAGE_COLOR_LOG=$'\e[32;01m'
PORTAGE_COLOR_NORMAL=$'\e[0m'
+ PORTAGE_COLOR_QAWARN=$'\e[33m'
+ PORTAGE_COLOR_WARN=$'\e[33;01m'
fi
}
diff --git a/bin/save-ebuild-env.sh b/bin/save-ebuild-env.sh
index 8d2ec67ed..98808814b 100644
--- a/bin/save-ebuild-env.sh
+++ b/bin/save-ebuild-env.sh
@@ -102,8 +102,10 @@ __save_ebuild_env() {
LAST_E_CMD LAST_E_LEN LD_PRELOAD MISC_FUNCTIONS_ARGS MOPREFIX \
NOCOLOR PKGDIR PKGUSE PKG_LOGDIR PKG_TMPDIR \
PORTAGE_BASHRC_FILES PORTAGE_BASHRCS_SOURCED \
- PORTAGE_COLOR_BAD PORTAGE_COLOR_BRACKET PORTAGE_COLOR_GOOD \
- PORTAGE_COLOR_HILITE PORTAGE_COLOR_NORMAL
[gentoo-portage-dev] [PATCH 1/4] Prefix color vars with "PORTAGE_COLOR_"
Rename color variables used by einfo etc. to use "PORTAGE_COLOR_"
prefix. Currently these variables are prone to being accidentally
ovewritten e.g. if an ebuild uses GOOD or BAD variables for some
purpose. Using PORTAGE prefix should keep us safe.
As an extra benefit, this makes it trivial to grep for all uses of color
variables.
Signed-off-by: Michał Górny
---
bin/install-qa-check.d/10ignored-flags | 4 +--
bin/isolated-functions.sh | 42 +-
bin/save-ebuild-env.sh | 16 +-
lib/portage/output.py | 4 +--
4 files changed, 34 insertions(+), 32 deletions(-)
diff --git a/bin/install-qa-check.d/10ignored-flags
b/bin/install-qa-check.d/10ignored-flags
index 89706cd4c..7cd073578 100644
--- a/bin/install-qa-check.d/10ignored-flags
+++ b/bin/install-qa-check.d/10ignored-flags
@@ -52,7 +52,7 @@ ignored_flag_check() {
f=$(<"${T}"/scanelf-ignored-CFLAGS.log)
if [[ -n ${f} ]] ; then
__vecho -ne '\n'
- eqawarn "${BAD}QA Notice: Files built without
respecting CFLAGS have been detected${NORMAL}"
+ eqawarn "${PORTAGE_COLOR_BAD}QA Notice: Files
built without respecting CFLAGS have been detected${PORTAGE_COLOR_NORMAL}"
eqawarn " Please include the following list of
files in your report:"
eqawarn "${f}"
__vecho -ne '\n'
@@ -82,7 +82,7 @@ ignored_flag_check() {
f=$(<"${T}"/scanelf-ignored-LDFLAGS.log)
if [[ -n ${f} ]] ; then
__vecho -ne '\n'
- eqawarn "${BAD}QA Notice: Files built without
respecting LDFLAGS have been detected${NORMAL}"
+ eqawarn "${PORTAGE_COLOR_BAD}QA Notice: Files
built without respecting LDFLAGS have been detected${PORTAGE_COLOR_NORMAL}"
eqawarn " Please include the following list of
files in your report:"
eqawarn "${f}"
__vecho -ne '\n'
diff --git a/bin/isolated-functions.sh b/bin/isolated-functions.sh
index 5630dcf4c..3e79ea6e3 100644
--- a/bin/isolated-functions.sh
+++ b/bin/isolated-functions.sh
@@ -255,7 +255,7 @@ __elog_base() {
shift
;;
*)
- __vecho -e " ${BAD}*${NORMAL} Invalid use of internal
function __elog_base(), next message will not be logged"
+ __vecho -e "
${PORTAGE_COLOR_BAD}*${PORTAGE_COLOR_NORMAL} Invalid use of internal function
__elog_base(), next message will not be logged"
return 1
;;
esac
@@ -270,7 +270,7 @@ eqawarn() {
__elog_base QA "$*"
[[ ${RC_ENDCOL} != "yes" && ${LAST_E_CMD} == "ebegin" ]] && echo >&2
echo -e "$@" | while read -r ; do
- echo " $WARN*$NORMAL $REPLY" >&2
+ echo " ${PORTAGE_COLOR_WARN}*${PORTAGE_COLOR_NORMAL} ${REPLY}"
>&2
done
LAST_E_CMD="eqawarn"
return 0
@@ -280,7 +280,7 @@ elog() {
__elog_base LOG "$*"
[[ ${RC_ENDCOL} != "yes" && ${LAST_E_CMD} == "ebegin" ]] && echo >&2
echo -e "$@" | while read -r ; do
- echo " $GOOD*$NORMAL $REPLY" >&2
+ echo " ${PORTAGE_COLOR_GOOD}*${PORTAGE_COLOR_NORMAL} ${REPLY}"
>&2
done
LAST_E_CMD="elog"
return 0
@@ -290,7 +290,7 @@ einfo() {
__elog_base INFO "$*"
[[ ${RC_ENDCOL} != "yes" && ${LAST_E_CMD} == "ebegin" ]] && echo >&2
echo -e "$@" | while read -r ; do
- echo " $GOOD*$NORMAL $REPLY" >&2
+ echo " ${PORTAGE_COLOR_GOOD}*${PORTAGE_COLOR_NORMAL} ${REPLY}"
>&2
done
LAST_E_CMD="einfo"
return 0
@@ -299,7 +299,7 @@ einfo() {
einfon() {
__elog_base INFO "$*"
[[ ${RC_ENDCOL} != "yes" && ${LAST_E_CMD} == "ebegin" ]] && echo >&2
- echo -ne " ${GOOD}*${NORMAL} $*" >&2
+ echo -ne " ${PORTAGE_COLOR_GOOD}*${PORTAGE_COLOR_NORMAL} $*" >&2
LAST_E_CMD="einfon"
return 0
}
@@ -308,7 +308,7 @@ ewarn() {
__elog_base WARN "$*"
[[ ${RC_ENDCOL} != "yes" && ${LAST_E_
[gentoo-portage-dev] [PATCH 0/4] Output rewrite for better clarify and greppability
Hi, Ok, so it's more major than I originally intended but I think it's a good direction (once you get used to it). Roughly: 1. All bash color vars are now prefixed with `PORTAGE_COLOR_` to avoid accidental collisions with ebuild vars (e.g. ebuild setting `GOOD=foo` broke `elog` before). 2. There are specific color vars for all kinds of output functions, and now `einfo` messages use distinct color (dark green) from `elog`, and `eqawarn` (brown) from `ewarn`. 3. Messages are now prefixed by their kind, making it possible to distinguish them without colors and grep for specific kind of logs: - `[--]` for einfo & ebegin - `[II]` for elog - `[WW]` for ewarn - `[QA]` for eqawarn - `[EE]` for eerror 4. Finally, I've replaced most of `>>>` and `!!!` in Portage output with four `>>>>` and `` to align the output again. PR (includes screenshot): https://github.com/gentoo/portage/pull/759 Michał Górny (4): Prefix color vars with "PORTAGE_COLOR_" Use distinct colors for output msg types Use verbose prefixes for output messages Use ">>>>" and "" for output prefixes bin/clean_locks | 2 +- bin/dohtml.py | 2 +- bin/doins.py | 2 +- bin/ebuild| 8 +- bin/ebuild-helpers/dobin | 2 +- bin/ebuild-helpers/dodoc | 2 +- bin/ebuild-helpers/doexe | 2 +- bin/ebuild-helpers/doinfo | 2 +- bin/ebuild-helpers/dolib | 2 +- bin/ebuild-helpers/doman | 2 +- bin/ebuild-helpers/domo | 2 +- bin/ebuild-helpers/dosbin | 2 +- bin/ebuild-helpers/dosed | 4 +- bin/ebuild-helpers/keepdir| 2 +- bin/ebuild-helpers/newins | 4 +- bin/env-update| 2 +- bin/fixpackages | 4 +- bin/glsa-check| 14 +- bin/install-qa-check.d/10ignored-flags| 4 +- bin/install.py| 2 +- bin/isolated-functions.sh | 50 --- bin/misc-functions.sh | 22 +-- bin/phase-functions.sh| 64 - bin/phase-helpers.sh | 18 +-- bin/portageq | 6 +- bin/save-ebuild-env.sh| 18 ++- cnf/repo.postsync.d/example | 4 +- lib/_emerge/Binpkg.py | 10 +- lib/_emerge/BinpkgExtractorAsync.py | 4 +- lib/_emerge/BinpkgVerifier.py | 12 +- lib/_emerge/BlockerCache.py | 2 +- lib/_emerge/BlockerDB.py | 2 +- lib/_emerge/EbuildBuild.py| 2 +- lib/_emerge/EbuildIpcDaemon.py| 2 +- lib/_emerge/EbuildPhase.py| 2 +- lib/_emerge/JobStatusDisplay.py | 2 +- lib/_emerge/MergeListItem.py | 2 +- lib/_emerge/PackageUninstall.py | 6 +- lib/_emerge/Scheduler.py | 20 +-- lib/_emerge/SpawnProcess.py | 2 +- lib/_emerge/SubProcess.py | 2 +- lib/_emerge/actions.py| 126 ++ lib/_emerge/countdown.py | 4 +- lib/_emerge/depgraph.py | 121 + lib/_emerge/main.py | 6 +- lib/_emerge/post_emerge.py| 4 +- lib/_emerge/resolver/output_helpers.py| 4 +- lib/_emerge/resolver/slot_collision.py| 16 +-- lib/_emerge/show_invalid_depstring_notice.py | 2 +- lib/_emerge/unmerge.py| 44 +++--- lib/portage/__init__.py | 24 ++-- lib/portage/_selinux.py | 4 +- lib/portage/_sets/shell.py| 2 +- lib/portage/binrepo/config.py | 4 +- lib/portage/data.py | 2 +- lib/portage/dbapi/__init__.py | 2 +- lib/portage/dbapi/bintree.py | 45 --- lib/portage/dbapi/porttree.py | 12 +- lib/portage/dbapi/vartree.py | 94 ++--- lib/portage/dep/__init__.py | 48 +++ lib/portage/elog/__init__.py | 2 +- lib/portage/elog/messages.py | 15 ++- lib/portage/elog/mod_custom.py| 4 +- lib/portage/elog/mod_echo.py | 4 +- lib/portage/elog/mod_mail_summary.py | 2 +- lib/portage/elog/mod_save_summary.py | 2 +- lib/portage/ema
[gentoo-portage-dev] [PATCH v2] install-qa-check.d: Add a QA check for installing xattrs
Warn the developers if ebuilds install files with xattrs to ${ED}.
The xattrs may or may not be preserved when installing the package,
making them unreliable on one hand, and somewhat suprising in other
cases (e.g. when they unintentionally leak from developer's system).
This is the first step towards restoring PMS compliance and *not*
preserving extended metadata.
Signed-off-by: Michał Górny
---
bin/install-qa-check.d/95xattr | 54 ++
1 file changed, 54 insertions(+)
create mode 100644 bin/install-qa-check.d/95xattr
Changes in v2:
- added a link to the tracker in place of in-place explanation
diff --git a/bin/install-qa-check.d/95xattr b/bin/install-qa-check.d/95xattr
new file mode 100644
index 0..4bf6631d3
--- /dev/null
+++ b/bin/install-qa-check.d/95xattr
@@ -0,0 +1,54 @@
+# Check for xattrs. See https://bugs.gentoo.org/815220.
+
+xattr_check() {
+ type -P getfattr >/dev/null || return
+
+ pushd "${ED}" >/dev/null || die
+ local x file= keys
+ local -A data=()
+ while read -r x; do
+ case ${x} in
+ "# file: "*)
+ file=${x#*: }
+ file=/${file#.}
+ ;;
+ btrfs.*)
+ # ignore btrfs xattrs, they're implicit fs
metadata
+ ;;
+ security.capability)
+ # don't report caps if we have fcaps.eclass
inherited
+ if ! has fcaps ${INHERITED}; then
+ data[${file}]+=" ${x}"
+ fi
+ ;;
+ ?*)
+ data[${file}]+=" ${x}"
+ ;;
+ esac
+ done < <(getfattr -R -h -m - . 2>/dev/null)
+ popd >/dev/null || die
+
+ if [[ ${data[@]} ]]; then
+ eqawarn "One or more files in \${ED} include extended
attributes."
+ eqawarn
+
+ for file in "${!data[@]}"; do
+ keys=( ${data[${file}]} )
+ for x in "${keys[@]}"; do
+ eqatag xattr "key=${x}" "${file}"
+ done
+ eqawarn " ${file} (${keys[*]})"
+ done
+
+ eqawarn
+ eqawarn "It is impossible to reliably guarantee that the
extended attributes"
+ eqawarn "will be reliably preserved while merging. Please file
a bug"
+ eqawarn "and make it block the tracker
https://bugs.gentoo.org/815220.;
+ eqawarn "More information about the problem can also be found
on the tracker."
+ fi
+}
+
+xattr_check
+: # guarantee successful exit
+
+# vim:ft=sh
--
2.33.0
Re: [gentoo-portage-dev] [PATCH] install-qa-check.d: Add a QA check for installing xattrs
On Mon, 2021-09-27 at 21:09 -0400, Mike Gilbert wrote: > On Mon, Sep 27, 2021 at 1:20 PM Michał Górny wrote: > > + eqawarn > > + eqawarn "It is impossible to reliably guarantee that the > > extended attributes" > > + eqawarn "will be reliably preserved while merging. Please > > ensure that any" > > + eqawarn "extended metadata necessary is applied in > > pkg_postinst() phase," > > + eqawarn "and that the implementation includes a fallback if > > necessary." > > This message suggests that applying xattrs in pkg_postinst is > acceptable. However, your patch offers no way to disable the QA > warning for ebuilds that do so. We'll cross that bridge when we get there. Ideally, we wouldn't need to silence the check because no packages would do that. If they do, then we'll probably want to work on an eclass like fcaps.eclas. -- Best regards, Michał Górny
Re: [gentoo-portage-dev] [PATCH] install-qa-check.d: Add a QA check for installing xattrs
On Mon, 2021-09-27 at 21:03 -0400, Mike Gilbert wrote:
> On Mon, Sep 27, 2021 at 1:20 PM Michał Górny wrote:
> >
> > Warn the developers if ebuilds install files with xattrs to ${ED}.
> > The xattrs may or may not be preserved when installing the package,
> > making them unreliable on one hand, and somewhat suprising in other
> > cases (e.g. when they unintentionally leak from developer's system).
> >
> > This is the first step towards restoring PMS compliance and *not*
> > preserving extended metadata.
>
> How does preserving xattrs conflict with PMS?
The PMS doesn't specify that xattrs, ACLs, caps etc. are preserved.
By doing that, Portage allows developers to commit ebuilds that are not
going to work reliably without even realizing it. In fact, this can't
even work reliably inside Portage itself, depending on the filesystem
used for $D.
Furthermore, doexe preserving stuff goes contrary to common sense. Why
would helpers preserve xattrs when they are supposed to reset things
like mode and ownership by design?
> Is there a bug report you could reference?
It starts with https://bugs.gentoo.org/814857.
--
Best regards,
Michał Górny
[gentoo-portage-dev] [PATCH] install-qa-check.d: Add a QA check for installing xattrs
Warn the developers if ebuilds install files with xattrs to ${ED}.
The xattrs may or may not be preserved when installing the package,
making them unreliable on one hand, and somewhat suprising in other
cases (e.g. when they unintentionally leak from developer's system).
This is the first step towards restoring PMS compliance and *not*
preserving extended metadata.
Signed-off-by: Michał Górny
---
bin/install-qa-check.d/95xattr | 54 ++
1 file changed, 54 insertions(+)
create mode 100644 bin/install-qa-check.d/95xattr
diff --git a/bin/install-qa-check.d/95xattr b/bin/install-qa-check.d/95xattr
new file mode 100644
index 0..07d8042a8
--- /dev/null
+++ b/bin/install-qa-check.d/95xattr
@@ -0,0 +1,54 @@
+# Check for xattrs.
+
+xattr_check() {
+ type -P getfattr >/dev/null || return
+
+ pushd "${ED}" >/dev/null || die
+ local x file= keys
+ local -A data=()
+ while read -r x; do
+ case ${x} in
+ "# file: "*)
+ file=${x#*: }
+ file=/${file#.}
+ ;;
+ btrfs.*)
+ # ignore btrfs xattrs, they're implicit fs
metadata
+ ;;
+ security.capability)
+ # don't report caps if we have fcaps.eclass
inherited
+ if ! has fcaps ${INHERITED}; then
+ data[${file}]+=" ${x}"
+ fi
+ ;;
+ ?*)
+ data[${file}]+=" ${x}"
+ ;;
+ esac
+ done < <(getfattr -R -h -m - . 2>/dev/null)
+ popd >/dev/null || die
+
+ if [[ ${data[@]} ]]; then
+ eqawarn "One or more files in \${ED} include extended
attributes."
+ eqawarn
+
+ for file in "${!data[@]}"; do
+ keys=( ${data[${file}]} )
+ for x in "${keys[@]}"; do
+ eqatag xattr "key=${x}" "${file}"
+ done
+ eqawarn " ${file} (${keys[*]})"
+ done
+
+ eqawarn
+ eqawarn "It is impossible to reliably guarantee that the
extended attributes"
+ eqawarn "will be reliably preserved while merging. Please
ensure that any"
+ eqawarn "extended metadata necessary is applied in
pkg_postinst() phase,"
+ eqawarn "and that the implementation includes a fallback if
necessary."
+ fi
+}
+
+xattr_check
+: # guarantee successful exit
+
+# vim:ft=sh
--
2.33.0
Re: [gentoo-portage-dev] [PATCH v3] Copy files/* into the work tree instead of symlinking it
On Mon, 2021-09-27 at 12:49 +0200, Ulrich Mueller wrote: > > > > > > On Sun, 26 Sep 2021, Michał Górny wrote: > > > Symlinking FILESDIR into the work tree has the unintended > > consequence > > of preserving all original file metadata, including system-specific > > ACLs > > and so on. When these files are installed, this could lead to > > unintentionally copying this metadata to the system and/or binary > > packages. > > > Let's copy all files instead and drop metadata in the process. > > Since > > FILESDIR is expected to be small by design, this shouldn't cause any > > major trouble. It is also easier and less likely to cause > > regressions > > than making sure stuff is not preserved when installing. > > > Unfortunately, a similar problem applies to DISTDIR. However, > > installing files from DISTDIR is rarer than from FILESDIR, so I > > guess > > we'll cross that bridge when we get to it. > > Sorry for the late reply, but this looks like the wrong solution to > me. > > Looking at the installation helpers (doins, doexe, etc.), they don't > preserve the normal permission bits, but reset them to a defined > state. > So why would they preserve xattrs? > > I don't see anything in PMS that would mandate that behaviour (on the > contrary, in section 13.3.1 there is "Other file attributes may be > discarded"). How do the other package managers handle this? Yes, I agree this is the wrong long-term solution but it's a fix that works today and shouldn't cause regressions. I'd like to revisit how files are installed long-term but this requires more time, and most importantly making sure we won't accidentally break stuff. I'd like to establish whether we have any ebuilds relying e.g. on caps or ACLs being preserved from src_install(), and get alternative solutions to that first (fcaps.eclass-alike?). Apparently PaX will also require special fixes but I'm not sure if it's worth fixing at this point. -- Best regards, Michał Górny
[gentoo-portage-dev] [PATCH v3] Copy files/* into the work tree instead of symlinking it
Symlinking FILESDIR into the work tree has the unintended consequence
of preserving all original file metadata, including system-specific ACLs
and so on. When these files are installed, this could lead to
unintentionally copying this metadata to the system and/or binary
packages.
Let's copy all files instead and drop metadata in the process. Since
FILESDIR is expected to be small by design, this shouldn't cause any
major trouble. It is also easier and less likely to cause regressions
than making sure stuff is not preserved when installing.
Unfortunately, a similar problem applies to DISTDIR. However,
installing files from DISTDIR is rarer than from FILESDIR, so I guess
we'll cross that bridge when we get to it.
Bug: https://bugs.gentoo.org/814857
Signed-off-by: Michał Górny
---
bin/phase-functions.sh| 2 +-
.../package/ebuild/prepare_build_dirs.py | 19 +--
2 files changed, 10 insertions(+), 11 deletions(-)
Changed in v3:
- use optimized copyfile from Portage (supports reflinking)
diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh
index d3221993d..9a4c97b16 100644
--- a/bin/phase-functions.sh
+++ b/bin/phase-functions.sh
@@ -296,7 +296,7 @@ __dyn_clean() {
rm -rf "${PORTAGE_BUILDDIR}/build-info"
rm -rf "${WORKDIR}"
- rm -f "${PORTAGE_BUILDDIR}/files"
+ rm -rf "${PORTAGE_BUILDDIR}/files"
fi
if [ -f "${PORTAGE_BUILDDIR}/.unpacked" ]; then
diff --git a/lib/portage/package/ebuild/prepare_build_dirs.py
b/lib/portage/package/ebuild/prepare_build_dirs.py
index 659198905..2e2ef73f4 100644
--- a/lib/portage/package/ebuild/prepare_build_dirs.py
+++ b/lib/portage/package/ebuild/prepare_build_dirs.py
@@ -1,4 +1,4 @@
-# Copyright 2010-2020 Gentoo Authors
+# Copyright 2010-2021 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2
__all__ = ["prepare_build_dirs"]
@@ -27,6 +27,7 @@ from portage.util import (
normalize_path,
writemsg,
)
+from portage.util.file_copy import copyfile
from portage.util.install_mask import _raise_exc
@@ -478,16 +479,14 @@ def _ensure_log_subdirs(logdir, subdir):
def _prepare_fake_filesdir(settings):
real_filesdir = settings["O"] + "/files"
-symlink_path = settings["FILESDIR"]
+filesdir = settings["FILESDIR"]
+portage.util.ensure_dirs(filesdir, mode=0o755)
-try:
-link_target = os.readlink(symlink_path)
-except OSError:
-os.symlink(real_filesdir, symlink_path)
-else:
-if link_target != real_filesdir:
-os.unlink(symlink_path)
-os.symlink(real_filesdir, symlink_path)
+# Copy files from real directory to ebuild directory (without metadata).
+if os.path.isdir(real_filesdir):
+shutil.copytree(
+real_filesdir, filesdir, copy_function=copyfile, dirs_exist_ok=True
+)
def _prepare_fake_distdir(settings, alist):
--
2.33.0
[gentoo-portage-dev] [PATCH v2] Copy files/* into the work tree instead of symlinking it
Symlinking FILESDIR into the work tree has the unintended consequence
of preserving all original file metadata, including system-specific ACLs
and so on. When these files are installed, this could lead to
unintentionally copying this metadata to the system and/or binary
packages.
Let's copy all files instead and drop metadata in the process. Since
FILESDIR is expected to be small by design, this shouldn't cause any
major trouble. It is also easier and less likely to cause regressions
than making sure stuff is not preserved when installing.
Unfortunately, a similar problem applies to DISTDIR. However,
installing files from DISTDIR is rarer than from FILESDIR, so I guess
we'll cross that bridge when we get to it.
Bug: https://bugs.gentoo.org/814857
Signed-off-by: Michał Górny
---
bin/phase-functions.sh | 2 +-
lib/portage/package/ebuild/prepare_build_dirs.py | 16 +++-
2 files changed, 8 insertions(+), 10 deletions(-)
diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh
index d3221993d..9a4c97b16 100644
--- a/bin/phase-functions.sh
+++ b/bin/phase-functions.sh
@@ -296,7 +296,7 @@ __dyn_clean() {
rm -rf "${PORTAGE_BUILDDIR}/build-info"
rm -rf "${WORKDIR}"
- rm -f "${PORTAGE_BUILDDIR}/files"
+ rm -rf "${PORTAGE_BUILDDIR}/files"
fi
if [ -f "${PORTAGE_BUILDDIR}/.unpacked" ]; then
diff --git a/lib/portage/package/ebuild/prepare_build_dirs.py
b/lib/portage/package/ebuild/prepare_build_dirs.py
index 659198905..9d83bacad 100644
--- a/lib/portage/package/ebuild/prepare_build_dirs.py
+++ b/lib/portage/package/ebuild/prepare_build_dirs.py
@@ -478,16 +478,14 @@ def _ensure_log_subdirs(logdir, subdir):
def _prepare_fake_filesdir(settings):
real_filesdir = settings["O"] + "/files"
-symlink_path = settings["FILESDIR"]
+filesdir = settings["FILESDIR"]
+portage.util.ensure_dirs(filesdir, mode=0o755)
-try:
-link_target = os.readlink(symlink_path)
-except OSError:
-os.symlink(real_filesdir, symlink_path)
-else:
-if link_target != real_filesdir:
-os.unlink(symlink_path)
-os.symlink(real_filesdir, symlink_path)
+# Copy files from real directory to ebuild directory (without metadata).
+if os.path.isdir(real_filesdir):
+shutil.copytree(
+real_filesdir, filesdir, copy_function=shutil.copyfile,
dirs_exist_ok=True
+)
def _prepare_fake_distdir(settings, alist):
--
2.33.0
[gentoo-portage-dev] [PATCH] Copy files/* into the work tree instead of symlinking it
Symlinking FILESDIR into the work tree has the unintended consequence
of preserving all original file metadata, including system-specific ACLs
and so on. When these files are installed, this could lead to
unintentionally copying this metadata to the system and/or binary
packages.
Let's copy all files instead and drop metadata in the process. Since
FILESDIR is expected to be small by design, this shouldn't cause any
major trouble. It is also easier and less likely to cause regressions
than making sure stuff is not preserved when installing.
Unfortunately, a similar problem applies to DISTDIR. However,
installing files from DISTDIR is rarer than from FILESDIR, so I guess
we'll cross that bridge when we get to it.
Bug: https://bugs.gentoo.org/814857
Signed-off-by: Michał Górny
---
bin/phase-functions.sh | 2 +-
lib/portage/package/ebuild/prepare_build_dirs.py | 15 ++-
2 files changed, 7 insertions(+), 10 deletions(-)
diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh
index d3221993d..9a4c97b16 100644
--- a/bin/phase-functions.sh
+++ b/bin/phase-functions.sh
@@ -296,7 +296,7 @@ __dyn_clean() {
rm -rf "${PORTAGE_BUILDDIR}/build-info"
rm -rf "${WORKDIR}"
- rm -f "${PORTAGE_BUILDDIR}/files"
+ rm -rf "${PORTAGE_BUILDDIR}/files"
fi
if [ -f "${PORTAGE_BUILDDIR}/.unpacked" ]; then
diff --git a/lib/portage/package/ebuild/prepare_build_dirs.py
b/lib/portage/package/ebuild/prepare_build_dirs.py
index 659198905..0e85ded8b 100644
--- a/lib/portage/package/ebuild/prepare_build_dirs.py
+++ b/lib/portage/package/ebuild/prepare_build_dirs.py
@@ -478,16 +478,13 @@ def _ensure_log_subdirs(logdir, subdir):
def _prepare_fake_filesdir(settings):
real_filesdir = settings["O"] + "/files"
-symlink_path = settings["FILESDIR"]
+filesdir = settings["FILESDIR"]
+portage.util.ensure_dirs(filesdir, mode=0o755)
-try:
-link_target = os.readlink(symlink_path)
-except OSError:
-os.symlink(real_filesdir, symlink_path)
-else:
-if link_target != real_filesdir:
-os.unlink(symlink_path)
-os.symlink(real_filesdir, symlink_path)
+# Copy files from real directory to ebuild directory (without metadata).
+shutil.copytree(
+real_filesdir, filesdir, copy_function=shutil.copyfile,
dirs_exist_ok=True
+)
def _prepare_fake_distdir(settings, alist):
--
2.33.0
Re: [gentoo-portage-dev] [PATCH] eend: Output QA notice when called without argument
On Fri, 2021-09-03 at 18:58 +0200, Ulrich Müller wrote:
> PMS says about eend: "Takes one fixed argument, which is a numeric
> return code, and an optional message in all subsequent arguments."
>
> Bug: https://bugs.gentoo.org/703520
> Signed-off-by: Ulrich Müller
> ---
> bin/isolated-functions.sh | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/bin/isolated-functions.sh b/bin/isolated-functions.sh
> index b495ae6c7..5b1f372d2 100644
> --- a/bin/isolated-functions.sh
> +++ b/bin/isolated-functions.sh
> @@ -364,6 +364,7 @@ __eend() {
> }
>
> eend() {
> + [[ -n $1 ]] || eqawarn "QA Notice: eend called without return code"
> local retval=${1:-0}
> shift
>
I think the message could be a bit confusing. Maybe say explicitly that
it's missing an argument.
--
Best regards,
Michał Górny
[gentoo-portage-dev] [PATCH 4/4] Include INHERIT value in generated cache
PkgCore uses an additional md5-cache INHERIT key to indicate eclasses
explicitly inherited in an ebuild. Update Portage to emit the same key
to restore cache compatibility.
Signed-off-by: Michał Górny
---
bin/ebuild.sh | 8 +++-
bin/phase-functions.sh | 2 +-
lib/portage/__init__.py| 2 +-
lib/portage/package/ebuild/_config/special_env_vars.py | 2 +-
4 files changed, 10 insertions(+), 4 deletions(-)
diff --git a/bin/ebuild.sh b/bin/ebuild.sh
index 381bcb5c8..07ca58d22 100755
--- a/bin/ebuild.sh
+++ b/bin/ebuild.sh
@@ -402,6 +402,9 @@ inherit() {
unset $__export_funcs_var
has $1 $INHERITED || export INHERITED="$INHERITED $1"
+ if [[ ${ECLASS_DEPTH} -eq 1 ]]; then
+ export
PORTAGE_EXPLICIT_INHERIT="${PORTAGE_EXPLICIT_INHERIT} $1"
+ fi
fi
shift
@@ -648,6 +651,7 @@ if ! has "$EBUILD_PHASE" clean cleanrm ; then
unset INHERITED IUSE REQUIRED_USE ECLASS E_IUSE E_REQUIRED_USE
unset E_DEPEND E_RDEPEND E_PDEPEND E_BDEPEND E_IDEPEND
E_PROPERTIES
unset E_RESTRICT PROVIDES_EXCLUDE REQUIRES_EXCLUDE
+ unset PORTAGE_EXPLICIT_INHERIT
if [[ $PORTAGE_DEBUG != 1 || ${-/x/} != $- ]] ; then
source "$EBUILD" || die "error sourcing ebuild"
@@ -757,7 +761,7 @@ if [[ $EBUILD_PHASE = depend ]] ; then
metadata_keys=(
DEPEND RDEPEND SLOT SRC_URI RESTRICT HOMEPAGE LICENSE
DESCRIPTION KEYWORDS INHERITED IUSE REQUIRED_USE PDEPEND BDEPEND
- EAPI PROPERTIES DEFINED_PHASES IDEPEND
+ EAPI PROPERTIES DEFINED_PHASES IDEPEND INHERIT
)
if ! ___eapi_has_BDEPEND; then
@@ -767,6 +771,8 @@ if [[ $EBUILD_PHASE = depend ]] ; then
unset IDEPEND
fi
+ INHERIT=${PORTAGE_EXPLICIT_INHERIT}
+
# The extra $(echo) commands remove newlines.
for f in "${metadata_keys[@]}" ; do
echo "${f}=$(echo ${!f})" >&${PORTAGE_PIPE_FD} || exit $?
diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh
index 0bb5d86e1..d3221993d 100644
--- a/bin/phase-functions.sh
+++ b/bin/phase-functions.sh
@@ -20,7 +20,7 @@ PORTAGE_READONLY_VARS="D EBUILD EBUILD_PHASE
EBUILD_PHASE_FUNC \
PORTAGE_BUILD_USER PORTAGE_BUNZIP2_COMMAND \
PORTAGE_BZIP2_COMMAND PORTAGE_COLORMAP PORTAGE_CONFIGROOT \
PORTAGE_DEBUG PORTAGE_DEPCACHEDIR PORTAGE_EBUILD_EXIT_FILE \
- PORTAGE_ECLASS_LOCATIONS \
+ PORTAGE_ECLASS_LOCATIONS PORTAGE_EXPLICIT_INHERIT \
PORTAGE_GID PORTAGE_GRPNAME PORTAGE_INST_GID PORTAGE_INST_UID \
PORTAGE_INTERNAL_CALLER PORTAGE_IPC_DAEMON PORTAGE_IUSE
PORTAGE_LOG_FILE \
PORTAGE_MUTABLE_FILTERED_VARS PORTAGE_OVERRIDE_EPREFIX
PORTAGE_PROPERTIES \
diff --git a/lib/portage/__init__.py b/lib/portage/__init__.py
index 232d77f0e..a41ca4323 100644
--- a/lib/portage/__init__.py
+++ b/lib/portage/__init__.py
@@ -519,7 +519,7 @@ auxdbkeys = (
'RESTRICT', 'HOMEPAGE', 'LICENSE', 'DESCRIPTION',
'KEYWORDS', 'INHERITED', 'IUSE', 'REQUIRED_USE',
'PDEPEND', 'BDEPEND', 'EAPI',
- 'PROPERTIES', 'DEFINED_PHASES', 'IDEPEND',
+ 'PROPERTIES', 'DEFINED_PHASES', 'IDEPEND', 'INHERIT',
)
def portageexit():
diff --git a/lib/portage/package/ebuild/_config/special_env_vars.py
b/lib/portage/package/ebuild/_config/special_env_vars.py
index 456af1838..8e314a6d6 100644
--- a/lib/portage/package/ebuild/_config/special_env_vars.py
+++ b/lib/portage/package/ebuild/_config/special_env_vars.py
@@ -22,7 +22,7 @@ env_blacklist = frozenset((
"KEYWORDS", "LICENSE", "MERGE_TYPE",
"PDEPEND", "PF", "PKGUSE", "PORTAGE_BACKGROUND",
"PORTAGE_BACKGROUND_UNMERGE", "PORTAGE_BUILDDIR_LOCKED",
- "PORTAGE_BUILT_USE", "PORTAGE_CONFIGROOT",
+ "PORTAGE_BUILT_USE", "PORTAGE_CONFIGROOT", "PORTAGE_EXPLICIT_INHERIT",
"PORTAGE_INTERNAL_CALLER", "PORTAGE_IUSE",
"PORTAGE_NONFATAL", "PORTAGE_PIPE_FD", "PORTAGE_REPO_NAME",
"PORTAGE_USE", "PROPERTIES", "RDEPEND", "REPOSITORY",
--
2.33.0
[gentoo-portage-dev] [PATCH 2/4] Switch internal metadata to key=value format
Signed-off-by: Michał Górny
---
bin/ebuild.sh | 13 +++--
lib/_emerge/EbuildMetadataPhase.py | 15 +--
2 files changed, 16 insertions(+), 12 deletions(-)
diff --git a/bin/ebuild.sh b/bin/ebuild.sh
index 32995d95b..381bcb5c8 100755
--- a/bin/ebuild.sh
+++ b/bin/ebuild.sh
@@ -754,10 +754,11 @@ if [[ $EBUILD_PHASE = depend ]] ; then
export SANDBOX_ON="0"
set -f
- auxdbkeys="DEPEND RDEPEND SLOT SRC_URI RESTRICT HOMEPAGE LICENSE
+ metadata_keys=(
+ DEPEND RDEPEND SLOT SRC_URI RESTRICT HOMEPAGE LICENSE
DESCRIPTION KEYWORDS INHERITED IUSE REQUIRED_USE PDEPEND BDEPEND
- EAPI PROPERTIES DEFINED_PHASES IDEPEND UNUSED_04
- UNUSED_03 UNUSED_02 UNUSED_01"
+ EAPI PROPERTIES DEFINED_PHASES IDEPEND
+ )
if ! ___eapi_has_BDEPEND; then
unset BDEPEND
@@ -767,10 +768,10 @@ if [[ $EBUILD_PHASE = depend ]] ; then
fi
# The extra $(echo) commands remove newlines.
- for f in ${auxdbkeys} ; do
- eval "echo \$(echo \${!f}) 1>&${PORTAGE_PIPE_FD}" || exit $?
+ for f in "${metadata_keys[@]}" ; do
+ echo "${f}=$(echo ${!f})" >&${PORTAGE_PIPE_FD} || exit $?
done
- eval "exec ${PORTAGE_PIPE_FD}>&-"
+ exec {PORTAGE_PIPE_FD}>&-
set +f
else
# Note: readonly variables interfere with __preprocess_ebuild_env(), so
diff --git a/lib/_emerge/EbuildMetadataPhase.py
b/lib/_emerge/EbuildMetadataPhase.py
index d00f194c2..5fd0e8a4d 100644
--- a/lib/_emerge/EbuildMetadataPhase.py
+++ b/lib/_emerge/EbuildMetadataPhase.py
@@ -151,13 +151,16 @@ class EbuildMetadataPhase(SubProcess):
metadata_lines =
_unicode_decode(b''.join(self._raw_metadata),
encoding=_encodings['repo.content'],
errors='replace').splitlines()
+ metadata = {}
metadata_valid = True
- if len(portage.auxdbkeys) != len(metadata_lines):
- # Don't trust bash's returncode if the
- # number of lines is incorrect.
- metadata_valid = False
- else:
- metadata = dict(zip(portage.auxdbkeys,
metadata_lines))
+ for l in metadata_lines:
+ if '=' not in l:
+ metadata_valid = False
+ break
+ key, value = l.split('=', 1)
+ metadata[key] = value
+
+ if metadata_valid:
parsed_eapi = self._eapi
if parsed_eapi is None:
parsed_eapi = "0"
--
2.33.0
[gentoo-portage-dev] [PATCH 3/4] Remove UNUSED* auxdbkeys
The UNUSED* auxdbkeys are a relict of old metadata cache format that
required a fixed number of lines. This format is no longer supported
by Portage, and all uses of auxdbkeys strip UNUSED values, so just
remove them entirely.
Signed-off-by: Michał Górny
---
bin/portageq | 3 +--
lib/_emerge/Package.py | 3 +--
lib/portage/__init__.py| 4 +---
lib/portage/dbapi/__init__.py | 3 +--
repoman/lib/repoman/qa_data.py | 2 +-
5 files changed, 5 insertions(+), 10 deletions(-)
diff --git a/bin/portageq b/bin/portageq
index 385259f9d..d3cb9f140 100755
--- a/bin/portageq
+++ b/bin/portageq
@@ -238,8 +238,7 @@ docstrings['metadata'] = """
[]+
Returns metadata values for the specified package.
Available keys: %s
-""" % ','.join(sorted(x for x in portage.auxdbkeys \
-if not x.startswith('UNUSED_')))
+""" % ','.join(sorted(x for x in portage.auxdbkeys))
metadata.__doc__ = docstrings['metadata']
diff --git a/lib/_emerge/Package.py b/lib/_emerge/Package.py
index e8809a89d..4e25619ae 100644
--- a/lib/_emerge/Package.py
+++ b/lib/_emerge/Package.py
@@ -791,8 +791,7 @@ class Package(Task):
pkg = self
return pkg
-_all_metadata_keys = set(x for x in portage.auxdbkeys \
- if not x.startswith("UNUSED_"))
+_all_metadata_keys = set(x for x in portage.auxdbkeys)
_all_metadata_keys.update(Package.metadata_keys)
_all_metadata_keys = frozenset(_all_metadata_keys)
diff --git a/lib/portage/__init__.py b/lib/portage/__init__.py
index 6e22a174b..232d77f0e 100644
--- a/lib/portage/__init__.py
+++ b/lib/portage/__init__.py
@@ -519,10 +519,8 @@ auxdbkeys = (
'RESTRICT', 'HOMEPAGE', 'LICENSE', 'DESCRIPTION',
'KEYWORDS', 'INHERITED', 'IUSE', 'REQUIRED_USE',
'PDEPEND', 'BDEPEND', 'EAPI',
- 'PROPERTIES', 'DEFINED_PHASES', 'IDEPEND', 'UNUSED_04',
- 'UNUSED_03', 'UNUSED_02', 'UNUSED_01',
+ 'PROPERTIES', 'DEFINED_PHASES', 'IDEPEND',
)
-auxdbkeylen = len(auxdbkeys)
def portageexit():
pass
diff --git a/lib/portage/dbapi/__init__.py b/lib/portage/dbapi/__init__.py
index d7facc9b6..3caefb816 100644
--- a/lib/portage/dbapi/__init__.py
+++ b/lib/portage/dbapi/__init__.py
@@ -28,8 +28,7 @@ class dbapi:
_category_re = re.compile(r'^\w[-.+\w]*$', re.UNICODE)
_categories = None
_use_mutable = False
- _known_keys = frozenset(x for x in auxdbkeys
- if not x.startswith("UNUSED_0"))
+ _known_keys = frozenset(auxdbkeys)
_pkg_str_aux_keys = ("EAPI", "KEYWORDS", "SLOT", "repository")
def __init__(self):
diff --git a/repoman/lib/repoman/qa_data.py b/repoman/lib/repoman/qa_data.py
index afb403d8d..4785581e2 100644
--- a/repoman/lib/repoman/qa_data.py
+++ b/repoman/lib/repoman/qa_data.py
@@ -79,7 +79,7 @@ class QAData:
self.missingvars = qadata.get("missingvars", [])
logging.debug("QAData: missingvars: %s", self.missingvars)
- self.allvars = set(x for x in portage.auxdbkeys if not
x.startswith("UNUSED_"))
+ self.allvars = set(portage.auxdbkeys)
self.allvars.update(Package.metadata_keys)
self.allvars = sorted(self.allvars)
--
2.33.0
[gentoo-portage-dev] [PATCH 1/4] Remove deprecated dbkey support from doebuild/ebuild.sh
Remove the support for the dbkey logic that used to write metadata into
a file. This logic has stopped being used and became deprecated
in 2013. If any external tool is still using it, it's probably been
broken by changes in metadata itself since, and would definitely
be broken by the incoming change in metadata format.
Signed-off-by: Michał Górny
---
bin/ebuild.sh | 23 ---
lib/portage/package/ebuild/doebuild.py | 22 +++---
2 files changed, 7 insertions(+), 38 deletions(-)
diff --git a/bin/ebuild.sh b/bin/ebuild.sh
index 3042e6c8c..32995d95b 100755
--- a/bin/ebuild.sh
+++ b/bin/ebuild.sh
@@ -754,14 +754,6 @@ if [[ $EBUILD_PHASE = depend ]] ; then
export SANDBOX_ON="0"
set -f
- if [ -n "${dbkey}" ] ; then
- if [ ! -d "${dbkey%/*}" ]; then
- install -d -g ${PORTAGE_GID} -m2775 "${dbkey%/*}"
- fi
- # Make it group writable. 666&~002==664
- umask 002
- fi
-
auxdbkeys="DEPEND RDEPEND SLOT SRC_URI RESTRICT HOMEPAGE LICENSE
DESCRIPTION KEYWORDS INHERITED IUSE REQUIRED_USE PDEPEND BDEPEND
EAPI PROPERTIES DEFINED_PHASES IDEPEND UNUSED_04
@@ -775,17 +767,10 @@ if [[ $EBUILD_PHASE = depend ]] ; then
fi
# The extra $(echo) commands remove newlines.
- if [ -n "${dbkey}" ] ; then
- > "${dbkey}"
- for f in ${auxdbkeys} ; do
- echo $(echo ${!f}) >> "${dbkey}" || exit $?
- done
- else
- for f in ${auxdbkeys} ; do
- eval "echo \$(echo \${!f}) 1>&${PORTAGE_PIPE_FD}" ||
exit $?
- done
- eval "exec ${PORTAGE_PIPE_FD}>&-"
- fi
+ for f in ${auxdbkeys} ; do
+ eval "echo \$(echo \${!f}) 1>&${PORTAGE_PIPE_FD}" || exit $?
+ done
+ eval "exec ${PORTAGE_PIPE_FD}>&-"
set +f
else
# Note: readonly variables interfere with __preprocess_ebuild_env(), so
diff --git a/lib/portage/package/ebuild/doebuild.py
b/lib/portage/package/ebuild/doebuild.py
index 366cbb9ca..5115ff6a3 100644
--- a/lib/portage/package/ebuild/doebuild.py
+++ b/lib/portage/package/ebuild/doebuild.py
@@ -568,7 +568,7 @@ _doebuild_commands_without_builddir = (
)
def doebuild(myebuild, mydo, _unused=DeprecationWarning, settings=None,
debug=0, listonly=0,
- fetchonly=0, cleanup=0, dbkey=DeprecationWarning, use_cache=1,
fetchall=0, tree=None,
+ fetchonly=0, cleanup=0, use_cache=1, fetchall=0, tree=None,
mydbapi=None, vartree=None, prev_mtimes=None,
fd_pipes=None, returnpid=False):
"""
@@ -591,9 +591,6 @@ def doebuild(myebuild, mydo, _unused=DeprecationWarning,
settings=None, debug=0,
@type fetchonly: Boolean
@param cleanup: Passed to prepare_build_dirs (TODO: what does it do?)
@type cleanup: Boolean
- @param dbkey: A file path where metadata generated by the 'depend' phase
- will be written.
- @type dbkey: String
@param use_cache: Enables the cache
@type use_cache: Boolean
@param fetchall: Used to wrap fetch(), fetches all URIs (even ones
invalid due to USE conditionals)
@@ -637,11 +634,6 @@ def doebuild(myebuild, mydo, _unused=DeprecationWarning,
settings=None, debug=0,
"settings['EROOT'] is used.",
DeprecationWarning, stacklevel=2)
- if dbkey is not DeprecationWarning:
- warnings.warn("portage.doebuild() called "
- "with deprecated dbkey argument.",
- DeprecationWarning, stacklevel=2)
-
if not tree:
writemsg("Warning: tree not specified to doebuild\n")
tree = "porttree"
@@ -836,16 +828,8 @@ def doebuild(myebuild, mydo, _unused=DeprecationWarning,
settings=None, debug=0,
# get possible slot information from the deps file
if mydo == "depend":
- writemsg("!!! DEBUG: dbkey: %s\n" % str(dbkey), 2)
- if returnpid:
- return _spawn_phase(mydo, mysettings,
- fd_pipes=fd_pipes, returnpid=returnpid)
- if dbkey and dbkey is not DeprecationWarning:
- mysettings["dbkey"] = dbkey
- else:
- mysettings["dbkey"] = \
- os.path.join(mysettings.depcachedir,
"aux_db_key_temp")
-
+ if not returnpid:
+ raise TypeError("returnpid must be True for
depend phase")
return _spawn_phase(mydo, mysettings,
fd_pipes=fd_pipes, returnpid=returnpid)
--
2.33.0
[gentoo-portage-dev] [PATCH 0/4] Modernize metadata passing & add INHERIT to md5-cache
Hi, Here's a patchset that: 1. Removes some leftover cruft from the old cache format. 2. Switches internal ebuild.sh/Python metadata logic from using a format resembling the old cache with hardcoded line number to data mapping (sic!) in favor of plain key=value format. 3. Adds INHERIT key to md5-cache for better compatibility with PkgCore. This key lists eclasses directly inherited by ebuild (vs _eclasses_ that lists indirect inherits as well). Michał Górny (4): Remove deprecated dbkey support from doebuild/ebuild.sh Switch internal metadata to key=value format Remove UNUSED* auxdbkeys Include INHERIT value in generated cache bin/ebuild.sh | 36 --- bin/phase-functions.sh| 2 +- bin/portageq | 3 +- lib/_emerge/EbuildMetadataPhase.py| 15 lib/_emerge/Package.py| 3 +- lib/portage/__init__.py | 4 +-- lib/portage/dbapi/__init__.py | 3 +- .../ebuild/_config/special_env_vars.py| 2 +- lib/portage/package/ebuild/doebuild.py| 22 ++-- repoman/lib/repoman/qa_data.py| 2 +- 10 files changed, 33 insertions(+), 59 deletions(-) -- 2.33.0
Re: [gentoo-portage-dev] [PATCH v2 1/2] Revert "Revert "Generate a QA Notice when EXPORT_FUNCTIONS is called before inherit""
On Mon, 2021-08-30 at 08:22 +0200, Ulrich Müller wrote:
> Reinstate the QA notice, because Portage behavior deviates from PMS,
> and breakage of eclasses with Pkgcore has been observed recently.
>
> This reverts commit f44d32550861cb25c209ef61dcd7ae1aa230da1f.
>
> Bug: https://bugs.gentoo.org/399039
> Signed-off-by: Ulrich Müller
> ---
> bin/ebuild.sh | 8
> 1 file changed, 8 insertions(+)
>
> diff --git a/bin/ebuild.sh b/bin/ebuild.sh
> index 5916bedfc..1bca2c965 100755
> --- a/bin/ebuild.sh
> +++ b/bin/ebuild.sh
> @@ -243,6 +243,14 @@ inherit() {
> ECLASS_DEPTH=$(($ECLASS_DEPTH + 1))
> if [[ ${ECLASS_DEPTH} -gt 1 ]]; then
> debug-print "*** Multiple Inheritence (Level: ${ECLASS_DEPTH})"
> +
> + # Since ECLASS_DEPTH > 1, the following variables are locals
> from the
> + # previous inherit call in the call stack.
> + if [[ -n ${ECLASS} && -n ${!__export_funcs_var} ]] ; then
> + eqawarn "QA Notice: EXPORT_FUNCTIONS is called before
> inherit in ${ECLASS}.eclass."
> + eqawarn "For compatibility with <=portage-2.1.6.7, only
> call EXPORT_FUNCTIONS"
> + eqawarn "after inherit(s)."
> + fi
> fi
>
> local -x ECLASS
Merged both.
--
Best regards,
Michał Górny
[gentoo-portage-dev] [PATCH] post_emerge: Display all preserved libs with --verbose
---
lib/_emerge/post_emerge.py | 2 +-
lib/portage/util/_dyn_libs/display_preserved_libs.py | 11 +++
2 files changed, 8 insertions(+), 5 deletions(-)
diff --git a/lib/_emerge/post_emerge.py b/lib/_emerge/post_emerge.py
index 0c4df0d32..c540308d3 100644
--- a/lib/_emerge/post_emerge.py
+++ b/lib/_emerge/post_emerge.py
@@ -140,7 +140,7 @@ def post_emerge(myaction, myopts, myfiles,
else:
print()
print(colorize("WARN", "!!!") + " existing preserved
libs:")
- display_preserved_libs(vardbapi)
+ display_preserved_libs(vardbapi, verbose="--verbose" in
myopts)
print("Use " + colorize("GOOD", "emerge
@preserved-rebuild") +
" to rebuild packages using these libraries")
diff --git a/lib/portage/util/_dyn_libs/display_preserved_libs.py
b/lib/portage/util/_dyn_libs/display_preserved_libs.py
index 8deafc25e..5818501bb 100644
--- a/lib/portage/util/_dyn_libs/display_preserved_libs.py
+++ b/lib/portage/util/_dyn_libs/display_preserved_libs.py
@@ -6,7 +6,7 @@ import portage
from portage.output import colorize
-def display_preserved_libs(vardb):
+def display_preserved_libs(vardb, verbose=False):
MAX_DISPLAY = 3
@@ -36,7 +36,8 @@ def display_preserved_libs(vardb):
consumers.append(c)
consumers.sort()
consumer_map[f] = consumers
-
search_for_owners.update(consumers[:MAX_DISPLAY+1])
+ max_search = None if verbose else MAX_DISPLAY +
1
+ search_for_owners.update(consumers[:max_search])
owners = {}
for f in search_for_owners:
@@ -75,7 +76,9 @@ def display_preserved_libs(vardb):
# they don't need to be rebuilt (see bug
#461908).
consumers = consumers_non_preserved
- if len(consumers) == MAX_DISPLAY + 1:
+ if verbose:
+ max_display = None
+ elif len(consumers) == MAX_DISPLAY + 1:
# Display 1 extra consumer, instead of
displaying
# "used by 1 other files".
max_display = MAX_DISPLAY + 1
@@ -91,6 +94,6 @@ def display_preserved_libs(vardb):
owners_desc = ", ".join(x.mycpv for x
in owners.get(c, []))
print(colorize("WARN", " * ") + " used by
%s (%s)" % \
(c, owners_desc))
- if len(consumers) > max_display:
+ if not verbose and len(consumers) > max_display:
print(colorize("WARN", " * ") + " used by
%d other files" %
(len(consumers) - max_display))
--
2.33.0
Re: [gentoo-portage-dev] [PATCH 2/2] ebuild.sh: Update QA notice in inherit()
On Mon, 2021-08-30 at 00:47 +0200, Ulrich Mueller wrote: > > > > > > On Sun, 29 Aug 2021, Michał Górny wrote: > > > On Sun, 2021-08-29 at 22:06 +0200, Ulrich Müller wrote: > > > - eqawarn "For compatibility with > > > <=portage-2.1.6.7, only call EXPORT_FUNCTIONS" > > > - eqawarn "after inherit(s)." > > > + eqawarn "For compatibility with PMS, only > > > call EXPORT_FUNCTIONS after inherit(s)." > > > Could you add a sentence that the current Portage behavior is going > > to > > change in the future? > > Sure, but do we actually know that? That's the plan today, can't be 100% sure it'll happen. You can say 'can change' if you think it's better -- but people need some motivation or they're just going to ignore it as 'another PMS noise'. Alternatively, we could mention that pkgcore is broken by this. My point is mostly to say 'yes, this breaks stuff for users today, it's not just some theory'. -- Best regards, Michał Górny
Re: [gentoo-portage-dev] [PATCH 2/2] ebuild.sh: Update QA notice in inherit()
On Sun, 2021-08-29 at 22:06 +0200, Ulrich Müller wrote:
> Bug: https://bugs.gentoo.org/399039
> Signed-off-by: Ulrich Müller
> ---
> bin/ebuild.sh | 3 +--
> 1 file changed, 1 insertion(+), 2 deletions(-)
>
> diff --git a/bin/ebuild.sh b/bin/ebuild.sh
> index 1bca2c965..f5f3d9eec 100755
> --- a/bin/ebuild.sh
> +++ b/bin/ebuild.sh
> @@ -248,8 +248,7 @@ inherit() {
> # previous inherit call in the call stack.
> if [[ -n ${ECLASS} && -n ${!__export_funcs_var} ]] ; then
> eqawarn "QA Notice: EXPORT_FUNCTIONS is called before
> inherit in ${ECLASS}.eclass."
> - eqawarn "For compatibility with <=portage-2.1.6.7, only
> call EXPORT_FUNCTIONS"
> - eqawarn "after inherit(s)."
> + eqawarn "For compatibility with PMS, only call
> EXPORT_FUNCTIONS after inherit(s)."
> fi
> fi
>
Could you add a sentence that the current Portage behavior is going to
change in the future?
--
Best regards,
Michał Górny
Re: [gentoo-portage-dev] In what phase are file "merged"?
On Wed, 2021-06-23 at 12:40 +0200, Ulrich Mueller wrote: > > > > > > On Wed, 23 Jun 2021, Fabian Groffen wrote: > > > On 23-06-2021 08:47:58 +0200, Ulrich Mueller wrote: > > > It's somewhat hidden, but it's there: > > > https://projects.gentoo.org/pms/8/pms.html#x1-950009.1.10 > > > > > > 9.1.10 pkg_preinst > > > ... immediately before merging the package to the live filesystem. > > > ... > > > > > > 9.1.11 pkg_postinst > > > ... immediately after merging the package to the live filesystem. > > > ... > > > Aha, so does this mean pkg_prerm and pkg_postrm are run with > > replacing > > package in place, e.g. if they refer to scripts installed by the > > replaced package they may no longer exist or be the same? > > PMS has similar wording for pkg_prerm and pkg_postrm, so (at least > according to the spec) the old package will still be present in > pkg_prerm and will be gone in pkg_postrm. > > I don't think that the ebuild can rely on any particular status of the > new package in pkg_*rm (of the old package), or the status of the old > package in pkg_*inst (of the new package). I would even say that it can't rely on the particular status of the old package in any case, if it's meant to be removed. In particular, its dependencies can be unmerged before the package itself. -- Best regards, Michał Górny
[gentoo-portage-dev] [PATCH] Emit QA notices for setuptools warnings
Emit QA notices for setuptools warnings such as:
* QA Notice: setuptools warnings detected:
*
* Usage of dash-separated 'build-requires' will not be supported in
future versions. Please use the underscore name 'build_requires' instead
* Usage of dash-separated 'description-file' will not be supported in
future versions. Please use the underscore name 'description_file' instead
* Usage of dash-separated 'upload-dir' will not be supported in
future versions. Please use the underscore name 'upload_dir' instead
Signed-off-by: Michał Górny
---
lib/portage/package/ebuild/doebuild.py | 14 ++
1 file changed, 14 insertions(+)
diff --git a/lib/portage/package/ebuild/doebuild.py
b/lib/portage/package/ebuild/doebuild.py
index 86c1d40b4..b1557edd7 100644
--- a/lib/portage/package/ebuild/doebuild.py
+++ b/lib/portage/package/ebuild/doebuild.py
@@ -1863,6 +1863,10 @@ def _check_build_log(mysettings, out=None):
re.compile(r'g?make\[\d+\]: warning: jobserver unavailable:')
make_jobserver = []
+ # we deduplicate these since they is repeated for every setup.py call
+ setuptools_warn = set()
+ setuptools_warn_re = re.compile(r'.*\/setuptools\/.*: UserWarning:
(.*)')
+
def _eerror(lines):
for line in lines:
eerror(line, phase="install", key=mysettings.mycpv,
out=out)
@@ -1892,6 +1896,10 @@ def _check_build_log(mysettings, out=None):
if make_jobserver_re.match(line) is not None:
make_jobserver.append(line.rstrip("\n"))
+ m = setuptools_warn_re.match(line)
+ if m is not None:
+ setuptools_warn.add(m.group(1))
+
except (EOFError, zlib.error) as e:
_eerror(["portage encountered a zlib error: '%s'" % (e,),
"while reading the log file: '%s'" % logfile])
@@ -1945,6 +1953,12 @@ def _check_build_log(mysettings, out=None):
msg.extend("\t" + line for line in make_jobserver)
_eqawarn(msg)
+ if setuptools_warn:
+ msg = [_("QA Notice: setuptools warnings detected:")]
+ msg.append("")
+ msg.extend("\t" + line for line in sorted(setuptools_warn))
+ _eqawarn(msg)
+
f.close()
if f_real is not None:
f_real.close()
--
2.31.1
Re: [gentoo-portage-dev] [PATCH v2] emerge --search: auto-detect regular expressions (bug 737480)
s for matches of the supplied string in the ebuild repository.
> By default emerge uses a case-insensitive simple search, but you can
> -enable a regular expression search by prefixing the search string with %.
> +enable a regular expression search by prefixing the search string with %
> +(the % prefix can often be omitted if the
> +\fB\-\-regex\-search\-auto\fR option is enabled).
> For example, \fBemerge \-\-search "%^kde"\fR searches for any package whose
> name starts with "kde"; \fBemerge \-\-search "%gcc$"\fR searches for any
> package that ends with "gcc"; \fBemerge \-\-search "office"\fR searches for
> @@ -764,6 +766,14 @@ matching packages due to \fB\-\-rebuild\fR.
> A space separated list of package names or slot atoms. Emerge will not
> rebuild
> packages that depend on matching packages due to \fB\-\-rebuild\fR.
> .TP
> +.BR "\-\-regex\-search\-auto < y | n >"
> +Enable or disable automatic regular expression detection for search actions.
> +If this option is enabled (the default), then regular expression search
> +is automatically enabled when the search string is a valid regular expression
> +which contains any of these commonly used regular expression characters or
> +character sequences:
> +^ $ * [ ] { } | ? .+
> +.TP
> .BR \-\-oneshot ", " \-1
> Emerge as normal, but do not add the packages to the world file
> for later updating.
--
Best regards,
Michał Górny
signature.asc
Description: This is a digitally signed message part
Re: [gentoo-portage-dev] profile masking
On Fri, 2020-08-14 at 15:42 +, Joakim Tjernlund wrote: > On Fri, 2020-08-14 at 17:31 +0200, Ulrich Mueller wrote: > > > > > > > On Fri, 14 Aug 2020, Joakim Tjernlund wrote: > > > When pkgs are masked in the profile, it affects all variants of that > > > pkgs, even the ones that are in other overlays. > > > Example: > > > !!! The following installed packages are masked: > > > - sys-auth/sssd-::transmode (masked by: package.mask) > > > /usr/portage/profiles/package.mask: > > > # Matt Turner (2020-08-13) > > > # Masked for testing > > > My sssd- is now masked. > > > Could the profile syntax be extended to include syntax allowed in > > > /etc/portage ? Then one could use the ::gentoo syntax (or so I hope) > > > > The :: syntax is Portage specific and doesn't exist in EAPI 7. > > So there's no chance to get it into the profile dir anytime soon > > (because that would imply :: to be added to a future EAPI and the > > top-level profile dir to be bumped to that EAPI). > > Is profile part of EAPI? masks are not defined/used in ebuilds directly. > > > You could override the mask in your overlay's profile/package.mask > > instead, using an entry with the "-" operator. > > Yes, I know I can add that in profile/package.mask but I am looking for the > bigger > picture here. This has to stop somehow, there need to be something that limits > the mask scope to the repo/overlay it is defined. > Why is that? I dare say the bigger picture needs to include different mask reasons. Sure, 'masked for testing' may or may not make little sense for live ebuilds. However, 'masked for security issues' may pretty apply to custom repo ebuilds as well. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-portage-dev] [PATCH 1/3] Add cached portage.getpid() function
On Fri, 2020-08-07 at 21:08 -0700, Zac Medico wrote: > Since getpid is a syscall, cache results, and update them > via an after fork hook. > > Signed-off-by: Zac Medico > --- > lib/portage/__init__.py | 14 +++ > .../tests/process/test_AsyncFunction.py | 24 +++ > 2 files changed, 38 insertions(+) > > diff --git a/lib/portage/__init__.py b/lib/portage/__init__.py > index 916c93510..52902ba7d 100644 > --- a/lib/portage/__init__.py > +++ b/lib/portage/__init__.py > @@ -14,6 +14,7 @@ try: > if not hasattr(errno, 'ESTALE'): > # ESTALE may not be defined on some systems, such as interix. > errno.ESTALE = -1 > + import multiprocessing.util > import re > import types > import platform > @@ -368,6 +369,19 @@ _internal_caller = False > > _sync_mode = False > > +def _fork_watcher(_fork_watcher): > + _fork_watcher.current_pid = _os.getpid() I don't really like the idea of putting variables on functions. Would there be any problem with using a proper class here? > + > +_fork_watcher(_fork_watcher) > + > +multiprocessing.util.register_after_fork(_fork_watcher, _fork_watcher) > + > +def getpid(): > + """ > + Cached version of os.getpid(). ForkProcess updates the cache. > + """ > + return _fork_watcher.current_pid > + > def _get_stdin(): > """ > Buggy code in python's multiprocessing/process.py closes sys.stdin > diff --git a/lib/portage/tests/process/test_AsyncFunction.py > b/lib/portage/tests/process/test_AsyncFunction.py > index 55857026d..3b360e02f 100644 > --- a/lib/portage/tests/process/test_AsyncFunction.py > +++ b/lib/portage/tests/process/test_AsyncFunction.py > @@ -3,6 +3,7 @@ > > import sys > > +import portage > from portage import os > from portage.tests import TestCase > from portage.util._async.AsyncFunction import AsyncFunction > @@ -36,3 +37,26 @@ class AsyncFunctionTestCase(TestCase): > def testAsyncFunctionStdin(self): > loop = asyncio._wrap_loop() > loop.run_until_complete(self._testAsyncFunctionStdin(loop)) > + > + def _test_getpid_fork(self): > + """ > + Verify that portage.getpid() cache is updated in a forked child > process. > + """ > + loop = asyncio._wrap_loop() > + proc = AsyncFunction(scheduler=loop, target=portage.getpid) > + proc.start() > + proc.wait() > + self.assertEqual(proc.pid, proc.result) > + > + def test_getpid_fork(self): > + self._test_getpid_fork() > + > + def test_getpid_double_fork(self): > + """ > + Verify that portage.getpid() cache is updated correctly after > + two forks. > + """ > + loop = asyncio._wrap_loop() > + proc = AsyncFunction(scheduler=loop, > target=self._test_getpid_fork) > + proc.start() > + self.assertEqual(proc.wait(), 0) -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-portage-dev] [PATCH] lib/*: Fix useless-return
f print_changelog(self):
> @@ -687,7 +683,6 @@ class Display:
> if ebuild_path_cl is not None:
> self.changelogs.extend(_calc_changelog(
> ebuild_path_cl, pkg_info.previous_pkg,
> pkg.cpv))
> - return
>
>
> def check_system_world(self, pkg):
> diff --git a/lib/portage/elog/mod_custom.py b/lib/portage/elog/mod_custom.py
> index 7cfafeccc..aaf1d3b1b 100644
> --- a/lib/portage/elog/mod_custom.py
> +++ b/lib/portage/elog/mod_custom.py
> @@ -18,4 +18,3 @@ def process(mysettings, key, logentries, fulltext):
> retval = portage.process.spawn_bash(mylogcmd)
> if retval != 0:
> raise portage.exception.PortageException("!!!
> PORTAGE_ELOG_COMMAND failed with exitcode %d" % retval)
> - return
> diff --git a/lib/portage/elog/mod_echo.py b/lib/portage/elog/mod_echo.py
> index 80f2b11ac..a026847b7 100644
> --- a/lib/portage/elog/mod_echo.py
> +++ b/lib/portage/elog/mod_echo.py
> @@ -3,9 +3,10 @@
> # Distributed under the terms of the GNU General Public License v2
>
> import sys
> -from portage.output import EOutput, colorize
> +
> from portage.const import EBUILD_PHASES
> from portage.localization import _
> +from portage.output import EOutput, colorize
>
>
> _items = []
> @@ -61,4 +62,3 @@ def _finalize():
> for line in msgcontent:
> fmap[msgtype](line.strip("\n"))
> _items = []
> - return
> diff --git a/lib/portage/elog/mod_mail.py b/lib/portage/elog/mod_mail.py
> index 38eaa277f..f737a80ce 100644
> --- a/lib/portage/elog/mod_mail.py
> +++ b/lib/portage/elog/mod_mail.py
> @@ -41,5 +41,3 @@ def process(mysettings, key, logentries, fulltext):
> portage.mail.send_mail(mysettings, mymessage)
> except PortageException as e:
> writemsg("%s\n" % str(e), noiselevel=-1)
> -
> - return
> diff --git a/lib/portage/glsa.py b/lib/portage/glsa.py
> index 9260e7e09..1870d9338 100644
> --- a/lib/portage/glsa.py
> +++ b/lib/portage/glsa.py
> @@ -492,7 +492,6 @@ class Glsa:
> finally:
> f.close()
>
> - return None
>
> def parse(self, myfile):
> """
> @@ -583,7 +582,6 @@ class Glsa:
> self.packages[name].append(tmp)
> # TODO: services aren't really used yet
> self.services = self.affected.getElementsByTagName("service")
> - return None
>
> def dump(self, outstream=sys.stdout, encoding="utf-8"):
> """
> @@ -684,7 +682,6 @@ class Glsa:
> mode='a+', encoding=_encodings['content'],
> errors='strict')
> checkfile.write(_unicode_decode(self.nr + "\n"))
> checkfile.close()
> - return None
>
> def getMergeList(self, least_change=True):
> """
> diff --git a/lib/portage/mail.py b/lib/portage/mail.py
> index 6503b4cc9..f4fccd8c2 100644
> --- a/lib/portage/mail.py
> +++ b/lib/portage/mail.py
> @@ -136,4 +136,3 @@ def send_mail(mysettings, message):
> raise portage.exception.PortageException(_("!!! An
> error occurred while trying to send logmail:\n")+str(e))
> except socket.error as e:
> raise portage.exception.PortageException(_("!!! A
> network error occurred while trying to send logmail:\n%s\nSure you configured
> PORTAGE_ELOG_MAILURI correctly?") % str(e))
> - return
> diff --git a/lib/portage/sync/controller.py b/lib/portage/sync/controller.py
> index cb68e2c37..24ebf4ff8 100644
> --- a/lib/portage/sync/controller.py
> +++ b/lib/portage/sync/controller.py
> @@ -174,7 +174,7 @@ class SyncManager:
>
>
> def do_callback(self, result):
> - #print("result:", result, "callback()", self.callback)
> + # print("result:", result, "callback()", self.callback)
> exitcode, updatecache_flg = result
> self.exitcode = exitcode
> self.updatecache_flg = updatecache_flg
> @@ -184,7 +184,6 @@ class SyncManager:
> writemsg_level(msg + "\n")
> if self.callback:
> self.callback(exitcode, updatecache_flg)
> - return
>
>
> def perform_post_sync_hook(self, reponame, dosyncuri='',
> repolocation='
[gentoo-portage-dev] [PATCH] Remove py<3.4 import compatibility
---
bin/binhost-snapshot | 5 +--
lib/_emerge/BinpkgFetcher.py | 5 +--
lib/_emerge/BlockerCache.py | 6 +---
lib/portage/cache/anydbm.py | 31 +---
lib/portage/dbapi/bintree.py | 5 +--
lib/portage/dbapi/porttree.py | 6 +---
lib/portage/dbapi/vartree.py | 6 +---
lib/portage/getbinpkg.py | 36 +--
lib/portage/glsa.py | 5 +--
lib/portage/package/ebuild/fetch.py | 12 ++-
.../asyncio/test_policy_wrapper_recursion.py | 8 +
lib/portage/tests/util/futures/test_retry.py | 5 +--
lib/portage/tests/util/test_socks5.py | 11 ++
lib/portage/tests/util/test_xattr.py | 14 +---
lib/portage/util/__init__.py | 5 +--
.../util/_dyn_libs/PreservedLibsRegistry.py | 6 +---
lib/portage/util/_eventloop/EventLoop.py | 10 ++
.../util/_eventloop/asyncio_event_loop.py | 9 ++---
lib/portage/util/_urlopen.py | 12 ++-
lib/portage/util/futures/_asyncio/__init__.py | 7 ++--
lib/portage/util/futures/_asyncio/tasks.py| 7 +---
lib/portage/util/futures/events.py| 12 +++
lib/portage/util/futures/futures.py | 36 ---
lib/portage/util/futures/transports.py| 5 +--
lib/portage/util/futures/unix_events.py | 21 ---
25 files changed, 61 insertions(+), 224 deletions(-)
diff --git a/bin/binhost-snapshot b/bin/binhost-snapshot
index d677e7568..afab23592 100755
--- a/bin/binhost-snapshot
+++ b/bin/binhost-snapshot
@@ -8,10 +8,7 @@ import os
import sys
import textwrap
-try:
- from urllib.parse import urlparse
-except ImportError:
- from urlparse import urlparse
+from urllib.parse import urlparse
from os import path as osp
if osp.isfile(osp.join(osp.dirname(osp.dirname(osp.realpath(__file__))),
".portage_not_installed")):
diff --git a/lib/_emerge/BinpkgFetcher.py b/lib/_emerge/BinpkgFetcher.py
index c26c9bd6b..01c9bc1e6 100644
--- a/lib/_emerge/BinpkgFetcher.py
+++ b/lib/_emerge/BinpkgFetcher.py
@@ -6,10 +6,7 @@ import functools
from _emerge.AsynchronousLock import AsynchronousLock
from _emerge.CompositeTask import CompositeTask
from _emerge.SpawnProcess import SpawnProcess
-try:
- from urllib.parse import urlparse as urllib_parse_urlparse
-except ImportError:
- from urlparse import urlparse as urllib_parse_urlparse
+from urllib.parse import urlparse as urllib_parse_urlparse
import stat
import sys
import portage
diff --git a/lib/_emerge/BlockerCache.py b/lib/_emerge/BlockerCache.py
index d0beb46ac..5f8fb360e 100644
--- a/lib/_emerge/BlockerCache.py
+++ b/lib/_emerge/BlockerCache.py
@@ -7,11 +7,7 @@ from portage.util import writemsg
from portage.data import secpass
import portage
from portage import os
-
-try:
- import cPickle as pickle
-except ImportError:
- import pickle
+import pickle
class BlockerCache(portage.cache.mappings.MutableMapping):
"""This caches blockers of installed packages so that dep_check does not
diff --git a/lib/portage/cache/anydbm.py b/lib/portage/cache/anydbm.py
index 121a4eaf2..c02d85b4f 100644
--- a/lib/portage/cache/anydbm.py
+++ b/lib/portage/cache/anydbm.py
@@ -4,29 +4,14 @@
from __future__ import absolute_import
-try:
- import anydbm as anydbm_module
-except ImportError:
- # python 3.x
- import dbm as anydbm_module
+import dbm
try:
import dbm.gnu as gdbm
except ImportError:
- try:
- import gdbm
- except ImportError:
- gdbm = None
+ gdbm = None
-try:
- from dbm import whichdb
-except ImportError:
- from whichdb import whichdb
-
-try:
- import cPickle as pickle
-except ImportError:
- import pickle
+import pickle
from portage import _unicode_encode
from portage import os
import sys
@@ -53,15 +38,15 @@ class database(fs_template.FsBased):
self._db_path = os.path.join(self.location,
fs_template.gen_label(self.location, self.label)+default_db)
self.__db = None
mode = "w"
- if whichdb(self._db_path) in ("dbm.gnu", "gdbm"):
+ if dbm.whichdb(self._db_path) in ("dbm.gnu", "gdbm"):
# Allow multiple concurrent writers (see bug #53607).
mode += "u"
try:
# dbm.open() will not work with bytes in python-3.1:
# TypeError: can't concat bytes to str
- self.__db = anydbm_module.open(self._db_path,
+ self.__db = dbm.open(self._db_path,
mode, self._perms)
- except anydbm_module.error:
+ except dbm.error:
# XXX handle this at some point
try:
Re: [gentoo-portage-dev] [PATCH 2/3] Remove support code for Python < 3.3
Dnia July 17, 2020 8:10:25 AM UTC, Ulrich Mueller napisał(a):
>>>>>> On Fri, 17 Jul 2020, Michał Górny wrote:
>
>> --- a/lib/portage/util/_dyn_libs/PreservedLibsRegistry.py
>> +++ b/lib/portage/util/_dyn_libs/PreservedLibsRegistry.py
>> @@ -34,12 +34,9 @@ class PreservedLibsRegistry(object):
>>
>> _json_write_opts = {
>> "ensure_ascii": False,
>> -"indent": "\t",
>> -"sort_keys": True
>> +"indent": 4,
>
>This is assigning 4 now unconditionally, while it was "\t" before for
>modern Python versions. Is this intentional?
I'm sorry, this is my mistakd. I'll fix it when I get home.
>
>> +"sort_keys": True,
>
>Why the added comma?
>
>> }
>> -if sys.hexversion < 0x30200F0:
>> -# indent only supports int number of spaces
>> -_json_write_opts["indent"] = 4
>>
>> def __init__(self, root, filename):
>> """
>
>
>> --- a/lib/portage/util/mtimedb.py
>> +++ b/lib/portage/util/mtimedb.py
>> @@ -29,12 +29,9 @@ class MtimeDB(dict):
>>
>> _json_write_opts = {
>> "ensure_ascii": False,
>> -"indent": "\t",
>> +"indent": 4,
>
>Same here.
>
>> "sort_keys": True
>> }
>> -if sys.hexversion < 0x30200F0:
>> -# indent only supports int number of spaces
>> -_json_write_opts["indent"] = 4
>>
>> def __init__(self, filename):
>> dict.__init__(self)
>
>Ulrich
--
Best regards,
Michał Górny
[gentoo-portage-dev] [PATCH 1/3] Remove support code for Python < 3.2
Signed-off-by: Michał Górny
---
lib/portage/tests/process/test_poll.py | 9 +++--
.../tests/util/futures/asyncio/test_pipe_closed.py | 7 ++-
lib/portage/util/configparser.py| 13 +++--
3 files changed, 8 insertions(+), 21 deletions(-)
diff --git a/lib/portage/tests/process/test_poll.py
b/lib/portage/tests/process/test_poll.py
index f505b5049..3ea176c0d 100644
--- a/lib/portage/tests/process/test_poll.py
+++ b/lib/portage/tests/process/test_poll.py
@@ -1,4 +1,4 @@
-# Copyright 1998-2019 Gentoo Authors
+# Copyright 1998-2020 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2
import functools
@@ -34,11 +34,8 @@ class PipeReaderTestCase(TestCase):
def test_domain_socket(self):
def make_pipes():
- if sys.version_info >= (3, 2):
- read_end, write_end = socket.socketpair()
- return (read_end.detach(), write_end.detach()),
None
- else:
- self.skipTest('socket detach not supported')
+ read_end, write_end = socket.socketpair()
+ return (read_end.detach(), write_end.detach()), None
self._do_test(make_pipes)
def test_named_pipe(self):
diff --git a/lib/portage/tests/util/futures/asyncio/test_pipe_closed.py
b/lib/portage/tests/util/futures/asyncio/test_pipe_closed.py
index 507385c04..b8e5556d0 100644
--- a/lib/portage/tests/util/futures/asyncio/test_pipe_closed.py
+++ b/lib/portage/tests/util/futures/asyncio/test_pipe_closed.py
@@ -1,4 +1,4 @@
-# Copyright 2018 Gentoo Foundation
+# Copyright 2018-2020 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2
import errno
@@ -32,10 +32,7 @@ class _PipeClosedTestCase(object):
self._do_test(read_end, write_end)
def test_domain_socket(self):
- if sys.version_info >= (3, 2):
- read_end, write_end = socket.socketpair()
- else:
- self.skipTest('socket detach not supported')
+ read_end, write_end = socket.socketpair()
self._do_test(read_end.detach(), write_end.detach())
def test_named_pipe(self):
diff --git a/lib/portage/util/configparser.py b/lib/portage/util/configparser.py
index f3452231f..9bd9f9722 100644
--- a/lib/portage/util/configparser.py
+++ b/lib/portage/util/configparser.py
@@ -11,16 +11,9 @@ __all__ = ['ConfigParserError', 'NoOptionError',
'ParsingError',
import io
import sys
-try:
- from configparser import (Error as ConfigParserError,
- NoOptionError, ParsingError, RawConfigParser)
- if sys.hexversion >= 0x302:
- from configparser import ConfigParser as SafeConfigParser
- else:
- from configparser import SafeConfigParser
-except ImportError:
- from ConfigParser import (Error as ConfigParserError,
- NoOptionError, ParsingError, RawConfigParser, SafeConfigParser)
+from configparser import (Error as ConfigParserError,
+ NoOptionError, ParsingError, RawConfigParser)
+from configparser import ConfigParser as SafeConfigParser
from portage import _encodings
from portage import _unicode_encode
--
2.27.0
[gentoo-portage-dev] [PATCH 2/3] Remove support code for Python < 3.3
Signed-off-by: Michał Górny
---
bin/chpathtool.py | 8 +--
bin/doins.py | 14 +
lib/portage/_emirrordist/FetchTask.py | 13 ++--
lib/portage/dbapi/vartree.py | 25 ++--
lib/portage/util/_compare_files.py| 22 +++
.../util/_dyn_libs/PreservedLibsRegistry.py | 7 +--
lib/portage/util/movefile.py | 60 +--
lib/portage/util/mtimedb.py | 7 +--
8 files changed, 39 insertions(+), 117 deletions(-)
diff --git a/bin/chpathtool.py b/bin/chpathtool.py
index fbd18b987..c036046ae 100755
--- a/bin/chpathtool.py
+++ b/bin/chpathtool.py
@@ -128,12 +128,8 @@ def chpath_inplace(filename, is_text_file, old, new):
f.close()
if modified:
- if sys.hexversion >= 0x303:
- orig_mtime = orig_stat.st_mtime_ns
- os.utime(filename, ns=(orig_mtime, orig_mtime))
- else:
- orig_mtime = orig_stat[stat.ST_MTIME]
- os.utime(filename, (orig_mtime, orig_mtime))
+ orig_mtime = orig_stat.st_mtime_ns
+ os.utime(filename, ns=(orig_mtime, orig_mtime))
return modified
def chpath_inplace_symlink(filename, st, old, new):
diff --git a/bin/doins.py b/bin/doins.py
index 98dc4f810..a08e3f8c9 100644
--- a/bin/doins.py
+++ b/bin/doins.py
@@ -110,10 +110,6 @@ def _parse_install_options(
parser.add_argument('-p', '--preserve-timestamps', action='store_true')
split_options = shlex.split(options)
namespace, remaining = parser.parse_known_args(split_options)
- if namespace.preserve_timestamps and sys.version_info < (3, 3):
- # -p is not supported in this case, since timestamps cannot
- # be preserved with full precision
- remaining.append('-p')
# Because parsing '--mode' option is partially supported. If unknown
# arg for --mode is passed, namespace.mode is set to None.
if remaining or namespace.mode is None:
@@ -151,15 +147,7 @@ def _set_timestamps(source_stat, dest):
source_stat: stat result for the source file.
dest: path to the dest file.
"""
- os.utime(dest, (source_stat.st_atime, source_stat.st_mtime))
-
-
-if sys.version_info >= (3, 3):
- def _set_timestamps_ns(source_stat, dest):
- os.utime(dest, ns=(source_stat.st_atime_ns,
source_stat.st_mtime_ns))
-
- _set_timestamps_ns.__doc__ = _set_timestamps.__doc__
- _set_timestamps = _set_timestamps_ns
+ os.utime(dest, ns=(source_stat.st_atime_ns, source_stat.st_mtime_ns))
class _InsInProcessInstallRunner(object):
diff --git a/lib/portage/_emirrordist/FetchTask.py
b/lib/portage/_emirrordist/FetchTask.py
index 322de79ba..a1ba58822 100644
--- a/lib/portage/_emirrordist/FetchTask.py
+++ b/lib/portage/_emirrordist/FetchTask.py
@@ -1,4 +1,4 @@
-# Copyright 2013-2019 Gentoo Authors
+# Copyright 2013-2020 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2
from __future__ import division
@@ -385,14 +385,9 @@ class FetchTask(CompositeTask):
# Apply the timestamp from the source file, but
# just rely on umask for permissions.
try:
- if sys.hexversion >= 0x303:
- os.utime(copier.dest_path,
-
ns=(self._current_stat.st_mtime_ns,
- self._current_stat.st_mtime_ns))
- else:
- os.utime(copier.dest_path,
-
(self._current_stat[stat.ST_MTIME],
-
self._current_stat[stat.ST_MTIME]))
+ os.utime(copier.dest_path,
+ ns=(self._current_stat.st_mtime_ns,
+ self._current_stat.st_mtime_ns))
except OSError as e:
msg = "%s %s utime failed unexpectedly: %s" % \
(self.distfile, current_mirror.name, e)
diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
index abb0a9308..2e29b25e5 100644
--- a/lib/portage/dbapi/vartree.py
+++ b/lib/portage/dbapi/vartree.py
@@ -435,10 +435,7 @@ class vardbapi(dbapi):
if mysplit[0] == '*':
mysplit[0] = mysplit[0][1:]
try:
- if sys.hexversion >= 0x303:
- mystat =
os.stat(self.getpath(mysplit[0])).st_mtime_ns
- else:
- mystat =
os
[gentoo-portage-dev] [PATCH 3/3] Remove support code for Python < 3.4
Signed-off-by: Michał Górny --- lib/_emerge/AsynchronousLock.py | 12 +--- lib/_emerge/EbuildMetadataPhase.py| 12 +--- lib/_emerge/FifoIpcDaemon.py | 30 +-- lib/_emerge/PipeReader.py | 12 +--- lib/_emerge/SpawnProcess.py | 16 -- lib/portage/dbapi/_MergeProcess.py| 10 --- lib/portage/locks.py | 11 --- lib/portage/process.py| 2 +- lib/portage/util/_async/PipeLogger.py | 10 --- lib/portage/util/_eventloop/EventLoop.py | 23 -- .../util/_eventloop/global_event_loop.py | 15 -- lib/portage/util/futures/_asyncio/__init__.py | 26 +--- lib/portage/util/futures/unix_events.py | 4 +-- 13 files changed, 20 insertions(+), 163 deletions(-) diff --git a/lib/_emerge/AsynchronousLock.py b/lib/_emerge/AsynchronousLock.py index aed1bcb15..d2a6773ff 100644 --- a/lib/_emerge/AsynchronousLock.py +++ b/lib/_emerge/AsynchronousLock.py @@ -1,4 +1,4 @@ -# Copyright 2010-2018 Gentoo Foundation +# Copyright 2010-2020 Gentoo Authors # Distributed under the terms of the GNU General Public License v2 import fcntl @@ -192,16 +192,6 @@ class _LockProcess(AbstractPollTask): fcntl.fcntl(in_pr, fcntl.F_SETFL, fcntl.fcntl(in_pr, fcntl.F_GETFL) | os.O_NONBLOCK) - # FD_CLOEXEC is enabled by default in Python >=3.4. - if sys.hexversion < 0x304: - try: - fcntl.FD_CLOEXEC - except AttributeError: - pass - else: - fcntl.fcntl(in_pr, fcntl.F_SETFD, - fcntl.fcntl(in_pr, fcntl.F_GETFD) | fcntl.FD_CLOEXEC) - self.scheduler.add_reader(in_pr, self._output_handler) self._registered = True self._proc = SpawnProcess( diff --git a/lib/_emerge/EbuildMetadataPhase.py b/lib/_emerge/EbuildMetadataPhase.py index efe71892c..d00f194c2 100644 --- a/lib/_emerge/EbuildMetadataPhase.py +++ b/lib/_emerge/EbuildMetadataPhase.py @@ -1,4 +1,4 @@ -# Copyright 1999-2018 Gentoo Foundation +# Copyright 1999-2020 Gentoo Authors # Distributed under the terms of the GNU General Public License v2 from _emerge.SubProcess import SubProcess @@ -93,16 +93,6 @@ class EbuildMetadataPhase(SubProcess): fcntl.fcntl(master_fd, fcntl.F_SETFL, fcntl.fcntl(master_fd, fcntl.F_GETFL) | os.O_NONBLOCK) - # FD_CLOEXEC is enabled by default in Python >=3.4. - if sys.hexversion < 0x304: - try: - fcntl.FD_CLOEXEC - except AttributeError: - pass - else: - fcntl.fcntl(master_fd, fcntl.F_SETFD, - fcntl.fcntl(master_fd, fcntl.F_GETFD) | fcntl.FD_CLOEXEC) - fd_pipes[slave_fd] = slave_fd settings["PORTAGE_PIPE_FD"] = str(slave_fd) diff --git a/lib/_emerge/FifoIpcDaemon.py b/lib/_emerge/FifoIpcDaemon.py index 2ec69d1cb..ab1fdb572 100644 --- a/lib/_emerge/FifoIpcDaemon.py +++ b/lib/_emerge/FifoIpcDaemon.py @@ -1,14 +1,8 @@ -# Copyright 2010-2018 Gentoo Foundation +# Copyright 2010-2020 Gentoo Authors # Distributed under the terms of the GNU General Public License v2 import sys -try: - import fcntl -except ImportError: - # http://bugs.jython.org/issue1074 - fcntl = None - from portage import os from _emerge.AbstractPollTask import AbstractPollTask from portage.cache.mappings import slot_dict_class @@ -28,17 +22,6 @@ class FifoIpcDaemon(AbstractPollTask): self._files.pipe_in = \ os.open(self.input_fifo, os.O_RDONLY|os.O_NONBLOCK) - # FD_CLOEXEC is enabled by default in Python >=3.4. - if sys.hexversion < 0x304 and fcntl is not None: - try: - fcntl.FD_CLOEXEC - except AttributeError: - pass - else: - fcntl.fcntl(self._files.pipe_in, fcntl.F_SETFD, - fcntl.fcntl(self._files.pipe_in, - fcntl.F_GETFD) | fcntl.FD_CLOEXEC) - self.scheduler.add_reader( self._files.pipe_in, self._input_handler) @@ -55,17 +38,6 @@ class FifoIpcDaemon(AbstractPollTask): self._files.pipe_in = \ os.open(self.input_fifo, os.O_RDONLY|os.O_NONBLOCK) - # FD_CLOEXEC is enable
[gentoo-portage-dev] [PATCH] Clean up more py2 conditional code
Closes: https://github.com/gentoo/portage/pull/575 Signed-off-by: Michał Górny --- lib/portage/cache/anydbm.py | 4 +- lib/portage/cache/mappings.py | 45 +--- lib/portage/cache/sql_template.py | 5 +- lib/portage/cache/template.py | 12 ++--- lib/portage/elog/messages.py | 3 +- lib/portage/output.py | 3 +- lib/portage/package/ebuild/config.py | 11 +--- .../futures/asyncio/test_subprocess_exec.py | 4 -- lib/portage/tests/util/futures/test_retry.py | 2 - lib/portage/tests/util/test_socks5.py | 16 ++ lib/portage/util/__init__.py | 53 +-- lib/portage/util/_dyn_libs/NeededEntry.py | 10 lib/portage/util/digraph.py | 3 -- lib/portage/util/listdir.py | 2 - lib/portage/util/whirlpool.py | 25 - lib/portage/xpak.py | 2 - 16 files changed, 52 insertions(+), 148 deletions(-) diff --git a/lib/portage/cache/anydbm.py b/lib/portage/cache/anydbm.py index 88d85b0da..121a4eaf2 100644 --- a/lib/portage/cache/anydbm.py +++ b/lib/portage/cache/anydbm.py @@ -112,5 +112,5 @@ class database(fs_template.FsBased): self.__db.sync() self.__db.close() - if sys.hexversion >= 0x300: - items = iteritems + # TODO: do we need iteritems()? + items = iteritems diff --git a/lib/portage/cache/mappings.py b/lib/portage/cache/mappings.py index 0432fdf60..0adecde4a 100644 --- a/lib/portage/cache/mappings.py +++ b/lib/portage/cache/mappings.py @@ -25,9 +25,6 @@ class Mapping(object): def __iter__(self): return iter(self.keys()) - def keys(self): - return list(self.__iter__()) - def __contains__(self, key): try: value = self[key] @@ -46,12 +43,6 @@ class Mapping(object): for _, v in self.items(): yield v - def values(self): - return [v for _, v in self.iteritems()] - - def items(self): - return list(self.iteritems()) - def get(self, key, default=None): try: return self[key] @@ -64,10 +55,10 @@ class Mapping(object): def __len__(self): return len(list(self)) - if sys.hexversion >= 0x300: - items = iteritems - keys = __iter__ - values = itervalues + # TODO: do we need to keep iter*? + items = iteritems + keys = __iter__ + values = itervalues class MutableMapping(Mapping): """ @@ -184,8 +175,8 @@ class UserDict(MutableMapping): def clear(self): self.data.clear() - if sys.hexversion >= 0x300: - keys = __iter__ + keys = __iter__ + class ProtectedDict(MutableMapping): """ @@ -234,8 +225,8 @@ class ProtectedDict(MutableMapping): def __contains__(self, key): return key in self.new or (key not in self.blacklist and key in self.orig) - if sys.hexversion >= 0x300: - keys = __iter__ + keys = __iter__ + class LazyLoad(Mapping): """ @@ -271,8 +262,8 @@ class LazyLoad(Mapping): self.pull = None return key in self.d - if sys.hexversion >= 0x300: - keys = __iter__ + keys = __iter__ + _slot_dict_classes = weakref.WeakValueDictionary() @@ -328,9 +319,6 @@ def slot_dict_class(keys, prefix="_val_"): l += 1 return l - def keys(self): - return list(self) - def iteritems(self): prefix = self._prefix for k in self.allowed_keys: @@ -339,16 +327,10 @@ def slot_dict_class(keys, prefix="_val_"): except AttributeError: pass - def items(self): - return list(self.iteritems()) - def itervalues(self): for k, v in self.iteritems(): yield v - def values(self): - return list(self.itervalues()) - def __delitem__(self, k): try: delattr(self, self._prefix + k) @@ -447,10 +429,9 @@ def slot_dict_class(keys, prefix="_val_"): def __repr__(self): re
[gentoo-portage-dev] [PATCH] Eliminate the most of explicit py3 conditionals
Eliminate the most of py2/py3 conditions in the code. Leave a few
where the relevant code is unclear, they will be addressed later.
Closes: https://github.com/gentoo/portage/pull/574
Signed-off-by: Michał Górny
---
bin/check-implicit-pointer-usage.py | 25 ++
bin/chmod-lite.py | 11 ++---
bin/dohtml.py | 11 ++---
bin/doins.py | 7 ++-
bin/ebuild| 15 +-
bin/filter-bash-environment.py| 11 ++---
bin/install.py| 19 ---
bin/pid-ns-init | 10 ++--
bin/xattr-helper.py | 32 +++-
lib/_emerge/DependencyArg.py | 10
lib/_emerge/JobStatusDisplay.py | 4 +-
lib/_emerge/Package.py| 8 ---
lib/_emerge/PackageVirtualDbapi.py| 3 --
lib/_emerge/SequentialTaskQueue.py| 3 --
lib/_emerge/TaskSequence.py | 3 --
lib/_emerge/UseFlagDisplay.py | 8 ---
lib/_emerge/UserQuery.py | 17 +++
lib/_emerge/actions.py| 2 -
lib/_emerge/resolver/DbapiProvidesIndex.py| 3 --
lib/_emerge/resolver/output_helpers.py| 16 --
lib/_emerge/resolver/slot_collision.py| 8 ---
lib/portage/__init__.py | 49 +++
lib/portage/_emirrordist/Config.py| 4 --
lib/portage/_selinux.py | 12 -
lib/portage/_sets/base.py | 3 --
lib/portage/dbapi/porttree.py | 7 +--
lib/portage/dep/__init__.py | 18 +--
lib/portage/dep/soname/SonameAtom.py | 8 ---
lib/portage/elog/mod_save_summary.py | 2 -
lib/portage/elog/mod_syslog.py| 4 --
lib/portage/exception.py | 45 -
lib/portage/mail.py | 49 +++
lib/portage/manifest.py | 7 ---
lib/portage/output.py | 4 +-
lib/portage/process.py| 16 +-
lib/portage/proxy/objectproxy.py | 6 ---
lib/portage/repository/config.py | 6 ---
.../tests/unicode/test_string_format.py | 44 -
lib/portage/util/_ShelveUnicodeWrapper.py | 45 -
39 files changed, 102 insertions(+), 453 deletions(-)
delete mode 100644 lib/portage/util/_ShelveUnicodeWrapper.py
diff --git a/bin/check-implicit-pointer-usage.py
b/bin/check-implicit-pointer-usage.py
index a49db8107..868e4b3c8 100755
--- a/bin/check-implicit-pointer-usage.py
+++ b/bin/check-implicit-pointer-usage.py
@@ -33,22 +33,10 @@ pointer_pattern = (
+ r"|"
+ r"cast to pointer from integer of different size)")
-if sys.hexversion < 0x300:
-# Use encoded byte strings in python-2.x, since the python ebuilds are
-# known to remove the encodings module when USE=build is enabled (thus
-# disabling unicode decoding/encoding). The portage module has a
-# workaround for this, but currently we don't import that here since we
-# don't want to trigger potential sandbox violations due to stale pyc
-# files for the portage module.
-unicode_quote_open = '\xE2\x80\x98'
-unicode_quote_close = '\xE2\x80\x99'
-def write(msg):
-sys.stdout.write(msg)
-else:
-unicode_quote_open = '\u2018'
-unicode_quote_close = '\u2019'
-def write(msg):
-sys.stdout.buffer.write(msg.encode('utf_8', 'backslashreplace'))
+unicode_quote_open = '\u2018'
+unicode_quote_close = '\u2019'
+def write(msg):
+sys.stdout.buffer.write(msg.encode('utf_8', 'backslashreplace'))
pointer_pattern = re.compile(pointer_pattern)
@@ -57,10 +45,7 @@ last_implicit_linenum = -1
last_implicit_func = ""
while True:
-if sys.hexversion >= 0x300:
-line = sys.stdin.buffer.readline().decode('utf_8', 'replace')
-else:
-line = sys.stdin.readline()
+line = sys.stdin.buffer.readline().decode('utf_8', 'replace')
if not line:
break
# translate unicode open/close quotes to ascii ones
diff --git a/bin/chmod-lite.py b/bin/chmod-lite.py
index 177be7eab..7fe743ed1 100755
--- a/bin/chmod-lite.py
+++ b/bin/chmod-lite.py
@@ -12,12 +12,11 @@ os.chdir(os.environ["__PORTAGE_HELPER_CWD"])
def main(files):
- if sys.hexversion >= 0x300:
- # We can't trust that the filesystem encoding (locale dependent)
- # correctly matches the arguments, so use surrogateescape to
- # pass through the original argv bytes for Python 3.
- fs_encoding = sys.getfilesystemencoding()
- files = [x.encode(fs_encoding, 'surrogateescape') for x in
files]
+ # We can't tru
[gentoo-portage-dev] [PATCH] Eliminate basestring/long/_unicode py3 compat
Replace basestring and _unicode hacks with str, and long with int. Signed-off-by: Michał Górny --- bin/egencache | 7 +-- lib/_emerge/BinpkgFetcher.py | 5 +- lib/_emerge/BlockerCache.py | 19 ++- lib/_emerge/BlockerDB.py | 4 +- lib/_emerge/FakeVartree.py| 11 +--- lib/_emerge/JobStatusDisplay.py | 7 +-- lib/_emerge/Package.py| 27 - lib/_emerge/Scheduler.py | 5 +- lib/_emerge/actions.py| 21 +++ lib/_emerge/create_world_atom.py | 8 +-- lib/_emerge/depgraph.py | 15 ++--- lib/_emerge/main.py | 4 +- lib/_emerge/resolver/output.py| 26 - lib/_emerge/resolver/output_helpers.py| 6 +- lib/_emerge/resolver/slot_collision.py| 6 +- lib/portage/__init__.py | 12 ++-- lib/portage/_emirrordist/MirrorDistTask.py| 7 +-- lib/portage/_emirrordist/main.py | 9 +-- lib/portage/_sets/base.py | 7 +-- lib/portage/cache/flat_hash.py| 5 +- lib/portage/cache/fs_template.py | 7 +-- lib/portage/cache/index/pkg_desc_index.py | 10 +--- lib/portage/cache/metadata.py | 8 +-- lib/portage/cache/sqlite.py | 7 +-- lib/portage/cache/template.py | 19 ++- lib/portage/cvstree.py| 3 - lib/portage/dbapi/bintree.py | 55 --- lib/portage/dbapi/porttree.py | 6 +- lib/portage/dbapi/vartree.py | 33 +-- lib/portage/dep/__init__.py | 50 +++-- lib/portage/eclass_cache.py | 10 +--- lib/portage/elog/__init__.py | 5 +- lib/portage/elog/mod_echo.py | 5 +- lib/portage/elog/mod_syslog.py| 5 +- lib/portage/emaint/modules/binhost/binhost.py | 9 +-- lib/portage/emaint/modules/sync/sync.py | 9 +-- lib/portage/exception.py | 9 +-- lib/portage/getbinpkg.py | 5 +- lib/portage/locks.py | 17 +++--- lib/portage/mail.py | 7 +-- lib/portage/manifest.py | 12 +--- lib/portage/package/ebuild/config.py | 11 ++-- lib/portage/package/ebuild/doebuild.py| 8 +-- .../package/ebuild/getmaskingstatus.py| 7 +-- lib/portage/process.py| 5 +- lib/portage/proxy/lazyimport.py | 7 +-- lib/portage/repository/config.py | 5 +- lib/portage/sync/getaddrinfo_validate.py | 6 +- lib/portage/sync/modules/rsync/rsync.py | 7 +-- lib/portage/tests/dep/test_match_from_list.py | 7 +-- .../tests/resolver/ResolverPlayground.py | 9 +-- .../tests/unicode/test_string_format.py | 7 +-- lib/portage/update.py | 16 ++ lib/portage/util/__init__.py | 8 +-- lib/portage/util/_dyn_libs/LinkageMapELF.py | 6 +- .../util/_dyn_libs/PreservedLibsRegistry.py | 7 +-- lib/portage/util/_urlopen.py | 9 +-- lib/portage/util/changelog.py | 10 ++-- lib/portage/util/compression_probe.py | 4 +- lib/portage/util/configparser.py | 9 +-- lib/portage/util/env_update.py| 7 +-- lib/portage/util/install_mask.py | 9 +-- lib/portage/versions.py | 17 ++ lib/portage/xml/metadata.py | 12 ++-- repoman/lib/repoman/__init__.py | 7 +-- repoman/lib/repoman/main.py | 2 - repoman/lib/repoman/metadata.py | 5 -- .../modules/scan/metadata/ebuild_metadata.py | 5 +- repoman/lib/repoman/utilities.py | 3 - 69 files changed, 227 insertions(+), 490 deletions(-) diff --git a/bin/egencache b/bin/egencache index d172319f8..0847d73a7 100755 --- a/bin/egencache +++ b/bin/egencache @@ -75,9 +75,6 @@ else: else: from portage.xml.metadata import parse_metadata_use -if sys.hexversion >= 0x300: - # pylint: disable=W0622 - long = int def parse_args(args): usage = "egencache [options] ... [atom] ..." @@ -344,7 +341,7 @@ class GenCache(object): max_mtime = ec_hash.mtime if max_mtime == sc.mtime: max_mtime += 1 - max_mtime = long(max_mtime) + max_mtime = int(max_mtime) try: os.utime(ebuild_hash.location, (max_mtime, max_mtime)) e
[gentoo-portage-dev] [PATCH] Remove unnecessary time.monotonic() compat
time.monotonic() is available since py3.3, so there's no need for
the compat anymore.
Signed-off-by: Michał Górny
---
lib/portage/dbapi/vartree.py | 7 ++--
lib/portage/tests/util/futures/test_retry.py | 8 ++---
lib/portage/util/_eventloop/EventLoop.py | 6 ++--
lib/portage/util/monotonic.py| 34
4 files changed, 10 insertions(+), 45 deletions(-)
delete mode 100644 lib/portage/util/monotonic.py
diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
index 3687b471b..80ca0ab80 100644
--- a/lib/portage/dbapi/vartree.py
+++ b/lib/portage/dbapi/vartree.py
@@ -1,4 +1,4 @@
-# Copyright 1998-2019 Gentoo Authors
+# Copyright 1998-2020 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2
from __future__ import division, unicode_literals
@@ -36,7 +36,6 @@ portage.proxy.lazyimport.lazyimport(globals(),
'portage.util.install_mask:install_mask_dir,InstallMask,_raise_exc',
'portage.util.listdir:dircache,listdir',
'portage.util.movefile:movefile',
- 'portage.util.monotonic:monotonic',
'portage.util.path:first_existing,iter_parents',
'portage.util.writeable_check:get_ro_checker',
'portage.util._xattr:xattr',
@@ -3616,7 +3615,7 @@ class dblink(object):
symlink_collisions = []
destroot = self.settings['ROOT']
totfiles = len(file_list) + len(symlink_list)
- previous = monotonic()
+ previous = time.monotonic()
progress_shown = False
report_interval = 1.7 # seconds
falign = len("%d" % totfiles)
@@ -3625,7 +3624,7 @@ class dblink(object):
for i, (f, f_type) in enumerate(chain(
((f, "reg") for f in file_list),
((f, "sym") for f in symlink_list))):
- current = monotonic()
+ current = time.monotonic()
if current - previous > report_interval:
showMessage(_("%3d%% done, %*d files
remaining ...\n") %
(i * 100 / totfiles,
falign, totfiles - i))
diff --git a/lib/portage/tests/util/futures/test_retry.py
b/lib/portage/tests/util/futures/test_retry.py
index 7a1e76280..4530bba83 100644
--- a/lib/portage/tests/util/futures/test_retry.py
+++ b/lib/portage/tests/util/futures/test_retry.py
@@ -1,4 +1,4 @@
-# Copyright 2018 Gentoo Foundation
+# Copyright 2018-2020 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2
try:
@@ -12,6 +12,7 @@ except ImportError:
import dummy_threading as threading
import sys
+import time
from portage.tests import TestCase
from portage.util._eventloop.global_event_loop import global_event_loop
@@ -19,7 +20,6 @@ from portage.util.backoff import RandomExponentialBackoff
from portage.util.futures import asyncio
from portage.util.futures.retry import retry
from portage.util.futures.executor.fork import ForkExecutor
-from portage.util.monotonic import monotonic
class SucceedLaterException(Exception):
@@ -31,12 +31,12 @@ class SucceedLater(object):
A callable object that succeeds some duration of time has passed.
"""
def __init__(self, duration):
- self._succeed_time = monotonic() + duration
+ self._succeed_time = time.monotonic() + duration
def __call__(self):
loop = global_event_loop()
result = loop.create_future()
- remaining = self._succeed_time - monotonic()
+ remaining = self._succeed_time - time.monotonic()
if remaining > 0:
loop.call_soon_threadsafe(lambda: None if result.done()
else
result.set_exception(SucceedLaterException(
diff --git a/lib/portage/util/_eventloop/EventLoop.py
b/lib/portage/util/_eventloop/EventLoop.py
index ffd12cff9..a3bea97aa 100644
--- a/lib/portage/util/_eventloop/EventLoop.py
+++ b/lib/portage/util/_eventloop/EventLoop.py
@@ -1,4 +1,4 @@
-# Copyright 1999-2018 Gentoo Foundation
+# Copyright 1999-2020 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2
from __future__ import division
@@ -11,6 +11,7 @@ import os
import select
import signal
import sys
+import time
import traceback
try:
@@ -37,7 +38,6 @@ portage.proxy.lazyimport.lazyimport(globals(),
)
from portage.util import writemsg_level
-from portage.util.monotonic import monotonic
from ..SlotObject import SlotObject
from .PollConstants import PollConstants
from .PollSelectAdapter import PollSelectAdapter
@@ -887,7 +887,7 @@ class EventLoop(
Re: [gentoo-portage-dev] [PATCH] travis.yml: drop python 2.7 (bug 731114)
On Mon, 2020-07-06 at 11:42 -0700, Zac Medico wrote: > It should be pretty safe to drop support for python2.7 at this point. > We should probably also change the trove classifier to ... Python :: 3 :: Only -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-portage-dev] Speeding up Tree Verification
On Tue, 2020-06-30 at 16:51 -0500, Sid Spry wrote: > On Tue, Jun 30, 2020, at 2:29 PM, Michał Górny wrote: > > On Tue, 2020-06-30 at 12:50 -0500, Sid Spry wrote: > > > On Tue, Jun 30, 2020, at 2:28 AM, Michał Górny wrote: > > > > Dnia June 30, 2020 2:13:43 AM UTC, Sid Spry napisał(a): > > > > > Hello, > > > > > > > > > > I have some runnable pseudocode outlining a faster tree verification > > > > > algorithm. > > > > > Before I create patches I'd like to see if there is any guidance on > > > > > making the > > > > > changes as unobtrusive as possible. If the radical change in algorithm > > > > > is > > > > > acceptable I can work on adding the changes. > > > > > > > > > > Instead of composing any kind of structured data out of the portage > > > > > tree my > > > > > algorithm just lists all files and then optionally batches them out to > > > > > threads. > > > > > There is a noticeable speedup by eliding the tree traversal operations > > > > > which > > > > > can be seen when running the algorithm with a single thread and > > > > > comparing it to > > > > > the current algorithm in gemato (which should still be discussed > > > > > here?). > > > > > > > > Without reading the code: does your algorithm correctly detect > > > > extraneous files? > > > > > > > > > > Yes and no. > > > > > > I am not sure why this is necessary. If the file does not appear in a > > > manifest it is > > > ignored. It makes the most sense to me to put the burden of not including > > > untracked files on the publisher. If the user puts an untracked file into > > > the tree it > > > will be ignored to no consequence; the authored files don't refer to it, > > > after all. > > > > This is necessary because a malicious third party can MITM you an rsync > > tree with extraneous files (say, -r1 baselayout ebuild) that do horrible > > things on your system. If you don't reject files not in Manifest, you > > open a huge security hole. > > > > Ok, I will refer to https://www.gentoo.org/glep/glep-0074.html and implement > the > checks in detail, but will still need to spend some time looking for the best > place > to insert the code. > > I think it best to address this from two fronts. On one hand rejecting extra > files > seems to have immediate benefit but the larger issue is portage exposing > untracked potentially malicious files to the user. I've pushed two branches with two exploits I could think of. Please test your tools against them. In both cases, the directory should be rejected without any ill effects. https://github.com/mgorny/glep74-test-cases > > Has anything like a verity loopback filesystem been explored? It might reduce > duplication of work. I don't understand what a 'verity loopback filesystem' is. Could you elaborate? -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-portage-dev] Speeding up Tree Verification
On Tue, 2020-06-30 at 12:50 -0500, Sid Spry wrote: > On Tue, Jun 30, 2020, at 2:28 AM, Michał Górny wrote: > > Dnia June 30, 2020 2:13:43 AM UTC, Sid Spry napisał(a): > > > Hello, > > > > > > I have some runnable pseudocode outlining a faster tree verification > > > algorithm. > > > Before I create patches I'd like to see if there is any guidance on > > > making the > > > changes as unobtrusive as possible. If the radical change in algorithm > > > is > > > acceptable I can work on adding the changes. > > > > > > Instead of composing any kind of structured data out of the portage > > > tree my > > > algorithm just lists all files and then optionally batches them out to > > > threads. > > > There is a noticeable speedup by eliding the tree traversal operations > > > which > > > can be seen when running the algorithm with a single thread and > > > comparing it to > > > the current algorithm in gemato (which should still be discussed > > > here?). > > > > Without reading the code: does your algorithm correctly detect extraneous > > files? > > > > Yes and no. > > I am not sure why this is necessary. If the file does not appear in a > manifest it is > ignored. It makes the most sense to me to put the burden of not including > untracked files on the publisher. If the user puts an untracked file into the > tree it > will be ignored to no consequence; the authored files don't refer to it, > after all. This is necessary because a malicious third party can MITM you an rsync tree with extraneous files (say, -r1 baselayout ebuild) that do horrible things on your system. If you don't reject files not in Manifest, you open a huge security hole. > But it would be easy enough to build a second list of all files and compare > it to > the list of files built from the manifests. If there are extras an error can > be > generated. This is actually the first test I did on my manifest parsing code. > I tried > to see if my tracked files roughly matched the total files in tree. That can > be > repurposed for this check. > > > > Some simple tests like counting all objects traversed and verified > > > returns the > > > same(ish). Once it is put into portage it could be tested in detail. > > > > > > There is also my partial attempt at removing the brittle interface to > > > GnuPG > > > (it's not as if the current code is badly designed, just that parsing > > > the > > > output of GnuPG directly is likely not the best idea). > > > > The 'brittle interface' is well-defined machine-readable output. > > > > Ok. I was aware there was a machine interface, but the classes that manipulate > a temporary GPG home seemed like not the best solution. I guess that is all > due to GPG assuming everything is in ~/.gnupg and keeping its state as a > directory structure. A temporary home directory guarantees that user configuration does not affect the verification result. > > > > Needs gemato, dnspython, and requests. Slightly better than random code > > > because > > > I took inspiration from the existing gemato classes. > > > > The code makes a lot of brittle assumptions about the structure. The > > GLEP was specifically designed to avoid that and let us adjust the > > structure in the future to meet our needs. > > > > These same assumptions are built into the code that operates on the > tree structure. If the GLEP were changed the existing code would also > potentially need changing. This code just uses the structure in a different > way. > The code that predates the GLEP, yes. It will eventually be changed to be more flexible, especially when we can assume that we start removing backwards compatibility. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-portage-dev] Speeding up Tree Verification
>> glsa.build_manifest_list()
>> glsa.parse_manifests()
>>
>> start = timeit.default_timer()
>> glsa.verify_manifests()
>> end = timeit.default_timer()
>> pprint(end - start)
>>
>> # Handled by checking for None.
>> #no_ignore = filter(lambda x: x._tag != 'ignore',
>glsa_manifest_results)
>>
>>
>> #pprint(glsa._manifest_results)
>> real_files = [x for x in filter(lambda x: x is not None,
>glsa._manifest_results)]
>> #pprint(real_files)
>> pprint(len(glsa._manifest_results))
>> pprint(len(real_files))
>>
>> all_files = []
>> for path, dirs, files in os.walk('/var/db/repos/gentoo'):
>> pass
>>
>> return 0
>>
>> if __name__ == '__main__':
>> sys.exit(main())
>> ```
>>
>> ```python (wkd.py, likely unneeded but I didn't want to redo these
>files yet)
>> #!/usr/bin/env python3
>> import sys, hashlib
>> import dns
>> from dns import (
>> name, query, dnssec,
>> message, resolver, rdatatype
>> )
>> import shutil, requests
>>
>> def check_domain_signature(domain: str) -> bool:
>> response = dns.resolver.query(domain, dns.rdatatype.NS)
>> nsname = response.rrset[0]
>> response = dns.resolver.query(str(nsname), dns.rdatatype.A)
>> nsaddr = response.rrset[0].to_text()
>>
>> # DNSKEY
>> request = dns.message.make_query(domain,
>> dns.rdatatype.DNSKEY, want_dnssec=True)
>> response = dns.query.udp(request, nsaddr)
>> if response.rcode() != 0:
>> raise Exception('Unable to request dnskey.')
>>
>> answer = response.answer
>> if len(answer) != 2:
>> raise Exception('Malformed answer to dnskey query.')
>>
>> name = dns.name.from_text(domain)
>> try:
>> dns.dnssec.validate(answer[0], answer[1], {name: answer[0]})
>> except dns.dnssec.ValidationFailure as exc:
>> # Validation failed. The raise causes python to abort with
>status 1.
>> #raise exc
>> return False
>> except AttributeError as exc:
>> # Validation may have failed; DNSKEY missing signer
>attribute. dig may report
>> # domain as valid.
>> #
>> # TODO: Additional state where subdomain of valid domain may
>fail with 3 nested
>> # KeyErrors. Avoid temptation to wildcard catch. Safer to put
>in process?
>> #raise exc
>> return False
>> else:
>> return True
>>
>> def hash_localpart(incoming: bytes) -> str:
>> '''Z-base32 the localpart of an e-mail address
>>
>>
>https://tools.ietf.org/html/draft-koch-openpgp-webkey-service-08#section-3.1
>> describes why this is needed.
>>
>> See https://tools.ietf.org/html/rfc6189#section-5.1.6 for a
>> description of the z-base32 scheme.
>> '''
>> zb32 = "ybndrfg8ejkmcpqxot1uwisza345h769"
>>
>> b = hashlib.sha1(incoming).digest()
>> ret = ""
>> assert(len(b) * 8 == 160)
>> for i in range(0, 160, 5):
>> byte = i // 8
>> offset = i - byte * 8
>> # offset | bits remaining in k+1 | right-shift k+1
>> # 3 | 0 | x
>> # 4 | 1 | 7
>> # 5 | 2 | 6
>> # 6 | 3 | 5
>> # 7 | 4 | 4
>> if offset < 4:
>> n = (b[byte] >> (3 - offset))
>> else:
>> n = (b[byte] << (offset - 3)) + (b[byte + 1] >> (11 -
>offset))
>>
>> ret += zb32[n & 0b1]
>> return ret
>>
>> def build_web_key_uri(address: str) -> str:
>> local, remote = address.split('@')
>> local = hash_localpart(local.encode('utf-8'))
>> return 'https://' + remote + '/.well-known/openpgpkey/hu/' + \
>> local
>>
>> def stream_to_file(uri: str, fname: str) -> None:
>> with requests.get(uri, verify=True, stream=True) as r:
>> from pprint import pprint
>> pprint(r.headers)
>> with open(fname, 'wb') as f:
>> shutil.copyfileobj(r.raw, f)
>> ```
>>
--
Best regards,
Michał Górny
Re: [gentoo-portage-dev] Speeding up Tree Verification
est, nsaddr)
>if response.rcode() != 0:
>raise Exception('Unable to request dnskey.')
>
>answer = response.answer
>if len(answer) != 2:
>raise Exception('Malformed answer to dnskey query.')
>
>name = dns.name.from_text(domain)
>try:
>dns.dnssec.validate(answer[0], answer[1], {name: answer[0]})
>except dns.dnssec.ValidationFailure as exc:
> # Validation failed. The raise causes python to abort with status 1.
>#raise exc
>return False
>except AttributeError as exc:
># Validation may have failed; DNSKEY missing signer attribute. dig may
>report
># domain as valid.
>#
># TODO: Additional state where subdomain of valid domain may fail with
>3 nested
># KeyErrors. Avoid temptation to wildcard catch. Safer to put in
>process?
>#raise exc
>return False
>else:
>return True
>
>def hash_localpart(incoming: bytes) -> str:
>'''Z-base32 the localpart of an e-mail address
>
>https://tools.ietf.org/html/draft-koch-openpgp-webkey-service-08#section-3.1
>describes why this is needed.
>
>See https://tools.ietf.org/html/rfc6189#section-5.1.6 for a
>description of the z-base32 scheme.
>'''
>zb32 = "ybndrfg8ejkmcpqxot1uwisza345h769"
>
>b = hashlib.sha1(incoming).digest()
>ret = ""
>assert(len(b) * 8 == 160)
>for i in range(0, 160, 5):
>byte = i // 8
>offset = i - byte * 8
># offset | bits remaining in k+1 | right-shift k+1
># 3 | 0 | x
># 4 | 1 | 7
># 5 | 2 | 6
># 6 | 3 | 5
># 7 | 4 | 4
>if offset < 4:
>n = (b[byte] >> (3 - offset))
>else:
> n = (b[byte] << (offset - 3)) + (b[byte + 1] >> (11 - offset))
>
>ret += zb32[n & 0b1]
>return ret
>
>def build_web_key_uri(address: str) -> str:
>local, remote = address.split('@')
>local = hash_localpart(local.encode('utf-8'))
>return 'https://' + remote + '/.well-known/openpgpkey/hu/' + \
>local
>
>def stream_to_file(uri: str, fname: str) -> None:
>with requests.get(uri, verify=True, stream=True) as r:
>from pprint import pprint
>pprint(r.headers)
>with open(fname, 'wb') as f:
>shutil.copyfileobj(r.raw, f)
>```
--
Best regards,
Michał Górny
Re: [gentoo-portage-dev] Add caching to a few commonly used functions
Dnia June 28, 2020 3:42:33 AM UTC, Zac Medico napisał(a): >On 6/27/20 8:12 PM, Michał Górny wrote: >> Dnia June 28, 2020 3:00:00 AM UTC, Zac Medico >napisał(a): >>> On 6/26/20 11:34 PM, Chun-Yu Shei wrote: >>>> Hi, >>>> >>>> I was recently interested in whether portage could be speed up, >since >>>> dependency resolution can sometimes take a while on slower >machines. >>>> After generating some flame graphs with cProfile and vmprof, I >found >>> 3 >>>> functions which seem to be called extremely frequently with the >same >>>> arguments: catpkgsplit, use_reduce, and match_from_list. In the >>> first >>>> two cases, it was simple to cache the results in dicts, while >>>> match_from_list was a bit trickier, since it seems to be a >>> requirement >>>> that it return actual entries from the input "candidate_list". I >>> also >>>> ran into some test failures if I did the caching after the >>>> mydep.unevaluated_atom.use and mydep.repo checks towards the end of >>> the >>>> function, so the caching is only done up to just before that point. >>>> >>>> The catpkgsplit change seems to definitely be safe, and I'm pretty >>> sure >>>> the use_reduce one is too, since anything that could possibly >change >>> the >>>> result is hashed. I'm a bit less certain about the match_from_list >>> one, >>>> although all tests are passing. >>>> >>>> With all 3 patches together, "emerge -uDvpU --with-bdeps=y @world" >>>> speeds up from 43.53 seconds to 30.96 sec -- a 40.6% speedup. >>> "emerge >>>> -ep @world" is just a tiny bit faster, going from 18.69 to 18.22 >sec >>>> (2.5% improvement). Since the upgrade case is far more common, >this >>>> would really help in daily use, and it shaves about 30 seconds off >>>> the time you have to wait to get to the [Yes/No] prompt (from ~90s >to >>>> 60s) on my old Sandy Bridge laptop when performing normal upgrades. >>>> >>>> Hopefully, at least some of these patches can be incorporated, and >>> please >>>> let me know if any changes are necessary. >>>> >>>> Thanks, >>>> Chun-Yu >>> >>> Using global variables for caches like these causes a form of memory >>> leak for use cases involving long-running processes that need to >work >>> with many different repositories (and perhaps multiple versions of >>> those >>> repositories). >>> >>> There are at least a couple of different strategies that we can use >to >>> avoid this form of memory leak: >>> >>> 1) Limit the scope of the caches so that they have some sort of >garbage >>> collection life cycle. For example, it would be natural for the >>> depgraph >>> class to have a local cache of use_reduce results, so that the cache >>> can >>> be garbage collected along with the depgraph. >>> >>> 2) Eliminate redundant calls. For example, redundant calls to >>> catpkgslit >>> can be avoided by constructing more _pkg_str instances, since >>> catpkgsplit is able to return early when its argument happens to be >a >>> _pkg_str instance. >> >> I think the weak stuff from the standard library might also be >helpful. >> >> -- >> Best regards, >> Michał Górny >> > >Hmm, maybe weak global caches are an option? It would probably be necessary to add hit/miss counter and compare results before and after. -- Best regards, Michał Górny
Re: [gentoo-portage-dev] Add caching to a few commonly used functions
Dnia June 28, 2020 3:00:00 AM UTC, Zac Medico napisał(a): >On 6/26/20 11:34 PM, Chun-Yu Shei wrote: >> Hi, >> >> I was recently interested in whether portage could be speed up, since >> dependency resolution can sometimes take a while on slower machines. >> After generating some flame graphs with cProfile and vmprof, I found >3 >> functions which seem to be called extremely frequently with the same >> arguments: catpkgsplit, use_reduce, and match_from_list. In the >first >> two cases, it was simple to cache the results in dicts, while >> match_from_list was a bit trickier, since it seems to be a >requirement >> that it return actual entries from the input "candidate_list". I >also >> ran into some test failures if I did the caching after the >> mydep.unevaluated_atom.use and mydep.repo checks towards the end of >the >> function, so the caching is only done up to just before that point. >> >> The catpkgsplit change seems to definitely be safe, and I'm pretty >sure >> the use_reduce one is too, since anything that could possibly change >the >> result is hashed. I'm a bit less certain about the match_from_list >one, >> although all tests are passing. >> >> With all 3 patches together, "emerge -uDvpU --with-bdeps=y @world" >> speeds up from 43.53 seconds to 30.96 sec -- a 40.6% speedup. >"emerge >> -ep @world" is just a tiny bit faster, going from 18.69 to 18.22 sec >> (2.5% improvement). Since the upgrade case is far more common, this >> would really help in daily use, and it shaves about 30 seconds off >> the time you have to wait to get to the [Yes/No] prompt (from ~90s to >> 60s) on my old Sandy Bridge laptop when performing normal upgrades. >> >> Hopefully, at least some of these patches can be incorporated, and >please >> let me know if any changes are necessary. >> >> Thanks, >> Chun-Yu > >Using global variables for caches like these causes a form of memory >leak for use cases involving long-running processes that need to work >with many different repositories (and perhaps multiple versions of >those >repositories). > >There are at least a couple of different strategies that we can use to >avoid this form of memory leak: > >1) Limit the scope of the caches so that they have some sort of garbage >collection life cycle. For example, it would be natural for the >depgraph >class to have a local cache of use_reduce results, so that the cache >can >be garbage collected along with the depgraph. > >2) Eliminate redundant calls. For example, redundant calls to >catpkgslit >can be avoided by constructing more _pkg_str instances, since >catpkgsplit is able to return early when its argument happens to be a >_pkg_str instance. I think the weak stuff from the standard library might also be helpful. -- Best regards, Michał Górny
Re: [gentoo-portage-dev] [PATCH 1/3] Add caching to catpkgsplit function
Dnia June 27, 2020 6:34:13 AM UTC, Chun-Yu Shei napisał(a):
>According to cProfile, catpkgsplit is called up to 1-5.5 million times
>during "emerge -uDvpU --with-bdeps=y @world". Adding a dict to cache
>its
>results reduces the time for this command from 43.53 -> 41.53 seconds
>--
>a 4.8% speedup.
Not saying caching is wrong for an interim solution but this is the kind of
function where refactoring may yield even more gain.
>---
> lib/portage/versions.py | 7 +++
> 1 file changed, 7 insertions(+)
>
>diff --git a/lib/portage/versions.py b/lib/portage/versions.py
>index 0c21373cc..ffec316ce 100644
>--- a/lib/portage/versions.py
>+++ b/lib/portage/versions.py
>@@ -312,6 +312,7 @@ def _pkgsplit(mypkg, eapi=None):
>
> _cat_re = re.compile('^%s$' % _cat, re.UNICODE)
> _missing_cat = 'null'
>+_catpkgsplit_cache = {}
>
> def catpkgsplit(mydata, silent=1, eapi=None):
> """
>@@ -331,6 +332,11 @@ def catpkgsplit(mydata, silent=1, eapi=None):
> return mydata.cpv_split
> except AttributeError:
> pass
>+
>+ cache_entry = _catpkgsplit_cache.get(mydata)
>+ if cache_entry is not None:
>+ return cache_entry
>+
> mysplit = mydata.split('/', 1)
> p_split = None
> if len(mysplit) == 1:
>@@ -343,6 +349,7 @@ def catpkgsplit(mydata, silent=1, eapi=None):
> if not p_split:
> return None
> retval = (cat, p_split[0], p_split[1], p_split[2])
>+ _catpkgsplit_cache[mydata] = retval
> return retval
>
> class _pkg_str(_unicode):
--
Best regards,
Michał Górny
Re: [gentoo-portage-dev] [PATCH] Use env to find python
On Wed, 2020-06-17 at 10:18 +0200, Ulrich Mueller wrote:
> > > > > > On Tue, 16 Jun 2020, Michał Górny wrote:
> > '/usr/bin/env python' (with no extra options) is the portable shebang.
>
> I still think this is a gross hack. You want your script to use the
> correct interpreter (which is in /usr/bin, or the path substituted for
> it at install time), not some random binary which happens to be found
> in the user's ${PATH}.
>
How do you presume what the correct interpreter is, and why do you
presume that users put 'random binaries' in their PATH? What if
the user needs to override 'correct interpreter'? Requiring people to
overwrite executables in /usr/bin sounds like a 'gross hack'.
--
Best regards,
Michał Górny
signature.asc
Description: This is a digitally signed message part
Re: [gentoo-portage-dev] erroneous behavior in 2-style USE dependencies?
On Tue, 2020-06-16 at 23:07 +, Michael Lienhardt wrote: > Dear all, > > My bad for not noticing it sooner, but when there is a dependency like > ">=sys-fs/udev-208-r1:0/0[static-libs?]" (that occurs in > virtual/libgudev-215-r3), > since 'static-libs' is not a use flags of sys-fs/udev-242, that cpv is > silently not considered during dependency solving by emerge. > However, the PMS states: > - it is an error for a use dependency to be applied to an ebuild which does > not have the flag in question in IUSE_REFERENCEABLE > This is a bit like undefined behavior. PMS says such a thing shouldn't happen (i.e. the ebuild is wrong) but does not force specific error handling. You could reject the ebuild entirely or reject dependencies that don't have the flag in question (even if it's disabled). You could also pretend it's 'static-libs(-)?' but that would be bad if the default was supposed to be '(+)'. > This is related to the tool I'm working on: should my tool allow this > behavior, or fail like it is currently doing (I guess the former)? > That depends on what the tool is doing. I suppose you probably don't need very strict behavior there. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-portage-dev] [PATCH] Use env to find python
On Tue, 2020-06-16 at 15:19 -0400, Mike Gilbert wrote: > On Tue, Jun 16, 2020 at 1:55 PM Zac Medico wrote: > > On 6/16/20 10:46 AM, Mike Gilbert wrote: > > > On Tue, Jun 16, 2020 at 1:45 PM Mike Gilbert wrote: > > > > On Mon, Jun 15, 2020 at 9:39 AM Sid Spry wrote: > > > > > On Mon, Jun 15, 2020, at 2:36 AM, Ulrich Mueller wrote: > > > > > > But we know that it is in /usr/bin, so why add yet another > > > > > > indirection? > > > > > > > > > > > > Attachments: > > > > > > * signature.asc > > > > > > > > > > Ah, sorry -- I forgot to note this here. If you wish to support > > > > > prefix it is possible it may not be in /usr/bin. Granted I am not > > > > > sure if the prefix stage3 I was using is old enough to be broken in > > > > > some way, but adding this would prevent future breakage. > > > > > > > > The portage ebuild and the python distutils module already take care > > > > of updating shebangs at install time. > > > > > > I suppose your patch might be useful if you are trying to run portage > > > from a git checkout on a prefix system. > > > > > > > So, given that the ebuild updates shebangs automatically, should't we > > optimize the default shebangs to be as flexible as possible? > > Yes, that makes sense. > > However, we should test to make sure that distutils is smart enough to > parse that "/usr/bin/env -S python" string and replace it with > version-specific python shebang. > '/usr/bin/env python' (with no extra options) is the portable shebang. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-portage-dev] [PATCH] config.environ: delay export of A and AA (bug 720180)
On Mon, 2020-05-25 at 21:31 -0700, Zac Medico wrote: > Since variables like A and AA can contain extremely large values which > may trigger E2BIG errors during attempts to execute subprocesses, delay > export until the last moment, and unexport when appropriate. > Please don't. This will only hide the problem from developers who will unknowingly commit broken ebuilds and cause users of alternative package managers to suffer. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
[gentoo-portage-dev] [PATCH] process: Provide libc fallback for sethostname() on PyPy
Signed-off-by: Michał Górny
---
lib/portage/process.py | 9 -
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/lib/portage/process.py b/lib/portage/process.py
index 79052b608..ceb454030 100644
--- a/lib/portage/process.py
+++ b/lib/portage/process.py
@@ -715,7 +715,14 @@ def _exec(binary, mycommand, opt_name, fd_pipes,
if unshare_net:
# use 'localhost' to
avoid hostname resolution problems
try:
-
socket.sethostname('localhost')
+ # pypy3 does
not implement socket.sethostname()
+ new_hostname =
b'localhost'
+ if
hasattr(socket, 'sethostname'):
+
socket.sethostname(new_hostname)
+ else:
+ if
libc.sethostname(new_hostname, len(new_hostname)) != 0:
+
errno_value = ctypes.get_errno()
+
raise OSError(errno_value, os.strerror(errno_value))
except Exception as e:
writemsg("Unable to set hostname: %s (for FEATURES=\"network-sandbox\")\n" % (
e,),
--
2.26.2
Re: [gentoo-portage-dev] [PATCH] Enable FEATURES=parallel-install by default (bug 715110)
On Tue, 2020-05-12 at 13:18 -0700, Zac Medico wrote: > On 5/12/20 12:28 PM, Michał Górny wrote: > > W dniu wto, 12.05.2020 o godzinie 10∶05 -0700, użytkownik Zac Medico > > napisał: > > > On 5/12/20 3:39 AM, Michał Górny wrote: > > > > I'm sorry but I was asking of a more high-level implications. > > > > > > > > I presume that this means that more than files of more than one > > > > package > > > > can be merged simultaneously. However: > > > > > > No, an exclusive lock must be held on vardbapi._fs_lock for this. > > > This > > > is currently required at least to guarantee that access to the config > > > memory file is serialized (config memory is the thing that emerge > > > --noconfmem disables, but --noconfmem does not disable this lock). > > > > > > We assume that it's probably not worthwhile to try to merge files for > > > more than one package at a time, since that would cause them to > > > compete > > > for IO bandwidth. > > > > > > > 1. Are collisions handled correctly then? i.e. if you start > > > > installing > > > > A, and then B, and the two packages collide will portage fail > > > > before > > > > starting to install any file from B? > > > > > > There are no guarantees here. However, the risk is minimal, since > > > it's > > > unlikely that a file collision of this sort would occur. file > > > collisions > > > are a QA problem that is generally detected and corrected log before > > > we > > > would encounter a collision of this sort. > > > > > > > 2. Are preinst/postinst phases called simultaneously or serialized? > > > > > > They're serialized. > > > > Now I'm lost here. Could you try to explain to me, without getting > > into the deep technicalities, how parallel-install achieves better > > speed or at doing what is non-parallel-install so slow? > > It allows preinst/postinst/prerm/postrm phases to run for one package > while files are concurrently being merged or unmerged for another > package. This makes it possible to approach saturation of IO bandwidth. Doesn't this imply that programs run in these phases could fail if Portage is simultaneously replacing their files? -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-portage-dev] [PATCH] Enable FEATURES=parallel-install by default (bug 715110)
W dniu wto, 12.05.2020 o godzinie 10∶05 -0700, użytkownik Zac Medico napisał: > On 5/12/20 3:39 AM, Michał Górny wrote: > > I'm sorry but I was asking of a more high-level implications. > > > > I presume that this means that more than files of more than one > > package > > can be merged simultaneously. However: > > No, an exclusive lock must be held on vardbapi._fs_lock for this. > This > is currently required at least to guarantee that access to the config > memory file is serialized (config memory is the thing that emerge > --noconfmem disables, but --noconfmem does not disable this lock). > > We assume that it's probably not worthwhile to try to merge files for > more than one package at a time, since that would cause them to > compete > for IO bandwidth. > > > 1. Are collisions handled correctly then? i.e. if you start > > installing > > A, and then B, and the two packages collide will portage fail > > before > > starting to install any file from B? > > There are no guarantees here. However, the risk is minimal, since > it's > unlikely that a file collision of this sort would occur. file > collisions > are a QA problem that is generally detected and corrected log before > we > would encounter a collision of this sort. > > > 2. Are preinst/postinst phases called simultaneously or serialized? > > They're serialized. Now I'm lost here. Could you try to explain to me, without getting into the deep technicalities, how parallel-install achieves better speed or at doing what is non-parallel-install so slow? -- Best regards, Michał Górny
Re: [gentoo-portage-dev] [PATCH] Enable FEATURES=parallel-install by default (bug 715110)
W dniu wto, 12.05.2020 o godzinie 01∶40 -0700, użytkownik Zac Medico napisał: > On 5/11/20 10:54 PM, Michał Górny wrote: > > W dniu nie, 10.05.2020 o godzinie 21∶32 -0700, użytkownik Zac > > Medico > > napisał: > > > The feature enables finer grained locks for install operations, > > > and > > > everyone agrees that it's safe to enable by default. > > > > Who's 'everyone' and where's their analysis of the problem? > > The manpage doesn't really help understand what this does, exactly. > > Before parallel-install there was just one big lock, so only one > package > slot could enter the merge/unmerge state at a given time. > > With parallel install, there are a few finer-grained locks that come > into play. [...] I'm sorry but I was asking of a more high-level implications. I presume that this means that more than files of more than one package can be merged simultaneously. However: 1. Are collisions handled correctly then? i.e. if you start installing A, and then B, and the two packages collide will portage fail before starting to install any file from B? 2. Are preinst/postinst phases called simultaneously or serialized? -- Best regards, Michał Górny
Re: [gentoo-portage-dev] [PATCH] Enable FEATURES=parallel-install by default (bug 715110)
W dniu nie, 10.05.2020 o godzinie 21∶32 -0700, użytkownik Zac Medico napisał: > The feature enables finer grained locks for install operations, and > everyone agrees that it's safe to enable by default. Who's 'everyone' and where's their analysis of the problem? The manpage doesn't really help understand what this does, exactly. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-portage-dev] [PATCH] Change BINPKG_COMPRESS default from bzip2 to xz
On Sun, 2020-04-26 at 18:23 -0400, Michael Orlitzky wrote: > On 4/26/20 3:25 PM, Ulrich Mueller wrote: > > > > > > > On Sun, 26 Apr 2020, Michael Orlitzky wrote: > > > Fuel for the fire: > > > * https://www.nongnu.org/lzip/lzip_benchmark.html > > > * https://www.nongnu.org/lzip/xz_inadequate.html > > > > Yep. That's why lzip is the dominant compression format now, and nobody > > is using xz any more. > > > > If you believed that argument, you would be using Windows =P > > Dude has GRAPHS, it must be true. > It is true but nobody cares. People managed to shove a lot of unused code into the kernel and nobody cares. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-portage-dev] [PATCH] Change BINPKG_COMPRESS default from bzip2 to xz
On Sun, 2020-04-26 at 09:55 -0700, Matt Turner wrote: > Bug: https://bugs.gentoo.org/715108 > Signed-off-by: Matt Turner > --- > Strawman patch. Bikeshed away. > xz is generally slow and doesn't do parallel good. If we want to change this, we should go for something cool like zstd that scales better. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-portage-dev] Need backup mentor for FUSE-based sandbox project
On Sat, 2020-04-25 at 21:40 -0700, Alec Warner wrote: > On Thu, Apr 23, 2020 at 12:09 PM Michał Górny wrote: > > > Hi, everyone. > > > > It seems that we *urgently* (read: in 6 days) need to find backup > > mentors for this year's GSoC projects. I'm mentoring the project to > > develop a FUSE-based sandbox alternative that's going to work reliably > > with more packages than the LD_PRELOAD hack [1]. > > > > Would anyone volunteer to be the backup maintainer for this? > > > > [1] > > https://summerofcode.withgoogle.com/dashboard/organization/5749981018849280/proposal/5572241732927488/ > > I get a 403 for that. > > https://wiki.gentoo.org/wiki/Google_Summer_of_Code/2020/Ideas#FUSE-powered_sandbox > is > your writeup; I assume the above is supposed to be a link to the student's > proposal? Yes and yes. I suppose idea of stable links didn't reach Google yet. However, I should have noted that we have a backup mentor now but there's no harm in having a second backup ;-). > > > > > > -- > > Best regards, > > Michał Górny > > > > -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
[gentoo-portage-dev] Need backup mentor for FUSE-based sandbox project
Hi, everyone. It seems that we *urgently* (read: in 6 days) need to find backup mentors for this year's GSoC projects. I'm mentoring the project to develop a FUSE-based sandbox alternative that's going to work reliably with more packages than the LD_PRELOAD hack [1]. Would anyone volunteer to be the backup maintainer for this? [1] https://summerofcode.withgoogle.com/dashboard/organization/5749981018849280/proposal/5572241732927488/ -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-portage-dev] [PATCH v2] process: Unshare UTS namespace, and set hostname to 'localhost'
On Sat, 2020-03-28 at 11:16 -0700, Zac Medico wrote:
> On 3/28/20 12:13 AM, Michał Górny wrote:
> > Use UTS namespace to override hostname when network-sandbox is enabled.
> > Set it to 'localhost' as that has a better chance of being present
> > in /etc/hosts. This fixes tests in some packages that try to connect
> > to localhost via hostname obtained using gethostname(), e.g. docker-py,
> > and suffer resolution problems due to the system hostname not being
> > defined in /etc/hosts.
> > ---
> > lib/portage/process.py | 11 ++-
> > 1 file changed, 10 insertions(+), 1 deletion(-)
> >
> > diff --git a/lib/portage/process.py b/lib/portage/process.py
> > index c1fc4bcf6..c48503208 100644
> > --- a/lib/portage/process.py
> > +++ b/lib/portage/process.py
> > @@ -348,12 +348,14 @@ def spawn(mycommand, env=None, opt_name=None,
> > fd_pipes=None, returnpid=False,
> > if unshare_net or unshare_ipc or unshare_mount or unshare_pid:
> > # from /usr/include/bits/sched.h
> > CLONE_NEWNS = 0x0002
> > + CLONE_NEWUTS = 0x0400
> > CLONE_NEWIPC = 0x0800
> > CLONE_NEWPID = 0x2000
> > CLONE_NEWNET = 0x4000
> >
> > if unshare_net:
> > - unshare_flags |= CLONE_NEWNET
> > + # UTS namespace to override hostname
> > + unshare_flags |= CLONE_NEWNET | CLONE_NEWUTS
> > if unshare_ipc:
> > unshare_flags |= CLONE_NEWIPC
> > if unshare_mount:
> > @@ -704,6 +706,13 @@ def _exec(binary, mycommand, opt_name, fd_pipes,
> >
> > noiselevel=-1)
> > os._exit(1)
> > if unshare_net:
> > + # use 'localhost' to
> > avoid hostname resolution problems
> > + try:
> > +
> > socket.sethostname('localhost')
> > + except Exception as e:
> > +
> > writemsg("Unable to set hostname: %s (for FEATURES=\"network-sandbox\")\n"
> > % (
> > + e),
>
> Existing code uses (e,) in cases like this, in order to wrap the
> exception in a tuple, preventing ambiguity in python2 where exceptions
> may behave like tuples. If you don't include the comma, then the
> parenthesis do nothing here, but these days the string formatting
> appears to work correctly with python2.7 either way.
Oops, I meant to leave it there. I'll fix and push.
>
> > +
> > noiselevel=-1)
> >
> > _configure_loopback_interface()
> > except AttributeError:
> > # unshare() not supported by libc
> >
>
> Looks good. Please merge.
--
Best regards,
Michał Górny
signature.asc
Description: This is a digitally signed message part
[gentoo-portage-dev] [PATCH v2] process: Unshare UTS namespace, and set hostname to 'localhost'
Use UTS namespace to override hostname when network-sandbox is enabled.
Set it to 'localhost' as that has a better chance of being present
in /etc/hosts. This fixes tests in some packages that try to connect
to localhost via hostname obtained using gethostname(), e.g. docker-py,
and suffer resolution problems due to the system hostname not being
defined in /etc/hosts.
---
lib/portage/process.py | 11 ++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/lib/portage/process.py b/lib/portage/process.py
index c1fc4bcf6..c48503208 100644
--- a/lib/portage/process.py
+++ b/lib/portage/process.py
@@ -348,12 +348,14 @@ def spawn(mycommand, env=None, opt_name=None,
fd_pipes=None, returnpid=False,
if unshare_net or unshare_ipc or unshare_mount or unshare_pid:
# from /usr/include/bits/sched.h
CLONE_NEWNS = 0x0002
+ CLONE_NEWUTS = 0x0400
CLONE_NEWIPC = 0x0800
CLONE_NEWPID = 0x2000
CLONE_NEWNET = 0x4000
if unshare_net:
- unshare_flags |= CLONE_NEWNET
+ # UTS namespace to override hostname
+ unshare_flags |= CLONE_NEWNET | CLONE_NEWUTS
if unshare_ipc:
unshare_flags |= CLONE_NEWIPC
if unshare_mount:
@@ -704,6 +706,13 @@ def _exec(binary, mycommand, opt_name, fd_pipes,
noiselevel=-1)
os._exit(1)
if unshare_net:
+ # use 'localhost' to
avoid hostname resolution problems
+ try:
+
socket.sethostname('localhost')
+ except Exception as e:
+
writemsg("Unable to set hostname: %s (for FEATURES=\"network-sandbox\")\n" % (
+ e),
+
noiselevel=-1)
_configure_loopback_interface()
except AttributeError:
# unshare() not supported by libc
--
2.26.0
[gentoo-portage-dev] [PATCH] process: Unshare UTS namespace, and set hostname to 'localhost'
Use UTS namespace to override hostname when network-sandbox is enabled.
Set it to 'localhost' as that has a better chance of being present
in /etc/hosts. This fixes tests in some packages that try to connect
to localhost via hostname obtained using gethostname(), e.g. docker-py,
and suffer resolution problems due to the system hostname not being
defined in /etc/hosts.
---
lib/portage/process.py | 6 +-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/lib/portage/process.py b/lib/portage/process.py
index c1fc4bcf6..0f9789acb 100644
--- a/lib/portage/process.py
+++ b/lib/portage/process.py
@@ -348,12 +348,14 @@ def spawn(mycommand, env=None, opt_name=None,
fd_pipes=None, returnpid=False,
if unshare_net or unshare_ipc or unshare_mount or unshare_pid:
# from /usr/include/bits/sched.h
CLONE_NEWNS = 0x0002
+ CLONE_NEWUTS = 0x0400
CLONE_NEWIPC = 0x0800
CLONE_NEWPID = 0x2000
CLONE_NEWNET = 0x4000
if unshare_net:
- unshare_flags |= CLONE_NEWNET
+ # UTS namespace to override hostname
+ unshare_flags |= CLONE_NEWNET | CLONE_NEWUTS
if unshare_ipc:
unshare_flags |= CLONE_NEWIPC
if unshare_mount:
@@ -704,6 +706,8 @@ def _exec(binary, mycommand, opt_name, fd_pipes,
noiselevel=-1)
os._exit(1)
if unshare_net:
+ # use 'localhost' to
avoid hostname resolution problems
+
socket.sethostname('localhost')
_configure_loopback_interface()
except AttributeError:
# unshare() not supported by libc
--
2.26.0
Re: [gentoo-portage-dev] [PATCH] repoman: remove check for addpredict
On Thu, 2020-03-05 at 15:30 -0500, Mike Gilbert wrote: > This function has not been deprecated, and developers generally have a > good reason for using it. A repoman warning for this is just noise. > > No reason was given when this check was added in 2010, and it was requested > by a developer who has since retired. > If you want to remove this class of warnings, I'd suggest removing the one for -j1 as well. Same argument. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
[gentoo-portage-dev] [PATCH] repoman: Restore eclass deprecations lost in rewrite
Signed-off-by: Michał Górny
---
repoman/lib/repoman/modules/linechecks/deprecated/inherit.py | 5 +
1 file changed, 5 insertions(+)
diff --git a/repoman/lib/repoman/modules/linechecks/deprecated/inherit.py
b/repoman/lib/repoman/modules/linechecks/deprecated/inherit.py
index 9cef086da..f307d46fd 100644
--- a/repoman/lib/repoman/modules/linechecks/deprecated/inherit.py
+++ b/repoman/lib/repoman/modules/linechecks/deprecated/inherit.py
@@ -11,6 +11,8 @@ class InheritDeprecated(LineCheck):
# deprecated eclass : new eclass (False if no new eclass)
deprecated_eclasses = {
+ "autotools-multilib": "multilib-minimal",
+ "autotools-utils": False,
"base": False,
"bash-completion": "bash-completion-r1",
"boost-utils": False,
@@ -18,8 +20,10 @@ class InheritDeprecated(LineCheck):
"confutils": False,
"distutils": "distutils-r1",
"epatch": "(eapply since EAPI 6)",
+ "fdo-mime": "xdg-utils",
"games": False,
"gems": "ruby-fakegem",
+ "git-2": "git-r3",
"gpe": False,
"gst-plugins-bad": "gstreamer",
"gst-plugins-base": "gstreamer",
@@ -33,6 +37,7 @@ class InheritDeprecated(LineCheck):
"user": "GLEP 81",
"versionator": "eapi7-ver (built-in since EAPI 7)",
"x-modular": "xorg-2",
+ "xfconf": False,
}
_inherit_re = re.compile(r'^\s*inherit\s(.*)$')
--
2.25.1
[gentoo-portage-dev] [PATCH] Clean up remnants of eapi5-hdepend, HDEPEND and targetroot
Signed-off-by: Michał Górny
---
bin/eapi.sh | 46 +--
bin/ebuild.sh | 23 +++---
bin/phase-functions.sh| 4 +-
doc/package/ebuild.docbook| 1 -
doc/package/ebuild/eapi/5-hdepend.docbook | 32 -
doc/portage.docbook | 1 -
lib/_emerge/Package.py| 6 +--
lib/_emerge/actions.py| 1 -
lib/_emerge/depgraph.py | 12 +
lib/portage/__init__.py | 2 +-
lib/portage/cache/metadata.py | 2 +-
lib/portage/dbapi/bintree.py | 7 ++-
lib/portage/dbapi/porttree.py | 2 +-
lib/portage/dbapi/vartree.py | 2 +-
lib/portage/dep/_slot_operator.py | 3 --
lib/portage/eapi.py | 18 ++--
.../ebuild/_config/special_env_vars.py| 2 +-
lib/portage/package/ebuild/config.py | 13 +-
man/emerge.1 | 10 ++--
repoman/cnf/repository/linechecks.yaml| 1 -
.../modules/linechecks/patches/patches.py | 2 +-
21 files changed, 53 insertions(+), 137 deletions(-)
delete mode 100644 doc/package/ebuild/eapi/5-hdepend.docbook
diff --git a/bin/eapi.sh b/bin/eapi.sh
index 4eafc1c87..29dfb008c 100644
--- a/bin/eapi.sh
+++ b/bin/eapi.sh
@@ -31,19 +31,15 @@ ___eapi_has_prefix_variables() {
}
___eapi_has_BROOT() {
- [[ ! ${1-${EAPI-0}} =~
^(0|1|2|3|4|4-python|4-slot-abi|5|5-hdepend|5-progress|6)$ ]]
+ [[ ! ${1-${EAPI-0}} =~ ^(0|1|2|3|4|4-python|4-slot-abi|5|5-progress|6)$
]]
}
___eapi_has_SYSROOT() {
- [[ ! ${1-${EAPI-0}} =~
^(0|1|2|3|4|4-python|4-slot-abi|5|5-hdepend|5-progress|6)$ ]]
-}
-
-___eapi_has_HDEPEND() {
- [[ ${1-${EAPI-0}} =~ ^(5-hdepend)$ ]]
+ [[ ! ${1-${EAPI-0}} =~ ^(0|1|2|3|4|4-python|4-slot-abi|5|5-progress|6)$
]]
}
___eapi_has_BDEPEND() {
- [[ ! ${1-${EAPI-0}} =~
^(0|1|2|3|4|4-python|4-slot-abi|5|5-hdepend|5-progress|6)$ ]]
+ [[ ! ${1-${EAPI-0}} =~ ^(0|1|2|3|4|4-python|4-slot-abi|5|5-progress|6)$
]]
}
___eapi_has_RDEPEND_DEPEND_fallback() {
@@ -65,11 +61,11 @@ ___eapi_has_dosed() {
}
___eapi_has_einstall() {
- [[ ${1-${EAPI-0}} =~
^(0|1|2|3|4|4-python|4-slot-abi|5|5-hdepend|5-progress)$ ]]
+ [[ ${1-${EAPI-0}} =~ ^(0|1|2|3|4|4-python|4-slot-abi|5|5-progress)$ ]]
}
___eapi_has_dohtml() {
- [[ ${1-${EAPI-0}} =~
^(0|1|2|3|4|4-python|4-slot-abi|5|5-hdepend|5-progress|6)$ ]]
+ [[ ${1-${EAPI-0}} =~ ^(0|1|2|3|4|4-python|4-slot-abi|5|5-progress|6)$ ]]
}
___eapi_has_dohtml_deprecated() {
@@ -77,7 +73,7 @@ ___eapi_has_dohtml_deprecated() {
}
___eapi_has_dolib_libopts() {
- [[ ${1-${EAPI-0}} =~
^(0|1|2|3|4|4-python|4-slot-abi|5|5-hdepend|5-progress|6)$ ]]
+ [[ ${1-${EAPI-0}} =~ ^(0|1|2|3|4|4-python|4-slot-abi|5|5-progress|6)$ ]]
}
___eapi_has_docompress() {
@@ -85,7 +81,7 @@ ___eapi_has_docompress() {
}
___eapi_has_dostrip() {
- [[ ! ${1-${EAPI-0}} =~
^(0|1|2|3|4|4-python|4-slot-abi|5|5-hdepend|5-progress|6)$ ]]
+ [[ ! ${1-${EAPI-0}} =~ ^(0|1|2|3|4|4-python|4-slot-abi|5|5-progress|6)$
]]
}
___eapi_has_nonfatal() {
@@ -101,23 +97,23 @@ ___eapi_has_usex() {
}
___eapi_has_get_libdir() {
- [[ ! ${1-${EAPI-0}} =~
^(0|1|2|3|4|4-python|4-slot-abi|5|5-hdepend|5-progress)$ ]]
+ [[ ! ${1-${EAPI-0}} =~ ^(0|1|2|3|4|4-python|4-slot-abi|5|5-progress)$ ]]
}
___eapi_has_einstalldocs() {
- [[ ! ${1-${EAPI-0}} =~
^(0|1|2|3|4|4-python|4-slot-abi|5|5-hdepend|5-progress)$ ]]
+ [[ ! ${1-${EAPI-0}} =~ ^(0|1|2|3|4|4-python|4-slot-abi|5|5-progress)$ ]]
}
___eapi_has_eapply() {
- [[ ! ${1-${EAPI-0}} =~
^(0|1|2|3|4|4-python|4-slot-abi|5|5-hdepend|5-progress)$ ]]
+ [[ ! ${1-${EAPI-0}} =~ ^(0|1|2|3|4|4-python|4-slot-abi|5|5-progress)$ ]]
}
___eapi_has_eapply_user() {
- [[ ! ${1-${EAPI-0}} =~
^(0|1|2|3|4|4-python|4-slot-abi|5|5-hdepend|5-progress)$ ]]
+ [[ ! ${1-${EAPI-0}} =~ ^(0|1|2|3|4|4-python|4-slot-abi|5|5-progress)$ ]]
}
___eapi_has_in_iuse() {
- [[ ! ${1-${EAPI-0}} =~
^(0|1|2|3|4|4-python|4-slot-abi|5|5-hdepend|5-progress)$ ]]
+ [[ ! ${1-${EAPI-0}} =~ ^(0|1|2|3|4|4-python|4-slot-abi|5|5-progress)$ ]]
}
___eapi_has_version_functions() {
@@ -167,7 +163,7 @@ ___eapi_unpack_supports_xz() {
}
___eapi_unpack_supports_txz() {
- [[ ! ${1-${EAPI-0}} =~
^(0|1|2|3|4|4-python|4-slot-abi|5|5-hdepend|5-progress)$ ]]
+ [[ ! ${1-${EAPI-0}} =~ ^(0|1|2|3|4|4-python|4-slot-abi|5|5-progress)$ ]]
}
___eapi_econf_passes_--disable-dependency-tracking() {
@@ -179,7 +175,7 @@ ___eapi_econf_passes_--disable-silent-rules() {
}
___eapi_econf_passes_--docdir_and_--htmldir() {
- [[ ! ${1-${EAPI-0}} =~
^(0|1|2|3|4|4-python|4-slot-abi|5|5-hdepend|5-progress)$ ]]
+ [[ ! ${1-${EAPI-0
Re: [gentoo-portage-dev] Re: RFC: [QA] notice with 'failed' seds [was PATCH: eapply drop -s option]
On Sun, 2019-12-15 at 13:29 -0800, Zac Medico wrote:
> On 12/13/19 2:12 PM, Michael 'veremitz' Everitt wrote:
> > On 13/12/19 20:36, Michał Górny wrote [excerpted]:
> > > Is this really an argument for or *against* it? Developers are entirely
> > > capable of keeping seds that do nothing for years, as well as patches
> > > that -- while apparently applying correctly -- are entirely meaningless.
> >
> >
> > I think there is some merit in some kind of feedback when sed's are doing
> > nothing, although how feasible it is to generate any useful feedback I
> > can't say. I wouldn't say it needs to explicitly fail or make lots of
> > noise, just an info message that could prompt some further investigation.
> >
>
> It's possible to implement a sed wrapper that detects file arguments for
> -i/--in-place mode, and compares file content before and after the sed call.
>
> There are also ways to make sed exit with an error but that won't be as
> easy to use as a sed wrapper:
>
> https://stackoverflow.com/questions/15965073/return-code-of-sed-for-no-match/15966279
Don't forget that there could be valid cases for sed not changing
a file. Not to mention corner cases where a working replacement results
in no change, e.g.:
# yuck!
sed -i -e "s^-O2^${CFLAGS}^" ...
--
Best regards,
Michał Górny
signature.asc
Description: This is a digitally signed message part
Re: [gentoo-portage-dev] [PATCH v2] eapply: Output verbosely only if patch fails to apply with -F0
On Fri, 2019-12-13 at 23:49 +0100, Ulrich Mueller wrote:
> > > > > > On Fri, 13 Dec 2019, Mike Gilbert wrote:
> > > > It also triggers pointless bug reports. Please remove this.
> > >
> > > I don't like that eqawarn either (see above).
> > >
> > > OTOH, users shouldn't normally have "qa" in PORTAGE_ELOG_CLASSES,
> > > so they won't see the warning?
> > Here's a bug report filed by a user, which is what prompted me to
> > reply on this thread in the first place.
> > https://bugs.gentoo.org/702608
>
> Well then, trivial patch included below.
>
>
> From 81000b32d330a1cc41a4541f7e4264918eb7e6c5 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Ulrich=20M=C3=BCller?=
> Date: Fri, 13 Dec 2019 23:41:23 +0100
> Subject: [PATCH] eapply: Drop QA warning for fuzz factor.
> MIME-Version: 1.0
> Content-Type: text/plain; charset=UTF-8
> Content-Transfer-Encoding: 8bit
>
> This didn't add any information beyond what is already present in the
> output of patch. Developers will know how to interpret its output, and
> users won't see the warning anyway with the standard configuration.
>
> Signed-off-by: Ulrich Müller
> ---
> bin/phase-helpers.sh | 2 --
> 1 file changed, 2 deletions(-)
>
> diff --git a/bin/phase-helpers.sh b/bin/phase-helpers.sh
> index b5691bd70..020862ba0 100644
> --- a/bin/phase-helpers.sh
> +++ b/bin/phase-helpers.sh
> @@ -1004,8 +1004,6 @@ if ___eapi_has_eapply; then
> if ${patch_cmd} "${all_opts[@]}" --dry-run -s -F0 \
> < "${f}" &>/dev/null; then
> all_opts+=( -s -F0 )
> - else
> - eqawarn "${f}: patch failed to apply
> without a fuzz factor, please rebase"
> fi
>
> ${patch_cmd} "${all_opts[@]}" < "${f}"
Actually, I added that because of your comment that people should be
rebasing patches rather than removing context.
--
Best regards,
Michał Górny
signature.asc
Description: This is a digitally signed message part
Re: [gentoo-portage-dev] [PATCH] eapply: Drop -s option for patch.
On Fri, 2019-12-13 at 17:15 -0500, Mike Gilbert wrote: > On Fri, Dec 13, 2019 at 4:42 PM Michał Górny wrote: > > On Fri, 2019-12-13 at 16:37 -0500, Mike Gilbert wrote: > > > On Fri, Dec 13, 2019 at 3:36 PM Michał Górny wrote: > > > > Just like 'many of the proposals lately', developers are going to be > > > > the ones disabling it (because they don't care), and users will be the > > > > ones enabling it (because they do care), just to learn that developers > > > > don't care and go complaining to the mailing lists that users dare > > > > report issues they don't care about. > > > > > > I care if the patch is actually broken, which the warning doesn't > > > really tell me. It's just not a very reliable indicator, and will > > > produce false-positives frequently. > > > > > > > You can also take less context into the patch and use -F0. Then you'll > > have the same effect, no warnings to bother you and no pretending that > > the patch applies when it doesn't. > > That really doesn't help me. My point is that I don't want to touch > the patch unless it is actually necessary to do so. > Then make patches with -U0. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-portage-dev] [PATCH] eapply: Drop -s option for patch.
On Fri, 2019-12-13 at 21:56 +, Michael 'veremitz' Everitt wrote: > On 13/12/19 21:42, Michał Górny wrote: > > On Fri, 2019-12-13 at 16:37 -0500, Mike Gilbert wrote: > > > On Fri, Dec 13, 2019 at 3:36 PM Michał Górny wrote: > > > > Just like 'many of the proposals lately', developers are going to be > > > > the ones disabling it (because they don't care), and users will be the > > > > ones enabling it (because they do care), just to learn that developers > > > > don't care and go complaining to the mailing lists that users dare > > > > report issues they don't care about. > > > I care if the patch is actually broken, which the warning doesn't > > > really tell me. It's just not a very reliable indicator, and will > > > produce false-positives frequently. > > > > > You can also take less context into the patch and use -F0. Then you'll > > have the same effect, no warnings to bother you and no pretending that > > the patch applies when it doesn't. > > > Is there any mileage in having a similar scheme to which we already apply > '-p' increments to the -F variable? > eg. > 1) attempt patch with -F0 > 2) if (1) fails, attempt with -F2/3 & display 'yellow' warning & QA notice That is precisely what my patch does and what people are complaining about. > 3) if (2) fails, attempt with, say, -F10 & display big fat 'red' warning > and QA notice That makes no sense as it exceeds context provided in most patches. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-portage-dev] [PATCH] eapply: Drop -s option for patch.
On Fri, 2019-12-13 at 16:37 -0500, Mike Gilbert wrote: > On Fri, Dec 13, 2019 at 3:36 PM Michał Górny wrote: > > Just like 'many of the proposals lately', developers are going to be > > the ones disabling it (because they don't care), and users will be the > > ones enabling it (because they do care), just to learn that developers > > don't care and go complaining to the mailing lists that users dare > > report issues they don't care about. > > I care if the patch is actually broken, which the warning doesn't > really tell me. It's just not a very reliable indicator, and will > produce false-positives frequently. > You can also take less context into the patch and use -F0. Then you'll have the same effect, no warnings to bother you and no pretending that the patch applies when it doesn't. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-portage-dev] [PATCH] eapply: Drop -s option for patch.
On Fri, 2019-12-13 at 21:31 +0100, Fabian Groffen wrote: > On 13-12-2019 14:24:33 -0500, Michael Orlitzky wrote: > > On 12/13/19 9:28 AM, Fabian Groffen wrote: > > > We are providing those patches, maybe. In reality very often the > > > patches originate from somewhere else though. And you don't want to > > > have to respin all of those just because. At least that's what I feel. > > > > > > > Just because... the context changed? A new "!" in a line of context can > > be the difference between letting someone log in with the right password > > and letting them log in with the wrong password. You should at least > > have to stop and verify that the patch does what you think it does when > > it "gains" fuzz. And at that point, git diff will give you a clean > > version of it. > > Counter argument is that we've been doing this for decades, and always > relied on maintainers to check the contents of their patches, without > problems. We didn't introduce a predictable random number generator or > something. Is this really an argument for or *against* it? Developers are entirely capable of keeping seds that do nothing for years, as well as patches that -- while apparently applying correctly -- are entirely meaningless. Should I remind you that epatch was entirely capable of creating meaningless files by randomly applying the wrong patch level? > Your very specific example just illustrates the niche this proposal is > targetting. > > As with many of the proposals lately, they just seem to aim at more work > for individual maintainers, with a very low gain ratio. > Better, allow this to be a FEATURE, or whatever, that devs should enable, > but don't spit this in the user's face by default. > Just like 'many of the proposals lately', developers are going to be the ones disabling it (because they don't care), and users will be the ones enabling it (because they do care), just to learn that developers don't care and go complaining to the mailing lists that users dare report issues they don't care about. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-portage-dev] [PATCH] eapply: Drop -s option for patch.
On Fri, 2019-12-13 at 09:06 -0500, Mike Gilbert wrote: > On Fri, Dec 13, 2019 at 8:52 AM Michał Górny wrote: > > On Fri, 2019-12-13 at 08:47 -0500, Mike Gilbert wrote: > > > On Thu, Dec 12, 2019 at 3:15 PM Ulrich Mueller wrote: > > > > > > > > > On Thu, 12 Dec 2019, Mike Gilbert wrote: > > > > > I think this should be reverted. It causes too much noise, and > > > > > "solves" a problem only very rarely. > > > > > > > > Now, how many lines of output does this typically produce, compared > > > > to the total size of a typical build log? Especially with mgorny's > > > > subsequent modification, which suppresses the output unless the patch > > > > doesn't apply cleanly. > > > > > > In most cases, I would be inclined to simply ignore the patch output > > > since there's really no need for me to take any action on it. > > > > > > On the other hand, it makes it more difficult to quickly identify the > > > list of patches being applied if there is junk output in the middle of > > > the list. > > > > > > > It was also suggested that we add -F0 in EAPI 8, but that would break > > > > the build in those cases that are producing extra output now. I don't > > > > think that would be preferable. > > > > > > I am opposed to including such a change in EAPI 8. It would make > > > ebuild maintenance more difficult for everyone, and I don't think the > > > potential benefit is worth it. > > > > ...and why do we consider it correct to apply patches when the context > > doesn't match? If our only goal is to make things 'easier' for > > 'everyone', then we could just pass -F and ignore all the context. > > > > Though I don't understand why include any context in the first place if > > you don't care about it matching. Sounds like a waste of space to me! > > The patch command defaults to -F2. If that makes no sense, why is it > the upstream default? > You should ask upstream, not me. But if I were to guess, the answer would be because patch(1) is used by random people trying to apply random patches they've found somewhere. We on the other hand are applying patches that *we* are supposed to provide. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-portage-dev] [PATCH] eapply: Drop -s option for patch.
On Fri, 2019-12-13 at 08:47 -0500, Mike Gilbert wrote: > On Thu, Dec 12, 2019 at 3:15 PM Ulrich Mueller wrote: > > > > > > > On Thu, 12 Dec 2019, Mike Gilbert wrote: > > > I think this should be reverted. It causes too much noise, and > > > "solves" a problem only very rarely. > > > > Now, how many lines of output does this typically produce, compared > > to the total size of a typical build log? Especially with mgorny's > > subsequent modification, which suppresses the output unless the patch > > doesn't apply cleanly. > > In most cases, I would be inclined to simply ignore the patch output > since there's really no need for me to take any action on it. > > On the other hand, it makes it more difficult to quickly identify the > list of patches being applied if there is junk output in the middle of > the list. > > > It was also suggested that we add -F0 in EAPI 8, but that would break > > the build in those cases that are producing extra output now. I don't > > think that would be preferable. > > I am opposed to including such a change in EAPI 8. It would make > ebuild maintenance more difficult for everyone, and I don't think the > potential benefit is worth it. ...and why do we consider it correct to apply patches when the context doesn't match? If our only goal is to make things 'easier' for 'everyone', then we could just pass -F and ignore all the context. Though I don't understand why include any context in the first place if you don't care about it matching. Sounds like a waste of space to me! -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-portage-dev] [PATCH v2] eapply: Output verbosely only if patch fails to apply with -F0
On Wed, 2019-11-27 at 23:49 +0100, Ulrich Mueller wrote: > > > > > > On Wed, 27 Nov 2019, Michael Orlitzky wrote: > > This now disagrees with the PMS algorithm, doesn't it? > > The difference is that there is now a QA warning for something that is > perfectly within the spec. Maybe the extra verboseness would be enough, > but the eqawarn line should be omitted? It doesn't provide any info that > isn't already present in the output of patch itself. > It helps people understand why some patches throw a wall of text while others don't. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
[gentoo-portage-dev] [PATCH v2] eapply: Output verbosely only if patch fails to apply with -F0
12d0c48ad disabled silent output for eapply, in order to obtain fuzz
factors in build logs. However, this also causes eapply to report all
patched files which can make logs unreadable when there are no fuzz
factors to be reported. Instead, use verbose output only when applying
the patch with -F0 fails.
To achieve that, attempt to apply each patch with -F0 --dry-run first.
If this succeeds, just silently apply the patch for real. If it
doesn't, output an explicit eqawarn that the patch does not apply
cleanly and retry with the default fuzz factor and verbose output.
Non-silenced output applies both to successful application with fuzz
and to failure.
Signed-off-by: Michał Górny
---
bin/phase-helpers.sh | 16 ++--
1 file changed, 14 insertions(+), 2 deletions(-)
Changes in v2:
- added original path to output
diff --git a/bin/phase-helpers.sh b/bin/phase-helpers.sh
index 60f8d3243..b5691bd70 100644
--- a/bin/phase-helpers.sh
+++ b/bin/phase-helpers.sh
@@ -995,8 +995,20 @@ if ___eapi_has_eapply; then
# -f to avoid interactivity
# -g0 to guarantee no VCS interaction
# --no-backup-if-mismatch not to pollute the sources
- ${patch_cmd} -p1 -f -g0 --no-backup-if-mismatch \
- "${patch_options[@]}" < "${f}"
+ local all_opts=(
+ -p1 -f -g0 --no-backup-if-mismatch
+ "${patch_options[@]}"
+ )
+ # try applying with -F0 first, output a verbose warning
+ # if fuzz factor is necessary
+ if ${patch_cmd} "${all_opts[@]}" --dry-run -s -F0 \
+ < "${f}" &>/dev/null; then
+ all_opts+=( -s -F0 )
+ else
+ eqawarn "${f}: patch failed to apply
without a fuzz factor, please rebase"
+ fi
+
+ ${patch_cmd} "${all_opts[@]}" < "${f}"
failed=${?}
if ! eend "${failed}"; then
__helpers_die "patch -p1 ${patch_options[*]}
failed with ${f}"
--
2.24.0
Re: [gentoo-portage-dev] [PATCH] eapply: Output verbosely only if patch fails to apply with -F0
On Wed, 2019-11-27 at 11:13 -0800, Zac Medico wrote:
> On 11/27/19 10:54 AM, Michał Górny wrote:
> > 12d0c48ad disabled silent output for eapply, in order to obtain fuzz
> > factors in build logs. However, this also causes eapply to report all
> > patched files which can make logs unreadable when there are no fuzz
> > factors to be reported. Instead, use verbose output only when applying
> > the patch with -F0 fails.
> >
> > To achieve that, attempt to apply each patch with -F0 --dry-run first.
> > If this succeeds, just silently apply the patch for real. If it
> > doesn't, output an explicit eqawarn that the patch does not apply
> > cleanly and retry with the default fuzz factor and verbose output.
> > Non-silenced output applies both to successful application with fuzz
> > and to failure.
> >
> > Signed-off-by: Michał Górny
> > ---
> > bin/phase-helpers.sh | 16 ++--
> > 1 file changed, 14 insertions(+), 2 deletions(-)
> >
> > diff --git a/bin/phase-helpers.sh b/bin/phase-helpers.sh
> > index 60f8d3243..6f9b52678 100644
> > --- a/bin/phase-helpers.sh
> > +++ b/bin/phase-helpers.sh
> > @@ -995,8 +995,20 @@ if ___eapi_has_eapply; then
> > # -f to avoid interactivity
> > # -g0 to guarantee no VCS interaction
> > # --no-backup-if-mismatch not to pollute the sources
> > - ${patch_cmd} -p1 -f -g0 --no-backup-if-mismatch \
> > - "${patch_options[@]}" < "${f}"
> > + local all_opts=(
> > + -p1 -f -g0 --no-backup-if-mismatch
> > + "${patch_options[@]}"
> > + )
> > + # try applying with -F0 first, output a verbose warning
> > + # if fuzz factor is necessary
> > + if ${patch_cmd} "${all_opts[@]}" --dry-run -s -F0 \
> > + < "${f}" &>/dev/null; then
> > + all_opts+=( -s -F0 )
> > + else
> > + eqawarn "patch failed to apply without a
> > fuzz factor, please rebase"
>
> The earlier ebegin call only outputs the basename, so there could be
> some confusion about the origin of the patch. This function is also
> called by eapply_user, it's possible that people could get confused and
> report bugs about fuzz in their local patches. Other than that it looks
> good.
>
Ok, I will add a full path.
--
Best regards,
Michał Górny
signature.asc
Description: This is a digitally signed message part
[gentoo-portage-dev] [PATCH] eapply: Output verbosely only if patch fails to apply with -F0
12d0c48ad disabled silent output for eapply, in order to obtain fuzz
factors in build logs. However, this also causes eapply to report all
patched files which can make logs unreadable when there are no fuzz
factors to be reported. Instead, use verbose output only when applying
the patch with -F0 fails.
To achieve that, attempt to apply each patch with -F0 --dry-run first.
If this succeeds, just silently apply the patch for real. If it
doesn't, output an explicit eqawarn that the patch does not apply
cleanly and retry with the default fuzz factor and verbose output.
Non-silenced output applies both to successful application with fuzz
and to failure.
Signed-off-by: Michał Górny
---
bin/phase-helpers.sh | 16 ++--
1 file changed, 14 insertions(+), 2 deletions(-)
diff --git a/bin/phase-helpers.sh b/bin/phase-helpers.sh
index 60f8d3243..6f9b52678 100644
--- a/bin/phase-helpers.sh
+++ b/bin/phase-helpers.sh
@@ -995,8 +995,20 @@ if ___eapi_has_eapply; then
# -f to avoid interactivity
# -g0 to guarantee no VCS interaction
# --no-backup-if-mismatch not to pollute the sources
- ${patch_cmd} -p1 -f -g0 --no-backup-if-mismatch \
- "${patch_options[@]}" < "${f}"
+ local all_opts=(
+ -p1 -f -g0 --no-backup-if-mismatch
+ "${patch_options[@]}"
+ )
+ # try applying with -F0 first, output a verbose warning
+ # if fuzz factor is necessary
+ if ${patch_cmd} "${all_opts[@]}" --dry-run -s -F0 \
+ < "${f}" &>/dev/null; then
+ all_opts+=( -s -F0 )
+ else
+ eqawarn "patch failed to apply without a
fuzz factor, please rebase"
+ fi
+
+ ${patch_cmd} "${all_opts[@]}" < "${f}"
failed=${?}
if ! eend "${failed}"; then
__helpers_die "patch -p1 ${patch_options[*]}
failed with ${f}"
--
2.24.0
Re: [gentoo-portage-dev] SLOT move of sys-libs/ncurses-5.9-r5(0/5) to sys-libs/ncurses-compat-5.9-r5(5/5) ?
On Thu, 2019-11-14 at 13:01 +, Joakim Tjernlund wrote: > Is there way to move the above with current portage ? > No. Slot moves work only within a package. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-portage-dev] [r...@gentoo.org: Cron /usr/local/bin/pidlock -s sync-distfiles /usr/bin/timeout -k 2h 1h /usr/local/bin/mastermirror/sync-distfiles.sh]
On Tue, 2019-11-05 at 18:13 +, Robin H. Johnson wrote: > Can Portage handle this error more gracefully please? > Is it still happening though? I thought the Portage upgrade fixed it. > The symlink test file is there to verify mirror behavior, so we don't > want to delete it either. I think it's fine to delete it now. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-portage-dev] [PATCH] install-qa-check.d: remove check that bans libtool files and static libs from /
On Mon, 2019-11-04 at 04:15 -0600, William Hubbs wrote: > I also don't like your tone in your response to Zac merging the patch. > > https://archives.gentoo.org/gentoo-portage-dev/message/1abfd0499e514b7d6b70b709e9e3ae18 > > If I say out here that since I'm a council member I'm above you and zac > should listen to me and apply the patch is that appropriate? I imagine > not, so I feel the same way about you bringing your qa membership into > the discussion. > In my opinion, all that kind of thing leads to is people becoming angry. > > I'm going to ask you to close https://bugs.gentoo.org/699254. I honestly > do not feel that this is an appropriate way to deal with this situation. > Excuse me but are you serious? So first you choose to claim that something is not policy because you don't see it stamped. Then you demand that QA doesn't vote on stamping it because... why exactly? Because it's 'not an appropriate way', apparently. So what's the appropriate way? -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-portage-dev] [PATCH] install-qa-check.d: remove check that bans libtool files and static libs from /
On Sun, 2019-11-03 at 15:26 -0600, William Hubbs wrote:
> On Sun, Nov 03, 2019 at 12:18:40PM -0800, Zac Medico wrote:
> > On 11/3/19 11:53 AM, Michał Górny wrote:
> > > On Sun, 2019-11-03 at 11:49 -0800, Zac Medico wrote:
> > > > On 10/27/19 10:40 AM, William Hubbs wrote:
> > > > > Most upstreams and build systems do not make this distinction, so this
> > > > > causes unnecessary hacks in ebuilds.
> > > > >
> > > > > Signed-off-by: William Hubbs
> > > > > ---
> > > > > bin/install-qa-check.d/80libraries | 10 --
> > > > > 1 file changed, 10 deletions(-)
> > > > >
> > > > > diff --git a/bin/install-qa-check.d/80libraries
> > > > > b/bin/install-qa-check.d/80libraries
> > > > > index d1d2c4fdd..e59369bf6 100644
> > > > > --- a/bin/install-qa-check.d/80libraries
> > > > > +++ b/bin/install-qa-check.d/80libraries
> > > > > @@ -152,16 +152,6 @@ lib_check() {
> > > > > done
> > > > > [[ ${abort} == "yes" ]] && die "add those ldscripts"
> > > > >
> > > > > - # Make sure people don't store libtool files or static libs in
> > > > > /lib
> > > > > - f=$(ls "${ED%/}"/lib*/*.{a,la} 2>/dev/null)
> > > > > - if [[ -n ${f} ]] ; then
> > > > > - __vecho -ne '\n'
> > > > > - eqawarn "QA Notice: Excessive files found in the /
> > > > > partition"
> > > > > - eqawarn "${f}"
> > > > > - __vecho -ne '\n'
> > > > > - die "static archives (*.a) and libtool library files
> > > > > (*.la) belong in /usr/lib*, not /lib*"
> > > > > - fi
> > > > > -
> > > > > # Verify that the libtool files don't contain bogus $D entries.
> > > > > local abort=no gentoo_bug=no always_overflow=no
> > > > > for a in "${ED%/}"/usr/lib*/*.la ; do
> > > > >
> > > >
> > > > Merged. Thanks!
> > > >
> > > > https://gitweb.gentoo.org/proj/portage.git/commit/?id=498900e5e51460502d8271f409a4c614a021613b
> > > >
> > >
> > > Please revert this. I should point out that this has been vetoed
> > > by a QA member, and is currently subject to QA vote. Therefore, I
> > > believe you shouldn't be making rash decisions based on patches
> > > submitted by a single developer. Especially given that so far nobody
> > > else has voiced his opinion either way, so it's 1:1.
>
> You being a qa member doesn't have a lot to do with this mgorny. you
> know there was no official policy when I posted this, and as far as I
> know there is not one now.
>
That is a really poor argument. Something that's respected for 10+
years and reported as QA violation is a standing policy as far as I'm
concerned. Just because it isn't backed by a formally stamped policy
(at least as far as we know -- maybe it was actually stamped somewhere
in the past?) doesn't mean you it's fine for one person to change it ad-
hoc because it stands in his way.
I should point that I'm very concerned that you're pushing this forward
even though:
1) I've objected to the change itself,
2) I've pointed out that it's been sent to the wrong mailing list,
and that this explicitly prevents a number of developers from even
knowing that this is happening,
3) removing it provides a way for regressions that can have major impact
on users and that involve much effort in reverting that.
So if I send a revert patch afterwards, and you object, should the patch
be accepted because only one person objected?
--
Best regards,
Michał Górny
signature.asc
Description: This is a digitally signed message part
Re: [gentoo-portage-dev] [PATCH] install-qa-check.d: remove check that bans libtool files and static libs from /
On Sun, 2019-11-03 at 11:49 -0800, Zac Medico wrote:
> On 10/27/19 10:40 AM, William Hubbs wrote:
> > Most upstreams and build systems do not make this distinction, so this
> > causes unnecessary hacks in ebuilds.
> >
> > Signed-off-by: William Hubbs
> > ---
> > bin/install-qa-check.d/80libraries | 10 --
> > 1 file changed, 10 deletions(-)
> >
> > diff --git a/bin/install-qa-check.d/80libraries
> > b/bin/install-qa-check.d/80libraries
> > index d1d2c4fdd..e59369bf6 100644
> > --- a/bin/install-qa-check.d/80libraries
> > +++ b/bin/install-qa-check.d/80libraries
> > @@ -152,16 +152,6 @@ lib_check() {
> > done
> > [[ ${abort} == "yes" ]] && die "add those ldscripts"
> >
> > - # Make sure people don't store libtool files or static libs in /lib
> > - f=$(ls "${ED%/}"/lib*/*.{a,la} 2>/dev/null)
> > - if [[ -n ${f} ]] ; then
> > - __vecho -ne '\n'
> > - eqawarn "QA Notice: Excessive files found in the / partition"
> > - eqawarn "${f}"
> > - __vecho -ne '\n'
> > - die "static archives (*.a) and libtool library files (*.la)
> > belong in /usr/lib*, not /lib*"
> > - fi
> > -
> > # Verify that the libtool files don't contain bogus $D entries.
> > local abort=no gentoo_bug=no always_overflow=no
> > for a in "${ED%/}"/usr/lib*/*.la ; do
> >
>
> Merged. Thanks!
>
> https://gitweb.gentoo.org/proj/portage.git/commit/?id=498900e5e51460502d8271f409a4c614a021613b
>
Please revert this. I should point out that this has been vetoed
by a QA member, and is currently subject to QA vote. Therefore, I
believe you shouldn't be making rash decisions based on patches
submitted by a single developer. Especially given that so far nobody
else has voiced his opinion either way, so it's 1:1.
--
Best regards,
Michał Górny
signature.asc
Description: This is a digitally signed message part
Re: [gentoo-portage-dev] [PATCH] install-qa-check.d: remove check that bans libtool files and static libs from /
On Sun, 2019-10-27 at 13:49 -0500, William Hubbs wrote: > On Sun, Oct 27, 2019 at 06:58:00PM +0100, Michał Górny wrote: > > On Sun, 2019-10-27 at 12:40 -0500, William Hubbs wrote: > > > Most upstreams and build systems do not make this distinction, so this > > > causes unnecessary hacks in ebuilds. > > > > > > > The hacks aren't 'unnecessary'. There is a very good reason that files > > that are used *purely at build time* don't land in /. That reason is > > disk space. Even if people nowadays are forced to use initramfs with > > separate /usr, it doesn't mean you should just let their rootfs fill up > > with useless files. > > The useless files argument really holds no water with me. We install > many files on / that are useless in one situation or another. > Some examples are logrotate files when logrotate isn't installed, Do we really install logrotate files to /? > systemd units for openrc systems and openrc init scripts for systemd > systems. Those files are at least needed in some scenarios. You're talking about files that are never used without /usr mounted. And that can't be used because they depend on headers in /usr/include. > > Talk to me about useless files on / after we put all of these, and > possibly others I can't think of, behind use flags. > > > Do you have any *real* argument? Because 'unnecessary hack' is > > basically your feeling of ebuild aesthetics. My aesthetics is more > > worried about useless clutter in /lib*. FHS agrees with me, as you > > yourself admitted yesterday. > > Any downstream hack means that we are being lazy and not reporting the > bug upstream and asking them to fix it. > > This particular issue is not a big deal to any other distro and has > never been. Shouldn't we try to get upstreams to do this if it is so > important? > Because 'any other distro' usually means a binary distro that moves files around after installing. And splits static libraries into separate packages. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-portage-dev] [PATCH] install-qa-check.d: remove check that bans libtool files and static libs from /
On Sun, 2019-10-27 at 12:40 -0500, William Hubbs wrote: > Most upstreams and build systems do not make this distinction, so this > causes unnecessary hacks in ebuilds. > The hacks aren't 'unnecessary'. There is a very good reason that files that are used *purely at build time* don't land in /. That reason is disk space. Even if people nowadays are forced to use initramfs with separate /usr, it doesn't mean you should just let their rootfs fill up with useless files. Do you have any *real* argument? Because 'unnecessary hack' is basically your feeling of ebuild aesthetics. My aesthetics is more worried about useless clutter in /lib*. FHS agrees with me, as you yourself admitted yesterday. So why do you believe we should introduce this regression? And why are you trying to sneak it past most of the developers via gentoo-portage- dev instead of gentoo-dev? -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-portage-dev] [PATCH] make.globals: Change FETCHCOMMAND_RSYNC to --copy-links
On Mon, 2019-10-21 at 17:25 +0200, Ulrich Mueller wrote: > > > > > > On Mon, 21 Oct 2019, Michał Górny wrote: > > This also removes most of the other options that are irrelevant or even > > undesirable to distfile fetching, that is: > > - '-r' since we always fetch a single file, so recursive operation is > > unnecessary > > - '-p', '-o', '-g' since we want to apply our permissions and ownership > > for distfiles rather than copying the one from mirrors, > > - '-D' since we do not expect any devices or specials in distfiles. > > That certainly makes sense, but I don't see it in the actual patch? Those were the options implied by '-a'. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
