Re: [gentoo-portage-dev] [PATCH gentoolkit 1/2] eclean: Rewrite findPackages()
On 21/02/20 05:29, Matt Turner wrote: > I found the original code to be nearly incomprehensible. Instead of > populating a dict of potential binpkgs to remove and then removing from > the to-be-removed list, just selectively add to-be-removed packages. > > Signed-off-by: Matt Turner > --- > I switched from tabs to spaces in the process. I can revert back if > desired. > Probably best to stick to tabs for consistency with the other portage code, although naturally Zac probably better to ACK/NACK that. Otherwise I think this is a good refresh. +1. signature.asc Description: OpenPGP digital signature
[gentoo-portage-dev] Re: [PATCH gentoolkit 1/2] eclean: Rewrite findPackages()
On Thu, Feb 20, 2020 at 9:29 PM Matt Turner wrote:
>
> I found the original code to be nearly incomprehensible. Instead of
> populating a dict of potential binpkgs to remove and then removing from
> the to-be-removed list, just selectively add to-be-removed packages.
>
> Signed-off-by: Matt Turner
> ---
> I switched from tabs to spaces in the process. I can revert back if
> desired.
>
> pym/gentoolkit/eclean/search.py | 189
> 1 file changed, 94 insertions(+), 95 deletions(-)
>
> diff --git a/pym/gentoolkit/eclean/search.py b/pym/gentoolkit/eclean/search.py
> index 58bd97e..831ba39 100644
> --- a/pym/gentoolkit/eclean/search.py
> +++ b/pym/gentoolkit/eclean/search.py
> @@ -489,98 +489,97 @@ class DistfilesSearch(object):
>
>
> def findPackages(
> - options,
> - exclude=None,
> - destructive=False,
> - time_limit=0,
> - package_names=False,
> - pkgdir=None,
> - port_dbapi=portage.db[portage.root]["porttree"].dbapi,
> - var_dbapi=portage.db[portage.root]["vartree"].dbapi
> - ):
> - """Find all obsolete binary packages.
> -
> - XXX: packages are found only by symlinks.
> - Maybe i should also return .tbz2 files from All/ that have
> - no corresponding symlinks.
> -
> - @param options: dict of options determined at runtime
> - @param exclude: an exclusion dict as defined in
> - exclude.parseExcludeFile class.
> - @param destructive: boolean, defaults to False
> - @param time_limit: integer time value as returned by parseTime()
> - @param package_names: boolean, defaults to False.
> - used only if destructive=True
> - @param pkgdir: path to the binary package dir being checked
> - @param port_dbapi: defaults to
> portage.db[portage.root]["porttree"].dbapi
> - can be overridden for tests.
> - @param var_dbapi: defaults to
> portage.db[portage.root]["vartree"].dbapi
> - can be overridden for tests.
> -
> - @rtype: dict
> - @return clean_me i.e. {'cat/pkg-ver.tbz2': [filepath],}
> - """
> - if exclude is None:
> - exclude = {}
> - clean_me = {}
> - # create a full package dictionary
> -
> - # now do an access test, os.walk does not error for "no read
> permission"
> - try:
> - test = os.listdir(pkgdir)
> - del test
> - except EnvironmentError as er:
> - if options['ignore-failure']:
> - exit(0)
> - print( pp.error("Error accessing PKGDIR." ), file=sys.stderr)
> - print( pp.error("(Check your make.conf file and
> environment)."), file=sys.stderr)
> - print( pp.error("Error: %s" %str(er)), file=sys.stderr)
> - exit(1)
> -
> - # if portage supports FEATURES=binpkg-multi-instance, then
> - # cpv_all can return multiple instances per cpv, where
> - # instances are distinguishable by some extra attributes
> - # provided by portage's _pkg_str class
> - bin_dbapi = portage.binarytree(pkgdir=pkgdir,
> settings=var_dbapi.settings).dbapi
> - for cpv in bin_dbapi.cpv_all():
> - mtime = int(bin_dbapi.aux_get(cpv, ['_mtime_'])[0])
> - if time_limit and mtime >= time_limit:
> - # time-limit exclusion
> - continue
> - # dict is cpv->[pkgs] (supports binpkg-multi-instance)
> - clean_me.setdefault(cpv, []).append(cpv)
> -
> - # keep only obsolete ones
> - if destructive and package_names:
> - cp_all = dict.fromkeys(var_dbapi.cp_all())
> - else:
> - cp_all = {}
> - for cpv in list(clean_me):
> - if exclDictMatchCP(exclude,portage.cpv_getkey(cpv)):
> - # exclusion because of the exclude file
> - del clean_me[cpv]
> - continue
> - if not destructive and port_dbapi.cpv_exists(cpv):
> - # exclusion because pkg still exists (in porttree)
> - del clean_me[cpv]
> - continue
> - if destructive and var_dbapi.cpv_exists(cpv):
> - buildtime = var_dbapi.aux_get(cpv, ['BUILD_TIME'])[0]
> - clean_me[cpv] = [pkg for pkg in clean_me[cpv]
> - # only keep path if BUILD_TIME is identical
> with vartree
> - if bin_dbapi.aux_get(pkg, ['BUILD_TIME'])[0]
> != buildtime]
> - if not clean_me[cpv]:
> - # nothing we can clean for this package
> - del clean_me[cpv]
> -
[gentoo-portage-dev] [PATCH gentoolkit 2/2] eclean: Add option to delete binpkgs with changed deps
Signed-off-by: Matt Turner
---
pym/gentoolkit/eclean/cli.py| 7 ++-
pym/gentoolkit/eclean/search.py | 30 +-
2 files changed, 35 insertions(+), 2 deletions(-)
diff --git a/pym/gentoolkit/eclean/cli.py b/pym/gentoolkit/eclean/cli.py
index 1a99b3e..39aafd3 100644
--- a/pym/gentoolkit/eclean/cli.py
+++ b/pym/gentoolkit/eclean/cli.py
@@ -147,6 +147,8 @@ def printUsage(_error=None, help=None):
or help in ('all','packages'):
print( "Available", yellow("options"),"for the",
green("packages"),"action:", file=out)
+ print( yellow(" --changed-deps")+
+ " - delete packages for which ebuild
dependencies have changed", file=out)
print( yellow(" -i, --ignore-failure")+
" - ignore failure to locate PKGDIR",
file=out)
print( file=out)
@@ -263,6 +265,8 @@ def parseArgs(options={}):
options['size-limit'] = parseSize(a)
elif o in ("-v", "--verbose") and not options['quiet']:
options['verbose'] = True
+ elif o in ("--changed-deps"):
+ options['changed-deps'] = True
elif o in ("-i", "--ignore-failure"):
options['ignore-failure'] = True
else:
@@ -290,7 +294,7 @@ def parseArgs(options={}):
getopt_options['short']['distfiles'] = "fs:"
getopt_options['long']['distfiles'] = ["fetch-restricted",
"size-limit="]
getopt_options['short']['packages'] = "i"
- getopt_options['long']['packages'] = ["ignore-failure"]
+ getopt_options['long']['packages'] = ["ignore-failure", "changed-deps"]
# set default options, except 'nocolor', which is set in main()
options['interactive'] = False
options['pretend'] = False
@@ -303,6 +307,7 @@ def parseArgs(options={}):
options['fetch-restricted'] = False
options['size-limit'] = 0
options['verbose'] = False
+ options['changed-deps'] = False
options['ignore-failure'] = False
# if called by a well-named symlink, set the action accordingly:
action = None
diff --git a/pym/gentoolkit/eclean/search.py b/pym/gentoolkit/eclean/search.py
index 831ba39..da8c286 100644
--- a/pym/gentoolkit/eclean/search.py
+++ b/pym/gentoolkit/eclean/search.py
@@ -13,6 +13,8 @@ import sys
from functools import partial
import portage
+from portage.dep import Atom, use_reduce
+from portage.dep._slot_operator import strip_slots
import gentoolkit.pprinter as pp
from gentoolkit.eclean.exclude import (exclDictMatchCP, exclDictExpand,
@@ -488,6 +490,17 @@ class DistfilesSearch(object):
return clean_me, saved
+def _deps_equal(deps_a, deps_b, eapi, uselist=None):
+"""Compare two dependency lists given a set of USE flags"""
+if deps_a == deps_b: return True
+
+deps_a = use_reduce(deps_a, uselist=uselist, eapi=eapi, token_class=Atom)
+deps_b = use_reduce(deps_b, uselist=uselist, eapi=eapi, token_class=Atom)
+strip_slots(deps_a)
+strip_slots(deps_b)
+return deps_a == deps_b
+
+
def findPackages(
options,
exclude=None,
@@ -564,7 +577,22 @@ def findPackages(
# Exclude if binpkg exists in the porttree and not --deep
if not destructive and port_dbapi.cpv_exists(cpv):
-continue
+if not options['changed-deps']:
+continue
+
+uselist = bin_dbapi.aux_get(cpv, ['USE'])[0].split()
+all_equal = True
+
+for k in ('RDEPEND', 'PDEPEND'):
+binpkg_deps = bin_dbapi.aux_get(cpv, [k])
+ebuild_deps = port_dbapi.aux_get(cpv, [k])
+
+if not _deps_equal(binpkg_deps, ebuild_deps, cpv.eapi,
uselist):
+all_equal = False
+break
+
+if all_equal:
+continue
if destructive and var_dbapi.cpv_exists(cpv):
# Exclude if an instance of the package is installed due to
--
2.24.1
[gentoo-portage-dev] [PATCH gentoolkit 1/2] eclean: Rewrite findPackages()
I found the original code to be nearly incomprehensible. Instead of
populating a dict of potential binpkgs to remove and then removing from
the to-be-removed list, just selectively add to-be-removed packages.
Signed-off-by: Matt Turner
---
I switched from tabs to spaces in the process. I can revert back if
desired.
pym/gentoolkit/eclean/search.py | 189
1 file changed, 94 insertions(+), 95 deletions(-)
diff --git a/pym/gentoolkit/eclean/search.py b/pym/gentoolkit/eclean/search.py
index 58bd97e..831ba39 100644
--- a/pym/gentoolkit/eclean/search.py
+++ b/pym/gentoolkit/eclean/search.py
@@ -489,98 +489,97 @@ class DistfilesSearch(object):
def findPackages(
- options,
- exclude=None,
- destructive=False,
- time_limit=0,
- package_names=False,
- pkgdir=None,
- port_dbapi=portage.db[portage.root]["porttree"].dbapi,
- var_dbapi=portage.db[portage.root]["vartree"].dbapi
- ):
- """Find all obsolete binary packages.
-
- XXX: packages are found only by symlinks.
- Maybe i should also return .tbz2 files from All/ that have
- no corresponding symlinks.
-
- @param options: dict of options determined at runtime
- @param exclude: an exclusion dict as defined in
- exclude.parseExcludeFile class.
- @param destructive: boolean, defaults to False
- @param time_limit: integer time value as returned by parseTime()
- @param package_names: boolean, defaults to False.
- used only if destructive=True
- @param pkgdir: path to the binary package dir being checked
- @param port_dbapi: defaults to
portage.db[portage.root]["porttree"].dbapi
- can be overridden for tests.
- @param var_dbapi: defaults to portage.db[portage.root]["vartree"].dbapi
- can be overridden for tests.
-
- @rtype: dict
- @return clean_me i.e. {'cat/pkg-ver.tbz2': [filepath],}
- """
- if exclude is None:
- exclude = {}
- clean_me = {}
- # create a full package dictionary
-
- # now do an access test, os.walk does not error for "no read permission"
- try:
- test = os.listdir(pkgdir)
- del test
- except EnvironmentError as er:
- if options['ignore-failure']:
- exit(0)
- print( pp.error("Error accessing PKGDIR." ), file=sys.stderr)
- print( pp.error("(Check your make.conf file and
environment)."), file=sys.stderr)
- print( pp.error("Error: %s" %str(er)), file=sys.stderr)
- exit(1)
-
- # if portage supports FEATURES=binpkg-multi-instance, then
- # cpv_all can return multiple instances per cpv, where
- # instances are distinguishable by some extra attributes
- # provided by portage's _pkg_str class
- bin_dbapi = portage.binarytree(pkgdir=pkgdir,
settings=var_dbapi.settings).dbapi
- for cpv in bin_dbapi.cpv_all():
- mtime = int(bin_dbapi.aux_get(cpv, ['_mtime_'])[0])
- if time_limit and mtime >= time_limit:
- # time-limit exclusion
- continue
- # dict is cpv->[pkgs] (supports binpkg-multi-instance)
- clean_me.setdefault(cpv, []).append(cpv)
-
- # keep only obsolete ones
- if destructive and package_names:
- cp_all = dict.fromkeys(var_dbapi.cp_all())
- else:
- cp_all = {}
- for cpv in list(clean_me):
- if exclDictMatchCP(exclude,portage.cpv_getkey(cpv)):
- # exclusion because of the exclude file
- del clean_me[cpv]
- continue
- if not destructive and port_dbapi.cpv_exists(cpv):
- # exclusion because pkg still exists (in porttree)
- del clean_me[cpv]
- continue
- if destructive and var_dbapi.cpv_exists(cpv):
- buildtime = var_dbapi.aux_get(cpv, ['BUILD_TIME'])[0]
- clean_me[cpv] = [pkg for pkg in clean_me[cpv]
- # only keep path if BUILD_TIME is identical
with vartree
- if bin_dbapi.aux_get(pkg, ['BUILD_TIME'])[0] !=
buildtime]
- if not clean_me[cpv]:
- # nothing we can clean for this package
- del clean_me[cpv]
- continue
- if portage.cpv_getkey(cpv) in cp_all and
port_dbapi.cpv_exists(cpv):
- # exclusion because of --package-names
- del clean_me[cpv]
-
- # the getname method correctly supports
Re: [gentoo-portage-dev] [PATCH] einstalldocs: Fix test for DOCS being unset.
> On Thu, 20 Feb 2020, Zac Medico wrote: > Looks good. Even though earlier EAPIs are deprecated, would there be a > problem with updating __eapi4_src_install for consistency? I'd rather not, because for EAPIs 4 and 5 PMS specifies that the return status of declare -p should be tested [1]. Presumably it wouldn't make much of a difference, because we don't account for the "assigned, but empty" case in these EAPIs. src_install format 4 will just fail for DOCS="" or DOCS=(). [1] https://projects.gentoo.org/pms/7/pms.html#x1-94001r7 signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] einstalldocs: Fix test for DOCS being unset.
On 2/20/20 5:04 AM, Ulrich Müller wrote:
> The current test does not exactly test for unset DOCS, because it also
> evaluates as true if the variable has attributes. Such attributes can
> be defined even for an unset variable.
>
> Therefore test the output of declare -p for presence of an = sign
> instead, which indicates that a value has been assigned to the
> variable (bug 710076 comment #2).
>
> PMS reference: Algorithm 12.4, line 7:
> https://projects.gentoo.org/pms/7/pms.html#x1-135011r183
>
> See also bash upstream discussion:
> https://lists.gnu.org/archive/html/bug-bash/2020-02/msg00045.html
>
> Closes: https://bugs.gentoo.org/710076
> Signed-off-by: Ulrich Müller
> ---
> bin/phase-helpers.sh | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/bin/phase-helpers.sh b/bin/phase-helpers.sh
> index 3deb28c68..9495465f9 100644
> --- a/bin/phase-helpers.sh
> +++ b/bin/phase-helpers.sh
> @@ -1,5 +1,5 @@
> #!/bin/bash
> -# Copyright 1999-2019 Gentoo Authors
> +# Copyright 1999-2020 Gentoo Authors
> # Distributed under the terms of the GNU General Public License v2
>
> if ___eapi_has_DESTTREE_INSDESTTREE; then
> @@ -953,7 +953,7 @@ fi
> if ___eapi_has_einstalldocs; then
> einstalldocs() {
> (
> - if ! declare -p DOCS &>/dev/null ; then
> + if [[ $(declare -p DOCS 2>/dev/null) != *=* ]]; then
> local d
> for d in README* ChangeLog AUTHORS NEWS TODO
> CHANGES \
> THANKS BUGS FAQ CREDITS
> CHANGELOG ; do
>
Looks good. Even though earlier EAPIs are deprecated, would there be a
problem with updating __eapi4_src_install for consistency?
--
Thanks,
Zac
signature.asc
Description: OpenPGP digital signature
[gentoo-portage-dev] [PATCH] einstalldocs: Fix test for DOCS being unset.
The current test does not exactly test for unset DOCS, because it also
evaluates as true if the variable has attributes. Such attributes can
be defined even for an unset variable.
Therefore test the output of declare -p for presence of an = sign
instead, which indicates that a value has been assigned to the
variable (bug 710076 comment #2).
PMS reference: Algorithm 12.4, line 7:
https://projects.gentoo.org/pms/7/pms.html#x1-135011r183
See also bash upstream discussion:
https://lists.gnu.org/archive/html/bug-bash/2020-02/msg00045.html
Closes: https://bugs.gentoo.org/710076
Signed-off-by: Ulrich Müller
---
bin/phase-helpers.sh | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/bin/phase-helpers.sh b/bin/phase-helpers.sh
index 3deb28c68..9495465f9 100644
--- a/bin/phase-helpers.sh
+++ b/bin/phase-helpers.sh
@@ -1,5 +1,5 @@
#!/bin/bash
-# Copyright 1999-2019 Gentoo Authors
+# Copyright 1999-2020 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2
if ___eapi_has_DESTTREE_INSDESTTREE; then
@@ -953,7 +953,7 @@ fi
if ___eapi_has_einstalldocs; then
einstalldocs() {
(
- if ! declare -p DOCS &>/dev/null ; then
+ if [[ $(declare -p DOCS 2>/dev/null) != *=* ]]; then
local d
for d in README* ChangeLog AUTHORS NEWS TODO
CHANGES \
THANKS BUGS FAQ CREDITS
CHANGELOG ; do
--
2.25.1
signature.asc
Description: PGP signature
