Re: [gentoo-portage-dev] [PATCH gentoolkit 2/2] eclean: Add option to delete binpkgs with changed deps
On Mon, Mar 2, 2020 at 9:15 PM Zac Medico wrote:
>
> On 3/2/20 1:11 PM, Matt Turner wrote:
> > On Sun, Mar 1, 2020 at 10:39 PM Zac Medico wrote:
> >>
> >> On 2/20/20 9:29 PM, Matt Turner wrote:
> >>> @@ -564,7 +577,22 @@ def findPackages(
> >>>
> >>> # Exclude if binpkg exists in the porttree and not --deep
> >>> if not destructive and port_dbapi.cpv_exists(cpv):
> >>> -continue
> >>> +if not options['changed-deps']:
> >>> +continue
> >>> +
> >>> +uselist = bin_dbapi.aux_get(cpv, ['USE'])[0].split()
> >>> +all_equal = True
> >>> +
> >>> +for k in ('RDEPEND', 'PDEPEND'):
> >>> +binpkg_deps = bin_dbapi.aux_get(cpv, [k])
> >>> +ebuild_deps = port_dbapi.aux_get(cpv, [k])
> >>> +
> >>> +if not _deps_equal(binpkg_deps, ebuild_deps, cpv.eapi,
> >>> uselist):
> >>> +all_equal = False
> >>> +break
> >>> +
> >>> +if all_equal:
> >>> +continue
> >>
> >> If all_equal is True, then none of the other filters have an opportunity
> >> to add this package to the dead_binpkgs set. That's not good is it?
> >
> > There are four cases we skip including packages: 1) exclude list, 2)
> > time limit, 3) non-destructive and package still exists (and now
> > optionally runtime deps haven't changed), 4) destructive and package
> > is installed. Cases (3) and (4) are non-overlapping.
> >
> > If none of those cases are true, then we add the package to the
> > dead_binpkgs set. The logic looks right to me.
> >
> > Maybe I'm not understanding.
>
> What I imagine is that you could have some old packages that you
> probably want to delete because they're so old, even though their deps
> have not changed. Meanwhile you have some packages that are
> relatively recent and you'd like to delete them if they have changed deps.
>
> Given the current logic, I guess you'd have to do separate passes, one
> to delete packages based on age and another to delete packages based on
> changed deps. Maybe it's fine to require separate passes for this kind
> of thing. I supposed the alternative would be to add an --or flag that
> would allow you to say that you want to delete packages if they are at
> least a certain age or they have changed deps.
Oh, I think I understand now.
I was confused about this. I expected that --time-limit=2w to mean
that eclean should delete everything older than two weeks, but it's
actually the opposite, more or less. It actually means to *keep*
everything with less than two weeks old. Surprised me...
> -t, --time-limit= - don't delete files modified since
So, with that surprising behavior I think my patch is doing the right
thing, but with the wrong comment. I'll fix those and send v2 patches
with the tabs restored, etc.
Thanks a bunch for the review.
Re: [gentoo-portage-dev] [PATCH gentoolkit 2/2] eclean: Add option to delete binpkgs with changed deps
On 3/2/20 1:11 PM, Matt Turner wrote:
> On Sun, Mar 1, 2020 at 10:39 PM Zac Medico wrote:
>>
>> On 2/20/20 9:29 PM, Matt Turner wrote:
>>> @@ -564,7 +577,22 @@ def findPackages(
>>>
>>> # Exclude if binpkg exists in the porttree and not --deep
>>> if not destructive and port_dbapi.cpv_exists(cpv):
>>> -continue
>>> +if not options['changed-deps']:
>>> +continue
>>> +
>>> +uselist = bin_dbapi.aux_get(cpv, ['USE'])[0].split()
>>> +all_equal = True
>>> +
>>> +for k in ('RDEPEND', 'PDEPEND'):
>>> +binpkg_deps = bin_dbapi.aux_get(cpv, [k])
>>> +ebuild_deps = port_dbapi.aux_get(cpv, [k])
>>> +
>>> +if not _deps_equal(binpkg_deps, ebuild_deps, cpv.eapi,
>>> uselist):
>>> +all_equal = False
>>> +break
>>> +
>>> +if all_equal:
>>> +continue
>>
>> If all_equal is True, then none of the other filters have an opportunity
>> to add this package to the dead_binpkgs set. That's not good is it?
>
> There are four cases we skip including packages: 1) exclude list, 2)
> time limit, 3) non-destructive and package still exists (and now
> optionally runtime deps haven't changed), 4) destructive and package
> is installed. Cases (3) and (4) are non-overlapping.
>
> If none of those cases are true, then we add the package to the
> dead_binpkgs set. The logic looks right to me.
>
> Maybe I'm not understanding.
What I imagine is that you could have some old packages that you
probably want to delete because they're so old, even though their deps
have not changed. Meanwhile you have some packages that are
relatively recent and you'd like to delete them if they have changed deps.
Given the current logic, I guess you'd have to do separate passes, one
to delete packages based on age and another to delete packages based on
changed deps. Maybe it's fine to require separate passes for this kind
of thing. I supposed the alternative would be to add an --or flag that
would allow you to say that you want to delete packages if they are at
least a certain age or they have changed deps.
--
Thanks,
Zac
signature.asc
Description: OpenPGP digital signature
Re: [gentoo-portage-dev] [PATCH gentoolkit 2/2] eclean: Add option to delete binpkgs with changed deps
On 3/2/20 1:02 PM, Matt Turner wrote:
> On Mon, Mar 2, 2020 at 12:40 PM Matt Turner wrote:
>>
>> On Sun, Mar 1, 2020 at 10:32 PM Zac Medico wrote:
>>>
>>> On 2/20/20 9:29 PM, Matt Turner wrote:
+
def findPackages(
options,
exclude=None,
@@ -564,7 +577,22 @@ def findPackages(
# Exclude if binpkg exists in the porttree and not --deep
if not destructive and port_dbapi.cpv_exists(cpv):
-continue
+if not options['changed-deps']:
+continue
+
+uselist = bin_dbapi.aux_get(cpv, ['USE'])[0].split()
+all_equal = True
+
+for k in ('RDEPEND', 'PDEPEND'):
+binpkg_deps = bin_dbapi.aux_get(cpv, [k])
+ebuild_deps = port_dbapi.aux_get(cpv, [k])
+
+if not _deps_equal(binpkg_deps, ebuild_deps, cpv.eapi,
uselist):
+all_equal = False
+break
+
+if all_equal:
+continue
if destructive and var_dbapi.cpv_exists(cpv):
# Exclude if an instance of the package is installed due to
>>>
>>> The aux_get calls are expensive, so it's more efficient to get multiple
>>> values with each call like:
>>> keys = ('RDEPEND', 'PDEPEND')
>>> binpkg_deps = dict(zip(keys, bin_dbapi.aux_get(cpv, keys))
>>> ebuild_deps = dict(zip(keys, port_dbapi.aux_get(cpv, keys))
>>>
>>> Otherwise, looks good.
>>
>> Thanks, that makes the code a lot nicer too.
>
> Actually, use_reduce wants a list (it calls .split). Wrapping those in
> list() looks like it works, but I suspect that's not as you intended.
> What does the zip add over just doing this?
>
> binpkg_deps = bin_dbapi.aux_get(cpv, keys)
> ebuild_deps = port_dbapi.aux_get(cpv, keys)
Using dict(zip(keys, port_dbapi.aux_get(cpv, keys)) is only useful if
you want to use a dictionary to access the values. However, if lists are
good enough then you might just use those instead. You could even join
the values together like this:
ebuild_deps = ' '.join(port_dbapi.aux_get(cpv, keys))
--
Thanks,
Zac
signature.asc
Description: OpenPGP digital signature
Re: [gentoo-portage-dev] [PATCH gentoolkit 2/2] eclean: Add option to delete binpkgs with changed deps
On Sun, Mar 1, 2020 at 10:39 PM Zac Medico wrote:
>
> On 2/20/20 9:29 PM, Matt Turner wrote:
> > @@ -564,7 +577,22 @@ def findPackages(
> >
> > # Exclude if binpkg exists in the porttree and not --deep
> > if not destructive and port_dbapi.cpv_exists(cpv):
> > -continue
> > +if not options['changed-deps']:
> > +continue
> > +
> > +uselist = bin_dbapi.aux_get(cpv, ['USE'])[0].split()
> > +all_equal = True
> > +
> > +for k in ('RDEPEND', 'PDEPEND'):
> > +binpkg_deps = bin_dbapi.aux_get(cpv, [k])
> > +ebuild_deps = port_dbapi.aux_get(cpv, [k])
> > +
> > +if not _deps_equal(binpkg_deps, ebuild_deps, cpv.eapi,
> > uselist):
> > +all_equal = False
> > +break
> > +
> > +if all_equal:
> > +continue
>
> If all_equal is True, then none of the other filters have an opportunity
> to add this package to the dead_binpkgs set. That's not good is it?
There are four cases we skip including packages: 1) exclude list, 2)
time limit, 3) non-destructive and package still exists (and now
optionally runtime deps haven't changed), 4) destructive and package
is installed. Cases (3) and (4) are non-overlapping.
If none of those cases are true, then we add the package to the
dead_binpkgs set. The logic looks right to me.
Maybe I'm not understanding.
With your other suggestion in place, the code looks like this, which
is hopefully clearer.
# Exclude if binpkg exists in the porttree and not --deep
if not destructive and port_dbapi.cpv_exists(cpv):
if not options['changed-deps']:
continue
keys = ('RDEPEND', 'PDEPEND')
binpkg_deps = bin_dbapi.aux_get(cpv, keys)
ebuild_deps = port_dbapi.aux_get(cpv, keys)
uselist = bin_dbapi.aux_get(cpv, ['USE'])[0].split()
if _deps_equal(binpkg_deps, ebuild_deps, cpv.eapi, uselist):
continue
Unfortunately I don't have any packages with changed-deps at the
moment for testing :(
Re: [gentoo-portage-dev] [PATCH gentoolkit 2/2] eclean: Add option to delete binpkgs with changed deps
On Mon, Mar 2, 2020 at 12:40 PM Matt Turner wrote:
>
> On Sun, Mar 1, 2020 at 10:32 PM Zac Medico wrote:
> >
> > On 2/20/20 9:29 PM, Matt Turner wrote:
> > > +
> > > def findPackages(
> > > options,
> > > exclude=None,
> > > @@ -564,7 +577,22 @@ def findPackages(
> > >
> > > # Exclude if binpkg exists in the porttree and not --deep
> > > if not destructive and port_dbapi.cpv_exists(cpv):
> > > -continue
> > > +if not options['changed-deps']:
> > > +continue
> > > +
> > > +uselist = bin_dbapi.aux_get(cpv, ['USE'])[0].split()
> > > +all_equal = True
> > > +
> > > +for k in ('RDEPEND', 'PDEPEND'):
> > > +binpkg_deps = bin_dbapi.aux_get(cpv, [k])
> > > +ebuild_deps = port_dbapi.aux_get(cpv, [k])
> > > +
> > > +if not _deps_equal(binpkg_deps, ebuild_deps, cpv.eapi,
> > > uselist):
> > > +all_equal = False
> > > +break
> > > +
> > > +if all_equal:
> > > +continue
> > >
> > > if destructive and var_dbapi.cpv_exists(cpv):
> > > # Exclude if an instance of the package is installed due to
> > >
> >
> > The aux_get calls are expensive, so it's more efficient to get multiple
> > values with each call like:
> > keys = ('RDEPEND', 'PDEPEND')
> > binpkg_deps = dict(zip(keys, bin_dbapi.aux_get(cpv, keys))
> > ebuild_deps = dict(zip(keys, port_dbapi.aux_get(cpv, keys))
> >
> > Otherwise, looks good.
>
> Thanks, that makes the code a lot nicer too.
Actually, use_reduce wants a list (it calls .split). Wrapping those in
list() looks like it works, but I suspect that's not as you intended.
What does the zip add over just doing this?
binpkg_deps = bin_dbapi.aux_get(cpv, keys)
ebuild_deps = port_dbapi.aux_get(cpv, keys)
Re: [gentoo-portage-dev] [PATCH gentoolkit 2/2] eclean: Add option to delete binpkgs with changed deps
On Sun, Mar 1, 2020 at 10:32 PM Zac Medico wrote:
>
> On 2/20/20 9:29 PM, Matt Turner wrote:
> > +
> > def findPackages(
> > options,
> > exclude=None,
> > @@ -564,7 +577,22 @@ def findPackages(
> >
> > # Exclude if binpkg exists in the porttree and not --deep
> > if not destructive and port_dbapi.cpv_exists(cpv):
> > -continue
> > +if not options['changed-deps']:
> > +continue
> > +
> > +uselist = bin_dbapi.aux_get(cpv, ['USE'])[0].split()
> > +all_equal = True
> > +
> > +for k in ('RDEPEND', 'PDEPEND'):
> > +binpkg_deps = bin_dbapi.aux_get(cpv, [k])
> > +ebuild_deps = port_dbapi.aux_get(cpv, [k])
> > +
> > +if not _deps_equal(binpkg_deps, ebuild_deps, cpv.eapi,
> > uselist):
> > +all_equal = False
> > +break
> > +
> > +if all_equal:
> > +continue
> >
> > if destructive and var_dbapi.cpv_exists(cpv):
> > # Exclude if an instance of the package is installed due to
> >
>
> The aux_get calls are expensive, so it's more efficient to get multiple
> values with each call like:
> keys = ('RDEPEND', 'PDEPEND')
> binpkg_deps = dict(zip(keys, bin_dbapi.aux_get(cpv, keys))
> ebuild_deps = dict(zip(keys, port_dbapi.aux_get(cpv, keys))
>
> Otherwise, looks good.
Thanks, that makes the code a lot nicer too.
Re: [gentoo-portage-dev] [PATCH gentoolkit 2/2] eclean: Add option to delete binpkgs with changed deps
On 2/20/20 9:29 PM, Matt Turner wrote:
> @@ -564,7 +577,22 @@ def findPackages(
>
> # Exclude if binpkg exists in the porttree and not --deep
> if not destructive and port_dbapi.cpv_exists(cpv):
> -continue
> +if not options['changed-deps']:
> +continue
> +
> +uselist = bin_dbapi.aux_get(cpv, ['USE'])[0].split()
> +all_equal = True
> +
> +for k in ('RDEPEND', 'PDEPEND'):
> +binpkg_deps = bin_dbapi.aux_get(cpv, [k])
> +ebuild_deps = port_dbapi.aux_get(cpv, [k])
> +
> +if not _deps_equal(binpkg_deps, ebuild_deps, cpv.eapi,
> uselist):
> +all_equal = False
> +break
> +
> +if all_equal:
> +continue
If all_equal is True, then none of the other filters have an opportunity
to add this package to the dead_binpkgs set. That's not good is it?
--
Thanks,
Zac
signature.asc
Description: OpenPGP digital signature
Re: [gentoo-portage-dev] [PATCH gentoolkit 2/2] eclean: Add option to delete binpkgs with changed deps
On 2/20/20 9:29 PM, Matt Turner wrote:
> +
> def findPackages(
> options,
> exclude=None,
> @@ -564,7 +577,22 @@ def findPackages(
>
> # Exclude if binpkg exists in the porttree and not --deep
> if not destructive and port_dbapi.cpv_exists(cpv):
> -continue
> +if not options['changed-deps']:
> +continue
> +
> +uselist = bin_dbapi.aux_get(cpv, ['USE'])[0].split()
> +all_equal = True
> +
> +for k in ('RDEPEND', 'PDEPEND'):
> +binpkg_deps = bin_dbapi.aux_get(cpv, [k])
> +ebuild_deps = port_dbapi.aux_get(cpv, [k])
> +
> +if not _deps_equal(binpkg_deps, ebuild_deps, cpv.eapi,
> uselist):
> +all_equal = False
> +break
> +
> +if all_equal:
> +continue
>
> if destructive and var_dbapi.cpv_exists(cpv):
> # Exclude if an instance of the package is installed due to
>
The aux_get calls are expensive, so it's more efficient to get multiple
values with each call like:
keys = ('RDEPEND', 'PDEPEND')
binpkg_deps = dict(zip(keys, bin_dbapi.aux_get(cpv, keys))
ebuild_deps = dict(zip(keys, port_dbapi.aux_get(cpv, keys))
Otherwise, looks good.
--
Thanks,
Zac
signature.asc
Description: OpenPGP digital signature
[gentoo-portage-dev] [PATCH gentoolkit 2/2] eclean: Add option to delete binpkgs with changed deps
Signed-off-by: Matt Turner
---
pym/gentoolkit/eclean/cli.py| 7 ++-
pym/gentoolkit/eclean/search.py | 30 +-
2 files changed, 35 insertions(+), 2 deletions(-)
diff --git a/pym/gentoolkit/eclean/cli.py b/pym/gentoolkit/eclean/cli.py
index 1a99b3e..39aafd3 100644
--- a/pym/gentoolkit/eclean/cli.py
+++ b/pym/gentoolkit/eclean/cli.py
@@ -147,6 +147,8 @@ def printUsage(_error=None, help=None):
or help in ('all','packages'):
print( "Available", yellow("options"),"for the",
green("packages"),"action:", file=out)
+ print( yellow(" --changed-deps")+
+ " - delete packages for which ebuild
dependencies have changed", file=out)
print( yellow(" -i, --ignore-failure")+
" - ignore failure to locate PKGDIR",
file=out)
print( file=out)
@@ -263,6 +265,8 @@ def parseArgs(options={}):
options['size-limit'] = parseSize(a)
elif o in ("-v", "--verbose") and not options['quiet']:
options['verbose'] = True
+ elif o in ("--changed-deps"):
+ options['changed-deps'] = True
elif o in ("-i", "--ignore-failure"):
options['ignore-failure'] = True
else:
@@ -290,7 +294,7 @@ def parseArgs(options={}):
getopt_options['short']['distfiles'] = "fs:"
getopt_options['long']['distfiles'] = ["fetch-restricted",
"size-limit="]
getopt_options['short']['packages'] = "i"
- getopt_options['long']['packages'] = ["ignore-failure"]
+ getopt_options['long']['packages'] = ["ignore-failure", "changed-deps"]
# set default options, except 'nocolor', which is set in main()
options['interactive'] = False
options['pretend'] = False
@@ -303,6 +307,7 @@ def parseArgs(options={}):
options['fetch-restricted'] = False
options['size-limit'] = 0
options['verbose'] = False
+ options['changed-deps'] = False
options['ignore-failure'] = False
# if called by a well-named symlink, set the action accordingly:
action = None
diff --git a/pym/gentoolkit/eclean/search.py b/pym/gentoolkit/eclean/search.py
index 831ba39..da8c286 100644
--- a/pym/gentoolkit/eclean/search.py
+++ b/pym/gentoolkit/eclean/search.py
@@ -13,6 +13,8 @@ import sys
from functools import partial
import portage
+from portage.dep import Atom, use_reduce
+from portage.dep._slot_operator import strip_slots
import gentoolkit.pprinter as pp
from gentoolkit.eclean.exclude import (exclDictMatchCP, exclDictExpand,
@@ -488,6 +490,17 @@ class DistfilesSearch(object):
return clean_me, saved
+def _deps_equal(deps_a, deps_b, eapi, uselist=None):
+"""Compare two dependency lists given a set of USE flags"""
+if deps_a == deps_b: return True
+
+deps_a = use_reduce(deps_a, uselist=uselist, eapi=eapi, token_class=Atom)
+deps_b = use_reduce(deps_b, uselist=uselist, eapi=eapi, token_class=Atom)
+strip_slots(deps_a)
+strip_slots(deps_b)
+return deps_a == deps_b
+
+
def findPackages(
options,
exclude=None,
@@ -564,7 +577,22 @@ def findPackages(
# Exclude if binpkg exists in the porttree and not --deep
if not destructive and port_dbapi.cpv_exists(cpv):
-continue
+if not options['changed-deps']:
+continue
+
+uselist = bin_dbapi.aux_get(cpv, ['USE'])[0].split()
+all_equal = True
+
+for k in ('RDEPEND', 'PDEPEND'):
+binpkg_deps = bin_dbapi.aux_get(cpv, [k])
+ebuild_deps = port_dbapi.aux_get(cpv, [k])
+
+if not _deps_equal(binpkg_deps, ebuild_deps, cpv.eapi,
uselist):
+all_equal = False
+break
+
+if all_equal:
+continue
if destructive and var_dbapi.cpv_exists(cpv):
# Exclude if an instance of the package is installed due to
--
2.24.1
