Re: [gentoo-portage-dev] [PATCH] Add @changed-subslot package set
On 1/18/21 8:42 PM, Alec Warner wrote:
> On Mon, Jan 18, 2021 at 8:09 PM Zac Medico wrote:
>>
>> On 1/18/21 6:07 PM, Alec Warner wrote:
>>> On Fri, Jan 15, 2021 at 6:47 PM Matt Turner wrote:
This set is the upgradable packages for which the highest visible
version has a different subslot than the currently installed version.
The primary purpose of this feature is for use in catalyst builds. We
update the "seed" stage3 before using it to build a new stage1.
Updating the entire stage is expensive and unnecessary (since we're
going to build the latest packages in stage1 and then rebuild everything
in stage3).
What we definitely do need to update in the original stage3 however, is
any package that would trigger a subslot rebuild.
For example: gcc links with libmpfr.so from dev-libs/mpfr. mpfr's SONAME
changes from libmpfr.so.4 (SLOT="0/4") to libmpfr.so.6 (SLOT="0/6"). If
the seed stage's dev-libs/mpfr is not updated before emerging gcc, gcc
will link with libmpfr.so.4, but the latest version of dev-libs/mpfr
will be built and libmpfr.so.6 included into the stage1. Since the old
libmpfr.so.4 is not included in the stage1, gcc will not work, breaking
subsequent stage builds.
Our current options to update the seed are too large a hammer (e.g.,
"--update --deep --newuse @world" or "--update --deep --newuse
--complete-graph --rebuild-if-new-ver gcc") and spend too much time
updating seed stages for no gain beyond updating only packages for whom
the subslot has changed.
With this set, catalyst will likely use
emerge @changed-subslot --ignore-built-slot-operator-deps y
to update the seed stage.
Thank you to Zac Medico for showing me how to do this.
Bug: https://bugs.gentoo.org/739004
Signed-off-by: Matt Turner
---
cnf/sets/portage.conf | 5 +
lib/portage/_sets/dbapi.py | 39 +-
2 files changed, 43 insertions(+), 1 deletion(-)
diff --git a/cnf/sets/portage.conf b/cnf/sets/portage.conf
index 22f0fa3a5..5651a9c53 100644
--- a/cnf/sets/portage.conf
+++ b/cnf/sets/portage.conf
@@ -84,6 +84,11 @@ exclude-files = /usr/bin/Xorg
[rebuilt-binaries]
class = portage.sets.dbapi.RebuiltBinaries
+# Installed packages for which the subslot of the highest visible ebuild
+# version is different than the currently installed version.
+[changed-subslot]
+class = portage.sets.dbapi.SubslotChangedSet
+
# Installed packages for which the highest visible ebuild
# version is lower than the currently installed version.
[downgrade]
diff --git a/lib/portage/_sets/dbapi.py b/lib/portage/_sets/dbapi.py
index 52367c4a6..46ba5c17d 100644
--- a/lib/portage/_sets/dbapi.py
+++ b/lib/portage/_sets/dbapi.py
@@ -15,7 +15,7 @@ from portage._sets import SetConfigError, get_boolean
import portage
__all__ = ["CategorySet", "ChangedDepsSet", "DowngradeSet",
- "EverythingSet", "OwnerSet", "VariableSet"]
+ "EverythingSet", "OwnerSet", "SubslotChangedSet", "VariableSet"]
class EverythingSet(PackageSet):
_operations = ["merge"]
@@ -167,6 +167,43 @@ class VariableSet(EverythingSet):
singleBuilder = classmethod(singleBuilder)
+class SubslotChangedSet(PackageSet):
+
+ _operations = ["merge", "unmerge"]
+
+ description = "Package set which contains all packages " + \
+ "for which the subslot of the highest visible ebuild is "
+ \
+ "different than the currently installed version."
>>>
>>> description = ("string1",
>>>"string2",
>>>"string3")
>>>
>>> vs concat + \ for line continuation?
>>>
>>> -A
>>
>> We also got flak on irc about the classmethod(singleBuilder) usage as
>> opposed to @classmethod. This package set is formatted exactly like
>> others in the file, so it's just a copy / paste thing.
>
> Ack, I can send a patch. I don't watch all these CLs.
>
> PEP8 is fairly clear about line continuations (\) and whey are
> appropriate (e.g. not here.)
> I don't think the pep cares about classmethod() vs @classmethod (and I
> don't really care either.)
>
> I personally find:
>
> a = "foo" + " bar" + \
> "baz"
>
> jarring; as compared to:
>
> a = ("foo", "bar",
> "baz")
>
Me too, parenthesis look much better.
> but then you have to know that python will implicitly join these
> strings together, which is unfortunate, butl I attempt to write python
> assuming readers know python (so I'm not worried about readers knowing
> implicit string join.) There are probably 5 other ways to write this
> block (""" strings, ''.join(), string.format(), some other stuff.)
Re: [gentoo-portage-dev] [PATCH] Add @changed-subslot package set
On Mon, Jan 18, 2021 at 8:09 PM Zac Medico wrote:
>
> On 1/18/21 6:07 PM, Alec Warner wrote:
> > On Fri, Jan 15, 2021 at 6:47 PM Matt Turner wrote:
> >>
> >> This set is the upgradable packages for which the highest visible
> >> version has a different subslot than the currently installed version.
> >>
> >> The primary purpose of this feature is for use in catalyst builds. We
> >> update the "seed" stage3 before using it to build a new stage1.
> >>
> >> Updating the entire stage is expensive and unnecessary (since we're
> >> going to build the latest packages in stage1 and then rebuild everything
> >> in stage3).
> >>
> >> What we definitely do need to update in the original stage3 however, is
> >> any package that would trigger a subslot rebuild.
> >>
> >> For example: gcc links with libmpfr.so from dev-libs/mpfr. mpfr's SONAME
> >> changes from libmpfr.so.4 (SLOT="0/4") to libmpfr.so.6 (SLOT="0/6"). If
> >> the seed stage's dev-libs/mpfr is not updated before emerging gcc, gcc
> >> will link with libmpfr.so.4, but the latest version of dev-libs/mpfr
> >> will be built and libmpfr.so.6 included into the stage1. Since the old
> >> libmpfr.so.4 is not included in the stage1, gcc will not work, breaking
> >> subsequent stage builds.
> >>
> >> Our current options to update the seed are too large a hammer (e.g.,
> >> "--update --deep --newuse @world" or "--update --deep --newuse
> >> --complete-graph --rebuild-if-new-ver gcc") and spend too much time
> >> updating seed stages for no gain beyond updating only packages for whom
> >> the subslot has changed.
> >>
> >> With this set, catalyst will likely use
> >>
> >> emerge @changed-subslot --ignore-built-slot-operator-deps y
> >>
> >> to update the seed stage.
> >>
> >> Thank you to Zac Medico for showing me how to do this.
> >>
> >> Bug: https://bugs.gentoo.org/739004
> >> Signed-off-by: Matt Turner
> >> ---
> >> cnf/sets/portage.conf | 5 +
> >> lib/portage/_sets/dbapi.py | 39 +-
> >> 2 files changed, 43 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/cnf/sets/portage.conf b/cnf/sets/portage.conf
> >> index 22f0fa3a5..5651a9c53 100644
> >> --- a/cnf/sets/portage.conf
> >> +++ b/cnf/sets/portage.conf
> >> @@ -84,6 +84,11 @@ exclude-files = /usr/bin/Xorg
> >> [rebuilt-binaries]
> >> class = portage.sets.dbapi.RebuiltBinaries
> >>
> >> +# Installed packages for which the subslot of the highest visible ebuild
> >> +# version is different than the currently installed version.
> >> +[changed-subslot]
> >> +class = portage.sets.dbapi.SubslotChangedSet
> >> +
> >> # Installed packages for which the highest visible ebuild
> >> # version is lower than the currently installed version.
> >> [downgrade]
> >> diff --git a/lib/portage/_sets/dbapi.py b/lib/portage/_sets/dbapi.py
> >> index 52367c4a6..46ba5c17d 100644
> >> --- a/lib/portage/_sets/dbapi.py
> >> +++ b/lib/portage/_sets/dbapi.py
> >> @@ -15,7 +15,7 @@ from portage._sets import SetConfigError, get_boolean
> >> import portage
> >>
> >> __all__ = ["CategorySet", "ChangedDepsSet", "DowngradeSet",
> >> - "EverythingSet", "OwnerSet", "VariableSet"]
> >> + "EverythingSet", "OwnerSet", "SubslotChangedSet", "VariableSet"]
> >>
> >> class EverythingSet(PackageSet):
> >> _operations = ["merge"]
> >> @@ -167,6 +167,43 @@ class VariableSet(EverythingSet):
> >>
> >> singleBuilder = classmethod(singleBuilder)
> >>
> >> +class SubslotChangedSet(PackageSet):
> >> +
> >> + _operations = ["merge", "unmerge"]
> >> +
> >> + description = "Package set which contains all packages " + \
> >> + "for which the subslot of the highest visible ebuild is "
> >> + \
> >> + "different than the currently installed version."
> >
> > description = ("string1",
> >"string2",
> >"string3")
> >
> > vs concat + \ for line continuation?
> >
> > -A
>
> We also got flak on irc about the classmethod(singleBuilder) usage as
> opposed to @classmethod. This package set is formatted exactly like
> others in the file, so it's just a copy / paste thing.
Ack, I can send a patch. I don't watch all these CLs.
PEP8 is fairly clear about line continuations (\) and whey are
appropriate (e.g. not here.)
I don't think the pep cares about classmethod() vs @classmethod (and I
don't really care either.)
I personally find:
a = "foo" + " bar" + \
"baz"
jarring; as compared to:
a = ("foo", "bar",
"baz")
but then you have to know that python will implicitly join these
strings together, which is unfortunate, butl I attempt to write python
assuming readers know python (so I'm not worried about readers knowing
implicit string join.) There are probably 5 other ways to write this
block (""" strings, ''.join(), string.format(), some other stuff.)
>
> On the topic of python formatting, maybe we should use something like
> https://github.com/psf/black to automate it?
I've used black
Re: [gentoo-portage-dev] [PATCH] Add @changed-subslot package set
On 1/18/21 6:07 PM, Alec Warner wrote:
> On Fri, Jan 15, 2021 at 6:47 PM Matt Turner wrote:
>>
>> This set is the upgradable packages for which the highest visible
>> version has a different subslot than the currently installed version.
>>
>> The primary purpose of this feature is for use in catalyst builds. We
>> update the "seed" stage3 before using it to build a new stage1.
>>
>> Updating the entire stage is expensive and unnecessary (since we're
>> going to build the latest packages in stage1 and then rebuild everything
>> in stage3).
>>
>> What we definitely do need to update in the original stage3 however, is
>> any package that would trigger a subslot rebuild.
>>
>> For example: gcc links with libmpfr.so from dev-libs/mpfr. mpfr's SONAME
>> changes from libmpfr.so.4 (SLOT="0/4") to libmpfr.so.6 (SLOT="0/6"). If
>> the seed stage's dev-libs/mpfr is not updated before emerging gcc, gcc
>> will link with libmpfr.so.4, but the latest version of dev-libs/mpfr
>> will be built and libmpfr.so.6 included into the stage1. Since the old
>> libmpfr.so.4 is not included in the stage1, gcc will not work, breaking
>> subsequent stage builds.
>>
>> Our current options to update the seed are too large a hammer (e.g.,
>> "--update --deep --newuse @world" or "--update --deep --newuse
>> --complete-graph --rebuild-if-new-ver gcc") and spend too much time
>> updating seed stages for no gain beyond updating only packages for whom
>> the subslot has changed.
>>
>> With this set, catalyst will likely use
>>
>> emerge @changed-subslot --ignore-built-slot-operator-deps y
>>
>> to update the seed stage.
>>
>> Thank you to Zac Medico for showing me how to do this.
>>
>> Bug: https://bugs.gentoo.org/739004
>> Signed-off-by: Matt Turner
>> ---
>> cnf/sets/portage.conf | 5 +
>> lib/portage/_sets/dbapi.py | 39 +-
>> 2 files changed, 43 insertions(+), 1 deletion(-)
>>
>> diff --git a/cnf/sets/portage.conf b/cnf/sets/portage.conf
>> index 22f0fa3a5..5651a9c53 100644
>> --- a/cnf/sets/portage.conf
>> +++ b/cnf/sets/portage.conf
>> @@ -84,6 +84,11 @@ exclude-files = /usr/bin/Xorg
>> [rebuilt-binaries]
>> class = portage.sets.dbapi.RebuiltBinaries
>>
>> +# Installed packages for which the subslot of the highest visible ebuild
>> +# version is different than the currently installed version.
>> +[changed-subslot]
>> +class = portage.sets.dbapi.SubslotChangedSet
>> +
>> # Installed packages for which the highest visible ebuild
>> # version is lower than the currently installed version.
>> [downgrade]
>> diff --git a/lib/portage/_sets/dbapi.py b/lib/portage/_sets/dbapi.py
>> index 52367c4a6..46ba5c17d 100644
>> --- a/lib/portage/_sets/dbapi.py
>> +++ b/lib/portage/_sets/dbapi.py
>> @@ -15,7 +15,7 @@ from portage._sets import SetConfigError, get_boolean
>> import portage
>>
>> __all__ = ["CategorySet", "ChangedDepsSet", "DowngradeSet",
>> - "EverythingSet", "OwnerSet", "VariableSet"]
>> + "EverythingSet", "OwnerSet", "SubslotChangedSet", "VariableSet"]
>>
>> class EverythingSet(PackageSet):
>> _operations = ["merge"]
>> @@ -167,6 +167,43 @@ class VariableSet(EverythingSet):
>>
>> singleBuilder = classmethod(singleBuilder)
>>
>> +class SubslotChangedSet(PackageSet):
>> +
>> + _operations = ["merge", "unmerge"]
>> +
>> + description = "Package set which contains all packages " + \
>> + "for which the subslot of the highest visible ebuild is " + \
>> + "different than the currently installed version."
>
> description = ("string1",
>"string2",
>"string3")
>
> vs concat + \ for line continuation?
>
> -A
We also got flak on irc about the classmethod(singleBuilder) usage as
opposed to @classmethod. This package set is formatted exactly like
others in the file, so it's just a copy / paste thing.
On the topic of python formatting, maybe we should use something like
https://github.com/psf/black to automate it?
--
Thanks,
Zac
signature.asc
Description: OpenPGP digital signature
Re: [gentoo-portage-dev] [PATCH] Add @changed-subslot package set
On Fri, Jan 15, 2021 at 6:47 PM Matt Turner wrote:
>
> This set is the upgradable packages for which the highest visible
> version has a different subslot than the currently installed version.
>
> The primary purpose of this feature is for use in catalyst builds. We
> update the "seed" stage3 before using it to build a new stage1.
>
> Updating the entire stage is expensive and unnecessary (since we're
> going to build the latest packages in stage1 and then rebuild everything
> in stage3).
>
> What we definitely do need to update in the original stage3 however, is
> any package that would trigger a subslot rebuild.
>
> For example: gcc links with libmpfr.so from dev-libs/mpfr. mpfr's SONAME
> changes from libmpfr.so.4 (SLOT="0/4") to libmpfr.so.6 (SLOT="0/6"). If
> the seed stage's dev-libs/mpfr is not updated before emerging gcc, gcc
> will link with libmpfr.so.4, but the latest version of dev-libs/mpfr
> will be built and libmpfr.so.6 included into the stage1. Since the old
> libmpfr.so.4 is not included in the stage1, gcc will not work, breaking
> subsequent stage builds.
>
> Our current options to update the seed are too large a hammer (e.g.,
> "--update --deep --newuse @world" or "--update --deep --newuse
> --complete-graph --rebuild-if-new-ver gcc") and spend too much time
> updating seed stages for no gain beyond updating only packages for whom
> the subslot has changed.
>
> With this set, catalyst will likely use
>
> emerge @changed-subslot --ignore-built-slot-operator-deps y
>
> to update the seed stage.
>
> Thank you to Zac Medico for showing me how to do this.
>
> Bug: https://bugs.gentoo.org/739004
> Signed-off-by: Matt Turner
> ---
> cnf/sets/portage.conf | 5 +
> lib/portage/_sets/dbapi.py | 39 +-
> 2 files changed, 43 insertions(+), 1 deletion(-)
>
> diff --git a/cnf/sets/portage.conf b/cnf/sets/portage.conf
> index 22f0fa3a5..5651a9c53 100644
> --- a/cnf/sets/portage.conf
> +++ b/cnf/sets/portage.conf
> @@ -84,6 +84,11 @@ exclude-files = /usr/bin/Xorg
> [rebuilt-binaries]
> class = portage.sets.dbapi.RebuiltBinaries
>
> +# Installed packages for which the subslot of the highest visible ebuild
> +# version is different than the currently installed version.
> +[changed-subslot]
> +class = portage.sets.dbapi.SubslotChangedSet
> +
> # Installed packages for which the highest visible ebuild
> # version is lower than the currently installed version.
> [downgrade]
> diff --git a/lib/portage/_sets/dbapi.py b/lib/portage/_sets/dbapi.py
> index 52367c4a6..46ba5c17d 100644
> --- a/lib/portage/_sets/dbapi.py
> +++ b/lib/portage/_sets/dbapi.py
> @@ -15,7 +15,7 @@ from portage._sets import SetConfigError, get_boolean
> import portage
>
> __all__ = ["CategorySet", "ChangedDepsSet", "DowngradeSet",
> - "EverythingSet", "OwnerSet", "VariableSet"]
> + "EverythingSet", "OwnerSet", "SubslotChangedSet", "VariableSet"]
>
> class EverythingSet(PackageSet):
> _operations = ["merge"]
> @@ -167,6 +167,43 @@ class VariableSet(EverythingSet):
>
> singleBuilder = classmethod(singleBuilder)
>
> +class SubslotChangedSet(PackageSet):
> +
> + _operations = ["merge", "unmerge"]
> +
> + description = "Package set which contains all packages " + \
> + "for which the subslot of the highest visible ebuild is " + \
> + "different than the currently installed version."
description = ("string1",
"string2",
"string3")
vs concat + \ for line continuation?
-A
> +
> + def __init__(self, portdb=None, vardb=None):
> + super(SubslotChangedSet, self).__init__()
> + self._portdb = portdb
> + self._vardb = vardb
> +
> + def load(self):
> + atoms = []
> + xmatch = self._portdb.xmatch
> + xmatch_level = "bestmatch-visible"
> + cp_list = self._vardb.cp_list
> + pkg_str = self._vardb._pkg_str
> + for cp in self._vardb.cp_all():
> + for cpv in cp_list(cp):
> + pkg = pkg_str(cpv, None)
> + slot_atom = "%s:%s" % (pkg.cp, pkg.slot)
> + ebuild = xmatch(xmatch_level, slot_atom)
> + if not ebuild:
> + continue
> + if pkg.sub_slot != ebuild.sub_slot:
> + atoms.append(slot_atom)
> +
> + self._setAtoms(atoms)
> +
> + def singleBuilder(cls, options, settings, trees):
> + return cls(portdb=trees["porttree"].dbapi,
> + vardb=trees["vartree"].dbapi)
> +
> + singleBuilder = classmethod(singleBuilder)
> +
> class DowngradeSet(PackageSet):
>
> _operations = ["merge", "unmerge"]
> --
> 2.26.2
>
>
