Re: subtle issue with systemd, dnf 'greedy' obsoletes behaviour, and multiple repos
Daniel Mach wrote: > This is nothing that can be fixed in DNF directly. > DNF passes packages to libsolv to resolve the transaction and it can > only cherry-pick packages for the transaction or set transaction flags. Yes, I also think that this needs to be fixed at the libsolv level. > If you want change in behavior, please open a discussion with libsolv's > upstream. I can try it, but the reaction to https://github.com/openSUSE/libsolv/issues/146 makes me sceptical that this one will be handled any better. But one can always try. > If I understand correctly how the sat-solver works, it's populated with > a set of rules before the transaction is resolved and it's not possible > to tweak in on fly as yum did. It could be possible to pre-process the > packages and hide Obsoletes of older packages, but what if the latest > package has broken deps and doesn't get into transaction? DNF in Fedora > has best=0. In that case the older package would get into transaction > and its Obsoletes would be ignored - and that's definitely something we > don't want. I rather think that this needs to be handled with some special SAT rule at the libsolv level (i.e., one such as: the transaction is not allowed to drag in both the old version of the package and the Obsoletes that it contains). It is true that this complexity did not exist with the old yum because it would never pick old versions of packages, but filter them out completely before even trying to compute a transaction (like dnf --best does, essentially). Kevin Kofler ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: subtle issue with systemd, dnf 'greedy' obsoletes behaviour, and multiple repos
Igor Raits wrote: > On Fri, 2020-06-05 at 08:38 +0200, Kevin Kofler wrote: >> The correct solution is to actually fix DNF. It MUST NOT take >> Obsoletes from >> outdated versions of packages (i.e., when there is a newer version in >> the >> repository which removes the Obsoletes) into account. > > https://github.com/openSUSE/libsolv/issues/146 That is actually the other issue, that splitting should be possible, which seems to be somehow partially fixed judging from Adam Williamson's tests. Not sure when and how though, given that libsolv upstream had refused to act on my report (see the comment history). I think the one about Obsoletes from outdated versions of packages is not filed at upstream libsolv yet, only downstream against DNF, and DNF probably cannot fix it without support from libsolv. But unfortunately, given the reaction to my issue 146, I doubt libsolv upstream will handle this any better if I file it there. Kevin Kofler ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: subtle issue with systemd, dnf 'greedy' obsoletes behaviour, and multiple repos
Dne 05. 06. 20 v 8:41 Kevin Kofler napsal(a): The fact that Obsoletes from outdated packages are being considered is the bug: https://bugzilla.redhat.com/show_bug.cgi?id=1748187 but the DNF developers refused to acknowledge this and just closed the bug when the issue was worked around in the font package somehow. This is nothing that can be fixed in DNF directly. DNF passes packages to libsolv to resolve the transaction and it can only cherry-pick packages for the transaction or set transaction flags. If you want change in behavior, please open a discussion with libsolv's upstream. The currently supported flags are: https://github.com/openSUSE/libsolv/blob/master/doc/libsolv-bindings.txt#constants If I understand correctly how the sat-solver works, it's populated with a set of rules before the transaction is resolved and it's not possible to tweak in on fly as yum did. It could be possible to pre-process the packages and hide Obsoletes of older packages, but what if the latest package has broken deps and doesn't get into transaction? DNF in Fedora has best=0. In that case the older package would get into transaction and its Obsoletes would be ignored - and that's definitely something we don't want. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: subtle issue with systemd, dnf 'greedy' obsoletes behaviour, and multiple repos
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On Fri, 2020-06-05 at 08:38 +0200, Kevin Kofler wrote: > Miro Hrončok wrote: > > > On 01. 06. 20 23:10, Adam Williamson wrote: > > > It was actually a bit tricky to come up with a solution for this. > > > I > > > hacked up a minimal reproducer with empty packages, and > > > experimented a > > > bit, and the solution I was able to find that works is to have > > > systemd- > > > udev Obsoletes: systemd < 245.6-1. This seems to correctly clue > > > DNF in > > > to the situation and cause it to leave out anything from 245.4- > > > 1.fc32 > > > in the upgrade. > > > > IMO this is the "correct" solution to the problem. Thanks. > > The correct solution is to actually fix DNF. It MUST NOT take > Obsoletes from > outdated versions of packages (i.e., when there is a newer version in > the > repository which removes the Obsoletes) into account. https://github.com/openSUSE/libsolv/issues/146 > Kevin Kofler > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: > https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org - -- Igor Raits -BEGIN PGP SIGNATURE- iQIzBAEBCgAdFiEEcwgJ58gsbV5f5dMcEV1auJxcHh4FAl7Z7FMACgkQEV1auJxc Hh7BvQ//UQxGN/Cvn0yab5qqBCDnKgtOLB55YnSY1tfqY2moUv+TndNSkbIHHnSw JlIPSBgVcuThDG2NYtX9t9hhYwlEve2se4TBTlI8fg4O8sYsfjbdvD2wpwBPf1cP nHn54izfGDerEbnY6Zo8NRg13ABIKbMYzqlA9r9zh9FeiY34EDK3TR1TzRJg8wnP OV+0bWI8xkyfb8MrSRKc9zo7Jq203OiVFb59G60507i3QyV0y8epg8Fq6+yMxwsZ vjUtaUZBNO+wkWqM1qiY8NQh8rbsgptZuc7vbc78LMsoUvFLWLvs+q8JtKYBYw6l iYXb8w1m6uSRvODnz3kzZnTii0uBf2uwGYuS7+ZPEWagE3mGl50gWqLgMZxT0gQL mrguNJa9nHViKVLgC2MAj2/tffaX6T4/JzAsKLfpaP9HJTqERy3h128j9S/Mqw1q e7kKVOZmpBJSCmRYfz1pYHCJJNCzxbIhs1uD4Guh0BFpFDANCCSg3uu/KzuzJQQZ VlvNGPjY1L8oDhdRFG0E0IgfJdCrsEegtopWFnZ/q0Pw65BjM/hZss0SKpUn5PZO bAyvMn8qQUvjHWGnXVAu7Dy2Y5A2eStbhNrKRnP5hUKDC/NU9s3DPcNOecjqvvsw HXPr+A9XJ03ICk6MiewrbwRSlu4PcWMQHSHI6QjUsAIyVt5Elj8= =Dk2S -END PGP SIGNATURE- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: subtle issue with systemd, dnf 'greedy' obsoletes behaviour, and multiple repos
Zbigniew Jędrzejewski-Szmek wrote: > On Mon, Jun 01, 2020 at 02:10:12PM -0700, Adam Williamson wrote: >> Another factor here is that DNF's behaviour when multiple different >> available packages obsolete an installed package on update is 'greedy'. >> If 'bar', 'moo' and 'meep' all obsolete 'foo' and are available on >> update, DNF will try to install *all three*. This is intentional and >> necessary when e.g. a package is split in two and we want both the new >> packages to be installed in place of the old one. (If it's just >> multiple available versions of the *same* package that all obsolete an >> installed package, DNF will simply try to select the newest one as part >> of the update, which is fine). > > I think this "greedy" behaviour is correct: we rely on this to allow > package splits. There was some discussion whether dnf is in the wrong > here, but it seems to be doing everything correctly. The greediness is correct: https://bugzilla.redhat.com/show_bug.cgi?id=1261034 The fact that Obsoletes from outdated packages are being considered is the bug: https://bugzilla.redhat.com/show_bug.cgi?id=1748187 but the DNF developers refused to acknowledge this and just closed the bug when the issue was worked around in the font package somehow. But this: Adam Williamson wrote: > I was worried that if you had u2f-hidraw-policy and systemd installed, > but not systemd-udev, this Obsoletes: might lead to systemd-udev > getting installed on upgrade. However, it does not seem to: DNF is > happy just updating systemd to 245-6.1.fc32, which is not obsoleted by > the same-versioned systemd-udev, correctly realizing that this > satisfies all constraints and it doesn't need to pull in systemd-udev. is also a bug rather than a feature, it shows that https://bugzilla.redhat.com/show_bug.cgi?id=1261034 is still not properly fixed. This will do the wrong thing on package splits. So I think it is a very bad idea to rely on this DNF bug to work around the other bug. We need to finally get DNF fixed to process Obsoletes in a sane way, the way the old YUM did. Kevin Kofler ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: subtle issue with systemd, dnf 'greedy' obsoletes behaviour, and multiple repos
Miro Hrončok wrote: > On 01. 06. 20 23:10, Adam Williamson wrote: >> It was actually a bit tricky to come up with a solution for this. I >> hacked up a minimal reproducer with empty packages, and experimented a >> bit, and the solution I was able to find that works is to have systemd- >> udev Obsoletes: systemd < 245.6-1. This seems to correctly clue DNF in >> to the situation and cause it to leave out anything from 245.4-1.fc32 >> in the upgrade. > > IMO this is the "correct" solution to the problem. Thanks. The correct solution is to actually fix DNF. It MUST NOT take Obsoletes from outdated versions of packages (i.e., when there is a newer version in the repository which removes the Obsoletes) into account. Kevin Kofler ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: subtle issue with systemd, dnf 'greedy' obsoletes behaviour, and multiple repos
On Tue, 2020-06-02 at 06:37 +, Zbigniew Jędrzejewski-Szmek wrote: > On Mon, Jun 01, 2020 at 02:10:12PM -0700, Adam Williamson wrote: > > Hi folks! openQA caught a subtle issue in a systemd update over the > > weekend, and I thought it would be worth flagging up here both to let > > people know about it and also to see if anyone has a better solution > > than the one I came up with. > > > > The issue affects this systemd update for F32: > > https://bodhi.fedoraproject.org/updates/FEDORA-2020-dd43dd05b1 > > that update provides systemd-245.6-1.fc32 . The stable release of > > Fedora 32 contains systemd-245.4-1.fc32. > > Hi Adam, > > thanks for the deep investigation (and also for setting up onenqa in the > first place, I don't think we'd have caught the issue otherwise.) > > > Another factor here is that DNF's behaviour when multiple different > > available packages obsolete an installed package on update is 'greedy'. > > If 'bar', 'moo' and 'meep' all obsolete 'foo' and are available on > > update, DNF will try to install *all three*. This is intentional and > > necessary when e.g. a package is split in two and we want both the new > > packages to be installed in place of the old one. (If it's just > > multiple available versions of the *same* package that all obsolete an > > installed package, DNF will simply try to select the newest one as part > > of the update, which is fine). > > I think this "greedy" behaviour is correct: we rely on this to allow > package splits. There was some discussion whether dnf is in the wrong here, > but it seems to be doing everything correctly. I can think of potential refinements to dnf's behaviour here, though I imagine the internal ordering of this whole dependency resolution process is a bunch of fun already. As a dumb simplification you could float: "only consider the *highest available version* of any given package as a candidate for obsoleting other packages", though I think it is too simple: what if we're in a case where the newer build of package Z in updates-testing has problems, so outside of Obsoletes: questions dnf would choose to update to the older build of package Z in updates? Still, it feels like an area where it *should* be possible to improve dnf's behaviour somehow. (though I guess this may well be in libsolv anyway). > > the solution I was able to find that works is to have systemd- > > udev Obsoletes: systemd < 245.6-1. This seems to correctly clue DNF in > > to the situation and cause it to leave out anything from 245.4-1.fc32 > > in the upgrade. > > Yep, this sounds like the correct solution. systemd-udev Requires systemd, > and systemd is only provided by systemd, so there doesn't seem to be any > chance of systemd getting uninstalled. Great. I see you went ahead and did that, so it looks like we're good now. Thanks! -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: subtle issue with systemd, dnf 'greedy' obsoletes behaviour, and multiple repos
On Mon, Jun 01, 2020 at 02:10:12PM -0700, Adam Williamson wrote: > Hi folks! openQA caught a subtle issue in a systemd update over the > weekend, and I thought it would be worth flagging up here both to let > people know about it and also to see if anyone has a better solution > than the one I came up with. > > The issue affects this systemd update for F32: > https://bodhi.fedoraproject.org/updates/FEDORA-2020-dd43dd05b1 > that update provides systemd-245.6-1.fc32 . The stable release of > Fedora 32 contains systemd-245.4-1.fc32. Hi Adam, thanks for the deep investigation (and also for setting up onenqa in the first place, I don't think we'd have caught the issue otherwise.) > Another factor here is that DNF's behaviour when multiple different > available packages obsolete an installed package on update is 'greedy'. > If 'bar', 'moo' and 'meep' all obsolete 'foo' and are available on > update, DNF will try to install *all three*. This is intentional and > necessary when e.g. a package is split in two and we want both the new > packages to be installed in place of the old one. (If it's just > multiple available versions of the *same* package that all obsolete an > installed package, DNF will simply try to select the newest one as part > of the update, which is fine). I think this "greedy" behaviour is correct: we rely on this to allow package splits. There was some discussion whether dnf is in the wrong here, but it seems to be doing everything correctly. > the solution I was able to find that works is to have systemd- > udev Obsoletes: systemd < 245.6-1. This seems to correctly clue DNF in > to the situation and cause it to leave out anything from 245.4-1.fc32 > in the upgrade. Yep, this sounds like the correct solution. systemd-udev Requires systemd, and systemd is only provided by systemd, so there doesn't seem to be any chance of systemd getting uninstalled. Zbyszek ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: subtle issue with systemd, dnf 'greedy' obsoletes behaviour, and multiple repos
On 01. 06. 20 23:10, Adam Williamson wrote: It was actually a bit tricky to come up with a solution for this. I hacked up a minimal reproducer with empty packages, and experimented a bit, and the solution I was able to find that works is to have systemd- udev Obsoletes: systemd < 245.6-1. This seems to correctly clue DNF in to the situation and cause it to leave out anything from 245.4-1.fc32 in the upgrade. IMO this is the "correct" solution to the problem. Thanks. -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
