On Mon, Dec 04, 2023 at 01:13:43PM +0100, Helmut Grohne wrote: > David Kalnischkies made me aware […]
Oh, did he? I think he wanted to tell you something else… 😉 As IRC seems to be really bad at transporting complicated things (who had guessed?) and I need to sort my thoughts anyhow let me recount the last few days in a lengthy mail… perhaps that helps me and whoever else might be interested. Disclaimer: /me is an independent APT developer for ~14 years aka super biased if the topic is upgrades and APT. Content warning: Gory details of APT and dpkg internals are described in text. Reader discretion advised. First of, I care only about APT (= meaning everything using libapt, be it apt, aptitude, unattended-upgrades or some python-apt script), not about someone who believes they could perform an upgrade from bookworm to trixie by hand with dpkg¹. I mean, APT can calculated it, so there obviously is a way, but its so tedious for a human being that nobody does that. Sure, there are people who believe "dpkg -i *.deb" would work, but a) that wouldn't be affected by this problem to begin with and b) it doesn't work as you have to spell out pre-depends, the order in which the debs are given is important and last but not least you have to work around a few things in dpkg (e.g. #832972, #844300). So whoever believes they can do it without APT are probably lying to themselves or at the very least wasting a lot of time such experts could use far better to improve APT and/or dpkg for the benefit of all… Source: I stumbled over many bugs while trying to simplify APT down to that dpkg call more than seven years ago, so that is not even a new thing and so not even worthy of grabbing your pitchforks because I supposedly impose new things you haven't adapted to yet… The release notes are actually saying you have to use 'apt', so I am affording a significant amount of leeway here talking about APT. (bootstrapping and such stuff which gets away with not using APT isn't upgrading anything, so this problem with Conflicts is of no concern) So, after clearing that one up, lets focus on the issue at hand: APT tells dpkg about all removes it will eventually schedule ahead of time so technically all (well… some?) Conflicts your yaml includes could be made to exhibit the problem, but APT usually always makes the removal of a package explicit as well and in that case you/we/usr-merge is fine. There is one exception we have to talk about: If a package is scheduled to be removed in one dpkg call and in the next one unpacked. I figured out today that I implemented that sorta by accident… actually, I was working on crossgrades (= a package changing architecture in an upgrade, which contrary to common usage of the word is at least for APT also the case for all <-> any and as such can happen even on a single architecture system) and accidentally its also catching temporary removals which don't change architecture, but are also removing a package before unpacking it (Its Debians fault for trusting me with this stuff…). While the former might be interesting to look at as another source of esoteric problems, lets focus on the temporary removals here: Unversioned conflicts [usually] do not cause temporary removal. They cause "permanent" removal as the packages aren't co-installable (yes, I mentioned you could conflict on a provides from bookworm which is removed in trixie. I don't think that actually happens in reality). A versioned conflict is discouraged by policy and by lintian. Okay, we need it here, so: It can cause it, but only if the conflict is mutual but the packages still co-installable on trixie AND they were also co-installable in bookworm (and for it to actually happen, they have to be installed both from bookworm on the user system of course). If "pkga conflicts pkgb (<< 2)" and "pkgb depends pkga (>= 2)" (like in a package rename perhaps) APT can just "unpack pkgb pkga" and all is fine. The problem you came with was instead "pkga conflicts pkgb (<< 2)" and "pkgb conflicts pkga (<< 2)" (its the same if ONE of them is a breaks, but not if both are) as APT can't just unpack them it has to remove one of them before unpack both. That is what I called a temporary removal as the package is only removed for a very short time. But that short amount of time is already too long if the packages involved are essential, which I suppose is the reason for §6.6 as if APT (as it currently does by happy accident as described above) just tells dpkg that it is allowed to deinstall one of them and unpacks both dpkg can do its §6.6 dance to avoid the actual removal from disk. Yeah! I implemented a generic improvement by accident instead of a bug! Sadly, for usr-merge we need the removals to happen explicitly as dpkg can't untangle the aliasing. The barrier idea achieves this by the way of pre-depends as APT has to spell out pre-depends for dpkg aka it has to configure the barrier package before it can unpack the ones pre-depending on it, so APT schedules "remove pkga; unpack barrier; configure barrier; unpack pkga pkgb". remove and unpack of pkga are not next to each other, so this is "clearly" not a crossgrade and so the removal remains explicit. So, in more natural words the situation might be described as: "pkga and pkgb need both be installed in bookworm. The maintainer moves a file from /lib to /usr/lib while also moving the file from pkga to pkgb in trixie and keeping the ability to co-install pkga and pkgb" Ability here means that they don't have to be installed together. I mention this explicitly as moving a file from -bin to a -data package is usually not affected as the -bin will usually depend rather than break the -data package which eliminates the need for a temporal removal in general. So, I have some hope that the actual amount of packages which need this sort of barrier trickery is countable with two hands (and I don't mean in binary). It happens rather seldom in normal upgrades, but usr-merge changes Breaks to Conflicts, so I suppose the number is not zero like it usually is. > barrier package approach can be used here, but it requires each provider > to have its own barrier package rather than one central barrier package. For the trick to work the amount of barrier packages doesn't matter at all and there are no real requirements except that the package it conflicts with is not (pseudo)essential (or protected), but APT complains loudly if that would be the case so people would notice… Julian and I were suggesting a single one as that /might/ work better with APTs ordering code. If it really does, we will know only on rollout as no amount of testing can actually replicate the ingenuity of the maintainers of 60000+ package with 2+ years each of coming up with interwoven dependency-linking between them all. My personal opinion was to not have it be usrmerge/usr-is-merged as especially the former has dependencies and dependencies mean they could theoretically form a loop. I think the more detached it is the better. Prior art in the form of multiarch-support suggests it can work, although I think that had only the Pre-Depends pointing at it (but a massive amount of them) and not the Conflicts from it… I mentioned in passing that we sorta could do both by having a barrier package provide a bunch of barrier-for-x and have packages pre-depend on those. Its another layer of indirection, but it should work™. A very tiny reason I favor a single/very few barrier packages is that they appear in the NEW packages listing of APT and that just looks silly if its many… (yes, THAT is the problem here to be concerned with… 😉) > For versioned conflicts (where the conflicted version generally is in > bookworm and not in trixie), a single central barrier package might do, > but we can also start with one and split it up later if providing > multiple barriers from the same binary package initially. For instance, > the files diverted by molly-guard would require a barrier package that > also handles bfh-container and progress-linux-container on one side and > systemd, finit, kexec-tools, runit and sysvinit on the other. Such At the time of writing of this mail none of those packages exhibits the required negative dependency setup in unstable. I suppose at least molly-guard will be if a version >> 0.8.2~ enters unstable which breaks/conflicts versioned with systemd-sysv (I only looked at the dependencies, not if the file moving is happening, I just assume it is or we would probably not speak about it). The rest are unversioned in systemd-sysv 255~rc4-2 and hence not co-installable. I suppose you are working with unreleased versions. I haven't looked at a lot of other examples from the yaml, I would predict most of them to be unversioned conflicts through and as such to be unaffected and even many of the versioned conflicts to have redeeming qualities disqualifying them for temporal removal trigger. > If we have a choice between introducing a barrier package and […] Yes. If its reasonably easy and contained dealing with it at the problems source (= individual packages) should be preferred. I have some trust in the barrier idea, but its a relatively high-level trick with drawbacks which doesn't always work (e.g. for essentials). It is also our usual approach even IF we discover that a package runs into a bug in APT/dpkg – the package has to somehow work around it as we usually can fix bugs only for stable+1 (well, given we do this because dpkg doesn't do aliasing, I suppose I have proven my point). Best regards David Kalnischkies ¹ Yes, I know of dselect and cupt. They don't even support M-A. Now have a guess who implemented that ~14 years ago in APT. I warned you that I am biased, didn't I? cupt doesn't use selections as far as I know and dselect uses them, but calls apt for the actual work if dselect-upgrade is any indication… haven't used either so no promises.
signature.asc
Description: PGP signature