Bug#1058937: /usr-move: Do we support upgrades without apt?
Hi Matthew, On Thu, Dec 21, 2023 at 02:42:56PM +, Matthew Vernon wrote: > On 21/12/2023 09:41, Helmut Grohne wrote: > > > Is it ok to call upgrade scenarios failures that cannot be reproduced > > using apt unsupported until we no longer deal with aliasing? Let me thank David for clarifying what "using apt" means in exactly the way I intended it. As a result, I think the only "no" reply, I've seen thus far is from Matthew here. > I incline towards "no"; if an upgrade has failed part-way (as does happen), > people may then reasonably use dpkg directly to try and un-wedge the upgrade > (e.g. to try and configure some part-installed packages, or try installing > some already-downloaded packages). I incline to agreeing with the scenario you depict. This can reasonably happen. I also think that David made a good case for it being unlikely to manage oneself into the buggy situation that way. And then the consequence is that you lost some possibly important files. If you ended up fiddling with dpkg in a failed upgrade, would it be too much to ask for running dpkg --verify? In the event you see missing files, you may reinstall affected packages and thus have cured the symptoms for your installation. Say we extended release-notes saying that you should dpkg --verify after the upgrade and more so if you happened to use dpkg directly in the process and review the output. Would that address your concern? > It may be that the mitigations necessary are worse than the risk, but I > think the behaviour as described in #1058937 is definitely buggy. I hope we all agree this is buggy. That's not the question. The question at hand is whether this is a bug worth fixing or mitigating. We face a lot of bugs in Debian and assign different severities. Here, the preliminary analysis assigned a rc-severity which generally means it is worth fixing. That's the thing I'm questioning here. Also keep in mind that probably the majority of bullseye -> bookworm upgrades have been performed already. In all those upgrades, nobody ran into the issue and reported it. As David pointed out, it was encountered by actively trying to make it break. It's the silent kind of failure, so it may just have happened without people noticing. Maybe we can all run dpkg --verify on our installations (in particular those upgraded to bookworm or later) and report if they show anything suspicious. Then we can better quantify how likely these issues happen in practice. I note that dpkg --verify does not currently work with --path-exclude. I'm not sure whether that's a bug. Being a user of --path-exclude, I note that I ran dpkg --verify on 5 very different systems and didn't spot unusual things. This is anecdotal evidence and cannot prove the absence of problems though. I'd be very keen to see at least one user reporting such problems in a real upgrade rather than me trying to find problems. Helmut
Bug#1058937: /usr-move: Do we support upgrades without apt?
On Thu, Dec 21, 2023 at 02:42:56PM +, Matthew Vernon wrote: > On 21/12/2023 09:41, Helmut Grohne wrote: > > Is it ok to call upgrade scenarios failures that cannot be reproduced > > using apt unsupported until we no longer deal with aliasing? > > I incline towards "no"; if an upgrade has failed part-way (as does happen), > people may then reasonably use dpkg directly to try and un-wedge the upgrade > (e.g. to try and configure some part-installed packages, or try installing > some already-downloaded packages). You can configure half-installed packages, no problem, this is about unpacking (which is the first step in an install, where only Conflicts and Pre-Depends matter, if you are not deep into dpkg vocabulary) The "try installing" part is less straight forward. In general, you are running into dpkg "features" (e.g. not handling pre-depends) or into dpkg bugs (e.g. #832972, #844300): In the best case your system state became a bit worse and hence harder to "un-wedge". In the worst case a maintainer script has run amok as nobody tested this. But yeah, most of the time you will indeed be lucky and hence come to the unreasonable conclusion that its reasonable to call dpkg directly. Anyway, if your upgrade failed part-way, you are probably in luck given that its more likely the upgrade failed in unpack/configure than in removal – so if you aren't too eager to install more packages by hand but limit yourself to e.g. (re)installing the ones who failed you are fine as apt will have removed the conflictors already for you (or upgraded them, if that can resolve the conflict). But lets assume you are not: As you are driving dpkg by hand you also have the time to read what it prints, which in the problematic case is (as exampled by #1058937): | dpkg: considering removing libnfsidmap-regex:amd64 in favour of libnfsidmap1:amd64 ... | dpkg: yes, will remove libnfsidmap-regex:amd64 in favour of libnfsidmap1:amd64 (and the same for libnfsidmap2:amd64 as well. If your terminal supports it parts of these messages will be in bold) Note that the similar "dpkg: considering deconfiguration of …" which is the result of Breaks relations is not a problematic case. (Also note that this exact situation is indeed another reason why interacting with dpkg by hand is somewhat dangerous as you might not expect packages to be removed from your system while you just told dpkg to unpack something… just remember that the next time you happen to "dpkg -i" some random deb file onto your system.) That is of course no hint that a file might have been lost due to aliasing if you don't know that this could be the case, but on the upside it is not entirely a silent file lose situation either. We could write something in the release notes if someone happens to read them AND also encounters this message. Query your memory: Did you encounter this message before? Nothing in the /usr merge plan makes that particularly more likely to be encountered for a user and not all of the encounters will actually exhibit the file lose. So if you haven't – and I would argue that most people haven't – there is a pretty good chance you wont have a problem in the future either… So, in summary: Yes, there are theoretic relatively easy ways to trigger it with dpkg directly. That isn't the question. The question is if a real person who isn't actively trying to trigger it is likely to run into it by accident (and/or if such a person can even reasonably exist) so that we have to help them by generating work for many people and potentially new upgrade problems for everyone – or if we declare them, existing or not, a non-issue at least for the upgrade to trixie. And on a sidenote: I would advise to reconsider interacting with dpkg too casually – but luck is probably on your side in any case. Best regards David Kalnischkies signature.asc Description: PGP signature
Bug#1058937: /usr-move: Do we support upgrades without apt?
On Thu, Dec 21, 2023 at 03:31:55PM +0100, Marc Haber wrote: > On Thu, Dec 21, 2023 at 11:19:48AM -0300, Antonio Terceiro wrote: > > On Thu, Dec 21, 2023 at 10:41:57AM +0100, Helmut Grohne wrote: > > > Is it ok to call upgrade scenarios failures that cannot be reproduced > > > using apt unsupported until we no longer deal with aliasing? > > > > I think so, yes. I don't think it's likely that there are people doing > > upgrades on running systems not using apt. > > Do those GUI frontends that work via packagekit or other frameworks > count as "using apt"? I explained that in full detail in my mail to the pause-thread: https://lists.debian.org/debian-devel/2023/12/msg00039.html In short: helmuts "apt" (my "APT") includes everything that uses libapt. That is apt, apt-get, python-apt, aptitude, synaptics, everything based on packagekit, … I know of only cupt and dselect which don't count, but I have some suspicion that they would work anyhow. IF you don't run into other problems with them, like that they do not implement Multi-Arch. So this thread is really about: How much are people REALLY fiddling with dpkg directly in an upgrade and can we just say its unsupported – because, at least that is my view, in practice nobody does it and its therefore also completely untested. Case in point: We have this thread not because someone found it while working with dpkg directly even through they had potentially years, but because Helmut ended up triggering an edge case in which apt interacts with dpkg in this way and only after that people looked for how to trigger it with dpkg because triggering it with apt is hard (= as Helmut checked, no package (pair) in current unstable is known to exhibit the required setup). (I will write another mail in another subthread about the finer details of what interacting with dpkg in an upgrade means and what might be problematic if you aren't careful – in general, not just with aliasing) Best regards David Kalnischkies signature.asc Description: PGP signature
Bug#1058937: /usr-move: Do we support upgrades without apt?
Am 21.12.23 um 11:50 schrieb Christoph Berg: Re: Helmut Grohne Is it ok to call upgrade scenarios failures that cannot be reproduced using apt unsupported until we no longer deal with aliasing? If the answer is yes here, we'll close #1058937 (Ben's libnfsidmap1 bug) with no action calling the scenario unsupported. I think we should only deal with problems that can reasonably happen in practice. If an extra hammer is required to hit the problem, we should not spend extra effort on it. A (dist-)upgrade not using apt is very much a corner case/niche use case. I'd be interested if #1058937 can be reproduced using aptitude, though. While the release notes explicitly recommend using apt/apt-get, I do think that dist-upgrades using aptitude should not run into those file loss issues. If aptitude is safe, I'd consider #1058937 a bug, that is not release critical and I'd assign low(er) priority to it. Other issues, like getting all packages updated to move their files to /usr, have higher priority. Michael OpenPGP_signature.asc Description: OpenPGP digital signature
Bug#1058937: /usr-move: Do we support upgrades without apt?
On Thu, 21 Dec 2023 at 15:31:55 +0100, Marc Haber wrote: > Do those GUI frontends that work via packagekit or other frameworks > count as "using apt"? Managing apt/dpkg packages via packagekit uses libapt-pkg6.0 (via /usr/lib/*/packagekit-backend/libpk_backend_apt.so). I don't know whether that's enough to give it the specific desirable behaviour around Conflicts that Helmut is referring to, but I hope it is. Other non-CLI package management like unattended-upgrades is generally in a similar situation, using libapt-pkg or its language bindings, but not the apt(8) or apt-get(8) CLIs specifically. smcv
Bug#1058937: /usr-move: Do we support upgrades without apt?
Hi, On 21/12/2023 09:41, Helmut Grohne wrote: Is it ok to call upgrade scenarios failures that cannot be reproduced using apt unsupported until we no longer deal with aliasing? I incline towards "no"; if an upgrade has failed part-way (as does happen), people may then reasonably use dpkg directly to try and un-wedge the upgrade (e.g. to try and configure some part-installed packages, or try installing some already-downloaded packages). It may be that the mitigations necessary are worse than the risk, but I think the behaviour as described in #1058937 is definitely buggy. Regards, Matthew
Bug#1058937: /usr-move: Do we support upgrades without apt?
On Thu, Dec 21, 2023 at 11:19:48AM -0300, Antonio Terceiro wrote: > On Thu, Dec 21, 2023 at 10:41:57AM +0100, Helmut Grohne wrote: > > Is it ok to call upgrade scenarios failures that cannot be reproduced > > using apt unsupported until we no longer deal with aliasing? > > I think so, yes. I don't think it's likely that there are people doing > upgrades on running systems not using apt. Do those GUI frontends that work via packagekit or other frameworks count as "using apt"? I now that WE recommend using apt in a text console outside of X, but even many of our own users do what their Desktop Environment does, and our downstreams like *b*nt* recommend other ways to upgrade as well. Greetings Marc -- - Marc Haber | "I don't trust Computers. They | Mailadresse im Header Leimen, Germany| lose things."Winona Ryder | Fon: *49 6224 1600402 Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421
Bug#1058937: /usr-move: Do we support upgrades without apt?
On Thu, Dec 21, 2023 at 10:41:57AM +0100, Helmut Grohne wrote: > ## Upgrading using dpkg directly? > > We already have quite a number of packages that use Conflicts to prevent > file loss in upgrades in a very similar way to #1058937 (Ben's > libnfsidmap1 bug) even in released versions of Debian. For instance, > dhcpcd-base's Replaces were upgraded to Conflicts, see #1053657. If you > employ dpkg, you can still experience the problem there. > > Is it ok to call upgrade scenarios failures that cannot be reproduced > using apt unsupported until we no longer deal with aliasing? I think so, yes. I don't think it's likely that there are people doing upgrades on running systems not using apt. If there are, they already need to deal with doing the dpkg calls in the right order anyway -- basically doing the apt dependency resolution by hand -- that this is just another corner case that they need to handle; there could be already Conflicts in there for other reasons than /usr-merge. signature.asc Description: PGP signature
Bug#1058937: /usr-move: Do we support upgrades without apt?
Re: Helmut Grohne > Is it ok to call upgrade scenarios failures that cannot be reproduced > using apt unsupported until we no longer deal with aliasing? > > If the answer is yes here, we'll close #1058937 (Ben's libnfsidmap1 bug) > with no action calling the scenario unsupported. I think we should only deal with problems that can reasonably happen in practice. If an extra hammer is required to hit the problem, we should not spend extra effort on it. Christoph
Bug#1058937: /usr-move: Do we support upgrades without apt?
Hi, this installment serves a dual purpose. Let me first give an update of the status quo and then pose a consensus question on how we want to deal with a particular problem. I Cc d-release@l.d.o as upgrades are an integral part of releases. I Cc d-ctte@l.d.o for advisory feedback with experience due to earlier decisions on merged-/usr. # Status As I detailed earlier, diversions have been proving more difficult than anticipated. I spent significant time on molly-guard to get to a working mitigation and thanks to Francois and Daniel, all of the diverters of /sbin/halt and others have been updated in experimental for wider testing. This is looking promising and passing all testing that has been performed thus far. Meanwhile Chris Hofstaedler and kind folks in Cambridge worked a lot on M-A:same shared file loss (DEP17 P7) and got us down to one (reintroduced) issue. Pending further reintroductions, this aspect is done. Cool! I've since uploaded debhelper and dh_installudev will now also install to /usr. udevdir in udev.pc has been changed in a NEW upload to experimental as well and is expected to hit unstable before too long (thanks Michael and Luca). Earlier, I requested a pause of /usr merges. Since we have a better understanding and solutions that seem to be working now, I am happy for you to move stuff again more widely. For moves involving diversions in any way, consider having me review your change ahead of upload. At the time of this writing, there are 1237 source packages in unstable that still ship something aliased. This is the number we need to get down to 0 for trixie. Of these 860 involve a systemd unit and of these 761 only have systemd units aliased many of which can be converted by a no-change upload due to changed debhelper and systemd.pc behaviour. # The problem with conflicts The idea in DEP17 was to use Conflicts as a mitigation strategy in agreement with a naive reading of Debian policy. As it turns out, that doesn't exactly match reality (#1057199 debian-policy) and there are situations where files can be lost despite Conflicts having been declared. In theory, this subtlety should be irrelevant and unobservable, but aliasing (which breaks dpkg's assumptions) makes this observable. We move a file from / to /usr in $PKGA. AND one of The file is also being moved to a different package (causing DEP17 P1). OR The file is being diverted (causing DEP17 P3). AND The mitigation involves declaring a Conflict for unpack ordering (i.e. M7 for P1 or M18 for P3). AND one of The upgrade is being performed using a direct dpkg invocation without apt in a way that unpacks the package declaring the conflict before the conflicted package is removed. Example: #1058937 (Ben's libnfsidmap1 bug) OR The involved packages declare a mutual conflict (or mutual conflicts + breaks) and therefore apt invokes dpkg as in the earlier point. Example: An earlier version of the molly-guard mitigation declared versioned Breaks for systemd-sysv. This condition is complex, so let me try to break it down into something simpler. We'll have somewhere between 20 and 100 instances of P1 + P3 I guess and we aimed for mitigating most of them using Conflicts (i.e. first two conditions). The horny part is the last one. It basically says that as long as we only ever use apt and avoid mutual conflicts, the issue is not practically observable. That mutual conflict condition is delicate on its own. There are basically two ways to trigger it. The way my molly-guard patch did it was having two versioned Conflicts or Breaks declarations. I checked the archive and there is no instance of any package combination doing this. Hypothetically, another way to trigger this is unversioned Conflicts combined with a package that drops Provides in a later version (thanks David), but we haven't seen any practical instance and I haven't figured a good way to gauge this problem yet. ## Options (combinations possible) When mitigating P1, we can opt for protective diversions (M8) instead of Conflicts (M7), though that is more fragile. When mitigating P3, we can avoid the mutual conflicts. For molly-guard that has been more involved, but it seems manageable. For other packages (that do not need to access diverted files), it becomes simpler. We can restore lost files in a postinst. For this to work, we must duplicate (e.g. hard link) affected files in the data.tar. Example: #1057220 (systemd-sysv upgrade file loss) Note that this approach is not policy compliant for essential packages as they must work when unpacked and this is relevant for gzip being diverted by zutils for instance. We can introduce "barrier" packages (one or more) and have them enforce conflicting packages removed before the conflictor being unpacked (thanks Julian). We can - and this is the crux of the matter - argue that upgrading with bare dpkg is unsupported and you get to keep the pieces if you do so anyway.
