Re: /usr-merge status update + next steps
On Tue, 22 Aug 2023 at 10:21, Helmut Grohne wrote: > Let me also put this into numbers. Across all suites, we have around > 2200 binary packages shipping files in aliased locations. If you > disregard systemd units, we're left with 1030 packages. In other words, > more than half of the binary packages shipping files in aliased > locations do so only via systemd units. > > I recognize that various people have repeatedly asked me to consider an > opt-out approach and to look at these numbers. Thanks for your > persistence. Does that also convince others to treat systemd units > separate from the rest? It seems plausible to move systemd units in an > opt-out fashion while moving other files in an opt-in fashion. The main > benefit here is that we could use binNMUs to canonicalizes 1/3 of the > archive. (This is less than half, because a number of packages shipping > systemd units are Arch:all.) > > To me, the risks and cost savings for forcefully moving systemd units > bear a trade-off that is worth considering (despite me earlier having > argued otherwise). Unfortunately, evaluating risk is a subjective > process to some extent and I know that we have quite some disagreement > on how severe these risks are. How can we move forward here? In this > instance, I welcome +1 and -1 style responses and you may send them > directly to me if you want to save the list from such traffic. Thanks for this - not only I agree that immediately converting all packages shipping units is a good idea and a great starting point, but I think (after doing that first bit separately if it's easier to make progress that way) we should go much further, and opt-out-update everything. I do not see any valid reason why any package should be allowed to ship files in the legacy directories once we have a strategy and a tool in place, besides some temporary workaround that may or may not be needed on a package-by-package case due to unforeseen circumstances - in such cases, allowing an opt-out while things are worked out (together with an RC bug to track it) would be fine of course. But to me the most important thing is that the message needs to be: we now have a strategy to approach this, hence everything either gets converted or removed from testing, period. The opt-out to follow your strategy for the pseudo-essential set of course can be treated differently as required. TL;DR: dh_usrmerge should allow an opt-out but default to convert, without any new flag/compat level change required, and anything not using debhelper should get an RC bug at the same time to perform whatever manual step is needed to achieve the equivalent output.
Re: /usr-merge status update + next steps
Control: forwarded -1 https://salsa.debian.org/debian/debhelper/-/merge_requests/108 Control: tags -1 + patch On Sun, Aug 20, 2023 at 11:19:56PM +0200, Michael Biebl wrote: > Related to that: > dh_installsystemd (and the old, deprecated dh_systemd_enable) currently only > consider systemd unit files that are installed to lib/ Thank you Michael and Niels (who privately pointed at the same thing). This is the kind of review that I was hoping for. > One could trick dh_installsystemd by running dh_usrmerge after > dh_installsystemd, but this approach obviously doesn't work, if you change > your package to build with --prefix=/usr, so the files are already in the > canonical location when dh_installsystemd runs. > > So this would need a corresponding change in dh_installsystemd. I guess for > the time being, it would make sense if the tool looked in both paths, at > least as long as the transition is ongoing. You are spot-on. Even before we released bookworm, we had a group of people (including Sebastian Ramacher and myself) advocating for doing this change. As far as I understand the discussion, the main argument against it was that it could encourage people to violate the moratorium. In reality, our refusal to fix this earlier did cause "reverse violations" of the moratorium where files previously shiped below /usr/lib in bullseye would be moved to /lib in bookworm. That happened to boinc-client, cfengine3, nvme-cli, podman, and powerman (see https://subdivi.de/~helmut/dumat.yaml). So I argue that the reasoning was wrong even back then. Keep in mind that Niels clarified that he wasn't really objecting the change, but didn't want to handle its fallout if any. Speaking of fallout, we now have DEP17 and dumat which allow as to quantitaively estimate what may break. * P1 is the main category we see here. This problem arises if we restructure packages and move files between / and /usr. Since we are rather early in the release cycle, not much restructuring has happened yet and all of the restructuring that would cause P1-style file loss, happened for bullseye->bookworm with nothing yet for bookworm->trixie (as of this writing). And since dumat.yaml is updated four times a day, we learn about such problems quickly. * P2 is a problem, but I've files patches for all in-archive instances already. No key packages are affected, so we can upgrade those bugs to RC-severity when problems arise. * P3 has had one instance that Luca Boccassi removed before bookworm, so for systemd units, no P3 problems are left in trixie and beyond. * P4/P5/P6/P7/P8/P9/P10 do not apply. If we were to lift the moratorium just for systemd units right now, we're likely to run into P1 problems due to later package restructuring, but there is little else that may go wrong. Due to these P1 problems, we still have the moratorium and I have repeatedly argued for an opt-in approach to moving files from / to /usr. Let me also put this into numbers. Across all suites, we have around 2200 binary packages shipping files in aliased locations. If you disregard systemd units, we're left with 1030 packages. In other words, more than half of the binary packages shipping files in aliased locations do so only via systemd units. I recognize that various people have repeatedly asked me to consider an opt-out approach and to look at these numbers. Thanks for your persistence. Does that also convince others to treat systemd units separate from the rest? It seems plausible to move systemd units in an opt-out fashion while moving other files in an opt-in fashion. The main benefit here is that we could use binNMUs to canonicalizes 1/3 of the archive. (This is less than half, because a number of packages shipping systemd units are Arch:all.) To me, the risks and cost savings for forcefully moving systemd units bear a trade-off that is worth considering (despite me earlier having argued otherwise). Unfortunately, evaluating risk is a subjective process to some extent and I know that we have quite some disagreement on how severe these risks are. How can we move forward here? In this instance, I welcome +1 and -1 style responses and you may send them directly to me if you want to save the list from such traffic. In any case, I implemented the changes to debhelper to recognize units in /usr. The change does not yet move units to /usr (as that is still prohibited by the moratorium and I don't think we have consensus on that aspect just yet). I am willing to handle the fallout of this change and have implemented the dumat service to quickly diagnose such fallout. Nevertheless, I welcome reviews of the debhelper MR referenced above. Niels already replied to the MR. He'll not interact (review/merge/upload) with the MR and authorized me to do those things (provided I handle possible fallout). Thank you. Helmut
Re: /usr-merge status update + next steps
Am 19.08.23 um 23:14 schrieb Helmut Grohne: ## dh_usrmerge I intend to add a new tool dh_usrmerge to debhelper (not yet implemented). Its purpose is performing the path canonicalization in binary packages. As long as the moratorium is in effect, this helper must not be used. It shall be possible to enable this helper via "Build-Depends: dh-sequence-usrmerge" or "--with=usrmerge". I discussed the possibility of adding this helper to the default sequence via a compatibility level. However, Niels Thykier pointed out that a new compatibility level typically takes 3 years to be adopted and we want 100% adoption before trixie. It will not be mandatory to use this helper. If dropping e.g. --prefix=/ and relying on the /usr default works, that's better. Related to that: dh_installsystemd (and the old, deprecated dh_systemd_enable) currently only consider systemd unit files that are installed to lib/ One could trick dh_installsystemd by running dh_usrmerge after dh_installsystemd, but this approach obviously doesn't work, if you change your package to build with --prefix=/usr, so the files are already in the canonical location when dh_installsystemd runs. So this would need a corresponding change in dh_installsystemd. I guess for the time being, it would make sense if the tool looked in both paths, at least as long as the transition is ongoing. A relevant/related discussion happened already in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1031695 It derailed a bit but I think still has some useful information. Michael OpenPGP_signature.asc Description: OpenPGP digital signature
Re: /usr-merge status update + next steps
Hi Helmut, On 19-08-2023 23:14, Helmut Grohne wrote: I recognize that this is quite a non-standard way to ask for a MBF. Does anyone object to me doing it in this way? I recall I said this before, but just in case. In my opinion (with my Release Team member hat on, but not on behalf of the team) this is fine. What I appreciate about using RC bugs is that in the weird case where the maintainer has good reasons why the package should migrate nevertheless (e.g. severe security implications), he has full power to achieve that. The history is also fully tracked in the BTS. Paul OpenPGP_signature.asc Description: OpenPGP digital signature
