Bug#1035904: dpkg currently warning about merged-usr systems (revisited)
Hi, Quoting Steve McIntyre (2023-05-15 02:54:02) > On Mon, May 15, 2023 at 12:24:15AM +0100, Luca Boccassi wrote: > >On Sun, 14 May 2023 at 22:37, Josh Triplett wrote: > > > >> The x86-64 ABI is set. Feel free to make the case to the next > >> architecture designer that their new ABI should have the dynamic linker > >> in `/usr/lib`. That would *not* have the same downsides, as long as > >> everyone agrees on a path. > > > >In practice it is not, though. There are other distributions that > >change PT_INTERP for their own purposes, they've already been listed > >in this thread. And I am still not hearing any concrete, factual use > >case that would be impaired by such a change. I'm beginning to > >seriously think there aren't any? Is that really the case? > > The ABI has been agreed and set down in documentation that *just > about* everybody has been following since its inception. This includes > the most basic set of definitions of what an x86-64 program must look > like, including the interpreter path. If this path is changed, then > *at the most basic level* we'd be making programs that are not valid > by the ABI we've agreed to. This is an *external interface contract*, > not something we should ever consider changing without significant > cross- and inter-project discussion. > > Pointing at gentoo or nixos as examples of projects that have decided > to break compatibility doesn't cut it, I'm afraid. They're well known > for changing fundamental things around Linux and (basically) not caring about > interoperability. That attitude is *not* Debian's. me and Luca have different ideas about how bootstrapping a Debian chroot should look like and I don't want to make an argument *for* changing PT_INTERP here as I think that keeping compatibility with others by following ABI is a good thing and because I think (and hope -- but Helmut is doing that analysis right now) that the debootstrap problem can be solved in a way I envision without changing PT_INTERP. But what I do not understand about the argument against Luca's proposal is: Obviously, with Luca's proposal, binaries from packages built with a different dynamic linker path in them would not work on distributions without merged-/usr symlinks. But if the property of stuff from Debian being able to run on non-Debian non-merged-/usr systems is an important one, then why was it okay to have merged-/usr as the default? Because with merged-/usr we already changed the interface contract for a lot of things because now binaries and libraries can also be found at other locations than on non-merged-/usr systems. A script with a /usr/bin/bash shebang built on and for Debian will not work on a system without the symlinks. So did we not years ago decide, that the result of the "cross- and inter-project discussion" is, that everybody is going merged-/usr and that's why we need it too and that's why it is okay to build a system where binaries and scripts built for it just may not run on those other systems that do not do it? With merged-/usr we already *did* "change fundamental things around" for reasons that are really not clear to me (but which i do not want to discuss here) and as a result did not "care about interoperability" (with those who do not also adopt it). In my own Debian work I so far only got extra work because of merged-/usr and I do not see the benefits (yet) and I was hoping that "changing fundamental things around Linux and (basically) not caring about interoperability" was *not* Debian's attitude but alas here we are. So have we not already burned the bridges to the non-merged-/usr world? Why was it okay back then to say "we can make this change because all other important players are doing merged-/usr so we can/have to as well". And now in the PT_INTERP discussion somehow we care again about those systems? I thought we already had the "cross- and inter-project discussion" about merged-/usr and because the result was "yes, go for it" we did it too. But if that is the case, why do we now care for a subset of the interoperability problems caused by merged-/usr for systems that don't have it? As I said, I don't care much about the PT_INTERP value but I don't understand yet, why this argument about interoperability with non-merged-/usr systems is working now but it didn't wasn't enough to stop another very fundamental change in how we build a Linux distro. Thanks! cheers, josch signature.asc Description: signature
Re: Bug#993161: pam: some remaining changes for DPKG_ROOT
Hi Steve, Quoting Steve Langasek (2022-09-09 07:09:32) > My feedback to you on IRC was that I think it's inappropriate for you to go > package-by-package in the BTS to the packages in the base set expecting > support for a feature that has to my knowledge never been surfaced for > project-wide discussion on debian-devel or similar. > > So if you want to take that discussion to the Technical Committee to ratify > as something that base packages must support, well, I don't think that's the > best use of the TC vs just starting a thread on debian-devel, I agree that this is not the best use of the TC's time. Since we both agree on that and since it has been more than three weeks since our post to d-devel [1] without any concerns or otherwise negative feedback, do you still want to wait for the TC to decide or do you want to cut it short by applying our patch? [1] https://lists.debian.org/166289720850.2390.3729551131862514967@localhost > but it does satisfy my expectation that there be a project-level review of > the design prior to obligating base package maintainers to support this > feature. We are working on some changes to Debian policy in #1020323 but we are not planning to put any "must" statements in the text. Our intention is not to obligate anybody to do anything other than apply reasonable patches that we provide. If it breaks, it is not your responsibility to fix it. Since the DPKG_ROOT variable is empty during normal installations I hope you agree that our patch will not be able to introduce any bugs during normal package installation. If the chrootless bootstrapping should fail in the future, it is not the obligation of package maintainers to fix it. We have said so multiple times in the past already... josch signature.asc Description: signature
Re: Bug#1019409: Bug#993161: pam: some remaining changes for DPKG_ROOT
Hi, Quoting Steve Langasek (2022-09-10 22:16:55) > > > > For the record I do not consider this an override requiring a > > > > supermajority and would abide by a majority TC decision. > > > > Thank you for your input. The TC can just issue advice after reviewing > > > the proposed changes, in this case. An alternative would be to word the > > > resolution such that it counts as advice if we have a simple majority > > > and an override if we have a supermajority. I'd prefer the former, but > > > it would be good to hear from Helmut about it. > > > AIUI, Steve's objection is substantially that this is quite an invasive > > change to make across our toolchain, and should be discussed on debian-devel > > before just being implemented package-by-package (rather than any particular > > objection to the approach). Is that correct? > > I think that's a fair characterization, yes. > > I support the goal of making it easier to bootstrap ports. I also don't > even see a cleaner way to accomplish this than what's proposed. But I think > there's a duty, when making distro-level changes, to have a project-level > discussion about what's being proposed and why, and to get consensus on it, > not just file a bunch of bug reports on individual packages. I think there's a duty, when maintaining a package, to at least send a short reply to bugs against your package and even more so, if pinged multiple times and your co-maintainer explicitly waiting for you and thus this non-action resulting in blocking other people's work. We invoked the TC not because we did not want to discuss on d-devel but because you have kept silent since February 2021 when we filed our initial bug #983427 against pam. In hindsight, we should've written to d-devel, yes. Helmut and myself are working on a mail to send to d-devel to get this done now in the sense of "better late than never". We didn't think that such a mail was necessary because there are only 10 source packages (including pam) that require the DPKG_ROOT variable in their maintainer script for this mechanism to work (that's why I wouldn't classify this as "distro-level change") and because the required changes to maintainer scripts result in zero behaviour changes on anything that is not early-bootstrap. >From my side, I'd be fine with the TC pausing any action on this issue and waiting for our mail to d-devel first. This is assuming that if there is no opposition to the DPKG_ROOT idea, that Steve then also has no objection against merging our patch. Helmut, what do you think? Thanks! cheers, josch signature.asc Description: signature
Re: tech-ctte: More specific advice regarding merged-/usr and implications of #978636
Quoting Josh Triplett (2022-07-26 17:29:43) > Johannes Schauer Marin Rodrigues wrote: > > I just remembered a pending MR that I'd say is a little more important than > > "good to have": > > > > https://salsa.debian.org/debian/debianutils/-/merge_requests/21 > > > > Without this one-line change, /etc/shells on merged-/usr systems will have > > some > > shells listed twice and some shells (like /usr/bin/bash) missing. The reason > > is, that dpkg-realpath doesn't support merged-/usr and thus a workaround is > > needed. > > Over the years, I've seen a few proposals floated to consider dropping > /etc/shells; this would just require dropping pam_shells.so from > /etc/pam.d/chsh. That would also have the side effect of solving this > problem, and making one less thing requiring maintainer scripts. > > Would it be worth reopening that discussion, and evaluating the net > value of continuing to maintain /etc/shells? "one less thing requiring maintainer scripts" is not an argument anymore because update-shells from debianutils is precisely the mechanism that avoids maintainer scripts for updating /etc/shells. It does so by using a dpkg trigger on /usr/share/debianutils/shells.d instead. We were able to remove maintainer scripts updating /etc/shells (partly or completely) from bash as well as dash thanks to update-shells in debianutils. Thanks! cheers, josch signature.asc Description: signature
Re: tech-ctte: More specific advice regarding merged-/usr and implications of #978636
Hi tech-ctte Quoting Luca Boccassi (2022-07-25 15:22:10) > There's also a pending MR for reportbug that would be good to have, but > doesn't need to block other progress: > > https://salsa.debian.org/reportbug-team/reportbug/-/merge_requests/77 I just remembered a pending MR that I'd say is a little more important than "good to have": https://salsa.debian.org/debian/debianutils/-/merge_requests/21 Without this one-line change, /etc/shells on merged-/usr systems will have some shells listed twice and some shells (like /usr/bin/bash) missing. The reason is, that dpkg-realpath doesn't support merged-/usr and thus a workaround is needed. Given the recent CTTE ruling over debianutils (#994275) and since the MR from five months ago has no maintainer activity, I wanted to ask you for informal advice on this. Should I just file a bug and NMU with maximum delay? Or is application of this fix deemed more important? Thanks! cheers, josch signature.asc Description: signature
Re: tech-ctte: More specific advice regarding merged-/usr and implications of #978636
Hi Luca, Quoting Luca Boccassi (2022-07-25 15:22:10) > usrmerge has migrated to testing, and the deboostrap MR has been merged and > just uploaded to unstable. Do you want to do the equivalent upload for > mmdebstrap now, so that it's ready as well? thank you for the heads-up! Yes, the mmdebstrap upload will not be optional because the upload of debootstrap 1.0.127 breaks the mmdebstrap autopkgtest and to make sure that debootstrap migrates, I'll have to do an mmdebstrap upload adapting its autopkgtest to the new debootstrap behaviour. > Once debootstrap 1.0.127 has migrated to testing we can arrange an upload to > bullseye-backports, and then ensure all the buildds are using that new > version. Then i-s-h can be updated. At the point that i-s-h is uploaded, I'll have to do yet another mmdebstrap upload because that again will break the expectations I encoded in mmdebstrap's autopkgtests. I am running the mmdebstrap autopkgtest daily on jenkins so I'll definitely get notified once the i-s-h upload breaks the autopkgtest once again. :) Thanks! cheers, josch signature.asc Description: signature
Re: tech-ctte: More specific advice regarding merged-/usr and implications of #978636
Hi, Quoting Luca Boccassi (2022-07-18 21:03:14) > It was renamed following a request on #debian-ftp while it was in NEW, as the > feedback was that 'usrmerge' and 'usrmerged' were too similar and thus easily > confused. The 'usrmerged' one can be disregarded and will be de-crufted. I think that's very sensible. > > Does that sound okay to you and does the patch look like it does the right > > thing? > > Yes, without knowing much about mmdebstrap, the diff looks good to me. > I'd only ask that in the comment of the script hooks/no-merged- > usr/setup00.sh if you could please mention explicitly that it creates an > unsupported system. Maybe even print a warning when it's called. Running mmdebstrap with --hook-dir=hooks/no-merged-usr will now print the following to stderr: Warning: starting with Debian 12 (Bookworm), systems without merged-/usr are not supported anymore > Also, I assume it is creating the metapackage on-the-fly because it doesn't > have the downloaded packages available at that point? Not a problem, just > double checking. That is correct. But I think it's not ideal if mmdebstrap creates a chroot containing a package that doesn't come from the archive. I now extended the hook script such that calling mmdebstrap with --hook-dir=hooks/merged-usr will first install the custom built metapackage, then install the essential packages and then upgrade to the real usr-is-merged package. I think then the roadmap is to release debootstrap with #71 merged, then upload init-system-helpers that depends "usrmerge | usr-is-merged" and then I test and upload mmdebstrap including those hook scripts. Since those are just hooks, nothing stops people from using them with the mmdebstrap version currently in unstable and testing, so nothing should be blocked by this in case I should need longer to release an mmdebstrap version shipping these hook scripts. Thanks! cheers, josch signature.asc Description: signature
Re: tech-ctte: More specific advice regarding merged-/usr and implications of #978636
Hi all, Quoting Luca Boccassi (2022-07-18 16:42:03) > On Mon, 2022-07-18 at 16:04 +0200, Johannes Schauer Marin Rodrigues > wrote: > > Hi Luca & Helmut, > > > > Quoting Helmut Grohne (2022-07-18 14:32:31) > > > On Sun, Jul 17, 2022 at 12:56:14AM +0100, Luca Boccassi wrote: > > > > A MR is pending for debootstrap [1] to make use of this new package and > > > > file flag when --variant=buildd is passed, so that buildds can continue > > > > to be unmerged-usr until the CTTE says otherwise, as per decision > > > > above. > > > > I don't think I have access to the mail to which Helmut replied but I guess > > [1] > > is this merge request? > > > > https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/71 > > Yes - the thread with the first mail is on the CTTE mailing list: > https://lists.debian.org/debian-ctte/2022/07/threads.html > > btw, any reason you dropped the list from CC? I just didn't think that mmdebstrap-specific stuff would be on-topic on tech-ctte. Luca agreed for me to send my full reply to their mail back to the list in public. > > > Can I ask you to also work with the mmdebstrap maintainer to make sure > > > that merged-/usr and opting out of it works well there? Specifically, I'd > > > like to have it work without pulling the usrmerge package (and perl > > > module dependencies) both in a merged and unmerged (buildd) layout. > > > mmdebstrap used to have switches, but now mostly ignores /usr-merge (and > > > thus doesn't merge). If we do nothing, it'll pull in usrmerge and its > > > perl module dependencies with no easy way to opt out. I'm not sure what > > > the solution is, but I'm positive that you'll find a reasonable > > > compromise once talking to one another. mmdebstrap has tried to stay away > > > from custom set-up code, which makes this slightly non-trivial. > > > > [SNIP RANT] > > > > Anyways, I think what makes the situation a bit better as far as mmdebstrap > > is > > concerned is, that we only need to be able to build an unmerged chroot until > > the transition is complete, correct? Does this mean that the following can > > be a > > solution for everybody that wants to build an unmerged buildd chroot? > > > > mmdebstrap --variant=buildd \ > > --setup-hook='echo "this system will not be supported in the future" > > > "$1/etc/unsupported-skip-usrmerge-conversion"' \ > > unstable chroot.tar > > > > I can throw this into one of the hook scripts in /usr/share/mmdebstrap/hooks > > and then the CLI invocation would be shorter if desired but I guess this > > will > > only be run from scripts anyways? This will still install the usrmerge > > package > > and its dependencies though, more on that below. > > If you can also add set it up so that usr-is-merged gets installed, > then you avoid the usrmerge pkg and its extra dependencies as well. But > yes that should work. Yes, that seems useful. But then usr-is-merged being installed would be a lie. I guess we accept that hack? Also, what is the relationship between the usr-is-merged package and the usrmerged package? Both were/are built by src:usrmerge but its changelog doesn't indicate when or why the naming changed so I'm left a bit confused about the relationship between these two packages. Since your deboootstrap merge request changed from usrmerged to usr-is-merged I guess the latter is the correct choice? > > > > Once deboostrap is updated and deployed on the buildds, a "usrmerge | > > > > usr-is-merged" dependency will be added to the Priority: essential > > > > init-system-helpers package, and uploaded to unstable to complete the > > > > transitions for all installations that are older than buster or that > > > > have been manually installed as unmerged-usr. Any system not upgraded > > > > > > I think this setup is prone to breakage. While apt prefers the first > > > alternative, it doesn't have to pick it and other resolvers are even > > > more prone to picking non-default alternatives. Imagine one of the perl > > > modules being uninstallable. Even apt will pick the usr-is-merged > > > package in that scenario. Are you ware of this potential problem? Do you > > > have any ideas on how to handle it? To be clear: I do not think this > > > aspect to be a show-stopper. Let's just talk about it on a technical > > > level. > > > > > > Let me propose something strange without having thought through the > > > implications
