Bug#924401: #924401 base-files fails postinst when base-passwd is unpacked
Hi, for everybody else who is reading this bug and was not reading debian-devel at the same time, here is the thread that Tim started there for more context: https://lists.debian.org/alpine.deb.2.20.2102211635290.6...@einstein.home.woodall.me.uk Quoting Tim Woodall (2021-02-22 18:28:56) > On Mon, 22 Feb 2021, Helmut Grohne wrote: > > On Mon, Feb 22, 2021 at 07:33:10AM +, Tim Woodall wrote: > >> A. /etc/passwd is part of base-passwd's interface and base-files is > >>right in relying on it working at all times. Then base-passwd is rc > >>buggy for violating a policy must. Fixing this violation is > >>technically impossible. > >> > >> > >> I seem to have hit this same issue independently. > >> > >> Could you explain why "Fixing this violation is technically impossible" > > > > The requirement here is that base-passwd needs to work when unpacked. > > The only way to make that work is making /etc/passwd a conffile. That > > would technically be possible, but it would be very annoying, because > > this file is different on virtually any Debian installation. So we > > cannot make it a conffile in practice. The next bet would be ensuring > > that base-passwd.postinst is run before other packages' postint somehow. > > Such an ordering mechanism does not exist at present and it would be > > prone to dependency loops. > > > >> As far as I can see, making base-passwd not essential, only required, > >> and then making passwd and base-files pre-depend on base-passwd the > >> system seems to bootstrap /etc/passed and /etc/group OK. > > > > What you write is almost certainly self-contradictory. base-files is > > essential. Anything it depends on (including base-passwd in your > > scenario) is pseudo-essential and thus inherits all the same > > requirements except for actually being essential. You gained nothing. > > And you didn't explain how you'd make base-passwd non-essential. > > > >> That also seems to conform to the debian policy. The oddity is that > >> base-files and passwd only actually need to depend on base-passwd, not > >> pre-depend on it as they only use /etc/passwd and /etc/group in the > >> postinst scripts but the debian policy doesn't seem to consider this > >> case. > > > > They don't have to depend on base-passwd at all, because dependencies on > > essential packages should be omitted. > > If base-passwd wasn't essential then passwd and base-files should depend > on it. > > If passwd and base-files (pre)depend on base-passwd then base-passwd > will be configured before passwd or base-files postinst is run. In your mails to debian-devel and to this bug you are mixing some orthogonal problems. Please note, that Debian policy does *not* describe how to create a Debian chroot from zero. Specifically it does not cover the situation in which a package was *never* configured. So when it says that packages in the Essential:yes set must work even without being configured, then this is talking about package upgrades where packages from the Essential:yes set can find themselves in the configure stage but still have to work. Yes, this should be clarified in Debian policy. Since Debian policy does *not* talk about the debootstrap scenario, you can *not* make the argument of "you have to add the Depends because Debian policy says so". Remember, that Debian policy documents existing practice, so if we want to at some point document the debootstrap scenario in Debian policy, we first have to make it work properly in the first place. For reasons that Colin and Helmut already listed, I don't think adding more dependencies between Essential:yes packages or dropping Essential:yes is the right way forward. If you start adding dependencies between packages in the Essential:yes set, depending on what they use, you will very quickly end up with a lot of circular dependencies and you are back at square one. As the author of one of the debootstrap programs (mmdebstrap) I would welcome you investing time in properly solving this problem. For this I suggest you open a bug against dpkg because I think it's there that this has to be solved. > > I suggest that you detail on the practical issue you have been hitting. > > Doing so allows evaluating prospective solutions against all relevant > > use cases. > > > > What I am doing - which seems similar to what others are doing with > other tools, is to: > > 1. unpack apt, dpkg and debian-archive-keyring data.tar.xz into tmpdir/ > (This provides a set of config that apt needs to run) > > 2. create a sources.list in tmpdir and touch image/var/lib/dpkg/status > (dpkg needs this file to exist) > > 3. apt-get install -o RootDir=tmpdir -d $ESSENTIAL > (This resolves all of the dependencies and downloads every package that > needs to be installed. I cannot see a way to get apt to do this without > jumping through the hoops of step 1 and 2 as --reinstall doesn't > re-download dependencies) > > 4. unpack the data.tar.xz from every deb downloaded in 3 into image/ >
Processed (with 1 error): merge
Processing commands for cont...@bugs.debian.org: > reassign 983251 base-passwd Bug #983251 [passwd] passwd is missing dependencies Bug reassigned from package 'passwd' to 'base-passwd'. No longer marked as found in versions shadow/1:4.5-1.1. Ignoring request to alter fixed versions of bug #983251 to the same values previously set > forcemerge 924401 983251 Bug #924401 [base-passwd,base-files,debian-policy] base-files fails postinst when base-passwd is unpacked Unable to merge bugs because: package of #983251 is 'base-passwd' not 'base-passwd,base-files,debian-policy' Failed to forcibly merge 924401: Did not alter merged bugs. > End of message, stopping processing here. Please contact me if you need assistance. -- 924401: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=924401 983251: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=983251 Debian Bug Tracking System Contact ow...@bugs.debian.org with problems
Bug#924401: #924401 base-files fails postinst when base-passwd is unpacked
On Mon, 22 Feb 2021, Helmut Grohne wrote: On Mon, Feb 22, 2021 at 07:33:10AM +, Tim Woodall wrote: A. /etc/passwd is part of base-passwd's interface and base-files is right in relying on it working at all times. Then base-passwd is rc buggy for violating a policy must. Fixing this violation is technically impossible. I seem to have hit this same issue independently. Could you explain why "Fixing this violation is technically impossible" The requirement here is that base-passwd needs to work when unpacked. The only way to make that work is making /etc/passwd a conffile. That would technically be possible, but it would be very annoying, because this file is different on virtually any Debian installation. So we cannot make it a conffile in practice. The next bet would be ensuring that base-passwd.postinst is run before other packages' postint somehow. Such an ordering mechanism does not exist at present and it would be prone to dependency loops. As far as I can see, making base-passwd not essential, only required, and then making passwd and base-files pre-depend on base-passwd the system seems to bootstrap /etc/passed and /etc/group OK. What you write is almost certainly self-contradictory. base-files is essential. Anything it depends on (including base-passwd in your scenario) is pseudo-essential and thus inherits all the same requirements except for actually being essential. You gained nothing. And you didn't explain how you'd make base-passwd non-essential. That also seems to conform to the debian policy. The oddity is that base-files and passwd only actually need to depend on base-passwd, not pre-depend on it as they only use /etc/passwd and /etc/group in the postinst scripts but the debian policy doesn't seem to consider this case. They don't have to depend on base-passwd at all, because dependencies on essential packages should be omitted. If base-passwd wasn't essential then passwd and base-files should depend on it. If passwd and base-files (pre)depend on base-passwd then base-passwd will be configured before passwd or base-files postinst is run. I suggest that you detail on the practical issue you have been hitting. Doing so allows evaluating prospective solutions against all relevant use cases. What I am doing - which seems similar to what others are doing with other tools, is to: 1. unpack apt, dpkg and debian-archive-keyring data.tar.xz into tmpdir/ (This provides a set of config that apt needs to run) 2. create a sources.list in tmpdir and touch image/var/lib/dpkg/status (dpkg needs this file to exist) 3. apt-get install -o RootDir=tmpdir -d $ESSENTIAL (This resolves all of the dependencies and downloads every package that needs to be installed. I cannot see a way to get apt to do this without jumping through the hoops of step 1 and 2 as --reinstall doesn't re-download dependencies) 4. unpack the data.tar.xz from every deb downloaded in 3 into image/ 5. repeat step 2 but in image/ 6. chroot image/ apt-get update apt-get install $ESSENTIAL And expect that to create a fully configured system. For buster at least, the _only_ sticking point is apt not configuring base-passwd early enough so one or the other of base-files and passwd postinst fails. I've created local copies of those two that have an explicit dependency on base-passwd and the problem is solved. I haven't tried with unstable or bullseye yet but I don't immediately see any other issues. Tim.
Bug#924401: #924401 base-files fails postinst when base-passwd is unpacked
On Mon, Feb 22, 2021 at 07:33:10AM +, Tim Woodall wrote: > A. /etc/passwd is part of base-passwd's interface and base-files is >right in relying on it working at all times. Then base-passwd is rc >buggy for violating a policy must. Fixing this violation is >technically impossible. > > > I seem to have hit this same issue independently. > > Could you explain why "Fixing this violation is technically impossible" The requirement here is that base-passwd needs to work when unpacked. The only way to make that work is making /etc/passwd a conffile. That would technically be possible, but it would be very annoying, because this file is different on virtually any Debian installation. So we cannot make it a conffile in practice. The next bet would be ensuring that base-passwd.postinst is run before other packages' postint somehow. Such an ordering mechanism does not exist at present and it would be prone to dependency loops. > As far as I can see, making base-passwd not essential, only required, > and then making passwd and base-files pre-depend on base-passwd the > system seems to bootstrap /etc/passed and /etc/group OK. What you write is almost certainly self-contradictory. base-files is essential. Anything it depends on (including base-passwd in your scenario) is pseudo-essential and thus inherits all the same requirements except for actually being essential. You gained nothing. And you didn't explain how you'd make base-passwd non-essential. > That also seems to conform to the debian policy. The oddity is that > base-files and passwd only actually need to depend on base-passwd, not > pre-depend on it as they only use /etc/passwd and /etc/group in the > postinst scripts but the debian policy doesn't seem to consider this > case. They don't have to depend on base-passwd at all, because dependencies on essential packages should be omitted. I suggest that you detail on the practical issue you have been hitting. Doing so allows evaluating prospective solutions against all relevant use cases. Helmut
Bug#983304: please document "Protected" field
Hi Tomas, That is indeed vague. However, `Protected` has already been documented in testing under deb-controls [0]. I hope this helps. [0] https://manpages.debian.org/testing/dpkg-dev/deb-control.5.en.html *--Sergio Moraes* *Data Management Analyst* On Mon, Feb 22, 2021 at 12:27 AM Tomas Pospisek wrote: > Source: debian-policy > Version: 4.5.1.0 > Severity: wishlist > > In Julian Andres Klode's blog I've [1] glimpsed: > > > New features > > [...] > > The Protected field is now supported. It replaces the previous Important > > field and is like Essential, but only for installed packages (some minor > > more differences maybe in terms of ordering the installs). > > So I've tried to find out what the "Protected" field is for. The only > info about it that I could find is from `man 1 dpkg`: > > > Protected packages contain mostly important system boot infrastructure. > > Removing them might cause the whole system to be unable to boot, so use > > with caution. > > That seems a bit vague and sparse. > > Is there maybe more rigid information for what *exactly* the "Protected" > field should and will be used? > > Either way, if there's a new control field, then I think it should get > documented in the `debian-policy`? > > Thanks, > *t > > [1] https://blog.jak-linux.org/2021/02/18/apt-2.2/ > > -- System Information: > Debian Release: 10.8 > APT prefers stable > APT policy: (500, 'stable') > Architecture: amd64 (x86_64) > Foreign Architectures: i386 > > Kernel: Linux 4.19.0-13-amd64 (SMP w/8 CPU cores) > Kernel taint flags: TAINT_WARN, TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE > Locale: LANG=de_CH.utf8, LC_CTYPE=de_CH.utf8 (charmap=UTF-8), > LANGUAGE=de_CH:de (charmap=UTF-8) > Shell: /bin/sh linked to /bin/dash > Init: systemd (via /run/systemd/system) > LSM: AppArmor: enabled > >
Bug#983304: please document "Protected" field
Hi! On Mon, 2021-02-22 at 11:30:08 +0100, Julian Andres Klode wrote: > On Mon, Feb 22, 2021 at 09:23:00AM +0100, Tomas Pospisek wrote: > > Source: debian-policy > > Version: 4.5.1.0 > > Severity: wishlist > > In Julian Andres Klode's blog I've [1] glimpsed: > > > > > New features > > > [...] > > > The Protected field is now supported. It replaces the previous Important > > > field and is like Essential, but only for installed packages (some minor > > > more differences maybe in terms of ordering the installs). > > > > So I've tried to find out what the "Protected" field is for. The only > > info about it that I could find is from `man 1 dpkg`: > You can also read the spec [1] > > [1] https://wiki.debian.org/Teams/Dpkg/Spec/ProtectedField I've updated that to point to the RFC thread on debian-devel and to the file shipped by dpkg (/usr/share/doc/dpkg/protected-field.txt), which are both better resources. In general once these get implemented in dpkg, the wiki spec pages have stopped being the core canonical source of information, and are kept for historical purposes or to keep information that is out-of-scope for dpkg itself (such as Debian-specific policy or implementation tracking, etc). But I guess that should be clearly reflected there or the pages kept in sync. I'll try to do both, and perhaps incorporate any of the following paragraphs, that might no be already covered from information in the pre-existing docs once merged. > What you use them for is obviously not fixed. What we have so far are: > > ## Essential for some use cases > > The init package uses the old Important: yes to be harder to remove so > you don't end up bricking your system into an unbootable one. > > That's the "Essential only for some use cases" use case, because init > obviously should not be Essential because we don't want it in an app > container, but we do want it to be Essential when it's already installed. > > The same applies to e2fsprogs, which is also marked like that, because > duh, you do not want to be able to remove e2fsprogs on a real system, > but again you don't want it installed automatically in a container. > > > ## Weird hacks in low-level packages > > As another example, we marked libgcc-s1 as Protected, because it Replaces: > libgcc1 but can't Conflict with it, so removing libgcc-s1 would remove > the library entirely, so we had to protect it. > > ## Local metapackages > > The original use case was local system configuration meta packages, > where you define a my-machine-foo meta package, and have it depend on > all packages in the system and mark that as Protected: yes[2]; back then > we used XB-Important: yes (well we still do, you gotta set both for > backwards compatibility with stable atm really). Because you then mark > all other packages auto and you then absolutely do not want to lose your > meta package :D > > [2] > https://blog.jak-linux.org/2012/01/24/managing-system-package-selections-using-custom-meta-packages/ > > ## De-essentializing Essential packages > > Packages that were Essential can be downgraded to Protected and > installed systems won't show any change in behavior (avoiding > regressions), whereas the package will no longer be "essential" > on new systems. Thanks, Guillem
Bug#924401: #924401 base-files fails postinst when base-passwd is unpacked
On Mon, 22 Feb 2021, Colin Watson wrote: On Mon, Feb 22, 2021 at 07:33:10AM +, Tim Woodall wrote: As far as I can see, making base-passwd not essential, only required, and then making passwd and base-files pre-depend on base-passwd the system seems to bootstrap /etc/passed and /etc/group OK. That also seems to conform to the debian policy. The oddity is that base-files and passwd only actually need to depend on base-passwd, not pre-depend on it as they only use /etc/passwd and /etc/group in the postinst scripts but the debian policy doesn't seem to consider this case. base-passwd is independently essential to the functioning of the system as it currently stands, not just because base-files and passwd need it. As such I do not consider it correct to remove the Essential flag from base-passwd and won't be doing so. My view on this is that policy's definition of Essential should, for the time being, be refined to indicate that it only applies after the package has been configured at least once. Independently, I agree with other comments on this bug to the effect that it would be useful to extend dpkg such that initial copies of the essential files provided by base-passwd could be written without having to run base-passwd's maintainer scripts (something like the ability to provide an initial version of a configuration file without making it a full conffile), which would then allow simplifying the process, but I see no reason to block policy refinements on that; the policy manual is a living document and we can always update it again later once it's possible to simplify the bootstrapping process further. This can work. but I think there's a separate piece of metadata that is missing. Essential: yes actually encodes two pieces of information: 1. That the package is available and usable even when it's not configured. 2. That you can rely on the existence of the package and so do not need to include an explicit dependency to it. Making base-passwd not essential would mean that every package that depends on it should include at least a transitive dependency to it. That seems ludicrous - so I agree with Colin that base-passwd should be Essential. But that then leaves the problem that essential also encodes 1. I think there should be another piece of metadata "ConfigureOptional: yes" (or something that encompasses packages that do not need configuring to work) That way, two, completely isolated packages A and B could be marked "ConfigureOptional: yes" and use each other in their postinst scripts (and declare themselves as dependent on each other) Apt would then need to understand that circular dependencies can be broken by unpacking, but not configuring ConfigureOptional packages. There are a number of ways of solving the bootstrap tools problem of automatically determining the correct order to configure the packages without having to remove base-passwd from Essential. 1. make passwd and base-files depend on base-passwd without removing the Essential: yes flag from base-passwd (Possibly by updating to policy manual to state that Essential: yes packages CAN (Should?) state a dependency on other Essential: yes packages. This would go in tandem with Colin's suggestion to update the policy to state that Essential: yes merely requires that the package guarantees to be available if it's been configured at least once - and anybody outside of the essential set itself can assume that) 2. update dpkg to explicitly create these files if they don't exist, possibly by way of a type of "create only" conffile. This would have general use as quite a lot of packages use a postinst script to install configuration for the first time. The first example I found was for python3: [ -f /etc/python3/debian_config ] || new_config_file base-passwd would then use this facility and would strictly conform to the current Essential: yes policy. FWIW, it is possible to bootstrap a "system" that only includes 14 of the 23 Essential: yes packages in Buster that has everything you need to install the remaining 9 without having bootstrapped them from outside the system. It is, I presume, missing something that would prevent anything from booting and it's only usable via a chroot on an already booted system. And there's absolutely nothing with "Essential:" anything other than yes, which suggests that another possibility would be to have Essential: yes and Essential: needs_configure but option 2 above feels like the correct solution. I've never looked at dpkg source, but if it doesn't look too daunting I might try to put together at least a proof of concept for the idea of "only create, never modify" conffiles Regards, Tim.
Bug#924401: #924401 base-files fails postinst when base-passwd is unpacked
On Mon, Feb 22, 2021 at 07:33:10AM +, Tim Woodall wrote: > As far as I can see, making base-passwd not essential, only required, > and then making passwd and base-files pre-depend on base-passwd the > system seems to bootstrap /etc/passed and /etc/group OK. > > That also seems to conform to the debian policy. The oddity is that > base-files and passwd only actually need to depend on base-passwd, not > pre-depend on it as they only use /etc/passwd and /etc/group in the > postinst scripts but the debian policy doesn't seem to consider this > case. base-passwd is independently essential to the functioning of the system as it currently stands, not just because base-files and passwd need it. As such I do not consider it correct to remove the Essential flag from base-passwd and won't be doing so. My view on this is that policy's definition of Essential should, for the time being, be refined to indicate that it only applies after the package has been configured at least once. Independently, I agree with other comments on this bug to the effect that it would be useful to extend dpkg such that initial copies of the essential files provided by base-passwd could be written without having to run base-passwd's maintainer scripts (something like the ability to provide an initial version of a configuration file without making it a full conffile), which would then allow simplifying the process, but I see no reason to block policy refinements on that; the policy manual is a living document and we can always update it again later once it's possible to simplify the bootstrapping process further. -- Colin Watson (he/him) [cjwat...@debian.org]
Bug#983304: please document "Protected" field
On Mon, Feb 22, 2021 at 09:23:00AM +0100, Tomas Pospisek wrote: > Source: debian-policy > Version: 4.5.1.0 > Severity: wishlist > > In Julian Andres Klode's blog I've [1] glimpsed: > > > New features > > [...] > > The Protected field is now supported. It replaces the previous Important > > field and is like Essential, but only for installed packages (some minor > > more differences maybe in terms of ordering the installs). > > So I've tried to find out what the "Protected" field is for. The only > info about it that I could find is from `man 1 dpkg`: You can also read the spec [1] [1] https://wiki.debian.org/Teams/Dpkg/Spec/ProtectedField Let's start declaring what this field is: Essential packages are automatically installed by apt when they are not, Protected packages are not, but they're otherwise the same. Well, OK not entirely, but you can think of them as: Protected <=> Essential (if installed) Given that they're not installed automatically, it follows that you need explicit dependencies on them. And APT also does not immediately configure them given that all dependencies are actually defined. Or well, the other way to think about it is that this is a package that cannot be removed without an extra check. > > > Protected packages contain mostly important system boot infrastructure. > > Removing them might cause the whole system to be unable to boot, so use > > with caution. > > That seems a bit vague and sparse. > > Is there maybe more rigid information for what *exactly* the "Protected" > field should and will be used? What you use them for is obviously not fixed. What we have so far are: ## Essential for some use cases The init package uses the old Important: yes to be harder to remove so you don't end up bricking your system into an unbootable one. That's the "Essential only for some use cases" use case, because init obviously should not be Essential because we don't want it in an app container, but we do want it to be Essential when it's already installed. The same applies to e2fsprogs, which is also marked like that, because duh, you do not want to be able to remove e2fsprogs on a real system, but again you don't want it installed automatically in a container. ## Weird hacks in low-level packages As another example, we marked libgcc-s1 as Protected, because it Replaces: libgcc1 but can't Conflict with it, so removing libgcc-s1 would remove the library entirely, so we had to protect it. ## Local metapackages The original use case was local system configuration meta packages, where you define a my-machine-foo meta package, and have it depend on all packages in the system and mark that as Protected: yes[2]; back then we used XB-Important: yes (well we still do, you gotta set both for backwards compatibility with stable atm really). Because you then mark all other packages auto and you then absolutely do not want to lose your meta package :D [2] https://blog.jak-linux.org/2012/01/24/managing-system-package-selections-using-custom-meta-packages/ ## De-essentializing Essential packages Packages that were Essential can be downgraded to Protected and installed systems won't show any change in behavior (avoiding regressions), whereas the package will no longer be "essential" on new systems. -- debian developer - deb.li/jak | jak-linux.org - free software dev ubuntu core developer i speak de, en
Bug#983304: please document "Protected" field
Source: debian-policy Version: 4.5.1.0 Severity: wishlist In Julian Andres Klode's blog I've [1] glimpsed: > New features > [...] > The Protected field is now supported. It replaces the previous Important > field and is like Essential, but only for installed packages (some minor > more differences maybe in terms of ordering the installs). So I've tried to find out what the "Protected" field is for. The only info about it that I could find is from `man 1 dpkg`: > Protected packages contain mostly important system boot infrastructure. > Removing them might cause the whole system to be unable to boot, so use > with caution. That seems a bit vague and sparse. Is there maybe more rigid information for what *exactly* the "Protected" field should and will be used? Either way, if there's a new control field, then I think it should get documented in the `debian-policy`? Thanks, *t [1] https://blog.jak-linux.org/2021/02/18/apt-2.2/ -- System Information: Debian Release: 10.8 APT prefers stable APT policy: (500, 'stable') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 4.19.0-13-amd64 (SMP w/8 CPU cores) Kernel taint flags: TAINT_WARN, TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE Locale: LANG=de_CH.utf8, LC_CTYPE=de_CH.utf8 (charmap=UTF-8), LANGUAGE=de_CH:de (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) LSM: AppArmor: enabled
