Bug#1109119: upgrade-reports: systemctl occasionally fails loading libcrypto.so
Hello, please unsubscribe me from all lists. best, Daniel On 7/27/25 15:37, Chris Hofstaedtler wrote: On Thu, Jul 24, 2025 at 02:43:27AM +0200, Chris Hofstaedtler wrote: On Sun, Jul 13, 2025 at 10:29:25AM +0200, Helmut Grohne wrote: Hi Chris, On Sat, Jul 12, 2025 at 12:36:20AM +0200, Chris Hofstaedtler wrote: I was somewhat hoping this is caused by the existing upgrade issue which Helmut is working on AFAIU. Was it proven that this is not the same problem? I believe this is independent and I merely caught it as I was carefully reading test results for your other issue. This particular one is solely caused by libssl3 having done a t64 transition when it was not appropriate to do so. I do not expect my proposed change to glibc to improve the situation regarding systemctl. Is this problem still happening? Maybe a better question to ask would have been: is there a known reproducer for this? Given there were unclear reasons for the deconfigure/unpack order of libssl3(t64), I do wonder if other changes in the archive might have improved the situation. Best, Chris
Bug#1109119: upgrade-reports: systemctl occasionally fails loading libcrypto.so
On Thu, Jul 24, 2025 at 02:43:27AM +0200, Chris Hofstaedtler wrote: > On Sun, Jul 13, 2025 at 10:29:25AM +0200, Helmut Grohne wrote: > > Hi Chris, > > > > On Sat, Jul 12, 2025 at 12:36:20AM +0200, Chris Hofstaedtler wrote: > > > I was somewhat hoping this is caused by the existing upgrade issue which > > > Helmut is working on AFAIU. Was it proven that this is not the same > > > problem? > > > > I believe this is independent and I merely caught it as I was carefully > > reading test results for your other issue. This particular one is solely > > caused by libssl3 having done a t64 transition when it was not > > appropriate to do so. I do not expect my proposed change to glibc to > > improve the situation regarding systemctl. > > Is this problem still happening? Maybe a better question to ask would have been: is there a known reproducer for this? Given there were unclear reasons for the deconfigure/unpack order of libssl3(t64), I do wonder if other changes in the archive might have improved the situation. Best, Chris
Bug#1109119: upgrade-reports: systemctl occasionally fails loading libcrypto.so
Hi Helmut Am 25.07.25 um 00:18 schrieb Helmut Grohne: I understand that having libsystemd-shared and systemctl can be upgraded separately and that doing so results in yet another window where systemctl is dysfunctional. The problem reported here is about systemctl linking libcrypto.so.3 however. Both bookworm and trixie do that. Therefore my understanding is that the aforementioned commit does not cause the libcrypto.so.3 linkage that is the problem here. Conversely, reverting it will not remove libcrypto.so.3 and therefore will not help with the t64-induced library rename. What I'm trying to argue here is that even if reverting it improves robustness in some way, it does not improve robustness regarding libssl3 to libssl3t64 upgrades. Do you concur here? My hope was, that the systemctl dependencies would be significantly trimmed down but it seems we regressed in that regard already in bookworm. In bullseye e.g. systemctl didn't link against libcrypto: bullseye# ldd /bin/systemctl /lib64/ld-linux-x86-64.so.2 (0x7fb9862de000) libblkid.so.1 => /usr/lib/x86_64-linux-gnu/libblkid.so.1 (0x7fb98602a000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7fb985d12000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x7fb985c74000) libgcrypt.so.20 => /usr/lib/x86_64-linux-gnu/libgcrypt.so.20 (0x7fb985f0a000) libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x7fb985c4e000) liblz4.so.1 => /usr/lib/x86_64-linux-gnu/liblz4.so.1 (0x7fb986156000) liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x7fb986179000) libpcre2-8.so.0 => /usr/lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x7fb985c7a000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x7fb985ee6000) libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x7fb9861a1000) libzstd.so.1 => /usr/lib/x86_64-linux-gnu/libzstd.so.1 (0x7fb98607b000) linux-vdso.so.1 (0x7fb9862dc000) bookworm# ldd /bin/systemctl /lib64/ld-linux-x86-64.so.2 (0x7f456416) libblkid.so.1 => /lib/x86_64-linux-gnu/libblkid.so.1 (0x7f4563d87000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7f456341f000) libcap.so.2 => /lib/x86_64-linux-gnu/libcap.so.2 (0x7f4563fff000) libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3 (0x7f456360) libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x7f4563c4) libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x7f4563b19000) liblz4.so.1 => /lib/x86_64-linux-gnu/liblz4.so.1 (0x7f4563f7c000) liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x7f4563fa2000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x7f4563dde000) libmount.so.1 => /lib/x86_64-linux-gnu/libmount.so.1 (0x7f4563bdd000) libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x7f4563b41000) libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x7f4563fd1000) libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1 (0x7f4563ec) linux-vdso.so.1 (0x7f456415e000) systemctl gained a libcap, libcrypto, libm and libmount dependency. trixie# ldd /bin/systemctl /lib64/ld-linux-x86-64.so.2 (0x7f4cd9b29000) libacl.so.1 => /lib/x86_64-linux-gnu/libacl.so.1 (0x7f4cd9acb000) libaudit.so.1 => /lib/x86_64-linux-gnu/libaudit.so.1 (0x7f4cd9299000) libblkid.so.1 => /lib/x86_64-linux-gnu/libblkid.so.1 (0x7f4cd9a6b000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7f4cd940a000) libcap-ng.so.0 => /lib/x86_64-linux-gnu/libcap-ng.so.0 (0x7f4cd9a3f000) libcap.so.2 => /lib/x86_64-linux-gnu/libcap.so.2 (0x7f4cd9a5f000) libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x7f4cd93ce000) libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3 (0x7f4cd8c0) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x7f4cd8b1) libmount.so.1 => /lib/x86_64-linux-gnu/libmount.so.1 (0x7f4cd9352000) libpam.so.0 => /lib/x86_64-linux-gnu/libpam.so.0 (0x7f4cd9a4b000) libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x7f4cd8997000) libseccomp.so.2 => /lib/x86_64-linux-gnu/libseccomp.so.2 (0x7f4cd9324000) libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x7f4cd92f) libsystemd-shared-257.so => /usr/lib/x86_64-linux-gnu/systemd/libsystemd-shared-257.so (0x7f4cd960) libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x7f4cd92d) libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1 (0x7f4cd8a46000) linux-vdso.so.1 (0x7f4cd9b27000) systemctl (also via libsystemd-shared) gained a libacl, libaudit, libblkid, libcap-ng, libcrypt, libpam and libseccomp dependency
Bug#1109119: upgrade-reports: systemctl occasionally fails loading libcrypto.so
Hi Michael, I appreciate that you chime in here. On Thu, Jul 24, 2025 at 11:18:31PM +0200, Michael Biebl wrote: > We are rather late in the trixie release cycle, so maybe the safest approach > is to revert 8ca2b266789bff0b1af348100e72da7245864174 for trixie and > re-apply it early during the forky release? Could you help me understand why reverting this commit would improve the symptoms of the bug report at hand? I understand that having libsystemd-shared and systemctl can be upgraded separately and that doing so results in yet another window where systemctl is dysfunctional. The problem reported here is about systemctl linking libcrypto.so.3 however. Both bookworm and trixie do that. Therefore my understanding is that the aforementioned commit does not cause the libcrypto.so.3 linkage that is the problem here. Conversely, reverting it will not remove libcrypto.so.3 and therefore will not help with the t64-induced library rename. What I'm trying to argue here is that even if reverting it improves robustness in some way, it does not improve robustness regarding libssl3 to libssl3t64 upgrades. Do you concur here? Helmut
Bug#1109119: upgrade-reports: systemctl occasionally fails loading libcrypto.so
Am 24.07.25 um 23:18 schrieb Michael Biebl: We are rather late in the trixie release cycle, so maybe the safest approach is to revert 8ca2b266789bff0b1af348100e72da7245864174 for trixie and re-apply it early during the forky release? Especially, since trixie contains both the t64 and usrmove related changes, which will no longer be an issue for forky. OpenPGP_signature.asc Description: OpenPGP digital signature
Bug#1109119: upgrade-reports: systemctl occasionally fails loading libcrypto.so
[looping in bluca as well] On Thu, 24 Jul 2025 02:43:27 +0200 Chris Hofstaedtler wrote: On Sun, Jul 13, 2025 at 10:29:25AM +0200, Helmut Grohne wrote: > Hi Chris, > > On Sat, Jul 12, 2025 at 12:36:20AM +0200, Chris Hofstaedtler wrote: > > I was somewhat hoping this is caused by the existing upgrade issue which > > Helmut is working on AFAIU. Was it proven that this is not the same problem? > > I believe this is independent and I merely caught it as I was carefully > reading test results for your other issue. This particular one is solely > caused by libssl3 having done a t64 transition when it was not > appropriate to do so. I do not expect my proposed change to glibc to > improve the situation regarding systemctl. Is this problem still happening? If so, are there any ideas on what to do about it? Not sure if it's relevant, but I post it here for completeness sake: To make systemctl more robust during dist-upgrades, in the past we built systemd with `-Dlink-systemctl-shared=false`, so systemctl has a minimal set of library dependencies and then we added all those library dependencies to Pre-Depends via https://salsa.debian.org/systemd-team/systemd/-/blob/debian/master/debian/rules?ref_type=heads#L302 This was under the assumption that systemctl is quasi-essential and can be triggered at arbitrary times during a dist-upgrades. Luca changed that (recently) in https://salsa.debian.org/systemd-team/systemd/-/commit/8ca2b266789bff0b1af348100e72da7245864174 I.e. systemctl now links against libsystemd-shared which in turn has a couple of side-effects a/ libsystemd-shared is not an so-versioned package, i.e. different versions can not be co-installed. So there is a time window where systemctl is in a non-functional state, e.g. when you have an old systemd + new libsystemd-shared or a new systemd + old libsystemd-shared unpacked This could be addressed by making libsystemd-shared so-versioned, i.e. naming it libsystemd-shared-$VER or going back to linking systemctl statically against libsystemd-shared. b/ If Luca want's to continue to use `-Dlink-systemctl-shared=true`, maybe it would make sense to treat all those library dependencies of libsystemd-shared as Pre-Depends. We are rather late in the trixie release cycle, so maybe the safest approach is to revert 8ca2b266789bff0b1af348100e72da7245864174 for trixie and re-apply it early during the forky release? Michael OpenPGP_signature.asc Description: OpenPGP digital signature
Bug#1109119: upgrade-reports: systemctl occasionally fails loading libcrypto.so
On Sun, Jul 13, 2025 at 10:29:25AM +0200, Helmut Grohne wrote: > Hi Chris, > > On Sat, Jul 12, 2025 at 12:36:20AM +0200, Chris Hofstaedtler wrote: > > I was somewhat hoping this is caused by the existing upgrade issue which > > Helmut is working on AFAIU. Was it proven that this is not the same problem? > > I believe this is independent and I merely caught it as I was carefully > reading test results for your other issue. This particular one is solely > caused by libssl3 having done a t64 transition when it was not > appropriate to do so. I do not expect my proposed change to glibc to > improve the situation regarding systemctl. Is this problem still happening? If so, are there any ideas on what to do about it? Chris
Bug#1109119: upgrade-reports: systemctl occasionally fails loading libcrypto.so
Hi Chris, On Sat, Jul 12, 2025 at 12:36:20AM +0200, Chris Hofstaedtler wrote: > I was somewhat hoping this is caused by the existing upgrade issue which > Helmut is working on AFAIU. Was it proven that this is not the same problem? I believe this is independent and I merely caught it as I was carefully reading test results for your other issue. This particular one is solely caused by libssl3 having done a t64 transition when it was not appropriate to do so. I do not expect my proposed change to glibc to improve the situation regarding systemctl. Helmut
Bug#1109119: upgrade-reports: systemctl occasionally fails loading libcrypto.so
Package: upgrade-reports Severity: important X-Debbugs-Cc: [email protected],debian-release.debian.org,[email protected],[email protected],[email protected] Hello, while working on a bookworm -> trixie upgrade failure, I noticed a strange line showing up. | Preparing to unpack .../openssh-server_1%3a10.0p1-5_amd64.deb ... | systemctl: error while loading shared libraries: libcrypto.so.3: cannot open shared object file: No such file or directory This is openssh-server.preinst failing to systemctl stop rescue-ssh.service. I talked to Colin and both of us agreed that this instance is probably practically irrelevant. However, I still think there is a problem. Due to the time64 transition, libssl3 was renamed to libssl3t64 and for some reason apt ends up removing libssl3 before unpacking libssl3t64. Given Breaks+Replaces, unpacking libssl3t64 after having deconfigured libssl3 before removing libssl3 should be fine, but apt does not like that solution. As a result, libcrypto.so.3 is temporarily removed. deb-systemd-invoke is part of init-system-helpers and therefore essential. It calls out to systemctl, which is not essential but for all practical matters we really should be treating it as if it was and maintainer scripts expect it to work at all times. libssl3 or libssl3t64 are pseudo-essential. Some part (apt or openssl) violates policy during the upgrade as being pseudo-essential requires it to work at all times even when unpacked. In practice, this means that systemctl cannot be expected to work in maintainer scripts. This will mostly affect preinst scripts (not just openssh-server) trying to stop services. For instance, it is conceivable that we could fail to stop mariadb or postgresql due to this (but there is no practical evidence of this ever having happened). Failure to stop services violates assumptions placed by package maintainers and that may have all sorts of consequences. I have several reports of systemctl having failed during release upgrades without having failed the upgrade transaction as a whole. It really is unclear whether this has practical consequences and whether there is a dataloss scenario something else that makes this problem practically relevant. We typically reboot after a dist upgrade (at least that's what release notes strongly recommend) and doing so tends to fix any failure to stop or start services. I have no evidence of this problem having caused a real issue (beyond that message). If you have earlier upgraded from bookworm to trixie. You should be able to search in your /var/log/apt/term.log* for the earlier message to see whether you were affected. In talking to Ivo and Paul, we agreed to report the problem to d-devel via upgrade-reports. At this stage we want to gauge the impact and better understand how serious this actually is, so following up on the bug report with evidence (dropping all lists if that's all you add) is highly appreciated. The options for fixing this are dim. Reverting the t64 transition for openssl and going dual-ABI seems highly unlikely even though it would fix the problem at the root. Other options are dim, because we have no scripts that are guarantueed to run before apt chooses to remove bookworm's libssl3. We considered doing changes to bookworm to mitigate. Conceivably, a bookworm update could add a libssl3.preinst that diverts the library to keep it around until it is overwritten by libssl3t64. I invite others to work on the problem as I have no capacity to do it. I'm still yak shaving another release upgrade problem and would like to enjoy DebConf. Thank you Helmut
