Hi,
I talked to Aurelien on IRC. This mail gives some context and summarizes
our discussion.
On Mon, Jun 23, 2025 at 04:58:34PM +0200, Helmut Grohne wrote:
> On Sun, Jun 22, 2025 at 11:05:03PM +0200, Helmut Grohne wrote:
> > mmdebstrap --variant=apt --architectures=amd64,i386
> > --include=systemd-timesyncd,libc6-dbg:i386 --chrooted-customize-hook='sed
> > -i -e s/bookworm/trixie/ /etc/apt/sources.list && apt update && apt-get -y
> > install apt' bookworm /dev/null
>
> Further observations.
> * We may install libc6 instead of apt.
> * We may include systemd-container instead of systemd-timesyncd.
> * If we include systemd-resolved or systemd-boot instead of
> systemd-timesyncd, the problem does not reproduce.
> * trixie's libc6 Breaks: system (<< trixie).
This is a key ingredient to the problem.
> * trixie's systemd Pre-Depends: libsystemd-shared (= ...)
> * trixie's libsystemd-shared Depends: libc6 (>= trixie)
> * All of systemd-container, systemd-resolved, systemd-boot and
> * systemd-timesyncd Conflicts: systemd (<< trixie).
> * In the apt log you may see that apt unpacks libsystemd-shared and
> only then considers deconfiguring systemd (which Depends:
> libsystemd-shared (= ...)). Why is that legal?
Fundamentally, apt has no chance in this situation, because libc6
Breaks: systemd (<< trixie) and systemd Pre-Depends: libc6 (>= trixie).
Configuring systemd before unpacking systemd violates the Breaks, doing
it the other way round violates the Pre-Depends. apt end up in a
situation where it temporarily has to do something bad. It tends to
involve further packages (such as systemd-timesyncd) to make the upgrade
fail, but that's the fundamental situation.
So let's look into why we have these relations.
systemctl is used in lots of maintainer scripts without issueing a
direct dependency on systemd. As a result of this practice, systemctl
needs to practically work at all times in a similar way that essential
packages must work at all times. If we were to demote the Pre-Depends
to Depends, apt would be entitled to unpack systemd before libc6 and
then other maintainer scripts could fail expecting a working systemctl.
We very much want the unpack order enforced by Pre-Depends. What we'd
not need here is having libc6 configured prior to unpacking systemd. I
am now wondering whether systemd could:
Depends: libc6 (>= trixie)
Conflicts: libc6 (<< trixie)
Those Conflicts sound very risky, and I haven't given this approach a
try. Is this worth pursuing or do we readily rule it out for some
reason? I note that this would get us into a Breaks + Conflicts cycle
and while that is solvable in theory (unpack conflicted, unpack broken,
configure both), apt tends to solve it by temporarily removing one and
we wouldn't want it to remove either not even temporarily.
The other side is what I discussed with Aurelien and much of what
follows is due to him (thank you). systemd needs to be restarted when
upgrading libc6, because it uses nss modules. Failing to do so results
in e.g. #993821. The way to restart systemd has changed over time. For
instance, restarting user managers and other daemons such as resolved
has become a thing. Due to these changes, the responsibility of
restarting was eventually transferred to systemd via means of a dpkg
trigger (see #1074607) and the Breaks ensures that systemd can handle
the trigger. The Breaks declaration shall ensure that systemd ends up
being restarted, but what also does is require that systemd.postinst is
run before libc6.postinst. We observe that we do not expect to bump the
Breaks version in forky as it is an artifact of transitioning to dpkg
triggers, so in later dist-upgrades the cycle at hand is expected to
disappear. If we are trying to relax it, we'd have to re-add code for
restarting systemd to libc6.postinst:
if the installed systemd does not support triggers; then
# Expect a bookworm systemd:
# * If it were older, we'd be doing a skip upgrade.
# * If it is newer (e.g. backports), it supports triggers.
restart systemd the bookworm way
#
https://sources.debian.org/src/systemd/252.36-1~deb12u1/debian/systemd.postinst/#L77
fi
I note that this is the code that Aurelien spent effort on to get
removed from libc6 and moved into glibc. He expressed that he does not
like this approach (for obvious reasons).
As far as I can see, these are the options to fix the root cause and I
see no alternatives. If we end up deciding that neither is sensible,
the next option is to whack-a-mole upgrade failures. The way to go here
is having systemd packages stop Conflicting with systemd. Originally,
those were Breaks+Replaces and they were upgraded to Conflicts
(overriding my objections) to mitigate /usr-move file loss. I think we
can downgrade them to Breaks+Replaces by adding a few protective
diversions to systemd. In situations where there are no such Conflicts,
apt tends to run into the problematic login less often, but this is far
from fixing it once and for all.
I also looked for further instances of the problem with mutual
relations by analyzing package metadata. I'm attaching the script that
produces the following output:
libc6 breaks systemd pre-depends
texlive-base conflicts texlive-extra-utils breaks
libreoffice-common conflicts libreoffice-base pre-depends
libreoffice-common conflicts libreoffice-base-nogui pre-depends
libreoffice-common conflicts libreoffice-calc pre-depends
libreoffice-common conflicts libreoffice-calc-nogui pre-depends
libreoffice-common conflicts libreoffice-draw pre-depends
libreoffice-common conflicts libreoffice-draw-nogui pre-depends
libreoffice-common conflicts libreoffice-gnome pre-depends
libreoffice-common conflicts libreoffice-impress pre-depends
libreoffice-common conflicts libreoffice-impress-nogui pre-depends
libreoffice-common conflicts libreoffice-math pre-depends
libreoffice-common conflicts libreoffice-math-nogui pre-depends
libreoffice-common conflicts libreoffice-report-builder pre-depends
libreoffice-common conflicts libreoffice-writer pre-depends
libreoffice-common conflicts libreoffice-writer-nogui pre-depends
libreoffice-common breaks libreoffice-gnome pre-depends
libreoffice-common breaks libreoffice-report-builder pre-depends
samba-ad-dc conflicts samba breaks
Generally, I think we may tolerate temporary removals in all but libc6
vs systemd. In case of texlive-base vs texlive-latex-extra I saw a
temporary removal in an attempted upgrade and in case of samba-ad-dc vs
samba, I saw a temporary deconfiguration. I'm also partially guilty of
the libreoffice situation as I recommended pre-depends on Rene as a
means of avoiding mutual Breaks + Conflicts. Not sure if mutual
Conflicts + Pre-Depends is any better.
Helmut
#!/usr/bin/python3
import subprocess
import debian.deb822
from debian.debian_support import version_compare
sidpkgs = {}
bookwormpkgs = {}
with subprocess.Popen(["chdist", "sid", "apt-cache", "show", "*"], stdout=subprocess.PIPE) as proc:
for pkg in debian.deb822.Packages.iter_paragraphs(proc.stdout):
if "Breaks" in pkg or "Conflicts" in pkg:
sidpkgs[pkg["Package"]] = pkg
with subprocess.Popen(["chdist", "bookworm", "apt-cache", "show", "*"], stdout=subprocess.PIPE) as proc:
for pkg in debian.deb822.Packages.iter_paragraphs(proc.stdout):
if pkg["Package"] in sidpkgs:
bookwormpkgs[pkg["Package"]] = pkg
for pc in sidpkgs.values():
if pc["Package"] not in bookwormpkgs:
continue
for kc in ("conflicts", "breaks", "pre-depends"):
if kc not in pc:
continue
for crs in pc.relations[kc]:
if kc == "pre-depends":
if len(crs) != 1:
continue
else:
assert len(crs) == 1
cr = crs[0]
if cr["version"] is None:
continue
if cr["version"][0][0] != ('>' if kc == "pre-depeneds" else '<'):
continue
if cr["name"] not in bookwormpkgs:
continue
if cr["name"] == pc["Package"]:
continue
if version_compare(bookwormpkgs[cr["name"]]["version"], cr["version"][1]) > 0:
continue
if cr["name"] not in sidpkgs:
continue
pb = sidpkgs[cr["name"]]
for kb in ("conflicts", "breaks", "pre-depends") if kc == "conflicts" else ("pre-depends",):
for brs in pb.relations[kb]:
if kb == "pre-depends":
if len(brs) != 1:
continue
else:
assert len(brs) == 1
br = brs[0]
if br["version"] is None:
continue
if br["version"][0][0] != ('>' if kb == 'pre-depends' else '<'):
continue
if br["name"] != pc["Package"]:
continue
if version_compare(bookwormpkgs[br["name"]]["version"], br["version"][1]) > 0:
continue
print(pc["Package"], kc, pb["Package"], kb)
