Disclaimer: I used Claude to help diagnose and solve this issue, but I'm a human fyi :D
Robot diagnosis beep boop: We hit this on Ubuntu 18.04 (bionic) hosts with Ubuntu Pro (ESM Infra) enabled and unattended-upgrades on. Multiple hosts were affected. Timeline on an affected host (UTC, 2026-10-15): - Around 06:40, unattended-upgrades installs systemd 237-3ubuntu10.57+esm4. - Within the same second, systemd-resolved restarts and fails with "Failed to add built-in positive trust anchor: No space left on device", then hits the start limit. The host was not rebooted. Findings: 1. No resource was exhausted: disk, inodes, inotify and /run were all fine. Under strace, no syscall returns ENOSPC. The last calls are ENOENT lookups of the dnssec-trust-anchors.d directories, then the error is printed, so it is generated inside systemd-resolved. 2. Copying /usr/share/dns/root.ds to /etc/dnssec-trust- anchors.d/root.positive did not fix it on esm4. resolved still failed with ENOSPC. 3. 237-3ubuntu10.57+esm5 fixes it. On esm5, resolved logs three positive trust anchors (root DS key tags 19036, 20326 and 38696) and starts normally, including after a reboot. 4. Canonical's esm.ubuntu.com IPv4 servers were inconsistent between about 15:00 and 16:00 UTC on 2026-10-15: - 185.125.190.23 and 185.125.190.24 returned 404 for the esm5 .deb files. - At least one server still served an index listing esm4 as the newest version: apt-get update fetched only InRelease and apt-cache policy kept esm4 as the candidate. - 185.125.190.75 served both the esm5 index and packages. Because of the stale index, a host still on esm3 could be upgraded to esm4 and lose DNS. Hosts already on esm4 cannot fix themselves, because apt needs DNS. Recovery that worked on the hosts where we tried it: 1. Write a temporary nameserver line (the DHCP-provided resolver) to /etc/resolv.conf. Create /run/systemd/resolve first if it is missing, and make the file mode 644, since apt downloads as the _apt user. 2. Pin esm.ubuntu.com in /etc/hosts to a server that serves esm5 (185.125.190.75 worked), run apt-get update, and check that apt-cache policy systemd shows esm5 as the candidate. 3. Upgrade systemd, libsystemd0, udev, libudev1, libpam-systemd, libnss- systemd and systemd-sysv, then remove the pin. 4. Run systemctl reset-failed systemd-resolved, start it, and restore the /etc/resolv.conf symlink if it was replaced. Requests for Canonical: remove esm4 from every esm.ubuntu.com server so the stale index stops offering it, and publish a notice for bionic ESM users, since affected hosts lose DNS and cannot recover through unattended-upgrades. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2170852 Title: Ubuntu 18.04: systemd-resolved fails to start for 237-3ubuntu10.57+esm4 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-pro/+bug/2170852/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
