Public bug reported:
We experienced a production DNS outage affecting more than 50 Ubuntu 18.04
Pro EC2 instances across the following AWS regions:
- ap-southeast-2
- ap-southeast-1
- ap-northeast-1
Incident time:
2026-10-15 approximately 00:50 UTC
Please note that our operational timezone is Asia/Taipei (UTC+8).
The affected instances reported repeated systemd-resolved startup
failures:
Failed to add built-in positive trust anchor:
No space left on device
systemd-resolved subsequently entered a restart/failure loop. DNS resolution
stopped working and applications began reporting errors such as:
java.net.UnknownHostException:
secretsmanager.ap-southeast-2.amazonaws.com
Immediate mitigation:
systemctl stop systemd-resolved
systemctl reset-failed systemd-resolved
rm -f /etc/resolv.conf
We then configured /etc/resolv.conf with:
nameserver 172.31.0.2
nameserver 169.254.169.253
This restored DNS resolution by bypassing systemd-resolved.
Based on Ubuntu Bug 2170852 and the package history currently available,
our current understanding is:
- systemd 237-3ubuntu10.57+esm4 introduced the regression.
- systemd 237-3ubuntu10.57+esm5 appears to contain the correction.
- The regression affected DNSSEC trust-anchor handling when loading the
new root KSK, including key tag 38696.
- There may have been Ubuntu ESM repository/CDN inconsistency while the
corrected package was being published.
We are reviewing dpkg and APT history from each affected host to confirm
the exact installed package version at the time of failure.
Business impact:
- More than 50 production instances were affected.
- The incident occurred across three AWS regions.
- Applications were unable to resolve AWS service endpoints.
- AWS Secrets Manager calls failed.
- Application services stopped or failed during startup.
- A manual DNS configuration workaround was required.
- Affected systems could not reliably download the corrected package
because DNS resolution was unavailable.
We request that Canonical:
1. Confirm whether systemd 237-3ubuntu10.57+esm4 introduced the regression.
2. Confirm that 237-3ubuntu10.57+esm5 is the complete supported fix.
3. Provide a formal root cause analysis.
4. Explain how the regression passed Ubuntu ESM package validation.
5. Confirm whether the ESM repositories or CDN endpoints served inconsistent
package indexes or returned 404 responses during the publication of esm5.
6. Confirm that every ESM repository endpoint now consistently provides the
corrected package.
7. Provide a supported fleet-wide detection and recovery procedure.
8. Confirm which systemd-family packages must be upgraded together.
9. Provide guidance for safely restoring /etc/resolv.conf and
systemd-resolved after upgrading.
10. Publish an official notice so other Ubuntu Pro customers can identify
and remediate affected systems.
Related AWS Support Case ID:
179205881800971
** Affects: lua-systemd (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2171369
Title:
Ubuntu 18.04 ESM systemd 237-3ubuntu10.57+esm4 caused systemd-resolved
outage across 50+ AWS EC2 instances
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/lua-systemd/+bug/2171369/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
