** Description changed:
+ [Impact]
+
+ When a TCP DNS socket receives a locally generated ICMP port-unreachable
error, epoll reports EPOLLERR to systemd-resolved.
+ The affected systemd versions in Jammy and Noble do not consume the pending
socket error, so epoll repeatedly wakes the daemon.
+ systemd-resolved can enter a busy loop, consume substantial CPU, and stall
resolvectl requests.
+
+ This can affect VPN DNS kill switches implemented with nftables REJECT
+ rules when a DNS query reaches TCP and the TCP request is locally
+ rejected with an ICMP port-unreachable error.
+
+ The fix reads SO_ERROR when EPOLLERR is reported and completes the
+ failed DNS stream. It also stops treating ECONNRESET as packet loss.
+
+
+ [Test Plan]
+
+ 1. install required pkgs.
+ - Jammy: sudo apt install systemd nftables dnsmasq-base
+ - Noble: sudo apt install systemd systemd-resolved nftables dnsmasq-base
+
+
+ 2. Create a test DNS server at 192.0.2.53. dnsmasq serves a TXT record larger
than its UDP limit, so it sets the TC bit and forces systemd-resolved to retry
over TCP.
+
+ sudo systemctl restart systemd-resolved
+ sudo ip link add lp2145709d type dummy
+ sudo ip address add 192.0.2.53/32 dev lp2145709d
+ sudo ip link set lp2145709d up
+
+ sudo dnsmasq --keep-in-foreground --conf-file=/dev/null --no-resolv \
+ --no-hosts --bind-interfaces --listen-address=192.0.2.53
--port=53 \
+ --edns-packet-max=512 \
+ --txt-record="lp2145709.test,$(printf '%1024s' '' | tr ' '
x)" \
+ </dev/null >/dev/null 2>&1 &
+ sleep 1
+
+ 3. Make the test server the route-only DNS server, allow its UDP replies, and
reject TCP DNS locally with ICMP port-unreachable.
+
+ sudo resolvectl dns lp2145709d 192.0.2.53
+ sudo resolvectl domain lp2145709d '~.'
+ sudo resolvectl default-route lp2145709d yes
+ sudo resolvectl reset-server-features
+
+ sudo nft add table inet lp2145709
+ sudo nft 'add chain inet lp2145709 output { type filter hook output
priority 0; policy accept; }'
+ sudo nft add rule inet lp2145709 output ip daddr 192.0.2.53 udp dport 53
counter accept
+ sudo nft add rule inet lp2145709 output ip daddr 192.0.2.53 tcp dport 53
counter reject
+
+ 4. Start the query in the background. Check the TCP counter and measure
+ CPU while the affected stream is still active; measuring only after the
+ client timeout can miss the busy loop.
+
+ timeout 20s resolvectl query --cache=no --type=TXT lp2145709.test &
+ query_pid=$!
+ sleep 2
+
+ sudo nft list chain inet lp2145709 output # TCP reject rule: packets
+ > 0
+
+ top -b -d 5 -n 2 -p "$(systemctl show --property MainPID --value
+ systemd-resolved.service)" | tail -n 3
+
+ timeout 5s resolvectl status
+ wait "$query_pid"
+ printf 'query_rc=%s\n' "$?"
+
+ The archive packages reproduced the bug with a nonzero TCP counter and
+ 100.2% CPU on Jammy and 99.8% on Noble during the five-second sample.
+ Both queries timed out with status 124. A zero TCP counter means the
+ affected path was not exercised and invalidates the result.
+
+ [Where problems could occur]
+
+ The change affects only DNS stream error handling in systemd-resolved.
+ When epoll reports EPOLLERR and SO_ERROR can be read, the stream is now
+ closed and the error is passed to its transactions. If getsockopt()
+ fails, execution falls through to the previous handling.
+
+ The second part changes feature-level accounting for ECONNRESET: a reset
+ TCP connection no longer increments the server's packet-loss counter.
+ The transaction still receives the connection error and the failed
+ stream is detached. A regression could cause a DNS query to fail sooner,
+ or systemd-resolved to select a different server or transport after a
+ TCP connection reset. The test above exercises the EPOLLERR handling
+ path but does not separately inject ECONNRESET.
+
+ [Other Info]
+
+ Upstream issue: https://github.com/systemd/systemd/issues/33719
+ Upstream pull request: https://github.com/systemd/systemd/pull/33903
+ Upstream v255-stable commit:
https://github.com/systemd/systemd-stable/commit/030dbbc39e54666bd0f393ef47f0b0d9b2dfe8b4
+
+
+
+ [Original Description]
+
+
# Launchpad Bug Report
**Package:** systemd (Ubuntu 24.04 Noble)
**Title:** systemd-resolved 100% CPU spin loop when DNS queries are rejected
or time out (fix available in v255-stable)
---
## Description
systemd-resolved enters a 100% CPU spin loop when DNS queries fail with
ECONNREFUSED (from firewall REJECT rules) or time out over TCP. This is
triggered by common VPN tools (ExpressVPN, Tailscale) that either block
non-VPN DNS traffic or register DNS servers that are intermittently
unreachable.
The bug is in `src/resolve/resolved-dns-transaction.c` where
`dns_server_packet_lost()` is called on connection reset errors, causing
`dns_transaction_retry()` to compute a timer deadline already in the
past. This creates an infinite tight loop:
```
epoll_wait → read(timerfd) → gettid → timerfd_settime(tv_sec=0, tv_nsec=1) →
repeat
```
Confirmed via strace — approximately 50,000 iterations per second,
consuming 100% of one CPU core indefinitely until the service is
restarted.
## Impact
- 100% CPU on one core, causing thermal throttling and fan noise
- Degrades system responsiveness
- Persists until systemd-resolved is manually restarted
- Affects any Ubuntu 24.04 system using VPN software (ExpressVPN, Tailscale,
OpenVPN, WireGuard) or any configuration where DNS queries receive ICMP
port-unreachable or TCP timeouts
## Steps to Reproduce
1. Install Ubuntu 24.04 with default systemd-resolved configuration
2. Connect a VPN that blocks non-VPN DNS traffic (e.g., ExpressVPN with
Network Lock, or Tailscale with MagicDNS)
3. Observe systemd-resolved consuming 100% CPU within seconds
Alternatively:
1. Configure a DNS server in systemd-resolved that is unreachable
2. Trigger DNS queries that fail over TCP with ECONNREFUSED
3. Observe the CPU spin
## Upstream Fix
This was reported upstream as
[systemd/systemd#33719](https://github.com/systemd/systemd/issues/33719)
and fixed in [PR #33903](https://github.com/systemd/systemd/pull/33903)
("resolved: don't treat conn reset as packet loss") by Ronan Pigott,
merged 2024-08-02.
The fix was cherry-picked to **v255-stable** as commit
[`030dbbc39e54666bd0f393ef47f0b0d9b2dfe8b4`](https://github.com/systemd/systemd-
stable/commit/030dbbc39e54666bd0f393ef47f0b0d9b2dfe8b4), included in
**v255.11** released 2024-08-16.
The fix distinguishes between ECONNRESET/ECONNREFUSED (connection
errors) and actual packet loss. Connection errors now properly fail the
transaction instead of triggering an immediate retry with a bogus timer
deadline.
## Current State
- **Ubuntu 24.04 (Noble):** Ships systemd 255.4-1ubuntu8.12. The fix is NOT
included. Verified by reviewing the full Ubuntu changelog from 255.4-1ubuntu8
through 255.4-1ubuntu8.12 — no mention of this fix.
- **Pop!_OS 24.04:** Ships 255.4-1ubuntu8.12pop0 (repackaged Ubuntu). Also
missing the fix.
- **Upstream v255-stable:** Fix has been stable since v255.11 (August 2024) —
18+ months with no regressions reported.
- **Fedora 41+, Debian Trixie:** Ship systemd 256+ or 257+ which include the
fix.
## Requested Action
Cherry-pick commit `030dbbc39e54` from the v255-stable branch into the
Ubuntu Noble systemd package. This is a minimal, self-contained fix to
`src/resolve/resolved-dns-transaction.c` that has been stable in the
v255 branch for over 18 months.
## System Information
```
$ systemd --version
systemd 255 (255.4-1ubuntu8.12pop0~1769790828~24.04~d4491c0)
$ lsb_release -a
Distributor ID: Pop
Description: Pop!_OS 24.04 LTS
Release: 24.04
Codename: noble
$ uname -r
6.18.7-76061807-generic
```
## References
- Upstream bug: https://github.com/systemd/systemd/issues/33719
- Upstream fix PR: https://github.com/systemd/systemd/pull/33903
- v255-stable cherry-pick:
https://github.com/systemd/systemd-stable/commit/030dbbc39e54666bd0f393ef47f0b0d9b2dfe8b4
- Related older bug (same class):
https://github.com/systemd/systemd/issues/20062
- Tailscale reports of same issue:
https://github.com/tailscale/tailscale/issues/10630,
https://github.com/tailscale/tailscale/issues/13802
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2145709
Title:
systemd-resolved 100% CPU spin loop when DNS queries are rejected or
time out (fix available in v255-stable)
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2145709/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
