Public bug reported:
### Impact
Any restart of `ubuntu-fan.service` kills the fan `dnsmasq` and does not
respawn it, while the
unit reports success. The fan bridge and routes stay up, so nothing appears
wrong until the
existing DHCP leases expire roughly an hour later — at which point every
container on the fan
loses its IPv4 address at once and cannot get a new one.
On our hosts this is triggered with no operator action at all, by the glibc
restart sweep of an
`unattended-upgrades` run. Two of three hosts in a Ceph cluster lost their
containers on the same
morning, ~20 minutes apart, matching the times each host installed the libc6
update. The delayed
onset makes the cause very hard to attribute: by the time anything breaks, the
upgrade is an hour
in the past and the service claims to be healthy.
This appears to be the `ubuntu-fan`-side counterpart of the still-open LP:
#1903520, where the
same dnsmasq-in-the-wrong-cgroup problem is triggered by restarting
`jujud-machine`. Here the
cgroup owner is `ubuntu-fan.service` itself and the trigger is an ordinary
security update, so no
Juju involvement is needed to hit it.
### Environment
- Ubuntu 24.04 (noble), kernel 6.8.0-136-generic
- ubuntu-fan 0.12.16+24.04.1
- Fan config (`/etc/network/fan` line 40, written by fanatic):
`10.10.104.0/24 252.0.0.0/8`
- `fanctl show`:
```
Bridge Underlay Overlay Flags
fan-252 10.10.104.60/24 252.0.0.0/8 enable dhcp
```
- LXD containers attached to `fan-252`, addressed by the fan dnsmasq.
### Steps to reproduce
1. Configure a fan with `dhcp` enabled and attach at least one running
container to the bridge.
2. Confirm dnsmasq is serving: `systemd-cgls -u ubuntu-fan.service`
3. `systemctl restart ubuntu-fan` (or let a libc6 upgrade's restart sweep do it)
4. `pgrep -af 'interface=fan-252'` → no dnsmasq
5. Wait for the DHCP lease time to elapse → containers lose their IPv4
addresses.
### What happens
dnsmasq runs with PPID 1 but is a member of the unit's cgroup:
```
# systemd-cgls -u ubuntu-fan.service
Unit ubuntu-fan.service (/system.slice/ubuntu-fan.service):
└─432679 dnsmasq -u dnsmasq --strict-order --bind-interfaces \
--pid-file=/run/ubuntu-fan/dnsmasq-fan-252.pid --conf-file= \
--listen-address 252.62.0.1 --dhcp-range 252.62.0.2,252.62.255.254 ...
```
So the default `KillMode=control-group` kills it on any stop of the unit. The
subsequent start does
not bring it back. From the libc6-triggered event on one host:
```
Jul 28 06:41:48 systemd[1]: Starting apt-daily-upgrade.service ...
(unattended-upgrades: libc-bin libc-dev-bin libc-devtools libc6 libc6-dev
locales)
Jul 28 06:41:55 dnsmasq[1565]: exiting on receipt of SIGTERM
Jul 28 06:41:55 systemd[1]: ubuntu-fan.service: Deactivated successfully.
Jul 28 06:41:55 systemd[1]: Stopped ubuntu-fan.service - Ubuntu FAN network
setup.
Jul 28 06:41:55 systemd[1]: Starting ubuntu-fan.service - Ubuntu FAN network
setup...
Jul 28 06:41:55 systemd[1]: Finished ubuntu-fan.service - Ubuntu FAN network
setup.
```
`ExecStart` exited `0/SUCCESS`. There is not a single DHCP log line after this
point, and no
dnsmasq process exists. Every file in `/run/ubuntu-fan/` still carries its
original boot mtime —
the restart touched none of them.
The last lease renewal on this host was 06:31:38; the containers kept working
until the lease
elapsed and then went dark, with the juju agents going `lost` and ceph-mon
losing quorum.
### Why the start does not respawn dnsmasq
`dhcp_reconfigure()` — the function that starts dnsmasq — is called at the very
end of the
bridge-configuration function in `/usr/sbin/fanctl` (~line 1229), after the
bridge, tunnel and NAT
setup. That function short-circuits when `/run/ubuntu-fan/brg-state-<bridge>`
already exists
(~line 1132), which it does, because the bridge was never torn down. So the
dnsmasq code path is
never reached.
Deleting `/run/ubuntu-fan/dnsmasq-flags-fan-252` — the file
`dhcp_reconfigure()` compares against
to decide whether to respawn — has no effect either, for the same reason: the
code that reads it
does not execute.
### Recovery is also blocked
1. `fanctl down` refuses while containers are attached:
`/usr/sbin/fanctl: fan-252: in use, will not destroy` → `err=1`
2. Because that makes the stop fail, the unit lands in `failed`. `systemctl
restart` on a failed
unit does not run `ExecStop`, so `fanctl down` never gets another chance,
and `ExecStart` keeps
failing:
`/usr/sbin/fanctl: fan-252: already configured` → `err=1`
This loops indefinitely — a plain restart can never recover the service.
3. The only working recovery is to stop every container on the bridge, run
`/usr/lib/ubuntu-fan/fan-net stop` by hand, `systemctl reset-failed`,
`systemctl start`, then
start the containers again.
Note the inconsistent exit codes: the libc-triggered `fan-net start` exited 0
while leaving the
fan without DHCP (silent failure), whereas later manual starts exit 1 with
`already configured`
(unrecoverable failure). Neither reflects the actual state of the fan. Compare
LP: #1719644 on
fanctl exit codes.
### Expected behaviour
Either of these would prevent the outage:
- `ubuntu-fan.service` should not let systemd kill dnsmasq out from under
`fanctl` — e.g. ship
`KillMode=process` in the unit, or run dnsmasq from its own unit.
- `fanctl up` should verify that a dnsmasq is actually running for a
`dhcp`-enabled fan and
respawn it if not, rather than inferring liveness from `/run` state files,
and should not exit 0
when the fan has no DHCP server.
Additionally, `fan-net stop` failing on `in use, will not destroy` should
probably not leave the
unit in a state from which `systemctl restart` can never recover.
### Tested mitigation
A drop-in makes the outage stop happening:
```ini
# /etc/systemd/system/ubuntu-fan.service.d/10-keep-dnsmasq.conf
[Service]
KillMode=process
```
The unit is `Type=oneshot` with `RemainAfterExit=yes` and its main process has
already exited, so
nothing is killed on stop and dnsmasq survives. Verified across a `systemctl
restart` with
containers running — same dnsmasq PID before and after, containers kept their
leases, and systemd
logs the intent explicitly:
```
ubuntu-fan.service: Unit process 933560 (dnsmasq) remains running after unit
stopped.
ubuntu-fan.service: Found left-over process 933560 (dnsmasq) in control group
while starting unit. Ignoring.
```
This does not fix the `already configured` failure — a restart while the fan is
up still leaves the
unit `failed` — but it does prevent the loss of container networking, which is
the damaging part.
### Related
- LP: #1903520 — same dnsmasq-killed-by-cgroup mechanism, triggered via
`jujud-machine`; open since 2020.
- LP: #1719644 — fanctl exit codes not reflecting failed up steps.
- LP: #1744296 — `already configured` after underlay ifdown/ifup; fixed in
0.12.10, different trigger.
** Affects: ubuntu-fan (Ubuntu)
Importance: Undecided
Status: New
** Tags: dnsmasq fan noble systemd
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2161974
Title:
ubuntu-fan.service restart silently leaves the fan bridge without a
DHCP server; containers lose addresses an hour later
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ubuntu-fan/+bug/2161974/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
