Package: dhcpcd-base
Version: 1:10.5.2-1
Severity: important
Dear Maintainer,
After upgrading dhcpcd-base/dhcpcd from 1:10.3.2-6 to 1:10.5.2-1, the next
shutdown/reboot takes about three minutes longer than before.
networking.service's ExecStop (ifdown -a) hangs and is eventually SIGKILLed
after two consecutive 90 s systemd timeouts:
systemd[1]: Stopping networking.service - Raise network interfaces...
ifdown[9640]: sending commands to dhcpcd process
dhcpcd[453]: control command: dhcpcd -k enp1s0
dhcpcd[453]: enp1s0: releasing lease of 172.31.0.122
[90 s pass]
ifdown[9621]: Got signal Terminated, terminating...
systemd[1]: networking.service: Control process exited, code=exited,
status=1/FAILURE
[90 s pass]
systemd[1]: networking.service: State 'final-sigterm' timed out. Killing.
systemd[1]: networking.service: Killing process 9640 (dhcpcd) with signal
SIGKILL.
systemd[1]: networking.service: Failed with result 'timeout'.
CAUSE
Upgrading the package does not restart dhcpcd.service. dhcpcd.postinst
contains only the dh_installsystemd *enable* snippet, and
dhcpcd-base.postinst only runs systemd-sysusers; neither invokes
deb-systemd-invoke restart. So after the upgrade the long-running manager is
still the old (10.3.2) binary, while ifup/ifdown now exec the new (10.5.2)
dhcpcd client.
The 10.5.2 client writes its command to the AF_UNIX control socket and then
blocks in read() forever. The 10.3.2 manager receives and fully executes the
command -- it removes the interface, releases the lease and deletes the routes
-- but never sends the reply the new client waits for. The client has no
timeout, so it hangs until something kills it.
The hung process, with the manager still at 10.3.2 and the client at 10.5.2:
PID PPID STAT WCHAN COMMAND
460 1 Ss do_epoll_wait dhcpcd: [manager] [ip4] [ip6]
848 846 S do_wait ifdown enp1s0
854 848 S do_wait /bin/sh -c dhcpcd -k enp1s0
855 854 S unix_stream_read_generic dhcpcd -k enp1s0
# cat /proc/855/stack
[<0>] unix_stream_read_generic+0x6f9/0xcb0
[<0>] unix_stream_recvmsg+0x87/0x90
[<0>] sock_recvmsg+0x9e/0xa0
[<0>] sock_read_iter+0x97/0x100
[<0>] vfs_read+0x358/0x390
[<0>] ksys_read+0xbf/0xf0
[<0>] do_syscall_64+0xe1/0x610
[<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e
# cat /proc/855/syscall
0 0x5 0x7ffc9bf4dd2c 0x4 0x0 0x0 0x0 0x7ffc9bf4dcb0 0x7f5dff69bca2
(syscall 0 = read, on fd 5, the control socket)
REPRODUCER
1. Boot a system with dhcpcd-base 1:10.3.2-6, dhcpcd.service enabled, and
/etc/network/interfaces containing:
allow-hotplug enp1s0
iface enp1s0 inet dhcp
2. apt install dhcpcd=1:10.5.2-1 dhcpcd-base=1:10.5.2-1
(do not restart dhcpcd.service -- the package does not do it either)
3. ifdown enp1s0
Expected: returns promptly.
Actual: prints "sending commands to dhcpcd process" and hangs forever.
At shutdown the same call is made by networking.service's ExecStop, which is
why the first reboot after this upgrade costs about three extra minutes.
BISECTION
Same VM, same kernel (7.1.12+deb14-cloud-amd64), same ifupdown (0.8.45+nmu1,
whose only change over 0.8.45 is packaging: DPKG_ROOT support and sysusers.d),
same /etc/network/interfaces. Only dhcpcd was varied:
manager 10.3.2 + client 10.3.2 -> ifdown returns in <1 s, clean shutdown
manager 10.5.2 + client 10.5.2 -> ifdown returns in <1 s, clean shutdown
manager 10.5.2 + client 10.3.2 -> ifdown returns in <1 s, clean shutdown
manager 10.3.2 + client 10.5.2 -> ifdown hangs indefinitely
Only the mismatched pair produced by upgrading without restarting the daemon
fails, and only in that direction. Reproduced on two independent VMs, and the
hang disappears on every subsequent reboot, once the manager is 10.5.2 too.
That is what makes it easy to miss: it bites exactly once per upgrade.
SUGGESTED FIX
Restart dhcpcd.service on upgrade (deb-systemd-invoke restart), so that the
manager and the client binaries cannot diverge.
Separately, and worth forwarding upstream: a client that blocks in read() on
the control socket with no timeout and no version handshake is fragile. A
manager that does not understand a command should still respond or close the
socket, and the client should not wait forever. I did not find an existing
upstream issue for this; the closest, NetworkConfiguration/dhcpcd#560, is a
different case in 10.3.0 and is closed. I have filed the upstream half as
NetworkConfiguration/dhcpcd#723:
https://github.com/NetworkConfiguration/dhcpcd/issues/723
WORKAROUND
systemctl restart dhcpcd.service
after upgrading dhcpcd-base, before the next reboot.
-- System Information (affected machine is a Debian VM, not the reporting host)
Debian Release: forky/sid
Architecture: amd64
Kernel: Linux 7.1.12+deb14-cloud-amd64
Locale: LANG=en_US.UTF-8
Versions of packages involved:
ii dhcpcd 1:10.5.2-1
ii dhcpcd-base 1:10.5.2-1
ii ifupdown 0.8.45+nmu1
ii systemd 261.2-1
ii libc6 2.43-4
(upgraded from dhcpcd/dhcpcd-base 1:10.3.2-6)
