** Description changed:
[ Impact ]
During unattended-upgrades or needrestart on systems with unreachable or
recovering remote network shares (NFS, CIFS) in /etc/fstab, `systemctl
daemon-reexec` triggers `systemd-fstab-generator`.
`fstab-generator` calls `canonicalize_mount_path()` which runs `chase()` into
the unreachable network mount, blocking indefinitely in uninterruptible sleep
(D-state).
After 90s, the generator process is killed by SIGALRM, returning -EPROTO.
During daemon-reexec, `manager_startup()` treats this error as fatal and
freezes PID 1 ("Freezing execution."), halting the entire OS and requiring a
hard reboot.
[ Fix ]
- Two-part systemic fix:
- 1. `core/manager`: During daemon-reexec on an already running system
(`serialization && !switching_root`), do not freeze PID 1 on generator
failures. Log an error acknowledging that on-disk generated units may be
missing, and proceed with deserializing in-memory units, matching
`daemon-reload` resilience. Boot-time, switch-root, and soft-reboot handling is
untouched.
- 2. `fstab-generator`: Avoid traversing into already mounted remote network
shares. Check `/proc/self/mountinfo` for the target path or resolved symlink
without path cache; if present as an active mount point, skip chasing the last
component to avoid entering the mounted remote filesystem.
+ `fstab-generator`: Avoid traversing into already mounted remote network
shares. Check `/proc/self/mountinfo` for the target path or resolved symlink
without path cache. If present as an active mount point, skip chasing the last
component to avoid entering the mounted remote filesystem and triggering
D-state hangs. Unmounted paths and parent directory symlinks continue to be
chased normally.
[ Test Plan ]
1. Mount a remote NFS share in /etc/fstab (e.g. /mnt/nfs).
2. Block traffic to the NFS server (e.g. `iptables -A OUTPUT -p tcp --dport
2049 -j DROP`).
3. Execute `systemctl daemon-reexec`.
- - Without fix: PID 1 blocks in D-state, generator times out after 90s, PID
1 prints "Freezing execution." and the system halts.
- - With fix: `fstab-generator` recognizes the active mount in mountinfo
without VFS lookup, skips traversal, and `daemon-reexec` completes in <1s. If
any generator times out, PID 1 logs an error and continues running without
freezing.
+ - Without fix: `fstab-generator` blocks in D-state, times out after 90s, PID
1 prints "Freezing execution." and the system halts.
+ - With fix: `fstab-generator` recognizes the active mount in mountinfo
without VFS lookup, skips traversal, and `daemon-reexec` completes in <1s.
[ Where problems could occur ]
- - `core/manager`: If a generator fails during reexec, units generated on disk
will be missing, relying on deserialized units from the previous manager
instance. Cold boot, switch-root, and soft-reboot error handling is untouched.
- - `fstab-generator`: Symlinked mount points whose target is already mounted
on a network share skip last-component chasing; unmounted symlinks continue to
be chased normally.
+ - `fstab-generator`: Symlinked mount points whose target is already mounted
on a network share skip last-component chasing; unmounted symlinks continue to
be chased normally. Multi-level symlink chains and nested mounts under
unreachable network shares remain out of scope for generator canonicalization.
[ Other Info ]
- Upstream PR: https://github.com/systemd/systemd/pull/44369
- Upstream issue: https://github.com/systemd/systemd/issues/39354
- NixOS issue: https://github.com/NixOS/nixpkgs/issues/375376
- Target Ubuntu series: noble (24.04 LTS), resolute (26.04 LTS), stonking
(26.10).
+ - Development series status: The fix lands first in devel (stonking) via
upstream sync/cherry-pick; stable SRUs target noble and resolute.
+ - Test suites: 0 regressions in systemd test suites.
--- [ Original Report ]
I am running Ubuntu 24.04 on a Hyper-V virtual machine.
The system became unusable after an unattended-upgrade triggered a
systemd daemon-reexec via needrestart.
Environment:
- Ubuntu 24.04
- systemd 255.4-1ubuntu8.17
- SSSD 2.9.4-1.1ubuntu6.7
- Hyper-V virtual machine
- unattended-upgrades enabled
- needrestart installed
Timeline from the journal:
Aug 27 06:16:00 systemd[1]: Starting apt-daily-upgrade.service - Daily apt
upgrade and clean activities...
Aug 27 06:16:38 unattended-upgrade started
Aug 27 06:16:42 unattended-upgrade finished upgrading:
- vim, vim-common, vim-runtime, vim-tiny, xxd,
- libssl3t64, openssl
+ vim, vim-common, vim-runtime, vim-tiny, xxd,
+ libssl3t64, openssl
Aug 27 06:16:44 systemd[1]: Reexecuting requested from client PID 769685
('systemctl') (unit apt-daily-upgrade.service)...
Aug 27 06:16:44 systemd[1]: Reexecuting.
Aug 27 06:16:44 systemd[1]: systemd 255.4-1ubuntu8.17 running in system mode
(...)
About 90 seconds later:
Aug 27 06:18:14 systemd[1]: Failed to fork off sandboxing environment for
executing generators: Protocol error
Aug 27 06:18:14 systemd[1]: Freezing execution.
After that, PID 1 became effectively unresponsive.
Examples of subsequent failures:
dbus-daemon:
- Failed to activate service 'org.freedesktop.systemd1': timed out
+ Failed to activate service 'org.freedesktop.systemd1': timed out
systemd-journald:
- Failed to send WATCHDOG=1 notification message
+ Failed to send WATCHDOG=1 notification message
SSSD socket activation also stopped working, so NSS lookups for domain
users failed.
Even a normal reboot was no longer possible:
Call to Reboot failed: Connection timed out
Failed to open /run/initctl: No such device or address
Failed to talk to init daemon: Connection timed out
The VM had to be hard-powered off from Hyper-V.
A SysRq task dump while the system was in the failed state showed PID 1
sleeping in waitid():
task:systemd state:S pid:1
Call Trace:
- __schedule
- schedule
- do_wait
- kernel_waitid
- __do_sys_waitid
- __x64_sys_waitid
- ...
+ __schedule
+ schedule
+ do_wait
+ kernel_waitid
+ __do_sys_waitid
+ __x64_sys_waitid
+ ...
The system itself was otherwise not under resource pressure:
- low load
- sufficient free memory
- no relevant D-state processes
- system D-Bus daemon still alive
The daemon-reexec appears to have been triggered by needrestart.
The installed needrestart hook contains:
/etc/needrestart/restart.d/systemd-manager:
- exec systemctl daemon-reexec
+ exec systemctl daemon-reexec
needrestart internally identifies the target as "systemd-manager".
As a workaround I added:
/etc/needrestart/conf.d/99-no-systemd-manager-reexec.conf
with:
$nrconf{override_rc}{qr(^systemd-manager$)} = 0;
This prevents automatic daemon-reexec of PID 1 while keeping unattended-
upgrades enabled.
A similar "Failed to fork off sandboxing environment for executing
generators: Protocol error" event had already occurred once on the
previous day after a systemd reexec, but the system recovered that time.
This looks like a systemd daemon-reexec / generator sandboxing failure
rather than an SSSD or Hyper-V failure. SSSD only became a downstream
casualty after PID 1 entered "Freezing execution".
Please let me know if further logs, coredumps, or a full SysRq task dump
would be useful.
ProblemType: Bug
DistroRelease: Ubuntu 24.04
Package: systemd 255.4-1ubuntu8.17
ProcVersionSignature: Ubuntu 6.8.0-138.138-generic 6.8.12
Uname: Linux 6.8.0-138-generic x86_64
ApportVersion: 2.28.3-0ubuntu0.1
Architecture: amd64
CasperMD5CheckResult: pass
CloudArchitecture: x86_64
CloudID: none
CloudName: none
CloudPlatform: none
CloudSubPlatform: config
Date: Thu Aug 27 11:00:24 2026
InstallationDate: Installed on 2025-03-28 (517 days ago)
InstallationMedia: Ubuntu-Server 22.04.4 LTS "Jammy Jellyfish" - Release
amd64 (20240216.1)
Lspci:
Lspci-vt:
Lsusb: Error: command ['lsusb'] failed with exit code 1:
Lsusb-t:
Lsusb-v: Error: command ['lsusb', '-v'] failed with exit code 1:
MachineType: Microsoft Corporation Virtual Machine
ProcEnviron:
- LANG=en_US.UTF-8
- PATH=(custom, no user)
- SHELL=/bin/bash
- TERM=xterm
- XDG_RUNTIME_DIR=<set>
+ LANG=en_US.UTF-8
+ PATH=(custom, no user)
+ SHELL=/bin/bash
+ TERM=xterm
+ XDG_RUNTIME_DIR=<set>
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-6.8.0-138-generic
root=/dev/mapper/ubuntu--vg-ubuntu--lv ro
SourcePackage: systemd
UpgradeStatus: Upgraded to noble on 2025-05-26 (458 days ago)
dmi.bios.date: 10/23/2025
dmi.bios.release: 4.1
dmi.bios.vendor: Microsoft Corporation
dmi.bios.version: Hyper-V UEFI Release v4.1
dmi.board.asset.tag: None
dmi.board.name: Virtual Machine
dmi.board.vendor: Microsoft Corporation
dmi.board.version: Hyper-V UEFI Release v4.1
dmi.chassis.asset.tag: 8801-4483-8005-5833-2836-0338-95
dmi.chassis.type: 3
dmi.chassis.vendor: Microsoft Corporation
dmi.chassis.version: Hyper-V UEFI Release v4.1
dmi.modalias:
dmi:bvnMicrosoftCorporation:bvrHyper-VUEFIReleasev4.1:bd10/23/2025:br4.1:svnMicrosoftCorporation:pnVirtualMachine:pvrHyper-VUEFIReleasev4.1:rvnMicrosoftCorporation:rnVirtualMachine:rvrHyper-VUEFIReleasev4.1:cvnMicrosoftCorporation:ct3:cvrHyper-VUEFIReleasev4.1:skuNone:
dmi.product.family: Virtual Machine
dmi.product.name: Virtual Machine
dmi.product.sku: None
dmi.product.version: Hyper-V UEFI Release v4.1
dmi.sys.vendor: Microsoft Corporation
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2165295
Title:
systemd freezes after daemon-reexec triggered by unattended-upgrades /
needrestart on Ubuntu 24.04
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2165295/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
