Upstream resolved this in commit
36344cd1f1283ed609746e29137c8439c41ccd5d
(https://github.com/systemd/systemd/pull/44443)
by adding `CHASE_NO_SYNC` (`AT_STATX_DONT_SYNC`) to `chase()` and
enabling it in `fstab-generator`, preventing `statx()` from blocking on
unreachable network mounts.
PR #44369 has been closed in favor of `36344cd1f128`.
** Description changed:
[ Impact ]
During unattended-upgrades or needrestart on systems with unreachable or
recovering remote network shares (NFS, CIFS) in /etc/fstab, `systemctl
daemon-reexec` triggers `systemd-fstab-generator`.
- `fstab-generator` calls `canonicalize_mount_path()` which runs `chase()` into
the unreachable network mount, blocking indefinitely in uninterruptible sleep
(D-state).
+ `fstab-generator` calls `canonicalize_mount_path()` which runs `chase()`.
Inside `chase()`, `statx()` without `AT_STATX_DONT_SYNC` forces the kernel to
revalidate attributes of the mount point root with the remote NFS server,
blocking indefinitely in uninterruptible sleep (D-state).
After 90s, the generator process is killed by SIGALRM, returning -EPROTO.
During daemon-reexec, `manager_startup()` treats this error as fatal and
freezes PID 1 ("Freezing execution."), halting the entire OS and requiring a
hard reboot.
[ Fix ]
- `fstab-generator`: Add `canonicalize_network_mount_path()` to avoid calling
`chase()` into already mounted remote network shares. Parse
`/proc/self/mountinfo` without a path cache and check whether the simplified
target path or its 1-level symlink target (`readlink_malloc()`) is already an
active mount point. If mounted, return the canonical path directly from
`/proc/self/mountinfo` without touching the remote VFS. Unmounted paths fall
back to `chase()` normally.
+ Backport upstream commit 36344cd1f1283ed609746e29137c8439c41ccd5d ("chase:
don't force a sync with the server in chase_statx() unless needed").
+ It adds the `CHASE_NO_SYNC` flag to pass `AT_STATX_DONT_SYNC` to `statx()`
and enables it in `fstab-generator`, preventing the kernel from querying
unreachable remote servers during path canonicalization.
[ Test Plan ]
1. Mount a remote NFS share in /etc/fstab (e.g. /mnt/nfs).
2. Block traffic to the NFS server (e.g. `iptables -A OUTPUT -p tcp --dport
2049 -j DROP`).
3. Execute `systemctl daemon-reexec`.
- - Without fix: `fstab-generator` blocks in D-state, times out after 90s, PID
1 prints "Freezing execution." and the system halts.
- - With fix: `fstab-generator` recognizes the active mount in
`/proc/self/mountinfo` without VFS lookup, skips `chase()`, and `daemon-reexec`
completes in <1s.
+ - Without fix: `fstab-generator` blocks in D-state in `statx()`, times out
after 90s, PID 1 prints "Freezing execution." and the system halts.
+ - With fix: `chase()` uses `AT_STATX_DONT_SYNC`, avoiding remote RPC
revalidation, and `daemon-reexec` completes in <1s.
[ Where problems could occur ]
- - `fstab-generator`: Network mount points that are already active in
`/proc/self/mountinfo` (directly or via a 1-level symlink) skip `chase()`;
unmounted paths continue to be chased normally. Multi-level symlink chains or
intermediate directory symlinks pointing to an active network mount will fall
back to `chase()`.
+ - `chase()` / `fstab-generator`: `CHASE_NO_SYNC` is only enabled for callers
that explicitly pass the flag (`fstab-generator`), which only canonicalizes the
mount path and does not rely on fresh remote `st_uid` or `st_mode` attributes.
All other `chase()` callers keep existing behavior.
[ Other Info ]
- - Upstream PR: https://github.com/systemd/systemd/pull/44369
- - Upstream issue: https://github.com/systemd/systemd/issues/39354
- - NixOS issue: https://github.com/NixOS/nixpkgs/issues/375376
- - Target Ubuntu series: tenacious (27.04), stonking (26.10), resolute (26.04
LTS), noble (24.04 LTS).
- - Development series status: The fix targets devel (tenacious) first via
upstream merge/cherry-pick; stable SRUs target stonking, resolute, and noble.
- - Test suites: 0 regressions in systemd test suites
(TEST-81-GENERATORS.fstab-generator.sh extended with 7 test cases).
-
+ - Upstream commit:
https://github.com/systemd/systemd/commit/36344cd1f1283ed609746e29137c8439c41ccd5d
(PR #44443)
+ - Upstream issues: https://github.com/systemd/systemd/issues/44396,
https://github.com/systemd/systemd/issues/39354
--- [ Original Report ]
I am running Ubuntu 24.04 on a Hyper-V virtual machine.
The system became unusable after an unattended-upgrade triggered a
systemd daemon-reexec via needrestart.
Environment:
- Ubuntu 24.04
- systemd 255.4-1ubuntu8.17
- SSSD 2.9.4-1.1ubuntu6.7
- Hyper-V virtual machine
- unattended-upgrades enabled
- needrestart installed
Timeline from the journal:
Aug 27 06:16:00 systemd[1]: Starting apt-daily-upgrade.service - Daily apt
upgrade and clean activities...
Aug 27 06:16:38 unattended-upgrade started
Aug 27 06:16:42 unattended-upgrade finished upgrading:
vim, vim-common, vim-runtime, vim-tiny, xxd,
libssl3t64, openssl
Aug 27 06:16:44 systemd[1]: Reexecuting requested from client PID 769685
('systemctl') (unit apt-daily-upgrade.service)...
Aug 27 06:16:44 systemd[1]: Reexecuting.
Aug 27 06:16:44 systemd[1]: systemd 255.4-1ubuntu8.17 running in system mode
(...)
About 90 seconds later:
Aug 27 06:18:14 systemd[1]: Failed to fork off sandboxing environment for
executing generators: Protocol error
Aug 27 06:18:14 systemd[1]: Freezing execution.
After that, PID 1 became effectively unresponsive.
Examples of subsequent failures:
dbus-daemon:
Failed to activate service 'org.freedesktop.systemd1': timed out
systemd-journald:
Failed to send WATCHDOG=1 notification message
SSSD socket activation also stopped working, so NSS lookups for domain
users failed.
Even a normal reboot was no longer possible:
Call to Reboot failed: Connection timed out
Failed to open /run/initctl: No such device or address
Failed to talk to init daemon: Connection timed out
The VM had to be hard-powered off from Hyper-V.
A SysRq task dump while the system was in the failed state showed PID 1
sleeping in waitid():
task:systemd state:S pid:1
Call Trace:
__schedule
schedule
do_wait
kernel_waitid
__do_sys_waitid
__x64_sys_waitid
...
The system itself was otherwise not under resource pressure:
- low load
- sufficient free memory
- no relevant D-state processes
- system D-Bus daemon still alive
The daemon-reexec appears to have been triggered by needrestart.
The installed needrestart hook contains:
/etc/needrestart/restart.d/systemd-manager:
exec systemctl daemon-reexec
needrestart internally identifies the target as "systemd-manager".
As a workaround I added:
/etc/needrestart/conf.d/99-no-systemd-manager-reexec.conf
with:
$nrconf{override_rc}{qr(^systemd-manager$)} = 0;
This prevents automatic daemon-reexec of PID 1 while keeping unattended-
upgrades enabled.
A similar "Failed to fork off sandboxing environment for executing
generators: Protocol error" event had already occurred once on the
previous day after a systemd reexec, but the system recovered that time.
This looks like a systemd daemon-reexec / generator sandboxing failure
rather than an SSSD or Hyper-V failure. SSSD only became a downstream
casualty after PID 1 entered "Freezing execution".
Please let me know if further logs, coredumps, or a full SysRq task dump
would be useful.
ProblemType: Bug
DistroRelease: Ubuntu 24.04
Package: systemd 255.4-1ubuntu8.17
ProcVersionSignature: Ubuntu 6.8.0-138.138-generic 6.8.12
Uname: Linux 6.8.0-138-generic x86_64
ApportVersion: 2.28.3-0ubuntu0.1
Architecture: amd64
CasperMD5CheckResult: pass
CloudArchitecture: x86_64
CloudID: none
CloudName: none
CloudPlatform: none
CloudSubPlatform: config
Date: Thu Aug 27 11:00:24 2026
InstallationDate: Installed on 2025-03-28 (517 days ago)
InstallationMedia: Ubuntu-Server 22.04.4 LTS "Jammy Jellyfish" - Release
amd64 (20240216.1)
Lspci:
Lspci-vt:
Lsusb: Error: command ['lsusb'] failed with exit code 1:
Lsusb-t:
Lsusb-v: Error: command ['lsusb', '-v'] failed with exit code 1:
MachineType: Microsoft Corporation Virtual Machine
ProcEnviron:
LANG=en_US.UTF-8
PATH=(custom, no user)
SHELL=/bin/bash
TERM=xterm
XDG_RUNTIME_DIR=<set>
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-6.8.0-138-generic
root=/dev/mapper/ubuntu--vg-ubuntu--lv ro
SourcePackage: systemd
UpgradeStatus: Upgraded to noble on 2025-05-26 (458 days ago)
dmi.bios.date: 10/23/2025
dmi.bios.release: 4.1
dmi.bios.vendor: Microsoft Corporation
dmi.bios.version: Hyper-V UEFI Release v4.1
dmi.board.asset.tag: None
dmi.board.name: Virtual Machine
dmi.board.vendor: Microsoft Corporation
dmi.board.version: Hyper-V UEFI Release v4.1
dmi.chassis.asset.tag: 8801-4483-8005-5833-2836-0338-95
dmi.chassis.type: 3
dmi.chassis.vendor: Microsoft Corporation
dmi.chassis.version: Hyper-V UEFI Release v4.1
dmi.modalias:
dmi:bvnMicrosoftCorporation:bvrHyper-VUEFIReleasev4.1:bd10/23/2025:br4.1:svnMicrosoftCorporation:pnVirtualMachine:pvrHyper-VUEFIReleasev4.1:rvnMicrosoftCorporation:rnVirtualMachine:rvrHyper-VUEFIReleasev4.1:cvnMicrosoftCorporation:ct3:cvrHyper-VUEFIReleasev4.1:skuNone:
dmi.product.family: Virtual Machine
dmi.product.name: Virtual Machine
dmi.product.sku: None
dmi.product.version: Hyper-V UEFI Release v4.1
dmi.sys.vendor: Microsoft Corporation
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2165295
Title:
systemd freezes after daemon-reexec triggered by unattended-upgrades /
needrestart on Ubuntu 24.04
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2165295/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs