Upstream resolved this in commit
36344cd1f1283ed609746e29137c8439c41ccd5d

(https://github.com/systemd/systemd/pull/44443)

by adding `CHASE_NO_SYNC` (`AT_STATX_DONT_SYNC`) to `chase()` and
enabling it in `fstab-generator`, preventing `statx()` from blocking on
unreachable network mounts.

PR #44369 has been closed in favor of `36344cd1f128`.

** Description changed:

  [ Impact ]
  During unattended-upgrades or needrestart on systems with unreachable or 
recovering remote network shares (NFS, CIFS) in /etc/fstab, `systemctl 
daemon-reexec` triggers `systemd-fstab-generator`.
- `fstab-generator` calls `canonicalize_mount_path()` which runs `chase()` into 
the unreachable network mount, blocking indefinitely in uninterruptible sleep 
(D-state).
+ `fstab-generator` calls `canonicalize_mount_path()` which runs `chase()`. 
Inside `chase()`, `statx()` without `AT_STATX_DONT_SYNC` forces the kernel to 
revalidate attributes of the mount point root with the remote NFS server, 
blocking indefinitely in uninterruptible sleep (D-state).
  After 90s, the generator process is killed by SIGALRM, returning -EPROTO. 
During daemon-reexec, `manager_startup()` treats this error as fatal and 
freezes PID 1 ("Freezing execution."), halting the entire OS and requiring a 
hard reboot.
  
  [ Fix ]
- `fstab-generator`: Add `canonicalize_network_mount_path()` to avoid calling 
`chase()` into already mounted remote network shares. Parse 
`/proc/self/mountinfo` without a path cache and check whether the simplified 
target path or its 1-level symlink target (`readlink_malloc()`) is already an 
active mount point. If mounted, return the canonical path directly from 
`/proc/self/mountinfo` without touching the remote VFS. Unmounted paths fall 
back to `chase()` normally.
+ Backport upstream commit 36344cd1f1283ed609746e29137c8439c41ccd5d ("chase: 
don't force a sync with the server in chase_statx() unless needed").
+ It adds the `CHASE_NO_SYNC` flag to pass `AT_STATX_DONT_SYNC` to `statx()` 
and enables it in `fstab-generator`, preventing the kernel from querying 
unreachable remote servers during path canonicalization.
  
  [ Test Plan ]
  1. Mount a remote NFS share in /etc/fstab (e.g. /mnt/nfs).
  2. Block traffic to the NFS server (e.g. `iptables -A OUTPUT -p tcp --dport 
2049 -j DROP`).
  3. Execute `systemctl daemon-reexec`.
- - Without fix: `fstab-generator` blocks in D-state, times out after 90s, PID 
1 prints "Freezing execution." and the system halts.
- - With fix: `fstab-generator` recognizes the active mount in 
`/proc/self/mountinfo` without VFS lookup, skips `chase()`, and `daemon-reexec` 
completes in <1s.
+ - Without fix: `fstab-generator` blocks in D-state in `statx()`, times out 
after 90s, PID 1 prints "Freezing execution." and the system halts.
+ - With fix: `chase()` uses `AT_STATX_DONT_SYNC`, avoiding remote RPC 
revalidation, and `daemon-reexec` completes in <1s.
  
  [ Where problems could occur ]
- - `fstab-generator`: Network mount points that are already active in 
`/proc/self/mountinfo` (directly or via a 1-level symlink) skip `chase()`; 
unmounted paths continue to be chased normally. Multi-level symlink chains or 
intermediate directory symlinks pointing to an active network mount will fall 
back to `chase()`.
+ - `chase()` / `fstab-generator`: `CHASE_NO_SYNC` is only enabled for callers 
that explicitly pass the flag (`fstab-generator`), which only canonicalizes the 
mount path and does not rely on fresh remote `st_uid` or `st_mode` attributes. 
All other `chase()` callers keep existing behavior.
  
  [ Other Info ]
- - Upstream PR: https://github.com/systemd/systemd/pull/44369
- - Upstream issue: https://github.com/systemd/systemd/issues/39354
- - NixOS issue: https://github.com/NixOS/nixpkgs/issues/375376
- - Target Ubuntu series: tenacious (27.04), stonking (26.10), resolute (26.04 
LTS), noble (24.04 LTS).
- - Development series status: The fix targets devel (tenacious) first via 
upstream merge/cherry-pick; stable SRUs target stonking, resolute, and noble.
- - Test suites: 0 regressions in systemd test suites 
(TEST-81-GENERATORS.fstab-generator.sh extended with 7 test cases).
- 
+ - Upstream commit: 
https://github.com/systemd/systemd/commit/36344cd1f1283ed609746e29137c8439c41ccd5d
 (PR #44443)
+ - Upstream issues: https://github.com/systemd/systemd/issues/44396, 
https://github.com/systemd/systemd/issues/39354
  
  --- [ Original Report ]
  
  I am running Ubuntu 24.04 on a Hyper-V virtual machine.
  
  The system became unusable after an unattended-upgrade triggered a
  systemd daemon-reexec via needrestart.
  
  Environment:
  - Ubuntu 24.04
  - systemd 255.4-1ubuntu8.17
  - SSSD 2.9.4-1.1ubuntu6.7
  - Hyper-V virtual machine
  - unattended-upgrades enabled
  - needrestart installed
  
  Timeline from the journal:
  
  Aug 27 06:16:00 systemd[1]: Starting apt-daily-upgrade.service - Daily apt 
upgrade and clean activities...
  Aug 27 06:16:38 unattended-upgrade started
  Aug 27 06:16:42 unattended-upgrade finished upgrading:
  vim, vim-common, vim-runtime, vim-tiny, xxd,
  libssl3t64, openssl
  
  Aug 27 06:16:44 systemd[1]: Reexecuting requested from client PID 769685 
('systemctl') (unit apt-daily-upgrade.service)...
  Aug 27 06:16:44 systemd[1]: Reexecuting.
  Aug 27 06:16:44 systemd[1]: systemd 255.4-1ubuntu8.17 running in system mode 
(...)
  
  About 90 seconds later:
  
  Aug 27 06:18:14 systemd[1]: Failed to fork off sandboxing environment for 
executing generators: Protocol error
  Aug 27 06:18:14 systemd[1]: Freezing execution.
  
  After that, PID 1 became effectively unresponsive.
  
  Examples of subsequent failures:
  
  dbus-daemon:
  Failed to activate service 'org.freedesktop.systemd1': timed out
  
  systemd-journald:
  Failed to send WATCHDOG=1 notification message
  
  SSSD socket activation also stopped working, so NSS lookups for domain
  users failed.
  
  Even a normal reboot was no longer possible:
  
  Call to Reboot failed: Connection timed out
  Failed to open /run/initctl: No such device or address
  Failed to talk to init daemon: Connection timed out
  
  The VM had to be hard-powered off from Hyper-V.
  
  A SysRq task dump while the system was in the failed state showed PID 1
  sleeping in waitid():
  
  task:systemd state:S pid:1
  
  Call Trace:
  __schedule
  schedule
  do_wait
  kernel_waitid
  __do_sys_waitid
  __x64_sys_waitid
  ...
  
  The system itself was otherwise not under resource pressure:
  - low load
  - sufficient free memory
  - no relevant D-state processes
  - system D-Bus daemon still alive
  
  The daemon-reexec appears to have been triggered by needrestart.
  
  The installed needrestart hook contains:
  
  /etc/needrestart/restart.d/systemd-manager:
  exec systemctl daemon-reexec
  
  needrestart internally identifies the target as "systemd-manager".
  
  As a workaround I added:
  
  /etc/needrestart/conf.d/99-no-systemd-manager-reexec.conf
  
  with:
  
  $nrconf{override_rc}{qr(^systemd-manager$)} = 0;
  
  This prevents automatic daemon-reexec of PID 1 while keeping unattended-
  upgrades enabled.
  
  A similar "Failed to fork off sandboxing environment for executing
  generators: Protocol error" event had already occurred once on the
  previous day after a systemd reexec, but the system recovered that time.
  
  This looks like a systemd daemon-reexec / generator sandboxing failure
  rather than an SSSD or Hyper-V failure. SSSD only became a downstream
  casualty after PID 1 entered "Freezing execution".
  
  Please let me know if further logs, coredumps, or a full SysRq task dump
  would be useful.
  
  ProblemType: Bug
  DistroRelease: Ubuntu 24.04
  Package: systemd 255.4-1ubuntu8.17
  ProcVersionSignature: Ubuntu 6.8.0-138.138-generic 6.8.12
  Uname: Linux 6.8.0-138-generic x86_64
  ApportVersion: 2.28.3-0ubuntu0.1
  Architecture: amd64
  CasperMD5CheckResult: pass
  CloudArchitecture: x86_64
  CloudID: none
  CloudName: none
  CloudPlatform: none
  CloudSubPlatform: config
  Date: Thu Aug 27 11:00:24 2026
  InstallationDate: Installed on 2025-03-28 (517 days ago)
  InstallationMedia: Ubuntu-Server 22.04.4 LTS "Jammy Jellyfish" - Release 
amd64 (20240216.1)
  Lspci:
  
  Lspci-vt:
  
  Lsusb: Error: command ['lsusb'] failed with exit code 1:
  Lsusb-t:
  
  Lsusb-v: Error: command ['lsusb', '-v'] failed with exit code 1:
  MachineType: Microsoft Corporation Virtual Machine
  ProcEnviron:
  LANG=en_US.UTF-8
  PATH=(custom, no user)
  SHELL=/bin/bash
  TERM=xterm
  XDG_RUNTIME_DIR=<set>
  ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-6.8.0-138-generic 
root=/dev/mapper/ubuntu--vg-ubuntu--lv ro
  SourcePackage: systemd
  UpgradeStatus: Upgraded to noble on 2025-05-26 (458 days ago)
  dmi.bios.date: 10/23/2025
  dmi.bios.release: 4.1
  dmi.bios.vendor: Microsoft Corporation
  dmi.bios.version: Hyper-V UEFI Release v4.1
  dmi.board.asset.tag: None
  dmi.board.name: Virtual Machine
  dmi.board.vendor: Microsoft Corporation
  dmi.board.version: Hyper-V UEFI Release v4.1
  dmi.chassis.asset.tag: 8801-4483-8005-5833-2836-0338-95
  dmi.chassis.type: 3
  dmi.chassis.vendor: Microsoft Corporation
  dmi.chassis.version: Hyper-V UEFI Release v4.1
  dmi.modalias: 
dmi:bvnMicrosoftCorporation:bvrHyper-VUEFIReleasev4.1:bd10/23/2025:br4.1:svnMicrosoftCorporation:pnVirtualMachine:pvrHyper-VUEFIReleasev4.1:rvnMicrosoftCorporation:rnVirtualMachine:rvrHyper-VUEFIReleasev4.1:cvnMicrosoftCorporation:ct3:cvrHyper-VUEFIReleasev4.1:skuNone:
  dmi.product.family: Virtual Machine
  dmi.product.name: Virtual Machine
  dmi.product.sku: None
  dmi.product.version: Hyper-V UEFI Release v4.1
  dmi.sys.vendor: Microsoft Corporation

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2165295

Title:
  systemd freezes after daemon-reexec triggered by unattended-upgrades /
  needrestart on Ubuntu 24.04

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2165295/+subscriptions


-- 
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Reply via email to