Adding another instance of this bug. I think I'm running into the same
issue on one of my desktop machines. Debugged it with the help of an LLM
and looks to be fixed in upstream. I was able to get a stack trace using
netconsole since the network stack is still functional when this occurs.
I can reproduce fairly reliably if someone needs a machine to test the
fix on.
LLM analysis and prescription below:
-----------------------------------------------
Confirming this on a discrete Navi 23 desktop, and I was able to capture the
kernel state during the hang. The traces below point at the AMD display Idle
State Manager (ISM), not at PCI/ASPM.
Hardware / software
Ubuntu 26.04.1 LTS, GNOME/Wayland
Kernel 7.0.0-30.30-generic (7.0.12)
GPU: Navi 23 [Radeon RX 6600] [1002:73ff] rev c7, Sapphire [1da2:e451]
Board: Gigabyte B450 I AORUS PRO WIFI, Ryzen, single dGPU
Symptom
Reboot or poweroff hangs after userspace is gone. The machine stays
powered, screens go to standby, and it never resets. It still answers
ICMP, but every TCP port is closed and SysRq does nothing (by that point
device_shutdown() has already torn down USB, so there is no input layer
left to reach). Only a hard power cycle recovers it.
Reproduced 2 times in 4 consecutive reboots, which matches the ~50% in
this bug's title.
Capture method
Nothing about this reaches the disk journal: the hang happens inside the
reboot(2) syscall, long after journald has exited. Every affected boot's
journal ends at the ordinary "Stopping systemd-journal-flush.service"
line, identical to a clean shutdown, which is why this is hard to
diagnose from logs alone.
I captured it with netconsole to another machine, plus ignore_loglevel
and systemd.log_target=kmsg. The network stack stays alive throughout,
so netconsole keeps streaming after the disk journal is gone.
hung_task output (repeated identically at 122s, 245s and 368s; no other
task ever joined):
INFO: task kworker/u50:3:136 blocked for more than 122 seconds.
Tainted: P O 7.0.0-30-generic #30-Ubuntu
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:kworker/u50:3 state:D stack:0 pid:136 tgid:136 ppid:2
task_flags:0x4208060 flags:0x00080000
Workqueue: events_unbound dm_ism_delayed_work_func [amdgpu]
Call Trace:
<TASK>
__schedule+0x2bb/0x650
schedule+0x27/0x90
schedule_preempt_disabled+0x15/0x30
__mutex_lock.constprop.0+0x550/0xaf0
? srso_return_thunk+0x5/0x5f
__mutex_lock_slowpath+0x13/0x20
mutex_lock+0x3b/0x50
dm_ism_delayed_work_func+0x2e/0x60 [amdgpu]
process_one_work+0x1ac/0x3d0
worker_thread+0x1b8/0x360
? _raw_spin_lock_irqsave+0xe/0x20
? srso_return_thunk+0x5/0x5f
? __pfx_worker_thread+0x10/0x10
kthread+0xf7/0x130
? __pfx_kthread+0x10/0x10
ret_from_fork+0x195/0x2a0
? __pfx_kthread+0x10/0x10
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
INFO: task kworker/u50:3:136 is blocked on a mutex likely owned by task
reboot:6348.
INFO: task reboot:6348 blocked for more than 122 seconds.
Tainted: P O 7.0.0-30-generic #30-Ubuntu
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:reboot state:D stack:0 pid:6348 tgid:6348 ppid:1
task_flags:0x400100 flags:0x00080800
Call Trace:
<TASK>
__schedule+0x2bb/0x650
schedule+0x27/0x90
schedule_timeout+0xcf/0x110
wait_for_completion+0x81/0x140
__flush_work+0x270/0x3c0
? __pfx_wq_barrier_func+0x10/0x10
disable_delayed_work_sync+0x6a/0x90
amdgpu_dm_ism_disable+0x3a/0xa0 [amdgpu]
dm_suspend+0x90/0x390 [amdgpu]
? srso_return_thunk+0x5/0x5f
? smuio_v11_0_6_update_rom_clock_gating+0x66/0x150 [amdgpu]
? srso_return_thunk+0x5/0x5f
? nv_common_set_clockgating_state+0xcd/0xe0 [amdgpu]
? srso_return_thunk+0x5/0x5f
amdgpu_ip_block_suspend+0x27/0x60 [amdgpu]
amdgpu_device_ip_suspend_phase1+0xb5/0x180 [amdgpu]
? srso_return_thunk+0x5/0x5f
? cancel_delayed_work_sync+0x4a/0x80
amdgpu_device_suspend+0xb3/0x2b0 [amdgpu]
amdgpu_pci_shutdown+0x63/0xa0 [amdgpu]
pci_device_shutdown+0x37/0x90
device_shutdown+0x148/0x200
? srso_return_thunk+0x5/0x5f
kernel_restart+0x3a/0x70
__do_sys_reboot+0x156/0x250
__x64_sys_reboot+0x1b/0x30
[...]
Analysis
This is an ABBA deadlock between the reboot path and an ISM worker:
reboot(2) -> device_shutdown() -> amdgpu_pci_shutdown()
-> amdgpu_device_suspend() -> dm_suspend()
-> amdgpu_dm_ism_disable() [holds dm->dc_lock]
-> disable_delayed_work_sync() -> waits for the worker
dm_ism_delayed_work_func() [needs dm->dc_lock]
-> mutex_lock() -> blocks forever
The kernel identifies it explicitly:
"task kworker/u50:3:136 is blocked on a mutex likely owned by task
reboot:6348"
It is intermittent because it requires an ISM delayed work item to be
in flight at the moment shutdown begins.
Upstream status: already fixed
Introduced: 754003486c3c "drm/amd/display: Add Idle state manager(ISM)"
Fixed: 3714fe242592 "drm/amd/display: Fix ISM dc_lock deadlock
during suspend" (Ray Wu, AMD)
The upstream commit message describes this exact failure:
"amdgpu_dm_ism_disable()
is called under dc_lock and waits for ISM delayed work via
disable_delayed_work_sync(). The work handlers themselves take dc_lock,
producing an ABBA deadlock when a worker is in flight at suspend time."
The fix splits the disable path into two phases with opposite locking
contracts, so workers are quiesced before dc_lock is taken. It is small:
3 files, +73/-16, confined to amdgpu_dm.c and amdgpu_dm_ism.{c,h}.
Release status:
v7.0 no ISM code at all
v7.1 ISM present, buggy ordering (mutex_lock(&dm->dc_lock) then
amdgpu_dm_ism_disable(dm))
v7.2 fixed
Ubuntu's 7.0.0-x backported ISM from 7.1 without the 7.2 fix, which is
why this appears on a 7.0-based kernel that upstream never shipped ISM
in. Note also that current upstream guards the contract with
lockdep_assert_not_held(&dm->dc_lock), which compiles to nothing without
CONFIG_PROVE_LOCKING, so production kernels give no warning before
wedging.
On the PCI/ASPM attribution in this bug's title
I do not think ASPM is involved. The reset instruction is never reached:
the deadlock is inside device_shutdown(), before any PCI power
transition or platform reset. That is also consistent with the later
comment here reporting the same regression on a Phoenix iGPU with no
discrete GPU and no PCIe switch.
Suggested action
Backport 3714fe242592 to the 7.0 kernel.
Likely duplicates / related
LP#2162928 - [amdgpu][Navi 44] intermittent poweroff hang on
7.0.0-27/28/29; reports 7.0.0-22 unaffected, which brackets
the ISM backport to between -22 and -27.
LP#2162679 - NULL deref in dc_allow_idle_optimizations_internal via
dm_ism_delayed_work_func; same subsystem.
Full netconsole capture available on request.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2162959
Title:
System hangs on shutdown/reboot ~50% of the time since kernel 7.0.0-28
(PCI/ASPM regression)
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2162959/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
