https://bugzilla.kernel.org/show_bug.cgi?id=221842
Bug ID: 221842
Summary: ACPI: AB/BA deadlock between kec_query
(acpi_ec_event_processor) and kacpi_notify
(acpi_battery_notify) workers on the AML mutex - HP
ZBook Firefly 14 G10
Product: ACPI
Version: 2.5
Hardware: Intel
OS: Linux
Status: NEW
Severity: high
Priority: P3
Component: EC
Assignee: [email protected]
Reporter: [email protected]
Regression: No
Created attachment 310583
--> https://bugzilla.kernel.org/attachment.cgi?id=310583&action=edit
Full ACPI table dump (MSDM body removed - OEM Windows key)
AB/BA deadlock between the kec_query and kacpi_notify workqueue workers on the
AML mutex, on an HP ZBook Firefly 14 inch G10. The machine hard-hangs every
8-23
days. Once wedged, every subsequent EC transaction blocks in uninterruptible
sleep and the count grows without bound until the machine is unusable; recovery
is only by forced power-off and the journal ends mid-line each time.
Six occurrences captured with per-boot journals, and the last two with a
30-second-interval watchdog sampling GPE rate and D-state task count, so the
onset window is bounded to 30 seconds and the precursor is captured.
THE DEADLOCK
============
Two workers each hold the mutex the other needs:
- kec_query / acpi_ec_event_processor, evaluating an EC _Qxx query method,
blocked in acpi_ex_acquire_mutex -> acpi_os_wait_semaphore
- kacpi_notify / acpi_os_execute_deferred -> acpi_battery_notify ->
acpi_battery_refresh -> acpi_battery_get_info, blocked on the EC/AML mutex
the query worker holds
On the most recent occurrence the kernel names the cycle itself. Every other
blocked worker reports being blocked on a semaphore last held by
kworker/0:0:2924563, while kworker/0:0:2924563 reports blocked on one last held
by kworker/2:2:2931861:
INFO: task kworker/2:1:2864589 blocked for more than 61 seconds.
Tainted: G OE 7.0.0-28-generic #28~24.04.1-Ubuntu
Workqueue: kec_query acpi_ec_event_processor
Call Trace:
__schedule+0x2d0/0x660
schedule+0x27/0x90
schedule_timeout+0x104/0x110
___down_common+0x14c/0x200
__down_common+0x5a/0x150
__down_timeout+0x16/0x30
down_timeout+0x6a/0x70
acpi_os_wait_semaphore+0x7c/0x1c0
acpi_ex_system_wait_mutex+0xdf/0x140
acpi_ex_acquire_mutex_object+0x68/0x170
acpi_ex_acquire_mutex+0x8c/0x2d0
acpi_ex_opcode_2A_0T_1R+0x172/0x1f0
acpi_ds_exec_end_op+0x1bb/0x970
acpi_ps_parse_loop+0x276/0x780
acpi_ps_parse_aml+0x226/0x600
acpi_ps_execute_method+0x172/0x3e0
acpi_ns_evaluate+0x175/0x5f0
acpi_evaluate_object+0x213/0x490
acpi_ec_event_processor+0xe6/0x140
process_one_work+0x1af/0x430
INFO: task kworker/2:1:2864589 blocked on a semaphore likely last held by
task kworker/0:0:2924563
This looks like a mutex-acquisition-ordering bug in the platform DSDT/AML (HP
firmware), exposed whenever a firmware-initiated EC _Qxx query races an
OS-initiated battery Notify(). I am filing here rather than only with the
vendor
because the kernel currently has no way to break or avoid the cycle, and
because
the unbounded pile-up afterwards may itself be worth addressing.
TIMELINE OF THE MOST RECENT OCCURRENCE (2026-08-06)
===================================================
Watchdog samples every 30 s. gpe6E is the GPE 0x6E (EC) interrupt rate, Dtasks
the count of tasks in uninterruptible sleep.
[09:02:12] gpe6E=1/s Dtasks=0
[09:02:42] gpe6E=5/s Dtasks=0 <- EC event burst; upower logs battery
99->97->99 at 09:02:38-39
[09:03:12] gpe6E=1/s Dtasks=0
[09:04:12] gpe6E=1/s Dtasks=0 <- last healthy sample
[09:04:42] gpe6E=2/s Dtasks=8 <- deadlock latches inside this 30 s window
[09:05:12] gpe6E=0/s Dtasks=12 <- EC stops completing entirely
[09:15:17] gpe6E=0/s Dtasks=34 (khungtaskd all-CPU backtrace at
09:15:18)
[09:42:14] gpe6E=0/s Dtasks=92 load average 90.29
Note gpe6E drops to exactly 0/s at latch and never recovers: the EC completes
no
further transactions. The load average of 90 is entirely blocked tasks, not CPU
use. The last upower sample ever written is the 09:02:39 battery reading -
upowerd is among the first victims, which is why no further battery state is
recorded.
The trigger is consistently a battery Notify(): a charge-state perturbation
appears in upower within a few minutes of every latch.
On an earlier occurrence (2026-07-28) the same deadlock latched inside the
s2idle suspend-entry window, about 4 s after "PM: suspend entry (s2idle)", most
likely from an AC-removal notify racing suspend. That variant is worse: the
machine never reaches a low-power state. It sat drawing ~8 W sustained in a
closed bag for 84 minutes.
REPRODUCIBILITY
===============
Six occurrences: 2026-06-16, 06-18, 07-11, 07-20, 07-28, 08-06 (gaps 2, 23, 9,
8, 9 days). No deliberate reproducer - it fires on ordinary desktop use,
correlated with battery/AC state changes.
Not a regression. The first three occurred on 6.17.0-35-generic, the last three
on 7.0.0-28-generic, verified per-boot from the journals. Both series affected.
SYSTEM
======
Machine: HP ZBook Firefly 14 inch G10 Mobile Workstation PC
BIOS: HP V70 Ver. 01.12.01, 2026-05-03 (latest; fwupd reports up to date)
CPU/GPU: i7-1355U + NVIDIA RTX A500 (hybrid)
Kernels: 7.0.0-28-generic and 6.17.0-35-generic (Ubuntu 24.04.4 LTS)
Cmdline: ro quiet splash vt.handoff=7 - no ACPI parameters set
Taint: 12288 = TAINT_OOT_MODULE + TAINT_UNSIGNED_MODULE (NVIDIA DKMS only;
no warn/oops bits)
On the taint and the distro kernel: this is an Ubuntu kernel with the NVIDIA
DKMS module loaded. NVIDIA appears in none of the six deadlock traces - every
blocked task is an ACPI workqueue worker in drivers/acpi. The decisive artifact
is the acpidump, which comes from firmware and is kernel- and
distro-independent.
I can test a vanilla mainline kernel on request, though given the 8-23 day
interval a negative result would take weeks to mean anything.
WHAT I ALREADY RULED OUT
========================
thermald was the amplifier, and masking it did NOT fix it. ftrace of
acpi_ec_space_handler / submit / complete showed thermald polling DPTF/int340x
thermal-zone AML accounted for 72 of 87 EC operations in a 3-second sample
(call
path: thermald -> acpi_ev_address_space_dispatch -> acpi_ec_space_handler ->
acpi_ec_transaction). An A/B test measured GPE 0x6E at ~17/s with thermald
running vs ~2/s stopped, about 87% of EC traffic. I masked it on 2026-07-20; EC
rate has held at 1-2/s since, and the machine has still deadlocked twice, at 8
and 9 day intervals, both inside the pre-mitigation range. So reducing EC
traffic
~8.5x did not prevent the race - baseline kworker EC activity alone is
sufficient
to hit it. I mention this because "your EC is being hammered by a userspace
poller" is a reasonable first read of the symptom, and it turned out not to be
the cause.
Not a firmware event storm. ftrace of acpi_ec recorded 0 query events with all
samples SCI_EVT=0; the GPE increments are EC completion IRQs from OS-initiated
reads, not spontaneous _Qxx events.
Not fixable by BIOS update. Installed BIOS 01.12.01 is newer than the latest
SoftPaq I could find (01.09.00 / sp158083, Apr 2025), and fwupd reports up to
date.
hung_task_panic would not have captured the suspend-time variant - khungtaskd
is
freezable and that hang is inside the suspend path, so no hung-task report was
emitted for the 2026-07-28 occurrence (verified: zero "INFO: task" lines in
that
boot).
PRIOR ART
=========
The historical attempts to run Notify() handlers off the main
deferred-execution
thread were both reverted, and the stated reason matches what I observe exactly
-
events get caught behind the AML semaphore and the workers pile up until the
system dies:
b8d35192c55fb055792ff0641408eaaec7c88988
"ACPI: execute Notify() handlers on new thread"
reverted in 72945b2b90a5554975b8f72673ab7139d232a121
37605a6900f6b4d886d995751fcfeef88c4e462c
"ACPI: created a dedicated workqueue for notify() execution"
reverted in b976fe19acc565e5137e6f12af7b6633a23e6b7c
The unbounded D-state accumulation after the latch (0 -> 92 blocked tasks in 38
minutes, still climbing when power was cut) looks like the same failure the
second revert describes.
ATTACHMENTS
===========
acpidump-sanitized.txt full ACPI table dump (the key artifact;
contains the DSDT with the offending mutex
ordering)
dmesg-boot-minus-1-sanitized.txt complete kernel log for the boot that
deadlocked
hungtask-first-report.txt first hung-task report at 09:06:05, before
the
all-CPU backtrace noise
semaphore-summary.txt every "blocked on a semaphore likely last
held
by" line, showing the cycle
watchdog-window-2026-08-06.txt 30 s watchdog samples across the latch and
the
following 38 minutes
Two redactions, neither touching anything ACPI-related: the MSDM table body is
removed (it carries the OEM Windows product key), and Wi-Fi MAC/BSSID values in
the kernel log are replaced with <mac-redacted-N> placeholders, consistently
per
unique value. All 39 SSDTs, the DSDT, and the FACP are intact. Happy to supply
anything further on request.
QUESTIONS
=========
1. Is there a supportable way to break or time-bound this cycle from the kernel
side, or is a DSDT override the only route?
2. Should the post-deadlock pile-up be bounded - should the EC query workqueue
stop accepting work once a transaction has been stuck past EC_CMD_TIMEOUT,
rather than accumulating unbounded blocked workers?
3. Is there anything further worth capturing on the next occurrence? I have a
watchdog in place and can add instrumentation; ramoops is not yet
configured.
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
_______________________________________________
acpi-bugzilla mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/acpi-bugzilla
