** Description changed:

+ SRU Justification for Resolute
+ 
+ [Impact]
+ On Dell systems with Intel PCIe Bluetooth controllers (CID: F14-DVT2-C1X), 
the btintel_pcie reset and coredump paths have races. Two hardware error events 
at the same time can pick different reset types for one recovery. A duplicate 
interrupt can requeue a coredump while the previous one is still finishing.
+ 
+ The earlier fix for the boot-time NULL pointer crash (LP: #2163723) did
+ not close all of these races. The ODM stress test could still trigger
+ them. This is a follow-up for the two commits that are still missing
+ from the resolute oem-7.0 tree.
+ 
+ Original crash, about 1 in 500 boots:
+ 
+ kernel: BUG: kernel NULL pointer dereference, address: 0000000000000db0
+ kernel: RIP: 0010:btintel_pcie_dump_debug_registers+0x20/0x8f0 [btintel_pcie]
+ kernel: Call Trace:
+ kernel: btintel_pcie_irq_msix_handler+0x1a2/0x230 [btintel_pcie]
+ kernel: irq_thread_fn+0x26/0x70
+ 
+ Other problems from the two races:
+ - A reset can run with the wrong type (PLDR instead of FLR).
+ - The debug register dump runs during an FLR. The Bluetooth MMIO window can 
read all 1s or trigger AER.
+ 
+ [Fix]
+ Two btintel_pcie patches. Both are in v7.3.
+ 
+ 86f866189361 Bluetooth: btintel_pcie: split coredump worker into per-trigger 
works
+ Splits the one coredump worker into three workers: coredump, hw exception and 
fw trigger. Each worker owns one guard bit. A new helper, 
btintel_pcie_queue_coredump(), is the only place that sets COREDUMP_INPROGRESS.
+ 
+ e0650618ee83 Bluetooth: btintel_pcie: serialize reset_type with 
RECOVERY_IN_PROGRESS
+ reset_type is now written only after RECOVERY_IN_PROGRESS is won. The 
sysfs/hci reset callback always requests FLR. hw_error() skips the register 
dump when a reset is already running.
+ 
+ The second patch builds on the first. Both are clean cherry-picks of the
+ upstream commits.
+ 
+ [Test Plan]
+ On a Dell system with an Intel PCIe Bluetooth controller, run the 
wireless/Bluetooth stress test over many boot cycles (about 1000):
+ $ dmesg | grep -E "btintel_pcie|NULL pointer"
+ 
+ Also trigger resets while Bluetooth is in use:
+ $ for i in $(seq 100); do echo 1 | sudo tee /sys/class/bluetooth/hci0/reset; 
sleep 5; done
+ $ dmesg | grep -E "btintel_pcie|BUG|Call Trace"
+ 
+ Without patches: NULL pointer dereference in 
btintel_pcie_dump_debug_registers shows up during boot or reset, or the wrong 
reset type runs.
+ With patches: no call trace, no NULL pointer dereference, Bluetooth comes 
back after each reset. The ODM confirmed these patches fix the issue on their 
test kernel.
+ 
+ [Where problems could occur]
+ Could break the btintel_pcie reset and coredump paths on Intel PCIe Bluetooth 
controllers (Blazar and Scorpius Peak families).
+ 
+ If the new worker split has a bug, a coredump could be dropped or
+ garbled. A guard bit that never clears would block later coredumps.
+ Teardown could also hang if the disable/enable counts for the three
+ workers do not match.
+ 
+ If the RECOVERY_IN_PROGRESS gate in hw_error() is too strict, a real
+ hardware error could be ignored while a reset is running. Bluetooth
+ would then stay dead until the next reset. If the FLR request is wrong,
+ sysfs and HCI timeout resets would fail to bring the controller back.
+ 
+ =======================================
+ 
  [Impact]
  Bluetooth call trace shows up during boot on Dell systems with Intel PCIe 
Bluetooth controllers (CID: F14-DVT2-C1X). It can hang the system's 
shutdown/boot flow. Hit rate is about 1 in 500 boots.
  
  Error log:
  
  kernel: BUG: kernel NULL pointer dereference, address: 0000000000000db0
  kernel: RIP: 0010:btintel_pcie_dump_debug_registers+0x20/0x8f0 [btintel_pcie]
  kernel: Call Trace:
  kernel:  ? bt_err+0x7a/0xa7 [bluetooth]
  kernel:  btintel_pcie_irq_msix_handler+0x1a2/0x230 [btintel_pcie]
  kernel:  irq_thread_fn+0x26/0x70
  kernel:  irq_thread+0xbc/0x1c0
  kernel:  kthread+0xf7/0x130
  kernel:  ret_from_fork+0x195/0x2a0
  kernel:  ret_from_fork_asm+0x1a/0x30
  
  The crash hits in the MSI-X interrupt thread while dumping debug
  registers, and the reset/coredump path around it has several races that
  can corrupt driver state or drop RX events during a coredump.
  
  [Fix]
  Eight btintel_pcie driver fixes, in dependency order. Merged in v7.2:
  
  e43b33bf8d671 Bluetooth: btintel_pcie: Load IOSF debug regs by controller 
variant
  2d00975c841fa Bluetooth: btintel_pcie: Support Product level reset
  975a70ff0aec8 Bluetooth: btintel_pcie: Add support for smart trigger dump
  4d62d88e7ae6d Bluetooth: btintel_pcie: Add 50 ms delay before MAC init on 
BlazarIW
  f70f7f2512c6b Bluetooth: btintel_pcie: Separate coredump work from RX work
  
  These add product-level reset support, smart trigger coredumps, and a
  timing fix for BlazarIW fast-restart, and move coredump processing off
  the RX workqueue so RX events don't get dropped while a coredump runs.
  They land ahead of the reset-path rework below.
  
  Not yet in a released kernel, merged in the bluetooth-next tree (not yet
  released upstream):
  
  603b91aeb20a2 Bluetooth: btintel_pcie: Refactor FLR to use device_reprobe()
  ca75417ab1793 Bluetooth: btintel_pcie: split coredump worker into per-trigger 
works
  bd8bee79e1fa8 Bluetooth: btintel_pcie: serialize reset_type with 
RECOVERY_IN_PROGRESS
  
  These rework the reset/coredump internals: FLR now shares the same
  teardown/re-init path as PLDR through device_reprobe() instead of an
  open-coded, easy-to-desync copy; the combined coredump worker is split
  into three single-purpose workers (coredump, hw-exception, fw-trigger)
  each owning one guard bit, closing a race where a duplicate IRQ could
  requeue while the previous coredump was still finishing; and reset_type
  is now written only after the RECOVERY_IN_PROGRESS bit is won, closing a
  window where two concurrent hardware-error events could pick different
  reset types for the same recovery attempt.
  
  These 3 follow-up patches were requested directly by the ODM (Hugh Chao)
  after testing showed the debug-register fix alone was not enough; they
  close races that the ODM's stress test could still trigger.
  
  [Test Plan]
  Reproduce on the affected Dell platform with the Intel PCIe Bluetooth 
controller:
  Run WB (wireless/Bluetooth) stress test across repeated boot cycles (the 
ODM's test hit the bug at roughly 1/500 boots).
  
  Without patches: btintel_pcie_dump_debug_registers NULL pointer
  dereference call trace shows up in dmesg during boot, can block
  shutdown.
  
  With patches: no call trace, no NULL pointer dereference across repeated
  boot/stress cycles. ODM confirmed the 3 bluetooth-next patches fixed the
  issue on their test kernel.
  
  [Where problems could occur]
  Could break the Bluetooth btintel_pcie driver's reset and coredump paths on 
Intel PCIe Bluetooth controllers (Blazar and Scorpius Peak families).
  
  If the IOSF base address selection by variant is wrong for some SKU,
  coredump reads would return garbage instead of crashing, making register
  dumps useless without an obvious symptom.
  
  If the FLR-to-device_reprobe() refactor has a bug, a function-level
  reset could leave the Bluetooth hci device half torn down, causing
  btintel_pcie_probe() to fail on reprobe and BT to stay non-functional
  after a reset event, instead of the driver silently working around it as
  before.
  
  If the split coredump workers or the RECOVERY_IN_PROGRESS serialization
  miss a case, two reset/coredump triggers could still race, corrupting a
  coredump buffer or leaving a stale reset_type, which would show up as an
  incomplete or garbled coredump, or the wrong reset variant (FLR vs PLDR)
  running when a hardware error hits.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2163723

Title:
  btintel_pcie NULL pointer call trace during boot on Intel Bluetooth
  PCIe controllers

To manage notifications about this bug go to:
https://bugs.launchpad.net/hwe-next/+bug/2163723/+subscriptions


-- 
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Reply via email to