** Description changed:
-
Package: linux (Ubuntu)
Source Package: linux
- Binary: linux-image-7.0.0-31-generic (Ubuntu 26.04 LTS / 24.04 HWE)
+ Binary: linux-image-7.0.0-31-generic (Ubuntu 26.04 LTS / 24.04 HWE / 24.10)
Upstream Subsystem: drivers/thunderbolt/ (Native Host Interface & Software
Connection Manager)
- Affected Hardware: Intel Meteor Lake / Arrow Lake USB4 Host Interface
[8086:7ec2 / 8086:7ec4], ASMedia ASM2464PD, and external PCIe NVMe direct-boot
topologies.
+ Affected Hardware: Intel Meteor Lake / Arrow Lake USB4 Host Interface
[8086:7ec2 / 8086:7ec4], AMD Ryzen 6000/7000/8000 USB4 routers, ASMedia
ASM2464PD, Intel Goshen Ridge / Titan Ridge bridges, and external PCIe NVMe
direct-boot topologies.
+
================================================================================
1. SUMMARY OF THE DEFECT
================================================================================
- When booting Linux directly from an external NVMe SSD over a USB4/Thunderbolt
4 PCIe Gen 4 x4 tunnel, motherboard UEFI firmware negotiates the link and
builds the PCIe tunnel. GRUB2 executes and loads vmlinuz and initrd into host
RAM across this tunnel.
- However, during early kernel initialization inside the initramfs,
thunderbolt.ko issues an unconditional Host Router Reset (host_reset=true).
This severs the pre-boot PCIe tunnel mid-boot, causing nvme_probe() to
encounter Master Abort (0xFFFFFFFF) and return terminal error -ENODEV. The root
filesystem device disappears permanently from the kernel bus, causing an
initramfs timeout and emergency rescue shell drop ("Gave up waiting for root
file system device").
+ When direct-booting Linux from an external NVMe SSD over a USB4/Thunderbolt 4
PCIe Gen 4 x4 tunnel, motherboard UEFI firmware negotiates the link and builds
the PCIe tunnel. GRUB2 executes and loads vmlinuz and initrd into host RAM
across this tunnel.
+
+ However, during early kernel initialization inside the initramfs,
+ thunderbolt.ko issues an unconditional Host Router Reset
+ (host_reset=true). This severs the pre-boot PCIe tunnel mid-boot,
+ causing nvme_probe() to encounter Master Abort (0xFFFFFFFF) and return
+ terminal error -ENODEV. Because the Linux driver core does not re-probe
+ devices that failed with -ENODEV, the root filesystem device disappears
+ permanently from the kernel bus, causing an initramfs timeout and
+ emergency rescue shell drop ("Gave up waiting for root file system
+ device").
+
================================================================================
2. FORENSIC ROOT CAUSE: THE TEARDOWN CASCADE
================================================================================
Tracing drivers/thunderbolt/nhi.c and drivers/thunderbolt/tb.c isolates the
exact sequence:
+
1. nhi_probe() (drivers/thunderbolt/nhi.c:1249):
Calls nhi_reset(nhi). On USB4 v2 controllers (REG_CAPS >= 0x40), because
module parameter 'host_reset' defaults to true, it writes REG_RESET_HRR (BIT 0)
to memory-mapped register REG_RESET (0x39898):
- iowrite32(REG_RESET_HRR, nhi->iobase + REG_RESET);
+ iowrite32(REG_RESET_HRR, nhi->iobase + REG_RESET);
This asserts a hardware Host Router Reset. Register ADP_PCIE_CS_0 bit
ADP_PCIE_CS_0_PE (Path Enable, BIT 31) is de-asserted, physically cutting the
PCIe tunnel.
+
2. tb_start() (drivers/thunderbolt/tb.c:3066-3070):
nhi_probe() invokes tb_domain_add(tb, host_reset), which calls
tb_start(tb, reset = true).
tb_start() enforces:
- if (reset && tb_switch_is_usb4(tb->root_switch)) {
- discover = false;
- if (usb4_switch_version(tb->root_switch) == 1)
- tb_switch_reset(tb->root_switch);
- }
+ if (reset && tb_switch_is_usb4(tb->root_switch)) {
+ discover = false;
+ if (usb4_switch_version(tb->root_switch) == 1)
+ tb_switch_reset(tb->root_switch);
+ }
Because discover is set to false, tb_discover_tunnels() and
tb_scan_switch() are completely bypassed.
+
3. Asynchronous Driver Collision (drivers/nvme/host/pci.c):
- Concurrently, nvme_probe() attempts to enumerate the storage controller at
the pre-boot ACPI address. Because the tunnel has been severed:
- nvme 0000:06:00.0: Unable to change power state from D3cold to D0,
device inaccessible
- nvme 0000:06:00.0: error -ENODEV: probe failed
- Under Linux driver core semantics, an endpoint that fails with -ENODEV is
never re-probed. Even when thunderbolt.ko eventually re-enumerates the
enclosure seconds later, it generates thunderbolt uevents, not PCI uevents. The
root partition UUID is never detected by dracut/systemd.
+ Concurrently, nvme_probe() attempts to enumerate the storage controller at
the pre-boot ACPI/PCI address. Because the tunnel has been severed:
+ nvme 0000:06:00.0: Unable to change power state from D3cold to D0,
device inaccessible
+ nvme 0000:06:00.0: error -ENODEV: probe failed
+ Under Linux driver core semantics, an endpoint that fails probe with
-ENODEV is never re-probed. Even when thunderbolt.ko eventually re-enumerates
the enclosure seconds later, it generates thunderbolt bus uevents, not PCI bus
uevents. The root partition UUID is never detected by dracut/systemd, rendering
'rootdelay=' parameters ineffective.
+
================================================================================
3. EMPIRICAL HARDWARE PROOF: BUILT-IN ADOPTION ALREADY EXISTS
================================================================================
Empirical testing on live physical hardware (Intel Core Ultra 9 275HX Arrow
Lake-HX with WD_BLACK SN7100 in ASM2464PD) proves that the Thunderbolt driver
ALREADY POSSESSES complete architectural infrastructure to support pre-boot
boot tunnels.
+
When booted with 'thunderbolt.host_reset=0':
- - nhi_reset() is bypassed: the hardware link remains uninterrupted.
+ - nhi_reset() is bypassed: the hardware PCIe tunnel remains uninterrupted.
- tb_start() retains discover = true.
- tb_discover_tunnels() executes tb_tunnel_discover_pci(), locates the active
pre-boot PCIe tunnel, and marks intermediate switches as parent->boot = true.
- In tb_scan_finalize_switch(), sw->boot triggers automated switch
authorization (sw->authorized = 1) before userspace uevents fire.
- Live sysfs telemetry confirms:
- $ cat /sys/bus/thunderbolt/devices/0-1/boot
- 1
- $ cat /sys/bus/thunderbolt/devices/0-1/authorized
- 1
- Performance when tunnel is preserved:
- - PCIe Gen 4.0 x4 at 16.0 GT/s (~64 Gbps link).
- - Buffered read: 3,587.60 MB/s; Direct write: 2,024.33 MB/s.
- - Host Memory Buffer (HMB): 64 MB host DDR5 RAM allocated via Intel VT-d
IOMMU (Write Amplification Factor dropped from 6.80 to 1.88, extending NAND
lifespan by 72%).
- The default 'host_reset=true' policy simply short-circuits this functional
subsystem.
+
+ Live sysfs telemetry confirms pre-boot adoption:
+ $ cat /sys/bus/thunderbolt/devices/0-1/boot
+ 1
+ $ cat /sys/bus/thunderbolt/devices/0-1/authorized
+ 1
+
+ Performance & Flash Endurance when tunnel is preserved:
+ - PCIe Gen 4.0 x4 at 16.0 GT/s (~64 Gbps physical link).
+ - Buffered disk read: 3,587.60 MB/s; Direct sequential write: 2,024.33 MB/s.
+ - Host Memory Buffer (HMB): 64 MB host DDR5 RAM allocated via Intel VT-d
IOMMU (Write Amplification Factor drops from 6.80 under UASP to 1.88 under
native NVMe, extending NAND endurance by 72%).
+
+ The default 'host_reset=true' policy simply short-circuits this
+ functional subsystem.
+
================================================================================
- 4. UPSTREAM COMMIT GENEALOGY & MAINTAINER ASSUMPTION
+ 4. UPSTREAM COMMIT GENEALOGY & MAINTAINER ASSUMPTIONS
================================================================================
- Commit 0fc70886569c (Mika Westerberg, Dec 2022): Introduced USB4 v2 host
router reset via REG_RESET_HRR.
- Commit 59a54c5f3dbd (Sanath S / Mario Limonciello, Jan 2024): Defaulted
host_reset to true to clear suboptimal boot firmware DisplayPort bandwidth
tunnels (HBR2 vs HBR3) and reclaim exhausted AMD PCIe BAR space for docking
stations.
- Commit 6faa39eea953 (Mika Westerberg, Feb 2024): Cemented 'discover =
false' for USB4 host routers.
- Upstream maintainers operated under the unexamined assumption that all USB4
devices are secondary, hotpluggable peripherals mounted after the OS has booted
from internal storage. They were unaware that the running root filesystem could
reside on the pre-boot PCIe tunnel.
- Related Issue: CVE-2024-53194 documents how commits 0fc70886569c and
59a54c5f3dbd cause unexpected PCIe presence drops triggering use-after-free
crashes in pciehp.
+
+ Upstream maintainers operated under the unexamined assumption that all
+ USB4 devices are secondary, hotpluggable peripherals mounted after the
+ OS has booted from internal storage. They were unaware that the running
+ root filesystem could reside on the pre-boot PCIe tunnel.
+
+ Related Vulnerability: CVE-2024-53194 documents how commits 0fc70886569c
+ and 59a54c5f3dbd cause unexpected PCIe presence drops triggering use-
+ after-free crashes in pciehp.
+
================================================================================
- 5. PROPOSED UPSTREAM LINUX KERNEL PATCH
+ 5. TESTED WORKAROUND (IMMEDIATE MITIGATION FOR USERS)
================================================================================
- We propose a two-tier safety guard for drivers/thunderbolt/:
- 1. In nhi.c: Before issuing REG_RESET_HRR, check if sibling PCIe bridges on
the root bus (external-facing or Thunderbolt ports) have active child devices
populated by boot firmware. If active downstream devices exist, skip
nhi_reset() and clear host_reset.
- 2. In tb.c: In tb_start(), inspect whether any PCIe downstream adapter is
enabled by boot firmware (tb_pci_port_is_enabled()). If active PCIe boot
tunnels exist, do not set discover = false and do not reset the router. Allow
tb_discover_tunnels() to adopt and authorize the boot storage.
- Full patch diff against upstream mainline:
-
https://github.com/StickwoodJr/usb4-nvme-direct-boot/blob/main/patches/0001-thunderbolt-preserve-pre-boot-pcie-tunnels.patch
+ Affected users direct-booting over USB4/TB4 can immediately work around the
failure without recompiling the kernel by appending the following to
GRUB_CMDLINE_LINUX in /etc/default/grub (or /etc/default/grub.d/99-usb4.cfg):
+
+ thunderbolt.host_reset=0 thunderbolt.clx=0 pcie_port_pm=off
+
+ Notes on parameters:
+ - thunderbolt.host_reset=0: Preserves the pre-boot UEFI PCIe tunnel across
kernel handover.
+ - thunderbolt.clx=0 & pcie_port_pm=off: Prevents low-power link state
retraining drops during early initqueue settling (may increase idle power draw
by 1–3W on battery).
+ - ASM2464PD Retimer Cold Reset: If transitioning between ports or warm
reboots fails to detect the drive in BIOS, a 30-second flea-power discharge
(unplug AC, hold power button 30s) resets the high-speed retimer PHY state.
+
+
================================================================================
+ 6. PROPOSED UPSTREAM LINUX KERNEL PATCH
+
================================================================================
+ We propose a two-tier safety guard in drivers/thunderbolt/:
+ 1. In nhi.c (nhi_has_active_boot_device()): Before issuing REG_RESET_HRR,
check if sibling PCIe bridges on the root bus (external-facing or Thunderbolt
ports) have active child devices populated by boot firmware. If active
downstream devices exist, skip nhi_reset() and clear host_reset.
+ 2. In tb.c (tb_switch_has_active_pcie_tunnel()): In tb_start(), inspect
whether any PCIe downstream adapter is enabled by boot firmware
(tb_pci_port_is_enabled()). If active PCIe boot tunnels exist, do not set
discover = false and do not reset the router. Allow tb_discover_tunnels() to
adopt and authorize the boot storage.
+
+ Engineering Suite & LKML Proposal Repository:
+ - GitHub: https://github.com/StickwoodJr/usb4-nvme-direct-boot
+ - Full Patch:
https://github.com/StickwoodJr/usb4-nvme-direct-boot/blob/main/patches/0001-thunderbolt-preserve-pre-boot-pcie-tunnels.patch
+ - Automated Patch Verification Tool: scripts/apply_kernel_patch.sh (--check /
--apply / --reverse)
+
```diff
--- a/drivers/thunderbolt/nhi.c
+++ b/drivers/thunderbolt/nhi.c
@@ -1158,6 +1158,11 @@ static void nhi_reset(struct tb_nhi *nhi)
return;
}
+ if (nhi_has_active_boot_device(nhi)) {
+ dev_info(nhi->dev, "preserving pre-boot PCIe tunnel for active
boot device\n");
+ return;
+ }
+
iowrite32(REG_RESET_HRR, nhi->iobase + REG_RESET);
msleep(100);
+
--- a/drivers/thunderbolt/tb.c
+++ b/drivers/thunderbolt/tb.c
@@ -3059,6 +3077,11 @@ static int tb_start(struct tb *tb, bool reset)
tb_switch_tmu_enable(tb->root_switch);
+ if (tb_switch_has_active_pcie_tunnel(tb->root_switch)) {
+ tb_info(tb, "active PCIe boot tunnel detected, preserving
topology\n");
+ reset = false;
+ }
+
if (reset && tb_switch_is_usb4(tb->root_switch)) {
discover = false;
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2167764
Title:
thunderbolt.host_reset=1 default tears down pre-boot UEFI PCIe
tunnels, breaking external NVMe direct boot over USB4
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2167764/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
