** Description changed: - System Environment: - - Ubuntu Release: Ubuntu 26.04.1 LTS - - Kernel Package: linux-image-7.0.0-31-generic (7.0.0-31.31) - - Host Hardware: Alienware 16X Aurora (Intel Core Ultra 9 275HX Arrow Lake-HX) - - Thunderbolt Root Port: Intel Meteor Lake/Arrow Lake TB4 Root Port #0 [8086:7ec4] - - External Enclosure: ASMedia ASM2464PD USB4 / PCIe bridge - - Storage Drive: WD_BLACK SN7100 1TB NVMe PCIe 4.0 SSD - 1. What was expected to happen: - When booting Ubuntu directly from an external NVMe SSD connected to a rear USB4 / Thunderbolt 4 port, the motherboard UEFI BIOS successfully negotiates the 40 Gbps link, builds a pre-boot PCIe Gen 4 x4 tunnel, and loads GRUB into memory. The kernel is expected to preserve this pre-existing tunnel across the handover and mount the root filesystem to load the desktop. - - 2. What happened instead: - During early kernel initialization, the thunderbolt.ko module probes and executes nhi_reset() because thunderbolt.host_reset defaults to 1 (upstream kernel commit 59a54c5f3dbd). - - This host router reset tears down the active pre-boot PCIe tunnel created by the UEFI firmware. As a consequence: - - The external NVMe storage controller disappears from the PCIe bus (-ENODEV). - - The initramfs bootloader waits indefinitely for the root partition UUID to appear. - - Boot drops into an emergency recovery shell: "ALERT! UUID=... does not exist." - - 3. Root Cause & Verified Workaround: - Passing `thunderbolt.host_reset=0` (along with `thunderbolt.clx=0 pcie_port_pm=off`) on the kernel command line prevents nhi_reset() during probe, leaving the pre-boot UEFI tunnel intact across the kernel handover. - - With `thunderbolt.host_reset=0`: - - The external drive stays enumerated at /dev/nvme0n1 without link drops. - - Boot proceeds immediately to the desktop. - - Full PCIe Gen 4 x4 link speed (16.0 GT/s x4, ~3,588 MB/s read) and Host Memory Buffer (64 MB HMB) are fully operational. - - 4. Suggested Resolution: - The kernel thunderbolt driver should detect whether a valid pre-boot tunnel is hosting the active boot filesystem before issuing nhi_reset(), or default host_reset to 0 when pre-existing boot tunnels are active. - - 5. Reference & Attached Logs: - - Attached: version.log, lspci-vnvn.log - - Reproduction scripts and standalone test suite: https://github.com/StickwoodJr/usb4-nvme-direct-boot + Package: linux (Ubuntu) + Source Package: linux + Binary: linux-image-7.0.0-31-generic (Ubuntu 26.04 LTS / 24.04 HWE) + Upstream Subsystem: drivers/thunderbolt/ (Native Host Interface & Software Connection Manager) + Affected Hardware: Intel Meteor Lake / Arrow Lake USB4 Host Interface [8086:7ec2 / 8086:7ec4], ASMedia ASM2464PD, and external PCIe NVMe direct-boot topologies. + ================================================================================ + 1. SUMMARY OF THE DEFECT + ================================================================================ + When booting Linux directly from an external NVMe SSD over a USB4/Thunderbolt 4 PCIe Gen 4 x4 tunnel, motherboard UEFI firmware negotiates the link and builds the PCIe tunnel. GRUB2 executes and loads vmlinuz and initrd into host RAM across this tunnel. + However, during early kernel initialization inside the initramfs, thunderbolt.ko issues an unconditional Host Router Reset (host_reset=true). This severs the pre-boot PCIe tunnel mid-boot, causing nvme_probe() to encounter Master Abort (0xFFFFFFFF) and return terminal error -ENODEV. The root filesystem device disappears permanently from the kernel bus, causing an initramfs timeout and emergency rescue shell drop ("Gave up waiting for root file system device"). + ================================================================================ + 2. FORENSIC ROOT CAUSE: THE TEARDOWN CASCADE + ================================================================================ + Tracing drivers/thunderbolt/nhi.c and drivers/thunderbolt/tb.c isolates the exact sequence: + 1. nhi_probe() (drivers/thunderbolt/nhi.c:1249): + Calls nhi_reset(nhi). On USB4 v2 controllers (REG_CAPS >= 0x40), because module parameter 'host_reset' defaults to true, it writes REG_RESET_HRR (BIT 0) to memory-mapped register REG_RESET (0x39898): + iowrite32(REG_RESET_HRR, nhi->iobase + REG_RESET); + This asserts a hardware Host Router Reset. Register ADP_PCIE_CS_0 bit ADP_PCIE_CS_0_PE (Path Enable, BIT 31) is de-asserted, physically cutting the PCIe tunnel. + 2. tb_start() (drivers/thunderbolt/tb.c:3066-3070): + nhi_probe() invokes tb_domain_add(tb, host_reset), which calls tb_start(tb, reset = true). + tb_start() enforces: + if (reset && tb_switch_is_usb4(tb->root_switch)) { + discover = false; + if (usb4_switch_version(tb->root_switch) == 1) + tb_switch_reset(tb->root_switch); + } + Because discover is set to false, tb_discover_tunnels() and tb_scan_switch() are completely bypassed. + 3. Asynchronous Driver Collision (drivers/nvme/host/pci.c): + Concurrently, nvme_probe() attempts to enumerate the storage controller at the pre-boot ACPI address. Because the tunnel has been severed: + nvme 0000:06:00.0: Unable to change power state from D3cold to D0, device inaccessible + nvme 0000:06:00.0: error -ENODEV: probe failed + Under Linux driver core semantics, an endpoint that fails with -ENODEV is never re-probed. Even when thunderbolt.ko eventually re-enumerates the enclosure seconds later, it generates thunderbolt uevents, not PCI uevents. The root partition UUID is never detected by dracut/systemd. + ================================================================================ + 3. EMPIRICAL HARDWARE PROOF: BUILT-IN ADOPTION ALREADY EXISTS + ================================================================================ + Empirical testing on live physical hardware (Intel Core Ultra 9 275HX Arrow Lake-HX with WD_BLACK SN7100 in ASM2464PD) proves that the Thunderbolt driver ALREADY POSSESSES complete architectural infrastructure to support pre-boot boot tunnels. + When booted with 'thunderbolt.host_reset=0': + - nhi_reset() is bypassed: the hardware link remains uninterrupted. + - tb_start() retains discover = true. + - tb_discover_tunnels() executes tb_tunnel_discover_pci(), locates the active pre-boot PCIe tunnel, and marks intermediate switches as parent->boot = true. + - In tb_scan_finalize_switch(), sw->boot triggers automated switch authorization (sw->authorized = 1) before userspace uevents fire. + Live sysfs telemetry confirms: + $ cat /sys/bus/thunderbolt/devices/0-1/boot + 1 + $ cat /sys/bus/thunderbolt/devices/0-1/authorized + 1 + Performance when tunnel is preserved: + - PCIe Gen 4.0 x4 at 16.0 GT/s (~64 Gbps link). + - Buffered read: 3,587.60 MB/s; Direct write: 2,024.33 MB/s. + - Host Memory Buffer (HMB): 64 MB host DDR5 RAM allocated via Intel VT-d IOMMU (Write Amplification Factor dropped from 6.80 to 1.88, extending NAND lifespan by 72%). + The default 'host_reset=true' policy simply short-circuits this functional subsystem. + ================================================================================ + 4. UPSTREAM COMMIT GENEALOGY & MAINTAINER ASSUMPTION + ================================================================================ + - Commit 0fc70886569c (Mika Westerberg, Dec 2022): Introduced USB4 v2 host router reset via REG_RESET_HRR. + - Commit 59a54c5f3dbd (Sanath S / Mario Limonciello, Jan 2024): Defaulted host_reset to true to clear suboptimal boot firmware DisplayPort bandwidth tunnels (HBR2 vs HBR3) and reclaim exhausted AMD PCIe BAR space for docking stations. + - Commit 6faa39eea953 (Mika Westerberg, Feb 2024): Cemented 'discover = false' for USB4 host routers. + Upstream maintainers operated under the unexamined assumption that all USB4 devices are secondary, hotpluggable peripherals mounted after the OS has booted from internal storage. They were unaware that the running root filesystem could reside on the pre-boot PCIe tunnel. + Related Issue: CVE-2024-53194 documents how commits 0fc70886569c and 59a54c5f3dbd cause unexpected PCIe presence drops triggering use-after-free crashes in pciehp. + ================================================================================ + 5. PROPOSED UPSTREAM LINUX KERNEL PATCH + ================================================================================ + We propose a two-tier safety guard for drivers/thunderbolt/: + 1. In nhi.c: Before issuing REG_RESET_HRR, check if sibling PCIe bridges on the root bus (external-facing or Thunderbolt ports) have active child devices populated by boot firmware. If active downstream devices exist, skip nhi_reset() and clear host_reset. + 2. In tb.c: In tb_start(), inspect whether any PCIe downstream adapter is enabled by boot firmware (tb_pci_port_is_enabled()). If active PCIe boot tunnels exist, do not set discover = false and do not reset the router. Allow tb_discover_tunnels() to adopt and authorize the boot storage. + Full patch diff against upstream mainline: + https://github.com/StickwoodJr/usb4-nvme-direct-boot/blob/main/patches/0001-thunderbolt-preserve-pre-boot-pcie-tunnels.patch + ```diff + --- a/drivers/thunderbolt/nhi.c + +++ b/drivers/thunderbolt/nhi.c + @@ -1158,6 +1158,11 @@ static void nhi_reset(struct tb_nhi *nhi) + return; + } + + + if (nhi_has_active_boot_device(nhi)) { + + dev_info(nhi->dev, "preserving pre-boot PCIe tunnel for active boot device\n"); + + return; + + } + + + iowrite32(REG_RESET_HRR, nhi->iobase + REG_RESET); + msleep(100); + --- a/drivers/thunderbolt/tb.c + +++ b/drivers/thunderbolt/tb.c + @@ -3059,6 +3077,11 @@ static int tb_start(struct tb *tb, bool reset) + tb_switch_tmu_enable(tb->root_switch); + + + if (tb_switch_has_active_pcie_tunnel(tb->root_switch)) { + + tb_info(tb, "active PCIe boot tunnel detected, preserving topology\n"); + + reset = false; + + } + + + if (reset && tb_switch_is_usb4(tb->root_switch)) { + discover = false;
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2167764 Title: thunderbolt.host_reset=1 default tears down pre-boot UEFI PCIe tunnels, breaking external NVMe direct boot over USB4 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2167764/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
