Hi Bjorn, Ilpo, and the PCI community, This patch series introduces upstream Linux kernel support for PCIe Lane Margining at Receiver (LMR) per PCI Express Base Specification Revision 7.0 / 6.0 (§ 7.7.11 and § 8.4.4).
During the review of the previous v7 standalone LMR submission (https://lore.kernel.org/linux-pci/[email protected]/), Ilpo Järvinen pointed out that feature drivers resolving link partners duplicate link traversal logic that is already present in drivers such as ASPM: "This feels like duplicating similar functionality with the aspm driver that also wants to infer ends of the link when giving a pci_dev in. The aspm driver currently does that within, but it kind of duplicating pci_bus. It would be nice to avoid the duplication and have something similar for this in PCI core." In response to this feedback, the implementation is factored into a clean, modular 3-patch stack: 1. Patch 1/3 (PCI: Add pcie_get_link_endpoints() helper): Standardized, race-safe helper in drivers/pci/pci.c and include/linux/pci.h to identify both ends of a point-to-point PCIe link. Safely inspects the subordinate bus under down_read(&pci_bus_sem) and acquires a reference (pci_dev_get()), paired symmetrically with pcie_put_link_endpoints(). Filters for Function 0 (with ARI support), validates bridge ownership (child->self == pdev) to prevent ABA pointer reuse, and handles RCiEP and empty downstream ports (-ENODEV). 2. Patch 2/3 (PCI/ASPM: Add pci_aspm_inhibit() helper for temporary link state suppression): Reference-counted mechanism (aspm_inhibit_cnt) in drivers/pci/pcie/aspm.c to temporarily disable ASPM state transitions (L0s, L1, L1SS) on an active link. Guarantees callers hold down_read(&pci_bus_sem) (via lockdep_assert_held_read), eliminating deadlocks and preventing power-saving transitions while hardware link characterization is in progress. 3. Patch 3/3 (PCI/pcie: Add PCIe Lane Margining at Receiver (LMR) support): Implements the core LMR driver (CONFIG_PCIE_LMR) in drivers/pci/pcie/margin.c exposing a debugfs interface under /sys/kernel/debug/pci/pcie_lmr_<dev>/. Strictly self-contained: 0 modified lines in drivers/pci/pci.c, pci-driver.c, or core drivers/pci/pci.h. Exposes clean lifecycle hooks in drivers/pci/probe.c and drivers/pci/remove.c, supports 2D timing and voltage margining across physical receivers (0..6), enforces Function 0 endpoint filtering (§ 7.7.11), and includes an exhaustive kselftest suite (tools/testing/selftests/pcie_lmt/). - Capability Discovery (§ 7.7.11): Probed via Extended Capability ID 0x27 on Gen4+ (>= 16 GT/s) physical links. Differentiates optional support on Gen4/Gen5 from mandatory capability presence on Gen6+ (>= 64 GT/s). - Multi-Function Device Scope (§ 7.7.11): Enforces that capability registration is strictly restricted to Function 0 on Endpoints / Upstream Ports. - Payload Decoding (Table 4-77, Cmd 88h): Accurately decodes Margin Control Capabilities (Bits 0..4); preserves Bits 7:5 as reserved 0b. - Direction Encodings (§ 4.2.18.1.2): Enforces that Left/Right and Up/Down direction bits remain 0b for symmetric receivers (Bits 6 & 7 reserved 0b). - Multi-Receiver Concurrency (§ 4.2.18.2): Governed strictly by MIndErrorSampler. If MIndErrorSampler == 0b (main data sampler), at most one receiver across the link is margined at a time. - Autonomous Speed & Width Transitions (§ 7.5.3.7, § 7.5.3.17, § 8.4.4): Software sequencing disables Downstream Component before Upstream Component on enablement, and restores Upstream Component before Downstream Component on teardown. Restoration is unconditional via pcie_capability_clear_and_set_word() (serialized by pci_lock), eliminating trylock failure traps. Ingress paths (margin_enable_write and pci_lmr_exit) strictly follow the canonical PCI locking DAG: down_read(&pci_bus_sem) -> pci_dev_lock(downstream_port) -> pci_dev_lock(upstream_port) -> mutex_lock(&mdev->lock) -> spin_lock_irqsave(&aspm_lock) Key Concurrency Guarantees: 1. Hot-Swap Identity Invariance: When disabling LMR, dynamic endpoint resolution is completely bypassed. Teardown binds strictly to the session-saved mdev->partner, which is read and pinned with pci_dev_get() under mutex_lock(&mdev->lock). 2. Surprise Removal & Fault Hardening: Hardware register accesses check pci_dev_is_disconnected() and PCI_POSSIBLE_ERROR() guards, preventing MMIO or config space bus aborts when hardware is pulled. 3. Power Management Synchronization: Both link partners are pinned in D0 using pm_runtime_resume_and_get() during enablement, and symmetrically released via pm_runtime_put() on teardown. System suspend hooks into pci_pm_prepare() where all devices in the hierarchy are guaranteed to be in D0. 4. Security Teardown Order: In pci_destroy_dev(), pci_lmr_exit() is called after pci_tsm_destroy(), preserving D0 state for TSM link operations (PCIe IDE unbind and SPDM cryptographic session teardown). - Build: Clean compile on x86_64 and ARM64; make W=1 drivers/pci/ (0 warnings). - Linters: checkpatch.pl clean across all 3 patches (0 errors, 0 warnings). - Kselftest: tools/testing/selftests/pcie_lmt/pcie_lmt.sh expanded to 287 lines, covering positive and negative boundary tests (syntax clean with bash -n). - Pre-Commit AI Review (Sashiko AI): * Patch 1 (f34fdf7f5c6b): Reviewed — Clean (0 issues). * Patch 2 (6adc680a727f): Reviewed — Clean (0 issues). * Patch 3 (c62b5f92af63): Reviewed — Clean (0 issues, "No issues found."). Signed-off-by: Priyank Rathod <[email protected]> --- Changes in v3: - Port Classification Modernization: * Replaced open-coded (PCI_EXP_TYPE_ROOT_PORT || PCI_EXP_TYPE_DOWNSTREAM) checks in drivers/pci/pcie/margin.c with canonical pcie_downstream_port(dev). * Why: Open-coded comparisons missed PCI_EXP_TYPE_PCIE_BRIDGE ports, which act as downstream ports in specific bridge topologies. This caused inverted downstream/upstream port classification, reversing lock acquisition order and causing configuration writes to link partners without holding appropriate locks. * How: Adopted core pcie_downstream_port() helper universally across margin.c. - Hardware Execution Payload Status Validation: * In pci_lmr_run_cmd(), added explicit checking of hardware execution status bits (LMR_STS_EXEC_MASK via pci_lmr_check_exec_status()) instead of relying solely on PCI configuration bus transaction return codes. * Why: A configuration register write/read succeeds at the bus transaction level (ret == 0) even when the physical receiver hardware rejects the command (NAK) or encounters excessive bit errors during stepping. * How: Explicitly map LMR_STS_EXEC_NAK (0x3) to -EOPNOTSUPP without mutating software margin state, and LMR_STS_EXEC_TOO_MANY_ERR (0x0) to -EIO while resetting both plane->timing_val and plane->voltage_val to nominal 0. - Hardware Step 0 Synchronization & Orthogonal State Preservation (Table 4-77): * Aligned step margin zeroing with PCIe Base Specification Revision 7.0 Table 4-77. * Why: Writing a Step Margin command with payload 0 is explicitly specified as a hardware NO-OP; the physical receiver sampler ignores it. The only specification- defined nominal reset is Command 0x0F ("Go to Normal Settings"), but issuing 0x0F clears both timing and voltage samplers simultaneously, causing orthogonal hardware drift in 2D margining. * How: When zeroing an axis (e.g. echo 0 > margin_timing) while the orthogonal axis is displaced (voltage_val != 0), issue 0x0F and immediately re-apply the saved orthogonal displacement via pci_lmr_issue_step(), ensuring 100% bit-accurate hardware/software state synchronization. - Multi-Receiver Concurrency Guard Hardening (§ 4.2.18.2 & § 8.4.4): * In pci_lmr_check_sample_multiple_rx(), enforced dual verification of both LMR_CAP_IND_ERROR_SAMPLER (Bit 4) and LMR_CAP_SAMPLE_MULTIPLE_RECEIVERS (Bit 5). * Why: If an active receiver uses the live data sampler (Bit 4 = 0) or cannot sample multiple receivers concurrently (Bit 5 = 0), running margining on another receiver injects errors into operational data traffic or exceeds hardware capabilities. * How: Replaced single-bit check with a bidirectional predicate requiring BOTH Bit 4 and Bit 5 to be asserted on both the targeted receiver and all currently active receivers across all lanes. If either bit is 0, reject with -EBUSY. - Non-Link Port Type Filtering: * In pci_lmr_init(), filtered out non-link PCIe port types (PCI_EXP_TYPE_RC_END, PCI_EXP_TYPE_RC_EC, PCI_EXP_TYPE_PCI_BRIDGE) per PCIe Base Spec § 7.7.11 to eliminate spurious pci_warn diagnostics on unsupported topologies. - DebugFS Subsystem Directory Portability: * Verified debugfs "pci" root directory coexistence on architectures like s390 via debugfs_lookup("pci", NULL) before debugfs_create_dir("pci", NULL), preventing -EEXIST from poisoning the root dentry pointer. - Pre-Submission Verification: * Pre-commit AI review (Sashiko AI bot): Reviewed — No issues found (0 findings, 0 concerns, Review ID 94722). * Compiler: make W=1 drivers/pci/ (0 warnings, 0 errors on x86_64 and ARM64). * Linter: checkpatch.pl clean across all patches (0 warnings, 0 errors). - Link to v2: https://lore.kernel.org/r/[email protected] Changes in v2: - Decomposed monolithic LMR driver into a 3-patch stack with dedicated core helpers: * Patch 1: PCI core link endpoint discovery helper (pcie_get_link_endpoints()). * Patch 2: Core ASPM temporary inhibition helper (pci_aspm_inhibit()). * Patch 3: PCIe Lane Margining at Receiver driver and debugfs interface. - Core Scope Discipline: Confined LMR logic exclusively to drivers/pci/pcie/margin.c and include/linux/pci.h forward declarations (0 modified lines in pci.c/pci-driver.c). - Endpoint Discovery Hardening: * Handled RCiEP and empty downstream buses returning -ENODEV. * Added Function 0 resolution (pcie_find_link_upstream_func0()) supporting PCIe ARI. * Added bridge ownership verification (child->self == pdev) to prevent ABA races. * Added symmetric pcie_put_link_endpoints() to balance pci_dev_get() references. - ASPM Inhibit Hardening: * Added lockdep_assert_held_read(&pci_bus_sem) to pci_aspm_inhibit_locked(). * Added reference counting (aspm_inhibit_cnt) to support nested/concurrent calls. - Hot-Swap & Concurrency Protections: * Eliminated hot-swap identity mismatch in margin_enable_write() by binding teardown strictly to session-saved mdev->partner instead of re-evaluating endpoints. * Added mutex protection when reading and pinning mdev->partner to eliminate UAF races. * Converted autonomous speed/width restoration to unconditional register write via pcie_capability_clear_and_set_word(), eliminating trylock failure degradation. - Specification Alignments (PCIe Base Spec r7.0): * Aligned Extended Capability ID 0x27 to § 7.7.11 and § 8.4.4. * Enforced Function 0 restriction on Endpoints / Upstream Ports (§ 7.7.11). * Preserved Reserved Bits 7:5 in Command 88h capabilities response. * Implemented MIndErrorSampler-based concurrency for multi-receiver links (§ 4.2.18.2). * Corrected direction bit encoding to 0b for symmetric receivers (§ 4.2.18.1.2). - Kselftest Overhaul: * Extended test coverage from 105 to 287 lines, adding comprehensive boundary and negative validation tests. - Link to v1: https://lore.kernel.org/r/[email protected] - Link to v7 (monolithic LMR): https://lore.kernel.org/linux-pci/[email protected]/ --- Priyank Rathod (3): PCI: Add pcie_get_link_endpoints() helper PCI/ASPM: Add pci_aspm_inhibit() helper for temporary link state suppression PCI/pcie: Add PCIe Lane Margining at Receiver (LMR) support Documentation/PCI/index.rst | 1 + Documentation/PCI/pcie-lmr.rst | 174 +++ MAINTAINERS | 8 + drivers/pci/pci.c | 152 +++ drivers/pci/pci.h | 8 + drivers/pci/pcie/Kconfig | 12 + drivers/pci/pcie/Makefile | 1 + drivers/pci/pcie/aspm.c | 100 ++ drivers/pci/pcie/margin.c | 1682 ++++++++++++++++++++++++++ drivers/pci/probe.c | 1 + drivers/pci/remove.c | 2 +- include/linux/pci.h | 17 + include/uapi/linux/pci_regs.h | 18 + tools/testing/selftests/Makefile | 1 + tools/testing/selftests/pcie_lmt/Makefile | 3 + tools/testing/selftests/pcie_lmt/pcie_lmt.sh | 405 +++++++ 16 files changed, 2584 insertions(+), 1 deletion(-) --- base-commit: cee9395acd8043be0644b25c34bfa86623f2b935 change-id: 20260831-pcie-link-endpoints-978e100d5d06 Best regards, -- Priyank Rathod <[email protected]>
