Thank you all for review suggestions. Closing the loop on v3. I will rework
patch series v4 as per the review feedback.
Current understanding of what changes in v4:
- Create v4 on top of cxl_reset v9 from Srirangan. Move the patch series to
Linus's RC tree and apply Srirangan's cxl-reset support patch v9 on it.
- Patch 01 is dropped; vfio-cxl reads pdev->hdm directly.
- The memdev is created at bind via devm_cxl_probe_mem() and is removed on
unbind. Probe takes no action that outlives it.
- Volatile state (DVSEC shadow, CM snapshot, ecap handlers) is acquired at
open_device() and released at close_device(). Nothing volatile is read at
bind.
- HDM mmap will make use of huge pfnmap as seen in vfio mmap routine.
- DVSEC is handled through ecap_perms readfn/writefn with a range check against
the CXL DVSEC body; other DVSECs pass through unchanged.
- UAPI: the flag is VFIO_DEVICE_FLAGS_CXL; region index fields are removed from
the device cap; component-register geometry moves to a region info cap; there
is no firmware-committed flag.
- The CM capability snapshot uses readq() throughout; the lock-on-commit state
machine is corrected so COMMIT is read-only while locked.
- The file moves to drivers/vfio/pci/cxl/ with a private header; duplicate
register defines are dropped for the upstream ones; devres uses
devm_add_action_or_reset().
- vfio-pci loads vfio-cxl through request_module() only when it finds a CXL
device; vfio-cxl registers its ops at module_init; each bound device holds a
module reference from probe to release.
- vfio-cxl owns the whole component BAR via pci_request_selected_regions(). A
cxl_register_map flag (skip_sub_bar_request) lets cxl-core skip its HDM/RAS
sub-block claim for an owning driver; cxl_pci is unchanged.
- The HPA is resolved before the guest sees the device; the guest only relocates
CXL memory in GPA space. Range readiness happens at enumeration in cxl-core,
so
vfio-cxl waits on nothing and consumes a ready resource (No await_range_active
exported)
--
Device lifecycle in v4
cxl-pci discovery / enumeration
- pdev->hdm is populated (decoder settings, count, base and size) via cxl_reset
v9.
- No memdev here; a passthrough device binds to vfio-pci, not cxl_pci.
vfio-pci probe / vfio-cxl bind (reversible)
- Read hdm_hpa and hdm_size from pdev->hdm as u64 on vfio_pci_core_device.
- Walk the PCIe ecap for the CXL DVSEC body range; record offset and size.
- Record the component-register BAR index.
- request_module("vfio-cxl"); take one module reference, held until release.
- Create the memdev via devm_cxl_probe_mem(); devm removes it on unbind.
- Register pfn_address_space over the HDM range for error containment.
- Request the whole component BAR; cxl-core skips its sub-block claim.
vfio-pci open_device (volatile state, device-open lifetime)
- Read the DVSEC body into a shadow; read the CM array with readq().
- Validate pdev->hdm against the hardware.
- Install the ecap_perms DVSEC handler (range check, fall through for others).
- Allocate cxl_vfio_state and set vdev->cxl.
vfio-pci close_device
- Remove the DVSEC handler; free the shadow and snapshot; clear vdev->cxl.
vfio-pci remove / unbind
- Release the component BAR; unregister pfn_address_space; drop the module
reference.
- The memdev is auto-removed by devm.
I will post v4 on this basis.
Thanks,
Manish
> -----Original Message-----
> From: Manish Honap <[email protected]>
> Sent: 13 July 2026 22:14
> To: Dave Jiang <[email protected]>; [email protected];
> [email protected]; [email protected]; [email protected]; Ankit Agrawal
> <[email protected]>; [email protected];
> [email protected]; [email protected]; [email protected];
> [email protected]; [email protected]
> Cc: Neo Jia <[email protected]>; Krishnakant Jaju <[email protected]>;
> Vikram Sethi <[email protected]>; Zhi Wang <[email protected]>;
> [email protected]; [email protected]; linux-
> [email protected]; [email protected]; linux-
> [email protected]; Manish Honap <[email protected]>
> Subject: RE: [PATCH v3 00/11] vfio/pci: Add CXL Type-2 device
> passthrough support
>
> > -----Original Message-----
> > From: Dave Jiang <[email protected]>
> > Sent: 10 July 2026 21:57
> > To: Manish Honap <[email protected]>; [email protected];
> > [email protected]; [email protected]; [email protected]; Ankit Agrawal
> > <[email protected]>; [email protected];
> > [email protected]; [email protected]; [email protected];
> > [email protected]; [email protected]
> > Cc: Neo Jia <[email protected]>; Krishnakant Jaju <[email protected]>;
> > Vikram Sethi <[email protected]>; Zhi Wang <[email protected]>;
> > [email protected]; [email protected]; linux-
> > [email protected]; [email protected]; linux-
> > [email protected]
> > Subject: Re: [PATCH v3 00/11] vfio/pci: Add CXL Type-2 device
> > passthrough support
> >
> > External email: Use caution opening links or attachments
> >
> >
> > On 6/25/26 9:53 AM, [email protected] wrote:
> > > From: Manish Honap <[email protected]>
> > >
> > > CXL Type-2 accelerators (CXL.mem-capable GPUs and similar) cannot be
> > > passed through to virtual machines with stock vfio-pci because the
> > > driver has no concept of HDM decoder management, HDM region
> > > exposure, or component register virtualization. This series adds
> > > those three pieces, sufficient for a guest to use the device's
> > > firmware-committed coherent memory under UVM / ATS.
> > >
> > > v3 is a rewrite of the v2 framework form, responding to Dan's
> > > request in the v2 review for "less emulation, narrower interfaces,
> > > and a closer mapping to the spec language."
> > > In this release, cxl-core exposes four EXPORT_SYMBOL_GPL helpers
> > > behind an opaque handle. vfio-pci becomes a thin transport on top
> > > of
> > those.
> > > Please see "Changes since v2" and "Reviewer feedback addressed"
> > > below for the per-area summary.
> > >
> > > Motivation
> > > ==========
> > >
> > > A CXL Type-2 device exposes its HDM-mapped device memory through HDM
> > > decoders that BIOS programs and commits at boot. To pass such a
> > > device to a guest, vfio-pci has to do three things at once:
> > >
> > > 1. Surface the firmware-committed HDM-mapped HPA range as a guest-
> > > mmappable region.
> > >
> > > 2. Surface a CXL-spec-compliant view of the CXL Device DVSEC body,
> > > the HDM Decoder Capability block, and the CXL.cache/mem cap-
> array
> > > prefix, so the guest's CXL driver enumerates the same topology
> > > the host saw.
> > >
> > > 3. Keep the host's committed decoder configuration intact (the
> > > physical decoder is never reprogrammed) while letting the guest
> > > observe and manage a shadow that follows the per-field write
> > > semantics in the spec.
> > >
> > > The series builds on Alejandro Lucero-Palau's v28 work applied on
> > > for-7.3/cxl-type2-enabling [1] (sfc is the in-tree consumer today).
> > > vfio-pci becomes the second consumer.
> > >
> > > Architecture
> > > ============
> > >
> > > cxl-core owns the CXL semantics. A new file
> > > drivers/cxl/core/passthrough.c (gated by hidden Kconfig
> > > CXL_VFIO_PASSTHROUGH) provides four exported symbols:
> > >
> > > struct cxl_passthrough *
> > > devm_cxl_passthrough_create(struct device *dev,
> > > struct cxl_dev_state *cxlds);
> > >
> > > int cxl_passthrough_dvsec_rw(p, off, val, sz, write);
> > > int cxl_passthrough_hdm_rw (p, off, val, write);
> > > int cxl_passthrough_cm_rw (p, off, val, write);
> > >
> > > cxl_passthrough is an opaque handle; vfio-pci sees no cxl-internal
> > > struct pointers. The shadows are snapshotted at create time: the
> > > DVSEC body from PCI config space dword by dword, the CM cap-array
> > > and HDM block from the cxl-core MMIO mapping at cxlds->reg_map.base.
> > > Per-field write semantics follow below:
> > > CXL r4.0 8.1.3 DVSEC:
> > > - LOCK is RWO,
> > > - CONTROL/CONTROL2 are RWL gated on CONFIG_LOCK,
> > > - STATUS/STATUS2 are RW1C,
> > > - RANGE1 is HwInit, RANGE2 is RsvdZ
> > > CXL r4.0 8.2.4.20 HDM:
> > > - GLOBAL_CTRL RW,
> > > - decoder CTRL implements COMMIT/COMMITTED,
> > > - decoder BASE/SIZE RWL gated on COMMITTED or LOCK_ON_COMMIT,
> > > - cap header HwInit).
> > >
> > > vfio-pci becomes a thin transport. The new module
> > > drivers/vfio/pci/cxl/ exposes two VFIO regions.
> > >
> > > VFIO_REGION_SUBTYPE_CXL (HDM region): mmappable view of the
> > > HDM-mapped HPA. The mmap fault handler calls vmf_insert_pfn() from
> > > the physical HPA. pread/pwrite go through the memremap_wb() kva
> > > captured at bind time.
> > >
> > > VFIO_REGION_SUBTYPE_CXL_COMP_REGS (component register shadow):
> > > pread/pwrite only, dword-aligned (-EINVAL on misalignment).
> > > Each dword dispatches by offset to cxl_passthrough_cm_rw() or
> > > cxl_passthrough_hdm_rw(). No shadow state on the vfio side; cxl-
> core
> > > enforces the spec.
> > >
> > > CXL DVSEC config-space accesses use a clipping shim in
> > > vfio_pci_config_rw_single(). A config-space chunk that crosses the
> > > DVSEC body boundary is split: header bytes go through the generic
> > > perm-bits path, body bytes go through cxl_passthrough_dvsec_rw().
> > > The shim replaces v2's approach of repointing ecap_perms[]
> > >
> > > Sparse-mmap is exposed on the component BAR so userspace can mmap
> > > the non-component portions directly; only the CXL component register
> > > sub-range goes through pread/pwrite emulation. The CXL sub-range is
> > > also skipped from vfio_pci-core's request_selected_regions() set
> > > because cxl-core's devm_cxl_probe_mem() already holds a
> > > request_mem_region() on it; the asymmetric skip is matched by an
> > > asymmetric release on disable().
> > >
> > > Scope and out-of-scope
> > > ======================
> > >
> > > In scope (rejected at create time with -EOPNOTSUPP otherwise):
> > >
> > > - Firmware-committed devices (HOST_FIRMWARE_COMMITTED set).
> > > - Single HDM decoder (hdm_count == 1).
> > > - No interleave (IW == 0).
> > >
> > > Out of scope, deferred for follow-on work:
> > >
> > > - Multi-decoder devices and interleave.
> > > - Guest-driven (non-firmware-committed) HDM commit.
> > > - Hotplug, FLR, and sibling-function reset of CXL Type-2 devices.
> > >
> > > Changes since v2
> > > ================
> > >
> > > This is a rewrite, not an incremental update. The structure of the
> > > series changed (20 patches in v2 to 11 in v3) because v3 collapses
> > > v2 patches 9-15 (detection, HDM emulation, media readiness, region
> > > management, HDM region, DVSEC emulation) into one cxl-core helper
> > > file and one vfio-pci consumer.
> > >
> > > Framework replaced by narrow opaque-handle helpers (patches 6, 8)
> > >
> > > v2 carried a generic register-emulation framework split across
> four
> > > state-machine files in cxl-core.
> > > v3 collapses it into one file: drivers/cxl/core/passthrough.c
> > > exposing the four EXPORT_SYMBOL_GPL helpers above behind a struct
> > > cxl_passthrough opaque handle.
> > >
> > > Shadow ownership moved into cxl-core (patches 6, 8)
> > >
> > > vfio-pci no longer keeps any per-field state. It forwards
> > > (offset, value) into cxl-core, and cxl-core enforces the spec
> > > (RWO, RWL, RW1C, HwInit, RsvdZ) with explicit CXL r4.0 section
> > > references in the switch arms.
> > >
> > > DVSEC config-space clipping shim (patch 8)
> > >
> > > v2 repointed ecap_perms[] to redirect CXL DVSEC reads and writes.
> > > v3 keeps ecap_perms[] untouched and clips per-config-access chunks
> > > at the DVSEC body boundary in vfio_pci_config_rw_single(); header
> > bytes
> > > go through the generic perm-bits path, body bytes go through
> > > cxl_passthrough_dvsec_rw(). The shim is local to the per-device
> > > path.
> > >
> > > CONFIG_VFIO_PCI_CXL gates the new module (patch 7)
> > >
> > > v2 had a CONFIG_VFIO_CXL_CORE Kconfig stub; v3 renames it to
> > > CONFIG_VFIO_PCI_CXL to match the vfio-pci naming convention.
> > > The hidden CXL_VFIO_PASSTHROUGH selects the cxl-core helper file
> > > on demand. With both disabled, the cxl-core size is unchanged.
> > >
> > > UAPI rewritten with named fields (patch 5)
> > >
> > > vfio_device_info_cap_cxl in v3 carries:
> > > flags + HOST_FIRMWARE_COMMITTED bit
> > > hdm_region_idx
> > > comp_reg_region_idx
> > > comp_reg_bar
> > > comp_reg_offset
> > > comp_reg_size
> > > The DPA terminology is renamed to HDM region throughout.
> > > CACHE_CAPABLE (HDM-DB indicator) is dropped;
> > > it was informational only in v2 with no caller, and re-adding it
> > > for an active CXL.cache plumbing series later.
> > >
> > > Selftests trimmed (patch 9)
> > >
> > > v2 carried selftests for device detection, capability parsing,
> > > region enumeration, HDM register emulation, HDM mmap with
> > > page-fault insertion, FLR invalidation, and DVSEC register
> > > emulation. v3 keeps a smoke-test set of six focused tests:
> > >
> > > device_is_cxl GET_INFO advertises FLAGS_CXL
> > > and a populated CAP_CXL.
> > > hdm_region_mmap_rw mmap one page, write+read back.
> > > component_bar_sparse_mmap SPARSE_MMAP cap excludes the
> > > CXL component register sub-range.
> > > comp_regs_cm_cap_array_read pread of the CM cap-array
> > > header at CXL_CM_OFFSET succeeds
> > > (CAP_ID == 1).
> > > dvsec_lock_byte_read pread of the DVSEC CONFIG_LOCK
> > > byte through the clipping shim
> > > succeeds.
> > > hdm_decoder_commit_fsm COMMIT / COMMITTED state machine
> > > and LOCK_ON_COMMIT behaviour.
> > >
> > > FLR invalidation, page-fault insertion under load, and full
> > > DVSEC field-by-field write coverage are deferred to a follow-on
> > > selftest series. The current six are the minimal set that
> > > exercises the kernel-side contract end-to-end.
> > >
> > > cxl-core prep patches split (patches 1-4)
> > >
> > > v3 keeps the cxl-side enablers from v2 patches 1-4 but each as
> > > a standalone change so the cxl maintainer can review the helper
> > > API independently of the vfio consumer:
> > >
> > > [1/11] cxl_get_hdm_info()
> > > [2/11] cxl_await_range_active() split from media-ready wait
> > > [3/11] cxl_register_map records BIR + BAR offset
> > > [4/11] component/HDM register defines moved to
> > > uapi/cxl/cxl_regs.h
> > >
> > > Reviewer feedback addressed
> > > ===========================
> > >
> > > Dan
> > > ---
> > >
> > > - VFIO exposes HDM/host-visible region, not raw DPA; docs/UAPI say
> HDM
> > > region, DPA only inside cxl-core where appropriate.
> > > - One vfio-pci device = one HDM region / one decoder, no interleave;
> > > hdm_count != 1 → -EOPNOTSUPP.
> > > - Global HDM on DVSEC Range Base treated as legacy; RANGE1/RANGE2
> > > read-only snapshot, guest writes dropped.
> > > - No guest/kernel lock games; DVSEC LOCK and HDM LOCK_ON_COMMIT RWO,
> > > fixed at create from firmware snapshot.
> > > - Opaque cxl_passthrough handle only; vfio gets HPA via memdev probe
> +
> > > layout via cxl_get_hdm_info(), rw via helpers.
> > > - No multi-region accelerator case in v3; single region enforced,
> > > multi-region deferred.
> > > - cxl_await_range_active stays in cxl-core probe; not exported, vfio
> > does
> > > not call it.
> > > - No guest LOCK→0 reprogram; guest cannot clear LOCK to remap host
> > HPA;
> > > kernel uncommit tied to COMMIT, not LOCK alone.
> > >
> > > Jason / Gregory / Dan
> > > ---------------------
> > >
> > > - memremap(WB) + request_mem_region on HPA; conflicting
> > > direct-map/EFI
> > use
> > > fails probe with -EBUSY.
> > >
> > > Jonathan
> > > --------
> > >
> > > - uapi/cxl/cxl_regs.h for register defines so VMMs need no private
> > > kernel headers.
> > > - __free() locals on cxl-core/passthrough error paths instead of
> > > struct-owned temporaries.
> > > - No "precommitted at probe" assumption; acquire checks COMMITTED in
> > > HDM shadow and refuses if missing.
> > >
> > > Dave
> > > ----
> > >
> > > - memremap(MEMREMAP_WB) for HDM host mapping (not ioremap_cache).
> > > - Renamed cap flag to VFIO_CXL_CAP_HOST_FIRMWARE_COMMITTED for
> > clarity.
> > > - __free() / DEFINE_FREE() cleanup in new passthrough.c create path.
> > >
> > > Patch series
> > > ============
> > >
> > > [1/11] cxl: Add cxl_get_hdm_info() helper for HDM decoder metadata
> > > [2/11] cxl: Split cxl_await_range_active() from media-ready wait
> > > [3/11] cxl: Record BIR and BAR offset in cxl_register_map [4/11]
> cxl:
> > > Move component/HDM register defines to
> > > uapi/cxl/cxl_regs.h
> > > [5/11] vfio: UAPI for CXL Type-2 device passthrough [6/11] cxl:
> > > Add register-virtualization helpers for vfio Type-2
> > > passthrough
> > > [7/11] vfio/pci: Add CONFIG_VFIO_PCI_CXL with bind-time CXL Type-2
> > > acquisition
> > > [8/11] vfio/pci/cxl: Add HDM + COMP_REGS regions and DVSEC clipping
> > > shim
> > > [9/11] selftests/vfio: Add CXL Type-2 device passthrough smoke test
> > > [10/11] docs: vfio-pci: Document CXL Type-2 device passthrough
> > > [11/11]
> > > vfio/pci: Provide opt-out for CXL Type-2 extensions
> > >
> > > Dependencies
> > > ============
> > >
> > > [1] [PATCH v28 0/5] Type2 device basic support
> > > https://lore.kernel.org/linux-cxl/20260618181806.118745-1-alejandro.
> > > lu
> > > [email protected]/
> > >
> > > [2] Previous version of this patch series [PATCH v2 00/20] vfio/pci:
> > > Add CXL Type-2 device passthrough support
> > > https://lore.kernel.org/linux-cxl/20260401143917.108413-1-mhonap@nvi
> > > di
> > > a.com/
> > >
> > > [3] Companion QEMU series
> > > [RFC 0/9] QEMU: CXL Type-2 device passthrough via vfio-pci
> > > https://lore.kernel.org/linux-cxl/20260427181235.3003865-1-mhonap@nv
> > > id
> > > ia.com/
> > >
> > > Manish Honap (11):
> > > cxl: Add cxl_get_hdm_info() helper for HDM decoder metadata
> > > cxl: Split cxl_await_range_active() from media-ready wait
> > > cxl: Record BIR and BAR offset in cxl_register_map
> > > cxl: Move component/HDM register defines to uapi/cxl/cxl_regs.h
> > > vfio: UAPI for CXL Type-2 device passthrough
> > > cxl: Add register-virtualization helpers for vfio Type-2
> passthrough
> > > vfio/pci: Add CONFIG_VFIO_PCI_CXL with bind-time CXL Type-2
> > > acquisition
> > > vfio/pci/cxl: Add HDM + COMP_REGS regions and DVSEC clipping shim
> > > selftests/vfio: Add CXL Type-2 device passthrough smoke test
> > > docs: vfio-pci: Document CXL Type-2 device passthrough
> > > vfio/pci: Provide opt-out for CXL Type-2 extensions
> > >
> > > Documentation/driver-api/index.rst | 1 +
> > > Documentation/driver-api/vfio-pci-cxl.rst | 282 ++++++
> > > drivers/cxl/Kconfig | 7 +
> > > drivers/cxl/core/Makefile | 1 +
> > > drivers/cxl/core/passthrough.c | 590 ++++++++++++
> > > drivers/cxl/core/pci.c | 70 +-
> > > drivers/cxl/core/regs.c | 35 +
> > > drivers/cxl/cxl.h | 52 +-
> > > drivers/vfio/pci/Kconfig | 2 +
> > > drivers/vfio/pci/Makefile | 1 +
> > > drivers/vfio/pci/cxl/Kconfig | 34 +
> > > drivers/vfio/pci/cxl/Makefile | 2 +
> > > drivers/vfio/pci/cxl/vfio_cxl_core.c | 889
> > ++++++++++++++++++
> > > drivers/vfio/pci/cxl/vfio_cxl_priv.h | 71 ++
> > > drivers/vfio/pci/vfio_pci.c | 9 +
> > > drivers/vfio/pci/vfio_pci_config.c | 31 +
> > > drivers/vfio/pci/vfio_pci_core.c | 68 +-
> > > drivers/vfio/pci/vfio_pci_priv.h | 93 ++
> > > drivers/vfio/pci/vfio_pci_rdwr.c | 17 +
> > > include/cxl/cxl.h | 18 +
> > > include/cxl/passthrough.h | 121 +++
> > > include/linux/vfio_pci_core.h | 8 +
> > > include/uapi/cxl/cxl_regs.h | 63 ++
> > > include/uapi/linux/vfio.h | 46 +
> > > tools/testing/selftests/vfio/Makefile | 1 +
> > > .../selftests/vfio/lib/vfio_pci_device.c | 11 +-
> > > .../selftests/vfio/vfio_cxl_type2_test.c | 350 +++++++
> > > 27 files changed, 2821 insertions(+), 52 deletions(-) create mode
> > > 100644 Documentation/driver-api/vfio-pci-cxl.rst
> > > create mode 100644 drivers/cxl/core/passthrough.c create mode
> > > 100644 drivers/vfio/pci/cxl/Kconfig create mode 100644
> > > drivers/vfio/pci/cxl/Makefile create mode 100644
> > > drivers/vfio/pci/cxl/vfio_cxl_core.c
> > > create mode 100644 drivers/vfio/pci/cxl/vfio_cxl_priv.h
> > > create mode 100644 include/cxl/passthrough.h create mode 100644
> > > include/uapi/cxl/cxl_regs.h create mode 100644
> > > tools/testing/selftests/vfio/vfio_cxl_type2_test.c
> > >
> > > base-commit: 90cf2e0d702c8a132ccbe72e7687f33c04c14658
> >
> > Hi Manish,
> > b4 seems to be failing with applying on top of this base-commit.
> >
>
> Sorry about this; The series does not apply against a plain mainline
> tree because it depends on Alejandro's v28 "Type2 device basic support"
>
> I created this patchseries on `origin/for-7.3/cxl-type2-enabling' with
> base commit as 90cf2e0d702c and applied Alejandro's patch-v28. on this
> tree, I see the patches can be applied correctly.
>
> I see Srirangan has based his reset series on Linus's tree. I will
> follow similar guidelines for v4.
>
