Hi Mostafa, On Wed, 21 Oct 2026 10:55:36 +0000 Mostafa Saleh <[email protected]> wrote:
> On Fri, Sep 25, 2026 at 12:07:33PM -0700, Jacob Pan wrote: > > Hi, > > > > This RFC supports guest VM device assignment from a Linux host > > using the Microsoft Hypervisor (MSHV). The goal is that a common > > set of flows and infrastructure can be applied for Xen and other > > type 1 hypervisors. > > > > Background: > > > > The IOMMUFD/VFIO stack used for device assignment has largely > > evolved around the KVM/QEMU model, at least as the first enabling > > target. In that model, the Linux host owns the host-side physical > > IOMMU programming as well as the physical PCI devices. > > > > Architecturally, MSHV and Xen differ from this KVM/QEMU ownership > > model. At a high level: > > > > +------------------+ > > | Guest VM | > > | virtual IOMMU | > > +--------+---------+ > > | vIOMMU ops > > v > > +------------------+ external attach +------------------+ > > | MSHV |<-----------------------| Linux host | > > +--------+---------+ +--------+---------+ > > | programs | owns > > v v > > +------------------+ +------------------+ > > | HW IOMMU |<---------------------->| PCI dev | > > +------------------+ DMA +------------------+ > > > > For MSHV, the specific ownership model is as follows: > > 1. The Linux host OS owns the physical device and exposes it > > through VFIO/IOMMUFD, but it does not directly own any of the IOMMU > > hardware. 2. The device node (/dev/mshv) owns the VM partition file > > used by userspace. 3. MSHV owns most of the physical IOMMU > > programming, including stage-2 I/O page tables. > > 4. The Linux MSHV root pvIOMMU driver bridges IOMMUFD/VFIO UAPIs > > to hardware IOMMU programming through MSHV hypercalls. > > 5. Guest virtual IOMMU emulation is done in MSHV rather than the > > userspace VMM. > > 6. Stage-2 IOPT may be shared between CPU and IOMMU. It is > > managed by MSHV and abstracted from the Linux host, including TLB > > synchronization. > > > > These structural differences from the KVM/QEMU model create several > > needs for IOMMUFD support: > > 1. Move a device to a guest whose I/O page table is externally > > managed. This patchset refers to that operation as external attach. > > Unlike conventional IOMMU vendor drivers, where translation is > > managed within the driver through paging domains, external attach > > means that translation is managed outside the driver, such as by > > the hypervisor. 2. Identify the guest-visible device across VMM, > > host pvIOMMU driver, and hypervisor namespaces. > > 3. Identify the guest partition in the host pvIOMMU driver for > > device attach hypercalls. > > 4. Tie the device-attachment lifetime to the hypervisor partition. > > 5. Coordinate guest-memory pinning and mapping for DMA between > > MSHV and the external IOMMU domain attachment. > > > > External attach has a few caveats compared with KVM/QEMU, where > > nesting is typically handled by the host IOMMU driver in one > > transaction after the userspace VMM uses IOMMUFD to form nested > > domains: > > > > 1. MSHV creates a temporal split between host and guest > > device-domain attach. The host first attaches S2 through a > > hypercall; later, if virtual IOMMU is present, the guest > > allocates S1 and establishes nesting with MSHV directly > > through a pvIOMMU or emulated vIOMMU. > > 2. Guest-memory pinning before mapping is covered by the > > MSHV/userspace VMM contract, not by the VFIO/IOMMUFD pin-and-map > > DMA flow. 3. Device capabilities such as ATS and PASID need a > > contract among MSHV, the userspace VMM, and the host pvIOMMU > > driver. At the time of host external attach, only S2 is attached, > > so the guest's later ATS/PASID usage is not yet known. For example, > > whether to expose and enable PASID in the device could depend on > > whether the userspace VMM creates a guest virtual IOMMU. > > > > This RFC proposes an IOMMUFD external attach model for those cases. > > The normal Linux-managed paging or nested domain is not the object > > that ultimately owns the translation context. Instead, userspace > > identifies a VM/domain object, binds a physical device to a > > VM-visible device identity, and asks the host pvIOMMU driver to > > attach the device to an externally managed address space. > > > > The series makes that relationship explicit in IOMMUFD. The model > > is anchored around a hypervisor vIOMMU object that represents the > > per-VM partition abstraction. It introduces: > > > > 1. IOMMU_DOMAIN_EXTERNAL for externally managed translation > > domains. 2. IOMMU_VIOMMU_TYPE_HYPERVISOR for the file-backed VM > > partition. 3. IOMMU_HWPT_EXTERNAL for user-visible external type > > IOMMU domains. > > > > It reuses the IOMMUFD vDEVICE object to carry the guest-VM-scoped > > logical device ID visible to the hypervisor. > > > > This is very similar to pKVM on arm64 also, where the IOMMU is managed > by the hypervisor, and for device assignment, the host doesn't control > it for the guest, but the hypervisor (in EL2), and there will be a > channel between the guest and pKVM directly after the guest runs. > > The host only sets the virtual to physical topology (how many vIOMMUs, > vSID -> SID mappings) and describes that for the hypervisor through > hypercalls. > In Android, we have an out of tree uAPI through KVM/VFIO bridge > IOCTLs that proxies those hypercalls. > > In that case, the host just creates a virtual IOMMU and the guest sees > a paravirtual one which runs with single stage collapsed translation, > although I don't see a difference on the host side if nested > translation was used beyond an argument change to the hypercalls. > By "argument change", do you mean that the host would pass the guest S1 context in the device-attach hypercall for nested translation? IMHO, I don't see any change needed for the host-side attach between hw nested and collapsed single stage, if guest and host attach are split. I think it is up to pKVM or MSHV to handle guest virtual iommu mapping for both cases. > My goal is to converge to IOMMUFD and I believe this series is the way > forward, By relying on the vDEVICE virtual toplogy uAPI and the > external attach introduced here. > > I will try to prototype something based on that to prove that it > works. However that work has a dependency on the host SMMUv3 support > [1] which needs to land first. > > [1] https://lore.kernel.org/all/[email protected]/ > Same here, our pviommu root driver must land first. https://lore.kernel.org/linux-iommu/[email protected]/ My next step is to rebase on Aneesh's v7 such that we have a converged iommufd viommu layer, i will remove mshv specific helper registration in this series. https://lore.kernel.org/linux-iommu/[email protected]/ > Thanks, > Mostafa > > > > Roughly, the VMM flow is as follows: > > 1. iommufd = open("/dev/iommu") > > 2. vfio_fd = open("/dev/vfio/devices/vfioN") > > 3. dev_id = bind_vfio_device(vfio_fd, iommufd) > > 4. create_mshv_partition(&mshv_fd, &vm_fd) > > // Creates the guest vIOMMU and configures guest-memory > > pinning policy. 5. alloc_hypervisor_viommu(iommufd, &viommu_id, > > dev_id, vm_fd) // Uses the hypervisor vIOMMU type. > > 6. alloc_vdevice(iommufd, viommu_id, dev_id, vm_device_id, > > &vdevice_id) 7. alloc_external_hwpt(iommufd, dev_id, viommu_id, > > &external_hwpt_id) 8. attach_external_hwpt(vfio_fd, > > external_hwpt_id) > > > > Step 4 may be completed before steps 1-3, but it must precede step > > 5 because the partition file is required to allocate the hypervisor > > vIOMMU. > > > > The attach_external_hwpt() operation enters the MSHV root pvIOMMU > > driver's external-HWPT device_attach path: > > -> resolve partition ID from hypervisor vIOMMU > > -> resolve logical device ID from vDEVICE > > -> invoke the partition prepare_attach callback to force pinning > > if policy requires > > -> MSHV attach-device hypercall > > > > Object lifetime: > > - The hypervisor vIOMMU takes a reference on the MSHV partition > > file and holds it until the vIOMMU is destroyed. Closing the > > userspace copy of the partition fd therefore does not destroy the > > partition while the vIOMMU still exists. > > - Each vDEVICE and external HWPT holds a reference on its parent > > vIOMMU. The vIOMMU cannot be destroyed until these child objects > > are removed. > > - An attached device holds a reference on the external HWPT, so > > userspace cannot destroy the HWPT while the device is attached. > > - Closing the IOMMUFD fd destroys the object graph from the > > leaves upward. Once the child objects are gone, destroying the > > vIOMMU drops its partition file reference and allows the MSHV > > partition to be released. > > > > Thus, the external-attach lifetime is anchored directly by IOMMUFD > > object references and the partition file. It does not require a > > separate VFIO-hypervisor bridge, such as kvm-vfio: the MSHV root > > IOMMU driver's detach operation directly tears down the > > hypervisor-visible external-attach state. > > > > The root pvIOMMU driver changes in this RFC address the following > > concerns received from the older version[1]: > > 1. The target partition comes from the fd-backed hypervisor > > vIOMMU created by userspace, not from the thread issuing the attach. > > 2. External attach is represented explicitly, avoiding > > paging-domain hacks. 3. Guest memory is pinned exactly once. In the > > older model, the same memory could be pinned once by /dev/mshv and > > again by the VFIO DMA map ioctl. > > > > This RFC builds on the MSHV PCI passthrough/root pvIOMMU base work > > [2]. The mergeable MSHV driver changes should be added on top of a > > future version of that base root pvIOMMU driver. > > > > Feedback would be especially useful on: > > > > 1. Whether IOMMU_DOMAIN_EXTERNAL is the right IOMMU core > > representation for an externally managed translation (Xen, perhaps > > TSM [4]). 2. Whether IOMMU_VIOMMU_TYPE_HYPERVISOR/VDEVICE can be > > common to MSHV, Xen, and similar type-1 hypervisor flows, and > > whether it can cover the semantics in CoCo TDISP that would > > otherwise require tsm_bind() [3]. 3. Whether a file-backed > > VM/domain reference is the right way to anchor the external attach > > lifetime. 4. Whether vDEVICE virt_id is the right generic source > > for the hypervisor-assigned logical device ID. > > > > At LPC 2026, I will be presenting this work and have open > > discussions. https://lpc.events/event/20/contributions/2520/ > > > > Future work: > > 1. ATS/PASID policy integration with the root pvIOMMU driver. > > ATS/PASID policy is intentionally left as a separate contract > > issue. I do not think the root pvIOMMU driver should blindly enable > > or disable PASID/ATS on a device without knowing the guest > > configuration or receiving acknowledgment from MSHV, which owns the > > device-TLB flush for S2 changes. > > > > 2. Live update support > > The current LUO proposal preserves DMA mappings through > > IOMMUFD-owned state. External attach requires a different > > preservation model because the S2 IOPT is managed by MSHV rather > > than IOMMUFD. > > > > 3. vDEVICE lifetime during external attachment > > An attached external HWPT does not currently retain the vDEVICE > > used by MSHV to resolve the guest logical device ID. We can either > > let the core retain the vDEVICE identity for external attach, or > > provide an opt-in mechanism to the vendor IOMMU drivers as needed. > > I am not sure if all type 1 hypervisors require a vDEVICE identity. > > > > 4. Revokable dma-buf support for P2PDMA > > Now that MSHV, instead of IOMMUFD, manages the S2 IOPT, we need to > > reconsider how revokable dma-bufs are handled. Perhaps MSHV can be > > the importer of revokable dma-bufs for P2PDMA and unmap them when > > the dma-buf is revoked. > > > > [1] > > https://lore.kernel.org/linux-iommu/[email protected]/ > > [2] https://lore.kernel.org/all/[email protected]/ > > [3] > > https://lore.kernel.org/all/[email protected]/T/#t > > [4] https://lore.kernel.org/linux-iommu/[email protected]/ > > > > Patch groups: > > > > IOMMUFD core and selftest: > > > > 1. iommu: Introduce external attach domain type > > 2. iommufd: Introduce hypervisor vIOMMU type > > 3. iommufd: Add external HWPT support > > 4. iommufd/selftest: Add hypervisor external attach backend > > > > MSHV: > > > > 5. mshv: Add partition file identity helper > > 6. mshv: Add prepare callback for external device attach > > > > MSHV root pIOMMU driver: > > > > 7. iommu/hyperv: Split root IOMMU declarations > > 8. iommu/hyperv: Add fd-backed vIOMMU support > > 9. iommu/hyperv: Add IOMMUFD external domains > > > > > > Thanks, > > Jacob > > > > Jacob Pan (9): > > iommu: Introduce external attach domain type > > iommufd: Introduce hypervisor vIOMMU type > > iommufd: Add external HWPT support > > iommufd/selftest: Add hypervisor external attach backend > > mshv: Add partition file identity helper > > mshv: Add prepare callback for external device attach > > iommu/hyperv: Split root IOMMU declarations > > iommu/hyperv: Add fd-backed vIOMMU support > > iommu/hyperv: Add IOMMUFD external domains > > > > drivers/hv/hv_common.c | 91 ++++++++++++++++ > > drivers/hv/mshv_regions.c | 6 ++ > > drivers/hv/mshv_root.h | 1 + > > drivers/hv/mshv_root_main.c | 71 ++++++++++++- > > drivers/iommu/amd/iommufd.c | 10 +- > > drivers/iommu/hyperv/Makefile | 2 +- > > drivers/iommu/hyperv/hv-iommu-iommufd.c | 135 > > ++++++++++++++++++++++++ drivers/iommu/hyperv/hv-iommu-root.c | > > 102 ++++++++++++++++-- drivers/iommu/hyperv/hv-iommu.h | > > 40 +++++++ drivers/iommu/iommu.c | 2 + > > drivers/iommu/iommufd/device.c | 9 +- > > drivers/iommu/iommufd/hw_pagetable.c | 88 +++++++++++++-- > > drivers/iommu/iommufd/iommufd_private.h | 7 ++ > > drivers/iommu/iommufd/main.c | 4 + > > drivers/iommu/iommufd/selftest.c | 120 +++++++++++++++++++-- > > drivers/iommu/iommufd/viommu.c | 37 ++++--- > > include/asm-generic/mshyperv.h | 42 ++++++++ > > include/linux/iommu.h | 17 ++- > > include/linux/iommufd.h | 11 ++ > > include/uapi/linux/iommufd.h | 42 +++++++- > > 20 files changed, 790 insertions(+), 47 deletions(-) > > create mode 100644 drivers/iommu/hyperv/hv-iommu-iommufd.c > > create mode 100644 drivers/iommu/hyperv/hv-iommu.h > > > > -- > > 2.43.0 > > > >
