Hi, This series relax the vIOMMU check and allows live migration with vIOMMU without VFs using device dirty tracking. It's rewritten based on first 4 patches of [1] from Joao.
Currently what block us is the lack of dirty bitmap query with iommufd before unmap. By adding that query and handle some corner case we can relax the check. Based on vfio-next branch: patch1-2: add dirty bitmap query with iommufd patch3: a ranaming cleanup patch4-5: unmap_bitmap optimization patch6-7: fixes to avoid losing dirty pages patch8: add a blocker if VM memory is really quite large for unmap_bitmap patch9: relax vIOMMU check We used [2] to test, it contains dom_switch series + this series + nesting series. I included nesting series just because I'd like to confirm the two patches optimizing out dirty tracking for readonly pages work. We tested VM live migration (running QAT workload in VM) with QAT device passthrough, below matrix configs with guest config 'iommu=pt' and 'iommu=nopt': 1.Scalable mode vIOMMU + IOMMUFD cdev mode 2.Scalable mode vIOMMU + legacy VFIO mode 3.legacy mode vIOMMU + IOMMUFD cdev mode 4.legacy mode vIOMMU + legacy VFIO mode The QAT workload is a user level app that utilizes VFIO to control QAT device. Thanks Zhenzhong [1] https://github.com/jpemartins/qemu/commits/vfio-migration-viommu/ [2] https://github.com/yiliu1765/qemu/tree/liuyi/zhenzhong/iommufd_nesting.v8.DS_LM.wip Changelog: v5: - drop the patch checking iommu_dirty_tracking (Avihai, Joao) - pass iotlb info to unmap_bitmap when switch out of system AS v4: - bypass memory size check for device dirty tracking as it's unrelated (Avihai) - split vfio_device_dirty_pages_disabled() helper out as a separate patch - add a patch to fix minor error on checking vbasedev->iommu_dirty_tracking v3: - return bitmap query failure to fail migration (Avihai) - refine patch7, set IOMMUFD backend 'dirty_pgsizes' and 'max_dirty_bitmap_size' (Cedric) - refine patch7, calculate memory limit instead of hardcode 8TB (Liuyi) - refine commit log (Cedric, Liuyi) v2: - add backend_flag parameter to pass DIRTY_BITMAP_NO_CLEAR (Joao, Cedric) - add a cleanup patch to rename vfio_dma_unmap_bitmap (Cedric) - add blocker if unmap_bitmap limit check fail (Liuyi) Joao Martins (1): vfio: Add a backend_flag parameter to vfio_contianer_query_dirty_bitmap() Zhenzhong Duan (8): vfio/iommufd: Add framework code to support getting dirty bitmap before unmap vfio/iommufd: Query dirty bitmap before DMA unmap vfio/container-legacy: rename vfio_dma_unmap_bitmap() to vfio_legacy_dma_unmap_get_dirty_bitmap() vfio/iommufd: Add IOMMU_HWPT_GET_DIRTY_BITMAP_NO_CLEAR flag support intel_iommu: Fix unmap_bitmap failure with legacy VFIO backend vfio/listener: Construct iotlb entry when unmap memory address space vfio/migration: Add migration blocker if VM memory is too large to cause unmap_bitmap failure vfio/migration: Allow live migration with vIOMMU without VFs using device dirty tracking include/hw/vfio/vfio-container.h | 8 +++-- include/hw/vfio/vfio-device.h | 10 ++++++ include/system/iommufd.h | 2 +- backends/iommufd.c | 5 +-- hw/i386/intel_iommu.c | 42 +++++++++++++++++++++++++ hw/vfio-user/container.c | 5 +-- hw/vfio/container-legacy.c | 15 +++++---- hw/vfio/container.c | 20 ++++++------ hw/vfio/device.c | 6 ++++ hw/vfio/iommufd.c | 53 +++++++++++++++++++++++++++++--- hw/vfio/listener.c | 21 ++++++++++--- hw/vfio/migration.c | 40 ++++++++++++++++++++++-- backends/trace-events | 2 +- hw/vfio/trace-events | 2 +- 14 files changed, 194 insertions(+), 37 deletions(-) -- 2.47.1
