On Sun, 27 Apr 2025 02:27:34 +0000 Kyo Liu <kyo....@nebula-matrix.com> wrote:
> I propose this patch for DPDK to enable coexistence between > DPDK and kernel drivers for regular NICs.This solution requires > adding a new pci_ops in rte_pci_driver, through which DPDK will > retrieve the required IOVA mode from the vendor driver. > This mechanism is necessary to handle different IOMMU > configurations and operating modes. Below is a detailed > analysis of various scenarios: > > 1. When IOMMU is enabled: > 1.1 With PT (Pass-Through) enabled: > In this case, the domain type is IOMMU_DOMAIN_IDENTITY, > which prevents vendor drivers from setting IOVA->PA mapping tables. > Therefore, DPDK must use PA mode. To achieve this: > The vendor kernel driver will register a character device (cdev) to > communicate with DPDK. This cdev handles device operations > (open, mmap, etc.) and ultimately > programs the hardware registers. > > 1.2 With PT disabled: > Here, the vendor driver doesn't enforce specific IOVA mode requirements. > Our implementation will: > Integrate a mediated device (mdev) in the vendor driver. > This mdev interacts with DPDK and manages IOVA->PA mapping configurations. > > 2. When IOMMU is disabled: > The vendor driver mandates PA mode (consistent with DPDK's PA mode > requirement in this scenario). > A character device (cdev) will similarly be registered for DPDK > communication. > > Summary: > The solution leverages multiple technologies: > mdev for IOVA management when IOMMU is partially enabled. > VFIO for device passthrough operations. > cdev for register programming coordination. > A new pci_ops interface in DPDK to dynamically determine IOVA modes. > This architecture enables clean coexistence by establishing standardized > communication channels between DPDK and vendor drivers across different > IOMMU configurations. > > Motivation for the Patch: > This patch is introduced to prepare for the upcoming open-source > contribution of our NebulaMatrix SNIC driver to DPDK. We aim to > ensure that our SNIC can seamlessly coexist with kernel drivers > using this mechanism. By adopting the proposed > architecture—leveraging dynamic IOVA mode negotiation via pci_ops, > mediated devices (mdev), and character device (cdev) > interactions—we enable our SNIC to operate in hybrid environments > here both DPDK and kernel drivers may manage the same hardware. > This design aligns with DPDK’s scalability goals and ensures > compatibility across diverse IOMMU configurations, which is critical > for real-world deployment scenarios. > > Signed-off-by: Kyo Liu <kyo....@nebula-matrix.com> Looks like this might also end up being ABI change so may have to wait for 25.11
