On 9/15/2026 4:32 PM, David Marchand wrote:
On Thu, 10 Sept 2026 at 14:53, Anatoly Burakov
<[email protected]> wrote:
This patchset introduces a major refactor of the VFIO subsystem in DPDK to
support character device (cdev) interface introduced in Linux kernel, as well as
make the API more streamlined and useful. The goal is to simplify device
management, improve compatibility, make the code readable, and clarify API.
The following sections outline the key issues addressed by this patchset and the
corresponding changes introduced.
1. Only group mode is supported
===============================
Since kernel version 4.14.327 (LTS), VFIO supports the new character device
(cdev)-based way of working with VFIO devices (otherwise known as IOMMUFD). This
is a device-centric mode and does away with all the complexity regarding groups
and IOMMU types, delegating it all to the kernel, and exposes a much simpler
interface to userspace. The old group-based implementation will still be around,
and will need to be kept in DPDK for compatibility reasons.
To enable this, VFIO is heavily refactored, so that the code can support both
modes while relying on (mostly) common infrastructure.
Additionally, new `vfio_get_mode` API is added for those cases that need
some introspection into VFIO's internals, with two modes: group (old-style),
and cdev (the new mode).
Historically, no-IOMMU mode was technically a variant of group mode, the
distinction is largely irrelevant to the user, as all usages of noiommu checks
in our codebase are for deciding whether to use IOVA or PA, not anything to do
with managing groups. However, now that upcoming kernel versions will support
no-IOMMU for both group, cdev compatibility, and full cdev paths, a new
`vfio_get_iommu_mode` is also added, with two modes: safe (full IOMMU backing),
and unsafe (no-IOMMU mode). The naming is chosen explicitly to emphasize that
using no-IOMMU mode is not ideal.
2. Custom container assignment API does not map to cdev mode
============================================================
The existing `rte_vfio_device_setup/release` model is fundamentally incompatible
with cdev mode, because for custom container cases, the expected flow is that
the user binds the IOMMU group (and thus, implicitly, the device itself) to a
specific container using `rte_vfio_container_group_bind`, whereas this step is
not needed for cdev as the device fd is assigned to the container straight away.
Therefore, what we do instead is introduce a new API for container device
assignment which, semantically, will assign a device to specified container, so
that when it is mapped using `rte_pci_map_device`, the appropriate container is
selected. Under the hood though, we essentially transition to getting device fd
straight away at assign stage, so that by the time the PCI bus attempts to map
the device, it is already mapped and we just return an fd. There is no
"unassign" API because `release_device` already performs that function.
Because the API is now unified around device assignment, the old group-specific
API's can be removed and, where appropriate, reimplemented using new API. There
were other users of VFIO which relied on group API but only for convenience
purposes; no actual VFIO functionality depended on those API's.
List of removed API's:
* `rte_vfio_get_group_fd`
* `rte_vfio_clear_group`
* `rte_vfio_container_group_bind` (replaced by container assign API)
* `rte_vfio_container_group_unbind`
* `rte_vfio_noiommu_is_enabled` (replaced by new mode API)
3. The API responsibilities aren't clear and bleed into each other
==================================================================
Some API's do multiple things at once. In particular:
* `rte_vfio_get_device_info` will setup the device
* `rte_vfio_setup_device` will get device info
These API's have been adjusted to do one thing only.
4. The API does not need to be public
=====================================
The initial idea for exposing VFIO API was to enable userspace applications to
directly map memory for DMA, but it turns out that in practice only drivers use
this API. Therefore, the entire VFIO API is made internal, driver-only, and is
renamed from `rte_vfio` to `dev_vfio`.
Hi David!
Thanks for the cleanup!
I am still in the process of reviewing.
Some first comments.
- About patch 1, I am not sure I understand your intention.
Without applying it, a conflict appears later in the series.
It's because the first patch is queued for next-net but our CI doesn't
seem to be capable of rebasing on top of that tree yet, so as a
workaround (to get compile checks etc.) I added it to the first patch
series. The cover letter states that.
- Do you know if some Linux capability is needed for using the cdev mode?
Did you test this change in (unpriviledged) containers for example?
I did not test such a scenario as I do not have a setup for this. I know
we've done internal testing but not for unprivileged/non-root scenarios.
- It was a ugly/gray area so far, but should we stop exposing an API
that do nothing on Windows and FreeBSD?
Especially now that we make it internal.
I am not 100% sure why this is necessary to do - my best guess is that
some of the drivers call into VFIO API which means it needs to at least
link correctly. I would gladly remove all such references, but I would
suggest this to be future work and out of scope for this patchset. Now
that we're removed the ABI surface we can do whatever!
Drivers calling the VFIO API may be broken/falsely announcing support
on other OSes.
I prefer a clear broken build rather than some runtime failure on
those OSes the day someone starts testing.
I agree.
--vfio-intr / --vfio-vf-token EAL options are already Linux only.
So EAL common code looks already ready.
There may be one complication on the PCI bus side, with its calls to
vfio_dma_map/unmap but it seems doable (implement per OS
dma_map/dma_unmap internal symbols ?).
Something like:
https://github.com/david-marchand/dpdk/commit/cc77440883b0d38b842d6c520292b540eabb78c4
https://github.com/david-marchand/dpdk/commit/8f426089f949e0694e5201193be88de673c4283d
That can be done, yes.
- Probably for later, but I see one more constant added in config/meson.build.
All VFIO objects seems to be local (or exchanged over MP messages).
How much effort would it take to remove those constants in config/meson.build?
I am thinking about RTE_MAX_VFIO_CONTAINERS, RTE_MAX_VFIO_GROUPS,
RTE_MAX_VFIO_DEVICES, EAL_VFIO_MAX_USER_MEM_MAPS.
Compile time constants is a balancing act between flexibility and and
ease of implementation. *Technically*, because all of the configuration
etc. is process local, I believe we can just do per process
malloc/realloc and expand these lists as they grow, thereby doing away
with any limitations. However, obviosly, it would require some plumbing
work.
That said, now that VFIO code is much cleaner and more structured, I do
not think it is difficult to do that.
- rte_eal_check_module() only user is VFIO.
No opensource project use it.
It could be removed in the future.
Agree.
--
Thanks,
Anatoly
