Add an mlx5 driver to VFIO self test. This is largely a remix of the
existing VFIO mlx5 driver in rdma-core. It uses an RDMA loopback QP
to issue RDMA WRITE operations which effectively perform memory
copies using DMA. Since mlx5 has a stable programming ABI this
should work on devices from CX5 to current HW. The device FW must
support the QP loopback configuration.
Also support send_msi by arming completion events of the RDMA WRITE
to trigger MSI delivery.
mlx5 device startup is very complex and most of this code is just
booting the device, with a smaller amount for operating the QP.
This entire series was coded by Claude Code in about 4 days. It
used about 4.5M output tokens, 30 individual sessions and 5600 lines
of AI-generated .md files. I spent an annoying amount of time
de-slopping and cleaning its work product to make it presentable.
However, previous VFIO drivers have taken on the order of 1-2
months to write, so getting one in a week is pretty remarkable.
For those interested, the flow I used was broadly a prompt sequence
sort of like:
- Hey Claude, go look at the falcon series, VFIO self test, the
mlx5 driver, rdma-core and some PDF documentation and make a
plan to put mlx5 under the selftest.
- Write an rdma-core application using the built-in VFIO provider
that can do the required memcpy operations that vfio selftests
wants.
(This resulted in a 1k loc C file that compiled and ran the
first time but had a few bugs related to device programming
that the AI resolved.)
- Replace the rdma-core components with open-coded versions to
create a fully stand-alone program that does the DMA memcpy.
- Review and audit the thing.
[Pause and de-slop it]
- Make it work on a PF too (this is surprisingly hard!).
[Move to a kernel tree and copy all the .md files and .c program
it made]
- Hey Claude, look at all this stuff and make a broad plan to
actually build a VFIO self test.
- Here is my 1 sentence advice on what each patch should look
like, make a detailed plan to make a patch for every one.
[Pause and polish the patch plans]
- Execute plan X then commit it [pause and de-slop each patch,
repeat].
[Review and final polish]
v2:
- Rebase on v7.1-rc3, drop falcon patches
- dev_dbg checks the format string even in non debug builds
- Sort includes
- Adjust comments/commit messageas
- Use linux/pci_ids.h
- Compute the driver.max_memcpy_size the same as mlx5 kernel driver, it
should be GBs now on most devices
- Put region_size into all drivers instead of allowing 0
v1: https://patch.msgid.link/r/[email protected]
Jason Gunthorpe (11):
net/mlx5: Add IFC structures for CQE and WQE
net/mlx5: Move HW constant groups from device.h/cq.h to mlx5_ifc.h
net/mlx5: Extract MLX5_SET/GET macros into mlx5_ifc_macros.h
net/mlx5: Add ONCE and MMIO accessor variants to mlx5_ifc_macros.h
selftests: Add additional kernel functions to tools/include/
selftests: Fix arm64 IO barriers to match kernel
vfio: selftests: Allow drivers to specify required region size
vfio: selftests: Add dev_dbg
vfio: selftests: Add mlx5 driver - HW init and command interface
vfio: selftests: Add mlx5 driver - data path and memcpy ops
vfio: selftests: mlx5 driver - add send_msi support
include/linux/mlx5/cq.h | 10 -
include/linux/mlx5/device.h | 231 +-
include/linux/mlx5/mlx5_ifc.h | 178 ++
include/linux/mlx5/mlx5_ifc_macros.h | 185 ++
tools/arch/arm64/include/asm/barrier.h | 18 +
tools/arch/x86/include/asm/barrier.h | 5 +
tools/include/asm-generic/io.h | 28 +
tools/include/asm/barrier.h | 8 +
tools/include/linux/stddef.h | 10 +
.../selftests/vfio/lib/drivers/dsa/dsa.c | 1 +
.../selftests/vfio/lib/drivers/ioat/ioat.c | 1 +
.../selftests/vfio/lib/drivers/mlx5/mlx5.c | 1928 +++++++++++++++++
.../selftests/vfio/lib/drivers/mlx5/mlx5_hw.h | 114 +
.../vfio/lib/drivers/mlx5/mlx5_ifc.h | 1 +
.../vfio/lib/drivers/mlx5/mlx5_ifc_fpga.h | 1 +
.../vfio/lib/drivers/mlx5/mlx5_ifc_macros.h | 1 +
.../lib/include/libvfio/vfio_pci_device.h | 10 +
.../lib/include/libvfio/vfio_pci_driver.h | 6 +
tools/testing/selftests/vfio/lib/libvfio.mk | 2 +
.../selftests/vfio/lib/vfio_pci_driver.c | 3 +
.../selftests/vfio/vfio_pci_driver_test.c | 7 +-
21 files changed, 2508 insertions(+), 240 deletions(-)
create mode 100644 include/linux/mlx5/mlx5_ifc_macros.h
create mode 100644 tools/include/linux/stddef.h
create mode 100644 tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5.c
create mode 100644 tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_hw.h
create mode 120000 tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc.h
create mode 120000
tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc_fpga.h
create mode 120000
tools/testing/selftests/vfio/lib/drivers/mlx5/mlx5_ifc_macros.h
base-commit: a9fe1263579aef5796ff68538b24e228ad0665df
--
2.43.0
