mlx5 PMD is a bifurcated driver, which means that instead of communicating with HW directly using UIO or VFIO, the driver uses existing kernel and userspace APIs for that purpose. mlx5 PMD requires that `mlx5_core`, `mlx5_ib` and `ib_uverbs` kernel modules are loaded, and that NIC is bound to `mlx5_core` driver.
Kernel bypass is achieved through rdma-core library which exposes required functionalities e.g., mapping memory to the device, FW access for setting up Rx and Tx queues (through mlx5dv_devx_* interfaces) and flow rule offloading. One benefit of such architecture is that it allows libraries and applications to build additional features on top of DPDK, utilizing rdma-core directly. For example it is possible to implement additional flow rule offloads. This can be achieved by creating an IBV (IB Verbs) context and PD (protection domain) through rdma-core and sharing it with mlx5 PMD through `cmd_fd` and `pd_handle` device arguments provided during probing [1]. mlx5 PMD will then create all necessary resources for DPDK using imported context and PD. What is missing for that approach to work, is that external libraries and applications are not aware of HW identifiers of Rx and Tx queues (returned from mlx5dv_devx_* APIs) created by DPDK through imported context and PD. Knowledge of these identifiers is crucial for correct flow rule offload. This patchset proposes the introduction of the following private PMD APIs to address the above use case: 1. mlx5 PMD driver events 2. mlx5 PMD steering toggle mlx5 PMD driver events, exposed through `rte_pmd_mlx5_driver_event_cb_register/unregister()` functions, allow libraries and applications to register a custom callback. This callback will be called whenever mlx5 PMD creates or destroys an Rx/Tx queue. All necessary HW identifiers along with corresponding DPDK port and queue indexes will be passed as parameters. mlx5 PMD steering toggle, exposed through `rte_pmd_mlx5_driver_disable/enable_steering()` functions, allow libraries and applications to disable/enable flow rule offloading provided by mlx5 PMD. This is added to support use cases where library/application needs full control of flow rule offloading. Library/application will use data provided through driver events API to configure flow rules by itself. When flow rule offloading is disabled, DPDK flow API will return ENOSYS for mlx5 PMD. Patches: - Patch 1 - Add mlx5 PMD driver events private API - Patches 2-3 - Minor clean ups in mlx5 PMD which make introduction of steering toggle private API easier. - Patch 4 - Add mlx5 PMD steering toggle private API [1]: https://doc.dpdk.org/guides/platform/mlx5.html#device-arguments Dariusz Sosnowski (4): net/mlx5: add driver event callbacks net/mlx5: move eCPRI release function to flex net/mlx5: rework Rx queue mark flag functions net/mlx5: add steering toggle API drivers/net/mlx5/meson.build | 1 + drivers/net/mlx5/mlx5.c | 18 -- drivers/net/mlx5/mlx5.h | 1 + drivers/net/mlx5/mlx5_devx.c | 17 ++ drivers/net/mlx5/mlx5_driver_event.c | 300 +++++++++++++++++++++++++++ drivers/net/mlx5/mlx5_driver_event.h | 18 ++ drivers/net/mlx5/mlx5_flow.c | 199 +++++++++++++++++- drivers/net/mlx5/mlx5_flow.h | 7 +- drivers/net/mlx5/mlx5_flow_flex.c | 18 ++ drivers/net/mlx5/mlx5_trigger.c | 30 +++ drivers/net/mlx5/rte_pmd_mlx5.h | 192 +++++++++++++++++ 11 files changed, 772 insertions(+), 29 deletions(-) create mode 100644 drivers/net/mlx5/mlx5_driver_event.c create mode 100644 drivers/net/mlx5/mlx5_driver_event.h -- 2.21.0
