NVIDIA NICs and DPUs have an embedded switch (E-Switch) to which all associated functions (PFs, VFs, SFs) and physical ports are connected through virtual ports (vports). When E-Switch is in switchdev mode, it is exposed as a single IB device with multiple ports, owned by the PF (E-Switch manager). Uplink and each of the representor netdevs are bound to one of the IB device ports. mlx5 PMD probes the PF device and spawns ethdev ports for the uplink and for representors selected through representor devargs. Type and location of each port in the E-Switch topology are derived from phys_port_name of the related netdev.
Future FW and Linux kernel releases will introduce nested E-Switch. Nested E-Switch groups a set of VFs or SFs into a new E-Switch instance. Its architecture is similar to the regular E-Switch: there is a single IB device with multiple ports, owned by one of the grouped functions, which serves as the E-Switch manager (transfer proxy) of the nested E-Switch. Remaining grouped functions are bound to other IB device ports and serve as representors. However, nested E-Switch differs from the regular one in ways which break assumptions made by mlx5 PMD: - Ports of nested E-Switch are netdevs of real VFs/SFs, not representor netdevs created on the PF. Each of these functions has its own PCI or auxiliary device, but the IB device can be opened only through the device of E-Switch manager. mlx5 PMD assumes a one-to-one relation between rte_device and mlx5 common device, and that all ethdev ports in a single E-Switch are spawned on the same rte_device. - These netdevs do not have phys_port_name. Instead, they are related to devlink ports with VIRTUAL flavour. Goal of this patchset is to allow mlx5 PMD to probe ports of nested E-Switch. Each of the grouped VFs/SFs is probed using its own PCI or auxiliary device, which mlx5 PMD associates with the IB device of the nested E-Switch. Representor devargs are not required for such ports and ethdev port name is the name of the probed device. Probing of ports not related to VIRTUAL devlink ports is unchanged. Other mlx5 class drivers (compress, crypto, regex, vdpa) still allow probing only through the IB device owner. - Patches 1-5 - Rework Netlink helpers in common/mlx5, so that they can be used in common driver code, before any ethdev port is spawned. IB port number and IB port to ifindex queries no longer require the IB port to netdev cache, generic netdev query is separated from switch info query, and parent device (bus and device name) of a netdev is queried. Also adds a check whether given device is an auxiliary device. - Patch 6 - When mlx5 common device is created, gathers a list of associated devices. For each IB device port, the parent device of related netdev is added to the list. - Patch 7 - Uses associated devices list to find mlx5 common device for the probed rte_device and passes the probed rte_device to mlx5 class drivers. Only class drivers which explicitly opt in can be probed through a device other than the IB device owner. - Patches 8-9 - Add Netlink queries for the devlink port related to a netdev (through IFLA_DEVLINK_PORT attribute, available since Linux 6.2) and for flavour of that devlink port. - Patch 10 - Adds a common check whether given device is managed by any mlx5 driver. - Patch 11 - Adds MLX5_ETH_FOREACH_SIBLING macro, which iterates over mlx5 ethdev ports sharing the same mlx5 common device. It replaces iteration over ports sharing the same rte_device, which is no longer valid for nested E-Switch. - Patch 12 - Allows probing mlx5 networking driver through associated PCI devices (VFs). Ports related to VIRTUAL devlink ports are recognized as nested E-Switch ports: port of IB device owner is the E-Switch manager, others are representors. Only the port related to the probed device is spawned. - Patch 13 - Allows probing mlx5 networking driver through associated auxiliary devices (SFs). Auxiliary device with multiport IB device is treated as nested E-Switch port. Dariusz Sosnowski (13): common/mlx5: rework IB port number query common/mlx5: allow ifindex query without cache common/mlx5: separate netdev info from switch info common/mlx5: query parent device over Netlink common/mlx5: add check for auxiliary device common/mlx5: gather associated devices common/mlx5: always use EAL device for probing common/mlx5: query devlink port for netdevs common/mlx5: get devlink port by port index common/mlx5: add check for managed device net/mlx5: add sibling device iteration macro net/mlx5: allow associated devices net/mlx5: allow multiport auxiliary device probe drivers/common/mlx5/linux/meson.build | 6 + drivers/common/mlx5/linux/mlx5_common_os.c | 164 ++++++ drivers/common/mlx5/linux/mlx5_common_os.h | 4 + drivers/common/mlx5/linux/mlx5_nl.c | 578 ++++++++++++++----- drivers/common/mlx5/linux/mlx5_nl.h | 67 ++- drivers/common/mlx5/mlx5_common.c | 108 +++- drivers/common/mlx5/mlx5_common.h | 54 ++ drivers/common/mlx5/windows/mlx5_common_os.c | 19 + drivers/common/mlx5/windows/mlx5_common_os.h | 1 + drivers/compress/mlx5/mlx5_compress.c | 1 + drivers/crypto/mlx5/mlx5_crypto.c | 1 + drivers/net/mlx5/linux/mlx5_ethdev_os.c | 24 +- drivers/net/mlx5/linux/mlx5_os.c | 280 +++++++-- drivers/net/mlx5/linux/mlx5_os.h | 2 +- drivers/net/mlx5/mlx5.c | 81 ++- drivers/net/mlx5/mlx5.h | 8 + drivers/net/mlx5/mlx5_ethdev.c | 40 +- drivers/net/mlx5/mlx5_flow.c | 24 +- drivers/net/mlx5/mlx5_mac.c | 8 +- drivers/net/mlx5/mlx5_trigger.c | 10 +- drivers/net/mlx5/windows/mlx5_os.c | 25 +- drivers/regex/mlx5/mlx5_regex.c | 1 + drivers/vdpa/mlx5/mlx5_vdpa.c | 1 + 23 files changed, 1221 insertions(+), 286 deletions(-) -- 2.43.0
