Also available here
https://github.com/pkitszel/linux/tree/xlvf-v2
The purpose of this series is to allow iavf to use more than 16 queue
pairs, in two modes: up to 64 and up to 256 queue pairs.
In order to support more queues for VF, we must give it a bigger RSS
table (GLOBAL LUT or PF LUT). There are 16 GLOBAL LUTs on E810, and one
PF LUT for every PF on given card. Both kinds could be (re)assigned to
a VF, while a PF must hold at least one of them at any given moment.
GLOBAL LUT allows VF to use up to 64 queues, PF LUT lets it use up to
256 queues.
RSS LUTs are exposed to the user for assignment via devlink resources
API, which I have extended to make it possible.
Devlink changes:
1. Extend shared devlink by two callbacks, that give the driver
a constructor/destructor for the priv data attached to the shared
devlink instance. The constructor gets an additional param, used in
"ice: represent RSS LUTs as devlink resources". mlx5 is touched only
to adjust to the API change.
ice uses the shared devlink instance as a "whole device" aggregate
over all PFs on given card, instead of its custom xarray.
2. Extend devlink resources API to allow user to assign resources.
Before, only the driver could assign resources, without any way for
the user to interact. Now the driver could register an occupancy
setter (.occ_set()) for a resource. ice RSS LUTs are exposed that
way. It was proposed as an RFC in Feb 2025, link in the patch.
There are also patches for, Admin Queue extension, GLOBAL RSS LUT
alloc/free, and two rather big "new opcodes" patches, by Ahmed (iavf)
and Brett (ice).
Finally there is a patch that adds devlink instance for VF and
registers devlink resources on it, combined with all the glue code to
make actual use of the whole series and accept the larger number of
queues for VF. This (the last) patch contains usage examples.
There is one resource added that just groups GLOBAL and PF LUTs under
it, so there are 3 rows of data for each device (PF/VF/whole-dev):
$ devlink resource show pci/0000:18:00.0
pci/0000:18:00.0:
name rss size 1 unit entry size_min 0 size_max 2 size_gran 1 dpipe_tables none
resources:
name lut_512 size 0 unit entry size_min 0 size_max 1 size_gran 1
dpipe_tables none
name lut_2048 size 1 unit entry size_min 0 size_max 1 size_gran 1
dpipe_tables none
Technically the aggregate "name rss" line could be eliminated, with
just the two last ones kept (then renamed to "rss_lut_2048" form, from
the current "rss/lut_2048"). I like it as it is, but this is just an
opinion, I'm open to discussion.
v2:
* drop "ice: simplify ice_vc_dis_qs_msg() a little", already applied
* devlink: propagate the .shd_init() error to the caller, so
devlink_shd_get() returns ERR_PTR() now; register the shared
instance only after successful .shd_init(); document the new ops
* devlink: document the .occ_set() mode of resources
* ice: prefix the shared devlink id with "ice:", handle errors of
devl_nested_devlink_set()
* iavf: stop multi-message virtchnl ops (queue config, queue-vector
mapping) at the first error or PF NACK, keep the op pending on
timeout
* iavf: fix queue and RSS (re)negotiation on init, reset and
ethtool -L
* ice: serialize devlink RSS LUT changes with ethtool -x/-X/-L, RXHASH
toggle and VSI (re)configuration, by new pf->rss_lut_lock; make ADQ
and devlink RSS LUT reassignment mutually exclusive
* ice: limit VF queue count and PF RSS size to the capacity of the
current RSS LUT
* ice: create VF devlink together with the VF, rework deferred VF reset
* many smaller fixes, mostly after Sashiko review; per-patch changelogs
are under "---", together with Sashiko suggestions that I decided
not to apply, and why
v1:
https://lore.kernel.org/netdev/[email protected]
Ahmed Zaki (1):
iavf: use new opcodes to request more than 16 queues
Brett Creeley (2):
ice: add VF queue ena/dis helper functions
ice: introduce handling of virtchnl LARGE VF opcodes
Przemek Kitszel (11):
devlink, mlx5: add init/fini ops for shared devlink
ice: use shared devlink to store ice_adapters instead of custom xarray
ice: add helpers for Global RSS LUT alloc, free, vsi_update
ice: rename ICE_MAX_RSS_QS_PER_VF to ICE_MAX_QS_PER_VF_VCV1
ice: bump to 256qs for VF
iavf: extend iavf_configure_queues() to support more queues
iavf: temporary rename of IAVF_MAX_REQ_QUEUES to
IAVF_MAX_REQ_QUEUES_VCV1
iavf: increase max number of queues to 256
devlink: give user option to allocate resources
ice: represent RSS LUTs as devlink resources
ice: support up to 256 VF queues
.../networking/devlink/devlink-resource.rst | 5 +
.../networking/devlink/devlink-shared.rst | 19 +-
drivers/net/ethernet/intel/ice/Makefile | 1 +
drivers/net/ethernet/intel/iavf/iavf.h | 21 +-
.../net/ethernet/intel/iavf/iavf_register.h | 4 +-
.../net/ethernet/intel/ice/devlink/devlink.h | 2 +-
.../net/ethernet/intel/ice/devlink/resource.h | 26 +
drivers/net/ethernet/intel/ice/ice.h | 3 +
drivers/net/ethernet/intel/ice/ice_adapter.h | 52 +-
.../net/ethernet/intel/ice/ice_adminq_cmd.h | 1 +
drivers/net/ethernet/intel/ice/ice_common.h | 1 +
drivers/net/ethernet/intel/ice/ice_lag.h | 2 +-
drivers/net/ethernet/intel/ice/ice_lib.h | 5 +
drivers/net/ethernet/intel/ice/ice_switch.h | 2 +
drivers/net/ethernet/intel/ice/ice_vf_lib.h | 36 +-
drivers/net/ethernet/intel/ice/virt/queues.h | 3 +
drivers/net/ethernet/intel/ice/virt/rss.h | 1 +
.../net/ethernet/intel/ice/virt/virtchnl.h | 4 +
include/linux/net/intel/virtchnl.h | 136 +++-
include/net/devlink.h | 37 +
.../net/ethernet/intel/iavf/iavf_ethtool.c | 3 +
drivers/net/ethernet/intel/iavf/iavf_main.c | 134 +++-
.../net/ethernet/intel/iavf/iavf_virtchnl.c | 334 ++++++++-
.../net/ethernet/intel/ice/devlink/devlink.c | 11 +-
.../net/ethernet/intel/ice/devlink/resource.c | 644 ++++++++++++++++++
drivers/net/ethernet/intel/ice/ice_adapter.c | 107 ++-
drivers/net/ethernet/intel/ice/ice_common.c | 2 +-
drivers/net/ethernet/intel/ice/ice_ethtool.c | 103 +--
drivers/net/ethernet/intel/ice/ice_lag.c | 6 +-
drivers/net/ethernet/intel/ice/ice_lib.c | 178 +++--
drivers/net/ethernet/intel/ice/ice_main.c | 60 +-
drivers/net/ethernet/intel/ice/ice_sriov.c | 15 +-
drivers/net/ethernet/intel/ice/ice_switch.c | 41 ++
drivers/net/ethernet/intel/ice/ice_vf_lib.c | 69 +-
.../net/ethernet/intel/ice/virt/allowlist.c | 8 +
drivers/net/ethernet/intel/ice/virt/queues.c | 468 +++++++++++--
drivers/net/ethernet/intel/ice/virt/rss.c | 37 +-
.../net/ethernet/intel/ice/virt/virtchnl.c | 42 +-
.../ethernet/mellanox/mlx5/core/sh_devlink.c | 6 +-
net/devlink/resource.c | 88 ++-
net/devlink/sh_dev.c | 57 +-
41 files changed, 2478 insertions(+), 296 deletions(-)
create mode 100644 drivers/net/ethernet/intel/ice/devlink/resource.h
create mode 100644 drivers/net/ethernet/intel/ice/devlink/resource.c
--
2.51.1
