MANA reconfigures a port by destroying its queues and building them again.
mana_detach() tears the whole queue set down, mana_attach() allocates a new
one, and every path that changes a queue property does this: the ethtool
channel, ring and private-flag setters, the MTU change and XDP attach.
If the second half fails there is nothing left to fall back to. The old
queues are already gone, so the port stays down and the failure cannot be
undone from userspace:
# ethtool -G ens1 rx 8192 tx 16384 # under memory pressure
netlink error: Cannot allocate memory
mana 7870:00:00.0 ens1: Failed to create 16 TX queues, -12
mana 7870:00:00.0 ens1: mana_attach failed: -12
# ip link show ens1
... state DOWN
On VM SKUs with no netvsc fallback interface this takes the VM off the
network entirely, and it is reachable from an ordinary ethtool ring resize
that happens to run when memory is tight.
v1 [1] tried to recover after the fact: schedule a reset and retry
mana_attach() with smaller values, falling back to the previous setting or
to the defaults. Review pointed out that silently replacing a user's
setting with a different one is the wrong behaviour, and that
pre-allocating the resources and replacing the queue set looked feasible
and should be investigated instead. This series does that, so there is no
failure to recover from and no user setting to override.
The model is to build the new queue set alongside the running one, publish
it, then retire the old one.
carrier off -> netif_tx_disable -> swap the queue pointers -> update the
netdev queue counts -> reprogram RSS/steering -> reattach XDP ->
netif_tx_start_all_queues -> restore carrier
Everything that can fail happens before anything is swapped. If allocation
fails the running queues have not been touched at all: the ethtool call
returns the error, the interface keeps forwarding traffic and the
configuration is unchanged. If the swap itself fails, the previous set is
put back and the port keeps running on it.
Patch layout:
1 the queue-set allocate/publish/free helpers.
2-6 convert the five callers: ethtool channels, rings and private
flags, the MTU change and XDP attach.
7 the remaining detach/attach users are the TX-timeout reset handler
and suspend; make mana_detach() always finish its teardown so the
reset handler cannot leave a port permanently dead.
8-11 keep behaviour the swap model would otherwise change: per-queue
counters move into the port context so a rebuild no longer zeroes
the interface statistics, the EQ pool is shared across a swap
instead of being duplicated, EQs left idle by a reduction are
released, and a user-configured RSS indirection table survives a
rebuild.
12-13 stop rebuilding queues that do not change. A reduction keeps the
surviving queues and an increase keeps the existing ones, so
32 -> 8 channels destroys 24 queue pairs instead of building 8 and
destroying 32, and 4 -> 8 builds 4 instead of 8.
Known trade-off, since it is deliberate: RX counters are now indexed by
queue number in the port context, so during the window between steering
moving to the new queues and the old ones being destroyed, both generations
can update one slot. MANA is 64-bit only, so u64_stats_sync carries no
seqcount and nothing can be corrupted; at worst a few increments are lost.
Serialising them would put a lock in the per-packet receive path, and
giving each set its own slots would make ndo_get_stats64() report a dip
during a swap, which is the regression this is meant to remove.
[1]
https://lore.kernel.org/netdev/[email protected]/
Dipayaan Roy (1):
net: mana: do not bail out of mana_detach on dealloc failure
Long Li (12):
net: mana: add queue-set allocation and teardown helpers
net: mana: swap queue sets in mana_set_channels
net: mana: swap queue sets in mana_set_ringparam
net: mana: swap queue sets in mana_set_priv_flags
net: mana: swap queue sets in mana_change_mtu
net: mana: swap queue sets in mana_xdp_set
net: mana: keep per-queue statistics in the port context
net: mana: share the EQ pool across a queue-set swap
net: mana: release EQs left idle by a channel-count reduction
net: mana: keep a user-configured RSS table across a queue rebuild
net: mana: keep the surviving queues when the channel count is reduced
net: mana: keep the existing queues when the channel count is raised
.../net/ethernet/microsoft/mana/mana_bpf.c | 117 +-
drivers/net/ethernet/microsoft/mana/mana_en.c | 1698 ++++++++++++++++-
.../ethernet/microsoft/mana/mana_ethtool.c | 333 +++-
include/net/mana/gdma.h | 11 +-
include/net/mana/mana.h | 130 +-
5 files changed, 2066 insertions(+), 223 deletions(-)
base-commit: d67e5dbda22604d0fcde32fce58c65f88676e676
--
2.43.0
