** Summary changed: - [SRU] "ethtool -x" fails with "netlink error: Operation not supported" on drivers without get_channels (noble, ethtool 6.7) + "ethtool -x" fails with "netlink error: Operation not supported" on interfaces without channels support (noble, ethtool 6.7)
** Description changed: - [ Impact ] + Ubuntu release: 24.04 LTS (noble), arm64 + Package: ethtool 1:6.7-1build1 (the only ethtool in noble; there is no noble-updates or noble-security upload) + Seen on: NVIDIA Jetson AGX Thor Developer Kit - On Ubuntu 24.04 (noble), "ethtool -x <iface>" (--show-rxfh) fails with + == What happens == - netlink error: Operation not supported + On noble, "ethtool -x <iface>" fails on some network interfaces with - on any network driver that implements RSS (ethtool_ops::get_rxfh and - ETHTOOL_GRXRINGS) but does not implement ethtool_ops::get_channels. The RSS - indirection table, hash key and hash function cannot be displayed, although - the kernel driver answers the RSS request correctly. + netlink error: Operation not supported - This is a regression in the ethtool userspace tool, not in the kernel. - ethtool 6.2 converted --show-rxfh from ioctl to netlink (commit ffab99c1f382 - "netlink: add netlink handler for get rss (-x)"). After receiving the RSS - reply, the netlink handler sends ETHTOOL_MSG_CHANNELS_GET only to learn the - RX ring count for display. Drivers that implement ETHTOOL_GRXRINGS do not - necessarily implement the channels API, so that second request returns - EOPNOTSUPP and ethtool discards the RSS data it already received. + and prints nothing else. The RSS indirection table, hash key and hash + function cannot be displayed, so RSS configuration cannot be checked on + these systems. - It is fixed upstream in ethtool 6.11 by: + It happens on any interface that supports RSS but does not support the + channels query, that is, where "ethtool -l <iface>" also answers + "Operation not supported". The kernel is fine: it answers the RSS + request correctly. - dff30c61ff76 "netlink: rss: retrieve ring count using ETHTOOL_GRXRINGS ioctl" - https://git.kernel.org/pub/scm/network/ethtool/ethtool.git/commit/?id=dff30c61ff76c4145031f524c5856bf5f1f7306b - Fixes: ffab99c1f382 ("netlink: add netlink handler for get rss (-x)") + Expected: the command prints the RSS indirection table, the hash key and + the hash function, and returns 0. - The fix goes back to the ETHTOOL_GRXRINGS ioctl for the ring count, as the - ioctl path (do_grxfh) has always done. + == Cause == - This is a regression from the previous LTS: jammy ships ethtool 5.16, which - predates the netlink handler and works on the same drivers. + ethtool 6.2 moved "-x" from ioctl to netlink. After it receives the RSS + data, the new code asks for the channel counts, only to print the RX + ring count in the header line. When that second request is not + supported, ethtool gives up and throws away the RSS data it already has. - The upstream report lists these in-tree drivers as affected at the time: - lan743x, enetc, mvpp2, mvneta, atlantic, cxgb4, sfc (ef100, falcon, siena) - and ixgbevf. We hit it on <AFFECTED HARDWARE AND DRIVER>, arm64, where it - blocks validation of RSS configuration. + == Fix == - Series status: - noble 1:6.7-1build1 affected - plucky 1:6.11-1 fixed, as are all later series - devel 1:7.2-1 fixed - jammy 1:5.16 not affected + Fixed upstream in ethtool 6.11 by one commit: - [ Test Plan ] + dff30c61ff76 "netlink: rss: retrieve ring count using ETHTOOL_GRXRINGS ioctl" + https://git.kernel.org/pub/scm/network/ethtool/ethtool.git/commit/?id=dff30c61ff76c4145031f524c5856bf5f1f7306b - A NIC is needed whose driver implements get_rxfh but not get_channels: any - of the in-tree drivers above (for example atlantic, or an ixgbevf VF), or - the hardware named in [ Impact ]. We will run the verification on our - affected hardware and report here. + It reads the ring count the way the old ioctl path always did. - 1. Reproduce with the current noble package: + == Upstream == - $ apt policy ethtool # 1:6.7-1build1 - $ sudo ethtool -x <iface> - netlink error: Operation not supported - $ echo $? - 1 - - 2. Install ethtool from noble-proposed and repeat: - - $ sudo ethtool -x <iface> - RX flow hash indirection table for <iface> with <N> RX ring(s): - 0: ... - RSS hash key: - ... - RSS hash function: - toeplitz: on - ... - $ echo $? - 0 - - 3. Regression check on a driver that does implement get_channels (for - example virtio_net with multiqueue, igb, ixgbe or mlx5): capture - "ethtool -x <iface>" with the old and the new package and confirm the - output is identical, including the ring count in the header line. - - 4. Repeat steps 1 to 3 on both the noble GA kernel and the current HWE - kernel. - - [ Where problems could occur ] - - The change is confined to the netlink path of --show-rxfh (-x): 3 files, - 22 insertions, 36 deletions. No other subcommand is touched. - - The RX ring count printed in the header is now read with the - ETHTOOL_GRXRINGS ioctl instead of being derived from the channel counts - (combined + rx). Possible regressions: - - - A driver that implements get_channels but not ETHTOOL_GRXRINGS works with - 6.7 today and would fail after this change with "Cannot get RX ring - count". That matches the behaviour of jammy (5.16) and of every release - from 6.11 on. - - On a driver where the two counts differ, the number in the "with N RX - ring(s)" header line changes. Scripts that parse that line could see a - different value. - - The netlink path now also opens the ioctl socket. In an environment - where that socket cannot be opened, -x would fail where it previously - succeeded. - - A regression would show up as "ethtool -x" failing or printing a different - ring count. It would not affect network configuration or traffic. - - [ Other Info ] - - - Upstream report and discussion: - https://lore.kernel.org/netdev/20240711114535.pfrlbih3ehajnpvh@skbuf/ - - The upstream commit applies to the 6.7 source without changes in our - testing. A build with only this patch was validated on the affected - arm64 hardware: "ethtool -x" succeeds and shows the ring count, the full - indirection table and the hash key. - - LP: #2146605 has a full-version backport, 1:7.1-1~24.04.1, waiting in the - noble Unapproved queue. That version contains this fix. If it is - accepted, this bug can be closed as fixed by it. If not, we ask for this - single cherry-pick as 1:6.7-1ubuntu0.1. - - Precedent for a one-patch SRU of the same class (a regression from an - ioctl-to-netlink conversion in ethtool): LP: #2100246, jammy - 1:5.16-1ubuntu0.2. + - Upstream report and discussion: https://lore.kernel.org/netdev/20240711114535.pfrlbih3ehajnpvh%40skbuf/ + - The upstream commit applies to the 6.7 source unchanged. We built noble's ethtool with only this patch and tested it on our hardware: "ethtool -x" succeeds and shows the ring count, the full indirection table and the hash key. ** Tags removed: regression-release -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2173817 Title: "ethtool -x" fails with "netlink error: Operation not supported" on interfaces without channels support (noble, ethtool 6.7) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ethtool/+bug/2173817/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
