On Thu, Dec 11, 2025, kernel test robot <[email protected]> wrote:
> kernel test robot noticed a 20.1% regression of
> sockperf.throughput.UDP.msg_per_sec on:
> commit: 5628f3fe3b16 ("net: add NUMA awareness to skb_attempt_defer_free()")
We independently reproduced this on a second architecture, ppc64le (POWER10),
and can add data that both confirms the commit and shows what amplifies it.
Commit: 5628f3fe3b16 ("net: add NUMA awareness to skb_attempt_defer_free()")
Parent: 844c9db7f7f5 ("net: use llist for sd->defer_list")
Two factors govern the magnitude, and your x86 report and our ppc64le data
agree once both are considered:
(a) How much of the workload runs the net_rx softirq deferred-free path.
Tight loopback UDP (sockperf, iperf3 loopback) is softirq-dominated and is
hit hard. A real-NIC test bounded by other costs is barely affected.
(b) How many *possible* NUMA nodes skb_defer_free_flush() now walks. The new
for_each_node() loop iterates node_possible_map; on a POWER10 LPAR that is
0..31 (32 possible) with only 1 node online, so 31 of every 32 iterations
touch a cold, always-empty per-node list on every softirq RX pass. This
shows up as _find_next_bit(): ~2.76% on POWER10 vs <1% in your x86 profile.
--- Data point 1: stress-ng UDP, isolated commit A/B (POWER10) ---
POWER10, 8 CPUs, 62 GiB, NUMA 1 online / 32 possible. Two upstream v7.2-rc4
kernels, same compiler (gcc 16.1.1) and config; the only difference is the
presence of 5628f3fe3b16. stress-ng 0.19.03, --udp 1 -t 23, 3 runs,
bogo-ops/sec (higher is better):
Kernel Avg Delta
v7.2-rc4 + revert (good) 265145 baseline
v7.2-rc4 stock (bad) 204854 -22.7%
perf (stock vs revert): skb_defer_free_flush 2.75% -> 0%, _find_next_bit
2.76% -> 0% -- both eliminated by the revert; consistent with your x86 profile
(skb_defer_free_flush 0 -> ~4.1%, _find_next_bit -> ~1%).
--- Data point 2: iperf3 UDP loopback A/B (POWER10, second host) ---
Independent reproduction on another POWER10 host (32 possible / 1 online),
iperf3 -u -b 0 -l 16k -t 20, both ends pinned to the online node, 5 runs,
Gbit/s (higher is better):
Kernel (has 5628f3fe3b16?) median mean
no (6.12-based) 6.34 6.31 baseline
yes (7.1-based) 4.85 4.88 -22.7%
(These two kernels differ in base version, so this delta is not a single-commit
isolation on its own -- Data point 1 is the clean isolation. Both land at
-22.7%, matching your 20.1% sockperf figure.)
--- Data point 3: workload dependence (real-NIC vs loopback) ---
The same commit-vs-parent A/B measured with a *real NIC* iperf3 UDP stream
(200G, 3 reboots/kernel) on a 2-possible-node ppc64le host shows only -2.6%
(7515.8 -> 7313.8 Mb/s, zero overlap across reboots). Where the deferred-free
softirq path is a small fraction of the work, the regression is small -- the
mechanism is the same, the exposure differs. This also matches our observation
that a same-CPU-heavy stress-ng udp run barely triggers the cross-CPU
skb_attempt_defer_free() path and shows ~no change.
--- On Jason's question ---
On Sun, Dec 14, 2025, Jason Xing <[email protected]> wrote:
> from what I've known, commit e20dfbad8aab2 and commit 21664814b89e altogether
> can lead to a similar regression ... could you also launch some experiments
> just around those two commits?
On POWER10 our isolation is a direct A/B of 5628f3fe3b16 against its parent
844c9db7f7f5 with identical compiler and config (Data point 1), and a full
revert of only 5628f3fe3b16 recovers the throughput and removes
skb_defer_free_flush()/_find_next_bit() from the profile. So on this hardware
the regression is attributable to 5628f3fe3b16 specifically; e20dfbad8aab2 and
21664814b89e are not in the delta. Happy to test those two separately if useful.
--- Suggested fix ---
Use for_each_online_node() instead of for_each_node() in skb_defer_free_flush()
(and cap what skb_attempt_defer_free() populates accordingly), reducing the loop
from N_possible to N_online -- 32 -> 1 on these POWER10 LPARs. We can build and
benchmark a candidate patch on the POWER10 hosts and report back on this thread.
If a fix is posted, the appropriate tags are:
Reported-by: kernel test robot <[email protected]>
Closes: https://lore.kernel.org/oe-lkp/[email protected]
Thanks,
Adrian Tomasov
Red Hat -- Kernel Performance QE
