Hello,
We run a native-XDP forwarding datapath (an eBPF GTP-U user plane) on an
Intel N100 mini-PC with two i226-V NICs (in-tree igc, firmware 2023:889d,
2.5GBASE-T; Ubuntu 24.04, 7.0.0-38-generic, iommu=pt). Small-packet
capacity is ~2.6 Mpps/port here. We found that after the port is
over-offered for a few seconds, the RX path keeps a persistent per-packet
cost regression after the offered load is reduced back to a clean level.
It never recovers on its own. A comparable storm-and-return test on a
second box with an ixgbe NIC did not leave a persistent regression, so we
would appreciate a look from the igc side.
1) Observation
Same workload (82-byte frames), same coalescing (fixed EITR, 65 us), same
IRQ rate (~15.6k/queue), no frequency change:
fresh after boot: 42.7% per-core softirq utilization at 2.4 Mpps/port
after one storm: 50.8% (+8 points; an earlier, less-tuned config
measured 55.2% -> 71.1%)
then keeps creeping: +5 points over the following 40 min of traffic
"Storm" = ~2.7 Mpps/port offered for ~15 s: rx_no_buffer_count +~0.2M/port,
rx_fifo_errors +0.06-0.6M, no adapter reset, dmesg silent.
perf stat, same offered rate before vs after (equal event set):
instructions: unchanged (216.5G vs 217.3G)
cycles: +17-23%
LLC-loads: +116%
DRAM-hit loads: x2.5 (37.8M -> 96.0M)
load-bound stalls: x2.5 (12.1G -> 30.1G)
igc_clean_rx_irq self-time ~8.5% -> 16.9%
PEBS (perf mem) attributes the delta to the two dependent per-packet
loads of struct page (+0x8 compound_head, +0x34 refcount), moving from
L1/L2 hits to DRAM hits.
2) RX ring page layout
Physical addresses of RX buffer pages, sampled via bpftrace
(igc_clean_rx_irq -> ring -> rx_buffer_info -> page):
fresh: 97-100% of adjacent slots within 128 KiB (median step 8 KiB)
stormed: 0-2% (median neighbour distance 115-150 MB); ~2/3 of the page
set replaced (intersection vs pre-storm snapshot: 23-37%)
During the storm __page_frag_cache_drain runs 0.45-0.73M times per 15 s;
storms that do not trigger this recycling leave layout and performance
unchanged. Our reading: storm -> buffer recycling churn -> refill from
fragmented blocks -> scattered ring pages -> per-packet page-metadata
misses.
We are not claiming this is igc-exclusive -- ixgbe shares a similar
RX-buffer lifetime shape; the X520 result above just means we cannot say
the effect is universal either.
3) Recovery (measured)
ip link down/up, ethtool -r, FLR, compact_memory: no effect
rmmod/modprobe x1: partial
rmmod/modprobe x2-3: near baseline, stable
reboot: full
(local workaround: force fresh ring allocations at refill: full)
4) Related observation (happy to split out or drop)
The default RSS indirection table is block-interleaved
(igc_setup_mrqc(): tbl[j] = (j * queues) / IGC_RETA_SIZE). With
correlated tunnel flows (GTP-U flows differing only in low tuple bits)
and a per-boot random RSS key, we saw the whole workload collapse into
one 32-bucket block -> a single RX queue (2 cores saturated, capacity
~halved). ethtool -X equal N restores the spread; we apply that as a
workaround. ixgbe initialises this table interleaved. Flagging because it
materially lowers this box's small-packet ceiling and makes the storm
above easier to reach. (Also: RSS key get/set was merged in July 2026 --
thanks; our distro kernel predates it.)
5) Ask
- Is this persistent post-storm state known for igc/I226? Any recommended
mitigation?
- Would keeping the RX ring pages collectively fresh (allocation
strategy, higher-order pages, a local reserve) at refill time be
feasible in the driver, or is this considered allocator-side?
- Happy to test patches, run mainline, or minimise the reproducer
(trivial XDP program).
Raw data available on request: perf reports, counter dumps, ring-page
dumps, recovery logs, reproducer notes (two-day study).
Thanks,
Frank
