Hello, after upgrading the kernel from 6.8 to 7.0 (Proxmox kernels 6.8.12-34-pve -> 7.0.14-26-pve), the software Rx byte counters of an E810 started showing phantom jumps of hundreds of MB to several GB. Each jump is exactly one packet. The cause is a length read after the buffer has already been freed on the XDP_DROP path. The bug is still present in current master.
Setup
-----
- Intel E810-XXV for SFP (rev 02), FW 4.30 0x8001b891,
DDP ICE OS Default Package 1.3.43.0
- in-tree ice, kernel 7.0.14 (Proxmox build 7.0.14-26-pve)
- 2 ports in an 802.3ad bond, native XDP program attached to the bond.
The program returns XDP_DROP for blocklisted source IPs and XDP_PASS
for everything else.
- 6.8 kernel with the same HW/FW and the same XDP program: not affected
What we see
-----------
One sample, 1 s interval, ethtool -S plus netdev stats:
rx_queue_4_packets: 76843 -> 76844 (+1)
rx_queue_4_bytes: 32920496416 -> 33701427810 (+780931394)
rx_bytes.nic (HW) on the same port grew by 2222 bytes in that second
Only the queues that receive the blocklisted flows are affected (3 of 16
here). All other queues show normal bytes per packet. Every jump is
smaller than 4 GiB. 780931394 is 0x2E8C1142, which looks like the low
32 bits of a kernel address.
Root cause
----------
In ice_clean_rx_irq() (drivers/net/ethernet/intel/ice/ice_txrx.c):
xdp_verdict = ice_run_xdp(rx_ring, xdp, xdp_prog, xdp_ring, rx_desc);
if (xdp_verdict == ICE_XDP_PASS)
goto construct_skb;
if (xdp_verdict & (ICE_XDP_TX | ICE_XDP_REDIR))
xdp_xmit |= xdp_verdict;
total_rx_bytes += xdp_get_buff_len(&xdp->base);
On XDP_DROP, XDP_ABORTED and a failed XDP_TX/XDP_REDIRECT, ice_run_xdp()
has already called libeth_xdp_return_buff(xdp). That function frees the
buffer and sets xdp->data = NULL. xdp_get_buff_len() then computes
xdp->data_end - NULL, so the low 32 bits of a pointer get added to
rx_bytes. For multi-buffer frames, xdp_get_shared_info_from_buff() would
additionally read skb_shared_info from a page that has already been
recycled.
This looks to have come in with "ice: switch to Page Pool"
(93f53db9f9dc). The old Rx path did not clear xdp->data on drop.
Proposed fix (untested)
-----------------------
Take the length before running the program:
--- a/drivers/net/ethernet/intel/ice/ice_txrx.c
+++ b/drivers/net/ethernet/intel/ice/ice_txrx.c
@@ static int ice_clean_rx_irq(struct ice_rx_ring *rx_ring, int budget)
- unsigned int size;
+ unsigned int size, len;
@@
if (ice_is_non_eop(rx_ring, rx_desc) ||
unlikely(!xdp->data))
continue;
+ /* ice_run_xdp() may free the buffer and clear xdp->data */
+ len = xdp_get_buff_len(&xdp->base);
xdp_verdict = ice_run_xdp(rx_ring, xdp, xdp_prog, xdp_ring,
rx_desc);
if (xdp_verdict == ICE_XDP_PASS)
goto construct_skb;
if (xdp_verdict & (ICE_XDP_TX | ICE_XDP_REDIR))
xdp_xmit |= xdp_verdict;
- total_rx_bytes += xdp_get_buff_len(&xdp->base);
+ total_rx_bytes += len;
total_rx_pkts++;
With this change TX/REDIRECT count the length before any
bpf_xdp_adjust_head/tail. If you would rather keep counting the
post-program length, the same thing can be done inside ice_run_xdp(),
before the buffer is returned.
The root cause analysis and the proposed fix were done with the help of
Claude Opus 5.5 (Anthropic). The symptoms and measurements come from our
production firewall. I have checked the code paths described above
against the v7.0 source.
I can test a patch on the affected box. The issue reproduces reliably:
each XDP_DROP produces one phantom jump.
Thanks,
Dominik Dvořák
Head of IT Services
Masaryk University | Faculty of Economics and Administration
IT Services
A: Lipová 507/41a | 602 00 Brno
T: +420 549 493 183 | M: +420 739 487 807
E: [email protected]
smime.p7s
Description: S/MIME cryptographic signature
