Thanks! Tested on the affected box and sent: https://lore.kernel.org/netdev/[email protected]/
Dominik ________________________________________ From: Maciej Fijalkowski <[email protected]> Sent: Thursday, November 5, 2026 12:19 To: Dominik Dvořák <[email protected]> Cc: [email protected] <[email protected]>; [email protected] <[email protected]>; Tony Nguyen <[email protected]>; Przemek Kitszel <[email protected]> Subject: Re: [BUG] ice: rx_bytes inflated by up to 4 GiB per XDP_DROP (buffer length read after libeth_xdp_return_buff()) [You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] On Thu, Nov 05, 2026 at 09:05:19AM +0000, Dominik Dvořák wrote: > Hello, > > after upgrading the kernel from 6.8 to 7.0 (Proxmox kernels > 6.8.12-34-pve -> 7.0.14-26-pve), the software Rx byte counters of an E810 > started showing phantom jumps of hundreds of MB to several GB. Each jump > is exactly one packet. The cause is a length read after the buffer has > already been freed on the XDP_DROP path. The bug is still present in > current master. > > Setup > ----- > - Intel E810-XXV for SFP (rev 02), FW 4.30 0x8001b891, > DDP ICE OS Default Package 1.3.43.0 > - in-tree ice, kernel 7.0.14 (Proxmox build 7.0.14-26-pve) > - 2 ports in an 802.3ad bond, native XDP program attached to the bond. > The program returns XDP_DROP for blocklisted source IPs and XDP_PASS > for everything else. > - 6.8 kernel with the same HW/FW and the same XDP program: not affected > > What we see > ----------- > One sample, 1 s interval, ethtool -S plus netdev stats: > > rx_queue_4_packets: 76843 -> 76844 (+1) > rx_queue_4_bytes: 32920496416 -> 33701427810 (+780931394) > rx_bytes.nic (HW) on the same port grew by 2222 bytes in that second > > Only the queues that receive the blocklisted flows are affected (3 of 16 > here). All other queues show normal bytes per packet. Every jump is > smaller than 4 GiB. 780931394 is 0x2E8C1142, which looks like the low > 32 bits of a kernel address. > > Root cause > ---------- > In ice_clean_rx_irq() (drivers/net/ethernet/intel/ice/ice_txrx.c): > > xdp_verdict = ice_run_xdp(rx_ring, xdp, xdp_prog, xdp_ring, rx_desc); > if (xdp_verdict == ICE_XDP_PASS) > goto construct_skb; > > if (xdp_verdict & (ICE_XDP_TX | ICE_XDP_REDIR)) > xdp_xmit |= xdp_verdict; > total_rx_bytes += xdp_get_buff_len(&xdp->base); > > On XDP_DROP, XDP_ABORTED and a failed XDP_TX/XDP_REDIRECT, ice_run_xdp() > has already called libeth_xdp_return_buff(xdp). That function frees the > buffer and sets xdp->data = NULL. xdp_get_buff_len() then computes > xdp->data_end - NULL, so the low 32 bits of a pointer get added to > rx_bytes. For multi-buffer frames, xdp_get_shared_info_from_buff() would > additionally read skb_shared_info from a page that has already been > recycled. > > This looks to have come in with "ice: switch to Page Pool" > (93f53db9f9dc). The old Rx path did not clear xdp->data on drop. > > Proposed fix (untested) > ----------------------- > Take the length before running the program: > > --- a/drivers/net/ethernet/intel/ice/ice_txrx.c > +++ b/drivers/net/ethernet/intel/ice/ice_txrx.c > @@ static int ice_clean_rx_irq(struct ice_rx_ring *rx_ring, int budget) > - unsigned int size; > + unsigned int size, len; > @@ > if (ice_is_non_eop(rx_ring, rx_desc) || > unlikely(!xdp->data)) > continue; > > + /* ice_run_xdp() may free the buffer and clear xdp->data */ > + len = xdp_get_buff_len(&xdp->base); > xdp_verdict = ice_run_xdp(rx_ring, xdp, xdp_prog, xdp_ring, > rx_desc); > if (xdp_verdict == ICE_XDP_PASS) > goto construct_skb; > > if (xdp_verdict & (ICE_XDP_TX | ICE_XDP_REDIR)) > xdp_xmit |= xdp_verdict; > - total_rx_bytes += xdp_get_buff_len(&xdp->base); > + total_rx_bytes += len; > total_rx_pkts++; > > With this change TX/REDIRECT count the length before any > bpf_xdp_adjust_head/tail. If you would rather keep counting the > post-program length, the same thing can be done inside ice_run_xdp(), > before the buffer is returned. > > The root cause analysis and the proposed fix were done with the help of > Claude Opus 5.5 (Anthropic). The symptoms and measurements come from our > production firewall. I have checked the code paths described above > against the v7.0 source. > > I can test a patch on the affected box. The issue reproduces reliably: > each XDP_DROP produces one phantom jump. Please test and send a fix. You've done the root cause analysis so would be nice if you get a credit for providing a fix to this issue. I can then test this on my setup and give you an ACK. Thanks! > > Thanks, > Dominik Dvořák > Head of IT Services > > Masaryk University | Faculty of Economics and Administration > IT Services > A: Lipová 507/41a | 602 00 Brno > T: +420 549 493 183 | M: +420 739 487 807 > E: [email protected] > >
