On 2025-07-31 18:36:04, Jesper Dangaard Brouer wrote: > > > On 30/07/2025 09.31, Dragos Tatulea wrote: > > On Tue, Jul 29, 2025 at 01:20:07PM -0700, Dipayaan Roy wrote: > > > On Tue, Jul 29, 2025 at 12:15:23PM +0200, Jesper Dangaard Brouer wrote: > > > > > > > > > > > > On 23/07/2025 21.07, Dipayaan Roy wrote: > > > > > This patch enhances RX buffer handling in the mana driver by > > > > > allocating > > > > > pages from a page pool and slicing them into MTU-sized fragments, > > > > > rather > > > > > than dedicating a full page per packet. This approach is especially > > > > > beneficial on systems with large page sizes like 64KB. > > > > > > > > > > Key improvements: > > > > > > > > > > - Proper integration of page pool for RX buffer allocations. > > > > > - MTU-sized buffer slicing to improve memory utilization. > > > > > - Reduce overall per Rx queue memory footprint. > > > > > - Automatic fallback to full-page buffers when: > > > > > * Jumbo frames are enabled (MTU > PAGE_SIZE / 2). > > > > > * The XDP path is active, to avoid complexities with fragment > > > > > reuse. > > > > > - Removal of redundant pre-allocated RX buffers used in scenarios > > > > > like MTU > > > > > changes, ensuring consistency in RX buffer allocation. > > > > > > > > > > Testing on VMs with 64KB pages shows around 200% throughput > > > > > improvement. > > > > > Memory efficiency is significantly improved due to reduced wastage in > > > > > page > > > > > allocations. Example: We are now able to fit 35 rx buffers in a > > > > > single 64kb > > > > > page for MTU size of 1500, instead of 1 rx buffer per page previously. > > > > > > > > > > Tested: > > > > > > > > > > - iperf3, iperf2, and nttcp benchmarks. > > > > > - Jumbo frames with MTU 9000. > > > > > - Native XDP programs (XDP_PASS, XDP_DROP, XDP_TX, XDP_REDIRECT) for > > > > > testing the XDP path in driver. > > > > > - Page leak detection (kmemleak). > > > > > - Driver load/unload, reboot, and stress scenarios. > > > > > > > > Chris (Cc) discovered a crash/bug[1] with page pool fragments used > > > > from the mlx5 driver. > > > > He put together a BPF program that reproduces the issue here: > > > > - [2] https://github.com/arges/xdp-redirector > > > > > > > > Can I ask you to test that your driver against this reproducer? > > > > > > > > > > > > [1] https://lore.kernel.org/all/aIEuZy6fUj_4wtQ6@861G6M3/ > > > > > > > > --Jesper > > > > > > > > > > Hi Jesper, > > > > > > I was unable to reproduce this issue on mana driver. > > > > > Please note that I had to make a few adjustments to get reprodduction on > > mlx5: > > > > - Make sure that the veth MACs are recognized by the device. Otherwise > > traffic might be dropped by the device. > > > > - Enable GRO on the veth device. Otherwise packets get dropped before > > they reach the devmap BPF program. > > > > Try starting the test program with one thread and see if you see packets > > coming through veth1-ns1 end of the veth pair. > > > > Hi Dipayaan, > > Enabling GRO on the veth device is quite important for the test to be valid. > > I've asked Chris to fix this in the reproducer. He can report back when > he have done this, so you can re-run the test. It is also good advice > from Dragos that you should check packets are coming through the veth > pair, to make sure the test is working. > > The setup.sh script also need to be modified, as it is loading xdp on a > net_device called "ext0" [0], which is specific to our systems (which > default also have GRO enabled for veth). > > [0] https://github.com/arges/xdp-redirector/blob/main/setup.sh#L28 > > --Jesper
I pushed some updates to the setup script to make it easier to use. If you have issues running the script, please share the output. --chris
