(I'm kinda reviewing this very late here. Some suggestions/comments but feel free to ignore if not useful).
On Wed, Jul 1, 2026 at 12:22 PM Bobby Eshleman <[email protected]> wrote: > > Every devmem dmabuf binding hands the page_pool PAGE_SIZE niovs today. > On NICs that consume one descriptor per netmem, this caps a single RX > descriptor at PAGE_SIZE and burns CPU on buffer churn. > > In this series, we add a bind-time netlink attribute, > NETDEV_A_DMABUF_RX_BUF_SIZE, that lets userspace request a larger niov size > (power of two >= PAGE_SIZE). FWIW we may be able to support arbitrary sizes with devmem. Because the genpool supports byte-aligned allocations AFAIR. Also the dma-mapping happens with the dma-buf size, so the actual niov size doesn't matter. The only thing I can think off which may not be flexible to arbitrary sizes is the driver itself. IDK what happens if you ask the driver to dma into a buffer that is frag size 5023 or something like that. But that is something that can be relaxed in the future. > Drivers must opt in via > queue_mgmt_ops.QCFG_RX_PAGE_SIZE. > nit that probably doesn't matter: ...QCFG_RX_NETMEM_SIZE, or (...NIOV_SIZE). This doesn't actually work with pages, right? If you decide to extend to arbrary sizes, I would add to the queue_mgmt ops supports_netmem_size(size_t size) function, and let the driver enforce "it has to be power of 2" if it needs to. AFAICT core doesn't need to. > Selftests use udmabuf, but udmabuf sgtables were previously hardcoded to > PAGE_SIZE. This series modifies udmabuf to respect folio sizes in its exported > sgtable. The result is that when backing udmabuf with MFD_HUGETLB 2MB pages, > the sgtable is populated with 2MB entries, allowing devmem's gen_pool to carve > out large (eg. 64K) niovs. > > Measurements > ------------ > > Setup: kperf devmem RX/TX cuda, 4 flows, 64 MB messages, 60s, dctcp, > num-rx-queues=4, dmabuf-rx/tx-size-mb=2048, 10 runs per niov size, > mlx5. > > niov RX dev Gbps RX flow avg Gbps app sys % > ----- ---------------- ----------------- ---------------- > 4K 300.63 +/- 53.21 75.16 +/- 13.30 54.15 +/- 10.23 > 16K 321.35 +/- 28.20 80.34 +/- 7.05 41.05 +/- 8.87 > 32K 347.63 +/- 2.20 86.91 +/- 0.55 44.54 +/- 3.51 > 64K 332.11 +/- 14.26 83.03 +/- 3.56 35.47 +/- 3.11 > > RX app sys % drops ~19% from 4K to 64K. > Hard to read the columns for me but seems like good perf data. Did performance become worse from 32K to 64K? I wonder why. I have some devmem performance fixes that are very critical for our production that I haven't gotten around to upstreaming yet. I wonder if I can send them to you for upstream submission. Are you potentially interested? -- Thanks, Mina
