On Sat, Sep 19, 2026 at 02:05:49PM -0700, Stephen Hemminger wrote: > On Fri, 18 Sep 2026 18:50:42 -0400 > Md Rayhanul Islam <[email protected]> wrote: > > > DPDK cannot transmit a single packet on a Raspberry Pi Compute Module 4 > > with an Intel I210. Nothing reports an error: rte_eth_tx_burst() returns > > the full count, the link is up at 1 Gbps, and testpmd in txonly mode > > still shows TX-packets: 0. Received data is DMA'd somewhere other than > > the mbuf. The same card works on x86, and the same board works through > > the kernel's igb driver. > > > > This has been reported twice and never explained: on Stack Overflow in > > October 2023 with DPDK 23.07 [1], where TX-dropped equalled TX-total at > > 55 million with both vfio-noiommu and uio_pci_generic, and on dpdk-users > > in January 2026 with DPDK 25.03 [2]. Two years apart, so this is the > > platform, not a regression. > > > > There are two independent causes, hence two patches. > > > > First, the PCIe host bridge does not present memory to devices at CPU > > physical addresses. Its device tree says so, and DPDK does not look: > > > > $ hexdump -C /proc/device-tree/scb/pcie@7d500000/dma-ranges > > 02000000 00000004 00000000 00000000 00000000 00000001 00000000 > > > > PCI memory space | bus 0x4_0000_0000 | CPU 0x0 | size 4 GiB > > > > A device reaching CPU physical address P must therefore be programmed > > with P + 0x4_0000_0000. In IOVA_PA mode every ring and mbuf address > > DPDK hands the NIC falls outside the inbound window and is discarded > > silently. Patch 1 reads the translation from the host bridge and > > applies it to IOVAs; that alone made the NIC transmit. > > > > Second, the bus is not cache coherent -- no "dma-coherent" on the bridge > > or any ancestor -- so the NIC read stale rings: zeroed descriptors with > > DD set and null buffer addresses. Patch 2 does the cache maintenance > > the kernel DMA API would do. Two details took the longest to find: > > > > - Cleaning to the point of unification (DC CVAU) was not enough. Only > > a clean to the point of coherency made the device see CPU stores. > > > > - A clean writes back a whole 64-byte line, which holds four > > descriptors, so cleaning one erased DD bits the NIC had just set on > > its neighbours. TX completion therefore comes from the hardware > > head register, and RX descriptors are refilled a whole line at a > > time, once every descriptor in that line has come back. > > > > The cache maintenance sits in the driver. Every driver on such a bus > > needs the same thing, so an arch or EAL helper may be the better home; I > > kept it local to e1000 for a first submission and am happy to move it. > > > > Each patch documents its own half, since the failure gives nothing to > > search for. > > > > The device is bound to uio_pci_generic throughout: this SoC has no > > IOMMU, so VFIO cannot be used to sidestep the address question. > > > > Tested on a Compute Module 4 (BCM2711, 4 GB, Cortex-A72, 64-byte cache > > lines) with an I210 (8086:1533 rev 03) on uio_pci_generic, Ubuntu 22.04 > > arm64, kernel 5.15, pcie_aspm=off. With this series applied to current > > main, testpmd txonly reaches 1.42 Mpps at 64-byte frames, which is 1 GbE > > line rate, with 0 TX errors; without it, TX-packets stays 0. The same > > code based on v25.03 also passed a 300k-frame MAC loopback with no loss > > and byte-exact payloads, and 390-run round-trip campaigns against a > > second board with ICMP and UDP probes, 100 to 1500 byte packets, 1k > > pkt/s to line rate, with no unexplained loss. > > > > Not tested: any other board, SoC or NIC. The offset is read from the > > device tree rather than hardcoded, but I have only seen this platform. > > Only e1000 was changed, so other drivers on a non-coherent bus still > > read stale descriptors. > > > > [1] https://stackoverflow.com/questions/77225289/ > > [2] https://mails.dpdk.org/archives/users/2026-January/008433.html > > > > Md Rayhanul Islam (2): > > eal/linux: apply PCIe inbound DMA translation > > net/e1000: maintain caches on non-coherent DMA > > Short observations: > 1. Too much AI generated slop, extra docs, comments on everything. > New code should look like the surrounding code. > This looks like AI wasn't quite sure and left lots of docs for future > self.
I really don't view this as a bad thing. A little too much detail in the comments is better for the future than too little. Given these patches touch areas outside the usual behaviour we expect from DPDK platforms, having extra comments is definitely useful. /Bruce
