On Wed, Mar 11, 2026 at 09:08:50PM +0200, Leon Romanovsky wrote: > From: Leon Romanovsky <[email protected]> > > The RDMA subsystem exposes DMA regions through the verbs interface, which > assumes a coherent system. Use the DMA_ATTR_REQUIRE_COHERENCE attribute to > ensure coherency and avoid taking the SWIOTLB path.
Lets elaborate a bit more so people understand why verbs is like this: The RDMA verbs programming model is like HMM and assumes concurrent DMA and CPU access to userspace memory in a process. The HW device and programming model has so-called "one-sided" operations which are initiated over the network by a remote CPU without notification or involvement of the local CPU. These include things like ATOMIC compare/swap, READ, and WRITE. Using these operations a remote CPU can traverse data structures, form locks, and so on without awareness of the host CPU. Having SWIOTLB substitute the memory or the DMA be cache incoherent completely breaks these use cases. RDMA in-kernel is OK with incoherence because none of the kernel use cases make use of one-sided operations that would cause problems. Jason
