On Thu, Oct 11, 2012 at 11:04:02PM +0200, Or Gerlitz wrote: > On Thu, Oct 11, 2012 at 10:44 PM, Roland Dreier <[email protected]> > wrote: > > > No one has really ever tried to deal with the issue of userspace > > RDMA on a cache-incoherent architecture. Basically if you try the > > current stack, the in-kernel users (IPoIB etc) should be OK but > > libibverbs etc. will be completely broken. > > I think the question might refer even to cache-coherent systems, e.g > in the kernel IB core and ULPs all buffers are dma mapped to/from the > device before/after they are touched by the CPU and vise versa, wheres > in user space, after the buffers are registered once, they are > repeatedly touched by the CPUs and provided to the HW for DMA, e.g all > user space buffers are treated like kernel DMA coherent ones.
The answer is the same, userspace is designed to rely on a DMA cache coherent platform where the only requirement is to issue barrier instructions, which is done in the providers. I'm not sure supporting non-DMA-coherent is even possible with the verbs API, yes we could add the cache ops to the providers, the information is mostly there. However non-DMA-coherent system all require that once you start a DMA WRITE into a cache line *that line is never dirtied by the CPU* - which requires application support that is not even contemplated by verbs. Indeed, I wonder if all the kernel ULPs meet that restriction? Hopefully the forthcoming server grade ARMs are fully DMA coherent.. Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
