Re: [vfio-users] VFIO and coherency with DMA
On Thu, 13 Feb 2020 10:02:26 + "Stark, Derek" wrote: > Hello, > > I've been experimenting with VFIO with one of our FPGA cards using a > Xilinx part and XDMA IP core. It's been smooth progress so far and > I've had no issues with bar access and also DMA mapping/transfers to > and from the card. All in all, I'm finding it a very nice userspace > driver framework. > > I'm hoping someone can help clarify my understanding of how VFIO > works for DMA in terms of coherence. I'm on a standard x86_64 Intel > Xeon platform. > > In the code I see: > VFIO > /* > * IOMMU enforces DMA cache coherence (ex. PCIe NoSnoop stripping). > This > * capability is subject to change as groups are added or removed. > */ > #define VFIO_DMA_CC_IOMMU 4 > > > Which implies that IOMMU sets the mappings up as coherent is this > understanding correct? No, this is a mechanism for reporting the cache coherency of the IOMMU. For example, KVM uses this to understand whether it needs to emulate wbinv instructions for cases where the DMA is not coherent. There's nothing vfio can specifically do in the IOMMU mapping to make a DMA coherent afaik. > I'm more used to having scatter gather based DMAs where you need to > sync for the CPU or the device depending upon who owns/accesses the > memory. > > The use case I am specifically looking at is if a DMA mapping is > setup through VFIO and then left open whilst data is transferred from > the device to host memory and then the CPU is processing this data. > The pinned/mapped data buffer is reused repeatedly as part of a ring > of buffers. It's only at the point of closing down this application > that the buffer would be unmapped in vfio. > > Is there any sync type functions or equivalents I need to be aware of > in this case? Can VFIO DMA mapped memory buffers be safely used in > this way? It can, but you need to test that cache coherence extension above to know whether the processor is coherent with DMA. If it's not then you need to invalidate the processor cache before you pull in new data from the device or else you might just be re-reading stale data from the cache. Thanks, Alex ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
[vfio-users] VFIO and coherency with DMA
Hello, I've been experimenting with VFIO with one of our FPGA cards using a Xilinx part and XDMA IP core. It's been smooth progress so far and I've had no issues with bar access and also DMA mapping/transfers to and from the card. All in all, I'm finding it a very nice userspace driver framework. I'm hoping someone can help clarify my understanding of how VFIO works for DMA in terms of coherence. I'm on a standard x86_64 Intel Xeon platform. In the code I see: VFIO /* * IOMMU enforces DMA cache coherence (ex. PCIe NoSnoop stripping). This * capability is subject to change as groups are added or removed. */ #define VFIO_DMA_CC_IOMMU 4 Which implies that IOMMU sets the mappings up as coherent is this understanding correct? I'm more used to having scatter gather based DMAs where you need to sync for the CPU or the device depending upon who owns/accesses the memory. The use case I am specifically looking at is if a DMA mapping is setup through VFIO and then left open whilst data is transferred from the device to host memory and then the CPU is processing this data. The pinned/mapped data buffer is reused repeatedly as part of a ring of buffers. It's only at the point of closing down this application that the buffer would be unmapped in vfio. Is there any sync type functions or equivalents I need to be aware of in this case? Can VFIO DMA mapped memory buffers be safely used in this way? Thanks in advance for any information you can help with. Derek CONFIDENTIALITY NOTICE: This message (including any attachments) may contain Molex confidential information, protected by law. If this message is confidential, forwarding it to individuals, other than those with a need to know, without the permission of the sender, is prohibited. This message is also intended for a specific individual. If you are not the intended recipient, you should delete this message and are hereby notified that any disclosure, copying, or distribution of this message or taking of any action based upon it, is strictly prohibited. English | Chinese | Japanese www.molex.com/confidentiality.html ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
