On Fri, 2015-01-30 at 19:12 +0000, Mark Hounschell wrote: > I've posted the following email to vger.kernel.org but got no response. I am > trying to adapt some of our out of kernel GPL drivers to use the AMD IOMMU. > Here is what I posted to LKML > > "start quote" > > Sorry for the noise. I've read everything DMA in the kernel Doc dir and > searched the web to no avail. So I thought I might get some useful info here. > > I'm currently using a 3.18.3 (x86_64) kernel on an AMD platform. I am > currently doing 8MB DMAs to and from our device using the in kernel CMA > "cma=64M@0-4G" with no problems. This device is not DAC or scatter/gather > capable so the in kernel CMA has been great and replaced our old bigphysarea > usage. > > We simply use dma_alloc_coherent and pass the dma_addr_t *dma_handle > returned from the dma_alloc_coherent function to our device as the "bus/pci" > address to use. > > We also use remap_pfn_range on that dma_addr_t *dma_handle returned from > the dma_alloc_coherent function to mmap userland to the buffer. All is good > until I enable the IOMMU. I then either get IO_PAGE_FAULTs, the DMA just > quietly never completes or the system gets borked.
The dma_addr_t is an I/O virtual address (IOVA), it's the address the *device* uses to access the buffer returned by dma_alloc_coherent. If you mmap that address through /dev/mem, you're getting the processor view of the address, which is not IOMMU translated. Only the device uses the dma_addr_t, processor accesses need to use the returned void*, or some sort of virt_to_phys() version of that to allow userspace to mmap it through devmem. Without an IOMMU, the dma_addr_t is simply a virt_to_bus() translation of the void* buffer, so the code happens to work, but is still and incorrect usage of the DMA API. BTW, depending on how much if your driver is in userspace, vfio might be a better choice for device access and IOMMU programming. Thanks, Alex > [ 106.115725] AMD-Vi: Event logged [IO_PAGE_FAULT device=03:00.0 > domain=0x001b address=0x00000000aa500000 flags=0x0010] > [ 106.115729] AMD-Vi: Event logged [IO_PAGE_FAULT device=03:00.0 > domain=0x001b address=0x00000000aa500040 flags=0x0010] > > Here are the IOMMU settings in my kernel config: > > #grep IOMMU .config > # CONFIG_GART_IOMMU is not set > # CONFIG_CALGARY_IOMMU is not set > CONFIG_IOMMU_HELPER=y > CONFIG_VFIO_IOMMU_TYPE1=m > CONFIG_IOMMU_API=y > CONFIG_IOMMU_SUPPORT=y > CONFIG_AMD_IOMMU=y > # CONFIG_AMD_IOMMU_STATS is not set > CONFIG_AMD_IOMMU_V2=m > CONFIG_INTEL_IOMMU=y > CONFIG_INTEL_IOMMU_DEFAULT_ON=y > CONFIG_INTEL_IOMMU_FLOPPY_WA=y > # CONFIG_IOMMU_STRESS is not set > > > From reading the in kernel doc it would appear that we could in fact, using > the IOMMU and the dma_map_sg function, get rid of the CMA requirement and > our device could DMA anywhere, even above the 4GB address space limit of our > device. But before going through this larger change to our GPL driver, I > want to understand if and/or why the dma_alloc_coherent function does not > appear to set up the IOMMU for me. Is the IOMMU only supported for > "streaming" DMA type and not for "coherent"? I read no reference to this in > the kernel doc? > > Any hints would be greatly appreciated. Again, sorry for the noise. > > > "end quote" > > Sorry if this is not correct place to get info on the AMD IOMMU support in > the kernel. If it's not could someone point me in the right direction? > > Thanks and Regards > Mark > > > > _______________________________________________ > iommu mailing list > [email protected] > https://lists.linuxfoundation.org/mailman/listinfo/iommu _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
