Re: IOAT DMA w/IOMMU

On Tue, Aug 21, 2018 at 4:53 PM, Logan Gunthorpe wrote: > > CPUs don't always have good support for routing PCI P2P transactions. > However, I would have expected a relatively new i7 CPU to support it > well. It may simply be that this CPU does not have good support, however > that comes as a

Re: IOAT DMA w/IOMMU

On 21/08/18 05:45 PM, Eric Pilmore wrote: > Well, the only difference between success and failure is running with the > call to dma_map_resource for the destination address, which is a PCI BAR > address. Prior to Kit introducing this call, we never created a mapping for > the > destination PCI

Re: IOAT DMA w/IOMMU

On Tue, Aug 21, 2018 at 4:35 PM, Logan Gunthorpe wrote: > > > On 21/08/18 05:28 PM, Eric Pilmore wrote: >> >> >> On Tue, Aug 21, 2018 at 4:20 PM, Logan Gunthorpe > > wrote: >> >> >> >> On 21/08/18 05:18 PM, Eric Pilmore wrote: >> > We have been running locally

Re: IOAT DMA w/IOMMU

On 21/08/18 05:28 PM, Eric Pilmore wrote: > > > On Tue, Aug 21, 2018 at 4:20 PM, Logan Gunthorpe > wrote: > > > > On 21/08/18 05:18 PM, Eric Pilmore wrote: > > We have been running locally with Kit's change for dma_map_resource and > its > >

Re: IOAT DMA w/IOMMU

On Tue, Aug 21, 2018 at 4:20 PM, Logan Gunthorpe wrote: > > > On 21/08/18 05:18 PM, Eric Pilmore wrote: >> We have been running locally with Kit's change for dma_map_resource and its >> incorporation in ntb_async_tx_submit for the destination address and >> it runs fine >> under "load" (iperf) on

Re: IOAT DMA w/IOMMU

On Tue, Aug 21, 2018 at 4:20 PM, Logan Gunthorpe wrote: > > > On 21/08/18 05:18 PM, Eric Pilmore wrote: > > We have been running locally with Kit's change for dma_map_resource and > its > > incorporation in ntb_async_tx_submit for the destination address and > > it runs fine > > under "load"

Re: IOAT DMA w/IOMMU

On 21/08/18 05:18 PM, Eric Pilmore wrote: > We have been running locally with Kit's change for dma_map_resource and its > incorporation in ntb_async_tx_submit for the destination address and > it runs fine > under "load" (iperf) on a Xeon (Xeon(R) CPU E5-2680 v4 @ 2.40GHz) based > system, >

Re: IOAT DMA w/IOMMU

On Thu, Aug 16, 2018 at 11:56 AM, Logan Gunthorpe wrote: > > > > On 16/08/18 12:53 PM, Kit Chow wrote: > > > > > > On 08/16/2018 10:21 AM, Logan Gunthorpe wrote: > >> > >> On 16/08/18 11:16 AM, Kit Chow wrote: > >>> I only have access to intel hosts for testing (and possibly an AMD > >>> host

Re: IOAT DMA w/IOMMU

On 16/08/18 12:53 PM, Kit Chow wrote: > > > On 08/16/2018 10:21 AM, Logan Gunthorpe wrote: >> >> On 16/08/18 11:16 AM, Kit Chow wrote: >>> I only have access to intel hosts for testing (and possibly an AMD >>> host currently collecting dust) and am not sure how to go about getting >>> the

Re: IOAT DMA w/IOMMU

On 08/16/2018 10:21 AM, Logan Gunthorpe wrote: On 16/08/18 11:16 AM, Kit Chow wrote: I only have access to intel hosts for testing (and possibly an AMD host currently collecting dust) and am not sure how to go about getting the proper test coverage for other architectures. Well, I thought

Re: IOAT DMA w/IOMMU

On 16/08/18 11:16 AM, Kit Chow wrote: > I only have access to intel hosts for testing (and possibly an AMD > host currently collecting dust) and am not sure how to go about getting > the proper test coverage for other architectures. Well, I thought you were only changing the Intel IOMMU

Re: IOAT DMA w/IOMMU

On 08/09/2018 02:36 PM, Logan Gunthorpe wrote: On 09/08/18 03:31 PM, Eric Pilmore wrote: On Thu, Aug 9, 2018 at 12:35 PM, Logan Gunthorpe wrote: Hey, On 09/08/18 12:51 PM, Eric Pilmore wrote: Was wondering if anybody here has used IOAT DMA engines with an IOMMU turned on (Xeon based

Re: IOAT DMA w/IOMMU

On 14/08/18 00:50, Logan Gunthorpe wrote: On 13/08/18 05:48 PM, Kit Chow wrote: On 08/13/2018 04:39 PM, Logan Gunthorpe wrote: On 13/08/18 05:30 PM, Kit Chow wrote: In arch/x86/include/asm/page.h, there is the following comment in regards to validating the virtual address. /*   *

Re: IOAT DMA w/IOMMU

On 08/13/2018 04:50 PM, Logan Gunthorpe wrote: On 13/08/18 05:48 PM, Kit Chow wrote: On 08/13/2018 04:39 PM, Logan Gunthorpe wrote: On 13/08/18 05:30 PM, Kit Chow wrote: In arch/x86/include/asm/page.h, there is the following comment in regards to validating the virtual address. /*   *

Re: IOAT DMA w/IOMMU

On 13/08/18 05:48 PM, Kit Chow wrote: > On 08/13/2018 04:39 PM, Logan Gunthorpe wrote: >> >> On 13/08/18 05:30 PM, Kit Chow wrote: >>> In arch/x86/include/asm/page.h, there is the following comment in >>> regards to validating the virtual address. >>> >>> /* >>>   * virt_to_page(kaddr) returns a

Re: IOAT DMA w/IOMMU

On 08/13/2018 04:39 PM, Logan Gunthorpe wrote: On 13/08/18 05:30 PM, Kit Chow wrote: In arch/x86/include/asm/page.h, there is the following comment in regards to validating the virtual address. /*  * virt_to_page(kaddr) returns a valid pointer if and only if  * virt_addr_valid(kaddr)

Re: IOAT DMA w/IOMMU

On 13/08/18 05:30 PM, Kit Chow wrote: > In arch/x86/include/asm/page.h, there is the following comment in > regards to validating the virtual address. > > /* >  * virt_to_page(kaddr) returns a valid pointer if and only if >  * virt_addr_valid(kaddr) returns true. >  */ > #define

Re: IOAT DMA w/IOMMU

Taking a step back, I was a little surprised that dma_map_single successfully returned an iommu address for the pci bar address passed into it during my initial experiment... Internally, dma_map_single calls virt_to_page() to translate the "virtual address" into a page and intel_map page

Re: IOAT DMA w/IOMMU

Taking a step back, I was a little surprised that dma_map_single successfully returned an iommu address for the pci bar address passed into it during my initial experiment... Internally, dma_map_single calls virt_to_page() to translate the "virtual address" into a page and intel_map page then

Re: IOAT DMA w/IOMMU

On 08/13/2018 07:59 AM, Robin Murphy wrote: On 13/08/18 15:23, Kit Chow wrote: On 08/10/2018 07:10 PM, Logan Gunthorpe wrote: On 10/08/18 06:53 PM, Kit Chow wrote: I was able to finally succeed in doing the dma transfers over ioat only when prot has DMA_PTE_WRITE set by setting the

Re: IOAT DMA w/IOMMU

On 13/08/18 15:23, Kit Chow wrote: On 08/10/2018 07:10 PM, Logan Gunthorpe wrote: On 10/08/18 06:53 PM, Kit Chow wrote: I was able to finally succeed in doing the dma transfers over ioat only when prot has DMA_PTE_WRITE set by setting the direction to either DMA_FROM_DEVICE or

Re: IOAT DMA w/IOMMU

On 08/10/2018 07:10 PM, Logan Gunthorpe wrote: On 10/08/18 06:53 PM, Kit Chow wrote: I was able to finally succeed in doing the dma transfers over ioat only when prot has DMA_PTE_WRITE set by setting the direction to either DMA_FROM_DEVICE or DMA_BIDIRECTIONAL. Any ideas if the prot settings

Re: IOAT DMA w/IOMMU

On 10/08/18 06:53 PM, Kit Chow wrote: > I was able to finally succeed in doing the dma transfers over ioat only > when prot has DMA_PTE_WRITE set by setting the direction to either > DMA_FROM_DEVICE or DMA_BIDIRECTIONAL. Any ideas if the prot settings > need to be changed? Are there any bad

Re: IOAT DMA w/IOMMU

Success! I've implemented a new intel_map_resource (and intel_unmap_resource) routine which is called by dma_map_resource. As mentioned previously, the primary job of dma_map_resource/intel_map_resource is to call the intel iommu internal mapping routine (__intel_map_single) without

Re: IOAT DMA w/IOMMU

On 08/10/2018 10:15 AM, Logan Gunthorpe wrote: > > > On 10/08/18 11:01 AM, Dave Jiang wrote: >> Or if the BIOS has provided mapping for the Intel NTB device >> specifically? Is that a possibility? NTB does go through the IOMMU. > > I don't know but if the BIOS is doing it, but that would

Re: IOAT DMA w/IOMMU

On 10/08/18 11:01 AM, Dave Jiang wrote: > Or if the BIOS has provided mapping for the Intel NTB device > specifically? Is that a possibility? NTB does go through the IOMMU. I don't know but if the BIOS is doing it, but that would only at best work for Intel NTB I see no hope in getting the

Re: IOAT DMA w/IOMMU

On 08/10/2018 09:33 AM, Logan Gunthorpe wrote: > > > On 10/08/18 10:31 AM, Dave Jiang wrote: >> >> >> On 08/10/2018 09:24 AM, Logan Gunthorpe wrote: >>> >>> >>> On 10/08/18 10:02 AM, Kit Chow wrote: Turns out there is no dma_map_resource routine on x86. get_dma_ops returns

Re: IOAT DMA w/IOMMU

On 10/08/18 10:31 AM, Dave Jiang wrote: > > > On 08/10/2018 09:24 AM, Logan Gunthorpe wrote: >> >> >> On 10/08/18 10:02 AM, Kit Chow wrote: >>> Turns out there is no dma_map_resource routine on x86. get_dma_ops >>> returns intel_dma_ops which has map_resource pointing to NULL. >> >> Oh, yup.

Re: IOAT DMA w/IOMMU

On 08/10/2018 09:24 AM, Logan Gunthorpe wrote: > > > On 10/08/18 10:02 AM, Kit Chow wrote: >> Turns out there is no dma_map_resource routine on x86. get_dma_ops >> returns intel_dma_ops which has map_resource pointing to NULL. > > Oh, yup. I wasn't aware of that. From a cursory view, it

Re: IOAT DMA w/IOMMU

On 10/08/18 10:23 AM, Kit Chow wrote: > There is an internal routine (__intel_map_single) inside the intel iommu > code that does the actual mapping using a phys_addr_t. Think I'll try to > implement a intel_map_resource routine that calls that routine directly > without all of the

Re: IOAT DMA w/IOMMU

On 10/08/18 10:02 AM, Kit Chow wrote: > Turns out there is no dma_map_resource routine on x86. get_dma_ops > returns intel_dma_ops which has map_resource pointing to NULL. Oh, yup. I wasn't aware of that. From a cursory view, it looks like it shouldn't be too hard to implement though. > Will

Re: IOAT DMA w/IOMMU

There is an internal routine (__intel_map_single) inside the intel iommu code that does the actual mapping using a phys_addr_t. Think I'll try to implement a intel_map_resource routine that calls that routine directly without all of the conversions done for dma_map_{single,page} (pci bar addr

Re: IOAT DMA w/IOMMU

Turns out there is no dma_map_resource routine on x86. get_dma_ops returns intel_dma_ops which has map_resource pointing to NULL. (gdb) p intel_dma_ops $7 = {alloc = 0x8150f310 ,   free = 0x8150ec20 ,   mmap = 0x0 , get_sgtable = 0x0 ,   map_page = 0x8150f2d0 ,  

Re: IOAT DMA w/IOMMU

On 08/09/2018 03:50 PM, Logan Gunthorpe wrote: On 09/08/18 04:48 PM, Kit Chow wrote: Based on Logan's comments, I am very hopeful that the dma_map_resource will make things work on the older platforms... Well, I *think* dma_map_single() would still work. So I'm not that confident that's the

Re: IOAT DMA w/IOMMU

On 09/08/18 04:48 PM, Kit Chow wrote: > Based on Logan's comments, I am very hopeful that the dma_map_resource > will make things work on the older platforms... Well, I *think* dma_map_single() would still work. So I'm not that confident that's the root of your problem. I'd still like to see

Re: IOAT DMA w/IOMMU

; David Woodhouse ; Alex Williamson ; iommu@lists.linux-foundation.org Subject: Re: IOAT DMA w/IOMMU On 08/09/2018 02:11 PM, Logan Gunthorpe wrote: On 09/08/18 02:57 PM, Kit Chow wrote: On 08/09/2018 01:11 PM, Logan Gunthorpe wrote: On 09/08/18 01:47 PM, Kit Chow wrote: I haven't tested

RE: IOAT DMA w/IOMMU

avid Woodhouse ; Alex > Williamson ; > iommu@lists.linux-foundation.org > Subject: Re: IOAT DMA w/IOMMU > > > > On 08/09/2018 02:11 PM, Logan Gunthorpe wrote: > > > > On 09/08/18 02:57 PM, Kit Chow wrote: > >> > >> On 08/09/2018 01:11 PM, Logan Guntho

Re: IOAT DMA w/IOMMU

On 08/09/2018 02:11 PM, Logan Gunthorpe wrote: On 09/08/18 02:57 PM, Kit Chow wrote: On 08/09/2018 01:11 PM, Logan Gunthorpe wrote: On 09/08/18 01:47 PM, Kit Chow wrote: I haven't tested this scenario but my guess would be that IOAT would indeed go through the IOMMU and the PCI BAR

Re: IOAT DMA w/IOMMU

On 09/08/18 03:31 PM, Eric Pilmore wrote: > On Thu, Aug 9, 2018 at 12:35 PM, Logan Gunthorpe wrote: >> Hey, >> >> On 09/08/18 12:51 PM, Eric Pilmore wrote: > Was wondering if anybody here has used IOAT DMA engines with an > IOMMU turned on (Xeon based system)? My specific question is

Re: IOAT DMA w/IOMMU

On Thu, Aug 9, 2018 at 12:35 PM, Logan Gunthorpe wrote: > Hey, > > On 09/08/18 12:51 PM, Eric Pilmore wrote: Was wondering if anybody here has used IOAT DMA engines with an IOMMU turned on (Xeon based system)? My specific question is really whether it is possible to DMA (w/IOAT) to

Re: IOAT DMA w/IOMMU

On 09/08/18 02:57 PM, Kit Chow wrote: > > > On 08/09/2018 01:11 PM, Logan Gunthorpe wrote: >> >> On 09/08/18 01:47 PM, Kit Chow wrote: I haven't tested this scenario but my guess would be that IOAT would indeed go through the IOMMU and the PCI BAR address would need to be

Re: IOAT DMA w/IOMMU

On 08/09/2018 01:11 PM, Logan Gunthorpe wrote: On 09/08/18 01:47 PM, Kit Chow wrote: I haven't tested this scenario but my guess would be that IOAT would indeed go through the IOMMU and the PCI BAR address would need to be properly mapped into the IOAT's IOVA. The fact that you see DMAR

Re: IOAT DMA w/IOMMU

On 09/08/18 01:47 PM, Kit Chow wrote: >> I haven't tested this scenario but my guess would be that IOAT would >> indeed go through the IOMMU and the PCI BAR address would need to be >> properly mapped into the IOAT's IOVA. The fact that you see DMAR errors >> is probably a good indication that

Re: IOAT DMA w/IOMMU

Hey, On 09/08/18 12:51 PM, Eric Pilmore wrote: >>> Was wondering if anybody here has used IOAT DMA engines with an >>> IOMMU turned on (Xeon based system)? My specific question is really >>> whether it is possible to DMA (w/IOAT) to a PCI BAR address as the >>> destination without having to map

Re: IOAT DMA w/IOMMU

On 08/09/2018 12:35 PM, Logan Gunthorpe wrote: Hey, On 09/08/18 12:51 PM, Eric Pilmore wrote: Was wondering if anybody here has used IOAT DMA engines with an IOMMU turned on (Xeon based system)? My specific question is really whether it is possible to DMA (w/IOAT) to a PCI BAR address as the

Re: IOAT DMA w/IOMMU

On Thu, Aug 9, 2018 at 11:43 AM, Bjorn Helgaas wrote: > [+cc David, Logan, Alex, iommu list] > > On Thu, Aug 09, 2018 at 11:14:13AM -0700, Eric Pilmore wrote: >> Didn't get any response on the IRC channel so trying here. >> >> Was wondering if anybody here has used IOAT DMA engines with an >>

Re: IOAT DMA w/IOMMU

[+cc David, Logan, Alex, iommu list] On Thu, Aug 09, 2018 at 11:14:13AM -0700, Eric Pilmore wrote: > Didn't get any response on the IRC channel so trying here. > > Was wondering if anybody here has used IOAT DMA engines with an > IOMMU turned on (Xeon based system)? My specific question is