Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches

2018-05-08 Thread Christian König
Hi Bjorn, Am 08.05.2018 um 01:13 schrieb Bjorn Helgaas: [+to Alex] Alex, Are you happy with this strategy of turning off ACS based on CONFIG_PCI_P2PDMA? We only check this at enumeration-time and I don't know if there are other places we would care? thanks for pointing this out, I totally

Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches

2018-05-09 Thread Christian König
Am 09.05.2018 um 15:12 schrieb Stephen Bates: Jerome and Christian I think there is confusion here, Alex properly explained the scheme PCIE-device do a ATS request to the IOMMU which returns a valid translation for a virtual address. Device can then use that address directly without going

Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches

2018-05-10 Thread Christian König
Am 09.05.2018 um 18:45 schrieb Logan Gunthorpe: On 09/05/18 07:40 AM, Christian König wrote: The key takeaway is that when any device has ATS enabled you can't disable ACS without breaking it (even if you unplug and replug it). I don't follow how you came to this conclusion... The ACS bits

Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches

2018-05-10 Thread Christian König
Am 10.05.2018 um 16:20 schrieb Stephen Bates: Hi Jerome As it is tie to PASID this is done using IOMMU so looks for caller of amd_iommu_bind_pasid() or intel_svm_bind_mm() in GPU the existing user is the AMD GPU driver see: Ah thanks. This cleared things up for me. A quick search shows

Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches

2018-05-11 Thread Christian König
Am 10.05.2018 um 19:15 schrieb Logan Gunthorpe: On 10/05/18 11:11 AM, Stephen Bates wrote: Not to me. In the p2pdma code we specifically program DMA engines with the PCI bus address. Ah yes of course. Brain fart on my part. We are not programming the P2PDMA initiator with an IOVA but with

Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches

2018-05-08 Thread Christian König
Am 08.05.2018 um 18:27 schrieb Logan Gunthorpe: On 08/05/18 01:17 AM, Christian König wrote: AMD APUs mandatory need the ACS flag set for the GPU integrated in the CPU when IOMMU is enabled or otherwise you will break SVM. Well, given that the current set only disables ACS bits on bridges

Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches

2018-05-08 Thread Christian König
Am 08.05.2018 um 16:25 schrieb Stephen Bates: Hi Christian AMD APUs mandatory need the ACS flag set for the GPU integrated in the CPU when IOMMU is enabled or otherwise you will break SVM. OK but in this case aren't you losing (many of) the benefits of P2P since all DMAs will now get

Re: [PATCH v4 00/14] Copy Offload in NVMe Fabrics with P2P PCI Memory

2018-05-04 Thread Christian König
Am 03.05.2018 um 20:43 schrieb Logan Gunthorpe: On 03/05/18 11:29 AM, Christian König wrote: Ok, that is the point where I'm stuck. Why do we need that in one function call in the PCIe subsystem? The problem at least with GPUs is that we seriously don't have that information here, cause

Re: [PATCH v4 00/14] Copy Offload in NVMe Fabrics with P2P PCI Memory

2018-05-02 Thread Christian König
Hi Logan, it would be rather nice to have if you could separate out the functions to detect if peer2peer is possible between two devices. That would allow me to reuse the same logic for GPU peer2peer where I don't really have ZONE_DEVICE. Regards, Christian. Am 24.04.2018 um 01:30 schrieb

Re: [PATCH v4 00/14] Copy Offload in NVMe Fabrics with P2P PCI Memory

2018-05-03 Thread Christian König
Am 02.05.2018 um 17:56 schrieb Logan Gunthorpe: Hi Christian, On 5/2/2018 5:51 AM, Christian König wrote: it would be rather nice to have if you could separate out the functions to detect if peer2peer is possible between two devices. This would essentially be pci_p2pdma_distance

Re: [PATCH v4 00/14] Copy Offload in NVMe Fabrics with P2P PCI Memory

2018-05-03 Thread Christian König
Am 03.05.2018 um 17:59 schrieb Logan Gunthorpe: On 03/05/18 03:05 AM, Christian König wrote: Second question is how to you want to handle things when device are not behind the same root port (which is perfectly possible in the cases I deal with)? I think we need to implement a whitelist