RE: PCIe host controller behind IOMMU on ARM
On 13 November 2015 14:00, Arnd Bergmann wrote: > On Friday 13 November 2015 13:03:11 Phil Edworthy wrote: > > > > > > Then pci_device_add() sets the devices coherent_dma_mask to 4GiB > before > > > > calling of_pci_dma_configure(). I assume it does this on the basis that > > > > this is > a > > > > good default for PCI drivers that don't call dma_set_mask(). > > > > So if arch_setup_dma_ops() walks up the parents to limit the mask, > > > > you'll > hit > > > > this mask. > > > > > > arch_setup_dma_ops() does not walk up the hierarchy, of_dma_configure() > > > does this before calling arch_setup_dma_ops(). The PCI devices start out > > > with the 32-bit mask, but the limit should be whatever PCI host uses. > > Ok, so of_dma_configure() could walk up the tree and restrict the dma > > mask to whatever parents limit it to. Then it could be overridden by > > a dma-ranges entry in the DT node, right? > > No, the dma-ranges properties tell you what the allowed masks are, > this is what of_dma_configure() looks at. Ok, I understand now. > > If so, one problem I can see is PCI controllers already use the > > dma-ranges binding but with 3 address cells since it also specifies > > the PCI address range. > > > > I noticed that of_dma_get_range() skips straight to the parent node. > > Shouldn't it attempt to get the dma-ranges for the device's node > > first? > > No, the dma-ranges explain the capabilities of the bus, this is > what you have to look at. The device itself may have additional > restrictions, but those are what the driver knows based on the > compatibility value when it passes the device specific mask into > dma_set_mask() Ok, this is making sense now. > > I mean most hardware is limited by the peripheral's > > capabilities, not the bus. If fact, of_dma_get_range() gets the number > > of address and size cells from the device node, but gets the dma-ranges > > from the parent. That seems a little odd to me. > > of_dma_get_range() calls of_n_addr_cells()/of_n_size_cells(), which get > the #address-cells/#size-cells property from the parent device (except > for the root, which is special). Right, I should have checked what of_n_addr/size_cell() actually did. > > The only other problem I can see is that currently all PCI drivers can > > try to set their dma mask to 64 bits. At the moment that succeeds > > because there are no checks. > > Right, this is the main bug we need to fix. Yep. > > Until devices using them have their DTs > > updated with dma-ranges, we would be limiting them to a 32 bit mask. I > > guess that's not much of an issue in practice. > > Correct. I've tried to tell everyone about this when they added device > nodes for DMA capable devices. In most cases, they want 32-bit masks > anyway. Thanks for your help & patience, much appreciated! Phil -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: PCIe host controller behind IOMMU on ARM
On Friday 13 November 2015 13:03:11 Phil Edworthy wrote: > > > > Then pci_device_add() sets the devices coherent_dma_mask to 4GiB before > > > calling of_pci_dma_configure(). I assume it does this on the basis that > > > this is a > > > good default for PCI drivers that don't call dma_set_mask(). > > > So if arch_setup_dma_ops() walks up the parents to limit the mask, you'll > > > hit > > > this mask. > > > > arch_setup_dma_ops() does not walk up the hierarchy, of_dma_configure() > > does this before calling arch_setup_dma_ops(). The PCI devices start out > > with the 32-bit mask, but the limit should be whatever PCI host uses. > Ok, so of_dma_configure() could walk up the tree and restrict the dma > mask to whatever parents limit it to. Then it could be overridden by > a dma-ranges entry in the DT node, right? No, the dma-ranges properties tell you what the allowed masks are, this is what of_dma_configure() looks at. > If so, one problem I can see is PCI controllers already use the > dma-ranges binding but with 3 address cells since it also specifies > the PCI address range. > > I noticed that of_dma_get_range() skips straight to the parent node. > Shouldn't it attempt to get the dma-ranges for the device's node > first? No, the dma-ranges explain the capabilities of the bus, this is what you have to look at. The device itself may have additional restrictions, but those are what the driver knows based on the compatibility value when it passes the device specific mask into dma_set_mask() > I mean most hardware is limited by the peripheral's > capabilities, not the bus. If fact, of_dma_get_range() gets the number > of address and size cells from the device node, but gets the dma-ranges > from the parent. That seems a little odd to me. of_dma_get_range() calls of_n_addr_cells()/of_n_size_cells(), which get the #address-cells/#size-cells property from the parent device (except for the root, which is special). > The only other problem I can see is that currently all PCI drivers can > try to set their dma mask to 64 bits. At the moment that succeeds > because there are no checks. Right, this is the main bug we need to fix. > Until devices using them have their DTs > updated with dma-ranges, we would be limiting them to a 32 bit mask. I > guess that's not much of an issue in practice. Correct. I've tried to tell everyone about this when they added device nodes for DMA capable devices. In most cases, they want 32-bit masks anyway. Arnd -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
RE: PCIe host controller behind IOMMU on ARM
Hi Arnd, On 12 November 2015 16:17, Arnd Bergmann wrote: > On Thursday 12 November 2015 15:33:41 Phil Edworthy wrote: > > On 12 November 2015 09:49, Arnd Bergmann wrote: > > > On Thursday 12 November 2015 09:26:33 Phil Edworthy wrote: > > > > On 11 November 2015 18:25, LIviu wrote: > > > > > On Mon, Nov 09, 2015 at 12:32:13PM +, Phil Edworthy wrote: > > > > > > of_dma_configure calls of_dma_get_range to do all this for the PCIe host, > > > and then calls arch_setup_dma_ops() so the architecture specific code can > > > enforce the limits in dma_set_mask and pick an appropriate set of dma > > > operations. The missing part is in the implementation of > arch_setup_dma_ops, > > > which currently happily ignores the base and limit. > > I don't think it's as simple as that, though I could be wrong! > > > > First off, of_dma_configure() sets a default coherent_dma_mask to 4GiB. > > This default is set for the 'platform soc' device. For my own testing I > > increased > > this to DMA_BIT_MASK(63). Note that setting it to DMA_BIT_MASK(64) causes > > boot failure that I haven't looked into. > > Most platform devices actually need the 32-bit mask, so we intentionally > followed what PCI does here and default to that and require platform drivers > to explicitly ask for a larger mask if they need it. Ok, that makes sense. > > Then pci_device_add() sets the devices coherent_dma_mask to 4GiB before > > calling of_pci_dma_configure(). I assume it does this on the basis that > > this is a > > good default for PCI drivers that don't call dma_set_mask(). > > So if arch_setup_dma_ops() walks up the parents to limit the mask, you'll > > hit > > this mask. > > arch_setup_dma_ops() does not walk up the hierarchy, of_dma_configure() > does this before calling arch_setup_dma_ops(). The PCI devices start out > with the 32-bit mask, but the limit should be whatever PCI host uses. Ok, so of_dma_configure() could walk up the tree and restrict the dma mask to whatever parents limit it to. Then it could be overridden by a dma-ranges entry in the DT node, right? If so, one problem I can see is PCI controllers already use the dma-ranges binding but with 3 address cells since it also specifies the PCI address range. I noticed that of_dma_get_range() skips straight to the parent node. Shouldn't it attempt to get the dma-ranges for the device's node first? I mean most hardware is limited by the peripheral's capabilities, not the bus. If fact, of_dma_get_range() gets the number of address and size cells from the device node, but gets the dma-ranges from the parent. That seems a little odd to me. The only other problem I can see is that currently all PCI drivers can try to set their dma mask to 64 bits. At the moment that succeeds because there are no checks. Until devices using them have their DTs updated with dma-ranges, we would be limiting them to a 32 bit mask. I guess that's not much of an issue in practice. > > Finally, dma_set_mask_and_coherent() is called from the PCI card driver > > but it doesn't check the parents dma masks either. > > The way I think this should work is that arch_setup_dma_ops() stores the > allowed mask in the struct device, and that dma_set_mask compares the > mask against that. That makes sense. Thanks for your help, Phil -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
RE: PCIe host controller behind IOMMU on ARM
Hi Arnd, On 12 November 2015 16:17, Arnd Bergmann wrote: > On Thursday 12 November 2015 15:33:41 Phil Edworthy wrote: > > On 12 November 2015 09:49, Arnd Bergmann wrote: > > > On Thursday 12 November 2015 09:26:33 Phil Edworthy wrote: > > > > On 11 November 2015 18:25, LIviu wrote: > > > > > On Mon, Nov 09, 2015 at 12:32:13PM +, Phil Edworthy wrote: > > > > > > of_dma_configure calls of_dma_get_range to do all this for the PCIe host, > > > and then calls arch_setup_dma_ops() so the architecture specific code can > > > enforce the limits in dma_set_mask and pick an appropriate set of dma > > > operations. The missing part is in the implementation of > arch_setup_dma_ops, > > > which currently happily ignores the base and limit. > > I don't think it's as simple as that, though I could be wrong! > > > > First off, of_dma_configure() sets a default coherent_dma_mask to 4GiB. > > This default is set for the 'platform soc' device. For my own testing I > > increased > > this to DMA_BIT_MASK(63). Note that setting it to DMA_BIT_MASK(64) causes > > boot failure that I haven't looked into. > > Most platform devices actually need the 32-bit mask, so we intentionally > followed what PCI does here and default to that and require platform drivers > to explicitly ask for a larger mask if they need it. Ok, that makes sense. > > Then pci_device_add() sets the devices coherent_dma_mask to 4GiB before > > calling of_pci_dma_configure(). I assume it does this on the basis that > > this is a > > good default for PCI drivers that don't call dma_set_mask(). > > So if arch_setup_dma_ops() walks up the parents to limit the mask, you'll > > hit > > this mask. > > arch_setup_dma_ops() does not walk up the hierarchy, of_dma_configure() > does this before calling arch_setup_dma_ops(). The PCI devices start out > with the 32-bit mask, but the limit should be whatever PCI host uses. Ok, so of_dma_configure() could walk up the tree and restrict the dma mask to whatever parents limit it to. Then it could be overridden by a dma-ranges entry in the DT node, right? If so, one problem I can see is PCI controllers already use the dma-ranges binding but with 3 address cells since it also specifies the PCI address range. I noticed that of_dma_get_range() skips straight to the parent node. Shouldn't it attempt to get the dma-ranges for the device's node first? I mean most hardware is limited by the peripheral's capabilities, not the bus. If fact, of_dma_get_range() gets the number of address and size cells from the device node, but gets the dma-ranges from the parent. That seems a little odd to me. The only other problem I can see is that currently all PCI drivers can try to set their dma mask to 64 bits. At the moment that succeeds because there are no checks. Until devices using them have their DTs updated with dma-ranges, we would be limiting them to a 32 bit mask. I guess that's not much of an issue in practice. > > Finally, dma_set_mask_and_coherent() is called from the PCI card driver > > but it doesn't check the parents dma masks either. > > The way I think this should work is that arch_setup_dma_ops() stores the > allowed mask in the struct device, and that dma_set_mask compares the > mask against that. That makes sense. Thanks for your help, Phil -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: PCIe host controller behind IOMMU on ARM
On Friday 13 November 2015 13:03:11 Phil Edworthy wrote: > > > > Then pci_device_add() sets the devices coherent_dma_mask to 4GiB before > > > calling of_pci_dma_configure(). I assume it does this on the basis that > > > this is a > > > good default for PCI drivers that don't call dma_set_mask(). > > > So if arch_setup_dma_ops() walks up the parents to limit the mask, you'll > > > hit > > > this mask. > > > > arch_setup_dma_ops() does not walk up the hierarchy, of_dma_configure() > > does this before calling arch_setup_dma_ops(). The PCI devices start out > > with the 32-bit mask, but the limit should be whatever PCI host uses. > Ok, so of_dma_configure() could walk up the tree and restrict the dma > mask to whatever parents limit it to. Then it could be overridden by > a dma-ranges entry in the DT node, right? No, the dma-ranges properties tell you what the allowed masks are, this is what of_dma_configure() looks at. > If so, one problem I can see is PCI controllers already use the > dma-ranges binding but with 3 address cells since it also specifies > the PCI address range. > > I noticed that of_dma_get_range() skips straight to the parent node. > Shouldn't it attempt to get the dma-ranges for the device's node > first? No, the dma-ranges explain the capabilities of the bus, this is what you have to look at. The device itself may have additional restrictions, but those are what the driver knows based on the compatibility value when it passes the device specific mask into dma_set_mask() > I mean most hardware is limited by the peripheral's > capabilities, not the bus. If fact, of_dma_get_range() gets the number > of address and size cells from the device node, but gets the dma-ranges > from the parent. That seems a little odd to me. of_dma_get_range() calls of_n_addr_cells()/of_n_size_cells(), which get the #address-cells/#size-cells property from the parent device (except for the root, which is special). > The only other problem I can see is that currently all PCI drivers can > try to set their dma mask to 64 bits. At the moment that succeeds > because there are no checks. Right, this is the main bug we need to fix. > Until devices using them have their DTs > updated with dma-ranges, we would be limiting them to a 32 bit mask. I > guess that's not much of an issue in practice. Correct. I've tried to tell everyone about this when they added device nodes for DMA capable devices. In most cases, they want 32-bit masks anyway. Arnd -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
RE: PCIe host controller behind IOMMU on ARM
On 13 November 2015 14:00, Arnd Bergmann wrote: > On Friday 13 November 2015 13:03:11 Phil Edworthy wrote: > > > > > > Then pci_device_add() sets the devices coherent_dma_mask to 4GiB > before > > > > calling of_pci_dma_configure(). I assume it does this on the basis that > > > > this is > a > > > > good default for PCI drivers that don't call dma_set_mask(). > > > > So if arch_setup_dma_ops() walks up the parents to limit the mask, > > > > you'll > hit > > > > this mask. > > > > > > arch_setup_dma_ops() does not walk up the hierarchy, of_dma_configure() > > > does this before calling arch_setup_dma_ops(). The PCI devices start out > > > with the 32-bit mask, but the limit should be whatever PCI host uses. > > Ok, so of_dma_configure() could walk up the tree and restrict the dma > > mask to whatever parents limit it to. Then it could be overridden by > > a dma-ranges entry in the DT node, right? > > No, the dma-ranges properties tell you what the allowed masks are, > this is what of_dma_configure() looks at. Ok, I understand now. > > If so, one problem I can see is PCI controllers already use the > > dma-ranges binding but with 3 address cells since it also specifies > > the PCI address range. > > > > I noticed that of_dma_get_range() skips straight to the parent node. > > Shouldn't it attempt to get the dma-ranges for the device's node > > first? > > No, the dma-ranges explain the capabilities of the bus, this is > what you have to look at. The device itself may have additional > restrictions, but those are what the driver knows based on the > compatibility value when it passes the device specific mask into > dma_set_mask() Ok, this is making sense now. > > I mean most hardware is limited by the peripheral's > > capabilities, not the bus. If fact, of_dma_get_range() gets the number > > of address and size cells from the device node, but gets the dma-ranges > > from the parent. That seems a little odd to me. > > of_dma_get_range() calls of_n_addr_cells()/of_n_size_cells(), which get > the #address-cells/#size-cells property from the parent device (except > for the root, which is special). Right, I should have checked what of_n_addr/size_cell() actually did. > > The only other problem I can see is that currently all PCI drivers can > > try to set their dma mask to 64 bits. At the moment that succeeds > > because there are no checks. > > Right, this is the main bug we need to fix. Yep. > > Until devices using them have their DTs > > updated with dma-ranges, we would be limiting them to a 32 bit mask. I > > guess that's not much of an issue in practice. > > Correct. I've tried to tell everyone about this when they added device > nodes for DMA capable devices. In most cases, they want 32-bit masks > anyway. Thanks for your help & patience, much appreciated! Phil -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: PCIe host controller behind IOMMU on ARM
On Thursday 12 November 2015 15:33:41 Phil Edworthy wrote: > On 12 November 2015 09:49, Arnd Bergmann wrote: > > On Thursday 12 November 2015 09:26:33 Phil Edworthy wrote: > > > On 11 November 2015 18:25, LIviu wrote: > > > > On Mon, Nov 09, 2015 at 12:32:13PM +, Phil Edworthy wrote: > > > > of_dma_configure calls of_dma_get_range to do all this for the PCIe host, > > and then calls arch_setup_dma_ops() so the architecture specific code can > > enforce the limits in dma_set_mask and pick an appropriate set of dma > > operations. The missing part is in the implementation of arch_setup_dma_ops, > > which currently happily ignores the base and limit. > I don't think it's as simple as that, though I could be wrong! > > First off, of_dma_configure() sets a default coherent_dma_mask to 4GiB. > This default is set for the 'platform soc' device. For my own testing I > increased > this to DMA_BIT_MASK(63). Note that setting it to DMA_BIT_MASK(64) causes > boot failure that I haven't looked into. Most platform devices actually need the 32-bit mask, so we intentionally followed what PCI does here and default to that and require platform drivers to explicitly ask for a larger mask if they need it. > Then pci_device_add() sets the devices coherent_dma_mask to 4GiB before > calling of_pci_dma_configure(). I assume it does this on the basis that this > is a > good default for PCI drivers that don't call dma_set_mask(). > So if arch_setup_dma_ops() walks up the parents to limit the mask, you'll hit > this mask. arch_setup_dma_ops() does not walk up the hierarchy, of_dma_configure() does this before calling arch_setup_dma_ops(). The PCI devices start out with the 32-bit mask, but the limit should be whatever PCI host uses. > Finally, dma_set_mask_and_coherent() is called from the PCI card driver > but it doesn't check the parents dma masks either. The way I think this should work is that arch_setup_dma_ops() stores the allowed mask in the struct device, and that dma_set_mask compares the mask against that. Arnd -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
RE: PCIe host controller behind IOMMU on ARM
Hi Arnd, On 12 November 2015 09:49, Arnd Bergmann wrote: > On Thursday 12 November 2015 09:26:33 Phil Edworthy wrote: > > On 11 November 2015 18:25, LIviu wrote: > > > On Mon, Nov 09, 2015 at 12:32:13PM +, Phil Edworthy wrote: > > > > I think you're mixing things a bit or not explaining them very well. > > > Having the > > > PCIe controller limited to 32-bit AXI does not mean that the PCIe bus > > > cannot > > > carry 64-bit addresses. It depends on how they get translated by the host > bridge > > > or its associated ATS block. I can't see why you can't have a setup where > > > the CPU addresses are 32-bit but the PCIe bus addresses are all 64-bit. > > > You just have to be careful on how you setup your mem64 ranges so that > they > > > don't > > > overlap with the 32-bit ranges when translated. > > From a HW point of view I agree that we can setup the PCI host bridge such > > that > > it uses 64-bit PCI address, with 32-bit cpu addresses. Though in practice > > doesn't > > this mean that the dma ops used by card drivers has to be provided by our > > PCI > > host bridge driver so we can apply the translation to those PCI addresses? > > This comes back to my point below about how to do this. Adding a bus > > notifier > > to do this may be too late, and arm64 doesn't implement set_dma_ops(). > > > > > And no, you should not limit at the card driver the DMA_BIT_MASK() unless > the > > > card is not capable of supporting more than 32-bit addresses. > > If there was infrastructure that checked all parents dma-ranges when the > > dma_set_mask() function is called as Arnd pointed out, this would nicely > > solve > > the problem. > > of_dma_configure calls of_dma_get_range to do all this for the PCIe host, > and then calls arch_setup_dma_ops() so the architecture specific code can > enforce the limits in dma_set_mask and pick an appropriate set of dma > operations. The missing part is in the implementation of arch_setup_dma_ops, > which currently happily ignores the base and limit. I don't think it's as simple as that, though I could be wrong! First off, of_dma_configure() sets a default coherent_dma_mask to 4GiB. This default is set for the 'platform soc' device. For my own testing I increased this to DMA_BIT_MASK(63). Note that setting it to DMA_BIT_MASK(64) causes boot failure that I haven't looked into. Then pci_device_add() sets the devices coherent_dma_mask to 4GiB before calling of_pci_dma_configure(). I assume it does this on the basis that this is a good default for PCI drivers that don't call dma_set_mask(). So if arch_setup_dma_ops() walks up the parents to limit the mask, you'll hit this mask. Finally, dma_set_mask_and_coherent() is called from the PCI card driver but it doesn't check the parents dma masks either. Thanks Phil -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: PCIe host controller behind IOMMU on ARM
On Thu, Nov 12, 2015 at 09:26:33AM +, Phil Edworthy wrote: > Hi Liviu, Arnd, > > On 11 November 2015 18:25, LIviu wrote: > > On Mon, Nov 09, 2015 at 12:32:13PM +, Phil Edworthy wrote: > > > Hi Liviu, Will, > > > > > > On 04 November 2015 15:19, Phil wrote: > > > > On 04 November 2015 15:02, Liviu wrote: > > > > > On Wed, Nov 04, 2015 at 02:48:38PM +, Phil Edworthy wrote: > > > > > > Hi Liviu, > > > > > > > > > > > > On 04 November 2015 14:24, Liviu wrote: > > > > > > > On Wed, Nov 04, 2015 at 01:57:48PM +, Phil Edworthy wrote: > > > > > > > > Hi, > > > > > > > > > > > > > > > > I am trying to hook up a PCIe host controller that sits behind > > > > > > > > an > > IOMMU, > > > > > > > > but having some problems. > > > > > > > > > > > > > > > > I'm using the pcie-rcar PCIe host controller and it works fine > > > > > > > > without > > > > > > > > the IOMMU, and I can attach the IOMMU to the controller such > > > > > > > > that > > any > > > > > calls > > > > > > > > to dma_alloc_coherent made by the controller driver uses the > > > > iommu_ops > > > > > > > > version of dma_ops. > > > > > > > > > > > > > > > > However, I can't see how to make the endpoints to utilise the > > dma_ops > > > > that > > > > > > > > the controller uses. Shouldn't the endpoints inherit the > > > > > > > > dma_ops from > > the > > > > > > > > controller? > > > > > > > > > > > > > > No, not directly. > > > > > > > > > > > > > > > Any pointers for this? > > > > > > > > > > > > > > You need to understand the process through which a driver for > > endpoint > > > > get > > > > > > > an address to be passed down to the device. Have a look at > > > > > > > Documentation/DMA-API-HOWTO.txt, there is a nice explanation > > > > > > > there. > > > > > > > (Hint: EP driver needs to call dma_map_single). > > > > > > > > > > > > > > Also, you need to make sure that the bus address that ends up > > > > > > > being set > > > > into > > > > > > > the endpoint gets translated correctly by the host controller > > > > > > > into an > > address > > > > > > > that the IOMMU can then translate into physical address. > > > > > > Sure, though since this is bog standard Intel PCIe ethernet card > > > > > > which > > works > > > > > > fine when the IOMMU is effectively unused, I don’t think there is a > > problem > > > > > > with that. > > > > > > > > > > > > The driver for the PCIe controller sets up the IOMMU mapping ok > > > > > > when I > > > > > > do a test call to dma_alloc_coherent() in the controller's driver. > > > > > > i.e. when I > > > > > > do this, it ends up in arm_iommu_alloc_attrs(), which calls > > > > > > __iommu_alloc_buffer() and __alloc_iova(). > > > > > > > > > > > > When an endpoint driver allocates and maps a dma coherent buffer it > > > > > > also needs to end up in arm_iommu_alloc_attrs(), but it doesn't. > > > > > > > > > > Why do you think that? Remember that the only thing attached to the > > IOMMU > > > > is > > > > > the > > > > > host controller. The endpoint is on the PCIe bus, which gets a > > > > > different > > > > > translation > > > > > that the IOMMU knows nothing about. If it helps you to visualise it > > > > > better, > > think > > > > > of the host controller as another IOMMU device. It's the ops of the > > > > > host > > > > > controller > > > > > that should be invoked, not the IOMMU's. > > > > Ok, that makes sense. I'll have a think and poke it a bit more... > > > > Hi Phil, > > > > Not trying to ignore your email, but I thought this is more in Will's > > backyard. > > > > > Somewhat related to this, since our PCIe controller HW is limited to > > > 32-bit AXI address range, before trying to hook up the IOMMU I have > > > tried to limit the dma_mask for PCI cards to DMA_BIT_MASK(32). The > > > reason being that Linux uses a 1 to 1 mapping between PCI addresses > > > and cpu (phys) addresses when there isn't an IOMMU involved, so I > > > think that we need to limit the PCI address space used. > > > > I think you're mixing things a bit or not explaining them very well. Having > > the > > PCIe controller limited to 32-bit AXI does not mean that the PCIe bus cannot > > carry 64-bit addresses. It depends on how they get translated by the host > > bridge > > or its associated ATS block. I can't see why you can't have a setup where > > the CPU addresses are 32-bit but the PCIe bus addresses are all 64-bit. > > You just have to be careful on how you setup your mem64 ranges so that they > > don't > > overlap with the 32-bit ranges when translated. > From a HW point of view I agree that we can setup the PCI host bridge such > that > it uses 64-bit PCI address, with 32-bit cpu addresses. Though in practice > doesn't > this mean that the dma ops used by card drivers has to be provided by our PCI > host bridge driver so we can apply the translation to those PCI addresses? I thought all addresses that are set into the cards go through pcibios_resource_to_bus() which give you the PCI address
Re: PCIe host controller behind IOMMU on ARM
On Thursday 12 November 2015 09:26:33 Phil Edworthy wrote: > On 11 November 2015 18:25, LIviu wrote: > > On Mon, Nov 09, 2015 at 12:32:13PM +, Phil Edworthy wrote: > > I think you're mixing things a bit or not explaining them very well. Having > > the > > PCIe controller limited to 32-bit AXI does not mean that the PCIe bus cannot > > carry 64-bit addresses. It depends on how they get translated by the host > > bridge > > or its associated ATS block. I can't see why you can't have a setup where > > the CPU addresses are 32-bit but the PCIe bus addresses are all 64-bit. > > You just have to be careful on how you setup your mem64 ranges so that they > > don't > > overlap with the 32-bit ranges when translated. > From a HW point of view I agree that we can setup the PCI host bridge such > that > it uses 64-bit PCI address, with 32-bit cpu addresses. Though in practice > doesn't > this mean that the dma ops used by card drivers has to be provided by our PCI > host bridge driver so we can apply the translation to those PCI addresses? > This comes back to my point below about how to do this. Adding a bus notifier > to do this may be too late, and arm64 doesn't implement set_dma_ops(). > > > And no, you should not limit at the card driver the DMA_BIT_MASK() unless > > the > > card is not capable of supporting more than 32-bit addresses. > If there was infrastructure that checked all parents dma-ranges when the > dma_set_mask() function is called as Arnd pointed out, this would nicely solve > the problem. of_dma_configure calls of_dma_get_range to do all this for the PCIe host, and then calls arch_setup_dma_ops() so the architecture specific code can enforce the limits in dma_set_mask and pick an appropriate set of dma operations. The missing part is in the implementation of arch_setup_dma_ops, which currently happily ignores the base and limit. Arnd -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
RE: PCIe host controller behind IOMMU on ARM
Hi Liviu, Arnd, On 11 November 2015 18:25, LIviu wrote: > On Mon, Nov 09, 2015 at 12:32:13PM +, Phil Edworthy wrote: > > Hi Liviu, Will, > > > > On 04 November 2015 15:19, Phil wrote: > > > On 04 November 2015 15:02, Liviu wrote: > > > > On Wed, Nov 04, 2015 at 02:48:38PM +, Phil Edworthy wrote: > > > > > Hi Liviu, > > > > > > > > > > On 04 November 2015 14:24, Liviu wrote: > > > > > > On Wed, Nov 04, 2015 at 01:57:48PM +, Phil Edworthy wrote: > > > > > > > Hi, > > > > > > > > > > > > > > I am trying to hook up a PCIe host controller that sits behind an > IOMMU, > > > > > > > but having some problems. > > > > > > > > > > > > > > I'm using the pcie-rcar PCIe host controller and it works fine > > > > > > > without > > > > > > > the IOMMU, and I can attach the IOMMU to the controller such that > any > > > > calls > > > > > > > to dma_alloc_coherent made by the controller driver uses the > > > iommu_ops > > > > > > > version of dma_ops. > > > > > > > > > > > > > > However, I can't see how to make the endpoints to utilise the > dma_ops > > > that > > > > > > > the controller uses. Shouldn't the endpoints inherit the dma_ops > > > > > > > from > the > > > > > > > controller? > > > > > > > > > > > > No, not directly. > > > > > > > > > > > > > Any pointers for this? > > > > > > > > > > > > You need to understand the process through which a driver for > endpoint > > > get > > > > > > an address to be passed down to the device. Have a look at > > > > > > Documentation/DMA-API-HOWTO.txt, there is a nice explanation there. > > > > > > (Hint: EP driver needs to call dma_map_single). > > > > > > > > > > > > Also, you need to make sure that the bus address that ends up being > > > > > > set > > > into > > > > > > the endpoint gets translated correctly by the host controller into > > > > > > an > address > > > > > > that the IOMMU can then translate into physical address. > > > > > Sure, though since this is bog standard Intel PCIe ethernet card which > works > > > > > fine when the IOMMU is effectively unused, I don’t think there is a > problem > > > > > with that. > > > > > > > > > > The driver for the PCIe controller sets up the IOMMU mapping ok when I > > > > > do a test call to dma_alloc_coherent() in the controller's driver. > > > > > i.e. when I > > > > > do this, it ends up in arm_iommu_alloc_attrs(), which calls > > > > > __iommu_alloc_buffer() and __alloc_iova(). > > > > > > > > > > When an endpoint driver allocates and maps a dma coherent buffer it > > > > > also needs to end up in arm_iommu_alloc_attrs(), but it doesn't. > > > > > > > > Why do you think that? Remember that the only thing attached to the > IOMMU > > > is > > > > the > > > > host controller. The endpoint is on the PCIe bus, which gets a different > > > > translation > > > > that the IOMMU knows nothing about. If it helps you to visualise it > > > > better, > think > > > > of the host controller as another IOMMU device. It's the ops of the host > > > > controller > > > > that should be invoked, not the IOMMU's. > > > Ok, that makes sense. I'll have a think and poke it a bit more... > > Hi Phil, > > Not trying to ignore your email, but I thought this is more in Will's > backyard. > > > Somewhat related to this, since our PCIe controller HW is limited to > > 32-bit AXI address range, before trying to hook up the IOMMU I have > > tried to limit the dma_mask for PCI cards to DMA_BIT_MASK(32). The > > reason being that Linux uses a 1 to 1 mapping between PCI addresses > > and cpu (phys) addresses when there isn't an IOMMU involved, so I > > think that we need to limit the PCI address space used. > > I think you're mixing things a bit or not explaining them very well. Having > the > PCIe controller limited to 32-bit AXI does not mean that the PCIe bus cannot > carry 64-bit addresses. It depends on how they get translated by the host > bridge > or its associated ATS block. I can't see why you can't have a setup where > the CPU addresses are 32-bit but the PCIe bus addresses are all 64-bit. > You just have to be careful on how you setup your mem64 ranges so that they > don't > overlap with the 32-bit ranges when translated. From a HW point of view I agree that we can setup the PCI host bridge such that it uses 64-bit PCI address, with 32-bit cpu addresses. Though in practice doesn't this mean that the dma ops used by card drivers has to be provided by our PCI host bridge driver so we can apply the translation to those PCI addresses? This comes back to my point below about how to do this. Adding a bus notifier to do this may be too late, and arm64 doesn't implement set_dma_ops(). > And no, you should not limit at the card driver the DMA_BIT_MASK() unless the > card is not capable of supporting more than 32-bit addresses. If there was infrastructure that checked all parents dma-ranges when the dma_set_mask() function is called as Arnd pointed out, this would nicely solve the problem. > > Since pci_setup_device() sets up
Re: PCIe host controller behind IOMMU on ARM
On Thursday 12 November 2015 09:26:33 Phil Edworthy wrote: > On 11 November 2015 18:25, LIviu wrote: > > On Mon, Nov 09, 2015 at 12:32:13PM +, Phil Edworthy wrote: > > I think you're mixing things a bit or not explaining them very well. Having > > the > > PCIe controller limited to 32-bit AXI does not mean that the PCIe bus cannot > > carry 64-bit addresses. It depends on how they get translated by the host > > bridge > > or its associated ATS block. I can't see why you can't have a setup where > > the CPU addresses are 32-bit but the PCIe bus addresses are all 64-bit. > > You just have to be careful on how you setup your mem64 ranges so that they > > don't > > overlap with the 32-bit ranges when translated. > From a HW point of view I agree that we can setup the PCI host bridge such > that > it uses 64-bit PCI address, with 32-bit cpu addresses. Though in practice > doesn't > this mean that the dma ops used by card drivers has to be provided by our PCI > host bridge driver so we can apply the translation to those PCI addresses? > This comes back to my point below about how to do this. Adding a bus notifier > to do this may be too late, and arm64 doesn't implement set_dma_ops(). > > > And no, you should not limit at the card driver the DMA_BIT_MASK() unless > > the > > card is not capable of supporting more than 32-bit addresses. > If there was infrastructure that checked all parents dma-ranges when the > dma_set_mask() function is called as Arnd pointed out, this would nicely solve > the problem. of_dma_configure calls of_dma_get_range to do all this for the PCIe host, and then calls arch_setup_dma_ops() so the architecture specific code can enforce the limits in dma_set_mask and pick an appropriate set of dma operations. The missing part is in the implementation of arch_setup_dma_ops, which currently happily ignores the base and limit. Arnd -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
RE: PCIe host controller behind IOMMU on ARM
Hi Liviu, Arnd, On 11 November 2015 18:25, LIviu wrote: > On Mon, Nov 09, 2015 at 12:32:13PM +, Phil Edworthy wrote: > > Hi Liviu, Will, > > > > On 04 November 2015 15:19, Phil wrote: > > > On 04 November 2015 15:02, Liviu wrote: > > > > On Wed, Nov 04, 2015 at 02:48:38PM +, Phil Edworthy wrote: > > > > > Hi Liviu, > > > > > > > > > > On 04 November 2015 14:24, Liviu wrote: > > > > > > On Wed, Nov 04, 2015 at 01:57:48PM +, Phil Edworthy wrote: > > > > > > > Hi, > > > > > > > > > > > > > > I am trying to hook up a PCIe host controller that sits behind an > IOMMU, > > > > > > > but having some problems. > > > > > > > > > > > > > > I'm using the pcie-rcar PCIe host controller and it works fine > > > > > > > without > > > > > > > the IOMMU, and I can attach the IOMMU to the controller such that > any > > > > calls > > > > > > > to dma_alloc_coherent made by the controller driver uses the > > > iommu_ops > > > > > > > version of dma_ops. > > > > > > > > > > > > > > However, I can't see how to make the endpoints to utilise the > dma_ops > > > that > > > > > > > the controller uses. Shouldn't the endpoints inherit the dma_ops > > > > > > > from > the > > > > > > > controller? > > > > > > > > > > > > No, not directly. > > > > > > > > > > > > > Any pointers for this? > > > > > > > > > > > > You need to understand the process through which a driver for > endpoint > > > get > > > > > > an address to be passed down to the device. Have a look at > > > > > > Documentation/DMA-API-HOWTO.txt, there is a nice explanation there. > > > > > > (Hint: EP driver needs to call dma_map_single). > > > > > > > > > > > > Also, you need to make sure that the bus address that ends up being > > > > > > set > > > into > > > > > > the endpoint gets translated correctly by the host controller into > > > > > > an > address > > > > > > that the IOMMU can then translate into physical address. > > > > > Sure, though since this is bog standard Intel PCIe ethernet card which > works > > > > > fine when the IOMMU is effectively unused, I don’t think there is a > problem > > > > > with that. > > > > > > > > > > The driver for the PCIe controller sets up the IOMMU mapping ok when I > > > > > do a test call to dma_alloc_coherent() in the controller's driver. > > > > > i.e. when I > > > > > do this, it ends up in arm_iommu_alloc_attrs(), which calls > > > > > __iommu_alloc_buffer() and __alloc_iova(). > > > > > > > > > > When an endpoint driver allocates and maps a dma coherent buffer it > > > > > also needs to end up in arm_iommu_alloc_attrs(), but it doesn't. > > > > > > > > Why do you think that? Remember that the only thing attached to the > IOMMU > > > is > > > > the > > > > host controller. The endpoint is on the PCIe bus, which gets a different > > > > translation > > > > that the IOMMU knows nothing about. If it helps you to visualise it > > > > better, > think > > > > of the host controller as another IOMMU device. It's the ops of the host > > > > controller > > > > that should be invoked, not the IOMMU's. > > > Ok, that makes sense. I'll have a think and poke it a bit more... > > Hi Phil, > > Not trying to ignore your email, but I thought this is more in Will's > backyard. > > > Somewhat related to this, since our PCIe controller HW is limited to > > 32-bit AXI address range, before trying to hook up the IOMMU I have > > tried to limit the dma_mask for PCI cards to DMA_BIT_MASK(32). The > > reason being that Linux uses a 1 to 1 mapping between PCI addresses > > and cpu (phys) addresses when there isn't an IOMMU involved, so I > > think that we need to limit the PCI address space used. > > I think you're mixing things a bit or not explaining them very well. Having > the > PCIe controller limited to 32-bit AXI does not mean that the PCIe bus cannot > carry 64-bit addresses. It depends on how they get translated by the host > bridge > or its associated ATS block. I can't see why you can't have a setup where > the CPU addresses are 32-bit but the PCIe bus addresses are all 64-bit. > You just have to be careful on how you setup your mem64 ranges so that they > don't > overlap with the 32-bit ranges when translated. From a HW point of view I agree that we can setup the PCI host bridge such that it uses 64-bit PCI address, with 32-bit cpu addresses. Though in practice doesn't this mean that the dma ops used by card drivers has to be provided by our PCI host bridge driver so we can apply the translation to those PCI addresses? This comes back to my point below about how to do this. Adding a bus notifier to do this may be too late, and arm64 doesn't implement set_dma_ops(). > And no, you should not limit at the card driver the DMA_BIT_MASK() unless the > card is not capable of supporting more than 32-bit addresses. If there was infrastructure that checked all parents dma-ranges when the dma_set_mask() function is called as Arnd pointed out, this would nicely solve the problem. > > Since pci_setup_device() sets up
Re: PCIe host controller behind IOMMU on ARM
On Thu, Nov 12, 2015 at 09:26:33AM +, Phil Edworthy wrote: > Hi Liviu, Arnd, > > On 11 November 2015 18:25, LIviu wrote: > > On Mon, Nov 09, 2015 at 12:32:13PM +, Phil Edworthy wrote: > > > Hi Liviu, Will, > > > > > > On 04 November 2015 15:19, Phil wrote: > > > > On 04 November 2015 15:02, Liviu wrote: > > > > > On Wed, Nov 04, 2015 at 02:48:38PM +, Phil Edworthy wrote: > > > > > > Hi Liviu, > > > > > > > > > > > > On 04 November 2015 14:24, Liviu wrote: > > > > > > > On Wed, Nov 04, 2015 at 01:57:48PM +, Phil Edworthy wrote: > > > > > > > > Hi, > > > > > > > > > > > > > > > > I am trying to hook up a PCIe host controller that sits behind > > > > > > > > an > > IOMMU, > > > > > > > > but having some problems. > > > > > > > > > > > > > > > > I'm using the pcie-rcar PCIe host controller and it works fine > > > > > > > > without > > > > > > > > the IOMMU, and I can attach the IOMMU to the controller such > > > > > > > > that > > any > > > > > calls > > > > > > > > to dma_alloc_coherent made by the controller driver uses the > > > > iommu_ops > > > > > > > > version of dma_ops. > > > > > > > > > > > > > > > > However, I can't see how to make the endpoints to utilise the > > dma_ops > > > > that > > > > > > > > the controller uses. Shouldn't the endpoints inherit the > > > > > > > > dma_ops from > > the > > > > > > > > controller? > > > > > > > > > > > > > > No, not directly. > > > > > > > > > > > > > > > Any pointers for this? > > > > > > > > > > > > > > You need to understand the process through which a driver for > > endpoint > > > > get > > > > > > > an address to be passed down to the device. Have a look at > > > > > > > Documentation/DMA-API-HOWTO.txt, there is a nice explanation > > > > > > > there. > > > > > > > (Hint: EP driver needs to call dma_map_single). > > > > > > > > > > > > > > Also, you need to make sure that the bus address that ends up > > > > > > > being set > > > > into > > > > > > > the endpoint gets translated correctly by the host controller > > > > > > > into an > > address > > > > > > > that the IOMMU can then translate into physical address. > > > > > > Sure, though since this is bog standard Intel PCIe ethernet card > > > > > > which > > works > > > > > > fine when the IOMMU is effectively unused, I don’t think there is a > > problem > > > > > > with that. > > > > > > > > > > > > The driver for the PCIe controller sets up the IOMMU mapping ok > > > > > > when I > > > > > > do a test call to dma_alloc_coherent() in the controller's driver. > > > > > > i.e. when I > > > > > > do this, it ends up in arm_iommu_alloc_attrs(), which calls > > > > > > __iommu_alloc_buffer() and __alloc_iova(). > > > > > > > > > > > > When an endpoint driver allocates and maps a dma coherent buffer it > > > > > > also needs to end up in arm_iommu_alloc_attrs(), but it doesn't. > > > > > > > > > > Why do you think that? Remember that the only thing attached to the > > IOMMU > > > > is > > > > > the > > > > > host controller. The endpoint is on the PCIe bus, which gets a > > > > > different > > > > > translation > > > > > that the IOMMU knows nothing about. If it helps you to visualise it > > > > > better, > > think > > > > > of the host controller as another IOMMU device. It's the ops of the > > > > > host > > > > > controller > > > > > that should be invoked, not the IOMMU's. > > > > Ok, that makes sense. I'll have a think and poke it a bit more... > > > > Hi Phil, > > > > Not trying to ignore your email, but I thought this is more in Will's > > backyard. > > > > > Somewhat related to this, since our PCIe controller HW is limited to > > > 32-bit AXI address range, before trying to hook up the IOMMU I have > > > tried to limit the dma_mask for PCI cards to DMA_BIT_MASK(32). The > > > reason being that Linux uses a 1 to 1 mapping between PCI addresses > > > and cpu (phys) addresses when there isn't an IOMMU involved, so I > > > think that we need to limit the PCI address space used. > > > > I think you're mixing things a bit or not explaining them very well. Having > > the > > PCIe controller limited to 32-bit AXI does not mean that the PCIe bus cannot > > carry 64-bit addresses. It depends on how they get translated by the host > > bridge > > or its associated ATS block. I can't see why you can't have a setup where > > the CPU addresses are 32-bit but the PCIe bus addresses are all 64-bit. > > You just have to be careful on how you setup your mem64 ranges so that they > > don't > > overlap with the 32-bit ranges when translated. > From a HW point of view I agree that we can setup the PCI host bridge such > that > it uses 64-bit PCI address, with 32-bit cpu addresses. Though in practice > doesn't > this mean that the dma ops used by card drivers has to be provided by our PCI > host bridge driver so we can apply the translation to those PCI addresses? I thought all addresses that are set into the cards go through pcibios_resource_to_bus() which give you the PCI address
RE: PCIe host controller behind IOMMU on ARM
Hi Arnd, On 12 November 2015 09:49, Arnd Bergmann wrote: > On Thursday 12 November 2015 09:26:33 Phil Edworthy wrote: > > On 11 November 2015 18:25, LIviu wrote: > > > On Mon, Nov 09, 2015 at 12:32:13PM +, Phil Edworthy wrote: > > > > I think you're mixing things a bit or not explaining them very well. > > > Having the > > > PCIe controller limited to 32-bit AXI does not mean that the PCIe bus > > > cannot > > > carry 64-bit addresses. It depends on how they get translated by the host > bridge > > > or its associated ATS block. I can't see why you can't have a setup where > > > the CPU addresses are 32-bit but the PCIe bus addresses are all 64-bit. > > > You just have to be careful on how you setup your mem64 ranges so that > they > > > don't > > > overlap with the 32-bit ranges when translated. > > From a HW point of view I agree that we can setup the PCI host bridge such > > that > > it uses 64-bit PCI address, with 32-bit cpu addresses. Though in practice > > doesn't > > this mean that the dma ops used by card drivers has to be provided by our > > PCI > > host bridge driver so we can apply the translation to those PCI addresses? > > This comes back to my point below about how to do this. Adding a bus > > notifier > > to do this may be too late, and arm64 doesn't implement set_dma_ops(). > > > > > And no, you should not limit at the card driver the DMA_BIT_MASK() unless > the > > > card is not capable of supporting more than 32-bit addresses. > > If there was infrastructure that checked all parents dma-ranges when the > > dma_set_mask() function is called as Arnd pointed out, this would nicely > > solve > > the problem. > > of_dma_configure calls of_dma_get_range to do all this for the PCIe host, > and then calls arch_setup_dma_ops() so the architecture specific code can > enforce the limits in dma_set_mask and pick an appropriate set of dma > operations. The missing part is in the implementation of arch_setup_dma_ops, > which currently happily ignores the base and limit. I don't think it's as simple as that, though I could be wrong! First off, of_dma_configure() sets a default coherent_dma_mask to 4GiB. This default is set for the 'platform soc' device. For my own testing I increased this to DMA_BIT_MASK(63). Note that setting it to DMA_BIT_MASK(64) causes boot failure that I haven't looked into. Then pci_device_add() sets the devices coherent_dma_mask to 4GiB before calling of_pci_dma_configure(). I assume it does this on the basis that this is a good default for PCI drivers that don't call dma_set_mask(). So if arch_setup_dma_ops() walks up the parents to limit the mask, you'll hit this mask. Finally, dma_set_mask_and_coherent() is called from the PCI card driver but it doesn't check the parents dma masks either. Thanks Phil -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: PCIe host controller behind IOMMU on ARM
On Thursday 12 November 2015 15:33:41 Phil Edworthy wrote: > On 12 November 2015 09:49, Arnd Bergmann wrote: > > On Thursday 12 November 2015 09:26:33 Phil Edworthy wrote: > > > On 11 November 2015 18:25, LIviu wrote: > > > > On Mon, Nov 09, 2015 at 12:32:13PM +, Phil Edworthy wrote: > > > > of_dma_configure calls of_dma_get_range to do all this for the PCIe host, > > and then calls arch_setup_dma_ops() so the architecture specific code can > > enforce the limits in dma_set_mask and pick an appropriate set of dma > > operations. The missing part is in the implementation of arch_setup_dma_ops, > > which currently happily ignores the base and limit. > I don't think it's as simple as that, though I could be wrong! > > First off, of_dma_configure() sets a default coherent_dma_mask to 4GiB. > This default is set for the 'platform soc' device. For my own testing I > increased > this to DMA_BIT_MASK(63). Note that setting it to DMA_BIT_MASK(64) causes > boot failure that I haven't looked into. Most platform devices actually need the 32-bit mask, so we intentionally followed what PCI does here and default to that and require platform drivers to explicitly ask for a larger mask if they need it. > Then pci_device_add() sets the devices coherent_dma_mask to 4GiB before > calling of_pci_dma_configure(). I assume it does this on the basis that this > is a > good default for PCI drivers that don't call dma_set_mask(). > So if arch_setup_dma_ops() walks up the parents to limit the mask, you'll hit > this mask. arch_setup_dma_ops() does not walk up the hierarchy, of_dma_configure() does this before calling arch_setup_dma_ops(). The PCI devices start out with the 32-bit mask, but the limit should be whatever PCI host uses. > Finally, dma_set_mask_and_coherent() is called from the PCI card driver > but it doesn't check the parents dma masks either. The way I think this should work is that arch_setup_dma_ops() stores the allowed mask in the struct device, and that dma_set_mask compares the mask against that. Arnd -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: PCIe host controller behind IOMMU on ARM
On Wednesday 11 November 2015 18:24:56 liviu.du...@arm.com wrote: > > > Somewhat related to this, since our PCIe controller HW is limited to > > 32-bit AXI address range, before trying to hook up the IOMMU I have > > tried to limit the dma_mask for PCI cards to DMA_BIT_MASK(32). The > > reason being that Linux uses a 1 to 1 mapping between PCI addresses > > and cpu (phys) addresses when there isn't an IOMMU involved, so I > > think that we need to limit the PCI address space used. > > I think you're mixing things a bit or not explaining them very well. Having > the > PCIe controller limited to 32-bit AXI does not mean that the PCIe bus cannot > carry 64-bit addresses. It depends on how they get translated by the host > bridge > or its associated ATS block. I can't see why you can't have a setup where > the CPU addresses are 32-bit but the PCIe bus addresses are all 64-bit. > You just have to be careful on how you setup your mem64 ranges so that they > don't > overlap with the 32-bit ranges when translated. > > And no, you should not limit at the card driver the DMA_BIT_MASK() unless the > card is not capable of supporting more than 32-bit addresses. I think we are missing one crucial bit of infrastructure on ARM64 at the moment: the dma_set_mask() function should fail if a driver asks for a mask that is larger than the dma-ranges property of the parent device (or any device higher up in the hierarchy) allows. Drivers that want a larger mask should try that first, and then fall back to a 32-bit mask, which is guaranteed to work. Arnd -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: PCIe host controller behind IOMMU on ARM
On Mon, Nov 09, 2015 at 12:32:13PM +, Phil Edworthy wrote: > Hi Liviu, Will, > > On 04 November 2015 15:19, Phil wrote: > > On 04 November 2015 15:02, Liviu wrote: > > > On Wed, Nov 04, 2015 at 02:48:38PM +, Phil Edworthy wrote: > > > > Hi Liviu, > > > > > > > > On 04 November 2015 14:24, Liviu wrote: > > > > > On Wed, Nov 04, 2015 at 01:57:48PM +, Phil Edworthy wrote: > > > > > > Hi, > > > > > > > > > > > > I am trying to hook up a PCIe host controller that sits behind an > > > > > > IOMMU, > > > > > > but having some problems. > > > > > > > > > > > > I'm using the pcie-rcar PCIe host controller and it works fine > > > > > > without > > > > > > the IOMMU, and I can attach the IOMMU to the controller such that > > > > > > any > > > calls > > > > > > to dma_alloc_coherent made by the controller driver uses the > > iommu_ops > > > > > > version of dma_ops. > > > > > > > > > > > > However, I can't see how to make the endpoints to utilise the > > > > > > dma_ops > > that > > > > > > the controller uses. Shouldn't the endpoints inherit the dma_ops > > > > > > from the > > > > > > controller? > > > > > > > > > > No, not directly. > > > > > > > > > > > Any pointers for this? > > > > > > > > > > You need to understand the process through which a driver for endpoint > > get > > > > > an address to be passed down to the device. Have a look at > > > > > Documentation/DMA-API-HOWTO.txt, there is a nice explanation there. > > > > > (Hint: EP driver needs to call dma_map_single). > > > > > > > > > > Also, you need to make sure that the bus address that ends up being > > > > > set > > into > > > > > the endpoint gets translated correctly by the host controller into an > > > > > address > > > > > that the IOMMU can then translate into physical address. > > > > Sure, though since this is bog standard Intel PCIe ethernet card which > > > > works > > > > fine when the IOMMU is effectively unused, I don’t think there is a > > > > problem > > > > with that. > > > > > > > > The driver for the PCIe controller sets up the IOMMU mapping ok when I > > > > do a test call to dma_alloc_coherent() in the controller's driver. i.e. > > > > when I > > > > do this, it ends up in arm_iommu_alloc_attrs(), which calls > > > > __iommu_alloc_buffer() and __alloc_iova(). > > > > > > > > When an endpoint driver allocates and maps a dma coherent buffer it > > > > also needs to end up in arm_iommu_alloc_attrs(), but it doesn't. > > > > > > Why do you think that? Remember that the only thing attached to the IOMMU > > is > > > the > > > host controller. The endpoint is on the PCIe bus, which gets a different > > > translation > > > that the IOMMU knows nothing about. If it helps you to visualise it > > > better, think > > > of the host controller as another IOMMU device. It's the ops of the host > > > controller > > > that should be invoked, not the IOMMU's. > > Ok, that makes sense. I'll have a think and poke it a bit more... Hi Phil, Not trying to ignore your email, but I thought this is more in Will's backyard. > Somewhat related to this, since our PCIe controller HW is limited to > 32-bit AXI address range, before trying to hook up the IOMMU I have > tried to limit the dma_mask for PCI cards to DMA_BIT_MASK(32). The > reason being that Linux uses a 1 to 1 mapping between PCI addresses > and cpu (phys) addresses when there isn't an IOMMU involved, so I > think that we need to limit the PCI address space used. I think you're mixing things a bit or not explaining them very well. Having the PCIe controller limited to 32-bit AXI does not mean that the PCIe bus cannot carry 64-bit addresses. It depends on how they get translated by the host bridge or its associated ATS block. I can't see why you can't have a setup where the CPU addresses are 32-bit but the PCIe bus addresses are all 64-bit. You just have to be careful on how you setup your mem64 ranges so that they don't overlap with the 32-bit ranges when translated. And no, you should not limit at the card driver the DMA_BIT_MASK() unless the card is not capable of supporting more than 32-bit addresses. > > Since pci_setup_device() sets up dma_mask, I added a bus notifier in the > PCIe controller driver so I can change the mask, if needed, on the > BUS_NOTIFY_BOUND_DRIVER action. > However, I think there is the potential for card drivers to allocate and > map buffers before the bus notifier get called. Additionally, I've seen > drivers change their behaviour based on the success or failure of > dma_set_mask_and_coherent(dev, DMA_BIT_MASK(64)), so the > driver could, theoretically at least, operate in a way that is not > compatible with a more restricted dma_mask (though I can't think > of any way this would not work with hardware I've seen). > > So, I think that using a bus notifier is the wrong way to go, but I don’t > know what other options I have. Any suggestions? I would first have a look at how the PCIe bus addresses are translated by the host controller.
Re: PCIe host controller behind IOMMU on ARM
On Mon, Nov 09, 2015 at 12:32:13PM +, Phil Edworthy wrote: > Hi Liviu, Will, > > On 04 November 2015 15:19, Phil wrote: > > On 04 November 2015 15:02, Liviu wrote: > > > On Wed, Nov 04, 2015 at 02:48:38PM +, Phil Edworthy wrote: > > > > Hi Liviu, > > > > > > > > On 04 November 2015 14:24, Liviu wrote: > > > > > On Wed, Nov 04, 2015 at 01:57:48PM +, Phil Edworthy wrote: > > > > > > Hi, > > > > > > > > > > > > I am trying to hook up a PCIe host controller that sits behind an > > > > > > IOMMU, > > > > > > but having some problems. > > > > > > > > > > > > I'm using the pcie-rcar PCIe host controller and it works fine > > > > > > without > > > > > > the IOMMU, and I can attach the IOMMU to the controller such that > > > > > > any > > > calls > > > > > > to dma_alloc_coherent made by the controller driver uses the > > iommu_ops > > > > > > version of dma_ops. > > > > > > > > > > > > However, I can't see how to make the endpoints to utilise the > > > > > > dma_ops > > that > > > > > > the controller uses. Shouldn't the endpoints inherit the dma_ops > > > > > > from the > > > > > > controller? > > > > > > > > > > No, not directly. > > > > > > > > > > > Any pointers for this? > > > > > > > > > > You need to understand the process through which a driver for endpoint > > get > > > > > an address to be passed down to the device. Have a look at > > > > > Documentation/DMA-API-HOWTO.txt, there is a nice explanation there. > > > > > (Hint: EP driver needs to call dma_map_single). > > > > > > > > > > Also, you need to make sure that the bus address that ends up being > > > > > set > > into > > > > > the endpoint gets translated correctly by the host controller into an > > > > > address > > > > > that the IOMMU can then translate into physical address. > > > > Sure, though since this is bog standard Intel PCIe ethernet card which > > > > works > > > > fine when the IOMMU is effectively unused, I don’t think there is a > > > > problem > > > > with that. > > > > > > > > The driver for the PCIe controller sets up the IOMMU mapping ok when I > > > > do a test call to dma_alloc_coherent() in the controller's driver. i.e. > > > > when I > > > > do this, it ends up in arm_iommu_alloc_attrs(), which calls > > > > __iommu_alloc_buffer() and __alloc_iova(). > > > > > > > > When an endpoint driver allocates and maps a dma coherent buffer it > > > > also needs to end up in arm_iommu_alloc_attrs(), but it doesn't. > > > > > > Why do you think that? Remember that the only thing attached to the IOMMU > > is > > > the > > > host controller. The endpoint is on the PCIe bus, which gets a different > > > translation > > > that the IOMMU knows nothing about. If it helps you to visualise it > > > better, think > > > of the host controller as another IOMMU device. It's the ops of the host > > > controller > > > that should be invoked, not the IOMMU's. > > Ok, that makes sense. I'll have a think and poke it a bit more... Hi Phil, Not trying to ignore your email, but I thought this is more in Will's backyard. > Somewhat related to this, since our PCIe controller HW is limited to > 32-bit AXI address range, before trying to hook up the IOMMU I have > tried to limit the dma_mask for PCI cards to DMA_BIT_MASK(32). The > reason being that Linux uses a 1 to 1 mapping between PCI addresses > and cpu (phys) addresses when there isn't an IOMMU involved, so I > think that we need to limit the PCI address space used. I think you're mixing things a bit or not explaining them very well. Having the PCIe controller limited to 32-bit AXI does not mean that the PCIe bus cannot carry 64-bit addresses. It depends on how they get translated by the host bridge or its associated ATS block. I can't see why you can't have a setup where the CPU addresses are 32-bit but the PCIe bus addresses are all 64-bit. You just have to be careful on how you setup your mem64 ranges so that they don't overlap with the 32-bit ranges when translated. And no, you should not limit at the card driver the DMA_BIT_MASK() unless the card is not capable of supporting more than 32-bit addresses. > > Since pci_setup_device() sets up dma_mask, I added a bus notifier in the > PCIe controller driver so I can change the mask, if needed, on the > BUS_NOTIFY_BOUND_DRIVER action. > However, I think there is the potential for card drivers to allocate and > map buffers before the bus notifier get called. Additionally, I've seen > drivers change their behaviour based on the success or failure of > dma_set_mask_and_coherent(dev, DMA_BIT_MASK(64)), so the > driver could, theoretically at least, operate in a way that is not > compatible with a more restricted dma_mask (though I can't think > of any way this would not work with hardware I've seen). > > So, I think that using a bus notifier is the wrong way to go, but I don’t > know what other options I have. Any suggestions? I would first have a look at how the PCIe bus addresses are translated by the host controller.
Re: PCIe host controller behind IOMMU on ARM
On Wednesday 11 November 2015 18:24:56 liviu.du...@arm.com wrote: > > > Somewhat related to this, since our PCIe controller HW is limited to > > 32-bit AXI address range, before trying to hook up the IOMMU I have > > tried to limit the dma_mask for PCI cards to DMA_BIT_MASK(32). The > > reason being that Linux uses a 1 to 1 mapping between PCI addresses > > and cpu (phys) addresses when there isn't an IOMMU involved, so I > > think that we need to limit the PCI address space used. > > I think you're mixing things a bit or not explaining them very well. Having > the > PCIe controller limited to 32-bit AXI does not mean that the PCIe bus cannot > carry 64-bit addresses. It depends on how they get translated by the host > bridge > or its associated ATS block. I can't see why you can't have a setup where > the CPU addresses are 32-bit but the PCIe bus addresses are all 64-bit. > You just have to be careful on how you setup your mem64 ranges so that they > don't > overlap with the 32-bit ranges when translated. > > And no, you should not limit at the card driver the DMA_BIT_MASK() unless the > card is not capable of supporting more than 32-bit addresses. I think we are missing one crucial bit of infrastructure on ARM64 at the moment: the dma_set_mask() function should fail if a driver asks for a mask that is larger than the dma-ranges property of the parent device (or any device higher up in the hierarchy) allows. Drivers that want a larger mask should try that first, and then fall back to a 32-bit mask, which is guaranteed to work. Arnd -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
RE: PCIe host controller behind IOMMU on ARM
Hi Liviu, Will, On 04 November 2015 15:19, Phil wrote: > On 04 November 2015 15:02, Liviu wrote: > > On Wed, Nov 04, 2015 at 02:48:38PM +, Phil Edworthy wrote: > > > Hi Liviu, > > > > > > On 04 November 2015 14:24, Liviu wrote: > > > > On Wed, Nov 04, 2015 at 01:57:48PM +, Phil Edworthy wrote: > > > > > Hi, > > > > > > > > > > I am trying to hook up a PCIe host controller that sits behind an > > > > > IOMMU, > > > > > but having some problems. > > > > > > > > > > I'm using the pcie-rcar PCIe host controller and it works fine without > > > > > the IOMMU, and I can attach the IOMMU to the controller such that any > > calls > > > > > to dma_alloc_coherent made by the controller driver uses the > iommu_ops > > > > > version of dma_ops. > > > > > > > > > > However, I can't see how to make the endpoints to utilise the dma_ops > that > > > > > the controller uses. Shouldn't the endpoints inherit the dma_ops from > > > > > the > > > > > controller? > > > > > > > > No, not directly. > > > > > > > > > Any pointers for this? > > > > > > > > You need to understand the process through which a driver for endpoint > get > > > > an address to be passed down to the device. Have a look at > > > > Documentation/DMA-API-HOWTO.txt, there is a nice explanation there. > > > > (Hint: EP driver needs to call dma_map_single). > > > > > > > > Also, you need to make sure that the bus address that ends up being set > into > > > > the endpoint gets translated correctly by the host controller into an > > > > address > > > > that the IOMMU can then translate into physical address. > > > Sure, though since this is bog standard Intel PCIe ethernet card which > > > works > > > fine when the IOMMU is effectively unused, I don’t think there is a > > > problem > > > with that. > > > > > > The driver for the PCIe controller sets up the IOMMU mapping ok when I > > > do a test call to dma_alloc_coherent() in the controller's driver. i.e. > > > when I > > > do this, it ends up in arm_iommu_alloc_attrs(), which calls > > > __iommu_alloc_buffer() and __alloc_iova(). > > > > > > When an endpoint driver allocates and maps a dma coherent buffer it > > > also needs to end up in arm_iommu_alloc_attrs(), but it doesn't. > > > > Why do you think that? Remember that the only thing attached to the IOMMU > is > > the > > host controller. The endpoint is on the PCIe bus, which gets a different > > translation > > that the IOMMU knows nothing about. If it helps you to visualise it better, > > think > > of the host controller as another IOMMU device. It's the ops of the host > > controller > > that should be invoked, not the IOMMU's. > Ok, that makes sense. I'll have a think and poke it a bit more... Somewhat related to this, since our PCIe controller HW is limited to 32-bit AXI address range, before trying to hook up the IOMMU I have tried to limit the dma_mask for PCI cards to DMA_BIT_MASK(32). The reason being that Linux uses a 1 to 1 mapping between PCI addresses and cpu (phys) addresses when there isn't an IOMMU involved, so I think that we need to limit the PCI address space used. Since pci_setup_device() sets up dma_mask, I added a bus notifier in the PCIe controller driver so I can change the mask, if needed, on the BUS_NOTIFY_BOUND_DRIVER action. However, I think there is the potential for card drivers to allocate and map buffers before the bus notifier get called. Additionally, I've seen drivers change their behaviour based on the success or failure of dma_set_mask_and_coherent(dev, DMA_BIT_MASK(64)), so the driver could, theoretically at least, operate in a way that is not compatible with a more restricted dma_mask (though I can't think of any way this would not work with hardware I've seen). So, I think that using a bus notifier is the wrong way to go, but I don’t know what other options I have. Any suggestions? Thanks for your help Phil
RE: PCIe host controller behind IOMMU on ARM
Hi Liviu, Will, On 04 November 2015 15:19, Phil wrote: > On 04 November 2015 15:02, Liviu wrote: > > On Wed, Nov 04, 2015 at 02:48:38PM +, Phil Edworthy wrote: > > > Hi Liviu, > > > > > > On 04 November 2015 14:24, Liviu wrote: > > > > On Wed, Nov 04, 2015 at 01:57:48PM +, Phil Edworthy wrote: > > > > > Hi, > > > > > > > > > > I am trying to hook up a PCIe host controller that sits behind an > > > > > IOMMU, > > > > > but having some problems. > > > > > > > > > > I'm using the pcie-rcar PCIe host controller and it works fine without > > > > > the IOMMU, and I can attach the IOMMU to the controller such that any > > calls > > > > > to dma_alloc_coherent made by the controller driver uses the > iommu_ops > > > > > version of dma_ops. > > > > > > > > > > However, I can't see how to make the endpoints to utilise the dma_ops > that > > > > > the controller uses. Shouldn't the endpoints inherit the dma_ops from > > > > > the > > > > > controller? > > > > > > > > No, not directly. > > > > > > > > > Any pointers for this? > > > > > > > > You need to understand the process through which a driver for endpoint > get > > > > an address to be passed down to the device. Have a look at > > > > Documentation/DMA-API-HOWTO.txt, there is a nice explanation there. > > > > (Hint: EP driver needs to call dma_map_single). > > > > > > > > Also, you need to make sure that the bus address that ends up being set > into > > > > the endpoint gets translated correctly by the host controller into an > > > > address > > > > that the IOMMU can then translate into physical address. > > > Sure, though since this is bog standard Intel PCIe ethernet card which > > > works > > > fine when the IOMMU is effectively unused, I don’t think there is a > > > problem > > > with that. > > > > > > The driver for the PCIe controller sets up the IOMMU mapping ok when I > > > do a test call to dma_alloc_coherent() in the controller's driver. i.e. > > > when I > > > do this, it ends up in arm_iommu_alloc_attrs(), which calls > > > __iommu_alloc_buffer() and __alloc_iova(). > > > > > > When an endpoint driver allocates and maps a dma coherent buffer it > > > also needs to end up in arm_iommu_alloc_attrs(), but it doesn't. > > > > Why do you think that? Remember that the only thing attached to the IOMMU > is > > the > > host controller. The endpoint is on the PCIe bus, which gets a different > > translation > > that the IOMMU knows nothing about. If it helps you to visualise it better, > > think > > of the host controller as another IOMMU device. It's the ops of the host > > controller > > that should be invoked, not the IOMMU's. > Ok, that makes sense. I'll have a think and poke it a bit more... Somewhat related to this, since our PCIe controller HW is limited to 32-bit AXI address range, before trying to hook up the IOMMU I have tried to limit the dma_mask for PCI cards to DMA_BIT_MASK(32). The reason being that Linux uses a 1 to 1 mapping between PCI addresses and cpu (phys) addresses when there isn't an IOMMU involved, so I think that we need to limit the PCI address space used. Since pci_setup_device() sets up dma_mask, I added a bus notifier in the PCIe controller driver so I can change the mask, if needed, on the BUS_NOTIFY_BOUND_DRIVER action. However, I think there is the potential for card drivers to allocate and map buffers before the bus notifier get called. Additionally, I've seen drivers change their behaviour based on the success or failure of dma_set_mask_and_coherent(dev, DMA_BIT_MASK(64)), so the driver could, theoretically at least, operate in a way that is not compatible with a more restricted dma_mask (though I can't think of any way this would not work with hardware I've seen). So, I think that using a bus notifier is the wrong way to go, but I don’t know what other options I have. Any suggestions? Thanks for your help Phil
RE: PCIe host controller behind IOMMU on ARM
Hi Will, On 04 November 2015 15:30, Will wrote: > On Wed, Nov 04, 2015 at 03:19:13PM +, Phil Edworthy wrote: > > On 04 November 2015 15:02, Liviu wrote: > > > On Wed, Nov 04, 2015 at 02:48:38PM +, Phil Edworthy wrote: > > > > Sure, though since this is bog standard Intel PCIe ethernet card which > > > > works > > > > fine when the IOMMU is effectively unused, I don’t think there is a > > > > problem > > > > with that. > > > > > > > > The driver for the PCIe controller sets up the IOMMU mapping ok when I > > > > do a test call to dma_alloc_coherent() in the controller's driver. i.e. > > > > when I > > > > do this, it ends up in arm_iommu_alloc_attrs(), which calls > > > > __iommu_alloc_buffer() and __alloc_iova(). > > > > > > > > When an endpoint driver allocates and maps a dma coherent buffer it > > > > also needs to end up in arm_iommu_alloc_attrs(), but it doesn't. > > > > > > Why do you think that? Remember that the only thing attached to the > IOMMU is > > > the > > > host controller. The endpoint is on the PCIe bus, which gets a different > > > translation > > > that the IOMMU knows nothing about. If it helps you to visualise it > > > better, > think > > > of the host controller as another IOMMU device. It's the ops of the host > > > controller > > > that should be invoked, not the IOMMU's. > > Ok, that makes sense. I'll have a think and poke it a bit more... > > Take a look at of_iommu_configure, which is currently lacking support > for PCI devices. It should be using a variant on the device-tree bindings > already in use for describing MSI device IDs, so that we can translate > the RequesterID of the endpoint into an ID that the IOMMU can understand. Whilst we want to introduce isolation at some point, right now we would like to use the IOMMU as this PCIe controller only uses a 32-bit AXI address. Thanks Phil
Re: PCIe host controller behind IOMMU on ARM
On Wed, Nov 04, 2015 at 03:19:13PM +, Phil Edworthy wrote: > On 04 November 2015 15:02, Liviu wrote: > > On Wed, Nov 04, 2015 at 02:48:38PM +, Phil Edworthy wrote: > > > Sure, though since this is bog standard Intel PCIe ethernet card which > > > works > > > fine when the IOMMU is effectively unused, I don’t think there is a > > > problem > > > with that. > > > > > > The driver for the PCIe controller sets up the IOMMU mapping ok when I > > > do a test call to dma_alloc_coherent() in the controller's driver. i.e. > > > when I > > > do this, it ends up in arm_iommu_alloc_attrs(), which calls > > > __iommu_alloc_buffer() and __alloc_iova(). > > > > > > When an endpoint driver allocates and maps a dma coherent buffer it > > > also needs to end up in arm_iommu_alloc_attrs(), but it doesn't. > > > > Why do you think that? Remember that the only thing attached to the IOMMU is > > the > > host controller. The endpoint is on the PCIe bus, which gets a different > > translation > > that the IOMMU knows nothing about. If it helps you to visualise it better, > > think > > of the host controller as another IOMMU device. It's the ops of the host > > controller > > that should be invoked, not the IOMMU's. > Ok, that makes sense. I'll have a think and poke it a bit more... Take a look at of_iommu_configure, which is currently lacking support for PCI devices. It should be using a variant on the device-tree bindings already in use for describing MSI device IDs, so that we can translate the RequesterID of the endpoint into an ID that the IOMMU can understand. Will -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
RE: PCIe host controller behind IOMMU on ARM
Hi Liviu, On 04 November 2015 15:02, Liviu wrote: > On Wed, Nov 04, 2015 at 02:48:38PM +, Phil Edworthy wrote: > > Hi Liviu, > > > > On 04 November 2015 14:24, Liviu wrote: > > > On Wed, Nov 04, 2015 at 01:57:48PM +, Phil Edworthy wrote: > > > > Hi, > > > > > > > > I am trying to hook up a PCIe host controller that sits behind an IOMMU, > > > > but having some problems. > > > > > > > > I'm using the pcie-rcar PCIe host controller and it works fine without > > > > the IOMMU, and I can attach the IOMMU to the controller such that any > calls > > > > to dma_alloc_coherent made by the controller driver uses the iommu_ops > > > > version of dma_ops. > > > > > > > > However, I can't see how to make the endpoints to utilise the dma_ops > > > > that > > > > the controller uses. Shouldn't the endpoints inherit the dma_ops from > > > > the > > > > controller? > > > > > > No, not directly. > > > > > > > Any pointers for this? > > > > > > You need to understand the process through which a driver for endpoint get > > > an address to be passed down to the device. Have a look at > > > Documentation/DMA-API-HOWTO.txt, there is a nice explanation there. > > > (Hint: EP driver needs to call dma_map_single). > > > > > > Also, you need to make sure that the bus address that ends up being set > > > into > > > the endpoint gets translated correctly by the host controller into an > > > address > > > that the IOMMU can then translate into physical address. > > Sure, though since this is bog standard Intel PCIe ethernet card which works > > fine when the IOMMU is effectively unused, I don’t think there is a problem > > with that. > > > > The driver for the PCIe controller sets up the IOMMU mapping ok when I > > do a test call to dma_alloc_coherent() in the controller's driver. i.e. > > when I > > do this, it ends up in arm_iommu_alloc_attrs(), which calls > > __iommu_alloc_buffer() and __alloc_iova(). > > > > When an endpoint driver allocates and maps a dma coherent buffer it > > also needs to end up in arm_iommu_alloc_attrs(), but it doesn't. > > Why do you think that? Remember that the only thing attached to the IOMMU is > the > host controller. The endpoint is on the PCIe bus, which gets a different > translation > that the IOMMU knows nothing about. If it helps you to visualise it better, > think > of the host controller as another IOMMU device. It's the ops of the host > controller > that should be invoked, not the IOMMU's. Ok, that makes sense. I'll have a think and poke it a bit more... Thanks for your comments Phil
Re: PCIe host controller behind IOMMU on ARM
On Wed, Nov 04, 2015 at 02:48:38PM +, Phil Edworthy wrote: > Hi Liviu, > > On 04 November 2015 14:24, Liviu wrote: > > On Wed, Nov 04, 2015 at 01:57:48PM +, Phil Edworthy wrote: > > > Hi, > > > > > > I am trying to hook up a PCIe host controller that sits behind an IOMMU, > > > but having some problems. > > > > > > I'm using the pcie-rcar PCIe host controller and it works fine without > > > the IOMMU, and I can attach the IOMMU to the controller such that any > > > calls > > > to dma_alloc_coherent made by the controller driver uses the iommu_ops > > > version of dma_ops. > > > > > > However, I can't see how to make the endpoints to utilise the dma_ops that > > > the controller uses. Shouldn't the endpoints inherit the dma_ops from the > > > controller? > > > > No, not directly. > > > > > Any pointers for this? > > > > You need to understand the process through which a driver for endpoint get > > an address to be passed down to the device. Have a look at > > Documentation/DMA-API-HOWTO.txt, there is a nice explanation there. > > (Hint: EP driver needs to call dma_map_single). > > > > Also, you need to make sure that the bus address that ends up being set into > > the endpoint gets translated correctly by the host controller into an > > address > > that the IOMMU can then translate into physical address. > Sure, though since this is bog standard Intel PCIe ethernet card which works > fine when the IOMMU is effectively unused, I don’t think there is a problem > with that. > > The driver for the PCIe controller sets up the IOMMU mapping ok when I > do a test call to dma_alloc_coherent() in the controller's driver. i.e. when I > do this, it ends up in arm_iommu_alloc_attrs(), which calls > __iommu_alloc_buffer() and __alloc_iova(). > > When an endpoint driver allocates and maps a dma coherent buffer it > also needs to end up in arm_iommu_alloc_attrs(), but it doesn't. Why do you think that? Remember that the only thing attached to the IOMMU is the host controller. The endpoint is on the PCIe bus, which gets a different translation that the IOMMU knows nothing about. If it helps you to visualise it better, think of the host controller as another IOMMU device. It's the ops of the host controller that should be invoked, not the IOMMU's. Best regards, Liviu > > Thanks > Phil -- | I would like to | | fix the world, | | but they're not | | giving me the | \ source code! / --- ¯\_(ツ)_/¯ -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
RE: PCIe host controller behind IOMMU on ARM
Hi Liviu, On 04 November 2015 14:24, Liviu wrote: > On Wed, Nov 04, 2015 at 01:57:48PM +, Phil Edworthy wrote: > > Hi, > > > > I am trying to hook up a PCIe host controller that sits behind an IOMMU, > > but having some problems. > > > > I'm using the pcie-rcar PCIe host controller and it works fine without > > the IOMMU, and I can attach the IOMMU to the controller such that any calls > > to dma_alloc_coherent made by the controller driver uses the iommu_ops > > version of dma_ops. > > > > However, I can't see how to make the endpoints to utilise the dma_ops that > > the controller uses. Shouldn't the endpoints inherit the dma_ops from the > > controller? > > No, not directly. > > > Any pointers for this? > > You need to understand the process through which a driver for endpoint get > an address to be passed down to the device. Have a look at > Documentation/DMA-API-HOWTO.txt, there is a nice explanation there. > (Hint: EP driver needs to call dma_map_single). > > Also, you need to make sure that the bus address that ends up being set into > the endpoint gets translated correctly by the host controller into an address > that the IOMMU can then translate into physical address. Sure, though since this is bog standard Intel PCIe ethernet card which works fine when the IOMMU is effectively unused, I don’t think there is a problem with that. The driver for the PCIe controller sets up the IOMMU mapping ok when I do a test call to dma_alloc_coherent() in the controller's driver. i.e. when I do this, it ends up in arm_iommu_alloc_attrs(), which calls __iommu_alloc_buffer() and __alloc_iova(). When an endpoint driver allocates and maps a dma coherent buffer it also needs to end up in arm_iommu_alloc_attrs(), but it doesn't. Thanks Phil
Re: PCIe host controller behind IOMMU on ARM
On Wed, Nov 04, 2015 at 01:57:48PM +, Phil Edworthy wrote: > Hi, > > I am trying to hook up a PCIe host controller that sits behind an IOMMU, > but having some problems. > > I'm using the pcie-rcar PCIe host controller and it works fine without > the IOMMU, and I can attach the IOMMU to the controller such that any calls > to dma_alloc_coherent made by the controller driver uses the iommu_ops > version of dma_ops. > > However, I can't see how to make the endpoints to utilise the dma_ops that > the controller uses. Shouldn't the endpoints inherit the dma_ops from the > controller? No, not directly. > Any pointers for this? You need to understand the process through which a driver for endpoint get an address to be passed down to the device. Have a look at Documentation/DMA-API-HOWTO.txt, there is a nice explanation there. (Hint: EP driver needs to call dma_map_single). Also, you need to make sure that the bus address that ends up being set into the endpoint gets translated correctly by the host controller into an address that the IOMMU can then translate into physical address. Best regards, Liviu > > Thanks > Phil > > -- > To unsubscribe from this list: send the line "unsubscribe linux-pci" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- | I would like to | | fix the world, | | but they're not | | giving me the | \ source code! / --- ¯\_(ツ)_/¯ -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
RE: PCIe host controller behind IOMMU on ARM
Hi Liviu, On 04 November 2015 14:24, Liviu wrote: > On Wed, Nov 04, 2015 at 01:57:48PM +, Phil Edworthy wrote: > > Hi, > > > > I am trying to hook up a PCIe host controller that sits behind an IOMMU, > > but having some problems. > > > > I'm using the pcie-rcar PCIe host controller and it works fine without > > the IOMMU, and I can attach the IOMMU to the controller such that any calls > > to dma_alloc_coherent made by the controller driver uses the iommu_ops > > version of dma_ops. > > > > However, I can't see how to make the endpoints to utilise the dma_ops that > > the controller uses. Shouldn't the endpoints inherit the dma_ops from the > > controller? > > No, not directly. > > > Any pointers for this? > > You need to understand the process through which a driver for endpoint get > an address to be passed down to the device. Have a look at > Documentation/DMA-API-HOWTO.txt, there is a nice explanation there. > (Hint: EP driver needs to call dma_map_single). > > Also, you need to make sure that the bus address that ends up being set into > the endpoint gets translated correctly by the host controller into an address > that the IOMMU can then translate into physical address. Sure, though since this is bog standard Intel PCIe ethernet card which works fine when the IOMMU is effectively unused, I don’t think there is a problem with that. The driver for the PCIe controller sets up the IOMMU mapping ok when I do a test call to dma_alloc_coherent() in the controller's driver. i.e. when I do this, it ends up in arm_iommu_alloc_attrs(), which calls __iommu_alloc_buffer() and __alloc_iova(). When an endpoint driver allocates and maps a dma coherent buffer it also needs to end up in arm_iommu_alloc_attrs(), but it doesn't. Thanks Phil
Re: PCIe host controller behind IOMMU on ARM
On Wed, Nov 04, 2015 at 02:48:38PM +, Phil Edworthy wrote: > Hi Liviu, > > On 04 November 2015 14:24, Liviu wrote: > > On Wed, Nov 04, 2015 at 01:57:48PM +, Phil Edworthy wrote: > > > Hi, > > > > > > I am trying to hook up a PCIe host controller that sits behind an IOMMU, > > > but having some problems. > > > > > > I'm using the pcie-rcar PCIe host controller and it works fine without > > > the IOMMU, and I can attach the IOMMU to the controller such that any > > > calls > > > to dma_alloc_coherent made by the controller driver uses the iommu_ops > > > version of dma_ops. > > > > > > However, I can't see how to make the endpoints to utilise the dma_ops that > > > the controller uses. Shouldn't the endpoints inherit the dma_ops from the > > > controller? > > > > No, not directly. > > > > > Any pointers for this? > > > > You need to understand the process through which a driver for endpoint get > > an address to be passed down to the device. Have a look at > > Documentation/DMA-API-HOWTO.txt, there is a nice explanation there. > > (Hint: EP driver needs to call dma_map_single). > > > > Also, you need to make sure that the bus address that ends up being set into > > the endpoint gets translated correctly by the host controller into an > > address > > that the IOMMU can then translate into physical address. > Sure, though since this is bog standard Intel PCIe ethernet card which works > fine when the IOMMU is effectively unused, I don’t think there is a problem > with that. > > The driver for the PCIe controller sets up the IOMMU mapping ok when I > do a test call to dma_alloc_coherent() in the controller's driver. i.e. when I > do this, it ends up in arm_iommu_alloc_attrs(), which calls > __iommu_alloc_buffer() and __alloc_iova(). > > When an endpoint driver allocates and maps a dma coherent buffer it > also needs to end up in arm_iommu_alloc_attrs(), but it doesn't. Why do you think that? Remember that the only thing attached to the IOMMU is the host controller. The endpoint is on the PCIe bus, which gets a different translation that the IOMMU knows nothing about. If it helps you to visualise it better, think of the host controller as another IOMMU device. It's the ops of the host controller that should be invoked, not the IOMMU's. Best regards, Liviu > > Thanks > Phil -- | I would like to | | fix the world, | | but they're not | | giving me the | \ source code! / --- ¯\_(ツ)_/¯ -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: PCIe host controller behind IOMMU on ARM
On Wed, Nov 04, 2015 at 01:57:48PM +, Phil Edworthy wrote: > Hi, > > I am trying to hook up a PCIe host controller that sits behind an IOMMU, > but having some problems. > > I'm using the pcie-rcar PCIe host controller and it works fine without > the IOMMU, and I can attach the IOMMU to the controller such that any calls > to dma_alloc_coherent made by the controller driver uses the iommu_ops > version of dma_ops. > > However, I can't see how to make the endpoints to utilise the dma_ops that > the controller uses. Shouldn't the endpoints inherit the dma_ops from the > controller? No, not directly. > Any pointers for this? You need to understand the process through which a driver for endpoint get an address to be passed down to the device. Have a look at Documentation/DMA-API-HOWTO.txt, there is a nice explanation there. (Hint: EP driver needs to call dma_map_single). Also, you need to make sure that the bus address that ends up being set into the endpoint gets translated correctly by the host controller into an address that the IOMMU can then translate into physical address. Best regards, Liviu > > Thanks > Phil > > -- > To unsubscribe from this list: send the line "unsubscribe linux-pci" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- | I would like to | | fix the world, | | but they're not | | giving me the | \ source code! / --- ¯\_(ツ)_/¯ -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
RE: PCIe host controller behind IOMMU on ARM
Hi Liviu, On 04 November 2015 15:02, Liviu wrote: > On Wed, Nov 04, 2015 at 02:48:38PM +, Phil Edworthy wrote: > > Hi Liviu, > > > > On 04 November 2015 14:24, Liviu wrote: > > > On Wed, Nov 04, 2015 at 01:57:48PM +, Phil Edworthy wrote: > > > > Hi, > > > > > > > > I am trying to hook up a PCIe host controller that sits behind an IOMMU, > > > > but having some problems. > > > > > > > > I'm using the pcie-rcar PCIe host controller and it works fine without > > > > the IOMMU, and I can attach the IOMMU to the controller such that any > calls > > > > to dma_alloc_coherent made by the controller driver uses the iommu_ops > > > > version of dma_ops. > > > > > > > > However, I can't see how to make the endpoints to utilise the dma_ops > > > > that > > > > the controller uses. Shouldn't the endpoints inherit the dma_ops from > > > > the > > > > controller? > > > > > > No, not directly. > > > > > > > Any pointers for this? > > > > > > You need to understand the process through which a driver for endpoint get > > > an address to be passed down to the device. Have a look at > > > Documentation/DMA-API-HOWTO.txt, there is a nice explanation there. > > > (Hint: EP driver needs to call dma_map_single). > > > > > > Also, you need to make sure that the bus address that ends up being set > > > into > > > the endpoint gets translated correctly by the host controller into an > > > address > > > that the IOMMU can then translate into physical address. > > Sure, though since this is bog standard Intel PCIe ethernet card which works > > fine when the IOMMU is effectively unused, I don’t think there is a problem > > with that. > > > > The driver for the PCIe controller sets up the IOMMU mapping ok when I > > do a test call to dma_alloc_coherent() in the controller's driver. i.e. > > when I > > do this, it ends up in arm_iommu_alloc_attrs(), which calls > > __iommu_alloc_buffer() and __alloc_iova(). > > > > When an endpoint driver allocates and maps a dma coherent buffer it > > also needs to end up in arm_iommu_alloc_attrs(), but it doesn't. > > Why do you think that? Remember that the only thing attached to the IOMMU is > the > host controller. The endpoint is on the PCIe bus, which gets a different > translation > that the IOMMU knows nothing about. If it helps you to visualise it better, > think > of the host controller as another IOMMU device. It's the ops of the host > controller > that should be invoked, not the IOMMU's. Ok, that makes sense. I'll have a think and poke it a bit more... Thanks for your comments Phil
Re: PCIe host controller behind IOMMU on ARM
On Wed, Nov 04, 2015 at 03:19:13PM +, Phil Edworthy wrote: > On 04 November 2015 15:02, Liviu wrote: > > On Wed, Nov 04, 2015 at 02:48:38PM +, Phil Edworthy wrote: > > > Sure, though since this is bog standard Intel PCIe ethernet card which > > > works > > > fine when the IOMMU is effectively unused, I don’t think there is a > > > problem > > > with that. > > > > > > The driver for the PCIe controller sets up the IOMMU mapping ok when I > > > do a test call to dma_alloc_coherent() in the controller's driver. i.e. > > > when I > > > do this, it ends up in arm_iommu_alloc_attrs(), which calls > > > __iommu_alloc_buffer() and __alloc_iova(). > > > > > > When an endpoint driver allocates and maps a dma coherent buffer it > > > also needs to end up in arm_iommu_alloc_attrs(), but it doesn't. > > > > Why do you think that? Remember that the only thing attached to the IOMMU is > > the > > host controller. The endpoint is on the PCIe bus, which gets a different > > translation > > that the IOMMU knows nothing about. If it helps you to visualise it better, > > think > > of the host controller as another IOMMU device. It's the ops of the host > > controller > > that should be invoked, not the IOMMU's. > Ok, that makes sense. I'll have a think and poke it a bit more... Take a look at of_iommu_configure, which is currently lacking support for PCI devices. It should be using a variant on the device-tree bindings already in use for describing MSI device IDs, so that we can translate the RequesterID of the endpoint into an ID that the IOMMU can understand. Will -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
RE: PCIe host controller behind IOMMU on ARM
Hi Will, On 04 November 2015 15:30, Will wrote: > On Wed, Nov 04, 2015 at 03:19:13PM +, Phil Edworthy wrote: > > On 04 November 2015 15:02, Liviu wrote: > > > On Wed, Nov 04, 2015 at 02:48:38PM +, Phil Edworthy wrote: > > > > Sure, though since this is bog standard Intel PCIe ethernet card which > > > > works > > > > fine when the IOMMU is effectively unused, I don’t think there is a > > > > problem > > > > with that. > > > > > > > > The driver for the PCIe controller sets up the IOMMU mapping ok when I > > > > do a test call to dma_alloc_coherent() in the controller's driver. i.e. > > > > when I > > > > do this, it ends up in arm_iommu_alloc_attrs(), which calls > > > > __iommu_alloc_buffer() and __alloc_iova(). > > > > > > > > When an endpoint driver allocates and maps a dma coherent buffer it > > > > also needs to end up in arm_iommu_alloc_attrs(), but it doesn't. > > > > > > Why do you think that? Remember that the only thing attached to the > IOMMU is > > > the > > > host controller. The endpoint is on the PCIe bus, which gets a different > > > translation > > > that the IOMMU knows nothing about. If it helps you to visualise it > > > better, > think > > > of the host controller as another IOMMU device. It's the ops of the host > > > controller > > > that should be invoked, not the IOMMU's. > > Ok, that makes sense. I'll have a think and poke it a bit more... > > Take a look at of_iommu_configure, which is currently lacking support > for PCI devices. It should be using a variant on the device-tree bindings > already in use for describing MSI device IDs, so that we can translate > the RequesterID of the endpoint into an ID that the IOMMU can understand. Whilst we want to introduce isolation at some point, right now we would like to use the IOMMU as this PCIe controller only uses a 32-bit AXI address. Thanks Phil
