Re: [RFC PATCH 0/7] Introduce automatic DMA configuration for IOMMU masters
Hi Will, On 2014-08-29 17:54, Will Deacon wrote: This patch series is an RFC to implement IOMMU master configuration into of_dma_configure. I haven't yet ported any IOMMU drivers to it, so it remains untested, but I wanted to get some early feedback to ensure that this doesn't end up going in the wrong direction. The idea comes out of my understanding following discussions with Arnd and David at Kernel Summit / LinuxCon in Chicago. Essentially: - Allow IOMMUs to be probed early and set up per-instance data on their of_node - Add a new callback to the iommu_ops structure for adding a device with a set of opaque IDs (e.g. Stream IDs or Requester IDs) - Add an of_iommu_configure function, called from of_dma_configure, which registers the master IDs with the correspond IOMMU before probing the master itself - Where applicable, create an IOMMU domain per device and swizzle the DMA ops for that device to use the IOMMU variants. I haven't bothered doing anything clever with the DMA masks yet, so I wouldn't be surprised if we end up chewing through tonnes of memory allocating IOMMU domains that are far too big at the moment. Again, this is just an RFC and I'd welcome comments on the general direction of the series. Thanks for your patches, I wasn't aware the fact that you are working on this. When do you plan to send a second version? I would like to rebase my Exynos IOMMU patches (https://lkml.org/lkml/2014/8/5/183) on top of your work, but I wonder if I should select this version as a base or wait a bit for an update. Best regards -- Marek Szyprowski, PhD Samsung RD Institute Poland ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH 0/7] Introduce automatic DMA configuration for IOMMU masters
On Tue, Sep 02, 2014 at 07:26:01AM +0100, Marek Szyprowski wrote: Hi Will, Hi Marek, On 2014-08-29 17:54, Will Deacon wrote: This patch series is an RFC to implement IOMMU master configuration into of_dma_configure. I haven't yet ported any IOMMU drivers to it, so it remains untested, but I wanted to get some early feedback to ensure that this doesn't end up going in the wrong direction. The idea comes out of my understanding following discussions with Arnd and David at Kernel Summit / LinuxCon in Chicago. Essentially: - Allow IOMMUs to be probed early and set up per-instance data on their of_node - Add a new callback to the iommu_ops structure for adding a device with a set of opaque IDs (e.g. Stream IDs or Requester IDs) - Add an of_iommu_configure function, called from of_dma_configure, which registers the master IDs with the correspond IOMMU before probing the master itself - Where applicable, create an IOMMU domain per device and swizzle the DMA ops for that device to use the IOMMU variants. I haven't bothered doing anything clever with the DMA masks yet, so I wouldn't be surprised if we end up chewing through tonnes of memory allocating IOMMU domains that are far too big at the moment. Again, this is just an RFC and I'd welcome comments on the general direction of the series. Thanks for your patches, I wasn't aware the fact that you are working on this. When do you plan to send a second version? I would like to rebase my Exynos IOMMU patches (https://lkml.org/lkml/2014/8/5/183) on top of your work, but I wonder if I should select this version as a base or wait a bit for an update. I'll try and get something out today/tomorrow depending on how easily the review comments fall out. It would be really great if you get an IOMMU working with this (I was going to look at the ARM SMMU once this stops moving) -- I have concerns that allocating one domain per master might be too much, but it's hard to tell without an IOMMU driver ported over. I'll CC you on v2. Cheers, Will ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH 0/7] Introduce automatic DMA configuration for IOMMU masters
Hi Will, On 2014-09-02 10:31, Will Deacon wrote: On Tue, Sep 02, 2014 at 07:26:01AM +0100, Marek Szyprowski wrote: On 2014-08-29 17:54, Will Deacon wrote: This patch series is an RFC to implement IOMMU master configuration into of_dma_configure. I haven't yet ported any IOMMU drivers to it, so it remains untested, but I wanted to get some early feedback to ensure that this doesn't end up going in the wrong direction. The idea comes out of my understanding following discussions with Arnd and David at Kernel Summit / LinuxCon in Chicago. Essentially: - Allow IOMMUs to be probed early and set up per-instance data on their of_node - Add a new callback to the iommu_ops structure for adding a device with a set of opaque IDs (e.g. Stream IDs or Requester IDs) - Add an of_iommu_configure function, called from of_dma_configure, which registers the master IDs with the correspond IOMMU before probing the master itself - Where applicable, create an IOMMU domain per device and swizzle the DMA ops for that device to use the IOMMU variants. I haven't bothered doing anything clever with the DMA masks yet, so I wouldn't be surprised if we end up chewing through tonnes of memory allocating IOMMU domains that are far too big at the moment. Again, this is just an RFC and I'd welcome comments on the general direction of the series. Thanks for your patches, I wasn't aware the fact that you are working on this. When do you plan to send a second version? I would like to rebase my Exynos IOMMU patches (https://lkml.org/lkml/2014/8/5/183) on top of your work, but I wonder if I should select this version as a base or wait a bit for an update. I'll try and get something out today/tomorrow depending on how easily the review comments fall out. It would be really great if you get an IOMMU working with this (I was going to look at the ARM SMMU once this stops moving) Great, I will wait then for v2. -- I have concerns that allocating one domain per master might be too much, but it's hard to tell without an IOMMU driver ported over. One domain per master is IMHO a sane default configuration. The only default alternative I see is to have only one domain (related with dma-mapping subsystem) and bind all devices to it. However I really don't see any disadvantage of having separate domain per each master and such configuration gives devices better separation. However we also need to figure out how to let drivers to make their own configuration, like it is required by Exynos DRM subsystem, which consist of several devices, each having its own IOMMU controller, but for convenience those drivers assume that they all have been bound to the same, single domain. Best regards -- Marek Szyprowski, PhD Samsung RD Institute Poland ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH 0/7] Introduce automatic DMA configuration for IOMMU masters
On Tuesday 02 September 2014 10:48:02 Marek Szyprowski wrote: -- I have concerns that allocating one domain per master might be too much, but it's hard to tell without an IOMMU driver ported over. One domain per master is IMHO a sane default configuration. The only default alternative I see is to have only one domain (related with dma-mapping subsystem) and bind all devices to it. However I really don't see any disadvantage of having separate domain per each master and such configuration gives devices better separation. I was expecting that the dma-mapping implementation would by default use one domain for all devices, since that is what the simpler IOMMUs without domain support have to do anyway. For isolation purposes, it can only help to have more domains, but I would guess that there is some space overhead in maintaining lots of page tables. However we also need to figure out how to let drivers to make their own configuration, like it is required by Exynos DRM subsystem, which consist of several devices, each having its own IOMMU controller, but for convenience those drivers assume that they all have been bound to the same, single domain. IIRC with the way we ended up putting the mask into the iommu descriptor of the ARM SMMU, you can put multiple devices into the same iommu group, and have them automatically share a domain. I don't know if the same would work for the Samsung implementation. Arnd ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH 0/7] Introduce automatic DMA configuration for IOMMU masters
Hi Arnd, On 2014-09-02 10:56, Arnd Bergmann wrote: On Tuesday 02 September 2014 10:48:02 Marek Szyprowski wrote: -- I have concerns that allocating one domain per master might be too much, but it's hard to tell without an IOMMU driver ported over. One domain per master is IMHO a sane default configuration. The only default alternative I see is to have only one domain (related with dma-mapping subsystem) and bind all devices to it. However I really don't see any disadvantage of having separate domain per each master and such configuration gives devices better separation. I was expecting that the dma-mapping implementation would by default use one domain for all devices, since that is what the simpler IOMMUs without domain support have to do anyway. For isolation purposes, it can only help to have more domains, but I would guess that there is some space overhead in maintaining lots of page tables. I'm okay with both approaches (separate domain for each device vs. single common domain for all devices). Maybe this can be some kind of Kconfig option added to DMA debugging? Separation might be really helpful when debugging strange device behavior. However we also need to figure out how to let drivers to make their own configuration, like it is required by Exynos DRM subsystem, which consist of several devices, each having its own IOMMU controller, but for convenience those drivers assume that they all have been bound to the same, single domain. IIRC with the way we ended up putting the mask into the iommu descriptor of the ARM SMMU, you can put multiple devices into the same iommu group, and have them automatically share a domain. I don't know if the same would work for the Samsung implementation. The question is how to transfer such information from the device drivers, that need/benefit from such configuration to iommu driver, which does all the setup? This is something completely internal to particular drivers and should not be exported to device tree or userspace. Thierry suggested to hardcode this information in the iommu driver, but I'm looking for other approaches. Maybe simply releasing device from the default dma-mapping domain before attaching to custom one will be the easiest solution. Best regards -- Marek Szyprowski, PhD Samsung RD Institute Poland ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH 0/7] Introduce automatic DMA configuration for IOMMU masters
On Tuesday 02 September 2014 12:42:13 Marek Szyprowski wrote: Hi Arnd, On 2014-09-02 10:56, Arnd Bergmann wrote: On Tuesday 02 September 2014 10:48:02 Marek Szyprowski wrote: -- I have concerns that allocating one domain per master might be too much, but it's hard to tell without an IOMMU driver ported over. One domain per master is IMHO a sane default configuration. The only default alternative I see is to have only one domain (related with dma-mapping subsystem) and bind all devices to it. However I really don't see any disadvantage of having separate domain per each master and such configuration gives devices better separation. I was expecting that the dma-mapping implementation would by default use one domain for all devices, since that is what the simpler IOMMUs without domain support have to do anyway. For isolation purposes, it can only help to have more domains, but I would guess that there is some space overhead in maintaining lots of page tables. I'm okay with both approaches (separate domain for each device vs. single common domain for all devices). Maybe this can be some kind of Kconfig option added to DMA debugging? Separation might be really helpful when debugging strange device behavior. We should probably support the iommu=strict command line option that some other architectures have. This is mainly meant to ensure that IOTLBs are shot down as soon as the driver unmaps some memory, which you often want to avoid for performance reasons. The iommu driver itself can then decide to also use separate domains for iommu=strict but a shared domain otherwise. For hardware on which the shared domain is hard to do, the driver might always use separate domains. However we also need to figure out how to let drivers to make their own configuration, like it is required by Exynos DRM subsystem, which consist of several devices, each having its own IOMMU controller, but for convenience those drivers assume that they all have been bound to the same, single domain. IIRC with the way we ended up putting the mask into the iommu descriptor of the ARM SMMU, you can put multiple devices into the same iommu group, and have them automatically share a domain. I don't know if the same would work for the Samsung implementation. The question is how to transfer such information from the device drivers, that need/benefit from such configuration to iommu driver, which does all the setup? This is something completely internal to particular drivers and should not be exported to device tree or userspace. Thierry suggested to hardcode this information in the iommu driver, but I'm looking for other approaches. Maybe simply releasing device from the default dma-mapping domain before attaching to custom one will be the easiest solution. For the ARM SMMU, the problem is that there is not necessarily a good way to partition the masters into IOMMU groups automatically, therefore we want to provide some hints in DT. On a machine that can have more domains than it has masters, this is not a problem and we can always use an all-ones mask, but for a machine on which this is not the case, the problem is simplified a lot of we hardcode the masks in a way that can always work, putting multiple devices into an iommu group if necessary. This is similar to how we do things for pinctrl, where you might have a theoretically endless space of options to set stuff up, but we can simplify it by defining the useful configurations. Arnd ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH 0/7] Introduce automatic DMA configuration for IOMMU masters
Hi Will, On 2014-09-02 12:57, Will Deacon wrote: On Tue, Sep 02, 2014 at 11:42:13AM +0100, Marek Szyprowski wrote: On 2014-09-02 10:56, Arnd Bergmann wrote: On Tuesday 02 September 2014 10:48:02 Marek Szyprowski wrote: -- I have concerns that allocating one domain per master might be too much, but it's hard to tell without an IOMMU driver ported over. One domain per master is IMHO a sane default configuration. The only default alternative I see is to have only one domain (related with dma-mapping subsystem) and bind all devices to it. However I really don't see any disadvantage of having separate domain per each master and such configuration gives devices better separation. I was expecting that the dma-mapping implementation would by default use one domain for all devices, since that is what the simpler IOMMUs without domain support have to do anyway. For isolation purposes, it can only help to have more domains, but I would guess that there is some space overhead in maintaining lots of page tables. I'm okay with both approaches (separate domain for each device vs. single common domain for all devices). Maybe this can be some kind of Kconfig option added to DMA debugging? Separation might be really helpful when debugging strange device behavior. One potential problem with a single domain is when you have multiple instances of a given IOMMU, each with different hardware restrictions. Then you can end up with multiple sets of page tables for the domain which, although not impossible to work with, is a bit of a mess. Maybe the default dma-mapping domain should be one per a given IOMMU instance? This will simplify a lot of things in such case. I think having one domain per IOMMU instance would make the most sense, but then you have to teach more of the stack about the IOMMU topology. I think we'll get there in the end, but that's a little way off right now. Right, those seems to be a details. Best regards -- Marek Szyprowski, PhD Samsung RD Institute Poland ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH 0/7] Introduce automatic DMA configuration for IOMMU masters
Hi Arnd, On 2014-09-02 14:22, Arnd Bergmann wrote: On Tuesday 02 September 2014 12:42:13 Marek Szyprowski wrote: On 2014-09-02 10:56, Arnd Bergmann wrote: On Tuesday 02 September 2014 10:48:02 Marek Szyprowski wrote: -- I have concerns that allocating one domain per master might be too much, but it's hard to tell without an IOMMU driver ported over. One domain per master is IMHO a sane default configuration. The only default alternative I see is to have only one domain (related with dma-mapping subsystem) and bind all devices to it. However I really don't see any disadvantage of having separate domain per each master and such configuration gives devices better separation. I was expecting that the dma-mapping implementation would by default use one domain for all devices, since that is what the simpler IOMMUs without domain support have to do anyway. For isolation purposes, it can only help to have more domains, but I would guess that there is some space overhead in maintaining lots of page tables. I'm okay with both approaches (separate domain for each device vs. single common domain for all devices). Maybe this can be some kind of Kconfig option added to DMA debugging? Separation might be really helpful when debugging strange device behavior. We should probably support the iommu=strict command line option that some other architectures have. This is mainly meant to ensure that IOTLBs are shot down as soon as the driver unmaps some memory, which you often want to avoid for performance reasons. The iommu driver itself can then decide to also use separate domains for iommu=strict but a shared domain otherwise. For hardware on which the shared domain is hard to do, the driver might always use separate domains. Just to let you know, lazy unmapping is not yet implemented in ARM dma-mapping implementation based on IOMMU. However we also need to figure out how to let drivers to make their own configuration, like it is required by Exynos DRM subsystem, which consist of several devices, each having its own IOMMU controller, but for convenience those drivers assume that they all have been bound to the same, single domain. IIRC with the way we ended up putting the mask into the iommu descriptor of the ARM SMMU, you can put multiple devices into the same iommu group, and have them automatically share a domain. I don't know if the same would work for the Samsung implementation. The question is how to transfer such information from the device drivers, that need/benefit from such configuration to iommu driver, which does all the setup? This is something completely internal to particular drivers and should not be exported to device tree or userspace. Thierry suggested to hardcode this information in the iommu driver, but I'm looking for other approaches. Maybe simply releasing device from the default dma-mapping domain before attaching to custom one will be the easiest solution. For the ARM SMMU, the problem is that there is not necessarily a good way to partition the masters into IOMMU groups automatically, therefore we want to provide some hints in DT. On a machine that can have more domains than it has masters, this is not a problem and we can always use an all-ones mask, but for a machine on which this is not the case, the problem is simplified a lot of we hardcode the masks in a way that can always work, putting multiple devices into an iommu group if necessary. Well, I was talking about the Exynos IOMMU case, where there are no hw restrictions and grouping is done just to make things easier for the Exynos DRM drivers (a buffer gets the same DMA address for all devices, which are a part of virtual Exynos DRM device). This is similar to how we do things for pinctrl, where you might have a theoretically endless space of options to set stuff up, but we can simplify it by defining the useful configurations. Right, if hardware is limited, a sane working configuration is something that should be encoded in device tree. Best regards -- Marek Szyprowski, PhD Samsung RD Institute Poland ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH 0/7] Introduce automatic DMA configuration for IOMMU masters
On Tuesday 02 September 2014 14:24:18 Marek Szyprowski wrote: For isolation purposes, it can only help to have more domains, but I would guess that there is some space overhead in maintaining lots of page tables. I'm okay with both approaches (separate domain for each device vs. single common domain for all devices). Maybe this can be some kind of Kconfig option added to DMA debugging? Separation might be really helpful when debugging strange device behavior. One potential problem with a single domain is when you have multiple instances of a given IOMMU, each with different hardware restrictions. Then you can end up with multiple sets of page tables for the domain which, although not impossible to work with, is a bit of a mess. Maybe the default dma-mapping domain should be one per a given IOMMU instance? This will simplify a lot of things in such case. Yes, that sounds like a good idea. Arnd ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH 0/7] Introduce automatic DMA configuration for IOMMU masters
On Tuesday 02 September 2014 14:30:36 Marek Szyprowski wrote: However we also need to figure out how to let drivers to make their own configuration, like it is required by Exynos DRM subsystem, which consist of several devices, each having its own IOMMU controller, but for convenience those drivers assume that they all have been bound to the same, single domain. IIRC with the way we ended up putting the mask into the iommu descriptor of the ARM SMMU, you can put multiple devices into the same iommu group, and have them automatically share a domain. I don't know if the same would work for the Samsung implementation. The question is how to transfer such information from the device drivers, that need/benefit from such configuration to iommu driver, which does all the setup? This is something completely internal to particular drivers and should not be exported to device tree or userspace. Thierry suggested to hardcode this information in the iommu driver, but I'm looking for other approaches. Maybe simply releasing device from the default dma-mapping domain before attaching to custom one will be the easiest solution. For the ARM SMMU, the problem is that there is not necessarily a good way to partition the masters into IOMMU groups automatically, therefore we want to provide some hints in DT. On a machine that can have more domains than it has masters, this is not a problem and we can always use an all-ones mask, but for a machine on which this is not the case, the problem is simplified a lot of we hardcode the masks in a way that can always work, putting multiple devices into an iommu group if necessary. Well, I was talking about the Exynos IOMMU case, where there are no hw restrictions and grouping is done just to make things easier for the Exynos DRM drivers (a buffer gets the same DMA address for all devices, which are a part of virtual Exynos DRM device). Does that mean you don't actually need to use multiple contexts here and could actually just use the normal dma-mapping interface if there is a way to ensure the mappings are always shared across these masters? Or do you need this in addition to being able to use multiple masters so you can have multiple rendering contexts in user space? Arnd ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH 0/7] Introduce automatic DMA configuration for IOMMU masters
On 2014-09-02 14:46, Arnd Bergmann wrote: On Tuesday 02 September 2014 14:30:36 Marek Szyprowski wrote: However we also need to figure out how to let drivers to make their own configuration, like it is required by Exynos DRM subsystem, which consist of several devices, each having its own IOMMU controller, but for convenience those drivers assume that they all have been bound to the same, single domain. IIRC with the way we ended up putting the mask into the iommu descriptor of the ARM SMMU, you can put multiple devices into the same iommu group, and have them automatically share a domain. I don't know if the same would work for the Samsung implementation. The question is how to transfer such information from the device drivers, that need/benefit from such configuration to iommu driver, which does all the setup? This is something completely internal to particular drivers and should not be exported to device tree or userspace. Thierry suggested to hardcode this information in the iommu driver, but I'm looking for other approaches. Maybe simply releasing device from the default dma-mapping domain before attaching to custom one will be the easiest solution. For the ARM SMMU, the problem is that there is not necessarily a good way to partition the masters into IOMMU groups automatically, therefore we want to provide some hints in DT. On a machine that can have more domains than it has masters, this is not a problem and we can always use an all-ones mask, but for a machine on which this is not the case, the problem is simplified a lot of we hardcode the masks in a way that can always work, putting multiple devices into an iommu group if necessary. Well, I was talking about the Exynos IOMMU case, where there are no hw restrictions and grouping is done just to make things easier for the Exynos DRM drivers (a buffer gets the same DMA address for all devices, which are a part of virtual Exynos DRM device). Does that mean you don't actually need to use multiple contexts here and could actually just use the normal dma-mapping interface if there is a way to ensure the mappings are always shared across these masters? Well, a default, shared single domain for dma-mapping interface will work with Exynos DRM and its multiple masters, although I never thought about such configuration. Or do you need this in addition to being able to use multiple masters so you can have multiple rendering contexts in user space? Such advanced IO space management is not yet implemented. However there are also devices (like multimedia codec - exynos mfc and camera capture subsystem exynos isp), which have limited DMA/IO window (256MiB in case of video codec), so they will still need to use their own separate domain. Best regards -- Marek Szyprowski, PhD Samsung RD Institute Poland ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH 0/7] Introduce automatic DMA configuration for IOMMU masters
On Tuesday 02 September 2014 14:43:18 Arnd Bergmann wrote: On Tuesday 02 September 2014 14:24:18 Marek Szyprowski wrote: For isolation purposes, it can only help to have more domains, but I would guess that there is some space overhead in maintaining lots of page tables. I'm okay with both approaches (separate domain for each device vs. single common domain for all devices). Maybe this can be some kind of Kconfig option added to DMA debugging? Separation might be really helpful when debugging strange device behavior. One potential problem with a single domain is when you have multiple instances of a given IOMMU, each with different hardware restrictions. Then you can end up with multiple sets of page tables for the domain which, although not impossible to work with, is a bit of a mess. Maybe the default dma-mapping domain should be one per a given IOMMU instance? This will simplify a lot of things in such case. Yes, that sounds like a good idea. That would work as a default configuration for the Renesas IPMMU IOMMU, which supports four TLBs to be shared between more than four bus masters. Ideally I'd like to make the bus master to TLB association somehow configurable, but associating all bus masters with a single TLB as a first step is fine. -- Regards, Laurent Pinchart ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
