Re: [PATCH 0/5] OMAP IOMMU fixes and IOMMU architecture questions
On Tue, Apr 08, 2014 at 05:02:37PM +0200, Laurent Pinchart wrote: On Tuesday 08 April 2014 15:43:22 Joerg Roedel wrote: Who is someone in this case? That's exactly the problem :-) The ARM DMA API implementation doesn't care who that someone is. Existing implementations call those functions either from the bus masters device drivers (in which case the drivers need to be IOMMU- aware, even if they use the DMA API and don't need to handle the IOMMU directly) or from the IOMMU drivers (in which case the bus masters device drivers don't have to care about the IOMMU, but without a way for drivers to handle the IOMMU directly when they need to). Possible other candidates are core IOMMU code or bus code. That doesn't sound very transparent for the device drivers. All what you describe above (attaching a device to its default domain for DMA-API) should happen in the IOMMU driver. For the device driver it should make no difference if there is an IOMMU or not. If we call arm_iommu_attach_device() from the IOMMU driver to get default transparent IOMMU handling, the function will then attach the device to the default domain with a call to iommu_attach_device(). If you have to call a function it is not transparent anymore. If a driver needs to handle the IOMMU directly, should it start by detaching the device from the ARM IOMMU domain ? We would need to be very careful with the assumptions made by the different layers, as they might not support a driver attaching a new domain to a device that already has a domain attached. I'd feel more comfortable with avoiding to attach the default domain to the device in the first place, but that might not be easily doable. The way this is solved by others is that iommu_attach_device() automatically detaches the device from its default (DMA-API) domain and attach it to the device drivers own domain. On iommu_detach_device() the device is attached back to the default domain. Joerg ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH 0/5] OMAP IOMMU fixes and IOMMU architecture questions
Hi Joerg, On Friday 04 April 2014 12:18:11 Joerg Roedel wrote: On Fri, Mar 14, 2014 at 12:00:16PM +0100, Laurent Pinchart wrote: Right, we indeed need two levels of API, one for drivers such as remoteproc that need direct control of the IOMMU, and one for drivers that only need to map buffers without any additional requirement. In the second case the drivers should ideally use the DMA mapping API not even be aware that an IOMMU is present. This would require moving the ARM mapping allocation out of the client driver. These two levels exist in the form of the DMA-API and the IOMMU-API. In fact, the IOMMU-API was created to provide a way to drivers to specifiy the IOVA-PHYS mappings on its own. I agree with you, the two levels are already present, but there's still rough edges that we need to soften. The ARM DMA API implementation requires someone to create the VA space mapping by calling arm_iommu_create_mapping() and attach devices to mappings by calling arm_iommu_attach_device(). This must only be performed for devices that use the DMA API, devices that manage their IOMMU directly will call to the IOMMU API directly. One obvious possibility is to call those functions in the IOMMU bus masters device drivers. This is pretty easy, but makes the drivers IOMMU-aware (which I'd like to avoid) and doesn't allow multiple bus masters to share a VA space mapping. Another possibility is to place the calls in the IOMMU drivers. This has the advantage of removing any IOMMU knowledge from bus masters device drivers and allowing multiple bus masters per IOMMU, but makes IOMMU management by the DMA API unconditional. Has anyone given this problem a though ? The IOMMU core or the IOMMU driver will need to know whether the driver expects to control the IOMMU directly or to have it managed transparently. As this is a software configuration I don't think the information belongs to DT. The question is, how should this information be conveyed ? The way this works on x86 is that a device driver can use the DMA-API per default. If it wants to use the IOMMU-API it has to allocate a domain and add the device it handles to this domain (it can't use DMA-API anymore then). Are drivers supposed to reimplement the DMA API internally in that case ? -- Regards, Laurent Pinchart ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH 0/5] OMAP IOMMU fixes and IOMMU architecture questions
Hi Marek, On Friday 04 April 2014 14:23:57 Marek Szyprowski wrote: Hello, I'm sorry for a delay, I've got back from my holidays and I'm still busy trying to get thought all the emails. No worries. On 2014-03-17 23:44, Laurent Pinchart wrote: On Monday 17 March 2014 14:58:24 Suman Anna wrote: On 03/16/2014 04:54 PM, Sakari Ailus wrote: On Fri, Mar 14, 2014 at 12:00:16PM +0100, Laurent Pinchart wrote: Hi Suman, (CC'ing Joerg Roedel and Marek Szyprowski for the core IOMMU discussion) On Thursday 13 March 2014 21:33:37 Suman Anna wrote: On 03/07/2014 06:46 PM, Laurent Pinchart wrote: Hello, This patch set fixes miscellaneous issues with the OMAP IOMMU driver, found when trying to port the OMAP3 ISP away from omap- iovmm to the ARM DMA API. The biggest issue is fixed by patch 5/5, while the other patches fix smaller problems that I've noticed when reading the code, without experiencing them at runtime. I'd like to take this as an opportunity to discuss OMAP IOMMU integration with the ARM DMA mapping implementation. The idea is to hide the IOMMU completely behind the DMA mapping API, making it completely transparent to drivers. Thanks for starting the discussion. A drivers will only need to allocate memory with dma_alloc_*, and behind the scene the ARM DMA mapping implementation will find out that the device is behind an IOMMU and will map the buffers through the IOMMU, returning an I/O VA address to the driver. No direct call to the OMAP IOMMU driver or to the IOMMU core should be necessary anymore. To use the IOMMU the ARM DMA implementation requires a VA mapping to be created with a call to arm_iommu_create_mapping() and to then be attached to the device with arm_iommu_attach_device(). This is currently not performed by the OMAP IOMMU driver, I have thus added that code to the OMAP3 ISP driver for now. I believe this to still be an improvement compared to the current situation, as it allows getting rid of custom memory allocation code in the OMAP3 ISP driver and custom I/O VA space management in omap-iovmm. However, that code should ideally be removed from the driver. The question is, where should it be moved to ? One possible solution would be to add the code to the OMAP IOMMU driver. However, this would require all OMAP IOMMU users to be converted to the ARM DMA API. I assume there would be issues that I don't foresee though. Yeah, I think this will pose some problems for the other main user of IOMMUs - the remoteproc devices (DSP, Dual-Cortex M3/M4 processors in OMAP4 and beyond). A remoteproc device also needs to map memory at specific addresses for its code and data sections, and not rely on a I/O VA address being given to it. The firmware sections are already linked at specific addresses, and so remoteproc needs to allocate memory, load the firmware and map into appropriate device addresses. We are doing this currently usage a combination of CMA memory to get contiguous memory (there are some restrictions for certain sections) and iommu_map/unmap API to program the MMU with these pages. This usage is different from what is expected from exchanging buffers, which can be allocated from a predefined mapping range. Even that one is tricky if we need to support different cache properties/attributes as the cache configuration is in general local to these processors. Right, we indeed need two levels of API, one for drivers such as remoteproc that need direct control of the IOMMU, and one for drivers that only need to map buffers without any additional requirement. In the second case the drivers should ideally use the DMA mapping API not even be aware that an IOMMU is present. This would require moving the ARM mapping allocation out of the client driver. Actually, I think there is one another use case, even with remoteprocs which is to runtime map buffers. This is different from the firmware management. The memory for buffers could have been allocated from other subsystems, but remoteproc would need just to manage the VA space and map. Right, although you might not always have to manage the VA space in that case. Anyway, if your driver needs to manage the VA space for the firmware, it should be able to manage the VA space for the buffers using the same IOMMU API. I've already seen some use cases which require to give a client device limited access to DMA mapping IOMMU related structures. If we agree on API, I see no problems to add such calls to dma-mapping. However in the most common case I would like to hide the presence of IOMMU from the client device at all. I agree with you. I'd like to hide the IOMMU completely by default, but some drivers will need fine-grained control over
Re: [PATCH 0/5] OMAP IOMMU fixes and IOMMU architecture questions
Hi Laurent, On Tue, Apr 08, 2014 at 02:50:38PM +0200, Laurent Pinchart wrote: I agree with you, the two levels are already present, but there's still rough edges that we need to soften. The ARM DMA API implementation requires someone to create the VA space mapping by calling arm_iommu_create_mapping() and attach devices to mappings by calling arm_iommu_attach_device(). Who is someone in this case? This must only be performed for devices that use the DMA API, devices that manage their IOMMU directly will call to the IOMMU API directly. One obvious possibility is to call those functions in the IOMMU bus masters device drivers. This is pretty easy, but makes the drivers IOMMU-aware (which I'd like to avoid) and doesn't allow multiple bus masters to share a VA space mapping. Another possibility is to place the calls in the IOMMU drivers. This has the advantage of removing any IOMMU knowledge from bus masters device drivers and allowing multiple bus masters per IOMMU, but makes IOMMU management by the DMA API unconditional. Why does that make IOMMU management by the DMA API unconditional? On x86 it works this way: The IOMMU drivers create DMA-API mappings for the devices by default. So any driver can use the DMA-API just out-of-the-box without being aware of an IOMMU. If the driver wants to deal with the IOMMU directly, it creates its own iommu-domain and attaches the device to it. The device uses the drivers domain then and not the DMA-API domain setup by the IOMMU driver. On iommu-detach the device is assigned back to its DMA-API domain. The way this works on x86 is that a device driver can use the DMA-API per default. If it wants to use the IOMMU-API it has to allocate a domain and add the device it handles to this domain (it can't use DMA-API anymore then). Are drivers supposed to reimplement the DMA API internally in that case? Usually not, but possible. Device drivers use the IOMMU-API when they want to control the io-addresses themselves. This is the case for VFIO for example, where a given range from a process address space is mapped into a device address space. In those cases the device driver usally does not need to implement its own address allocator. Regards, Joerg ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH 0/5] OMAP IOMMU fixes and IOMMU architecture questions
Hi Joerg, On Tuesday 08 April 2014 15:43:22 Joerg Roedel wrote: On Tue, Apr 08, 2014 at 02:50:38PM +0200, Laurent Pinchart wrote: I agree with you, the two levels are already present, but there's still rough edges that we need to soften. The ARM DMA API implementation requires someone to create the VA space mapping by calling arm_iommu_create_mapping() and attach devices to mappings by calling arm_iommu_attach_device(). Who is someone in this case? That's exactly the problem :-) The ARM DMA API implementation doesn't care who that someone is. Existing implementations call those functions either from the bus masters device drivers (in which case the drivers need to be IOMMU- aware, even if they use the DMA API and don't need to handle the IOMMU directly) or from the IOMMU drivers (in which case the bus masters device drivers don't have to care about the IOMMU, but without a way for drivers to handle the IOMMU directly when they need to). Possible other candidates are core IOMMU code or bus code. This must only be performed for devices that use the DMA API, devices that manage their IOMMU directly will call to the IOMMU API directly. One obvious possibility is to call those functions in the IOMMU bus masters device drivers. This is pretty easy, but makes the drivers IOMMU- aware (which I'd like to avoid) and doesn't allow multiple bus masters to share a VA space mapping. Another possibility is to place the calls in the IOMMU drivers. This has the advantage of removing any IOMMU knowledge from bus masters device drivers and allowing multiple bus masters per IOMMU, but makes IOMMU management by the DMA API unconditional. Why does that make IOMMU management by the DMA API unconditional? On x86 it works this way: The IOMMU drivers create DMA-API mappings for the devices by default. So any driver can use the DMA-API just out-of-the-box without being aware of an IOMMU. If the driver wants to deal with the IOMMU directly, it creates its own iommu-domain and attaches the device to it. The device uses the drivers domain then and not the DMA-API domain setup by the IOMMU driver. On iommu-detach the device is assigned back to its DMA-API domain. If we call arm_iommu_attach_device() from the IOMMU driver to get default transparent IOMMU handling, the function will then attach the device to the default domain with a call to iommu_attach_device(). If a driver needs to handle the IOMMU directly, should it start by detaching the device from the ARM IOMMU domain ? We would need to be very careful with the assumptions made by the different layers, as they might not support a driver attaching a new domain to a device that already has a domain attached. I'd feel more comfortable with avoiding to attach the default domain to the device in the first place, but that might not be easily doable. The way this works on x86 is that a device driver can use the DMA-API per default. If it wants to use the IOMMU-API it has to allocate a domain and add the device it handles to this domain (it can't use DMA- API anymore then). Are drivers supposed to reimplement the DMA API internally in that case? Usually not, but possible. Device drivers use the IOMMU-API when they want to control the io-addresses themselves. This is the case for VFIO for example, where a given range from a process address space is mapped into a device address space. In those cases the device driver usally does not need to implement its own address allocator. -- Regards, Laurent Pinchart ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH 0/5] OMAP IOMMU fixes and IOMMU architecture questions
On Fri, Mar 14, 2014 at 12:00:16PM +0100, Laurent Pinchart wrote: Right, we indeed need two levels of API, one for drivers such as remoteproc that need direct control of the IOMMU, and one for drivers that only need to map buffers without any additional requirement. In the second case the drivers should ideally use the DMA mapping API not even be aware that an IOMMU is present. This would require moving the ARM mapping allocation out of the client driver. These two levels exist in the form of the DMA-API and the IOMMU-API. In fact, the IOMMU-API was created to provide a way to drivers to specifiy the IOVA-PHYS mappings on its own. The IOMMU core or the IOMMU driver will need to know whether the driver expects to control the IOMMU directly or to have it managed transparently. As this is a software configuration I don't think the information belongs to DT. The question is, how should this information be conveyed ? The way this works on x86 is that a device driver can use the DMA-API per default. If it wants to use the IOMMU-API it has to allocate a domain and add the device it handles to this domain (it can't use DMA-API anymore then). Joerg ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH 0/5] OMAP IOMMU fixes and IOMMU architecture questions
Hello, I'm sorry for a delay, I've got back from my holidays and I'm still busy trying to get thought all the emails. On 2014-03-17 23:44, Laurent Pinchart wrote: Hi Suman and Sakari, On Monday 17 March 2014 14:58:24 Suman Anna wrote: On 03/16/2014 04:54 PM, Sakari Ailus wrote: On Fri, Mar 14, 2014 at 12:00:16PM +0100, Laurent Pinchart wrote: Hi Suman, (CC'ing Joerg Roedel and Marek Szyprowski for the core IOMMU discussion) On Thursday 13 March 2014 21:33:37 Suman Anna wrote: On 03/07/2014 06:46 PM, Laurent Pinchart wrote: Hello, This patch set fixes miscellaneous issues with the OMAP IOMMU driver, found when trying to port the OMAP3 ISP away from omap-iovmm to the ARM DMA API. The biggest issue is fixed by patch 5/5, while the other patches fix smaller problems that I've noticed when reading the code, without experiencing them at runtime. I'd like to take this as an opportunity to discuss OMAP IOMMU integration with the ARM DMA mapping implementation. The idea is to hide the IOMMU completely behind the DMA mapping API, making it completely transparent to drivers. Thanks for starting the discussion. A drivers will only need to allocate memory with dma_alloc_*, and behind the scene the ARM DMA mapping implementation will find out that the device is behind an IOMMU and will map the buffers through the IOMMU, returning an I/O VA address to the driver. No direct call to the OMAP IOMMU driver or to the IOMMU core should be necessary anymore. To use the IOMMU the ARM DMA implementation requires a VA mapping to be created with a call to arm_iommu_create_mapping() and to then be attached to the device with arm_iommu_attach_device(). This is currently not performed by the OMAP IOMMU driver, I have thus added that code to the OMAP3 ISP driver for now. I believe this to still be an improvement compared to the current situation, as it allows getting rid of custom memory allocation code in the OMAP3 ISP driver and custom I/O VA space management in omap-iovmm. However, that code should ideally be removed from the driver. The question is, where should it be moved to ? One possible solution would be to add the code to the OMAP IOMMU driver. However, this would require all OMAP IOMMU users to be converted to the ARM DMA API. I assume there would be issues that I don't foresee though. Yeah, I think this will pose some problems for the other main user of IOMMUs - the remoteproc devices (DSP, Dual-Cortex M3/M4 processors in OMAP4 and beyond). A remoteproc device also needs to map memory at specific addresses for its code and data sections, and not rely on a I/O VA address being given to it. The firmware sections are already linked at specific addresses, and so remoteproc needs to allocate memory, load the firmware and map into appropriate device addresses. We are doing this currently usage a combination of CMA memory to get contiguous memory (there are some restrictions for certain sections) and iommu_map/unmap API to program the MMU with these pages. This usage is different from what is expected from exchanging buffers, which can be allocated from a predefined mapping range. Even that one is tricky if we need to support different cache properties/attributes as the cache configuration is in general local to these processors. Right, we indeed need two levels of API, one for drivers such as remoteproc that need direct control of the IOMMU, and one for drivers that only need to map buffers without any additional requirement. In the second case the drivers should ideally use the DMA mapping API not even be aware that an IOMMU is present. This would require moving the ARM mapping allocation out of the client driver. Actually, I think there is one another use case, even with remoteprocs which is to runtime map buffers. This is different from the firmware management. The memory for buffers could have been allocated from other subsystems, but remoteproc would need just to manage the VA space and map. Right, although you might not always have to manage the VA space in that case. Anyway, if your driver needs to manage the VA space for the firmware, it should be able to manage the VA space for the buffers using the same IOMMU API. I've already seen some use cases which require to give a client device limited access to DMA mapping IOMMU related structures. If we agree on API, I see no problems to add such calls to dma-mapping. However in the most common case I would like to hide the presence of IOMMU from the client device at all. The IOMMU core or the IOMMU driver will need to know whether the driver expects to control the IOMMU directly or to have it managed transparently. As this is a software configuration I don't think the information belongs to DT. The question is, how should this information be conveyed? Can this be done through iommu_domain_set_attr? But that means the
Re: [PATCH 0/5] OMAP IOMMU fixes and IOMMU architecture questions
Hi Laurent, Sakari, On 03/16/2014 04:54 PM, Sakari Ailus wrote: Hi Laurent and Suman, On Fri, Mar 14, 2014 at 12:00:16PM +0100, Laurent Pinchart wrote: Hi Suman, (CC'ing Joerg Roedel and Marek Szyprowski for the core IOMMU discussion) On Thursday 13 March 2014 21:33:37 Suman Anna wrote: On 03/07/2014 06:46 PM, Laurent Pinchart wrote: Hello, This patch set fixes miscellaneous issues with the OMAP IOMMU driver, found when trying to port the OMAP3 ISP away from omap-iovmm to the ARM DMA API. The biggest issue is fixed by patch 5/5, while the other patches fix smaller problems that I've noticed when reading the code, without experiencing them at runtime. I'd like to take this as an opportunity to discuss OMAP IOMMU integration with the ARM DMA mapping implementation. The idea is to hide the IOMMU completely behind the DMA mapping API, making it completely transparent to drivers. Thanks for starting the discussion. A drivers will only need to allocate memory with dma_alloc_*, and behind the scene the ARM DMA mapping implementation will find out that the device is behind an IOMMU and will map the buffers through the IOMMU, returning an I/O VA address to the driver. No direct call to the OMAP IOMMU driver or to the IOMMU core should be necessary anymore. To use the IOMMU the ARM DMA implementation requires a VA mapping to be created with a call to arm_iommu_create_mapping() and to then be attached to the device with arm_iommu_attach_device(). This is currently not performed by the OMAP IOMMU driver, I have thus added that code to the OMAP3 ISP driver for now. I believe this to still be an improvement compared to the current situation, as it allows getting rid of custom memory allocation code in the OMAP3 ISP driver and custom I/O VA space management in omap-iovmm. However, that code should ideally be removed from the driver. The question is, where should it be moved to ? One possible solution would be to add the code to the OMAP IOMMU driver. However, this would require all OMAP IOMMU users to be converted to the ARM DMA API. I assume there would be issues that I don't foresee though. Yeah, I think this will pose some problems for the other main user of IOMMUs - the remoteproc devices (DSP, Dual-Cortex M3/M4 processors in OMAP4 and beyond). A remoteproc device also needs to map memory at specific addresses for its code and data sections, and not rely on a I/O VA address being given to it. The firmware sections are already linked at specific addresses, and so remoteproc needs to allocate memory, load the firmware and map into appropriate device addresses. We are doing this currently usage a combination of CMA memory to get contiguous memory (there are some restrictions for certain sections) and iommu_map/unmap API to program the MMU with these pages. This usage is different from what is expected from exchanging buffers, which can be allocated from a predefined mapping range. Even that one is tricky if we need to support different cache properties/attributes as the cache configuration is in general local to these processors. Right, we indeed need two levels of API, one for drivers such as remoteproc that need direct control of the IOMMU, and one for drivers that only need to map buffers without any additional requirement. In the second case the drivers should ideally use the DMA mapping API not even be aware that an IOMMU is present. This would require moving the ARM mapping allocation out of the client driver. Actually, I think there is one another use case, even with remoteprocs which is to runtime map buffers. This is different from the firmware management. The memory for buffers could have been allocated from other subsystems, but remoteproc would need just to manage the VA space and map. The IOMMU core or the IOMMU driver will need to know whether the driver expects to control the IOMMU directly or to have it managed transparently. As this is a software configuration I don't think the information belongs to DT. The question is, how should this information be conveyed? Can this be done through iommu_domain_set_attr? But that means the client driver has to dictate this. The iommu driver can be configured appropriately based on this. The driver would need to know that, I think. Currently the DMA mapping API doesn't allow explicit addressing to IOVA address space AFAIK. The IOMMU API does. It's a good question how to do this as I don't think there's even a way for the driver to explicitly obtain access to the IOMMU. The virtual address space allocation would need to take into account that some of the address space is actually mapped outside it. The iova library can do this already. I'm not even sure whether the OMAP IOMMU driver would be the best place to put that code. Ideally VA spaces should be created by the platform somehow, and mapping of devices to IOMMUs should be handled by the IOMMU core instead of the IOMMU drivers. We're not there yet, and the path might not
Re: [PATCH 0/5] OMAP IOMMU fixes and IOMMU architecture questions
Hi Suman and Sakari, On Monday 17 March 2014 14:58:24 Suman Anna wrote: On 03/16/2014 04:54 PM, Sakari Ailus wrote: On Fri, Mar 14, 2014 at 12:00:16PM +0100, Laurent Pinchart wrote: Hi Suman, (CC'ing Joerg Roedel and Marek Szyprowski for the core IOMMU discussion) On Thursday 13 March 2014 21:33:37 Suman Anna wrote: On 03/07/2014 06:46 PM, Laurent Pinchart wrote: Hello, This patch set fixes miscellaneous issues with the OMAP IOMMU driver, found when trying to port the OMAP3 ISP away from omap-iovmm to the ARM DMA API. The biggest issue is fixed by patch 5/5, while the other patches fix smaller problems that I've noticed when reading the code, without experiencing them at runtime. I'd like to take this as an opportunity to discuss OMAP IOMMU integration with the ARM DMA mapping implementation. The idea is to hide the IOMMU completely behind the DMA mapping API, making it completely transparent to drivers. Thanks for starting the discussion. A drivers will only need to allocate memory with dma_alloc_*, and behind the scene the ARM DMA mapping implementation will find out that the device is behind an IOMMU and will map the buffers through the IOMMU, returning an I/O VA address to the driver. No direct call to the OMAP IOMMU driver or to the IOMMU core should be necessary anymore. To use the IOMMU the ARM DMA implementation requires a VA mapping to be created with a call to arm_iommu_create_mapping() and to then be attached to the device with arm_iommu_attach_device(). This is currently not performed by the OMAP IOMMU driver, I have thus added that code to the OMAP3 ISP driver for now. I believe this to still be an improvement compared to the current situation, as it allows getting rid of custom memory allocation code in the OMAP3 ISP driver and custom I/O VA space management in omap-iovmm. However, that code should ideally be removed from the driver. The question is, where should it be moved to ? One possible solution would be to add the code to the OMAP IOMMU driver. However, this would require all OMAP IOMMU users to be converted to the ARM DMA API. I assume there would be issues that I don't foresee though. Yeah, I think this will pose some problems for the other main user of IOMMUs - the remoteproc devices (DSP, Dual-Cortex M3/M4 processors in OMAP4 and beyond). A remoteproc device also needs to map memory at specific addresses for its code and data sections, and not rely on a I/O VA address being given to it. The firmware sections are already linked at specific addresses, and so remoteproc needs to allocate memory, load the firmware and map into appropriate device addresses. We are doing this currently usage a combination of CMA memory to get contiguous memory (there are some restrictions for certain sections) and iommu_map/unmap API to program the MMU with these pages. This usage is different from what is expected from exchanging buffers, which can be allocated from a predefined mapping range. Even that one is tricky if we need to support different cache properties/attributes as the cache configuration is in general local to these processors. Right, we indeed need two levels of API, one for drivers such as remoteproc that need direct control of the IOMMU, and one for drivers that only need to map buffers without any additional requirement. In the second case the drivers should ideally use the DMA mapping API not even be aware that an IOMMU is present. This would require moving the ARM mapping allocation out of the client driver. Actually, I think there is one another use case, even with remoteprocs which is to runtime map buffers. This is different from the firmware management. The memory for buffers could have been allocated from other subsystems, but remoteproc would need just to manage the VA space and map. Right, although you might not always have to manage the VA space in that case. Anyway, if your driver needs to manage the VA space for the firmware, it should be able to manage the VA space for the buffers using the same IOMMU API. The IOMMU core or the IOMMU driver will need to know whether the driver expects to control the IOMMU directly or to have it managed transparently. As this is a software configuration I don't think the information belongs to DT. The question is, how should this information be conveyed? Can this be done through iommu_domain_set_attr? But that means the client driver has to dictate this. The iommu driver can be configured appropriately based on this. The problem with that approach is that IOMMU client (bus master) drivers would need a pointer to the IOMMU domain. That's what we want to avoid in the first place :-) The driver would need to know that, I think. The IOMMU client driver would know that. Or, to be accurate, it should either know that it wants to manage the IOMMU directly, or should ignore
Re: [PATCH 0/5] OMAP IOMMU fixes and IOMMU architecture questions
Hi Laurent and Suman, On Fri, Mar 14, 2014 at 12:00:16PM +0100, Laurent Pinchart wrote: Hi Suman, (CC'ing Joerg Roedel and Marek Szyprowski for the core IOMMU discussion) On Thursday 13 March 2014 21:33:37 Suman Anna wrote: On 03/07/2014 06:46 PM, Laurent Pinchart wrote: Hello, This patch set fixes miscellaneous issues with the OMAP IOMMU driver, found when trying to port the OMAP3 ISP away from omap-iovmm to the ARM DMA API. The biggest issue is fixed by patch 5/5, while the other patches fix smaller problems that I've noticed when reading the code, without experiencing them at runtime. I'd like to take this as an opportunity to discuss OMAP IOMMU integration with the ARM DMA mapping implementation. The idea is to hide the IOMMU completely behind the DMA mapping API, making it completely transparent to drivers. Thanks for starting the discussion. A drivers will only need to allocate memory with dma_alloc_*, and behind the scene the ARM DMA mapping implementation will find out that the device is behind an IOMMU and will map the buffers through the IOMMU, returning an I/O VA address to the driver. No direct call to the OMAP IOMMU driver or to the IOMMU core should be necessary anymore. To use the IOMMU the ARM DMA implementation requires a VA mapping to be created with a call to arm_iommu_create_mapping() and to then be attached to the device with arm_iommu_attach_device(). This is currently not performed by the OMAP IOMMU driver, I have thus added that code to the OMAP3 ISP driver for now. I believe this to still be an improvement compared to the current situation, as it allows getting rid of custom memory allocation code in the OMAP3 ISP driver and custom I/O VA space management in omap-iovmm. However, that code should ideally be removed from the driver. The question is, where should it be moved to ? One possible solution would be to add the code to the OMAP IOMMU driver. However, this would require all OMAP IOMMU users to be converted to the ARM DMA API. I assume there would be issues that I don't foresee though. Yeah, I think this will pose some problems for the other main user of IOMMUs - the remoteproc devices (DSP, Dual-Cortex M3/M4 processors in OMAP4 and beyond). A remoteproc device also needs to map memory at specific addresses for its code and data sections, and not rely on a I/O VA address being given to it. The firmware sections are already linked at specific addresses, and so remoteproc needs to allocate memory, load the firmware and map into appropriate device addresses. We are doing this currently usage a combination of CMA memory to get contiguous memory (there are some restrictions for certain sections) and iommu_map/unmap API to program the MMU with these pages. This usage is different from what is expected from exchanging buffers, which can be allocated from a predefined mapping range. Even that one is tricky if we need to support different cache properties/attributes as the cache configuration is in general local to these processors. Right, we indeed need two levels of API, one for drivers such as remoteproc that need direct control of the IOMMU, and one for drivers that only need to map buffers without any additional requirement. In the second case the drivers should ideally use the DMA mapping API not even be aware that an IOMMU is present. This would require moving the ARM mapping allocation out of the client driver. The IOMMU core or the IOMMU driver will need to know whether the driver expects to control the IOMMU directly or to have it managed transparently. As this is a software configuration I don't think the information belongs to DT. The question is, how should this information be conveyed ? The driver would need to know that, I think. Currently the DMA mapping API doesn't allow explicit addressing to IOVA address space AFAIK. The IOMMU API does. It's a good question how to do this as I don't think there's even a way for the driver to explicitly obtain access to the IOMMU. The virtual address space allocation would need to take into account that some of the address space is actually mapped outside it. The iova library can do this already. I'm not even sure whether the OMAP IOMMU driver would be the best place to put that code. Ideally VA spaces should be created by the platform somehow, and mapping of devices to IOMMUs should be handled by the IOMMU core instead of the IOMMU drivers. We're not there yet, and the path might not be straightforward, hence this attempt to start a constructive discussion. A second completely unrelated problem that I'd like to get feedback on is the suspend/resume support in the OMAP IOMMU driver, or rather the lack thereof. The driver exports omap_iommu_save_ctx() and omap_iommu_restore_ctx() functions and expects the IOMMU users to call
Re: [PATCH 0/5] OMAP IOMMU fixes and IOMMU architecture questions
Hi Suman, (CC'ing Joerg Roedel and Marek Szyprowski for the core IOMMU discussion) On Thursday 13 March 2014 21:33:37 Suman Anna wrote: On 03/07/2014 06:46 PM, Laurent Pinchart wrote: Hello, This patch set fixes miscellaneous issues with the OMAP IOMMU driver, found when trying to port the OMAP3 ISP away from omap-iovmm to the ARM DMA API. The biggest issue is fixed by patch 5/5, while the other patches fix smaller problems that I've noticed when reading the code, without experiencing them at runtime. I'd like to take this as an opportunity to discuss OMAP IOMMU integration with the ARM DMA mapping implementation. The idea is to hide the IOMMU completely behind the DMA mapping API, making it completely transparent to drivers. Thanks for starting the discussion. A drivers will only need to allocate memory with dma_alloc_*, and behind the scene the ARM DMA mapping implementation will find out that the device is behind an IOMMU and will map the buffers through the IOMMU, returning an I/O VA address to the driver. No direct call to the OMAP IOMMU driver or to the IOMMU core should be necessary anymore. To use the IOMMU the ARM DMA implementation requires a VA mapping to be created with a call to arm_iommu_create_mapping() and to then be attached to the device with arm_iommu_attach_device(). This is currently not performed by the OMAP IOMMU driver, I have thus added that code to the OMAP3 ISP driver for now. I believe this to still be an improvement compared to the current situation, as it allows getting rid of custom memory allocation code in the OMAP3 ISP driver and custom I/O VA space management in omap-iovmm. However, that code should ideally be removed from the driver. The question is, where should it be moved to ? One possible solution would be to add the code to the OMAP IOMMU driver. However, this would require all OMAP IOMMU users to be converted to the ARM DMA API. I assume there would be issues that I don't foresee though. Yeah, I think this will pose some problems for the other main user of IOMMUs - the remoteproc devices (DSP, Dual-Cortex M3/M4 processors in OMAP4 and beyond). A remoteproc device also needs to map memory at specific addresses for its code and data sections, and not rely on a I/O VA address being given to it. The firmware sections are already linked at specific addresses, and so remoteproc needs to allocate memory, load the firmware and map into appropriate device addresses. We are doing this currently usage a combination of CMA memory to get contiguous memory (there are some restrictions for certain sections) and iommu_map/unmap API to program the MMU with these pages. This usage is different from what is expected from exchanging buffers, which can be allocated from a predefined mapping range. Even that one is tricky if we need to support different cache properties/attributes as the cache configuration is in general local to these processors. Right, we indeed need two levels of API, one for drivers such as remoteproc that need direct control of the IOMMU, and one for drivers that only need to map buffers without any additional requirement. In the second case the drivers should ideally use the DMA mapping API not even be aware that an IOMMU is present. This would require moving the ARM mapping allocation out of the client driver. The IOMMU core or the IOMMU driver will need to know whether the driver expects to control the IOMMU directly or to have it managed transparently. As this is a software configuration I don't think the information belongs to DT. The question is, how should this information be conveyed ? I'm not even sure whether the OMAP IOMMU driver would be the best place to put that code. Ideally VA spaces should be created by the platform somehow, and mapping of devices to IOMMUs should be handled by the IOMMU core instead of the IOMMU drivers. We're not there yet, and the path might not be straightforward, hence this attempt to start a constructive discussion. A second completely unrelated problem that I'd like to get feedback on is the suspend/resume support in the OMAP IOMMU driver, or rather the lack thereof. The driver exports omap_iommu_save_ctx() and omap_iommu_restore_ctx() functions and expects the IOMMU users to call them directly. This is really hackish to say the least. A proper suspend/resume implementation shouldn't be difficult to implement, and I'm wondering whether the lack of proper power management support comes from historical reasons only, or if there are problems I might not have considered. Agreed, the current code definitely needs a cleanup and better organization (like folding into runtime suspend ops). The main thing is that we need the IOMMU to be activated as long as its parent device is active (like the omap3isp or a remoteproc device). And this behaviour needs separation from configuring it and
Re: [PATCH 0/5] OMAP IOMMU fixes and IOMMU architecture questions
Hi Laurent, On 03/07/2014 06:46 PM, Laurent Pinchart wrote: Hello, This patch set fixes miscellaneous issues with the OMAP IOMMU driver, found when trying to port the OMAP3 ISP away from omap-iovmm to the ARM DMA API. The biggest issue is fixed by patch 5/5, while the other patches fix smaller problems that I've noticed when reading the code, without experiencing them at runtime. I'd like to take this as an opportunity to discuss OMAP IOMMU integration with the ARM DMA mapping implementation. The idea is to hide the IOMMU completely behind the DMA mapping API, making it completely transparent to drivers. Thanks for starting the discussion. A drivers will only need to allocate memory with dma_alloc_*, and behind the scene the ARM DMA mapping implementation will find out that the device is behind an IOMMU and will map the buffers through the IOMMU, returning an I/O VA address to the driver. No direct call to the OMAP IOMMU driver or to the IOMMU core should be necessary anymore. To use the IOMMU the ARM DMA implementation requires a VA mapping to be created with a call to arm_iommu_create_mapping() and to then be attached to the device with arm_iommu_attach_device(). This is currently not performed by the OMAP IOMMU driver, I have thus added that code to the OMAP3 ISP driver for now. I believe this to still be an improvement compared to the current situation, as it allows getting rid of custom memory allocation code in the OMAP3 ISP driver and custom I/O VA space management in omap-iovmm. However, that code should ideally be removed from the driver. The question is, where should it be moved to ? One possible solution would be to add the code to the OMAP IOMMU driver. However, this would require all OMAP IOMMU users to be converted to the ARM DMA API. I assume there would be issues that I don't foresee though. Yeah, I think this will pose some problems for the other main user of IOMMUs - the remoteproc devices (DSP, Dual-Cortex M3/M4 processors in OMAP4 and beyond). A remoteproc device also needs to map memory at specific addresses for its code and data sections, and not rely on a I/O VA address being given to it. The firmware sections are already linked at specific addresses, and so remoteproc needs to allocate memory, load the firmware and map into appropriate device addresses. We are doing this currently usage a combination of CMA memory to get contiguous memory (there are some restrictions for certain sections) and iommu_map/unmap API to program the MMU with these pages. This usage is different from what is expected from exchanging buffers, which can be allocated from a predefined mapping range. Even that one is tricky if we need to support different cache properties/attributes as the cache configuration is in general local to these processors. I'm not even sure whether the OMAP IOMMU driver would be the best place to put that code. Ideally VA spaces should be created by the platform somehow, and mapping of devices to IOMMUs should be handled by the IOMMU core instead of the IOMMU drivers. We're not there yet, and the path might not be straightforward, hence this attempt to start a constructive discussion. A second completely unrelated problem that I'd like to get feedback on is the suspend/resume support in the OMAP IOMMU driver, or rather the lack thereof. The driver exports omap_iommu_save_ctx() and omap_iommu_restore_ctx() functions and expects the IOMMU users to call them directly. This is really hackish to say the least. A proper suspend/resume implementation shouldn't be difficult to implement, and I'm wondering whether the lack of proper power management support comes from historical reasons only, or if there are problems I might not have considered. Agreed, the current code definitely needs a cleanup and better organization (like folding into runtime suspend ops). The main thing is that we need the IOMMU to be activated as long as its parent device is active (like the omap3isp or a remoteproc device). And this behaviour needs separation from configuring it and programming for the first time. Only a user can dictate when an IOMMU is not needed. So we either need an API to activate or have access to the iommu's dev object somehow so that we can use the pm_runtime_get/put API to hold a usage counter and let the runtime suspend ops take care of save/restore without tearing down the domain or detaching the device. Last but not least, the patches are available at git://linuxtv.org/pinchartl/media.git omap3isp/dma along with a patch series that ports the OMAP3 ISP driver to the DMA API. I will submit that one for review once the IOMMU patches get accepted and after fixing a couple of remaining bugs (I'm aware that I have broken userspace PFNMAP buffers). I have looked at your tree, and as you already stated, it seems to be right intermediate solution for now to get rid of omap-iovmm. regards Suman Laurent Pinchart (5): iommu/omap: Use
Re: [PATCH 0/5] OMAP IOMMU fixes and IOMMU architecture questions
Hi Laurent, Laurent Pinchart wrote: along with a patch series that ports the OMAP3 ISP driver to the DMA API. I will submit that one for review once the IOMMU patches get accepted and after fixing a couple of remaining bugs (I'm aware that I have broken userspace PFNMAP buffers). Laurent Pinchart (5): iommu/omap: Use the cache cleaning API iommu/omap: Fix 'no page for' debug message in flush_iotlb_page() iommu/omap: Flush the TLB only after updating translation table entries iommu/omap: Remove comment about supporting single page mappings only iommu/omap: Fix map protection value handling drivers/iommu/omap-iommu.c | 68 -- 1 file changed, 23 insertions(+), 45 deletions(-) Until I have time to read and think about the rest, for the series, Reviewed-by: Sakari Ailus sakari.ai...@iki.fi -- Sakari Ailus sakari.ai...@iki.fi ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
[PATCH 0/5] OMAP IOMMU fixes and IOMMU architecture questions
Hello, This patch set fixes miscellaneous issues with the OMAP IOMMU driver, found when trying to port the OMAP3 ISP away from omap-iovmm to the ARM DMA API. The biggest issue is fixed by patch 5/5, while the other patches fix smaller problems that I've noticed when reading the code, without experiencing them at runtime. I'd like to take this as an opportunity to discuss OMAP IOMMU integration with the ARM DMA mapping implementation. The idea is to hide the IOMMU completely behind the DMA mapping API, making it completely transparent to drivers. A drivers will only need to allocate memory with dma_alloc_*, and behind the scene the ARM DMA mapping implementation will find out that the device is behind an IOMMU and will map the buffers through the IOMMU, returning an I/O VA address to the driver. No direct call to the OMAP IOMMU driver or to the IOMMU core should be necessary anymore. To use the IOMMU the ARM DMA implementation requires a VA mapping to be created with a call to arm_iommu_create_mapping() and to then be attached to the device with arm_iommu_attach_device(). This is currently not performed by the OMAP IOMMU driver, I have thus added that code to the OMAP3 ISP driver for now. I believe this to still be an improvement compared to the current situation, as it allows getting rid of custom memory allocation code in the OMAP3 ISP driver and custom I/O VA space management in omap-iovmm. However, that code should ideally be removed from the driver. The question is, where should it be moved to ? One possible solution would be to add the code to the OMAP IOMMU driver. However, this would require all OMAP IOMMU users to be converted to the ARM DMA API. I assume there would be issues that I don't foresee though. I'm not even sure whether the OMAP IOMMU driver would be the best place to put that code. Ideally VA spaces should be created by the platform somehow, and mapping of devices to IOMMUs should be handled by the IOMMU core instead of the IOMMU drivers. We're not there yet, and the path might not be straightforward, hence this attempt to start a constructive discussion. A second completely unrelated problem that I'd like to get feedback on is the suspend/resume support in the OMAP IOMMU driver, or rather the lack thereof. The driver exports omap_iommu_save_ctx() and omap_iommu_restore_ctx() functions and expects the IOMMU users to call them directly. This is really hackish to say the least. A proper suspend/resume implementation shouldn't be difficult to implement, and I'm wondering whether the lack of proper power management support comes from historical reasons only, or if there are problems I might not have considered. Last but not least, the patches are available at git://linuxtv.org/pinchartl/media.git omap3isp/dma along with a patch series that ports the OMAP3 ISP driver to the DMA API. I will submit that one for review once the IOMMU patches get accepted and after fixing a couple of remaining bugs (I'm aware that I have broken userspace PFNMAP buffers). Laurent Pinchart (5): iommu/omap: Use the cache cleaning API iommu/omap: Fix 'no page for' debug message in flush_iotlb_page() iommu/omap: Flush the TLB only after updating translation table entries iommu/omap: Remove comment about supporting single page mappings only iommu/omap: Fix map protection value handling drivers/iommu/omap-iommu.c | 68 -- 1 file changed, 23 insertions(+), 45 deletions(-) -- Regards, Laurent Pinchart ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
