There are some devices in which a hypervisor may only allow 1 DMA window to exist at a time, and in those cases, a DDW is never created to them, since the default DMA window keeps using this resource.
LoPAR recommends this procedure: 1. Remove the default DMA window, 2. Query for which configs the DDW can be created, 3. Create a DDW. Patch #1: Create defines for outputs of ibm,ddw-applicable, so it's easier to identify them. Patch #2: - After LoPAR level 2.8, there is an extension that can make ibm,query-pe-dma-windows to have 6 outputs instead of 5. This changes the order of the outputs, and that can cause some trouble. - query_ddw() was updated to check how many outputs the ibm,query-pe-dma-windows is supposed to have, update the rtas_call() and deal correctly with the outputs in both cases. - This patch looks somehow unrelated to the series, but it can avoid future problems on DDW creation. Patch #3 moves the window-removing code from remove_ddw() to remove_dma_window(), creating a way to delete any DMA window, so it can be used to delete the default DMA window. Patch #4 makes use of the remove_dma_window() from patch #3 to remove the default DMA window before query_ddw(). It also implements a new rtas call to recover the default DMA window, in case anything fails after it was removed, and a DDW couldn't be created. Patch #5: Instead of destroying the created DDW if it doesn't map the whole partition, make use of it instead of the default DMA window as it improves performance. Patch #6: Does some renaming of 'direct window' to 'dma window', given the DDW created can now be also used in indirect mapping if direct mapping is not available. All patches were tested into an LPAR with an Ethernet VF: 4005:01:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function] Patch #5 It was tested with a 64GB DDW which did not map the whole partition (128G). Performance improvement noticed by using the DDW instead of the default DMA window: 64 thread write throughput: +203.0% 64 thread read throughput: +17.5% 1 thread write throughput: +20.5% 1 thread read throughput: +3.43% Average write latency: -23.0% Average read latency: -2.26% --- Changes since v2: - Change the way ibm,ddw-extensions is accessed, using a proper function instead of doing this inline everytime it's used. - Remove previous patch #6, as it doesn't look like it would be useful. - Add new patch, for changing names from direct* to dma*, as indirect mapping can be used from now on. - Fix some typos, corrects some define usage. - v2 link: http://patchwork.ozlabs.org/project/linuxppc-dev/list/?series=185433&state=%2A&archive=both Changes since v1: - Add defines for ibm,ddw-applicable and ibm,ddw-extensions outputs - Merge aux function query_ddw_out_sz() into query_ddw() - Merge reset_dma_window() patch (prev. #2) into remove default DMA window patch (#4). - Keep device_node *np name instead of using pdn in remove_*() - Rename 'device_node *pdn' into 'parent' in new functions - Rename dfl_win to default_win - Only remove the default DMA window if there is no window available in first query. - Check if default DMA window can be restored before removing it. - Fix 'unitialized use' (found by travis mpe:ci-test) - New patches #5 and #6 - v1 link: http://patchwork.ozlabs.org/project/linuxppc-dev/list/?series=184420&state=%2A&archive=both Special thanks for Alexey Kardashevskiy and Oliver O'Halloran for the feedback provided! Leonardo Bras (6): powerpc/pseries/iommu: Create defines for operations in ibm,ddw-applicable powerpc/pseries/iommu: Update call to ibm,query-pe-dma-windows powerpc/pseries/iommu: Move window-removing part of remove_ddw into remove_dma_window powerpc/pseries/iommu: Remove default DMA window before creating DDW powerpc/pseries/iommu: Make use of DDW even if it does not map the partition powerpc/pseries/iommu: Rename "direct window" to "dma window" arch/powerpc/platforms/pseries/iommu.c | 379 ++++++++++++++++++------- 1 file changed, 269 insertions(+), 110 deletions(-) -- 2.25.4