On 13/05/2019 13:42, Jean-Philippe Brucker wrote:
On 13/05/2019 08:09, Pankaj Bansal wrote:
Hi Jean,
-----Original Message-----
From: Jean-Philippe Brucker <[email protected]>
Sent: Friday, 10 May, 2019 07:07 PM
To: Pankaj Bansal <[email protected]>; Will Deacon
<[email protected]>; Robin Murphy <[email protected]>; Joerg
Roedel <[email protected]>
Cc: [email protected]; Varun Sethi <[email protected]>; linux-
[email protected]; Nipun Gupta <[email protected]>
Subject: Re: [ARM SMMU] Dynamic StreamID allocation
On 10/05/2019 13:33, Pankaj Bansal wrote:
Hi Will/Robin/Joerg,
I am s/w engineer from NXP India Pvt. Ltd.
We are using SMMU-V3 in one of NXP SOC.
I have a question about the SMMU Stream ID allocation in linux.
Right now the Stream IDs allocated to a device are mapped via device tree to
the device.
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Felix
ir.bootlin.com%2Flinux%2Flatest%2Fsource%2FDocumentation%2Fdevicetree%
2Fbindings%2Fiommu%2Farm%2Csmmu-
v3.txt%23L39&data=02%7C01%7Cpankaj
.bansal%40nxp.com%7C3cbe8bd4827e425afd0f08d6d54c925e%7C686ea1d3b
c2b4c6
fa92cd99c5c301635%7C0%7C0%7C636930922220665343&sdata=vIG5u5n
XR5iRp
uuuGjeFxKBtA5f5ohf91znXX0QWm1c%3D&reserved=0
As the device tree is passed from bootloader to linux, we detect all the stream
IDs needed by a device in bootloader and add their IDs in respective device
nodes.
For each PCIE Endpoint (a unique BDF (Bus Device Function)) on PCIE bus, we
are assigning a unique Stream ID in bootloader.
However, this poses an issue with PCIE hot plug.
If we plug in a pcie device while linux is running, a unique BDF is assigned to
the device, for which there is no stream ID in device tree.
How can this problem be solved in linux?
Assuming the streamID associated to a BDF is predictable (streamID = BDF
+ constant), using the iommu-map property should just work:
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Felixir.boo
tlin.com%2Flinux%2Flatest%2Fsource%2FDocumentation%2Fdevicetree%2Fbind
ings%2Fpci%2Fpci-
iommu.txt&data=02%7C01%7Cpankaj.bansal%40nxp.com%7C3cbe8bd482
7e425afd0f08d6d54c925e%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0
%7C636930922220665343&sdata=GkkovEnvhd5dN%2BGdh%2FnKCyW5Cd
EnLDP3cWTrk%2B%2FO7EQ%3D&reserved=0
It describes the streamIDs of all possible BDFs, including hotplugged functions.
You mean that we should increase the "length" parameter (in
(rid-base,iommu,iommu-base,length) touple) ?
This would cater to any *new* Bus Device Function being detected on PCIE bus?
Is that right ?
No, the iommu-map solution only works when you can predict at boot time
which streamID will correspond to any BDF, for example if the PCIe
controller or firmware automatically assigns streamID = BDF.
Right now when we make iommu-map in bootloader, we are giving one RID per BDF:
https://elixir.bootlin.com/u-boot/latest/source/drivers/pci/pcie_layerscape_fixup.c#L168
Right, that won't work with hotplug. You can't derive a unique streamID
from any BDF at boot, if the streamID range is limited to 16 values on
this hardware.
Furthermore, does U-Boot enumerate all possible SR-IOV capabilities, or
is this already broken regardless of hotplug?
Maybe you could move this allocator to the Linux Layerscape driver, and
call iommu_fwspec_add_ids() from there, rather than using iommu-map? Not
sure how feasible that is, but it might still be the simplest.
Assuming there's still the same silly 7-bit limitation as LS2085, you
could in theory carve out that entire space of ICIDs to allow a static
mapping like this:
iommu-map-mask = <0x0f07>;
iommu-map = <0x000 &smmu 0x00 0x8>,
<0x100 &smmu 0x08 0x8>,
<0x200 &smmu 0x10 0x8>,
<0x300 &smmu 0x18 0x8>,
<0x400 &smmu 0x20 0x8>,
<0x500 &smmu 0x28 0x8>,
<0x600 &smmu 0x30 0x8>,
<0x700 &smmu 0x38 0x8>,
<0x800 &smmu 0x40 0x8>,
<0x900 &smmu 0x48 0x8>,
<0xa00 &smmu 0x50 0x8>,
<0xb00 &smmu 0x58 0x8>,
<0xc00 &smmu 0x60 0x8>,
<0xd00 &smmu 0x68 0x8>,
<0xe00 &smmu 0x70 0x8>,
<0xf00 &smmu 0x78 0x8>;
That gives you 16 buses before IDs start aliasing awkwardly (any devices
sharing a bus will alias per-function, but that's probably acceptable
since they'd likely get grouped together anyway).
Either way the IOMMU layer itself is actually relatively easy-going
these days. Even with the existing approach, if you could just update
Linux's internal idea of the firmware data before the bus_add_device()
call happens for the hotplugged device, then things should just work.
You're also going to face the exact same problem for ITS Device IDs,
though, and things may be less forgiving there.
Robin.
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
