On 02/02/2018 11:56, Peter Xu wrote:
> On Thu, Feb 01, 2018 at 07:51:31PM +0000, Dr. David Alan Gilbert wrote:
>> * Peter Xu (pet...@redhat.com) wrote:
>>> In the past, we prioritized IOMMU migration so that we have such a
>>> priority order:
>>> IOMMU > PCI Devices
>>> When migrating a guest with both vIOMMU and pcie-root-port, we'll always
>>> migrate vIOMMU first, since pcie-root-port will be seen to have the same
>>> priority of general PCI devices.
>>> That's problematic.
>>> The thing is that PCI bus number information is stored in the root port,
>>> and that is needed by vIOMMU during post_load(), e.g., to figure out
>>> context entry for a device. If we don't have correct bus numbers for
>>> devices, we won't be able to recover device state of the DMAR memory
>>> regions, and things will be messed up.
>>> So let's boost the PCIe root ports to be even with higher priority:
>>> PCIe Root Port > IOMMU > PCI Devices
>>> A smoke test shows that this patch fixes bug 1538953.
>> Two questions (partially overlapping with what I replied to Michaels):
>> a) What happens with multiple IOMMUs?
> If there are more IOMMUs, then the patch will let all the vIOMMUs be
> migrated after pcie root ports.
> But a more true answer is that: I don't really know. :)
> Because I even don't know how multiple vIOMMUs will coop with each
> other, especially nested.
I am not aware of "nested" IOMMUs. Multiple IOMMUs work together
by dividing the bus ranges, when each of them declares in the
corresponding ACPI table the bus/device/range is in charge of.
However there was a kernel bug some time ago preventing several
IOMMUs to work together, I am not sure the problem is solved yet.
In nested case, maybe there will be
> dependency between vIOMMUs, but I'll avoid thinking about that until
> we support more than one vIOMMUs.
>> b) What happens with multiple root ports?
> Same answer as previous one: all of them will be migrated before any
> Note that IMHO we don't care which pcie root port is migrated first -
> IMHO they should not depend on each other, but Marcel may correct me.
Right, each Root Port is independent from each other.
>> c) How correct is this ordering on different implementations
>> (e.g. ARM/Power/etc)
> Currently it won't affect since Intel IOMMU is the only user for
> MIG_PRI_IOMMU. After SMMU is merged it may affect (if it uses this
> bit), but IMHO it's fine too as long as pcie root ports won't depend
> on anything related to SMMU.