On 20/1/26 04:37, Jason Gunthorpe wrote:
On Mon, Jan 19, 2026 at 12:00:47PM +1100, Alexey Kardashevskiy wrote:
On 18/1/26 02:43, Jason Gunthorpe wrote:
On Sat, Jan 17, 2026 at 03:54:52PM +1100, Alexey Kardashevskiy wrote:
I am trying this with TEE-IO on AMD SEV and hitting problems.
My understanding is that if you want to use SEV today you also have to
use the kernel command line parameter to force 4k IOMMU pages?
No, not only 4K. I do not enforce any page size by default so it is
"everything but 512G", only when the device is "accepted" - I unmap
everything in QEMU, "accept" the device, then map everything again
but this time IOMMU uses the (4K|2M) pagemask and takes RMP entry
sizes into account.
I mean, I'm telling you how things work in upstream right now. If you
want this to work you set the 4k only cmdline option and it
works. None of what you are describing is upstream. Upstream does not
support > 4K IOPTEs if RMP is used.
ah, that. Well, even now if you force swiotlb, then IOMMU should be able to use
huge pages. But ok, point taken.
Now, from time to time the guest will share 4K pages which makes the
host OS smash NPT's 2MB PDEs to 4K PTEs, and 2M RMP entries to 4K
RMP entries, and since the IOMMU performs RMP checks - IOMMU PDEs
have to use the same granularity as NPT and RMP.
IMHO this is a bad hardware choice, it is going to make some very
troublesome software, so sigh.
afaik the Other OS is still not using 2MB pages (or does but not much?) and
runs on the same hw :)
Sure we can force some rules in Linux to make the sw simpler though.
I mean that the HW requires multiple SW controlled tables to all be
sizes must be matched. Instead the HW should read all the tables and
compute the appropriate smallest size automatically.
Not sure I follow. IOMMU table matches the QEMU page table, it is two tables
already and IOMMU cannot just blindly use 2M PTEs if the guest is backed with
4K pages.
Doing it at mapping time doesn't seem right to me, AFAICT the RMP can
change dynamically whenever the guest decides to change the
private/shared status of memory?
The guest requests page state conversion which makes KVM change RMPs
and potentially smash huge pages, the guest only (in)validates the
RMP entry but does not change ASID+GPA+otherbits, the host does. But
yeah a race is possible here.
It is not even a "race", it is just something the VMM has to deal with
whenever the RMP changes.
My expectation for AMD was that the VMM would be monitoring the RMP
granularity and use cut or "increase/decrease page size" through
iommupt to adjust the S2 mapping so it works with these RMP
limitations.
Those don't fully exist yet, but they are in the plans.
I remember the talks about hitless smashing but in case of RMPs atomic xchg is
not enough (we have a HW engine for that).
I don't think you need hitless here, if the guest is doing
encrpyed/decrypted conversions then it can be expected to not do DMA
at the same time, or at least it is OK if DMA during this period
fails.
The guest converts only a handful of 4Ks (say, the guest userspace wants to read
certificates from guest-os->host-os->fw) and only that converted part is not
expected for DMA but the rest of 2MB page is DMA-able.
So long as the VMM gets a chance to fix the iommu before the guest
understands the RMP change is completed it would be OK.
The IOMMU HW needs to understand the change too. After I smash IO PDE, there is
a small window before smashing an RMP entry when incoming trafic may hit
not-converted part of a 2MB page and RMP check in the IOMMU will fail. That
mentioned above HW+FW engine can stall DMA for a few ms while it is smashing
things.
I'm assuming there is a VMM call involved here?
Yes.
It assumes that the VMM is continually aware of what all the RMP PTEs
look like and when they are changing so it can make the required
adjustments.
The flow would be some thing like..
1) Create an IOAS
2) Create a HWPT. If there is some known upper bound on RMP/etc page
size then limit the HWPT page size to the upper bound
3) Map stuff into the ioas
4) Build the RMP/etc and map ranges of page granularity
5) Call iommufd to adjust the page size within ranges
Say, I hotplug a device into a VM with a mix of 4K and 2M RMPs. QEMU
will ask iommufd to map everything (and that would be 2M/1G), should
then QEMU ask KVM to walk through ranges and call iommufd directly
to make IO PDEs/PTEs match RMPs?
Yes, assuming it isn't already tracking it on its own.
I mean, I have to do the KVM->iommufd part anyway when 2M->4K
smashing happens in runtime but the initial mapping could be simpler
if iommufd could check RMP.
Yeah, but then we have to implement two completely different
flows. You can't do without the above since you have to deal with
dynamic changes to the RMP by the guest.
Making it so map can happen right the first time is an
optimization. Lets get the basics and then think about optimising. I
think optimizing hot plug is not important, nor do I know how good an
optimization this would even be.
Got it.
For the time being I do bypass IOMMU and make KVM call another FW+HW DMA engine
to smash IOPDEs.
I don't even want to know what that means :\ You can't change the
IOMMU page tables owned by linux from FW or you are creating bugs.
oh but I can :) It is a FW call which takes a pointer to an 2MB IOPDE, a new
table of 4K PTEs filled with the old PDE's pfn plus offsets and then the FW
exchanges the old IOPDE with a new table and smashes the corresponding RMP, and
it suspends the DMA while doing so.
If I get it right, for other platforms, the entire IOMMU table is going to live
in a secure space so there will be similar FW calls so it is not that different.
ps. I am still curious about:
btw just realized - does the code check that the folio_size
matches IO pagesize? Or batch_to_domain() is expected to start a
new batch if the next page size is not the same as previous? With
THP, we can have a mix of page sizes"
The batch has a linear chunk of consecutive physical addreses. It has
nothing to do with folios. The batch can start and end on any physical
address so long as all addresses within the range are contiguously
mapped.
The iommu mapping logic accept contiguous physical ranges and breaks
them back down into IOPTEs. There is no direct relationship between
folio size and IOPTE construction.
Ah right, pfn_reader_first/next take care of that constructiveness. Never mind.
Thanks,
For example the iommufd selftest often has a scenario where it lucks
into maybe 16k of consecutive PFNs because that is just what the MM
does on a fresh boot. Even though they are actually 4k folios they
will be mapped into AMDv1's 16k IOPTE encoding.
Jason
--
Alexey
