From: Denis V. Lunev <[email protected]>
increase_address_space() publishes the new root and mode and only then
points the DTEs at the new root. alloc_pte() reads pgtable->mode
without the domain lock, so a mapper on another CPU that sees the new
mode installs its PTEs under the new root and hands the IOVA to the
device while the IOMMU still walks the old table:
nvme 0000:63:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT
domain=0x0004 address=0xfffffffffff40000 flags=0x0030]
The v1 table starts with 3 levels and grows to 6 on the first IOVA
dma-iommu takes from the top of the 64-bit space, once the 32-bit space
runs out. That happens under heavy DMA load, so many CPUs map into the
window at once. A faulting device read of memory fails the command with
Data Transfer Error. A faulting device write to memory is posted and
gets dropped silently: an NVMe READ completes successfully and the
buffer keeps whatever it held before.
Update the DTEs inside the seqcount write section. alloc_pte() and
fetch_pte() spin in read_seqcount_begin() until it ends, so nothing is
mapped under the new root before the devices use it.
Mainline is not affected since commit 789a5913b29c ("iommu/amd: Use
the generic iommu page table"): increase_top() calls change_top()
before it publishes the new top.
https://virtuozzo.atlassian.net/browse/VSTOR-154276
Fixes: 6eedb59c18a3 ("iommu/amd: Remove amd_iommu_domain_get_pgtable")
Feature: fix ms/iommu
Signed-off-by: Denis V. Lunev <[email protected]>
---
Reproduced on epyc06 (Supermicro H12DSU-iN, EPYC 7352, four Optane in
one IOMMU group): an out-of-tree module eats the group's 32-bit IOVA
space through a root port while fio reads the drives, and a kprobe
delays amd_iommu_update_and_flush_device_table() to hold the window
open. Stock 6.12.0-211.52.1.22.9.vz10 faults on every try, and fio
verify sees reads return stale buffers with no I/O error. With this
patch: no faults, no verify mismatches, and the first 64-bit IOVA is
mapped only after the last DTE flush.
drivers/iommu/amd/io_pgtable.c | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/drivers/iommu/amd/io_pgtable.c b/drivers/iommu/amd/io_pgtable.c
index 70c2f5b1631b..46dfed9bd7e2 100644
--- a/drivers/iommu/amd/io_pgtable.c
+++ b/drivers/iommu/amd/io_pgtable.c
@@ -131,12 +131,15 @@ static bool increase_address_space(struct amd_io_pgtable
*pgtable,
*pte = PM_LEVEL_PDE(pgtable->mode, iommu_virt_to_phys(pgtable->root));
+ /*
+ * Keep the seqcount odd until the DTEs point at the new root, or a
+ * concurrent alloc_pte() maps an IOVA the device cannot translate yet.
+ */
write_seqcount_begin(&pgtable->seqcount);
pgtable->root = pte;
pgtable->mode += 1;
- write_seqcount_end(&pgtable->seqcount);
-
amd_iommu_update_and_flush_device_table(domain);
+ write_seqcount_end(&pgtable->seqcount);
pte = NULL;
ret = true;
--
2.53.0
_______________________________________________
Devel mailing list
[email protected]
https://lists.openvz.org/mailman/listinfo/devel
