Hi Daniel,
On 1/19/2022 2:47 AM, Daniel Jordan wrote:
Hi,
I've hit a memory leak while testing qemu v6.2.0-rc4 on an AMD EPYC 7J13
(Milan) system. Starting an almost 1T guest, the leak is over 1.5G per
qemu invocation. I haven't checked whether the leak is proportional to
guest size. It happens with a vfio device, and only when the guest's
memory is preallocated using qemu prealloc (this latter part is kinda
strange). It happens when the guest memory uses THP but not hugetlb.
Bisection:
# bad: [df0cc57e057f18e44dac8e6c18aba47ab53202f9] Linux 5.16
# good: [f40ddce88593482919761f74910f42f4b84c004b] Linux 5.11
git bisect start 'df0cc57e057f1' 'f40ddce885934' '--' 'drivers/vfio'
'drivers/iommu' 'include/linux/amd-iommu.h' 'include/linux/dma-iommu.h'
'include/linux/intel-iommu.h' 'include/linux/iommu-helper.h'
'include/linux/of_iommu.h' 'include/
linux/omap-iommu.h' 'include/linux/platform_data/iommu-omap.h'
'include/linux/iommu.h' 'include/trace/events/intel_iommu.h'
'include/trace/events/iommu.h' 'include/uapi/linux/iommu.h'
'include/uapi/linux/virtio_iommu.h' 'arch/x86/events/a
md/iommu.h' 'arch/x86/events/amd/iommu.c' 'arch/x86/include/asm/iommu.h'
'arch/x86/include/asm/iommu_table.h' 'arch/x86/kernel/pci-iommu_table.c'
# bad: [cee57d4fe74e82e784f6566bad3e3bb1ca51a211] iommu/vt-d: Remove
unnecessary braces
git bisect bad cee57d4fe74e82e784f6566bad3e3bb1ca51a211
# bad: [9fb5fad562fa0a41c84691714d99c23f54168a9e] iommu: remove
DOMAIN_ATTR_PAGING
git bisect bad 9fb5fad562fa0a41c84691714d99c23f54168a9e
# bad: [45e606f2726926b04094e1c9bf809bca4884c57f] Merge branches 'arm/renesas',
'arm/smmu', 'x86/amd', 'x86/vt-d' and 'core' into next
git bisect bad 45e606f2726926b04094e1c9bf809bca4884c57f
# good: [7060377ce06f9cd3ed6274c0f2310463feb5baec] Merge branch 'for-joerg/mtk'
into for-joerg/arm-smmu/updates
git bisect good 7060377ce06f9cd3ed6274c0f2310463feb5baec
# bad: [6778ff5b21bd8e78c8bd547fd66437cf2657fd9b] iommu/amd: Fix performance
counter initialization
git bisect bad 6778ff5b21bd8e78c8bd547fd66437cf2657fd9b
# good: [f9b4df790aa4372bfa11b7d212e537b763295429] iommu/amd: Declare functions
as extern
git bisect good f9b4df790aa4372bfa11b7d212e537b763295429
# bad: [33aef9786046d9a5744cd1e8d5d0ce800d611fdc] iommu/amd: Rename variables
to be consistent with struct io_pgtable_ops
git bisect bad 33aef9786046d9a5744cd1e8d5d0ce800d611fdc
# bad: [e42ba0633064ef23eb1c8c21edf96bac1541bd4b] iommu/amd: Restructure code
for freeing page table
git bisect bad e42ba0633064ef23eb1c8c21edf96bac1541bd4b
# good: [18954252a1d0b12e1b77087b55c37fb43b09e12a] iommu/amd: Move IO page
table related functions
git bisect good 18954252a1d0b12e1b77087b55c37fb43b09e12a
# first bad commit: [e42ba0633064ef23eb1c8c21edf96bac1541bd4b] iommu/amd:
Restructure code for freeing page table
commit e42ba0633064ef23eb1c8c21edf96bac1541bd4b
Author: Suravee Suthikulpanit <suravee.suthikulpa...@amd.com>
Date: Tue Dec 15 01:36:59 2020 -0600
iommu/amd: Restructure code for freeing page table
By consolidate logic into v1_free_pgtable helper function,
which is called from IO page table framework.
Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpa...@amd.com>
Link:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.kernel.org%2Fr%2F20201215073705.123786-8-suravee.suthikulpanit%40amd.com&data=04%7C01%7Csuravee.suthikulpanit%40amd.com%7C143de50116b0431302ce08d9dabb5dab%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637781323390114388%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=RK%2F8spS7L5qdvaBYx0OE9T75TOfb9QiA8%2BKk4C00Jqo%3D&reserved=0
Signed-off-by: Joerg Roedel <jroe...@suse.de>
drivers/iommu/amd/amd_iommu.h | 1 -
drivers/iommu/amd/io_pgtable.c | 41 ++++++++++++++++++++++++-----------------
drivers/iommu/amd/iommu.c | 21 ++++-----------------
3 files changed, 28 insertions(+), 35 deletions(-)
Qemu command line:
numactl -m 1 -N 1 "$QEMU" \
-name vmol74 \
-machine q35,accel=kvm,usb=off,dump-guest-core=off,memory-backend=pc.ram \
-cpu host,host-phys-bits=true \
-smp cpus=32 \
-no-user-config \
-nodefaults \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-boot strict=on \
-drive file=${vm_image},format=raw,if=none,id=drive-ide0-0-0 \
-device
ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=2 \
-device vfio-pci,host=${pci_addr},id=net2,bus=pcie.0 \
-msg timestamp=on \
-nographic \
-object
memory-backend-ram,id=pc.ram,size=980g,prealloc=on,prealloc-threads=16 -m 980g \
-daemonize
Kernel config attached, and I can provide other details too if I left
anything out.
thanks,
Daniel
Could you please try the following change to see if this fix the problem?
diff --git a/drivers/iommu/amd/io_pgtable.c b/drivers/iommu/amd/io_pgtable.c
index 182c93a43efd..1eddf557636d 100644
--- a/drivers/iommu/amd/io_pgtable.c
+++ b/drivers/iommu/amd/io_pgtable.c
@@ -519,12 +519,6 @@ static void v1_free_pgtable(struct io_pgtable *iop)
dom = container_of(pgtable, struct protection_domain, iop);
- /* Update data structure */
- amd_iommu_domain_clr_pt_root(dom);
-
- /* Make changes visible to IOMMUs */
- amd_iommu_domain_update(dom);
-
/* Page-table is not visible to IOMMU anymore, so free it */
BUG_ON(pgtable->mode < PAGE_MODE_NONE ||
pgtable->mode > PAGE_MODE_6_LEVEL);
@@ -532,6 +526,12 @@ static void v1_free_pgtable(struct io_pgtable *iop)
root = (unsigned long)pgtable->root;
freelist = free_sub_pt(root, pgtable->mode, freelist);
+ /* Update data structure */
+ amd_iommu_domain_clr_pt_root(dom);
+
+ /* Make changes visible to IOMMUs */
+ amd_iommu_domain_update(dom);
+
free_page_list(freelist);
}
---
Thanks,
Suravee
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
