On Tue, 30 Jun 2026 at 21:51, Jason Gunthorpe <[email protected]> wrote: > > On Tue, Jun 30, 2026 at 01:24:57PM +0200, Christian König wrote: > > > > Today I finally hit up iommu.strict=1 makes things a lot happier, > > > non-strict IOMMU seems to allow a race between dma_unmap_page and > > > free_page where the unmap goes into the IOMMU flush queue, where the > > > actual unmap are delayed until something triggers a flush later, and > > > the ATS translations stay alive past when they should, and after the > > > page has been allocated by some subsequent user. > > > > Try disabling ATS, that was a common source of problems during HW > > bringup for us in the past. > > The spark HW cannot run without ATS. > > > > 2. split ttm page unmap/free into two stages, and force an iommu > > > flush/ATS sync operation after the unmap before the free. (I don't > > > think iommu has an interface for this). > > > > Yeah that idea already came up on similar issues as well. I > > certainly wouldn't be opposed to that. > > Only for debugging, it is completely wrong to rely on this for > functionality. > > > You either have to use memory fences which use the same path as the > > DMA operations itself > > Yes, in a PCIe system generally something like the GPU TLB > invalidation command will complete and flush the fabric path to quiet > the DMAs. That won't be true on spark though. > > Dave could be facing a fabric issue. AFAIK the ATS invalidation that > the iommu=strict pushses should fence the fabric so would resolve a > missing fabric fence. I suppose if you don't see any evidence of a > non-present ATS response explosion with strict then this would be a > really good theory.
It's definitely a fabric issue, the iommu ATS invalidation is what "fixes" it, but I'm not figuring out how to flush it from the GPU side, and I'm not seeing where the official driver does this either, guess I have to keep digging. Maybe Alistair knows something. > > However, the GPU must have another way to inject a fabric fence. The > production driver does not rely on ATS fencing to synchronize memory > operations! > > > So far the record for grace periods you need after a TLB flush is 6 > > seconds on production HW with up to 10 seconds on engineering > > samples (because of HW bugs). > > Yeah, if the HW mechanism to fence the fabric don't work spark would > need similar times to guarentee a fabric drain. >From the ubuntu kernel this thing ships with: https://bugs.launchpad.net/ubuntu/+source/linux-nvidia-6.14/+bug/2132033 "Add two more Spark iGPU IDs for the existing iommu quirk " diff -u linux-nvidia-6.14-6.14.0/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c linux-nvidia-6.14-6.14.0/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c --- linux-nvidia-6.14-6.14.0/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ linux-nvidia-6.14-6.14.0/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -3654,7 +3654,9 @@ if (IS_HISI_PTT_DEVICE(pdev)) return IOMMU_DOMAIN_IDENTITY; - if (pdev->vendor == PCI_VENDOR_ID_NVIDIA && pdev->device == 0x2E12) + if (pdev->vendor == PCI_VENDOR_ID_NVIDIA && + (pdev->device == 0x2E12 || pdev->device == 0x2E2A || + pdev->device == 0x2E2B)) return IOMMU_DOMAIN_DMA; } I should probably work out if anything upstream does this or if I can mess up the production driver against an upstream kernel. Dave.
