On Wed, Sep 2, 2026 at 9:25 AM Alex Deucher <[email protected]> wrote: > > On Wed, Sep 2, 2026 at 9:20 AM Christian König <[email protected]> > wrote: > > > > On 9/2/26 15:06, Alex Deucher wrote: > > > On Wed, Sep 2, 2026 at 3:25 AM Christian König <[email protected]> > > > wrote: > > >> > > >> On 9/1/26 22:10, Alex Deucher wrote: > > >>> GMC 9-12 use KIQ or MES for TLB invalidations to avoid using MMIO which > > >>> would > > >>> require disallowing GFXOFF. KIQ and MES are management queues however > > >>> and if > > >>> they hang, they cannot be recovered by a queue reset since they are the > > >>> mechanisms which handle queue resets. Since using KIQ or MES will exit > > >>> GFXOFF > > >>> anyway, explicitly disallow it on the MMIO path and use that. Next, > > >>> switch to > > >>> using SDMA for TLB invalidations. SDMA 4.4.x and newer have special > > >>> packets > > >>> specifically for this purpose. > > >> > > >> That packet was just introduced to work around SRIOV issues and proved > > >> to cause stability issues as well. > > >> > > > > > > It was introduced because of gfxoff and having to toggle it to use > > > MMIO. > > > > No, that was completely unrelated to GFXOFF. This also applies to HW where > > the SDMA is not even in the GFX domain. > > > > The problem was the SRIOV could interrupt the SDMA while it waited for the > > ACK and when the VF was scheduled in again the ACK bit was resetted. > > > > So a single packet was introduced which couldn't be interrupted by the > > hypervisor. > > > > > WIndows uses SDMA pretty much exclusively for TLB > > > invalidations. It's also what the memory hubs team recommends. > > > > That is rather interesting. > > >> Since Arun now found that on basically all Navi generations the SDMA can > > >> hang while doing a TLB invalidation I have to clearly NAK this approach. > > >> > > >> In the long run we should do the TLB invalidations on newer HW > > >> completely with the MES, the SDMA was always just a workaround we should > > >> not push forward. > > >> > > > > > > Well KIQ and MES also seem to be problematic as well. See the thread > > > from Denis about KIQ and his renoir system. There are also various > > > reports of MES timeouts doing TLB invalidations. > > > > The MES is the only instance which knows the process to VMID mapping for > > user queues. > > > > The hack to read out the PASID->VMID mapping register directly from the IH > > is not really something we can use in the future. > > > > So the plan was to move all of that into MES in the near term. IIRC the MES > > even added a new packet for that. > > Sure, this patch set retains the use of MES for pasid invalidation > where it's supported (navi4x and newer with new enough firmware). We > could try and get the new packet added to gfx11 MES as well, but then > there is still gfx9 and 10 which still use KIQ. >
Regarding gfx 11 mes, there is only a single scheduler queue, similar to KIQ on previous generations. That micro controller handles both driver commands and queue scheduling. On gfx12, there are now two MES queues, one for scheduling and one for misc commands to avoid contention while scheduling. This series doesn't actually remove any functionality, it just provides the option of using: - KIQ - SDMA - MES - MMIO and cleans up a lot of duplicate code in the gmc modules. The default is changed to SDMA or MES depending on what the device supports. We can adjust the defaults for each gmc generation as needed. Alex > Alex > > > > > Regards, > > Christian. > > > > > > > > Alex > > > > > >> Regards, > > >> Christian. > > >> > > >>> If SDMA hangs while doing the invalidation for > > >>> some reason, it's easier to reset the SDMA queue than KIQ or MES. > > >>> Finally, > > >>> most of the TLB invalidation code between GMC 9 through 12 was > > >>> identical, so > > >>> move it to common GMC helpers and remove the IP specific code. If the > > >>> SMDA > > >>> and MMIO pathes prove to be stable, the KIQ pathes can be removed in > > >>> the future > > >>> to further simplify things. SDMA 4.x could also be updated to support > > >>> PASID > > >>> invalidation via SDMA using either the new packet (SDMA 4.4.x) or via > > >>> REG_WRITE/REG_WAIT packets (SDMA 4.0.x). > > >>> > > >>> Code is available on this branch as well: > > >>> https://gitlab.freedesktop.org/agd5f/linux/-/commits/tlb_inv_rework?ref_type=heads > > >>> > > >>> V2: > > >>> - Add missing hub callbacks in gmc9 hubs > > >>> > > >>> Alex Deucher (31): > > >>> drm/amdgpu/gmc9: disallow gfxoff around TLB flushes > > >>> drm/amdgpu/gmc10: disallow gfxoff around TLB flushes > > >>> drm/amdgpu/gmc11: disallow gfxoff around TLB flushes > > >>> drm/amdgpu/gmc12: disallow gfxoff around TLB flushes > > >>> drm/amdgpu/gmc9: set vmhub funcs for gfxhub and mmhub > > >>> drm/amdgpu: add a gmc flag for using MMIO for TLB flush > > >>> drm/amdgpu/gmc9: use MMIO for TLB flushes > > >>> drm/amdgpu/gmc10: use MMIO for TLB flushes > > >>> drm/amdgpu/gmc11: use MMIO for TLB flushes > > >>> drm/amdgpu/gmc12: use MMIO for TLB flushes > > >>> drm/amdgpu: add a buffer funcs callback for TLB invalidation > > >>> drm/amdgpu/sdma5.0: add tlb invalidation buffer func callback > > >>> drm/amdgpu/sdma5.2: add tlb invalidation buffer func callback > > >>> drm/amdgpu/sdma6: add tlb invalidation buffer func callback > > >>> drm/amdgpu/sdma7: add tlb invalidation buffer func callback > > >>> drm/amdgpu: simplify amdgpu_gmc_flush_gpu_tlb() > > >>> drm/amdgpu: add tlb invalidation method enum > > >>> drm/amdgpu: plumb tlb inv method in amdgpu_gmc_flush_gpu_tlb_gart() > > >>> drm/amdgpu: uplevel reset check in amdgpu_gmc_flush_gpu_tlb_gart() > > >>> drm/amdgpu/gmc: add new callback to lookup vmid to pasid mapping > > >>> drm/amdgpu: add a gmc callback for the inv semaphore > > >>> drm/amdgpu/gmc: rework pasid flushing > > >>> drm/amdgpu/gmc9: use SDMA for gart TLB invalidation > > >>> drm/amdgpu/gmc10: use SDMA for gart TLB invalidation > > >>> drm/amdgpu/gmc11: use SDMA for gart TLB invalidation > > >>> drm/amdgpu/gmc12: use SDMA for gart TLB invalidation > > >>> drm/amdgpu/gmc10: use SDMA for pasid TLB invalidation > > >>> drm/amdgpu/gmc11: use SDMA for pasid TLB invalidation > > >>> drm/amdgpu/gmc12: use MES or SDMA for pasid TLB invalidation > > >>> drm/amdgpu/gmc12: drop MES tlb inv in gmc callbacks > > >>> drm/amdgpu/gmc: add helpers for various tlb inv functions > > >>> > > >>> drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c | 2 +- > > >>> drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c | 486 +++++++++++++++++++---- > > >>> drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h | 33 +- > > >>> drivers/gpu/drm/amd/amdgpu/amdgpu_sdma.h | 18 + > > >>> drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c | 2 + > > >>> drivers/gpu/drm/amd/amdgpu/gfxhub_v1_0.c | 33 ++ > > >>> drivers/gpu/drm/amd/amdgpu/gfxhub_v1_2.c | 33 ++ > > >>> drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c | 202 +--------- > > >>> drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c | 207 +--------- > > >>> drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c | 250 ++---------- > > >>> drivers/gpu/drm/amd/amdgpu/gmc_v12_1.c | 225 +---------- > > >>> drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c | 135 ++----- > > >>> drivers/gpu/drm/amd/amdgpu/mes_v12_0.c | 4 + > > >>> drivers/gpu/drm/amd/amdgpu/mes_v12_1.c | 4 + > > >>> drivers/gpu/drm/amd/amdgpu/mmhub_v1_0.c | 33 ++ > > >>> drivers/gpu/drm/amd/amdgpu/mmhub_v1_7.c | 32 ++ > > >>> drivers/gpu/drm/amd/amdgpu/mmhub_v1_8.c | 32 ++ > > >>> drivers/gpu/drm/amd/amdgpu/mmhub_v9_4.c | 32 ++ > > >>> drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c | 49 +++ > > >>> drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c | 49 +++ > > >>> drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c | 49 +++ > > >>> drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c | 48 +++ > > >>> 22 files changed, 943 insertions(+), 1015 deletions(-) > > >>> > > >> > >
