Correction to my earlier test report: it turns out my system updated linux-firmware-amdgpu (20260622 -> 20260810), amd-ucode, and the kernel (7.1.8 -> 7.1.9) between the failing reproductions and the tlb_inv_rework test, so the clean result on your branch is confounded.
Stock 7.1.9 + firmware 20260810 is now also clean: two hibernate cycles today, zero "failed to write reg" errors (previously 4/4 cycles failed on firmware 20260622). Upstream 7.1.9 only has two trivial amdgpu display fixes, so my theory is that the amdgpu firmware refresh fixed the S4 KIQ wedge. If it's useful I can downgrade linux-firmware-amdgpu to 20260622 from the pacman cache and try to reproduce again on stock to confirm the firmware attribution. On Tue, 1 Sept 2026 at 16:45, Denis Pisarev <[email protected]> wrote: > > Hi Alex, > > Tested tlb_inv_rework (tip: cd3e1ff4f (HEAD -> > amdgpu-kiq-mmio-fallback-v3) drm/amdgpu: fall back to MMIO > TLB invalidation when KIQ is unresponsive, kernelrelease > 7.1.0-MANJARO+) on the affected machine (Cezanne / Ryzen > 7 PRO 5850U, previously reproducing 100% across hibernate > cycles). > > Result: hibernate resume is clean. > > - stock conditions (no GFXOFF workaround, hook removed) > - fresh boot: 0 "failed to write reg" errors > - after S4 resume + 30 min of normal desktop use: 0 errors > (control on 7.1.8 stock) > - the amdgpu MODE2 reset at resume still occurs as on stock, but no > invalidation failures follow > - one caveat from my testing: kernel was built with a trimmed > localmodconfig from the running distro config, not the full distro > config > > The GFXOFF correlation stands from the earlier single-variable test > (GFXOFF held off across S4 -> zero errors), but with your rework the > KIQ dependency for invalidation is gone, so the wedge no longer > matters for this workload. > > Anything else you'd like tested on this machine (multiple hibernate > cycles, sleep, specific tracing)? > > > On Tue, 25 Aug 2026 at 19:20, Alex Deucher <[email protected]> wrote: > > > > On Tue, Aug 25, 2026 at 9:19 AM Denis Pisarev <[email protected]> wrote: > > > > > > Hi Christian, Alex, > > > > > > Understood regarding the fallback - dropping that approach. A working > > > KIQ after S4 resume is the real fix. > > > > > > Regarding GFXOFF, I agree that a gated-off CP/ME/KIQ would also fail > > > the ring test, which aligns with the data. The ring test passes upon > > > resuming and sched.ready is set; however, the write-reg/wait-ack > > > handshake begins to fail during runtime, starting in the same second > > > as the thaw and persisting for hours (80-140 errors per hour, measured > > > over 9+ hours). So the KIQ initialises fine, but then stops completing > > > those packets. > > > > > > The GFXOFF attribution comes from a single-variable experiment only: > > > same workload, one hibernation cycle each. GFXOFF held off across the > > > whole S4 cycle (disabled before hibernating and verified to be still > > > off after resuming): zero errors. Also zero errors over 15 minutes > > > after re-enabling it post-resuming. Plain S3 suspend never triggers > > > it. This is merely a correlation; I cannot prove that it is a > > > mechanism. The existing semaphore workaround comment in gmc_v9_0.c > > > about losing invalidate acknowledge state across power-gating cycles > > > appears to be relevant, but I cannot prove that. I am happy to run > > > whatever tracing is required to identify the mechanism from the > > > affected machine (RLC/ME state, gfx_off_state sampled at the time of > > > failure, KIQ fence timestamps and register dumps). > > > > Can you try this branch? > > https://gitlab.freedesktop.org/agd5f/linux/-/commits/tlb_inv_rework?ref_type=heads > > > > Alex > > > > > > > > Thanks, > > > Denis > > > > > > On Mon, 24 Aug 2026 at 20:31, Alex Deucher <[email protected]> wrote: > > > > > > > > On Mon, Aug 24, 2026 at 12:30 PM Denis Pisarev <[email protected]> > > > > wrote: > > > > > > > > > > Hi, > > > > > > > > > > v3 addresses the two findings from the sashiko-bot review of v2. > > > > > Failure data and trigger isolation are unchanged (bugzilla 219492): > > > > > S4 resume on Cezanne (gmc_v9, GFXOFF) wedges KIQ TLB flushes at > > > > > 80-140/hour for 9+ hours with sched.ready true throughout; holding > > > > > GFXOFF off across the S4 cycle produces zero errors. > > > > > > > > > > 1. [High] "VFs and interrupt contexts silently drop TLB flushes once > > > > > the threshold is reached" - correct, and fixed. In v3 the latch > > > > > only reroutes bare metal process context to MMIO. VFs and IRQ > > > > > contexts keep submitting to KIQ exactly as before this patch, with > > > > > per-failure logging, because they have no MMIO alternative; there > > > > > is no longer any code path that drops a flush without attempting > > > > > and logging. > > > > > > > > > > 2. [High] "KIQ and MMIO race on the same invalidation engine if KIQ > > > > > recovers" - this remains the documented open question; no code > > > > > change in v3. Our analysis: once latched, this path submits no new > > > > > KIQ commands, so the exposure is limited to already-queued stale > > > > > commands and the recovery transition window. The engine serializes > > > > > requests internally, so the realistic worst case is a lost flush > > > > > request caught by the existing ACK timeout ("Timeout waiting for VM > > > > > flush ACK!"), not silent state corruption. If maintainers consider > > > > > a fence necessary (or a dedicated invalidate engine for the MMIO > > > > > path), guidance on the preferred mechanism would be welcome. > > > > > > > > > > Full patch history: v1 (initial fallback+counter), v2 (GFXOFF hold, > > > > > per-instance counter, irqsave, VF/IRQ restrictions, MES error > > > > > propagation) - all from bot review; v3 (this one) fixes the VF/IRQ > > > > > drop regression the bot found in v2. > > > > > > > > > > Also still open from the cover letters: the alternative direction of > > > > > fixing the S4 resume ordering itself (RLC/ME vs GFXOFF) instead of a > > > > > runtime fallback. Happy to run tracing on the affected hardware. > > > > > > > > > > > > > If that's a problem, that should be fixed. That said, did you have > > > > any success with the branch I proposed? > > > > > > > > Alex > > > > > > > > > Patch 1/1 follows. > > > > > > > > > > Denis Pisarev > > > > > > > > > > Denis Pisarev (1): > > > > > drm/amdgpu: fall back to MMIO TLB invalidation when KIQ is > > > > > unresponsive > > > > > > > > > > drivers/gpu/drm/amd/amdgpu/amdgpu.h | 2 + > > > > > drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h | 2 + > > > > > drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c | 18 ++-- > > > > > drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h | 2 +- > > > > > drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c | 123 > > > > > +++++++++++++++++++----- > > > > > 5 files changed, 114 insertions(+), 33 deletions(-) > > > > > > > > > > -- > > > > > 2.55.0 > > > > > > > > > -- > Best regards, > Denis Pisarev. > [email protected] -- Best regards, Denis Pisarev. [email protected]
