The KDE session is actually still live on tty1 during that test, I've just switched to TTY3. I'm away for a few days so can't confirm if I have logs of exactly what sent those, but it may be regular background system activity like maybe Wayland cleaning up context(?)
But the NOP is just the smallest reproducer I could come up with - I originally caught the behavior just watching regular system activity. When I saw sclk toggle from 0 to high and stay there with no other jobs dispatched, iirc it was always <10 jobs showing up on that log line. My first attempt to trigger it myself was a simple open gl task, but that didn't work - iirc setting up the context, running it, and destructing dispatched a larger bunch of tasks (20-something). It wasn't until I held the context between "pokes" that I could get the manual clock toggling behavior. Maybe worth noting the Python version of the "poke" reproducer (used in the demo) is using a GL stack to dispatch an empty compute shader, whereas the c poke is the more barebones PM4 packet with NOP, but either one acts as a toggle. In the wild, the pattern seems to be: GPU idles, sclk->0 Lone task comes in and completes with nothing behind it sclk->max Later, another task comes in sclk->0 (my speculation is queuing or completing a task triggers a test of % of recent idle clocks, and since it's been sitting at max clock with nothing to do, it sees 100% idle clocks and immediately throttles back down to 0) - Kevin On Mon, Aug 24, 2026, 9:04 PM Feng, Kenneth <[email protected]> wrote: > AMD General > > Thanks Kevin. > > At the beginning of the video, right after gpu-monitor.py is executed, > there are 13 jobs dispatched. > > Are they also the NOP packets? > > Thanks. > > > > *From:* Kevin Hake <[email protected]> > *Sent:* Tuesday, August 25, 2026 3:22 AM > *To:* Feng, Kenneth <[email protected]> > *Cc:* [email protected]; Deucher, Alexander < > [email protected]> > *Subject:* Re: [drm/amd #3549] Navi 33 idle sclk latch: repro, analysis, > userspace workaround > > > > You don't often get email from [email protected]. Learn why this is important > <https://aka.ms/LearnAboutSenderIdentification> > > Hi Kenneth, > > I think that's the bug - it is reporting 100% utilization but the GPU is > idle - I know it's idle because I traced job submissions and dispatch, and > separately drm_sched_job_done, debugfs pending fences, the kernel's > drm-engine-* nanoseconds - everything points to nothing actually crunching, > at least nothing real from userspace. Not to mention I'm in TTY with > minimal system activity. You can see the behavior in this demo: > https://asciinema.org/a/Qc5CQBW6ycb89ldd > > > I'm wondering if there's a bug where utilization is calculated something > something like, "how many idle clocks have there been since we last > checked?". If the gpu has been asleep with sclk == 0, then it sees 0 idle > clocks in the past n seconds (no idle clocks?? we must be really busy!), > and sets utilization to 100. But I'm just speculating, what I can confirm > is that with sclk at 0, a single NOP to the gpu toggles utilization to 100% > and sclk to max. It seems odd numbers of very small work trigger the > toggle, When larger groups of work come in it behaves more normally and > doesn't "stick" high. > > > Thanks, > > Kevin Hake > > +1 716 432 7106 > > +52 81 1300 5479 > > > > > > On Mon, Aug 24, 2026 at 8:10 AM Feng, Kenneth <[email protected]> > wrote: > > AMD General > > > > What’s the GPU utilization(observed by sudo cat > /sys/kernel/debug/dri/x/amdgpu_pm_info) when the sclk is highest? > > The policy is that if the utilization is 100%, the clock has to stay > highest. > > Could you please elaborate what ‘a tiny job’ is to trigger the issue? > > Thanks. > > > > > > *From:* amd-gfx <[email protected]> *On Behalf Of *Kevin > Hake > *Sent:* Sunday, August 23, 2026 11:05 PM > *To:* [email protected] > *Cc:* Deucher, Alexander <[email protected]> > *Subject:* [drm/amd #3549] Navi 33 idle sclk latch: repro, analysis, > userspace workaround > > > > Some people who received this message don't often get email from > [email protected]. Learn why this is important > <https://aka.ms/LearnAboutSenderIdentification> > > > Hi all, > > Follow-up to drm/amd issue #3549 ("AMD GPU usage peaks with system in idle > — > Radeon RX 7600 XT"), where I posted a root-cause analysis and reproducer > a few weeks ago (note 3586398). The thread's been quiet, so I'm summarizing > here in case it's useful to whoever owns PMFW/swsmu for SMU 13.0.7. > > Behavior (RX 7600 XT / Navi 33, kernels 6.x–7.1, likely any DE (tho I'm on > Plasma w/Wayland) or bare TTY): > > - With the gfx job stream verifiably empty, a single tiny job submitted > while sclk is at 0 (deep sleep) latches sclk at max (~2990 MHz, ~115 W) > indefinitely. A second completed gfx job clears it. So at true idle the > card toggles between 0 and max on every stray job, and can sit at max > for an hour with the screen off. > - Light sustained load (video, a compositor, radeontop) masks it, which > is why the earlier "monitoring tools toggle gfxoff" hypothesis looked > plausible. It reproduces on demand from a TTY with nothing polling. > - Umio's finding that disabling GFXOFF via ppfeaturemask stops the spikes > is consistent with this model: without GFXOFF there is no "single job > from deep sleep" transition to mis-handle. > > How I measured: > > - Job stream: a private ftrace instance on amdgpu_cs_ioctl / > amdgpu_sched_run_job / drm_sched_job_done, cross-checked against fdinfo > drm-engine counters and debugfs amdgpu_fence_info, so I could prove the > kernel submitted nothing while clocks moved. > - Clock: sampling the firmware's gpu_metrics (v1.3) at 1 kHz, matched to > the driver's 1 ms refresh of that table, to catch the ramp edges. > - Stimulus: a minimal compute dispatch with a held context (no setup/ > teardown jobs), and later a one-packet PM4 NOP via libdrm — either one > toggles the latch. > > Everything points at the firmware's clock selection rather than the driver. > Since PMFW can't be patched from outside, I published a small userspace > workaround: a C daemon over libdrm/render node that submits one PM4 NOP > when it sees sclk latched (same binary is the manual reproducer): > > https://github.com/KevinHake/amdgpu-unstick > > Scripts, logs, and an asciinema demo are attached to the issue. If a > narrower repro would help (specific firmware versions, mesa-free path, > other kernels), tell me what you need and I'll run it. > > Thanks, > Kevin Hake > >
