On 8/4/2026 4:00 PM, Tales A. Mendonça wrote:
Are we seeing this on i915?
I have not tested i915 on the affected machines yet - the
instrumentation that measured the stalls (late-ack logging, kick
results) is xe-only, so I have no comparable i915 data. I can boot one
of the ARL machines with i915 for a few days and watch for TLB
invalidation timeouts there, if that data helps.
That would definitely help, because if the issue does not happen on i915
it likely means that we're missing a WA or something like that in Xe.
More generally: both machines here reproduce reliably (~1 stall/hour
on a desktop workload, much more under memory pressure), so I am happy
to test anything on them - including any GuC build the firmware team
would like data on.
On Stuart's masking concern: fully agreed, that is why patch 3 is
marked RFC. Patches 1-2 are pure diagnostics and stand on their own; I
am fine holding patch 3 until the firmware side has been looked at.
The data point it adds is that a doorbell ring unblocks the ack in the
majority of episodes, while the severe ones ignore 8-9 consecutive
rings - hopefully that narrows where to look inside the GuC.
Just a bit of a terminology update here, to make sure we're on the same
page: we usually refer to the notification you're sending to the GuC as
an H2G interrupt and not a doorbell. I'm making this clarification
because the GuC supports a separate per-context notification mechanism
that is referred to as doorbell and which we currently do not implement
in neither i915 nor Xe.
When receiving the H2G interrupt, the only thing that the GuC does is
look into the CTB and process anything in there; however, you've said
that the contents of the H2G CTB are processed immediately, so the
follow up interrupt should result in the GuC just bailing out and doing
nothing because there is no data to process. It feels like when the
issue occurs something is stuck in HW rather than GuC FW and triggering
the interrupt causes the HW to get unstuck.
I have pushed the latest GuC FW for MTL here in case you want to give it
a go:
https://gitlab.com/dceraolo/drm-firmware/-/blob/f783b931555be057dafc2400b7eb4d445c953fec/i915/mtl_guc_70.72.1.bin
. You can override the GuC firmware used by the driver via the
xe.guc_firmware_path modparam; the path is relative to /lib/firmware/
and the firmware needs to be in initramfs for the driver to find it at
boot. Note that we haven't tested this image on MTL, so it might have
unexpected results.
Also, would you be able to capture the GuC logs when the issue occurs?
The default guc log size is relatively small, so you'd have to capture
right when the issue happens. However, you can make them bigger by
building the kernel with CONFIG_DRM_XE_DEBUG or by simply modifying the
xe_guc_log.h file to pick the bigger size by default. If you go with the
latter, please also set xe.guc_log_level=3 on the command line (this is
automatically added by the kconfig).
Thanks,
Daniele
I will send a v2 addressing Matt's review comments (the
__xe_devcoredump unification and the fixes on patch 2).
Thanks,
Tales
Em ter., 4 de ago. de 2026 às 19:08, Daniele Ceraolo Spurio
<[email protected]> escreveu:
On 8/4/2026 2:33 PM, Summers, Stuart wrote:
On Tue, 2026-08-04 at 14:27 -0700, Matthew Brost wrote:
On Tue, Aug 04, 2026 at 03:02:44PM -0600, Summers, Stuart wrote:
On Mon, 2026-08-03 at 23:14 -0300, Tales A. Mendonça wrote:
Hi,
This series is a follow-up to the TLB invalidation ack stall I
have
been debugging on ARL, tracked in:
https://gitlab.freedesktop.org/drm/xe/kernel/-/work_items/8678
Summary of the issue: with GuC 70.53.0 on ARL (reproduced on 7d51
and
7dd1 machines here, plus an independent Arc Pro 130T report on
the
issue above), TLB invalidation acks intermittently stall for
~2.3s.
The H2G request is consumed from the CTB immediately and the G2H
CTB
is empty the whole time - the firmware simply does not send the
ack
until much later. The fence timeout fires at 2.25s and the ack
lands
tens of ms after it. Userspace blocked on the invalidation
(compositor
buffer unmaps etc.) hitches for the full window.
Firstly, thanks for the patch!
I haven't looked in to all the details of the sighting you were
debugging, but we have had similar issues that were fixed in a
later
GuC version. I think around 70.60.0? It might be worth trying on
something later than that to see if that helps... (+Daniele)
I think this would require an AR on our end to make a new firmware
version available.
The upstream repo only has 70.53.0 available for ARL [1] (iirc, ARL
aliases to MTL for firmware). (+Julia too).
Presumably, the GuC changelogs should indicate whether an issue
related
this has been fixed. If so, we need to update all GuC versions across
both i915 and Xe.
Right... I guess I'd still like to see if we can test this in GuC (or
get confirmation we can't for some reason) before committing something.
My worry is we will prevent bug reports like this by working around it
and miss critical bugs that need to be fixed in the right component.
[1]
https://gitlab.com/kernel-firmware/linux-firmware/-/blob/main/i915/mtl_guc_70.bin?ref_type=heads
Patch 1 adds xe_devcoredump_gt() so this kind of hang - which has
no
Is there a reason we don't just re-use the main xe_devcoredump()?
This is my suggestion: the main devcoredump infrastructure is job-
based,
so it cannot be used for hangs that are not associated with a job.
In my opinion, this is a gap on our end. Introducing something like
`xe_devcoredump_gt()`, which can be used for non-job-based hangs
(e.g.,
TLB invalidation timeouts like those addressed in this series, or
more
generally any GuC protocol hang), makes sense to me.
Ok makes sense. We can do that here. It would be nice to have a more
inclusive implementation that lets us call this from anywhere so we
aren't duplicating things around for different use cases. But not a
blocker here.
I haven't looked at the patch yet, but at a high level, adding
`xe_devcoredump_gt()` seems like a reasonable approach.
exec queue or job to blame - leaves a devcoredump with the GuC
log
and
CT state behind (Matt suggested capturing devcoredumps when we
discussed the issue; devcoredumps from both machines are attached
to
the issue above).
Patch 2 logs when the ack for a timed out invalidation finally
arrives. This is what established that the acks are late rather
than
lost.
Patch 3 is the RFC part: a delayed work that pokes the GuC
(status
register read, CT flush, doorbell ring) every 250ms while an ack
is
overdue. On my machines this converts the guaranteed 2.3s stall
into
I'm a little worried we're just papering over something here that
needs
to be addressed in GuC, particularly around GT going to sleep or
something around the time we're expecting a response, so the pings
on
registers might be prematurely waking things up which is something
we'd
want to happen in GuC, not the KMD.
In general, I agree with this. We should avoid papering over the
issue
and instead fix it properly in the GuC. That said, this workaround
provides a pretty strong data point, since it appears to get the TLB
invalidation unstuck.
So if we hit this issue I guess we're already going to have some
performance degredation and the workaround makes that better. I need to
look at the implementation, but we could be potentially introducing
performance penalties in other areas doing these pings.
Again, I'd like to see if we can fix this in the right place before
implementing a workaround for it. Hopefully Daniele or Julia can give
some direction there.
Are we seeing this on i915 at all? Given that Xe does not officially
support MTL/ARL and is missing several critical WAs for those platforms,
the approach so far has been to only update the GuC FW if it is required
for i915.
Looking at the GuC release notes, there have been a couple of
TLB-related fixes after 70.53, but they're both marked as only affecting
PVC and Xe2+ platforms, so no fixes seem to be available for ARL (or at
least they're not listed in the release notes).
Daniele
Thanks,
Stuart
Matt
Thanks,
Stuart
a
sub-500ms hiccup for the majority of occurrences; a minority of
severe
episodes ignore 8-9 consecutive doorbells, which points at the
GuC
firmware being internally blocked for the whole window. Full data
on
the issue. I am happy to rework the approach (different delay,
tying it to the G2H handler, dropping the status read, etc.) -
mainly
I would like the firmware side investigated, since no host-side
poke
can fix the severe cases.
Based on drm-tip. Tested for several days on both ARL machines
under
desktop and VM-heavy workloads.
Thanks,
Tales
Tales A. Mendonça (3):
drm/xe: Capture devcoredump on TLB invalidation timeout
drm/xe: Log when a timed out TLB invalidation ack finally
arrives
drm/xe: Kick GuC while TLB invalidation acks are overdue
drivers/gpu/drm/xe/xe_devcoredump.c | 68 ++++++++++++
drivers/gpu/drm/xe/xe_devcoredump.h | 6 ++
drivers/gpu/drm/xe/xe_tlb_inval.c | 131
+++++++++++++++++++++++-
drivers/gpu/drm/xe/xe_tlb_inval_types.h | 42 ++++++++
4 files changed, 243 insertions(+), 4 deletions(-)
