On 8/25/26 09:34, Christian König wrote:
On 8/25/26 07:05, SHANMUGAM, SRINIVASAN wrote:
AMD General
-----Original Message-----
From: Natalie Vock <[email protected]>
Sent: Thursday, August 20, 2026 3:29 PM
To: SHANMUGAM, SRINIVASAN <[email protected]>;
Koenig, Christian <[email protected]>; Deucher, Alexander
<[email protected]>
Cc: [email protected]; Kuehling, Felix <[email protected]>;
Zhu, James <[email protected]>; Lazar, Lijo <[email protected]>; Six,
Lancelot <[email protected]>; Pelloux-Prayer, Pierre-Eric <Pierre-
[email protected]>; Timur Kristóf <[email protected]>; Samuel
Pitoiset <[email protected]>
Subject: Re: [RFC PATCH 1/3] drm/amdgpu/uapi: Add second-level trap handler
ops to VM ioctl
Hi,
first of all: Thanks for working on this! It's great seeing trap handler
support come
together.
On 8/20/26 09:01, Srinivasan Shanmugam wrote:
When a GPU shader hits an exception, memory fault, or debug
breakpoint, the hardware jumps to the first-level trap handler. The
first-level handler (managed by the kernel via CWSR) checks the TMA
buffer for a second-level handler address. If one is installed, it
forwards the trap to that userspace handler, allowing the runtime or
debugger to handle shader exceptions without modifying the kernel trap handler.
KFD already supports this for compute workloads. Render-node user
queues had no equivalent mechanism. Add it.
The second-level handler is a per-VM setting — it applies to all
shader waves executing under that VMID regardless of queue type. GFX
and compute queues from the same process share the same VMID, so one
SET_L2_TRAP call covers all queue types for that process. This
configuration is not CWSR-specific; CWSR is only the first-level
handler mechanism. The correct home for this setting is the VM ioctl
(DRM_AMDGPU_VM), following the same pattern as
AMDGPU_VM_OP_RESERVE_VMID.
Add two new VM ioctl operations:
AMDGPU_VM_OP_SET_L2_TRAP (op = 3) — install second-level handler
AMDGPU_VM_OP_CLEAR_L2_TRAP (op = 4) — remove second-level
handler
Extend drm_amdgpu_vm_in with a 32-byte union for op-specific data. The
l2trap member carries the GPU virtual addresses and sizes of the TBA
(handler code) and TMA (handler scratch memory).
This should be a BO handle and offset+size, instead. The BOs associated with the
TBA/TMA must be tracked as used by every submission from the VM that has this
trap handler installed, otherwise you introduce a ton of race conditions. Off
the top
of my head, here are a few:
1. The GEM VA ioctl can spuriously fail to actually update page tables.
This is okay and intentional, and BOs with outdated page tables will
be updated on the next submit if they're used by the submission. If
the TBA/TMA BOs aren't marked in the set of used buffers, the PTs may
end up never being updated and subsequent accesses will fault.
2. The TBA/TMA may be evicted/moved around concurrently with executing
submissions if these submissions didn't add their fences to the
TBA/TMA resv, which would likely randomly corrupt things or hang.
A simpler solution could be requiring the TBA/TMA buffers to be
VM_ALWAYS_VALID, in which case synchronization to all submissions in the VM
is taken care of automagically. This prevents exporting the TBA/TMA to an fd,
but I
don't expect anyone would want to do this.
Hi Natalie,
Thanks for the feedback.
We will update the UAPI in v2 to pass BO handles alongside
the GPU VA for TBA/TMA. This ensures the kernel can properly
track and pin the buffers.
Please don't. It is the responsibility of userspace to make sure that the BOs
are either in the used list or valid per VM.
Oh, right, I guess we can do it this way too. That's fine by me, but I
think my point about syncing TBA/TMA binds/unbinds in the same way as
VM map/unmap ops still applies. I suppose for user queues the existing
approach could be fine, but as Timur points out RADV won't be able to
use this unless support for kernel queues is added.
As far as I remember, it's convention that before merging new uAPI, you
also have a userspace-side use of that uAPI lined up. I or someone else
working on RADV can easily write that up, but for that we need an API
that RADV can actually use :)
Best,
Natalie
Regards,
Christian.
Thanks,
Srini
Regards,
Natalie
