On GC 9.4.3, 9.4.4 and 9.5.0, doorbell writes issued from shaders are
routed to an SDMA engine at 32-byte (4 doorbell index) granularity.
SDMA engine doorbell ranges are 10 indices apart, so the range of every
odd engine starts 2 indices into a 4-index block that begins in the
previous engine's range. A shader ringing the doorbell of a queue on an
odd engine therefore rings the previous engine instead. The intended
queue never sees the doorbell, so its work stalls and the application
hangs.
Align the per-engine doorbell base up to a 4-index boundary on these
ASICs. Each engine only uses 4 of its 10 indices, so the aligned base
plus the queue offset stays within the engine's range. Even engines are
already aligned, so only the doorbell IDs of odd-engine queues change.
Only doorbells rung by the device itself are routed this way. Doorbell
writes from the host, including the kernel's own SDMA rings, are not
affected.
The doorbell ID of an SDMA queue is saved in CRIU checkpoints and
checked again on restore. A checkpoint taken before this change that
contains an SDMA queue on an odd engine will fail to restore with
-EINVAL. This is intentional: the restored process still holds the old
doorbell address, so accepting the stale ID would bring back the same
stall.
Tested on MI300X with a shader-rung SDMA doorbell test: queues on odd
engines hung before this change and complete with it.
v2: use u32 instead of uint32_t for the new locals (checkpatch)
v3: name the routing granularity, use dev->node_id, document the
layout invariants, describe impact
v4: express the granularity in doorbell indices, drop unrelated
cleanups, explain the CRIU restore behavior, add test note
Fixes: 6df442a03d1a ("drm/amdgpu: add new doorbell assignment table for
aqua_vanjaram")
Cc: [email protected]
Signed-off-by: Saleel Kudchadker <[email protected]>
---
.../drm/amd/amdkfd/kfd_device_queue_manager.c | 32 +++++++++++++++----
drivers/gpu/drm/amd/amdkfd/kfd_priv.h | 7 ++++
2 files changed, 33 insertions(+), 6 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
index 6d12f9dc7f04..5e57576fba21 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
@@ -599,12 +599,32 @@ static int allocate_doorbell(struct qcm_process_device
*qpd,
* we need the physical sdma engine id in order to get the
* correct doorbell offset.
*/
- uint32_t valid_id = idx_offset[qpd->dqm->dev->node_id *
-
get_num_all_sdma_engines(qpd->dqm) +
- q->properties.sdma_engine_id]
- + (q->properties.sdma_queue_id
& 1)
- *
KFD_QUEUE_DOORBELL_MIRROR_OFFSET
- + (q->properties.sdma_queue_id
>> 1);
+ u32 engine_base = idx_offset[dev->node_id *
+ get_num_all_sdma_engines(qpd->dqm)
+
+ q->properties.sdma_engine_id];
+ u32 valid_id;
+
+ /*
+ * With the aqua_vanjaram doorbell layout, a shader doorbell
+ * write whose 32-byte block starts in the previous engine's
+ * range rings that engine instead. Engine ranges are 10
+ * indices apart (sdma_doorbell_range = 20 dwords, halved for
+ * 8-byte doorbells, in aqua_vanjaram_doorbell_index_init()),
+ * which puts odd engines 2 indices into a shared block.
+ * Aligning costs at most 2 indices, and 8 queues per engine use
+ * only 4 of the 10, so the aligned base stays in range. The
+ * 512-index mirror offset keeps the alignment.
+ */
+ if (KFD_GC_VERSION(dev) == IP_VERSION(9, 4, 3) ||
+ KFD_GC_VERSION(dev) == IP_VERSION(9, 4, 4) ||
+ KFD_GC_VERSION(dev) == IP_VERSION(9, 5, 0))
+ engine_base = ALIGN(engine_base,
+
KFD_SDMA_SHADER_DOORBELL_GRANULARITY);
+
+ valid_id = engine_base +
+ (q->properties.sdma_queue_id & 1) *
+ KFD_QUEUE_DOORBELL_MIRROR_OFFSET +
+ (q->properties.sdma_queue_id >> 1);
if (restore_id && *restore_id != valid_id)
return -EINVAL;
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
index 2847a5ec5ede..b9a31489b1c8 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
@@ -133,6 +133,13 @@
*/
#define KFD_QUEUE_DOORBELL_MIRROR_OFFSET 512
+/*
+ * On GC 9.4.3, 9.4.4 and 9.5.0, doorbell writes issued from shaders are
+ * routed to an SDMA engine at 32-byte granularity, i.e. 4 indices with the
+ * 8-byte SDMA doorbells used on SOC15.
+ */
+#define KFD_SDMA_SHADER_DOORBELL_GRANULARITY 4
+
/**
* enum kfd_ioctl_flags - KFD ioctl flags
* Various flags that can be set in &amdkfd_ioctl_desc.flags to control how
--
2.34.1
