Hi Christian,

> That is clearly a really bad idea. Waiting for the termination *must*
> be unbounded. The real question is why those KFD processes doesn't
> terminate?

Ok, dropped timeout and chased the termination instead. Reproduced
it reliably (MI50 on Thunderbolt 3, ~2GB of SVM managed ranges +
queues, then pciehp Surprise+ slot power-off). The answer from the
hung-task dumps:

The process can't terminate because its own exit path needs the GPU.
SIGBUS -> elf_core_dump -> get_dump_page -> __get_user_pages ->
do_swap_page hits the SVM managed ranges and calls
svm_migrate_to_ram(). That submits an SDMA copy job to a killed
drm_sched entity (refcount_t warns + "push to a killed entity"), then
svm_migrate_copy_done() waits uninterruptibly on a job fence that can
never signal. The task wedges in D holding svms->lock, which wedges
svm_range_restore_work (rwsem writer) and the userptr restore worker
behind it -> kfd_processes_count never drains.

So the bounded wait was indeed masking the bug. v3 keeps the wait
unbounded and instead:

1/3 teardown ordering: on the disconnected path, fini_hw frees kfd.dev
before teardown_processes runs, so the drain wait can't even observe
the count (that's also where the NULL deref came from). Signal +
drain now happens while kfd.dev is still valid; the NULL guard covers
the second call.

2/3 unchanged from the v2 I posted (no ret pre-init, no warn).

3/3 new: SVM migration fails fast with -ENODEV on unplugged /
pci_dev_is_disconnected device (entry points + the copy-done fence
wait + stop rescheduling svm_range_restore_work on -ENODEV). The
coredump fault then just turns into VM_FAULT_SIGBUS and the task dies
normally.

Retested with the same workload + removal: process died by SIGBUS +
core in ~4s, count drained, removal completed in ~7s, zero hung tasks
or refcount warnings, and the eGPU re-enumerated cleanly when the slot
re-powered. Host survived; previously it wedged hard.

One thing worth flagging separately: the refcount_t saturation /
"push to a killed entity" warns show amdgpu_job_submit /
amdgpu_copy_buffer happily arm jobs on dead sched entities - worth a
gate there too, but that's beyond this series.

Related: the April "drm/amdkfd: Terminate queues on surprise unplug
with running processes" thread touches the same area (process
termination around kfd_cleanup_nodes) - 1/3 lands teardown earlier for
the same reason, and is compatible with that approach.

Series + repro details below.

Reply via email to