On surprise removal the killed process's own exit path can fault on its SVM managed ranges: SIGBUS -> coredump -> get_dump_page() -> __get_user_pages() -> do_swap_page() -> svm_migrate_to_ram(). Without an unplug check, svm_migrate_vram_to_ram() submits an SDMA copy job to an already-killed drm_sched entity (observed as "drm_sched_rq_add_entity: Trying to push to a killed entity" plus refcount_t saturation warnings), then svm_migrate_copy_done() waits uninterruptibly on a job fence that can never signal. The task dies in D state holding svms->lock, which wedges svm_range_restore_work() on the rwsem writer side, so kfd_processes_count never drains and the pciehp removal thread wedges for good.
Block unplug for the duration of a migration with drm_dev_enter()/ drm_dev_exit() instead of racy flag checks: the enter section covers both the job submission and the fence wait, so the sched entity cannot be torn down underneath. pci_dev_is_disconnected() is kept as an early-out since the PCI flag can be set well before drm_dev_unplug() runs. The fence wait in svm_migrate_copy_done() is additionally bounded by AMDGPU_FENCE_JIFFIES_TIMEOUT as a fallback for the submit->wait window on a device that dies electrically without a running removal. Verified on MI50 behind Thunderbolt: with these checks the workload takes SIGBUS, coredump completes, kfd_processes_count drains in ~4s, pciehp removal finishes, and the device re-enumerates cleanly on slot re-power. Previously the same repro wedged the host. Cc: Xiaogang Chen <[email protected]> Signed-off-by: Olivier Kaloudoff <[email protected]> --- Changes in v4 (per Felix Kuehling's review): - replace the bare drm_dev_is_unplugged()/pci_dev_is_disconnected() checks with drm_dev_enter()/drm_dev_exit() around the migration; pci_dev_is_disconnected() is kept as the early-out for the physically-gone-before-unplug window - bound the fence wait with AMDGPU_FENCE_JIFFIES_TIMEOUT instead of an open-coded 10 s - move the svm_range_restore_work() -ENODEV hunk to patch 2 New in v3. diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c index 253365a..e5faa2b 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c @@ -202,7 +202,21 @@ svm_migrate_copy_done(struct amdgpu_device *adev, struct dma_fence *mfence) int r = 0; if (mfence) { - r = dma_fence_wait(mfence, false); + /* Job fences on a dead device never signal */ + if (drm_dev_is_unplugged(adev_to_drm(adev)) || + pci_dev_is_disconnected(adev->pdev)) { + r = -ENODEV; + } else { + long ret; + + /* survive unplug between submit and wait */ + ret = dma_fence_wait_timeout(mfence, false, + AMDGPU_FENCE_JIFFIES_TIMEOUT); + if (!ret) + r = -ETIMEDOUT; + else if (ret < 0) + r = ret; + } dma_fence_put(mfence); pr_debug("sdma copy memory fence done\n"); } @@ -494,6 +508,7 @@ svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc, u64 ttm_res_offset; struct kfd_node *node; unsigned long mpages = 0; + int idx; long r = 0; if (start_mgr < prange->start || last_mgr > prange->last) { @@ -507,6 +522,12 @@ svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc, pr_debug("failed to get kfd node by id 0x%x\n", best_loc); return -ENODEV; } + /* The device may be electrically gone before drm_dev_unplug() runs */ + if (pci_dev_is_disconnected(node->adev->pdev)) + return -ENODEV; + /* Block unplug while the migration submits jobs and waits on them */ + if (!drm_dev_enter(adev_to_drm(node->adev), &idx)) + return -ENODEV; pr_debug("svms 0x%p [0x%lx 0x%lx] in [0x%lx 0x%lx] to gpu 0x%x\n", prange->svms, start_mgr, last_mgr, prange->start, prange->last, @@ -521,6 +542,7 @@ svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc, node->xcp ? node->xcp->id : 0); if (r) { dev_dbg(node->adev->dev, "failed to reserve VRAM, r: %ld\n", r); + drm_dev_exit(idx); return -ENOSPC; } @@ -565,6 +587,7 @@ out: prange->npages * PAGE_SIZE, KFD_IOC_ALLOC_MEM_FLAGS_VRAM, node->xcp ? node->xcp->id : 0); + drm_dev_exit(idx); return r < 0 ? r : 0; } @@ -821,6 +844,7 @@ int svm_migrate_vram_to_ram(struct svm_range *prange, struct mm_struct *mm, unsigned long start; unsigned long end; unsigned long mpages = 0; + int idx; long r = 0; /* this pragne has no any vram page to migrate to sys ram */ @@ -841,6 +865,12 @@ int svm_migrate_vram_to_ram(struct svm_range *prange, struct mm_struct *mm, pr_debug("failed to get kfd node by id 0x%x\n", prange->actual_loc); return -ENODEV; } + /* The device may be electrically gone before drm_dev_unplug() runs */ + if (pci_dev_is_disconnected(node->adev->pdev)) + return -ENODEV; + /* Block unplug while the migration submits jobs and waits on them */ + if (!drm_dev_enter(adev_to_drm(node->adev), &idx)) + return -ENODEV; pr_debug("svms 0x%p prange 0x%p [0x%lx 0x%lx] from gpu 0x%x to ram\n", prange->svms, prange, start_mgr, last_mgr, prange->actual_loc); @@ -885,6 +915,7 @@ int svm_migrate_vram_to_ram(struct svm_range *prange, struct mm_struct *mm, } } + drm_dev_exit(idx); return r < 0 ? r : 0; }
