AMD General The issue here is specific to runpm resume: SW still keeps the forced level, but D3 can drop perfmon-related HW programming, and during resume smu_late_init() is not the final step inside amdgpu_device_ip_late_init().
After smu_late_init(), amdgpu_device_ip_late_init() still executes global amdgpu_device_set_cg_state() / amdgpu_device_set_pg_state(). That is why this fix restores at the end of amdgpu_pmops_runtime_resume(): once resume programming is done, re-assert UMD/perfmon state if current level is profile-forced. Thanks, Candice -----Original Message----- From: Lazar, Lijo <[email protected]> Sent: Friday, July 24, 2026 11:41 AM To: Li, Candice <[email protected]>; Zhang, Hawking <[email protected]>; [email protected] Subject: RE: [PATCH] drm/amdgpu: restore perf state after runtime resume AMD General > If we do this in smu_late_init(), later resume stages can still touch > CG/PG/GC and clear PERFMON_CLOCK_STATE again, so the restore is not > persistent. If this happens, the check that user level is enforced is not correct (since it is not really enforced). The same could happen when a user applies force profile mode also. Basically, we need to prevent this for all other cases also. Where do you see this happening? Thanks, Lijo -----Original Message----- From: Li, Candice <[email protected]> Sent: Friday, July 24, 2026 8:38 AM To: Li, Candice <[email protected]>; Lazar, Lijo <[email protected]>; Zhang, Hawking <[email protected]>; [email protected] Subject: RE: [PATCH] drm/amdgpu: restore perf state after runtime resume AMD General Hi Lijo, I want to revise my earlier agreement after rechecking the full runtime-resume flow. I agree profile/forced DPM is a user policy and D3 can lose HW programming, but for this bug the key is restore timing. If we do this in smu_late_init(), later resume stages can still touch CG/PG/GC and clear PERFMON_CLOCK_STATE again, so the restore is not persistent. For that reason, I’d like to keep the original placement: restore at the end of amdgpu_pmops_runtime_resume(), and only re-assert UMD stable pstate when current level is a profile forced level. This keeps smu_late_init() for normal policy replay, while ensuring perfmon state is restored at a point where it is not overwritten afterward. Thanks, Candice -----Original Message----- From: amd-gfx <[email protected]> On Behalf Of Li, Candice Sent: Thursday, July 23, 2026 2:16 PM To: Lazar, Lijo <[email protected]>; Zhang, Hawking <[email protected]>; [email protected] Subject: RE: [PATCH] drm/amdgpu: restore perf state after runtime resume AMD General -----Original Message----- From: Lazar, Lijo <[email protected]> Sent: Thursday, July 23, 2026 11:46 AM To: Li, Candice <[email protected]>; Zhang, Hawking <[email protected]>; [email protected] Subject: Re: [PATCH] drm/amdgpu: restore perf state after runtime resume On 23-Jul-26 9:00 AM, Li, Candice wrote: > AMD General > > -----Original Message----- > From: Lazar, Lijo <[email protected]> > Sent: Wednesday, July 22, 2026 8:11 PM > To: Zhang, Hawking <[email protected]>; Li, Candice > <[email protected]>; [email protected] > Subject: Re: [PATCH] drm/amdgpu: restore perf state after runtime > resume > > > > On 22-Jul-26 4:43 PM, Zhang, Hawking wrote: >> AMD General >> >> Reviewed-by: Hawking Zhang <[email protected]> >> >> Regards, >> Hawking >> -----Original Message----- >> From: amd-gfx <[email protected]> On Behalf Of >> Candice Li >> Sent: Wednesday, July 22, 2026 6:36 PM >> To: [email protected] >> Cc: Li, Candice <[email protected]> >> Subject: [PATCH] drm/amdgpu: restore perf state after runtime resume >> >> Runtime suspend runs GFX hw_fini and clears perfmon clock gating while KFD >> keeps running, so block perf counters read zero after resume even though >> profiling remains enabled in software. >> >> Re-apply UMD stable pstate for profile DPM modes and re-enable KFD perfcount >> after a successful runtime resume. >> >> Signed-off-by: Candice Li <[email protected]> >> --- >> drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c | 46 ++++++++++++++++++++++ >> drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h | 1 + >> drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 17 ++++++++ >> drivers/gpu/drm/amd/amdkfd/kfd_chardev.c | 8 ++++ >> drivers/gpu/drm/amd/amdkfd/kfd_priv.h | 2 + >> drivers/gpu/drm/amd/amdkfd/kfd_process.c | 1 + >> 6 files changed, 75 insertions(+) >> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c >> b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c >> index 121282dd30c161..38974beb98ef88 100644 >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c >> @@ -38,6 +38,7 @@ >> #include "amdgpu_reset.h" >> #if IS_ENABLED(CONFIG_HSA_AMD) >> #include "kfd_priv.h" >> +#include "kfd_device_queue_manager.h" >> #endif >> >> /* Total memory size in system memory and all GPU VRAM. Used to @@ -280,6 >> +281,51 @@ int amdgpu_amdkfd_resume(struct amdgpu_device *adev, bool >> resume_proc) >> return r; >> } >> >> +void amdgpu_amdkfd_restore_profiler_after_runpm(struct amdgpu_device >> +*adev) { #if IS_ENABLED(CONFIG_HSA_AMD) >> + struct kfd_dev *kfd; >> + struct kfd_process *p; >> + struct kfd_process_device *pdd; >> + struct device_queue_manager *dqm; >> + int i; >> + >> + if (!adev->kfd.dev || !adev->kfd.init_complete) >> + return; >> + >> + kfd = adev->kfd.dev; >> + >> + mutex_lock(&kfd->profiler_lock); >> + p = kfd->profiler_process; >> + if (!p || !kfd->profiler_perfcount_enabled) { >> + mutex_unlock(&kfd->profiler_lock); >> + return; >> + } >> + mutex_unlock(&kfd->profiler_lock); >> + >> + mutex_lock(&p->mutex); >> + >> + mutex_lock(&kfd->profiler_lock); >> + if (kfd->profiler_process != p || !kfd->profiler_perfcount_enabled) { >> + mutex_unlock(&kfd->profiler_lock); >> + mutex_unlock(&p->mutex); >> + return; >> + } >> + mutex_unlock(&kfd->profiler_lock); >> + >> + for (i = 0; i < p->n_pdds; i++) { >> + pdd = p->pdds[i]; >> + if (!pdd || !pdd->dev || pdd->dev->kfd != kfd) >> + continue; >> + >> + dqm = pdd->dev->dqm; >> + if (dqm && dqm->ops.set_perfcount) >> + dqm->ops.set_perfcount(dqm, 1); >> + } >> + mutex_unlock(&p->mutex); >> +#endif >> +} >> + >> void amdgpu_amdkfd_suspend_process(struct amdgpu_device *adev) { >> if (adev->kfd.dev) >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h >> b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h >> index 338412a750edf0..1be9d5ef97d6d0 100644 >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h >> @@ -164,6 +164,7 @@ void amdgpu_amdkfd_teardown_processes(struct >> amdgpu_device *adev); >> >> void amdgpu_amdkfd_suspend(struct amdgpu_device *adev, bool >> suspend_proc); int amdgpu_amdkfd_resume(struct amdgpu_device *adev, >> bool resume_proc); >> +void amdgpu_amdkfd_restore_profiler_after_runpm(struct amdgpu_device >> +*adev); >> void amdgpu_amdkfd_suspend_process(struct amdgpu_device *adev); >> int amdgpu_amdkfd_resume_process(struct amdgpu_device *adev); void >> amdgpu_amdkfd_interrupt(struct amdgpu_device *adev, diff --git >> a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c >> b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c >> index 5362705143bc75..f7a1064c340692 100644 >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c >> @@ -2920,6 +2920,21 @@ static int amdgpu_pmops_runtime_suspend(struct device >> *dev) >> return 0; >> } >> >> +static void amdgpu_restore_perf_state_after_runpm(struct >> +amdgpu_device >> +*adev) { >> + enum amd_dpm_forced_level level; >> + uint32_t profile_mode_mask = AMD_DPM_FORCED_LEVEL_PROFILE_STANDARD | >> + AMD_DPM_FORCED_LEVEL_PROFILE_MIN_SCLK | >> + AMD_DPM_FORCED_LEVEL_PROFILE_MIN_MCLK | >> + AMD_DPM_FORCED_LEVEL_PROFILE_PEAK; >> + >> + level = amdgpu_dpm_get_performance_level(adev); >> + if (level & profile_mode_mask) >> + amdgpu_asic_update_umd_stable_pstate(adev, true); > > This is supposed to be called before entering the force dpm level mode. > It's better to move this logic inside swsmu late_init before > reapplying the level. > > Thanks, > Lijo > > Thanks, Lijo. > > I think this logic should remain in runtime resume. The > amdgpu_asic_update_umd_stable_pstate(adev, true) call here is a > restore/re-apply step for an already-selected profile forced level, not a > force-level entry. > Enter/exit behavior is already covered by the regular DPM force path > (amdgpu_dpm_force_performance_level() / SMU force handling) when level > transitions occur. > > During runpm suspend/resume, the software forced level can remain profile > mode, but HW-side stable pstate (and KFD profiler state) may be cleared by > hw_fini/hw_init. > That is why this restore is done after amdgpu_device_resume(). > > Moving this into swsmu late_init would apply the logic to broader flows (not > only runpm), so I’d prefer to keep it in runpm resume. > Profile setting is a user level setting and it's not the default SOC behavior. During runtime PM, chip is in ~D3 state and some settings are lost. For proper restoration, all the steps that are followed for a regular user profile setting needs to be done (you don't need to go by the 'force' name). That is done in the logic here also - it's checking if the level is one of the forced dpm levels. To properly reapply, it needs to follow the same process. This is followed in other user setting scenarios also. smu late init has a specific call to restore user setttings. Thanks, Lijo Thanks, Lijo. Good point - profile mode is user-level state and after runpm we should replay it through the same user-setting restore path, not only re-enable UMD stable pstate. I’ll move this logic into the swsmu late init restore flow (alongside user profile restoration) so profile reapply follows the same sequence as normal user profile programming. Then runtime-resume side will keep only kfd profiler restore hook. Thanks, Candice > > Thanks, > Candice > >> + >> + amdgpu_amdkfd_restore_profiler_after_runpm(adev); >> +} >> + >> static int amdgpu_pmops_runtime_resume(struct device *dev) { >> struct pci_dev *pdev = to_pci_dev(dev); @@ -2964,6 +2979,8 >> @@ static int amdgpu_pmops_runtime_resume(struct device *dev) >> >> if (adev->pm.rpm_mode == AMDGPU_RUNPM_PX) >> drm_dev->switch_power_state = DRM_SWITCH_POWER_ON; >> + >> + amdgpu_restore_perf_state_after_runpm(adev); >> adev->in_runpm = false; >> return 0; >> } >> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c >> b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c >> index 2c1a936459ac87..2a140c40b53f9c 100644 >> --- a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c >> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c >> @@ -3347,6 +3347,7 @@ static inline uint32_t profile_lock_device(struct >> kfd_process *p, >> if (op == 1) { >> if (!kfd->profiler_process) { >> kfd->profiler_process = p; >> + kfd->profiler_perfcount_enabled = false; >> status = 0; >> mutex_unlock(&kfd->profiler_lock); >> if (ptl->hw_supported) { @@ -3364,6 +3365,7 >> @@ static inline uint32_t profile_lock_device(struct kfd_process *p, >> } >> } else if (op == 0 && kfd->profiler_process == p) { >> kfd->profiler_process = NULL; >> + kfd->profiler_perfcount_enabled = false; >> status = 0; >> mutex_unlock(&kfd->profiler_lock); >> >> @@ -3404,6 +3406,12 @@ static inline int kfd_profiler_pmc(struct kfd_process >> *p, >> mutex_unlock(&p->mutex); >> >> dqm->ops.set_perfcount(dqm, args->perfcount_enable); >> + >> + mutex_lock(&pdd->dev->kfd->profiler_lock); >> + if (pdd->dev->kfd->profiler_process == p) >> + pdd->dev->kfd->profiler_perfcount_enabled = >> !!args->perfcount_enable; >> + mutex_unlock(&pdd->dev->kfd->profiler_lock); >> + >> return status; >> } >> >> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h >> b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h >> index 88191a4c165796..81055ada4f80b8 100644 >> --- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h >> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h >> @@ -389,6 +389,8 @@ struct kfd_dev { >> struct mutex profiler_lock; >> /* Process currently holding the lock */ >> struct kfd_process *profiler_process; >> + /* Whether perfcount is enabled for profiler_process */ >> + bool profiler_perfcount_enabled; >> }; >> >> enum kfd_mempool { >> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process.c >> b/drivers/gpu/drm/amd/amdkfd/kfd_process.c >> index eb508fe3ded780..8b7020fc0577a2 100644 >> --- a/drivers/gpu/drm/amd/amdkfd/kfd_process.c >> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_process.c >> @@ -1180,6 +1180,7 @@ static void kfd_process_profiler_release(struct >> kfd_process *p, struct kfd_proce >> if (pdd->dev->kfd->profiler_process == p) { >> pdd->qpd.dqm->ops.set_perfcount(pdd->qpd.dqm, 0); >> pdd->dev->kfd->profiler_process = NULL; >> + pdd->dev->kfd->profiler_perfcount_enabled = false; >> } >> mutex_unlock(&pdd->dev->kfd->profiler_lock); >> } >> -- >> 2.25.1 >> >
