AMD General But SMU debug mail box register is itself a part of PCIe MMIO BAR, we can't block PCIE BARs before SMU debug mail box write...
-----Original Message----- From: Koenig, Christian <[email protected]> Sent: Wednesday, September 30, 2026 3:56 PM To: Zhang, Yifan <[email protected]>; Lazar, Lijo <[email protected]>; Wang, Kevin <[email protected]>; Deucher, Alexander <[email protected]>; [email protected] Cc: Yuan, Perry <[email protected]> Subject: Re: [PATCH] amdgpu/pm: block GPU PCI bar access during mode1 reset on gfx12 If the SMU debug mail box register is the start of the mode 1 reset procedure and if that procedure requires that we stop VRAM and doorbell accesses then we need to do that *before* we do the SMU debug mail box write. Everything else is racy and not something we can do upstream. Regards, Christian. On 9/30/26 09:51, Zhang, Yifan wrote: > AMD General > > SMU debug mail box registers are accessed prior to disabling > PCI_COMMAND_MEMORY, and PSP bootloader status are accessed after enabling > PCI_COMMAND_MEMORY. There is no MMIO BAR access in b/w. > > -----Original Message----- > From: Koenig, Christian <[email protected]> > Sent: Wednesday, September 30, 2026 3:12 PM > To: Zhang, Yifan <[email protected]>; Lazar, Lijo > <[email protected]>; Wang, Kevin <[email protected]>; Deucher, > Alexander <[email protected]>; [email protected] > Cc: Yuan, Perry <[email protected]> > Subject: Re: [PATCH] amdgpu/pm: block GPU PCI bar access during mode1 > reset on gfx12 > > Well it doesn't matter who gets the request and what do we pool. > > The real question is if that goes over the MMIO BAR or the PCIe config space. > > If the SMU debug mailbox registers or PSP bootloader status is accessed over > the MMIO BAR space then I agree with Lijo that the approach of disabling all > MMIO BAR accesses won't work. > > Regards, > Christian. > > On 9/30/26 08:35, Zhang, Yifan wrote: >> AMD General >> >> >> On Navi3x/Navi4x, the driver doesn't poll for a response. It sends the mode1 >> reset request through the SMU debug mailbox and waits a fixed interval >> (SMU13_MODE1_RESET_WAIT_TIME_IN_MS) for the reset to proceed. After that, it >> polls the PSP bootloader status and the memsize register to confirm the ASIC >> is out of reset. On ASICs where the regular SMU message path is used, the >> polled response only acknowledges the message; it doesn't indicate that the >> reset has completed. >> >> >> >> *From:*Lazar, Lijo <[email protected]> >> *Sent:* Wednesday, September 30, 2026 2:28 PM >> *To:* Zhang, Yifan <[email protected]>; Wang, Kevin >> <[email protected]>; Deucher, Alexander <[email protected]>; >> Koenig, Christian <[email protected]>; >> [email protected] >> *Cc:* Yuan, Perry <[email protected]> >> *Subject:* Re: [PATCH] amdgpu/pm: block GPU PCI bar access during >> mode1 reset on gfx12 >> >> >> >> AMD General >> >> >> >> The actual reset sequence is run by the FW while you are polling for the >> response. This patch doesn't make any sense in that way. >> >> >> >> Thanks, >> >> Lijo >> >> --------------------------------------------------------------------- >> - >> --------------------------------------------------------------------- >> - >> --------------------------------------------------------------------- >> - >> --------------------------------------------------------------------- >> - >> --------------------------------------------------------------------- >> - >> --------------------------------------------------------------------- >> - >> --------------------------------------------------------------------- >> - >> --------------------------------------------------------------------- >> - >> --------------------------------------------------------------------- >> - >> --------------------------------------------------------------------- >> - >> --------------------------------------------------------------------- >> - >> --------------------------------------------------------------------- >> - >> --------------------------------------------------------------------- >> - >> --------------------------------------------------------------------- >> - >> ---------- >> >> *From:*Zhang, Yifan <[email protected] >> <mailto:[email protected]>> >> *Sent:* Wednesday, 30 September 2026 11:47:46 >> *To:* Wang, Kevin <[email protected] <mailto:[email protected]>>; >> Lazar, Lijo <[email protected] <mailto:[email protected]>>; >> Deucher, Alexander <[email protected] >> <mailto:[email protected]>>; Koenig, Christian >> <[email protected] <mailto:[email protected]>>; >> [email protected] <mailto:[email protected]> >> <[email protected] >> <mailto:[email protected]>> >> *Cc:* Yuan, Perry <[email protected] <mailto:[email protected]>> >> *Subject:* RE: [PATCH] amdgpu/pm: block GPU PCI bar access during >> mode1 reset on gfx12 >> >> >> >> AMD General >> >> Yes. In addition, the polling happens inside smu_cmn_send_smc_msg(), which >> runs before PCI_COMMAND_MEMORY is cleared. During the mode1 reset process >> there should be no traffic to the PCIe BARs; otherwise, the PCI bus may hang. >> >> Also, every PCI config write in the patch targets only the GPU device itself >> (pci_write_config_dword is per device), so no other PCI devices are affected. >> >> -----Original Message----- >> From: Wang, Kevin <[email protected] <mailto:[email protected]>> >> Sent: Wednesday, September 30, 2026 2:09 PM >> To: Lazar, Lijo <[email protected] <mailto:[email protected]>>; >> Zhang, Yifan <[email protected] <mailto:[email protected]>>; >> Deucher, Alexander <[email protected] >> <mailto:[email protected]>>; Koenig, Christian >> <[email protected] <mailto:[email protected]>>; >> [email protected] <mailto:[email protected]> >> Cc: Yuan, Perry <[email protected] <mailto:[email protected]>> >> Subject: RE: [PATCH] amdgpu/pm: block GPU PCI bar access during mode1 >> reset on gfx12 >> >> AMD General >> >>> On a second look, the patch doesn't seem right. Reset sequence is >>> supposed to be complete by the time FW responds back to the mode-1 reset >>> message. >>> During reset sequence, we need Mem space to be enabled to poll >>> success status of the message and that's most volatile window during >>> the whole sequence. >> >> That is not right. When the OS driver polls and sees the response register >> become "1", it does **NOT** mean the mode1 sequence is completed. >> The PMFW will continue forward the message to PSP and wait for PMFW ucode >> reload to finish at a 2nd-half stage. >> >> Best Regards, >> Kevin >> >>> -----Original Message----- >>> From: amd-gfx <[email protected] >>> <mailto:[email protected]>> On Behalf Of Lazar, >>> Lijo >>> Sent: Wednesday, September 30, 2026 1:41 PM >>> To: Zhang, Yifan <[email protected] >>> <mailto:[email protected]>>; Deucher, Alexander >>> <[email protected] <mailto:[email protected]>>; >>> Koenig, Christian <[email protected] >>> <mailto:[email protected]>>; [email protected] >>> <mailto:[email protected]> >>> Cc: Yuan, Perry <[email protected] <mailto:[email protected]>> >>> Subject: Re: [PATCH] amdgpu/pm: block GPU PCI bar access during >>> mode1 reset on gfx12 >>> >>> On a second look, the patch doesn't seem right. Reset sequence is >>> supposed to be complete by the time FW responds back to the mode-1 reset >>> message. >>> During reset sequence, we need Mem space to be enabled to poll >>> success status of the message and that's most volatile window during >>> the whole sequence. >>> >>> Disabling the memory space after reset sequence is complete looks to >>> be the wrong diagnosis of the issue. Access during reinit sequence >>> after reset is safe and shouldn't cause any issue in general. >>> >>> That said, we should try to resolve the issue without touching any >>> PCI config settings. PCI config space is open for others as well. >>> >>> Thanks, >>> Lijo >>> >>> On 30-Sep-26 10:57 AM, Zhang, Yifan wrote: >>>> AMD General >>>> >>>> How about disable UE during mode1 reset ? E.g. >>>> >>>> Before: >>>> pcie_capability_read_word(pdev, PCI_EXP_DEVCTL, >>>> &saved_devctl); >>>> pcie_capability_write_word(pdev, PCI_EXP_DEVCTL, >>>> saved_devctl & >>>> ~PCI_EXP_DEVCTL_URRE); >>>> pci_read_config_dword(pdev, aer + PCI_ERR_UNCOR_MASK, >>>> &saved_uemsk); >>>> pci_write_config_dword(pdev, aer + PCI_ERR_UNCOR_MASK, >>>> saved_uemsk | >>>> PCI_ERR_UNC_UNSUP); >>>> >>>> >>>> After: >>>> pci_write_config_dword(pdev, aer + PCI_ERR_UNCOR_MASK, >>>> saved_uemsk); >>>> pcie_capability_write_word(pdev, PCI_EXP_DEVCTL, >>>> saved_devctl); >>>> >>>> -----Original Message----- >>>> From: Lazar, Lijo <[email protected] <mailto:[email protected]>> >>>> Sent: Wednesday, September 30, 2026 11:37 AM >>>> To: Zhang, Yifan <[email protected] >>>> <mailto:[email protected]>>; Deucher, Alexander >>>> <[email protected] <mailto:[email protected]>>; >>>> Koenig, Christian <[email protected] >>>> <mailto:[email protected]>>; [email protected] >>>> <mailto:[email protected]> >>>> Cc: Yuan, Perry <[email protected] <mailto:[email protected]>> >>>> Subject: Re: [PATCH] amdgpu/pm: block GPU PCI bar access during >>>> mode1 reset on gfx12 >>>> >>>> >>>> This is the behaviour of this bit - >>>> >>>> Memory Space Enable - Controls a Function's response to >>>> Memory Space accesses. When this bit is Clear, all received Memory >>>> Space >>> accesses are caused to be handled as "Unsupported Requests" >>>> >>>> On AER enabled systems, depending on how the platform is configured >>>> to >>> classify the severity of UR errors, that could result in System error. >>>> >>>> On other hand, if you are sure that there is no such access during >>>> mode-1 reset, we don't need this patch in the first place. >>>> >>>> Thanks, >>>> Lijo >>>> >>>> >>>> On 30-Sep-26 8:53 AM, Zhang, Yifan wrote: >>>>> AMD General >>>>> >>>>> Hi Lijo, >>>>> >>>>> Why does clearing PCI_COMMAND_MEMORY cause AER errors on data >>> center GPUs? Are their PCIe BARs still being accessed during a mode1 reset? >>>>> >>>>> Best Regards, >>>>> Yifan >>>>> >>>>> -----Original Message----- >>>>> From: Lazar, Lijo <[email protected] <mailto:[email protected]>> >>>>> Sent: Wednesday, September 30, 2026 11:06 AM >>>>> To: Zhang, Yifan <[email protected] >>>>> <mailto:[email protected]>>; Deucher, Alexander >>>>> <[email protected] <mailto:[email protected]>>; >>>>> Koenig, Christian <[email protected] >>>>> <mailto:[email protected]>>; [email protected] >>>>> <mailto:[email protected]> >>>>> Cc: Yuan, Perry <[email protected] <mailto:[email protected]>> >>>>> Subject: Re: [PATCH] amdgpu/pm: block GPU PCI bar access during >>>>> mode1 reset on gfx12 >>>>> >>>>> This doesn't work for all (at least on some datacenter GPUs). With >>>>> AER >>> enabled, this will result in AER errors. Depending on the platform >>> config, some may classify this access as fatal. >>>>> >>>>> Thanks, >>>>> Lijo >>>>> >>>>> On 30-Sep-26 8:25 AM, Zhang, Yifan wrote: >>>>>> Public >>>>>> >>>>>> >>>>>> Hi Alex, >>>>>> >>>>>> I think it is reasonable to apply the patch to all ASICs. I will >>>>>> sync with the hardware team about this change; so far we have >>>>>> only tested it on Navi31. >>>>>> >>>>>> How about adding a common helper and calling it from >>>>>> ASIC-specific code, starting with Navi31? For example: >>>>>> >>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c >>>>>> b/drivers/gpu/ drm/amd/amdgpu/amdgpu_device.c >>>>>> >>>>>> index b690a07e31ab..075de681ab86 100644 >>>>>> >>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c >>>>>> >>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c >>>>>> >>>>>> @@ -4931,6 +4931,28 @@ bool >>> amdgpu_device_should_recover_gpu(struct >>>>>> amdgpu_device *adev) >>>>>> >>>>>> return false; >>>>>> >>>>>> } >>>>>> >>>>>> +void amdgpu_device_mode1_reset_wait(struct amdgpu_device *adev, >>>>>> >>>>>> + unsigned int delay_ms) >>>>>> >>>>>> +{ >>>>>> >>>>>> + u16 saved_cmd, cmd; >>>>>> >>>>>> + >>>>>> >>>>>> + pci_read_config_word(adev->pdev, PCI_COMMAND, >>>>>> +&saved_cmd); >>>>>> >>>>>> + cmd = saved_cmd & ~PCI_COMMAND_MEMORY; >>>>>> >>>>>> + pci_write_config_word(adev->pdev, PCI_COMMAND, cmd); >>>>>> >>>>>> + >>>>>> >>>>>> + msleep(delay_ms); >>>>>> >>>>>> + >>>>>> >>>>>> + pci_write_config_word(adev->pdev, PCI_COMMAND, >>>>>> +saved_cmd); >>>>>> >>>>>> +} >>>>>> >>>>>> + >>>>>> >>>>>> diff --git a/drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0.c >>>>>> b/drivers/ gpu/drm/amd/pm/swsmu/smu13/smu_v13_0.c >>>>>> >>>>>> index b250c5e41e45..356e09b7eef9 100644 >>>>>> >>>>>> --- a/drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0.c >>>>>> >>>>>> +++ b/drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0.c >>>>>> >>>>>> @@ -2242,7 +2242,8 @@ int smu_v13_0_mode1_reset(struct >>> smu_context >>>>>> *smu) >>>>>> >>>>>> ret = smu_cmn_send_smc_msg(smu, SMU_MSG_Mode1Reset, >>>>>> NULL); >>>>>> >>>>>> if (!ret) >>>>>> >>>>>> - msleep(SMU13_MODE1_RESET_WAIT_TIME_IN_MS); >>>>>> >>>>>> + amdgpu_device_mode1_reset_wait(smu->adev, >>>>>> >>>>>> + >>>>>> SMU13_MODE1_RESET_WAIT_TIME_IN_MS); >>>>>> >>>>>> return ret; >>>>>> >>>>>> } >>>>>> >>>>>> Best Regards, >>>>>> >>>>>> Yifan >>>>>> >>>>>> *From:*Deucher, Alexander <[email protected] >>>>>> <mailto:[email protected]>> >>>>>> *Sent:* Wednesday, September 30, 2026 2:59 AM >>>>>> *To:* Zhang, Yifan <[email protected] >>>>>> <mailto:[email protected]>>; Koenig, Christian >>>>>> <[email protected] <mailto:[email protected]>>; >>>>>> [email protected] >>>>>> <mailto:[email protected]> >>>>>> *Cc:* Yuan, Perry <[email protected] >>>>>> <mailto:[email protected]>> >>>>>> *Subject:* Re: [PATCH] amdgpu/pm: block GPU PCI bar access during >>>>>> mode1 reset on gfx12 >>>>>> >>>>>> Public >>>>>> >>>>>> I think this should be asic agnostic. Any asic doing mode1 reset >>>>>> should do this? >>>>>> >>>>>> Alex >>>>>> >>>>>> ----------------------------------------------------------------- >>>>>> - >>>>>> -- >>>>>> - >>>>>> - >>>>>> -- >>>>>> >>>>>> *From:*Zhang, Yifan <[email protected] >> <mailto:[email protected]%0b>> >>> <mailto:[email protected] >> <mailto:[email protected]>>> >>>>>> *Sent:* Tuesday, September 29, 2026 5:36 AM >>>>>> *To:* Koenig, Christian <[email protected] >> <mailto:[email protected]%0b>> >>> >> <mailto:[email protected] <mailto:[email protected]>>>; >> [email protected] <mailto:[email protected]> >>>>>> <mailto:[email protected] >>>>>> <mailto:[email protected]>> >>>>>> <[email protected] >> <mailto:[email protected]%0b>> >>> >> <mailto:[email protected] >> <mailto:[email protected]>>> >>>>>> *Cc:* Deucher, Alexander <[email protected] >> <mailto:[email protected]%0b>> >>> >> <mailto:[email protected] >> <mailto:[email protected]>>>; Yuan, Perry >>> <[email protected] >> <mailto:[email protected]%0b>> >>> <mailto:[email protected] >> <mailto:[email protected]>>> >>>>>> *Subject:* RE: [PATCH] amdgpu/pm: block GPU PCI bar access during >>>>>> mode1 reset on gfx12 >>>>>> >>>>>> Public >>>>>> >>>>>> Hi Christian, >>>>>> >>>>>> The no_hw_access flag was first set in commit >>>>>> 0de604d0357d0d22cbf03af1077d174b641707b6. This was intended to >>>>>> fix the Mode 1 reset hang, but it proved insufficient. >>>>>> I think the flag was set in the SMU code so that it would apply >>>>>> only to >>>>>> SMU13/14 ASICs, without affecting the Mode 1 reset process on >>>>>> other >>> ASICs. >>>>>> I'm not sure whether we can drop the no_hw_access flag after >>>>>> disabling PCI_COMMAND_MEMORY. The PCI hang occurs intermittently, >>> so >>>>>> we may need another round of stress testing to confirm that the >>>>>> flag is no longer needed. Should we remove no_hw_access in a >>>>>> separate >>> patch? >>>>>> >>>>>> Best Regards, >>>>>> Yifan >>>>>> >>>>>> -----Original Message----- >>>>>> From: Koenig, Christian <[email protected] >> <mailto:[email protected]%0b>> >>> >> <mailto:[email protected] <mailto:[email protected]>>> >>>>>> Sent: Tuesday, September 29, 2026 5:06 PM >>>>>> To: Zhang, Yifan <[email protected] >> <mailto:[email protected]%0b>> >>> <mailto:[email protected] >> <mailto:[email protected]>>>; [email protected] >> <mailto:[email protected]> >>>>>> <mailto:[email protected] >>>>>> <mailto:[email protected]>> >>>>>> Cc: Deucher, Alexander <[email protected] >> <mailto:[email protected]%0b>> >>> >> <mailto:[email protected] >> <mailto:[email protected]>>>; Yuan, Perry >>> <[email protected] >> <mailto:[email protected]%0b>> >>> <mailto:[email protected] >> <mailto:[email protected]>>> >>>>>> Subject: Re: [PATCH] amdgpu/pm: block GPU PCI bar access during >>>>>> mode1 reset on gfx12 >>>>>> >>>>>> On 9/29/26 04:25, Yifan Zhang wrote: >>>>>>> block GPU PCI bar access during mode1 reset on gfx12, to resolve >>>>>>> the random mode1 reset hang issue. >>>>>> >>>>>> Interesting idea, that is much simpler than trying to mess with >>>>>> CPU page tables in the GPU reset path. >>>>>> >>>>>> The only downside I can see is that we now drop VRAM and doorbell >>>>>> writes, but I think that won't matter since VRAM is lost by a >>>>>> mode1 reset anyway. >>>>>> >>>>>> MMIO writes are dropped as well which means that we can also drop >>>>>> the no_hw_access flag. >>>>>> >>>>>> And BTW the SMU code should absolutely *NOT* touch that >>> no_hw_access >>>>>> flag in any way. Who added that there? >>>>>> >>>>>> Regards, >>>>>> Christian. >>>>>> >>>>>>> >>>>>>> Signed-off-by: Yifan Zhang <[email protected] >> <mailto:[email protected]%0b>> >>>> <mailto:[email protected] >> <mailto:[email protected]>>> >>>>>>> --- >>>>>>> .../drm/amd/pm/swsmu/smu14/smu_v14_0_2_ppt.c | 29 >>>>>>> ++++++++++++------- >>>>>>> 1 file changed, 19 insertions(+), 10 deletions(-) >>>>>>> >>>>>>> diff --git >>> a/drivers/gpu/drm/amd/pm/swsmu/smu14/smu_v14_0_2_ppt.c >>>>>>> b/drivers/gpu/drm/amd/pm/swsmu/smu14/smu_v14_0_2_ppt.c >>>>>>> index 072d9c4e202d..25b9085ead71 100644 >>>>>>> --- a/drivers/gpu/drm/amd/pm/swsmu/smu14/smu_v14_0_2_ppt.c >>>>>>> +++ b/drivers/gpu/drm/amd/pm/swsmu/smu14/smu_v14_0_2_ppt.c >>>>>>> @@ -2078,21 +2078,30 @@ static int >>>>>>> smu_v14_0_2_set_df_cstate(struct smu_context *smu, >>>>>>> >>>>>>> static int smu_v14_0_2_mode1_reset(struct smu_context *smu) { >>>>>>> + struct amdgpu_device *adev = smu->adev; >>>>>>> + u16 saved_cmd, cmd; >>>>>>> int ret = 0; >>>>>>> >>>>>>> ret = smu_cmn_send_debug_smc_msg(smu, >>> DEBUGSMC_MSG_Mode1Reset); >>>>>>> - if (!ret) { >>>>>>> - if (amdgpu_emu_mode == 1) { >>>>>>> - msleep(50000); >>>>>>> - } else { >>>>>>> - /* disable mmio access while doing mode 1 >>>>>>> reset*/ >>>>>>> - smu->adev->no_hw_access = true; >>>>>>> - /* ensure no_hw_access is globally visible >>>>>>> before any MMIO */ >>>>>>> - smp_mb(); >>>>>>> - msleep(1000); >>>>>>> - } >>>>>>> + if (ret) >>>>>>> + return ret; >>>>>>> + >>>>>>> + pci_read_config_word(adev->pdev, PCI_COMMAND, &saved_cmd); >>>>>>> + cmd = saved_cmd & ~PCI_COMMAND_MEMORY; >>>>>>> + pci_write_config_word(adev->pdev, PCI_COMMAND, cmd); >>>>>>> + >>>>>>> + if (amdgpu_emu_mode == 1) { >>>>>>> + msleep(50000); >>>>>>> + } else { >>>>>>> + /* disable mmio access while doing mode 1 reset*/ >>>>>>> + smu->adev->no_hw_access = true; >>>>>>> + /* ensure no_hw_access is globally visible before >>>>>>> +any MMIO */ >>>>>>> + smp_mb(); >>>>>>> + msleep(1000); >>>>>>> } >>>>>>> >>>>>>> + pci_write_config_word(adev->pdev, PCI_COMMAND, saved_cmd); >>>>>>> + >>>>>>> return ret; >>>>>>> } >>>>>>> >>>>>> >>>>> >>>> >> >
