v2: Some legacy ASICs do not send amdgpu_virt_request_init_data(). Only keep the full GPU access request early when request_init_data is not sent.
v1: Move the initialization of non-GPU resources out of the full GPU access region during AMDGPU device initialization. Background: In SR-IOV, the guest sends GPU_INIT_DATA, then the host enables VF_FB_EN and places early initialization data, such as IP discovery, VBIOS, and PF-VF exchange data, in the VF FB. The guest should then be able to read this data before requesting full GPU access. Before this patch, the VF still requested full GPU access in amdgpu_device_ip_early_init(). At that point TTM is not initialized yet, so the normal VRAM aperture mapping is unavailable and the guest falls back to MM_INDEX/MM_DATA register access. That register path requires full GPU access. Use the BAR0 framebuffer read path, amdgpu_device_read_fb_via_bar0(), for the early init-data copy instead of MM_INDEX/MM_DATA. This lets the driver delay the full GPU access request until after the early init data has been copied. Signed-off-by: chong li <[email protected]> Co-authored-by: Cursor <[email protected]> --- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 21 +++++++++++++++++---- drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c | 4 ++-- 2 files changed, 19 insertions(+), 6 deletions(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c index 610d82b79de3..ac66796e8634 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c @@ -38,6 +38,10 @@ #include <linux/apple-gmux.h> #include <linux/nospec.h> +#ifdef CONFIG_X86 +#include <asm/hypervisor.h> +#endif + #include <drm/drm_atomic_helper.h> #include <drm/drm_client_event.h> #include <drm/drm_crtc_helper.h> @@ -1987,16 +1991,19 @@ static int amdgpu_device_ip_early_init(struct amdgpu_device *adev) { struct amdgpu_ip_block *ip_block; struct pci_dev *parent; - bool total, skip_bios; + bool total, skip_bios, early_full_gpu_access = false; uint32_t bios_flags; int i, r; amdgpu_device_enable_virtual_display(adev); if (amdgpu_sriov_vf(adev)) { - r = amdgpu_virt_request_full_gpu(adev, true); - if (r) - return r; + early_full_gpu_access = (adev->virt.req_init_data_ver == 0); + if (early_full_gpu_access) { + r = amdgpu_virt_request_full_gpu(adev, true); + if (r) + return r; + } r = amdgpu_virt_init_critical_region(adev); if (r) @@ -2159,6 +2166,12 @@ static int amdgpu_device_ip_early_init(struct amdgpu_device *adev) if (!total) return -ENODEV; + if (amdgpu_sriov_vf(adev) && !early_full_gpu_access) { + r = amdgpu_virt_request_full_gpu(adev, true); + if (r) + return r; + } + if (adev->gmc.xgmi.supported) amdgpu_xgmi_early_init(adev); diff --git a/drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c b/drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c index 9a40107a0869..340703d89d6b 100644 --- a/drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +++ b/drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c @@ -185,8 +185,8 @@ static int xgpu_ai_send_access_requests(struct amdgpu_device *adev, } else if (req == IDH_REQ_GPU_INIT_DATA){ /* Dummy REQ_GPU_INIT_DATA handling */ r = xgpu_ai_poll_msg(adev, IDH_REQ_GPU_INIT_DATA_READY); - /* version set to 0 since dummy */ - adev->virt.req_init_data_ver = 0; + /* Version is set to 1 since GPU_CRIT_REGION_V1 */ + adev->virt.req_init_data_ver = GPU_CRIT_REGION_V1; } return 0; -- 2.48.1 Hi, Christian and Lijo. Sorry, I made a mistake. Host access to the VF FB is not platform-dependent. After the host driver sets VF_FB_EN in response to GPU_INIT_DATA, the host can access the VF FB. I borrowed an ESXi server and tested this patch there. The amdgpu driver works normally. I rewrote the background. Is anything still unclear? >On 07-Jul-26 5:25 PM, Christian K??nig wrote: >> On 7/7/26 13:30, chong li wrote: >>> v2: >>> Some legacy ASICs do not support amdgpu_virt_request_init_data(). >>> On platform ESXi, the PF can access VF VRAM only after the VF enters >>> full gpu access mode. >>> Only move the full access request later when the required conditions >>> are met >> >> This is adjusting the driver to the hypervisor and not the hypervisor to the >> driver. >> >> If the existing init order doesn't work on ESXi then that is an ESXi problem >> and needs to be fixed there. >> >> So still absolutely clear NAK. >> >>> >>> v1: >>> Move the initialization of non-GPU resources out of the full GPU >>> access region during AMDGPU device initialization >>> >>> background: >>> >>> After the amdgpu driver sends GPU_INIT_DATA to the host SR-IOV >>> driver, the host dumps the IP discovery/VBIOS/PF-VF exchange data into the >>> VF FB. >>> The VF can then read these data blocks without requesting full GPU access. >>> This is the purpose of the GPU_INIT_DATA event in the SR-IOV init flow. >>> >>> However, during ip_early_init, TTM is not initialized yet. >>> The current amdgpu driver therefore falls back to reading these data >>> through MM_INDEX/MM_DATA. That path requires register access, which >>> still needs full GPU access. >>> >>> As a result, even though amdgpu sends GPU_INIT_DATA and the host >>> prepares the init data early, the full GPU access window is not >>> reduced because the guest still needs full GPU access to copy the >>> data. >>> >>> To fix this, use amdgpu_device_read_fb_via_bar0() to copy the init >>> data from VF FB before TTM is ready. >>> For this early copy, the guest no longer needs full GPU access so the >>> full GPU access request can be moved later. >> >> You are just explaining the chicken and egg problem here instead of giving >> an actual justification. >> > >I guess there is no other technical reason to disallow accessing FB using >visible BAR range. The only reason of going through HDP indirect path could be >because the offsets could be outside of visible range. In SRIOV case, if host >driver guarantees discovery offsets in visible FB range, there is no reason to >disallow that. > >Thanks, >Lijo > > >> Regards, >> Christian. >> >>> >>> Signed-off-by: chong li <[email protected]> >>> Co-authored-by: Cursor <[email protected]> >>> --- >>> drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 32 +++++++++++++++++++--- >>> drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c | 4 +-- >>> 2 files changed, 30 insertions(+), 6 deletions(-) >>> >>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c >>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c >>> index 610d82b79de3..ec353f4dd0d2 100644 >>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c >>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c >>> @@ -38,6 +38,10 @@ >>> #include <linux/apple-gmux.h> >>> #include <linux/nospec.h> >>> >>> +#ifdef CONFIG_X86 >>> +#include <asm/hypervisor.h> >>> +#endif >>> + >>> #include <drm/drm_atomic_helper.h> >>> #include <drm/drm_client_event.h> >>> #include <drm/drm_crtc_helper.h> >>> @@ -1973,6 +1977,17 @@ static struct pci_dev >>> *amdgpu_device_find_parent(struct amdgpu_device *adev) >>> return parent; >>> } >>> >>> +static bool amdgpu_device_delay_full_gpu_access(struct amdgpu_device >>> +*adev) { #ifdef CONFIG_X86 >>> + return adev->virt.req_init_data_ver > 0 && >>> + (hypervisor_is_type(X86_HYPER_VMWARE) || >>> + hypervisor_is_type(X86_HYPER_MS_HYPERV)); >>> +#else >>> + return false; >>> +#endif >>> +} >>> + >>> /** >>> * amdgpu_device_ip_early_init - run early init for hardware IPs >>> * >>> @@ -1987,16 +2002,19 @@ static int amdgpu_device_ip_early_init(struct >>> amdgpu_device *adev) >>> { >>> struct amdgpu_ip_block *ip_block; >>> struct pci_dev *parent; >>> - bool total, skip_bios; >>> + bool total, skip_bios, delay_full_gpu_access = false; >>> uint32_t bios_flags; >>> int i, r; >>> >>> amdgpu_device_enable_virtual_display(adev); >>> >>> if (amdgpu_sriov_vf(adev)) { >>> - r = amdgpu_virt_request_full_gpu(adev, true); >>> - if (r) >>> - return r; >>> + delay_full_gpu_access = >>> amdgpu_device_delay_full_gpu_access(adev); >>> + if (!delay_full_gpu_access) { >>> + r = amdgpu_virt_request_full_gpu(adev, true); >>> + if (r) >>> + return r; >>> + } >>> >>> r = amdgpu_virt_init_critical_region(adev); >>> if (r) >>> @@ -2159,6 +2177,12 @@ static int amdgpu_device_ip_early_init(struct >>> amdgpu_device *adev) >>> if (!total) >>> return -ENODEV; >>> >>> + if (amdgpu_sriov_vf(adev) && delay_full_gpu_access) { >>> + r = amdgpu_virt_request_full_gpu(adev, true); >>> + if (r) >>> + return r; >>> + } >>> + >>> if (adev->gmc.xgmi.supported) >>> amdgpu_xgmi_early_init(adev); >>> >>> diff --git a/drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c >>> b/drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c >>> index 9a40107a0869..340703d89d6b 100644 >>> --- a/drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c >>> +++ b/drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c >>> @@ -185,8 +185,8 @@ static int xgpu_ai_send_access_requests(struct >>> amdgpu_device *adev, >>> } else if (req == IDH_REQ_GPU_INIT_DATA){ >>> /* Dummy REQ_GPU_INIT_DATA handling */ >>> r = xgpu_ai_poll_msg(adev, IDH_REQ_GPU_INIT_DATA_READY); >>> - /* version set to 0 since dummy */ >>> - adev->virt.req_init_data_ver = 0; >>> + /* Version is set to 1 since GPU_CRIT_REGION_V1 */ >>> + adev->virt.req_init_data_ver = GPU_CRIT_REGION_V1; >>> } >>> >>> return 0; >>
