[AMD Official Use Only - Internal Distribution Only]
Thanks Alex and Christian!
Hi Christian,
On ASICs with gmc v10, I find amdgpu_bo_late_init() is not invoked, so stolen
memory never get free, on other ASICs with gmc v9/v8/v7/v6, stolen memory was
freed in gmc_v*_0_late_init(). I don't know why, are there some special reasons
or just missed by coding?
Thanks!
Rico
________________________________
From: Christian König <ckoenig.leichtzumer...@gmail.com>
Sent: Wednesday, January 8, 2020 23:04
To: Alex Deucher <alexdeuc...@gmail.com>; Koenig, Christian
<christian.koe...@amd.com>
Cc: Long, Gang <gang.l...@amd.com>; Wang, Kevin(Yang) <kevin1.w...@amd.com>;
Xu, Feifei <feifei...@amd.com>; Yin, Tianci (Rico) <tianci....@amd.com>;
amd-gfx list <amd-gfx@lists.freedesktop.org>; Tuikov, Luben
<luben.tui...@amd.com>; Deucher, Alexander <alexander.deuc...@amd.com>; Yuan,
Xiaojie <xiaojie.y...@amd.com>; Zhang, Hawking <hawking.zh...@amd.com>
Subject: Re: [PATCH] drm/amdgpu: fix modprobe failure of the 2nd GPU when GDDR6
training enabled
Am 08.01.20 um 15:49 schrieb Alex Deucher:
> On Wed, Jan 8, 2020 at 7:52 AM Christian König <christian.koe...@amd.com>
> wrote:
>> Am 08.01.20 um 13:36 schrieb Tianci Yin:
>>> From: "Tianci.Yin" <tianci....@amd.com>
>>>
>>> [why]
>>> In dual GPUs scenario, stolen_size is assigned to zero on the 2nd GPU,
>>> then the bottom region of VRAM was allocated as GTT, unfortunately
>>> a small region of bottom VRAM was encroached by UMC firmware during
>>> GDDR6 BIST training, this cause pagefault.
>> What I'm missing here is why is the stolen size zero on the 2nd GPU?
>>
>> Maybe we need to read the stolen size after posting the GPU instead?
> There is no stolen memory on secondary GPUs because there is no pre-OS
> console using that memory.
Ah! Yeah that makes sense.
But in this case I would say we should change the patch like the following:
adev->gmc.stolen_size = min(adev->gmc.stolen_size,
AMDGPU_STOLEN_VGA_DEFAULT_SIZE);
And in amdgpu_ttm_late_init():
/* Can't free the stolen VGA memory when it might be used for memory
training again. */
if (!adev->fw_vram_usage.mem_train_support)
amdgpu_bo_free_kernel(....
Regards,
Christian.
>
> Alex
>
>> Regards,
>> Christian.
>>
>>> [how]
>>> Forcing stolen_size to 3MB, then the bottom region of VRAM was
>>> allocated as stolen memory, GTT corruption avoid.
>>> The stolen memory of the 2nd GPU will be free in late_init phase,
>>> no memory wasted.
>>>
>>> Change-Id: Icd0ad7de41333282949bb1e3e676c6c307ddd081
>>> Signed-off-by: Tianci.Yin <tianci....@amd.com>
>>> ---
>>> drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h | 6 ++++++
>>> drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c | 21 +++++++++++++++++++++
>>> 2 files changed, 27 insertions(+)
>>>
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>>> index c91dd602d5f1..440b793316df 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
>>> @@ -60,6 +60,11 @@
>>> */
>>> #define AMDGPU_GMC_FAULT_TIMEOUT 5000ULL
>>>
>>> +/*
>>> + * Default stolen memory size, 1024 * 768 * 4
>>> + */
>>> +#define AMDGPU_STOLEN_VGA_DEFAULT_SIZE 0x300000
>>> +
>>> struct firmware;
>>>
>>> /*
>>> @@ -192,6 +197,7 @@ struct amdgpu_gmc {
>>> uint32_t srbm_soft_reset;
>>> bool prt_warning;
>>> uint64_t stolen_size;
>>> + bool stolen_temp_reserved;
>>> /* apertures */
>>> u64 shared_aperture_start;
>>> u64 shared_aperture_end;
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c
>>> b/drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c
>>> index 7dc8c068c62a..0c96b67d6ca7 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c
>>> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c
>>> @@ -566,6 +566,11 @@ static int gmc_v10_0_late_init(void *handle)
>>> struct amdgpu_device *adev = (struct amdgpu_device *)handle;
>>> int r;
>>>
>>> + if (adev->gmc.stolen_temp_reserved) {
>>> + amdgpu_bo_late_init(adev);
>>> + adev->gmc.stolen_temp_reserved = false;
>>> + }
>>> +
>>> r = amdgpu_gmc_allocate_vm_inv_eng(adev);
>>> if (r)
>>> return r;
>>> @@ -756,6 +761,22 @@ static int gmc_v10_0_sw_init(void *handle)
>>> return r;
>>>
>>> adev->gmc.stolen_size = gmc_v10_0_get_vbios_fb_size(adev);
>>> + /*
>>> + * In dual GPUs scenario, stolen_size is assigned to zero on the 2nd
>>> GPU,
>>> + * then the bottom region of VRAM was allocated as GTT, unfortunately
>>> + * a small region of bottom VRAM was encroached by UMC firmware during
>>> + * GDDR6 BIST training, this cause pagefault.
>>> + * The page fault can be fixed by forcing stolen_size to 3MB, then
>>> the bottom
>>> + * region of VRAM was allocated as stolen memory, GTT corruption
>>> avoid.
>>> + * The stolen memory of the 2nd GPU will be free in late_init phase,
>>> + * no memory wasted.
>>> + */
>>> + if (adev->fw_vram_usage.mem_train_support &&
>>> + adev->gmc.stolen_size == 0) {
>>> + adev->gmc.stolen_size = AMDGPU_STOLEN_VGA_DEFAULT_SIZE;
>>> + adev->gmc.stolen_temp_reserved = true;
>>> + } else
>>> + adev->gmc.stolen_temp_reserved = false;
>>>
>>> /* Memory manager */
>>> r = amdgpu_bo_init(adev);
>> _______________________________________________
>> amd-gfx mailing list
>> amd-gfx@lists.freedesktop.org
>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=02%7C01%7Ctianci.yin%40amd.com%7C1365f16aef4c418f0db308d7944c0f26%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637140926723776401&sdata=ubt7GGRaAvLzv%2F%2BUXxaTW%2FgGbKkRKWv6%2BB50fHHW3Xc%3D&reserved=0
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=02%7C01%7Ctianci.yin%40amd.com%7C1365f16aef4c418f0db308d7944c0f26%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637140926723776401&sdata=ubt7GGRaAvLzv%2F%2BUXxaTW%2FgGbKkRKWv6%2BB50fHHW3Xc%3D&reserved=0
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
