RE: [PATCH 3/3] drm/amdgpu: add gtt_sys_limit

2017-06-27 Thread He, Roger 
Yeah, recently we indeed encountered an issue as below:
On workstation machine it has 32G system memory and thus the GTT table size 
takes up about 192M visible VRAM.
Maybe if system has bigger memory, GTT table size will exceed 256M, so 
Limitation  for max GTT table size is very necessary.

Thanks
Roger(Hongbo.He)
-Original Message-
From: amd-gfx [mailto:amd-gfx-boun...@lists.freedesktop.org] On Behalf Of Felix 
Kuehling
Sent: Wednesday, June 28, 2017 3:25 AM
To: Christian König <deathsim...@vodafone.de>; Michel Dänzer 
<mic...@daenzer.net>; John Brooks <j...@fastquake.com>
Cc: amd-gfx@lists.freedesktop.org
Subject: Re: [PATCH 3/3] drm/amdgpu: add gtt_sys_limit


On 17-06-27 04:01 AM, Christian König wrote:
> Am 27.06.2017 um 04:57 schrieb Michel Dänzer:
>> On 27/06/17 08:39 AM, John Brooks wrote:
>>> On Mon, Jun 26, 2017 at 03:39:57PM +0200, Christian König wrote:
>>>> From: Christian König <christian.koe...@amd.com>
>>>>
>>>> Limit the size of the GART table for the system domain.
>>>>
>>>> This saves us a bunch of visible VRAM, but also limitates the 
>>>> maximum BO size we can swap out.
>>>>
>>>> Signed-off-by: Christian König <christian.koe...@amd.com>
>>> Hmm.
>>>
>>> On my system, GTT is 4096MiB (1048576 pages). For this, the table 
>>> takes up
>>> 1048576*8 bytes = 8MiB. Reducing GTT to 256MiB (65536 pages) would 
>>> reduce the size of the table to 512 KiB. A relatively large saving, 
>>> to be sure.
>>> But in the
>>> grander scheme of things, is saving 7.5MiB (3% of visible VRAM @
>>> 256M) worth
>>> cutting GTT memory by a factor of 16?
>
> The amount of GTT memory the application can use through the VM page 
> tables stays the same.
>
> Only the system VM is limited to 256MB and so saves us a whole bunch 
> of space.
>
>> I'm afraid not, especially since it would limit the maximum BO size 
>> to < 256MB, if I understand correctly. Pretty sure that would cause 
>> failures with real world apps.
> Yes, exactly that's the major problem here. I should have put a WIP 
> mark on the patch.

OK, I should adapt this for the KFD branch. Currently we make our GART much 
bigger. On a system with lots of system memory, we can use up half the visible 
VRAM for the GART. With something like this we can get it back to something 
reasonable. But a 256MB limit on single allocation size is definitely too small.

Also, if this breaks S3, we have to make sure the hybrid branches don't pick it 
up accidentally.

Regards,
  Felix

>
> Christian.

___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

RE: [PATCH 3/3] drm/amdgpu: add gtt_sys_limit

2017-06-27 Thread He, Roger 
Setting limitation for max GTT table size is necessary.
But my only concern is that is it enough for amdgpu_gart_sys_limit 256M. 
whether it will trigger GTT eviction from GTT to CPU domain easily?
Because GTT size always equals to VRAM size before it scales with system memory 
size.

Thanks
Roger(Hongbo.He)
-Original Message-
From: He, Roger 
Sent: Wednesday, June 28, 2017 9:58 AM
To: 'Felix Kuehling' <felix.kuehl...@amd.com>; Christian König 
<deathsim...@vodafone.de>; Michel Dänzer <mic...@daenzer.net>; John Brooks 
<j...@fastquake.com>
Cc: amd-gfx@lists.freedesktop.org
Subject: RE: [PATCH 3/3] drm/amdgpu: add gtt_sys_limit

Yeah, recently we indeed encountered an issue as below:
On workstation machine it has 32G system memory and thus the GTT table size 
takes up about 192M visible VRAM.
Maybe if system has bigger memory, GTT table size will exceed 256M, so 
Limitation  for max GTT table size is very necessary.

Thanks
Roger(Hongbo.He)
-Original Message-
From: amd-gfx [mailto:amd-gfx-boun...@lists.freedesktop.org] On Behalf Of Felix 
Kuehling
Sent: Wednesday, June 28, 2017 3:25 AM
To: Christian König <deathsim...@vodafone.de>; Michel Dänzer 
<mic...@daenzer.net>; John Brooks <j...@fastquake.com>
Cc: amd-gfx@lists.freedesktop.org
Subject: Re: [PATCH 3/3] drm/amdgpu: add gtt_sys_limit


On 17-06-27 04:01 AM, Christian König wrote:
> Am 27.06.2017 um 04:57 schrieb Michel Dänzer:
>> On 27/06/17 08:39 AM, John Brooks wrote:
>>> On Mon, Jun 26, 2017 at 03:39:57PM +0200, Christian König wrote:
>>>> From: Christian König <christian.koe...@amd.com>
>>>>
>>>> Limit the size of the GART table for the system domain.
>>>>
>>>> This saves us a bunch of visible VRAM, but also limitates the 
>>>> maximum BO size we can swap out.
>>>>
>>>> Signed-off-by: Christian König <christian.koe...@amd.com>
>>> Hmm.
>>>
>>> On my system, GTT is 4096MiB (1048576 pages). For this, the table 
>>> takes up
>>> 1048576*8 bytes = 8MiB. Reducing GTT to 256MiB (65536 pages) would 
>>> reduce the size of the table to 512 KiB. A relatively large saving, 
>>> to be sure.
>>> But in the
>>> grander scheme of things, is saving 7.5MiB (3% of visible VRAM @
>>> 256M) worth
>>> cutting GTT memory by a factor of 16?
>
> The amount of GTT memory the application can use through the VM page 
> tables stays the same.
>
> Only the system VM is limited to 256MB and so saves us a whole bunch 
> of space.
>
>> I'm afraid not, especially since it would limit the maximum BO size 
>> to < 256MB, if I understand correctly. Pretty sure that would cause 
>> failures with real world apps.
> Yes, exactly that's the major problem here. I should have put a WIP 
> mark on the patch.

OK, I should adapt this for the KFD branch. Currently we make our GART much 
bigger. On a system with lots of system memory, we can use up half the visible 
VRAM for the GART. With something like this we can get it back to something 
reasonable. But a 256MB limit on single allocation size is definitely too small.

Also, if this breaks S3, we have to make sure the hybrid branches don't pick it 
up accidentally.

Regards,
  Felix

>
> Christian.

___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 3/3] drm/amdgpu: add gtt_sys_limit

2017-06-27 Thread Felix Kuehling 

On 17-06-27 04:01 AM, Christian König wrote:
> Am 27.06.2017 um 04:57 schrieb Michel Dänzer:
>> On 27/06/17 08:39 AM, John Brooks wrote:
>>> On Mon, Jun 26, 2017 at 03:39:57PM +0200, Christian König wrote:
 From: Christian König 

 Limit the size of the GART table for the system domain.

 This saves us a bunch of visible VRAM, but also limitates the
 maximum BO size we can swap out.

 Signed-off-by: Christian König 
>>> Hmm.
>>>
>>> On my system, GTT is 4096MiB (1048576 pages). For this, the table
>>> takes up
>>> 1048576*8 bytes = 8MiB. Reducing GTT to 256MiB (65536 pages) would
>>> reduce the
>>> size of the table to 512 KiB. A relatively large saving, to be sure.
>>> But in the
>>> grander scheme of things, is saving 7.5MiB (3% of visible VRAM @
>>> 256M) worth
>>> cutting GTT memory by a factor of 16?
>
> The amount of GTT memory the application can use through the VM page
> tables stays the same.
>
> Only the system VM is limited to 256MB and so saves us a whole bunch
> of space.
>
>> I'm afraid not, especially since it would limit the maximum BO size to <
>> 256MB, if I understand correctly. Pretty sure that would cause failures
>> with real world apps.
> Yes, exactly that's the major problem here. I should have put a WIP
> mark on the patch.

OK, I should adapt this for the KFD branch. Currently we make our GART
much bigger. On a system with lots of system memory, we can use up half
the visible VRAM for the GART. With something like this we can get it
back to something reasonable. But a 256MB limit on single allocation
size is definitely too small.

Also, if this breaks S3, we have to make sure the hybrid branches don't
pick it up accidentally.

Regards,
  Felix

>
> Christian.

___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 3/3] drm/amdgpu: add gtt_sys_limit

2017-06-27 Thread Christian König 

Am 26.06.2017 um 22:12 schrieb Felix Kuehling:

I'm wondering what makes this possible. Let me quote the last discussion
we had about GART:


On 17-04-04 06:26 PM, Felix Kuehling wrote:

Even with GART address space being allocated on demand, it still seems
to be limiting the maximum available system memory that can be allocated
from TTM. We have a test that allocates a bunch of 128MB buffers. On a
32GB system memory system with a 4GB GPU it can only get 31 buffers, so
a bit under 4GB. Looks like BOs remain bound to GART after being
initialized or migrated to GTT. For KFD that limits the amount of usable
system memory, for amdgpu_cs, I think it limits the amount of system
memory that can be used in a single command submission.


That's why I added a new limit instead of modifying the existing one. 
The gartsize parameter still works as it did before.



On 17-04-05 02:55 AM, Christian König wrote:

Are these two effects intentional?

Yes, thought about dropping this as well but testing showed that it is
still necessary.

The total amount of memory bound to the GPU must be limited by the
GART size or otherwise the swapping code won't work any more. E.g.
suspend/resume fails immediately if I remove that.

Has that changed? I don't remember seeing any changes to that effect.


No, there are still a bunch of problems to solve. I should have put a 
WIP mark on that patch.


The maximum BO size is limited with this patch and will probably result 
in a bunch of S3 problems.


But as long as you don't care about those limitations it should work.

Christian.



Regards,
   Felix


On 17-06-26 09:39 AM, Christian König wrote:

From: Christian König 

Limit the size of the GART table for the system domain.

This saves us a bunch of visible VRAM, but also limitates the maximum BO size 
we can swap out.

Signed-off-by: Christian König 
---
  drivers/gpu/drm/amd/amdgpu/amdgpu.h | 2 ++
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  | 6 ++
  drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 4 
  drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c| 8 ++--
  drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 2 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_gtt_mgr.c | 6 --
  drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 6 +++---
  7 files changed, 26 insertions(+), 8 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h 
b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
index ab1dad2..a511029 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -76,6 +76,7 @@
  extern int amdgpu_modeset;
  extern int amdgpu_vram_limit;
  extern int amdgpu_gart_size;
+extern unsigned amdgpu_gart_sys_limit;
  extern int amdgpu_moverate;
  extern int amdgpu_benchmarking;
  extern int amdgpu_testing;
@@ -602,6 +603,7 @@ struct amdgpu_mc {
u64 mc_vram_size;
u64 visible_vram_size;
u64 gtt_size;
+   u64 gtt_sys_limit;
u64 gtt_start;
u64 gtt_end;
u64 vram_start;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index 44484bb..b6edb83 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -1122,6 +1122,12 @@ static void amdgpu_check_arguments(struct amdgpu_device 
*adev)
}
}
  
+	if (amdgpu_gart_sys_limit < 32) {

+   dev_warn(adev->dev, "gart sys limit (%d) too small\n",
+amdgpu_gart_sys_limit);
+   amdgpu_gart_sys_limit = 32;
+   }
+
amdgpu_check_vm_size(adev);
  
  	amdgpu_check_block_size(adev);

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index 5a1d794..907ae5e 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
@@ -75,6 +75,7 @@
  
  int amdgpu_vram_limit = 0;

  int amdgpu_gart_size = -1; /* auto */
+unsigned amdgpu_gart_sys_limit = 256;
  int amdgpu_moverate = -1; /* auto */
  int amdgpu_benchmarking = 0;
  int amdgpu_testing = 0;
@@ -124,6 +125,9 @@ module_param_named(vramlimit, amdgpu_vram_limit, int, 0600);
  MODULE_PARM_DESC(gartsize, "Size of PCIE/IGP gart to setup in megabytes (32, 64, 
etc., -1 = auto)");
  module_param_named(gartsize, amdgpu_gart_size, int, 0600);
  
+MODULE_PARM_DESC(gartlimit, "GART limit for the system domain in megabytes (default 256)");

+module_param_named(gartlimit, amdgpu_gart_sys_limit, int, 0600);
+
  MODULE_PARM_DESC(moverate, "Maximum buffer migration rate in MB/s. (32, 64, etc., 
-1=auto, 0=1=disabled)");
  module_param_named(moverate, amdgpu_moverate, int, 0600);
  
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c

index 8877015..5c6a461 100644
---

Re: [PATCH 3/3] drm/amdgpu: add gtt_sys_limit

2017-06-27 Thread Christian König 

Am 27.06.2017 um 04:57 schrieb Michel Dänzer:

On 27/06/17 08:39 AM, John Brooks wrote:

On Mon, Jun 26, 2017 at 03:39:57PM +0200, Christian König wrote:

From: Christian König 

Limit the size of the GART table for the system domain.

This saves us a bunch of visible VRAM, but also limitates the maximum BO size 
we can swap out.

Signed-off-by: Christian König 

Hmm.

On my system, GTT is 4096MiB (1048576 pages). For this, the table takes up
1048576*8 bytes = 8MiB. Reducing GTT to 256MiB (65536 pages) would reduce the
size of the table to 512 KiB. A relatively large saving, to be sure. But in the
grander scheme of things, is saving 7.5MiB (3% of visible VRAM @ 256M) worth
cutting GTT memory by a factor of 16?


The amount of GTT memory the application can use through the VM page 
tables stays the same.


Only the system VM is limited to 256MB and so saves us a whole bunch of 
space.



I'm afraid not, especially since it would limit the maximum BO size to <
256MB, if I understand correctly. Pretty sure that would cause failures
with real world apps.
Yes, exactly that's the major problem here. I should have put a WIP mark 
on the patch.


Christian.
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 3/3] drm/amdgpu: add gtt_sys_limit

2017-06-26 Thread Michel Dänzer 
On 27/06/17 08:39 AM, John Brooks wrote:
> On Mon, Jun 26, 2017 at 03:39:57PM +0200, Christian König wrote:
>> From: Christian König 
>>
>> Limit the size of the GART table for the system domain.
>>
>> This saves us a bunch of visible VRAM, but also limitates the maximum BO 
>> size we can swap out.
>>
>> Signed-off-by: Christian König 
> 
> Hmm.
> 
> On my system, GTT is 4096MiB (1048576 pages). For this, the table takes up
> 1048576*8 bytes = 8MiB. Reducing GTT to 256MiB (65536 pages) would reduce the
> size of the table to 512 KiB. A relatively large saving, to be sure. But in 
> the
> grander scheme of things, is saving 7.5MiB (3% of visible VRAM @ 256M) worth
> cutting GTT memory by a factor of 16?

I'm afraid not, especially since it would limit the maximum BO size to <
256MB, if I understand correctly. Pretty sure that would cause failures
with real world apps.


-- 
Earthling Michel Dänzer   |   http://www.amd.com
Libre software enthusiast | Mesa and X developer
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 3/3] drm/amdgpu: add gtt_sys_limit

2017-06-26 Thread John Brooks 
On Mon, Jun 26, 2017 at 03:39:57PM +0200, Christian König wrote:
> From: Christian König 
> 
> Limit the size of the GART table for the system domain.
> 
> This saves us a bunch of visible VRAM, but also limitates the maximum BO size 
> we can swap out.
> 
> Signed-off-by: Christian König 

Hmm.

On my system, GTT is 4096MiB (1048576 pages). For this, the table takes up
1048576*8 bytes = 8MiB. Reducing GTT to 256MiB (65536 pages) would reduce the
size of the table to 512 KiB. A relatively large saving, to be sure. But in the
grander scheme of things, is saving 7.5MiB (3% of visible VRAM @ 256M) worth
cutting GTT memory by a factor of 16?

John

> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu.h | 2 ++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  | 6 ++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 4 
>  drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c| 8 ++--
>  drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 2 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_gtt_mgr.c | 6 --
>  drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 6 +++---
>  7 files changed, 26 insertions(+), 8 deletions(-)
> 
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> index ab1dad2..a511029 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> @@ -76,6 +76,7 @@
>  extern int amdgpu_modeset;
>  extern int amdgpu_vram_limit;
>  extern int amdgpu_gart_size;
> +extern unsigned amdgpu_gart_sys_limit;
>  extern int amdgpu_moverate;
>  extern int amdgpu_benchmarking;
>  extern int amdgpu_testing;
> @@ -602,6 +603,7 @@ struct amdgpu_mc {
>   u64 mc_vram_size;
>   u64 visible_vram_size;
>   u64 gtt_size;
> + u64 gtt_sys_limit;
>   u64 gtt_start;
>   u64 gtt_end;
>   u64 vram_start;
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> index 44484bb..b6edb83 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -1122,6 +1122,12 @@ static void amdgpu_check_arguments(struct 
> amdgpu_device *adev)
>   }
>   }
>  
> + if (amdgpu_gart_sys_limit < 32) {
> + dev_warn(adev->dev, "gart sys limit (%d) too small\n",
> +  amdgpu_gart_sys_limit);
> + amdgpu_gart_sys_limit = 32;
> + }
> +
>   amdgpu_check_vm_size(adev);
>  
>   amdgpu_check_block_size(adev);
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> index 5a1d794..907ae5e 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> @@ -75,6 +75,7 @@
>  
>  int amdgpu_vram_limit = 0;
>  int amdgpu_gart_size = -1; /* auto */
> +unsigned amdgpu_gart_sys_limit = 256;
>  int amdgpu_moverate = -1; /* auto */
>  int amdgpu_benchmarking = 0;
>  int amdgpu_testing = 0;
> @@ -124,6 +125,9 @@ module_param_named(vramlimit, amdgpu_vram_limit, int, 
> 0600);
>  MODULE_PARM_DESC(gartsize, "Size of PCIE/IGP gart to setup in megabytes (32, 
> 64, etc., -1 = auto)");
>  module_param_named(gartsize, amdgpu_gart_size, int, 0600);
>  
> +MODULE_PARM_DESC(gartlimit, "GART limit for the system domain in megabytes 
> (default 256)");
> +module_param_named(gartlimit, amdgpu_gart_sys_limit, int, 0600);
> +
>  MODULE_PARM_DESC(moverate, "Maximum buffer migration rate in MB/s. (32, 64, 
> etc., -1=auto, 0=1=disabled)");
>  module_param_named(moverate, amdgpu_moverate, int, 0600);
>  
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c
> index 8877015..5c6a461 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c
> @@ -70,6 +70,9 @@ void amdgpu_gart_set_defaults(struct amdgpu_device *adev)
>   adev->mc.mc_vram_size);
>   else
>   adev->mc.gtt_size = (uint64_t)amdgpu_gart_size << 20;
> +
> + adev->mc.gtt_sys_limit = min((uint64_t)amdgpu_gart_sys_limit << 20,
> +  adev->mc.gtt_size);
>  }
>  
>  /**
> @@ -350,8 +353,9 @@ int amdgpu_gart_init(struct amdgpu_device *adev)
>   if (r)
>   return r;
>   /* Compute table size */
> - adev->gart.num_cpu_pages = adev->mc.gtt_size / PAGE_SIZE;
> - adev->gart.num_gpu_pages = adev->mc.gtt_size / AMDGPU_GPU_PAGE_SIZE;
> + adev->gart.num_cpu_pages = adev->mc.gtt_sys_limit / PAGE_SIZE;
> + adev->gart.num_gpu_pages = adev->mc.gtt_sys_limit /
> + AMDGPU_GPU_PAGE_SIZE;
>   DRM_INFO("GART: num cpu pages %u, num gpu pages %u\n",
>adev->gart.num_cpu_pages, adev->gart.num_gpu_pages);
>  
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c 
>

Re: [PATCH 3/3] drm/amdgpu: add gtt_sys_limit

2017-06-26 Thread Felix Kuehling 
I'm wondering what makes this possible. Let me quote the last discussion
we had about GART:

> On 17-04-04 06:26 PM, Felix Kuehling wrote:
>> Even with GART address space being allocated on demand, it still seems
>> to be limiting the maximum available system memory that can be allocated
>> from TTM. We have a test that allocates a bunch of 128MB buffers. On a
>> 32GB system memory system with a 4GB GPU it can only get 31 buffers, so
>> a bit under 4GB. Looks like BOs remain bound to GART after being
>> initialized or migrated to GTT. For KFD that limits the amount of usable
>> system memory, for amdgpu_cs, I think it limits the amount of system
>> memory that can be used in a single command submission.

> On 17-04-05 02:55 AM, Christian König wrote:
>>> Are these two effects intentional?
>> Yes, thought about dropping this as well but testing showed that it is
>> still necessary.
>>
>> The total amount of memory bound to the GPU must be limited by the
>> GART size or otherwise the swapping code won't work any more. E.g.
>> suspend/resume fails immediately if I remove that.
Has that changed? I don't remember seeing any changes to that effect.

Regards,
  Felix


On 17-06-26 09:39 AM, Christian König wrote:
> From: Christian König 
>
> Limit the size of the GART table for the system domain.
>
> This saves us a bunch of visible VRAM, but also limitates the maximum BO size 
> we can swap out.
>
> Signed-off-by: Christian König 
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu.h | 2 ++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  | 6 ++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 4 
>  drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c| 8 ++--
>  drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 2 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_gtt_mgr.c | 6 --
>  drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 6 +++---
>  7 files changed, 26 insertions(+), 8 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> index ab1dad2..a511029 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> @@ -76,6 +76,7 @@
>  extern int amdgpu_modeset;
>  extern int amdgpu_vram_limit;
>  extern int amdgpu_gart_size;
> +extern unsigned amdgpu_gart_sys_limit;
>  extern int amdgpu_moverate;
>  extern int amdgpu_benchmarking;
>  extern int amdgpu_testing;
> @@ -602,6 +603,7 @@ struct amdgpu_mc {
>   u64 mc_vram_size;
>   u64 visible_vram_size;
>   u64 gtt_size;
> + u64 gtt_sys_limit;
>   u64 gtt_start;
>   u64 gtt_end;
>   u64 vram_start;
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> index 44484bb..b6edb83 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -1122,6 +1122,12 @@ static void amdgpu_check_arguments(struct 
> amdgpu_device *adev)
>   }
>   }
>  
> + if (amdgpu_gart_sys_limit < 32) {
> + dev_warn(adev->dev, "gart sys limit (%d) too small\n",
> +  amdgpu_gart_sys_limit);
> + amdgpu_gart_sys_limit = 32;
> + }
> +
>   amdgpu_check_vm_size(adev);
>  
>   amdgpu_check_block_size(adev);
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> index 5a1d794..907ae5e 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> @@ -75,6 +75,7 @@
>  
>  int amdgpu_vram_limit = 0;
>  int amdgpu_gart_size = -1; /* auto */
> +unsigned amdgpu_gart_sys_limit = 256;
>  int amdgpu_moverate = -1; /* auto */
>  int amdgpu_benchmarking = 0;
>  int amdgpu_testing = 0;
> @@ -124,6 +125,9 @@ module_param_named(vramlimit, amdgpu_vram_limit, int, 
> 0600);
>  MODULE_PARM_DESC(gartsize, "Size of PCIE/IGP gart to setup in megabytes (32, 
> 64, etc., -1 = auto)");
>  module_param_named(gartsize, amdgpu_gart_size, int, 0600);
>  
> +MODULE_PARM_DESC(gartlimit, "GART limit for the system domain in megabytes 
> (default 256)");
> +module_param_named(gartlimit, amdgpu_gart_sys_limit, int, 0600);
> +
>  MODULE_PARM_DESC(moverate, "Maximum buffer migration rate in MB/s. (32, 64, 
> etc., -1=auto, 0=1=disabled)");
>  module_param_named(moverate, amdgpu_moverate, int, 0600);
>  
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c 
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c
> index 8877015..5c6a461 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c
> @@ -70,6 +70,9 @@ void amdgpu_gart_set_defaults(struct amdgpu_device *adev)
>   adev->mc.mc_vram_size);
>   else
>   adev->mc.gtt_size = (uint64_t)amdgpu_gart_size << 20;
> +
> +