Re: [PATCH] drm/amdgpu: disable job timeout on GPU reset disabled

2018-03-20 Thread Christian König 
That's a good point as well, maybe we should have separate timeouts for 
gfx and compute?


Something like 5 seconds for gfx and 1 minute (or even longer) for compute?

Anyway I agree that we can worry about that later on, patch is 
Reviewed-by: Christian König <christian.koe...@amd.com> for now.


Regards,
Christian.

Am 20.03.2018 um 15:16 schrieb Deucher, Alexander:


My concern was that compute will always have the timeout disabled with 
no way to override it even if you enable GPU reset.  I guess we can 
address that down the road.



Acked-by: Alex Deucher <alexander.deuc...@amd.com>


*From:* Koenig, Christian
*Sent:* Tuesday, March 20, 2018 6:14:29 AM
*To:* Quan, Evan; amd-gfx@lists.freedesktop.org
*Cc:* Deucher, Alexander
*Subject:* Re: [PATCH] drm/amdgpu: disable job timeout on GPU reset 
disabled

Hi Evan,

that one is perfect if you ask me. Just reading up on the history of
that patch, Alex what was your concern with that?

Regarding printing this as error, that's a really good point as well. We
should probably reduce it to a warning or even info severity.

Regards,
Christian.

Am 20.03.2018 um 03:11 schrieb Quan, Evan:
> Hi Christian,
>
> The messages prompted on timeout are Errors not just Warnings 
although we did not see any real problem(for the dgemm special case). 
That's why we say it confusing.

> And i suppose you want a fix like my previous patch(see attachment).
>
> Regards,
> Evan
>> -Original Message-
>> From: Christian König [mailto:ckoenig.leichtzumer...@gmail.com]
>> Sent: Monday, March 19, 2018 5:42 PM
>> To: Quan, Evan <evan.q...@amd.com>; amd-gfx@lists.freedesktop.org
>> Cc: Deucher, Alexander <alexander.deuc...@amd.com>
>> Subject: Re: [PATCH] drm/amdgpu: disable job timeout on GPU reset
>> disabled
>>
>> Am 19.03.2018 um 07:08 schrieb Evan Quan:
>>> Since under some heavy computing environment(dgemm test), it takes the
>>> asic over 10+ seconds to finish the dispatched single job which will
>>> trigger the timeout. It's quite confusing although it does not seem to
>>> bring any real problems.
>>> As a quick workround, we choose to disable timeout when GPU reset is
>>> disabled.
>> NAK, I enabled those warning intentionally even when the GPU 
recovery is

>> disabled to have a hint in the logs what goes wrong.
>>
>> Please only increase the timeout for the compute queue and/or add a
>> separate timeout for them.
>>
>> Regards,
>> Christian.
>>
>>
>>> Change-Id: I3a95d856ba4993094dc7b6269649e470c5b053d2
>>> Signed-off-by: Evan Quan <evan.q...@amd.com>
>>> ---
>>>    drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 7 +++
>>>    1 file changed, 7 insertions(+)
>>>
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>>> index 8bd9c3f..9d6a775 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>>> @@ -861,6 +861,13 @@ static void
>> amdgpu_device_check_arguments(struct amdgpu_device *adev)
>>>  amdgpu_lockup_timeout = 1;
>>>  }
>>>
>>> +   /*
>>> +    * Disable timeout when GPU reset is disabled to avoid confusing
>>> +    * timeout messages in the kernel log.
>>> +    */
>>> +   if (amdgpu_gpu_recovery == 0 || amdgpu_gpu_recovery == -1)
>>> +   amdgpu_lockup_timeout = INT_MAX;
>>> +
>>>  adev->firmware.load_type = amdgpu_ucode_get_load_type(adev,
>> amdgpu_fw_load_type);
>>>    }
>>>



___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH] drm/amdgpu: disable job timeout on GPU reset disabled

2018-03-20 Thread Christian König 

Hi Evan,

that one is perfect if you ask me. Just reading up on the history of 
that patch, Alex what was your concern with that?


Regarding printing this as error, that's a really good point as well. We 
should probably reduce it to a warning or even info severity.


Regards,
Christian.

Am 20.03.2018 um 03:11 schrieb Quan, Evan:

Hi Christian,

The messages prompted on timeout are Errors not just Warnings although we did 
not see any real problem(for the dgemm special case). That's why we say it 
confusing.
And i suppose you want a fix like my previous patch(see attachment).

Regards,
Evan

-Original Message-
From: Christian König [mailto:ckoenig.leichtzumer...@gmail.com]
Sent: Monday, March 19, 2018 5:42 PM
To: Quan, Evan <evan.q...@amd.com>; amd-gfx@lists.freedesktop.org
Cc: Deucher, Alexander <alexander.deuc...@amd.com>
Subject: Re: [PATCH] drm/amdgpu: disable job timeout on GPU reset
disabled

Am 19.03.2018 um 07:08 schrieb Evan Quan:

Since under some heavy computing environment(dgemm test), it takes the
asic over 10+ seconds to finish the dispatched single job which will
trigger the timeout. It's quite confusing although it does not seem to
bring any real problems.
As a quick workround, we choose to disable timeout when GPU reset is
disabled.

NAK, I enabled those warning intentionally even when the GPU recovery is
disabled to have a hint in the logs what goes wrong.

Please only increase the timeout for the compute queue and/or add a
separate timeout for them.

Regards,
Christian.



Change-Id: I3a95d856ba4993094dc7b6269649e470c5b053d2
Signed-off-by: Evan Quan <evan.q...@amd.com>
---
   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 7 +++
   1 file changed, 7 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index 8bd9c3f..9d6a775 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -861,6 +861,13 @@ static void

amdgpu_device_check_arguments(struct amdgpu_device *adev)

amdgpu_lockup_timeout = 1;
}

+   /*
+* Disable timeout when GPU reset is disabled to avoid confusing
+* timeout messages in the kernel log.
+*/
+   if (amdgpu_gpu_recovery == 0 || amdgpu_gpu_recovery == -1)
+   amdgpu_lockup_timeout = INT_MAX;
+
adev->firmware.load_type = amdgpu_ucode_get_load_type(adev,

amdgpu_fw_load_type);

   }



___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

RE: [PATCH] drm/amdgpu: disable job timeout on GPU reset disabled

2018-03-19 Thread Quan, Evan 
Hi Christian,

The messages prompted on timeout are Errors not just Warnings although we did 
not see any real problem(for the dgemm special case). That's why we say it 
confusing.
And i suppose you want a fix like my previous patch(see attachment).

Regards,
Evan
> -Original Message-
> From: Christian König [mailto:ckoenig.leichtzumer...@gmail.com]
> Sent: Monday, March 19, 2018 5:42 PM
> To: Quan, Evan <evan.q...@amd.com>; amd-gfx@lists.freedesktop.org
> Cc: Deucher, Alexander <alexander.deuc...@amd.com>
> Subject: Re: [PATCH] drm/amdgpu: disable job timeout on GPU reset
> disabled
> 
> Am 19.03.2018 um 07:08 schrieb Evan Quan:
> > Since under some heavy computing environment(dgemm test), it takes the
> > asic over 10+ seconds to finish the dispatched single job which will
> > trigger the timeout. It's quite confusing although it does not seem to
> > bring any real problems.
> > As a quick workround, we choose to disable timeout when GPU reset is
> > disabled.
> 
> NAK, I enabled those warning intentionally even when the GPU recovery is
> disabled to have a hint in the logs what goes wrong.
> 
> Please only increase the timeout for the compute queue and/or add a
> separate timeout for them.
> 
> Regards,
> Christian.
> 
> 
> >
> > Change-Id: I3a95d856ba4993094dc7b6269649e470c5b053d2
> > Signed-off-by: Evan Quan <evan.q...@amd.com>
> > ---
> >   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 7 +++
> >   1 file changed, 7 insertions(+)
> >
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > index 8bd9c3f..9d6a775 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > @@ -861,6 +861,13 @@ static void
> amdgpu_device_check_arguments(struct amdgpu_device *adev)
> > amdgpu_lockup_timeout = 1;
> > }
> >
> > +   /*
> > +* Disable timeout when GPU reset is disabled to avoid confusing
> > +* timeout messages in the kernel log.
> > +*/
> > +   if (amdgpu_gpu_recovery == 0 || amdgpu_gpu_recovery == -1)
> > +   amdgpu_lockup_timeout = INT_MAX;
> > +
> > adev->firmware.load_type = amdgpu_ucode_get_load_type(adev,
> amdgpu_fw_load_type);
> >   }
> >

--- Begin Message ---
Under some heavy computing test(dgemm) environment, it may takes
the asic over 50+ seconds to finish the dispatched single job
which will trigger the timeout. It's quite annoying although it
does not seem to bring any real problems.
As a quick workround, we choose to not enfoce the timeout
setting on compute queues.

Change-Id: I210011a90898617367e897a90e9f8fb2639281a3
Signed-off-by: Evan Quan <evan.q...@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c
index 008e198..455a81e 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c
@@ -435,7 +435,9 @@ int amdgpu_fence_driver_init_ring(struct amdgpu_ring *ring,
if (ring->funcs->type != AMDGPU_RING_TYPE_KIQ) {
r = drm_sched_init(>sched, _sched_ops,
   num_hw_submission, amdgpu_job_hang_limit,
-  msecs_to_jiffies(amdgpu_lockup_timeout), 
ring->name);
+  (ring->funcs->type == 
AMDGPU_RING_TYPE_COMPUTE) ?
+  MAX_SCHEDULE_TIMEOUT : 
msecs_to_jiffies(amdgpu_lockup_timeout),
+  ring->name);
if (r) {
DRM_ERROR("Failed to create scheduler on ring %s.\n",
  ring->name);
--
2.7.4

--- End Message ---
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH] drm/amdgpu: disable job timeout on GPU reset disabled

2018-03-19 Thread Christian König 

Am 19.03.2018 um 07:08 schrieb Evan Quan:

Since under some heavy computing environment(dgemm test), it takes
the asic over 10+ seconds to finish the dispatched single job
which will trigger the timeout. It's quite confusing although it
does not seem to bring any real problems.
As a quick workround, we choose to disable timeout when GPU reset
is disabled.


NAK, I enabled those warning intentionally even when the GPU recovery is 
disabled to have a hint in the logs what goes wrong.


Please only increase the timeout for the compute queue and/or add a 
separate timeout for them.


Regards,
Christian.




Change-Id: I3a95d856ba4993094dc7b6269649e470c5b053d2
Signed-off-by: Evan Quan 
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 7 +++
  1 file changed, 7 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index 8bd9c3f..9d6a775 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -861,6 +861,13 @@ static void amdgpu_device_check_arguments(struct 
amdgpu_device *adev)
amdgpu_lockup_timeout = 1;
}
  
+	/*

+* Disable timeout when GPU reset is disabled to avoid confusing
+* timeout messages in the kernel log.
+*/
+   if (amdgpu_gpu_recovery == 0 || amdgpu_gpu_recovery == -1)
+   amdgpu_lockup_timeout = INT_MAX;
+
adev->firmware.load_type = amdgpu_ucode_get_load_type(adev, 
amdgpu_fw_load_type);
  }
  


___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

RE: [PATCH] drm/amdgpu: disable job timeout on GPU reset disabled

2018-03-19 Thread Quan, Evan 
Hi Alex,

INT_MAX is used instead of MAX_SCHEDULE_TIMEOUT(which we discussed in another 
mail thread) since the amdgpu_lockup_timeout is with data type int.
Using MAX_SCHEDULE_TIMEOUT(data type:long) will get compile warnings.

Regards,
Evan
-Original Message-
From: Evan Quan [mailto:evan.q...@amd.com] 
Sent: Monday, March 19, 2018 2:08 PM
To: amd-gfx@lists.freedesktop.org
Cc: Deucher, Alexander <alexander.deuc...@amd.com>; Quan, Evan 
<evan.q...@amd.com>
Subject: [PATCH] drm/amdgpu: disable job timeout on GPU reset disabled

Since under some heavy computing environment(dgemm test), it takes the asic 
over 10+ seconds to finish the dispatched single job which will trigger the 
timeout. It's quite confusing although it does not seem to bring any real 
problems.
As a quick workround, we choose to disable timeout when GPU reset is disabled.

Change-Id: I3a95d856ba4993094dc7b6269649e470c5b053d2
Signed-off-by: Evan Quan <evan.q...@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 7 +++
 1 file changed, 7 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index 8bd9c3f..9d6a775 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -861,6 +861,13 @@ static void amdgpu_device_check_arguments(struct 
amdgpu_device *adev)
amdgpu_lockup_timeout = 1;
}
 
+   /*
+* Disable timeout when GPU reset is disabled to avoid confusing
+* timeout messages in the kernel log.
+*/
+   if (amdgpu_gpu_recovery == 0 || amdgpu_gpu_recovery == -1)
+   amdgpu_lockup_timeout = INT_MAX;
+
adev->firmware.load_type = amdgpu_ucode_get_load_type(adev, 
amdgpu_fw_load_type);  }
 
--
2.7.4

___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH] drm/amdgpu: disable job timeout on GPU reset disabled

2018-03-19 Thread Evan Quan 
Since under some heavy computing environment(dgemm test), it takes
the asic over 10+ seconds to finish the dispatched single job
which will trigger the timeout. It's quite confusing although it
does not seem to bring any real problems.
As a quick workround, we choose to disable timeout when GPU reset
is disabled.

Change-Id: I3a95d856ba4993094dc7b6269649e470c5b053d2
Signed-off-by: Evan Quan 
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 7 +++
 1 file changed, 7 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index 8bd9c3f..9d6a775 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -861,6 +861,13 @@ static void amdgpu_device_check_arguments(struct 
amdgpu_device *adev)
amdgpu_lockup_timeout = 1;
}
 
+   /*
+* Disable timeout when GPU reset is disabled to avoid confusing
+* timeout messages in the kernel log.
+*/
+   if (amdgpu_gpu_recovery == 0 || amdgpu_gpu_recovery == -1)
+   amdgpu_lockup_timeout = INT_MAX;
+
adev->firmware.load_type = amdgpu_ucode_get_load_type(adev, 
amdgpu_fw_load_type);
 }
 
-- 
2.7.4

___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx