subject:"\[Intel\-gfx\] ✗ Fi.CI.BAT\: failure for drm\/i915\: ttm for stolen"

Re: [Intel-gfx] ✗ Fi.CI.BAT: failure for drm/i915: ttm for stolen (rev5)

2022-07-01 Thread Tvrtko Ursulin

On 30/06/2022 15:20, Robert Beckett wrote:

On 29/06/2022 13:51, Robert Beckett wrote:

On 28/06/2022 17:22, Robert Beckett wrote:

On 28/06/2022 09:46, Tvrtko Ursulin wrote:

On 27/06/2022 18:08, Robert Beckett wrote:

On 22/06/2022 10:05, Tvrtko Ursulin wrote:

On 21/06/2022 20:11, Robert Beckett wrote:

On 21/06/2022 18:37, Patchwork wrote:

*Patch Details*
*Series:* drm/i915: ttm for stolen (rev5)
*URL:* https://patchwork.freedesktop.org/series/101396/

*State:* failure
*Details:*
https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_101396v5/index.html

CI Bug Log - changes from CI_DRM_11790 -> Patchwork_101396v5

Summary

*FAILURE*

Serious unknown changes coming with Patchwork_101396v5
absolutely need to be

verified manually.

If you think the reported changes have nothing to do with the
changes
introduced in Patchwork_101396v5, please notify your bug team to
allow them
to document this new failure mode, which will reduce false
positives in CI.

External URL:
https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_101396v5/index.html

Participating hosts (40 -> 41)

Additional (2): fi-icl-u2 bat-dg2-9
Missing (1): fi-bdw-samus

Possible new issues

Here are the unknown changes that may have been introduced in
Patchwork_101396v5:

IGT changes

Possible regressions

* igt@i915_selftest@live@reset:
o bat-adlp-4: PASS

-> DMESG-FAIL

I keep hitting clobbered pages during engine resets on bat-adlp-4.
It seems to happen most of the time on that machine and
occasionally on bat-adlp-6.

Should bat-adlp-4 be considered an unreliable machine like
bat-adlp-6 is for now?

Alternatively, seeing the history of this in

commit 3da3c5c1c9825c24168f27b021339e90af37e969 "drm/i915:
Exclude low pages (128KiB) of stolen from use"

could this be an indication that maybe the original issue is
worse on adlp machines?
I have only ever seen page page 135 or 136 clobbered across many
runs via trybot, so it looks fairly consistent.

Though excluding the use of over 540K of stolen might be too severe.

Don't know but I see that on the latest version you even hit pages
165/166.

Any history of hitting this in CI without your series? If not, are
there some other changes which could explain it? Are you touching
the selftest itself?

Hexdump of the clobbered page looks quite complex. Especially
POISON_FREE. Any idea how that ends up there?

(see
https://intel-gfx-ci.01.org/tree/drm-tip/Trybot_105517v4/fi-rkl-guc/igt@i915_selftest@l...@reset.html#dmesg-warnings702)

after lots of slow debug via CI, it looks like the issue is that a
ring buffer was allocated and taking up that page during the
initial crc capture in the test, but by the time it came to check
for corruption, it had been freed from that page.

The test has a number of weaknesses:

1. the busy check is done twice, without taking in to account any
change in between. I assume previously this could be relied on
never to occur, but now it can for some reason (more on that later)

You mean the stolen page used/unused test? Probably the premise is
that the test controls the driver completely ie. is the sole user
and the two checks are run at the time where nothing else could have
changed the state.

With the nerfed request (as with GuC) this actually should hold. In
the generic case I am less sure, my working knowledge faded a bit,
but perhaps there was something guaranteeing the spinner couldn't
have been retired yet at the time of the second check. Would need
clarifying at least in comments.

2. the engine reset returns early with an error for guc submission
engines, but it is silently ignored in the test. Perhaps it should
ignore guc submission engines as it is a largely useless test for
those situations.

Yes looks dodgy indeed. You will need to summon the owners of the
GuC backend to comment on this.

However even if the test should be skipped with GuC it is extremely
interesting that you are hitting this so I suspect there is a more
serious issue at play.

indeed. That's why I am keen to get to the root cause instead of just
slapping in a fix.

A quick obvious fix is to have a busy bitmask that remembers each
page's busy state initially and only check for corruption if it was
busy during both checks.

However, the main question is why this is occurring now with my
changes.
I have added more debug to check where the stolen memory is being
freed, but the first run last night didn't hit the issue for once.
I am running again now, will report back if I figure out where it
is being freed.

I am pretty

Re: [Intel-gfx] ✗ Fi.CI.BAT: failure for drm/i915: ttm for stolen (rev5)

2022-06-30 Thread Robert Beckett

On 30/06/2022 15:52, Hellstrom, Thomas wrote:

Hi!

On Thu, 2022-06-30 at 15:20 +0100, Robert Beckett wrote:

On 29/06/2022 13:51, Robert Beckett wrote:

On 28/06/2022 17:22, Robert Beckett wrote:

On 28/06/2022 09:46, Tvrtko Ursulin wrote:

On 27/06/2022 18:08, Robert Beckett wrote:

On 22/06/2022 10:05, Tvrtko Ursulin wrote:

On 21/06/2022 20:11, Robert Beckett wrote:

On 21/06/2022 18:37, Patchwork wrote:

*Patch Details*
*Series:* drm/i915: ttm for stolen (rev5)
*URL:*
https://patchwork.freedesktop.org/series/101396/

*State:* failure
*Details:*
https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_101396v5/index.html

https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_101396v5/index.html

CI Bug Log - changes from CI_DRM_11790 ->
Patchwork_101396v5

Summary

*FAILURE*

Serious unknown changes coming with Patchwork_101396v5
absolutely
need to be
verified manually.

If you think the reported changes have nothing to do
with the
changes
introduced in Patchwork_101396v5, please notify your
bug team to
allow them
to document this new failure mode, which will reduce
false
positives in CI.

External URL:
https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_101396v5/index.html

Participating hosts (40 -> 41)

Additional (2): fi-icl-u2 bat-dg2-9
Missing (1): fi-bdw-samus

Possible new issues

Here are the unknown changes that may have been
introduced in
Patchwork_101396v5:

IGT changes

Possible regressions

* igt@i915_selftest@live@reset:
o bat-adlp-4: PASS
<
https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11790/bat-adlp-4/igt@i915_selftest@l...@reset.html

-> DMESG-FAIL
<
https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_101396v5/bat-adlp-4/igt@i915_selftest@l...@reset.html

I keep hitting clobbered pages during engine resets on
bat-adlp-4.
It seems to happen most of the time on that machine and
occasionally on bat-adlp-6.

Should bat-adlp-4 be considered an unreliable machine
like
bat-adlp-6 is for now?

Alternatively, seeing the history of this in

commit 3da3c5c1c9825c24168f27b021339e90af37e969
"drm/i915: Exclude
low pages (128KiB) of stolen from use"

could this be an indication that maybe the original issue
is worse
on adlp machines?
I have only ever seen page page 135 or 136 clobbered
across many
runs via trybot, so it looks fairly consistent.
Though excluding the use of over 540K of stolen might be
too severe.

Don't know but I see that on the latest version you even
hit pages
165/166.

Any history of hitting this in CI without your series? If
not, are
there some other changes which could explain it? Are you
touching
the selftest itself?

Hexdump of the clobbered page looks quite complex.
Especially
POISON_FREE. Any idea how that ends up there?

(see
https://intel-gfx-ci.01.org/tree/drm-tip/Trybot_105517v4/fi-rkl-guc/igt@i915_selftest@l...@reset.html#dmesg-warnings702
)

after lots of slow debug via CI, it looks like the issue is
that a
ring buffer was allocated and taking up that page during the
initial
crc capture in the test, but by the time it came to check for
corruption, it had been freed from that page.

The test has a number of weaknesses:

1. the busy check is done twice, without taking in to account
any
change in between. I assume previously this could be relied
on never
to occur, but now it can for some reason (more on that later)

You mean the stolen page used/unused test? Probably the premise
is
that the test controls the driver completely ie. is the sole
user and
the two checks are run at the time where nothing else could
have
changed the state.

With the nerfed request (as with GuC) this actually should
hold. In
the generic case I am less sure, my working knowledge faded a
bit,
but perhaps there was something guaranteeing the spinner
couldn't
have been retired yet at the time of the second check. Would
need
clarifying at least in comments.

2. the engine reset returns early with an error for guc
submission
engines, but it is silently ignored in the test. Perhaps it
should
ignore guc submission engines as it is a largely useless test
for
those situations.

Yes looks dodgy indeed. You will need to summon the owners of
the GuC
backend to comment on this.

However even if the test should be skipped with GuC it is
extremely
interesting that you are hitting this so I suspect there is a
more
serious issue at play.

indeed. That's why I am keen to get to the root cause instead of
just
slapping in a fix.

A quick obvious fix is to have a busy bitmask that remembers
each
page's busy state initially and only check for corruption if
it was
busy during both checks.

However, the main question is why this is occurring now with
my
changes.
I have added more debug to check where the stolen memory is
being
freed, but the first run last night didn't hit the issue for
once.
I am running again now, will report back if I figure out
where it is
being

Re: [Intel-gfx] ✗ Fi.CI.BAT: failure for drm/i915: ttm for stolen (rev5)

2022-06-30 Thread Hellstrom, Thomas

Hi!

On Thu, 2022-06-30 at 15:20 +0100, Robert Beckett wrote:
> 
> 
> On 29/06/2022 13:51, Robert Beckett wrote:
> > 
> > 
> > On 28/06/2022 17:22, Robert Beckett wrote:
> > > 
> > > 
> > > On 28/06/2022 09:46, Tvrtko Ursulin wrote:
> > > > 
> > > > On 27/06/2022 18:08, Robert Beckett wrote:
> > > > > 
> > > > > 
> > > > > On 22/06/2022 10:05, Tvrtko Ursulin wrote:
> > > > > > 
> > > > > > On 21/06/2022 20:11, Robert Beckett wrote:
> > > > > > > 
> > > > > > > 
> > > > > > > On 21/06/2022 18:37, Patchwork wrote:
> > > > > > > > *Patch Details*
> > > > > > > > *Series:*    drm/i915: ttm for stolen (rev5)
> > > > > > > > *URL:*   
> > > > > > > > https://patchwork.freedesktop.org/series/101396/ 
> > > > > > > > 
> > > > > > > > *State:*    failure
> > > > > > > > *Details:* 
> > > > > > > > https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_101396v5/index.html
> > > > > > > >  
> > > > > > > > <
> > > > > > > > https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_101396v5/index.html
> > > > > > > > >
> > > > > > > > 
> > > > > > > > 
> > > > > > > > 
> > > > > > > >   CI Bug Log - changes from CI_DRM_11790 ->
> > > > > > > > Patchwork_101396v5
> > > > > > > > 
> > > > > > > > 
> > > > > > > >     Summary
> > > > > > > > 
> > > > > > > > *FAILURE*
> > > > > > > > 
> > > > > > > > Serious unknown changes coming with Patchwork_101396v5
> > > > > > > > absolutely 
> > > > > > > > need to be
> > > > > > > > verified manually.
> > > > > > > > 
> > > > > > > > If you think the reported changes have nothing to do
> > > > > > > > with the 
> > > > > > > > changes
> > > > > > > > introduced in Patchwork_101396v5, please notify your
> > > > > > > > bug team to 
> > > > > > > > allow them
> > > > > > > > to document this new failure mode, which will reduce
> > > > > > > > false 
> > > > > > > > positives in CI.
> > > > > > > > 
> > > > > > > > External URL: 
> > > > > > > > https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_101396v5/index.html
> > > > > > > >  
> > > > > > > > 
> > > > > > > > 
> > > > > > > > 
> > > > > > > >     Participating hosts (40 -> 41)
> > > > > > > > 
> > > > > > > > Additional (2): fi-icl-u2 bat-dg2-9
> > > > > > > > Missing (1): fi-bdw-samus
> > > > > > > > 
> > > > > > > > 
> > > > > > > >     Possible new issues
> > > > > > > > 
> > > > > > > > Here are the unknown changes that may have been
> > > > > > > > introduced in 
> > > > > > > > Patchwork_101396v5:
> > > > > > > > 
> > > > > > > > 
> > > > > > > >   IGT changes
> > > > > > > > 
> > > > > > > > 
> > > > > > > >     Possible regressions
> > > > > > > > 
> > > > > > > >   * igt@i915_selftest@live@reset:
> > > > > > > >   o bat-adlp-4: PASS
> > > > > > > > <
> > > > > > > > https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11790/bat-adlp-4/igt@i915_selftest@l...@reset.html
> > > > > > > > >
> > > > > > > > 
> > > > > > > >     -> DMESG-FAIL
> > > > > > > > <
> > > > > > > > https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_101396v5/bat-adlp-4/igt@i915_selftest@l...@reset.html
> > > > > > > > >
> > > > > > > > 
> > > > > > > > 
> > > > > > > 
> > > > > > > I keep hitting clobbered pages during engine resets on
> > > > > > > bat-adlp-4.
> > > > > > > It seems to happen most of the time on that machine and 
> > > > > > > occasionally on bat-adlp-6.
> > > > > > > 
> > > > > > > Should bat-adlp-4 be considered an unreliable machine
> > > > > > > like 
> > > > > > > bat-adlp-6 is for now?
> > > > > > > 
> > > > > > > Alternatively, seeing the history of this in
> > > > > > > 
> > > > > > > commit 3da3c5c1c9825c24168f27b021339e90af37e969
> > > > > > > "drm/i915: Exclude 
> > > > > > > low pages (128KiB) of stolen from use"
> > > > > > > 
> > > > > > > could this be an indication that maybe the original issue
> > > > > > > is worse 
> > > > > > > on adlp machines?
> > > > > > > I have only ever seen page page 135 or 136 clobbered
> > > > > > > across many 
> > > > > > > runs via trybot, so it looks fairly consistent.
> > > > > > > Though excluding the use of over 540K of stolen might be
> > > > > > > too severe.
> > > > > > 
> > > > > > Don't know but I see that on the latest version you even
> > > > > > hit pages 
> > > > > > 165/166.
> > > > > > 
> > > > > > Any history of hitting this in CI without your series? If
> > > > > > not, are 
> > > > > > there some other changes which could explain it? Are you
> > > > > > touching 
> > > > > > the selftest itself?
> > > > > > 
> > > > > > Hexdump of the clobbered page looks quite complex.
> > > > > > Especially 
> > > > > > POISON_FREE. Any idea how that ends up there?
> > > > > 
> > > > > 
> > > > > (see 
> > > > > https://intel-gfx-ci.01.org/tree/drm-tip/Trybot_105517v4/fi-rkl-guc/igt@i915_selftest@l...@reset.html#dmesg-warnings702
> > > > > )
> > > > > 
> > > > > 
> > > > > after lots of slow debug via CI, it looks like the issue is
> > > > > that a 
> > > > > ring buffer was allocated and taking up that page during the
> > > > >

Re: [Intel-gfx] ✗ Fi.CI.BAT: failure for drm/i915: ttm for stolen (rev5)

2022-06-30 Thread Robert Beckett

On 29/06/2022 13:51, Robert Beckett wrote:

On 28/06/2022 17:22, Robert Beckett wrote:

On 28/06/2022 09:46, Tvrtko Ursulin wrote:

On 27/06/2022 18:08, Robert Beckett wrote:

On 22/06/2022 10:05, Tvrtko Ursulin wrote:

On 21/06/2022 20:11, Robert Beckett wrote:

On 21/06/2022 18:37, Patchwork wrote:

*Patch Details*
*Series:* drm/i915: ttm for stolen (rev5)
*URL:* https://patchwork.freedesktop.org/series/101396/

*State:* failure
*Details:*
https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_101396v5/index.html

CI Bug Log - changes from CI_DRM_11790 -> Patchwork_101396v5

Summary

*FAILURE*

Serious unknown changes coming with Patchwork_101396v5 absolutely
need to be

verified manually.

External URL:
https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_101396v5/index.html

Participating hosts (40 -> 41)

Additional (2): fi-icl-u2 bat-dg2-9
Missing (1): fi-bdw-samus

Possible new issues

Here are the unknown changes that may have been introduced in
Patchwork_101396v5:

IGT changes

Possible regressions

* igt@i915_selftest@live@reset:
o bat-adlp-4: PASS

-> DMESG-FAIL

I keep hitting clobbered pages during engine resets on bat-adlp-4.
It seems to happen most of the time on that machine and
occasionally on bat-adlp-6.

Should bat-adlp-4 be considered an unreliable machine like
bat-adlp-6 is for now?

Alternatively, seeing the history of this in

commit 3da3c5c1c9825c24168f27b021339e90af37e969 "drm/i915: Exclude
low pages (128KiB) of stolen from use"

could this be an indication that maybe the original issue is worse
on adlp machines?
I have only ever seen page page 135 or 136 clobbered across many
runs via trybot, so it looks fairly consistent.

Though excluding the use of over 540K of stolen might be too severe.

Don't know but I see that on the latest version you even hit pages
165/166.

Any history of hitting this in CI without your series? If not, are
there some other changes which could explain it? Are you touching
the selftest itself?

Hexdump of the clobbered page looks quite complex. Especially
POISON_FREE. Any idea how that ends up there?

(see
https://intel-gfx-ci.01.org/tree/drm-tip/Trybot_105517v4/fi-rkl-guc/igt@i915_selftest@l...@reset.html#dmesg-warnings702)

after lots of slow debug via CI, it looks like the issue is that a
ring buffer was allocated and taking up that page during the initial
crc capture in the test, but by the time it came to check for
corruption, it had been freed from that page.

The test has a number of weaknesses:

1. the busy check is done twice, without taking in to account any
change in between. I assume previously this could be relied on never
to occur, but now it can for some reason (more on that later)

You mean the stolen page used/unused test? Probably the premise is
that the test controls the driver completely ie. is the sole user and
the two checks are run at the time where nothing else could have
changed the state.

Yes looks dodgy indeed. You will need to summon the owners of the GuC
backend to comment on this.

However even if the test should be skipped with GuC it is extremely
interesting that you are hitting this so I suspect there is a more
serious issue at play.

indeed. That's why I am keen to get to the root cause instead of just
slapping in a fix.

A quick obvious fix is to have a busy bitmask that remembers each
page's busy state initially and only check for corruption if it was
busy during both checks.

I am pretty sure the "corruption" (which isn't actually

Re: [Intel-gfx] ✗ Fi.CI.BAT: failure for drm/i915: ttm for stolen (rev5)

2022-06-29 Thread Robert Beckett

On 28/06/2022 17:22, Robert Beckett wrote:

On 28/06/2022 09:46, Tvrtko Ursulin wrote:

On 27/06/2022 18:08, Robert Beckett wrote:

On 22/06/2022 10:05, Tvrtko Ursulin wrote:

On 21/06/2022 20:11, Robert Beckett wrote:

On 21/06/2022 18:37, Patchwork wrote:

*Patch Details*
*Series:* drm/i915: ttm for stolen (rev5)
*URL:* https://patchwork.freedesktop.org/series/101396/

*State:* failure
*Details:*
https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_101396v5/index.html

CI Bug Log - changes from CI_DRM_11790 -> Patchwork_101396v5

Summary

*FAILURE*

Serious unknown changes coming with Patchwork_101396v5 absolutely
need to be

verified manually.

If you think the reported changes have nothing to do with the changes
introduced in Patchwork_101396v5, please notify your bug team to
allow them
to document this new failure mode, which will reduce false
positives in CI.