Re: [Intel-gfx] ✗ Fi.CI.BAT: failure for drm/i915: Set PROBE_PREFER_ASYNCHRONOUS

2022-11-04 Thread Brian Norris
On Fri, Nov 04, 2022 at 02:38:03PM +, Matthew Auld wrote:
> On Thu, 3 Nov 2022 at 00:14, Brian Norris  wrote:
> > I'm still curious about the reported failures, but maybe they require
> > some particular sequence of tests? I also don't have the full
> > igt-gpu-tools set running, so maybe they do something a little
> > differently than my steps in [1]?
> >
> > Brian
> >
> > [1] I have a GLk system, if it matters. I figured I can run some of
> > these with any one of the following:
> >
> >   modprobe i915 live_selftests=1
> >   modprobe i915 live_selftests=1 igt__20__live_workarounds=Y
> >   modprobe i915 live_selftests=1 igt__19__live_uncore=Y
> >   modprobe i915 live_selftests=1 igt__18__live_sanitycheck=Y
> >   ...
> 
> CI should be using the IGT wrapper to run them, AFAIK. So something like:
> 
> ./build/tests/i915_selftest
> 
> Or to just run the live, mock or perf:
> 
> ./build/tests/i915_selftest --run-subtest live
> ./build/tests/i915_selftest --run-subtest mock
> ./build/tests/i915_selftest --run-subtest perf
> 
> Or if you want to run some particular selftest, like live mman tests:
> 
> ./build/tests/i915_selftest --run-subtest live --dyn mman

Thanks. I'm running through those now, and it seems like I'm doing
closer to what the CI logs show [1], but I'm still not reproducing on my
GLK. (I've now managed to run it with drm-tip; still no luck.)

So far, now I've managed to just reproduced *different* known problems:

https://lore.kernel.org/all/y2wfplbx1sedt...@google.com/

But after working around those, I run without any similar lockup
failures.

I might poke around some more next week, but I've probably spent more
time than reasonable on this already.

Anyway, thanks for the help!

Regards,
Brian

[1] For one, I've run through a test list, in order, based on this:
https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_110277v1/fi-glk-j4005/testlist0.txt


Re: [Intel-gfx] ✗ Fi.CI.BAT: failure for drm/i915: Set PROBE_PREFER_ASYNCHRONOUS

2022-11-04 Thread Matthew Auld
On Thu, 3 Nov 2022 at 00:14, Brian Norris  wrote:
>
> On Wed, Nov 02, 2022 at 12:18:37PM +, Matthew Auld wrote:
> > On Tue, 1 Nov 2022 at 21:58, Brian Norris  wrote:
> > >
> > > On Fri, Oct 28, 2022 at 5:24 PM Patchwork
> > >  wrote:
> > > >
> > > > Patch Details
> > > > Series:drm/i915: Set PROBE_PREFER_ASYNCHRONOUS
> > > > URL:https://patchwork.freedesktop.org/series/110277/
> > > > State:failure
> > > > Details:https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_110277v1/index.html
> > > >
> > > > CI Bug Log - changes from CI_DRM_12317 -> Patchwork_110277v1
> > > >
> > > > Summary
> > > >
> > > > FAILURE
> > > >
> > > > Serious unknown changes coming with Patchwork_110277v1 absolutely need 
> > > > to be
> > > > verified manually.
> > > >
> > > > If you think the reported changes have nothing to do with the changes
> > > > introduced in Patchwork_110277v1, please notify your bug team to allow 
> > > > them
> > > > to document this new failure mode, which will reduce false positives in 
> > > > CI.
> > > >
> > > > External URL: 
> > > > https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_110277v1/index.html
> > >
> > > For the record, I have almost zero idea what to do with this. From
> > > what I can tell, most (all?) of these failures are flaky(?) already
> > > and are probably not related to my change.
> >
> > https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_110277v1/index.html
> >
> > According to that link, this change appears to break every platform
> > when running the live selftests (looking at the purple squares).
> > Running the selftests normally involves loading and unloading the
> > module. Looking at the logs there is scary stuff like:
> >
> [...]
>
> Ah, thanks. I'm not sure what made me think the tests were failing the
> same way on drm-tip, but maybe just chalk that up to my unfamiliarity
> with this particular dashboard... (There are a few isolated failure
> and/or flakes on drm-tip, but they don't look like this.)
>
> Anyway, I think I managed to run some of these tests on my own platforms
> [1], and I don't reproduce those failures. I do see other failures
> (crashes) though, like in i915_gem_mman_live_selftests/igt_mmap, where
> igt_mmap_offset() (selftest-only code) -> vm_mmap() assumes we have a
> valid |current->mm|. But that's borrowing the modprobe process's memory
> map, and with async probe, the selftest sequence happens in a kernel
> worker instead (and current->mm is NULL). So that clearly won't work.

Semi related:
https://lore.kernel.org/intel-gfx/20221104134703.3770b371@maurocar-mobl2/T/#m888972bb1ffb0a913e3db8b4099dffdc2ec7a0dc

Sounds like a similar issue when trying to convert the live selftests
over to kunit.

>
> I suppose I could disable async probe when built as a module (I believe
> it doesn't really have any value, since the module load task just waits
> for the async task anyway). I'm not familiar enough with MM to know what
> the vm_mmap() alternatives are, but this particular bit of code does
> feel odd.
>
> Additionally, I think this implies that live_selftests will break if
> i915 is built-in (i.e., =y, not =m), as we'll again run in a
> kernel-thread context at boot time. But I would hope nobody is trying to
> run them that way? I guess this gets even hairier, because even if the
> driver is built into the kernel, it's possible to kick them off from a
> process context by tweaking the module parameters later, and then
> re-binding the device... So all in all, this bug leaves an ugly
> situation, with or without my patch.
>
> I'm still curious about the reported failures, but maybe they require
> some particular sequence of tests? I also don't have the full
> igt-gpu-tools set running, so maybe they do something a little
> differently than my steps in [1]?
>
> Brian
>
> [1] I have a GLk system, if it matters. I figured I can run some of
> these with any one of the following:
>
>   modprobe i915 live_selftests=1
>   modprobe i915 live_selftests=1 igt__20__live_workarounds=Y
>   modprobe i915 live_selftests=1 igt__19__live_uncore=Y
>   modprobe i915 live_selftests=1 igt__18__live_sanitycheck=Y
>   ...


Re: [Intel-gfx] ✗ Fi.CI.BAT: failure for drm/i915: Set PROBE_PREFER_ASYNCHRONOUS

2022-11-04 Thread Matthew Auld
On Thu, 3 Nov 2022 at 00:14, Brian Norris  wrote:
>
> On Wed, Nov 02, 2022 at 12:18:37PM +, Matthew Auld wrote:
> > On Tue, 1 Nov 2022 at 21:58, Brian Norris  wrote:
> > >
> > > On Fri, Oct 28, 2022 at 5:24 PM Patchwork
> > >  wrote:
> > > >
> > > > Patch Details
> > > > Series:drm/i915: Set PROBE_PREFER_ASYNCHRONOUS
> > > > URL:https://patchwork.freedesktop.org/series/110277/
> > > > State:failure
> > > > Details:https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_110277v1/index.html
> > > >
> > > > CI Bug Log - changes from CI_DRM_12317 -> Patchwork_110277v1
> > > >
> > > > Summary
> > > >
> > > > FAILURE
> > > >
> > > > Serious unknown changes coming with Patchwork_110277v1 absolutely need 
> > > > to be
> > > > verified manually.
> > > >
> > > > If you think the reported changes have nothing to do with the changes
> > > > introduced in Patchwork_110277v1, please notify your bug team to allow 
> > > > them
> > > > to document this new failure mode, which will reduce false positives in 
> > > > CI.
> > > >
> > > > External URL: 
> > > > https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_110277v1/index.html
> > >
> > > For the record, I have almost zero idea what to do with this. From
> > > what I can tell, most (all?) of these failures are flaky(?) already
> > > and are probably not related to my change.
> >
> > https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_110277v1/index.html
> >
> > According to that link, this change appears to break every platform
> > when running the live selftests (looking at the purple squares).
> > Running the selftests normally involves loading and unloading the
> > module. Looking at the logs there is scary stuff like:
> >
> [...]
>
> Ah, thanks. I'm not sure what made me think the tests were failing the
> same way on drm-tip, but maybe just chalk that up to my unfamiliarity
> with this particular dashboard... (There are a few isolated failure
> and/or flakes on drm-tip, but they don't look like this.)
>
> Anyway, I think I managed to run some of these tests on my own platforms
> [1], and I don't reproduce those failures. I do see other failures
> (crashes) though, like in i915_gem_mman_live_selftests/igt_mmap, where
> igt_mmap_offset() (selftest-only code) -> vm_mmap() assumes we have a
> valid |current->mm|. But that's borrowing the modprobe process's memory
> map, and with async probe, the selftest sequence happens in a kernel
> worker instead (and current->mm is NULL). So that clearly won't work.
>
> I suppose I could disable async probe when built as a module (I believe
> it doesn't really have any value, since the module load task just waits
> for the async task anyway). I'm not familiar enough with MM to know what
> the vm_mmap() alternatives are, but this particular bit of code does
> feel odd.
>
> Additionally, I think this implies that live_selftests will break if
> i915 is built-in (i.e., =y, not =m), as we'll again run in a
> kernel-thread context at boot time. But I would hope nobody is trying to
> run them that way? I guess this gets even hairier, because even if the
> driver is built into the kernel, it's possible to kick them off from a
> process context by tweaking the module parameters later, and then
> re-binding the device... So all in all, this bug leaves an ugly
> situation, with or without my patch.
>
> I'm still curious about the reported failures, but maybe they require
> some particular sequence of tests? I also don't have the full
> igt-gpu-tools set running, so maybe they do something a little
> differently than my steps in [1]?
>
> Brian
>
> [1] I have a GLk system, if it matters. I figured I can run some of
> these with any one of the following:
>
>   modprobe i915 live_selftests=1
>   modprobe i915 live_selftests=1 igt__20__live_workarounds=Y
>   modprobe i915 live_selftests=1 igt__19__live_uncore=Y
>   modprobe i915 live_selftests=1 igt__18__live_sanitycheck=Y
>   ...

CI should be using the IGT wrapper to run them, AFAIK. So something like:

./build/tests/i915_selftest

Or to just run the live, mock or perf:

./build/tests/i915_selftest --run-subtest live
./build/tests/i915_selftest --run-subtest mock
./build/tests/i915_selftest --run-subtest perf

Or if you want to run some particular selftest, like live mman tests:

./build/tests/i915_selftest --run-subtest live --dyn mman


Re: [Intel-gfx] ✗ Fi.CI.BAT: failure for drm/i915: Set PROBE_PREFER_ASYNCHRONOUS

2022-11-02 Thread Brian Norris
On Wed, Nov 02, 2022 at 12:18:37PM +, Matthew Auld wrote:
> On Tue, 1 Nov 2022 at 21:58, Brian Norris  wrote:
> >
> > On Fri, Oct 28, 2022 at 5:24 PM Patchwork
> >  wrote:
> > >
> > > Patch Details
> > > Series:drm/i915: Set PROBE_PREFER_ASYNCHRONOUS
> > > URL:https://patchwork.freedesktop.org/series/110277/
> > > State:failure
> > > Details:https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_110277v1/index.html
> > >
> > > CI Bug Log - changes from CI_DRM_12317 -> Patchwork_110277v1
> > >
> > > Summary
> > >
> > > FAILURE
> > >
> > > Serious unknown changes coming with Patchwork_110277v1 absolutely need to 
> > > be
> > > verified manually.
> > >
> > > If you think the reported changes have nothing to do with the changes
> > > introduced in Patchwork_110277v1, please notify your bug team to allow 
> > > them
> > > to document this new failure mode, which will reduce false positives in 
> > > CI.
> > >
> > > External URL: 
> > > https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_110277v1/index.html
> >
> > For the record, I have almost zero idea what to do with this. From
> > what I can tell, most (all?) of these failures are flaky(?) already
> > and are probably not related to my change.
> 
> https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_110277v1/index.html
> 
> According to that link, this change appears to break every platform
> when running the live selftests (looking at the purple squares).
> Running the selftests normally involves loading and unloading the
> module. Looking at the logs there is scary stuff like:
> 
[...]

Ah, thanks. I'm not sure what made me think the tests were failing the
same way on drm-tip, but maybe just chalk that up to my unfamiliarity
with this particular dashboard... (There are a few isolated failure
and/or flakes on drm-tip, but they don't look like this.)

Anyway, I think I managed to run some of these tests on my own platforms
[1], and I don't reproduce those failures. I do see other failures
(crashes) though, like in i915_gem_mman_live_selftests/igt_mmap, where
igt_mmap_offset() (selftest-only code) -> vm_mmap() assumes we have a
valid |current->mm|. But that's borrowing the modprobe process's memory
map, and with async probe, the selftest sequence happens in a kernel
worker instead (and current->mm is NULL). So that clearly won't work.

I suppose I could disable async probe when built as a module (I believe
it doesn't really have any value, since the module load task just waits
for the async task anyway). I'm not familiar enough with MM to know what
the vm_mmap() alternatives are, but this particular bit of code does
feel odd.

Additionally, I think this implies that live_selftests will break if
i915 is built-in (i.e., =y, not =m), as we'll again run in a
kernel-thread context at boot time. But I would hope nobody is trying to
run them that way? I guess this gets even hairier, because even if the
driver is built into the kernel, it's possible to kick them off from a
process context by tweaking the module parameters later, and then
re-binding the device... So all in all, this bug leaves an ugly
situation, with or without my patch.

I'm still curious about the reported failures, but maybe they require
some particular sequence of tests? I also don't have the full
igt-gpu-tools set running, so maybe they do something a little
differently than my steps in [1]?

Brian

[1] I have a GLk system, if it matters. I figured I can run some of
these with any one of the following:

  modprobe i915 live_selftests=1
  modprobe i915 live_selftests=1 igt__20__live_workarounds=Y
  modprobe i915 live_selftests=1 igt__19__live_uncore=Y
  modprobe i915 live_selftests=1 igt__18__live_sanitycheck=Y
  ...


Re: [Intel-gfx] ✗ Fi.CI.BAT: failure for drm/i915: Set PROBE_PREFER_ASYNCHRONOUS

2022-11-02 Thread Matthew Auld
On Tue, 1 Nov 2022 at 21:58, Brian Norris  wrote:
>
> On Fri, Oct 28, 2022 at 5:24 PM Patchwork
>  wrote:
> >
> > Patch Details
> > Series:drm/i915: Set PROBE_PREFER_ASYNCHRONOUS
> > URL:https://patchwork.freedesktop.org/series/110277/
> > State:failure
> > Details:https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_110277v1/index.html
> >
> > CI Bug Log - changes from CI_DRM_12317 -> Patchwork_110277v1
> >
> > Summary
> >
> > FAILURE
> >
> > Serious unknown changes coming with Patchwork_110277v1 absolutely need to be
> > verified manually.
> >
> > If you think the reported changes have nothing to do with the changes
> > introduced in Patchwork_110277v1, please notify your bug team to allow them
> > to document this new failure mode, which will reduce false positives in CI.
> >
> > External URL: 
> > https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_110277v1/index.html
>
> For the record, I have almost zero idea what to do with this. From
> what I can tell, most (all?) of these failures are flaky(?) already
> and are probably not related to my change.

https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_110277v1/index.html

According to that link, this change appears to break every platform
when running the live selftests (looking at the purple squares).
Running the selftests normally involves loading and unloading the
module. Looking at the logs there is scary stuff like:

<3> [371.938060] INFO: task kworker/u12:1:46 blocked for more than 61 seconds.
<3> [371.944992] Tainted: G U W 6.1.0-rc2-Patchwork_110277v1-g0f9cbc285f7d+ #1
<3> [371.953546] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
<6> [371.961428] task:kworker/u12:1 state:D stack:11512 pid:46 ppid:2
flags:0x4000
<6> [371.961433] Workqueue: events_unbound async_run_entry_fn
<6> [371.961452] Call Trace:
<6> [371.961454] 
<6> [371.961457] __schedule+0x30e/0xa70
<6> [371.961481] ? usleep_range_state+0xb0/0xb0
<6> [371.961484] ? __wait_for_common+0xd3/0x1d0
<6> [371.961487] schedule+0x4e/0xd0
<6> [371.961489] schedule_timeout+0x237/0x2e0
<6> [371.961493] ? usleep_range_state+0xb0/0xb0
<6> [371.961495] ? __wait_for_common+0xd3/0x1d0
<6> [371.961497] ? mark_held_locks+0x48/0x80
<6> [371.961500] ? _raw_spin_unlock_irq+0x1f/0x50
<6> [371.961504] ? usleep_range_state+0xb0/0xb0
<6> [371.961506] __wait_for_common+0xf5/0x1d0
<6> [371.961526] __flush_work+0x2c4/0x4e0
<6> [371.961530] ? flush_workqueue_prep_pwqs+0x120/0x120
<6> [371.961553] ? get_work_pool+0x90/0x90
<6> [371.961556] __cancel_work_timer+0x14e/0x1f0
<6> [371.961559] ? _raw_spin_unlock_irqrestore+0x54/0x70
<6> [371.961562] ? lockdep_hardirqs_on+0xbf/0x140
<6> [371.961567] intel_display_driver_unregister+0x27/0x40 [i915]
<6> [371.961722] i915_driver_remove+0x50/0x100 [i915]
<6> [371.961862] i915_pci_probe+0x123/0x240 [i915]
<6> [371.961996] ? _raw_spin_unlock_irqrestore+0x3d/0x70
<6> [371.962001] pci_device_probe+0x95/0x110
<6> [371.962005] really_probe+0xd6/0x350
<6> [371.962008] ? pm_runtime_barrier+0x4b/0xa0
<6> [371.962013] __driver_probe_device+0x73/0x170
<6> [371.962016] driver_probe_device+0x1a/0x90
<6> [371.962019] __driver_attach_async_helper+0x31/0x80
<6> [371.962023] async_run_entry_fn+0x28/0x130
<6> [371.962026] process_one_work+0x272/0x5b0
<6> [371.962032] worker_thread+0x37/0x370
<6> [371.962036] ? process_one_work+0x5b0/0x5b0
<6> [371.962038] kthread+0xed/0x120
<6> [371.962040] ? kthread_complete_and_exit+0x20/0x20
<6> [371.962044] ret_from_fork+0x1f/0x30
<6> [371.962053] 
<3> [371.962055] INFO: task kworker/3:1:57 blocked for more than 61 seconds.
<3> [371.968696] Tainted: G U W 6.1.0-rc2-Patchwork_110277v1-g0f9cbc285f7d+ #1
<3> [371.977219] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
<6> [371.985049] task:kworker/3:1 state:D stack:12488 pid:57 ppid:2
flags:0x4000
<6> [371.985053] Workqueue: events output_poll_execute [drm_kms_helper]
<6> [371.985080] Call Trace:
<6> [371.985081] 
<6> [371.985102] __schedule+0x30e/0xa70
<6> [371.985108] schedule+0x4e/0xd0
<6> [371.985110] async_synchronize_cookie_domain+0xe7/0x120
<6> [371.985113] ? finish_swait+0x90/0x90
<6> [371.985117] intel_fbdev_output_poll_changed+0x82/0x90 [i915]
<6> [371.985281] drm_kms_helper_hotplug_event+0x1e/0x30 [drm_kms_helper]
<6> [371.985291] output_poll_execute+0x1f5/0x200 [drm_kms_helper]
<6> [371.985303] process_one_work+0x272/0x5b0
<6> [371.985310] worker_thread+0x37/0x370
<6> [371.985313] ? process_one_work+0x5b0/0x5b0
<6> [371.985316] kthread+0xed/0x120
<6> [371.985317] ? kthread_complete_and_exit+0x20/0x20
<6> [371.985321] ret_from_fork+0x1f/0x30
<6> [371.985329] 
<3> [371.985348] INFO: task i915_selftest:6017 blocked for more than 61 seconds.


>
> But if someone believes to actually be a blocking issue with my patch,
> let me know, and I can see if I can figure out a better answer than
> that.
>
> Thanks,
> Brian


Re: [Intel-gfx] ✗ Fi.CI.BAT: failure for drm/i915: Set PROBE_PREFER_ASYNCHRONOUS

2022-11-01 Thread Brian Norris
On Fri, Oct 28, 2022 at 5:24 PM Patchwork
 wrote:
>
> Patch Details
> Series:drm/i915: Set PROBE_PREFER_ASYNCHRONOUS
> URL:https://patchwork.freedesktop.org/series/110277/
> State:failure
> Details:https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_110277v1/index.html
>
> CI Bug Log - changes from CI_DRM_12317 -> Patchwork_110277v1
>
> Summary
>
> FAILURE
>
> Serious unknown changes coming with Patchwork_110277v1 absolutely need to be
> verified manually.
>
> If you think the reported changes have nothing to do with the changes
> introduced in Patchwork_110277v1, please notify your bug team to allow them
> to document this new failure mode, which will reduce false positives in CI.
>
> External URL: 
> https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_110277v1/index.html

For the record, I have almost zero idea what to do with this. From
what I can tell, most (all?) of these failures are flaky(?) already
and are probably not related to my change.

But if someone believes to actually be a blocking issue with my patch,
let me know, and I can see if I can figure out a better answer than
that.

Thanks,
Brian


[Intel-gfx] ✗ Fi.CI.BAT: failure for drm/i915: Set PROBE_PREFER_ASYNCHRONOUS

2022-10-28 Thread Patchwork
== Series Details ==

Series: drm/i915: Set PROBE_PREFER_ASYNCHRONOUS
URL   : https://patchwork.freedesktop.org/series/110277/
State : failure

== Summary ==

CI Bug Log - changes from CI_DRM_12317 -> Patchwork_110277v1


Summary
---

  **FAILURE**

  Serious unknown changes coming with Patchwork_110277v1 absolutely need to be
  verified manually.
  
  If you think the reported changes have nothing to do with the changes
  introduced in Patchwork_110277v1, please notify your bug team to allow them
  to document this new failure mode, which will reduce false positives in CI.

  External URL: 
https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_110277v1/index.html

Participating hosts (40 -> 41)
--

  Additional (3): fi-kbl-soraka fi-tgl-dsi fi-pnv-d510 
  Missing(2): fi-ctg-p8600 fi-cfl-8700k 

Possible new issues
---

  Here are the unknown changes that may have been introduced in 
Patchwork_110277v1:

### IGT changes ###

 Possible regressions 

  * igt@i915_selftest@live@gt_engines:
- fi-apl-guc: [PASS][1] -> [INCOMPLETE][2]
   [1]: 
https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12317/fi-apl-guc/igt@i915_selftest@live@gt_engines.html
   [2]: 
https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_110277v1/fi-apl-guc/igt@i915_selftest@live@gt_engines.html

  * igt@i915_selftest@live@gt_timelines:
- fi-bdw-gvtdvm:  [PASS][3] -> [INCOMPLETE][4]
   [3]: 
https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12317/fi-bdw-gvtdvm/igt@i915_selftest@live@gt_timelines.html
   [4]: 
https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_110277v1/fi-bdw-gvtdvm/igt@i915_selftest@live@gt_timelines.html

  * igt@i915_selftest@live@sanitycheck:
- fi-bsw-nick:[PASS][5] -> [INCOMPLETE][6]
   [5]: 
https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12317/fi-bsw-nick/igt@i915_selftest@l...@sanitycheck.html
   [6]: 
https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_110277v1/fi-bsw-nick/igt@i915_selftest@l...@sanitycheck.html
- fi-hsw-g3258:   [PASS][7] -> [INCOMPLETE][8]
   [7]: 
https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12317/fi-hsw-g3258/igt@i915_selftest@l...@sanitycheck.html
   [8]: 
https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_110277v1/fi-hsw-g3258/igt@i915_selftest@l...@sanitycheck.html
- fi-bsw-kefka:   [PASS][9] -> [INCOMPLETE][10]
   [9]: 
https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12317/fi-bsw-kefka/igt@i915_selftest@l...@sanitycheck.html
   [10]: 
https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_110277v1/fi-bsw-kefka/igt@i915_selftest@l...@sanitycheck.html
- fi-elk-e7500:   [PASS][11] -> [INCOMPLETE][12]
   [11]: 
https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12317/fi-elk-e7500/igt@i915_selftest@l...@sanitycheck.html
   [12]: 
https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_110277v1/fi-elk-e7500/igt@i915_selftest@l...@sanitycheck.html
- fi-glk-j4005:   [PASS][13] -> [INCOMPLETE][14]
   [13]: 
https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12317/fi-glk-j4005/igt@i915_selftest@l...@sanitycheck.html
   [14]: 
https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_110277v1/fi-glk-j4005/igt@i915_selftest@l...@sanitycheck.html

  * igt@i915_selftest@live@uncore:
- fi-rkl-guc: [PASS][15] -> [INCOMPLETE][16]
   [15]: 
https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12317/fi-rkl-guc/igt@i915_selftest@l...@uncore.html
   [16]: 
https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_110277v1/fi-rkl-guc/igt@i915_selftest@l...@uncore.html
- fi-skl-guc: [PASS][17] -> [INCOMPLETE][18]
   [17]: 
https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12317/fi-skl-guc/igt@i915_selftest@l...@uncore.html
   [18]: 
https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_110277v1/fi-skl-guc/igt@i915_selftest@l...@uncore.html
- fi-rkl-11600:   [PASS][19] -> [INCOMPLETE][20]
   [19]: 
https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12317/fi-rkl-11600/igt@i915_selftest@l...@uncore.html
   [20]: 
https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_110277v1/fi-rkl-11600/igt@i915_selftest@l...@uncore.html
- fi-adl-ddr5:[PASS][21] -> [INCOMPLETE][22]
   [21]: 
https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12317/fi-adl-ddr5/igt@i915_selftest@l...@uncore.html
   [22]: 
https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_110277v1/fi-adl-ddr5/igt@i915_selftest@l...@uncore.html
- fi-cfl-guc: [PASS][23] -> [INCOMPLETE][24]
   [23]: 
https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12317/fi-cfl-guc/igt@i915_selftest@l...@uncore.html
   [24]: 
https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_110277v1/fi-cfl-guc/igt@i915_selftest@l...@uncore.html
- fi-skl-6700k2:  [PASS][25] -> [INCOMPLETE][26]
   [25]: 
https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12317/fi-skl-6700k2/igt@i915_selftest@l...@uncore.html
   [26]: 
https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_110277v1/fi-skl-6700k2/igt@i915_selftest@l...@uncore.html