Re: [PATCH v5] drm: Add initial ci/ subdirectory
On 8/10/22 8:24 PM, Rodrigo Siqueira Jordao wrote:
Hi Tomeu,
First of all, nice patch! I just saw it, and I have some basic questions
(I don't understand many of these CI details). I also CC some CI folks
from the display team at AMD.
Thanks for the feedback!
On 2022-07-26 14:16, Tomeu Vizoso wrote:
And use it to store expectations about what the DRM drivers are
supposed to pass in the IGT test suite.
Also include a configuration file that points to the out-of-tree CI
scripts.
By storing the test expectations along the code we can make sure both
stay in sync with each other, and so we can know when a code change
breaks those expectations.
This will allow all contributors to drm to reuse the infrastructure
already in gitlab.freedesktop.org to test the driver on several
generations of the hardware.
v2:
- Fix names of result expectation files to match SoC
- Don't execute tests that are going to skip on all boards
v3:
- Remove tracking of dmesg output during test execution
v4:
- Move up to drivers/gpu/drm
- Add support for a bunch of other drivers
- Explain how to incorporate fixes for CI from a
${TARGET_BRANCH}-external-fixes branch
- Remove tests that pass from expected results file, to reduce the
size of in-tree files
- Add docs about how to deal with outages in automated testing labs
- Specify the exact SHA of the CI scripts to be used
v5:
- Remove unneeded skips from Meson expectations file
- Use a more advanced runner that detects flakes automatically
- Use a more succint format for the expectations
- Run many more tests (and use sharding to finish in time)
- Use skip lists to avoid hanging machines
- Add some build testing
- Build IGT in each pipeline for faster uprevs
- List failures in the GitLab UI
Signed-off-by: Tomeu Vizoso
Reviewed-by: Neil Armstrong
---
Documentation/gpu/automated_testing.rst | 84 ++
drivers/gpu/drm/ci/amdgpu-stoney-fails.txt | 13 +++
drivers/gpu/drm/ci/amdgpu-stoney-flakes.txt | 20 +
drivers/gpu/drm/ci/amdgpu-stoney-skips.txt | 2 +
drivers/gpu/drm/ci/gitlab-ci.yml | 13 +++
drivers/gpu/drm/ci/i915-amly-flakes.txt | 32 +++
drivers/gpu/drm/ci/i915-amly-skips.txt | 2 +
drivers/gpu/drm/ci/i915-apl-fails.txt | 29 +++
drivers/gpu/drm/ci/i915-apl-flakes.txt | 1 +
drivers/gpu/drm/ci/i915-apl-skips.txt | 2 +
drivers/gpu/drm/ci/i915-cml-flakes.txt | 36
drivers/gpu/drm/ci/i915-glk-flakes.txt | 40 +
drivers/gpu/drm/ci/i915-glk-skips.txt | 2 +
drivers/gpu/drm/ci/i915-kbl-fails.txt | 8 ++
drivers/gpu/drm/ci/i915-kbl-flakes.txt | 24 ++
drivers/gpu/drm/ci/i915-kbl-skips.txt | 2 +
drivers/gpu/drm/ci/i915-tgl-fails.txt | 19
drivers/gpu/drm/ci/i915-tgl-flakes.txt | 6 ++
drivers/gpu/drm/ci/i915-tgl-skips.txt | 8 ++
drivers/gpu/drm/ci/i915-whl-fails.txt | 30 +++
drivers/gpu/drm/ci/i915-whl-flakes.txt | 1 +
drivers/gpu/drm/ci/mediatek-mt8173-fails.txt | 29 +++
drivers/gpu/drm/ci/mediatek-mt8183-fails.txt | 10 +++
drivers/gpu/drm/ci/mediatek-mt8183-flakes.txt | 14 +++
drivers/gpu/drm/ci/meson-g12b-fails.txt | 5 ++
drivers/gpu/drm/ci/meson-g12b-flakes.txt | 4 +
drivers/gpu/drm/ci/msm-apq8016-fails.txt | 15
drivers/gpu/drm/ci/msm-apq8016-flakes.txt | 4 +
drivers/gpu/drm/ci/msm-apq8096-fails.txt | 2 +
drivers/gpu/drm/ci/msm-apq8096-flakes.txt | 4 +
drivers/gpu/drm/ci/msm-apq8096-skips.txt | 2 +
drivers/gpu/drm/ci/msm-sc7180-fails.txt | 22 +
drivers/gpu/drm/ci/msm-sc7180-flakes.txt | 14 +++
drivers/gpu/drm/ci/msm-sc7180-skips.txt | 18
drivers/gpu/drm/ci/msm-sdm845-fails.txt | 44 ++
drivers/gpu/drm/ci/msm-sdm845-flakes.txt | 33 +++
drivers/gpu/drm/ci/msm-sdm845-skips.txt | 2 +
drivers/gpu/drm/ci/rockchip-rk3288-fails.txt | 75
drivers/gpu/drm/ci/rockchip-rk3288-flakes.txt | 5 ++
drivers/gpu/drm/ci/rockchip-rk3288-skips.txt | 46 ++
drivers/gpu/drm/ci/rockchip-rk3399-fails.txt | 86 +++
drivers/gpu/drm/ci/rockchip-rk3399-flakes.txt | 25 ++
drivers/gpu/drm/ci/rockchip-rk3399-skips.txt | 5 ++
drivers/gpu/drm/ci/virtio_gpu-none-fails.txt | 38
drivers/gpu/drm/ci/virtio_gpu-none-flakes.txt | 0
drivers/gpu/drm/ci/virtio_gpu-none-skips.txt | 6 ++
46 files changed, 882 insertions(+)
create mode 100644 Documentation/gpu/automated_testing.rst
create mode 100644 drivers/gpu/drm/ci/amdgpu-stoney-fails.txt
create mode 100644 drivers/gpu/drm/ci/amdgpu-stoney-flakes.txt
create mode 100644 drivers/gpu/drm/ci/amdgpu-stoney-skips.txt
create mode 100644 drivers/gpu/drm/ci/gitlab-ci.yml
create mode 100644 drivers/gpu/drm/ci/i915-amly-flakes.txt
create mode 100644
Re: [PATCH v5] drm: Add initial ci/ subdirectory
On Wed, Aug 10, 2022 at 11:25 AM Rodrigo Siqueira Jordao
wrote:
>
> Hi Tomeu,
>
> First of all, nice patch! I just saw it, and I have some basic questions
> (I don't understand many of these CI details). I also CC some CI folks
> from the display team at AMD.
>
> On 2022-07-26 14:16, Tomeu Vizoso wrote:
> > And use it to store expectations about what the DRM drivers are
> > supposed to pass in the IGT test suite.
> >
> > Also include a configuration file that points to the out-of-tree CI
> > scripts.
> >
> > By storing the test expectations along the code we can make sure both
> > stay in sync with each other, and so we can know when a code change
> > breaks those expectations.
> >
> > This will allow all contributors to drm to reuse the infrastructure
> > already in gitlab.freedesktop.org to test the driver on several
> > generations of the hardware.
> >
> > v2:
> >- Fix names of result expectation files to match SoC
> >- Don't execute tests that are going to skip on all boards
> >
> > v3:
> >- Remove tracking of dmesg output during test execution
> >
> > v4:
> >- Move up to drivers/gpu/drm
> >- Add support for a bunch of other drivers
> >- Explain how to incorporate fixes for CI from a
> > ${TARGET_BRANCH}-external-fixes branch
> >- Remove tests that pass from expected results file, to reduce the
> > size of in-tree files
> >- Add docs about how to deal with outages in automated testing labs
> >- Specify the exact SHA of the CI scripts to be used
> >
> > v5:
> >- Remove unneeded skips from Meson expectations file
> >- Use a more advanced runner that detects flakes automatically
> >- Use a more succint format for the expectations
> >- Run many more tests (and use sharding to finish in time)
> >- Use skip lists to avoid hanging machines
> >- Add some build testing
> >- Build IGT in each pipeline for faster uprevs
> >- List failures in the GitLab UI
> >
> > Signed-off-by: Tomeu Vizoso
> > Reviewed-by: Neil Armstrong
> > ---
> > Documentation/gpu/automated_testing.rst | 84 ++
> > drivers/gpu/drm/ci/amdgpu-stoney-fails.txt| 13 +++
> > drivers/gpu/drm/ci/amdgpu-stoney-flakes.txt | 20 +
> > drivers/gpu/drm/ci/amdgpu-stoney-skips.txt| 2 +
> > drivers/gpu/drm/ci/gitlab-ci.yml | 13 +++
> > drivers/gpu/drm/ci/i915-amly-flakes.txt | 32 +++
> > drivers/gpu/drm/ci/i915-amly-skips.txt| 2 +
> > drivers/gpu/drm/ci/i915-apl-fails.txt | 29 +++
> > drivers/gpu/drm/ci/i915-apl-flakes.txt| 1 +
> > drivers/gpu/drm/ci/i915-apl-skips.txt | 2 +
> > drivers/gpu/drm/ci/i915-cml-flakes.txt| 36
> > drivers/gpu/drm/ci/i915-glk-flakes.txt| 40 +
> > drivers/gpu/drm/ci/i915-glk-skips.txt | 2 +
> > drivers/gpu/drm/ci/i915-kbl-fails.txt | 8 ++
> > drivers/gpu/drm/ci/i915-kbl-flakes.txt| 24 ++
> > drivers/gpu/drm/ci/i915-kbl-skips.txt | 2 +
> > drivers/gpu/drm/ci/i915-tgl-fails.txt | 19
> > drivers/gpu/drm/ci/i915-tgl-flakes.txt| 6 ++
> > drivers/gpu/drm/ci/i915-tgl-skips.txt | 8 ++
> > drivers/gpu/drm/ci/i915-whl-fails.txt | 30 +++
> > drivers/gpu/drm/ci/i915-whl-flakes.txt| 1 +
> > drivers/gpu/drm/ci/mediatek-mt8173-fails.txt | 29 +++
> > drivers/gpu/drm/ci/mediatek-mt8183-fails.txt | 10 +++
> > drivers/gpu/drm/ci/mediatek-mt8183-flakes.txt | 14 +++
> > drivers/gpu/drm/ci/meson-g12b-fails.txt | 5 ++
> > drivers/gpu/drm/ci/meson-g12b-flakes.txt | 4 +
> > drivers/gpu/drm/ci/msm-apq8016-fails.txt | 15
> > drivers/gpu/drm/ci/msm-apq8016-flakes.txt | 4 +
> > drivers/gpu/drm/ci/msm-apq8096-fails.txt | 2 +
> > drivers/gpu/drm/ci/msm-apq8096-flakes.txt | 4 +
> > drivers/gpu/drm/ci/msm-apq8096-skips.txt | 2 +
> > drivers/gpu/drm/ci/msm-sc7180-fails.txt | 22 +
> > drivers/gpu/drm/ci/msm-sc7180-flakes.txt | 14 +++
> > drivers/gpu/drm/ci/msm-sc7180-skips.txt | 18
> > drivers/gpu/drm/ci/msm-sdm845-fails.txt | 44 ++
> > drivers/gpu/drm/ci/msm-sdm845-flakes.txt | 33 +++
> > drivers/gpu/drm/ci/msm-sdm845-skips.txt | 2 +
> > drivers/gpu/drm/ci/rockchip-rk3288-fails.txt | 75
> > drivers/gpu/drm/ci/rockchip-rk3288-flakes.txt | 5 ++
> > drivers/gpu/drm/ci/rockchip-rk3288-skips.txt | 46 ++
> > drivers/gpu/drm/ci/rockchip-rk3399-fails.txt | 86 +++
> > drivers/gpu/drm/ci/rockchip-rk3399-flakes.txt | 25 ++
> > drivers/gpu/drm/ci/rockchip-rk3399-skips.txt | 5 ++
> > drivers/gpu/drm/ci/virtio_gpu-none-fails.txt | 38
> > drivers/gpu/drm/ci/virtio_gpu-none-flakes.txt | 0
> > drivers/gpu/drm/ci/virtio_gpu-none-skips.txt | 6 ++
> > 46 files changed, 882 insertions(+)
> > create mode 100644
Re: [PATCH v5] drm: Add initial ci/ subdirectory
Hi Tomeu,
First of all, nice patch! I just saw it, and I have some basic questions
(I don't understand many of these CI details). I also CC some CI folks
from the display team at AMD.
On 2022-07-26 14:16, Tomeu Vizoso wrote:
And use it to store expectations about what the DRM drivers are
supposed to pass in the IGT test suite.
Also include a configuration file that points to the out-of-tree CI
scripts.
By storing the test expectations along the code we can make sure both
stay in sync with each other, and so we can know when a code change
breaks those expectations.
This will allow all contributors to drm to reuse the infrastructure
already in gitlab.freedesktop.org to test the driver on several
generations of the hardware.
v2:
- Fix names of result expectation files to match SoC
- Don't execute tests that are going to skip on all boards
v3:
- Remove tracking of dmesg output during test execution
v4:
- Move up to drivers/gpu/drm
- Add support for a bunch of other drivers
- Explain how to incorporate fixes for CI from a
${TARGET_BRANCH}-external-fixes branch
- Remove tests that pass from expected results file, to reduce the
size of in-tree files
- Add docs about how to deal with outages in automated testing labs
- Specify the exact SHA of the CI scripts to be used
v5:
- Remove unneeded skips from Meson expectations file
- Use a more advanced runner that detects flakes automatically
- Use a more succint format for the expectations
- Run many more tests (and use sharding to finish in time)
- Use skip lists to avoid hanging machines
- Add some build testing
- Build IGT in each pipeline for faster uprevs
- List failures in the GitLab UI
Signed-off-by: Tomeu Vizoso
Reviewed-by: Neil Armstrong
---
Documentation/gpu/automated_testing.rst | 84 ++
drivers/gpu/drm/ci/amdgpu-stoney-fails.txt| 13 +++
drivers/gpu/drm/ci/amdgpu-stoney-flakes.txt | 20 +
drivers/gpu/drm/ci/amdgpu-stoney-skips.txt| 2 +
drivers/gpu/drm/ci/gitlab-ci.yml | 13 +++
drivers/gpu/drm/ci/i915-amly-flakes.txt | 32 +++
drivers/gpu/drm/ci/i915-amly-skips.txt| 2 +
drivers/gpu/drm/ci/i915-apl-fails.txt | 29 +++
drivers/gpu/drm/ci/i915-apl-flakes.txt| 1 +
drivers/gpu/drm/ci/i915-apl-skips.txt | 2 +
drivers/gpu/drm/ci/i915-cml-flakes.txt| 36
drivers/gpu/drm/ci/i915-glk-flakes.txt| 40 +
drivers/gpu/drm/ci/i915-glk-skips.txt | 2 +
drivers/gpu/drm/ci/i915-kbl-fails.txt | 8 ++
drivers/gpu/drm/ci/i915-kbl-flakes.txt| 24 ++
drivers/gpu/drm/ci/i915-kbl-skips.txt | 2 +
drivers/gpu/drm/ci/i915-tgl-fails.txt | 19
drivers/gpu/drm/ci/i915-tgl-flakes.txt| 6 ++
drivers/gpu/drm/ci/i915-tgl-skips.txt | 8 ++
drivers/gpu/drm/ci/i915-whl-fails.txt | 30 +++
drivers/gpu/drm/ci/i915-whl-flakes.txt| 1 +
drivers/gpu/drm/ci/mediatek-mt8173-fails.txt | 29 +++
drivers/gpu/drm/ci/mediatek-mt8183-fails.txt | 10 +++
drivers/gpu/drm/ci/mediatek-mt8183-flakes.txt | 14 +++
drivers/gpu/drm/ci/meson-g12b-fails.txt | 5 ++
drivers/gpu/drm/ci/meson-g12b-flakes.txt | 4 +
drivers/gpu/drm/ci/msm-apq8016-fails.txt | 15
drivers/gpu/drm/ci/msm-apq8016-flakes.txt | 4 +
drivers/gpu/drm/ci/msm-apq8096-fails.txt | 2 +
drivers/gpu/drm/ci/msm-apq8096-flakes.txt | 4 +
drivers/gpu/drm/ci/msm-apq8096-skips.txt | 2 +
drivers/gpu/drm/ci/msm-sc7180-fails.txt | 22 +
drivers/gpu/drm/ci/msm-sc7180-flakes.txt | 14 +++
drivers/gpu/drm/ci/msm-sc7180-skips.txt | 18
drivers/gpu/drm/ci/msm-sdm845-fails.txt | 44 ++
drivers/gpu/drm/ci/msm-sdm845-flakes.txt | 33 +++
drivers/gpu/drm/ci/msm-sdm845-skips.txt | 2 +
drivers/gpu/drm/ci/rockchip-rk3288-fails.txt | 75
drivers/gpu/drm/ci/rockchip-rk3288-flakes.txt | 5 ++
drivers/gpu/drm/ci/rockchip-rk3288-skips.txt | 46 ++
drivers/gpu/drm/ci/rockchip-rk3399-fails.txt | 86 +++
drivers/gpu/drm/ci/rockchip-rk3399-flakes.txt | 25 ++
drivers/gpu/drm/ci/rockchip-rk3399-skips.txt | 5 ++
drivers/gpu/drm/ci/virtio_gpu-none-fails.txt | 38
drivers/gpu/drm/ci/virtio_gpu-none-flakes.txt | 0
drivers/gpu/drm/ci/virtio_gpu-none-skips.txt | 6 ++
46 files changed, 882 insertions(+)
create mode 100644 Documentation/gpu/automated_testing.rst
create mode 100644 drivers/gpu/drm/ci/amdgpu-stoney-fails.txt
create mode 100644 drivers/gpu/drm/ci/amdgpu-stoney-flakes.txt
create mode 100644 drivers/gpu/drm/ci/amdgpu-stoney-skips.txt
create mode 100644 drivers/gpu/drm/ci/gitlab-ci.yml
create mode 100644 drivers/gpu/drm/ci/i915-amly-flakes.txt
create mode 100644 drivers/gpu/drm/ci/i915-amly-skips.txt
create mode 100644
Re: [PATCH v5] drm: Add initial ci/ subdirectory
On Tue, Jul 26, 2022 at 11:17 AM Tomeu Vizoso
wrote:
>
> And use it to store expectations about what the DRM drivers are
> supposed to pass in the IGT test suite.
>
> Also include a configuration file that points to the out-of-tree CI
> scripts.
>
> By storing the test expectations along the code we can make sure both
> stay in sync with each other, and so we can know when a code change
> breaks those expectations.
>
> This will allow all contributors to drm to reuse the infrastructure
> already in gitlab.freedesktop.org to test the driver on several
> generations of the hardware.
>
> v2:
> - Fix names of result expectation files to match SoC
> - Don't execute tests that are going to skip on all boards
>
> v3:
> - Remove tracking of dmesg output during test execution
>
> v4:
> - Move up to drivers/gpu/drm
> - Add support for a bunch of other drivers
> - Explain how to incorporate fixes for CI from a
> ${TARGET_BRANCH}-external-fixes branch
> - Remove tests that pass from expected results file, to reduce the
> size of in-tree files
> - Add docs about how to deal with outages in automated testing labs
> - Specify the exact SHA of the CI scripts to be used
>
> v5:
> - Remove unneeded skips from Meson expectations file
> - Use a more advanced runner that detects flakes automatically
> - Use a more succint format for the expectations
> - Run many more tests (and use sharding to finish in time)
> - Use skip lists to avoid hanging machines
> - Add some build testing
> - Build IGT in each pipeline for faster uprevs
> - List failures in the GitLab UI
>
> Signed-off-by: Tomeu Vizoso
> Reviewed-by: Neil Armstrong
This looks good to me
Reviewed-by: Rob Clark
> ---
> Documentation/gpu/automated_testing.rst | 84 ++
> drivers/gpu/drm/ci/amdgpu-stoney-fails.txt| 13 +++
> drivers/gpu/drm/ci/amdgpu-stoney-flakes.txt | 20 +
> drivers/gpu/drm/ci/amdgpu-stoney-skips.txt| 2 +
> drivers/gpu/drm/ci/gitlab-ci.yml | 13 +++
> drivers/gpu/drm/ci/i915-amly-flakes.txt | 32 +++
> drivers/gpu/drm/ci/i915-amly-skips.txt| 2 +
> drivers/gpu/drm/ci/i915-apl-fails.txt | 29 +++
> drivers/gpu/drm/ci/i915-apl-flakes.txt| 1 +
> drivers/gpu/drm/ci/i915-apl-skips.txt | 2 +
> drivers/gpu/drm/ci/i915-cml-flakes.txt| 36
> drivers/gpu/drm/ci/i915-glk-flakes.txt| 40 +
> drivers/gpu/drm/ci/i915-glk-skips.txt | 2 +
> drivers/gpu/drm/ci/i915-kbl-fails.txt | 8 ++
> drivers/gpu/drm/ci/i915-kbl-flakes.txt| 24 ++
> drivers/gpu/drm/ci/i915-kbl-skips.txt | 2 +
> drivers/gpu/drm/ci/i915-tgl-fails.txt | 19
> drivers/gpu/drm/ci/i915-tgl-flakes.txt| 6 ++
> drivers/gpu/drm/ci/i915-tgl-skips.txt | 8 ++
> drivers/gpu/drm/ci/i915-whl-fails.txt | 30 +++
> drivers/gpu/drm/ci/i915-whl-flakes.txt| 1 +
> drivers/gpu/drm/ci/mediatek-mt8173-fails.txt | 29 +++
> drivers/gpu/drm/ci/mediatek-mt8183-fails.txt | 10 +++
> drivers/gpu/drm/ci/mediatek-mt8183-flakes.txt | 14 +++
> drivers/gpu/drm/ci/meson-g12b-fails.txt | 5 ++
> drivers/gpu/drm/ci/meson-g12b-flakes.txt | 4 +
> drivers/gpu/drm/ci/msm-apq8016-fails.txt | 15
> drivers/gpu/drm/ci/msm-apq8016-flakes.txt | 4 +
> drivers/gpu/drm/ci/msm-apq8096-fails.txt | 2 +
> drivers/gpu/drm/ci/msm-apq8096-flakes.txt | 4 +
> drivers/gpu/drm/ci/msm-apq8096-skips.txt | 2 +
> drivers/gpu/drm/ci/msm-sc7180-fails.txt | 22 +
> drivers/gpu/drm/ci/msm-sc7180-flakes.txt | 14 +++
> drivers/gpu/drm/ci/msm-sc7180-skips.txt | 18
> drivers/gpu/drm/ci/msm-sdm845-fails.txt | 44 ++
> drivers/gpu/drm/ci/msm-sdm845-flakes.txt | 33 +++
> drivers/gpu/drm/ci/msm-sdm845-skips.txt | 2 +
> drivers/gpu/drm/ci/rockchip-rk3288-fails.txt | 75
> drivers/gpu/drm/ci/rockchip-rk3288-flakes.txt | 5 ++
> drivers/gpu/drm/ci/rockchip-rk3288-skips.txt | 46 ++
> drivers/gpu/drm/ci/rockchip-rk3399-fails.txt | 86 +++
> drivers/gpu/drm/ci/rockchip-rk3399-flakes.txt | 25 ++
> drivers/gpu/drm/ci/rockchip-rk3399-skips.txt | 5 ++
> drivers/gpu/drm/ci/virtio_gpu-none-fails.txt | 38
> drivers/gpu/drm/ci/virtio_gpu-none-flakes.txt | 0
> drivers/gpu/drm/ci/virtio_gpu-none-skips.txt | 6 ++
> 46 files changed, 882 insertions(+)
> create mode 100644 Documentation/gpu/automated_testing.rst
> create mode 100644 drivers/gpu/drm/ci/amdgpu-stoney-fails.txt
> create mode 100644 drivers/gpu/drm/ci/amdgpu-stoney-flakes.txt
> create mode 100644 drivers/gpu/drm/ci/amdgpu-stoney-skips.txt
> create mode 100644 drivers/gpu/drm/ci/gitlab-ci.yml
> create mode 100644 drivers/gpu/drm/ci/i915-amly-flakes.txt
> create mode 100644 drivers/gpu/drm/ci/i915-amly-skips.txt
> create mode 100644
