Hi everyone,
As some of you may already know, we have made great strides in making
our CI system usable, especially in the last 6 months when everything
started clicking together.
The CI team is no longer overwhelmed with fires and bug reports, so we
started working on increasing the coverage from just fast-feedback, to a
bigger set of IGT tests.
As some of you may know, running IGT has been a challenge that few
manage to overcome. Not only is the execution time counted in machine
months, but it can also lead to disk corruption, which does not
encourage developers to run it either. One test takes 21 days, on its
own, and it is a subset of another test which we never ran for obvious
reasons.
I would thus like to get the CI team and developers to work together to
decrease sharply the execution time of IGT, and get these tests run
multiple times per day!
There are three usages that the CI team envision (up for debate):
- Basic acceptance testing: Meant for developers and CI to check
quickly if a patch series is not completely breaking the world (< 10
minutes, timeout per test of 30s)
- Full run: Meant to be ran overnight by developers and users (< 6 hours)
- Stress tests: They can be in the test suite as a way to catch rare
issues, but they cannot be part of the default run mode. They likely
should be run on a case-by-case basis, on demand of a developer. Each
test could be allowed to take up to 1h.
There are multiple ways of getting to this situation (up for debate):
1) All the tests exposed by default are fast and meant to be run:
- Fast-feedback is provided by a testlist, for BAT
- Stress tests ran using a special command, kept for on-demand testing
2) Tests are all tagged with information about their exec time:
- igt@basic@.*: Meant for BAT
- igt@complete@.*: Meant for FULL
- igt@stress@.*: The stress tests
3) Testlists all the way:
- fast-feedback: for BAT
- all: the tests that people are expected to run (CI will run them)
- Stress tests will not be part of any testlist.
Whatever decision is being accepted, the CI team is mandating global
timeouts for both BAT and FULL testing, in order to guarantee
throughput. This will require the team as a whole to agree on time
quotas per sub-systems, and enforce them.
Can we try to get some healthy debate and reach a consensus on this? Our
CI efforts are being limited by this issue right now, and we will be
doing whatever we can until the test suite becomes saner and runnable,
but this may be unfair to some developers.
Looking forward to some constructive feedback and intelligent discussions!
Martin
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
