On Wed, 22 Sep 2021 02:05:35 GMT, David Holmes <dhol...@openjdk.org> wrote:
>> I'll take another stab at it tomorrow. >> >> The whole tiered testing thing encapsulates that some tests are more >> important to run (at least, first or regularly) than others. So we need to >> somehow capture that. Discussing what tests are actually running is not >> relevant, because `TEST.groups` is the source of truth there. >> >> For a contributor, these descriptions should tell why, when and how often >> would you use a particular tier. In other words, good docs capture some >> thinking about how contributors are supposed to use the existing tests. It >> seems to be a broad consensus that we run `tier1` before non-trivial >> integrations. It seems to be a rule of thumb that we care a lot about >> `tier1` stability and speed, and we push out less stable, slower or >> specially-configured tests to higher tiers. As I inspect the test groups, I >> think `tier2` is for regular but non-`tier1`-grade tests, `tier3` is mostly >> for stress tests, and `tier4` is (somewhat by construction) is "everything >> else". >> >> From here, this what I'd like to tell contributors: run `tier1` early and >> often, the bugs there are likely to be real product bugs and should be >> looked at as soon as possible, run `tier1` + `tier2` to execute all regular >> tests but suspect test bugs more, run `tier3` to add even more stress >> testing and suspect environmental problems when they fail, run `tier4` to >> run everything that previous tiers missed and hope for the best. Maybe I >> should just put that in the docs :) > > Ideal world: run every test on every platform before every integration. > Real world: run as many tests as is practical having consideration for time > of testing and the cost of testing resources, and the scope of the change. > > It is assumed/expected you will have run all tests pertaining to the area of > code change, that can reasonably be expected to be run. > > The tiered testing is intended to give broader coverage to ensure no > unintended consequences of a change. At a minimum run all tier1 tests on all > non-trivial integrations. If you have time and resources, and think a change > may have broader impact than what has been explicitly tested, then run > further tiers. All right, thanks for the inputs. Let me try again, see new commit. ------------- PR: https://git.openjdk.java.net/jdk/pull/5615
