Re: RFR: 8274083: Update testing docs to mention tiered testing [v5]
On 9/23/2021 11:00 PM, Alan Bateman wrote:
On Thu, 23 Sep 2021 12:53:23 GMT, Aleksey Shipilev wrote:
Now that OpenJDK has more or less complete `tier{1,2,3,4}` definitions, let's
mention them in `testing.md`.
Current patch is my braindump, I am open for suggestions :)
Aleksey Shipilev has updated the pull request incrementally with one additional
commit since the last revision:
More fixes
Marked as reviewed by alanb (Reviewer).
doc/testing.html line 80:
78:
79: tier1: This is the lowest test tier. Multiple developers run these tests every day. Because of the
widespread use, the tests in tier1 are carefully selected and optimized to run fast, and to run in the most stable manner. The
test failures in tier1 are usually followed up on quickly, either with fixes, or adding relevant tests to problem list. GitHub
Actions workflows, if enabled, run tier1 tests.
80: tier2: This test group covers even more ground. These contain, among other
things, tests that either run for too long to be at tier1, or may require special configuration, or
tests that are less stable, or cover the broader range of less critical JVM and JDK features/components (for example,
jaxp).
Thanks for the updates, I think it reads much better now.
A small suggestion is to change "less critical" to "non-core", and "jaxp" to
"XML" as I don't expect too many people know what JAXP is.
Catching up on email, for some belated context, the initial tired
testing email thread did included some discussion about what the tiered
were intended to mean including:
http://mail.openjdk.java.net/pipermail/jdk9-dev/2015-March/001991.html
Thanks,
-Joe
-
PR: https://git.openjdk.java.net/jdk/pull/5615
Re: RFR: 8274083: Update testing docs to mention tiered testing [v6]
On Fri, 24 Sep 2021 06:12:20 GMT, Aleksey Shipilev wrote:
>> Now that OpenJDK has more or less complete `tier{1,2,3,4}` definitions,
>> let's mention them in `testing.md`.
>>
>> Current patch is my braindump, I am open for suggestions :)
>
> Aleksey Shipilev has updated the pull request incrementally with one
> additional commit since the last revision:
>
> Another minor touchup
Looks good to me.
-
PR: https://git.openjdk.java.net/jdk/pull/5615
Re: RFR: 8274083: Update testing docs to mention tiered testing [v6]
On Fri, 24 Sep 2021 06:12:20 GMT, Aleksey Shipilev wrote:
>> Now that OpenJDK has more or less complete `tier{1,2,3,4}` definitions,
>> let's mention them in `testing.md`.
>>
>> Current patch is my braindump, I am open for suggestions :)
>
> Aleksey Shipilev has updated the pull request incrementally with one
> additional commit since the last revision:
>
> Another minor touchup
Marked as reviewed by alanb (Reviewer).
-
PR: https://git.openjdk.java.net/jdk/pull/5615
Re: RFR: 8274083: Update testing docs to mention tiered testing [v6]
> Now that OpenJDK has more or less complete `tier{1,2,3,4}` definitions, let's
> mention them in `testing.md`.
>
> Current patch is my braindump, I am open for suggestions :)
Aleksey Shipilev has updated the pull request incrementally with one additional
commit since the last revision:
Another minor touchup
-
Changes:
- all: https://git.openjdk.java.net/jdk/pull/5615/files
- new: https://git.openjdk.java.net/jdk/pull/5615/files/efd28804..94496373
Webrevs:
- full: https://webrevs.openjdk.java.net/?repo=jdk=5615=05
- incr: https://webrevs.openjdk.java.net/?repo=jdk=5615=04-05
Stats: 3 lines in 2 files changed: 0 ins; 0 del; 3 mod
Patch: https://git.openjdk.java.net/jdk/pull/5615.diff
Fetch: git fetch https://git.openjdk.java.net/jdk pull/5615/head:pull/5615
PR: https://git.openjdk.java.net/jdk/pull/5615
Re: RFR: 8274083: Update testing docs to mention tiered testing [v5]
On Fri, 24 Sep 2021 05:58:09 GMT, Alan Bateman wrote: >> Aleksey Shipilev has updated the pull request incrementally with one >> additional commit since the last revision: >> >> More fixes > > doc/testing.html line 80: > >> 78: >> 79: tier1: This is the lowest test tier. Multiple >> developers run these tests every day. Because of the widespread use, the >> tests in tier1 are carefully selected and optimized to run >> fast, and to run in the most stable manner. The test failures in >> tier1 are usually followed up on quickly, either with fixes, or >> adding relevant tests to problem list. GitHub Actions workflows, if enabled, >> run tier1 tests. >> 80: tier2: This test group covers even more ground. >> These contain, among other things, tests that either run for too long to be >> at tier1, or may require special configuration, or tests that >> are less stable, or cover the broader range of less critical JVM and JDK >> features/components (for example, jaxp). > > Thanks for the updates, I think it reads much better now. > A small suggestion is to change "less critical" to "non-core", and "jaxp" to > "XML" as I don't expect too many people know what JAXP is. All right, changed. I'll be integrating this version soon. - PR: https://git.openjdk.java.net/jdk/pull/5615
Re: RFR: 8274083: Update testing docs to mention tiered testing [v5]
On Thu, 23 Sep 2021 12:53:23 GMT, Aleksey Shipilev wrote:
>> Now that OpenJDK has more or less complete `tier{1,2,3,4}` definitions,
>> let's mention them in `testing.md`.
>>
>> Current patch is my braindump, I am open for suggestions :)
>
> Aleksey Shipilev has updated the pull request incrementally with one
> additional commit since the last revision:
>
> More fixes
Marked as reviewed by alanb (Reviewer).
doc/testing.html line 80:
> 78:
> 79: tier1: This is the lowest test tier. Multiple
> developers run these tests every day. Because of the widespread use, the
> tests in tier1 are carefully selected and optimized to run fast,
> and to run in the most stable manner. The test failures in tier1
> are usually followed up on quickly, either with fixes, or adding relevant
> tests to problem list. GitHub Actions workflows, if enabled, run
> tier1 tests.
> 80: tier2: This test group covers even more ground. These
> contain, among other things, tests that either run for too long to be at
> tier1, or may require special configuration, or tests that are
> less stable, or cover the broader range of less critical JVM and JDK
> features/components (for example, jaxp).
Thanks for the updates, I think it reads much better now.
A small suggestion is to change "less critical" to "non-core", and "jaxp" to
"XML" as I don't expect too many people know what JAXP is.
-
PR: https://git.openjdk.java.net/jdk/pull/5615
Re: RFR: 8274083: Update testing docs to mention tiered testing [v5]
On Thu, 23 Sep 2021 12:53:23 GMT, Aleksey Shipilev wrote:
>> Now that OpenJDK has more or less complete `tier{1,2,3,4}` definitions,
>> let's mention them in `testing.md`.
>>
>> Current patch is my braindump, I am open for suggestions :)
>
> Aleksey Shipilev has updated the pull request incrementally with one
> additional commit since the last revision:
>
> More fixes
Marked as reviewed by jiefu (Reviewer).
-
PR: https://git.openjdk.java.net/jdk/pull/5615
Re: RFR: 8274083: Update testing docs to mention tiered testing [v5]
On Thu, 23 Sep 2021 12:53:23 GMT, Aleksey Shipilev wrote:
>> Now that OpenJDK has more or less complete `tier{1,2,3,4}` definitions,
>> let's mention them in `testing.md`.
>>
>> Current patch is my braindump, I am open for suggestions :)
>
> Aleksey Shipilev has updated the pull request incrementally with one
> additional commit since the last revision:
>
> More fixes
Marked as reviewed by dholmes (Reviewer).
-
PR: https://git.openjdk.java.net/jdk/pull/5615
Re: RFR: 8274083: Update testing docs to mention tiered testing [v5]
On Thu, 23 Sep 2021 12:53:23 GMT, Aleksey Shipilev wrote:
>> Now that OpenJDK has more or less complete `tier{1,2,3,4}` definitions,
>> let's mention them in `testing.md`.
>>
>> Current patch is my braindump, I am open for suggestions :)
>
> Aleksey Shipilev has updated the pull request incrementally with one
> additional commit since the last revision:
>
> More fixes
Change looks good to me, thank you.
-
Marked as reviewed by mseledtsov (Committer).
PR: https://git.openjdk.java.net/jdk/pull/5615
Re: RFR: 8274083: Update testing docs to mention tiered testing [v3]
On 23/09/2021 8:33 pm, Thomas Stuefe wrote:
On Thu, 23 Sep 2021 09:19:42 GMT, Aleksey Shipilev wrote:
Now that OpenJDK has more or less complete `tier{1,2,3,4}` definitions, let's
mention them in `testing.md`.
Current patch is my braindump, I am open for suggestions :)
Aleksey Shipilev has updated the pull request incrementally with one additional
commit since the last revision:
More fixes
I like this. Thanks for doing this (and for the actual definition of the
tiers). This hopefully makes communicating about test coverage much easier.
As I understood, tier1-3 are the same as Oracle tier1-3?
Not exactly no. Our tier definitions are tuples of (tests, platforms,
flags). The sets of tests will be close but not exactly the same.
David
-
Marked as reviewed by stuefe (Reviewer).
PR: https://git.openjdk.java.net/jdk/pull/5615
Re: RFR: 8274083: Update testing docs to mention tiered testing [v4]
On Thu, 23 Sep 2021 11:58:02 GMT, Alan Bateman wrote: >> Aleksey Shipilev has updated the pull request incrementally with one >> additional commit since the last revision: >> >> Drop GH link > > doc/testing.html line 79: > >> 77: >> 78: tier1: This is the lowest test tier. Multiple >> developers run these tests every day. Normally, at least this tier should be >> clean before integration. Because of the widespread use, the tests in >> tier1 are carefully selected and optimized to run fast, and to >> run in the most stable manner. The test failures in tier1 are >> usually followed up on quickly, either with fixes, or adding relevant tests >> to problem list. GitHub Actions workflows, if enabled, run >> tier1 tests. >> 79: tier2: This test group covers even more ground. >> These contain, among other things, tests that either run for too long to be >> at tier1, tests for less stable and/or experimental features, >> tests for less essential JDK components (for example, jaxp). > > I think this a bit unfair on tier2. Tests in tier2 need be run by anyone > making changes in the security area, I/O, networking and many other core > APIs. In general I think anyone changing code in the libraries area needs to > look at TEST.groups to get some idea which tier or finer test group to run. > > I'm not sure about words like "less stable" and "experimental features". Do > we have tests for hotspot experimental features in hotspot_tier2? See new commit. > Tests in tier2 need be run by anyone making changes in the security area, > I/O, networking and many other core APIs. In general I think anyone changing > code in the libraries area needs to look at TEST.groups to get some idea > which tier or finer test group to run. True. That is why the text says that component tests come first, and then you run first N tiers to capture anything else. If any component test is not in the component test group, and only in `tier2`, I'd say that is a test grouping bug, and the test should be in component test group to begin with. In other words, `tier2` testing does not replace the component group testing, and thus the requirement to run `tier2` if particular areas are changed would feel awkward to me. > Do we have tests for hotspot experimental features in hotspot_tier2? We do move deprecating things out of `tier1` (see e.g. recent JDK-8272838). - PR: https://git.openjdk.java.net/jdk/pull/5615
Re: RFR: 8274083: Update testing docs to mention tiered testing [v5]
> Now that OpenJDK has more or less complete `tier{1,2,3,4}` definitions, let's
> mention them in `testing.md`.
>
> Current patch is my braindump, I am open for suggestions :)
Aleksey Shipilev has updated the pull request incrementally with one additional
commit since the last revision:
More fixes
-
Changes:
- all: https://git.openjdk.java.net/jdk/pull/5615/files
- new: https://git.openjdk.java.net/jdk/pull/5615/files/32e9b5b4..efd28804
Webrevs:
- full: https://webrevs.openjdk.java.net/?repo=jdk=5615=04
- incr: https://webrevs.openjdk.java.net/?repo=jdk=5615=03-04
Stats: 21 lines in 2 files changed: 4 ins; 2 del; 15 mod
Patch: https://git.openjdk.java.net/jdk/pull/5615.diff
Fetch: git fetch https://git.openjdk.java.net/jdk pull/5615/head:pull/5615
PR: https://git.openjdk.java.net/jdk/pull/5615
Re: RFR: 8274083: Update testing docs to mention tiered testing [v4]
On Thu, 23 Sep 2021 11:54:06 GMT, Alan Bateman wrote: >> Aleksey Shipilev has updated the pull request incrementally with one >> additional commit since the last revision: >> >> Drop GH link > > doc/testing.html line 72: > >> 70: The test specifications given in TEST is parsed into >> fully qualified test descriptors, which clearly and unambigously show which >> tests will be run. As an example, :tier1 will expand to >> jtreg:$(TOPDIR)/test/hotspot/jtreg:tier1 >> jtreg:$(TOPDIR)/test/jdk:tier1 jtreg:$(TOPDIR)/test/langtools:tier1 >> jtreg:$(TOPDIR)/test/nashorn:tier1 jtreg:$(TOPDIR)/test/jaxp:tier1. >> You can always submit a list of fully qualified test descriptors in the >> TEST variable if you want to shortcut the parser. >> 71: Common Test Groups >> 72: In the ideal world, contributors would be advised to run all the >> tests for every change. But in real world, one could only be expected to run >> as many tests as practical, while being mindful of the scope for the change, >> the testing resources available, etc. > > A suggestion here is to start with something like "Ideally, all tests are run > for every change but this may not be practical. Contributors are expected to > at least run tier1 (see below) and the tests for the areas that are changed." Added a version of that, thanks. - PR: https://git.openjdk.java.net/jdk/pull/5615
Re: RFR: 8274083: Update testing docs to mention tiered testing [v4]
On Thu, 23 Sep 2021 11:30:20 GMT, Aleksey Shipilev wrote:
>> Now that OpenJDK has more or less complete `tier{1,2,3,4}` definitions,
>> let's mention them in `testing.md`.
>>
>> Current patch is my braindump, I am open for suggestions :)
>
> Aleksey Shipilev has updated the pull request incrementally with one
> additional commit since the last revision:
>
> Drop GH link
doc/testing.html line 72:
> 70: The test specifications given in TEST is parsed into
> fully qualified test descriptors, which clearly and unambigously show which
> tests will be run. As an example, :tier1 will expand to
> jtreg:$(TOPDIR)/test/hotspot/jtreg:tier1 jtreg:$(TOPDIR)/test/jdk:tier1
> jtreg:$(TOPDIR)/test/langtools:tier1 jtreg:$(TOPDIR)/test/nashorn:tier1
> jtreg:$(TOPDIR)/test/jaxp:tier1. You can always submit a list of fully
> qualified test descriptors in the TEST variable if you want to
> shortcut the parser.
> 71: Common Test Groups
> 72: In the ideal world, contributors would be advised to run all the tests
> for every change. But in real world, one could only be expected to run as
> many tests as practical, while being mindful of the scope for the change, the
> testing resources available, etc.
A suggestion here is to start with something like "Ideally, all tests are run
for every change but this may not be practical. Contributors are expected to at
least run tier1 (see below) and the tests for the areas that are changed."
doc/testing.html line 79:
> 77:
> 78: tier1: This is the lowest test tier. Multiple
> developers run these tests every day. Normally, at least this tier should be
> clean before integration. Because of the widespread use, the tests in
> tier1 are carefully selected and optimized to run fast, and to
> run in the most stable manner. The test failures in tier1 are
> usually followed up on quickly, either with fixes, or adding relevant tests
> to problem list. GitHub Actions workflows, if enabled, run tier1
> tests.
> 79: tier2: This test group covers even more ground. These
> contain, among other things, tests that either run for too long to be at
> tier1, tests for less stable and/or experimental features, tests
> for less essential JDK components (for example, jaxp).
I think this a bit unfair on tier2. Tests in tier2 need be run by anyone making
changes in the security area, I/O, networking and many other core APIs. In
general I think anyone changing code in the libraries area needs to look at
TEST.groups to get some idea which tier or finer test group to run.
I'm not sure about words like "less stable" and "experimental features". Do we
have tests for hotspot experimental features in hotspot_tier2?
-
PR: https://git.openjdk.java.net/jdk/pull/5615
Re: RFR: 8274083: Update testing docs to mention tiered testing [v4]
On Thu, 23 Sep 2021 11:30:20 GMT, Aleksey Shipilev wrote:
>> Now that OpenJDK has more or less complete `tier{1,2,3,4}` definitions,
>> let's mention them in `testing.md`.
>>
>> Current patch is my braindump, I am open for suggestions :)
>
> Aleksey Shipilev has updated the pull request incrementally with one
> additional commit since the last revision:
>
> Drop GH link
Marked as reviewed by ihse (Reviewer).
-
PR: https://git.openjdk.java.net/jdk/pull/5615
Re: RFR: 8274083: Update testing docs to mention tiered testing [v4]
> Now that OpenJDK has more or less complete `tier{1,2,3,4}` definitions, let's
> mention them in `testing.md`.
>
> Current patch is my braindump, I am open for suggestions :)
Aleksey Shipilev has updated the pull request incrementally with one additional
commit since the last revision:
Drop GH link
-
Changes:
- all: https://git.openjdk.java.net/jdk/pull/5615/files
- new: https://git.openjdk.java.net/jdk/pull/5615/files/cf2bda57..32e9b5b4
Webrevs:
- full: https://webrevs.openjdk.java.net/?repo=jdk=5615=03
- incr: https://webrevs.openjdk.java.net/?repo=jdk=5615=02-03
Stats: 3 lines in 2 files changed: 0 ins; 0 del; 3 mod
Patch: https://git.openjdk.java.net/jdk/pull/5615.diff
Fetch: git fetch https://git.openjdk.java.net/jdk pull/5615/head:pull/5615
PR: https://git.openjdk.java.net/jdk/pull/5615
Re: RFR: 8274083: Update testing docs to mention tiered testing [v3]
On Thu, 23 Sep 2021 10:56:02 GMT, Magnus Ihse Bursie wrote: >> Aleksey Shipilev has updated the pull request incrementally with one >> additional commit since the last revision: >> >> More fixes > > doc/testing.md line 95: > >> 93: the tests in `tier1` are carefully selected and optimized to run fast, >> and to run in the >> 94: most stable manner. The test failures in `tier1` are usually followed up >> on quickly, either >> 95: with fixes, or adding relevant tests to problem list. [GitHub Actions >> workflows](../.github/workflows/), > > I don't think the link to the `.github` directory makes sense here. If there > should be a link at all, it should point to information on how GHA works and > how to enable these. And frankly I'm not sure if any such comprehensive > documentation exists. :-( Perhaps in the Skara wiki? Otherwise the link is > just distracting. Sure, I actually put the link there very optimistically :) Dropped it in the new commit. - PR: https://git.openjdk.java.net/jdk/pull/5615
Re: RFR: 8274083: Update testing docs to mention tiered testing [v3]
On Thu, 23 Sep 2021 09:19:42 GMT, Aleksey Shipilev wrote:
>> Now that OpenJDK has more or less complete `tier{1,2,3,4}` definitions,
>> let's mention them in `testing.md`.
>>
>> Current patch is my braindump, I am open for suggestions :)
>
> Aleksey Shipilev has updated the pull request incrementally with one
> additional commit since the last revision:
>
> More fixes
I think this looks overall good. Just do something about that GHA link and I'm
perfectly happy! :)
doc/testing.md line 95:
> 93: the tests in `tier1` are carefully selected and optimized to run fast,
> and to run in the
> 94: most stable manner. The test failures in `tier1` are usually followed up
> on quickly, either
> 95: with fixes, or adding relevant tests to problem list. [GitHub Actions
> workflows](../.github/workflows/),
I don't think the link to the `.github` directory makes sense here. If there
should be a link at all, it should point to information on how GHA works and
how to enable these. And frankly I'm not sure if any such comprehensive
documentation exists. :-( Perhaps in the Skara wiki? Otherwise the link is just
distracting.
-
Changes requested by ihse (Reviewer).
PR: https://git.openjdk.java.net/jdk/pull/5615
Re: RFR: 8274083: Update testing docs to mention tiered testing [v3]
On Thu, 23 Sep 2021 09:19:42 GMT, Aleksey Shipilev wrote:
>> Now that OpenJDK has more or less complete `tier{1,2,3,4}` definitions,
>> let's mention them in `testing.md`.
>>
>> Current patch is my braindump, I am open for suggestions :)
>
> Aleksey Shipilev has updated the pull request incrementally with one
> additional commit since the last revision:
>
> More fixes
I like this. Thanks for doing this (and for the actual definition of the
tiers). This hopefully makes communicating about test coverage much easier.
As I understood, tier1-3 are the same as Oracle tier1-3?
-
Marked as reviewed by stuefe (Reviewer).
PR: https://git.openjdk.java.net/jdk/pull/5615
Re: RFR: 8274083: Update testing docs to mention tiered testing [v3]
On Thu, 23 Sep 2021 09:37:49 GMT, Aleksey Shipilev wrote: > No, I think "every other X" puts "X" in singular form: > https://www.merriam-webster.com/dictionary/every%20other. Okay, I agree. Thanks. - PR: https://git.openjdk.java.net/jdk/pull/5615
Re: RFR: 8274083: Update testing docs to mention tiered testing [v3]
On Thu, 23 Sep 2021 09:19:42 GMT, Aleksey Shipilev wrote:
>> Now that OpenJDK has more or less complete `tier{1,2,3,4}` definitions,
>> let's mention them in `testing.md`.
>>
>> Current patch is my braindump, I am open for suggestions :)
>
> Aleksey Shipilev has updated the pull request incrementally with one
> additional commit since the last revision:
>
> More fixes
LGTM.
Thanks for this kind of doc update.
-
Marked as reviewed by jiefu (Reviewer).
PR: https://git.openjdk.java.net/jdk/pull/5615
Re: RFR: 8274083: Update testing docs to mention tiered testing [v3]
On Thu, 23 Sep 2021 09:28:29 GMT, Jie Fu wrote: >> Aleksey Shipilev has updated the pull request incrementally with one >> additional commit since the last revision: >> >> More fixes > > doc/testing.md line 107: > >> 105: (`JTREG_KEYWORDS=\!headful`), or both. >> 106: >> 107: - `tier4`: This test group includes every other test not covered by >> previous tiers. It includes, > > `other test` => `other tests` No, I think "every other X" puts "X" in singular form: https://www.merriam-webster.com/dictionary/every%20other. - PR: https://git.openjdk.java.net/jdk/pull/5615
Re: RFR: 8274083: Update testing docs to mention tiered testing [v3]
On Thu, 23 Sep 2021 09:19:42 GMT, Aleksey Shipilev wrote:
>> Now that OpenJDK has more or less complete `tier{1,2,3,4}` definitions,
>> let's mention them in `testing.md`.
>>
>> Current patch is my braindump, I am open for suggestions :)
>
> Aleksey Shipilev has updated the pull request incrementally with one
> additional commit since the last revision:
>
> More fixes
doc/testing.md line 107:
> 105: (`JTREG_KEYWORDS=\!headful`), or both.
> 106:
> 107: - `tier4`: This test group includes every other test not covered by
> previous tiers. It includes,
`other test` => `other tests`
-
PR: https://git.openjdk.java.net/jdk/pull/5615
Re: RFR: 8274083: Update testing docs to mention tiered testing [v3]
> Now that OpenJDK has more or less complete `tier{1,2,3,4}` definitions, let's
> mention them in `testing.md`.
>
> Current patch is my braindump, I am open for suggestions :)
Aleksey Shipilev has updated the pull request incrementally with one additional
commit since the last revision:
More fixes
-
Changes:
- all: https://git.openjdk.java.net/jdk/pull/5615/files
- new: https://git.openjdk.java.net/jdk/pull/5615/files/86cbd115..cf2bda57
Webrevs:
- full: https://webrevs.openjdk.java.net/?repo=jdk=5615=02
- incr: https://webrevs.openjdk.java.net/?repo=jdk=5615=01-02
Stats: 7 lines in 2 files changed: 1 ins; 0 del; 6 mod
Patch: https://git.openjdk.java.net/jdk/pull/5615.diff
Fetch: git fetch https://git.openjdk.java.net/jdk pull/5615/head:pull/5615
PR: https://git.openjdk.java.net/jdk/pull/5615
Re: RFR: 8274083: Update testing docs to mention tiered testing [v2]
On Thu, 23 Sep 2021 05:44:57 GMT, David Holmes wrote: >> Aleksey Shipilev has updated the pull request incrementally with two >> additional commits since the last revision: >> >> - Revert accidental edit >> - More discussion > > doc/testing.html line 73: > >> 71: Common Test Groups >> 72: In the ideal world, contributors would be advised to run all the >> tests for every change. But in real world, one could only be expected to run >> as many tests as practical, while being mindful of the scope for the change, >> the testing resources available, etc. >> 73: The source tree currently defines a few common test groups in the >> relevant TEST.groups files. The test groups are either covering >> a specific component, for example hotspot_gc. It is a good idea >> to look into TEST.groups files to get a sense what tests are >> relevant to a particular JDK component. > > You state "The test groups are either ..." but there is no "or ..." in that > sentence. Thanks, see new commit. - PR: https://git.openjdk.java.net/jdk/pull/5615
Re: RFR: 8274083: Update testing docs to mention tiered testing [v2]
On Wed, 22 Sep 2021 12:28:18 GMT, Aleksey Shipilev wrote:
>> Now that OpenJDK has more or less complete `tier{1,2,3,4}` definitions,
>> let's mention them in `testing.md`.
>>
>> Current patch is my braindump, I am open for suggestions :)
>
> Aleksey Shipilev has updated the pull request incrementally with two
> additional commits since the last revision:
>
> - Revert accidental edit
> - More discussion
Hi Aleksey,
I don't have any significant misgivings about this version - thanks.
One correction needed below.
Thanks,
David
doc/testing.html line 73:
> 71: Common Test Groups
> 72: In the ideal world, contributors would be advised to run all the tests
> for every change. But in real world, one could only be expected to run as
> many tests as practical, while being mindful of the scope for the change, the
> testing resources available, etc.
> 73: The source tree currently defines a few common test groups in the
> relevant TEST.groups files. The test groups are either covering
> a specific component, for example hotspot_gc. It is a good idea
> to look into TEST.groups files to get a sense what tests are
> relevant to a particular JDK component.
You state "The test groups are either ..." but there is no "or ..." in that
sentence.
-
Marked as reviewed by dholmes (Reviewer).
PR: https://git.openjdk.java.net/jdk/pull/5615
Re: RFR: 8274083: Update testing docs to mention tiered testing [v2]
> Now that OpenJDK has more or less complete `tier{1,2,3,4}` definitions, let's
> mention them in `testing.md`.
>
> Current patch is my braindump, I am open for suggestions :)
Aleksey Shipilev has updated the pull request incrementally with two additional
commits since the last revision:
- Revert accidental edit
- More discussion
-
Changes:
- all: https://git.openjdk.java.net/jdk/pull/5615/files
- new: https://git.openjdk.java.net/jdk/pull/5615/files/63c51dc7..86cbd115
Webrevs:
- full: https://webrevs.openjdk.java.net/?repo=jdk=5615=01
- incr: https://webrevs.openjdk.java.net/?repo=jdk=5615=00-01
Stats: 34 lines in 2 files changed: 16 ins; 0 del; 18 mod
Patch: https://git.openjdk.java.net/jdk/pull/5615.diff
Fetch: git fetch https://git.openjdk.java.net/jdk pull/5615/head:pull/5615
PR: https://git.openjdk.java.net/jdk/pull/5615
Re: RFR: 8274083: Update testing docs to mention tiered testing [v2]
On Wed, 22 Sep 2021 02:05:35 GMT, David Holmes wrote: >> I'll take another stab at it tomorrow. >> >> The whole tiered testing thing encapsulates that some tests are more >> important to run (at least, first or regularly) than others. So we need to >> somehow capture that. Discussing what tests are actually running is not >> relevant, because `TEST.groups` is the source of truth there. >> >> For a contributor, these descriptions should tell why, when and how often >> would you use a particular tier. In other words, good docs capture some >> thinking about how contributors are supposed to use the existing tests. It >> seems to be a broad consensus that we run `tier1` before non-trivial >> integrations. It seems to be a rule of thumb that we care a lot about >> `tier1` stability and speed, and we push out less stable, slower or >> specially-configured tests to higher tiers. As I inspect the test groups, I >> think `tier2` is for regular but non-`tier1`-grade tests, `tier3` is mostly >> for stress tests, and `tier4` is (somewhat by construction) is "everything >> else". >> >> From here, this what I'd like to tell contributors: run `tier1` early and >> often, the bugs there are likely to be real product bugs and should be >> looked at as soon as possible, run `tier1` + `tier2` to execute all regular >> tests but suspect test bugs more, run `tier3` to add even more stress >> testing and suspect environmental problems when they fail, run `tier4` to >> run everything that previous tiers missed and hope for the best. Maybe I >> should just put that in the docs :) > > Ideal world: run every test on every platform before every integration. > Real world: run as many tests as is practical having consideration for time > of testing and the cost of testing resources, and the scope of the change. > > It is assumed/expected you will have run all tests pertaining to the area of > code change, that can reasonably be expected to be run. > > The tiered testing is intended to give broader coverage to ensure no > unintended consequences of a change. At a minimum run all tier1 tests on all > non-trivial integrations. If you have time and resources, and think a change > may have broader impact than what has been explicitly tested, then run > further tiers. All right, thanks for the inputs. Let me try again, see new commit. - PR: https://git.openjdk.java.net/jdk/pull/5615
Re: RFR: 8274083: Update testing docs to mention tiered testing
On Tue, 21 Sep 2021 16:06:28 GMT, Aleksey Shipilev wrote: >> doc/testing.html line 77: >> >>> 75: tier1: This test group is the first line of defense >>> against bugs. Multiple developers run these tests every day. Normally, at >>> least this tier is ran before integration. Because of the widespread, the >>> tests in tier1 are carefully selected and optimized to run >>> fast, and to run in the most stable manner. The test failures in >>> tier1 are usually followed up on quickly, either with fixes, >>> or adding relevant tests to problem list. >> href="../.github/workflows/">GitHub Actions workflows, if enabled, run >>> tier1 tests. >>> 76: tier2: This test group covers even more ground. >>> These contain, among other things, tests that either run for too long to be >>> at tier1, test unstable and/or experimental features, test >>> less essential JDK components. >>> 77: tier3: This test group covers more stressful tests, >>> the tests for corner cases not covered by previous tiers, plus the tests >>> that require GUIs. As such, this suite should either be run with low >>> concurrency (TEST_JOBS=1), or without headful tests >>> (JTREG_KEYWORDS=\!headful), or both. >> >> It's not clear to me that these descriptions are useful. Instead of "First >> line of defense against bugs" then it might be better to say something about >> the hotspot and core library tests that are typically run in tier1. Folks >> contributing to some areas of the core, I/O, and networking libraries will >> need to learn about tier2 as that is where many of the tests for these APIs >> and implementations are. So I wouldn't use words like "unstable", >> "experimental", "less essential" but maybe say that it includes tests that >> may require some configuration, like changing firewall rules to allow >> multicast or other traffic. > > I'll take another stab at it tomorrow. > > The whole tiered testing thing encapsulates that some tests are more > important to run (at least, first or regularly) than others. So we need to > somehow capture that. Discussing what tests are actually running is not > relevant, because `TEST.groups` is the source of truth there. > > For a contributor, these descriptions should tell why, when and how often > would you use a particular tier. In other words, good docs capture some > thinking about how contributors are supposed to use the existing tests. It > seems to be a broad consensus that we run `tier1` before non-trivial > integrations. It seems to be a rule of thumb that we care a lot about `tier1` > stability and speed, and we push out less stable, slower or > specially-configured tests to higher tiers. As I inspect the test groups, I > think `tier2` is for regular but non-`tier1`-grade tests, `tier3` is mostly > for stress tests, and `tier4` is (somewhat by construction) is "everything > else". > > From here, this what I'd like to tell contributors: run `tier1` early and > often, the bugs there are likely to be real product bugs and should be looked > at as soon as possible, run `tier1` + `tier2` to execute all regular tests > but suspect test bugs more, run `tier3` to add even more stress testing and > suspect environmental problems when they fail, run `tier4` to run everything > that previous tiers missed and hope for the best. Maybe I should just put > that in the docs :) Ideal world: run every test on every platform before every integration. Real world: run as many tests as is practical having consideration for time of testing and the cost of testing resources, and the scope of the change. It is assumed/expected you will have run all tests pertaining to the area of code change, that can reasonably be expected to be run. The tiered testing is intended to give broader coverage to ensure no unintended consequences of a change. At a minimum run all tier1 tests on all non-trivial integrations. If you have time and resources, and think a change may have broader impact than what has been explicitly tested, then run further tiers. - PR: https://git.openjdk.java.net/jdk/pull/5615
Re: RFR: 8274083: Update testing docs to mention tiered testing
On Tue, 21 Sep 2021 15:20:44 GMT, Alan Bateman wrote:
>> Now that OpenJDK has more or less complete `tier{1,2,3,4}` definitions,
>> let's mention them in `testing.md`.
>>
>> Current patch is my braindump, I am open for suggestions :)
>
> doc/testing.html line 77:
>
>> 75: tier1: This test group is the first line of defense
>> against bugs. Multiple developers run these tests every day. Normally, at
>> least this tier is ran before integration. Because of the widespread, the
>> tests in tier1 are carefully selected and optimized to run
>> fast, and to run in the most stable manner. The test failures in
>> tier1 are usually followed up on quickly, either with fixes, or
>> adding relevant tests to problem list. > href="../.github/workflows/">GitHub Actions workflows, if enabled, run
>> tier1 tests.
>> 76: tier2: This test group covers even more ground.
>> These contain, among other things, tests that either run for too long to be
>> at tier1, test unstable and/or experimental features, test less
>> essential JDK components.
>> 77: tier3: This test group covers more stressful tests,
>> the tests for corner cases not covered by previous tiers, plus the tests
>> that require GUIs. As such, this suite should either be run with low
>> concurrency (TEST_JOBS=1), or without headful tests
>> (JTREG_KEYWORDS=\!headful), or both.
>
> It's not clear to me that these descriptions are useful. Instead of "First
> line of defense against bugs" then it might be better to say something about
> the hotspot and core library tests that are typically run in tier1. Folks
> contributing to some areas of the core, I/O, and networking libraries will
> need to learn about tier2 as that is where many of the tests for these APIs
> and implementations are. So I wouldn't use words like "unstable",
> "experimental", "less essential" but maybe say that it includes tests that
> may require some configuration, like changing firewall rules to allow
> multicast or other traffic.
I'll take another stab at it tomorrow.
The whole tiered testing thing encapsulates that some tests are more important
to run (at least, first or regularly) than others. So we need to somehow
capture that. Discussing what tests are actually running is not relevant,
because `TEST.groups` is the source of truth there.
For a contributor, these descriptions should tell why, when and how often would
you use a particular tier. In other words, good docs capture some thinking
about how contributors are supposed to use the existing tests. It seems to be a
broad consensus that we run `tier1` before non-trivial integrations. It seems
to be a rule of thumb that we care a lot about `tier1` stability and speed, and
we push out less stable, slower or specially-configured tests to higher tiers.
As I inspect the test groups, I think `tier2` is for regular but
non-`tier1`-grade tests, `tier3` is mostly for stress tests, and `tier4` is
(somewhat by construction) is "everything else".
>From here, this what I'd like to tell contributors: run `tier1` early and
>often, the bugs there are likely to be real product bugs and should be looked
>at as soon as possible, run `tier1` + `tier2` to execute all regular tests but
>suspect test bugs more, run `tier3` to add even more stress testing and
>suspect environmental problems when they fail, run `tier4` to run everything
>that previous tiers missed and hope for the best. Maybe I should just put that
>in the docs :)
-
PR: https://git.openjdk.java.net/jdk/pull/5615
Re: RFR: 8274083: Update testing docs to mention tiered testing
On Tue, 21 Sep 2021 14:52:26 GMT, Aleksey Shipilev wrote:
> Now that OpenJDK has more or less complete `tier{1,2,3,4}` definitions, let's
> mention them in `testing.md`.
>
> Current patch is my braindump, I am open for suggestions :)
doc/testing.html line 77:
> 75: tier1: This test group is the first line of defense
> against bugs. Multiple developers run these tests every day. Normally, at
> least this tier is ran before integration. Because of the widespread, the
> tests in tier1 are carefully selected and optimized to run fast,
> and to run in the most stable manner. The test failures in tier1
> are usually followed up on quickly, either with fixes, or adding relevant
> tests to problem list. GitHub Actions
> workflows, if enabled, run tier1 tests.
> 76: tier2: This test group covers even more ground. These
> contain, among other things, tests that either run for too long to be at
> tier1, test unstable and/or experimental features, test less
> essential JDK components.
> 77: tier3: This test group covers more stressful tests,
> the tests for corner cases not covered by previous tiers, plus the tests that
> require GUIs. As such, this suite should either be run with low concurrency
> (TEST_JOBS=1), or without headful tests
> (JTREG_KEYWORDS=\!headful), or both.
It's not clear to me that these descriptions are useful. Instead of "First line
of defense against bugs" then it might be better to say something about the
hotspot and core library tests that are typically run in tier1. Folks
contributing to some areas of the core, I/O, and networking libraries will need
to learn about tier2 as that is where many of the tests for these APIs and
implementations are. So I wouldn't use words like "unstable", "experimental",
"less essential" but maybe say that it includes tests that may require some
configuration, like changing firewall rules to allow multicast or other traffic.
-
PR: https://git.openjdk.java.net/jdk/pull/5615
RFR: 8274083: Update testing docs to mention tiered testing
Now that OpenJDK has more or less complete `tier{1,2,3,4}` definitions, let's
mention them in `testing.md`.
Current patch is my braindump, I am open for suggestions :)
-
Commit messages:
- Fix
Changes: https://git.openjdk.java.net/jdk/pull/5615/files
Webrev: https://webrevs.openjdk.java.net/?repo=jdk=5615=00
Issue: https://bugs.openjdk.java.net/browse/JDK-8274083
Stats: 41 lines in 2 files changed: 41 ins; 0 del; 0 mod
Patch: https://git.openjdk.java.net/jdk/pull/5615.diff
Fetch: git fetch https://git.openjdk.java.net/jdk pull/5615/head:pull/5615
PR: https://git.openjdk.java.net/jdk/pull/5615
