Re: RFR: 8273314: Add tier4 test groups [v3]
On Mon, 6 Sep 2021 13:22:03 GMT, Aleksey Shipilev wrote:
>> During the review of JDK-8272914 that added hotspot:tier{2,3} groups,
>> @iignatev suggested to create tier4 groups that capture all tests not in
>> tiers{1,2,3}.
>>
>> Caveats:
>> - I excluded `applications` from `hotspot:tier4`, because they require test
>> dependencies (e.g. jcstress).
>> - `jdk:tier4` only runs well with `JTREG_KEYWORDS=!headful` or reduced
>> concurrency with `TEST_JOBS=1`, because headful tests cannot run in parallel
>>
>> Sample run with `JTREG_KEYWORDS=!headful`:
>>
>>
>> ==
>> Test summary
>> ==
>>TEST TOTAL PASS FAIL ERROR
>>
jtreg:test/hotspot/jtreg:tier4 3585 3584 0 1
<<
jtreg:test/jdk:tier4 2893 2887 5 1
<<
>>jtreg:test/langtools:tier40 0 0 0
>>
>>jtreg:test/jaxp:tier4 0 0 0 0
>>
>> ==
>>
>> real 699m39.462s
>> user 6626m8.448s
>> sys 1110m43.704s
>>
>>
>> There are interesting test failures on my machine, which I would address
>> separately.
>
> Aleksey Shipilev has updated the pull request incrementally with one
> additional commit since the last revision:
>
> Drop applications and fix the comment
All right, I am convinced that current patch is as good as it gets. GUI tests
still do not run well with default parallelism, but I see no reason to block
this integration before that is resolved. Either run `tier4` in headless mode,
or limit the parallelism.
@iignatev, @mrserb, @dholmes-ora -- are you good with this?
-
PR: https://git.openjdk.java.net/jdk/pull/5357
Re: RFR: 8273314: Add tier4 test groups [v3]
On Mon, 6 Sep 2021 13:22:03 GMT, Aleksey Shipilev wrote:
>> During the review of JDK-8272914 that added hotspot:tier{2,3} groups,
>> @iignatev suggested to create tier4 groups that capture all tests not in
>> tiers{1,2,3}. I have excluded `vmTestbase` and `hotspot:tier4,` because they
>> take 10+ hours on my highly parallel machine. I have also excluded
>> `applications` from `hotspot:tier4`, because they require test dependencies
>> (e.g. jcstress).
>>
>> Sample run:
>>
>>
>> ==
>> Test summary
>> ==
>>TEST TOTAL PASS FAIL ERROR
>>
jtreg:test/hotspot/jtreg:tier4 426 425 1 0
<<
jtreg:test/jdk:tier4 2891 2885 4 2
<<
>>jtreg:test/langtools:tier40 0 0 0
>>
>>jtreg:test/jaxp:tier4 0 0 0 0
>>
>> ==
>>
>> real 64m13.994s
>> user 1462m1.213s
>> sys 39m38.032s
>>
>>
>> There are interesting test failures on my machine, which I would address
>> separately.
>
> Aleksey Shipilev has updated the pull request incrementally with one
> additional commit since the last revision:
>
> Drop applications and fix the comment
Progress:
a) `hotspot:tier4` still runs cleanly, and a bit faster due to recent
`vmTestbase` parallelism improvements.
b) `jdk:tier4` has a lot of failures in headful mode, probably because the
tests do not like to run in parallel, see for example #5533. It would take a
while to resolve for all GUI tests. If we are in agreement that current `tier4`
definition is good, maybe it would be proper to integrate this PR, and then
make `tier4` clean for headful mode?
-
PR: https://git.openjdk.java.net/jdk/pull/5357
Re: RFR: 8273314: Add tier4 test groups [v3]
On Mon, 6 Sep 2021 13:22:03 GMT, Aleksey Shipilev wrote:
>> During the review of JDK-8272914 that added hotspot:tier{2,3} groups,
>> @iignatev suggested to create tier4 groups that capture all tests not in
>> tiers{1,2,3}. I have excluded `vmTestbase` and `hotspot:tier4,` because they
>> take 10+ hours on my highly parallel machine. I have also excluded
>> `applications` from `hotspot:tier4`, because they require test dependencies
>> (e.g. jcstress).
>>
>> Sample run:
>>
>>
>> ==
>> Test summary
>> ==
>>TEST TOTAL PASS FAIL ERROR
>>
jtreg:test/hotspot/jtreg:tier4 426 425 1 0
<<
jtreg:test/jdk:tier4 2891 2885 4 2
<<
>>jtreg:test/langtools:tier40 0 0 0
>>
>>jtreg:test/jaxp:tier4 0 0 0 0
>>
>> ==
>>
>> real 64m13.994s
>> user 1462m1.213s
>> sys 39m38.032s
>>
>>
>> There are interesting test failures on my machine, which I would address
>> separately.
>
> Aleksey Shipilev has updated the pull request incrementally with one
> additional commit since the last revision:
>
> Drop applications and fix the comment
Hi Aleksey,
I've discussed this with Igor and while I don't agree with the rationale I
won't "block it".
Cheers,
David
-
PR: https://git.openjdk.java.net/jdk/pull/5357
Re: RFR: 8273314: Add tier4 test groups [v3]
On Mon, 6 Sep 2021 13:22:03 GMT, Aleksey Shipilev wrote:
>> During the review of JDK-8272914 that added hotspot:tier{2,3} groups,
>> @iignatev suggested to create tier4 groups that capture all tests not in
>> tiers{1,2,3}. I have excluded `vmTestbase` and `hotspot:tier4,` because they
>> take 10+ hours on my highly parallel machine. I have also excluded
>> `applications` from `hotspot:tier4`, because they require test dependencies
>> (e.g. jcstress).
>>
>> Sample run:
>>
>>
>> ==
>> Test summary
>> ==
>>TEST TOTAL PASS FAIL ERROR
>>
jtreg:test/hotspot/jtreg:tier4 426 425 1 0
<<
jtreg:test/jdk:tier4 2891 2885 4 2
<<
>>jtreg:test/langtools:tier40 0 0 0
>>
>>jtreg:test/jaxp:tier4 0 0 0 0
>>
>> ==
>>
>> real 64m13.994s
>> user 1462m1.213s
>> sys 39m38.032s
>>
>>
>> There are interesting test failures on my machine, which I would address
>> separately.
>
> Aleksey Shipilev has updated the pull request incrementally with one
> additional commit since the last revision:
>
> Drop applications and fix the comment
Once again, the disconnect between Oracle and OpenJDK test definitions seems to
be the problem for Oracle's side. Rectifying that disconnect might fall under
the scope of this PR, but I have to point out that it is a courtesy of upstream
open-source project to care about proprietary downstream definitions.
More to the point: since `tier4` as "catch-all-the-rest" was Igor's idea, I
assumed that would be in agreement with Oracle's test definitions. Following
this discussion, it seems I assumed wrong. So it puts me in a weird position to
be between two Oracle engineers arguing about proprietary test definitions I
cannot really know about, and have no decision power about. For all I care for
OpenJDK, we might as well model `tier4` after what Oracle does, as to minimize
confusion for Oracle engineers. But then again, I have no idea what Oracle
means by `tier4`. So as the alternative, I can postpone this PR until you folks
have a coherent view on this, or I can just give up on this PR and re-assign
the RFE to Igor, assuming he is willing to work this out.
Tell me what you want me to do here.
-
PR: https://git.openjdk.java.net/jdk/pull/5357
Re: RFR: 8273314: Add tier4 test groups [v3]
On 7/09/2021 2:20 pm, Igor Ignatyev wrote:
On Mon, 6 Sep 2021 13:22:03 GMT, Aleksey Shipilev wrote:
During the review of JDK-8272914 that added hotspot:tier{2,3} groups, @iignatev
suggested to create tier4 groups that capture all tests not in tiers{1,2,3}. I
have excluded `vmTestbase` and `hotspot:tier4,` because they take 10+ hours on
my highly parallel machine. I have also excluded `applications` from
`hotspot:tier4`, because they require test dependencies (e.g. jcstress).
Sample run:
==
Test summary
==
TEST TOTAL PASS FAIL ERROR
jtreg:test/hotspot/jtreg:tier4 426 425 1 0 <<
jtreg:test/jdk:tier4 2891 2885 4 2 <<
jtreg:test/langtools:tier40 0 0 0
jtreg:test/jaxp:tier4 0 0 0 0
==
real64m13.994s
user1462m1.213s
sys 39m38.032s
There are interesting test failures on my machine, which I would address
separately.
Aleksey Shipilev has updated the pull request incrementally with one additional
commit since the last revision:
Drop applications and fix the comment
_Mailing list message from [David Holmes](mailto:david.hol...@oracle.com) on
[hotspot-dev](mailto:hotspot-...@mail.openjdk.java.net):_
On 7/09/2021 1:17 am, Aleksey Shipilev wrote:
@dholmes-ora: Generally speaking, all `tierX` definitions are rather arbitrary, as there
seem to be nothing intrinsic about the tests to be in a particular tier. In other words,
what `tierX` consists of is a matter of agreement. I'd say `hotspot:tier4` is "all
assorted Hotspot tests that are not application-specific suites"
The difference is that your previous work just consolidated the existing
subsystem tier 1-3 definitions, but here you are choosing to define "all
the rest" as tier 4. I don't think it is actually helpful/useful to
anyone - and it bears no resemblance whatsoever to what we call "tier
4", so that will just lead to unnecessary confusion IMO.
@dholmes-ora , although I fully agree that this might lead to some misunderstanding b/w Oracle and non-Oracle folks, I don't see how it's different from the previous patch, which introduced `hotspot:tier2` and `hotspot:tier3`.
Because hotspot:tier2 and hotspot:tier3 simply grouped existing
component definitions for tiers 2 and 3:
+tier2 = \
+ :hotspot_tier2_runtime \
+ :hotspot_tier2_runtime_platform_agnostic \
+ :hotspot_tier2_serviceability \
+ :tier2_gc_epsilon \
+ :tier2_gc_shenandoah
+
+tier3 = \
+ :hotspot_tier3_runtime \
+ :tier3_gc_shenandoah
but that is not the case for tier4.
even if we reduce `tierN` to just a set of tests, the test groups added by
8272914 bear as much resemblance to the test sets used in Oracle's tier2-3 as
the suggested `hotspot:tier4` groups in this patch to the actual `tier4`
definition used in Oracle's internal system, e.g. `hotspot:tier2` group has 0
tests from `test/hotspot/jtreg/compiler`, but Oracle's `tier2` does include a
number of `test/hotspot/jtreg/compiler` tests (which aren't part of `:tier1`).
I believe that this patch actually moves us closer to a convergence point, as
the union of `hotspot:tier1` -- `hotspot:tier4` test groups is very close to
the test sets used in hotspot parts of Oracle's `tier1` -- `tier4` definitions.
We can discuss Oracle's tier definitions privately - I don't see any
connection between those and this tier4 however.
Thanks,
David
-
Thanks,
-- Igor
-
PR: https://git.openjdk.java.net/jdk/pull/5357
Re: RFR: 8273314: Add tier4 test groups [v3]
On Mon, 6 Sep 2021 13:22:03 GMT, Aleksey Shipilev wrote:
>> During the review of JDK-8272914 that added hotspot:tier{2,3} groups,
>> @iignatev suggested to create tier4 groups that capture all tests not in
>> tiers{1,2,3}. I have excluded `vmTestbase` and `hotspot:tier4,` because they
>> take 10+ hours on my highly parallel machine. I have also excluded
>> `applications` from `hotspot:tier4`, because they require test dependencies
>> (e.g. jcstress).
>>
>> Sample run:
>>
>>
>> ==
>> Test summary
>> ==
>>TEST TOTAL PASS FAIL ERROR
>>
jtreg:test/hotspot/jtreg:tier4 426 425 1 0
<<
jtreg:test/jdk:tier4 2891 2885 4 2
<<
>>jtreg:test/langtools:tier40 0 0 0
>>
>>jtreg:test/jaxp:tier4 0 0 0 0
>>
>> ==
>>
>> real 64m13.994s
>> user 1462m1.213s
>> sys 39m38.032s
>>
>>
>> There are interesting test failures on my machine, which I would address
>> separately.
>
> Aleksey Shipilev has updated the pull request incrementally with one
> additional commit since the last revision:
>
> Drop applications and fix the comment
> _Mailing list message from [David Holmes](mailto:david.hol...@oracle.com) on
> [hotspot-dev](mailto:hotspot-...@mail.openjdk.java.net):_
>
> On 7/09/2021 1:17 am, Aleksey Shipilev wrote:
>
> > @dholmes-ora: Generally speaking, all `tierX` definitions are rather
> > arbitrary, as there seem to be nothing intrinsic about the tests to be in a
> > particular tier. In other words, what `tierX` consists of is a matter of
> > agreement. I'd say `hotspot:tier4` is "all assorted Hotspot tests that are
> > not application-specific suites"
>
> The difference is that your previous work just consolidated the existing
> subsystem tier 1-3 definitions, but here you are choosing to define "all
> the rest" as tier 4. I don't think it is actually helpful/useful to
> anyone - and it bears no resemblance whatsoever to what we call "tier
> 4", so that will just lead to unnecessary confusion IMO.
@dholmes-ora , although I fully agree that this might lead to some
misunderstanding b/w Oracle and non-Oracle folks, I don't see how it's
different from the previous patch, which introduced `hotspot:tier2` and
`hotspot:tier3`. even if we reduce `tierN` to just a set of tests, the test
groups added by 8272914 bear as much resemblance to the test sets used in
Oracle's tier2-3 as the suggested `hotspot:tier4` groups in this patch to the
actual `tier4` definition used in Oracle's internal system, e.g.
`hotspot:tier2` group has 0 tests from `test/hotspot/jtreg/compiler`, but
Oracle's `tier2` does include a number of `test/hotspot/jtreg/compiler` tests
(which aren't part of `:tier1`). I believe that this patch actually moves us
closer to a convergence point, as the union of `hotspot:tier1` --
`hotspot:tier4` test groups is very close to the test sets used in hotspot
parts of Oracle's `tier1` -- `tier4` definitions.
Thanks,
-- Igor
-
PR: https://git.openjdk.java.net/jdk/pull/5357
Re: RFR: 8273314: Add tier4 test groups [v3]
> During the review of JDK-8272914 that added hotspot:tier{2,3} groups,
> @iignatev suggested to create tier4 groups that capture all tests not in
> tiers{1,2,3}. I have excluded `vmTestbase` and `hotspot:tier4,` because they
> take 10+ hours on my highly parallel machine. I have also excluded
> `applications` from `hotspot:tier4`, because they require test dependencies
> (e.g. jcstress).
>
> Sample run:
>
>
> ==
> Test summary
> ==
>TEST TOTAL PASS FAIL ERROR
>
>>> jtreg:test/hotspot/jtreg:tier4 426 425 1 0 <<
>>> jtreg:test/jdk:tier4 2891 2885 4 2 <<
>jtreg:test/langtools:tier40 0 0 0
>
>jtreg:test/jaxp:tier4 0 0 0 0
>
> ==
>
> real 64m13.994s
> user 1462m1.213s
> sys 39m38.032s
>
>
> There are interesting test failures on my machine, which I would address
> separately.
Aleksey Shipilev has updated the pull request incrementally with one additional
commit since the last revision:
Drop applications and fix the comment
-
Changes:
- all: https://git.openjdk.java.net/jdk/pull/5357/files
- new: https://git.openjdk.java.net/jdk/pull/5357/files/afb77062..160c13c7
Webrevs:
- full: https://webrevs.openjdk.java.net/?repo=jdk=5357=02
- incr: https://webrevs.openjdk.java.net/?repo=jdk=5357=01-02
Stats: 3 lines in 2 files changed: 0 ins; 0 del; 3 mod
Patch: https://git.openjdk.java.net/jdk/pull/5357.diff
Fetch: git fetch https://git.openjdk.java.net/jdk pull/5357/head:pull/5357
PR: https://git.openjdk.java.net/jdk/pull/5357
