On Thu, 25 Jan 2024 12:02:35 GMT, Aleksey Shipilev <sh...@openjdk.org> wrote:
> When looking at [JDK-8324647](https://bugs.openjdk.org/browse/JDK-8324647), I > was surprised to see that GHA has this output: > > > Running test 'jtreg:test/lib-test:tier1' > /home/runner/work/jdk/jdk/test/lib-test/TEST.groups: group all: group > includes itself > Error: One or more groups are invalid > Finished running test 'jtreg:test/lib-test:tier1' > Test report is stored in > build/run-test-prebuilt/test-results/jtreg_test_lib_test_tier1 > > ============================== > Test summary > ============================== > TEST TOTAL PASS FAIL ERROR >>> jtreg:test/lib-test:tier1 1 0 0 1 << > ============================== > TEST FAILURE > > > ...and yet, the tests are recorded as "green"! I think this is because our > error reporting code is tad buggy. AFAICS, `make test-prebuilt` does not > report a non-zero exit code on failure, and the external script looks for > `build/run-test-prebuilt/make-support/exit-with-error` to check for error. > But it does not _set_ the exit code on its own when the failure is > discovered, neither it sets `failure=true` on generic failure. > > I think at very minimum we should report `failure=true` on generic failure. I > thought about reporting non-zero exit code from the parsing script, but it > does not feel clean to return non-zero exit code from the _parsing script_. > Non-zero exit code from that script should signify the error in the script > itself. > > Additional testing: > - [x] Current GHA with lib-test failure (now red) > - [x] Current GHA with lib-test fix (now green) > - [x] GHA with artificial errors: > https://github.com/shipilev/jdk/actions/runs/7652797646/job/20854472816 Marked as reviewed by erikj (Reviewer). ------------- PR Review: https://git.openjdk.org/jdk/pull/17568#pullrequestreview-1844334099
