Branch: refs/heads/main
Home: https://github.com/WebKit/WebKit
Commit: 96d2789262f79ea36dad789e9a3dae5a334b3ff1
https://github.com/WebKit/WebKit/commit/96d2789262f79ea36dad789e9a3dae5a334b3ff1
Author: Carlos Alberto Lopez Perez <[email protected]>
Date: 2026-01-28 (Wed, 28 Jan 2026)
Changed paths:
M Tools/Scripts/webkitpy/common/host_mock.py
M Tools/Scripts/webkitpy/layout_tests/models/test_run_results.py
M Tools/Scripts/webkitpy/layout_tests/models/test_run_results_unittest.py
Log Message:
-----------
[webkitpy][run-webkit-tests] Wrong exit code and report when a test is
repeated (via --repeat-each=X) and there is a mix of unexpected and expected
results
https://bugs.webkit.org/show_bug.cgi?id=306336
Reviewed by Nikolas Zimmermann.
When a test is run multiple times (via --repeat-each=X) then it should only
be considered a regression if _all_ of the results it generated where
unexpected. Otherwise, if there is only one PASS or only one expected failure
it should be considered flaky instead.
And run-webkit-tests should not exit with non-zero error in case of flaky test.
Only the tests results marked as regression should increment the exit code.
This has been causing issues on the WPE WK2 EWS when on the retry step with
patch (layout-tests-repeat-failures) there are flaky tests but not regressions
and the return code of the step is non-zero. That confuses the EWS logic that
expects a list of failed tests when the result is non-zero to guard against
a patch that breaks the runner itself, and since there isn't such list it
ends trigerring an insfrastructure error.
This patch changes the code to count the number of regressions and number
of flakies after the loop processing all the test results is finished.
That avoids counting as regression a case that later will be reclassified
as flaky when the retry-results for the tests are processed.
This also fixes a logic error when the retry results are examined to determine
if a test should be marked as flaky that was introduced in 243493@main:
The test should be marked flaky if there are more than one result and any of
those results match the expectation or PASS.
Also several unit tests are added to ensure to test better this logic.
* Tools/Scripts/webkitpy/common/host_mock.py:
(MockHost.__init__):
* Tools/Scripts/webkitpy/layout_tests/models/test_run_results.py:
(count_tests_with_report_result):
(summarize_results):
* Tools/Scripts/webkitpy/layout_tests/models/test_run_results_unittest.py:
(get_result):
(SummarizedResultsTest.test_summarized_run_metadata):
(summarized_results_repeat_tests):
(SummarizedResultsRepeatResultsFlakiesTest):
(SummarizedResultsRepeatResultsFlakiesTest.setUp):
Canonical link: https://commits.webkit.org/306367@main
To unsubscribe from these emails, change your notification settings at
https://github.com/WebKit/WebKit/settings/notifications
