Branch: refs/heads/main
Home: https://github.com/WebKit/WebKit
Commit: 66615c600f5c30183209d9383444224e91d5d347
https://github.com/WebKit/WebKit/commit/66615c600f5c30183209d9383444224e91d5d347
Author: Issac Roy <[email protected]>
Date: 2026-09-02 (Wed, 02 Sep 2026)
Changed paths:
M Tools/CISupport/ews-build/factories_unittest.py
M Tools/CISupport/ews-build/steps.py
M Tools/CISupport/ews-build/steps_unittest.py
M Tools/Scripts/webkitpy/api_tests/manager.py
Log Message:
-----------
[EWS] API test results are not reported to the results database
https://bugs.webkit.org/show_bug.cgi?id=323180
rdar://186426179
Reviewed by Aakash Jain.
Extend results database reporting to API tests, and use the flakiness verdicts
that reporting
makes possible so a pull request is not blamed for a flaky API test failure.
Every exit of AnalyzeAPITestsResults ends the build, so reporting happens
inside the steps that
derive each report rather than in a step queued after them, which would never
run. The API run
properties are prefixed to namespace them from the identically named layout
ones.
AnalyzeAPITestsResults blames the change for a test that failed both runs with
the change and
passed on the clean tree. RunAPITests already drops failures the results
database records as
pre-existing, but a test recorded as flaky rather than failing survives that
filter and fails that
way routinely, so the pull request is blamed for a failure it did not cause and
the author is left
to rerun the queue until the flake goes the other way.
The analyzer now asks the database for a flakiness verdict on each of those
tests and drops the
ones it recognizes, which turns the build green and names them in the build
summary when they are
the only new failures. A CleanTree verdict can never come back for an API test:
no API step reports
WithinStepCleanTree, so the database holds no clean-tree row one could match
against. A query that
fails leaves the failure attributed to the change and records the test in
results-db_api_flaky_unknown. A verdict outside INCLUDED_FLAKY_VERDICTS is
logged as
would-have-ignored rather than excused; nothing narrows the set today, so the
path exists for a
queue that later does.
A BetweenBuilds verdict with no intra-build evidence is recorded in
results-db_api_flaky_unsupported and not acted on. A test that fails on some
builds and passes on
others, never twice within one build, is as consistent with a
configuration-dependent failure as
with a flake, and the queue would excuse it on every build.
The report to the database covers the failures the clean-tree run did not
explain, before the
excused ones are removed from that set. Reporting the narrowed set instead
would starve the verdict
that earned the excuse: the rows behind it would stop being written the moment
it started working,
and it would age out of the flakiness window.
The read belongs in the analyzer rather than beside the pre-existing-failure
filter in RunAPITests.
run-api-tests has no per-test retry, so the only flakiness an API build
observes for itself is the
symmetric difference between the two runs with the change, which ReRunAPITests
reports as
BetweenStepsDirtyTree. Excusing a failure during the first run would pass the
step before the rerun
that produces that evidence ever happens.
* Tools/CISupport/ews-build/factories_unittest.py:
* Tools/CISupport/ews-build/steps.py:
(RunAPITests): Prefix the run properties, and split the json parse so the
per-test results survive
rather than only the failure names.
(RunAPITests.parse_and_set_failures):
(RunAPITests.parse_api_failures_from_string):
(RunAPITests.api_failures_from_json):
(RunAPITests.parse_api_test_json):
(ReRunAPITests.parse_and_set_failures): Report the tests that failed one of the
two runs and passed
the other.
(AnalyzeAPITestsResults):
(AnalyzeAPITestsResults.run): Report the failures attributed to the change
before the build is
finished, and drop the ones the results database recognizes as flaky.
(AnalyzeAPITestsResults.flaky_new_failures_using_results_db):
* Tools/CISupport/ews-build/steps_unittest.py:
* Tools/Scripts/webkitpy/api_tests/manager.py:
(Manager.run): Expose the per-test results already collected for the results
database upload,
limited to the tests that did not pass so a build log is not swamped.
Canonical link: https://commits.webkit.org/320321@main
To unsubscribe from these emails, change your notification settings at
https://github.com/WebKit/WebKit/settings/notifications
