Branch: refs/heads/main
Home: https://github.com/WebKit/WebKit
Commit: 09e7bd88eb16e6698f416042af509c085382d892
https://github.com/WebKit/WebKit/commit/09e7bd88eb16e6698f416042af509c085382d892
Author: Issac Roy <[email protected]>
Date: 2026-09-03 (Thu, 03 Sep 2026)
Changed paths:
M Tools/CISupport/ews-build/steps.py
M Tools/CISupport/ews-build/steps_unittest.py
M Tools/Scripts/run-javascriptcore-tests
Log Message:
-----------
[EWS] JSC EWS blames the pull request author for pre-existing flaky test
failures
https://bugs.webkit.org/show_bug.cgi?id=323155
rdar://186409107
Reviewed by Aakash Jain.
Extend results database reporting to JSC. The runner retries stress tests
itself, so flakiness comes from its own retries rather than a re-run pair, and a
test it retried that still failed is reported as a failure rather than as
evidence of flakiness. Every exit of `AnalyzeJSCTestsResults` ends the build, so
reporting happens inside the steps that produce each result rather than in a
step queued after them, which would never run.
`runJSCStressTests` writes only the notable per-test results to
`jsc_results.json`. `%reportData` carries a row for every test the stress run
executed, which the `uploadResults` path needs and the json file's consumers do
not: they read the failures and the flakes. Writing all of it made the file a
10 MB step log, parsed in the master and persisted as a build property on every
JSC run and again on the clean-tree retry, to carry two rows.
`AnalyzeJSCTestsResults` blamed the pull request author for pre-existing test
failures. It computed the failures it attributes to the change from the raw
`jsc_stress_test_failures` and `jsc_binary_failures` properties, so a test the
results database already recorded as failing was reported as new when it failed
with the change and passed on the clean tree. The filtered lists
`RunJavaScriptCoreTests` writes for exactly this purpose were read nowhere. The
analyzer now prefers `jsc_stress_test_failures_filtered` and
`jsc_binary_failures_filtered`, falling back to the raw properties for a build
that is not against the default branch, and filters below the check for no
failures at all, which needs the raw lists.
`_check_for_preexisting_failures` no longer stops at the first failure the
database does not recognize. That short circuit left the filtered lists
incomplete, which is harmless while nothing reads them and wrong now that the
analyzer does. `MAX_FAILURES_TO_CHECK_RESULTS_DB` bounds the queries instead, at
50 to match API tests. A failure past the cap is treated as new by both the
pre-existing check and the flakiness read.
Three property writes in `RunJavaScriptCoreTests.runCommand` hardcoded a `jsc_`
prefix while every write around them derives from `self.prefix`.
`RunJSCTestsWithoutChange` inherits `runCommand`, so a clean-tree run with any
failure of its own overwrote the with-change run's filtered lists, and the
analyzer would then subtract the clean-tree failures from themselves and report
nothing at all.
`RunJavaScriptCoreTests` also asks the database for a flakiness verdict on each
stress test failure it cannot otherwise explain, dropping the ones it recognizes
from the filtered lists so the analyzer never attributes them to the change. It
acts on the same three verdicts `RunWebKitTests` acts on. Binary test failures
are left out: nothing reports them to the database, so a verdict for one could
only ever come back unknown.
A build whose failures the database explains now ends at the run step, green
with a warning, instead of paying a revert, a recompile and two more JSC runs to
reach the same conclusion. Reaching that branch on a flakiness verdict rather
than only on a pre-existing failure is what makes the read worth running there.
`results_db_ignore_message` moved from `RunWebKitTests` to
`ResultsDBReportMixin` so both suites word that summary the same way, which
pluralizes JSC's pre-existing-only message and appends "based on results-db".
A `BetweenBuilds` verdict with no intra-build evidence is recorded in
`results-db_jsc_flaky_unsupported` and not acted on. A test that fails on some
builds and passes on others, never twice within one build, is as consistent with
a configuration-dependent failure as with a flake, and the queue would excuse it
on every build. `RunJSCTestsWithoutChange` excuses nothing at all:
`AnalyzeJSCTestsResults` subtracts the clean-tree run's raw failures, so its
filtered lists are read by no one and the queries behind them buy nothing.
A verdict outside `INCLUDED_FLAKY_VERDICTS` is logged as would-have-ignored
rather than excused; nothing narrows the set today, so the path exists for a
queue that later does. The four new `results-db_jsc_` properties are read by
nothing, and are there for the build page and for measuring the read.
`jsc_results` is set below the failure-threshold return, so a run with more
failures than the step reports individually does not put the map back for the
analyzer to read. `jsc_flaky_and_passed` starts being set for the first time:
the property was read from a `flakyAndPassed` key the runner has never emitted,
so `send_email_for_flaky_failure` also fires for the first time.
Canonical link: https://commits.webkit.org/320409@main
To unsubscribe from these emails, change your notification settings at
https://github.com/WebKit/WebKit/settings/notifications
