Branch: refs/heads/main
Home: https://github.com/WebKit/WebKit
Commit: 24279dccdd79ccf2e03c043ea4c499f8738530a2
https://github.com/WebKit/WebKit/commit/24279dccdd79ccf2e03c043ea4c499f8738530a2
Author: Issac Roy <[email protected]>
Date: 2026-08-21 (Fri, 21 Aug 2026)
Changed paths:
M Tools/CISupport/Shared/steps.py
M Tools/CISupport/ews-build/results_db.py
M Tools/CISupport/ews-build/steps.py
M Tools/CISupport/ews-build/steps_unittest.py
M Tools/CISupport/ews-build/twisted_additions.py
Log Message:
-----------
TLF: Use historical flakiness information from EWS to ignore flaky tests
rdar://157892382
https://bugs.webkit.org/show_bug.cgi?id=318291
Reviewed by Aakash Jain.
`claude` summary:
EWS reports test failures and flakiness to results.webkit.org but never reads
any of it back, so a test already known to flake still fails the build and
blames whoever happened to be testing. `flaky_verdicts_for` consults that
history for every failure a step surfaced. A clean-tree flake convicts on one
build, since the change cannot have caused it. Convicting with the change in
the tree takes `PRS_FOR_DIRTY_TREE_FLAKE` pull requests, and blaming other
people's patches takes `PRS_FOR_BETWEEN_BUILD_FLAKE` and
`AUTHORS_FOR_BETWEEN_BUILD_FLAKE`. Counting builds would not do: `build_url`
carries the build number, so retrying one pull request would let a change that
genuinely made a test non-deterministic excuse its own failure.
Both rules tally the same three sets over the same rows, so they share
`_evidence_in`, `_convict` and `_rows_by_flaky_type`, and both report
`build_urls` — `results_db_details` always wrote them and only the reader
ignored them. A row with a null `flaky_type` no longer raises from `sorted()`
inside the code meant to make bad data visible.
Test names ride in the query string, and a step's worth of long WPT paths
overflows nginx's 8k request line, so `_query_flaky` asks about
`TESTS_PER_FLAKY_QUERY` at a time, each chunk allowed
`FLAKY_QUERY_TIMEOUT_SECONDS` rather than the 10s default that used to cover a
single test. A failed chunk fails the whole query, since a partial answer reads
as an absence of history for the rest. `TwistedAdditions.request` builds the
query with `urlencode(doseq=True)`, which repeats a key for a list value and
escapes names that carry their own query string.
Nothing is ignored yet. `SHOULD_IGNORE_FLAKY_TESTS` is False, so a verdict is
recorded and logged but no test leaves the failing set and no build changes
colour. `SetBuildSummary` reads a `force_build_success` property rather than
matching `SUCCESS_MSGS` against a step's summary text, so turning that flag on
does not depend on wording. `FlakyVerdict.intra_build_evidence` records whether
any build saw the test behave inconsistently, exported per run as
`results-db_*_run_flaky`, `_flaky_unsupported` and `_flaky_unknown`: the
between-builds rule is the only one that can convict without that evidence, so
whether it earns its place is a property query rather than a log scrape.
* Tools/CISupport/Shared/steps.py:
(SetBuildSummary):
(SetBuildSummary.run):
* Tools/CISupport/ews-build/results_db.py:
(FlakyVerdict):
(FlakyVerdict.is_flaky):
(FlakyVerdict.evidence):
(ResultsDatabase):
(ResultsDatabase._evidence_in):
(ResultsDatabase._convict):
(ResultsDatabase._rows_by_flaky_type):
(ResultsDatabase._is_intra_build_flake):
(ResultsDatabase._is_inter_build_flake):
(ResultsDatabase._parse_results_ews_response):
(ResultsDatabase._query_flaky):
(ResultsDatabase.flaky_verdicts_for):
(ResultsDatabase.flaky_verdicts_for.logger):
* Tools/CISupport/ews-build/steps.py:
(RunWebKitTests):
(RunWebKitTests.__init__):
(RunWebKitTests.runCommand):
(RunWebKitTests.filter_failures_using_results_db):
(RunWebKitTests.results_db_ignore_message):
(RunWebKitTests.evaluateCommand):
(RunWebKitTests.getResultSummary):
(ReRunWebKitTests.evaluateCommand):
(ReRunWebKitTests.runCommand):
* Tools/CISupport/ews-build/steps_unittest.py:
(TestFilterLayoutTestFailuresUsingResultsDB._configure):
(TestFilterLayoutTestFailuresUsingResultsDB):
(TestFilterLayoutTestFailuresUsingResultsDB.test_a_flaky_verdict_is_recorded_without_ignoring_the_failure):
(TestFilterLayoutTestFailuresUsingResultsDB.test_a_test_with_no_verdict_is_not_treated_as_sound):
(TestFilterLayoutTestFailuresUsingResultsDB.test_the_ignore_message_covers_both_categories):
(TestFilterLayoutTestFailuresUsingResultsDB.test_set_build_summary_restores_success_from_the_property):
(TestFilterLayoutTestFailuresUsingResultsDB.test_ignoring_every_failure_sets_the_property_set_build_summary_reads):
(TestFilterLayoutTestFailuresUsingResultsDB.test_ignoring_every_failure_sets_the_property_set_build_summary_reads.Cmd):
(TestFilterLayoutTestFailuresUsingResultsDB.test_wpe_platform_is_translated_to_uppercase_for_results_db):
(TestFilterLayoutTestFailuresUsingResultsDB.test_caps_number_of_results_db_queries):
(TestLayoutTestStepsReportAsTheyRun):
(TestLayoutTestStepsReportAsTheyRun.test_the_first_run_exports_the_verdicts_it_relied_on):
(TestLayoutTestStepsReportAsTheyRun.test_the_first_run_exports_the_verdicts_it_relied_on.fake_filter):
* Tools/CISupport/ews-build/twisted_additions.py:
(TwistedAdditions.request):
Canonical link: https://commits.webkit.org/319616@main
To unsubscribe from these emails, change your notification settings at
https://github.com/WebKit/WebKit/settings/notifications
