Branch: refs/heads/main
Home: https://github.com/WebKit/WebKit
Commit: 4d47d17af08a370737e69269fb7ad8bd63aa9786
https://github.com/WebKit/WebKit/commit/4d47d17af08a370737e69269fb7ad8bd63aa9786
Author: Issac Roy <[email protected]>
Date: 2026-08-14 (Fri, 14 Aug 2026)
Changed paths:
M Tools/CISupport/ews-build/factories_unittest.py
M Tools/CISupport/ews-build/layout_test_failures.py
M Tools/CISupport/ews-build/layout_test_failures_unittest.py
M Tools/CISupport/ews-build/results_db.py
M Tools/CISupport/ews-build/steps.py
M Tools/CISupport/ews-build/steps_unittest.py
M Tools/Scripts/libraries/resultsdbpy/resultsdbpy/model/ews_context.py
M
Tools/Scripts/libraries/resultsdbpy/resultsdbpy/model/ews_context_unittest.py
Log Message:
-----------
TLF: Report EWS test failures and flakiness to the results database
https://bugs.webkit.org/show_bug.cgi?id=318291
rdar://157892382
Reviewed by Aakash Jain and Sam Sneddon.
Record what layout tests did on EWS so a later change can consult that history.
New failures and
each category of flake are uploaded with the real per-test result, the commit
EWS tested against,
and enough build context to attribute a row later.
Each report leaves the step that derived it rather than a trailing step of its
own. EWS calls
buildFinished() as ordinary control flow, and alwaysRun does not survive it, so
a step queued
behind the analyze steps never runs at all: on a failing build, which is the
only interesting
kind, everything the build observed would be discarded.
A test can flake three ways, and they are not equally damning. It can flake
inside a single step's
own retries, fail one of the two runs and pass the other, or flake on the clean
tree, where the
change cannot be responsible. Each is reported under its own flaky_type as soon
as it is known,
and a report spanning both runs keeps each run's own outcome alongside the
merged sequence, since
flattening them loses which run saw what.
The timestamp is per run rather than per build, so the several reports one
build makes do not
collide on a primary key whose uuid is commit-derived. Two classifications of
the same test within
one run collide the same way, so flaky_type joins the flaky table's clustering
key. Cassandra
cannot add a clustering column to an existing table, so that table is created
under a new name
instead of requiring a manual DROP TABLE wherever it has already been deployed.
Reporting is skipped unless the build can describe itself completely enough to
be found again.
The architecture comes from the machine that ran the tests, not from the
builder configuration,
which for the macOS-Sequoia queues names both architectures at once and so
matches no machine.
The comparison between the two runs is skipped when either hit its failure
limit, since a
truncated run covers a different subset of the suite and a test missing from
one of them was
never run rather than passing.
An unreachable results database never changes a step's verdict.
* Tools/CISupport/ews-build/results_db.py:
(ResultsDatabase.report_ews): Upload to /api/upload/ews, and log the tests the
server says it
stored rather than the ones we sent.
(ResultsDatabase):
(ResultsDatabase.make_request):
(ResultsDatabase.has_commit):
* Tools/CISupport/ews-build/steps.py:
(ResultsDBReportMixin): Reporting shared by every suite; hosts supply the suite
and the reports.
(ResultsDBReportMixin.suite):
(ResultsDBReportMixin.results_db_query_configuration):
(ResultsDBReportMixin.results_db_configuration):
(ResultsDBReportMixin.results_db_details):
(ResultsDBReportMixin.merged_results): One result per test across runs, keeping
each run's own
outcome as well as the merged sequence.
(ResultsDBReportMixin.report_to_results_db):
(ShowIdentifier):
(ShowIdentifier.run): Read the commit timestamp from git-webkit find --json, so
the commit is
fully defined without reinterpreting a human-readable date.
(RunWebKitTests):
(RunWebKitTests.run):
(RunWebKitTests.runCommand):
(RunWebKitTests.filter_failures_using_results_db):
(RunWebKitTestsInStressMode):
(RunWebKitTestsEWSSiteIsolation):
(RunWebKitTestsEWSSiteIsolation.results_db_query_configuration):
(RunWebKitTestsEWSSiteIsolation.filter_failures_using_results_db):
(ReRunWebKitTests.runCommand):
(RunWebKitTestsWithoutChange.runCommand):
(AnalyzeLayoutTestsResults):
(AnalyzeLayoutTestsResults.report_failure):
(AnalyzeLayoutTestsResults.run):
(RunWebKitTestsRepeatFailuresRedTree.runCommand):
(RunWebKitTestsRepeatFailuresWithoutChangeRedTree.runCommand):
* Tools/CISupport/ews-build/layout_test_failures.py:
(LayoutTestFailures): Keep each reported test's result leaf.
(LayoutTestFailures.__init__):
(LayoutTestFailures.results_from_string):
(LayoutTestFailures.results_from_string.collect_tests_that_did_not_pass):
(LayoutTestFailures.results_from_string.get_failing_tests): Deleted.
* Tools/CISupport/ews-build/factories_unittest.py:
(TestExpectedBuildSteps.test_steps_that_report_declare_the_suite_they_run):
* Tools/CISupport/ews-build/layout_test_failures_unittest.py:
(TestLayoutTestFailures.test_results_from_string_valid_jsonp):
(TestLayoutTestFailures.test_results_from_string_valid_json):
(TestLayoutTestFailures.test_results_records_the_result_leaf_for_every_reported_test):
(TestLayoutTestFailures):
(TestLayoutTestFailures.test_results_skips_tests_that_are_neither_failing_nor_flaky):
(TestLayoutTestFailures.test_results_from_string_split_at_4096):
* Tools/CISupport/ews-build/steps_unittest.py:
(BuildStepMixinAdditions):
(BuildStepMixinAdditions.layout_test_failures):
(BuildStepMixinAdditions.setup_test_build_step):
(BuildStepMixinAdditions.setup_test_build_step.record_report_ews):
(TestFilterLayoutTestFailuresUsingResultsDB.test_queries_include_the_flavor_being_tested):
(TestFilterLayoutTestFailuresUsingResultsDB):
(TestFilterLayoutTestFailuresUsingResultsDB.test_site_isolation_queries_across_platforms):
(TestFilterLayoutTestFailuresUsingResultsDB.test_caps_number_of_results_db_queries):
(TestRunWebKitTestsRedTree.test_set_properties_when_executed_scope_this_class):
(TestReportToResultsDB):
(TestReportToResultsDB.setUp):
(TestReportToResultsDB.tearDown):
(TestReportToResultsDB.configureStep):
(TestReportToResultsDB.test_reports_the_configuration_the_tests_ran_in):
(TestReportToResultsDB.test_folds_build_provenance_into_details):
(TestReportToResultsDB.test_authors_fall_back_to_the_patch_author):
(TestReportToResultsDB.test_reports_the_commit_ews_tested):
(TestReportToResultsDB.test_commit_is_left_resolvable_without_a_timestamp):
(TestReportToResultsDB.test_reports_when_the_run_started):
(TestReportToResultsDB.test_falls_back_to_the_current_time_when_no_run_was_timed):
(TestReportToResultsDB.test_merge_skips_a_test_no_run_produced_a_result_for):
(TestReportToResultsDB.test_records_results_against_the_branch_the_change_targets):
(TestReportToResultsDB.test_reports_nothing_without_a_version):
(TestReportToResultsDB.test_reports_nothing_without_an_architecture):
(TestReportToResultsDB.test_reports_the_architecture_the_tests_ran_on):
(TestReportToResultsDB.test_reports_nothing_for_several_architectures_at_once):
(TestReportToResultsDB.test_reports_for_a_wildcard_builder_that_resolves_to_apple):
(TestReportToResultsDB.test_a_step_that_provokes_flakiness_reports_nothing):
(TestReportToResultsDB.test_an_unreachable_results_database_does_not_fail_the_step):
(TestReportToResultsDB.test_an_unreachable_results_database_does_not_fail_the_step.explode):
(TestReportToResultsDB.test_uploads_the_platform_name_queries_ask_for):
(TestReportToResultsDB.test_merges_a_tests_outcomes_across_runs_without_repeating_them):
(TestLayoutTestStepsReportAsTheyRun):
(TestLayoutTestStepsReportAsTheyRun.FakeLogObserver):
(TestLayoutTestStepsReportAsTheyRun.FakeLogObserver.__init__):
(TestLayoutTestStepsReportAsTheyRun.FakeLogObserver.getStdout):
(TestLayoutTestStepsReportAsTheyRun.FakeLogObserver.getStderr):
(TestLayoutTestStepsReportAsTheyRun.FakeLogObserver.getHeaders):
(TestLayoutTestStepsReportAsTheyRun.setUp):
(TestLayoutTestStepsReportAsTheyRun.tearDown):
(TestLayoutTestStepsReportAsTheyRun.configureStep):
(TestLayoutTestStepsReportAsTheyRun.configureStep.fake_filter):
(TestLayoutTestStepsReportAsTheyRun.reported):
(TestLayoutTestStepsReportAsTheyRun.test_the_first_run_reports_its_own_flakes):
(TestLayoutTestStepsReportAsTheyRun.test_the_first_run_records_its_failures_for_later_reports):
(TestLayoutTestStepsReportAsTheyRun.test_the_rerun_reports_its_flakes_and_what_differed_from_the_first_run):
(TestLayoutTestStepsReportAsTheyRun.test_the_rerun_reports_no_inter_step_flakes_when_its_own_run_was_truncated):
(TestLayoutTestStepsReportAsTheyRun.test_the_rerun_reports_no_inter_step_flakes_when_the_first_run_was_truncated):
(TestLayoutTestStepsReportAsTheyRun.test_the_rerun_reports_no_inter_step_flakes_without_a_first_run_to_compare):
(TestLayoutTestStepsReportAsTheyRun.test_the_clean_tree_run_reports_flakes_the_change_cannot_have_caused):
(TestLayoutTestStepsReportAsTheyRun.test_the_redtree_repeat_run_reports_with_change_flakes):
(TestLayoutTestStepsReportAsTheyRun.test_the_redtree_clean_tree_repeat_run_reports_without_change_flakes):
(TestLayoutTestStepsReportAsTheyRun.test_the_analyze_step_reports_the_failures_it_attributes_to_the_change):
(TestRunWebKitTestsRepeatFailuresRedTree.test_set_properties_when_executed_scope_this_class):
(TestRunWebKitTestsRepeatFailuresWithoutChangeRedTree.test_set_properties_when_executed_scope_this_class):
(MockLayoutTestFailures): Deleted.
(MockLayoutTestFailures.__init__): Deleted.
* Tools/Scripts/libraries/resultsdbpy/resultsdbpy/model/ews_context.py:
(EWSContext.EWSFlakesByStartTime): flaky_type joins the clustering key, under a
new table name
since a clustering column cannot be added to an existing table.
(EWSContext.__init__):
(EWSContext.record_results):
(EWSContext.find_for_test):
(EWSContext.EWSFlakyTestsByStartTime): Deleted.
(EWSContext.EWSFlakyTestsByStartTime.unpack): Deleted.
* Tools/Scripts/libraries/resultsdbpy/resultsdbpy/model/ews_context_unittest.py:
(EWSContextTest):
(EWSContextTest.test_both_flaky_types_from_one_run_are_kept):
Canonical link: https://commits.webkit.org/319195@main
To unsubscribe from these emails, change your notification settings at
https://github.com/WebKit/WebKit/settings/notifications
