Branch: refs/heads/main
Home: https://github.com/WebKit/WebKit
Commit: cb97940984aef5babb9025633eb65d32a76100cb
https://github.com/WebKit/WebKit/commit/cb97940984aef5babb9025633eb65d32a76100cb
Author: Basuke Suzuki <[email protected]>
Date: 2026-03-11 (Wed, 11 Mar 2026)
Changed paths:
M
Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_finder_legacy.py
M Tools/Scripts/webkitpy/layout_tests/controllers/manager.py
M Tools/Scripts/webkitpy/layout_tests/models/test_expectations.py
M Tools/Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py
Log Message:
-----------
[run-webkit-tests] Extend --test-list to support inline TestExpectations
syntax.
https://bugs.webkit.org/show_bug.cgi?id=309696
rdar://172301315
Reviewed by Jonathan Bedard.
The `--test-list` option for `run-webkit-tests` accepts a text file listing
test paths to
run. Currently, the file format only supports plain test paths (one per line)
with `//`
comments. There is no way to annotate individual tests with expectations.
This change extends the `--test-list` file format to support inline
expectations using the
standard TestExpectations bracket syntax:
```
fast/dom/test1.html
fast/dom/test2.html [ Skip ]
fast/dom/test3.html [ Slow ]
fast/dom/test4.html [ Pass Failure ]
```
The full TestExpectations syntax is supported (Skip, Slow, Failure, Timeout,
Crash, Pass,
etc.). Both `#` and `//` comments are supported.
**Behavior:**
- **`test.html`** (plain path) — Include the test in the run, keep existing
expectations
from TestExpectations files. If the test is marked Skip in TestExpectations,
it is
un-skipped (existing behavior, unchanged).
- **`test.html [ Pass ]`** — Include and explicitly override the expectation to
Pass,
regardless of what TestExpectations files say.
- **`test.html [ Skip ]`** — Include in the test set but skip it. This Skip is
honored
and not overridden by the explicit-specification logic.
- **`test.html [ Slow ]`** — Include with extended timeout.
- **`test.html [ Failure ]`** — Include and expect failure. A failure result
will not be
reported as unexpected.
Inline expectations from `--test-list` files are injected with highest
priority, overriding
expectations from all other sources (generic TestExpectations,
platform-specific, overrides,
and Skipped files). This means a test marked `[ Skip ]` in the standard
TestExpectations
can be un-skipped by listing it in the test-list as a plain path or with `[
Pass ]`.
Conversely, tests marked `[ Skip ]` in the test-list file stay skipped.
Plain paths without brackets do not alter expectations — they only control
which tests are
included in the run. This is consistent with the existing `--test-list`
behavior.
**Motivation:**
When investigating specific sets of tests (e.g., triaging failures, running a
curated
subset for a feature area), it is useful to annotate individual tests in the
list file
itself rather than editing the global TestExpectations files. This avoids
polluting the
shared expectations and allows ad-hoc, per-session control over test behavior.
* Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_finder_legacy.py:
(LayoutTestFinder.__init__):
(LayoutTestFinder._read_test_names_from_file):
(LayoutTestFinder.get_test_list_expectations):
(LayoutTestFinder):
(LayoutTestFinder.get_test_list_skip_paths):
(LayoutTestFinder._strip_comments):
(LayoutTestFinder._strip_hash_and_whitespace):
* Tools/Scripts/webkitpy/layout_tests/controllers/manager.py:
(Manager._collect_tests):
(Manager.run):
* Tools/Scripts/webkitpy/layout_tests/models/test_expectations.py:
(TestExpectations.add_test_list_expectations):
* Tools/Scripts/webkitpy/layout_tests/run_webkit_tests_integrationtest.py:
(RunTest.test_test_list_with_skip_expectation):
(RunTest):
(RunTest.test_test_list_with_hash_comments):
(RunTest.test_test_list_with_mixed_content):
(RunTest.test_test_list_with_slow_expectation):
(RunTest.test_test_list_with_failure_expectation):
(RunTest.test_test_list_with_double_slash_comments):
(RunTest.test_test_list_skip_not_overridden):
(RunTest.test_test_list_unskip_with_pass):
Canonical link: https://commits.webkit.org/309087@main
To unsubscribe from these emails, change your notification settings at
https://github.com/WebKit/WebKit/settings/notifications
