Thanks to the tireless efforts of Dirk Pranke, Ojan Vafai, and numerous other engineers, WebKit now has a new test harness for running the Layout Tests.
It's FAST: 4-core Mac Pro, Debug build: run-webkit-tests: 11m25s new-run-webkit-tests: 3m24s new-run-webkit-tests --experimental-fully-parallel: 2m42 [1] WebKitTools/Scripts/new-run-webkit-tests is ready for you to play with today, however it is not ready to replace run-webkit-tests as our main testing script yet. Pros: - Fast (see above) - Skipping is no longer our only tool, can mark tests as flaky (see test_expectations.txt). - Automatically retries failing tests to avoid failing due to flakiness. - Outputs json for easier integration with other tools (like flakiness dashboard). Cons: - Does not run on all core platforms yet (like Windows). - results.html output is not as pretty as run-webkit-tests' output. - Missing useful flags (e.g. --guard, --leaks). - Can't run webarchive, mathml or wml tests yet. - Exposes more flaky tests due to running tests in a non-deterministic order. The test_expectations.txt file format is currently documented in the file itself: http://trac.webkit.org/browser/trunk/LayoutTests/platform/mac/test_expectations.txt I would invite you to try running new-run-webkit-tests on your Mac today. You should feel encouraged to file bugs and hunt me down at next week's contributer meeting. If you're interested in contributing to the code, it can all be found here: http://trac.webkit.org/browser/trunk/WebKitTools/Scripts/webkitpy/layout_tests/ Eventually we will rename run-webkit-tests to old-run-webkit-tests and new-run-webkit-tests to run-webkit-tests. Since new-run-webkit-tests is a complete re-write, some features of "run-webkit-tests" may go missing when we move. We will be moving them over to new-run-webkit-tests in priority order. Thanks for your time. -eric [1] new-run-webkit-tests currently defaults to a limited parallel-by-directory mode. Once we get new-run-webkit-tests on by default and weed out the existing flaky tests we'll turn on the fully parallel-by-file mode for a greater speed boost. _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
