A simple in-browser test runner is now available in the
web-platform-tests repository. This will automatically run
testharness.js tests and provides UI for manually marking the results of
reftests and manual tests. This runner is designed to be helpful when
developing tests and implementations, allowing all tests, or a subset of
tests, to be run with minimal labour, and to be helpful in comparing
various implementations.
To facilitate the latter use case, the runner can produce JSON output
from the test run, and there is a simple command line script for taking
this output from one or more browsers and producing an implementation
report (this script could be ported to run entirely in-browser).
I suggest that henceforth we stop compiling implementation reports by
hand on wiki pages and instead ask implementors to provide the JSON
output from their implementation. This could be from the runner in the
repository or from implementor-specific test harnesses used in
automation. In the longer term we should look to get a vendor-controlled
URL at which the latest test result data for each implementation is
published. Given this we will be able to automatically collate a test
report from the latest available data.
In order to use the in-repository runner it is necessary to:
a) Set up the latest web-platform-tests checkout, following the
instructions in README.md to install all the relevant submodules and set
up the hosts.
b) Generate a test manifest using
python tools/scripts/manifest.py MANIFEST.json
from the web-platform-tests root (this step must be repeated whenever
new tests are added).
c) Start the local server using
python serve.py
b) Navigate to
http://web-platform.test:8000/tools/runner/index.html
This is all documented in the README.md file in the web-platform-tests
repository.