This is just an update to make sure everyone is aware of recent
improvements to the web-platform-tests, which should improve the UX for
gecko hackers and make them suitable for more testing situations.
== Summary ==
* Now possible to set per-test prefs so experimental features can be tested.
* It is possible to check in test changes directly to mozilla-inbound
and have them upstreamed when we next sync.
* The command line behaviour is more compatible with mochitest. In
particular individual tests can now be run using positional arguments
containing paths to the test files.
* The runner will now pause before closing the browser if only a single
test is run.
* js stack traces are reported when an assert fails a test, making
debugging easier.
* Reftest semantics have been aligned with the conventions of the CSS
testsuite.
== Introduction ==
web-platform-tests is a W3C-coordinated attempt to build a cross-browser
testsuite for the web-platform stack. Writing tests in a way that allows
them to be run in all browsers gives us confidence that we are shipping
software that is compatible with other implementations, and that later
implementations will be compatible with us. This in turn gives authors
confidence that they can actually rely on the web platform to deliver on
the promise of working across browsers and devices without needing extra
layers of abstraction to paper over the gaps we left.
It is possible to run the full web-platform-tests suite in gecko with
|mach web-platform-tests| and these tests are run on buildbot for
desktop opt builds. A subset, corresponding to the implemented features,
of the tests are also run on CI by Servo.
Given these advantages you should give serious consideration to writing
web-platform-tests in place of Mozilla-only tests for platform development.
== Recent Improvements ==
=== Setting Prefs ==
It's now possible to set per-test preferences. In order to do this, the
test must have a corresponding ini file (also used for expectation
data). For example a test in
/testing/web-platform/tests/example/example-test.html would have an ini
file in /testing/web-platform/meta/example/example-test.html.ini. To set
a pref use a key with name "prefs" with a value that's a list of the
form [pref.name:pref value]. For example if the above test needs the
dom.example pref set to true, and the dom.example.value pref set to 5 it
would have an ini file like:
[example-test.html]
type: testharness
prefs: [dom.example:true,
dom.example.value:5]
=== Upstreaming Tests ===
The script used to sync changes from upstream has grown the ability to
push changes made in the Mozilla tree upstream as part of the sync
process. Therefore it should be possible to work on web-platform-tests
in your Mozilla tree just like other test types, and have them
upstreamed once they reach inbound. There are a couple of things to be
aware of:
* If you add new tests or change metadata (particularly timeouts) of
existing tests the manifest file needs to be regenerated. You could do
this by hand, but it is recommended to use |mach web-platform-tests
--manifest-update| the first time after adding the tests.
* With great power comes great responsibility. If we start upstreaming
tests that depend on Mozilla-specific behaviour or are poorly written
people won't be enthused about continuing with the system treating r+ in
bugzilla as sufficient review for landing upstream. If possible test
that your tests at least behave sensibly in other browsers. I am always
happy to give feedback on tests for style and harness usage.
Typically syncs happen about once a week, although this process still
needs to be manually initiated.
=== Command Line Behaviour ===
|mach web-platform-tests| now works more like |mach mochitest| in its
command line behaviour. In particular to run the test in
testing/web-platform/tests/dom/historical.html you can now write
./mach testing/web-platform/tests/dom/historical.html
It is also possible to use the path of the test relative to the server i.e.
./mach dom/historical.html
which is shorter but won't have command line completion :)
=== Runner Pause ===
The test runner will now pause until the browser closes if only a single
test file is run. In this case in-page output will not be surpressed as
it is for normal runs. It's also possible to pause after any test that
gets an unexpected result with --pause-on-unexpected.
=== js stacks ===
testharness.js now provides nicely formatted js stacks whenever an
assert_*() causes a test to fail. Hopefully this helps debugging,
particularly in the face of tests that don't provide lots of annotations
of what's being asserted.
=== RefTest changes ===
web-plaform-tests reftests have now been aligned with CSSWG reftests in
terms of the way that references, particularly multiple references, are
specified. This means that wptrunner can run the CSS testsuite as well
(Servo plans to take advantage of this fact).
== Future Work ==
Depends on the pain points that people experience :) CI-grade
reliability with debug builds is a big ticket item, but if there are
areas where wpt needs to improve to offer a developer experience on par
with existing test suites, please file bugs.
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
