There have been a few recent changes related to wpt that may be of
interest to a wider audience; for brevity I'm coalescing them into a
single email:
* New wpt dashboard focused on interop problems
* Support for fuzzy annotations in reftests
* Better support for debugging failing reftests on wpt.fyi
== Interop Dashboard ==
https://jgraham.github.io/wptdash/ (url may change) is a dashboard
intended to allow identifying web-platorm-test failures that present an
web-compat risk.
By default it shows tests that are failing in Gecko but passing in Both
Blink and WebKit, based on wpt.fyi runs of Firefox, Chrome and Safari.
Tests are divided by bug component.
In addition it presents a summary of the in-tree metadata in expectation
ini files to provide an overview of where tests are disabled in gecko,
and various test problems that aren't visible in wpt.fyi data (e.g.
leaks, or debug-only crashes).
There is upstream worked planned to add a system for annotating test
results from wpt.fyi; this will allow us to associate each gecko-only
failure with a bug and eventually to ensure that we triage all failures
that look like interop issues.
Please let me know about any changes that would make the dashboard more
useful to you.
== Fuzzy annotations available in wpt reftests ==
wpt has adopted a system to mark reftests as fuzzy. This is semantically
identical to the system used by gecko reftests; the fuzziness annotation
consists of two ranges; one for the number of pixels that may contain
differences and one for the maximum difference in any colour channel.
In the case where the test is known to (possibly) be an inexact match in
any configuration, the annotation may be put directly in the test file
as a <meta name=fuzzy> element e.g.
<meta name=fuzzy content="maxDifference=10-15;totalPixels=300">
(note that as with the recent changes to reftests, this requires exactly
300 pixels different i.e. the implied range is 300-300 not 0-300).
More usually, however the fuzziness will be browser and configuration
specific; in this case the annotation must be put in the wpt expectation
ini file (i.e. the one under testing/web-platform/meta). In this case
the basic syntax is:
[test.html]
fuzzy: maxDifference=8;totalPixels=0-1
The "if" syntax used for configuration-specific expectations also works
in this case.
In reftests involving multiple possible references or long chains of
references it may be necessary to specify exactly which comparison
requires the annotation or have multiple comparisons with different
annotations. This can be done as in the example below:
[test.html]
fuzzy: [ref1.html:maxDifference=8;totalPixels=0-1,
ref2.html==ref3.html:maxDifference=4;totalPixels=0-2]
The first annotation in the list applies to any comparison involving
ref1.html as the reference, the second only applies to the specific
comparison ref2.html == ref3.html (all paths are resolved relative to
the test).
More documentation is available at [1] and [2], the latter should
obviously move somewhere more useful.
== Reftest Comparisons ==
Debugging wpt reftest failures should now be easier since, thanks to
Chrome's ecosystem-infra team, wpt.fyi has gained reftest-analyzer like
functionality. See [3] for an example. wpt.fyi shows links to the
analyzer for all failing reftests [4]. Again, feedback about
improvements to make this more useful is very much encouraged.
[1]
https://web-platform-tests.org/writing-tests/reftests.html#fuzzy-matching
[2]
https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/wptrunner/wptrunner/manifestexpected.py#102
[3]
https://wpt.fyi/analyzer?screenshot=sha1%3A78ed15d1532e4134d8e3560c060538fd0b0a80d9&screenshot=sha1%3A5b8f16d25cb907619183b551c6b3e4d670991268
[4]
https://wpt.fyi/results/css/css-backgrounds/background-image-first-line.html?label=master&label=experimental&product=chrome&product=firefox&product=safari
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
