Great stuff! Thanks to you and everyone involved for pushing the state of
cross-browser testing forward.

On Fri, Dec 7, 2018 at 7:45 AM James Graham <> wrote:

> Welcome to the second annual update on progress in cross-browser interop
> testing through web-platform-tests. This year has seen big improvements
> to the platform coverage of wpt, as well as increasing the number of
> features that can be tested with wpt, and the visibility of test results
> in Gecko and in other browsers. In addition there have been numerous
> fixes intended to make the experience of working with wpt better and the
> results more reliable. So in no particular order:
> == Gecko CI ==
> * web-platform-tests are now enabled on CI on all tier-1 platforms i.e.
> Desktop Firefox on Linux, Windows and macOS, and mobile Firefox on
> Android (using the x86 emulator)
> * web-platform-tests reftests run the full set of CSS tests on all
> platforms.
> * web-platform-tests run with leak checking enabled in debug, and also
> run under LSAN. When tests are imported, any existing leaks are
> automatically added to the allowed list for those tests.
> * wpt wasm tests are running in the jsshell.
> == Testability ==
> * Substantially increased the scope of the testdriver.js [1] API to
> allow writing tests that are not possible using standard DOM APIs. In
> particular the following things are now possible:
>    - Sending trusted key and mouse events
>    - Sending complex series of trusted pointer and key interactions for
> things like in-content drag and drop or pinch zoom
>    - File upload
>    Also added the 'bless' API for cases where something trusted is
> required but the details are unimportant.
> * Implemented support for running tests on HTTP/2 (this is currently
> disabled by default due to problems with upstream CI that will be solved
> by migrating from Travis).
> * Added support for fuzzy matching in reftests, to allow reftests to
> pass in the face of small unavoidable differences between test and ref
> (this is not yet merged but is expected to be landed by the end of the
> year).
> * Added multiple top-level domains to wpt to allow cross-site tests.
> Changed wpt runner to run tests in a true top level browsing context
> (i.e. with null `opener`).
> == Results Collection and Viewability ==
> * Many fixes and improvements to the interface to allow
> efficient selection and comparison of multiple test runs.
> * Set up Taskcluster to run all tests after each upstream commit on
> Firefox nightly and Chrome dev on Linux.
> * Set up daily runs of stable and weekly runs of beta, also using
> Taskcluster.
> * Regular, reliable, Edge and Safari Stable/TP runs using a custom
> buildbot setup and Sauce Labs for Edge.
> == Sync ==
> * We deployed a new, faster, sync for mozilla-central that enables us to
> sync test changes made in Gecko repositories as soon as they land in our
> repositories, creates Bugzilla bugs to track upstream PRs and
> corresponding changes to Firefox test results, and allows us to
> downstream changes from web-platform-tests much more frequently.
> * Servo's sync system was updated to allow continuous syncing of PRs
> from servo/servo to web-platform-tests and frequent downstreaming.
> * Since launching the new sync we have upstreamed over 500 changes from
> Gecko bugs to the web-platform-tests repository.
> * Over the same time period we have seen about 1,500 changes synced from
> the Chromium repository to upstream web-platform-tests.
> == Developer Ergonomics ==
> * Stopped using the in-tree MANIFEST.json file and instead downloaded it
> on demand. Apart from avoiding merge conflicts, this helps avoid
> breaking many pieces of tooling including Phabricator and git-cinnabar.
> * Added reftest-analyzer compatible output to the default logger used by
> mach wpt.
> * Improved multi-global tests by extending .any.js tests to be more
> flexible and support more kinds of global scope.
> == GitHub Infrastructure ==
> * Moved test-stability checks from Travis to Taskcluster for performance
> and reliability
> * Started running PR checks in Safari using Azure Pipelines.
> * Started checking for tests that are close to the timeout value and
> likely to become intermittent.
> * Continued making more infrastructure Python 3 compatible.
> * Running affected tests on Chrome, Firefox and Safari and showing
> regressions as PR statuses (currently in beta)
> == Meta ==
> * Moved the web-platform-tests GitHub repository to its own organisation.
> * Started the process of creating a more formal governance structure for
> web-platform-tests, forming a provisional core team with membership from
> a range of stakeholders.
> A couple of the mentioned items are still under review, but will almost
> certainly merge before the end of the year :) These improvements have
> been the result of a close collaboration between people across the
> web-platform-tests community, including the Interop Testing team at
> Mozilla, the Ecosystem Infra team at Google and many, many, others. Huge
> thanks to everyone involved.
> [1]
> _______________________________________________
> dev-platform mailing list
dev-platform mailing list

Reply via email to