Great stuff! Thanks to you and everyone involved for pushing the state of cross-browser testing forward.
On Fri, Dec 7, 2018 at 7:45 AM James Graham <ja...@hoppipolla.co.uk> wrote: > Welcome to the second annual update on progress in cross-browser interop > testing through web-platform-tests. This year has seen big improvements > to the platform coverage of wpt, as well as increasing the number of > features that can be tested with wpt, and the visibility of test results > in Gecko and in other browsers. In addition there have been numerous > fixes intended to make the experience of working with wpt better and the > results more reliable. So in no particular order: > > == Gecko CI == > > * web-platform-tests are now enabled on CI on all tier-1 platforms i.e. > Desktop Firefox on Linux, Windows and macOS, and mobile Firefox on > Android (using the x86 emulator) > > * web-platform-tests reftests run the full set of CSS tests on all > platforms. > > * web-platform-tests run with leak checking enabled in debug, and also > run under LSAN. When tests are imported, any existing leaks are > automatically added to the allowed list for those tests. > > * wpt wasm tests are running in the jsshell. > > == Testability == > * Substantially increased the scope of the testdriver.js [1] API to > allow writing tests that are not possible using standard DOM APIs. In > particular the following things are now possible: > > - Sending trusted key and mouse events > - Sending complex series of trusted pointer and key interactions for > things like in-content drag and drop or pinch zoom > - File upload > > Also added the 'bless' API for cases where something trusted is > required but the details are unimportant. > > * Implemented support for running tests on HTTP/2 (this is currently > disabled by default due to problems with upstream CI that will be solved > by migrating from Travis). > > * Added support for fuzzy matching in reftests, to allow reftests to > pass in the face of small unavoidable differences between test and ref > (this is not yet merged but is expected to be landed by the end of the > year). > > * Added multiple top-level domains to wpt to allow cross-site tests. > Changed wpt runner to run tests in a true top level browsing context > (i.e. with null `opener`). > > == Results Collection and Viewability == > > * Many fixes and improvements to the wpt.fyi interface to allow > efficient selection and comparison of multiple test runs. > > * Set up Taskcluster to run all tests after each upstream commit on > Firefox nightly and Chrome dev on Linux. > > * Set up daily runs of stable and weekly runs of beta, also using > Taskcluster. > > * Regular, reliable, Edge and Safari Stable/TP runs using a custom > buildbot setup and Sauce Labs for Edge. > > == Sync == > > * We deployed a new, faster, sync for mozilla-central that enables us to > sync test changes made in Gecko repositories as soon as they land in our > repositories, creates Bugzilla bugs to track upstream PRs and > corresponding changes to Firefox test results, and allows us to > downstream changes from web-platform-tests much more frequently. > > * Servo's sync system was updated to allow continuous syncing of PRs > from servo/servo to web-platform-tests and frequent downstreaming. > > * Since launching the new sync we have upstreamed over 500 changes from > Gecko bugs to the web-platform-tests repository. > > * Over the same time period we have seen about 1,500 changes synced from > the Chromium repository to upstream web-platform-tests. > > == Developer Ergonomics == > > * Stopped using the in-tree MANIFEST.json file and instead downloaded it > on demand. Apart from avoiding merge conflicts, this helps avoid > breaking many pieces of tooling including Phabricator and git-cinnabar. > > * Added reftest-analyzer compatible output to the default logger used by > mach wpt. > > * Improved multi-global tests by extending .any.js tests to be more > flexible and support more kinds of global scope. > > == GitHub Infrastructure == > > * Moved test-stability checks from Travis to Taskcluster for performance > and reliability > > * Started running PR checks in Safari using Azure Pipelines. > > * Started checking for tests that are close to the timeout value and > likely to become intermittent. > > * Continued making more infrastructure Python 3 compatible. > > * Running affected tests on Chrome, Firefox and Safari and showing > regressions as PR statuses (currently in beta) > > == Meta == > > * Moved the web-platform-tests GitHub repository to its own organisation. > > * Started the process of creating a more formal governance structure for > web-platform-tests, forming a provisional core team with membership from > a range of stakeholders. > > A couple of the mentioned items are still under review, but will almost > certainly merge before the end of the year :) These improvements have > been the result of a close collaboration between people across the > web-platform-tests community, including the Interop Testing team at > Mozilla, the Ecosystem Infra team at Google and many, many, others. Huge > thanks to everyone involved. > > [1] https://web-platform-tests.org/writing-tests/testdriver.html > _______________________________________________ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
