Re: Next year in web-platform-tests
On 12/16/2017 04:57 AM, L. David Baron wrote: > On Friday 2017-12-15 16:00 -0600, Andrew McCreight wrote: >> For me, a blocker to only having WPT is leak checking, both in terms of >> XPCOM leak checking and LeakSanitizer. (The latter is probably going to >> automatically work if you run them in ASan, but it would be good to check.) >> I know there's a bug for XPCOM leak checking open already. > > There are also some other pieces that our existing test harnesses > (reftest and mochitest) do but the WPT harnesses don't, such as > assertion checking (in a way that allows us to mark existing > NS_ASSERTIONs as known failures, and get reports of unexpected > changes in either direction, without losing the rest of the > capability of the test). These make me somewhat hesitant to port > even existing reftests that can be ported to use WPT (see bug > 1263568 for previous discussion of these issues), particularly in > areas where assertions have been useful at catching problems. For > example, I'd be hesitant to add new reftests for css-multicol > directly to a wpt directory rather than to a directory that is run > by the layout/tools/reftest harness since assertion checking is > particularly important for multicol tests. Probably not all of them may be workable over the next year, but assertions are the main thing blocking bug 1425167, so if that could be sorted out it'd be great. Otherwise, I guess I can just make a bunch of assertions fatal, which may be the right thing to do anyway. James, do you know if there's a bug on file to track these? Otherwise I can file. -- Emilio > There are also a bunch of other features of the reftest harness > documented in layout/tools/reftest/README.txt that aren't part of > WPT that have been important for testing various things related to > painting, layers, async pan/zoom, scrolling, printing, etc., that > are things that simply can't be done in WPT. > > -David > > > > ___ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Next year in web-platform-tests
On Friday 2017-12-15 16:00 -0600, Andrew McCreight wrote: > For me, a blocker to only having WPT is leak checking, both in terms of > XPCOM leak checking and LeakSanitizer. (The latter is probably going to > automatically work if you run them in ASan, but it would be good to check.) > I know there's a bug for XPCOM leak checking open already. There are also some other pieces that our existing test harnesses (reftest and mochitest) do but the WPT harnesses don't, such as assertion checking (in a way that allows us to mark existing NS_ASSERTIONs as known failures, and get reports of unexpected changes in either direction, without losing the rest of the capability of the test). These make me somewhat hesitant to port even existing reftests that can be ported to use WPT (see bug 1263568 for previous discussion of these issues), particularly in areas where assertions have been useful at catching problems. For example, I'd be hesitant to add new reftests for css-multicol directly to a wpt directory rather than to a directory that is run by the layout/tools/reftest harness since assertion checking is particularly important for multicol tests. There are also a bunch of other features of the reftest harness documented in layout/tools/reftest/README.txt that aren't part of WPT that have been important for testing various things related to painting, layers, async pan/zoom, scrolling, printing, etc., that are things that simply can't be done in WPT. -David -- 𝄞 L. David Baron http://dbaron.org/ 𝄂 𝄢 Mozilla https://www.mozilla.org/ 𝄂 Before I built a wall I'd ask to know What I was walling in or walling out, And to whom I was like to give offense. - Robert Frost, Mending Wall (1914) signature.asc Description: PGP signature ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Next year in web-platform-tests
On Fri, Dec 15, 2017 at 4:59 PM, Andreas Tolfsen wrote: > I would be interested to hear more about what primitives for user > input emulation are required. Touch input emulation would also be good. There are some CSS properties like touch-action which require emulating user touch input for proper testing. APZ testing would also be made easier if we had a good mechanism for emulating user input. We have some stuff we've cobbled together over time that we're using now but we have to add to it for almost every new test we write which increases the pain barrier for writing tests. :( Cheers, kats ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Next year in web-platform-tests
On Fri, Dec 15, 2017 at 9:38 AM, James Graham wrote: > Following the summary of what we achieved in wpt in the last year, I'd > like to solicit input from the gecko community to inform the > priorities of various pieces of wpt work for the next year. > > In order to maximise the compatibility benefit we have two long term > objectives for the web-platform-tests work at Mozilla: > > * Make writing writing web-platform-tests the default for all > web-exposed features, so that we only write unshared tests in > exceptional cases (e.g. where there is a need to poke at Gecko > internals). > > * Ensure that gecko developers are using the web-platform-tests > results to avoid interoperability issues in the features they > develop. > > Obviously we are some way from achieving those goals. I have a large > list of things that I think will make a difference, but obviously I > have a different perspective to gecko developers, so getting some > feedback on the priorities that you have would be good (I know I have > already have conversations with several people, but it seems good to > open up the question to a broader audience). In particular > it would help to hear about things that you would consider blockers to > removing tests from non-wpt suites that are duplicated in wpt > (assuming exact duplication), and limitations either in the > capabilities of wpt or in the workflow that lead to you writing other > test types for cross-browser features. > For me, a blocker to only having WPT is leak checking, both in terms of XPCOM leak checking and LeakSanitizer. (The latter is probably going to automatically work if you run them in ASan, but it would be good to check.) I know there's a bug for XPCOM leak checking open already. > Thanks > > (Note: I set the reply-to for the email version of this message to be > off-list as an experiment to see if that avoids the anchoring effect where > early replies set the direction of all subsequent discussion. But I'm very > happy to have an on-list conversation about anything that you feel merits a > broader audience). > ___ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Next year in web-platform-tests
Also sprach smaug: > On 12/15/2017 05:38 PM, James Graham wrote: >> In particular it would help to hear about things that you would >> consider blockers to removing tests from non-wpt suites that are >> duplicated in wpt (assuming exact duplication), and limitations >> either in the capabilities of wpt > > Not being able to send (DOM) events which mimic user input prevents > converting many event handling tests to wpt. Not sure if WebDriver > could easily do some of this, or should browsers have some testing > mode which exposes APIs for this kinds of cases. This is what testdriver [1] is supposed to help with. It only exposes a single method for issuing trusted click events [2] at the moment, but can be extended in vendor-specific ways to do more things. One could imagine hooking the click command up to the WebDriver session that already controls the browser-under-test, or by exposing a test-only web API that calls down to nsIWindowUtils’ synthetic click primitive (such as Marionette). I would be interested to hear more about what primitives for user input emulation are required. [1] https://searchfox.org/mozilla-central/source/testing/web-platform/tests/docs/_writing-tests/testdriver.md [2] https://searchfox.org/mozilla-central/rev/c8e791091973825680bbba807fc1c4f5bda0f5a1/testing/web-platform/tests/resources/testdriver.js#49-86 ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Next year in web-platform-tests
On 15 December 2017 at 21:29, smaug wrote: > > Not being able to send (DOM) events which mimic user input prevents > converting > many event handling tests to wpt. > Not sure if WebDriver could easily do some of this, or should browsers > have some testing mode which exposes > APIs for this kinds of cases. > The TestDriver work does a lot of this and is currently being worked on and hopefully will solve this use case. It uses webdriver under the hood to do the user mimicing. David > > > -Olli > > > > or in the workflow that lead to you writing other >> test types for cross-browser features. >> >> Thanks >> >> (Note: I set the reply-to for the email version of this message to be >> off-list as an experiment to see if that avoids the anchoring effect where >> early >> replies set the direction of all subsequent discussion. But I'm very >> happy to have an on-list conversation about anything that you feel merits a >> broader audience). >> > > ___ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Next year in web-platform-tests
Great work on wpt! On 12/15/2017 05:38 PM, James Graham wrote: Following the summary of what we achieved in wpt in the last year, I'd like to solicit input from the gecko community to inform the priorities of various pieces of wpt work for the next year. In order to maximise the compatibility benefit we have two long term objectives for the web-platform-tests work at Mozilla: * Make writing writing web-platform-tests the default for all web-exposed features, so that we only write unshared tests in exceptional cases (e.g. where there is a need to poke at Gecko internals). * Ensure that gecko developers are using the web-platform-tests results to avoid interoperability issues in the features they develop. Obviously we are some way from achieving those goals. I have a large list of things that I think will make a difference, but obviously I have a different perspective to gecko developers, so getting some feedback on the priorities that you have would be good (I know I have already have conversations with several people, but it seems good to open up the question to a broader audience). In particular it would help to hear about things that you would consider blockers to removing tests from non-wpt suites that are duplicated in wpt (assuming exact duplication), and limitations either in the capabilities of wpt Not being able to send (DOM) events which mimic user input prevents converting many event handling tests to wpt. Not sure if WebDriver could easily do some of this, or should browsers have some testing mode which exposes APIs for this kinds of cases. -Olli or in the workflow that lead to you writing other test types for cross-browser features. Thanks (Note: I set the reply-to for the email version of this message to be off-list as an experiment to see if that avoids the anchoring effect where early replies set the direction of all subsequent discussion. But I'm very happy to have an on-list conversation about anything that you feel merits a broader audience). ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Intent to implement: individaul transform
Behind a preference(about:flags - enable individual transforms) On Sat, Dec 16, 2017 at 12:10 AM, Emilio Cobos Álvarez wrote: > > > On 12/15/2017 08:51 AM, Ku(顧思捷)CJ wrote: > > Summary: > > The translate, rotate, and scale properties allow authors to specify > > simple transforms independently, in a way that maps to typical user > > interface usage, rather than having to remember the order in transform > that > > keeps the actions of transform(), rotate() and scale() independent and > > acting in screen coordinates. > > Both Blink and Edge have implemented this feature. > > > > Bug: > > https://bugzilla.mozilla.org/show_bug.cgi?id=1207734 > > > > Link to standard: > > https://drafts.csswg.org/css-transforms-2/#individual-transforms > > > > Platform coverage: > > All platforms > > > > Target release: > > FF60 > > > > Preference behind which this will be implemented: > > "layout.css.individual-transform.enabled" > > > > Do other browser engines implement this? > > Blink/ Edge > > Just curious, has Edge shipped this? Blink still hasn't. Not that it > makes much of a difference, but I guess it's worth noting. > > > Tests: > > WPT test > > ___ > > dev-platform mailing list > > dev-platform@lists.mozilla.org > > https://lists.mozilla.org/listinfo/dev-platform > > > ___ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Intent to implement: individaul transform
On 12/15/2017 08:51 AM, Ku(顧思捷)CJ wrote: > Summary: > The translate, rotate, and scale properties allow authors to specify > simple transforms independently, in a way that maps to typical user > interface usage, rather than having to remember the order in transform that > keeps the actions of transform(), rotate() and scale() independent and > acting in screen coordinates. > Both Blink and Edge have implemented this feature. > > Bug: > https://bugzilla.mozilla.org/show_bug.cgi?id=1207734 > > Link to standard: > https://drafts.csswg.org/css-transforms-2/#individual-transforms > > Platform coverage: > All platforms > > Target release: > FF60 > > Preference behind which this will be implemented: > "layout.css.individual-transform.enabled" > > Do other browser engines implement this? > Blink/ Edge Just curious, has Edge shipped this? Blink still hasn't. Not that it makes much of a difference, but I guess it's worth noting. > Tests: > WPT test > ___ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Intent to implement: individaul transform
There have been a series of attacks[0] that allow SOP bypasses by applying non-constant-time transforms to cross-domain resources and using timing attacks to infer the contents. I'm not sure to what extent we have been tracking our exposure to these attacks over the years, but it's something I'm hoping to start understanding. Do we know how these transforms behave in this regard? -tom [0] This is a really incomplete list of these but it's a start: https://bugzilla.mozilla.org/show_bug.cgi?id=655987 https://dl.acm.org/citation.cfm?id=2516712&dl=ACM&coll=DL&CFID=1016908573&CFTOKEN=45471182 https://www.contextis.com/media/downloads/Pixel_Perfect_Timing_Attacks_with_HTML5_Whitepaper.pdf On Fri, Dec 15, 2017 at 1:51 AM, Ku(顧思捷)CJ wrote: > Summary: > The translate, rotate, and scale properties allow authors to specify > simple transforms independently, in a way that maps to typical user > interface usage, rather than having to remember the order in transform that > keeps the actions of transform(), rotate() and scale() independent and > acting in screen coordinates. > Both Blink and Edge have implemented this feature. > > Bug: > https://bugzilla.mozilla.org/show_bug.cgi?id=1207734 > > Link to standard: > https://drafts.csswg.org/css-transforms-2/#individual-transforms > > Platform coverage: > All platforms > > Target release: > FF60 > > Preference behind which this will be implemented: > "layout.css.individual-transform.enabled" > > Do other browser engines implement this? > Blink/ Edge > > Tests: > WPT test > ___ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Next year in web-platform-tests
Following the summary of what we achieved in wpt in the last year, I'd like to solicit input from the gecko community to inform the priorities of various pieces of wpt work for the next year. In order to maximise the compatibility benefit we have two long term objectives for the web-platform-tests work at Mozilla: * Make writing writing web-platform-tests the default for all web-exposed features, so that we only write unshared tests in exceptional cases (e.g. where there is a need to poke at Gecko internals). * Ensure that gecko developers are using the web-platform-tests results to avoid interoperability issues in the features they develop. Obviously we are some way from achieving those goals. I have a large list of things that I think will make a difference, but obviously I have a different perspective to gecko developers, so getting some feedback on the priorities that you have would be good (I know I have already have conversations with several people, but it seems good to open up the question to a broader audience). In particular it would help to hear about things that you would consider blockers to removing tests from non-wpt suites that are duplicated in wpt (assuming exact duplication), and limitations either in the capabilities of wpt or in the workflow that lead to you writing other test types for cross-browser features. Thanks (Note: I set the reply-to for the email version of this message to be off-list as an experiment to see if that avoids the anchoring effect where early replies set the direction of all subsequent discussion. But I'm very happy to have an on-list conversation about anything that you feel merits a broader audience). ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
This year in web-platform-tests
I thought that people might be interested in a retrospective on the progress with web-platform-tests in the last year. 2017 has seen big advance in the adoption of web-platform-tests by the wider browser community, and has brought considerable improvements to the associated tooling and infrastructure. So, in no particular order: == Sync == * About 600 commits from mozilla-central were upstreamed to GitHub. * Google deployed a rapid 2-way sync process between web-platform-tests and the Blink repository. As a result, contributions from Chromium engineers to wpt more than tripled, with over 800 CLs upstreamed. * WebKit and Edge made promising progress on running more tests and making contribution easier. * We developed an improved two-way sync for Firefox which is expected to go live in the new year. This is expected to dramatically improve latency of our synchronisations, and, by tracking upstream changes in bugzilla, improve the visibility of wpt changes to gecko developers. == Test Coverage == * A policy of requiring test changes with normative spec changes was widely adopted by W3C and WHATWG specs. Over half of all browser-relevant specifications now have some variant of this policy. * The CSS test suite was merged into web-platform-tests, bringing almost 30,000 layout tests into the suite. To support running this, we overhauled the wpt reftest runner to significantly improve the performance. * Initial work on "testdriver" landed. This will provide a wpt frontend to test features that aren't available to web content. The initial implementation allows providing user clicks via WebDriver, but there are plans to significantly expand the capabilities in the future. * Significantly improved the support for testing the WebDriver spec. == Infrastructure == * https://wpt.fyi launched. This displays the results of the tests in Firefox, Chrome, Edge and Safari from daily runs of the full suite. There are plans to improve this in the future to make it clearer which test failures are identifying interoperability problems. * wpt-verify job enabled as Tier 2 on treeherder. This is the wpt equivalent of the test-verify job for locating tests that are intermittent as they are initially committed. * Added a CLI in upstream web-platform-tests which provides support for running tests in multiple browsers using a `wpt run ` command. This also supports running on Sauce Labs if you need to get a result for a browser that isn't available locally. * Improved the usability of upstream PRs by consolidating all the relevant information into a single dashboard rather than spreading it over multiple GitHub comments. * Improved the upstream CI testing to be faster and cover more browsers. Further improvements to increase reliability and improve the visibility of test results are planned for the near future. Thanks to everyone, both at Mozilla, and in the wider wpt community, who contributed to these changes. I think we've really made a lot of progress on making cross-browser testing in integral part of the process of browser engineering. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Intent to implement: individaul transform
(cc'ing the dev-developer-tools list, please keep responses to dev-platform) Very happy to see this being implemented. I would like to point out that this does have some DevTools impact that we might want to align with the target release for this. There's a (sort of hidden) feature in DevTools where, if you position your cursor over a `transform` property name inside the CSS Rules panel of the inspector, then an overlay appears in the page. This overlay shows you 2 things: where the element selected in the inspector currently is on the page, and where it would have been if that `transform` property had not been there. It gives a sense of how CSS transform works. This MDN page describes that feature: https://developer.mozilla.org/en-US/docs/Tools/Page_Inspector/How_to/Visualize_transforms If we also want to support the new properties being introduced here, that will require some work in DevTools, and I'm not sure how the feature would even work. I think it would be a good opportunity for us to revise the UX of the feature and possibly come up with an even better transform tool. Note that none of this is a blocker, people using `transform` will still benefit from this feature. The new properties will just act as normal properties in the Rules panel, and hovering over them will just do nothing. On Fri, Dec 15, 2017 at 8:51 AM, Ku(顧思捷)CJ wrote: > Summary: > The translate, rotate, and scale properties allow authors to specify > simple transforms independently, in a way that maps to typical user > interface usage, rather than having to remember the order in transform that > keeps the actions of transform(), rotate() and scale() independent and > acting in screen coordinates. > Both Blink and Edge have implemented this feature. > > Bug: > https://bugzilla.mozilla.org/show_bug.cgi?id=1207734 > > Link to standard: > https://drafts.csswg.org/css-transforms-2/#individual-transforms > > Platform coverage: > All platforms > > Target release: > FF60 > > Preference behind which this will be implemented: > "layout.css.individual-transform.enabled" > > Do other browser engines implement this? > Blink/ Edge > > Tests: > WPT test > ___ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
