Re: Next Year in web-platform-tests - 2018/19 Edition

[Nils Ohlmeier]

> On 7Dec, 2018, at 07:45, James Graham  wrote:
> 
> I would also very much like to hear about remaining pain points and reasons 
> that developers choose to write browser-specific tests for web-platform 
> features. Those are issues we need to prioritize fixing.

Without going too much into the details, but in my opinion WebRTC is far, far 
away from being able to be tested sufficiently via web-platform tests. The list 
is long: browser interoperability, different network topologies affecting call 
establishment, network conditions affecting call quality, dependencies on 
external servers (depending on the network), not a common A/V test input 
format,… We do have improved in some of these areas for example in mochitests 
only.

I don’t think this is something the WebRTC ecosystem can solve short term. In 
the long run some of these could probably be tackled. But I’m raising this to 
clarify that WPT as of right now can not solved all existing problems.

Best
  Nils Ohlmeier
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Next year in web-platform-tests

[Ben Kelly]

I know you aware, but just mentioning for the list:

We need WPT coverage on Android to fully rely on it as our primary test
suite.  Particularly while Android's config deviates so far from desktop.

On Dec 15, 2017 10:40 AM, "James Graham"  wrote:

> Following the summary of what we achieved in wpt in the last year, I'd
> like to solicit input from the gecko community to inform the
> priorities of various pieces of wpt work for the next year.
>
> In order to maximise the compatibility benefit we have two long term
> objectives for the web-platform-tests work at Mozilla:
>
> * Make writing writing web-platform-tests the default for all
>   web-exposed features, so that we only write unshared tests in
>   exceptional cases (e.g. where there is a need to poke at Gecko
>   internals).
>
> * Ensure that gecko developers are using the web-platform-tests
>   results to avoid interoperability issues in the features they
>   develop.
>
> Obviously we are some way from achieving those goals. I have a large
> list of things that I think will make a difference, but obviously I
> have a different perspective to gecko developers, so getting some
> feedback on the priorities that you have would be good (I know I have
> already have conversations with several people, but it seems good to
> open up the question to a broader audience). In particular
> it would help to hear about things that you would consider blockers to
> removing tests from non-wpt suites that are duplicated in wpt
> (assuming exact duplication), and limitations either in the
> capabilities of wpt or in the workflow that lead to you writing other
> test types for cross-browser features.
>
> Thanks
>
> (Note: I set the reply-to for the email version of this message to be
> off-list as an experiment to see if that avoids the anchoring effect where
> early replies set the direction of all subsequent discussion. But I'm very
> happy to have an on-list conversation about anything that you feel merits a
> broader audience).
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Next year in web-platform-tests

[Emilio Cobos Álvarez]

On 12/16/2017 04:57 AM, L. David Baron wrote:
> On Friday 2017-12-15 16:00 -0600, Andrew McCreight wrote:
>> For me, a blocker to only having WPT is leak checking, both in terms of
>> XPCOM leak checking and LeakSanitizer. (The latter is probably going to
>> automatically work if you run them in ASan, but it would be good to check.)
>> I know there's a bug for XPCOM leak checking open already.
> 
> There are also some other pieces that our existing test harnesses
> (reftest and mochitest) do but the WPT harnesses don't, such as
> assertion checking (in a way that allows us to mark existing
> NS_ASSERTIONs as known failures, and get reports of unexpected
> changes in either direction, without losing the rest of the
> capability of the test).  These make me somewhat hesitant to port
> even existing reftests that can be ported to use WPT (see bug
> 1263568 for previous discussion of these issues), particularly in
> areas where assertions have been useful at catching problems.  For
> example, I'd be hesitant to add new reftests for css-multicol
> directly to a wpt directory rather than to a directory that is run
> by the layout/tools/reftest harness since assertion checking is
> particularly important for multicol tests.

Probably not all of them may be workable over the next year, but
assertions are the main thing blocking bug 1425167, so if that could be
sorted out it'd be great.

Otherwise, I guess I can just make a bunch of assertions fatal, which
may be the right thing to do anyway.

James, do you know if there's a bug on file to track these? Otherwise I
can file.

 -- Emilio

> There are also a bunch of other features of the reftest harness
> documented in layout/tools/reftest/README.txt that aren't part of
> WPT that have been important for testing various things related to
> painting, layers, async pan/zoom, scrolling, printing, etc., that
> are things that simply can't be done in WPT.
> 
> -David
> 
> 
> 
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
> 
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Next year in web-platform-tests

[L. David Baron]

On Friday 2017-12-15 16:00 -0600, Andrew McCreight wrote:
> For me, a blocker to only having WPT is leak checking, both in terms of
> XPCOM leak checking and LeakSanitizer. (The latter is probably going to
> automatically work if you run them in ASan, but it would be good to check.)
> I know there's a bug for XPCOM leak checking open already.

There are also some other pieces that our existing test harnesses
(reftest and mochitest) do but the WPT harnesses don't, such as
assertion checking (in a way that allows us to mark existing
NS_ASSERTIONs as known failures, and get reports of unexpected
changes in either direction, without losing the rest of the
capability of the test).  These make me somewhat hesitant to port
even existing reftests that can be ported to use WPT (see bug
1263568 for previous discussion of these issues), particularly in
areas where assertions have been useful at catching problems.  For
example, I'd be hesitant to add new reftests for css-multicol
directly to a wpt directory rather than to a directory that is run
by the layout/tools/reftest harness since assertion checking is
particularly important for multicol tests.

There are also a bunch of other features of the reftest harness
documented in layout/tools/reftest/README.txt that aren't part of
WPT that have been important for testing various things related to
painting, layers, async pan/zoom, scrolling, printing, etc., that
are things that simply can't be done in WPT.

-David

-- 
턞   L. David Baron http://dbaron.org/   턂
턢   Mozilla  https://www.mozilla.org/   턂
 Before I built a wall I'd ask to know
 What I was walling in or walling out,
 And to whom I was like to give offense.
   - Robert Frost, Mending Wall (1914)


signature.asc
Description: PGP signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Next year in web-platform-tests

[Kartikaya Gupta]

On Fri, Dec 15, 2017 at 4:59 PM, Andreas Tolfsen  wrote:
> I would be interested to hear more about what primitives for user
> input emulation are required.

Touch input emulation would also be good. There are some CSS
properties like touch-action which require emulating user touch input
for proper testing. APZ testing would also be made easier if we had a
good mechanism for emulating user input. We have some stuff we've
cobbled together over time that we're using now but we have to add to
it for almost every new test we write which increases the pain barrier
for writing tests. :(

Cheers,
kats
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Next year in web-platform-tests

[Andrew McCreight]

On Fri, Dec 15, 2017 at 9:38 AM, James Graham 
wrote:

> Following the summary of what we achieved in wpt in the last year, I'd
> like to solicit input from the gecko community to inform the
> priorities of various pieces of wpt work for the next year.
>
> In order to maximise the compatibility benefit we have two long term
> objectives for the web-platform-tests work at Mozilla:
>
> * Make writing writing web-platform-tests the default for all
>   web-exposed features, so that we only write unshared tests in
>   exceptional cases (e.g. where there is a need to poke at Gecko
>   internals).
>
> * Ensure that gecko developers are using the web-platform-tests
>   results to avoid interoperability issues in the features they
>   develop.
>
> Obviously we are some way from achieving those goals. I have a large
> list of things that I think will make a difference, but obviously I
> have a different perspective to gecko developers, so getting some
> feedback on the priorities that you have would be good (I know I have
> already have conversations with several people, but it seems good to
> open up the question to a broader audience). In particular
> it would help to hear about things that you would consider blockers to
> removing tests from non-wpt suites that are duplicated in wpt
> (assuming exact duplication), and limitations either in the
> capabilities of wpt or in the workflow that lead to you writing other
> test types for cross-browser features.
>

For me, a blocker to only having WPT is leak checking, both in terms of
XPCOM leak checking and LeakSanitizer. (The latter is probably going to
automatically work if you run them in ASan, but it would be good to check.)
I know there's a bug for XPCOM leak checking open already.


> Thanks
>
> (Note: I set the reply-to for the email version of this message to be
> off-list as an experiment to see if that avoids the anchoring effect where
> early replies set the direction of all subsequent discussion. But I'm very
> happy to have an on-list conversation about anything that you feel merits a
> broader audience).
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Next year in web-platform-tests

[Andreas Tolfsen]

Also sprach smaug:

> On 12/15/2017 05:38 PM, James Graham wrote:
>> In particular it would help to hear about things that you would
>> consider blockers to removing tests from non-wpt suites that are
>> duplicated in wpt (assuming exact duplication), and limitations
>> either in the capabilities of wpt
> 
> Not being able to send (DOM) events which mimic user input prevents
> converting many event handling tests to wpt.  Not sure if WebDriver
> could easily do some of this, or should browsers have some testing
> mode which exposes APIs for this kinds of cases.

This is what testdriver [1] is supposed to help with.  It only
exposes a single method for issuing trusted click events [2] at the
moment, but can be extended in vendor-specific ways to do more
things.

One could imagine hooking the click command up to the WebDriver
session that already controls the browser-under-test, or by exposing
a test-only web API that calls down to nsIWindowUtils’ synthetic
click primitive (such as Marionette).

I would be interested to hear more about what primitives for user
input emulation are required.

  [1] 
https://searchfox.org/mozilla-central/source/testing/web-platform/tests/docs/_writing-tests/testdriver.md
  [2] 
https://searchfox.org/mozilla-central/rev/c8e791091973825680bbba807fc1c4f5bda0f5a1/testing/web-platform/tests/resources/testdriver.js#49-86

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Next year in web-platform-tests

[David Burns]

On 15 December 2017 at 21:29, smaug  wrote:

>
> Not being able to send (DOM) events which mimic user input prevents
> converting
> many event handling tests to wpt.
> Not sure if WebDriver could easily do some of this, or should browsers
> have some testing mode which exposes
> APIs for this kinds of cases.
>

The TestDriver work does a lot of this and is currently being worked on and
hopefully will solve this use case. It uses webdriver under the hood to do
the user mimicing.

David



>
>
> -Olli
>
>
>
> or in the workflow that lead to you writing other
>> test types for cross-browser features.
>>
>> Thanks
>>
>> (Note: I set the reply-to for the email version of this message to be
>> off-list as an experiment to see if that avoids the anchoring effect where
>> early
>> replies set the direction of all subsequent discussion. But I'm very
>> happy to have an on-list conversation about anything that you feel merits a
>> broader audience).
>>
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Next year in web-platform-tests

[smaug]

Great work on wpt!

On 12/15/2017 05:38 PM, James Graham wrote:

Following the summary of what we achieved in wpt in the last year, I'd
like to solicit input from the gecko community to inform the
priorities of various pieces of wpt work for the next year.

In order to maximise the compatibility benefit we have two long term
objectives for the web-platform-tests work at Mozilla:

* Make writing writing web-platform-tests the default for all
  web-exposed features, so that we only write unshared tests in
  exceptional cases (e.g. where there is a need to poke at Gecko
  internals).

* Ensure that gecko developers are using the web-platform-tests
  results to avoid interoperability issues in the features they
  develop.

Obviously we are some way from achieving those goals. I have a large
list of things that I think will make a difference, but obviously I
have a different perspective to gecko developers, so getting some
feedback on the priorities that you have would be good (I know I have
already have conversations with several people, but it seems good to
open up the question to a broader audience). In particular
it would help to hear about things that you would consider blockers to
removing tests from non-wpt suites that are duplicated in wpt
(assuming exact duplication), and limitations either in the
capabilities of wpt


Not being able to send (DOM) events which mimic user input prevents converting
many event handling tests to wpt.
Not sure if WebDriver could easily do some of this, or should browsers have 
some testing mode which exposes
APIs for this kinds of cases.


-Olli



or in the workflow that lead to you writing other
test types for cross-browser features.

Thanks

(Note: I set the reply-to for the email version of this message to be off-list 
as an experiment to see if that avoids the anchoring effect where early
replies set the direction of all subsequent discussion. But I'm very happy to 
have an on-list conversation about anything that you feel merits a
broader audience).


___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform