[webkit-dev] Upstreaming Tests from WebKit to Web Platform Tests

[Ryosuke Niwa]

Hi all,

*Background*

Web Platform Tests is a suite of tests written for W3C specifications
=,
and Blink, Edge, Gecko, and WebKit all run them as a part of their
continuous build system.

Historically, each working group had their own repository for tests,
but with CSS WG's tests  finally getting
merged into the Web Platform Tests,
we will have a single repository with all the tests from W3C.

A few of us attended a meeting to discuss the future of this test suite

last Monday.
One of the major topic was that Blink and Gecko have adopted the process to
write the tests
using testharness.js in their own test suites, and automatically merge into
Web Platform Tests
without having to go through another round of code review for Web Platform
Tests.

Given we do test-driven development in WebKit, I think WebKit should do the
same.
This will benefit other browser vendors to catch their bugs with our tests,
and we will also benefit
from other browser vendors "reviewing" our tests against their
implementation and specifications.

In the long term, I believe this will drastically improve the
interoperability of the Web,
and dramatically improve the quality of the tests we run against WebKit.

In fact, there was a general consensus that we should even upstream
pass-if-not-crash tests
as well as style and layout invalidation tests to web-platform-tests as
they tend to be undertested
right now and almost all engines have bugs in this area.

*Problems & Questions*

In order to auto-merge tests from WebKit to Web Platform Tests, there are a
few problems.

*1. We need to start writing more if not all tests using testharness.js
instead of our own js-test(-pre).js*

I've heard of many complaints about testharness.js being too verbose and
cumbersome to use.
I agree with all those complaints but I don't think we can convince all
other browser vendors
to use our own testing script either since both Blink & Gecko are onboard
with using testharness.js

At this point, I'd argue that the benefit outweighs the cost of adopting
testharness.js.
Also, W3C have dropped many styling requirements for tests so we no longer
need to have
link/meta elements, etc... You simply include testharness.js and
testharness-report.js.

See my recent PR

for
example.

Do people still object to using testharness.js? If so, what are your
reasons?


*2. Where should we put upstremable tests?*

We need a mechanism to identify & merge tests back from WebKit into Web
Platform Tests.

One way to do this is to start adding tests directly into
LayoutTests/imported/w3c/web-platform-tests
and then automatically identify those tests and upstream them.

Assuming all upstream merges happen in timely happen, this should work fine.
And it has a benefit that you get to see all related tests in one place.

Another approach is to create another top-level directory like
LayoutTests/exports/web-platform-tests.
One downside of this approach is that we need to match the directory
structure
in order for any script to upstream tests to appropriate locations.

Do people prefer one approach over another? Or have another idea?


*3. What would be the process for upstreaming tests?*

One possible approach is to create a PR request for every patch posted on
Bugzilla
with tests that can be merged into Web Platform Tests.

Because we don't want to make Web Platform Tests less reliably,
every PR request to Web Platform Tests go through Travis CI

which runs the test hundreds of times to make sure it's not flaky.

This Travis CI job (stability check) will then show up as an additional EWS
bubble.


Another approach is to wait until a patch is landed, and automatically
create a PR
for tests that can be upstream'ed. This has a downside that the stability
check
wouldn't run until the patch is landed, and it's unclear what happens they
do fail.

Do committers get emails about them? Do the patches then need to be rolled
back?
If we don't come up with an appropriate process, we can end up with a lot of
broken PRs that never get fixed like those failing test expectations :(


Again, do you prefer one or the other? Or do you have another proposal?

- R. Niwa
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Upstreaming Tests from WebKit to Web Platform Tests

[Chris Dumez]

I also think we should do 2-way sync-ing of WPT tests, similarly to other 
browser engines. These tests are extremely useful for interoperability. Every 
major browser engine is involved.

Google also has a nice dashboard showing the pass rates for each WPT test in 
each major browser. It makes it easier to identify areas we should improve on.

I - for one - would be happy to write tests using testharness.js when it makes 
sense if it means I can just land them in WebKit with my code change and have 
them being automagically upstreamed, and run by other browsers as well.

Note that I am not advocating we use testharness.js for everything (although I 
would be OK with this). I propose we make it opt-in. We have a lot of tests 
that are not necessarily interesting to upstream. However, the ones that are, 
using testharness.js makes a lot of sense.

Chris Dumez

> On Feb 5, 2017, at 5:04 PM, youenn fablet  wrote:
> 
> I too am a big proponent of us moving more and more towards WPT.
> As part of the streams and fetch API implementation, most of the related 
> functional tests have been made as WPT.
> The benefits of having tests being improved and updated largely outweighs the 
> testharness.js initial cost.
> Somehow, these tests are becoming part of their related specs.
> 
> The two complaints I heard against testharness.js are:
> - They are less easy to debug as test harness.js does not print out messages 
> for each assert. There might be room for updating testharness to support that
> - Async tests are less easy to write. While this is probably true, 
> testharness has support for promise-based tests which are not verbose.
> WPT has also some benefits of its own, like the possibility to run easily the 
> same tests in worker/window environments, the flexible python-based server...
> 
> One complaint I heard against WPT tests is that they require being run behind 
> a server.
> This is becoming more and more true. 
> There is still a large portion of tests that could be run locally if we 
> update the paths to test harness.js/testharnessreport,js.
> I am not a big fan of modify any WPT test but maybe we should consider 
> switching back to modifying these paths at import and export time.
> 
> I would also like to go in a direction where writing WPT tests is the default 
> for WebKit layout test.
> For that to happen, we need an easy way to upstream tests from WebKit to WPT 
> GitHub.
> 
> I would tend to do the following:
> - Write/submit WPT tests in WebKit as regular WebKit testharness.js layout 
> tests through bugzilla
> - At commit queue time, automatically create a PR to WPT GitHub containing 
> the changes of the WebKit patch
> - Ask committer to fix the WPT PR should there be any issue with it 
> (committer being informed of such issues through bugzilla).
> 
>> Le dim. 5 févr. 2017 à 15:42, Ryosuke Niwa  a écrit :
>> Hi all,
>> 
>> Background
>> 
>> Web Platform Tests is a suite of tests written for W3C specifications=,
>> and Blink, Edge, Gecko, and WebKit all run them as a part of their 
>> continuous build system.
>> 
>> Historically, each working group had their own repository for tests,
>> but with CSS WG's tests finally getting merged into the Web Platform Tests,
>> we will have a single repository with all the tests from W3C.
>> 
>> A few of us attended a meeting to discuss the future of this test suite last 
>> Monday.
>> One of the major topic was that Blink and Gecko have adopted the process to 
>> write the tests
>> using testharness.js in their own test suites, and automatically merge into 
>> Web Platform Tests
>> without having to go through another round of code review for Web Platform 
>> Tests.
>> 
>> Given we do test-driven development in WebKit, I think WebKit should do the 
>> same.
>> This will benefit other browser vendors to catch their bugs with our tests, 
>> and we will also benefit
>> from other browser vendors "reviewing" our tests against their 
>> implementation and specifications.
>> 
>> In the long term, I believe this will drastically improve the 
>> interoperability of the Web,
>> and dramatically improve the quality of the tests we run against WebKit.
>> 
>> In fact, there was a general consensus that we should even upstream 
>> pass-if-not-crash tests
>> as well as style and layout invalidation tests to web-platform-tests as they 
>> tend to be undertested
>> right now and almost all engines have bugs in this area.
>> 
>> Problems & Questions
>> 
>> In order to auto-merge tests from WebKit to Web Platform Tests, there are a 
>> few problems.
>> 
>> 1. We need to start writing more if not all tests using testharness.js 
>> instead of our own js-test(-pre).js
>> 
>> I've heard of many complaints about testharness.js being too verbose and 
>> cumbersome to use.
>> I agree with all those complaints but I don't think we can convince all 
>> other browser vendors
>> to use our own testing script either since both Blink & Gecko are 

Re: [webkit-dev] Upstreaming Tests from WebKit to Web Platform Tests

[youenn fablet]

I too am a big proponent of us moving more and more towards WPT.
As part of the streams and fetch API implementation, most of the related
functional tests have been made as WPT.
The benefits of having tests being improved and updated largely outweighs
the testharness.js initial cost.
Somehow, these tests are becoming part of their related specs.

The two complaints I heard against testharness.js are:
- They are less easy to debug as test harness.js does not print out
messages for each assert. There might be room for updating testharness to
support that
- Async tests are less easy to write. While this is probably true,
testharness has support for promise-based tests which are not verbose.
WPT has also some benefits of its own, like the possibility to run easily
the same tests in worker/window environments, the flexible python-based
server...

One complaint I heard against WPT tests is that they require being run
behind a server.
This is becoming more and more true.
There is still a large portion of tests that could be run locally if we
update the paths to test harness.js/testharnessreport,js.
I am not a big fan of modify any WPT test but maybe we should consider
switching back to modifying these paths at import and export time.

I would also like to go in a direction where writing WPT tests is the
default for WebKit layout test.
For that to happen, we need an easy way to upstream tests from WebKit to
WPT GitHub.

I would tend to do the following:
- Write/submit WPT tests in WebKit as regular WebKit testharness.js layout
tests through bugzilla
- At commit queue time, automatically create a PR to WPT GitHub containing
the changes of the WebKit patch
- Ask committer to fix the WPT PR should there be any issue with it
(committer being informed of such issues through bugzilla).

Le dim. 5 févr. 2017 à 15:42, Ryosuke Niwa  a écrit :

> Hi all,
>
> *Background*
>
> Web Platform Tests is a suite of tests written for W3C specifications
> =,
> and Blink, Edge, Gecko, and WebKit all run them as a part of their
> continuous build system.
>
> Historically, each working group had their own repository for tests,
> but with CSS WG's tests  finally
> getting merged into the Web Platform Tests,
> we will have a single repository with all the tests from W3C.
>
> A few of us attended a meeting to discuss the future of this test suite
> 
> last Monday.
> One of the major topic was that Blink and Gecko have adopted the process
> to write the tests
> using testharness.js in their own test suites, and automatically merge
> into Web Platform Tests
> without having to go through another round of code review for Web Platform
> Tests.
>
> Given we do test-driven development in WebKit, I think WebKit should do
> the same.
> This will benefit other browser vendors to catch their bugs with our
> tests, and we will also benefit
> from other browser vendors "reviewing" our tests against their
> implementation and specifications.
>
> In the long term, I believe this will drastically improve the
> interoperability of the Web,
> and dramatically improve the quality of the tests we run against WebKit.
>
> In fact, there was a general consensus that we should even upstream
> pass-if-not-crash tests
> as well as style and layout invalidation tests to web-platform-tests as
> they tend to be undertested
> right now and almost all engines have bugs in this area.
>
> *Problems & Questions*
>
> In order to auto-merge tests from WebKit to Web Platform Tests, there are
> a few problems.
>
> *1. We need to start writing more if not all tests using testharness.js
> instead of our own js-test(-pre).js*
>
> I've heard of many complaints about testharness.js being too verbose and
> cumbersome to use.
> I agree with all those complaints but I don't think we can convince all
> other browser vendors
> to use our own testing script either since both Blink & Gecko are onboard
> with using testharness.js
>
> At this point, I'd argue that the benefit outweighs the cost of adopting
> testharness.js.
> Also, W3C have dropped many styling requirements for tests so we no longer
> need to have
> link/meta elements, etc... You simply include testharness.js and
> testharness-report.js.
>
> See my recent PR
> 
>  for
> example.
>
> Do people still object to using testharness.js? If so, what are your
> reasons?
>
>
> *2. Where should we put upstremable tests?*
>
> We need a mechanism to identify & merge tests back from WebKit into Web
> Platform Tests.
>
> One way to do this is to start adding tests directly into
> LayoutTests/imported/w3c/web-platform-tests
> and then automatically identify those tests and upstream them.
>
> Assuming all upstream merges happen in timely happen, 

Re: [webkit-dev] Upstreaming Tests from WebKit to Web Platform Tests

[Philip Jägenstedt]

FWIW, in Blink we stopped rewriting the testharness.js paths before
switching to wptserve, by instead rewriting those URLs only when running
LayoutTests:
https://cs.chromium.org/chromium/src/content/shell/renderer/layout_test/blink_test_runner.cc?type=cs=content/shell/renderer/layout_test/blink_test_runner.cc=221

So, we know that it's possible to run a lot of the tests without wptserve
without modifying the tests. However, trying to list all the tests that do
or don't need wptserve seems like a lot of work, and I think we're now
using wptserve for all tests.

On Sun, Feb 5, 2017 at 8:05 PM youenn fablet  wrote:

> I too am a big proponent of us moving more and more towards WPT.
> As part of the streams and fetch API implementation, most of the related
> functional tests have been made as WPT.
> The benefits of having tests being improved and updated largely outweighs
> the testharness.js initial cost.
> Somehow, these tests are becoming part of their related specs.
>
> The two complaints I heard against testharness.js are:
> - They are less easy to debug as test harness.js does not print out
> messages for each assert. There might be room for updating testharness to
> support that
> - Async tests are less easy to write. While this is probably true,
> testharness has support for promise-based tests which are not verbose.
> WPT has also some benefits of its own, like the possibility to run easily
> the same tests in worker/window environments, the flexible python-based
> server...
>
> One complaint I heard against WPT tests is that they require being run
> behind a server.
> This is becoming more and more true.
> There is still a large portion of tests that could be run locally if we
> update the paths to test harness.js/testharnessreport,js.
> I am not a big fan of modify any WPT test but maybe we should consider
> switching back to modifying these paths at import and export time.
>
> I would also like to go in a direction where writing WPT tests is the
> default for WebKit layout test.
> For that to happen, we need an easy way to upstream tests from WebKit to
> WPT GitHub.
>
> I would tend to do the following:
> - Write/submit WPT tests in WebKit as regular WebKit testharness.js layout
> tests through bugzilla
> - At commit queue time, automatically create a PR to WPT GitHub containing
> the changes of the WebKit patch
> - Ask committer to fix the WPT PR should there be any issue with it
> (committer being informed of such issues through bugzilla).
>
> Le dim. 5 févr. 2017 à 15:42, Ryosuke Niwa  a écrit :
>
> Hi all,
>
> *Background*
>
> Web Platform Tests is a suite of tests written for W3C specifications
> =,
> and Blink, Edge, Gecko, and WebKit all run them as a part of their
> continuous build system.
>
> Historically, each working group had their own repository for tests,
> but with CSS WG's tests  finally
> getting merged into the Web Platform Tests,
> we will have a single repository with all the tests from W3C.
>
> A few of us attended a meeting to discuss the future of this test suite
> 
> last Monday.
> One of the major topic was that Blink and Gecko have adopted the process
> to write the tests
> using testharness.js in their own test suites, and automatically merge
> into Web Platform Tests
> without having to go through another round of code review for Web Platform
> Tests.
>
> Given we do test-driven development in WebKit, I think WebKit should do
> the same.
> This will benefit other browser vendors to catch their bugs with our
> tests, and we will also benefit
> from other browser vendors "reviewing" our tests against their
> implementation and specifications.
>
> In the long term, I believe this will drastically improve the
> interoperability of the Web,
> and dramatically improve the quality of the tests we run against WebKit.
>
> In fact, there was a general consensus that we should even upstream
> pass-if-not-crash tests
> as well as style and layout invalidation tests to web-platform-tests as
> they tend to be undertested
> right now and almost all engines have bugs in this area.
>
> *Problems & Questions*
>
> In order to auto-merge tests from WebKit to Web Platform Tests, there are
> a few problems.
>
> *1. We need to start writing more if not all tests using testharness.js
> instead of our own js-test(-pre).js*
>
> I've heard of many complaints about testharness.js being too verbose and
> cumbersome to use.
> I agree with all those complaints but I don't think we can convince all
> other browser vendors
> to use our own testing script either since both Blink & Gecko are onboard
> with using testharness.js
>
> At this point, I'd argue that the benefit outweighs the cost of adopting
> testharness.js.
> Also, W3C have dropped many styling requirements for tests so