Re: [webkit-dev] Another WPT bite

[Rick Byers]

On Tue, May 16, 2017 at 4:54 AM, Maciej Stachowiak  wrote:

>
>
> > On May 16, 2017, at 1:16 AM, Ryosuke Niwa  wrote:
> >
> > On Mon, May 15, 2017 at 11:57 PM, Anne van Kesteren 
> wrote:
> >> On Tue, May 16, 2017 at 7:29 AM, Ryosuke Niwa  wrote:
> >>> Given we're talking about how these tests are ran inside WebKit,
> >>> whether there is an agreement about this or not is sort of irrelevant.
> >>> If a test doesn't run as expected, we can run it inside a HTTP server.
> >>
> >> I was just trying to help clarify why what makes sense for WebKit,
> >> might not make sense for tests designed to run on all engines. If
> >> that's not desired here, I'll stop.
> >
> > Sure, I don't think we're interested in changing the way tests are ran
> > elsewhere.
> >
> > It would be great if upstream web-platform-tests could use relative
> > paths whenever possible or allowed annotation (e.g. we'd add such
> > annotation) as to which tests could be ran locally via file URL.
> >
> > However, that's more or less a secondary concern (nice-to-have)
> > whereas how web-platform-tests are imported and ran in WebKit are a
> > lot more important to us due to the impact it has on our development
> > process, tooling, as well as the maintenance cost.
>
> I agree with Ryosuke. We don't want to impose on others, but these two
> changes would be convenient for WebKit's use. Perhaps somewhere in WPT
> space (a GitHub issue?) would be the appropriate venue to discuss it. I am
> assuming here the tradeoff is just the maintenance burden of keeping the
> relative paths up to date, but maybe there is some deeper reason not to do
> it.
>

Filed https://github.com/w3c/web-platform-tests/issues/5945

Even though (after much debate) chromium has switched to using wptserve all
the time, there are still some situations where I'd find it handy to be
able to load a test (which I know doesn't depend on wptserve) from some
other server or filesystem.  So I'd support changing this upstream.


>  - Maciej
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Ben Kelly]

On Sat, May 13, 2017 at 8:55 PM, Michael[tm] Smith  wrote:

> Anne van Kesteren , 2017-05-13 06:20 +0200:
> >
> > On Fri, May 12, 2017 at 11:49 PM, Maciej Stachowiak 
> wrote:
> > > It seems like there's two unusual things about WPT:
> > > - At least according to Alexey, WPT tests are somewhat prone to
> flakiness in Safari.
> >
> > Although they haven't always been working perfectly, changes to
> > web-platform-tests run through some kind of stability check in both
> > Chrome and Firefox (run the new tests 10 times and fail if the results
> > are inconsistent). Ideally Safari is added to that mix, though I think
> > the last time folks looked into that it wasn't possible for some
> > reason. That might be something to put some resources on.
>
> I think we already have that now un-blocked and we will by the end of this
> week
> (May 19) start having Travis running the stability checker for Safari too
> (and
> Edge too) and providing the results as comments to PRs (just as we now do
> for Chrome and Firefox).
>

FWIW safari results are now showing up:

https://github.com/w3c/web-platform-tests/pull/5921#issuecomment-301517865
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Maciej Stachowiak]

> On May 16, 2017, at 1:16 AM, Ryosuke Niwa  wrote:
> 
> On Mon, May 15, 2017 at 11:57 PM, Anne van Kesteren  wrote:
>> On Tue, May 16, 2017 at 7:29 AM, Ryosuke Niwa  wrote:
>>> Given we're talking about how these tests are ran inside WebKit,
>>> whether there is an agreement about this or not is sort of irrelevant.
>>> If a test doesn't run as expected, we can run it inside a HTTP server.
>> 
>> I was just trying to help clarify why what makes sense for WebKit,
>> might not make sense for tests designed to run on all engines. If
>> that's not desired here, I'll stop.
> 
> Sure, I don't think we're interested in changing the way tests are ran
> elsewhere.
> 
> It would be great if upstream web-platform-tests could use relative
> paths whenever possible or allowed annotation (e.g. we'd add such
> annotation) as to which tests could be ran locally via file URL.
> 
> However, that's more or less a secondary concern (nice-to-have)
> whereas how web-platform-tests are imported and ran in WebKit are a
> lot more important to us due to the impact it has on our development
> process, tooling, as well as the maintenance cost.

I agree with Ryosuke. We don't want to impose on others, but these two changes 
would be convenient for WebKit's use. Perhaps somewhere in WPT space (a GitHub 
issue?) would be the appropriate venue to discuss it. I am assuming here the 
tradeoff is just the maintenance burden of keeping the relative paths up to 
date, but maybe there is some deeper reason not to do it.

 - Maciej
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Ryosuke Niwa]

On Mon, May 15, 2017 at 11:57 PM, Anne van Kesteren  wrote:
> On Tue, May 16, 2017 at 7:29 AM, Ryosuke Niwa  wrote:
>> Given we're talking about how these tests are ran inside WebKit,
>> whether there is an agreement about this or not is sort of irrelevant.
>> If a test doesn't run as expected, we can run it inside a HTTP server.
>
> I was just trying to help clarify why what makes sense for WebKit,
> might not make sense for tests designed to run on all engines. If
> that's not desired here, I'll stop.

Sure, I don't think we're interested in changing the way tests are ran
elsewhere.

It would be great if upstream web-platform-tests could use relative
paths whenever possible or allowed annotation (e.g. we'd add such
annotation) as to which tests could be ran locally via file URL.

However, that's more or less a secondary concern (nice-to-have)
whereas how web-platform-tests are imported and ran in WebKit are a
lot more important to us due to the impact it has on our development
process, tooling, as well as the maintenance cost.

- R. Niwa
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Anne van Kesteren]

On Tue, May 16, 2017 at 7:29 AM, Ryosuke Niwa  wrote:
> Given we're talking about how these tests are ran inside WebKit,
> whether there is an agreement about this or not is sort of irrelevant.
> If a test doesn't run as expected, we can run it inside a HTTP server.

I was just trying to help clarify why what makes sense for WebKit,
might not make sense for tests designed to run on all engines. If
that's not desired here, I'll stop.


-- 
https://annevankesteren.nl/
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Ryosuke Niwa]

On Mon, May 15, 2017 at 10:07 PM, Anne van Kesteren  wrote:
> On Tue, May 16, 2017 at 5:45 AM, Ryosuke Niwa  wrote:
>>> I think the main problem with not running a server is that behavior
>>> for file URLs is not defined. And browsers tend to impose different
>>> restrictions there. So you might end up debugging something only to
>>> later found out it didn't work due to file URL restrictions. And you
>>> can't guarantee that something will work from a file URL, because
>>> there's no agreement on what will.
>>
>> That's not an issue with most ref or testharness.js tests which tests
>> WebIDL, CSS OM, etc... because none of them have to do network
>> requests.
>
> Regardless of the feature, there's no defined agreement on how it
> should work when loaded from a file URL. A test author cannot divide
> "this can load from a file URL" from "this needs to be loaded over
> HTTP(S)". Except that loading over HTTP(S) always ought to work as
> that much we've written down in standards.

Given we're talking about how these tests are ran inside WebKit,
whether there is an agreement about this or not is sort of irrelevant.
If a test doesn't run as expected, we can run it inside a HTTP server.

Also, I'm not aware of any browser putting restrictions on how script
or link elements are loaded. For example, if z.js exits at
/Volumes/Data/bin/ and /Volumes/Data/bin/x/y.html loads z.js by
 then Chrome, Firefox, and Safari all
happily execute z.js.

Always using HTTP just because that's the lowest common denominator is
not great given that it significantly affects the common development
workflow people follow in WebKit.

I really don't think there is a viable path for importing and using
testharness.js in WebKit if we required that everyone must use HTTP
sever. Even I wouldn't do it even though I'm otherwise supportive of
writing more tests using testharness.js and upstreaming those tests
back to web-platform-tests.

- R. Niwa
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[youenn fablet]

It makes sense to run WPT tests as HTTP URLs for conformance/regression
purposes.
It is fine to run WPT tests as file based URLs for development purposes.

Tooling should make it possible to run WPT tests as HTTP URLs for
development purposes with minimum to no cost.
We are not there yet.

Le lun. 15 mai 2017 à 22:08, Anne van Kesteren  a écrit :

> On Tue, May 16, 2017 at 5:45 AM, Ryosuke Niwa  wrote:
> >> I think the main problem with not running a server is that behavior
> >> for file URLs is not defined. And browsers tend to impose different
> >> restrictions there. So you might end up debugging something only to
> >> later found out it didn't work due to file URL restrictions. And you
> >> can't guarantee that something will work from a file URL, because
> >> there's no agreement on what will.
> >
> > That's not an issue with most ref or testharness.js tests which tests
> > WebIDL, CSS OM, etc... because none of them have to do network
> > requests.
>
> Regardless of the feature, there's no defined agreement on how it
> should work when loaded from a file URL. A test author cannot divide
> "this can load from a file URL" from "this needs to be loaded over
> HTTP(S)". Except that loading over HTTP(S) always ought to work as
> that much we've written down in standards.
>
>
> --
> https://annevankesteren.nl/
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Anne van Kesteren]

On Tue, May 16, 2017 at 5:45 AM, Ryosuke Niwa  wrote:
>> I think the main problem with not running a server is that behavior
>> for file URLs is not defined. And browsers tend to impose different
>> restrictions there. So you might end up debugging something only to
>> later found out it didn't work due to file URL restrictions. And you
>> can't guarantee that something will work from a file URL, because
>> there's no agreement on what will.
>
> That's not an issue with most ref or testharness.js tests which tests
> WebIDL, CSS OM, etc... because none of them have to do network
> requests.

Regardless of the feature, there's no defined agreement on how it
should work when loaded from a file URL. A test author cannot divide
"this can load from a file URL" from "this needs to be loaded over
HTTP(S)". Except that loading over HTTP(S) always ought to work as
that much we've written down in standards.


-- 
https://annevankesteren.nl/
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Ryosuke Niwa]

On Sun, May 14, 2017 at 12:35 AM, Anne van Kesteren  wrote:
> On Sun, May 14, 2017 at 8:24 AM, Maciej Stachowiak  wrote:
>> For the engineer use case, we can make a command-line tool to launch the 
>> server and load the test. But it's kind of sad that in ~95% of cases, the 
>> only value provided by the server is resolving the path to testharness.js. 
>> If tests referenced testharness.js via relative path, then most of the time 
>> they could be loaded as local files just fine, which would be more 
>> convenient (as well as, I believe, solving the "external trac link" issue).
>
> I think the main problem with not running a server is that behavior
> for file URLs is not defined. And browsers tend to impose different
> restrictions there. So you might end up debugging something only to
> later found out it didn't work due to file URL restrictions. And you
> can't guarantee that something will work from a file URL, because
> there's no agreement on what will.

That's not an issue with most ref or testharness.js tests which tests
WebIDL, CSS OM, etc... because none of them have to do network
requests.

> I personally just keep web-platform-tests running and don't notice
> much overhead (MacBook Air, Mid 2012).

People at Apple tend to reboot their machines and install new OS all
the time so this is clearly an annoyance for us. I've rarely debugged
or created our own HTTP tests for the extra step that involves.

Even with all the shadow DOM and custom elements I wrote, I used
relative paths when I write them, and only removed relative path right
before I uploaded them to WPT repository. Furthermore, whenever I
debug those tests, the first thing I do is to rewrite the paths to
relative paths although these days, I've worked around that by adding
symlinks from /resources to LayoutTests/resources/ on multiple OS
installations.  Nonetheless, this remains to be one of the biggest
issue I have with WPT tests.

- R. Niwa
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Anne van Kesteren]

On Sun, May 14, 2017 at 8:24 AM, Maciej Stachowiak  wrote:
> For the engineer use case, we can make a command-line tool to launch the 
> server and load the test. But it's kind of sad that in ~95% of cases, the 
> only value provided by the server is resolving the path to testharness.js. If 
> tests referenced testharness.js via relative path, then most of the time they 
> could be loaded as local files just fine, which would be more convenient (as 
> well as, I believe, solving the "external trac link" issue).

I think the main problem with not running a server is that behavior
for file URLs is not defined. And browsers tend to impose different
restrictions there. So you might end up debugging something only to
later found out it didn't work due to file URL restrictions. And you
can't guarantee that something will work from a file URL, because
there's no agreement on what will.

I personally just keep web-platform-tests running and don't notice
much overhead (MacBook Air, Mid 2012).


-- 
https://annevankesteren.nl/
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Maciej Stachowiak]

> On May 13, 2017, at 5:26 PM, Michael[tm] Smith  wrote:
> 
> Maciej Stachowiak , 2017-05-13 14:58 -0700:
>> ... From what I gather, there are a lot of tests where only the paths to
>> the test harness end up requiring the server.
> 
> Yeah that’s the case for the vast majority of tests. Relatively few — less 
> than
> 5% altogether, I’d estimate — actually rely on any special server behavior.
> 
> Of the ones needing special server behavior, I think even there by far most 
> are
> just cases of an accompanying foo.html.headers file in the tree along with the
> foo.html test, to specify that the server send particular response headers.
> 
> Out of ~50,000 test files in WPT, there are less than 1000 with a .headers 
> file.
> 
> I think the next-most-common case that require special server behavior are 
> cases
> where the server is doing some parsing and substitution of special parameters.
> In those cases the test file itself will be named in the pattern foo.sub.html,
> or one of its JS assets in the pattern foo.sub.js.
> 
> Out of ~50,000 test files in WPT, less than 300 are *.sub.* files.
> 
> So those two cases take together amount to only 3% of the total. So even with
> whatever else I’m missing added to that I’d estimate the number of tests that
> don’t rely on special server behavior is on the order of 95%.
> 
> So those ~95% all only need the “/resources/testharness.js” path to the test
> harness to resolve and then they’ll just work.
> 
>> Doing the fixup on import seems bad to me, since it seems safer and
>> cleaner for our WPT checkout to match WPT. But we could follow the
>> practice of using relative URLs for self-created tests, and perhaps not
>> even run them under the server when they don't need it.
>> 
>> For upstream, perhaps we could advocate with WPT to use relative paths to
>> load the harness
> 
> Given that a specific problem case Alexey mentioned was linking to tests 
> within
> the WebKit Trac and having them run as expected, I wonder if at least y’all
> could find a way to just make https://trac.webkit.org/resources/testharness.js
> work — I guess by making it redirect to some place where you have a current 
> WPT
> checkout. That’d at least solve things for the main specific case that’s been
> brought up so far being a real problem.

Alexey mentioned trac because it's something we couldn't easily solve with a 
command-line tool. The far more common case is engineers loading individual 
test cases to debug them. As you mentioned ~95% of test cases would load fine 
as a local file, except for the path where they expect testharness.js.

For the engineer use case, we can make a command-line tool to launch the server 
and load the test. But it's kind of sad that in ~95% of cases, the only value 
provided by the server is resolving the path to testharness.js. If tests 
referenced testharness.js via relative path, then most of the time they could 
be loaded as local files just fine, which would be more convenient (as well as, 
I believe, solving the "external trac link" issue).


>> (and perhaps make sure that tests that absolutely require the server fail in 
>> a
>> way that clearly indicates this for the tests that truly do need networking 
>> or
>> some other facility of the WPT server).
> 
> Yeah that would be good to make happen in general regardless. But I think 
> right
> now that maybe can mostly be (programatically) determined by (1) checking if 
> the
> name of the test file itself is in the *.sub.* pattern, or (2) checking to see
> if there’s an accompanying *.headers file for the test.

It would only be relevant for tests to give better diagnostics if they were set 
up to run outside the WPT server at all, which they aren't. This suggestion is 
somewhat conditional on consideration of that other suggestion (which is 
basically to load required static resources via relative path).

Regards,
Maciej

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Michael[tm] Smith]

Anne van Kesteren , 2017-05-13 06:20 +0200:
> 
> On Fri, May 12, 2017 at 11:49 PM, Maciej Stachowiak  wrote:
> > It seems like there's two unusual things about WPT:
> > - At least according to Alexey, WPT tests are somewhat prone to flakiness 
> > in Safari.
> 
> Although they haven't always been working perfectly, changes to
> web-platform-tests run through some kind of stability check in both
> Chrome and Firefox (run the new tests 10 times and fail if the results
> are inconsistent). Ideally Safari is added to that mix, though I think
> the last time folks looked into that it wasn't possible for some
> reason. That might be something to put some resources on.

I think we already have that now un-blocked and we will by the end of this week
(May 19) start having Travis running the stability checker for Safari too (and
Edge too) and providing the results as comments to PRs (just as we now do
for Chrome and Firefox).

  —Mike

-- 
Michael[tm] Smith https://people.w3.org/mike


signature.asc
Description: PGP signature
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Michael[tm] Smith]

Brady Eidson , 2017-05-13 17:11 -0700:
> 
> 
> > On May 12, 2017, at 7:43 PM, Ryosuke Niwa  wrote:
> > 
> > On Fri, May 12, 2017 at 7:31 PM, Alexey Proskuryakov  
> > wrote:
> >> 
> >> When there is a test failure that I need to communicate to others, I say
> >> something "please open
> >> 
> >> in Safari to reproduce". That's very easy to do, and makes it very easy for
> >> others to work on the issue.
> >> ...
> > 
> > Note that W3C's web-plaform-tests are hosted on
> > http://w3c-test.org/tools/runner/index.html so you can could do the
> > same thing.
> 
> Note Alexey's URL example points to a specific revision of a specific test.
> 
> A frustration that has often come up while I've been working on
> leading-edge features in WebKit is that the w3c-test.org
>  tests are constantly updated and therefore are a
> moving target.  One cannot use the same test repeatedly over some length
> of time and expect it to remain consistent.

True. But of course the tests are under version control and you can always
locate any specific revision of one you want to test against; e.g.:

  
https://raw.githubusercontent.com/w3c/web-platform-tests/b2e3ab7/service-workers/service-worker/activate-event-after-install-state-change.https.html

For practical reasons on w3c-test.org we can’t directly archive a runnable copy
of every single revision of of every single test. But I would be very willing to
look into adding a service somewhere that does something similar to what
rawgit.com or whatever do—that is, (1) given a URL like the above for specific
revision of a file in a github repo, then (2) load and run it.

rawgit.com itself can of course already load specific revisions of WPT
tests, but when you try it you’ll run into the general problem of the test
files specifying a /resources/testharness.js path to the harness; e.g.:

  
https://rawgit.com/w3c/web-platform-tests/b2e3ab7/service-workers/service-worker/activate-event-after-install-state-change.https.html

...which expects to find https://rawgit.com/resources/testharness.js.

But that’s not a hard problem to solve. As I mentioned earlier, I imagine
that with very little effort needed, I could get a similar service set up
that’s WPT-specific and WPT-aware in that it will have the test harness
available at for the /resources/testharness.js URL path.

  —Mike

-- 
Michael[tm] Smith https://people.w3.org/mike


signature.asc
Description: PGP signature
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Michael[tm] Smith]

Maciej Stachowiak , 2017-05-13 14:58 -0700:
> ... From what I gather, there are a lot of tests where only the paths to
> the test harness end up requiring the server.

Yeah that’s the case for the vast majority of tests. Relatively few — less than
5% altogether, I’d estimate — actually rely on any special server behavior.

Of the ones needing special server behavior, I think even there by far most are
just cases of an accompanying foo.html.headers file in the tree along with the
foo.html test, to specify that the server send particular response headers.

Out of ~50,000 test files in WPT, there are less than 1000 with a .headers file.

I think the next-most-common case that require special server behavior are cases
where the server is doing some parsing and substitution of special parameters.
In those cases the test file itself will be named in the pattern foo.sub.html,
or one of its JS assets in the pattern foo.sub.js.

Out of ~50,000 test files in WPT, less than 300 are *.sub.* files.

So those two cases take together amount to only 3% of the total. So even with
whatever else I’m missing added to that I’d estimate the number of tests that
don’t rely on special server behavior is on the order of 95%.

So those ~95% all only need the “/resources/testharness.js” path to the test
harness to resolve and then they’ll just work.

> Doing the fixup on import seems bad to me, since it seems safer and
> cleaner for our WPT checkout to match WPT. But we could follow the
> practice of using relative URLs for self-created tests, and perhaps not
> even run them under the server when they don't need it.
> 
> For upstream, perhaps we could advocate with WPT to use relative paths to
> load the harness

Given that a specific problem case Alexey mentioned was linking to tests within
the WebKit Trac and having them run as expected, I wonder if at least y’all
could find a way to just make https://trac.webkit.org/resources/testharness.js
work — I guess by making it redirect to some place where you have a current WPT
checkout. That’d at least solve things for the main specific case that’s been
brought up so far being a real problem.

> (and perhaps make sure that tests that absolutely require the server fail in a
> way that clearly indicates this for the tests that truly do need networking or
> some other facility of the WPT server).

Yeah that would be good to make happen in general regardless. But I think right
now that maybe can mostly be (programatically) determined by (1) checking if the
name of the test file itself is in the *.sub.* pattern, or (2) checking to see
if there’s an accompanying *.headers file for the test.

  —Mike

-- 
Michael[tm] Smith https://people.w3.org/mike


signature.asc
Description: PGP signature
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Brady Eidson]

> On May 12, 2017, at 7:43 PM, Ryosuke Niwa  wrote:
> 
> On Fri, May 12, 2017 at 7:31 PM, Alexey Proskuryakov  wrote:
>> 
>> When there is a test failure that I need to communicate to others, I say
>> something "please open
>> 
>> in Safari to reproduce". That's very easy to do, and makes it very easy for
>> others to work on the issue.
>> ...
> 
> Note that W3C's web-plaform-tests are hosted on
> http://w3c-test.org/tools/runner/index.html so you can could do the
> same thing.

Note Alexey's URL example points to a specific revision of a specific test.

A frustration that has often come up while I've been working on leading-edge 
features in WebKit is that the w3c-test.org  tests are 
constantly updated and therefore are a moving target.
One cannot use the same test repeatedly over some length of time and expect it 
to remain consistent.

Thanks,
~Brady



___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Maciej Stachowiak]

> On May 12, 2017, at 7:49 PM, a...@webkit.org wrote:
> 
> 
>> 12 мая 2017 г., в 19:38, Brian Burg > > написал(а):
>> 
>>> I think that I explained it very clearly, but let me try again.
>>> 
>>> When there is a test failure that I need to communicate to others, I say 
>>> something "please open 
>>> >>  
>>> >
>>>  in Safari to reproduce". That's very easy to do, and makes it very easy 
>>> for others to work on the issue.
>>> If your test requires complex setup, like WPT does, then I may not have the 
>>> time to write up complicated steps to reproduce, or the person who gets the 
>>> bug may not have the time to follow them. Those people don't have a WebKit 
>>> checkout, so scripts won't help. This makes the test less effective, as 
>>> problems that it finds are less likely to be addressed.
>> 
>> If the person works on WebKit, then it seems unreasonable that they would do 
>> work without a checkout.
> 
> It is correct that people who work on WebKit usually have a checkout. So I 
> was taking about people who don't work on WebKit.
> 
>> If they don’t work on WebKit, then you could run wptserve on a machine 
>> somewhere and link to that copy. We have several servers that exist solely 
>> to host test content, it doesn’t seem like a big deal to make one of them 
>> update regularly and relaunch wptserve to pick up test harness changes.
> 
> Yes, there is a number of things one could do. Those things would work in 
> some cases but not in others - I mentioned linking to a stable version that 
> won't change, which is something that trac gives us for free, and it would be 
> non-trivial to implement otherwise.
> 
> In practice, the best approach would be to reduce the test to a minimum that 
> doesn't use complex harnesses before ending it over. Everyone likes minimal 
> test cases.

I think WPT is becoming enough of a broad community norm that it's not 
unreasonable to point to a WPT server. It does seem regrettable that tests 
become dependent on being run through the server even when in principle they 
could run as local files. From what I gather, there are a lot of tests where 
only the paths to the test harness end up requiring the server. Doing the fixup 
on import seems bad to me, since it seems safer and cleaner for our WPT 
checkout to match WPT. But we could follow the practice of using relative URLs 
for self-created tests, and perhaps not even run them under the server when 
they don't need it.

For upstream, perhaps we could advocate with WPT to use relative paths to load 
the harness (and perhaps make sure that tests that absolutely require the 
server fail in a way that clearly indicates this for the tests that truly do 
need networking or some other facility of the WPT server).

Regards,
Maciej

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[youenn fablet]

I filed https://bugs.webkit.org/show_bug.cgi?id=172068 to track the need
for some extra tooling for HTTP/WPT served tests.
We already gathered information about related requirements & workflows here.
Let's add more there!

Le ven. 12 mai 2017 à 19:50,  a écrit :

>
> 12 мая 2017 г., в 19:38, Brian Burg  написал(а):
>
>
> I think that I explained it very clearly, but let me try again.
>
> When there is a test failure that I need to communicate to others, I say
> something "please open <
> https://trac.webkit.org/export/216812/webkit/trunk/LayoutTests/fast/images/destroyed-image-load-event.html
> >
> in Safari to reproduce". That's very easy to do, and makes it very easy for
> others to work on the issue.
> If your test requires complex setup, like WPT does, then I may not have
> the time to write up complicated steps to reproduce, or the person who gets
> the bug may not have the time to follow them. Those people don't have a
> WebKit checkout, so scripts won't help. This makes the test less effective,
> as problems that it finds are less likely to be addressed.
>
>
> If the person works on WebKit, then it seems unreasonable that they would
> do work without a checkout.
>
>
> It is correct that people who work on WebKit usually have a checkout. So I
> was taking about people who don't work on WebKit.
>
> If they don’t work on WebKit, then you could run wptserve on a machine
> somewhere and link to that copy. We have several servers that exist solely
> to host test content, it doesn’t seem like a big deal to make one of them
> update regularly and relaunch wptserve to pick up test harness changes.
>
>
> Yes, there is a number of things one could do. Those things would work in
> some cases but not in others - I mentioned linking to a stable version that
> won't change, which is something that trac gives us for free, and it would
> be non-trivial to implement otherwise.
>
> In practice, the best approach would be to reduce the test to a minimum
> that doesn't use complex harnesses before ending it over. Everyone likes
> minimal test cases.
>
> - Alexey
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Anne van Kesteren]

On Fri, May 12, 2017 at 11:49 PM, Maciej Stachowiak  wrote:
> It seems like there's two unusual things about WPT:
> - At least according to Alexey, WPT tests are somewhat prone to flakiness in 
> Safari.

Although they haven't always been working perfectly, changes to
web-platform-tests run through some kind of stability check in both
Chrome and Firefox (run the new tests 10 times and fail if the results
are inconsistent). Ideally Safari is added to that mix, though I think
the last time folks looked into that it wasn't possible for some
reason. That might be something to put some resources on.


-- 
https://annevankesteren.nl/
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Brian Burg]

> On May 12, 2017, at 7:31 PM, Alexey Proskuryakov  wrote:
> 
>> 
>> 12 мая 2017 г., в 16:12, Sam Weinig > > написал(а):
>> 
>> 
>> 
>>> On May 12, 2017, at 2:49 PM, Alexey Proskuryakov >> > wrote:
>>> 
>>> 
 12 мая 2017 г., в 14:38, Sam Weinig > написал(а):
 
 I regret piling on here, as I think this thread has diverged from it’s 
 original purpose, but…I understand this frustration. That said, perhaps 
 this is something we can solve with some tooling. For instance, a 
 run-test-in-safari (as a parallel to run-safari) script could be made 
 which starts the server, and then loads the test with the right URL in 
 your built Safari (or MiniBrowser, or whatever).  
>>> 
>>> 
>>> That's still not good enough, as this means that I can't point someone else 
>>> to an instance of the test on trac.webkit.org  to 
>>> reproduce an issue. There is of course be another place to point to when/if 
>>> the test gets upstreamed, but even that doesn't support stable links like 
>>> trac does.
>>> 
>>> That's not to mention that learning the name of this proposed script is no 
>>> easier than learning about run-webkit-httpd.
>>> 
>>> - Alexey
>>> 
>> 
>> 
>> I’m not sure what you mean by “good enough”.  Good enough for what?  What 
>> are we debating here?
> 
> I think that I explained it very clearly, but let me try again.
> 
> When there is a test failure that I need to communicate to others, I say 
> something "please open 
>   
> >
>  in Safari to reproduce". That's very easy to do, and makes it very easy for 
> others to work on the issue.
> If your test requires complex setup, like WPT does, then I may not have the 
> time to write up complicated steps to reproduce, or the person who gets the 
> bug may not have the time to follow them. Those people don't have a WebKit 
> checkout, so scripts won't help. This makes the test less effective, as 
> problems that it finds are less likely to be addressed.

If the person works on WebKit, then it seems unreasonable that they would do 
work without a checkout.

If they don’t work on WebKit, then you could run wptserve on a machine 
somewhere and link to that copy. We have several servers that exist solely to 
host test content, it doesn’t seem like a big deal to make one of them update 
regularly and relaunch wptserve to pick up test harness changes.

> 
> - Alexey
> 
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org 
> https://lists.webkit.org/mailman/listinfo/webkit-dev 
> 
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[ap]

> 12 мая 2017 г., в 19:38, Brian Burg  написал(а):
> 
>> I think that I explained it very clearly, but let me try again.
>> 
>> When there is a test failure that I need to communicate to others, I say 
>> something "please open 
>> >  
>> >
>>  in Safari to reproduce". That's very easy to do, and makes it very easy for 
>> others to work on the issue.
>> If your test requires complex setup, like WPT does, then I may not have the 
>> time to write up complicated steps to reproduce, or the person who gets the 
>> bug may not have the time to follow them. Those people don't have a WebKit 
>> checkout, so scripts won't help. This makes the test less effective, as 
>> problems that it finds are less likely to be addressed.
> 
> If the person works on WebKit, then it seems unreasonable that they would do 
> work without a checkout.

It is correct that people who work on WebKit usually have a checkout. So I was 
taking about people who don't work on WebKit.

> If they don’t work on WebKit, then you could run wptserve on a machine 
> somewhere and link to that copy. We have several servers that exist solely to 
> host test content, it doesn’t seem like a big deal to make one of them update 
> regularly and relaunch wptserve to pick up test harness changes.

Yes, there is a number of things one could do. Those things would work in some 
cases but not in others - I mentioned linking to a stable version that won't 
change, which is something that trac gives us for free, and it would be 
non-trivial to implement otherwise.

In practice, the best approach would be to reduce the test to a minimum that 
doesn't use complex harnesses before ending it over. Everyone likes minimal 
test cases.

- Alexey

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Ryosuke Niwa]

On Fri, May 12, 2017 at 7:38 PM, Brian Burg  wrote:
>
> If the person works on WebKit, then it seems unreasonable that they would do 
> work without a checkout.

This is not entirely true. If I have an iOS device, it's a lot easier
to load up a web page on a public server somewhere rather than setting
up a server on my mac, and then making it visible to my iPhone / iPad.

Also, I find it extremely useful to be able to run a test without
having a WebKit checkout since I install many different OS'es on many
Macs in order to figure out when a regression got introduced in the
underlying system itself, etc...

> If they don’t work on WebKit, then you could run wptserve on a machine
> somewhere and link to that copy. We have several servers that exist solely
> to host test content, it doesn’t seem like a big deal to make one of them
> update regularly and relaunch wptserve to pick up test harness changes.

Indeed such a server already exists for web-platform-tests.:
http://w3c-test.org/tools/runner/index.html

We could setup a similar server for layout tests but I'm not certain
if layout tests are written to be secure to be hosted on a publicly
visible server.

- R. Niwa
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Ryosuke Niwa]

On Fri, May 12, 2017 at 7:31 PM, Alexey Proskuryakov  wrote:
>
> 12 мая 2017 г., в 16:12, Sam Weinig  написал(а):
>
>
>
> On May 12, 2017, at 2:49 PM, Alexey Proskuryakov  wrote:
>
>
> 12 мая 2017 г., в 14:38, Sam Weinig  написал(а):
>
> I regret piling on here, as I think this thread has diverged from it’s
> original purpose, but…I understand this frustration. That said, perhaps this
> is something we can solve with some tooling. For instance, a
> run-test-in-safari (as a parallel to run-safari) script could be made which
> starts the server, and then loads the test with the right URL in your built
> Safari (or MiniBrowser, or whatever).
>
>
> That's still not good enough, as this means that I can't point someone else
> to an instance of the test on trac.webkit.org to reproduce an issue. There
> is of course be another place to point to when/if the test gets upstreamed,
> but even that doesn't support stable links like trac does.
>
> That's not to mention that learning the name of this proposed script is no
> easier than learning about run-webkit-httpd.
>
> - Alexey
>
>
> I’m not sure what you mean by “good enough”.  Good enough for what?  What
> are we debating here?
>
>
> I think that I explained it very clearly, but let me try again.
>
> When there is a test failure that I need to communicate to others, I say
> something "please open
> 
> in Safari to reproduce". That's very easy to do, and makes it very easy for
> others to work on the issue.
>
> If your test requires complex setup, like WPT does, then I may not have the
> time to write up complicated steps to reproduce, or the person who gets the
> bug may not have the time to follow them. Those people don't have a WebKit
> checkout, so scripts won't help. This makes the test less effective, as
> problems that it finds are less likely to be addressed.

Note that W3C's web-plaform-tests are hosted on
http://w3c-test.org/tools/runner/index.html so you can could do the
same thing.

So this is about WPT server tests written in WebKit?  But if it had to
be a WPT server test, then the alternative would have been HTTP tests
so the situation wouldn't have been different.

If you're talking about tests that use testharness.js, then we would
be using a relative path, so that should continue to work.

- R. Niwa
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Alexey Proskuryakov]

> 12 мая 2017 г., в 16:12, Sam Weinig  написал(а):
> 
> 
> 
>> On May 12, 2017, at 2:49 PM, Alexey Proskuryakov > > wrote:
>> 
>> 
>>> 12 мая 2017 г., в 14:38, Sam Weinig >> > написал(а):
>>> 
>>> I regret piling on here, as I think this thread has diverged from it’s 
>>> original purpose, but…I understand this frustration. That said, perhaps 
>>> this is something we can solve with some tooling. For instance, a 
>>> run-test-in-safari (as a parallel to run-safari) script could be made which 
>>> starts the server, and then loads the test with the right URL in your built 
>>> Safari (or MiniBrowser, or whatever).  
>> 
>> 
>> That's still not good enough, as this means that I can't point someone else 
>> to an instance of the test on trac.webkit.org  to 
>> reproduce an issue. There is of course be another place to point to when/if 
>> the test gets upstreamed, but even that doesn't support stable links like 
>> trac does.
>> 
>> That's not to mention that learning the name of this proposed script is no 
>> easier than learning about run-webkit-httpd.
>> 
>> - Alexey
>> 
> 
> 
> I’m not sure what you mean by “good enough”.  Good enough for what?  What are 
> we debating here?

I think that I explained it very clearly, but let me try again.

When there is a test failure that I need to communicate to others, I say 
something "please open 
>
 in Safari to reproduce". That's very easy to do, and makes it very easy for 
others to work on the issue.

If your test requires complex setup, like WPT does, then I may not have the 
time to write up complicated steps to reproduce, or the person who gets the bug 
may not have the time to follow them. Those people don't have a WebKit 
checkout, so scripts won't help. This makes the test less effective, as 
problems that it finds are less likely to be addressed.

- Alexey

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[youenn fablet]

> On the topic of LayoutTest/imported tests, can someone describe the
> current process of working with LayoutTest/imported?
>
> How do we handle a broken test in our tree?
>
> • Do we modify our expectations?
>
> - If so, how do we remember to change the expectations in a later import?
>
>
If a test breaks, you are supposed to rebase the expected file.
A later import will rebase the expected file.

When importing, some subtests may not be passing.
This is ok, keep the expectation as is and do not add a FAIL line in
TestExpectations.


> • Do we modify the test? More generally, are any modifications allowed
> inside of LayoutTest/imported?
> - If so, how do we track the fact that we have modifications so that a
> later import doesn't just wipe out our changes?
> - Without tooling, is the person who modifies the test responsible for
> making an upstream pull request?
>
>
It is ok to modify the test only if you are making sure the test change is
upstreamed at landing time.
The author of the patch is currently responsible for upstreaming the
changes.
We will make things better in a not-too-far future.



> How do we handle a new imported test that WebKit fails?
>
> To avoid bots being red this test will end up getting skipped or marked as
> flakey.
>
> If it is flaky, mark it as such or mark it as skip
If it is consistently failing, it is doing some level of regression testing.
Another possibility is to not import it at all.
Please use LayoutTests/imported/w3c/resources/import-expectation.json for
that purpose.

How do we ensure this actually gets addressed and not ignored / forgotten
> about?
>
>
This is currently the responsibility of the developer doing the feature
work.
When a feature is complete, we should look at how much tests are passing
vs. skipped/failing.
This is conformance testing, we can use our own copy of WPT or W3C one.


>
> To make upstream changes someone has to make a pull-request:
>
>
> I've had some upstream patches reviewed immediately and others are still
> in review after 3+ months.
>
> In an earlier webkit-dev email Youenn mentioned:
> https://lists.webkit.org/pipermail/webkit-dev/2017-April/028932.html
>
> As done by other browser teams, a test that is reviewed in WebKit bugzilla
> and
>
> landed in WebKit can be readily merged into WPT without additional review
>
> (although additional review is always great!).
>
>
> I don't have commit access to the WPT GitHub. So ultimately I still have
> to go through another review to get my pull requests accepted.
>
>
You might get commit access.
Otherwise, some WebKit members have it and will do the cq for you.
Please ping me for that purpose. I am sure others might help too.

>
> - Joe
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Joseph Pecoraro]

> On May 12, 2017, at 2:39 PM, Ryosuke Niwa  wrote:
> 
> On Fri, May 12, 2017 at 12:04 PM, Alexey Proskuryakov  > wrote:
> 
>> We don't have a concept of "first class", but I hope that when choosing
>> between looking into a simple test that was added for a known important bug,
>> and looking into an imported test whose importance is unclear, any WebKit
>> engineer will pick the former. And since no one can fix all the things, such
>> prioritization makes imported tests less effective.
> 
> This is absolutely not how I operate at all. Since almost all custom
> elements and shadow DOM API tests I wrote are written using
> testharness.js and upstreamed to web-platform-tests, they have been
> deleted from LayoutTests/fast/shadow-dom &
> LayoutTests/fast/custom-elements.
> 
> As such, if any new shadow DOM or custom elements tests under
> LayoutTests/imported/ start to fail, then we must fix them since we
> idon'thave any other test coverage.


On the topic of LayoutTest/imported tests, can someone describe the current 
process of working with LayoutTest/imported?

How do we handle a broken test in our tree?

• Do we modify our expectations?
- If so, how do we remember to change the expectations in a later 
import?

• Do we modify the test? More generally, are any modifications allowed inside 
of LayoutTest/imported?
- If so, how do we track the fact that we have modifications so that a 
later import doesn't just wipe out our changes?
- Without tooling, is the person who modifies the test responsible for 
making an upstream pull request?

How do we handle a new imported test that WebKit fails?

To avoid bots being red this test will end up getting skipped or marked as 
flakey.

How do we ensure this actually gets addressed and not ignored / forgotten about?

To make upstream changes someone has to make a pull-request:

I've had some upstream patches reviewed immediately and others are still in 
review after 3+ months.

In an earlier webkit-dev email Youenn mentioned:
https://lists.webkit.org/pipermail/webkit-dev/2017-April/028932.html

> As done by other browser teams, a test that is reviewed in WebKit bugzilla and
> landed in WebKit can be readily merged into WPT without additional review
> (although additional review is always great!).


I don't have commit access to the WPT GitHub. So ultimately I still have to go 
through another review to get my pull requests accepted.

- Joe

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Sam Weinig]

> On May 12, 2017, at 2:49 PM, Alexey Proskuryakov  wrote:
> 
> 
>> 12 мая 2017 г., в 14:38, Sam Weinig > > написал(а):
>> 
>> I regret piling on here, as I think this thread has diverged from it’s 
>> original purpose, but…I understand this frustration. That said, perhaps this 
>> is something we can solve with some tooling. For instance, a 
>> run-test-in-safari (as a parallel to run-safari) script could be made which 
>> starts the server, and then loads the test with the right URL in your built 
>> Safari (or MiniBrowser, or whatever).  
> 
> 
> That's still not good enough, as this means that I can't point someone else 
> to an instance of the test on trac.webkit.org  to 
> reproduce an issue. There is of course be another place to point to when/if 
> the test gets upstreamed, but even that doesn't support stable links like 
> trac does.
> 
> That's not to mention that learning the name of this proposed script is no 
> easier than learning about run-webkit-httpd.
> 
> - Alexey
> 


I’m not sure what you mean by “good enough”.  Good enough for what?  What are 
we debating here?  I was suggesting a mechanism to ease Simon’s specific 
problem, and I do think a script which both starts the server, and loads the 
correct URL would be helpful.  I often forget exactly what path to copy when 
writing http tests.

- Sam___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[youenn fablet]

I would guess Chromium and Mozilla to have the same issues there.
Incidentally, some work is being done right now to ease the
run-a-server-then-launch-a-browser thing.
We should be able to piggy-back on that effort and also handle the same
thing for LayoutTetsts/http/tests.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Alexey Proskuryakov]

> 12 мая 2017 г., в 14:39, Ryosuke Niwa  написал(а):
> 
> This is absolutely not how I operate at all. Since almost all custom
> elements and shadow DOM API tests I wrote are written using
> testharness.js and upstreamed to web-platform-tests, they have been
> deleted from LayoutTests/fast/shadow-dom &
> LayoutTests/fast/custom-elements.
> 
> As such, if any new shadow DOM or custom elements tests under
> LayoutTests/imported/ start to fail, then we must fix them since we
> idon'thave any other test coverage.


I'll try to remember that. I do think that this means that shadow DOM and 
custom elements now have less effective tests, which may result in overlooking 
important issues.

- Alexey

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Maciej Stachowiak]

> On May 12, 2017, at 2:39 PM, Ryosuke Niwa  wrote:
> 
> On Fri, May 12, 2017 at 12:04 PM, Alexey Proskuryakov  wrote:
>> 
>> 12 мая 2017 г., в 11:52, Ben Kelly  написал(а):
>> 
>> On Fri, May 12, 2017 at 2:26 PM, Rick Byers  wrote:
>>> 
>>> On Fri, May 12, 2017 at 2:06 PM, Alexey Proskuryakov 
>>> wrote:
 
 Since imported WPT tests are very flaky, and are not necessarily written
 to defend against important regressions, investigating issues with them is
 relatively lower priority than investigating issues observed with WebKit
 tests. So I would recommend not mixing tests for WebKit regressions with 
 WPT
 tests - if your test eventually ends up in LayoutTests/imported, it will
 become a lot less effective.
>>> 
>>> 
>>> FWIW this is absolutely NOT how we're treating this in chromium.  If this
>>> is how things end up in practice then we will have failed massively in this
>>> effort.
>>> 
>>> We figure if we want the web to behave consistently, we really have no
>>> choice but to treat web-platform-tests as first class with all the
>>> discipline we give to our own tests.  As such we are actively moving many of
>>> our LayoutTests to web-platform-tests and depending entirely on the
>>> regression prevention they provide us from there.  Obviously there will be
>>> hiccups, but because our product quality will depend on web-platform-tests
>>> being an effective and non-flaky testsuite (and because we're starting to
>>> require most new features have web-platform-tests before they ship), I'm
>>> confident that we've got the incentives in place to lead to constant
>>> ratcheting up the engineering discipline and quality of the test suite.
>> 
>> 
>> FWIW, mozilla also treats WPT as first class tests.  We're not actively
>> moving old tests to WPT like google, but all new tests (at least in DOM) are
>> being written in WPT.  Of course, we do have exceptions for some tests that
>> require gecko-specific features (forcing GC, etc).
>> 
>> 
>> We don't have a concept of "first class", but I hope that when choosing
>> between looking into a simple test that was added for a known important bug,
>> and looking into an imported test whose importance is unclear, any WebKit
>> engineer will pick the former. And since no one can fix all the things, such
>> prioritization makes imported tests less effective.
> 
> This is absolutely not how I operate at all. Since almost all custom
> elements and shadow DOM API tests I wrote are written using
> testharness.js and upstreamed to web-platform-tests, they have been
> deleted from LayoutTests/fast/shadow-dom &
> LayoutTests/fast/custom-elements.
> 
> As such, if any new shadow DOM or custom elements tests under
> LayoutTests/imported/ start to fail, then we must fix them since we
> idon'thave any other test coverage.

Our normal approach to imported conformance test suites is to treat them as 
seriously as any other test. Even when a test was originally written along with 
a specific bug report, we don't necessarily think about that when prioritizing 
its importance, since it often fails in a way that does not make clear whether 
the exact origins bug is back.

It seems like there's two unusual things about WPT:
- We pull from upstream more often, and upstream is evolving at a good pace. So 
it's more of a moving target than something like the old W3C DOM test suite.
- At least according to Alexey, WPT tests are somewhat prone to flakiness in 
Safari.

It seems like the first issue is something we need to adapt to, to get the best 
value from WPT. But the second issue is something that has to be resolved 
within WPT (perhaps with our help). We can't have a lot of our testing depend 
on a flaky process.


This is separate from the issues about ease of running those tests. On that, I 
agree with Sam:

> On May 12, 2017, at 2:38 PM, Sam Weinig  wrote:
> 
> I regret piling on here, as I think this thread has diverged from it’s 
> original purpose, but…I understand this frustration. That said, perhaps this 
> is something we can solve with some tooling. For instance, a 
> run-test-in-safari (as a parallel to run-safari) script could be made which 
> starts the server, and then loads the test with the right URL in your built 
> Safari (or MiniBrowser, or whatever).  


I think the pain can be reduced with tooling. The right tools might need to be 
a bit more subtle. You might want to reload the test repeatedly in the same 
Safari instance, or perhaps load it into already-running Safari. So maybe 
load-test-in-safari (that ensures the server is running or launches it, then 
loads the right URL for a test, and maybe even does the right thing for http, 
web sockets or plain-file tests) would be closer to the mark. It may still be a 
little inconvenient but it seems like we can make it significantly better.

Regards,
Maciej

Re: [webkit-dev] Another WPT bite

[Alexey Proskuryakov]

> 12 мая 2017 г., в 14:38, Sam Weinig  написал(а):
> 
> I regret piling on here, as I think this thread has diverged from it’s 
> original purpose, but…I understand this frustration. That said, perhaps this 
> is something we can solve with some tooling. For instance, a 
> run-test-in-safari (as a parallel to run-safari) script could be made which 
> starts the server, and then loads the test with the right URL in your built 
> Safari (or MiniBrowser, or whatever). 


That's still not good enough, as this means that I can't point someone else to 
an instance of the test on trac.webkit.org to reproduce an issue. There is of 
course be another place to point to when/if the test gets upstreamed, but even 
that doesn't support stable links like trac does.

That's not to mention that learning the name of this proposed script is no 
easier than learning about run-webkit-httpd.

- Alexey

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Brian Burg]

> On May 12, 2017, at 2:10 PM, Simon Fraser  wrote:
> 
>> 
>> On May 12, 2017, at 12:04 PM, Alexey Proskuryakov > > wrote:
>> 
>> 
>>> 12 мая 2017 г., в 11:52, Ben Kelly >> > написал(а):
>>> 
>>> On Fri, May 12, 2017 at 2:26 PM, Rick Byers >> > wrote:
>>> On Fri, May 12, 2017 at 2:06 PM, Alexey Proskuryakov >> > wrote:
>>> Since imported WPT tests are very flaky, and are not necessarily written to 
>>> defend against important regressions, investigating issues with them is 
>>> relatively lower priority than investigating issues observed with WebKit 
>>> tests. So I would recommend not mixing tests for WebKit regressions with 
>>> WPT tests - if your test eventually ends up in LayoutTests/imported, it 
>>> will become a lot less effective.
>>> 
>>> FWIW this is absolutely NOT how we're treating this in chromium.  If this 
>>> is how things end up in practice then we will have failed massively in this 
>>> effort.
>>> 
>>> We figure if we want the web to behave consistently, we really have no 
>>> choice but to treat web-platform-tests as first class with all the 
>>> discipline we give to our own tests.  As such we are actively moving 
>>>  many of our LayoutTests to 
>>> web-platform-tests and depending entirely on the regression prevention they 
>>> provide us from there.  Obviously there will be hiccups, but because our 
>>> product quality will depend on web-platform-tests being an effective and 
>>> non-flaky testsuite (and because we're starting to require most new 
>>> features have web-platform-tests before they ship), I'm confident that 
>>> we've got the incentives in place to lead to constant ratcheting up the 
>>> engineering discipline and quality of the test suite.
>>> 
>>> FWIW, mozilla also treats WPT as first class tests.  We're not actively 
>>> moving old tests to WPT like google, but all new tests (at least in DOM) 
>>> are being written in WPT.  Of course, we do have exceptions for some tests 
>>> that require gecko-specific features (forcing GC, etc).
>> 
>> 
>> We don't have a concept of "first class", but I hope that when choosing 
>> between looking into a simple test that was added for a known important bug, 
>> and looking into an imported test whose importance is unclear, any WebKit 
>> engineer will pick the former. And since no one can fix all the things, such 
>> prioritization makes imported tests less effective.
> 
> I just ran into a classic example of how a WPT incurred more overhead. I made 
> a code change that broke 
> LayoutTests/imported/w3c/web-platform-tests/cssom-view/elementFromPoint.html. 
> I tried loading it in Safari and it doesn't run the test code because it 
> can't find /resources/testharness.js when loaded from a local file.
> 
> So then I have to figure out how to fire up the WPT server 
> (run-webkit-httpd), then figure out which host to use (localhost or 
> 128.0.0.1?) and which port, and finally to figure out the right path to the 
> test.

It sound like the pain point is just getting the http:// served test open in 
the browser. This workflow can and should be automated.
Maybe it could be an option passed to run-safari that does all the setup?

One thing I have considered authoring at some point is a catchall driver script 
for launching tests in a browser, attaching lldb to a layout test or API test, 
etc. I do all these things manually right now and it makes investigating test 
failures a drag. It would easiest to start a new script for these 
debug/triaging workflows than to than to try and tack it all onto 
./Tools/Scripts/run-safari.

> There's no reason this test should not work when loaded from file://.

If it’s easy to inspect a test served over http, then I am ambivalent. Loading 
over file:// protocol can have strange consequences, especially when CORS, 
caching, and resource loading are involved.

> 
> Simon
> 
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org 
> https://lists.webkit.org/mailman/listinfo/webkit-dev 
> 
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Sam Weinig]

> On May 12, 2017, at 2:10 PM, Simon Fraser  wrote:
> 
> 
>> On May 12, 2017, at 12:04 PM, Alexey Proskuryakov > > wrote:
>> 
>> 
>>> 12 мая 2017 г., в 11:52, Ben Kelly >> > написал(а):
>>> 
>>> On Fri, May 12, 2017 at 2:26 PM, Rick Byers >> > wrote:
>>> On Fri, May 12, 2017 at 2:06 PM, Alexey Proskuryakov >> > wrote:
>>> Since imported WPT tests are very flaky, and are not necessarily written to 
>>> defend against important regressions, investigating issues with them is 
>>> relatively lower priority than investigating issues observed with WebKit 
>>> tests. So I would recommend not mixing tests for WebKit regressions with 
>>> WPT tests - if your test eventually ends up in LayoutTests/imported, it 
>>> will become a lot less effective.
>>> 
>>> FWIW this is absolutely NOT how we're treating this in chromium.  If this 
>>> is how things end up in practice then we will have failed massively in this 
>>> effort.
>>> 
>>> We figure if we want the web to behave consistently, we really have no 
>>> choice but to treat web-platform-tests as first class with all the 
>>> discipline we give to our own tests.  As such we are actively moving 
>>>  many of our LayoutTests to 
>>> web-platform-tests and depending entirely on the regression prevention they 
>>> provide us from there.  Obviously there will be hiccups, but because our 
>>> product quality will depend on web-platform-tests being an effective and 
>>> non-flaky testsuite (and because we're starting to require most new 
>>> features have web-platform-tests before they ship), I'm confident that 
>>> we've got the incentives in place to lead to constant ratcheting up the 
>>> engineering discipline and quality of the test suite.
>>> 
>>> FWIW, mozilla also treats WPT as first class tests.  We're not actively 
>>> moving old tests to WPT like google, but all new tests (at least in DOM) 
>>> are being written in WPT.  Of course, we do have exceptions for some tests 
>>> that require gecko-specific features (forcing GC, etc).
>> 
>> 
>> We don't have a concept of "first class", but I hope that when choosing 
>> between looking into a simple test that was added for a known important bug, 
>> and looking into an imported test whose importance is unclear, any WebKit 
>> engineer will pick the former. And since no one can fix all the things, such 
>> prioritization makes imported tests less effective.
> 
> I just ran into a classic example of how a WPT incurred more overhead. I made 
> a code change that broke 
> LayoutTests/imported/w3c/web-platform-tests/cssom-view/elementFromPoint.html. 
> I tried loading it in Safari and it doesn't run the test code because it 
> can't find /resources/testharness.js when loaded from a local file.
> 
> So then I have to figure out how to fire up the WPT server 
> (run-webkit-httpd), then figure out which host to use (localhost or 
> 128.0.0.1?) and which port, and finally to figure out the right path to the 
> test.
> 
> There's no reason this test should not work when loaded from file://.

I regret piling on here, as I think this thread has diverged from it’s original 
purpose, but…I understand this frustration. That said, perhaps this is 
something we can solve with some tooling. For instance, a run-test-in-safari 
(as a parallel to run-safari) script could be made which starts the server, and 
then loads the test with the right URL in your built Safari (or MiniBrowser, or 
whatever).  

- Sam



___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Ryosuke Niwa]

On Fri, May 12, 2017 at 2:10 PM, Simon Fraser  wrote:
>
> On May 12, 2017, at 12:04 PM, Alexey Proskuryakov  wrote:
>
>
> 12 мая 2017 г., в 11:52, Ben Kelly  написал(а):
>
> On Fri, May 12, 2017 at 2:26 PM, Rick Byers  wrote:
>>
>> On Fri, May 12, 2017 at 2:06 PM, Alexey Proskuryakov 
>> wrote:
>>>
>>> Since imported WPT tests are very flaky, and are not necessarily written
>>> to defend against important regressions, investigating issues with them is
>>> relatively lower priority than investigating issues observed with WebKit
>>> tests. So I would recommend not mixing tests for WebKit regressions with WPT
>>> tests - if your test eventually ends up in LayoutTests/imported, it will
>>> become a lot less effective.
>>
>>
>> FWIW this is absolutely NOT how we're treating this in chromium.  If this
>> is how things end up in practice then we will have failed massively in this
>> effort.
>>
>> We figure if we want the web to behave consistently, we really have no
>> choice but to treat web-platform-tests as first class with all the
>> discipline we give to our own tests.  As such we are actively moving many of
>> our LayoutTests to web-platform-tests and depending entirely on the
>> regression prevention they provide us from there.  Obviously there will be
>> hiccups, but because our product quality will depend on web-platform-tests
>> being an effective and non-flaky testsuite (and because we're starting to
>> require most new features have web-platform-tests before they ship), I'm
>> confident that we've got the incentives in place to lead to constant
>> ratcheting up the engineering discipline and quality of the test suite.
>
>
> FWIW, mozilla also treats WPT as first class tests.  We're not actively
> moving old tests to WPT like google, but all new tests (at least in DOM) are
> being written in WPT.  Of course, we do have exceptions for some tests that
> require gecko-specific features (forcing GC, etc).
>
>
> We don't have a concept of "first class", but I hope that when choosing
> between looking into a simple test that was added for a known important bug,
> and looking into an imported test whose importance is unclear, any WebKit
> engineer will pick the former. And since no one can fix all the things, such
> prioritization makes imported tests less effective.
>
>
> I just ran into a classic example of how a WPT incurred more overhead. I
> made a code change that broke
> LayoutTests/imported/w3c/web-platform-tests/cssom-view/elementFromPoint.html.
> I tried loading it in Safari and it doesn't run the test code because it
> can't find /resources/testharness.js when loaded from a local file.
>
> So then I have to figure out how to fire up the WPT server
> (run-webkit-httpd), then figure out which host to use (localhost or
> 128.0.0.1?) and which port, and finally to figure out the right path to the
> test.
>
> There's no reason this test should not work when loaded from file://.
>

This is an issue we can solve. We can rewrite link elements to use
relative path so that you can just open in the browser.
It's something we've wanted to do but haven't gotten around to yet.

- R. Niwa
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Chris Dumez]

Our test importer script is perfectly able to rewrite those paths to use 
relative paths. However, Youenn, who imports and re-syncs most tests does not 
like this option I believe.
I think, part of the issue is that *some* tests do not do the right thing when 
loading over file:// (e.g. security ones) and it is not necessarily obvious 
just by looking at the test.

--
 Chris Dumez




> On May 12, 2017, at 2:10 PM, Simon Fraser  wrote:
> 
>> 
>> On May 12, 2017, at 12:04 PM, Alexey Proskuryakov > > wrote:
>> 
>> 
>>> 12 мая 2017 г., в 11:52, Ben Kelly >> > написал(а):
>>> 
>>> On Fri, May 12, 2017 at 2:26 PM, Rick Byers >> > wrote:
>>> On Fri, May 12, 2017 at 2:06 PM, Alexey Proskuryakov >> > wrote:
>>> Since imported WPT tests are very flaky, and are not necessarily written to 
>>> defend against important regressions, investigating issues with them is 
>>> relatively lower priority than investigating issues observed with WebKit 
>>> tests. So I would recommend not mixing tests for WebKit regressions with 
>>> WPT tests - if your test eventually ends up in LayoutTests/imported, it 
>>> will become a lot less effective.
>>> 
>>> FWIW this is absolutely NOT how we're treating this in chromium.  If this 
>>> is how things end up in practice then we will have failed massively in this 
>>> effort.
>>> 
>>> We figure if we want the web to behave consistently, we really have no 
>>> choice but to treat web-platform-tests as first class with all the 
>>> discipline we give to our own tests.  As such we are actively moving 
>>>  many of our LayoutTests to 
>>> web-platform-tests and depending entirely on the regression prevention they 
>>> provide us from there.  Obviously there will be hiccups, but because our 
>>> product quality will depend on web-platform-tests being an effective and 
>>> non-flaky testsuite (and because we're starting to require most new 
>>> features have web-platform-tests before they ship), I'm confident that 
>>> we've got the incentives in place to lead to constant ratcheting up the 
>>> engineering discipline and quality of the test suite.
>>> 
>>> FWIW, mozilla also treats WPT as first class tests.  We're not actively 
>>> moving old tests to WPT like google, but all new tests (at least in DOM) 
>>> are being written in WPT.  Of course, we do have exceptions for some tests 
>>> that require gecko-specific features (forcing GC, etc).
>> 
>> 
>> We don't have a concept of "first class", but I hope that when choosing 
>> between looking into a simple test that was added for a known important bug, 
>> and looking into an imported test whose importance is unclear, any WebKit 
>> engineer will pick the former. And since no one can fix all the things, such 
>> prioritization makes imported tests less effective.
> 
> I just ran into a classic example of how a WPT incurred more overhead. I made 
> a code change that broke 
> LayoutTests/imported/w3c/web-platform-tests/cssom-view/elementFromPoint.html. 
> I tried loading it in Safari and it doesn't run the test code because it 
> can't find /resources/testharness.js when loaded from a local file.
> 
> So then I have to figure out how to fire up the WPT server 
> (run-webkit-httpd), then figure out which host to use (localhost or 
> 128.0.0.1?) and which port, and finally to figure out the right path to the 
> test.
> 
> There's no reason this test should not work when loaded from file://.
> 
> Simon
> 
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org 
> https://lists.webkit.org/mailman/listinfo/webkit-dev 
> 
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Christian Biesinger]

For what it's worth, my personal solution to this was to put a symlink
in / for the resources directory:

lrwxrwxrwx 1 root root 55 Dec 11  2015 /resources ->
/home/cbiesinger/csswg-test/resources/

Hm, I guess I should update that to web-platform-tests now!

Christian

On Fri, May 12, 2017 at 5:10 PM, Simon Fraser  wrote:
>
> On May 12, 2017, at 12:04 PM, Alexey Proskuryakov  wrote:
>
>
> 12 мая 2017 г., в 11:52, Ben Kelly  написал(а):
>
> On Fri, May 12, 2017 at 2:26 PM, Rick Byers  wrote:
>>
>> On Fri, May 12, 2017 at 2:06 PM, Alexey Proskuryakov 
>> wrote:
>>>
>>> Since imported WPT tests are very flaky, and are not necessarily written
>>> to defend against important regressions, investigating issues with them is
>>> relatively lower priority than investigating issues observed with WebKit
>>> tests. So I would recommend not mixing tests for WebKit regressions with WPT
>>> tests - if your test eventually ends up in LayoutTests/imported, it will
>>> become a lot less effective.
>>
>>
>> FWIW this is absolutely NOT how we're treating this in chromium.  If this
>> is how things end up in practice then we will have failed massively in this
>> effort.
>>
>> We figure if we want the web to behave consistently, we really have no
>> choice but to treat web-platform-tests as first class with all the
>> discipline we give to our own tests.  As such we are actively moving many of
>> our LayoutTests to web-platform-tests and depending entirely on the
>> regression prevention they provide us from there.  Obviously there will be
>> hiccups, but because our product quality will depend on web-platform-tests
>> being an effective and non-flaky testsuite (and because we're starting to
>> require most new features have web-platform-tests before they ship), I'm
>> confident that we've got the incentives in place to lead to constant
>> ratcheting up the engineering discipline and quality of the test suite.
>
>
> FWIW, mozilla also treats WPT as first class tests.  We're not actively
> moving old tests to WPT like google, but all new tests (at least in DOM) are
> being written in WPT.  Of course, we do have exceptions for some tests that
> require gecko-specific features (forcing GC, etc).
>
>
> We don't have a concept of "first class", but I hope that when choosing
> between looking into a simple test that was added for a known important bug,
> and looking into an imported test whose importance is unclear, any WebKit
> engineer will pick the former. And since no one can fix all the things, such
> prioritization makes imported tests less effective.
>
>
> I just ran into a classic example of how a WPT incurred more overhead. I
> made a code change that broke
> LayoutTests/imported/w3c/web-platform-tests/cssom-view/elementFromPoint.html.
> I tried loading it in Safari and it doesn't run the test code because it
> can't find /resources/testharness.js when loaded from a local file.
>
> So then I have to figure out how to fire up the WPT server
> (run-webkit-httpd), then figure out which host to use (localhost or
> 128.0.0.1?) and which port, and finally to figure out the right path to the
> test.
>
> There's no reason this test should not work when loaded from file://.
>
> Simon
>
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Simon Fraser]

> On May 12, 2017, at 12:04 PM, Alexey Proskuryakov  wrote:
> 
> 
>> 12 мая 2017 г., в 11:52, Ben Kelly > > написал(а):
>> 
>> On Fri, May 12, 2017 at 2:26 PM, Rick Byers > > wrote:
>> On Fri, May 12, 2017 at 2:06 PM, Alexey Proskuryakov > > wrote:
>> Since imported WPT tests are very flaky, and are not necessarily written to 
>> defend against important regressions, investigating issues with them is 
>> relatively lower priority than investigating issues observed with WebKit 
>> tests. So I would recommend not mixing tests for WebKit regressions with WPT 
>> tests - if your test eventually ends up in LayoutTests/imported, it will 
>> become a lot less effective.
>> 
>> FWIW this is absolutely NOT how we're treating this in chromium.  If this is 
>> how things end up in practice then we will have failed massively in this 
>> effort.
>> 
>> We figure if we want the web to behave consistently, we really have no 
>> choice but to treat web-platform-tests as first class with all the 
>> discipline we give to our own tests.  As such we are actively moving 
>>  many of our LayoutTests to 
>> web-platform-tests and depending entirely on the regression prevention they 
>> provide us from there.  Obviously there will be hiccups, but because our 
>> product quality will depend on web-platform-tests being an effective and 
>> non-flaky testsuite (and because we're starting to require most new features 
>> have web-platform-tests before they ship), I'm confident that we've got the 
>> incentives in place to lead to constant ratcheting up the engineering 
>> discipline and quality of the test suite.
>> 
>> FWIW, mozilla also treats WPT as first class tests.  We're not actively 
>> moving old tests to WPT like google, but all new tests (at least in DOM) are 
>> being written in WPT.  Of course, we do have exceptions for some tests that 
>> require gecko-specific features (forcing GC, etc).
> 
> 
> We don't have a concept of "first class", but I hope that when choosing 
> between looking into a simple test that was added for a known important bug, 
> and looking into an imported test whose importance is unclear, any WebKit 
> engineer will pick the former. And since no one can fix all the things, such 
> prioritization makes imported tests less effective.

I just ran into a classic example of how a WPT incurred more overhead. I made a 
code change that broke 
LayoutTests/imported/w3c/web-platform-tests/cssom-view/elementFromPoint.html. I 
tried loading it in Safari and it doesn't run the test code because it can't 
find /resources/testharness.js when loaded from a local file.

So then I have to figure out how to fire up the WPT server (run-webkit-httpd), 
then figure out which host to use (localhost or 128.0.0.1?) and which port, and 
finally to figure out the right path to the test.

There's no reason this test should not work when loaded from file://.

Simon

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[youenn fablet]

Filed https://github.com/w3c/web-platform-tests/issues/5909 and
https://github.com/w3c/web-platform-tests/issues/5910.


Le ven. 12 mai 2017 à 12:08, Rick Byers  a écrit :

> On Fri, May 12, 2017 at 2:43 PM, youenn fablet  wrote:
>
>>
>> Le ven. 12 mai 2017 à 11:07, Alexey Proskuryakov  a
>> écrit :
>>
>>>
>>> 9 мая 2017 г., в 11:27, Simon Fraser 
>>> написал(а):
>>>
>>>
>>> Another consideration here is "would my test be useful for other browser
>>> vendors". I don't think the answer is a unanimous "yes", so I think we
>>> should only use WPT for tests that will think are worth sharing.
>>>
>>>
>>> Since imported WPT tests are very flaky, and are not necessarily written
>>> to defend against important regressions, investigating issues with them is
>>> relatively lower priority than investigating issues observed with WebKit
>>> tests. So I would recommend not mixing tests for WebKit regressions with
>>> WPT tests - if your test eventually ends up in LayoutTests/imported, it
>>> will become a lot less effective.
>>>
>>
>> WPT tests are flaky in WebKit because WPT stability bots do not run yet
>> Safari, and most tests are written in non-WebKit environment.
>> Often, it is due to the fact that tests are not passing and flakiness is
>> only seen with failing assertions.
>>
>> From my experience with fetch API tests, I disagree that broken WPT tests
>> are lower priority.
>> I think it will change as more WebKit contributors will author WPT tests.
>>
>> I agree that tests doing subtle WebKit-specific regression checks are not
>> good candidates for WPT upstream.
>> When the test is all about conformance with a spec, it is a very good
>> candidate.
>>
>>
>>> Using the complicated harness has a similar consequence - if you use any
>>> WPT goodies like templates or server side scripts, the cost of
>>> investigating issues on the test increases, and makes the test less
>>> valuable.
>>>
>>
>> It is true that WPT put some emphasis on easing authoring of tests.
>> I guess there is a learning curve here in WPT test debugging.
>>
>> If you have a file with 20 tests, it is harder to debug.
>> It is also increasing the chances for flakiness/timeouts.
>> Maybe we could send that feedback.
>>
>> WPT infra could also be improved: more verbose debug-only output,
>> enabling running selected subtest only...
>> testharness.js is actively maintained and is continuously improving.
>> Since we have specific requirements as you are describing, we should push
>> them.
>>
>
> Concretely, please file issues with the "infra" label
> 
>  to
> track upstream WPT infrastructure bugs and feature requests.  Google has an
> ongoing contract with Bocoup (Bob and Mike cc'd) who are improving the
> infrastructure every day.  Of course there's an opportunity to make it even
> better fast with more funding from additional organizations who would
> benefit :-)
>
>>
>> I don't know if there is any way to adopt WPT that won't make testing
>>> less effective. WPT tests may be useful in very rare cases, where we
>>> actively want to communicate certain subtle behavior details to other
>>> vendors - but even so, I imagine that other vendors may not put high
>>> priority on those, for the same reasons.
>>>
>>
>> My own experience is that WPT tests are actually very valuable, at least
>> when we are talking about interoperability/spec conformance.
>> I also see WPT as an efficient way to author tests.
>>
>>
>>>
>>> - Alexey
>>>
>>> ___
>>> webkit-dev mailing list
>>> webkit-dev@lists.webkit.org
>>> https://lists.webkit.org/mailman/listinfo/webkit-dev
>>>
>>
>> ___
>> webkit-dev mailing list
>> webkit-dev@lists.webkit.org
>> https://lists.webkit.org/mailman/listinfo/webkit-dev
>>
>>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Rick Byers]

On Fri, May 12, 2017 at 2:43 PM, youenn fablet  wrote:

>
> Le ven. 12 mai 2017 à 11:07, Alexey Proskuryakov  a écrit :
>
>>
>> 9 мая 2017 г., в 11:27, Simon Fraser  написал(а):
>>
>>
>> Another consideration here is "would my test be useful for other browser
>> vendors". I don't think the answer is a unanimous "yes", so I think we
>> should only use WPT for tests that will think are worth sharing.
>>
>>
>> Since imported WPT tests are very flaky, and are not necessarily written
>> to defend against important regressions, investigating issues with them is
>> relatively lower priority than investigating issues observed with WebKit
>> tests. So I would recommend not mixing tests for WebKit regressions with
>> WPT tests - if your test eventually ends up in LayoutTests/imported, it
>> will become a lot less effective.
>>
>
> WPT tests are flaky in WebKit because WPT stability bots do not run yet
> Safari, and most tests are written in non-WebKit environment.
> Often, it is due to the fact that tests are not passing and flakiness is
> only seen with failing assertions.
>
> From my experience with fetch API tests, I disagree that broken WPT tests
> are lower priority.
> I think it will change as more WebKit contributors will author WPT tests.
>
> I agree that tests doing subtle WebKit-specific regression checks are not
> good candidates for WPT upstream.
> When the test is all about conformance with a spec, it is a very good
> candidate.
>
>
>> Using the complicated harness has a similar consequence - if you use any
>> WPT goodies like templates or server side scripts, the cost of
>> investigating issues on the test increases, and makes the test less
>> valuable.
>>
>
> It is true that WPT put some emphasis on easing authoring of tests.
> I guess there is a learning curve here in WPT test debugging.
>
> If you have a file with 20 tests, it is harder to debug.
> It is also increasing the chances for flakiness/timeouts.
> Maybe we could send that feedback.
>
> WPT infra could also be improved: more verbose debug-only output, enabling
> running selected subtest only...
> testharness.js is actively maintained and is continuously improving.
> Since we have specific requirements as you are describing, we should push
> them.
>

Concretely, please file issues with the "infra" label

to
track upstream WPT infrastructure bugs and feature requests.  Google has an
ongoing contract with Bocoup (Bob and Mike cc'd) who are improving the
infrastructure every day.  Of course there's an opportunity to make it even
better fast with more funding from additional organizations who would
benefit :-)

>
> I don't know if there is any way to adopt WPT that won't make testing less
>> effective. WPT tests may be useful in very rare cases, where we actively
>> want to communicate certain subtle behavior details to other vendors - but
>> even so, I imagine that other vendors may not put high priority on those,
>> for the same reasons.
>>
>
> My own experience is that WPT tests are actually very valuable, at least
> when we are talking about interoperability/spec conformance.
> I also see WPT as an efficient way to author tests.
>
>
>>
>> - Alexey
>>
>> ___
>> webkit-dev mailing list
>> webkit-dev@lists.webkit.org
>> https://lists.webkit.org/mailman/listinfo/webkit-dev
>>
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Alexey Proskuryakov]

> 12 мая 2017 г., в 11:52, Ben Kelly  написал(а):
> 
> On Fri, May 12, 2017 at 2:26 PM, Rick Byers  > wrote:
> On Fri, May 12, 2017 at 2:06 PM, Alexey Proskuryakov  > wrote:
> Since imported WPT tests are very flaky, and are not necessarily written to 
> defend against important regressions, investigating issues with them is 
> relatively lower priority than investigating issues observed with WebKit 
> tests. So I would recommend not mixing tests for WebKit regressions with WPT 
> tests - if your test eventually ends up in LayoutTests/imported, it will 
> become a lot less effective.
> 
> FWIW this is absolutely NOT how we're treating this in chromium.  If this is 
> how things end up in practice then we will have failed massively in this 
> effort.
> 
> We figure if we want the web to behave consistently, we really have no choice 
> but to treat web-platform-tests as first class with all the discipline we 
> give to our own tests.  As such we are actively moving 
>  many of our LayoutTests to 
> web-platform-tests and depending entirely on the regression prevention they 
> provide us from there.  Obviously there will be hiccups, but because our 
> product quality will depend on web-platform-tests being an effective and 
> non-flaky testsuite (and because we're starting to require most new features 
> have web-platform-tests before they ship), I'm confident that we've got the 
> incentives in place to lead to constant ratcheting up the engineering 
> discipline and quality of the test suite.
> 
> FWIW, mozilla also treats WPT as first class tests.  We're not actively 
> moving old tests to WPT like google, but all new tests (at least in DOM) are 
> being written in WPT.  Of course, we do have exceptions for some tests that 
> require gecko-specific features (forcing GC, etc).


We don't have a concept of "first class", but I hope that when choosing between 
looking into a simple test that was added for a known important bug, and 
looking into an imported test whose importance is unclear, any WebKit engineer 
will pick the former. And since no one can fix all the things, such 
prioritization makes imported tests less effective.

- Alexey

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Ben Kelly]

On Fri, May 12, 2017 at 2:26 PM, Rick Byers  wrote:

> On Fri, May 12, 2017 at 2:06 PM, Alexey Proskuryakov 
> wrote:
>
>> Since imported WPT tests are very flaky, and are not necessarily written
>> to defend against important regressions, investigating issues with them is
>> relatively lower priority than investigating issues observed with WebKit
>> tests. So I would recommend not mixing tests for WebKit regressions with
>> WPT tests - if your test eventually ends up in LayoutTests/imported, it
>> will become a lot less effective.
>>
>
> FWIW this is absolutely NOT how we're treating this in chromium.  If this
> is how things end up in practice then we will have failed massively in this
> effort.
>
> We figure if we want the web to behave consistently, we really have no
> choice but to treat web-platform-tests as first class with all the
> discipline we give to our own tests.  As such we are actively moving
>  many of our LayoutTests to
> web-platform-tests and depending entirely on the regression prevention they
> provide us from there.  Obviously there will be hiccups, but because our
> product quality will depend on web-platform-tests being an effective and
> non-flaky testsuite (and because we're starting to require most new
> features have web-platform-tests before they ship), I'm confident that
> we've got the incentives in place to lead to constant ratcheting up the
> engineering discipline and quality of the test suite.
>

FWIW, mozilla also treats WPT as first class tests.  We're not actively
moving old tests to WPT like google, but all new tests (at least in DOM)
are being written in WPT.  Of course, we do have exceptions for some tests
that require gecko-specific features (forcing GC, etc).
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[youenn fablet]

Le ven. 12 mai 2017 à 11:07, Alexey Proskuryakov  a écrit :

>
> 9 мая 2017 г., в 11:27, Simon Fraser  написал(а):
>
>
> Another consideration here is "would my test be useful for other browser
> vendors". I don't think the answer is a unanimous "yes", so I think we
> should only use WPT for tests that will think are worth sharing.
>
>
> Since imported WPT tests are very flaky, and are not necessarily written
> to defend against important regressions, investigating issues with them is
> relatively lower priority than investigating issues observed with WebKit
> tests. So I would recommend not mixing tests for WebKit regressions with
> WPT tests - if your test eventually ends up in LayoutTests/imported, it
> will become a lot less effective.
>

WPT tests are flaky in WebKit because WPT stability bots do not run yet
Safari, and most tests are written in non-WebKit environment.
Often, it is due to the fact that tests are not passing and flakiness is
only seen with failing assertions.

>From my experience with fetch API tests, I disagree that broken WPT tests
are lower priority.
I think it will change as more WebKit contributors will author WPT tests.

I agree that tests doing subtle WebKit-specific regression checks are not
good candidates for WPT upstream.
When the test is all about conformance with a spec, it is a very good
candidate.


> Using the complicated harness has a similar consequence - if you use any
> WPT goodies like templates or server side scripts, the cost of
> investigating issues on the test increases, and makes the test less
> valuable.
>

It is true that WPT put some emphasis on easing authoring of tests.
I guess there is a learning curve here in WPT test debugging.

If you have a file with 20 tests, it is harder to debug.
It is also increasing the chances for flakiness/timeouts.
Maybe we could send that feedback.

WPT infra could also be improved: more verbose debug-only output, enabling
running selected subtest only...
testharness.js is actively maintained and is continuously improving.
Since we have specific requirements as you are describing, we should push
them.

I don't know if there is any way to adopt WPT that won't make testing less
> effective. WPT tests may be useful in very rare cases, where we actively
> want to communicate certain subtle behavior details to other vendors - but
> even so, I imagine that other vendors may not put high priority on those,
> for the same reasons.
>

My own experience is that WPT tests are actually very valuable, at least
when we are talking about interoperability/spec conformance.
I also see WPT as an efficient way to author tests.


>
> - Alexey
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Rick Byers]

On Fri, May 12, 2017 at 2:06 PM, Alexey Proskuryakov  wrote:

>
> 9 мая 2017 г., в 11:27, Simon Fraser  написал(а):
>
> Another consideration here is "would my test be useful for other browser
> vendors". I don't think the answer is a unanimous "yes", so I think we
> should only use WPT for tests that will think are worth sharing.
>
>
> Since imported WPT tests are very flaky, and are not necessarily written
> to defend against important regressions, investigating issues with them is
> relatively lower priority than investigating issues observed with WebKit
> tests. So I would recommend not mixing tests for WebKit regressions with
> WPT tests - if your test eventually ends up in LayoutTests/imported, it
> will become a lot less effective.
>

FWIW this is absolutely NOT how we're treating this in chromium.  If this
is how things end up in practice then we will have failed massively in this
effort.

We figure if we want the web to behave consistently, we really have no
choice but to treat web-platform-tests as first class with all the
discipline we give to our own tests.  As such we are actively moving
 many of our LayoutTests to
web-platform-tests and depending entirely on the regression prevention they
provide us from there.  Obviously there will be hiccups, but because our
product quality will depend on web-platform-tests being an effective and
non-flaky testsuite (and because we're starting to require most new
features have web-platform-tests before they ship), I'm confident that
we've got the incentives in place to lead to constant ratcheting up the
engineering discipline and quality of the test suite.

Pragmatically to get there incrementally we're applying different policies
to different WPT directories.  Eg. for some directories if a WPT import
results in new failures we probably won't care enough to prioritize
investigating them.  But for many (which map to a feature team on the hook
for quality in that area) we'll treat it with the same priority as a newly
failing LayoutTest on our bots.

Of course the WebKit project might have a higher test quality bar than the
chromium project, so the right choice could be incredibly different for you
than for us.  But should you choose to treat WPT as a first-class test
suite for your project, we're committed to working together to make it work
well in practice for all of us.  We figure that in the long-run, if we
shift investments from our own LayoutTests to web-platform-tests, even
though there will be some additional costs and risks, it'll eventually lead
to both lower testing costs and higher product quality because we're
leveraging the work of spec editors like Anne and other implementors who
are also working to improve the test suite.

Using the complicated harness has a similar consequence - if you use any
> WPT goodies like templates or server side scripts, the cost of
> investigating issues on the test increases, and makes the test less
> valuable.
>
> I don't know if there is any way to adopt WPT that won't make testing less
> effective. WPT tests may be useful in very rare cases, where we actively
> want to communicate certain subtle behavior details to other vendors - but
> even so, I imagine that other vendors may not put high priority on those,
> for the same reasons.
>
> - Alexey
>
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Alexey Proskuryakov]

> 9 мая 2017 г., в 11:27, Simon Fraser  написал(а):
> 
> Another consideration here is "would my test be useful for other browser 
> vendors". I don't think the answer is a unanimous "yes", so I think we should 
> only use WPT for tests that will think are worth sharing.

Since imported WPT tests are very flaky, and are not necessarily written to 
defend against important regressions, investigating issues with them is 
relatively lower priority than investigating issues observed with WebKit tests. 
So I would recommend not mixing tests for WebKit regressions with WPT tests - 
if your test eventually ends up in LayoutTests/imported, it will become a lot 
less effective.

Using the complicated harness has a similar consequence - if you use any WPT 
goodies like templates or server side scripts, the cost of investigating issues 
on the test increases, and makes the test less valuable.

I don't know if there is any way to adopt WPT that won't make testing less 
effective. WPT tests may be useful in very rare cases, where we actively want 
to communicate certain subtle behavior details to other vendors - but even so, 
I imagine that other vendors may not put high priority on those, for the same 
reasons.

- Alexey

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Ryosuke Niwa]

On Thu, May 11, 2017 at 11:32 PM, Anne van Kesteren  wrote:
> On Fri, May 12, 2017 at 6:46 AM, Simon Fraser  wrote:
>> I was under the impression that tests upstreamed from vendor repositories 
>> would land in WPT tests
>> with minimal review, based on the fact that they had been reviewed when 
>> landed in the vendor repo.
>
> I think that's under the expectation that those reviewing WPT changes
> have some familiarity with the structure of WPT.
>
>
>> What's to stop 4 vendors from upstreaming similar but non-identical tests 
>> for a given feature?
>
> Usually when a new feature is in development developers from the
> various vendors use the relevant standards forum to coordinate on
> testing and avoid duplication.

Also, if someone is fixing a bug or implementing a feature with a new
test, and some other starts passing / failing, then the person would
be compelled to either modify the existing test or de-duplicating the
test.

We all have to be diligent in this regard but it's not much worse than
what we have to do in WebKit to avoid keep adding similar tests.

- R. Niwa
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Anne van Kesteren]

On Fri, May 12, 2017 at 6:46 AM, Simon Fraser  wrote:
> I was under the impression that tests upstreamed from vendor repositories 
> would land in WPT tests
> with minimal review, based on the fact that they had been reviewed when 
> landed in the vendor repo.

I think that's under the expectation that those reviewing WPT changes
have some familiarity with the structure of WPT.


> What's to stop 4 vendors from upstreaming similar but non-identical tests for 
> a given feature?

Usually when a new feature is in development developers from the
various vendors use the relevant standards forum to coordinate on
testing and avoid duplication.


-- 
https://annevankesteren.nl/
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Simon Fraser]

> On May 11, 2017, at 9:30 PM, Anne van Kesteren  wrote:
> 
> On Tue, May 9, 2017 at 8:27 PM, Simon Fraser  wrote:
>> I'm also concerned that with 4 vendors upstreaming their WPT tests, the WPT
>> repo will just become a morass of partially overlapping tests that takes 4x
>> longer to run than a curated repo.
> 
> Why do you think WPT is not curated? It's an actively maintained test
> suite. Treating it as a dumping ground would be a mistake I think.
> 
> For WHATWG standards we make sure to update corresponding tests for
> each change to the standard as well at which point any needed cleanup
> occurs (we didn't start out with this process unfortunately so there's
> definitely some cleanup left to be done here and there, but that's
> true for each software project).

I was under the impression that tests upstreamed from vendor repositories would 
land in WPT tests
with minimal review, based on the fact that they had been reviewed when landed 
in the vendor repo.

What's to stop 4 vendors from upstreaming similar but non-identical tests for a 
given feature?

Simon

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Anne van Kesteren]

On Tue, May 9, 2017 at 8:27 PM, Simon Fraser  wrote:
> I'm also concerned that with 4 vendors upstreaming their WPT tests, the WPT
> repo will just become a morass of partially overlapping tests that takes 4x
> longer to run than a curated repo.

Why do you think WPT is not curated? It's an actively maintained test
suite. Treating it as a dumping ground would be a mistake I think.

For WHATWG standards we make sure to update corresponding tests for
each change to the standard as well at which point any needed cleanup
occurs (we didn't start out with this process unfortunately so there's
definitely some cleanup left to be done here and there, but that's
true for each software project).


-- 
https://annevankesteren.nl/
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[youenn fablet]

Patch just landed.
Location is LayoutTests/http/wpt.
I also forgot to say that, should you want to write http served tests, it
might make sense to use this folder instead of http/tests.
   Y

Le jeu. 11 mai 2017 à 19:05, Sam Weinig  a écrit :

>
> On May 8, 2017, at 9:31 PM, youenn fablet  wrote:
>
> Hi all,
>
> Discussing with some WebKittens, testharness.js is more and more used in
> WebKit.
> Is it time to make testharness.js the recommended way of writing
> LayoutTests?
>
>
> I am in favor of this. If we simplified the question to some form of, “do
> we really need both testharness.js/testharnessreport.js and 
> js-test-pre.js/js-test-post.js?”
> I am even more in favor, as having two test harnesses seems unnecessary,
> cumbersome and unfriendly to new contributors,
>
> Do I think all tests should use testharness.js? No. Just as currently I
> don’t think all tests should use testharness.js/testharnessreport.js. But
> for many tests of new web platform features, it seems quite reasonable to
> start using this harness, as the benefits, which include a good feature
> set, easier interoperability with other browsers, and a reduced cost to
> upstreaming to web-platform-tests, out weigh the costs, leaning something
> new (there are probably other costs I am forgetting).
>
>
> To continue moving forward, some of us are proposing to serve all tests in
> LayoutTests/wpt through the WPT server [1].
> This would serve some purposes like increasing the use of WPT goodies:
> file-specific headers, templated tests (*.any.js), IDLParser, server-side
> scripts...
> It could also ease test migration from WebKit to W3C WPT.
>
>
> This seems uncontroversial and great to me (which would make sense since I
> asked you if we could do it).  It’s just a new directory, like
> LayoutTests/http where we can put tests that use the WPT server.
>
> - Sam
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Sam Weinig]

> On May 8, 2017, at 9:31 PM, youenn fablet  wrote:
> 
> Hi all,
> 
> Discussing with some WebKittens, testharness.js is more and more used in 
> WebKit.
> Is it time to make testharness.js the recommended way of writing LayoutTests?

I am in favor of this. If we simplified the question to some form of, “do we 
really need both testharness.js/testharnessreport.js and 
js-test-pre.js/js-test-post.js?” I am even more in favor, as having two test 
harnesses seems unnecessary, cumbersome and unfriendly to new contributors,

Do I think all tests should use testharness.js? No. Just as currently I don’t 
think all tests should use testharness.js/testharnessreport.js. But for many 
tests of new web platform features, it seems quite reasonable to start using 
this harness, as the benefits, which include a good feature set, easier 
interoperability with other browsers, and a reduced cost to upstreaming to 
web-platform-tests, out weigh the costs, leaning something new (there are 
probably other costs I am forgetting).


> To continue moving forward, some of us are proposing to serve all tests in 
> LayoutTests/wpt through the WPT server [1].
> This would serve some purposes like increasing the use of WPT goodies: 
> file-specific headers, templated tests (*.any.js), IDLParser, server-side 
> scripts...
> It could also ease test migration from WebKit to W3C WPT.

This seems uncontroversial and great to me (which would make sense since I 
asked you if we could do it).  It’s just a new directory, like LayoutTests/http 
where we can put tests that use the WPT server.  

- Sam___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[youenn fablet]

> Another consideration here is "would my test be useful for other browser
> vendors". I don't think the answer is a unanimous "yes", so I think we
> should only use WPT for tests that will think are worth sharing.
>

Agreed that some tests, especially the ones dedicated to WebKit
specificities should be kept in WebKit.
The risk is that these changes might be changed upstreamed in
small/stylistic ways and no longer cover the initial case.

That does not prevent to use WPT infrastructure for these tests, especially
if it eases authoring.
As a temporary measure, Chris mentioned the possibility to have inside the
WPT folder a folder dedicated to "to-be-upstreamed" tests.


> I'm also concerned that with 4 vendors upstreaming their WPT tests, the
> WPT repo will just become a morass of partially overlapping tests that
> takes 4x longer to run than a curated repo.
>

This is an issue in WebKit repository as well.
I saw some clean-up done in WPT though.
Sharing this goal and effort amongst the different communities might end up
with a better result.
Note also that spec editors are usually involved in WPT, which might help
organizing the test suites in better ways.


Simon
>
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Mike Pennisi]

> One question I have is whether web platform tests can run under a regular
> HTTP server (maybe with appropriate configuration) or do we need something
> special? Is the WPT server more than just a web server with specific
> configuration settings?

While many tests can probably be run in this way, many others are dependent
on the functionality of the Web Platform Test Server. It's written in Python
and supports (among other things) arbitrary HTTP request handling via
standalone Python scripts. Its capabilities are documented here:

https://wptserve.readthedocs.io/en/latest/

Like with `testharness.js`, this additional infrastructure is a case of
"simplicity versus power."

> Another consideration here is "would my test be useful for other browser
> vendors". I don't think the answer is a unanimous "yes", so I think we
> should only use WPT for tests that will think are worth sharing.

This is a good point. I'm currently migrating service worker tests from
Chromium to WPT, and I'm frequently coming across tests (or sometimes
specific assertions) that aren't appropriate for WPT.

> I'm also concerned that with 4 vendors upstreaming their WPT tests, the
> WPT repo will just become a morass of partially overlapping tests that
> takes 4x longer to run than a curated repo.

This gets at another consideration that hasn't been mentioned yet:
participating in WPT like this expands the responsibilities of authors and
reviewers.  Contributors may have to spend extra time cataloging their new
tests (or finding the correct test to extend). Likewise, reviewers will need
to develop some familiarity with certain sub-directories of WPT.

This can be kind of a drag, but only if you aren't committed to the
long-term benefits of sharing tests. From that perspective, meaningful
organization is just another part of the job, and the value accrues with
every new test.

Which is all to say: buy-in is important. If the team decides to participate in
WPT, folks ought to be excited about it!
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Ryosuke Niwa]

Forgot to CC webkit-dev.
- R. Niwa


On Tue, May 9, 2017 at 12:36 PM, Ryosuke Niwa  wrote:
> On Tue, May 9, 2017 at 1:12 AM, Maciej Stachowiak  wrote:
>>
>>
>> On May 8, 2017, at 11:15 PM, Ryosuke Niwa  wrote:
>>
>> On Mon, May 8, 2017 at 11:01 PM, Brady Eidson  wrote:
>>
>> On May 8, 2017, at 10:44 PM, Ryosuke Niwa  wrote:
>> On Mon, May 8, 2017 at 10:17 PM, Brady Eidson  wrote:
>>
>> But now talking about testharness.js directly, I object on the grounds of "a
>> file:// regression test is dirt easy to hack on and work with, whereas
>> anything that requires me to have an httpd running is a PITA"
>>
>> I think whether we use file:// or http:// is orthogonal point to using
>> testharness.js.  Many of the tests Chris and I have written using
>> testharness.js are checked into regular LayoutTests/ directories, and
>> they work just fine.
>>
>> Yes, I misunderstood this in Youenn's original message. Good to know!
>>
>> I just object to making it the "recommended way" of writing tests.
>>
>> Would you equally object to making js-test.js / js-test-pre.js the
>> recommended way of writing tests?
>>
>> Yes.
>>
>> If not, why?
>>
>> N/A
>>
>> What we're suggesting is to give preferential treatments to
>> testharness.js over js-test.js / js-test-pre.js when you were already
>> planning to write a test with the latter two scripts.
>>
>> "It's okay to write your test however you'd like. If you were considering
>> using js-test, maybe you should consider using testharness instead."
>> Is that's what's being proposed?
>>
>>
>> On May 8, 2017, at 10:44 PM, Ryosuke Niwa  wrote:
>>
>> On Mon, May 8, 2017 at 10:17 PM, Brady Eidson  wrote:
>>
>> But now talking about testharness.js directly, I object on the grounds of "a
>> file:// regression test is dirt easy to hack on and work with, whereas
>> anything that requires me to have an httpd running is a PITA"
>>
>>
>> I think whether we use file:// or http:// is orthogonal point to using
>> testharness.js.  Many of the tests Chris and I have written using
>> testharness.js are checked into regular LayoutTests/ directories, and
>> they work just fine.
>>
>>
>> Yes, I misunderstood this in Youenn's original message. Good to know!
>>
>>
>> I just object to making it the "recommended way" of writing tests.
>>
>>
>> Would you equally object to making js-test.js / js-test-pre.js the
>> recommended way of writing tests?
>>
>>
>> Yes.
>>
>> If not, why?
>>
>>
>> N/A
>>
>> What we're suggesting is to give preferential treatments to
>> testharness.js over js-test.js / js-test-pre.js when you were already
>> planning to write a test with the latter two scripts.
>>
>>
>> "It's okay to write your test however you'd like. If you were considering
>> using js-test, maybe you should consider using testharness instead."
>>
>> Is that's what's being proposed?
>>
>>
>> Besides other issues mentioned, testharness tends to result in more verbose
>> tests compared to js-test, at least for simple cases.
>>
>>
>> The thing I specifically asked Youenn to ask is, whether we should
>> place a test inside LayoutTests/wpt like LayoutTests/http/tests when
>> we want to write a test using testharness.js which requires some sort
>> of network code.
>>
>> Since people have had some opinions about directory structures in the past.
>>
>>
>> It seems like we need a few different directories, here are my opinions on
>> them:
>>
>> (1) Imported web platform tests that don't need a server
>> Currently LayoutTests/imported/w3c/web-platform-tests, which seems fine.
>> (2) Imported web platform tests that do need a server
>> Probably should be under LayoutTests/imported/w3c/ somewhere, or maybe
>
> I don't think it's a good idea to split web-platform-tests into two
> separate directories like that because I'd imagine there are resource
> dependencies between tests.  Note that this whole process of detecting
> & separating tests that can be ran locally is the work we'd have to
> do, and upstream won't maintain it.  Given that, we should be making
> the minimal amount of differences to the upstream directory structure
> as well.
>
> Granted, this would make it harder to know which tests require a
> server and which one doesn't. Perhaps this is bad enough that we need
> two directories after all but perhaps there is some other way to solve
> this problem like adding suffix to test names. (Although some ref
> tests in web-platform-tests use other tests as expectation but ref
> tests don't tend to require servers anyway; I know this is a terrible
> idea but we don't really have a control over what upstream does).
>
>> under http/tests/ per point (4)
>> (3) Custom testharness.js tests that don't need a server
>> Probably these should just go in their normal topic-specific directories
>> and should not need a special directory
>> (4) Custom testharness.js tests that do 

Re: [webkit-dev] Another WPT bite

[Ryosuke Niwa]

On Tue, May 9, 2017 at 11:06 AM, Maciej Stachowiak 
>
> If we run all the w3c-imported web platform tests under a web server, then
> obviously we only need one directory. My understanding is that we don't run
> them under a server at all. So it seems like one part of this proposal is
> "run everything under LayoutTests/imported/w3c/ from a web server".

We currently run all web-platform-tests using WPT server. In theory,
there's nothing preventing us from running these tests using file://
so long as tests don't rely on the networking aspect. However, we
don't currently rewrite URLs to testharness.js and
testharness-report.js both of which refers to the top-level directory
at the moment.

IMO, the entirety of the effort to make these imported tests runnable
in file scheme should be about re-enabling these URL rewrites that got
broken sometime ago.

>
> I think it would be cleaner to have a separate directory of tests intended
> for import, separate from imported tests.

You mean intended for exports?
> It could be right next to imported/w3c/web-platform-tests. I think mixing 
> tests that are imported from
> upstream and tests intended for eventual upstreaming is more confusing than
> helpful.

Okay. I've actually advocated for this approach for the same reason.

One annoying thing in this approach is that if we wanted to add a test
to, say, html/infrastructure/conformance-requirements/extensibility/
then we either have to mirror the same directory structure in this
upstream directory, or add some sort of WebKit-only annotation that
this test needs to be added there for the upstream process to be
automated.

>
> One question I have is whether web platform tests can run under a regular
> HTTP server (maybe with appropriate configuration) or do we need something
> special? Is the WPT server more than just a web server with specific
> configuration settings?

No. Most of web-platform-tests that do require HTTP access have an
accompanying Python file which use WPT server's functionality as far
as I know.

In fact, I'd even argue that we should try to use the WPT server for
writing tests for Web APIs and security checks since those tests would
be extremely valuable for other browser vendors as well.

- R. Niwa
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Maciej Stachowiak]

> On May 9, 2017, at 8:11 AM, Geoffrey Garen  wrote:
> 
>> What we're suggesting is to give preferential treatments to
>> testharness.js over js-test.js / js-test-pre.js when you were already
>> planning to write a test with the latter two scripts.
> 
> OK, I think this makes sense.
> 
> But I still think the very best kind of test is a flat file with 10-20 lines 
> of code in it. Particularly for debugging JavaScript issues, large wrapper 
> frameworks get in the way.
> 
>> - Tests would be more easily upstreamable to web-platform-tests, which are 
>> run by all major browser engines. This would help a lot in terms of 
>> interoperability. As previously discussed, Gecko and Blink already do 
>> automated export of tests to web-platform-tests. I believe we should do in 
>> the same direction and contribute more tests back.
> 
> I wonder why these other projects do automated export instead of 
> incorporating testharness.js directly.

I don't think that's an "also", not an "instead". My understanding is that they 
do two-way sync with the web-platform-tests GitHub, so there's a process for 
downloading tests and upstreaming tests authored by their team. But they still 
have their own copy.

Regards,
Maciej

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Maciej Stachowiak]

> On May 9, 2017, at 8:44 AM, youenn fablet  wrote:
> 
> 
> Besides other issues mentioned, testharness tends to result in more verbose 
> tests compared to js-test, at least for simple cases.
> 
> For synchronous tests, I am not sure there is any big difference one way or 
> the other.
> With asynchronous tests, it might be true, but using 
> testharness.js/promise_test usually improves things there.
>  I personally find it easier to not wrap code-to-be-tested into quotes.

It seems like it's a matter of personal preference to some extent. So I am not 
sure we should recommend one or the other (besides for tests that are intended 
to be contributed to web-platform-tests).

> 
> Another concern is the lack of verbose output which reduces the ability to 
> debug failing tests.
> This can be partially fixed by authoring tests with that issue in mind.
> For instance, having a big promise_test to handle the asynchronous aspect of 
> a test and nested test() inside it.

I don't think we've ever had a problem with debugging js-test.js tests when 
they fail.

> 
>> The thing I specifically asked Youenn to ask is, whether we should
>> place a test inside LayoutTests/wpt like LayoutTests/http/tests when
>> we want to write a test using testharness.js which requires some sort
>> of network code.
>> 
>> Since people have had some opinions about directory structures in the past.
> 
> 
> It seems like we need a few different directories, here are my opinions on 
> them:
> 
> (1) Imported web platform tests that don't need a server
> Currently LayoutTests/imported/w3c/web-platform-tests, which seems fine.
> 
> All WPT tests are expected to run behind the WPT server.
> That is the way tests are authored and tested elsewhere.
> If we have an issue with that, it is best to bring that and fix that directly 
> in WPT.
> I encountered several times small issues due to file:// based origins which 
> makes me think defaulting to http is a safe choice.
> 
> One concern is efficiency. We should study that and improve on that.
> Another concern is the ease of running tests for developers: drag 
> tests into a browser instead of running a server.
> We can partially accommodate this by rewriting 
> testharness.js/testharnessreport.js urls.
> A significant and growing amount of wpt tests will not behave as expected 
> (other resources loaded, origins, need for specific headers, need for 
> https...)
> 
> (2) Imported web platform tests that do need a server
> Probably should be under LayoutTests/imported/w3c/ somewhere, or maybe 
> under http/tests/ per point (4)
> 
> I don't think this will work, web-platform-tests is organized in terms of 
> features.
> There is no clear separation between file based compatible tests and http 
> based tests like in WebKit.

If we run all the w3c-imported web platform tests under a web server, then 
obviously we only need one directory. My understanding is that we don't run 
them under a server at all. So it seems like one part of this proposal is "run 
everything under LayoutTests/imported/w3c/ from a web server".

> 
> (3) Custom testharness.js tests that don't need a server 
> Probably these should just go in their normal topic-specific directories 
> and should not need a special directory
> 
> Right.
> The only case where it might make sense to put such tests in a specific 
> WPT-enabled directory is if the plan is to upstream these tests at some point.
> Such tests could be added in imported/w3c/web-platform-tests directly but 
> this requires coordination with resyncing tests at the moment.
> In a not-too-far-future, I hope such tests would directly be authored in 
> imported/w3c/web-platform-tests.

I think it would be cleaner to have a separate directory of tests intended for 
import, separate from imported tests. It could be right next to 
imported/w3c/web-platform-tests. I think mixing tests that are imported from 
upstream and tests intended for eventual upstreaming is more confusing than 
helpful.

>  
> (4) Custom testharness.js tests that do need a server
> Can these just be a subdirectory of http/tests/? We have websocket and 
> ssl/tls tests in there too. Would be nice to not need a separate directory 
> for networking tests that to use a particular test framework.
> 
> I do not have strong feelings there, http/wpt might make sense if it is found 
> easier to understand to everybody.
> I'll update the patch accordingly and will land it sometimes this week if 
> there is no additional feedback.

One question I have is whether web platform tests can run under a regular HTTP 
server (maybe with appropriate configuration) or do we need something special? 
Is the WPT server more than just a web server with specific configuration 
settings?

Regards,
Maciej


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Geoffrey Garen]

> Another concern is the ease of running tests for developers: drag 
> tests into a browser instead of running a server.

Yeah, it’s a pretty big concern if you can’t just drop a simple test case into 
a browser.

> We can partially accommodate this by rewriting 
> testharness.js/testharnessreport.js urls.
> A significant and growing amount of wpt tests will not behave as expected 
> (other resources loaded, origins, need for specific headers, need for https…)

This might be a reason to prefer WPT tests when you know you’re testing 
something whose behavior depends on http. But in all other cases, the ability 
for a test to be a self-contained file that can run in any browser, upload to a 
bug tracker, and so on, is super valuable.

Geoff
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[youenn fablet]

>
>
> Besides other issues mentioned, testharness tends to result in more
> verbose tests compared to js-test, at least for simple cases.
>

For synchronous tests, I am not sure there is any big difference one way or
the other.
With asynchronous tests, it might be true, but using
testharness.js/promise_test usually improves things there.
 I personally find it easier to not wrap code-to-be-tested into quotes.

Another concern is the lack of verbose output which reduces the ability to
debug failing tests.
This can be partially fixed by authoring tests with that issue in mind.
For instance, having a big promise_test to handle the asynchronous aspect
of a test and nested test() inside it.

The thing I specifically asked Youenn to ask is, whether we should
> place a test inside LayoutTests/wpt like LayoutTests/http/tests when
> we want to write a test using testharness.js which requires some sort
> of network code.
>
> Since people have had some opinions about directory structures in the past.
>
>
> It seems like we need a few different directories, here are my opinions on
> them:
>
> (1) Imported web platform tests that don't need a server
> Currently LayoutTests/imported/w3c/web-platform-tests, which seems
> fine.
>

All WPT tests are expected to run behind the WPT server.
That is the way tests are authored and tested elsewhere.
If we have an issue with that, it is best to bring that and fix that
directly in WPT.
I encountered several times small issues due to file:// based origins which
makes me think defaulting to http is a safe choice.

One concern is efficiency. We should study that and improve on that.
Another concern is the ease of running tests for developers: drag
tests into a browser instead of running a server.
We can partially accommodate this by rewriting
testharness.js/testharnessreport.js urls.
A significant and growing amount of wpt tests will not behave as expected
(other resources loaded, origins, need for specific headers, need for
https...)

(2) Imported web platform tests that do need a server
> Probably should be under LayoutTests/imported/w3c/ somewhere, or maybe
> under http/tests/ per point (4)
>

I don't think this will work, web-platform-tests is organized in terms of
features.
There is no clear separation between file based compatible tests and http
based tests like in WebKit.

(3) Custom testharness.js tests that don't need a server
> Probably these should just go in their normal topic-specific
> directories and should not need a special directory
>

Right.
The only case where it might make sense to put such tests in a specific
WPT-enabled directory is if the plan is to upstream these tests at some
point.
Such tests could be added in imported/w3c/web-platform-tests directly but
this requires coordination with resyncing tests at the moment.
In a not-too-far-future, I hope such tests would directly be authored in
imported/w3c/web-platform-tests.


> (4) Custom testharness.js tests that do need a server
> Can these just be a subdirectory of http/tests/? We have websocket and
> ssl/tls tests in there too. Would be nice to not need a separate directory
> for networking tests that to use a particular test framework.
>

I do not have strong feelings there, http/wpt might make sense if it is
found easier to understand to everybody.
I'll update the patch accordingly and will land it sometimes this week if
there is no additional feedback.

Thanks!
   y
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Geoffrey Garen]

> What we're suggesting is to give preferential treatments to
> testharness.js over js-test.js / js-test-pre.js when you were already
> planning to write a test with the latter two scripts.

OK, I think this makes sense.

But I still think the very best kind of test is a flat file with 10-20 lines of 
code in it. Particularly for debugging JavaScript issues, large wrapper 
frameworks get in the way.

> - Tests would be more easily upstreamable to web-platform-tests, which are 
> run by all major browser engines. This would help a lot in terms of 
> interoperability. As previously discussed, Gecko and Blink already do 
> automated export of tests to web-platform-tests. I believe we should do in 
> the same direction and contribute more tests back.

I wonder why these other projects do automated export instead of incorporating 
testharness.js directly.

Geoff
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Maciej Stachowiak]

> On May 8, 2017, at 11:15 PM, Ryosuke Niwa  wrote:
> 
> On Mon, May 8, 2017 at 11:01 PM, Brady Eidson  > wrote:
>
> On May 8, 2017, at 10:44 PM, Ryosuke Niwa < 
> rn...@webkit.org > 
> wrote:
>>> On Mon, May 8, 2017 at 10:17 PM, Brady Eidson >> > wrote:
 But now talking about testharness.js directly, I object on the grounds of 
 "a
 file:// regression test is dirt easy to hack on and work with, whereas
 anything that requires me to have an httpd running is a PITA"
>>> I think whether we use file:// or http:// is orthogonal point to using
>>> testharness.js.  Many of the tests Chris and I have written using
>>> testharness.js are checked into regular LayoutTests/ directories, and
>>> they work just fine.
>> Yes, I misunderstood this in Youenn's original message. Good to know!
 I just object to making it the "recommended way" of writing tests.
>>> Would you equally object to making js-test.js / js-test-pre.js the
>>> recommended way of writing tests?
>> Yes.
>>> If not, why?
>> N/A
>>> What we're suggesting is to give preferential treatments to
>>> testharness.js over js-test.js / js-test-pre.js when you were already
>>> planning to write a test with the latter two scripts.
>> "It's okay to write your test however you'd like. If you were considering 
>> using js-test, maybe you should consider using testharness instead."
>> Is that's what's being proposed?
> 
>> 
>>> On May 8, 2017, at 10:44 PM, Ryosuke Niwa >> > wrote:
>>> 
>>> On Mon, May 8, 2017 at 10:17 PM, Brady Eidson >> > wrote:
>>> 
 But now talking about testharness.js directly, I object on the grounds of 
 "a
 file:// regression test is dirt easy to hack on and work with, whereas
 anything that requires me to have an httpd running is a PITA"
>>> 
>>> I think whether we use file:// or http:// is orthogonal point to using
>>> testharness.js.  Many of the tests Chris and I have written using
>>> testharness.js are checked into regular LayoutTests/ directories, and
>>> they work just fine.
>> 
>> Yes, I misunderstood this in Youenn's original message. Good to know!
>>> 
 I just object to making it the "recommended way" of writing tests.
>>> 
>>> Would you equally object to making js-test.js / js-test-pre.js the
>>> recommended way of writing tests?
>> 
>> Yes.
>> 
>>> If not, why?
>> 
>> N/A
>> 
>>> What we're suggesting is to give preferential treatments to
>>> testharness.js over js-test.js / js-test-pre.js when you were already
>>> planning to write a test with the latter two scripts.
>> 
>> "It's okay to write your test however you'd like. If you were considering 
>> using js-test, maybe you should consider using testharness instead."
>> 
>> Is that's what's being proposed?

Besides other issues mentioned, testharness tends to result in more verbose 
tests compared to js-test, at least for simple cases.

> 
> The thing I specifically asked Youenn to ask is, whether we should
> place a test inside LayoutTests/wpt like LayoutTests/http/tests when
> we want to write a test using testharness.js which requires some sort
> of network code.
> 
> Since people have had some opinions about directory structures in the past.

It seems like we need a few different directories, here are my opinions on them:

(1) Imported web platform tests that don't need a server
Currently LayoutTests/imported/w3c/web-platform-tests, which seems fine.
(2) Imported web platform tests that do need a server
Probably should be under LayoutTests/imported/w3c/ somewhere, or maybe 
under http/tests/ per point (4)
(3) Custom testharness.js tests that don't need a server 
Probably these should just go in their normal topic-specific directories 
and should not need a special directory
(4) Custom testharness.js tests that do need a server
Can these just be a subdirectory of http/tests/? We have websocket and 
ssl/tls tests in there too. Would be nice to not need a separate directory for 
networking tests that to use a particular test framework.

Regards,
Maciej

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Ryosuke Niwa]

On Mon, May 8, 2017 at 11:01 PM, Brady Eidson  wrote:
>
>> On May 8, 2017, at 10:44 PM, Ryosuke Niwa  wrote:
>>
>> On Mon, May 8, 2017 at 10:17 PM, Brady Eidson  wrote:
>>
>>> But now talking about testharness.js directly, I object on the grounds of "a
>>> file:// regression test is dirt easy to hack on and work with, whereas
>>> anything that requires me to have an httpd running is a PITA"
>>
>> I think whether we use file:// or http:// is orthogonal point to using
>> testharness.js.  Many of the tests Chris and I have written using
>> testharness.js are checked into regular LayoutTests/ directories, and
>> they work just fine.
>
> Yes, I misunderstood this in Youenn's original message. Good to know!
>>
>>> I just object to making it the "recommended way" of writing tests.
>>
>> Would you equally object to making js-test.js / js-test-pre.js the
>> recommended way of writing tests?
>
> Yes.
>
>> If not, why?
>
> N/A
>
>> What we're suggesting is to give preferential treatments to
>> testharness.js over js-test.js / js-test-pre.js when you were already
>> planning to write a test with the latter two scripts.
>
> "It's okay to write your test however you'd like. If you were considering 
> using js-test, maybe you should consider using testharness instead."
>
> Is that's what's being proposed?

The thing I specifically asked Youenn to ask is, whether we should
place a test inside LayoutTests/wpt like LayoutTests/http/tests when
we want to write a test using testharness.js which requires some sort
of network code.

Since people have had some opinions about directory structures in the past.

- R. Niwa
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Brady Eidson]

> On May 8, 2017, at 10:44 PM, Ryosuke Niwa  wrote:
> 
> On Mon, May 8, 2017 at 10:17 PM, Brady Eidson  wrote:
> 
>> But now talking about testharness.js directly, I object on the grounds of "a
>> file:// regression test is dirt easy to hack on and work with, whereas
>> anything that requires me to have an httpd running is a PITA"
> 
> I think whether we use file:// or http:// is orthogonal point to using
> testharness.js.  Many of the tests Chris and I have written using
> testharness.js are checked into regular LayoutTests/ directories, and
> they work just fine.

Yes, I misunderstood this in Youenn's original message. Good to know!
> 
>> I just object to making it the "recommended way" of writing tests.
> 
> Would you equally object to making js-test.js / js-test-pre.js the
> recommended way of writing tests?

Yes.

> If not, why?

N/A

> What we're suggesting is to give preferential treatments to
> testharness.js over js-test.js / js-test-pre.js when you were already
> planning to write a test with the latter two scripts.

"It's okay to write your test however you'd like. If you were considering using 
js-test, maybe you should consider using testharness instead."

Is that's what's being proposed?

The WebKit project usually doesn't spend much time worrying about highly 
qualified non-binding suggestions. ;)

Thanks,
 Brady
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Brady Eidson]

> On May 8, 2017, at 10:42 PM, youenn fablet  wrote:
> 
> testharness.js does not need an http server. Some WPT goodies need the WPT 
> server.

I misunderstood since we were also discussing:

>> To continue moving forward, some of us are proposing to serve all tests in 
>> LayoutTests/wpt through the WPT server [1].

I think if somebody is writing a testharness.js test and it does *not* require 
the WPT server, it should be tested as a file URL.

This would imply putting it outside of LayoutTests/wpt though, right?

> I agree different frameworks offer different benefits. There is no reason we 
> should mandate one framework in particular.

Let me make it even more clear what I'm trying to defend. An ad-hoc custom test 
that uses no "framework" but is entirely self contained.

> In case there is no specific needs, it makes sense to me to recommend using 
> testharness.js, at least for those hacking WebCore.

I think it's valuable for engineers to be aware of the "frameworks" like 
testharness.js and js-test, and familiar with the facilities they provide, but 
to not necessarily imply that using any framework is expected.

Thanks,
 Brady___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Ryosuke Niwa]

On Mon, May 8, 2017 at 10:17 PM, Brady Eidson  wrote:
> On May 8, 2017, at 9:31 PM, youenn fablet  wrote:
>
> Hi all,
>
> Discussing with some WebKittens, testharness.js is more and more used in
> WebKit.
> Is it time to make testharness.js the recommended way of writing
> LayoutTests?
>
>
> Setting aside the pros or cons of testharness.js itself, I disagree with the
> principle of "1 single way to write all regression tests"
>
> In the past 11 years I've heard from multiple members of the team commenting
> on the benefits of different people writing regression tests in their own
> styles using their own techniques. We're capable of covering more edge cases
> when we don't have enforced uniformity. And I agree wholeheartedly.

Agreed.

> But now talking about testharness.js directly, I object on the grounds of "a
> file:// regression test is dirt easy to hack on and work with, whereas
> anything that requires me to have an httpd running is a PITA"

I think whether we use file:// or http:// is orthogonal point to using
testharness.js.  Many of the tests Chris and I have written using
testharness.js are checked into regular LayoutTests/ directories, and
they work just fine.

The imported web platform tests use WPT today because that's how
upstream works. But we're looking at ways to reduce the number of
tests that need to be ran under WPT for imported tests as well.

> I just object to making it the "recommended way" of writing tests.

Would you equally object to making js-test.js / js-test-pre.js the
recommended way of writing tests? If not, why?

What we're suggesting is to give preferential treatments to
testharness.js over js-test.js / js-test-pre.js when you were already
planning to write a test with the latter two scripts.

- R. Niwa
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[youenn fablet]

testharness.js does not need an http server. Some WPT goodies need the WPT
server.

I agree different frameworks offer different benefits. There is no reason
we should mandate one framework in particular.

In case there is no specific needs, it makes sense to me to recommend using
testharness.js, at least for those hacking WebCore. Chances are high that
another browser community might like running (and improving and
maintaining) it.
Chances are also high that one will have to debug/update such tests, so it
is good to learn this framework anyway.
Le lun. 8 mai 2017 à 22:17, Brady Eidson  a écrit :

> On May 8, 2017, at 9:31 PM, youenn fablet  wrote:
>
> Hi all,
>
> Discussing with some WebKittens, testharness.js is more and more used in
> WebKit.
> Is it time to make testharness.js the recommended way of writing
> LayoutTests?
>
>
> Setting aside the pros or cons of testharness.js itself, I disagree with
> the principle of "1 single way to write all regression tests"
>
> In the past 11 years I've heard from multiple members of the team
> commenting on the benefits of different people writing regression tests in
> their own styles using their own techniques. We're capable of covering more
> edge cases when we don't have enforced uniformity. And I agree
> wholeheartedly.
>
> But now talking about testharness.js directly, I object on the grounds of
> "a file:// regression test is dirt easy to hack on and work with, whereas
> anything that requires me to have an httpd running is a PITA"
>
> Note: I don't intend for any of this to mean I discourage the use of
> testharness.js tests. I don't. By all means, write them.
>
> I just object to making it the "recommended way" of writing tests.
>
> Thanks,
>  Brady
>
>
> To continue moving forward, some of us are proposing to serve all tests in
> LayoutTests/wpt through the WPT server [1].
> This would serve some purposes like increasing the use of WPT goodies:
> file-specific headers, templated tests (*.any.js), IDLParser, server-side
> scripts...
> It could also ease test migration from WebKit to W3C WPT.
>
> Some rules can guide whether adding tests to LayoutTests/wpt or
> LayoutTests/imported/w3c/web-platform-tests:
> - WebKit specific tests (crash tests, tests using internals...) in
> LayoutTests/wpt
> - Spec conformance/interoperability tests in
> LayoutTests/imported/w3c/web-platform-tests
>
>y
>
> [1]: bug 171479 
>
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Brady Eidson]

> On May 8, 2017, at 9:31 PM, youenn fablet  wrote:
> 
> Hi all,
> 
> Discussing with some WebKittens, testharness.js is more and more used in 
> WebKit.
> Is it time to make testharness.js the recommended way of writing LayoutTests?

Setting aside the pros or cons of testharness.js itself, I disagree with the 
principle of "1 single way to write all regression tests"

In the past 11 years I've heard from multiple members of the team commenting on 
the benefits of different people writing regression tests in their own styles 
using their own techniques. We're capable of covering more edge cases when we 
don't have enforced uniformity. And I agree wholeheartedly.

But now talking about testharness.js directly, I object on the grounds of "a 
file:// regression test is dirt easy to hack on and work with, whereas anything 
that requires me to have an httpd running is a PITA"

Note: I don't intend for any of this to mean I discourage the use of 
testharness.js tests. I don't. By all means, write them.

I just object to making it the "recommended way" of writing tests.

Thanks,
 Brady

> 
> To continue moving forward, some of us are proposing to serve all tests in 
> LayoutTests/wpt through the WPT server [1].
> This would serve some purposes like increasing the use of WPT goodies: 
> file-specific headers, templated tests (*.any.js), IDLParser, server-side 
> scripts...
> It could also ease test migration from WebKit to W3C WPT.
> 
> Some rules can guide whether adding tests to LayoutTests/wpt or 
> LayoutTests/imported/w3c/web-platform-tests:
> - WebKit specific tests (crash tests, tests using internals...) in 
> LayoutTests/wpt
> - Spec conformance/interoperability tests in 
> LayoutTests/imported/w3c/web-platform-tests
> 
>y
> 
> [1]: bug 171479 
> 
> 
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Chris Dumez]

> On May 8, 2017, at 9:44 PM, Geoffrey Garen  wrote:
> 
>> Is it time to make testharness.js the recommended way of writing LayoutTests?
> 
> What are the costs and benefits of testharness.js?

Benefit: 
- Tests would be more easily upstreamable to web-platform-tests, which are run 
by all major browser engines. This would help a lot in terms of 
interoperability. As previously discussed, Gecko and Blink already do automated 
export of tests to web-platform-tests. I believe we should do in the same 
direction and contribute more tests back.
- It is quite powerful (see documentation [1], has things like Promise tests, 
Worker tests and advanced assertions.

Cost:
- People are not necessarily used to it so there is a little bit of a learning 
curve. It is well documented though [1].

> 
> We usually try to make regression tests reductions of some larger problem to 
> aid debugging and to make testing fast. But testharness.js is 95kB. That's 
> kind of the opposite of a reduction.

It is proposed as a replacement to js-test.js, which a lot of us are already 
using in our layout tests. Using js-test.js or testharness.js has rarely 
interfered in my reductions, although it has happened to me.
For some tests, we’ll probably not use any framework. However, for most of 
them, I personally don’t see an issue.

[1] http://testthewebforward.org/docs/testharness-library.html 



> 
> Geoff
> 
>> 
>> To continue moving forward, some of us are proposing to serve all tests in 
>> LayoutTests/wpt through the WPT server [1].
>> This would serve some purposes like increasing the use of WPT goodies: 
>> file-specific headers, templated tests (*.any.js), IDLParser, server-side 
>> scripts...
>> It could also ease test migration from WebKit to W3C WPT.
>> 
>> Some rules can guide whether adding tests to LayoutTests/wpt or 
>> LayoutTests/imported/w3c/web-platform-tests:
>> - WebKit specific tests (crash tests, tests using internals...) in 
>> LayoutTests/wpt
>> - Spec conformance/interoperability tests in 
>> LayoutTests/imported/w3c/web-platform-tests
>> 
>>y
>> 
>> [1]: bug 171479 
>> 
>> 
>> ___
>> webkit-dev mailing list
>> webkit-dev@lists.webkit.org 
>> https://lists.webkit.org/mailman/listinfo/webkit-dev
> 
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Geoffrey Garen]

> Is it time to make testharness.js the recommended way of writing LayoutTests?

What are the costs and benefits of testharness.js?

We usually try to make regression tests reductions of some larger problem to 
aid debugging and to make testing fast. But testharness.js is 95kB. That's kind 
of the opposite of a reduction.

Geoff

> 
> To continue moving forward, some of us are proposing to serve all tests in 
> LayoutTests/wpt through the WPT server [1].
> This would serve some purposes like increasing the use of WPT goodies: 
> file-specific headers, templated tests (*.any.js), IDLParser, server-side 
> scripts...
> It could also ease test migration from WebKit to W3C WPT.
> 
> Some rules can guide whether adding tests to LayoutTests/wpt or 
> LayoutTests/imported/w3c/web-platform-tests:
> - WebKit specific tests (crash tests, tests using internals...) in 
> LayoutTests/wpt
> - Spec conformance/interoperability tests in 
> LayoutTests/imported/w3c/web-platform-tests
> 
>y
> 
> [1]: bug 171479 
> 
> 
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

[webkit-dev] Another WPT bite

[youenn fablet]

Hi all,

Discussing with some WebKittens, testharness.js is more and more used in
WebKit.
Is it time to make testharness.js the recommended way of writing
LayoutTests?

To continue moving forward, some of us are proposing to serve all tests in
LayoutTests/wpt through the WPT server [1].
This would serve some purposes like increasing the use of WPT goodies:
file-specific headers, templated tests (*.any.js), IDLParser, server-side
scripts...
It could also ease test migration from WebKit to W3C WPT.

Some rules can guide whether adding tests to LayoutTests/wpt or
LayoutTests/imported/w3c/web-platform-tests:
- WebKit specific tests (crash tests, tests using internals...) in
LayoutTests/wpt
- Spec conformance/interoperability tests in
LayoutTests/imported/w3c/web-platform-tests

   y

[1]: bug 171479 
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev