Re: [webkit-dev] Another WPT bite

[Maciej Stachowiak]

> On May 12, 2017, at 7:49 PM, a...@webkit.org wrote:
> 
> 
>> 12 мая 2017 г., в 19:38, Brian Burg > > написал(а):
>> 
>>> I think that I explained it very clearly, but let me try again.
>>> 
>>> When there is a test failure that I need to communicate to others, I say 
>>> something "please open 
>>> >>  
>>> >
>>>  in Safari to reproduce". That's very easy to do, and makes it very easy 
>>> for others to work on the issue.
>>> If your test requires complex setup, like WPT does, then I may not have the 
>>> time to write up complicated steps to reproduce, or the person who gets the 
>>> bug may not have the time to follow them. Those people don't have a WebKit 
>>> checkout, so scripts won't help. This makes the test less effective, as 
>>> problems that it finds are less likely to be addressed.
>> 
>> If the person works on WebKit, then it seems unreasonable that they would do 
>> work without a checkout.
> 
> It is correct that people who work on WebKit usually have a checkout. So I 
> was taking about people who don't work on WebKit.
> 
>> If they don’t work on WebKit, then you could run wptserve on a machine 
>> somewhere and link to that copy. We have several servers that exist solely 
>> to host test content, it doesn’t seem like a big deal to make one of them 
>> update regularly and relaunch wptserve to pick up test harness changes.
> 
> Yes, there is a number of things one could do. Those things would work in 
> some cases but not in others - I mentioned linking to a stable version that 
> won't change, which is something that trac gives us for free, and it would be 
> non-trivial to implement otherwise.
> 
> In practice, the best approach would be to reduce the test to a minimum that 
> doesn't use complex harnesses before ending it over. Everyone likes minimal 
> test cases.

I think WPT is becoming enough of a broad community norm that it's not 
unreasonable to point to a WPT server. It does seem regrettable that tests 
become dependent on being run through the server even when in principle they 
could run as local files. From what I gather, there are a lot of tests where 
only the paths to the test harness end up requiring the server. Doing the fixup 
on import seems bad to me, since it seems safer and cleaner for our WPT 
checkout to match WPT. But we could follow the practice of using relative URLs 
for self-created tests, and perhaps not even run them under the server when 
they don't need it.

For upstream, perhaps we could advocate with WPT to use relative paths to load 
the harness (and perhaps make sure that tests that absolutely require the 
server fail in a way that clearly indicates this for the tests that truly do 
need networking or some other facility of the WPT server).

Regards,
Maciej

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Brady Eidson]

> On May 12, 2017, at 7:43 PM, Ryosuke Niwa  wrote:
> 
> On Fri, May 12, 2017 at 7:31 PM, Alexey Proskuryakov  wrote:
>> 
>> When there is a test failure that I need to communicate to others, I say
>> something "please open
>> 
>> in Safari to reproduce". That's very easy to do, and makes it very easy for
>> others to work on the issue.
>> ...
> 
> Note that W3C's web-plaform-tests are hosted on
> http://w3c-test.org/tools/runner/index.html so you can could do the
> same thing.

Note Alexey's URL example points to a specific revision of a specific test.

A frustration that has often come up while I've been working on leading-edge 
features in WebKit is that the w3c-test.org  tests are 
constantly updated and therefore are a moving target.
One cannot use the same test repeatedly over some length of time and expect it 
to remain consistent.

Thanks,
~Brady



___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Michael[tm] Smith]

Maciej Stachowiak , 2017-05-13 14:58 -0700:
> ... From what I gather, there are a lot of tests where only the paths to
> the test harness end up requiring the server.

Yeah that’s the case for the vast majority of tests. Relatively few — less than
5% altogether, I’d estimate — actually rely on any special server behavior.

Of the ones needing special server behavior, I think even there by far most are
just cases of an accompanying foo.html.headers file in the tree along with the
foo.html test, to specify that the server send particular response headers.

Out of ~50,000 test files in WPT, there are less than 1000 with a .headers file.

I think the next-most-common case that require special server behavior are cases
where the server is doing some parsing and substitution of special parameters.
In those cases the test file itself will be named in the pattern foo.sub.html,
or one of its JS assets in the pattern foo.sub.js.

Out of ~50,000 test files in WPT, less than 300 are *.sub.* files.

So those two cases take together amount to only 3% of the total. So even with
whatever else I’m missing added to that I’d estimate the number of tests that
don’t rely on special server behavior is on the order of 95%.

So those ~95% all only need the “/resources/testharness.js” path to the test
harness to resolve and then they’ll just work.

> Doing the fixup on import seems bad to me, since it seems safer and
> cleaner for our WPT checkout to match WPT. But we could follow the
> practice of using relative URLs for self-created tests, and perhaps not
> even run them under the server when they don't need it.
> 
> For upstream, perhaps we could advocate with WPT to use relative paths to
> load the harness

Given that a specific problem case Alexey mentioned was linking to tests within
the WebKit Trac and having them run as expected, I wonder if at least y’all
could find a way to just make https://trac.webkit.org/resources/testharness.js
work — I guess by making it redirect to some place where you have a current WPT
checkout. That’d at least solve things for the main specific case that’s been
brought up so far being a real problem.

> (and perhaps make sure that tests that absolutely require the server fail in a
> way that clearly indicates this for the tests that truly do need networking or
> some other facility of the WPT server).

Yeah that would be good to make happen in general regardless. But I think right
now that maybe can mostly be (programatically) determined by (1) checking if the
name of the test file itself is in the *.sub.* pattern, or (2) checking to see
if there’s an accompanying *.headers file for the test.

  —Mike

-- 
Michael[tm] Smith https://people.w3.org/mike


signature.asc
Description: PGP signature
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Michael[tm] Smith]

Brady Eidson , 2017-05-13 17:11 -0700:
> 
> 
> > On May 12, 2017, at 7:43 PM, Ryosuke Niwa  wrote:
> > 
> > On Fri, May 12, 2017 at 7:31 PM, Alexey Proskuryakov  
> > wrote:
> >> 
> >> When there is a test failure that I need to communicate to others, I say
> >> something "please open
> >> 
> >> in Safari to reproduce". That's very easy to do, and makes it very easy for
> >> others to work on the issue.
> >> ...
> > 
> > Note that W3C's web-plaform-tests are hosted on
> > http://w3c-test.org/tools/runner/index.html so you can could do the
> > same thing.
> 
> Note Alexey's URL example points to a specific revision of a specific test.
> 
> A frustration that has often come up while I've been working on
> leading-edge features in WebKit is that the w3c-test.org
>  tests are constantly updated and therefore are a
> moving target.  One cannot use the same test repeatedly over some length
> of time and expect it to remain consistent.

True. But of course the tests are under version control and you can always
locate any specific revision of one you want to test against; e.g.:

  
https://raw.githubusercontent.com/w3c/web-platform-tests/b2e3ab7/service-workers/service-worker/activate-event-after-install-state-change.https.html

For practical reasons on w3c-test.org we can’t directly archive a runnable copy
of every single revision of of every single test. But I would be very willing to
look into adding a service somewhere that does something similar to what
rawgit.com or whatever do—that is, (1) given a URL like the above for specific
revision of a file in a github repo, then (2) load and run it.

rawgit.com itself can of course already load specific revisions of WPT
tests, but when you try it you’ll run into the general problem of the test
files specifying a /resources/testharness.js path to the harness; e.g.:

  
https://rawgit.com/w3c/web-platform-tests/b2e3ab7/service-workers/service-worker/activate-event-after-install-state-change.https.html

...which expects to find https://rawgit.com/resources/testharness.js.

But that’s not a hard problem to solve. As I mentioned earlier, I imagine
that with very little effort needed, I could get a similar service set up
that’s WPT-specific and WPT-aware in that it will have the test harness
available at for the /resources/testharness.js URL path.

  —Mike

-- 
Michael[tm] Smith https://people.w3.org/mike


signature.asc
Description: PGP signature
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Michael[tm] Smith]

Anne van Kesteren , 2017-05-13 06:20 +0200:
> 
> On Fri, May 12, 2017 at 11:49 PM, Maciej Stachowiak  wrote:
> > It seems like there's two unusual things about WPT:
> > - At least according to Alexey, WPT tests are somewhat prone to flakiness 
> > in Safari.
> 
> Although they haven't always been working perfectly, changes to
> web-platform-tests run through some kind of stability check in both
> Chrome and Firefox (run the new tests 10 times and fail if the results
> are inconsistent). Ideally Safari is added to that mix, though I think
> the last time folks looked into that it wasn't possible for some
> reason. That might be something to put some resources on.

I think we already have that now un-blocked and we will by the end of this week
(May 19) start having Travis running the stability checker for Safari too (and
Edge too) and providing the results as comments to PRs (just as we now do
for Chrome and Firefox).

  —Mike

-- 
Michael[tm] Smith https://people.w3.org/mike


signature.asc
Description: PGP signature
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Another WPT bite

[Maciej Stachowiak]

> On May 13, 2017, at 5:26 PM, Michael[tm] Smith  wrote:
> 
> Maciej Stachowiak , 2017-05-13 14:58 -0700:
>> ... From what I gather, there are a lot of tests where only the paths to
>> the test harness end up requiring the server.
> 
> Yeah that’s the case for the vast majority of tests. Relatively few — less 
> than
> 5% altogether, I’d estimate — actually rely on any special server behavior.
> 
> Of the ones needing special server behavior, I think even there by far most 
> are
> just cases of an accompanying foo.html.headers file in the tree along with the
> foo.html test, to specify that the server send particular response headers.
> 
> Out of ~50,000 test files in WPT, there are less than 1000 with a .headers 
> file.
> 
> I think the next-most-common case that require special server behavior are 
> cases
> where the server is doing some parsing and substitution of special parameters.
> In those cases the test file itself will be named in the pattern foo.sub.html,
> or one of its JS assets in the pattern foo.sub.js.
> 
> Out of ~50,000 test files in WPT, less than 300 are *.sub.* files.
> 
> So those two cases take together amount to only 3% of the total. So even with
> whatever else I’m missing added to that I’d estimate the number of tests that
> don’t rely on special server behavior is on the order of 95%.
> 
> So those ~95% all only need the “/resources/testharness.js” path to the test
> harness to resolve and then they’ll just work.
> 
>> Doing the fixup on import seems bad to me, since it seems safer and
>> cleaner for our WPT checkout to match WPT. But we could follow the
>> practice of using relative URLs for self-created tests, and perhaps not
>> even run them under the server when they don't need it.
>> 
>> For upstream, perhaps we could advocate with WPT to use relative paths to
>> load the harness
> 
> Given that a specific problem case Alexey mentioned was linking to tests 
> within
> the WebKit Trac and having them run as expected, I wonder if at least y’all
> could find a way to just make https://trac.webkit.org/resources/testharness.js
> work — I guess by making it redirect to some place where you have a current 
> WPT
> checkout. That’d at least solve things for the main specific case that’s been
> brought up so far being a real problem.

Alexey mentioned trac because it's something we couldn't easily solve with a 
command-line tool. The far more common case is engineers loading individual 
test cases to debug them. As you mentioned ~95% of test cases would load fine 
as a local file, except for the path where they expect testharness.js.

For the engineer use case, we can make a command-line tool to launch the server 
and load the test. But it's kind of sad that in ~95% of cases, the only value 
provided by the server is resolving the path to testharness.js. If tests 
referenced testharness.js via relative path, then most of the time they could 
be loaded as local files just fine, which would be more convenient (as well as, 
I believe, solving the "external trac link" issue).


>> (and perhaps make sure that tests that absolutely require the server fail in 
>> a
>> way that clearly indicates this for the tests that truly do need networking 
>> or
>> some other facility of the WPT server).
> 
> Yeah that would be good to make happen in general regardless. But I think 
> right
> now that maybe can mostly be (programatically) determined by (1) checking if 
> the
> name of the test file itself is in the *.sub.* pattern, or (2) checking to see
> if there’s an accompanying *.headers file for the test.

It would only be relevant for tests to give better diagnostics if they were set 
up to run outside the WPT server at all, which they aren't. This suggestion is 
somewhat conditional on consideration of that other suggestion (which is 
basically to load required static resources via relative path).

Regards,
Maciej

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev