Re: [webkit-dev] Another WPT bite

2017-05-14 Thread Anne van Kesteren
On Sun, May 14, 2017 at 8:24 AM, Maciej Stachowiak  wrote:
> For the engineer use case, we can make a command-line tool to launch the 
> server and load the test. But it's kind of sad that in ~95% of cases, the 
> only value provided by the server is resolving the path to testharness.js. If 
> tests referenced testharness.js via relative path, then most of the time they 
> could be loaded as local files just fine, which would be more convenient (as 
> well as, I believe, solving the "external trac link" issue).

I think the main problem with not running a server is that behavior
for file URLs is not defined. And browsers tend to impose different
restrictions there. So you might end up debugging something only to
later found out it didn't work due to file URL restrictions. And you
can't guarantee that something will work from a file URL, because
there's no agreement on what will.

I personally just keep web-platform-tests running and don't notice
much overhead (MacBook Air, Mid 2012).


-- 
https://annevankesteren.nl/
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev


Re: [webkit-dev] Another WPT bite

2017-05-14 Thread Maciej Stachowiak


> On May 13, 2017, at 5:26 PM, Michael[tm] Smith  wrote:
> 
> Maciej Stachowiak , 2017-05-13 14:58 -0700:
>> ... From what I gather, there are a lot of tests where only the paths to
>> the test harness end up requiring the server.
> 
> Yeah that’s the case for the vast majority of tests. Relatively few — less 
> than
> 5% altogether, I’d estimate — actually rely on any special server behavior.
> 
> Of the ones needing special server behavior, I think even there by far most 
> are
> just cases of an accompanying foo.html.headers file in the tree along with the
> foo.html test, to specify that the server send particular response headers.
> 
> Out of ~50,000 test files in WPT, there are less than 1000 with a .headers 
> file.
> 
> I think the next-most-common case that require special server behavior are 
> cases
> where the server is doing some parsing and substitution of special parameters.
> In those cases the test file itself will be named in the pattern foo.sub.html,
> or one of its JS assets in the pattern foo.sub.js.
> 
> Out of ~50,000 test files in WPT, less than 300 are *.sub.* files.
> 
> So those two cases take together amount to only 3% of the total. So even with
> whatever else I’m missing added to that I’d estimate the number of tests that
> don’t rely on special server behavior is on the order of 95%.
> 
> So those ~95% all only need the “/resources/testharness.js” path to the test
> harness to resolve and then they’ll just work.
> 
>> Doing the fixup on import seems bad to me, since it seems safer and
>> cleaner for our WPT checkout to match WPT. But we could follow the
>> practice of using relative URLs for self-created tests, and perhaps not
>> even run them under the server when they don't need it.
>> 
>> For upstream, perhaps we could advocate with WPT to use relative paths to
>> load the harness
> 
> Given that a specific problem case Alexey mentioned was linking to tests 
> within
> the WebKit Trac and having them run as expected, I wonder if at least y’all
> could find a way to just make https://trac.webkit.org/resources/testharness.js
> work — I guess by making it redirect to some place where you have a current 
> WPT
> checkout. That’d at least solve things for the main specific case that’s been
> brought up so far being a real problem.

Alexey mentioned trac because it's something we couldn't easily solve with a 
command-line tool. The far more common case is engineers loading individual 
test cases to debug them. As you mentioned ~95% of test cases would load fine 
as a local file, except for the path where they expect testharness.js.

For the engineer use case, we can make a command-line tool to launch the server 
and load the test. But it's kind of sad that in ~95% of cases, the only value 
provided by the server is resolving the path to testharness.js. If tests 
referenced testharness.js via relative path, then most of the time they could 
be loaded as local files just fine, which would be more convenient (as well as, 
I believe, solving the "external trac link" issue).


>> (and perhaps make sure that tests that absolutely require the server fail in 
>> a
>> way that clearly indicates this for the tests that truly do need networking 
>> or
>> some other facility of the WPT server).
> 
> Yeah that would be good to make happen in general regardless. But I think 
> right
> now that maybe can mostly be (programatically) determined by (1) checking if 
> the
> name of the test file itself is in the *.sub.* pattern, or (2) checking to see
> if there’s an accompanying *.headers file for the test.

It would only be relevant for tests to give better diagnostics if they were set 
up to run outside the WPT server at all, which they aren't. This suggestion is 
somewhat conditional on consideration of that other suggestion (which is 
basically to load required static resources via relative path).

Regards,
Maciej

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev