Hi Zan!

> On Feb 12, 2018, at 7:06 AM, z...@falconsigh.net wrote:
> Hi,
> the web-platform-tests repository includes tooling that enables running those 
> tests against a supported browser product. I'd like to propose adding generic 
> WebKit support there.
> Current changes only assume usage of the WebDriver protocol, and the 
> WebDriver binary accepting the --port flag. Selenium executors are used for 
> test harness and reftests. Same WebDriver implementation can also be tested 
> against the WebDriver tests included in the web-platform-tests directory, 
> presuming the tests are enabled or explicitly specified.
> Only port-specific bit is the specification of capabilities that are passed 
> to the WebDriver binary, idea being that these capabilities are the same as 
> those supported by the WebDriver implementation.

Yep, these capabilities already differ between safaridriver and 

> GTK is for now the only port that's supported, and it's leveraging the 
> WebDriver implementation under Source/WebDriver/ in WebKit. WPE will be doing 
> the same. Safari I suppose could use its own WebDriver implementation, or 
> perhaps even a separate product.

Whether it makes sense depends who is running the tests. If it's for public WPT 
CI, then it won't have access to trunk safaridriver bits, so it won't be very 
useful to detect new passes or failures. If it's me or another Apple engineer 
using it to find conformance problems, then it can run against trunk 
safaridriver and be quite useful. It might also be useful to run Safari 
Technology Preview safaridriver in public CI like wpt.fyi just to have an 
independent pass/fail metric that is a lot more current (every ~2 weeks) than 
/usr/bin/safaridriver (only updates when system Safari is updated).

Currently, safaridriver implements Selenium protocol, so this will be usable 
for running WPT tests in general (modulo bugs). However, the WebDriver tests in 
web-platform-tests repository only target W3C protocol and use a custom client 
library (not Selenium). I'm currently implementing W3C protocol; when it's 
ready for public testing via Safari Technology Preview, then I will make sure 
it's possible to target a 'safari' browser via wptrunner for WebDriver 
web-platform-tests. Until then it doesn't make much sense to add it upstream as 
a target for WebDriver tests.

> Here's the current set of changes:
> https://github.com/zdobersek/web-platform-tests/commit/c2ee920876ca6df7c4739feb8a6e03c77dffdb7f
> The web-platform-tests suite can then be run like this for the GTK port, 
> assuming a tip-of-trunk build:
> $ /work/web-platform-tests/wpt run --webkit-port=gtk \
>    --webdriver-binary=WebKitBuild/Release/bin/WebKitWebDriver \
>    --binary=WebKitBuild/Release/bin/MiniBrowser \
>    --binary-arg=--automation \
>    --binary-arg=--javascript-can-open-windows-automatically=true \
>    --binary-arg=--enable-xss-auditor=false \
>    webkit /2dcontext
> This can be further wrapped into a python script and run as part of the 
> continuous integration system. These changes add a run-web-platform-tests 
> script that invokes the web-platform-tests runner tool, also allowing each 
> port to specify what tests to enable and what the expected failures are:
> https://github.com/Igalia/webkit/commit/df1aeeb9476c6dd220067f4fc3c6ad69a8f948ba
> Only a small subset of tests is enabled there, for prototype purposes. The 
> expected results system could also be improved to avoid each expected failure 
> having to be marked as such in separate .ini files.

Carlos has a JSON-based expectations system for WPT WebDriver tests. Should we 
use that? I'm not sure if it is flexible enough to capture multiple port 
expectations. We'd probably want expectations to vary depending on enabled 
features somehow. Also, I'm not sure if we want to combine multiple port 
expectations, as this is a maintenance nightmare in the current layout test 
TestExpectation files.

> But foremost, I'd like to have a consensus of sorts about how various WebKit 
> ports should be handled in the web-platform-tests repository, so that the 
> changes there can proceed -- whether it's fine to implement a generic WebKit 
> product, or whether Safari would like to be treated as a separate browser[1], 
> etc.

For WebDriver testing purposes, I would prefer that there be a generic WebKit 
driver (uses Source/WebDriver/ and MiniBrowser) and a Safari driver (uses 
safaridriver and Safari). They are two separate REST API implementations with 
different bugs and capabilities.

It might be worth the effort to make Source/WebDriver work with MiniBrowser.app 
on Mac just for the purpose of having the entire system under test be compiled 
from one repository. However, this is generally not reflective of how 
safaridriver will perform since only the WebKit parts of WebDriver are shared 
between safaridriver and webkit[gtk]driver. It would just make it easier to run 
web-platform-tests against WebKit itself on Mac.

I don't have an opinion about how to handle webkit vs safari for the rest of 
web-platform-tests. Again, we need to think about who will be running it that 
way (engineers? CI?).

There is also the issue of WebKit on iOS... since safaridriver doesn't work 
with iOS Safari, we can't run web-platform-tests via WebDriver on iOS until 
such support exists.

> Regards,
> Zan
> [1] There's for instance this from a year ago (though not sure about its 
> functionality):
> https://github.com/w3c/web-platform-tests/tree/wptrunner-safari 
> <https://github.com/w3c/web-platform-tests/tree/wptrunner-safari>

I can clean up and submit a new PR for this, if that's the direction folks want 
to go now.

> _______________________________________________
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev

webkit-dev mailing list

Reply via email to