I filed https://bugs.webkit.org/show_bug.cgi?id=167911.
Le lun. 6 févr. 2017 à 16:22, youenn fablet <youe...@gmail.com> a écrit : > It seems we agree in moving this forward. Any objection? > If so, let's start with small practical steps, something like a script to > automatically generate WPT PR requests from a WebKit repo. > > Le lun. 6 févr. 2017 à 12:28, Ryosuke Niwa <rn...@webkit.org> a écrit : > > > On Monday, February 6, 2017, youenn fablet <youe...@gmail.com> wrote: > > The two complaints I heard against testharness.js are: > > - They are less easy to debug as test harness.js does not print out > messages for each assert. There might be room for updating testharness to > support that > - Async tests are less easy to write. While this is probably true, > testharness has support for promise-based tests which are not verbose. > WPT has also some benefits of its own, like the possibility to run easily > the same tests in worker/window environments, the flexible python-based > server... > > One complaint I heard against WPT tests is that they require being run > behind a server. > This is becoming more and more true. > There is still a large portion of tests that could be run locally if we > update the paths to test harness.js/testharnessreport,js. > I am not a big fan of modify any WPT test but maybe we should consider > switching back to modifying these paths at import and export time. > > > We should probably do this for the sake of making tests more debuggable. > Having to run a HTTP/HTTPS server makes testing WebCore against a test > needlessly harder especially for things like core DOM API which doesn't > require any network stack. > > > Or we could make our tooling better to also handle LayoutTests/http tests > more easily. > In any case, our CI infrastructure should run WPT tests as closely as > specified by WPT. > > > The concern I've heard about is not how run-webkit-tests run the tests. > It's about how a test is opened inside a browser, DRT, WTR while debugging. > > > I would tend to do the following: > - Write/submit WPT tests in WebKit as regular WebKit testharness.js layout > tests through bugzilla > - At commit queue time, automatically create a PR to WPT GitHub containing > the changes of the WebKit patch > > > I think commit queue time is too late. Remember that not everyone uses > commit queue. Also, if there was any issue with a test, we should be able > to tell that prior to landing the patch. > > > Ideally, having one EWS bullet/bot capturing/handling PR status would be > very good. > Knowing the state of a test from various browsers would be helpful at > review time. > > But this might require some extra work to support PR updates and so on. > CQ time seems like a reasonable target for step 1. > Well, step 0 might be to have a script that committers would run > themselves locally. > > > Again, many people don't use CQ. What do those people do? Manually run a > script? I don't think that's a workable solution. There are many people > just land patches from Xcode UI for example. > > In general, I don't think it's acceptable to introduce any extra step for > WebKit contributors. Let it be running a new script, potentially fix an > issue in GitHub PR, etc... > > There should be absolutely no new thing contributor has to run or monitor. > That should be the minimum bar for this upstreaming system to exist. > > Again, we can't even keep up with test failures on our own bots, and > people frequently land patches with broken tests or tests that fail on some > bots. There is no way we can succeed by introducing yet another step / > place humans has to care. That's simply unacceptable. > > - Ask committer to fix the WPT PR should there be any issue with it > (committer being informed of such issues through bugzilla). > > > I think this is too much of an overhead / a burden for WebKit > contributors. Now patch contributors need to worry about EWS, > build.webkit.org, and some random PR that gets created on Github failing. > > > Contributors would just need to worry about monitoring bugzilla, like they > are doing for EWS/CQ. > > > It depends on what kind of "monitoring" is required. > > > The number of conflicts should be fairly small hopefully. > I am not sure that we can come up with a solution that would handle > conflicts automatically. > > > In the case of a conflict, the patch should be rejected in EWS & CQ just > the same way a WebKit patch conflict would. > > - R. Niwa > > > -- > - R. Niwa > >
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
