The two complaints I heard against testharness.js are: > - They are less easy to debug as test harness.js does not print out > messages for each assert. There might be room for updating testharness to > support that > - Async tests are less easy to write. While this is probably true, > testharness has support for promise-based tests which are not verbose. > WPT has also some benefits of its own, like the possibility to run easily > the same tests in worker/window environments, the flexible python-based > server... > > One complaint I heard against WPT tests is that they require being run > behind a server. > This is becoming more and more true. > There is still a large portion of tests that could be run locally if we > update the paths to test harness.js/testharnessreport,js. > I am not a big fan of modify any WPT test but maybe we should consider > switching back to modifying these paths at import and export time. > > > We should probably do this for the sake of making tests more debuggable. > Having to run a HTTP/HTTPS server makes testing WebCore against a test > needlessly harder especially for things like core DOM API which doesn't > require any network stack. >
Or we could make our tooling better to also handle LayoutTests/http tests more easily. In any case, our CI infrastructure should run WPT tests as closely as specified by WPT. > I would tend to do the following: > - Write/submit WPT tests in WebKit as regular WebKit testharness.js layout > tests through bugzilla > - At commit queue time, automatically create a PR to WPT GitHub containing > the changes of the WebKit patch > > > I think commit queue time is too late. Remember that not everyone uses > commit queue. Also, if there was any issue with a test, we should be able > to tell that prior to landing the patch. > Ideally, having one EWS bullet/bot capturing/handling PR status would be very good. Knowing the state of a test from various browsers would be helpful at review time. But this might require some extra work to support PR updates and so on. CQ time seems like a reasonable target for step 1. Well, step 0 might be to have a script that committers would run themselves locally. > - Ask committer to fix the WPT PR should there be any issue with it > (committer being informed of such issues through bugzilla). > > > I think this is too much of an overhead / a burden for WebKit > contributors. Now patch contributors need to worry about EWS, > build.webkit.org, and some random PR that gets created on Github failing. > Contributors would just need to worry about monitoring bugzilla, like they are doing for EWS/CQ. The number of conflicts should be fairly small hopefully. I am not sure that we can come up with a solution that would handle conflicts automatically. > The fact people have to worry about EWS & build.webkit.org separately is > bad enough. We shouldn't be introducing yet another place people have to > monitor / care about. > > - R. Niwa > >
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev