FWIW, in Blink we stopped rewriting the testharness.js paths before switching to wptserve, by instead rewriting those URLs only when running LayoutTests: https://cs.chromium.org/chromium/src/content/shell/renderer/layout_test/blink_test_runner.cc?type=cs&q=content/shell/renderer/layout_test/blink_test_runner.cc&l=221
So, we know that it's possible to run a lot of the tests without wptserve without modifying the tests. However, trying to list all the tests that do or don't need wptserve seems like a lot of work, and I think we're now using wptserve for all tests. On Sun, Feb 5, 2017 at 8:05 PM youenn fablet <youe...@gmail.com> wrote: > I too am a big proponent of us moving more and more towards WPT. > As part of the streams and fetch API implementation, most of the related > functional tests have been made as WPT. > The benefits of having tests being improved and updated largely outweighs > the testharness.js initial cost. > Somehow, these tests are becoming part of their related specs. > > The two complaints I heard against testharness.js are: > - They are less easy to debug as test harness.js does not print out > messages for each assert. There might be room for updating testharness to > support that > - Async tests are less easy to write. While this is probably true, > testharness has support for promise-based tests which are not verbose. > WPT has also some benefits of its own, like the possibility to run easily > the same tests in worker/window environments, the flexible python-based > server... > > One complaint I heard against WPT tests is that they require being run > behind a server. > This is becoming more and more true. > There is still a large portion of tests that could be run locally if we > update the paths to test harness.js/testharnessreport,js. > I am not a big fan of modify any WPT test but maybe we should consider > switching back to modifying these paths at import and export time. > > I would also like to go in a direction where writing WPT tests is the > default for WebKit layout test. > For that to happen, we need an easy way to upstream tests from WebKit to > WPT GitHub. > > I would tend to do the following: > - Write/submit WPT tests in WebKit as regular WebKit testharness.js layout > tests through bugzilla > - At commit queue time, automatically create a PR to WPT GitHub containing > the changes of the WebKit patch > - Ask committer to fix the WPT PR should there be any issue with it > (committer being informed of such issues through bugzilla). > > Le dim. 5 févr. 2017 à 15:42, Ryosuke Niwa <rn...@webkit.org> a écrit : > > Hi all, > > *Background* > > Web Platform Tests is a suite of tests written for W3C specifications > <https://github.com/w3c/web-platform-tests/>=, > and Blink, Edge, Gecko, and WebKit all run them as a part of their > continuous build system. > > Historically, each working group had their own repository for tests, > but with CSS WG's tests <https://github.com/w3c/csswg-test> finally > getting merged into the Web Platform Tests, > we will have a single repository with all the tests from W3C. > > A few of us attended a meeting to discuss the future of this test suite > <https://docs.google.com/document/d/1lqOBTvaoTUIFuxK0IQL1UfXtybq7Kw8JxBvD6FZSdYE/edit#heading=h.aha7njnu40vz> > last Monday. > One of the major topic was that Blink and Gecko have adopted the process > to write the tests > using testharness.js in their own test suites, and automatically merge > into Web Platform Tests > without having to go through another round of code review for Web Platform > Tests. > > Given we do test-driven development in WebKit, I think WebKit should do > the same. > This will benefit other browser vendors to catch their bugs with our > tests, and we will also benefit > from other browser vendors "reviewing" our tests against their > implementation and specifications. > > In the long term, I believe this will drastically improve the > interoperability of the Web, > and dramatically improve the quality of the tests we run against WebKit. > > In fact, there was a general consensus that we should even upstream > pass-if-not-crash tests > as well as style and layout invalidation tests to web-platform-tests as > they tend to be undertested > right now and almost all engines have bugs in this area. > > *Problems & Questions* > > In order to auto-merge tests from WebKit to Web Platform Tests, there are > a few problems. > > *1. We need to start writing more if not all tests using testharness.js > instead of our own js-test(-pre).js* > > I've heard of many complaints about testharness.js being too verbose and > cumbersome to use. > I agree with all those complaints but I don't think we can convince all > other browser vendors > to use our own testing script either since both Blink & Gecko are onboard > with using testharness.js > > At this point, I'd argue that the benefit outweighs the cost of adopting > testharness.js. > Also, W3C have dropped many styling requirements for tests so we no longer > need to have > link/meta elements, etc... You simply include testharness.js and > testharness-report.js. > > See my recent PR > <https://github.com/w3c/web-platform-tests/commit/c4725e56c91954eaf51759feabec8c6560f10ade> > for > example. > > Do people still object to using testharness.js? If so, what are your > reasons? > > > *2. Where should we put upstremable tests?* > > We need a mechanism to identify & merge tests back from WebKit into Web > Platform Tests. > > One way to do this is to start adding tests directly into > LayoutTests/imported/w3c/web-platform-tests > and then automatically identify those tests and upstream them. > > Assuming all upstream merges happen in timely happen, this should work > fine. > And it has a benefit that you get to see all related tests in one place. > > Another approach is to create another top-level directory like > LayoutTests/exports/web-platform-tests. > One downside of this approach is that we need to match the directory > structure > in order for any script to upstream tests to appropriate locations. > > Do people prefer one approach over another? Or have another idea? > > > *3. What would be the process for upstreaming tests?* > > One possible approach is to create a PR request for every patch posted on > Bugzilla > with tests that can be merged into Web Platform Tests. > > Because we don't want to make Web Platform Tests less reliably, > every PR request to Web Platform Tests go through Travis CI > <https://travis-ci.org/> > which runs the test hundreds of times to make sure it's not flaky. > > This Travis CI job (stability check) will then show up as an additional > EWS bubble. > > > Another approach is to wait until a patch is landed, and automatically > create a PR > for tests that can be upstream'ed. This has a downside that the stability > check > wouldn't run until the patch is landed, and it's unclear what happens they > do fail. > > Do committers get emails about them? Do the patches then need to be rolled > back? > If we don't come up with an appropriate process, we can end up with a lot > of > broken PRs that never get fixed like those failing test expectations :( > > > Again, do you prefer one or the other? Or do you have another proposal? > > - R. Niwa > > _______________________________________________ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev > > _______________________________________________ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev >
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev