Hi all, *Background*
Web Platform Tests is a suite of tests written for W3C specifications <https://github.com/w3c/web-platform-tests/>=, and Blink, Edge, Gecko, and WebKit all run them as a part of their continuous build system. Historically, each working group had their own repository for tests, but with CSS WG's tests <https://github.com/w3c/csswg-test> finally getting merged into the Web Platform Tests, we will have a single repository with all the tests from W3C. A few of us attended a meeting to discuss the future of this test suite <https://docs.google.com/document/d/1lqOBTvaoTUIFuxK0IQL1UfXtybq7Kw8JxBvD6FZSdYE/edit#heading=h.aha7njnu40vz> last Monday. One of the major topic was that Blink and Gecko have adopted the process to write the tests using testharness.js in their own test suites, and automatically merge into Web Platform Tests without having to go through another round of code review for Web Platform Tests. Given we do test-driven development in WebKit, I think WebKit should do the same. This will benefit other browser vendors to catch their bugs with our tests, and we will also benefit from other browser vendors "reviewing" our tests against their implementation and specifications. In the long term, I believe this will drastically improve the interoperability of the Web, and dramatically improve the quality of the tests we run against WebKit. In fact, there was a general consensus that we should even upstream pass-if-not-crash tests as well as style and layout invalidation tests to web-platform-tests as they tend to be undertested right now and almost all engines have bugs in this area. *Problems & Questions* In order to auto-merge tests from WebKit to Web Platform Tests, there are a few problems. *1. We need to start writing more if not all tests using testharness.js instead of our own js-test(-pre).js* I've heard of many complaints about testharness.js being too verbose and cumbersome to use. I agree with all those complaints but I don't think we can convince all other browser vendors to use our own testing script either since both Blink & Gecko are onboard with using testharness.js At this point, I'd argue that the benefit outweighs the cost of adopting testharness.js. Also, W3C have dropped many styling requirements for tests so we no longer need to have link/meta elements, etc... You simply include testharness.js and testharness-report.js. See my recent PR <https://github.com/w3c/web-platform-tests/commit/c4725e56c91954eaf51759feabec8c6560f10ade> for example. Do people still object to using testharness.js? If so, what are your reasons? *2. Where should we put upstremable tests?* We need a mechanism to identify & merge tests back from WebKit into Web Platform Tests. One way to do this is to start adding tests directly into LayoutTests/imported/w3c/web-platform-tests and then automatically identify those tests and upstream them. Assuming all upstream merges happen in timely happen, this should work fine. And it has a benefit that you get to see all related tests in one place. Another approach is to create another top-level directory like LayoutTests/exports/web-platform-tests. One downside of this approach is that we need to match the directory structure in order for any script to upstream tests to appropriate locations. Do people prefer one approach over another? Or have another idea? *3. What would be the process for upstreaming tests?* One possible approach is to create a PR request for every patch posted on Bugzilla with tests that can be merged into Web Platform Tests. Because we don't want to make Web Platform Tests less reliably, every PR request to Web Platform Tests go through Travis CI <https://travis-ci.org/> which runs the test hundreds of times to make sure it's not flaky. This Travis CI job (stability check) will then show up as an additional EWS bubble. Another approach is to wait until a patch is landed, and automatically create a PR for tests that can be upstream'ed. This has a downside that the stability check wouldn't run until the patch is landed, and it's unclear what happens they do fail. Do committers get emails about them? Do the patches then need to be rolled back? If we don't come up with an appropriate process, we can end up with a lot of broken PRs that never get fixed like those failing test expectations :( Again, do you prefer one or the other? Or do you have another proposal? - R. Niwa
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
