> On Nov 17, 2017, at 7:53 AM, Frédéric WANG <fred.w...@free.fr> wrote: > > On 17/11/2017 16:26, Maciej Stachowiak wrote: >> WebKit has a lot of tests that were regression tests for a specific >> bug fix, not as conformance tests (though they might b useful for that >> too). In such cases, the association with the bugs.webkit.org bug is >> more important than the spec URL. That’s particularly the case when >> the test has been changed multiple times to reflect further WebKit >> behavior changes. I know that I’ve personally found it very useful to >> look at revision history of tests, or search for bug numbers or >> keywords in LayoutTests/ChangeLog to find related tests. >> Not saying this is the sole purpose of tests, but losing this ability or >> making it more awkward would be a downside to deleting tests from the WebKit >> repo. > Well, I think we agree here, that's why "conformance" tests was > specified in my message. For such tests, connecting the history of tests > with the development of the specs is actually more important. For > example I recently noticed that some bugs with WOFF2 (on Apple's ports > only) and WebVTT have been ignored in WebKit because we don't > import/sync WPT tests for these specs.
I think importing new test suites is a different question than upstreaming/removing tests. In general importing more test suites is probably good, but we probably need to tackle the WPT performance problem before we pull in too many new WPT suites. I should also note that pulling in the test suite won't automatically get the bugs fixed or prioritized, or even filed in the bug tracker necessarily.eratwerATWeRTWAerATWertwAerer >> Maybe I’m missing something, but isn’t that already how it works in WebKit? >> EWS and buildbot run all the tests, but on your own machine you can get >> run-webkit-tests to run any subset you want. > Not everybody has the hardware to produce debug builds of WebKit on all > existing ports and run a big amount of tests :-) > > AFAIK, EWS run all the builds as well as all the tests on macOS/iOS, you > don't have options to select a subset of the tests or the ports. > Buildbots run all the builds & tests on all platforms *after* a patch > lands in the main repo. So we do too much for WIP patches (extra runtime > cost I'm mentioning here) and not enough to check the final patch before > landing (which causes different issues like extra rollout or gardening > commits or difficult regression bisecting). I find it super convenient that EWS runs all the tests even on WIP patches. It often catches test failures in tests I didn't think to run myself. I think it would be great if EWS could run tests on more platforms. Just running release regression tests on more platforms would be a big win, if debug builds are the gating factor. I guess it would be kind of neat to have a feature of "run the tests across all configurations, but only some of them". But I don't think I would prioritize it over having more kinds of tests, or finding ways to make the full test suite run faster. > > -- > Frédéric Wang - frederic-wang.fr > > _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
