Hi all, I have recently landed support for 'virtual test suites' in new-run-webkit-tests (as the subject suggests ;) ). A virtual test suite tells NRWT to run all the tests in directory 'foo' with an additional set of command line arguments being passed to DRT, and to look for baselines in directory 'bar' before looking in directory 'foo'.
As a concrete example, we're going to try and use this to remove the 'chromium-gpu' ports and just view gpu-specific tests as a virtual suite on top of the existing tests. So, we have a mapping that says: run all the tests in fast/canvas with --accelerated-2d-canvas being passed to DRT (as well as the normal flags), and look for results in LayoutTests/platform/chromium/virtual/gpu/fast/canvas before looking in the normal places. The mapping of virtual test suites is currently hardcoded in the NRWT python code (in the webkitpy/layout_tests/port/*.py files, to be precise), but it would be easy to make the mappings live as files in the tree as well. All of the normal tools (webkit-patch garden-o-matic, webkit-patch rebaseline, the flakiness dashboard) should work as if the virtual tests are regular tests. I doubt they do at the moment - I'm still testing and bug hunting. I believe this feature will be generally useful down the road. For example, one can imagine running a set of tests with a mocked-out set of controls (e.g., fake scrollbars) so that we can have platform-independent baselines, and then re-running a subset of the tests with native controls to ensure that they also work. I realize that this is a bit different than the normal practice for LayoutTests, where you often run tests differently by setting up a test differently and then including a SCRIPT to do the real test; the downside of the latter approach is that there's more boilerplate and you have to remember to do this every time a new test is added. That said, there are definitely still times (most times?) when that is the better approach. At any rate, like I said, it's an experiment and feedback is welcome. If you have ideas for how something like this might be useful but need additional features, etc., do let me know. Thanks, -- Dirk _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
