On Wed, Apr 11, 2012 at 12:31 PM, Robert Hogan <[email protected]> wrote: > On Wednesday 11 April 2012 20:27:23 Dirk Pranke wrote: >> >> What does "clean up the existing folder" entail? >> > > Just move any -expected.html file out of there and generate pixel results. > Or move the test and its -expected.html into a folder in fast/css.
Okay, following a conversation on #irc, I think I understand this better. There do appear to be differences of opinion over what to do here, so I will try to describe this from ground zero. The CSS test suite has ~9000 tests. the correctness of which is established manually (and visually). We would like to import these tests and run them automatically. Some small number of tests (a few hundred?) do have reference html, but most don't. Since most tests do not come with reference html or expected PNGs, we have to add one or the other. Currently we have added ~400 or so to the tree, and when we add them we have been writing -expected.html reference results to avoid generating pixel results where possible. This raises at least two questions, which have been confusing the discussion until now: 1) What is the best way to add these tests to the tree? Should we add them one-at-a-time with -expected files that we have created? Should we add them all at once and SKIP the tests until we have generated -expected results for each test? Or should we only import subsets that have official -expected files? 2) Is it acceptable for someone other than the test author to write -expected html references? What bar do we need to have to ensure that the reference file will catch the bug the initial test is intended to catch? Do we (or should we) maintain a similar bar for pixel tests today? Personally, I don't care that much about (1), as long as we can track the provenance of the tests (where they came from) ... I would be happy with us importing tests one at a time and adding our own results, and hopefully feeding them back up stream to the W3C. I care much more about (2): I believe we should not be introducing new tests that only have pixel tests for results unless absolutely necessary - the cost of maintaining pixel test results vastly outweighs the potential benefit we get by having a reference file that won't break the same way the test breaks. I believe it is reasonably for someone other than the test author to write a reference html for a test and get it to be good enough that it will provide more value than a pixel test result, and we should be encouraging that. I believe we're hoping to discuss this more broadly at the contributors meeting, but it might be good to discuss here first. Thoughts? -- Dirk _______________________________________________ webkit-dev mailing list [email protected] http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
