The best way to judge whether a reference result is correct is to submit the result to the W3C CSS 2.1 test suite and have it reviewed. The only way this test suite will get more reference results is if people like us volunteer to submit references. If it's useful to us to have these 'homebrew reference results' then it will be useful to everyone else who uses the suite.
The previous thread mentioned checking Mozilla to see if their test suite had references for particular tests. If that's the case, then we should either encourage them to submit the references to the W3C, or just do that ourselves on their behalf. Alan From: Ryosuke Niwa <[email protected]> As we have previous discussed following https://lists.webkit.org/pipermail/webkit-dev/2012-March/019782.html, it's hard to judge whether a given reference result is enough to cover everything the test intends to test. e.g. you can have a bug such that both the test and the reference file ends up having the same rendering result. - Ryosuke On Tue, Apr 10, 2012 at 2:26 PM, Robert Hogan <[email protected]> wrote: Hi there, We currently add tests from the CSS 2.1 suite as we fix them. They get added to the css2.1/20110323 folder in LayoutTests. Most of them don't have native reference tests yet in the suite so we (mostly I) have been adding homebrew reference results to the folder to avoid generating pixel results on all platforms. (see http://webkitmemes.tumblr.com/post/20788159625 !) These reference-results are easily removed once superseded but it might be cleaner just to move them, and the associated css tests, to a folder of their own in fast/css. That will allow css2.1/20110323 to be a clean import that the 8500 or so passing tests can be added to in 20 or 30 batches.[1] It will also make NRWT's reftests harness work with the suite. Does anyone object to that approach? The only thing going against it seems to be the principle that imported tests should be stored separately and together but this is more a case of using them to fix bugs and prevent future regressions while allowing a clean import of the CSS 2.1 test suite to take place in parallel. The problem this does not solve is how we avoid creating pixel results for tests that already pass but which do not have reftests of their own. Again I would be in favour of putting these in fast/css and keeping them there until reftests are added to the suite. This would allow us to prevent them regressing and come up with a reftest for them that can be submitted to the CSS test suite guys. The end result would be that we only directly import to the css2.1 folder those tests in the CSS test suite that have ref tests native to the suite. Thanks, Robert [1] I keep a local and relatively up to date copy of the passing and failing tests in separate folders in my checkout. Yes I know I should create bugs for them and get started on landing the passes. _______________________________________________ webkit-dev mailing list [email protected] http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev _______________________________________________ webkit-dev mailing list [email protected] http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
