As we have previous discussed following https://lists.webkit.org/pipermail/webkit-dev/2012-March/019782.html, it's hard to judge whether a given reference result is enough to cover everything the test intends to test. e.g. you can have a bug such that both the test and the reference file ends up having the same rendering result.
- Ryosuke On Tue, Apr 10, 2012 at 2:26 PM, Robert Hogan <[email protected]> wrote: > Hi there, > > We currently add tests from the CSS 2.1 suite as we fix them. They get > added > to the css2.1/20110323 folder in LayoutTests. Most of them don't have > native reference tests yet in the suite so we (mostly I) have been adding > homebrew reference results to the folder to avoid generating pixel results > on all platforms. (see http://webkitmemes.tumblr.com/post/20788159625 !) > > These reference-results are easily removed once superseded but it might be > cleaner just to move them, and the associated css tests, to a folder of > their own in fast/css. That will allow css2.1/20110323 to be a clean import > that the 8500 or so passing tests can be added to in 20 or 30 batches.[1] > It will also make NRWT's reftests harness work with the suite. > > Does anyone object to that approach? The only thing going against it seems > to be the principle that imported tests should be stored separately and > together but this is more a case of using them to fix bugs and prevent > future regressions while allowing a clean import of the CSS 2.1 test suite > to take place in parallel. > > The problem this does not solve is how we avoid creating pixel results for > tests that already pass but which do not have reftests of their own. Again > I would be in favour of putting these in fast/css and keeping them there > until reftests are added to the suite. This would allow us to prevent them > regressing and come up with a reftest for them that can be submitted to the > CSS test suite guys. > > The end result would be that we only directly import to the css2.1 folder > those tests in the CSS test suite that have ref tests native to the suite. > > Thanks, > Robert > > > [1] I keep a local and relatively up to date copy of the passing and > failing tests in separate folders in my checkout. Yes I know I should > create bugs for them and get started on landing the passes. > > _______________________________________________ > webkit-dev mailing list > [email protected] > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev >
_______________________________________________ webkit-dev mailing list [email protected] http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
