On Tue, May 8, 2012 at 1:04 PM, Dirk Pranke <[email protected]> wrote: > > On Tue, May 8, 2012 at 12:42 PM, Ryosuke Niwa <[email protected]> wrote: > > On Wed, Apr 25, 2012 at 2:18 PM, Dirk Pranke <[email protected]> > wrote: > >> I am also more than a little leery of mixing -expected.{txt,png} > >> results with -expected.html results; I feel like it would be very > >> confusing, and it would lose many of the advantages of reftests, since > >> we'd presumably have to update the reference every time as often as we > >> update pixel tests. [We could create a "fake" reftest that just > >> contained the 800x600 pixel dump, but I'm not sure if that's better or > >> not]. > > > > I don't think we will lose advantages. Without some sort of the current > > result, we will not be able to catch regressions. > > According to Maciej, we've caught many regressions by using conformance > > tests this way. > > I have no doubt that you can catch regressions by using conformance > tests this way. My concern -- which is expressed throughout and which > you seemed to either miss or downplay -- is that adding more tests > creates more work, especially for pixel tests (and it slows down build > and test cycles, obviously). I don't think we should just add tests > because they exist somewhere on the web; they may provide no > additional coverage beyond the tests we already have. >
Yes, that's why we want reviews. And that's why we should only import tests from W3C instead of other browser vendors. We expect W3C to have some guideline on avoiding test duplicates. I feel like we need a stronger mechanism to either check that new test > suites do cover more functionality or we move to obsolete tests we > already have. > We can't really verify that two tests test same functionality, etc... automatically. Also, the most general form of this question is undecidable. However, there are a couple of ways to mitigate this issue: 1. Upstream as many layout tests as possible to W3C 2. Delete duplicate layout tests as we import more tests from W3C To be clear, I am all for importing test suites when we believe they > are comprehensive or do cover things we don't cover well now. This should probably be judged by individual reviewers. But, for example, rather than having four different test suites for > flexbox, I > would rather see us have one good one. > Sure but I'd that's a really hard problem to solve. >> Also, I was under the impression that (a) the W3C is mostly focused on > >> ref tests going forward and (b) we had agreed in that discussion that > >> we wouldn't import non-ref tests? Did something change in a discussion > >> after that session? > > > > No. We had agreed to import all tests regardless of whether they're > reftests > > or not because non-reftests can still catch future regressions or > > progressions. And the number of PNG files added to the repository wasn't > > considered as a valid counter-argument due to this utility. > > Well, I certainly didn't agree to it :) My concern is not the # of > PNGs so much as the cost of maintenance. > Sure, that's a valid concern but the overwhelming majority of the people in the room (e.g. Darin, Maciej, etc...) seemed to agree that this is a good idea. > I don't understand your proposal about adding platform/webkit. Why do we > > want that? As far as I know, there are no files in W3C test directories > that > > end with -expected.txt or -expected.png. > > The idea would be that no webkit-specific files would live in the test > directory, only files received from upstream. My thinking was that it > would make importing new versions easier and it would be easier to > understand what was ours vs. what was theirs. I don't feel that > strongly about this, though, it was just an idea. That'll be nice indeed. But if we're going this route, we should probably move all existing -expected.* to this directory as well. So this is probably a tangential issue. Similar to: >> By "ultimately move all existing tests", I assume you're including > >> tests that are currently in LayoutTests that have not come from (or > >> been submitted to) the W3C, e.g., the tests in fast/ ? > > > > Yes. > > I think reorganizing our existing test tree is an entirely different > discussion. I'm all for it, I just don't want to confuse it with the > discussion about importing test suites. > - Ryosuke
_______________________________________________ webkit-dev mailing list [email protected] http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
