-failing should trump -expected. I also like Ojan's idea.
I do not believe that -expected should be used to track "incorrect" results, because that makes understanding how tests are supposed to run dependent on the knowledge of the bug database as well. I think Ryosuke's concern is legitimate, both out of concern for Chromium's long list of failures and for what would happen if other ports started also running pixel tests, but I don't know if it's a big enough concern to kill the idea. It would be interesting to see how big of an impact there is, and, obviously, a given port could chose not to use -failure files if it didn't want to. -- Dirk On Fri, Jul 1, 2011 at 12:56 PM, Eric Seidel <e...@webkit.org> wrote: > I like the idea of -failing. But what happens when you have both > -failing and -expected in the same directory? Are either accepted? > (in which case it's like a file-system version of test-expetations > flaky lists) > > On Fri, Jul 1, 2011 at 12:49 PM, Ojan Vafai <o...@chromium.org> wrote: >> I proposed a while back to chromium folk that we minimize the use of TEXT >> and IMAGE and instead check in the failing results the way we do with the >> non-chromium ports*. I don't like that we rely on bugs to track that the >> result is incorrect though, so my suggestion was that we change the filename >> to indicate it. So, foo/bar-expected.txt becomes foo/bar-failing.txt and we >> just add the -failing version to the fallback order. >> The main thing I like about this approach is that it allows you to still >> have a clear list of failing tests that need fixing. I believe that with the >> current model of checking in a failing result and filing a bug, failing >> tests are forgotten about. >> Ojan >> * My original proposal to Chromium folk wanted to get rid of TEXT and IMAGE >> entirely from the expectations format. It was generally well received except >> it it makes handling certain temporary failures considerably more difficult >> (e.g. pulling in a new version of Skia). >> >> On Fri, Jul 1, 2011 at 11:09 AM, Adam Barth <aba...@webkit.org> wrote: >>> >>> You can do the same thing with NRWT that you can do with ORWT in this >>> regard, but nothing new. The test_expectation.txt file does give you >>> more fine-grained control than Skipped in the sense that you can >>> distinguish between TEXT, IMAGE, CRASH, and TIMEOUT failures, but it >>> doesn't let you distinguish between different sorts of TEXT failures, >>> for example. >>> >>> My sense is that the test_expectation.txt file is already somewhat >>> over complicated for the problem it solves. In this case, the >>> workflow of changing the expected results and filing a bug to track >>> the failure seems like a reasonable solution, especially if there's a >>> keyword or master bug that lets you find all these bugs easily. >>> >>> Adam >>> >>> >>> On Fri, Jul 1, 2011 at 10:58 AM, Adam Roben <aro...@apple.com> wrote: >>> > When a test starts failing on a bot that uses old-run-webkit-tests, we >>> > typically check in expected failure results for that test (e.g., >>> > <http://trac.webkit.org/changeset/90235>). That way we can find out if the >>> > test's behavior changes in the future even though the test is still >>> > failing. >>> > This is particularly useful for tests that are actually testing multiple >>> > things at once. (Maybe they should be broken up into multiple tests, but >>> > that's a different discussion.) >>> > >>> > Is there a way to achieve this with new-run-webkit-tests? I know that >>> > you can mark a test as an expected failure (either a text diff, or an >>> > image >>> > diff, or both). Does it let you specify that the test should fail in a >>> > particular way? >>> _______________________________________________ >>> webkit-dev mailing list >>> webkit-dev@lists.webkit.org >>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev >> >> >> _______________________________________________ >> webkit-dev mailing list >> webkit-dev@lists.webkit.org >> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev >> >> > _______________________________________________ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev > _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
