So then you'd never want both in the same directory, doing so shoudl be an error?
-eric On Fri, Jul 1, 2011 at 2:18 PM, Ojan Vafai <o...@chromium.org> wrote: > What Dirk said. It's just adding another layer into the fallback order. > > On Fri, Jul 1, 2011 at 1:54 PM, Dirk Pranke <dpra...@chromium.org> wrote: >> >> -failing should trump -expected. >> >> I also like Ojan's idea. >> >> I do not believe that -expected should be used to track "incorrect" >> results, because that makes understanding how tests are supposed to >> run dependent on the knowledge of the bug database as well. >> >> I think Ryosuke's concern is legitimate, both out of concern for >> Chromium's long list of failures and for what would happen if other >> ports started also running pixel tests, but I don't know if it's a big >> enough concern to kill the idea. It would be interesting to see how >> big of an impact there is, and, obviously, a given port could chose >> not to use -failure files if it didn't want to. >> >> -- Dirk >> >> On Fri, Jul 1, 2011 at 12:56 PM, Eric Seidel <e...@webkit.org> wrote: >> > I like the idea of -failing. But what happens when you have both >> > -failing and -expected in the same directory? Are either accepted? >> > (in which case it's like a file-system version of test-expetations >> > flaky lists) >> > >> > On Fri, Jul 1, 2011 at 12:49 PM, Ojan Vafai <o...@chromium.org> wrote: >> >> I proposed a while back to chromium folk that we minimize the use of >> >> TEXT >> >> and IMAGE and instead check in the failing results the way we do with >> >> the >> >> non-chromium ports*. I don't like that we rely on bugs to track that >> >> the >> >> result is incorrect though, so my suggestion was that we change the >> >> filename >> >> to indicate it. So, foo/bar-expected.txt becomes foo/bar-failing.txt >> >> and we >> >> just add the -failing version to the fallback order. >> >> The main thing I like about this approach is that it allows you to >> >> still >> >> have a clear list of failing tests that need fixing. I believe that >> >> with the >> >> current model of checking in a failing result and filing a bug, failing >> >> tests are forgotten about. >> >> Ojan >> >> * My original proposal to Chromium folk wanted to get rid of TEXT and >> >> IMAGE >> >> entirely from the expectations format. It was generally well received >> >> except >> >> it it makes handling certain temporary failures considerably more >> >> difficult >> >> (e.g. pulling in a new version of Skia). >> >> >> >> On Fri, Jul 1, 2011 at 11:09 AM, Adam Barth <aba...@webkit.org> wrote: >> >>> >> >>> You can do the same thing with NRWT that you can do with ORWT in this >> >>> regard, but nothing new. The test_expectation.txt file does give you >> >>> more fine-grained control than Skipped in the sense that you can >> >>> distinguish between TEXT, IMAGE, CRASH, and TIMEOUT failures, but it >> >>> doesn't let you distinguish between different sorts of TEXT failures, >> >>> for example. >> >>> >> >>> My sense is that the test_expectation.txt file is already somewhat >> >>> over complicated for the problem it solves. In this case, the >> >>> workflow of changing the expected results and filing a bug to track >> >>> the failure seems like a reasonable solution, especially if there's a >> >>> keyword or master bug that lets you find all these bugs easily. >> >>> >> >>> Adam >> >>> >> >>> >> >>> On Fri, Jul 1, 2011 at 10:58 AM, Adam Roben <aro...@apple.com> wrote: >> >>> > When a test starts failing on a bot that uses old-run-webkit-tests, >> >>> > we >> >>> > typically check in expected failure results for that test (e.g., >> >>> > <http://trac.webkit.org/changeset/90235>). That way we can find out >> >>> > if the >> >>> > test's behavior changes in the future even though the test is still >> >>> > failing. >> >>> > This is particularly useful for tests that are actually testing >> >>> > multiple >> >>> > things at once. (Maybe they should be broken up into multiple tests, >> >>> > but >> >>> > that's a different discussion.) >> >>> > >> >>> > Is there a way to achieve this with new-run-webkit-tests? I know >> >>> > that >> >>> > you can mark a test as an expected failure (either a text diff, or >> >>> > an image >> >>> > diff, or both). Does it let you specify that the test should fail in >> >>> > a >> >>> > particular way? >> >>> _______________________________________________ >> >>> webkit-dev mailing list >> >>> webkit-dev@lists.webkit.org >> >>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev >> >> >> >> >> >> _______________________________________________ >> >> webkit-dev mailing list >> >> webkit-dev@lists.webkit.org >> >> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev >> >> >> >> >> > _______________________________________________ >> > webkit-dev mailing list >> > webkit-dev@lists.webkit.org >> > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev >> > > > _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev