I like the idea of -failing. But what happens when you have both -failing and -expected in the same directory? Are either accepted? (in which case it's like a file-system version of test-expetations flaky lists)
On Fri, Jul 1, 2011 at 12:49 PM, Ojan Vafai <o...@chromium.org> wrote: > I proposed a while back to chromium folk that we minimize the use of TEXT > and IMAGE and instead check in the failing results the way we do with the > non-chromium ports*. I don't like that we rely on bugs to track that the > result is incorrect though, so my suggestion was that we change the filename > to indicate it. So, foo/bar-expected.txt becomes foo/bar-failing.txt and we > just add the -failing version to the fallback order. > The main thing I like about this approach is that it allows you to still > have a clear list of failing tests that need fixing. I believe that with the > current model of checking in a failing result and filing a bug, failing > tests are forgotten about. > Ojan > * My original proposal to Chromium folk wanted to get rid of TEXT and IMAGE > entirely from the expectations format. It was generally well received except > it it makes handling certain temporary failures considerably more difficult > (e.g. pulling in a new version of Skia). > > On Fri, Jul 1, 2011 at 11:09 AM, Adam Barth <aba...@webkit.org> wrote: >> >> You can do the same thing with NRWT that you can do with ORWT in this >> regard, but nothing new. The test_expectation.txt file does give you >> more fine-grained control than Skipped in the sense that you can >> distinguish between TEXT, IMAGE, CRASH, and TIMEOUT failures, but it >> doesn't let you distinguish between different sorts of TEXT failures, >> for example. >> >> My sense is that the test_expectation.txt file is already somewhat >> over complicated for the problem it solves. In this case, the >> workflow of changing the expected results and filing a bug to track >> the failure seems like a reasonable solution, especially if there's a >> keyword or master bug that lets you find all these bugs easily. >> >> Adam >> >> >> On Fri, Jul 1, 2011 at 10:58 AM, Adam Roben <aro...@apple.com> wrote: >> > When a test starts failing on a bot that uses old-run-webkit-tests, we >> > typically check in expected failure results for that test (e.g., >> > <http://trac.webkit.org/changeset/90235>). That way we can find out if the >> > test's behavior changes in the future even though the test is still >> > failing. >> > This is particularly useful for tests that are actually testing multiple >> > things at once. (Maybe they should be broken up into multiple tests, but >> > that's a different discussion.) >> > >> > Is there a way to achieve this with new-run-webkit-tests? I know that >> > you can mark a test as an expected failure (either a text diff, or an image >> > diff, or both). Does it let you specify that the test should fail in a >> > particular way? >> _______________________________________________ >> webkit-dev mailing list >> webkit-dev@lists.webkit.org >> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev > > > _______________________________________________ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev > > _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
