What Dirk said. It's just adding another layer into the fallback order. On Fri, Jul 1, 2011 at 1:54 PM, Dirk Pranke <dpra...@chromium.org> wrote:
> -failing should trump -expected. > > I also like Ojan's idea. > > I do not believe that -expected should be used to track "incorrect" > results, because that makes understanding how tests are supposed to > run dependent on the knowledge of the bug database as well. > > I think Ryosuke's concern is legitimate, both out of concern for > Chromium's long list of failures and for what would happen if other > ports started also running pixel tests, but I don't know if it's a big > enough concern to kill the idea. It would be interesting to see how > big of an impact there is, and, obviously, a given port could chose > not to use -failure files if it didn't want to. > > -- Dirk > > On Fri, Jul 1, 2011 at 12:56 PM, Eric Seidel <e...@webkit.org> wrote: > > I like the idea of -failing. But what happens when you have both > > -failing and -expected in the same directory? Are either accepted? > > (in which case it's like a file-system version of test-expetations > > flaky lists) > > > > On Fri, Jul 1, 2011 at 12:49 PM, Ojan Vafai <o...@chromium.org> wrote: > >> I proposed a while back to chromium folk that we minimize the use of > TEXT > >> and IMAGE and instead check in the failing results the way we do with > the > >> non-chromium ports*. I don't like that we rely on bugs to track that the > >> result is incorrect though, so my suggestion was that we change the > filename > >> to indicate it. So, foo/bar-expected.txt becomes foo/bar-failing.txt and > we > >> just add the -failing version to the fallback order. > >> The main thing I like about this approach is that it allows you to still > >> have a clear list of failing tests that need fixing. I believe that with > the > >> current model of checking in a failing result and filing a bug, failing > >> tests are forgotten about. > >> Ojan > >> * My original proposal to Chromium folk wanted to get rid of TEXT and > IMAGE > >> entirely from the expectations format. It was generally well received > except > >> it it makes handling certain temporary failures considerably more > difficult > >> (e.g. pulling in a new version of Skia). > >> > >> On Fri, Jul 1, 2011 at 11:09 AM, Adam Barth <aba...@webkit.org> wrote: > >>> > >>> You can do the same thing with NRWT that you can do with ORWT in this > >>> regard, but nothing new. The test_expectation.txt file does give you > >>> more fine-grained control than Skipped in the sense that you can > >>> distinguish between TEXT, IMAGE, CRASH, and TIMEOUT failures, but it > >>> doesn't let you distinguish between different sorts of TEXT failures, > >>> for example. > >>> > >>> My sense is that the test_expectation.txt file is already somewhat > >>> over complicated for the problem it solves. In this case, the > >>> workflow of changing the expected results and filing a bug to track > >>> the failure seems like a reasonable solution, especially if there's a > >>> keyword or master bug that lets you find all these bugs easily. > >>> > >>> Adam > >>> > >>> > >>> On Fri, Jul 1, 2011 at 10:58 AM, Adam Roben <aro...@apple.com> wrote: > >>> > When a test starts failing on a bot that uses old-run-webkit-tests, > we > >>> > typically check in expected failure results for that test (e.g., > >>> > <http://trac.webkit.org/changeset/90235>). That way we can find out > if the > >>> > test's behavior changes in the future even though the test is still > failing. > >>> > This is particularly useful for tests that are actually testing > multiple > >>> > things at once. (Maybe they should be broken up into multiple tests, > but > >>> > that's a different discussion.) > >>> > > >>> > Is there a way to achieve this with new-run-webkit-tests? I know that > >>> > you can mark a test as an expected failure (either a text diff, or an > image > >>> > diff, or both). Does it let you specify that the test should fail in > a > >>> > particular way? > >>> _______________________________________________ > >>> webkit-dev mailing list > >>> webkit-dev@lists.webkit.org > >>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev > >> > >> > >> _______________________________________________ > >> webkit-dev mailing list > >> webkit-dev@lists.webkit.org > >> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev > >> > >> > > _______________________________________________ > > webkit-dev mailing list > > webkit-dev@lists.webkit.org > > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev > > >
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
