No. Leave the -expected test alone with the "correct but different on this platform" expectation (of course, if the platform matches the generic version or some upstream version, you don't need an -expected.txt file at all).
-- Dirk On Fri, Jul 1, 2011 at 2:21 PM, Eric Seidel <e...@webkit.org> wrote: > So then you'd never want both in the same directory, doing so shoudl > be an error? > > -eric > > On Fri, Jul 1, 2011 at 2:18 PM, Ojan Vafai <o...@chromium.org> wrote: >> What Dirk said. It's just adding another layer into the fallback order. >> >> On Fri, Jul 1, 2011 at 1:54 PM, Dirk Pranke <dpra...@chromium.org> wrote: >>> >>> -failing should trump -expected. >>> >>> I also like Ojan's idea. >>> >>> I do not believe that -expected should be used to track "incorrect" >>> results, because that makes understanding how tests are supposed to >>> run dependent on the knowledge of the bug database as well. >>> >>> I think Ryosuke's concern is legitimate, both out of concern for >>> Chromium's long list of failures and for what would happen if other >>> ports started also running pixel tests, but I don't know if it's a big >>> enough concern to kill the idea. It would be interesting to see how >>> big of an impact there is, and, obviously, a given port could chose >>> not to use -failure files if it didn't want to. >>> >>> -- Dirk >>> >>> On Fri, Jul 1, 2011 at 12:56 PM, Eric Seidel <e...@webkit.org> wrote: >>> > I like the idea of -failing. But what happens when you have both >>> > -failing and -expected in the same directory? Are either accepted? >>> > (in which case it's like a file-system version of test-expetations >>> > flaky lists) >>> > >>> > On Fri, Jul 1, 2011 at 12:49 PM, Ojan Vafai <o...@chromium.org> wrote: >>> >> I proposed a while back to chromium folk that we minimize the use of >>> >> TEXT >>> >> and IMAGE and instead check in the failing results the way we do with >>> >> the >>> >> non-chromium ports*. I don't like that we rely on bugs to track that >>> >> the >>> >> result is incorrect though, so my suggestion was that we change the >>> >> filename >>> >> to indicate it. So, foo/bar-expected.txt becomes foo/bar-failing.txt >>> >> and we >>> >> just add the -failing version to the fallback order. >>> >> The main thing I like about this approach is that it allows you to >>> >> still >>> >> have a clear list of failing tests that need fixing. I believe that >>> >> with the >>> >> current model of checking in a failing result and filing a bug, failing >>> >> tests are forgotten about. >>> >> Ojan >>> >> * My original proposal to Chromium folk wanted to get rid of TEXT and >>> >> IMAGE >>> >> entirely from the expectations format. It was generally well received >>> >> except >>> >> it it makes handling certain temporary failures considerably more >>> >> difficult >>> >> (e.g. pulling in a new version of Skia). >>> >> >>> >> On Fri, Jul 1, 2011 at 11:09 AM, Adam Barth <aba...@webkit.org> wrote: >>> >>> >>> >>> You can do the same thing with NRWT that you can do with ORWT in this >>> >>> regard, but nothing new. The test_expectation.txt file does give you >>> >>> more fine-grained control than Skipped in the sense that you can >>> >>> distinguish between TEXT, IMAGE, CRASH, and TIMEOUT failures, but it >>> >>> doesn't let you distinguish between different sorts of TEXT failures, >>> >>> for example. >>> >>> >>> >>> My sense is that the test_expectation.txt file is already somewhat >>> >>> over complicated for the problem it solves. In this case, the >>> >>> workflow of changing the expected results and filing a bug to track >>> >>> the failure seems like a reasonable solution, especially if there's a >>> >>> keyword or master bug that lets you find all these bugs easily. >>> >>> >>> >>> Adam >>> >>> >>> >>> >>> >>> On Fri, Jul 1, 2011 at 10:58 AM, Adam Roben <aro...@apple.com> wrote: >>> >>> > When a test starts failing on a bot that uses old-run-webkit-tests, >>> >>> > we >>> >>> > typically check in expected failure results for that test (e.g., >>> >>> > <http://trac.webkit.org/changeset/90235>). That way we can find out >>> >>> > if the >>> >>> > test's behavior changes in the future even though the test is still >>> >>> > failing. >>> >>> > This is particularly useful for tests that are actually testing >>> >>> > multiple >>> >>> > things at once. (Maybe they should be broken up into multiple tests, >>> >>> > but >>> >>> > that's a different discussion.) >>> >>> > >>> >>> > Is there a way to achieve this with new-run-webkit-tests? I know >>> >>> > that >>> >>> > you can mark a test as an expected failure (either a text diff, or >>> >>> > an image >>> >>> > diff, or both). Does it let you specify that the test should fail in >>> >>> > a >>> >>> > particular way? >>> >>> _______________________________________________ >>> >>> webkit-dev mailing list >>> >>> webkit-dev@lists.webkit.org >>> >>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev >>> >> >>> >> >>> >> _______________________________________________ >>> >> webkit-dev mailing list >>> >> webkit-dev@lists.webkit.org >>> >> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev >>> >> >>> >> >>> > _______________________________________________ >>> > webkit-dev mailing list >>> > webkit-dev@lists.webkit.org >>> > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev >>> > >> >> > _______________________________________________ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev > _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
