I have a proposal that hopefully addresses everyone's concerns, is minimally different from the current format *and* unifies the format with Skipped lists (i.e. Skipped lists as they exist today are valid test_expectations.txt format). The changes from the current format are as follows: -Leaving out any "outcomes" means the test is skipped, unless the test has a SLOW modifier, in which case the implied outcome is PASS. -Remove the SKIP modifier entirely. -Make everything but the test name case-insensitive. -Have the test name be the last item on the line -Separate modifiers/outcomes/testname with a common delimiter (i.e. ":") -Any line starting with // or # is a comment -Including a bug entry is optional (maybe only if the test is skipped or wontfix?) -Bugs are listed as URLs, except for the bug_ojan format.
Examples: foo/bar.html # Skipped wontfix : foo/bar.html # Skipped and we never intend to fix it. For things like dashboard compatibility tests that only Apple will ever want to make pass wontfix : text : foo/bar.html # We never intend to fix this, but we expect it to run and fail text diff. Will still fail if the test times out or crashes. webkit.org/b/12345 : text image : foo/bar.html # Flaky. Sometimes only fails text diff, sometimes only fails pixel diff. slow mac debug : foo/bar.html # Given extra time to run on mac debug, but is expected to pass. image+text : foo/bar.html # Fails both text and pixel diffs bug_ojan : fail : foo/bar.html # Fails and ojan is taking responsibility to address the failure. bug_ojan : foo/bar.html # Skipped and ojan is taking responsibility to address it. # The following would give lint errors image : text : foo/bar.html # two outcomes listed in separate groupings slow text : foo/bar.html # outcome listed with non-outcome modifier crbug.com/12345 text : foo/bar.html # outcome listed with non-outcome modifier crbug.com/12345 : wontfix : foo/bar.html # two non-outcomes modifiers listed in separate groupings On Thu, May 17, 2012 at 1:12 AM, Ryosuke Niwa <[email protected]> wrote: > On Thu, May 17, 2012 at 1:06 AM, Maciej Stachowiak <[email protected]> wrote: >> >> On May 16, 2012, at 10:39 PM, Dirk Pranke <[email protected]> wrote: >> > There was a semi-logical basis, in that the stuff on the right of the >> > test clearly specified the outcome of the test (PASS, IMAGE, TEXT, >> > etc.). The stuff on the left was less well defined: there's the bug >> > numbers, the platform/configuration info (MAC LINUX RELEASE, etc.), >> > and some other stuff that there is less of a good place for (SLOW, >> > SKIP, WONTFIX). >> > >> > I think it makes sense to syntactically separate at least the >> > platform/configuration keywords from the outcome keywords. It might be >> > nice to separate the other things somehow as well, but I don't have >> > any great ideas for things that are clearly better than the existing >> > left/right convention. >> >> SKIP and WONTFIX seem parallel to PASS to me. I assume TEXT means a >> failure of text output and IMAGE means a pixel test failure? >> > > Yes. > > What does the build configuration info do? Does it apply the line to only >> those configurations? If that is the case, it does seem potentially >> different in kind, though maybe also better expressed by being able to >> combine multiple test_expectations files fro different platform/ >> directories. >> > > Yes. e.g. if you have: > BUGWK12345 WIN DEBUG : test.html = TEXT > then we expect test.html to have a failure of text output (i.e. > -expected.txt doesn't match) on Windows debug builds and expect it to pass > on all other platforms. > > I actually don't like the fact we're sort of duplicating platform fallback > structure here (e.g. we should be able to have a separate > text_expectations.txt in platform/chromium-win, platform/chromium-mac, > etc... to do the same but we don't currently support that) but there was a > strong resistance to do so among Chromium contributors the last time I > checked because that'll increase the number of files to maintain. > > - Ryosuke > > > _______________________________________________ > webkit-dev mailing list > [email protected] > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev > >
_______________________________________________ webkit-dev mailing list [email protected] http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
