On Aug 21, 2012, at 3:23 PM, Ojan Vafai <[email protected]> wrote: > On Mon, Aug 20, 2012 at 6:03 PM, Maciej Stachowiak <[email protected]> wrote: > Here's how I imagine the workflow when a sheriff or just innocent bystander > notices a deterministically failing test. Follow this two-step algorithm: > > 1) Are you confident that the new result is an improvement or no worse? If > so, then simply update -expected.txt. > 2) Otherwise, copy the old result to > -<whatever-we-call-the-unexpected-pass-result>.txt, and check in the new > result as -<whatever-we-call-the-expected-failure-result.txt>. > > I think we should do this. I don't care much about the naming. > > This replaces all other approaches to marking expected failures, including > the Skipped list, overwriting -expected even you know the result is a > regression, marking the test in TestExpectations as Skip, Wontfix, Image, > Text, or Text+Image, or any of the other legacy techniques for marking an > expected failure reult. > > Don't forget suffixing the test with "-disabled"! We have 109 such tests at > the moment according to > http://code.google.com/searchframe#search/&exact_package=chromium&q=file:third_party/WebKit/LayoutTests/.*%5C-disabled$&type=cs. > I think we should also get rid of this. If we need a way to disable a test > across ports (e.g. because it crashes in cross-platform code), we should make > a Skipped/TestExpectations file in LayoutTest/platform instead of renaming > the test file.
I agree that renaming to -disabled should be phased out as well. I specifically did not cover failure modes that produce no result, such as crashes or hangs. Those should still be tracked via TestExpectations IMO. Likewise for nondeterministic expectations failures. Regards, Maciej
_______________________________________________ webkit-dev mailing list [email protected] http://lists.webkit.org/mailman/listinfo/webkit-dev
