I had assumed that test_shell was also going to be modified, in order to produce IMAGEFAIL or TEXTFAIL automatically in "regressions.txt". So it wouldn't require any intervention other than a first pass through test_expectations.txt to mark the partial failures. I'm sure how often IMAGEFAIL -> FAIL regressions would happen (which I think is what Avi was trying to catch originally), but it might be useful for other things. For example, when triaging a bunch of new webkit merge failures without having to run the tests locally. IMAGEFAIL probably points to a Skia or CG issue, whereas the other might be a DOM issue (not 100% guaranteed, but a hint for grouping, anyway). Stephen
On Wed, Sep 23, 2009 at 11:34 PM, Pam Greene <[email protected]> wrote: > Call me a wet blanket, but I don't think there's a strong need for more > divergence in the file. Anything not passing is failing and needs looking > at; having a way to say "oh, it's 'only' the image that's bad" will increase > maintenance burden and support ignoring problems. Situations where we're > willing to ignore one type of failure for an extended time should be rare. > I'd vote for keeping FAIL meaning image and/or text, and IMAGEFAIL for > temporary use meaning image-only. > - Pam > > > On Wed, Sep 23, 2009 at 6:52 PM, Ojan Vafai <[email protected]> wrote: > >> I prefer IMAGE and TEXT since people maintaining these lists need to type >> these all the time. Also, longer names make for more bloat in the file and >> in the dashboard. Anyone who works with these lists even a small amount will >> know that IMAGE and TEXT refer to failures. >> >> We should really get rid of FAIL as a valid expectation (forces us to be >> more strict about what kind of failure it is), so the overlap of FAIL with >> IMAGE and TEXT is just temporary. >> >> Really, the FAIL part is redundant, no? Why not rename TIMEOUT/CRASH to >> TIMEOUT-FAIL/CRASH-FAIL? >> >> Ojan >> >> On Wed, Sep 23, 2009 at 6:32 PM, Dirk Pranke <[email protected]> wrote: >> >>> I think this plan sounds good, too. >>> >>> I'm mucking with those scripts a bit at the moment for the LTTF >>> reporting, so I can make this change tomorrow, unless someone else >>> would rather do it. >>> >>> I might actually prefer FAIL-TEXT and FAIL-IMAGE, but that's just me. >>> I agree that TEXTFAIL is better than TEXT. Anyone else care to express >>> a preference? >>> >>> -- Dirk >>> >>> On Wed, Sep 23, 2009 at 1:50 PM, Stephen White <[email protected]> >>> wrote: >>> > Could we make them TEXTFAIL and IMAGEFAIL, just to be clear? >>> > Stephen >>> > >>> > (And then post them to failblog if they're really embarassing.. J/K ;) >>> > On Wed, Sep 23, 2009 at 3:33 PM, Ojan Vafai <[email protected]> wrote: >>> >> >>> >> +pam, tc, darin in case they disagree with what I'm saying here. >>> >> >>> >> Also a bunch of current expectations would need to be modified. All >>> >> the cases where there is currently FAIL would need to be changed to >>> >> either FAIL or IMAGE or both if it's a text and image failure. You >>> >> should be able to get most of the data for this by looking at the >>> >> layout test dashboard. The only exception is you won't be able to >>> >> distinguish tests that fail both image and text from tests that only >>> >> fail image. >>> >> >>> >> A short-term solution could be to leave FAIL meaning IMAGE and/or TEXT >>> >> and adding IMAGE and TEXT for image-only and text-only failures. Then >>> >> we can gradually excise the FAIL lines from text_expectations. >>> >> >>> >> I think this would be a good permanent change, but I can see arguments >>> >> to the contrary. >>> >> >>> >> Ojan >>> >> >>> >> On Wed, Sep 23, 2009 at 12:25 PM, Ojan Vafai <[email protected]> >>> wrote: >>> >> > There is not. But adding it would be easy. There's been mention of >>> >> > doing this for a while, but noone has made the effort to make it >>> work. >>> >> > All you'd have to do is: >>> >> > -modify a few lines in TestExpectationsFile in >>> >> > src/webkit/tools/layout_tests/layout_package/test_expectations.py to >>> >> > add support for IMAGE in test_expectations. >>> >> > -treat IMAGE and other failures separately in >>> >> > src/webkit/tools/layout_tests/layout_package/compare_failures.py. >>> >> > Specifically, take test_failures.FailureImageHashMismatch out of >>> >> > FAILURE_TYPES and add an IMAGE_FAILURE_TYPE and use it below. >>> >> > >>> >> > Ojan >>> >> > >>> >> > On Wed, Sep 23, 2009 at 12:16 PM, Avi Drissman <[email protected]> >>> wrote: >>> >> >> I've been looking into the pixel test situation on the Mac, and it >>> >> >> isn't bad >>> >> >> at all. Of ~5300 tests that have png results, we're failing ~800, >>> most >>> >> >> of >>> >> >> which fall into huge buckets of easily-separable fail. >>> >> >> >>> >> >> Is there a way to specify that we're expecting an image compare to >>> fail >>> >> >> but >>> >> >> still want the layout to succeed? We don't want to turn off the >>> tests >>> >> >> entirely while we fix them and run the chance of breaking something >>> >> >> that >>> >> >> layout would have caught. >>> >> >> >>> >> >> Avi >>> >> >> >>> >> >> >> >>> >> >> >>> >> > >>> >> >>> >> >>> > >>> > >>> > >>> > -- >>> > All truth passes through three stages. First, it is ridiculed. Second, >>> it is >>> > violently opposed. Third, it is accepted as being self-evident. -- >>> > Schopenhauer >>> > >>> > > >>> > >>> >> >> > > > > -- All truth passes through three stages. First, it is ridiculed. Second, it is violently opposed. Third, it is accepted as being self-evident. -- Schopenhauer --~--~---------~--~----~------------~-------~--~----~ Chromium Developers mailing list: [email protected] View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~----------~----~----~----~------~----~------~--~---
