I don't think this is just about ignoring image-only results on mac for the short-term. My subjective sense is that we have many tests that start out as failing only image comparisons (e.g. due to theming), but over time another failure creeps in that causes a text failure that goes unnoticed. We'd ideally like to notice that extra regression when it happens as it might be easy to identify and fix right at the time the regression occurs.
That said, I share your reluctance to add more complexity to this file. Consider me on the fence. :) Ojan On Wed, Sep 23, 2009 at 8:34 PM, Pam Greene <[email protected]> wrote: > Call me a wet blanket, but I don't think there's a strong need for more > divergence in the file. Anything not passing is failing and needs looking > at; having a way to say "oh, it's 'only' the image that's bad" will increase > maintenance burden and support ignoring problems. Situations where we're > willing to ignore one type of failure for an extended time should be rare. > I'd vote for keeping FAIL meaning image and/or text, and IMAGEFAIL for > temporary use meaning image-only. > - Pam > > > On Wed, Sep 23, 2009 at 6:52 PM, Ojan Vafai <[email protected]> wrote: > >> I prefer IMAGE and TEXT since people maintaining these lists need to type >> these all the time. Also, longer names make for more bloat in the file and >> in the dashboard. Anyone who works with these lists even a small amount will >> know that IMAGE and TEXT refer to failures. >> >> We should really get rid of FAIL as a valid expectation (forces us to be >> more strict about what kind of failure it is), so the overlap of FAIL with >> IMAGE and TEXT is just temporary. >> >> Really, the FAIL part is redundant, no? Why not rename TIMEOUT/CRASH to >> TIMEOUT-FAIL/CRASH-FAIL? >> >> Ojan >> >> On Wed, Sep 23, 2009 at 6:32 PM, Dirk Pranke <[email protected]> wrote: >> >>> I think this plan sounds good, too. >>> >>> I'm mucking with those scripts a bit at the moment for the LTTF >>> reporting, so I can make this change tomorrow, unless someone else >>> would rather do it. >>> >>> I might actually prefer FAIL-TEXT and FAIL-IMAGE, but that's just me. >>> I agree that TEXTFAIL is better than TEXT. Anyone else care to express >>> a preference? >>> >>> -- Dirk >>> >>> On Wed, Sep 23, 2009 at 1:50 PM, Stephen White <[email protected]> >>> wrote: >>> > Could we make them TEXTFAIL and IMAGEFAIL, just to be clear? >>> > Stephen >>> > >>> > (And then post them to failblog if they're really embarassing.. J/K ;) >>> > On Wed, Sep 23, 2009 at 3:33 PM, Ojan Vafai <[email protected]> wrote: >>> >> >>> >> +pam, tc, darin in case they disagree with what I'm saying here. >>> >> >>> >> Also a bunch of current expectations would need to be modified. All >>> >> the cases where there is currently FAIL would need to be changed to >>> >> either FAIL or IMAGE or both if it's a text and image failure. You >>> >> should be able to get most of the data for this by looking at the >>> >> layout test dashboard. The only exception is you won't be able to >>> >> distinguish tests that fail both image and text from tests that only >>> >> fail image. >>> >> >>> >> A short-term solution could be to leave FAIL meaning IMAGE and/or TEXT >>> >> and adding IMAGE and TEXT for image-only and text-only failures. Then >>> >> we can gradually excise the FAIL lines from text_expectations. >>> >> >>> >> I think this would be a good permanent change, but I can see arguments >>> >> to the contrary. >>> >> >>> >> Ojan >>> >> >>> >> On Wed, Sep 23, 2009 at 12:25 PM, Ojan Vafai <[email protected]> >>> wrote: >>> >> > There is not. But adding it would be easy. There's been mention of >>> >> > doing this for a while, but noone has made the effort to make it >>> work. >>> >> > All you'd have to do is: >>> >> > -modify a few lines in TestExpectationsFile in >>> >> > src/webkit/tools/layout_tests/layout_package/test_expectations.py to >>> >> > add support for IMAGE in test_expectations. >>> >> > -treat IMAGE and other failures separately in >>> >> > src/webkit/tools/layout_tests/layout_package/compare_failures.py. >>> >> > Specifically, take test_failures.FailureImageHashMismatch out of >>> >> > FAILURE_TYPES and add an IMAGE_FAILURE_TYPE and use it below. >>> >> > >>> >> > Ojan >>> >> > >>> >> > On Wed, Sep 23, 2009 at 12:16 PM, Avi Drissman <[email protected]> >>> wrote: >>> >> >> I've been looking into the pixel test situation on the Mac, and it >>> >> >> isn't bad >>> >> >> at all. Of ~5300 tests that have png results, we're failing ~800, >>> most >>> >> >> of >>> >> >> which fall into huge buckets of easily-separable fail. >>> >> >> >>> >> >> Is there a way to specify that we're expecting an image compare to >>> fail >>> >> >> but >>> >> >> still want the layout to succeed? We don't want to turn off the >>> tests >>> >> >> entirely while we fix them and run the chance of breaking something >>> >> >> that >>> >> >> layout would have caught. >>> >> >> >>> >> >> Avi >>> >> >> >>> >> >> >> >>> >> >> >>> >> > >>> >> >>> >> >>> > >>> > >>> > >>> > -- >>> > All truth passes through three stages. First, it is ridiculed. Second, >>> it is >>> > violently opposed. Third, it is accepted as being self-evident. -- >>> > Schopenhauer >>> > >>> > >>> > >>> > >>> >> >> > --~--~---------~--~----~------------~-------~--~----~ Chromium Developers mailing list: [email protected] View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~----------~----~----~----~------~----~------~--~---
