On Thu, Jan 7, 2010 at 5:17 PM, Ojan Vafai <o...@chromium.org> wrote:
> Do we really need a separate set of bots for pixel tests? Lets just turn > the pixel tests on for the current bots. The only thing stopping us doing > that is the currently failing tests, hence the suggestion for adding an > expectations file (or we could skip all the failures). > > I don't know enough about text metrics changes between Mac releases. With > Windows releases, we've been able to support XP, Vista and 7 pretty easily > by using a generic theme for OS controls. Also, I think we have some hooks > to turn off cleartype or something. > ...and making sure all of the right / same fonts are installed :) -darin > There are only ~10 tests that needed custom results for Vista && Windows 7. > I wonder if a similar set of steps could be taken for supporting different > Mac releases. > > Ojan > > On Thu, Jan 7, 2010 at 5:08 PM, Eric Seidel <e...@webkit.org> wrote: > >> I'm totally in favor of adding test_expectations.txt like >> functionality to webkit (and we'll get it for free when Dirk finishes >> up-streaming run_webkit_tests.py) >> >> But the troubles with the pixel tests in the past were more to do with >> text metrics changing between OS releases, and individual font >> differences between machines. I suspect that those issues are very >> solvable. >> >> I think we mostly need someone willing to set up the pixel test bots. >> >> -eric >> >> On Thu, Jan 7, 2010 at 5:01 PM, Ojan Vafai <o...@chromium.org> wrote: >> > On Thu, Jan 7, 2010 at 10:22 AM, Darin Adler <da...@apple.com> wrote: >> >> >> >> On Jan 7, 2010, at 10:19 AM, Dimitri Glazkov wrote: >> >> > Are we planning to run pixel tests on the build bots? >> >> >> >> If we can get them green, we should. It’s a lot of work. We need a >> >> volunteer to do that work. We’ve tried before. >> > >> > Two possible long-term solutions come to mind: >> > 1. Turn the bots orange on pixel failures. They still need fixing, but >> are >> > not as severe as text diff failures. I'm not a huge fan of this, but >> it's an >> > option. >> > 2. Add in a concept of expected failures and only turn the bots red for >> > *unexpected* failurs. More details on this below. >> > In chromium-land, there's an expectations file that lists expected >> failures >> > and allows for distinguishing different types of failures (e.g. IMAGE >> vs. >> > TEXT). It's like Skipped lists, but doesn't necessarily skip the test. >> > Fixing the expected failures still needs doing of course, but can be >> done >> > asynchronously. The primary advantage of this approach is that we can >> turn >> > on pixel tests, keep the bots green and avoid further regressions. >> > Would something like that make sense for WebKit as a whole? To be clear, >> we >> > would be nearly as loathe to add tests to this file as we are about >> adding >> > them to the Skipped lists. This just provides a way forward. >> > While it's true that the bots used to be red more frequently with pixel >> > tests turned on, for the most part, there weren't significant pixel >> > regressions. Now, if you run the pixel tests on a clean build, there are >> a >> > number of failures and a very large number of hash-mismatches that are >> > within the failure tolerance level. >> > -Ojan >> > For reference, the format of the expectations file is something like >> this: >> > // Fails the image diff but not the text diff. >> > fast/forms/foo.html = IMAGE >> > // Fails just the text diff. >> > fast/forms/bar.html = TEXT >> > // Fails both the image and text diffs. >> > fast/forms/baz.html = IMAGE+TEXT >> > // Skips this test (e.g. because it hangs run-webkit-tests or causes >> other >> > tests to fail). >> > SKIP : fast/forms/foo1.html = IMAGE >> > _______________________________________________ >> > webkit-dev mailing list >> > webkit-dev@lists.webkit.org >> > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev >> > >> > >> > > > _______________________________________________ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev > >
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
