I think the best solution to this pixel matching problem is ref tests. How practical would it be to use ref tests for SVG?
Simon On Oct 8, 2010, at 12:43 PM, Dirk Pranke wrote: > Jeremy is correct; the Chromium port has seen real regressions that > virtually no concept of a fuzzy match that I can imagine would've > caught. > new-run-webkit-tests doesn't currently support the tolerance concept > at al, and I am inclined to argue that it shouldn't. > > However, I frequently am wrong about things, so it's quite possible > that there are good arguments for supporting it that I'm not aware of. > I'm not particularly interested in working on a tool that doesn't do > what the group wants it to do, and I would like all of the other > WebKit ports to be running pixel tests by default (and > new-run-webkit-tests ;) ) since I think it catches bugs. > > As far as I know, the general sentiment on the list has been that we > should be running pixel tests by default, and the reason that we > aren't is largely due to the work involved in getting them back up to > date and keeping them up to date. I'm sure that fuzzy matching reduces > the work load, especially for the sort of mismatches caused by > differences in the text antialiasing. > > In addition, I have heard concerns that we'd like to keep fuzzy > matching because people might potentially get different results on > machines with different hardware configurations, but I don't know that > we have any confirmed cases of that (except for arguably the case of > different code paths for gpu-accelerated rendering vs. unaccelerated > rendering). > > If we made it easier to maintain the baselines (improved tooling like > the chromium's rebaselining tool, add reftest support, etc.) are there > still compelling reasons for supporting --tolerance -based testing as > opposed to exact matching? > > -- Dirk > > On Fri, Oct 8, 2010 at 11:14 AM, Jeremy Orlow <jor...@chromium.org> wrote: >> I'm not an expert on Pixel tests, but my understanding is that in Chromium >> (where we've always run with tolerance 0) we've seen real regressions that >> would have slipped by with something like tolerance 0.1. When you have >> 0 tolerance, it is more maintenance work, but if we can avoid regressions, >> it seems worth it. >> J >> >> On Fri, Oct 8, 2010 at 10:58 AM, Nikolas Zimmermann >> <zimmerm...@physik.rwth-aachen.de> wrote: >>> >>> Am 08.10.2010 um 19:53 schrieb Maciej Stachowiak: >>> >>>> >>>> On Oct 8, 2010, at 12:46 AM, Nikolas Zimmermann wrote: >>>> >>>>> >>>>> Am 08.10.2010 um 00:44 schrieb Maciej Stachowiak: >>>>> >>>>>> >>>>>> On Oct 7, 2010, at 6:34 AM, Nikolas Zimmermann wrote: >>>>>> >>>>>>> Good evening webkit folks, >>>>>>> >>>>>>> I've finished landing svg/ pixel test baselines, which pass with >>>>>>> --tolerance 0 on my 10.5 & 10.6 machines. >>>>>>> As the pixel testing is very important for the SVG tests, I'd like to >>>>>>> run them on the bots, experimentally, so we can catch regressions >>>>>>> easily. >>>>>>> >>>>>>> Maybe someone with direct access to the leopard & snow leopard bots, >>>>>>> could just run "run-webkit-tests --tolerance 0 -p svg" and mail me the >>>>>>> results? >>>>>>> If it passes, we could maybe run the pixel tests for the svg/ >>>>>>> subdirectory on these bots? >>>>>> >>>>>> Running pixel tests would be great, but can we really expect the >>>>>> results to be stable cross-platform with tolerance 0? Perhaps we should >>>>>> start with a higher tolerance level. >>>>> >>>>> Sure, we could do that. But I'd really like to get a feeling, for what's >>>>> problematic first. If we see 95% of the SVG tests pass with --tolerance 0, >>>>> and only a few need higher tolerances >>>>> (64bit vs. 32bit aa differences, etc.), I could come up with a per-file >>>>> pixel test tolerance extension to DRT, if it's needed. >>>>> >>>>> How about starting with just one build slave (say. Mac Leopard) that >>>>> runs the pixel tests for SVG, with --tolerance 0 for a while. I'd be happy >>>>> to identify the problems, and see >>>>> if we can make it work, somehow :-) >>>> >>>> The problem I worry about is that on future Mac OS X releases, rendering >>>> of shapes may change in some tiny way that is not visible but enough to >>>> cause failures at tolerance 0. In the past, such false positives arose from >>>> time to time, which is one reason we added pixel test tolerance in the >>>> first >>>> place. I don't think running pixel tests on just one build slave will help >>>> us understand that risk. >>> >>> I think we'd just update the baseline to the newer OS X release, then, >>> like it has been done for the tiger -> leopard, leopard -> snow leopard >>> switch? >>> platform/mac/ should always contain the newest release baseline, when >>> therere are differences on leopard, the results go into >>> platform/mac-leopard/ >>> >>>> Why not start with some low but non-zero tolerance (0.1?) and see if we >>>> can at least make that work consistently, before we try the bolder step of >>>> tolerance 0? >>>> Also, and as a side note, we probably need to add more build slaves to >>>> run pixel tests at all, since just running the test suite without pixel >>>> tests is already slow enough that the testers are often significantly >>>> behind >>>> the builders. >>> >>> Well, I thought about just running the pixel tests for the svg/ >>> subdirectory as a seperate step, hence my request for tolerance 0, as the >>> baseline passes without problems at least on my & Dirks machine already. >>> I wouldnt' want to argue running 20.000+ pixel tests with tolerance 0 as >>> first step :-) But the 1000 SVG tests, might be fine, with tolerance 0? >>> >>> Even tolerance 0.1 as default for SVG would be fine with me, as long as we >>> can get the bots to run the SVG pixel tests :-) >>> >>> Cheers, >>> Niko >>> >>> _______________________________________________ >>> webkit-dev mailing list >>> webkit-dev@lists.webkit.org >>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev >> >> >> _______________________________________________ >> webkit-dev mailing list >> webkit-dev@lists.webkit.org >> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev >> >> > _______________________________________________ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
