Am 12.10.2010 um 22:43 schrieb James Robinson:
To add a concrete data point, http://trac.webkit.org/changeset/69517
caused a number of SVG tests to fail. It required 14 text
rebaselines for Mac and a further two more for Leopard (done by Adam
Barth). In order to pass the pixel tests in Chromium, it required
1506 new pixel baselines (checked in by the very brave Albert Wong, http://trac.webkit.org/changeset/69543)
. None of the rebaselining was done by the patch authors and in
general I would not expect a patch author that didn't work in
Chromium to be expected to update Chromium-specific baselines. I'm
a little skeptical of the claim that all SVG changes are run through
the pixel tests given that to date none of the affected platform/mac
SVG pixel baselines have been updated. This sort of mass-
rebaselining is required fairly regularly for minor changes in SVG
and in other parts of the codebase.
Dirk & me are running _every_ single SVG change through run-webkit-
tests --tolerance 0 -p svg.
Andreas Kling doesn't have access to a mac, and Dirk & me agreed to
rebaseline all affected SVG tests after his comment. Dirk did the 10.6
baseline updates, I added platform exceptions for 10.5 after Dirks
commit.
You can trust me, that we're running pixel tests.
I'd really like for the bots to run the pixel tests on every run,
preferably with 0 tolerance. We catch a lot of regressions by
running these tests on the Chromium bots that would probably
otherwise go unnoticed. However there is a large maintenance cost
associated with this coverage. We normally have two engineers (one
in PST, one elsewhere in the world) who watch the Chromium bots to
triage, suppress, and rebaseline tests as churn is introduced.
Questions:
- If the pixel tests were running either with a tolerance of 0 or
0.1, what would the expectation be for a patch like http://trac.webkit.org/changeset/69517
which requires hundreds of pixel rebaselines? Would the patch
author be expected to update the baselines for the platform/mac
port, or would someone else? Thus far the Chromium folks have been
the only ones actively maintaining the pixel baselines - which I
think is entirely reasonable since we're the only ones trying to run
the pixel tests on bots.
As I said before, Dirk & me maintain the mac pixel baselines for SVG.
If I would have written the patch, I had included the rebaselines,
though ppl like Andreas who don't have access to pixel tests, have to
be able to produce patches as well.
- Do we have the tools and infrastructure needed to do mass
rebaselines in WebKit currently? We've built a number of tools to
deal with the Chromium expectations, but since this has been a need
unique to Chromium so far the tools only work for Chromium.
Yes, webkit-patch rebaseline but I think Adam already mentioned it.
Cheers,
Niko
_______________________________________________
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
