> Mozilla has been using this technique for years. Perhaps we can pick their > brains for some good tricks. Or, dare I say it, share some tests. > > Hi, hope I'm not crashing the party, and sorry I'm late :-). Let me just say a few things about reftests...
Maciej mentioned that a reftest can assert that <b> and style="font-weight:bold" are equivalent, but would not catch a bug that completely disabled font-weight. That is true. However, in our reftest framework you can also assert that two pages render differently, so you can test that font-weight:normal renders differently from font-weight:bold, which would catch a bug like that. If you look in the Mozilla reftests, there are many tests with 'sanity' in the name ensuring that pages that should render differently, do. You could still miss a weird bug like font-weight:bold making the text italic instead. But in practice, we hardly ever encounter bugs that cause some reftests to render incorrectly and still all tests pass. Over time we have learned a number of tricks for writing reftests. Here are a few: -- In general we require tests to match pixel-perfectly. It's always tempting to allow a fudge factor, but tiny differences can indicate significant bugs. -- We have nifty SVG filter tricks to help find those tiny differences, e.g. http://mxr.mozilla.org/mozilla-central/source/layout/tools/reftest/reftest-analyzer.xhtml -- If some pixels of a test are unpredictable or platform-dependent, but not relevant to the test, you can censor them out by placing an opaque element over them in the test and reference. -- In other tests with unpredictable pixel values, e.g. video, we use SVG filters to threshold pixel channel values. -- The Web usually has More Than One Way To Do It. E.g. for CSS gradients, a lot of our reference pages use canvas to render a reference gradient. -- Many Web features have particular cases that are easy to reduce to reftests even when general cases aren't. For example, box-shadow and text-shadow with no blur are trivial to test with reftests. You can create gradients using repeated stops to achieve abrupt transitions and test them against colored boxes. You could use a similar trick to test patterns in SVG text. -- Aggressive subpixel antialiasing is a real pain. For example, wrapping text in an overflow:hidden container isn't always a no-op even if the container is sized to its contents. Using sans-serif fonts helps, using CSS padding helps more. -- For speed, conciseness and overall expediency, you can often pack many variations of a test into a single reftest page, and in practice there's no significant downside to that. One might argue that although reftests are a good way to test what they test, they don't provide enough broad functional testing, e.g. to make sure that blurs "look right". I don't have good data to refute or confirm that. However, if we want to, we can get an image test just by making the reference page a PNG ... but I can only see two cases out of over 4000 where we felt we needed to do that. Overall, I'm extremely enthusiastic about reftests! As others mentioned, we'd like to get official CSS and SVG test suites using reftests; they're the best approach I know of for writing robust automated layout and rendering tests across platforms and browsers. Rob -- "He was pierced for our transgressions, he was crushed for our iniquities; the punishment that brought us peace was upon him, and by his wounds we are healed. We all, like sheep, have gone astray, each of us has turned to his own way; and the LORD has laid on him the iniquity of us all." [Isaiah 53:5-6]
_______________________________________________ webkit-dev mailing list [email protected] http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
