That's about the awesomest thing ever. I want this in my build.webkit.org!
-eric On Fri, Nov 20, 2009 at 1:58 PM, Ojan Vafai <o...@chromium.org> wrote: > As of yesterday, we now retry any unexpected webkit failures. If they pass > the second time around, then we turn the bot orange and list the unexpected > flaky tests on the waterfall and at the end of the stdio of > run_webkit_test.py. If they fail the second time around we turn the bot red > as usual. > Q: Why not retry more than once? > A: We don't want to encourage ignoring flakiness. Even with retrying, > increased flakiness hurts us. But we have so many tests that fail ~1% of the > time that it's worth doing this as a stopgap while we try to fix flakiness. > Q: Won't this increase flakiness since people will now just ignore it? > A: The layout test dashboard will still show the tests as failing, so > they'll still show up as flaky. Hopefully the bot turning orange is enough > to encourage people to fix flaky tests. Also, tests that are very flaky will > still occasionally turn the bot red as they'll sometimes fail twice in a > row, so those really need to be fixed. > Q: Can I help fixing flakiness? That sounds hard. > A: Actually it turns out to be fairly straightforward to fix much of the > flakiness in webkit tests. A lot of it is just fixing obvious race > conditions in the test. > As proof that it's not that bad, here's a few flaky tests that Julie fixed > recently. > timeout increase in transition test: http://trac.webkit.org/changeset/51150 > remove setTimeout that was extra > watchdog: http://trac.webkit.org/changeset/51120 > remove setTimeout that was working around old > issue: http://trac.webkit.org/changeset/51088 > use onload to detect iframe loading: http://trac.webkit.org/changeset/49592 > And on the layout test dashboard, you can see they stopped being flaky: > http://src.chromium.org/viewvc/chrome/trunk/src/webkit/tools/layout_tests/flakiness_dashboard.html#tests=LayoutTests%2Ftransitions%2Fzero-duration-with-non-zero-delay-end.html%20LayoutTests%2Ffast%2Fencoding%2Fchar-encoding.html%20%20LayoutTests%2Ffast%2Floader%2Ffile-URL-with-port-number.html > > -- > Chromium Developers mailing list: chromium-dev@googlegroups.com > View archives, change email options, or unsubscribe: > http://groups.google.com/group/chromium-dev -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
