On Fri, Jan 8, 2010 at 12:14 PM, Alexey Proskuryakov <[email protected]> wrote: > > It's not clear where the bug is though. If there is an uninitialized memory > read in a later test, then earlier tests that randomly affect it can be > perfectly correct. The fact that on other platforms the problem was not > fixed support this hypothesis to some degree.
Agreed, we don't fully understand this issue, thus things are still broken. They're just differently broken than they were before. We could continue a slightly altered version of the previous brokenness by checking in failing results. I encouraged Ossy to skip them instead because at least with a skipped list we have a single list of issues to fix. With checked in failures it's never clear whether they're failures or not. I'm happy with either solution if you'd prefer checking in failing results, that's doable. Although I think rolling out the run-webkit-tests change is OK, I don't think it's the best solution. We shouldn't be afraid of re-ordering tests. We just have to make sure we try and discover the tests which affect other tests and properly skip them as we tried to do here. > As I said before, it's good when flaky tests cause pain and impede progress > (as long as those tests are correct, of course). That's their way of making > us investigate problems which can quickly turn into exploitable security > bugs in shipping software otherwise. I disagree. I see flaky tests as no value to the project whatsoever. The build bots are about maintaining a known good state, not about being a todo list. Skipped lists and bug lists should be our todo lists. A green bot running a few fewer tests is much more valuable than a bot which is occasionally red but runs all the tests. WebKit just needs to make an effort to drive our Skipped list to zero over time and be less scared to use it to keep the bots green. -eric _______________________________________________ webkit-dev mailing list [email protected] http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
