I think it's fine to add some support for sampling "unexpected" timeouts. However if the timeout is expected in test_expectations, we shouldn't bother to sample. :)
Then again, since new-run-webkit-tests is all parallel and stuff we need not block for the sample like how the current run-webkit-tests does. -eric On Sat, Apr 10, 2010 at 8:57 PM, Adam Barth <[email protected]> wrote: > https://bugs.webkit.org/show_bug.cgi?id=37402 > > On Sat, Apr 10, 2010 at 8:55 PM, Maciej Stachowiak <[email protected]> wrote: >> The reason for sampling timeouts is so that you can diagnose what happened >> more easily when you get an unexpected timeout. Same reason we save >> backtraces from crashes. Having some expected timeouts does not remove the >> need to diagnose regressions that manifest as a timeout. >> >> >> >> On Apr 10, 2010, at 8:39 PM, Eric Seidel <[email protected]> wrote: >> >>> new-run-webkit-tests does not sample tests when they timeout. timeout >>> is a perfectly "reasonable" test expectation. new-run-webkit-tests is >>> about running all the tests and making sure their behavior matches >>> what we have in test_expectations.txt. This is unlike >>> run-webkit-tests for which the only valid expectation is "PASS". >>> >>> new-run-webkit-tests makes sense in a world where you want to run all >>> the tests, but have no prayer of passing them all. I think most ports >>> are in this boat. Possibly even the Mac port these days. >>> >>> new-run-webkit-tests "always run every test, no matter its expected >>> outcome" philosophy enables a bunch of neat fringe benefits, including >>> documenting and tracking flaky tests w/o having them break your >>> builders or requiring skipping. :) >>> >>> -eric >>> >>> On Sat, Apr 10, 2010 at 7:49 PM, Alexey Proskuryakov <[email protected]> >>> wrote: >>>> >>>> 10.04.2010, в 17:44, Eric Seidel написал(а): >>>> >>>>> Yes, I think we should keep Chromium's default low timeout. Having >>>>> such a low timeout allows new-run-webkit-tests to easily run all flaky >>>>> tests every time, even ones which occasionally timeout. >>>> >>>> >>>> Does new-run-webkit-tests not sample tests that time out? Sampling is so >>>> slow that is easily outweighs the difference between 3 or 30 seconds that >>>> a >>>> test can run for. >>>> >>>> - WBR, Alexey Proskuryakov >>>> >>>> >>> _______________________________________________ >>> webkit-dev mailing list >>> [email protected] >>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev >> >> _______________________________________________ >> webkit-dev mailing list >> [email protected] >> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev >> > _______________________________________________ webkit-dev mailing list [email protected] http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
