The reason for sampling timeouts is so that you can diagnose what
happened more easily when you get an unexpected timeout. Same reason
we save backtraces from crashes. Having some expected timeouts does
not remove the need to diagnose regressions that manifest as a timeout.
On Apr 10, 2010, at 8:39 PM, Eric Seidel <[email protected]> wrote:
new-run-webkit-tests does not sample tests when they timeout. timeout
is a perfectly "reasonable" test expectation. new-run-webkit-tests is
about running all the tests and making sure their behavior matches
what we have in test_expectations.txt. This is unlike
run-webkit-tests for which the only valid expectation is "PASS".
new-run-webkit-tests makes sense in a world where you want to run all
the tests, but have no prayer of passing them all. I think most ports
are in this boat. Possibly even the Mac port these days.
new-run-webkit-tests "always run every test, no matter its expected
outcome" philosophy enables a bunch of neat fringe benefits, including
documenting and tracking flaky tests w/o having them break your
builders or requiring skipping. :)
-eric
On Sat, Apr 10, 2010 at 7:49 PM, Alexey Proskuryakov <[email protected]>
wrote:
10.04.2010, в 17:44, Eric Seidel написал(а):
Yes, I think we should keep Chromium's default low timeout. Having
such a low timeout allows new-run-webkit-tests to easily run all
flaky
tests every time, even ones which occasionally timeout.
Does new-run-webkit-tests not sample tests that time out? Sampling
is so
slow that is easily outweighs the difference between 3 or 30
seconds that a
test can run for.
- WBR, Alexey Proskuryakov
_______________________________________________
webkit-dev mailing list
[email protected]
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
_______________________________________________
webkit-dev mailing list
[email protected]
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
