Recording to a local file the order in which tests are run on each thread
should be relatively trivial. I filed
https://bugs.webkit.org/show_bug.cgi?id=37396. That will help with flakiness
that is simply order dependent, which I think is most of it. It won't help
with flakiness that has to do with
On Fri, Apr 9, 2010 at 5:03 PM, Yuzo Fujishima y...@google.com wrote:
Sorry, (mostly) false positives.
I've synced to r57383 and all but 3 failures are gone.
I don't see /tmp/layout-test-results. Where are the errors logged?
The following is the console output.
$
Yes, I think we should keep Chromium's default low timeout. Having
such a low timeout allows new-run-webkit-tests to easily run all flaky
tests every time, even ones which occasionally timeout.
We just need to mark slow tests as SLOW in
LayoutTests/platform/mac/test_expectations.txt
On Fri, Apr
On Sat, Apr 10, 2010 at 5:44 PM, Eric Seidel e...@webkit.org wrote:
We just need to mark slow tests as SLOW in
LayoutTests/platform/mac/test_expectations.txt
Done.
Adam
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
10.04.2010, в 17:44, Eric Seidel написал(а):
Yes, I think we should keep Chromium's default low timeout. Having
such a low timeout allows new-run-webkit-tests to easily run all flaky
tests every time, even ones which occasionally timeout.
Does new-run-webkit-tests not sample tests that
new-run-webkit-tests does not sample tests when they timeout. timeout
is a perfectly reasonable test expectation. new-run-webkit-tests is
about running all the tests and making sure their behavior matches
what we have in test_expectations.txt. This is unlike
run-webkit-tests for which the only
The reason for sampling timeouts is so that you can diagnose what
happened more easily when you get an unexpected timeout. Same reason
we save backtraces from crashes. Having some expected timeouts does
not remove the need to diagnose regressions that manifest as a timeout.
On Apr 10,
https://bugs.webkit.org/show_bug.cgi?id=37402
On Sat, Apr 10, 2010 at 8:55 PM, Maciej Stachowiak m...@apple.com wrote:
The reason for sampling timeouts is so that you can diagnose what happened
more easily when you get an unexpected timeout. Same reason we save
backtraces from crashes. Having
I think it's fine to add some support for sampling unexpected
timeouts. However if the timeout is expected in test_expectations, we
shouldn't bother to sample. :)
Then again, since new-run-webkit-tests is all parallel and stuff we
need not block for the sample like how the current
Hi, Eric,
In my development environment, new-run-webkit-tests reports 149 errors
(out of 12692 tests) while run-webkit-tests none. Is this expected?
Are all of the errors what you mean by Exposes more flaky tests due to
running tests in a non-deterministic order ?
Yuzo
On Sat, Apr 10, 2010 at
Sounds like a bug. You should expect 0 failures on leopard or snow leopard.
Please file a bug with a log of the failures and I will fix.
On Apr 9, 2010 4:11 PM, Yuzo Fujishima y...@google.com wrote:
Hi, Eric,
In my development environment, new-run-webkit-tests reports 149 errors
(out of 12692
On Apr 9, 2010, at 3:35 PM, Eric Seidel wrote:
Thanks to the tireless efforts of Dirk Pranke, Ojan Vafai, and
numerous other engineers, WebKit now has a new test harness for
running the Layout Tests.
...
Cons:
- Exposes more flaky tests due to running tests in a non-deterministic order.
On 09.04.2010, at 16:32, Brady Eidson wrote:
- Exposes more flaky tests due to running tests in a non-
deterministic order.
This sounds like a pro to me.
Existing run-webkit-tests can also do that, via a non-default option.
Unreproducible results with default options do sound like a
On Apr 9, 2010, at 4:41 PM, Alexey Proskuryakov wrote:
On 09.04.2010, at 16:32, Brady Eidson wrote:
- Exposes more flaky tests due to running tests in a non-deterministic
order.
This sounds like a pro to me.
Existing run-webkit-tests can also do that, via a non-default option.
Sorry, (mostly) false positives.
I've synced to r57383 and all but 3 failures are gone.
I don't see /tmp/layout-test-results. Where are the errors logged?
The following is the console output.
$ ./WebKitTools/Scripts/new-run-webkit-tests
stopping DumpRenderTree timed out, killing it
killed
On Fri, Apr 9, 2010 at 4:43 PM, Brady Eidson beid...@apple.com wrote:
We should make the new tool record the order so order-specific failures can
be explored.
That sounds like a good idea. If the tool doesn't do this yet, we can
certainly add that.
Fixing some of these have been easy so far.
The new tool had a much tighter timeout than the old tool. We'll
probably have to increase that. You can also mark a test as SLOW in
test_expectations.txt, and then it will be given a longer timeout. It
might be worth having a shorter timeout for developers so we know when
we write slow tests.
On Fri, Apr 9, 2010 at 4:43 PM, Brady Eidson beid...@apple.com wrote:
On Apr 9, 2010, at 4:41 PM, Alexey Proskuryakov wrote:
On 09.04.2010, at 16:32, Brady Eidson wrote:
- Exposes more flaky tests due to running tests in a non-deterministic
order.
This sounds like a pro to me.
On Fri, Apr 9, 2010 at 5:03 PM, Yuzo Fujishima y...@google.com wrote:
I don't see /tmp/layout-test-results. Where are the errors logged?
/tmp/run-chromium-webkit-tests-layout-test-results/*
This was done intentionally at the time to not collide with
run-webkit-tests, but it should be changed
We should make the new tool record the order so order-specific failures
can be explored.
A custom random generator would be enough. The seed can be given by a user
or a just starts with a random value which is reported when the test
finish. Much easier to share seed-revision pairs than
20 matches
Mail list logo
