Re: [webkit-dev] Flaky test hit list

2010-10-19 Thread Alexey Proskuryakov
15.10.2010, в 07:39, Eric Seidel написал(а): BTW, the commit-queue has started complaining publicly about flaky tests: https://bugs.webkit.org/show_bug.cgi?id=47698#c5 Hopefully this will bring further awareness to the issue. I find this extremely annoying and offensive. Half of my

Re: [webkit-dev] Flaky test hit list

2010-10-15 Thread Eric Seidel
BTW, the commit-queue has started complaining publicly about flaky tests: https://bugs.webkit.org/show_bug.cgi?id=47698#c5 Hopefully this will bring further awareness to the issue. Test flakes are the second most common reason for delays with the commit-queue (after the Snow Leopard build being

Re: [webkit-dev] Flaky test hit list

2010-10-01 Thread 鵜飼文敏
On Fri, Oct 1, 2010 at 12:03, Adam Barth aba...@webkit.org wrote: https://bugs.webkit.org/show_bug.cgi?id=46956 Unfortunately, it doesn't solve the whole problem. It does seem to reduce the flakiness by a lot though. Thanks! I think it might be race between DRT and pywebsocekt server,

Re: [webkit-dev] Flaky test hit list

2010-10-01 Thread Adam Barth
2010/10/1 Fumitoshi Ukai (鵜飼文敏) u...@chromium.org: On Fri, Oct 1, 2010 at 12:03, Adam Barth aba...@webkit.org wrote: https://bugs.webkit.org/show_bug.cgi?id=46956 Unfortunately, it doesn't solve the whole problem. It does seem to reduce the flakiness by a lot though. Thanks! I think it

[webkit-dev] Flaky test hit list

2010-09-30 Thread Adam Barth
Tonight I wrote a new webkit-patch command for detecting flaky tests. Here the tests that have flaked out on the Snow Leopard (Tests) build bot during the last 2000 revisions. This makes a good hit list of tests to fix to reduce flakiness. (The worst offenders are at the bottom). Take aways:

Re: [webkit-dev] Flaky test hit list

2010-09-30 Thread Adam Barth
On Wed, Sep 29, 2010 at 11:31 PM, Adam Barth aba...@webkit.org wrote:        50 compositing/geometry/limit-layer-bounds-transformed-overflow.html Here's the failure diff:

Re: [webkit-dev] Flaky test hit list

2010-09-30 Thread 鵜飼文敏
On Thu, Sep 30, 2010 at 15:31, Adam Barth aba...@webkit.org wrote: Tonight I wrote a new webkit-patch command for detecting flaky tests. Here the tests that have flaked out on the Snow Leopard (Tests) build bot during the last 2000 revisions. This makes a good hit list of tests to fix to

Re: [webkit-dev] Flaky test hit list

2010-09-30 Thread Simon Fraser
On Sep 30, 2010, at 12:26 AM, Adam Barth wrote: On Wed, Sep 29, 2010 at 11:31 PM, Adam Barth aba...@webkit.org wrote: 50 compositing/geometry/limit-layer-bounds-transformed-overflow.html Here's the failure diff:

Re: [webkit-dev] Flaky test hit list

2010-09-30 Thread Alexey Proskuryakov
29.09.2010, в 23:31, Adam Barth написал(а): 3) The appcache tests also have a serious flakiness problem. Yes, there's a frequent crash that I'm investigating. - WBR, Alexey Proskuryakov ___ webkit-dev mailing list webkit-dev@lists.webkit.org

Re: [webkit-dev] Flaky test hit list

2010-09-30 Thread Adam Barth
2010/9/30 Fumitoshi Ukai (鵜飼文敏) u...@chromium.org: AFAIK the websocket tests are not so flaky on chromium (*). I wonder this is because of difference in platform code, but seems not so flaky on Leopard too. So, I suspect issues in test fixtures of websocket test (e.g. failed to start up

Re: [webkit-dev] Flaky test hit list

2010-09-30 Thread Adam Barth
On Wed, Sep 29, 2010 at 11:31 PM, Adam Barth aba...@webkit.org wrote: I'll run the last 1 revisions over night and report on the results. I have the data for this if anyone's interested, but it's not as helpful as I thought it would be. The data is similar to the data from the past 2000

Re: [webkit-dev] Flaky test hit list

2010-09-30 Thread Eric Seidel
On Thu, Sep 30, 2010 at 11:06 AM, Eric Seidel esei...@google.com wrote: I've filed bugs for the top two:        19 media/audio-controls-rendering.html https://bugs.webkit.org/show_bug.cgi?id=46924        50 compositing/geometry/limit-layer-bounds-transformed-overflow.html

Re: [webkit-dev] Flaky test hit list

2010-09-30 Thread Adam Barth
I'm investigating the websocket issue. It seems these tests are flaky because they time out. If you know about websockets, I'd appreciate any tips you have via #webkit. Adam On Wed, Sep 29, 2010 at 11:31 PM, Adam Barth aba...@webkit.org wrote: Tonight I wrote a new webkit-patch command for

Re: [webkit-dev] Flaky test hit list

2010-09-30 Thread Eric Seidel
I see: [2010-09-30 16:52:57,560] [CRITICAL] root: [Errno 48] Address already in use in my pywebsocket.ws.log-30Sep2010-165257-err.txt after an error. Not sure if it's related, but sounds possible. :) On Thu, Sep 30, 2010 at 3:10 PM, Adam Barth aba...@webkit.org wrote: I'm investigating the

Re: [webkit-dev] Flaky test hit list

2010-09-30 Thread Adam Barth
In case you're interested in Leopard Debug, here's the hit list from the last 2000 revisions. Interesting notes: 1) Leopard seems way less flaky than Snow Leopard. 2) The websocket tests aren't flaky here. 3) http/tests/appcache/idempotent-update.html fails with an ASSERT about a webView being

Re: [webkit-dev] Flaky test hit list

2010-09-30 Thread Evan Martin
On Wed, Sep 29, 2010 at 11:31 PM, Adam Barth aba...@webkit.org wrote: Tonight I wrote a new webkit-patch command for detecting flaky tests. Are you familiar with the Chromium flakiness dashboard? It provides a continuous summary of test results aggregated across multiple builders, including

Re: [webkit-dev] Flaky test hit list

2010-09-30 Thread Adam Barth
Currently the WebSocket tests are served over HTTP from the WebSocket server itself (which is written in Python). It looks like we can resolve the flakiness by serving the WebSocket tests from the normal Apache server that servers the rest of our HTTP tests. I'm going to work up a patch that

Re: [webkit-dev] Flaky test hit list

2010-09-30 Thread Adam Barth
https://bugs.webkit.org/show_bug.cgi?id=46956 Unfortunately, it doesn't solve the whole problem. It does seem to reduce the flakiness by a lot though. Adam On Thu, Sep 30, 2010 at 6:12 PM, Adam Barth aba...@webkit.org wrote: Currently the WebSocket tests are served over HTTP from the