Re: [webkit-dev] Flaky test hit list
15.10.2010, в 07:39, Eric Seidel написал(а): BTW, the commit-queue has started complaining publicly about flaky tests: https://bugs.webkit.org/show_bug.cgi?id=47698#c5 Hopefully this will bring further awareness to the issue. I find this extremely annoying and offensive. Half of my bugmail is already about bugs that I'm not interested in. - WBR, Alexey Proskuryakov ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Flaky test hit list
BTW, the commit-queue has started complaining publicly about flaky tests: https://bugs.webkit.org/show_bug.cgi?id=47698#c5 Hopefully this will bring further awareness to the issue. Test flakes are the second most common reason for delays with the commit-queue (after the Snow Leopard build being broken). -eric p.s. Eventually we may improve the commit-queue to know how to file bugs against flaky tests itself. On Fri, Oct 1, 2010 at 7:29 AM, Adam Barth aba...@webkit.org wrote: 2010/10/1 Fumitoshi Ukai (鵜飼文敏) u...@chromium.org: On Fri, Oct 1, 2010 at 12:03, Adam Barth aba...@webkit.org wrote: https://bugs.webkit.org/show_bug.cgi?id=46956 Unfortunately, it doesn't solve the whole problem. It does seem to reduce the flakiness by a lot though. Thanks! I think it might be race between DRT and pywebsocekt server, that is, DRT tries to run tests while pywebsocket server is still initializing and not yet ready to serve html. I'll try to fix pywebsocket to reduce initialization latency. That doesn't seem that likely. With my patch, this happens after running something like 2000 WebSockets. It doesn't feel like an initialization race to me. It feels more like the the WebSocket server is dropping a connection once in a while. Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Flaky test hit list
On Fri, Oct 1, 2010 at 12:03, Adam Barth aba...@webkit.org wrote: https://bugs.webkit.org/show_bug.cgi?id=46956 Unfortunately, it doesn't solve the whole problem. It does seem to reduce the flakiness by a lot though. Thanks! I think it might be race between DRT and pywebsocekt server, that is, DRT tries to run tests while pywebsocket server is still initializing and not yet ready to serve html. I'll try to fix pywebsocket to reduce initialization latency. -- ukai Adam On Thu, Sep 30, 2010 at 6:12 PM, Adam Barth aba...@webkit.org wrote: Currently the WebSocket tests are served over HTTP from the WebSocket server itself (which is written in Python). It looks like we can resolve the flakiness by serving the WebSocket tests from the normal Apache server that servers the rest of our HTTP tests. I'm going to work up a patch that does that. Please let me know if there's any reason we shouldn't make that change. Thanks, Adam On Thu, Sep 30, 2010 at 3:10 PM, Adam Barth aba...@webkit.org wrote: I'm investigating the websocket issue. It seems these tests are flaky because they time out. If you know about websockets, I'd appreciate any tips you have via #webkit. Adam On Wed, Sep 29, 2010 at 11:31 PM, Adam Barth aba...@webkit.org wrote: Tonight I wrote a new webkit-patch command for detecting flaky tests. Here the tests that have flaked out on the Snow Leopard (Tests) build bot during the last 2000 revisions. This makes a good hit list of tests to fix to reduce flakiness. (The worst offenders are at the bottom). Take aways: 1) compositing/geometry/limit-layer-bounds-transformed-overflow.html is by far the worst offender. 2) The websocket tests are ridiculously flaky. 3) The appcache tests also have a serious flakiness problem. I'll run the last 1 revisions over night and report on the results. Adam === Results === Occurances Test name 1 compositing/reflections/nested-reflection-animated.html 1 fast/css/font-face-download-error.html 1 fast/dom/collection-null-like-arguments.html 1 fast/history/history-subframe-with-name.html 1 fast/js/sputnik/Conformance/13_Function_Definition/13.2_Creating_Function_Objects/S13.2.2_A5_T1.html 1 fast/js/sputnik/Conformance/13_Function_Definition/13.2_Creating_Function_Objects/S13.2.2_A5_T2.html 1 fast/js/vardecl-preserve-arguments.html 1 http/tests/appcache/different-https-origin-resource-main.html 1 http/tests/appcache/fallback.html 1 http/tests/appcache/manifest-redirect.html 1 http/tests/appcache/origin-quota.html 1 http/tests/appcache/resource-redirect.html 1 http/tests/appcache/top-frame-3.html 1 http/tests/appcache/update-cache.html 1 http/tests/appcache/xhr-foreign-resource.html 1 http/tests/cache/subresource-expiration.html 1 http/tests/loading/basic-credentials-sent-automatically.html 1 http/tests/misc/uncacheable-script-repeated.html 1 http/tests/navigation/changing-frame-hierarchy-in-onload.html 1 http/tests/navigation/ping-cross-origin-from-https.html 1 http/tests/navigation/ping-cross-origin.html 1 http/tests/navigation/post-goback-same-url.html 1 http/tests/plugins/get-url.html 1 http/tests/plugins/npapi-response-headers.html 1 http/tests/plugins/third-party-cookie-accept-policy.html 1 http/tests/security/credentials-in-referer.html 1 http/tests/security/cross-frame-access-protocol-explicit-domain.html 1 inspector/debugger-pause-on-breakpoint.html 1 inspector/extensions-events.html 1 media/audio-constructor.html 1 media/video-currentTime-set.html 1 plugins/destroy-stream-twice.html 1 plugins/return-error-from-new-stream-doesnt-invoke-destroy-stream.html 1 svg/custom/use-invalid-style.svg 1 transitions/transition-end-event-transform.html 1 websocket/tests/send.html 1 websocket/tests/simple-stress.html 1 websocket/tests/sub-protocol-with-space.html 1 websocket/tests/url-no-trailing-slash.html 1 websocket/tests/url-with-empty-query.html 1 websocket/tests/url-with-query.html 1 websocket/tests/websocket-pending-activity.html 1 websocket/tests/workers/close-in-onmessage-crash.html 2 http/tests/appcache/foreign-iframe-main.html 2 http/tests/appcache/local-content.html 2 http/tests/appcache/main-resource-hash.html 2 http/tests/appcache/non-html.xhtml 2 http/tests/appcache/reload.html 2 http/tests/css/css-image-loading.html 2 http/tests/plugins/cross-frame-object-access.html 2 http/tests/security/cross-frame-access-port-explicit-domain.html 2
Re: [webkit-dev] Flaky test hit list
2010/10/1 Fumitoshi Ukai (鵜飼文敏) u...@chromium.org: On Fri, Oct 1, 2010 at 12:03, Adam Barth aba...@webkit.org wrote: https://bugs.webkit.org/show_bug.cgi?id=46956 Unfortunately, it doesn't solve the whole problem. It does seem to reduce the flakiness by a lot though. Thanks! I think it might be race between DRT and pywebsocekt server, that is, DRT tries to run tests while pywebsocket server is still initializing and not yet ready to serve html. I'll try to fix pywebsocket to reduce initialization latency. That doesn't seem that likely. With my patch, this happens after running something like 2000 WebSockets. It doesn't feel like an initialization race to me. It feels more like the the WebSocket server is dropping a connection once in a while. Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] Flaky test hit list
Tonight I wrote a new webkit-patch command for detecting flaky tests. Here the tests that have flaked out on the Snow Leopard (Tests) build bot during the last 2000 revisions. This makes a good hit list of tests to fix to reduce flakiness. (The worst offenders are at the bottom). Take aways: 1) compositing/geometry/limit-layer-bounds-transformed-overflow.html is by far the worst offender. 2) The websocket tests are ridiculously flaky. 3) The appcache tests also have a serious flakiness problem. I'll run the last 1 revisions over night and report on the results. Adam === Results === Occurances Test name 1 compositing/reflections/nested-reflection-animated.html 1 fast/css/font-face-download-error.html 1 fast/dom/collection-null-like-arguments.html 1 fast/history/history-subframe-with-name.html 1 fast/js/sputnik/Conformance/13_Function_Definition/13.2_Creating_Function_Objects/S13.2.2_A5_T1.html 1 fast/js/sputnik/Conformance/13_Function_Definition/13.2_Creating_Function_Objects/S13.2.2_A5_T2.html 1 fast/js/vardecl-preserve-arguments.html 1 http/tests/appcache/different-https-origin-resource-main.html 1 http/tests/appcache/fallback.html 1 http/tests/appcache/manifest-redirect.html 1 http/tests/appcache/origin-quota.html 1 http/tests/appcache/resource-redirect.html 1 http/tests/appcache/top-frame-3.html 1 http/tests/appcache/update-cache.html 1 http/tests/appcache/xhr-foreign-resource.html 1 http/tests/cache/subresource-expiration.html 1 http/tests/loading/basic-credentials-sent-automatically.html 1 http/tests/misc/uncacheable-script-repeated.html 1 http/tests/navigation/changing-frame-hierarchy-in-onload.html 1 http/tests/navigation/ping-cross-origin-from-https.html 1 http/tests/navigation/ping-cross-origin.html 1 http/tests/navigation/post-goback-same-url.html 1 http/tests/plugins/get-url.html 1 http/tests/plugins/npapi-response-headers.html 1 http/tests/plugins/third-party-cookie-accept-policy.html 1 http/tests/security/credentials-in-referer.html 1 http/tests/security/cross-frame-access-protocol-explicit-domain.html 1 inspector/debugger-pause-on-breakpoint.html 1 inspector/extensions-events.html 1 media/audio-constructor.html 1 media/video-currentTime-set.html 1 plugins/destroy-stream-twice.html 1 plugins/return-error-from-new-stream-doesnt-invoke-destroy-stream.html 1 svg/custom/use-invalid-style.svg 1 transitions/transition-end-event-transform.html 1 websocket/tests/send.html 1 websocket/tests/simple-stress.html 1 websocket/tests/sub-protocol-with-space.html 1 websocket/tests/url-no-trailing-slash.html 1 websocket/tests/url-with-empty-query.html 1 websocket/tests/url-with-query.html 1 websocket/tests/websocket-pending-activity.html 1 websocket/tests/workers/close-in-onmessage-crash.html 2 http/tests/appcache/foreign-iframe-main.html 2 http/tests/appcache/local-content.html 2 http/tests/appcache/main-resource-hash.html 2 http/tests/appcache/non-html.xhtml 2 http/tests/appcache/reload.html 2 http/tests/css/css-image-loading.html 2 http/tests/plugins/cross-frame-object-access.html 2 http/tests/security/cross-frame-access-port-explicit-domain.html 2 security/block-test.html 2 websocket/tests/bad-sub-protocol-non-ascii.html 2 websocket/tests/handshake-fail-by-sub-protocol-mismatch.html 2 websocket/tests/simple.html 2 websocket/tests/unicode.html 3 http/tests/appcache/cyrillic-uri.html 3 http/tests/appcache/deferred-events-delete-while-raising.html 3 http/tests/appcache/remove-cache.html 3 http/tests/appcache/top-frame-4.html 3 http/tests/navigation/image-load-in-unload-handler.html 3 websocket/tests/handshake-fail-by-cross-origin.html 3 websocket/tests/httponly-cookie.pl 3 websocket/tests/long-invalid-header.html 3 websocket/tests/sub-protocol.html 3 websocket/tests/url-with-query-for-no-query.html 4 fast/canvas/webgl/gl-object-get-calls.html 4 http/tests/navigation/anchor-basic.html 4 websocket/tests/bad-sub-protocol-control-chars.html 4 websocket/tests/frame-length-longer-than-buffer.html 4 websocket/tests/frame-length-skip.html 4 websocket/tests/frame-lengths.html 4 websocket/tests/reload-crash.html 4 websocket/tests/url-with-credential.html 5 websocket/tests/bad-sub-protocol-empty.html 5 websocket/tests/error-detect.html 5 websocket/tests/handshake-error.html 6
Re: [webkit-dev] Flaky test hit list
On Wed, Sep 29, 2010 at 11:31 PM, Adam Barth aba...@webkit.org wrote: 50 compositing/geometry/limit-layer-bounds-transformed-overflow.html Here's the failure diff: http://build.webkit.org/results/SnowLeopard%20Intel%20Release%20(Tests)/r68736%20(18480)/compositing/geometry/limit-layer-bounds-transformed-overflow-pretty-diff.html --- /Volumes/Data/WebKit-BuildSlave/snowleopard-intel-release-tests/build/layout-test-results/compositing/geometry/limit-layer-bounds-transformed-overflow-expected.txt 2010-09-29 19:08:55.0 -0700 +++ /Volumes/Data/WebKit-BuildSlave/snowleopard-intel-release-tests/build/layout-test-results/compositing/geometry/limit-layer-bounds-transformed-overflow-actual.txt 2010-09-29 19:08:55.0 -0700 @@ -27,7 +27,6 @@ (GraphicsLayer (position -980.00 20.00) (bounds 2050.00 100.00) - (usingTiledLayer 1) (drawsContent 1) ) (GraphicsLayer I don't really know what that means. Do any of the compositing folks know what's going on? This test is far and away at the top of the flaky test distribution. Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Flaky test hit list
On Thu, Sep 30, 2010 at 15:31, Adam Barth aba...@webkit.org wrote: Tonight I wrote a new webkit-patch command for detecting flaky tests. Here the tests that have flaked out on the Snow Leopard (Tests) build bot during the last 2000 revisions. This makes a good hit list of tests to fix to reduce flakiness. (The worst offenders are at the bottom). Take aways: 1) compositing/geometry/limit-layer-bounds-transformed-overflow.html is by far the worst offender. 2) The websocket tests are ridiculously flaky. AFAIK the websocket tests are not so flaky on chromium (*). I wonder this is because of difference in platform code, but seems not so flaky on Leopard too. So, I suspect issues in test fixtures of websocket test (e.g. failed to start up pywebsocket server)? (*) http://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=websocket -- ukai 3) The appcache tests also have a serious flakiness problem. I'll run the last 1 revisions over night and report on the results. Adam === Results === Occurances Test name 1 compositing/reflections/nested-reflection-animated.html 1 fast/css/font-face-download-error.html 1 fast/dom/collection-null-like-arguments.html 1 fast/history/history-subframe-with-name.html 1 fast/js/sputnik/Conformance/13_Function_Definition/13.2_Creating_Function_Objects/S13.2.2_A5_T1.html 1 fast/js/sputnik/Conformance/13_Function_Definition/13.2_Creating_Function_Objects/S13.2.2_A5_T2.html 1 fast/js/vardecl-preserve-arguments.html 1 http/tests/appcache/different-https-origin-resource-main.html 1 http/tests/appcache/fallback.html 1 http/tests/appcache/manifest-redirect.html 1 http/tests/appcache/origin-quota.html 1 http/tests/appcache/resource-redirect.html 1 http/tests/appcache/top-frame-3.html 1 http/tests/appcache/update-cache.html 1 http/tests/appcache/xhr-foreign-resource.html 1 http/tests/cache/subresource-expiration.html 1 http/tests/loading/basic-credentials-sent-automatically.html 1 http/tests/misc/uncacheable-script-repeated.html 1 http/tests/navigation/changing-frame-hierarchy-in-onload.html 1 http/tests/navigation/ping-cross-origin-from-https.html 1 http/tests/navigation/ping-cross-origin.html 1 http/tests/navigation/post-goback-same-url.html 1 http/tests/plugins/get-url.html 1 http/tests/plugins/npapi-response-headers.html 1 http/tests/plugins/third-party-cookie-accept-policy.html 1 http/tests/security/credentials-in-referer.html 1 http/tests/security/cross-frame-access-protocol-explicit-domain.html 1 inspector/debugger-pause-on-breakpoint.html 1 inspector/extensions-events.html 1 media/audio-constructor.html 1 media/video-currentTime-set.html 1 plugins/destroy-stream-twice.html 1 plugins/return-error-from-new-stream-doesnt-invoke-destroy-stream.html 1 svg/custom/use-invalid-style.svg 1 transitions/transition-end-event-transform.html 1 websocket/tests/send.html 1 websocket/tests/simple-stress.html 1 websocket/tests/sub-protocol-with-space.html 1 websocket/tests/url-no-trailing-slash.html 1 websocket/tests/url-with-empty-query.html 1 websocket/tests/url-with-query.html 1 websocket/tests/websocket-pending-activity.html 1 websocket/tests/workers/close-in-onmessage-crash.html 2 http/tests/appcache/foreign-iframe-main.html 2 http/tests/appcache/local-content.html 2 http/tests/appcache/main-resource-hash.html 2 http/tests/appcache/non-html.xhtml 2 http/tests/appcache/reload.html 2 http/tests/css/css-image-loading.html 2 http/tests/plugins/cross-frame-object-access.html 2 http/tests/security/cross-frame-access-port-explicit-domain.html 2 security/block-test.html 2 websocket/tests/bad-sub-protocol-non-ascii.html 2 websocket/tests/handshake-fail-by-sub-protocol-mismatch.html 2 websocket/tests/simple.html 2 websocket/tests/unicode.html 3 http/tests/appcache/cyrillic-uri.html 3 http/tests/appcache/deferred-events-delete-while-raising.html 3 http/tests/appcache/remove-cache.html 3 http/tests/appcache/top-frame-4.html 3 http/tests/navigation/image-load-in-unload-handler.html 3 websocket/tests/handshake-fail-by-cross-origin.html 3 websocket/tests/httponly-cookie.pl 3 websocket/tests/long-invalid-header.html 3 websocket/tests/sub-protocol.html 3 websocket/tests/url-with-query-for-no-query.html 4 fast/canvas/webgl/gl-object-get-calls.html 4 http/tests/navigation/anchor-basic.html 4 websocket/tests/bad-sub-protocol-control-chars.html 4
Re: [webkit-dev] Flaky test hit list
On Sep 30, 2010, at 12:26 AM, Adam Barth wrote: On Wed, Sep 29, 2010 at 11:31 PM, Adam Barth aba...@webkit.org wrote: 50 compositing/geometry/limit-layer-bounds-transformed-overflow.html Here's the failure diff: http://build.webkit.org/results/SnowLeopard%20Intel%20Release%20(Tests)/r68736%20(18480)/compositing/geometry/limit-layer-bounds-transformed-overflow-pretty-diff.html --- /Volumes/Data/WebKit-BuildSlave/snowleopard-intel-release-tests/build/layout-test-results/compositing/geometry/limit-layer-bounds-transformed-overflow-expected.txt 2010-09-29 19:08:55.0 -0700 +++ /Volumes/Data/WebKit-BuildSlave/snowleopard-intel-release-tests/build/layout-test-results/compositing/geometry/limit-layer-bounds-transformed-overflow-actual.txt 2010-09-29 19:08:55.0 -0700 @@ -27,7 +27,6 @@ (GraphicsLayer (position -980.00 20.00) (bounds 2050.00 100.00) - (usingTiledLayer 1) (drawsContent 1) ) (GraphicsLayer I don't really know what that means. Do any of the compositing folks know what's going on? This test is far and away at the top of the flaky test distribution. I'll take a look. Is there a bugzilla? Simon ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Flaky test hit list
29.09.2010, в 23:31, Adam Barth написал(а): 3) The appcache tests also have a serious flakiness problem. Yes, there's a frequent crash that I'm investigating. - WBR, Alexey Proskuryakov ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Flaky test hit list
2010/9/30 Fumitoshi Ukai (鵜飼文敏) u...@chromium.org: AFAIK the websocket tests are not so flaky on chromium (*). I wonder this is because of difference in platform code, but seems not so flaky on Leopard too. So, I suspect issues in test fixtures of websocket test (e.g. failed to start up pywebsocket server)? (*) http://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=websocket This reminds me of an issue with used to have with the HTTP tests and IPv6. Maybe the WebSocket server isn't handling machines with IPv6 support correctly? On Thu, Sep 30, 2010 at 8:07 AM, Simon Fraser simon.fra...@apple.com wrote: On Sep 30, 2010, at 12:26 AM, Adam Barth wrote: On Wed, Sep 29, 2010 at 11:31 PM, Adam Barth aba...@webkit.org wrote: 50 compositing/geometry/limit-layer-bounds-transformed-overflow.html Here's the failure diff: I'll take a look. Is there a bugzilla? Thanks Simon. I don't think there's a bug on file yet. On Thu, Sep 30, 2010 at 8:43 AM, Alexey Proskuryakov a...@webkit.org wrote: 29.09.2010, в 23:31, Adam Barth написал(а): 3) The appcache tests also have a serious flakiness problem. Yes, there's a frequent crash that I'm investigating. Thanks Alexey. I appreciate it. Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Flaky test hit list
On Wed, Sep 29, 2010 at 11:31 PM, Adam Barth aba...@webkit.org wrote: I'll run the last 1 revisions over night and report on the results. I have the data for this if anyone's interested, but it's not as helpful as I thought it would be. The data is similar to the data from the past 2000 revisions, but with the addition of tests that used to be flaky but no longer seem to be a problem. Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Flaky test hit list
On Thu, Sep 30, 2010 at 11:06 AM, Eric Seidel esei...@google.com wrote: I've filed bugs for the top two: 19 media/audio-controls-rendering.html https://bugs.webkit.org/show_bug.cgi?id=46924 50 compositing/geometry/limit-layer-bounds-transformed-overflow.html https://bugs.webkit.org/show_bug.cgi?id=46912 The websocket tests already have a bug: https://bugs.webkit.org/show_bug.cgi?id=33307 As well as lots of other individual websocket test bugs: https://bugs.webkit.org/show_bug.cgi?id=33444 https://bugs.webkit.org/show_bug.cgi?id=37124 https://bugs.webkit.org/show_bug.cgi?id=37147 https://bugs.webkit.org/show_bug.cgi?id=45918 ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Flaky test hit list
I'm investigating the websocket issue. It seems these tests are flaky because they time out. If you know about websockets, I'd appreciate any tips you have via #webkit. Adam On Wed, Sep 29, 2010 at 11:31 PM, Adam Barth aba...@webkit.org wrote: Tonight I wrote a new webkit-patch command for detecting flaky tests. Here the tests that have flaked out on the Snow Leopard (Tests) build bot during the last 2000 revisions. This makes a good hit list of tests to fix to reduce flakiness. (The worst offenders are at the bottom). Take aways: 1) compositing/geometry/limit-layer-bounds-transformed-overflow.html is by far the worst offender. 2) The websocket tests are ridiculously flaky. 3) The appcache tests also have a serious flakiness problem. I'll run the last 1 revisions over night and report on the results. Adam === Results === Occurances Test name 1 compositing/reflections/nested-reflection-animated.html 1 fast/css/font-face-download-error.html 1 fast/dom/collection-null-like-arguments.html 1 fast/history/history-subframe-with-name.html 1 fast/js/sputnik/Conformance/13_Function_Definition/13.2_Creating_Function_Objects/S13.2.2_A5_T1.html 1 fast/js/sputnik/Conformance/13_Function_Definition/13.2_Creating_Function_Objects/S13.2.2_A5_T2.html 1 fast/js/vardecl-preserve-arguments.html 1 http/tests/appcache/different-https-origin-resource-main.html 1 http/tests/appcache/fallback.html 1 http/tests/appcache/manifest-redirect.html 1 http/tests/appcache/origin-quota.html 1 http/tests/appcache/resource-redirect.html 1 http/tests/appcache/top-frame-3.html 1 http/tests/appcache/update-cache.html 1 http/tests/appcache/xhr-foreign-resource.html 1 http/tests/cache/subresource-expiration.html 1 http/tests/loading/basic-credentials-sent-automatically.html 1 http/tests/misc/uncacheable-script-repeated.html 1 http/tests/navigation/changing-frame-hierarchy-in-onload.html 1 http/tests/navigation/ping-cross-origin-from-https.html 1 http/tests/navigation/ping-cross-origin.html 1 http/tests/navigation/post-goback-same-url.html 1 http/tests/plugins/get-url.html 1 http/tests/plugins/npapi-response-headers.html 1 http/tests/plugins/third-party-cookie-accept-policy.html 1 http/tests/security/credentials-in-referer.html 1 http/tests/security/cross-frame-access-protocol-explicit-domain.html 1 inspector/debugger-pause-on-breakpoint.html 1 inspector/extensions-events.html 1 media/audio-constructor.html 1 media/video-currentTime-set.html 1 plugins/destroy-stream-twice.html 1 plugins/return-error-from-new-stream-doesnt-invoke-destroy-stream.html 1 svg/custom/use-invalid-style.svg 1 transitions/transition-end-event-transform.html 1 websocket/tests/send.html 1 websocket/tests/simple-stress.html 1 websocket/tests/sub-protocol-with-space.html 1 websocket/tests/url-no-trailing-slash.html 1 websocket/tests/url-with-empty-query.html 1 websocket/tests/url-with-query.html 1 websocket/tests/websocket-pending-activity.html 1 websocket/tests/workers/close-in-onmessage-crash.html 2 http/tests/appcache/foreign-iframe-main.html 2 http/tests/appcache/local-content.html 2 http/tests/appcache/main-resource-hash.html 2 http/tests/appcache/non-html.xhtml 2 http/tests/appcache/reload.html 2 http/tests/css/css-image-loading.html 2 http/tests/plugins/cross-frame-object-access.html 2 http/tests/security/cross-frame-access-port-explicit-domain.html 2 security/block-test.html 2 websocket/tests/bad-sub-protocol-non-ascii.html 2 websocket/tests/handshake-fail-by-sub-protocol-mismatch.html 2 websocket/tests/simple.html 2 websocket/tests/unicode.html 3 http/tests/appcache/cyrillic-uri.html 3 http/tests/appcache/deferred-events-delete-while-raising.html 3 http/tests/appcache/remove-cache.html 3 http/tests/appcache/top-frame-4.html 3 http/tests/navigation/image-load-in-unload-handler.html 3 websocket/tests/handshake-fail-by-cross-origin.html 3 websocket/tests/httponly-cookie.pl 3 websocket/tests/long-invalid-header.html 3 websocket/tests/sub-protocol.html 3 websocket/tests/url-with-query-for-no-query.html 4 fast/canvas/webgl/gl-object-get-calls.html 4 http/tests/navigation/anchor-basic.html 4 websocket/tests/bad-sub-protocol-control-chars.html 4 websocket/tests/frame-length-longer-than-buffer.html 4 websocket/tests/frame-length-skip.html 4 websocket/tests/frame-lengths.html 4
Re: [webkit-dev] Flaky test hit list
I see: [2010-09-30 16:52:57,560] [CRITICAL] root: [Errno 48] Address already in use in my pywebsocket.ws.log-30Sep2010-165257-err.txt after an error. Not sure if it's related, but sounds possible. :) On Thu, Sep 30, 2010 at 3:10 PM, Adam Barth aba...@webkit.org wrote: I'm investigating the websocket issue. It seems these tests are flaky because they time out. If you know about websockets, I'd appreciate any tips you have via #webkit. Adam On Wed, Sep 29, 2010 at 11:31 PM, Adam Barth aba...@webkit.org wrote: Tonight I wrote a new webkit-patch command for detecting flaky tests. Here the tests that have flaked out on the Snow Leopard (Tests) build bot during the last 2000 revisions. This makes a good hit list of tests to fix to reduce flakiness. (The worst offenders are at the bottom). Take aways: 1) compositing/geometry/limit-layer-bounds-transformed-overflow.html is by far the worst offender. 2) The websocket tests are ridiculously flaky. 3) The appcache tests also have a serious flakiness problem. I'll run the last 1 revisions over night and report on the results. Adam === Results === Occurances Test name 1 compositing/reflections/nested-reflection-animated.html 1 fast/css/font-face-download-error.html 1 fast/dom/collection-null-like-arguments.html 1 fast/history/history-subframe-with-name.html 1 fast/js/sputnik/Conformance/13_Function_Definition/13.2_Creating_Function_Objects/S13.2.2_A5_T1.html 1 fast/js/sputnik/Conformance/13_Function_Definition/13.2_Creating_Function_Objects/S13.2.2_A5_T2.html 1 fast/js/vardecl-preserve-arguments.html 1 http/tests/appcache/different-https-origin-resource-main.html 1 http/tests/appcache/fallback.html 1 http/tests/appcache/manifest-redirect.html 1 http/tests/appcache/origin-quota.html 1 http/tests/appcache/resource-redirect.html 1 http/tests/appcache/top-frame-3.html 1 http/tests/appcache/update-cache.html 1 http/tests/appcache/xhr-foreign-resource.html 1 http/tests/cache/subresource-expiration.html 1 http/tests/loading/basic-credentials-sent-automatically.html 1 http/tests/misc/uncacheable-script-repeated.html 1 http/tests/navigation/changing-frame-hierarchy-in-onload.html 1 http/tests/navigation/ping-cross-origin-from-https.html 1 http/tests/navigation/ping-cross-origin.html 1 http/tests/navigation/post-goback-same-url.html 1 http/tests/plugins/get-url.html 1 http/tests/plugins/npapi-response-headers.html 1 http/tests/plugins/third-party-cookie-accept-policy.html 1 http/tests/security/credentials-in-referer.html 1 http/tests/security/cross-frame-access-protocol-explicit-domain.html 1 inspector/debugger-pause-on-breakpoint.html 1 inspector/extensions-events.html 1 media/audio-constructor.html 1 media/video-currentTime-set.html 1 plugins/destroy-stream-twice.html 1 plugins/return-error-from-new-stream-doesnt-invoke-destroy-stream.html 1 svg/custom/use-invalid-style.svg 1 transitions/transition-end-event-transform.html 1 websocket/tests/send.html 1 websocket/tests/simple-stress.html 1 websocket/tests/sub-protocol-with-space.html 1 websocket/tests/url-no-trailing-slash.html 1 websocket/tests/url-with-empty-query.html 1 websocket/tests/url-with-query.html 1 websocket/tests/websocket-pending-activity.html 1 websocket/tests/workers/close-in-onmessage-crash.html 2 http/tests/appcache/foreign-iframe-main.html 2 http/tests/appcache/local-content.html 2 http/tests/appcache/main-resource-hash.html 2 http/tests/appcache/non-html.xhtml 2 http/tests/appcache/reload.html 2 http/tests/css/css-image-loading.html 2 http/tests/plugins/cross-frame-object-access.html 2 http/tests/security/cross-frame-access-port-explicit-domain.html 2 security/block-test.html 2 websocket/tests/bad-sub-protocol-non-ascii.html 2 websocket/tests/handshake-fail-by-sub-protocol-mismatch.html 2 websocket/tests/simple.html 2 websocket/tests/unicode.html 3 http/tests/appcache/cyrillic-uri.html 3 http/tests/appcache/deferred-events-delete-while-raising.html 3 http/tests/appcache/remove-cache.html 3 http/tests/appcache/top-frame-4.html 3 http/tests/navigation/image-load-in-unload-handler.html 3 websocket/tests/handshake-fail-by-cross-origin.html 3 websocket/tests/httponly-cookie.pl 3 websocket/tests/long-invalid-header.html 3 websocket/tests/sub-protocol.html 3 websocket/tests/url-with-query-for-no-query.html 4 fast/canvas/webgl/gl-object-get-calls.html 4
Re: [webkit-dev] Flaky test hit list
In case you're interested in Leopard Debug, here's the hit list from the last 2000 revisions. Interesting notes: 1) Leopard seems way less flaky than Snow Leopard. 2) The websocket tests aren't flaky here. 3) http/tests/appcache/idempotent-update.html fails with an ASSERT about a webView being null. === Results === Occurances Test name 1 compositing/geometry/composited-html-size.html 1 compositing/iframes/composited-parent-iframe.html 1 fast/dom/collection-null-like-arguments.html 1 fast/events/tabindex-focus-blur-all.html 1 fast/js/vardecl-preserve-arguments.html 1 inspector/debugger-pause-on-breakpoint.html 1 inspector/extensions-events.html 1 media/video-load-networkState.html 1 media/video-pause-immediately.html 1 media/video-size-intrinsic-scale.html 1 media/video-zoom.html 1 platform/mac/accessibility/area-with-aria-label.html 1 platform/mac/fast/events/objc-event-api.html 1 svg/dynamic-updates/SVGMaskElement-svgdom-maskUnits-prop.html 2 animations/dynamic-stylesheet-loading.html 2 http/tests/cache/subresource-expiration.html 2 media/video-played-reset.html 2 media/video-preload.html 3 fast/events/wheelevent-in-scrolling-div.html 3 svg/dom/SVGScriptElement/script-set-href.svg 6 fast/files/workers/worker-read-blob-sync.html 7 svg/W3C-SVG-1.1/fonts-glyph-02-t.svg 8 media/video-display-aspect-ratio.html 10 fast/files/workers/worker-read-blob-async.html 22 media/video-src-change.html 76 http/tests/appcache/idempotent-update.html On Wed, Sep 29, 2010 at 11:31 PM, Adam Barth aba...@webkit.org wrote: Tonight I wrote a new webkit-patch command for detecting flaky tests. Here the tests that have flaked out on the Snow Leopard (Tests) build bot during the last 2000 revisions. This makes a good hit list of tests to fix to reduce flakiness. (The worst offenders are at the bottom). Take aways: 1) compositing/geometry/limit-layer-bounds-transformed-overflow.html is by far the worst offender. 2) The websocket tests are ridiculously flaky. 3) The appcache tests also have a serious flakiness problem. I'll run the last 1 revisions over night and report on the results. Adam === Results === Occurances Test name 1 compositing/reflections/nested-reflection-animated.html 1 fast/css/font-face-download-error.html 1 fast/dom/collection-null-like-arguments.html 1 fast/history/history-subframe-with-name.html 1 fast/js/sputnik/Conformance/13_Function_Definition/13.2_Creating_Function_Objects/S13.2.2_A5_T1.html 1 fast/js/sputnik/Conformance/13_Function_Definition/13.2_Creating_Function_Objects/S13.2.2_A5_T2.html 1 fast/js/vardecl-preserve-arguments.html 1 http/tests/appcache/different-https-origin-resource-main.html 1 http/tests/appcache/fallback.html 1 http/tests/appcache/manifest-redirect.html 1 http/tests/appcache/origin-quota.html 1 http/tests/appcache/resource-redirect.html 1 http/tests/appcache/top-frame-3.html 1 http/tests/appcache/update-cache.html 1 http/tests/appcache/xhr-foreign-resource.html 1 http/tests/cache/subresource-expiration.html 1 http/tests/loading/basic-credentials-sent-automatically.html 1 http/tests/misc/uncacheable-script-repeated.html 1 http/tests/navigation/changing-frame-hierarchy-in-onload.html 1 http/tests/navigation/ping-cross-origin-from-https.html 1 http/tests/navigation/ping-cross-origin.html 1 http/tests/navigation/post-goback-same-url.html 1 http/tests/plugins/get-url.html 1 http/tests/plugins/npapi-response-headers.html 1 http/tests/plugins/third-party-cookie-accept-policy.html 1 http/tests/security/credentials-in-referer.html 1 http/tests/security/cross-frame-access-protocol-explicit-domain.html 1 inspector/debugger-pause-on-breakpoint.html 1 inspector/extensions-events.html 1 media/audio-constructor.html 1 media/video-currentTime-set.html 1 plugins/destroy-stream-twice.html 1 plugins/return-error-from-new-stream-doesnt-invoke-destroy-stream.html 1 svg/custom/use-invalid-style.svg 1 transitions/transition-end-event-transform.html 1 websocket/tests/send.html 1 websocket/tests/simple-stress.html 1 websocket/tests/sub-protocol-with-space.html 1 websocket/tests/url-no-trailing-slash.html 1 websocket/tests/url-with-empty-query.html 1 websocket/tests/url-with-query.html 1 websocket/tests/websocket-pending-activity.html 1 websocket/tests/workers/close-in-onmessage-crash.html 2 http/tests/appcache/foreign-iframe-main.html 2
Re: [webkit-dev] Flaky test hit list
On Wed, Sep 29, 2010 at 11:31 PM, Adam Barth aba...@webkit.org wrote: Tonight I wrote a new webkit-patch command for detecting flaky tests. Are you familiar with the Chromium flakiness dashboard? It provides a continuous summary of test results aggregated across multiple builders, including views into the expected/actual results, timing, logs, etc. See for example http://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=perf%2Ftyping-at-end-of-line.html and click the show results link in the lower left to see the failure diff. The front page, as I understand it, provides a list of the flakiest tests in order. (It also can cause your browser to melt, be forewarned.) And the json input the dashboard requires is already produced by new-run-webkit-tests, so I think it'd be trivial to plug the other WebKit builders into it. (Given how small your hit list is, I doubt the WebKit results would cause the dashboard to become as heavyweight as the Chromium one is.) PS: along those lines, here's an hierarchical view into test runtime generated from the results json archived by those bots. I hope to hook this into the dashboard once I figure a few other things out. http://neugierig.org/software/chromium/tests/#Layout Demo (Copy that URL including the space, sorry, it's a quick hack). ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Flaky test hit list
Currently the WebSocket tests are served over HTTP from the WebSocket server itself (which is written in Python). It looks like we can resolve the flakiness by serving the WebSocket tests from the normal Apache server that servers the rest of our HTTP tests. I'm going to work up a patch that does that. Please let me know if there's any reason we shouldn't make that change. Thanks, Adam On Thu, Sep 30, 2010 at 3:10 PM, Adam Barth aba...@webkit.org wrote: I'm investigating the websocket issue. It seems these tests are flaky because they time out. If you know about websockets, I'd appreciate any tips you have via #webkit. Adam On Wed, Sep 29, 2010 at 11:31 PM, Adam Barth aba...@webkit.org wrote: Tonight I wrote a new webkit-patch command for detecting flaky tests. Here the tests that have flaked out on the Snow Leopard (Tests) build bot during the last 2000 revisions. This makes a good hit list of tests to fix to reduce flakiness. (The worst offenders are at the bottom). Take aways: 1) compositing/geometry/limit-layer-bounds-transformed-overflow.html is by far the worst offender. 2) The websocket tests are ridiculously flaky. 3) The appcache tests also have a serious flakiness problem. I'll run the last 1 revisions over night and report on the results. Adam === Results === Occurances Test name 1 compositing/reflections/nested-reflection-animated.html 1 fast/css/font-face-download-error.html 1 fast/dom/collection-null-like-arguments.html 1 fast/history/history-subframe-with-name.html 1 fast/js/sputnik/Conformance/13_Function_Definition/13.2_Creating_Function_Objects/S13.2.2_A5_T1.html 1 fast/js/sputnik/Conformance/13_Function_Definition/13.2_Creating_Function_Objects/S13.2.2_A5_T2.html 1 fast/js/vardecl-preserve-arguments.html 1 http/tests/appcache/different-https-origin-resource-main.html 1 http/tests/appcache/fallback.html 1 http/tests/appcache/manifest-redirect.html 1 http/tests/appcache/origin-quota.html 1 http/tests/appcache/resource-redirect.html 1 http/tests/appcache/top-frame-3.html 1 http/tests/appcache/update-cache.html 1 http/tests/appcache/xhr-foreign-resource.html 1 http/tests/cache/subresource-expiration.html 1 http/tests/loading/basic-credentials-sent-automatically.html 1 http/tests/misc/uncacheable-script-repeated.html 1 http/tests/navigation/changing-frame-hierarchy-in-onload.html 1 http/tests/navigation/ping-cross-origin-from-https.html 1 http/tests/navigation/ping-cross-origin.html 1 http/tests/navigation/post-goback-same-url.html 1 http/tests/plugins/get-url.html 1 http/tests/plugins/npapi-response-headers.html 1 http/tests/plugins/third-party-cookie-accept-policy.html 1 http/tests/security/credentials-in-referer.html 1 http/tests/security/cross-frame-access-protocol-explicit-domain.html 1 inspector/debugger-pause-on-breakpoint.html 1 inspector/extensions-events.html 1 media/audio-constructor.html 1 media/video-currentTime-set.html 1 plugins/destroy-stream-twice.html 1 plugins/return-error-from-new-stream-doesnt-invoke-destroy-stream.html 1 svg/custom/use-invalid-style.svg 1 transitions/transition-end-event-transform.html 1 websocket/tests/send.html 1 websocket/tests/simple-stress.html 1 websocket/tests/sub-protocol-with-space.html 1 websocket/tests/url-no-trailing-slash.html 1 websocket/tests/url-with-empty-query.html 1 websocket/tests/url-with-query.html 1 websocket/tests/websocket-pending-activity.html 1 websocket/tests/workers/close-in-onmessage-crash.html 2 http/tests/appcache/foreign-iframe-main.html 2 http/tests/appcache/local-content.html 2 http/tests/appcache/main-resource-hash.html 2 http/tests/appcache/non-html.xhtml 2 http/tests/appcache/reload.html 2 http/tests/css/css-image-loading.html 2 http/tests/plugins/cross-frame-object-access.html 2 http/tests/security/cross-frame-access-port-explicit-domain.html 2 security/block-test.html 2 websocket/tests/bad-sub-protocol-non-ascii.html 2 websocket/tests/handshake-fail-by-sub-protocol-mismatch.html 2 websocket/tests/simple.html 2 websocket/tests/unicode.html 3 http/tests/appcache/cyrillic-uri.html 3 http/tests/appcache/deferred-events-delete-while-raising.html 3 http/tests/appcache/remove-cache.html 3 http/tests/appcache/top-frame-4.html 3 http/tests/navigation/image-load-in-unload-handler.html 3 websocket/tests/handshake-fail-by-cross-origin.html 3 websocket/tests/httponly-cookie.pl 3
Re: [webkit-dev] Flaky test hit list
https://bugs.webkit.org/show_bug.cgi?id=46956 Unfortunately, it doesn't solve the whole problem. It does seem to reduce the flakiness by a lot though. Adam On Thu, Sep 30, 2010 at 6:12 PM, Adam Barth aba...@webkit.org wrote: Currently the WebSocket tests are served over HTTP from the WebSocket server itself (which is written in Python). It looks like we can resolve the flakiness by serving the WebSocket tests from the normal Apache server that servers the rest of our HTTP tests. I'm going to work up a patch that does that. Please let me know if there's any reason we shouldn't make that change. Thanks, Adam On Thu, Sep 30, 2010 at 3:10 PM, Adam Barth aba...@webkit.org wrote: I'm investigating the websocket issue. It seems these tests are flaky because they time out. If you know about websockets, I'd appreciate any tips you have via #webkit. Adam On Wed, Sep 29, 2010 at 11:31 PM, Adam Barth aba...@webkit.org wrote: Tonight I wrote a new webkit-patch command for detecting flaky tests. Here the tests that have flaked out on the Snow Leopard (Tests) build bot during the last 2000 revisions. This makes a good hit list of tests to fix to reduce flakiness. (The worst offenders are at the bottom). Take aways: 1) compositing/geometry/limit-layer-bounds-transformed-overflow.html is by far the worst offender. 2) The websocket tests are ridiculously flaky. 3) The appcache tests also have a serious flakiness problem. I'll run the last 1 revisions over night and report on the results. Adam === Results === Occurances Test name 1 compositing/reflections/nested-reflection-animated.html 1 fast/css/font-face-download-error.html 1 fast/dom/collection-null-like-arguments.html 1 fast/history/history-subframe-with-name.html 1 fast/js/sputnik/Conformance/13_Function_Definition/13.2_Creating_Function_Objects/S13.2.2_A5_T1.html 1 fast/js/sputnik/Conformance/13_Function_Definition/13.2_Creating_Function_Objects/S13.2.2_A5_T2.html 1 fast/js/vardecl-preserve-arguments.html 1 http/tests/appcache/different-https-origin-resource-main.html 1 http/tests/appcache/fallback.html 1 http/tests/appcache/manifest-redirect.html 1 http/tests/appcache/origin-quota.html 1 http/tests/appcache/resource-redirect.html 1 http/tests/appcache/top-frame-3.html 1 http/tests/appcache/update-cache.html 1 http/tests/appcache/xhr-foreign-resource.html 1 http/tests/cache/subresource-expiration.html 1 http/tests/loading/basic-credentials-sent-automatically.html 1 http/tests/misc/uncacheable-script-repeated.html 1 http/tests/navigation/changing-frame-hierarchy-in-onload.html 1 http/tests/navigation/ping-cross-origin-from-https.html 1 http/tests/navigation/ping-cross-origin.html 1 http/tests/navigation/post-goback-same-url.html 1 http/tests/plugins/get-url.html 1 http/tests/plugins/npapi-response-headers.html 1 http/tests/plugins/third-party-cookie-accept-policy.html 1 http/tests/security/credentials-in-referer.html 1 http/tests/security/cross-frame-access-protocol-explicit-domain.html 1 inspector/debugger-pause-on-breakpoint.html 1 inspector/extensions-events.html 1 media/audio-constructor.html 1 media/video-currentTime-set.html 1 plugins/destroy-stream-twice.html 1 plugins/return-error-from-new-stream-doesnt-invoke-destroy-stream.html 1 svg/custom/use-invalid-style.svg 1 transitions/transition-end-event-transform.html 1 websocket/tests/send.html 1 websocket/tests/simple-stress.html 1 websocket/tests/sub-protocol-with-space.html 1 websocket/tests/url-no-trailing-slash.html 1 websocket/tests/url-with-empty-query.html 1 websocket/tests/url-with-query.html 1 websocket/tests/websocket-pending-activity.html 1 websocket/tests/workers/close-in-onmessage-crash.html 2 http/tests/appcache/foreign-iframe-main.html 2 http/tests/appcache/local-content.html 2 http/tests/appcache/main-resource-hash.html 2 http/tests/appcache/non-html.xhtml 2 http/tests/appcache/reload.html 2 http/tests/css/css-image-loading.html 2 http/tests/plugins/cross-frame-object-access.html 2 http/tests/security/cross-frame-access-port-explicit-domain.html 2 security/block-test.html 2 websocket/tests/bad-sub-protocol-non-ascii.html 2 websocket/tests/handshake-fail-by-sub-protocol-mismatch.html 2 websocket/tests/simple.html 2 websocket/tests/unicode.html 3 http/tests/appcache/cyrillic-uri.html 3 http/tests/appcache/deferred-events-delete-while-raising.html 3 http/tests/appcache/remove-cache.html 3
