Re: [websockets] Test results available
On Thu, 02 Apr 2015 14:15:10 +0200, Simon Pieters sim...@opera.com wrote: http://www.w3c-test.org/websockets/keeping-connection-open/001.html -- the test is wrong. Passing undefined means the argument is not present per Web IDL, so this should not throw. I assume you mean some other test since that test doesn't use undefined. (I'll have a look and fix ones that use undefined.) https://github.com/w3c/web-platform-tests/pull/1702 -- Simon Pieters Opera Software
Re: [websockets] Test results available
On Thu, 26 Mar 2015 18:24:22 +0100, Boris Zbarsky bzbar...@mit.edu wrote: On 3/26/15 10:51 AM, Arthur Barstow wrote: If anyone is willing to help with the failure analysis, that would be very much appreciated. Taking a brief look at some of the failures in Firefox, in addition to the ones Olli already posted about: http://www.w3c-test.org/websockets/keeping-connection-open/001.html -- the test is wrong. Passing undefined means the argument is not present per Web IDL, so this should not throw. I assume you mean some other test since that test doesn't use undefined. (I'll have a look and fix ones that use undefined.) http://www.w3c-test.org/websockets/cookies/001.html seems racy to me: it kicks off an async test and then immediately removes the cookie, so it's not obvious to me why it expects that cookie to be present in the websocket connection; the cookie may well be removed before the connection is set up. I agree. The spec says to return from the constructor and establish the connection in parallel, so it is not guaranteed which cookies to include. Fix: https://github.com/w3c/web-platform-tests/pull/1701 But arguably that is a spec bug. It would be nice if it was deterministic. Filed https://www.w3.org/Bugs/Public/show_bug.cgi?id=28393 http://www.w3c-test.org/websockets/interfaces/WebSocket/readyState/003.html looks wrong to me: the value it should get is in fact undefined, since the property got deleted from the prototype. (Will have a look.) -- Simon Pieters Opera Software
Re: [websockets] Test results available
On Thu, 02 Apr 2015 14:15:10 +0200, Simon Pieters sim...@opera.com wrote: http://www.w3c-test.org/websockets/interfaces/WebSocket/readyState/003.html looks wrong to me: the value it should get is in fact undefined, since the property got deleted from the prototype. (Will have a look.) http://heycam.github.io/webidl/#es-attributes [[ configurable is false if the attribute was declared with the [Unforgeable] extended attribute and true otherwise; ]] Fix: https://github.com/w3c/web-platform-tests/pull/1703 -- Simon Pieters Opera Software
Re: [websockets] Test results available
On 4/2/15 8:15 AM, Simon Pieters wrote: http://www.w3c-test.org/websockets/keeping-connection-open/001.html -- the test is wrong. Passing undefined means the argument is not present per Web IDL, so this should not throw. I assume you mean some other test since that test doesn't use undefined. Er, yes. I meant http://www.w3c-test.org/websockets/Close-undefined.htm I think, which has since been fixed looks like?. I agree. The spec says to return from the constructor and establish the connection in parallel, so it is not guaranteed which cookies to include. Fix: https://github.com/w3c/web-platform-tests/pull/1701 Thanks! -Boris
Re: [websockets] Test results available
On Thu, 26 Mar 2015 18:06:15 +0100, Boris Zbarsky bzbar...@mit.edu wrote: On 3/26/15 1:02 PM, Boris Zbarsky wrote: It looks like the tests that are failed with an Error as opposed to a Fail are not being counted in the 2 passes list? And the for http://www.w3c-test.org/websockets/keeping-connection-open/001.html which is all-Timeout. This test passes for me in Blink and Gecko. It has a meta indicating it needs a long timeout. Is it a bug in the test runner being used here? -- Simon Pieters Opera Software
Re: [websockets] Test results available
On Thu, 26 Mar 2015 16:37:28 +0100, Olli Pettay o...@pettay.fi wrote: websockets/interfaces.html the test itself has bugs (uses old idlharness.js?). https://github.com/w3c/web-platform-tests/pull/1714 -- Simon Pieters Opera Software
Re: [websockets] Test results available
On Thu, 26 Mar 2015 16:37:28 +0100, Olli Pettay o...@pettay.fi wrote: Also websockets/interfaces/WebSocket/events/013.html is buggy. Seems to rely on blink/presto's EventHandler behavior, which is not what the specs says should happen. It probably matched the spec at the time the test was written. Spec now says [TreatNonObjectAsNull] so this should return the object, not null. https://html.spec.whatwg.org/multipage/webappapis.html#eventhandlernonnull Fix: https://github.com/w3c/web-platform-tests/pull/1704 -- Simon Pieters Opera Software
Re: [websockets] Test results available
On 3/26/15 10:51 AM, Arthur Barstow wrote: If anyone is willing to help with the failure analysis, that would be very much appreciated. Taking a brief look at some of the failures in Firefox, in addition to the ones Olli already posted about: http://www.w3c-test.org/websockets/keeping-connection-open/001.html -- the test is wrong. Passing undefined means the argument is not present per Web IDL, so this should not throw. http://www.w3c-test.org/websockets/cookies/001.html seems racy to me: it kicks off an async test and then immediately removes the cookie, so it's not obvious to me why it expects that cookie to be present in the websocket connection; the cookie may well be removed before the connection is set up. http://www.w3c-test.org/websockets/interfaces/WebSocket/readyState/003.html looks wrong to me: the value it should get is in fact undefined, since the property got deleted from the prototype. -Boris
Re: [websockets] Test results available
On 26/03/15 15:37, Olli Pettay wrote: websockets/interfaces.html the test itself has bugs (uses old idlharness.js?). Also websockets/interfaces/WebSocket/events/013.html is buggy. Seems to rely on blink/presto's EventHandler behavior, which is not what the specs says should happen. If you are inclined to fix these you can either do it in GitHub or in mozilla-inbound, from where the changes will get upstreamed.
Re: [websockets] Test results available
On 3/26/15 10:51 AM, Arthur Barstow wrote: * All results http://w3c.github.io/test-results/websockets/all.html * 2 passes http://w3c.github.io/test-results/websockets/less-than-2.html Overall these results are pretty good: 97% of the 495 tests have two or more passes. Arthur, It looks like the tests that are failed with an Error as opposed to a Fail are not being counted in the 2 passes list? For example, http://www.w3c-test.org/websockets/Create-Secure-valid-url-array-protocols.htm is not in that list even though everyone fails it. Is that an accident, or a purposeful decision? -Boris
Re: [websockets] Test results available
On 3/26/15 1:02 PM, Boris Zbarsky wrote: It looks like the tests that are failed with an Error as opposed to a Fail are not being counted in the 2 passes list? And the for http://www.w3c-test.org/websockets/keeping-connection-open/001.html which is all-Timeout. -Boris
Re: [websockets] Test results available
On 03/26/2015 04:51 PM, Arthur Barstow wrote: Earlier today I ran the Web Sockets tests on Chrome 41, Chrome/Canary 43, FF Nightly 39, IE 11, and Opera 12 and pushed the results to the test-results repo: * All results http://w3c.github.io/test-results/websockets/all.html * 2 passes http://w3c.github.io/test-results/websockets/less-than-2.html Overall these results are pretty good: 97% of the 495 tests have two or more passes. If anyone is willing to help with the failure analysis, that would be very much appreciated. Odin, Simon - for the purposes of evaluating these results and the Candidate Recommendation (exit criteria), should the Opera data be included? -Thanks, ArtB websockets/interfaces.html the test itself has bugs (uses old idlharness.js?). Also websockets/interfaces/WebSocket/events/013.html is buggy. Seems to rely on blink/presto's EventHandler behavior, which is not what the specs says should happen. -Olli
