Re: [websockets] Test results available

[Boris Zbarsky]

On 4/2/15 8:15 AM, Simon Pieters wrote:

http://www.w3c-test.org/websockets/keeping-connection-open/001.html --
the test is wrong.  Passing undefined means the argument is not
present per Web IDL, so this should not throw.


I assume you mean some other test since that test doesn't use undefined.


Er, yes.  I meant http://www.w3c-test.org/websockets/Close-undefined.htm 
I think, which has since been fixed looks like?.



I agree. The spec says to return from the constructor and establish the
connection in parallel, so it is not guaranteed which cookies to
include. Fix: https://github.com/w3c/web-platform-tests/pull/1701


Thanks!

-Boris

Re: [websockets] Test results available

[Simon Pieters]

On Thu, 26 Mar 2015 18:06:15 +0100, Boris Zbarsky  wrote:


On 3/26/15 1:02 PM, Boris Zbarsky wrote:

It looks like the tests that are failed with an "Error" as opposed to a
"Fail" are not being counted in the "<2 passes" list?


And the for  
http://www.w3c-test.org/websockets/keeping-connection-open/001.html  
which is all-"Timeout".


This test passes for me in Blink and Gecko. It has a  indicating it  
needs a long timeout. Is it a bug in the test runner being used here?


--
Simon Pieters
Opera Software

Re: [websockets] Test results available

[Simon Pieters]

On Thu, 26 Mar 2015 16:37:28 +0100, Olli Pettay  wrote:

websockets/interfaces.html  the test itself has bugs (uses old  
idlharness.js?).


https://github.com/w3c/web-platform-tests/pull/1714

--
Simon Pieters
Opera Software

Re: [websockets] Test results available

[Simon Pieters]

On Thu, 26 Mar 2015 16:37:28 +0100, Olli Pettay  wrote:

Also websockets/interfaces/WebSocket/events/013.html is buggy. Seems to  
rely on blink/presto's EventHandler behavior, which is not

what the specs says should happen.


It probably matched the spec at the time the test was written.

Spec now says [TreatNonObjectAsNull] so this should return the object, not  
null.

https://html.spec.whatwg.org/multipage/webappapis.html#eventhandlernonnull

Fix: https://github.com/w3c/web-platform-tests/pull/1704

--
Simon Pieters
Opera Software

Re: [websockets] Test results available

[Simon Pieters]

On Thu, 02 Apr 2015 14:15:10 +0200, Simon Pieters  wrote:

http://www.w3c-test.org/websockets/interfaces/WebSocket/readyState/003.html  
looks wrong to me: the value it should get is in fact undefined, since  
the property got deleted from the prototype.


(Will have a look.)


http://heycam.github.io/webidl/#es-attributes
[[
configurable is false if the attribute was declared with the [Unforgeable]  
extended attribute and true otherwise;

]]

Fix: https://github.com/w3c/web-platform-tests/pull/1703

--
Simon Pieters
Opera Software

Re: [websockets] Test results available

[Simon Pieters]

On Thu, 02 Apr 2015 14:15:10 +0200, Simon Pieters  wrote:

http://www.w3c-test.org/websockets/keeping-connection-open/001.html --  
the test is wrong.  Passing undefined means the argument is not present  
per Web IDL, so this should not throw.


I assume you mean some other test since that test doesn't use undefined.  
(I'll have a look and fix ones that use undefined.)


https://github.com/w3c/web-platform-tests/pull/1702

--
Simon Pieters
Opera Software

Re: [websockets] Test results available

[Simon Pieters]

On Thu, 26 Mar 2015 18:24:22 +0100, Boris Zbarsky  wrote:


On 3/26/15 10:51 AM, Arthur Barstow wrote:

If anyone is willing to help with the failure analysis, that would be
very much appreciated.


Taking a brief look at some of the failures in Firefox, in addition to  
the ones Olli already posted about:


http://www.w3c-test.org/websockets/keeping-connection-open/001.html --  
the test is wrong.  Passing undefined means the argument is not present  
per Web IDL, so this should not throw.


I assume you mean some other test since that test doesn't use undefined.  
(I'll have a look and fix ones that use undefined.)


http://www.w3c-test.org/websockets/cookies/001.html seems racy to me: it  
kicks off an async test and then immediately removes the cookie, so it's  
not obvious to me why it expects that cookie to be present in the  
websocket connection; the cookie may well be removed before the  
connection is set up.


I agree. The spec says to return from the constructor and establish the  
connection in parallel, so it is not guaranteed which cookies to include.  
Fix: https://github.com/w3c/web-platform-tests/pull/1701


But arguably that is a spec bug. It would be nice if it was deterministic.  
Filed https://www.w3.org/Bugs/Public/show_bug.cgi?id=28393


http://www.w3c-test.org/websockets/interfaces/WebSocket/readyState/003.html  
looks wrong to me: the value it should get is in fact undefined, since  
the property got deleted from the prototype.


(Will have a look.)

--
Simon Pieters
Opera Software

Re: [websockets] Test results available

[Arthur Barstow]

On 3/26/15 1:02 PM, Boris Zbarsky wrote:

On 3/26/15 10:51 AM, Arthur Barstow wrote:

* All results 

* <2 passes 



Overall these results are pretty good: 97% of the 495 tests have two or
more passes.


Arthur,

It looks like the tests that are failed with an "Error" as opposed to 
a "Fail" are not being counted in the "<2 passes" list?  For example, 
http://www.w3c-test.org/websockets/Create-Secure-valid-url-array-protocols.htm 
is not in that list even though everyone fails it.


Is that an accident, or a purposeful decision?


I'm glad you reported this Boris! Indeed the less-than-2 table is 
missing several tests that have 4 Errors + 1 Timeout. I don't know why 
they aren't included. Such tests are also missing from the other 
automatically generated file "complete-fails" [cf] (which is a proper 
subset of the <2 table).


I'm not sure of the reason(s) to omit such errors from the less-than-2 
table or the complete-fail table. It could be a bug in the tool 
[wptreport] and perhaps there is a switch that would include the missing 
errors. I also vaguely recall a discussion about pervasive errors being 
a relatively strong indicator of a test case bug. Perhaps Robin can help 
here; Robin?


Regardless, we will also need to investigate all of these other failures 
missing from <2.


-Thanks, AB

[cf] 
[wptreport] 

Re: [websockets] Test results available

[Boris Zbarsky]

On 3/26/15 10:51 AM, Arthur Barstow wrote:

If anyone is willing to help with the failure analysis, that would be
very much appreciated.


Taking a brief look at some of the failures in Firefox, in addition to 
the ones Olli already posted about:


http://www.w3c-test.org/websockets/keeping-connection-open/001.html -- 
the test is wrong.  Passing undefined means the argument is not present 
per Web IDL, so this should not throw.


http://www.w3c-test.org/websockets/cookies/001.html seems racy to me: it 
kicks off an async test and then immediately removes the cookie, so it's 
not obvious to me why it expects that cookie to be present in the 
websocket connection; the cookie may well be removed before the 
connection is set up.


http://www.w3c-test.org/websockets/interfaces/WebSocket/readyState/003.html 
looks wrong to me: the value it should get is in fact undefined, since 
the property got deleted from the prototype.


-Boris

Re: [websockets] Test results available

[James Graham]

On 26/03/15 15:37, Olli Pettay wrote:

> websockets/interfaces.html  the test itself has bugs (uses old
> idlharness.js?).
> 
> Also websockets/interfaces/WebSocket/events/013.html is buggy. Seems to
> rely on blink/presto's EventHandler behavior, which is not
> what the specs says should happen.

If you are inclined to fix these you can either do it in GitHub or in
mozilla-inbound, from where the changes will get upstreamed.

Re: [websockets] Test results available

[Boris Zbarsky]

On 3/26/15 1:02 PM, Boris Zbarsky wrote:

It looks like the tests that are failed with an "Error" as opposed to a
"Fail" are not being counted in the "<2 passes" list?


And the for 
http://www.w3c-test.org/websockets/keeping-connection-open/001.html 
which is all-"Timeout".


-Boris

Re: [websockets] Test results available

[Boris Zbarsky]

On 3/26/15 10:51 AM, Arthur Barstow wrote:

* All results 

* <2 passes 

Overall these results are pretty good: 97% of the 495 tests have two or
more passes.


Arthur,

It looks like the tests that are failed with an "Error" as opposed to a 
"Fail" are not being counted in the "<2 passes" list?  For example, 
http://www.w3c-test.org/websockets/Create-Secure-valid-url-array-protocols.htm 
is not in that list even though everyone fails it.


Is that an accident, or a purposeful decision?

-Boris

Re: [websockets] Test results available

[Olli Pettay]

On 03/26/2015 04:51 PM, Arthur Barstow wrote:

Earlier today I ran the Web Sockets tests on Chrome 41, Chrome/Canary 43, FF 
Nightly 39, IE 11, and Opera 12 and pushed the results to the
test-results repo:

* All results 

* <2 passes 

Overall these results are pretty good: 97% of the 495 tests have two or more 
passes.

If anyone is willing to help with the failure analysis, that would be very much 
appreciated.

Odin, Simon - for the purposes of evaluating these results and the Candidate 
Recommendation (exit criteria), should the Opera data be included?

-Thanks, ArtB






websockets/interfaces.html  the test itself has bugs (uses old idlharness.js?).

Also websockets/interfaces/WebSocket/events/013.html is buggy. Seems to rely on 
blink/presto's EventHandler behavior, which is not
what the specs says should happen.


-Olli

[websockets] Test results available

[Arthur Barstow]

Earlier today I ran the Web Sockets tests on Chrome 41, Chrome/Canary 
43, FF Nightly 39, IE 11, and Opera 12 and pushed the results to the 
test-results repo:


* All results 

* <2 passes 

Overall these results are pretty good: 97% of the 495 tests have two or 
more passes.


If anyone is willing to help with the failure analysis, that would be 
very much appreciated.


Odin, Simon - for the purposes of evaluating these results and the 
Candidate Recommendation (exit criteria), should the Opera data be included?


-Thanks, ArtB