Re: [websockets] Test results available

2015-04-02 Thread Simon Pieters

On Thu, 02 Apr 2015 14:15:10 +0200, Simon Pieters sim...@opera.com wrote:

http://www.w3c-test.org/websockets/keeping-connection-open/001.html --  
the test is wrong.  Passing undefined means the argument is not present  
per Web IDL, so this should not throw.


I assume you mean some other test since that test doesn't use undefined.  
(I'll have a look and fix ones that use undefined.)


https://github.com/w3c/web-platform-tests/pull/1702

--
Simon Pieters
Opera Software



Re: [websockets] Test results available

2015-04-02 Thread Simon Pieters

On Thu, 26 Mar 2015 18:24:22 +0100, Boris Zbarsky bzbar...@mit.edu wrote:


On 3/26/15 10:51 AM, Arthur Barstow wrote:

If anyone is willing to help with the failure analysis, that would be
very much appreciated.


Taking a brief look at some of the failures in Firefox, in addition to  
the ones Olli already posted about:


http://www.w3c-test.org/websockets/keeping-connection-open/001.html --  
the test is wrong.  Passing undefined means the argument is not present  
per Web IDL, so this should not throw.


I assume you mean some other test since that test doesn't use undefined.  
(I'll have a look and fix ones that use undefined.)


http://www.w3c-test.org/websockets/cookies/001.html seems racy to me: it  
kicks off an async test and then immediately removes the cookie, so it's  
not obvious to me why it expects that cookie to be present in the  
websocket connection; the cookie may well be removed before the  
connection is set up.


I agree. The spec says to return from the constructor and establish the  
connection in parallel, so it is not guaranteed which cookies to include.  
Fix: https://github.com/w3c/web-platform-tests/pull/1701


But arguably that is a spec bug. It would be nice if it was deterministic.  
Filed https://www.w3.org/Bugs/Public/show_bug.cgi?id=28393


http://www.w3c-test.org/websockets/interfaces/WebSocket/readyState/003.html  
looks wrong to me: the value it should get is in fact undefined, since  
the property got deleted from the prototype.


(Will have a look.)

--
Simon Pieters
Opera Software



Re: [websockets] Test results available

2015-04-02 Thread Simon Pieters

On Thu, 02 Apr 2015 14:15:10 +0200, Simon Pieters sim...@opera.com wrote:

http://www.w3c-test.org/websockets/interfaces/WebSocket/readyState/003.html  
looks wrong to me: the value it should get is in fact undefined, since  
the property got deleted from the prototype.


(Will have a look.)


http://heycam.github.io/webidl/#es-attributes
[[
configurable is false if the attribute was declared with the [Unforgeable]  
extended attribute and true otherwise;

]]

Fix: https://github.com/w3c/web-platform-tests/pull/1703

--
Simon Pieters
Opera Software



Re: [websockets] Test results available

2015-04-02 Thread Boris Zbarsky

On 4/2/15 8:15 AM, Simon Pieters wrote:

http://www.w3c-test.org/websockets/keeping-connection-open/001.html --
the test is wrong.  Passing undefined means the argument is not
present per Web IDL, so this should not throw.


I assume you mean some other test since that test doesn't use undefined.


Er, yes.  I meant http://www.w3c-test.org/websockets/Close-undefined.htm 
I think, which has since been fixed looks like?.



I agree. The spec says to return from the constructor and establish the
connection in parallel, so it is not guaranteed which cookies to
include. Fix: https://github.com/w3c/web-platform-tests/pull/1701


Thanks!

-Boris



Re: [websockets] Test results available

2015-04-02 Thread Simon Pieters

On Thu, 26 Mar 2015 18:06:15 +0100, Boris Zbarsky bzbar...@mit.edu wrote:


On 3/26/15 1:02 PM, Boris Zbarsky wrote:

It looks like the tests that are failed with an Error as opposed to a
Fail are not being counted in the 2 passes list?


And the for  
http://www.w3c-test.org/websockets/keeping-connection-open/001.html  
which is all-Timeout.


This test passes for me in Blink and Gecko. It has a meta indicating it  
needs a long timeout. Is it a bug in the test runner being used here?


--
Simon Pieters
Opera Software



Re: [websockets] Test results available

2015-04-02 Thread Simon Pieters

On Thu, 26 Mar 2015 16:37:28 +0100, Olli Pettay o...@pettay.fi wrote:

websockets/interfaces.html  the test itself has bugs (uses old  
idlharness.js?).


https://github.com/w3c/web-platform-tests/pull/1714

--
Simon Pieters
Opera Software



Re: [websockets] Test results available

2015-04-02 Thread Simon Pieters

On Thu, 26 Mar 2015 16:37:28 +0100, Olli Pettay o...@pettay.fi wrote:

Also websockets/interfaces/WebSocket/events/013.html is buggy. Seems to  
rely on blink/presto's EventHandler behavior, which is not

what the specs says should happen.


It probably matched the spec at the time the test was written.

Spec now says [TreatNonObjectAsNull] so this should return the object, not  
null.

https://html.spec.whatwg.org/multipage/webappapis.html#eventhandlernonnull

Fix: https://github.com/w3c/web-platform-tests/pull/1704

--
Simon Pieters
Opera Software



Re: [websockets] Test results available

2015-03-26 Thread Boris Zbarsky

On 3/26/15 10:51 AM, Arthur Barstow wrote:

If anyone is willing to help with the failure analysis, that would be
very much appreciated.


Taking a brief look at some of the failures in Firefox, in addition to 
the ones Olli already posted about:


http://www.w3c-test.org/websockets/keeping-connection-open/001.html -- 
the test is wrong.  Passing undefined means the argument is not present 
per Web IDL, so this should not throw.


http://www.w3c-test.org/websockets/cookies/001.html seems racy to me: it 
kicks off an async test and then immediately removes the cookie, so it's 
not obvious to me why it expects that cookie to be present in the 
websocket connection; the cookie may well be removed before the 
connection is set up.


http://www.w3c-test.org/websockets/interfaces/WebSocket/readyState/003.html 
looks wrong to me: the value it should get is in fact undefined, since 
the property got deleted from the prototype.


-Boris



Re: [websockets] Test results available

2015-03-26 Thread James Graham
On 26/03/15 15:37, Olli Pettay wrote:

 websockets/interfaces.html  the test itself has bugs (uses old
 idlharness.js?).
 
 Also websockets/interfaces/WebSocket/events/013.html is buggy. Seems to
 rely on blink/presto's EventHandler behavior, which is not
 what the specs says should happen.

If you are inclined to fix these you can either do it in GitHub or in
mozilla-inbound, from where the changes will get upstreamed.




Re: [websockets] Test results available

2015-03-26 Thread Boris Zbarsky

On 3/26/15 10:51 AM, Arthur Barstow wrote:

* All results http://w3c.github.io/test-results/websockets/all.html

* 2 passes http://w3c.github.io/test-results/websockets/less-than-2.html

Overall these results are pretty good: 97% of the 495 tests have two or
more passes.


Arthur,

It looks like the tests that are failed with an Error as opposed to a 
Fail are not being counted in the 2 passes list?  For example, 
http://www.w3c-test.org/websockets/Create-Secure-valid-url-array-protocols.htm 
is not in that list even though everyone fails it.


Is that an accident, or a purposeful decision?

-Boris



Re: [websockets] Test results available

2015-03-26 Thread Arthur Barstow

On 3/26/15 1:02 PM, Boris Zbarsky wrote:

On 3/26/15 10:51 AM, Arthur Barstow wrote:

* All results http://w3c.github.io/test-results/websockets/all.html

* 2 passes 
http://w3c.github.io/test-results/websockets/less-than-2.html


Overall these results are pretty good: 97% of the 495 tests have two or
more passes.


Arthur,

It looks like the tests that are failed with an Error as opposed to 
a Fail are not being counted in the 2 passes list?  For example, 
http://www.w3c-test.org/websockets/Create-Secure-valid-url-array-protocols.htm 
is not in that list even though everyone fails it.


Is that an accident, or a purposeful decision?


I'm glad you reported this Boris! Indeed the less-than-2 table is 
missing several tests that have 4 Errors + 1 Timeout. I don't know why 
they aren't included. Such tests are also missing from the other 
automatically generated file complete-fails [cf] (which is a proper 
subset of the 2 table).


I'm not sure of the reason(s) to omit such errors from the less-than-2 
table or the complete-fail table. It could be a bug in the tool 
[wptreport] and perhaps there is a switch that would include the missing 
errors. I also vaguely recall a discussion about pervasive errors being 
a relatively strong indicator of a test case bug. Perhaps Robin can help 
here; Robin?


Regardless, we will also need to investigate all of these other failures 
missing from 2.


-Thanks, AB

[cf] http://w3c.github.io/test-results/websockets/complete-fails.html
[wptreport] https://github.com/darobin/wptreport




Re: [websockets] Test results available

2015-03-26 Thread Boris Zbarsky

On 3/26/15 1:02 PM, Boris Zbarsky wrote:

It looks like the tests that are failed with an Error as opposed to a
Fail are not being counted in the 2 passes list?


And the for 
http://www.w3c-test.org/websockets/keeping-connection-open/001.html 
which is all-Timeout.


-Boris



[websockets] Test results available

2015-03-26 Thread Arthur Barstow
Earlier today I ran the Web Sockets tests on Chrome 41, Chrome/Canary 
43, FF Nightly 39, IE 11, and Opera 12 and pushed the results to the 
test-results repo:


* All results http://w3c.github.io/test-results/websockets/all.html

* 2 passes http://w3c.github.io/test-results/websockets/less-than-2.html

Overall these results are pretty good: 97% of the 495 tests have two or 
more passes.


If anyone is willing to help with the failure analysis, that would be 
very much appreciated.


Odin, Simon - for the purposes of evaluating these results and the 
Candidate Recommendation (exit criteria), should the Opera data be included?


-Thanks, ArtB





Re: [websockets] Test results available

2015-03-26 Thread Olli Pettay

On 03/26/2015 04:51 PM, Arthur Barstow wrote:

Earlier today I ran the Web Sockets tests on Chrome 41, Chrome/Canary 43, FF 
Nightly 39, IE 11, and Opera 12 and pushed the results to the
test-results repo:

* All results http://w3c.github.io/test-results/websockets/all.html

* 2 passes http://w3c.github.io/test-results/websockets/less-than-2.html

Overall these results are pretty good: 97% of the 495 tests have two or more 
passes.

If anyone is willing to help with the failure analysis, that would be very much 
appreciated.

Odin, Simon - for the purposes of evaluating these results and the Candidate 
Recommendation (exit criteria), should the Opera data be included?

-Thanks, ArtB






websockets/interfaces.html  the test itself has bugs (uses old idlharness.js?).

Also websockets/interfaces/WebSocket/events/013.html is buggy. Seems to rely on 
blink/presto's EventHandler behavior, which is not
what the specs says should happen.


-Olli