On Wed, Oct 17, 2018 at 11:53 PM Boris Zbarsky <bzbar...@mit.edu> wrote:
> On 10/13/18 3:27 AM, Philip Jägenstedt wrote:
> > Fiddling with these rules can reveal lots
> > more potential issues, and if you like I could provide reports on that too.
> I would be pretty interested in that, yes.  In particular, a report
> where there is 1 "not PASS and not FAIL" and 3 "PASS" would be pretty
> helpful, I suspect.

Rerunning my script it's apparent that unreliable Edge results [1]
leads to the same tests being considered lone failures or not for the
other browsers. So, I've use the same set of runs for this report of
what you suggested:

It's not a long list, so I checked them all and they are timeouts.
This is sometimes the failure mode for genuine problems, so looking
over these might be valuable.

> By the way, I recently found some tests that fail when run directly but
> pass in the harness.  :(  For example
> http://w3c-test.org/html/infrastructure/common-dom-interfaces/collections/htmlallcollection.html
> fails various subtests in all browsers due to the <div id="log"> being
> in the DOM when running directly.  Not really sure what we can do with that.

That's a bit odd, the <div id="log"> is in the markup and would be
when running manually or under automation. Are you sure that explains
the difference? If it does, then just removing it from the markup and
adapting any affected tests would be the way to go. I updated the test
pretty recently, if you're confident it's broken can you file a wpt
issue and assign me?

[1] https://github.com/web-platform-tests/results-collection/issues/563
dev-platform mailing list

Reply via email to