A lot of the test results I'm seeing there are the "harness status",
which has been a common cause of confusion:
https://github.com/web-platform-tests/wpt.fyi/issues/62. Don't know
quite what the right solution is here, but it's definitely still
confusing.
/Sam
On Mon, Feb 25, 2019 at 9:57 PM Philip Jägenstedt wrote:
>
> I think I know what's going on there. When drilling down into tests and
> subtests, only those matching the filter are shown. Clearing the filter
> things look a bit different in the directories you mentioned:
> https://wpt.fyi/results/ambient-light?label=master=experimental=chrome%5Btaskcluster%5D=firefox%5Btaskcluster%5D=safari%5Bazure%5D
> https://wpt.fyi/results/bluetooth?label=master=experimental=chrome%5Btaskcluster%5D=firefox%5Btaskcluster%5D=safari%5Bazure%5D
>
> In particular for idlharness.js tests some subtests will pass because they're
> preconditions for the real tests. There will also be tests that check that
> something doesn't work, which will pass even if the feature is entirely
> unsupported if "not working" results in the same thing, e.g. throwing an
> exception. Sometimes tests can be tweaked to fail if the feature is
> unsupported.
>
> Drilling down into a directory somewhat at random and clearing filters, it
> does look like this is legit:
> https://wpt.fyi/results/fetch/api/cors?label=master=experimental=chrome%5Btaskcluster%5D=firefox%5Btaskcluster%5D=safari%5Bazure%5D
>
> On Mon, Feb 25, 2019 at 8:31 PM Maciej Stachowiak wrote:
>>
>>
>> Neat.
>>
>> I see some obvious areas for focus, where Safari fails lots of tests that
>> the other browser don’t.
>>
>> For context, I tried looking at this view, which shows all tests that Safari
>> and Firefox pass with Safari results regardless of result:
>> https://wpt.fyi/results/?label=master=experimental=chrome%5Btaskcluster%5D=firefox%5Btaskcluster%5D=safari%5Bazure%5D=%28chrome%3Apass%7Cchrome%3Aok%29+%28firefox%3Apass%7Cfirefox%3Aok%29
>>
>> I noticed some puzzling results there: Safari passes all the ambient-light
>> and bluetooth tests that Chrome and Firefox do, despite not supporting these
>> standards at all. (For that matter I’m not sure Firefox supports these specs
>> either.) Not sure if harness problem, or dubious tests that don’t actually
>> test the standard.
>>
>> Regards,
>> Maciej
>>
>> On Feb 25, 2019, at 5:48 AM, Philip Jägenstedt wrote:
>>
>> I'd like to point out right away that diagnosing reftest failures is
>> currently cumbersome because we don't store the screenshots. This is
>> also a work in progress:
>> https://docs.google.com/document/d/1IhZa4mrjK1msUMhtamKwKJ_HhXD-nqh_4-BcPWM6soQ/edit?usp=sharing
>>
>> Until that has launched, I would recommend ignoring reftest failures
>> if the cause of failure isn't obvious.
>>
>> On Mon, Feb 25, 2019 at 2:30 PM Philip Jägenstedt
>> wrote:
>>
>>
>> Hi all,
>>
>> Following the improved Safari results last year [1] and the discussion
>> that generated, I'm happy to announce that the filtering requested as
>> now available in the search box. The full syntax is documented [2] but
>> there's also a new insights view [3] with some useful searches.
>>
>> Especially interesting for this list could be this view, of Chrome
>> Dev, Firefox Nightly and Safari Technology Preview, filtered to the
>> Safari-specific failures:
>> https://wpt.fyi/results/?label=master=experimental=chrome%5Btaskcluster%5D=firefox%5Btaskcluster%5D=safari%5Bazure%5D=%28chrome%3Apass%7Cchrome%3Aok%29+%28firefox%3Apass%7Cfirefox%3Aok%29+%28safari%3A%21pass%26safari%3A%21ok%29
>>
>> Both Google and Mozilla have efforts [4][5] to reduce the number of
>> Chrome/Firefox-specific failures, as this seems like a category of
>> problems which especially valuable, where changing just one browser
>> can remove a pain point for web developers.
>>
>> No doubt some failures are spurious, but hopefully there is value to
>> be found by looking into where the largest numbers of failures appear
>> to be. If something seems to be wrong with the search/filtering,
>> please file an issue for us! [6]
>>
>> Credit to Mark Dittmer and Luke Bjerring who owned this project.
>>
>> P.S. We are also working on triage metadata for wpt.fyi, to make it
>> possible to burn down a list of failures like this and not later have
>> to re-triage to find the new failures. [7]
>>
>> [1] https://lists.webkit.org/pipermail/webkit-dev/2018-October/030209.html
>> [2]
>> https://github.com/web-platform-tests/wpt.fyi/blob/master/api/query/README.md
>> [3] https://staging.wpt.fyi/insights
>> [4] https://bugs.chromium.org/p/chromium/issues/detail?id=896242
>> [5] https://bugzilla.mozilla.org/show_bug.cgi?id=1498357
>> [6]
>> https://github.com/web-platform-tests/wpt.fyi/issues/new?title=Structured+Queries+issue=web-platform-tests/wpt.fyi/8=bug=search.md
>> [7]
>> https://docs.google.com/document/d/1oWYVkc2ztANCGUxwNVTQHlWV32zq6Ifq9jkkbYNbSAg/edit?usp=sharing
>>
>> ___
>>
