Alright, I've written a one-off script [1] to find the Safari-only failures, and here's the output: https://gist.github.com/foolip/4d410ce79416bcdce71feb212159a02e
Barring bugs, each of linked tests or one of its subtests should be failing in Safari Technology Preview and passing in stable versions of Chrome, Edge and Firefox. Numerically, most of the failures are in css (622), encoding (135) and html (60). With css, it's mostly css/CSS2. I hope looking through this may be of use to you! [1] https://github.com/foolip/ad-hoc-wpt-results-analysis On Mon, Oct 8, 2018 at 11:50 PM Philip Jägenstedt <foo...@chromium.org> wrote: > > That filtering capability unfortunately does not yet exist on wpt.fyi > but it's a high priority and actively being worked on: > https://github.com/web-platform-tests/wpt.fyi/issues/201 > > FWIW, I suspect that these purposes, comparing to the stable versions > of all *other* browsers might be the most useful: > https://wpt.fyi/results/?product=chrome%5Bstable%5D&product=edge%5Bstable%5D&product=firefox%5Bstable%5D&product=safari%5Bexperimental%5D&aligned > > Again, no way to filter on wpt.fyi, but I'll see if I can download the > full results and write a quick script. > > On Thu, Oct 4, 2018 at 11:49 PM Ryosuke Niwa <rn...@webkit.org> wrote: > > > > Thanks for the intriguing data, Philip. > > > > Is there a way to get a list of tests where all other browsers pass but > > Safari / WebKit fail? > > > > That would allow us to quickly identify the set of tests we can fix to > > improve the interoperability across browsers right away. > > > > - R. Niwa > > > > On Tue, Oct 2, 2018 at 3:45 AM Philip Jägenstedt <foo...@chromium.org> > > wrote: > >> > >> Hi WebKittens, > >> > >> Fresh off the bots, I'm excited to report more robust Safari results, > >> and that Safari WPT pass rates are clearly improving! Thanks to the > >> hard work of Mike Pennisi [1] we now have the first Safari 12 results: > >> https://wpt.fyi/results/?sha=ee2e69bfb1&product=safari-12.0 > >> > >> This uses the same setup as for Safari Technology Preview, which has > >> been running for a while [2] and are the results you see on the > >> "experimental" view: > >> https://wpt.fyi/results/?label=experimental > >> > >> This appears much more robust than the Safari 11 data we've collected > >> from Sauce Labs, and we can see a massive improvement between Safari > >> 11 and 12: > >> https://wpt.fyi/results/?sha=ee2e69bfb1&product=safari-11.1&product=safari-12.0&diff > >> > >> This lumps together infrastructure improvements as well as Safari > >> 11->12 improvements, but improvements in service-workers/ [3] stands > >> out, as well as in webdriver/, referrer-policy/, css/css-align/, and > >> others. (The effect of moving away from Sauce is mainly less > >> timeouts.) > >> > >> Also very interesting is to compare Safari 12 stable to TP: > >> https://wpt.fyi/results/?sha=ee2e69bfb1&product=safari-12.0&product=safari-12.1&diff > >> > >> One can tell that work is going in canvas-related things, > >> web-animations/, css/css-logical/ and more! \o/ > >> > >> I hope you'll all find these results valuable, and please report bugs > >> or feature requests here: > >> https://github.com/web-platform-tests/wpt.fyi/issues > >> > >> P.S. We're also trying to use use these diff views to spot > >> regressions. It's a bit hard to use, [4] but a fix in in progress [5] > >> and I might check back here when that works. I'll append to the end of > >> this email a non-exhaustive list of possible regressions already > >> possible to spot. > >> > >> [1] https://github.com/web-platform-tests/results-collection/issues/604 > >> [2] https://wpt.fyi/test-runs?labels=safari,experimental > >> [3] > >> https://wpt.fyi/results/service-workers?sha=ee2e69bfb1&product=safari-11.1&product=safari-12.0&diff=true > >> [4] https://github.com/web-platform-tests/wpt.fyi/issues/411 > >> [5] https://github.com/web-platform-tests/wpt.fyi/pull/609 > >> > >> P.P.S. Possible regressions in Safari TP: > >> https://wpt.fyi/results/css/vendor-imports/mozilla/mozilla-central-reftests/shapes1?sha=ee2e69bfb1&product=safari-12.0&product=safari-12.1 > >> https://wpt.fyi/results/service-workers/service-worker/extendable-event-async-waituntil.https.html?sha=ee2e69bfb1&product=safari-12.0&product=safari-12.1 > >> https://wpt.fyi/results/service-workers/service-worker/skip-waiting-installed.https.html?sha=ee2e69bfb1&product=safari-12.0&product=safari-12.1 > >> _______________________________________________ > >> webkit-dev mailing list > >> webkit-dev@lists.webkit.org > >> https://lists.webkit.org/mailman/listinfo/webkit-dev _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
