Re: [webkit-dev] Position on emerging standard: video.requestAnimationFrame()

[Philip Jägenstedt]

Hi Simon,

Naming is hard as usual and was discussed on
https://github.com/w3ctag/design-reviews/issues/429, where you've already
commented.

Is there a pair of names that you think would work better
here? onFrameAvailable() would IMHO be quite unidiomatic, the web platform
doesn't have any other onFoo() methods, and what would the "cancel" variant
be called?

Can you file an issue in https://github.com/WICG/video-raf/issues if you
see a good alternative?

Also curious if +Eric Carlson  has any feedback on
this?

On Wed, Jan 22, 2020 at 10:49 PM Simon Fraser 
wrote:

>
> On Jan 21, 2020, at 5:27 PM, Thomas Guilbert  wrote:
>
> The idea was to reuse an API name that developers are already
> familiar with, in a similar context. The name is also being used in
> XRSession (
> https://developer.mozilla.org/en-US/docs/Web/API/XRSession/requestAnimationFrame),
> and in OffscreenCanvas (or technically DedicatedWorkerGlobalScope). The 
> AnimationFrameProvider
> mixin
> 
>  could
> also be updated so HTMLVideoElement can extend it.
>
> Yes, this isn't formally spec'ed out, but it will be. For now, they are
> added to the task queue and run like any other task. So, going off the spec
> you linked, I think this would be "5) Perform oldestTask's step"  and not
> "10) Rendering: [...] 11. Foreach document run animation frame callbacks
> for that Document".
>
>
> I would expect something that's called "requestAnimationFrame" to only
> fire in the "update the rendering" steps; requestAnimationFrame is a
> "before rendering" callback. So firing a callback with the same name at
> other times seems like it will lead to author confusion.
>
> The author's expectation should be that any content/style changes they
> make inside a requestAnimationFrame callback will appear on-screen in the
> same frame as other changes in the same event loop cycle, and that
> requestAnimationFrame won't be called more often than is necessary to
> update the screen at the appropriate frame rate.
>
> Simon
>
>
> On Tue, Jan 21, 2020 at 1:01 PM Simon Fraser 
> wrote:
>
>> On Jan 21, 2020, at 12:37 PM, Thomas Guilbert 
>> wrote:
>>
>> Hello,
>>
>> I'm reaching out to see if webkit would like to weigh in on the following 
>> proposal:https://discourse.wicg.io/t/proposal-video-requestanimationframe/3691
>>
>> The HTMLVideoElement.requestAnimationFrame() API allows web developers to be 
>> notified when a video frame has been presented for composition, and provides 
>> metadata for that frame.
>>
>> If you want to try it out, a prototype is available in Chromium Dev,
>> behind the enable-experimental-web-platform-features flag.
>>
>>
>> This is not official feedback, but I have some issues with the proposal.
>>
>> First, the name is confusing. It sounds like you're requesting a frame
>> from the video, but it's really a "frame available" callback. Why not call
>> it onFrameAvailable()?
>>
>> Second, its interaction with normal requestAnimationFrame() and the HTML
>> event loop needs to be better defined. Where in in the
>> https://html.spec.whatwg.org/multipage/webappapis.html#event-loop-processing-model
>>  do
>> these callbacks fire?
>>
>> Simon
>>
>>
>>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Safari Tech Preview 96 available on wpt.fyi!

[Philip Jägenstedt]

On Tue, Dec 3, 2019 at 3:05 AM Jiewen Tan  wrote:

> Hi Maciej,
>
> On Dec 2, 2019, at 4:10 PM, Maciej Stachowiak  wrote:
>
>
> There’s a number of mysterious timeouts with 96. Not sure if flakiness or
> real?
>
> The new WebCrypto failures are surprising, but likely real and should be
> investigated:
> https://wpt.fyi/results/WebCryptoAPI/wrapKey_unwrapKey?diff=ADC=is%3Adifferent_id=347530011_id=381930013
> 
>
>
> I believe these tests are flaky. I have made a PR to improve it a while
> ago. I should probably get those improvement landed sometime.
> https://github.com/web-platform-tests/wpt/pull/6102
>

That sounds very doable. I added you to the reviewers team to make this
easier and commented here:
https://github.com/web-platform-tests/wpt/pull/6102#issuecomment-561115470
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] WPT at the WebKit Contributors Meeting

[Philip Jägenstedt]

Thanks Ryosuke, this was definitely a worthwhile trip for me! I was
especially encouraged to see that the focus on engine-specific tests
resonated and in fact that there was already work in motion to look
into it.

While smoother import/export wasn't a big topic and not seen as urgent
by anyone on the WebKit side, I think the extra friction would soon
become evident when trying to address WebKit-specific failures at
scale. Pablo at Igalia is look into the options for this now, and
feedback from the wider WebKit community would be great:
https://github.com/web-platform-tests/wpt-pr-bot/issues/100#issuecomment-553558439


On Wed, Nov 13, 2019 at 11:22 PM Ryosuke Niwa  wrote:
>
> Hi Stephen,
>
> Thanks for coming to the contributor's meeting and making a
> representation. We enjoyed your presentation & we had a really
> productive discussion. The WebKit team at Apple is putting a lot of
> effort into improving our WPT pass rate going forward as Maciej
> presented so we're looking forward to working with you all :)
>
> Thanks
> - R. Niwa
>
> On Sun, Nov 10, 2019 at 6:21 PM Stephen Mcgruer  wrote:
> >
> > Hi all,
> >
> > Philip (foo...@chromium.org) and I wanted to thank you all for welcoming us 
> > at the WebKit Contributors Meeting last week to talk about Improving 
> > Interop with WPT[0] (web-platform-tests). We felt there was great dialog 
> > about WebKit and WPT, and we hope to keep that going looking forward.
> >
> > We have opened a few issues tracking items that were discussed at the 
> > meeting, including adding WebKitGTK to the UI[1] (fixed!), improving WebKit 
> > --> WPT export[2], and adding download links on the wpt.fyi dashboard[3]. 
> > We also intend to publish our methodology for the 'browser-specific 
> > failures' graph by the end of the year (including making it easy for anyone 
> > to run the same analysis).
> >
> > Please feel free to voice questions, concerns, or ideas via:
> >
> > Issues filed on our github repository for web-platform-tests[4] or the 
> > wpt.fyi dashboard[5], or
> > Email to smcgr...@chromium.org and foo...@chromium.org.
> >
> > We also accept pull requests, of course!
> >
> > Thanks,
> > Stephen & Philip
> >
> > [0]: https://trac.webkit.org/wiki/ImprovingInteropwithWPT
> > [1]: https://github.com/web-platform-tests/wpt.fyi/issues/1576
> > [2]: https://github.com/web-platform-tests/wpt-pr-bot/issues/100
> > [3]: https://github.com/web-platform-tests/wpt.fyi/issues/1480
> > [4]: https://github.com/web-platform-tests/wpt/issues
> > [5]: https://github.com/web-platform-tests/wpt.fyi/issues
> > ___
> > webkit-dev mailing list
> > webkit-dev@lists.webkit.org
> > https://lists.webkit.org/mailman/listinfo/webkit-dev
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Remove SVGTests.hasExtension?

[Philip Jägenstedt]

FWIW, here's the original intent to remove it from Chromium:
https://groups.google.com/a/chromium.org/d/msg/blink-dev/Ae_lmov16_o/Wa5XhHFoAAAJ

We had use counters, and I wrote "Usage is zero for all of them."

When concerned about usage in the wild, the tool we usually use for
Chromium is httparchive, currently containing ~460k pages. I just did a
quick search and found 230 matches for ".hasExtension(". But most are
".hasExtension()" and obviously not use of this API. A search for the
regexp r'\.hasExtension\([^)]' gives only 34 matches, listed here:
https://gist.github.com/foolip/5081f13794baafcd400d8630778dad86

A larger CSV with the bodies included:
https://drive.google.com/file/d/1neFKBtmyjOLaRb_sppP3cmMeFEXfiGqr/view?usp=sharing

Skimming that, I can't find any real usage of SVG's hasExtension, it seems
to be mostly utility code around file extensions.

Ryosuke, does that address the concern of existing content, or is there
anything else you'd check?

Note that in the case of Range's detach, I don't think anyone made an
attempt to remove it:
https://www.w3.org/Bugs/Public/show_bug.cgi?id=14591

There are 865,911 resources (more than there are pages) matching
".detach()" in httparchive, probably also mostly not real usage, but still
putting it in a very different situation than hasExtension :)

HTH

On Sat, Jun 8, 2019 at 12:42 AM Ryosuke Niwa  wrote:

> The concern here isn't with the future versions of Edge or Firefox but
> rather with the content that may be relying on this feature even
> unintentionally.
> Removing a JS method or object is very high risk because that could result
> in an immediate JS exception and break a key functionality of a website.
>
> This is why HTML and DOM standards has a bunch of methods that don't do
> anything like Range's detach:
> https://dom.spec.whatwg.org/#dom-range-detach
>
> - R. Niwa
>
> On Fri, Jun 7, 2019 at 11:54 AM Frédéric Wang  wrote:
>
>> Patch for Firefox was uploaded 4 years ago and they announced plan for
>> removal at that time. We pinged Mozilla some weeks ago about it, they
>> landed the patch and announced it:
>>
>> https://bugzilla.mozilla.org/show_bug.cgi?id=1133175
>>
>> https://www.fxsitecompat.com/en-CA/docs/2019/hasextension-has-been-removed-from-some-svg-interfaces/
>>
>> https://www.fxsitecompat.com/en-CA/docs/2015/several-properties-will-be-removed-from-svgsvgelement/
>>
>> I didn't check Edge, but Microsoft is migrating to Chromium so that's not
>> going to be there in future versions.
>>
>> On 07/06/2019 20:40, Ryosuke Niwa wrote:
>>
>> Does Edge support it? When did Firefox remove this feature?
>>
>> On Fri, Jun 7, 2019 at 1:55 AM Frédéric Wang  wrote:
>>
>>> Hi,
>>>
>>> 4 years ago, the SVG WG resolved to remove SVGTests.hasExtension due to 
>>> lack of use and being a poor API.
>>> It was removed from Chromium at that time and has been removed from Firefox 
>>> recently.
>>>
>>> I'm not sure whether the WebKit community is willing to keep that feature. 
>>> Anyway I opened a bug entry to track possible removal:
>>> https://bugs.webkit.org/show_bug.cgi?id=198652
>>>
>>> --
>>> Frédéric Wang
>>>
>>> ___
>>> webkit-dev mailing list
>>> webkit-dev@lists.webkit.org
>>> https://lists.webkit.org/mailman/listinfo/webkit-dev
>>>
>> --
>> - R. Niwa
>>
>>
>> --
>> Frédéric Wang
>>
>> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Filtering results on wpt.fyi, Safari-specific failures

[Philip Jägenstedt]

On Fri, Mar 29, 2019 at 6:16 PM Robert Ma  wrote:

> On Mon, Feb 25, 2019 at 8:49 AM Philip Jägenstedt 
> wrote:
>
>> I'd like to point out right away that diagnosing reftest failures is
>> currently cumbersome because we don't store the screenshots. This is
>> also a work in progress:
>>
>> https://docs.google.com/document/d/1IhZa4mrjK1msUMhtamKwKJ_HhXD-nqh_4-BcPWM6soQ/edit?usp=sharing
>>
>> Until that has launched, I would recommend ignoring reftest failures
>> if the cause of failure isn't obvious.
>>
>>
>
> Great news! Reftest screenshots are now available on wpt.fyi. No more
> guesswork for why a reftest fails!
>
> For example, this
> <https://wpt.fyi/results/css/css-flexbox/flex-wrap-002.html?label=master=experimental=chrome%5Btaskcluster%5D=firefox%5Btaskcluster%5D=safari%5Bazure%5D=%28chrome%3Apass%7Cchrome%3Aok%29+%28firefox%3Apass%7Cfirefox%3Aok%29+%28safari%3A%21pass%26safari%3A%21ok%29>
> is one of the Safari-only reftest failures you can find using the search
> link posted earlier. Now you can click the "compare" button (you might need
> to force-reload the page to see it) to view the screenshots. This example
> looks like a genuine failure, while some others are probably caused by font
> antialiasing/kerning (they should most likely use the Ahem font instead).
>
> We are also working on another feature to triage the failures
> <https://docs.google.com/document/d/1oWYVkc2ztANCGUxwNVTQHlWV32zq6Ifq9jkkbYNbSAg/edit>
> (e.g. to mark a test as a genuine failure and link it to bug trackers, or
> as flaky/broken). Stay tuned!
>

The screenshots can also come in handy when comparing Safari stable to
Technology Preview:
https://wpt.fyi/results/?diff=ADC=seq%28status%3Apass+status%3Afail%29_id=5130810281689088_id=5197532699295744

/css/css-contain/contain-layout-baseline-003.html is one reftest that
appears to have regressed in Technology Preview, and one can see the
failure here:
https://wpt.fyi/analyzer?screenshot=sha1%3A66e5479ec5db9b860338e89803b563f7e99510f6=sha1%3A385fc160998db876af7fce0e6a9fbf8ad06b4a45
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Filtering results on wpt.fyi, Safari-specific failures

[Philip Jägenstedt]

I think I know what's going on there. When drilling down into tests and
subtests, only those matching the filter are shown. Clearing the filter
things look a bit different in the directories you mentioned:
https://wpt.fyi/results/ambient-light?label=master=experimental=chrome%5Btaskcluster%5D=firefox%5Btaskcluster%5D=safari%5Bazure%5D
https://wpt.fyi/results/bluetooth?label=master=experimental=chrome%5Btaskcluster%5D=firefox%5Btaskcluster%5D=safari%5Bazure%5D

In particular for idlharness.js tests some subtests will pass because
they're preconditions for the real tests. There will also be tests that
check that something doesn't work, which will pass even if the feature is
entirely unsupported if "not working" results in the same thing, e.g.
throwing an exception. Sometimes tests can be tweaked to fail if the
feature is unsupported.

Drilling down into a directory somewhat at random and clearing filters, it
does look like this is legit:
https://wpt.fyi/results/fetch/api/cors?label=master=experimental=chrome%5Btaskcluster%5D=firefox%5Btaskcluster%5D=safari%5Bazure%5D


On Mon, Feb 25, 2019 at 8:31 PM Maciej Stachowiak  wrote:

>
> Neat.
>
> I see some obvious areas for focus, where Safari fails lots of tests that
> the other browser don’t.
>
> For context, I tried looking at this view, which shows all tests that
> Safari and Firefox pass with Safari results regardless of result:
>
> https://wpt.fyi/results/?label=master=experimental=chrome%5Btaskcluster%5D=firefox%5Btaskcluster%5D=safari%5Bazure%5D=%28chrome%3Apass%7Cchrome%3Aok%29+%28firefox%3Apass%7Cfirefox%3Aok%29
> <https://wpt.fyi/results/?label=master=experimental=chrome[taskcluster]=firefox[taskcluster]=safari[azure]=(chrome:pass%7Cchrome:ok)+(firefox:pass%7Cfirefox:ok)>
>
> I noticed some puzzling results there: Safari passes all the ambient-light
> and bluetooth tests that Chrome and Firefox do, despite not supporting
> these standards at all. (For that matter I’m not sure Firefox supports
> these specs either.) Not sure if harness problem, or dubious tests that
> don’t actually test the standard.
>
> Regards,
> Maciej
>
> On Feb 25, 2019, at 5:48 AM, Philip Jägenstedt 
> wrote:
>
> I'd like to point out right away that diagnosing reftest failures is
> currently cumbersome because we don't store the screenshots. This is
> also a work in progress:
>
> https://docs.google.com/document/d/1IhZa4mrjK1msUMhtamKwKJ_HhXD-nqh_4-BcPWM6soQ/edit?usp=sharing
>
> Until that has launched, I would recommend ignoring reftest failures
> if the cause of failure isn't obvious.
>
> On Mon, Feb 25, 2019 at 2:30 PM Philip Jägenstedt 
> wrote:
>
>
> Hi all,
>
> Following the improved Safari results last year [1] and the discussion
> that generated, I'm happy to announce that the filtering requested as
> now available in the search box. The full syntax is documented [2] but
> there's also a new insights view [3] with some useful searches.
>
> Especially interesting for this list could be this view, of Chrome
> Dev, Firefox Nightly and Safari Technology Preview, filtered to the
> Safari-specific failures:
>
> https://wpt.fyi/results/?label=master=experimental=chrome%5Btaskcluster%5D=firefox%5Btaskcluster%5D=safari%5Bazure%5D=%28chrome%3Apass%7Cchrome%3Aok%29+%28firefox%3Apass%7Cfirefox%3Aok%29+%28safari%3A%21pass%26safari%3A%21ok%29
>
> Both Google and Mozilla have efforts [4][5] to reduce the number of
> Chrome/Firefox-specific failures, as this seems like a category of
> problems which especially valuable, where changing just one browser
> can remove a pain point for web developers.
>
> No doubt some failures are spurious, but hopefully there is value to
> be found by looking into where the largest numbers of failures appear
> to be. If something seems to be wrong with the search/filtering,
> please file an issue for us! [6]
>
> Credit to Mark Dittmer and Luke Bjerring who owned this project.
>
> P.S. We are also working on triage metadata for wpt.fyi, to make it
> possible to burn down a list of failures like this and not later have
> to re-triage to find the new failures. [7]
>
> [1] https://lists.webkit.org/pipermail/webkit-dev/2018-October/030209.html
> [2]
> https://github.com/web-platform-tests/wpt.fyi/blob/master/api/query/README.md
> [3] https://staging.wpt.fyi/insights
> [4] https://bugs.chromium.org/p/chromium/issues/detail?id=896242
> [5] https://bugzilla.mozilla.org/show_bug.cgi?id=1498357
> [6]
> https://github.com/web-platform-tests/wpt.fyi/issues/new?title=Structured+Queries+issue=web-platform-tests/wpt.fyi/8=bug=search.md
> [7]
> https://docs.google.com/document/d/1oWYVkc2ztANCGUxwNVTQHlWV32zq6Ifq9jkkbYNbSAg/edit?usp=sharing
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
>
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Filtering results on wpt.fyi, Safari-specific failures

[Philip Jägenstedt]

I'd like to point out right away that diagnosing reftest failures is
currently cumbersome because we don't store the screenshots. This is
also a work in progress:
https://docs.google.com/document/d/1IhZa4mrjK1msUMhtamKwKJ_HhXD-nqh_4-BcPWM6soQ/edit?usp=sharing

Until that has launched, I would recommend ignoring reftest failures
if the cause of failure isn't obvious.

On Mon, Feb 25, 2019 at 2:30 PM Philip Jägenstedt  wrote:
>
> Hi all,
>
> Following the improved Safari results last year [1] and the discussion
> that generated, I'm happy to announce that the filtering requested as
> now available in the search box. The full syntax is documented [2] but
> there's also a new insights view [3] with some useful searches.
>
> Especially interesting for this list could be this view, of Chrome
> Dev, Firefox Nightly and Safari Technology Preview, filtered to the
> Safari-specific failures:
> https://wpt.fyi/results/?label=master=experimental=chrome%5Btaskcluster%5D=firefox%5Btaskcluster%5D=safari%5Bazure%5D=%28chrome%3Apass%7Cchrome%3Aok%29+%28firefox%3Apass%7Cfirefox%3Aok%29+%28safari%3A%21pass%26safari%3A%21ok%29
>
> Both Google and Mozilla have efforts [4][5] to reduce the number of
> Chrome/Firefox-specific failures, as this seems like a category of
> problems which especially valuable, where changing just one browser
> can remove a pain point for web developers.
>
> No doubt some failures are spurious, but hopefully there is value to
> be found by looking into where the largest numbers of failures appear
> to be. If something seems to be wrong with the search/filtering,
> please file an issue for us! [6]
>
> Credit to Mark Dittmer and Luke Bjerring who owned this project.
>
> P.S. We are also working on triage metadata for wpt.fyi, to make it
> possible to burn down a list of failures like this and not later have
> to re-triage to find the new failures. [7]
>
> [1] https://lists.webkit.org/pipermail/webkit-dev/2018-October/030209.html
> [2] 
> https://github.com/web-platform-tests/wpt.fyi/blob/master/api/query/README.md
> [3] https://staging.wpt.fyi/insights
> [4] https://bugs.chromium.org/p/chromium/issues/detail?id=896242
> [5] https://bugzilla.mozilla.org/show_bug.cgi?id=1498357
> [6] 
> https://github.com/web-platform-tests/wpt.fyi/issues/new?title=Structured+Queries+issue=web-platform-tests/wpt.fyi/8=bug=search.md
> [7] 
> https://docs.google.com/document/d/1oWYVkc2ztANCGUxwNVTQHlWV32zq6Ifq9jkkbYNbSAg/edit?usp=sharing
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

[webkit-dev] Filtering results on wpt.fyi, Safari-specific failures

[Philip Jägenstedt]

Hi all,

Following the improved Safari results last year [1] and the discussion
that generated, I'm happy to announce that the filtering requested as
now available in the search box. The full syntax is documented [2] but
there's also a new insights view [3] with some useful searches.

Especially interesting for this list could be this view, of Chrome
Dev, Firefox Nightly and Safari Technology Preview, filtered to the
Safari-specific failures:
https://wpt.fyi/results/?label=master=experimental=chrome%5Btaskcluster%5D=firefox%5Btaskcluster%5D=safari%5Bazure%5D=%28chrome%3Apass%7Cchrome%3Aok%29+%28firefox%3Apass%7Cfirefox%3Aok%29+%28safari%3A%21pass%26safari%3A%21ok%29

Both Google and Mozilla have efforts [4][5] to reduce the number of
Chrome/Firefox-specific failures, as this seems like a category of
problems which especially valuable, where changing just one browser
can remove a pain point for web developers.

No doubt some failures are spurious, but hopefully there is value to
be found by looking into where the largest numbers of failures appear
to be. If something seems to be wrong with the search/filtering,
please file an issue for us! [6]

Credit to Mark Dittmer and Luke Bjerring who owned this project.

P.S. We are also working on triage metadata for wpt.fyi, to make it
possible to burn down a list of failures like this and not later have
to re-triage to find the new failures. [7]

[1] https://lists.webkit.org/pipermail/webkit-dev/2018-October/030209.html
[2] 
https://github.com/web-platform-tests/wpt.fyi/blob/master/api/query/README.md
[3] https://staging.wpt.fyi/insights
[4] https://bugs.chromium.org/p/chromium/issues/detail?id=896242
[5] https://bugzilla.mozilla.org/show_bug.cgi?id=1498357
[6] 
https://github.com/web-platform-tests/wpt.fyi/issues/new?title=Structured+Queries+issue=web-platform-tests/wpt.fyi/8=bug=search.md
[7] 
https://docs.google.com/document/d/1oWYVkc2ztANCGUxwNVTQHlWV32zq6Ifq9jkkbYNbSAg/edit?usp=sharing
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Huge improvement in Safari results on wpt.fyi

[Philip Jägenstedt]

Hi Dean,

On the run of Safari that was used for this report, the infrastructure
test for ahem was actually passing:
https://wpt.fyi/results/infrastructure/assumptions?sha=67152fdecd=chrome[stable]=edge[stable]=firefox[stable]=safari[experimental]

Are you sure that Ahem is the explanation for the failures, do you
have a test that you think is actually passing and the wpt.fyi results
are wrong? Clearly, having screenshots would make it easier to
understand a situation like this, and it's something we've discussed a
bit today:
https://github.com/web-platform-tests/wpt.fyi/issues/57
On Fri, Oct 12, 2018 at 3:01 AM Dean Jackson  wrote:
>
> It turns out that many (most?) of the CSS failures are because we no longer 
> expose user-installed fonts, e.g. Ahem.
>
> Options:
>
> - update lots of tests to load Ahem via @font-face (yuck)
> - allow Ahem to be used if installed (weird to special case one font, but 
> probably ok)
>
> Dean
>
> > On 12 Oct 2018, at 03:26, Philip Jägenstedt  wrote:
> >
> > Alright, I've written a one-off script [1] to find the Safari-only
> > failures, and here's the output:
> > https://gist.github.com/foolip/4d410ce79416bcdce71feb212159a02e
> >
> > Barring bugs, each of linked tests or one of its subtests should be
> > failing in Safari Technology Preview and passing in stable versions of
> > Chrome, Edge and Firefox.
> >
> > Numerically, most of the failures are in css (622), encoding (135) and
> > html (60). With css, it's mostly css/CSS2.
> >
> > I hope looking through this may be of use to you!
> >
> > [1] https://github.com/foolip/ad-hoc-wpt-results-analysis
> >
> > On Mon, Oct 8, 2018 at 11:50 PM Philip Jägenstedt  
> > wrote:
> >>
> >> That filtering capability unfortunately does not yet exist on wpt.fyi
> >> but it's a high priority and actively being worked on:
> >> https://github.com/web-platform-tests/wpt.fyi/issues/201
> >>
> >> FWIW, I suspect that these purposes, comparing to the stable versions
> >> of all *other* browsers might be the most useful:
> >> https://wpt.fyi/results/?product=chrome%5Bstable%5D=edge%5Bstable%5D=firefox%5Bstable%5D=safari%5Bexperimental%5D
> >>
> >> Again, no way to filter on wpt.fyi, but I'll see if I can download the
> >> full results and write a quick script.
> >>
> >> On Thu, Oct 4, 2018 at 11:49 PM Ryosuke Niwa  wrote:
> >>>
> >>> Thanks for the intriguing data, Philip.
> >>>
> >>> Is there a way to get a list of tests where all other browsers pass but 
> >>> Safari / WebKit fail?
> >>>
> >>> That would allow us to quickly identify the set of tests we can fix to 
> >>> improve the interoperability across browsers right away.
> >>>
> >>> - R. Niwa
> >>>
> >>> On Tue, Oct 2, 2018 at 3:45 AM Philip Jägenstedt  
> >>> wrote:
> >>>>
> >>>> Hi WebKittens,
> >>>>
> >>>> Fresh off the bots, I'm excited to report more robust Safari results,
> >>>> and that Safari WPT pass rates are clearly improving! Thanks to the
> >>>> hard work of Mike Pennisi [1] we now have the first Safari 12 results:
> >>>> https://wpt.fyi/results/?sha=ee2e69bfb1=safari-12.0
> >>>>
> >>>> This uses the same setup as for Safari Technology Preview, which has
> >>>> been running for a while [2] and are the results you see on the
> >>>> "experimental" view:
> >>>> https://wpt.fyi/results/?label=experimental
> >>>>
> >>>> This appears much more robust than the Safari 11 data we've collected
> >>>> from Sauce Labs, and we can see a massive improvement between Safari
> >>>> 11 and 12:
> >>>> https://wpt.fyi/results/?sha=ee2e69bfb1=safari-11.1=safari-12.0
> >>>>
> >>>> This lumps together infrastructure improvements as well as Safari
> >>>> 11->12 improvements, but improvements in service-workers/ [3] stands
> >>>> out, as well as in webdriver/, referrer-policy/, css/css-align/, and
> >>>> others. (The effect of moving away from Sauce is mainly less
> >>>> timeouts.)
> >>>>
> >>>> Also very interesting is to compare Safari 12 stable to TP:
> >>>> https://wpt.fyi/results/?sha=ee2e69bfb1=safari-12.0=safari-12.1
> >>>>
> >>>> One can tell that work is going in canvas-related things,
> >>>> web-animations/, css/css-logical/ and more

Re: [webkit-dev] Huge improvement in Safari results on wpt.fyi

[Philip Jägenstedt]

On Fri, Oct 12, 2018 at 4:07 PM Geoffrey Sneddon  wrote:
>
> On Fri, Oct 12, 2018 at 4:23 AM Emilio Cobos Álvarez  wrote:
> >
> > On 10/12/18 3:59 AM, Geoffrey Garen wrote:
> > > Honest question: What’s gross about using @font-face?
> > >
> > > It would be lots of test edits. That’s a bummer.
> > >
> > > But maybe it’s clearer for the tests to specify the font they want to 
> > > use. It makes the test self-describing, eliminating the requirement that 
> > > the user take a step outside the test to get the right result.
>
> See https://github.com/web-platform-tests/wpt/issues/9105 about this.
>
> > Note that there's also the opposite opinion of loading a web font
> > potentially hiding bugs:
> >
> >https://lists.w3.org/Archives/Public/www-style/2017Jan/0053.html
> >
> > Though I don't have such a strong opinion myself, I think @font-face is
> > a fine solution for that problem (and other people seemed to be ok with
> > that as well, looking at how that thread continues...).
>
> I don't have a strong opinion here, but: a) it certainly seems
> annoying to flush layout and avoid triggering any layout invalidation
> bugs; b) we have plenty of other (manual ) tests for the font
> matching algorithm (and parts of that obviously do need to use system
> installed fonts).

I don't think we should change a bunch of tests, it's useful to be
able to depend on *some* system font existing across the board, and
Ahem is it. We already need root access in our CI systems because of
/etc/hosts, so just putting Ahem in /Library/Fonts as part of the
setup is fine.

> As an aside: when did user installed fonts stop being allowed by
> default? r226172 states nothing is using the SPI yet (though did it
> already default to No? in which case it has been disallowed by default
> since r225641). wpt.fyi seems to have Ahem being installed okay for
> STP but not stable, based on infrastructure/assumptions/ahem.html, and
> all that does it copy the font to ~/Library/Fonts, which confuses me!

I'd also like to know when this change happened, because in
https://github.com/foolip/wpt/pull/5 I had to work around it for Azure
Pipelines, which has macOS 10.13.6, while all the other CI systems I
tried worked with the code as-is. They are all running the same
version of STP. (This PR is still just me experimenting, but the goal
is to get Safari coverage for PRs pretty soon.)
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Huge improvement in Safari results on wpt.fyi

[Philip Jägenstedt]

That filtering capability unfortunately does not yet exist on wpt.fyi
but it's a high priority and actively being worked on:
https://github.com/web-platform-tests/wpt.fyi/issues/201

FWIW, I suspect that these purposes, comparing to the stable versions
of all *other* browsers might be the most useful:
https://wpt.fyi/results/?product=chrome%5Bstable%5D=edge%5Bstable%5D=firefox%5Bstable%5D=safari%5Bexperimental%5D

Again, no way to filter on wpt.fyi, but I'll see if I can download the
full results and write a quick script.

On Thu, Oct 4, 2018 at 11:49 PM Ryosuke Niwa  wrote:
>
> Thanks for the intriguing data, Philip.
>
> Is there a way to get a list of tests where all other browsers pass but 
> Safari / WebKit fail?
>
> That would allow us to quickly identify the set of tests we can fix to 
> improve the interoperability across browsers right away.
>
> - R. Niwa
>
> On Tue, Oct 2, 2018 at 3:45 AM Philip Jägenstedt  wrote:
>>
>> Hi WebKittens,
>>
>> Fresh off the bots, I'm excited to report more robust Safari results,
>> and that Safari WPT pass rates are clearly improving! Thanks to the
>> hard work of Mike Pennisi [1] we now have the first Safari 12 results:
>> https://wpt.fyi/results/?sha=ee2e69bfb1=safari-12.0
>>
>> This uses the same setup as for Safari Technology Preview, which has
>> been running for a while [2] and are the results you see on the
>> "experimental" view:
>> https://wpt.fyi/results/?label=experimental
>>
>> This appears much more robust than the Safari 11 data we've collected
>> from Sauce Labs, and we can see a massive improvement between Safari
>> 11 and 12:
>> https://wpt.fyi/results/?sha=ee2e69bfb1=safari-11.1=safari-12.0
>>
>> This lumps together infrastructure improvements as well as Safari
>> 11->12 improvements, but improvements in service-workers/ [3] stands
>> out, as well as in webdriver/, referrer-policy/, css/css-align/, and
>> others. (The effect of moving away from Sauce is mainly less
>> timeouts.)
>>
>> Also very interesting is to compare Safari 12 stable to TP:
>> https://wpt.fyi/results/?sha=ee2e69bfb1=safari-12.0=safari-12.1
>>
>> One can tell that work is going in canvas-related things,
>> web-animations/, css/css-logical/ and more! \o/
>>
>> I hope you'll all find these results valuable, and please report bugs
>> or feature requests here:
>> https://github.com/web-platform-tests/wpt.fyi/issues
>>
>> P.S. We're also trying to use use these diff views to spot
>> regressions. It's a bit hard to use, [4] but a fix in in progress [5]
>> and I might check back here when that works. I'll append to the end of
>> this email a non-exhaustive list of possible regressions already
>> possible to spot.
>>
>> [1] https://github.com/web-platform-tests/results-collection/issues/604
>> [2] https://wpt.fyi/test-runs?labels=safari,experimental
>> [3] 
>> https://wpt.fyi/results/service-workers?sha=ee2e69bfb1=safari-11.1=safari-12.0=true
>> [4] https://github.com/web-platform-tests/wpt.fyi/issues/411
>> [5] https://github.com/web-platform-tests/wpt.fyi/pull/609
>>
>> P.P.S. Possible regressions in Safari TP:
>> https://wpt.fyi/results/css/vendor-imports/mozilla/mozilla-central-reftests/shapes1?sha=ee2e69bfb1=safari-12.0=safari-12.1
>> https://wpt.fyi/results/service-workers/service-worker/extendable-event-async-waituntil.https.html?sha=ee2e69bfb1=safari-12.0=safari-12.1
>> https://wpt.fyi/results/service-workers/service-worker/skip-waiting-installed.https.html?sha=ee2e69bfb1=safari-12.0=safari-12.1
>> ___
>> webkit-dev mailing list
>> webkit-dev@lists.webkit.org
>> https://lists.webkit.org/mailman/listinfo/webkit-dev
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

[webkit-dev] Huge improvement in Safari results on wpt.fyi

[Philip Jägenstedt]

Hi WebKittens,

Fresh off the bots, I'm excited to report more robust Safari results,
and that Safari WPT pass rates are clearly improving! Thanks to the
hard work of Mike Pennisi [1] we now have the first Safari 12 results:
https://wpt.fyi/results/?sha=ee2e69bfb1=safari-12.0

This uses the same setup as for Safari Technology Preview, which has
been running for a while [2] and are the results you see on the
"experimental" view:
https://wpt.fyi/results/?label=experimental

This appears much more robust than the Safari 11 data we've collected
from Sauce Labs, and we can see a massive improvement between Safari
11 and 12:
https://wpt.fyi/results/?sha=ee2e69bfb1=safari-11.1=safari-12.0

This lumps together infrastructure improvements as well as Safari
11->12 improvements, but improvements in service-workers/ [3] stands
out, as well as in webdriver/, referrer-policy/, css/css-align/, and
others. (The effect of moving away from Sauce is mainly less
timeouts.)

Also very interesting is to compare Safari 12 stable to TP:
https://wpt.fyi/results/?sha=ee2e69bfb1=safari-12.0=safari-12.1

One can tell that work is going in canvas-related things,
web-animations/, css/css-logical/ and more! \o/

I hope you'll all find these results valuable, and please report bugs
or feature requests here:
https://github.com/web-platform-tests/wpt.fyi/issues

P.S. We're also trying to use use these diff views to spot
regressions. It's a bit hard to use, [4] but a fix in in progress [5]
and I might check back here when that works. I'll append to the end of
this email a non-exhaustive list of possible regressions already
possible to spot.

[1] https://github.com/web-platform-tests/results-collection/issues/604
[2] https://wpt.fyi/test-runs?labels=safari,experimental
[3] 
https://wpt.fyi/results/service-workers?sha=ee2e69bfb1=safari-11.1=safari-12.0=true
[4] https://github.com/web-platform-tests/wpt.fyi/issues/411
[5] https://github.com/web-platform-tests/wpt.fyi/pull/609

P.P.S. Possible regressions in Safari TP:
https://wpt.fyi/results/css/vendor-imports/mozilla/mozilla-central-reftests/shapes1?sha=ee2e69bfb1=safari-12.0=safari-12.1
https://wpt.fyi/results/service-workers/service-worker/extendable-event-async-waituntil.https.html?sha=ee2e69bfb1=safari-12.0=safari-12.1
https://wpt.fyi/results/service-workers/service-worker/skip-waiting-installed.https.html?sha=ee2e69bfb1=safari-12.0=safari-12.1
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Standard process for exporting new WPT tests?

[Philip Jägenstedt]

On Fri, May 25, 2018 at 6:09 PM youenn fablet  wrote:

>
> However, one thing I really like about your order is that it makes it more
>> straightforward to make WPT repo failures block browser repo merging. In
>> the WPT Travis job, we already detect flakiness for Chrome and Firefox and
>> I think we should also detect harness errors, and eventually also flag when
>> a test goes from passing everywhere to failing everywhere, which is
>> probably a mistake. We haven't really figured out how to make that block
>> Chromium's CQ yet, and maybe flipping the order would do it.
>>
>
> Exactly, WPT infrastructure has some really nice sanity checks that we do
> not have in WebKit and it sounds really great to benefit from it.
>
> That said, this will put some burden on WPT infrastructure like being fast
> enough to not slow down our own process.
>

If you run into trouble here, please let us all know so that things that
are holding back WebKit contributions don't linger and are forgotten.


> We should also probably run as much as possible the WPT sanity checks
> locally.
> We are now checking WPT linter as you suggested a while back.
> Maybe there are some other WPT checks that could be run in the WebKit
> context.
>

The one that comes to mind is stability checking. We're now running each
new/modified test 10 times in Chrome and Firefox on Travis, and it does
catch flaky tests. We have no straightforward way of doing the same for
Safari, which means no stability checking for Safari. Perhaps we should be
testing with the GTK port, or what is the Linux port that you run on WebKit
bots and want to stay in good shape?
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Standard process for exporting new WPT tests?

[Philip Jägenstedt]

On Fri, May 25, 2018, 00:52 Ryosuke Niwa <rn...@webkit.org> wrote:

>
> On Thu, May 24, 2018 at 3:22 AM, Philip Jägenstedt <foo...@chromium.org>
> wrote:
>
>> On Wed, May 23, 2018, 23:43 youenn fablet <youe...@gmail.com> wrote:
>>
>>> Le mer. 23 mai 2018 à 14:11, Frédéric Wang <fw...@igalia.com> a écrit :
>>>
>>>> On 23/05/2018 22:50, Ryosuke Niwa wrote:
>>>> > As we have preciously discussed, we should NEVER commit new tests into
>>>> > LayoutTests/imported/w3c/web-platform-tests.
>>>>
>>>
>>> Ryosuke, correct me if I am wrong, I think you are pointing out the
>>> following rule:
>>> Changes to LayoutTests/imported/w3c/web-platform-tests tests should land
>>> first in WPT repository, then in WebKit repository.
>>>
>>
>> Oh, that is surprising.
>>
>> https://github.com/w3c/web-platform-tests/pull/10964 is a recent WebKit
>> export, and https://trac.webkit.org/changeset/231788/webkit did modify
>> the test in place. Do you mean that the WPT PR was merged first, or should
>> be in general? Chromium and Gecko do it in the other order, and I'd be
>> interested to understand the trade-offs of flipping the order.
>>
>
> Yes, it's okay for a WebKit commit to merge the change which got merged
> into WPT but it's never okay to first commit the test into WebKit and then
> later upstream it to WPT...
>
> Any process like this where changes end up in WebKit trunk via anything
>> except a full WPT import will mean that divergence is possible.
>>
>
> Precisely to avoid these problems.
>

I'm saying that there's temporary divergence with the WPT-then-WebKit order
as well.

Whichever side is merged first, the other side can fail to merge because
any number of reasons, and something/someone then needs to deal with that.
(See question in reply to Youenn.)

However, one thing I really like about your order is that it makes it more
straightforward to make WPT repo failures block browser repo merging. In
the WPT Travis job, we already detect flakiness for Chrome and Firefox and
I think we should also detect harness errors, and eventually also flag when
a test goes from passing everywhere to failing everywhere, which is
probably a mistake. We haven't really figured out how to make that block
Chromium's CQ yet, and maybe flipping the order would do it.

>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Standard process for exporting new WPT tests?

[Philip Jägenstedt]

On Thu, May 24, 2018, 21:40 youenn fablet  wrote:

>
>
>>> Ryosuke, correct me if I am wrong, I think you are pointing out the
>>> following rule:
>>> Changes to LayoutTests/imported/w3c/web-platform-tests tests should land
>>> first in WPT repository, then in WebKit repository.
>>>
>>
>> Oh, that is surprising.
>>
>> https://github.com/w3c/web-platform-tests/pull/10964 is a recent WebKit
>> export, and https://trac.webkit.org/changeset/231788/webkit did modify
>> the test in place. Do you mean that the WPT PR was merged first, or should
>> be in general? Chromium and Gecko do it in the other order, and I'd be
>> interested to understand the trade-offs of flipping the order.
>>
>
> Yep, WPT PR merged first, then WebKit patch landed.
> It makes sure that whenever we are reimporting tests, we are not loosing
> any WebKit specific change.
> Conflict resolution also happens at a time the patch is being committed in
> WebKit.
> So far, I encountered very few conflicts.
>

Interesting. What happens happens if the WebKit patch then fails to land in
WebKit, perhaps because some bot fails the test, a conflict, or anything
else? Is resolving that a manual affair?

When imports are automated and much more frequent, I take it the upside is
that you never need to reapply still-being-exported patches like we do in
Blink. But what about the above case, when the change hasn't been applied
in WebKit yet? Seems like the importer instead needs to *revert* the
in-flight change?

>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Standard process for exporting new WPT tests?

[Philip Jägenstedt]

On Wed, May 23, 2018, 23:43 youenn fablet  wrote:

> Le mer. 23 mai 2018 à 14:11, Frédéric Wang  a écrit :
>
>> On 23/05/2018 22:50, Ryosuke Niwa wrote:
>> > As we have preciously discussed, we should NEVER commit new tests into
>> > LayoutTests/imported/w3c/web-platform-tests.
>>
>
> Ryosuke, correct me if I am wrong, I think you are pointing out the
> following rule:
> Changes to LayoutTests/imported/w3c/web-platform-tests tests should land
> first in WPT repository, then in WebKit repository.
>

Oh, that is surprising.

https://github.com/w3c/web-platform-tests/pull/10964 is a recent WebKit
export, and https://trac.webkit.org/changeset/231788/webkit did modify the
test in place. Do you mean that the WPT PR was merged first, or should be
in general? Chromium and Gecko do it in the other order, and I'd be
interested to understand the trade-offs of flipping the order.

Any process like this where changes end up in WebKit trunk via anything
except a full WPT import will mean that divergence is possible. In
Chromium, regular imports are the safeguard that will eventually resolve
any issuesinvolves. In practice, much to my surprise, we've really never
had something funny happen due to the temporary divergence the process
involves.

>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] WebKit support in web-platform-tests

[Philip Jägenstedt]

Thanks for your work on this, Žan!

That PR is now up at https://github.com/w3c/web-platform-tests/pull/9666 and
in it I linked to https://github.com/w3c/web-platform-tests/pull/8979,
which is a PR I updated last week to support Safari, including Technology
Review. I wish I'd payed closer attention to webkit-dev to see this thread
earlier :)

Given that safaridriver and webkitdriver aren't the same piece of code, it
makes sense to treat them as separate products in wpt run, and whether some
code is shared is wpt-internal concern.

The reason I tried to get Safari (TP) running is ultimately for wpt.fyi
and, and I don't imagine it'll be useful for WebKit CI.

Brian, I'll ask you on the PR about the WebDriver tests specifically.

On Fri, Feb 23, 2018 at 7:25 PM  wrote:

> Hi,
>
> in case of no additional comments against this, I'll be opening a pull
> request to merge the web-platform-tests changes into that repository in the
> following days. The WebKit changes will follow after some further work.
>
> Cheers,
> Zan
>
> On Mon, Feb 12, 2018, at 4:06 PM, z...@falconsigh.net wrote:
> > Hi,
> >
> > the web-platform-tests repository includes tooling that enables running
> > those tests against a supported browser product. I'd like to propose
> > adding generic WebKit support there.
> >
> >
> > Current changes only assume usage of the WebDriver protocol, and the
> > WebDriver binary accepting the --port flag. Selenium executors are used
> > for test harness and reftests. Same WebDriver implementation can also be
> > tested against the WebDriver tests included in the web-platform-tests
> > directory, presuming the tests are enabled or explicitly specified.
> >
> > Only port-specific bit is the specification of capabilities that are
> > passed to the WebDriver binary, idea being that these capabilities are
> > the same as those supported by the WebDriver implementation.
> >
> > GTK is for now the only port that's supported, and it's leveraging the
> > WebDriver implementation under Source/WebDriver/ in WebKit. WPE will be
> > doing the same. Safari I suppose could use its own WebDriver
> > implementation, or perhaps even a separate product.
> >
> > Here's the current set of changes:
> >
> https://github.com/zdobersek/web-platform-tests/commit/c2ee920876ca6df7c4739feb8a6e03c77dffdb7f
> >
> >
> > The web-platform-tests suite can then be run like this for the GTK port,
> > assuming a tip-of-trunk build:
> >
> > $ /work/web-platform-tests/wpt run --webkit-port=gtk \
> > --webdriver-binary=WebKitBuild/Release/bin/WebKitWebDriver \
> > --binary=WebKitBuild/Release/bin/MiniBrowser \
> > --binary-arg=--automation \
> > --binary-arg=--javascript-can-open-windows-automatically=true \
> > --binary-arg=--enable-xss-auditor=false \
> > webkit /2dcontext
> >
> > This can be further wrapped into a python script and run as part of the
> > continuous integration system. These changes add a run-web-platform-
> > tests script that invokes the web-platform-tests runner tool, also
> > allowing each port to specify what tests to enable and what the expected
> > failures are:
> >
> https://github.com/Igalia/webkit/commit/df1aeeb9476c6dd220067f4fc3c6ad69a8f948ba
> >
> > Only a small subset of tests is enabled there, for prototype purposes.
> > The expected results system could also be improved to avoid each
> > expected failure having to be marked as such in separate .ini files.
> >
> >
> > But foremost, I'd like to have a consensus of sorts about how various
> > WebKit ports should be handled in the web-platform-tests repository, so
> > that the changes there can proceed -- whether it's fine to implement a
> > generic WebKit product, or whether Safari would like to be treated as a
> > separate browser[1], etc.
> >
> >
> > Regards,
> > Zan
> >
> >
> > [1] There's for instance this from a year ago (though not sure about its
> > functionality):
> > https://github.com/w3c/web-platform-tests/tree/wptrunner-safari
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[Philip Jägenstedt]

On Tue, Nov 28, 2017 at 3:37 PM Ryosuke Niwa <rn...@webkit.org> wrote:

> On Mon, Nov 27, 2017 at 5:58 PM, Philip Jägenstedt <foo...@chromium.org>
> wrote:
>
>> (From Nov 24, I used the wrong email, resending for the archives.)
>>
>> On Sat, Nov 18, 2017 at 8:02 AM, youenn fablet <youe...@gmail.com> wrote:
>> > Thanks for taking the time to share this additional information.
>> > I think this is helpful to make progress.
>> > Please see inline for some more comments related to the points you are
>> > bringing to the table.
>> >
>> > Stepping back from the WPT server specific issue, being optimistic and
>> > assuming that we are able to run WPT tests with good enough
>> performances.
>> > Migration potential downsides I heard so far:
>> > - Prioritization of tests to investigate might be harder.
>> > - Sharing tests with other teams might be harder if subresource links
>> are
>> > not relative.
>> > These two seem like solvable issues to me.
>> >
>> > Le ven. 17 nov. 2017 à 10:09, Alexey Proskuryakov <a...@apple.com> a
>> écrit :
>> >>
>> >>
>> >> 17 нояб. 2017 г., в 9:18, youenn fablet <youe...@gmail.com>
>> написал(а):
>> >>
>> >>
>> >> Chris recently noticed that some heavily used files (testharness*) were
>> >> cacheable through Apache but not WPT.
>> >> This is now fixed and should improve WPT performances.
>> >>
>> >>
>> >> This is part of <https://bugs.webkit.org/show_bug.cgi?id=178277>,
>> another
>> >> part is that the server is 10x slower than Apache.
>> >
>> >
>> > Other measurements showed 3x slower, which makes it still worthwhile to
>> > explore.
>> > We need to be cautious here though since optimization is all about
>> chasing
>> > where time is actually spent.
>> >
>> > If we can prove to ourselves that this is an important issue, we should
>> > discuss with the WPT community to see how to fix this issue.
>> > In addition to better caching, other optimization like
>> > https://github.com/w3c/wptserve/pull/86 may bring some additional
>> benefit.
>> >
>> > If we do not find any good solution at WPT server level, we still have
>> the
>> > option to run some tests as file-based.
>> > Ryosuke mentioned the possibility to classify tests at import time by
>> > checking their results when served through WPT server and as file URLs.
>>
>> Thanks for filing https://github.com/w3c/web-platform-tests/issues/8391,
>> Youenn!
>>
>> I've put a placeholder around this in our planning for next quarter.
>> At a minimum, I'd like to investigate this and see how much slower the
>> tests are in Chromium's infra than they would be without wptserve, and
>> where that extra time is spent. But the reasons may not be the same in
>> WebKit, so please don't count on any improvement :)
>>
>> >> I just tested on my MacBook Pro, and WPT tests took 23% of time while
>> >> being only 9% of the total count. Taking in mind that WebKit own tests
>> have
>> >> higher value due to the way we choose what to test (see below), that's
>> not a
>> >> great story in my opinion.
>> >
>> >
>> > These numbers are difficult to interpret.
>> > WPT authoring style is multiple tests in a single file, which is bad for
>> > stability but potentially good for performances.
>> > WebKit usually prefers one file/one test.
>> >
>> > If we want to talk to WPT community about this, we need to provide some
>> more
>> > tangible numbers.
>> > We could try to run a large subset of WPT tests that run the same
>> through
>> > Apache and WPT.
>> > I just did that on a small subset of IDB tests. This seems to show
>> something
>> > like a 25% slowdown.
>> >
>> >> One other thing that we discussed before was the operational
>> complexity of
>> >> running WPT tests. We frequently need to share tests with people who
>> don't
>> >> work on WebKit directly, but have the need to edit and run our tests.
>> >> Inability to drag and drop a local copy into a Safari window is a
>> deterrent
>> >> to addressing problems caught by the tests. I think that the response
>> we got
>> >> was that tests will continue to require a server to run.
>> >
>> >
>> > Tests are available 

Re: [webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[Philip Jägenstedt]

(From Nov 24, I used the wrong email, resending for the archives.)

On Sat, Nov 18, 2017 at 8:02 AM, youenn fablet  wrote:
> Thanks for taking the time to share this additional information.
> I think this is helpful to make progress.
> Please see inline for some more comments related to the points you are
> bringing to the table.
>
> Stepping back from the WPT server specific issue, being optimistic and
> assuming that we are able to run WPT tests with good enough performances.
> Migration potential downsides I heard so far:
> - Prioritization of tests to investigate might be harder.
> - Sharing tests with other teams might be harder if subresource links are
> not relative.
> These two seem like solvable issues to me.
>
> Le ven. 17 nov. 2017 à 10:09, Alexey Proskuryakov  a écrit :
>>
>>
>> 17 нояб. 2017 г., в 9:18, youenn fablet  написал(а):
>>
>>
>> Chris recently noticed that some heavily used files (testharness*) were
>> cacheable through Apache but not WPT.
>> This is now fixed and should improve WPT performances.
>>
>>
>> This is part of , another
>> part is that the server is 10x slower than Apache.
>
>
> Other measurements showed 3x slower, which makes it still worthwhile to
> explore.
> We need to be cautious here though since optimization is all about chasing
> where time is actually spent.
>
> If we can prove to ourselves that this is an important issue, we should
> discuss with the WPT community to see how to fix this issue.
> In addition to better caching, other optimization like
> https://github.com/w3c/wptserve/pull/86 may bring some additional benefit.
>
> If we do not find any good solution at WPT server level, we still have the
> option to run some tests as file-based.
> Ryosuke mentioned the possibility to classify tests at import time by
> checking their results when served through WPT server and as file URLs.

Thanks for filing https://github.com/w3c/web-platform-tests/issues/8391, Youenn!

I've put a placeholder around this in our planning for next quarter.
At a minimum, I'd like to investigate this and see how much slower the
tests are in Chromium's infra than they would be without wptserve, and
where that extra time is spent. But the reasons may not be the same in
WebKit, so please don't count on any improvement :)

>> I just tested on my MacBook Pro, and WPT tests took 23% of time while
>> being only 9% of the total count. Taking in mind that WebKit own tests have
>> higher value due to the way we choose what to test (see below), that's not a
>> great story in my opinion.
>
>
> These numbers are difficult to interpret.
> WPT authoring style is multiple tests in a single file, which is bad for
> stability but potentially good for performances.
> WebKit usually prefers one file/one test.
>
> If we want to talk to WPT community about this, we need to provide some more
> tangible numbers.
> We could try to run a large subset of WPT tests that run the same through
> Apache and WPT.
> I just did that on a small subset of IDB tests. This seems to show something
> like a 25% slowdown.
>
>> One other thing that we discussed before was the operational complexity of
>> running WPT tests. We frequently need to share tests with people who don't
>> work on WebKit directly, but have the need to edit and run our tests.
>> Inability to drag and drop a local copy into a Safari window is a deterrent
>> to addressing problems caught by the tests. I think that the response we got
>> was that tests will continue to require a server to run.
>
>
> Tests are available at https://w3c-test.org which makes it easy to share
> through any tool supporting hyperlinks.
> A webarchive can also be made so that it is easy to share and probably edit
> such tests.
> Tools like jsfiddle are also a great way to create/share/edit tests.
> I received several bug reports on bugzilla using it and this proved to be
> efficient.
>
>>
>> Let me explain why I think that WebKit tests are often more valuable as
>> regression tests than WPT tests are. We add tests as we fix bugs, so we know
>> that the tests are generally for problems that have a high impact on users
>> and developers - that's because someone actually discovered the problem, and
>> someone prioritized it highly enough to fix. We also know that our tests
>> cover code that is error prone, which is why we had bugs in the first place.
>> Of course, anything can be broken, but certain things are less likely to.
>> Compliance tests written for specs are also valuable, but at some point we
>> need to prioritize which tests to investigate and even to run.
>
>
> I don't really see why we should prioritize the tests to run when all of
> them provide clear value to some WebKit members.
> I agree that we need to prioritize tests we investigate. There can be a
> solution inside WPT, like adding WebKit specific metadata so that:
> - WPT contributors would communicate with 

Re: [webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[Philip Jägenstedt]

On Fri, Nov 17, 2017 at 5:23 PM, Maciej Stachowiak  wrote:
>
>
>> On Nov 17, 2017, at 7:53 AM, Frédéric WANG  wrote:
>>
>> On 17/11/2017 16:26, Maciej Stachowiak wrote:
>>> WebKit has a lot of tests that were regression tests for a specific
>>> bug fix, not as conformance tests (though they might b useful for that
>>> too). In such cases, the association with the bugs.webkit.org bug is
>>> more important than the spec URL. That’s particularly the case when
>>> the test has been changed multiple times to reflect further WebKit
>>> behavior changes. I know that I’ve personally found it very useful to
>>> look at revision history of tests, or search for bug numbers or
>>> keywords in LayoutTests/ChangeLog to find related tests.
>>> Not saying this is the sole purpose of tests, but losing this ability
or making it more awkward would be a downside to deleting tests from the
WebKit repo.
>> Well, I think we agree here, that's why "conformance" tests was
>> specified in my message. For such tests, connecting the history of tests
>> with the development of the specs is actually more important. For
>> example I recently noticed that some bugs with WOFF2 (on Apple's ports
>> only) and WebVTT have been ignored in WebKit because we don't
>> import/sync WPT tests for these specs.
>
> I think importing new test suites is a different question than
upstreaming/removing tests. In general importing more test suites is
probably good, but we probably need to tackle the WPT performance problem
before we pull in too many new WPT suites.
>
> I should also note that pulling in the test suite won't automatically get
the bugs fixed or prioritized, or even filed in the bug tracker necessarily.

This is the case in Chromium as well, although Robert Ma is working on
import notifications:
https://docs.google.com/document/d/1W3V81l94slAC_rPcTKWXgv3YxRxtlSIAxi3yj6NsbBw/edit?usp=sharing

We don't know yet what's going to work well, but I think that somehow,
before too long, we need to at least triage all new failures. Far from all
will be worth prioritizing of course, but sometimes the existence of the
test can reduce the effort and make it easier to prioritize.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[Philip Jägenstedt]

Hi Danyao,

I'm afraid that I don't yet have proper breakdown of this, but this is from
some quick grepping in WebKit:

   - 3200k *-expected.html outside of LayoutTests/imported, that might work
   as reftests.
   - 60k files that aren't called *-expected.*, again outside of
   LayoutTests/imported/
   - 765 of those have "testharness.js" somewhere.
   - 3800 of those have "internals." or "eventSender." somewhere

So, it's certainly not the case that there are tens of thousands of tests
that could easily be upstreamed. And no doubt tens of thousands of tests
that will best be left alone for a long time to come.

But, at the very least I think there will be some hundreds of tests that
could be contributed with some mechanical text transforms that don't
introduce much chance of regressing the tests.

On Wed, Nov 15, 2017 at 5:16 PM Danyao <dan...@chromium.org> wrote:

> Hi Philip,
>
> Ali and I would be willing to help this wherever we can. It seems to us
> that there may be a few different classes of tests that require varying
> degree of effort to migrate. Do you have any estimates on how many tests
> fall into each category, maybe based on your analysis of Blink LayoutTests?
>
>1. braindead to migrate
>- 1a) ref tests not using window.internals
>   - 1b) tests using internals API that have trivial equivalent in
>   testharness
>2. tests using non-trivial internals API so require some domain
>expertise in the functionality under test
>   - 2a) priority features: maybe active development so more likely to
>   diverge in the near future?
>   - 2b) long tail
>3. tests that can't be migrated to WPT
>   - 3a) tests that explicit use layerTreeAsText (tests in the
>   compositing layer)
>   - 3b) tests that dump render object tree (testing browser internals)
>
>
> Thanks,
> Danyao
>
> On Wed, Nov 15, 2017 at 5:02 AM, Philip Jägenstedt <foo...@chromium.org>
> wrote:
>
>> Hello webkit-dev! (ecosystem-infra in Bcc)
>>
>> TPAC was last week, and there was much talk about web-platform-tests.
>> Some notes are at
>> https://lists.w3.org/Archives/Public/public-test-infra/2017OctDec/thread.html
>>  and
>> in particular https://www.w3.org/2017/11/07-testing-minutes.html.
>>
>> In Blink, we're thinking seriously about what it'd take to upstream large
>> parts of LayoutTests into web-platform-tests, and we've realized that there
>> are some risks here, beyond just making sure the tests are good and per
>> spec. Specifically, one bad outcome would be if Blink successfully
>> upstreamed thousands of tests that are also in WebKit, which then begin to
>> diverge in small and large ways. That'd leave WebKit with more duplication
>> between its own tests and web-platform-tests than any other engine, and a
>> headache that grows over time.
>>
>> So, I think this would require close cooperation with the WebKit project.
>> Some different ways this might happen:
>>
>>1. A Blink developer and WebKit developer work together at the same
>>time to move their common tests in some area into web-platform-tests, each
>>removing identical tests that were upstreamed by the other and
>>consolidating what remains.
>>2. A Blink developer works to first upstream tests from WebKit (using
>>WebKit's coming export mechanism), waits for it to be imported into Blink,
>>and then removes the tests from Blink.
>>3. The reverse situation.
>>
>>
>> Are there already some areas where the first approach might be doable,
>> where the Blink and WebKit folks already know each other and are eager to
>> do this? Is LayoutTests/css3/flexbox/ by any chance such a case?
>>
>> I realize it's hard to judge these approaches in the abstract, but am
>> curious to hear any enthusiasm or concerns about it.
>>
>> Thanks!
>>
>> ___
>> webkit-dev mailing list
>> webkit-dev@lists.webkit.org
>> https://lists.webkit.org/mailman/listinfo/webkit-dev
>>
>>
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

[webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[Philip Jägenstedt]

Hello webkit-dev! (ecosystem-infra in Bcc)

TPAC was last week, and there was much talk about web-platform-tests. Some
notes are at
https://lists.w3.org/Archives/Public/public-test-infra/2017OctDec/thread.html
and
in particular https://www.w3.org/2017/11/07-testing-minutes.html.

In Blink, we're thinking seriously about what it'd take to upstream large
parts of LayoutTests into web-platform-tests, and we've realized that there
are some risks here, beyond just making sure the tests are good and per
spec. Specifically, one bad outcome would be if Blink successfully
upstreamed thousands of tests that are also in WebKit, which then begin to
diverge in small and large ways. That'd leave WebKit with more duplication
between its own tests and web-platform-tests than any other engine, and a
headache that grows over time.

So, I think this would require close cooperation with the WebKit project.
Some different ways this might happen:

   1. A Blink developer and WebKit developer work together at the same time
   to move their common tests in some area into web-platform-tests, each
   removing identical tests that were upstreamed by the other and
   consolidating what remains.
   2. A Blink developer works to first upstream tests from WebKit (using
   WebKit's coming export mechanism), waits for it to be imported into Blink,
   and then removes the tests from Blink.
   3. The reverse situation.


Are there already some areas where the first approach might be doable,
where the Blink and WebKit folks already know each other and are eager to
do this? Is LayoutTests/css3/flexbox/ by any chance such a case?

I realize it's hard to judge these approaches in the abstract, but am
curious to hear any enthusiasm or concerns about it.

Thanks!
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Editing WPT tests in a WebKit checkout

[Philip Jägenstedt]

On Sun, Oct 15, 2017 at 2:39 PM, youenn fablet  wrote:
> Hi,
>
> I am working on a script [1] to easily export edits made to WPT tests from a
> local WebKit checkout into GitHub.
> The procedure would be something like:
> 1. Develop a WebKit patch
> 2. As part of step 1, update/create WPT tests and test them in WebKit
> environment
> 3. Export WPT-related changes to a WPT GitHub fork and/or PR to W3C WPT
> repository using the script
>
> This allows authoring WPT tests and WebKit patch at the same time.
> This removes the annoying steps of manually copying WebKit changes to an
> external WPT checkout.
>
> At that point, one can wait for WPT PR to land or start uploading changes to
> bugzilla in parallel.
> The usual WebKit review process will happen with the possibility to merge
> the corresponding WPT PR when related WebKit patch is r+ed.
> Changes to WPT tests should be committed in WebKit trunk when already merged
> to W3C WPT.
>
> There are some benefits of doing both PR and bugzilla at the same time.
> It allows for instance validating stability of test, style of test and
> results on other browsers through WPT GitHub infrastructure.

I'd like to lend my support to the value of "authoring WPT tests and
WebKit patch at the same time".

A month ago, I looked into what effect 2-way sync had in Chromium, and
summarized here:
https://groups.google.com/a/chromium.org/d/msg/ecosystem-infra/yDAnUCUqFcw/sk5gxzjbBgAJ

"Chromium contributions have more than tripled, and it seems very
likely that automatic export of changes has something to do with it."

Gecko is now improving its sync with web-platform-tests, and frequent
sync with WebKit is at the very top of my wishlist for things that
would enable improved interop of browser engines.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] WPT tests and runtime-enabled features

[Philip Jägenstedt]

FWIW, in Chromium, if we had an off-by-default feature that we wanted to
test using wpt, we would probably use the
https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/VirtualTestSuites
mechanism.

On Sat, Jun 3, 2017 at 1:03 AM, youenn fablet  wrote:

> Hi Ali,
>
> Yes this is something we should have a solution for. Right now, we have
> two ways:
> - Enabling the runtime-enabled flag only in DumpRenderTree/
> WebKitTestRunner
> - Updating WebKit testharnessreport.js to call internal APIs for some/all
> WPT tests
>
> Ideally, we should be able to set additional configuration information to
> WPT tests without modifying them.
> A configuration file alongside the WPT test might be feasible and could
> supersede the 
> technique.
> Folder-based hierarchical configuration might be nice but I am unsure
> whether that would meet the complexity tradeoff bar.
> Nothing yet started there so any help would be greatly appreciated!
> y
>
> Le ven. 2 juin 2017 à 13:37, Ali Juma  a écrit :
>
>> Hi all,
>>
>> While a feature is in development behind a runtime-enabled flag, is there
>> a way to write tests in web-platform-tests that enable the feature when
>> they're run as part of LayoutTests? Adding "" tags would technically work but seems a bit
>> questionable.
>>
>> The point of this would be to be able to use web platform tests as
>> regression tests while developing a feature, allowing the tests to be
>> upstreamed sooner and saving having to write them outside of
>> web-platform-tests first and then move them over later.
>>
>> Thanks,
>> Ali
>>
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Upstreaming Tests from WebKit to Web Platform Tests

[Philip Jägenstedt]

On Mon, Feb 6, 2017 at 7:32 PM Ryosuke Niwa  wrote:

> On Mon, Feb 6, 2017 at 4:22 PM, youenn fablet  wrote:
>
> It seems we agree in moving this forward. Any objection?
> If so, let's start with small practical steps, something like a script to
> automatically generate WPT PR requests from a WebKit repo.
>
>
> I think we need to first agree on where to put new tests. If we're placing
> next to existing tests inside LayoutTests/imported/w3c/web-platform-tests/
> then identifying those new tests will be an issue.
>
> Also, there needs to be a mechanism to detect these new tests that have
> been merged into upstream when we're re-importing. It's possible for tests
> newly added in WebKit to be renamed, moved, or otherwise modified before we
> re-import them back.
>
> All these situations need to be carefully thought out first.
>

If it might be of any use, here's a doc where we hash out a lot of question
for 2-way sync in Chromium:
https://docs.google.com/document/d/1JgPTyIWmjlXyhyatiZ9A6fSbUQPjCyM26budEY90VDE/edit?usp=sharing

Linked from that is also an older doc with a lot of discussion:
https://docs.google.com/document/d/1JOcZsURB3ITsRtundqkpVVDrqo6x-_jdif7s_0rLJD0/edit?usp=sharing

When it comes to identifying which commits to export next, we look for the
most recent exported commit in web-platform-tests, map that to a Chromium
commit, and then inspect all later commits that touch
LayoutTests/external/wpt/. This scheme only works if commits are always
exported in order. Note also that commits before the most recent import can
also be candidates, because of imports racing with test changes in CQ.
Automatic imports are also blocked on all exports being finished, to not
(temporarily) revert changes.

Happy to provide more details on design and trade-offs if useful to you.
Regardless, I'm quite excited to see this happening in WebKit.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Upstreaming Tests from WebKit to Web Platform Tests

[Philip Jägenstedt]

FWIW, in Blink we stopped rewriting the testharness.js paths before
switching to wptserve, by instead rewriting those URLs only when running
LayoutTests:
https://cs.chromium.org/chromium/src/content/shell/renderer/layout_test/blink_test_runner.cc?type=cs=content/shell/renderer/layout_test/blink_test_runner.cc=221

So, we know that it's possible to run a lot of the tests without wptserve
without modifying the tests. However, trying to list all the tests that do
or don't need wptserve seems like a lot of work, and I think we're now
using wptserve for all tests.

On Sun, Feb 5, 2017 at 8:05 PM youenn fablet  wrote:

> I too am a big proponent of us moving more and more towards WPT.
> As part of the streams and fetch API implementation, most of the related
> functional tests have been made as WPT.
> The benefits of having tests being improved and updated largely outweighs
> the testharness.js initial cost.
> Somehow, these tests are becoming part of their related specs.
>
> The two complaints I heard against testharness.js are:
> - They are less easy to debug as test harness.js does not print out
> messages for each assert. There might be room for updating testharness to
> support that
> - Async tests are less easy to write. While this is probably true,
> testharness has support for promise-based tests which are not verbose.
> WPT has also some benefits of its own, like the possibility to run easily
> the same tests in worker/window environments, the flexible python-based
> server...
>
> One complaint I heard against WPT tests is that they require being run
> behind a server.
> This is becoming more and more true.
> There is still a large portion of tests that could be run locally if we
> update the paths to test harness.js/testharnessreport,js.
> I am not a big fan of modify any WPT test but maybe we should consider
> switching back to modifying these paths at import and export time.
>
> I would also like to go in a direction where writing WPT tests is the
> default for WebKit layout test.
> For that to happen, we need an easy way to upstream tests from WebKit to
> WPT GitHub.
>
> I would tend to do the following:
> - Write/submit WPT tests in WebKit as regular WebKit testharness.js layout
> tests through bugzilla
> - At commit queue time, automatically create a PR to WPT GitHub containing
> the changes of the WebKit patch
> - Ask committer to fix the WPT PR should there be any issue with it
> (committer being informed of such issues through bugzilla).
>
> Le dim. 5 févr. 2017 à 15:42, Ryosuke Niwa  a écrit :
>
> Hi all,
>
> *Background*
>
> Web Platform Tests is a suite of tests written for W3C specifications
> =,
> and Blink, Edge, Gecko, and WebKit all run them as a part of their
> continuous build system.
>
> Historically, each working group had their own repository for tests,
> but with CSS WG's tests  finally
> getting merged into the Web Platform Tests,
> we will have a single repository with all the tests from W3C.
>
> A few of us attended a meeting to discuss the future of this test suite
> 
> last Monday.
> One of the major topic was that Blink and Gecko have adopted the process
> to write the tests
> using testharness.js in their own test suites, and automatically merge
> into Web Platform Tests
> without having to go through another round of code review for Web Platform
> Tests.
>
> Given we do test-driven development in WebKit, I think WebKit should do
> the same.
> This will benefit other browser vendors to catch their bugs with our
> tests, and we will also benefit
> from other browser vendors "reviewing" our tests against their
> implementation and specifications.
>
> In the long term, I believe this will drastically improve the
> interoperability of the Web,
> and dramatically improve the quality of the tests we run against WebKit.
>
> In fact, there was a general consensus that we should even upstream
> pass-if-not-crash tests
> as well as style and layout invalidation tests to web-platform-tests as
> they tend to be undertested
> right now and almost all engines have bugs in this area.
>
> *Problems & Questions*
>
> In order to auto-merge tests from WebKit to Web Platform Tests, there are
> a few problems.
>
> *1. We need to start writing more if not all tests using testharness.js
> instead of our own js-test(-pre).js*
>
> I've heard of many complaints about testharness.js being too verbose and
> cumbersome to use.
> I agree with all those complaints but I don't think we can convince all
> other browser vendors
> to use our own testing script either since both Blink & Gecko are onboard
> with using testharness.js
>
> At this point, I'd argue that the benefit outweighs the cost of adopting
> testharness.js.
> Also, W3C have dropped many styling requirements for tests so