Re: [webkit-dev] Safari Tech Preview 96 available on wpt.fyi!

[Philip Jägenstedt]

On Tue, Dec 3, 2019 at 3:05 AM Jiewen Tan  wrote:

> Hi Maciej,
>
> On Dec 2, 2019, at 4:10 PM, Maciej Stachowiak  wrote:
>
>
> There’s a number of mysterious timeouts with 96. Not sure if flakiness or
> real?
>
> The new WebCrypto failures are surprising, but likely real and should be
> investigated:
> https://wpt.fyi/results/WebCryptoAPI/wrapKey_unwrapKey?diff=ADC=is%3Adifferent_id=347530011_id=381930013
> 
>
>
> I believe these tests are flaky. I have made a PR to improve it a while
> ago. I should probably get those improvement landed sometime.
> https://github.com/web-platform-tests/wpt/pull/6102
>

That sounds very doable. I added you to the reviewers team to make this
easier and commented here:
https://github.com/web-platform-tests/wpt/pull/6102#issuecomment-561115470
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Safari Tech Preview 96 available on wpt.fyi!

[Stephen Mcgruer]

>
> There’s a number of mysterious timeouts with 96. Not sure if flakiness or
> real?
> Many large new chunks of not-run tests are caused by a harness error or
> timeout. E.g. html/ and webauthn


We have unfortunately struggled historically with getting reliable +
consistent results for Safari, with spurious infrastructure + other
problems. That said, Igalia has been working on improving reliability and
have made good progress, and we have also invested in projects like the
WebkitGTK runs to help with this problem (e.g. if a test fails in both
Safari and WebKitGTK, there's a better chance it's a real failure[0])

We have had many Safari TP 96 runs since my announcement (
https://wpt.fyi/runs?label=master=100=safari); you can
click on a single run (the icon) to view the run itself, or click on two
runs to get a diff view - which can be useful for spotting flakes (as most
SHAs are only changing a few tests, so differences between consecutive runs
are usually flakes). There is also an explicit flake analyzer on our
insights page - https://wpt.fyi/insights (it's a little clunky but can
still be useful).

I believe these tests are flaky. I have made a PR to improve it a while
> ago. I should probably get those improvement landed sometime.


Improvements would be great! Looking at that PR, it is quite old, so will
need to be rebased and force pushed. Let me know if you need any help on
that :).

STP 96 has enabled user agent UI for WebAuthn, which could cause crashes.
> This could be the reason why webauthn is 0 all the time.


It does look like there are still some harness failures in further STP 96
runs (e.g. https://wpt.fyi/results/webauthn?run_id=361070008); I see a
mention in the failing tests of a "UnexpectedAlertOpenException: unexpected
alert open (500):" - is that what you mean? Is there some flag we could be
setting on STP, or something Safaridriver could be doing to avoid these?

[0]:
https://wpt.fyi/results/?label=master=safari%5Bexperimental%5D=webkitgtk=chrome%5Bexperimental%5D=firefox%5Bexperimental%5D=safari%3Afail%20webkitgtk%3Afail%20chrome%3Apass%20firefox%3Apass
is
an example search for tests that fail in both Safari + WebKitGTK, but pass
in both Chrome and Firefox.

On Mon, 2 Dec 2019 at 21:04, Jiewen Tan  wrote:

> Hi Maciej,
>
> On Dec 2, 2019, at 4:10 PM, Maciej Stachowiak  wrote:
>
>
> There’s a number of mysterious timeouts with 96. Not sure if flakiness or
> real?
>
> The new WebCrypto failures are surprising, but likely real and should be
> investigated:
> https://wpt.fyi/results/WebCryptoAPI/wrapKey_unwrapKey?diff=ADC=is%3Adifferent_id=347530011_id=381930013
> 
>
>
> I believe these tests are flaky. I have made a PR to improve it a while
> ago. I should probably get those improvement landed sometime.
> https://github.com/web-platform-tests/wpt/pull/6102
>
>
> Many large new chunks of not-run tests are caused by a harness error or
> timeout. E.g. html/ and webauthn/
>
>
> STP 96 has enabled user agent UI for WebAuthn, which could cause crashes.
> This could be the reason why webauthn is 0 all the time.
>
> Best,
> Jiewen
>
>
>
> On Nov 27, 2019, at 7:07 AM, Stephen Mcgruer 
> wrote:
>
> Excited to announce that Safari Tech Preview 96 is now available on
> wpt.fyi!
>
> Example run:
> https://wpt.fyi/results/?label=master=experimental=chrome=firefox=safari
>
> Diff against Safari Tech Preview 95 (not at exactly the same WPT sha, but
> very close):
> https://wpt.fyi/results/?diff=ADC=is%3Adifferent_id=347530011_id=381930013
>
> If you are surprised by the results in the diff view, I would love to hear
> about it. One thing that surprised me is that the release blog post[0]
> mentioned Web Animations being enabled by default, but we see no
> differences in the test diff for web-animations/. It's possible we enable
> some flag that turned on Web Animations already, but I can't see it
> obviously in our safari setup[1].
>
> [0]:
> https://webkit.org/blog/9658/release-notes-for-safari-technology-preview-96/
> [1]:
> https://github.com/web-platform-tests/wpt/blob/master/tools/ci/azure/install_safari.yml
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
>
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Safari Tech Preview 96 available on wpt.fyi!

[Jiewen Tan]

Hi Maciej,

> On Dec 2, 2019, at 4:10 PM, Maciej Stachowiak  wrote:
> 
> 
> There’s a number of mysterious timeouts with 96. Not sure if flakiness or 
> real?
> 
> The new WebCrypto failures are surprising, but likely real and should be 
> investigated: 
> https://wpt.fyi/results/WebCryptoAPI/wrapKey_unwrapKey?diff=ADC=is%3Adifferent_id=347530011_id=381930013
>  
> 
I believe these tests are flaky. I have made a PR to improve it a while ago. I 
should probably get those improvement landed sometime.
https://github.com/web-platform-tests/wpt/pull/6102 

> 
> Many large new chunks of not-run tests are caused by a harness error or 
> timeout. E.g. html/ and webauthn/

STP 96 has enabled user agent UI for WebAuthn, which could cause crashes. This 
could be the reason why webauthn is 0 all the time.

Best,
Jiewen

> 
> 
>> On Nov 27, 2019, at 7:07 AM, Stephen Mcgruer > > wrote:
>> 
>> Excited to announce that Safari Tech Preview 96 is now available on wpt.fyi 
>> !
>> 
>> Example run: 
>> https://wpt.fyi/results/?label=master=experimental=chrome=firefox=safari
>>  
>> 
>> 
>> Diff against Safari Tech Preview 95 (not at exactly the same WPT sha, but 
>> very close): 
>> https://wpt.fyi/results/?diff=ADC=is%3Adifferent_id=347530011_id=381930013
>>  
>> 
>> 
>> If you are surprised by the results in the diff view, I would love to hear 
>> about it. One thing that surprised me is that the release blog post[0] 
>> mentioned Web Animations being enabled by default, but we see no differences 
>> in the test diff for web-animations/. It's possible we enable some flag that 
>> turned on Web Animations already, but I can't see it obviously in our safari 
>> setup[1].
>> 
>> [0]: 
>> https://webkit.org/blog/9658/release-notes-for-safari-technology-preview-96/ 
>> 
>> [1]: 
>> https://github.com/web-platform-tests/wpt/blob/master/tools/ci/azure/install_safari.yml
>>  
>> ___
>> webkit-dev mailing list
>> webkit-dev@lists.webkit.org 
>> https://lists.webkit.org/mailman/listinfo/webkit-dev
> 
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Safari Tech Preview 96 available on wpt.fyi!

[Maciej Stachowiak]

There’s a number of mysterious timeouts with 96. Not sure if flakiness or real?

The new WebCrypto failures are surprising, but likely real and should be 
investigated: 
https://wpt.fyi/results/WebCryptoAPI/wrapKey_unwrapKey?diff=ADC=is%3Adifferent_id=347530011_id=381930013
 


Many large new chunks of not-run tests are caused by a harness error or 
timeout. E.g. html/ and webauthn/


> On Nov 27, 2019, at 7:07 AM, Stephen Mcgruer  wrote:
> 
> Excited to announce that Safari Tech Preview 96 is now available on wpt.fyi 
> !
> 
> Example run: 
> https://wpt.fyi/results/?label=master=experimental=chrome=firefox=safari
>  
> 
> 
> Diff against Safari Tech Preview 95 (not at exactly the same WPT sha, but 
> very close): 
> https://wpt.fyi/results/?diff=ADC=is%3Adifferent_id=347530011_id=381930013
>  
> 
> 
> If you are surprised by the results in the diff view, I would love to hear 
> about it. One thing that surprised me is that the release blog post[0] 
> mentioned Web Animations being enabled by default, but we see no differences 
> in the test diff for web-animations/. It's possible we enable some flag that 
> turned on Web Animations already, but I can't see it obviously in our safari 
> setup[1].
> 
> [0]: 
> https://webkit.org/blog/9658/release-notes-for-safari-technology-preview-96/ 
> 
> [1]: 
> https://github.com/web-platform-tests/wpt/blob/master/tools/ci/azure/install_safari.yml
>  
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Safari Tech Preview 96 available on wpt.fyi!

[Stephen Mcgruer]

>
> Safari Technology Preview had been shipping with Web Animations on by
> default prior to that change. The value
> DEFAULT_EXPERIMENTAL_FEATURES_ENABLED is true for STP builds. So it’s
> correct that you don’t see any change.


Thanks for the explanation! :)

On Thu, 28 Nov 2019 at 07:34, Antoine Quint  wrote:

> Hi Stephen,
>
> On 27 Nov 2019, at 16:07, Stephen Mcgruer  wrote:
>
> One thing that surprised me is that the release blog post[0] mentioned Web
> Animations being enabled by default, but we see no differences in the test
> diff for web-animations/. It's possible we enable some flag that turned on
> Web Animations already, but I can't see it obviously in our safari setup[1].
>
>
> Safari Technology Preview had been shipping with Web Animations on by
> default prior to that change. The value
> DEFAULT_EXPERIMENTAL_FEATURES_ENABLED is true for STP builds. So it’s
> correct that you don’t see any change.
>
> Antoine
>
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Safari Tech Preview 96 available on wpt.fyi!

[Antoine Quint]

Hi Stephen,

> On 27 Nov 2019, at 16:07, Stephen Mcgruer  wrote:
> 
> One thing that surprised me is that the release blog post[0] mentioned Web 
> Animations being enabled by default, but we see no differences in the test 
> diff for web-animations/. It's possible we enable some flag that turned on 
> Web Animations already, but I can't see it obviously in our safari setup[1].

Safari Technology Preview had been shipping with Web Animations on by default 
prior to that change. The value DEFAULT_EXPERIMENTAL_FEATURES_ENABLED is true 
for STP builds. So it’s correct that you don’t see any change.

Antoine

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

[webkit-dev] Safari Tech Preview 96 available on wpt.fyi!

[Stephen Mcgruer]

Excited to announce that Safari Tech Preview 96 is now available on wpt.fyi!

Example run:
https://wpt.fyi/results/?label=master=experimental=chrome=firefox=safari

Diff against Safari Tech Preview 95 (not at exactly the same WPT sha, but
very close):
https://wpt.fyi/results/?diff=ADC=is%3Adifferent_id=347530011_id=381930013

If you are surprised by the results in the diff view, I would love to hear
about it. One thing that surprised me is that the release blog post[0]
mentioned Web Animations being enabled by default, but we see no
differences in the test diff for web-animations/. It's possible we enable
some flag that turned on Web Animations already, but I can't see it
obviously in our safari setup[1].

[0]:
https://webkit.org/blog/9658/release-notes-for-safari-technology-preview-96/
[1]:
https://github.com/web-platform-tests/wpt/blob/master/tools/ci/azure/install_safari.yml
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev