Re: [webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[Alexey Proskuryakov]

> 17 нояб. 2017 г., в 23:02, youenn fablet  написал(а):
> 
> Tests are available at https://w3c-test.org  which 
> makes it easy to share through any tool supporting hyperlinks.
> A webarchive can also be made so that it is easy to share and probably edit 
> such tests.
> Tools like jsfiddle are also a great way to create/share/edit tests.
> I received several bug reports on bugzilla using it and this proved to be 
> efficient.

I don't think that these are applicable verbatim, but there may be some 
solution. Let's find it before assuming that we can have it.

In particular, 
- editing a WebArchive is not really feasible;
- jsfiddle is cool, but I can't just copy a WPT test to jsfiddle to share it.

> Let me explain why I think that WebKit tests are often more valuable as 
> regression tests than WPT tests are. We add tests as we fix bugs, so we know 
> that the tests are generally for problems that have a high impact on users 
> and developers - that's because someone actually discovered the problem, and 
> someone prioritized it highly enough to fix. We also know that our tests 
> cover code that is error prone, which is why we had bugs in the first place. 
> Of course, anything can be broken, but certain things are less likely to. 
> Compliance tests written for specs are also valuable, but at some point we 
> need to prioritize which tests to investigate and even to run.
>  
> I don't really see why we should prioritize the tests to run when all of them 
> provide clear value to some WebKit members.

We prioritize which tests to run all the time, there are whole configurations 
for which we don't run any tests. That's largely because it's not enough to 
simply run the tests - keeping them in a state where they produce actionable 
results requires a lot of attention.

The time it takes to run tests definitely matters a lot for WebKit. So far we 
haven't taken a route like the one Robert mentioned, with a huge amount of 
hardware running shards of tests. There are multiple reasons to that, one of 
those being that we are very interested in finding bugs below WebKit too, and 
having dedicated testers helps with that. When the machine gets into a weird 
state after a few weeks of uptime, it is much easier to start isolating the 
problem when we can see which specific queues are hitting it. It is also much 
easier to reason about data collected by centralized systems, such as crash log 
collection services.

- Alexey


> I agree that we need to prioritize tests we investigate. There can be a 
> solution inside WPT, like adding WebKit specific metadata so that:
> - WPT contributors would communicate with WebKit members whenever changing 
> such tests
> - WebKit contributors would prioritize WPT-WebKit-metadata failing tests
> 
> That said, if these tests are so beneficial to WebKit, they are potentially 
> very useful to other teams as well.
> And vice-versa, we might find really good WPT tests that show useful crashes 
> and failures in WebKit.
> I am experiencing that nowadays with WPT service worker tests.


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[Frédéric WANG]

On 28/11/2017 16:24, Philip Jägenstedt wrote:
>
> To do the upstreaming from WebKit first would get around this problem,
> and is something I could see us doing. If a Chromium developer
> upstreamed tests from WebKit and got review from a WebKit reviewer, I
> think we would rubberstamp the corresponding removal in Chromium.
>
> But that would be a futile exercise if the WebKit reviewers don't want
> to delete the tests from WebKit. The only remaining options would then
> be not sharing the tests with EdgeHTML and Gecko, or creating more
> duplication for WebKit.
Note that we can also have duplication in WebKit but for each port
decide which version will be executed by adding "skip" tags in
TestExpectation. Not ideal in my opinion, but that would allow e.g.
GTK/WPE ports to run the latest WPT tests while macOS/iOS ports keep the
legacy LayoutTests/ versions / history.

-- 
Frédéric Wang - frederic-wang.fr



signature.asc
Description: OpenPGP digital signature
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[Philip Jägenstedt]

On Tue, Nov 28, 2017 at 3:37 PM Ryosuke Niwa  wrote:

> On Mon, Nov 27, 2017 at 5:58 PM, Philip Jägenstedt 
> wrote:
>
>> (From Nov 24, I used the wrong email, resending for the archives.)
>>
>> On Sat, Nov 18, 2017 at 8:02 AM, youenn fablet  wrote:
>> > Thanks for taking the time to share this additional information.
>> > I think this is helpful to make progress.
>> > Please see inline for some more comments related to the points you are
>> > bringing to the table.
>> >
>> > Stepping back from the WPT server specific issue, being optimistic and
>> > assuming that we are able to run WPT tests with good enough
>> performances.
>> > Migration potential downsides I heard so far:
>> > - Prioritization of tests to investigate might be harder.
>> > - Sharing tests with other teams might be harder if subresource links
>> are
>> > not relative.
>> > These two seem like solvable issues to me.
>> >
>> > Le ven. 17 nov. 2017 à 10:09, Alexey Proskuryakov  a
>> écrit :
>> >>
>> >>
>> >> 17 нояб. 2017 г., в 9:18, youenn fablet 
>> написал(а):
>> >>
>> >>
>> >> Chris recently noticed that some heavily used files (testharness*) were
>> >> cacheable through Apache but not WPT.
>> >> This is now fixed and should improve WPT performances.
>> >>
>> >>
>> >> This is part of ,
>> another
>> >> part is that the server is 10x slower than Apache.
>> >
>> >
>> > Other measurements showed 3x slower, which makes it still worthwhile to
>> > explore.
>> > We need to be cautious here though since optimization is all about
>> chasing
>> > where time is actually spent.
>> >
>> > If we can prove to ourselves that this is an important issue, we should
>> > discuss with the WPT community to see how to fix this issue.
>> > In addition to better caching, other optimization like
>> > https://github.com/w3c/wptserve/pull/86 may bring some additional
>> benefit.
>> >
>> > If we do not find any good solution at WPT server level, we still have
>> the
>> > option to run some tests as file-based.
>> > Ryosuke mentioned the possibility to classify tests at import time by
>> > checking their results when served through WPT server and as file URLs.
>>
>> Thanks for filing https://github.com/w3c/web-platform-tests/issues/8391,
>> Youenn!
>>
>> I've put a placeholder around this in our planning for next quarter.
>> At a minimum, I'd like to investigate this and see how much slower the
>> tests are in Chromium's infra than they would be without wptserve, and
>> where that extra time is spent. But the reasons may not be the same in
>> WebKit, so please don't count on any improvement :)
>>
>> >> I just tested on my MacBook Pro, and WPT tests took 23% of time while
>> >> being only 9% of the total count. Taking in mind that WebKit own tests
>> have
>> >> higher value due to the way we choose what to test (see below), that's
>> not a
>> >> great story in my opinion.
>> >
>> >
>> > These numbers are difficult to interpret.
>> > WPT authoring style is multiple tests in a single file, which is bad for
>> > stability but potentially good for performances.
>> > WebKit usually prefers one file/one test.
>> >
>> > If we want to talk to WPT community about this, we need to provide some
>> more
>> > tangible numbers.
>> > We could try to run a large subset of WPT tests that run the same
>> through
>> > Apache and WPT.
>> > I just did that on a small subset of IDB tests. This seems to show
>> something
>> > like a 25% slowdown.
>> >
>> >> One other thing that we discussed before was the operational
>> complexity of
>> >> running WPT tests. We frequently need to share tests with people who
>> don't
>> >> work on WebKit directly, but have the need to edit and run our tests.
>> >> Inability to drag and drop a local copy into a Safari window is a
>> deterrent
>> >> to addressing problems caught by the tests. I think that the response
>> we got
>> >> was that tests will continue to require a server to run.
>> >
>> >
>> > Tests are available at https://w3c-test.org which makes it easy to
>> share
>> > through any tool supporting hyperlinks.
>> > A webarchive can also be made so that it is easy to share and probably
>> edit
>> > such tests.
>> > Tools like jsfiddle are also a great way to create/share/edit tests.
>> > I received several bug reports on bugzilla using it and this proved to
>> be
>> > efficient.
>> >
>> >>
>> >> Let me explain why I think that WebKit tests are often more valuable as
>> >> regression tests than WPT tests are. We add tests as we fix bugs, so
>> we know
>> >> that the tests are generally for problems that have a high impact on
>> users
>> >> and developers - that's because someone actually discovered the
>> problem, and
>> >> someone prioritized it highly enough to fix. We also know that our
>> tests
>> >> cover code that is error prone, which is why we had bugs in the first
>> place.
>> >> Of course, anything 

Re: [webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[Ryosuke Niwa]

On Mon, Nov 27, 2017 at 5:58 PM, Philip Jägenstedt 
wrote:

> (From Nov 24, I used the wrong email, resending for the archives.)
>
> On Sat, Nov 18, 2017 at 8:02 AM, youenn fablet  wrote:
> > Thanks for taking the time to share this additional information.
> > I think this is helpful to make progress.
> > Please see inline for some more comments related to the points you are
> > bringing to the table.
> >
> > Stepping back from the WPT server specific issue, being optimistic and
> > assuming that we are able to run WPT tests with good enough performances.
> > Migration potential downsides I heard so far:
> > - Prioritization of tests to investigate might be harder.
> > - Sharing tests with other teams might be harder if subresource links are
> > not relative.
> > These two seem like solvable issues to me.
> >
> > Le ven. 17 nov. 2017 à 10:09, Alexey Proskuryakov  a
> écrit :
> >>
> >>
> >> 17 нояб. 2017 г., в 9:18, youenn fablet  написал(а):
> >>
> >>
> >> Chris recently noticed that some heavily used files (testharness*) were
> >> cacheable through Apache but not WPT.
> >> This is now fixed and should improve WPT performances.
> >>
> >>
> >> This is part of ,
> another
> >> part is that the server is 10x slower than Apache.
> >
> >
> > Other measurements showed 3x slower, which makes it still worthwhile to
> > explore.
> > We need to be cautious here though since optimization is all about
> chasing
> > where time is actually spent.
> >
> > If we can prove to ourselves that this is an important issue, we should
> > discuss with the WPT community to see how to fix this issue.
> > In addition to better caching, other optimization like
> > https://github.com/w3c/wptserve/pull/86 may bring some additional
> benefit.
> >
> > If we do not find any good solution at WPT server level, we still have
> the
> > option to run some tests as file-based.
> > Ryosuke mentioned the possibility to classify tests at import time by
> > checking their results when served through WPT server and as file URLs.
>
> Thanks for filing https://github.com/w3c/web-platform-tests/issues/8391,
> Youenn!
>
> I've put a placeholder around this in our planning for next quarter.
> At a minimum, I'd like to investigate this and see how much slower the
> tests are in Chromium's infra than they would be without wptserve, and
> where that extra time is spent. But the reasons may not be the same in
> WebKit, so please don't count on any improvement :)
>
> >> I just tested on my MacBook Pro, and WPT tests took 23% of time while
> >> being only 9% of the total count. Taking in mind that WebKit own tests
> have
> >> higher value due to the way we choose what to test (see below), that's
> not a
> >> great story in my opinion.
> >
> >
> > These numbers are difficult to interpret.
> > WPT authoring style is multiple tests in a single file, which is bad for
> > stability but potentially good for performances.
> > WebKit usually prefers one file/one test.
> >
> > If we want to talk to WPT community about this, we need to provide some
> more
> > tangible numbers.
> > We could try to run a large subset of WPT tests that run the same through
> > Apache and WPT.
> > I just did that on a small subset of IDB tests. This seems to show
> something
> > like a 25% slowdown.
> >
> >> One other thing that we discussed before was the operational complexity
> of
> >> running WPT tests. We frequently need to share tests with people who
> don't
> >> work on WebKit directly, but have the need to edit and run our tests.
> >> Inability to drag and drop a local copy into a Safari window is a
> deterrent
> >> to addressing problems caught by the tests. I think that the response
> we got
> >> was that tests will continue to require a server to run.
> >
> >
> > Tests are available at https://w3c-test.org which makes it easy to share
> > through any tool supporting hyperlinks.
> > A webarchive can also be made so that it is easy to share and probably
> edit
> > such tests.
> > Tools like jsfiddle are also a great way to create/share/edit tests.
> > I received several bug reports on bugzilla using it and this proved to be
> > efficient.
> >
> >>
> >> Let me explain why I think that WebKit tests are often more valuable as
> >> regression tests than WPT tests are. We add tests as we fix bugs, so we
> know
> >> that the tests are generally for problems that have a high impact on
> users
> >> and developers - that's because someone actually discovered the
> problem, and
> >> someone prioritized it highly enough to fix. We also know that our tests
> >> cover code that is error prone, which is why we had bugs in the first
> place.
> >> Of course, anything can be broken, but certain things are less likely
> to.
> >> Compliance tests written for specs are also valuable, but at some point
> we
> >> need to prioritize which tests to investigate and even to run.
> >
> 

Re: [webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[Philip Jägenstedt]

(From Nov 24, I used the wrong email, resending for the archives.)

On Sat, Nov 18, 2017 at 8:02 AM, youenn fablet  wrote:
> Thanks for taking the time to share this additional information.
> I think this is helpful to make progress.
> Please see inline for some more comments related to the points you are
> bringing to the table.
>
> Stepping back from the WPT server specific issue, being optimistic and
> assuming that we are able to run WPT tests with good enough performances.
> Migration potential downsides I heard so far:
> - Prioritization of tests to investigate might be harder.
> - Sharing tests with other teams might be harder if subresource links are
> not relative.
> These two seem like solvable issues to me.
>
> Le ven. 17 nov. 2017 à 10:09, Alexey Proskuryakov  a écrit :
>>
>>
>> 17 нояб. 2017 г., в 9:18, youenn fablet  написал(а):
>>
>>
>> Chris recently noticed that some heavily used files (testharness*) were
>> cacheable through Apache but not WPT.
>> This is now fixed and should improve WPT performances.
>>
>>
>> This is part of , another
>> part is that the server is 10x slower than Apache.
>
>
> Other measurements showed 3x slower, which makes it still worthwhile to
> explore.
> We need to be cautious here though since optimization is all about chasing
> where time is actually spent.
>
> If we can prove to ourselves that this is an important issue, we should
> discuss with the WPT community to see how to fix this issue.
> In addition to better caching, other optimization like
> https://github.com/w3c/wptserve/pull/86 may bring some additional benefit.
>
> If we do not find any good solution at WPT server level, we still have the
> option to run some tests as file-based.
> Ryosuke mentioned the possibility to classify tests at import time by
> checking their results when served through WPT server and as file URLs.

Thanks for filing https://github.com/w3c/web-platform-tests/issues/8391, Youenn!

I've put a placeholder around this in our planning for next quarter.
At a minimum, I'd like to investigate this and see how much slower the
tests are in Chromium's infra than they would be without wptserve, and
where that extra time is spent. But the reasons may not be the same in
WebKit, so please don't count on any improvement :)

>> I just tested on my MacBook Pro, and WPT tests took 23% of time while
>> being only 9% of the total count. Taking in mind that WebKit own tests have
>> higher value due to the way we choose what to test (see below), that's not a
>> great story in my opinion.
>
>
> These numbers are difficult to interpret.
> WPT authoring style is multiple tests in a single file, which is bad for
> stability but potentially good for performances.
> WebKit usually prefers one file/one test.
>
> If we want to talk to WPT community about this, we need to provide some more
> tangible numbers.
> We could try to run a large subset of WPT tests that run the same through
> Apache and WPT.
> I just did that on a small subset of IDB tests. This seems to show something
> like a 25% slowdown.
>
>> One other thing that we discussed before was the operational complexity of
>> running WPT tests. We frequently need to share tests with people who don't
>> work on WebKit directly, but have the need to edit and run our tests.
>> Inability to drag and drop a local copy into a Safari window is a deterrent
>> to addressing problems caught by the tests. I think that the response we got
>> was that tests will continue to require a server to run.
>
>
> Tests are available at https://w3c-test.org which makes it easy to share
> through any tool supporting hyperlinks.
> A webarchive can also be made so that it is easy to share and probably edit
> such tests.
> Tools like jsfiddle are also a great way to create/share/edit tests.
> I received several bug reports on bugzilla using it and this proved to be
> efficient.
>
>>
>> Let me explain why I think that WebKit tests are often more valuable as
>> regression tests than WPT tests are. We add tests as we fix bugs, so we know
>> that the tests are generally for problems that have a high impact on users
>> and developers - that's because someone actually discovered the problem, and
>> someone prioritized it highly enough to fix. We also know that our tests
>> cover code that is error prone, which is why we had bugs in the first place.
>> Of course, anything can be broken, but certain things are less likely to.
>> Compliance tests written for specs are also valuable, but at some point we
>> need to prioritize which tests to investigate and even to run.
>
>
> I don't really see why we should prioritize the tests to run when all of
> them provide clear value to some WebKit members.
> I agree that we need to prioritize tests we investigate. There can be a
> solution inside WPT, like adding WebKit specific metadata so that:
> - WPT contributors would communicate with 

Re: [webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[Robert Ma]

On Fri, Nov 24, 2017 at 9:09 AM, Frédéric WANG  wrote:
> Right, running all the tests is nice. But sometimes I personally feel
> guilty to have to execute everything on EWS for example to retrieve
> results for macOS/iOS (when I did not have access to a mac) or to fix
> compilation failures on Windows ports. So better granularity would be
> helpful in my opinion. In any cases, this was just one idea and people
> working on CI and infrastructure are in better position to know how to
> improve running time of WPT tests. I'm surprised that I don't hear such
> complaints from Mozilla and Chromium (but maybe I'm not aware of them).

Chromium has build bots that are similar to EWS. Some bots run before every
patch lands (CQ bots), while the other bots can be requested to run any
time with a WIP patch (try bots). CQ bots run *all* layout tests + WPT,
among other tests, on all major platforms. Try bots are similar, but one
select a subset of bots (platforms) to run. This is made possible by the
infrastructure team with the fancy LUCI system, especially the distributed
test execution system Swarming

.

Running all layout tests + WPT on a single machine takes hours, so I don't
think Blink engineers often do that locally. Generally speaking, people
will use try bots liberally to test 'em all whenever they want to run more
than just a handful of tests.

That said, we can of course benefit from WPT performance improvement as
well.

Robert
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[Frédéric WANG]

On 17/11/2017 17:23, Maciej Stachowiak wrote:
> I should also note that pulling in the test suite won't automatically
> get the bugs fixed or prioritized, or even filed in the bug tracker
> necessarily.eratwerATWeRTWAerATWertwAerer 
Sure. However, I believe in general having more people importing the WPT
tests and trying to match the behavior of the spec and of other browsers
would help, if only to be aware of issues (as Philip mentioned we could
also improve notifications etc). In the particular cases I mentioned
https://bugs.webkit.org/show_bug.cgi?id=162668 was reported one year ago
after Khaled Hosny (invited expert for the WebFonts WG) fixed the bug in
the GTK port while https://bugs.webkit.org/show_bug.cgi?id=179237 just
requires to backport the changes in Chromium to match the latest WebVTT
spec. These are of course only two examples, but in general more tests
shared with the rest of the Web Platform community (developers and spec
authors) are very useful for improving WebKit and interoperability.
> I find it super convenient that EWS runs all the tests even on WIP
> patches. It often catches test failures in tests I didn't think to run
> myself. I think it would be great if EWS could run tests on more
> platforms. Just running release regression tests on more platforms
> would be a big win, if debug builds are the gating factor. I guess it
> would be kind of neat to have a feature of "run the tests across all
> configurations, but only some of them". But I don't think I would
> prioritize it over having more kinds of tests, or finding ways to make
> the full test suite run faster.
Right, running all the tests is nice. But sometimes I personally feel
guilty to have to execute everything on EWS for example to retrieve
results for macOS/iOS (when I did not have access to a mac) or to fix
compilation failures on Windows ports. So better granularity would be
helpful in my opinion. In any cases, this was just one idea and people
working on CI and infrastructure are in better position to know how to
improve running time of WPT tests. I'm surprised that I don't hear such
complaints from Mozilla and Chromium (but maybe I'm not aware of them).

-- 
Frédéric Wang - frederic-wang.fr




signature.asc
Description: OpenPGP digital signature
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[Philip Jägenstedt]

On Fri, Nov 17, 2017 at 5:23 PM, Maciej Stachowiak  wrote:
>
>
>> On Nov 17, 2017, at 7:53 AM, Frédéric WANG  wrote:
>>
>> On 17/11/2017 16:26, Maciej Stachowiak wrote:
>>> WebKit has a lot of tests that were regression tests for a specific
>>> bug fix, not as conformance tests (though they might b useful for that
>>> too). In such cases, the association with the bugs.webkit.org bug is
>>> more important than the spec URL. That’s particularly the case when
>>> the test has been changed multiple times to reflect further WebKit
>>> behavior changes. I know that I’ve personally found it very useful to
>>> look at revision history of tests, or search for bug numbers or
>>> keywords in LayoutTests/ChangeLog to find related tests.
>>> Not saying this is the sole purpose of tests, but losing this ability
or making it more awkward would be a downside to deleting tests from the
WebKit repo.
>> Well, I think we agree here, that's why "conformance" tests was
>> specified in my message. For such tests, connecting the history of tests
>> with the development of the specs is actually more important. For
>> example I recently noticed that some bugs with WOFF2 (on Apple's ports
>> only) and WebVTT have been ignored in WebKit because we don't
>> import/sync WPT tests for these specs.
>
> I think importing new test suites is a different question than
upstreaming/removing tests. In general importing more test suites is
probably good, but we probably need to tackle the WPT performance problem
before we pull in too many new WPT suites.
>
> I should also note that pulling in the test suite won't automatically get
the bugs fixed or prioritized, or even filed in the bug tracker necessarily.

This is the case in Chromium as well, although Robert Ma is working on
import notifications:
https://docs.google.com/document/d/1W3V81l94slAC_rPcTKWXgv3YxRxtlSIAxi3yj6NsbBw/edit?usp=sharing

We don't know yet what's going to work well, but I think that somehow,
before too long, we need to at least triage all new failures. Far from all
will be worth prioritizing of course, but sometimes the existence of the
test can reduce the effort and make it easier to prioritize.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[youenn fablet]

I filed https://github.com/w3c/web-platform-tests/issues/8391 for the WPT
server performances issues.
As of upstreaming the tests, would the following solve identified issues
and keep the upstream process lightweight enough?
- Meta tag each upstreamed test to identify this is an upstreamed version
of a WebKit test (and maybe which WebKit test).
- Keep relative paths for upstreamed tests.
y

Le ven. 17 nov. 2017 à 23:02, youenn fablet  a écrit :

> Thanks for taking the time to share this additional information.
> I think this is helpful to make progress.
> Please see inline for some more comments related to the points you are
> bringing to the table.
>
> Stepping back from the WPT server specific issue, being optimistic and
> assuming that we are able to run WPT tests with good enough performances.
> Migration potential downsides I heard so far:
> - Prioritization of tests to investigate might be harder.
> - Sharing tests with other teams might be harder if subresource links are
> not relative.
> These two seem like solvable issues to me.
>
> Le ven. 17 nov. 2017 à 10:09, Alexey Proskuryakov  a écrit :
>
>>
>> 17 нояб. 2017 г., в 9:18, youenn fablet  написал(а):
>>
>>
>> Chris recently noticed that some heavily used files (testharness*) were
>> cacheable through Apache but not WPT.
>> This is now fixed and should improve WPT performances.
>>
>>
>> This is part of ,
>> another part is that the server is 10x slower than Apache.
>>
>
> Other measurements showed 3x slower, which makes it still worthwhile to
> explore.
> We need to be cautious here though since optimization is all about chasing
> where time is actually spent.
>
> If we can prove to ourselves that this is an important issue, we should
> discuss with the WPT community to see how to fix this issue.
> In addition to better caching, other optimization like
> https://github.com/w3c/wptserve/pull/86 may bring some additional benefit.
>
> If we do not find any good solution at WPT server level, we still have the
> option to run some tests as file-based.
> Ryosuke mentioned the possibility to classify tests at import time by
> checking their results when served through WPT server and as file URLs.
>
> I just tested on my MacBook Pro, and WPT tests took 23% of time while
>> being only 9% of the total count. Taking in mind that WebKit own tests have
>> higher value due to the way we choose what to test (see below), that's not
>> a great story in my opinion.
>>
>
> These numbers are difficult to interpret.
> WPT authoring style is multiple tests in a single file, which is bad for
> stability but potentially good for performances.
> WebKit usually prefers one file/one test.
>
> If we want to talk to WPT community about this, we need to provide some
> more tangible numbers.
> We could try to run a large subset of WPT tests that run the same through
> Apache and WPT.
> I just did that on a small subset of IDB tests. This seems to show
> something like a 25% slowdown.
>
> One other thing that we discussed before was the operational complexity of
>> running WPT tests. We frequently need to share tests with people who don't
>> work on WebKit directly, but have the need to edit and run our tests.
>> Inability to drag and drop a local copy into a Safari window is a deterrent
>> to addressing problems caught by the tests. I think that the response we
>> got was that tests will continue to require a server to run.
>>
>
> Tests are available at https://w3c-test.org which makes it easy to share
> through any tool supporting hyperlinks.
> A webarchive can also be made so that it is easy to share and probably
> edit such tests.
> Tools like jsfiddle are also a great way to create/share/edit tests.
> I received several bug reports on bugzilla using it and this proved to be
> efficient.
>
>
>> Let me explain why I think that WebKit tests are often more valuable as
>> regression tests than WPT tests are. We add tests as we fix bugs, so we
>> know that the tests are generally for problems that have a high impact on
>> users and developers - that's because someone actually discovered the
>> problem, and someone prioritized it highly enough to fix. We also know that
>> our tests cover code that is error prone, which is why we had bugs in the
>> first place. Of course, anything can be broken, but certain things are less
>> likely to. Compliance tests written for specs are also valuable, but at
>> some point we need to prioritize which tests to investigate and even to run.
>>
>
> I don't really see why we should prioritize the tests to run when all of
> them provide clear value to some WebKit members.
> I agree that we need to prioritize tests we investigate. There can be a
> solution inside WPT, like adding WebKit specific metadata so that:
> - WPT contributors would communicate with WebKit members whenever changing
> such tests
> - WebKit contributors would prioritize 

Re: [webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[youenn fablet]

Thanks for taking the time to share this additional information.
I think this is helpful to make progress.
Please see inline for some more comments related to the points you are
bringing to the table.

Stepping back from the WPT server specific issue, being optimistic and
assuming that we are able to run WPT tests with good enough performances.
Migration potential downsides I heard so far:
- Prioritization of tests to investigate might be harder.
- Sharing tests with other teams might be harder if subresource links are
not relative.
These two seem like solvable issues to me.

Le ven. 17 nov. 2017 à 10:09, Alexey Proskuryakov  a écrit :

>
> 17 нояб. 2017 г., в 9:18, youenn fablet  написал(а):
>
>
> Chris recently noticed that some heavily used files (testharness*) were
> cacheable through Apache but not WPT.
> This is now fixed and should improve WPT performances.
>
>
> This is part of , another
> part is that the server is 10x slower than Apache.
>

Other measurements showed 3x slower, which makes it still worthwhile to
explore.
We need to be cautious here though since optimization is all about chasing
where time is actually spent.

If we can prove to ourselves that this is an important issue, we should
discuss with the WPT community to see how to fix this issue.
In addition to better caching, other optimization like
https://github.com/w3c/wptserve/pull/86 may bring some additional benefit.

If we do not find any good solution at WPT server level, we still have the
option to run some tests as file-based.
Ryosuke mentioned the possibility to classify tests at import time by
checking their results when served through WPT server and as file URLs.

I just tested on my MacBook Pro, and WPT tests took 23% of time while being
> only 9% of the total count. Taking in mind that WebKit own tests have
> higher value due to the way we choose what to test (see below), that's not
> a great story in my opinion.
>

These numbers are difficult to interpret.
WPT authoring style is multiple tests in a single file, which is bad for
stability but potentially good for performances.
WebKit usually prefers one file/one test.

If we want to talk to WPT community about this, we need to provide some
more tangible numbers.
We could try to run a large subset of WPT tests that run the same through
Apache and WPT.
I just did that on a small subset of IDB tests. This seems to show
something like a 25% slowdown.

One other thing that we discussed before was the operational complexity of
> running WPT tests. We frequently need to share tests with people who don't
> work on WebKit directly, but have the need to edit and run our tests.
> Inability to drag and drop a local copy into a Safari window is a deterrent
> to addressing problems caught by the tests. I think that the response we
> got was that tests will continue to require a server to run.
>

Tests are available at https://w3c-test.org which makes it easy to share
through any tool supporting hyperlinks.
A webarchive can also be made so that it is easy to share and probably edit
such tests.
Tools like jsfiddle are also a great way to create/share/edit tests.
I received several bug reports on bugzilla using it and this proved to be
efficient.


> Let me explain why I think that WebKit tests are often more valuable as
> regression tests than WPT tests are. We add tests as we fix bugs, so we
> know that the tests are generally for problems that have a high impact on
> users and developers - that's because someone actually discovered the
> problem, and someone prioritized it highly enough to fix. We also know that
> our tests cover code that is error prone, which is why we had bugs in the
> first place. Of course, anything can be broken, but certain things are less
> likely to. Compliance tests written for specs are also valuable, but at
> some point we need to prioritize which tests to investigate and even to run.
>

I don't really see why we should prioritize the tests to run when all of
them provide clear value to some WebKit members.
I agree that we need to prioritize tests we investigate. There can be a
solution inside WPT, like adding WebKit specific metadata so that:
- WPT contributors would communicate with WebKit members whenever changing
such tests
- WebKit contributors would prioritize WPT-WebKit-metadata failing tests

That said, if these tests are so beneficial to WebKit, they are potentially
very useful to other teams as well.
And vice-versa, we might find really good WPT tests that show useful
crashes and failures in WebKit.
I am experiencing that nowadays with WPT service worker tests.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[Alexey Proskuryakov]

(re-sending from a correct address)

> 17 нояб. 2017 г., в 9:18, youenn fablet  > написал(а):
> 
> Chris recently noticed that some heavily used files (testharness*) were 
> cacheable through Apache but not WPT.
> This is now fixed and should improve WPT performances.

This is part of >, another part is that the 
server is 10x slower than Apache.

I just tested on my MacBook Pro, and WPT tests took 23% of time while being 
only 9% of the total count. Taking in mind that WebKit own tests have higher 
value due to the way we choose what to test (see below), that's not a great 
story in my opinion.

One other thing that we discussed before was the operational complexity of 
running WPT tests. We frequently need to share tests with people who don't work 
on WebKit directly, but have the need to edit and run our tests. Inability to 
drag and drop a local copy into a Safari window is a deterrent to addressing 
problems caught by the tests. I think that the response we got was that tests 
will continue to require a server to run.

Let me explain why I think that WebKit tests are often more valuable as 
regression tests than WPT tests are. We add tests as we fix bugs, so we know 
that the tests are generally for problems that have a high impact on users and 
developers - that's because someone actually discovered the problem, and 
someone prioritized it highly enough to fix. We also know that our tests cover 
code that is error prone, which is why we had bugs in the first place. Of 
course, anything can be broken, but certain things are less likely to. 
Compliance tests written for specs are also valuable, but at some point we need 
to prioritize which tests to investigate and even to run.

- Alexey

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[Chris Dumez]

> On Nov 17, 2017, at 9:18 AM, youenn fablet  wrote:
> 
> Chris recently noticed that some heavily used files (testharness*) were 
> cacheable through Apache but not WPT.

Wasn’t me. Alexey reported this via Description 
.___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[youenn fablet]

>From my experience, the people behind WPT are quite responsive to WebKit
community.
Here are two examples:
Chris recently noticed that some heavily used files (testharness*) were
cacheable through Apache but not WPT.
This is now fixed and should improve WPT performances.
In January, was discussed the idea to automate WPT manual tests.
WebKit requirement to use test runner API instead of WebDriver was taken
into account.
I am pretty confident that WPT can accommodate other WebKit issues.

We have a lot of conformance-only tests in WebKit LayoutTests.
It would make a lot of sense to upstream them/replace them.
Having an up-to-date version and an obsolete version of the same test at
the same time is not appealing.
It makes writing a conformance-fixing patch longer and it takes more time
to run the test suite for no good reason.

Regression tests are of course more difficult to process and probably less
of a priority.  This was discussed quickly at TPAC.
Upstreaming such tests could be done as long as these tests are marked as
WebKit-specific and changes to them would require validation from WebKit
community.
Keeping history through git should be possible.

I would also like to mention that, from my personal experience, WPT tests
are of a high quality.
This probably comes from WPT nature which is focused just on tests and
developed a specific review process and validation process for the sole
purpose of tests.

y
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[Maciej Stachowiak]

> On Nov 17, 2017, at 7:53 AM, Frédéric WANG  wrote:
> 
> On 17/11/2017 16:26, Maciej Stachowiak wrote:
>> WebKit has a lot of tests that were regression tests for a specific
>> bug fix, not as conformance tests (though they might b useful for that
>> too). In such cases, the association with the bugs.webkit.org bug is
>> more important than the spec URL. That’s particularly the case when
>> the test has been changed multiple times to reflect further WebKit
>> behavior changes. I know that I’ve personally found it very useful to
>> look at revision history of tests, or search for bug numbers or
>> keywords in LayoutTests/ChangeLog to find related tests.
>> Not saying this is the sole purpose of tests, but losing this ability or 
>> making it more awkward would be a downside to deleting tests from the WebKit 
>> repo.
> Well, I think we agree here, that's why "conformance" tests was
> specified in my message. For such tests, connecting the history of tests
> with the development of the specs is actually more important. For
> example I recently noticed that some bugs with WOFF2 (on Apple's ports
> only) and WebVTT have been ignored in WebKit because we don't
> import/sync WPT tests for these specs.

I think importing new test suites is a different question than 
upstreaming/removing tests. In general importing more test suites is probably 
good, but we probably need to tackle the WPT performance problem before we pull 
in too many new WPT suites.

I should also note that pulling in the test suite won't automatically get the 
bugs fixed or prioritized, or even filed in the bug tracker 
necessarily.eratwerATWeRTWAerATWertwAerer

>> Maybe I’m missing something, but isn’t that already how it works in WebKit? 
>> EWS and buildbot run all the tests, but on your own machine you can get 
>> run-webkit-tests to run any subset you want.
> Not everybody has the hardware to produce debug builds of WebKit on all
> existing ports and run a big amount of tests :-)
> 
> AFAIK, EWS run all the builds as well as all the tests on macOS/iOS, you
> don't have options to select a subset of the tests or the ports.
> Buildbots run all the builds & tests on all platforms *after* a patch
> lands in the main repo. So we do too much for WIP patches (extra runtime
> cost I'm mentioning here) and not enough to check the final patch before
> landing (which causes different issues like extra rollout or gardening
> commits or difficult regression bisecting).

I find it super convenient that EWS runs all the tests even on WIP patches. It 
often catches test failures in tests I didn't think to run myself. I think it 
would be great if EWS could run tests on more platforms. Just running release 
regression tests on more platforms would be a big win, if debug builds are the 
gating factor.

I guess it would be kind of neat to have a feature of "run the tests across all 
configurations, but only some of them". But I don't think I would prioritize it 
over having more kinds of tests, or finding ways to make the full test suite 
run faster.

> 
> -- 
> Frédéric Wang - frederic-wang.fr
> 
> 

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[Philip Jägenstedt]

Hi Danyao,

I'm afraid that I don't yet have proper breakdown of this, but this is from
some quick grepping in WebKit:

   - 3200k *-expected.html outside of LayoutTests/imported, that might work
   as reftests.
   - 60k files that aren't called *-expected.*, again outside of
   LayoutTests/imported/
   - 765 of those have "testharness.js" somewhere.
   - 3800 of those have "internals." or "eventSender." somewhere

So, it's certainly not the case that there are tens of thousands of tests
that could easily be upstreamed. And no doubt tens of thousands of tests
that will best be left alone for a long time to come.

But, at the very least I think there will be some hundreds of tests that
could be contributed with some mechanical text transforms that don't
introduce much chance of regressing the tests.

On Wed, Nov 15, 2017 at 5:16 PM Danyao  wrote:

> Hi Philip,
>
> Ali and I would be willing to help this wherever we can. It seems to us
> that there may be a few different classes of tests that require varying
> degree of effort to migrate. Do you have any estimates on how many tests
> fall into each category, maybe based on your analysis of Blink LayoutTests?
>
>1. braindead to migrate
>- 1a) ref tests not using window.internals
>   - 1b) tests using internals API that have trivial equivalent in
>   testharness
>2. tests using non-trivial internals API so require some domain
>expertise in the functionality under test
>   - 2a) priority features: maybe active development so more likely to
>   diverge in the near future?
>   - 2b) long tail
>3. tests that can't be migrated to WPT
>   - 3a) tests that explicit use layerTreeAsText (tests in the
>   compositing layer)
>   - 3b) tests that dump render object tree (testing browser internals)
>
>
> Thanks,
> Danyao
>
> On Wed, Nov 15, 2017 at 5:02 AM, Philip Jägenstedt 
> wrote:
>
>> Hello webkit-dev! (ecosystem-infra in Bcc)
>>
>> TPAC was last week, and there was much talk about web-platform-tests.
>> Some notes are at
>> https://lists.w3.org/Archives/Public/public-test-infra/2017OctDec/thread.html
>>  and
>> in particular https://www.w3.org/2017/11/07-testing-minutes.html.
>>
>> In Blink, we're thinking seriously about what it'd take to upstream large
>> parts of LayoutTests into web-platform-tests, and we've realized that there
>> are some risks here, beyond just making sure the tests are good and per
>> spec. Specifically, one bad outcome would be if Blink successfully
>> upstreamed thousands of tests that are also in WebKit, which then begin to
>> diverge in small and large ways. That'd leave WebKit with more duplication
>> between its own tests and web-platform-tests than any other engine, and a
>> headache that grows over time.
>>
>> So, I think this would require close cooperation with the WebKit project.
>> Some different ways this might happen:
>>
>>1. A Blink developer and WebKit developer work together at the same
>>time to move their common tests in some area into web-platform-tests, each
>>removing identical tests that were upstreamed by the other and
>>consolidating what remains.
>>2. A Blink developer works to first upstream tests from WebKit (using
>>WebKit's coming export mechanism), waits for it to be imported into Blink,
>>and then removes the tests from Blink.
>>3. The reverse situation.
>>
>>
>> Are there already some areas where the first approach might be doable,
>> where the Blink and WebKit folks already know each other and are eager to
>> do this? Is LayoutTests/css3/flexbox/ by any chance such a case?
>>
>> I realize it's hard to judge these approaches in the abstract, but am
>> curious to hear any enthusiasm or concerns about it.
>>
>> Thanks!
>>
>> ___
>> webkit-dev mailing list
>> webkit-dev@lists.webkit.org
>> https://lists.webkit.org/mailman/listinfo/webkit-dev
>>
>>
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[Frédéric WANG]

On 17/11/2017 16:26, Maciej Stachowiak wrote:
> WebKit has a lot of tests that were regression tests for a specific
> bug fix, not as conformance tests (though they might b useful for that
> too). In such cases, the association with the bugs.webkit.org bug is
> more important than the spec URL. That’s particularly the case when
> the test has been changed multiple times to reflect further WebKit
> behavior changes. I know that I’ve personally found it very useful to
> look at revision history of tests, or search for bug numbers or
> keywords in LayoutTests/ChangeLog to find related tests.
> Not saying this is the sole purpose of tests, but losing this ability or 
> making it more awkward would be a downside to deleting tests from the WebKit 
> repo.
Well, I think we agree here, that's why "conformance" tests was
specified in my message. For such tests, connecting the history of tests
with the development of the specs is actually more important. For
example I recently noticed that some bugs with WOFF2 (on Apple's ports
only) and WebVTT have been ignored in WebKit because we don't
import/sync WPT tests for these specs.
> Maybe I’m missing something, but isn’t that already how it works in WebKit? 
> EWS and buildbot run all the tests, but on your own machine you can get 
> run-webkit-tests to run any subset you want.
Not everybody has the hardware to produce debug builds of WebKit on all
existing ports and run a big amount of tests :-)

AFAIK, EWS run all the builds as well as all the tests on macOS/iOS, you
don't have options to select a subset of the tests or the ports.
Buildbots run all the builds & tests on all platforms *after* a patch
lands in the main repo. So we do too much for WIP patches (extra runtime
cost I'm mentioning here) and not enough to check the final patch before
landing (which causes different issues like extra rollout or gardening
commits or difficult regression bisecting).

-- 
Frédéric Wang - frederic-wang.fr




signature.asc
Description: OpenPGP digital signature
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[Maciej Stachowiak]

> On Nov 17, 2017, at 7:05 AM, Frédéric WANG  wrote:
> 
> Hi Philip,
> 
> We (at Igalia) are strong supporters of WPT tests and we've appreciated a lot 
> the effort made by Youenn and others to make them possible in WebKit. They 
> are very important for interoperability and make life much easier for 
> developers working on different WebKit ports and different Web engines.
> 
> Ideally, we believe that future conformance tests in WebKit should be written 
> using the WPT format and submitted to web-platform-tests repository (of 
> course there are exceptions when the WPT API is too limited). WPT tests have 
> references to spec URLs and their own git history, so it should always be 
> possible to know why they have been written. This also reduces review cost on 
> WebKit's side, as test review can be done by experts in the relevant specs.

WebKit has a lot of tests that were regression tests for a specific bug fix, 
not as conformance tests (though they might b useful for that too). In such 
cases, the  association with the bugs.webkit.org bug is more important than the 
spec URL. That’s particularly the case when the test has been changed multiple 
times to reflect further WebKit behavior changes. I know that I’ve personally 
found it very useful to look at revision history of tests, or search for bug 
numbers or keywords in LayoutTests/ChangeLog to find related tests.

Not saying this is the sole purpose of tests, but losing this ability or making 
it more awkward would be a downside to deleting tests from the WebKit repo.

> 
> Probably it would be nice to convert old tests to WPT and check duplication 
> to avoid increasing runtime cost but that's indeed going to be a big effort. 
> It could also be possible to improve how the CI tests are executed in the 
> WebKit project, taking inspiration from Chromium or Mozilla projects (i.e. 
> run all tests before a patch lands in the main repo, but allowing devs to run 
> a relevant subset during development).

Maybe I’m missing something, but isn’t that already how it works in WebKit? EWS 
and buildbot run all the tests, but on your own machine you can get 
run-webkit-tests to run any subset you want.

> 
> In any case, Igalia definitely supports this proposal and is happy to help 
> where possible.
> 
> Frédéric, for Igalia's Web Platform Team
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[Konstantin Tokarev]

17.11.2017, 18:19, "Maciej Stachowiak" :
>>  On Nov 17, 2017, at 7:15 AM, Konstantin Tokarev  wrote:
>>
>>  16.11.2017, 20:10, "Alexey Proskuryakov" :
>>>  Migrating WebKit tests seems undesirable to me, as that would make us lose 
>>> all their modification history. As mentioned before, WPT tests are 
>>> inherently getting less attention in CI support, largely because of unclear 
>>> benefit when there isn't any history for why the particular test was added.
>>
>>  Possible workaround would be to keep original file path as a comment inside 
>> new file. In this case history of ex-WebKit tests can be easily retraced 
>> from WebKit repository.
>
> From Apple’s point of view, we are ok with leaving duplicate tests rather 
> than removing them, especially when they were created as regression tests for 
> specific bugs rather than as broad standards compliance tests.
>
>>>  Regression tests are taking an excessive amount of time to run for us 
>>> (more than 1.5 hours in a debug build), a very large proportion of which is 
>>> WPT. I do not think that the strategy of adopting WPT tests is working well 
>>> for WebKit.
>>
>>  If existing tests are converted to WPT, do they take more time to run? If 
>> yes, it's quite unfortunate and needs to be fixed in WPT infrastructure.
>
> I talked to Alexey bout this and he says there is indeed a higher per-test 
> time cost, and that it’s a likely fixable problem with the WPT server. If 
> anyone fixed this, EWS and buildbot turnaround time would improve 
> significantly.

Is it possible to run tests that don't require server directly, like it's done 
now for regression tests?

>
>>>  - Alexey
>>>
  15 нояб. 2017 г., в 2:02, Philip Jägenstedt  
 написал(а):

  Hello webkit-dev! (ecosystem-infra in Bcc)

  TPAC was last week, and there was much talk about web-platform-tests. 
 Some notes are at 
 https://lists.w3.org/Archives/Public/public-test-infra/2017OctDec/thread.html
  and in particular https://www.w3.org/2017/11/07-testing-minutes.html.

  In Blink, we're thinking seriously about what it'd take to upstream large 
 parts of LayoutTests into web-platform-tests, and we've realized that 
 there are some risks here, beyond just making sure the tests are good and 
 per spec. Specifically, one bad outcome would be if Blink successfully 
 upstreamed thousands of tests that are also in WebKit, which then begin to 
 diverge in small and large ways. That'd leave WebKit with more duplication 
 between its own tests and web-platform-tests than any other engine, and a 
 headache that grows over time.

  So, I think this would require close cooperation with the WebKit project. 
 Some different ways this might happen:

  * A Blink developer and WebKit developer work together at the same time 
 to move their common tests in some area into web-platform-tests, each 
 removing identical tests that were upstreamed by the other and 
 consolidating what remains.
  * A Blink developer works to first upstream tests from WebKit (using 
 WebKit's coming export mechanism), waits for it to be imported into Blink, 
 and then removes the tests from Blink.
  * The reverse situation.

  Are there already some areas where the first approach might be doable, 
 where the Blink and WebKit folks already know each other and are eager to 
 do this? Is LayoutTests/css3/flexbox/ by any chance such a case?

  I realize it's hard to judge these approaches in the abstract, but am 
 curious to hear any enthusiasm or concerns about it.

  Thanks!
  ___
  webkit-dev mailing list
  webkit-dev@lists.webkit.org
  https://lists.webkit.org/mailman/listinfo/webkit-dev
>>>
>>>  ,
>>>
>>>  ___
>>>  webkit-dev mailing list
>>>  webkit-dev@lists.webkit.org
>>>  https://lists.webkit.org/mailman/listinfo/webkit-dev
>>
>>  --
>>  Regards,
>>  Konstantin
>>  ___
>>  webkit-dev mailing list
>>  webkit-dev@lists.webkit.org
>>  https://lists.webkit.org/mailman/listinfo/webkit-dev

-- 
Regards,
Konstantin
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[Michael Catanzaro]

On Thu, Nov 16, 2017 at 11:10 AM, Alexey Proskuryakov  
wrote:
Regression tests are taking an excessive amount of time to run for us 
(more than 1.5 hours in a debug build), a very large proportion of 
which is WPT. I do not think that the strategy of adopting WPT tests 
is working well for WebKit.


Perhaps treating WPT tests as regression tests is a mistake. I suspect 
the real value of these tests comes from using them as conformance 
tests: treating FAILs as test failures rather than committing them to 
the expected results, and working to reduce the number of test 
failures. That's not to say that regression testing is not useful... 
just that we're not currently getting full value from the WPT tests.


Adapting WebKit's behavior to more closely conform to the behavior of 
other web engines should reduce the number of web compatibility issues 
that negatively affect WebKit users. The benefits to that are clear, 
and improving our WPT passrate seems like the best way to achieve it.


My $0.02,

Michael (who is not volunteering to do any of this work, and whose 
opinion probably doesn't count for much here)


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[Maciej Stachowiak]

> On Nov 17, 2017, at 7:15 AM, Konstantin Tokarev  wrote:
> 
> 
> 
> 16.11.2017, 20:10, "Alexey Proskuryakov" :
>> Migrating WebKit tests seems undesirable to me, as that would make us lose 
>> all their modification history. As mentioned before, WPT tests are 
>> inherently getting less attention in CI support, largely because of unclear 
>> benefit when there isn't any history for why the particular test was added.
> 
> Possible workaround would be to keep original file path as a comment inside 
> new file. In this case history of ex-WebKit tests can be easily retraced from 
> WebKit repository.

From Apple’s point of view, we are ok with leaving duplicate tests rather than 
removing them, especially when they were created as regression tests for 
specific bugs rather than as broad standards compliance tests.

>> Regression tests are taking an excessive amount of time to run for us (more 
>> than 1.5 hours in a debug build), a very large proportion of which is WPT. I 
>> do not think that the strategy of adopting WPT tests is working well for 
>> WebKit.
> 
> If existing tests are converted to WPT, do they take more time to run? If 
> yes, it's quite unfortunate and needs to be fixed in WPT infrastructure.

I talked to Alexey bout this and he says there is indeed a higher per-test time 
cost, and that it’s a likely fixable problem with the WPT server. If anyone 
fixed this, EWS and buildbot turnaround time would improve significantly.

> 
>> 
>> - Alexey
>> 
>>> 15 нояб. 2017 г., в 2:02, Philip Jägenstedt  
>>> написал(а):
>>> 
>>> Hello webkit-dev! (ecosystem-infra in Bcc)
>>> 
>>> TPAC was last week, and there was much talk about web-platform-tests. Some 
>>> notes are at 
>>> https://lists.w3.org/Archives/Public/public-test-infra/2017OctDec/thread.html
>>>  and in particular https://www.w3.org/2017/11/07-testing-minutes.html.
>>> 
>>> In Blink, we're thinking seriously about what it'd take to upstream large 
>>> parts of LayoutTests into web-platform-tests, and we've realized that there 
>>> are some risks here, beyond just making sure the tests are good and per 
>>> spec. Specifically, one bad outcome would be if Blink successfully 
>>> upstreamed thousands of tests that are also in WebKit, which then begin to 
>>> diverge in small and large ways. That'd leave WebKit with more duplication 
>>> between its own tests and web-platform-tests than any other engine, and a 
>>> headache that grows over time.
>>> 
>>> So, I think this would require close cooperation with the WebKit project. 
>>> Some different ways this might happen:
>>> 
>>> * A Blink developer and WebKit developer work together at the same time to 
>>> move their common tests in some area into web-platform-tests, each removing 
>>> identical tests that were upstreamed by the other and consolidating what 
>>> remains.
>>> * A Blink developer works to first upstream tests from WebKit (using 
>>> WebKit's coming export mechanism), waits for it to be imported into Blink, 
>>> and then removes the tests from Blink.
>>> * The reverse situation.
>>> 
>>> Are there already some areas where the first approach might be doable, 
>>> where the Blink and WebKit folks already know each other and are eager to 
>>> do this? Is LayoutTests/css3/flexbox/ by any chance such a case?
>>> 
>>> I realize it's hard to judge these approaches in the abstract, but am 
>>> curious to hear any enthusiasm or concerns about it.
>>> 
>>> Thanks!
>>> ___
>>> webkit-dev mailing list
>>> webkit-dev@lists.webkit.org
>>> https://lists.webkit.org/mailman/listinfo/webkit-dev
>> 
>> ,
>> 
>> ___
>> webkit-dev mailing list
>> webkit-dev@lists.webkit.org
>> https://lists.webkit.org/mailman/listinfo/webkit-dev
> 
> 
> -- 
> Regards,
> Konstantin
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[Konstantin Tokarev]

16.11.2017, 20:10, "Alexey Proskuryakov" :
> Migrating WebKit tests seems undesirable to me, as that would make us lose 
> all their modification history. As mentioned before, WPT tests are inherently 
> getting less attention in CI support, largely because of unclear benefit when 
> there isn't any history for why the particular test was added.

Possible workaround would be to keep original file path as a comment inside new 
file. In this case history of ex-WebKit tests can be easily retraced from 
WebKit repository.

>
> Regression tests are taking an excessive amount of time to run for us (more 
> than 1.5 hours in a debug build), a very large proportion of which is WPT. I 
> do not think that the strategy of adopting WPT tests is working well for 
> WebKit.

If existing tests are converted to WPT, do they take more time to run? If yes, 
it's quite unfortunate and needs to be fixed in WPT infrastructure.

>
> - Alexey
>
>> 15 нояб. 2017 г., в 2:02, Philip Jägenstedt  написал(а):
>>
>> Hello webkit-dev! (ecosystem-infra in Bcc)
>>
>> TPAC was last week, and there was much talk about web-platform-tests. Some 
>> notes are at 
>> https://lists.w3.org/Archives/Public/public-test-infra/2017OctDec/thread.html
>>  and in particular https://www.w3.org/2017/11/07-testing-minutes.html.
>>
>> In Blink, we're thinking seriously about what it'd take to upstream large 
>> parts of LayoutTests into web-platform-tests, and we've realized that there 
>> are some risks here, beyond just making sure the tests are good and per 
>> spec. Specifically, one bad outcome would be if Blink successfully 
>> upstreamed thousands of tests that are also in WebKit, which then begin to 
>> diverge in small and large ways. That'd leave WebKit with more duplication 
>> between its own tests and web-platform-tests than any other engine, and a 
>> headache that grows over time.
>>
>> So, I think this would require close cooperation with the WebKit project. 
>> Some different ways this might happen:
>>
>> * A Blink developer and WebKit developer work together at the same time to 
>> move their common tests in some area into web-platform-tests, each removing 
>> identical tests that were upstreamed by the other and consolidating what 
>> remains.
>> * A Blink developer works to first upstream tests from WebKit (using 
>> WebKit's coming export mechanism), waits for it to be imported into Blink, 
>> and then removes the tests from Blink.
>> * The reverse situation.
>>
>> Are there already some areas where the first approach might be doable, where 
>> the Blink and WebKit folks already know each other and are eager to do this? 
>> Is LayoutTests/css3/flexbox/ by any chance such a case?
>>
>> I realize it's hard to judge these approaches in the abstract, but am 
>> curious to hear any enthusiasm or concerns about it.
>>
>> Thanks!
>> ___
>> webkit-dev mailing list
>> webkit-dev@lists.webkit.org
>> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
> ,
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev


-- 
Regards,
Konstantin
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[Frédéric WANG]

Hi Philip,

We (at Igalia) are strong supporters of WPT tests and we've appreciated
a lot the effort made by Youenn and others to make them possible in
WebKit. They are very important for interoperability and make life much
easier for developers working on different WebKit ports and different
Web engines.

Ideally, we believe that future conformance tests in WebKit should be
written using the WPT format and submitted to web-platform-tests
repository (of course there are exceptions when the WPT API is too
limited). WPT tests have references to spec URLs and their own git
history, so it should always be possible to know why they have been
written. This also reduces review cost on WebKit's side, as test review
can be done by experts in the relevant specs.

Probably it would be nice to convert old tests to WPT and check
duplication to avoid increasing runtime cost but that's indeed going to
be a big effort. It could also be possible to improve how the CI tests
are executed in the WebKit project, taking inspiration from Chromium or
Mozilla projects (i.e. run all tests before a patch lands in the main
repo, but allowing devs to run a relevant subset during development).

In any case, Igalia definitely supports this proposal and is happy to
help where possible.

Frédéric, for Igalia's Web Platform Team


signature.asc
Description: OpenPGP digital signature
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[Alexey Proskuryakov]

Migrating WebKit tests seems undesirable to me, as that would make us lose all 
their modification history. As mentioned before, WPT tests are inherently 
getting less attention in CI support, largely because of unclear benefit when 
there isn't any history for why the particular test was added.

Regression tests are taking an excessive amount of time to run for us (more 
than 1.5 hours in a debug build), a very large proportion of which is WPT. I do 
not think that the strategy of adopting WPT tests is working well for WebKit.

- Alexey


> 15 нояб. 2017 г., в 2:02, Philip Jägenstedt  написал(а):
> 
> Hello webkit-dev! (ecosystem-infra in Bcc)
> 
> TPAC was last week, and there was much talk about web-platform-tests. Some 
> notes are at 
> https://lists.w3.org/Archives/Public/public-test-infra/2017OctDec/thread.html 
> 
>  and in particular https://www.w3.org/2017/11/07-testing-minutes.html 
> .
> 
> In Blink, we're thinking seriously about what it'd take to upstream large 
> parts of LayoutTests into web-platform-tests, and we've realized that there 
> are some risks here, beyond just making sure the tests are good and per spec. 
> Specifically, one bad outcome would be if Blink successfully upstreamed 
> thousands of tests that are also in WebKit, which then begin to diverge in 
> small and large ways. That'd leave WebKit with more duplication between its 
> own tests and web-platform-tests than any other engine, and a headache that 
> grows over time.
> 
> So, I think this would require close cooperation with the WebKit project. 
> Some different ways this might happen:
> A Blink developer and WebKit developer work together at the same time to move 
> their common tests in some area into web-platform-tests, each removing 
> identical tests that were upstreamed by the other and consolidating what 
> remains.
> A Blink developer works to first upstream tests from WebKit (using WebKit's 
> coming export mechanism), waits for it to be imported into Blink, and then 
> removes the tests from Blink.
> The reverse situation.
> 
> Are there already some areas where the first approach might be doable, where 
> the Blink and WebKit folks already know each other and are eager to do this? 
> Is LayoutTests/css3/flexbox/ by any chance such a case?
> 
> I realize it's hard to judge these approaches in the abstract, but am curious 
> to hear any enthusiasm or concerns about it.
> 
> Thanks!
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[Danyao]

Hi Philip,

Ali and I would be willing to help this wherever we can. It seems to us
that there may be a few different classes of tests that require varying
degree of effort to migrate. Do you have any estimates on how many tests
fall into each category, maybe based on your analysis of Blink LayoutTests?

   1. braindead to migrate
   - 1a) ref tests not using window.internals
  - 1b) tests using internals API that have trivial equivalent in
  testharness
   2. tests using non-trivial internals API so require some domain
   expertise in the functionality under test
  - 2a) priority features: maybe active development so more likely to
  diverge in the near future?
  - 2b) long tail
   3. tests that can't be migrated to WPT
  - 3a) tests that explicit use layerTreeAsText (tests in the
  compositing layer)
  - 3b) tests that dump render object tree (testing browser internals)


Thanks,
Danyao

On Wed, Nov 15, 2017 at 5:02 AM, Philip Jägenstedt 
wrote:

> Hello webkit-dev! (ecosystem-infra in Bcc)
>
> TPAC was last week, and there was much talk about web-platform-tests. Some
> notes are at https://lists.w3.org/Archives/Public/public-test-
> infra/2017OctDec/thread.html and in particular https://www.w3.org/
> 2017/11/07-testing-minutes.html.
>
> In Blink, we're thinking seriously about what it'd take to upstream large
> parts of LayoutTests into web-platform-tests, and we've realized that there
> are some risks here, beyond just making sure the tests are good and per
> spec. Specifically, one bad outcome would be if Blink successfully
> upstreamed thousands of tests that are also in WebKit, which then begin to
> diverge in small and large ways. That'd leave WebKit with more duplication
> between its own tests and web-platform-tests than any other engine, and a
> headache that grows over time.
>
> So, I think this would require close cooperation with the WebKit project.
> Some different ways this might happen:
>
>1. A Blink developer and WebKit developer work together at the same
>time to move their common tests in some area into web-platform-tests, each
>removing identical tests that were upstreamed by the other and
>consolidating what remains.
>2. A Blink developer works to first upstream tests from WebKit (using
>WebKit's coming export mechanism), waits for it to be imported into Blink,
>and then removes the tests from Blink.
>3. The reverse situation.
>
>
> Are there already some areas where the first approach might be doable,
> where the Blink and WebKit folks already know each other and are eager to
> do this? Is LayoutTests/css3/flexbox/ by any chance such a case?
>
> I realize it's hard to judge these approaches in the abstract, but am
> curious to hear any enthusiasm or concerns about it.
>
> Thanks!
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev