Re: [webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

2017-11-28 Thread Alexey Proskuryakov 

> 17 нояб. 2017 г., в 23:02, youenn fablet  написал(а):
> 
> Tests are available at https://w3c-test.org  which 
> makes it easy to share through any tool supporting hyperlinks.
> A webarchive can also be made so that it is easy to share and probably edit 
> such tests.
> Tools like jsfiddle are also a great way to create/share/edit tests.
> I received several bug reports on bugzilla using it and this proved to be 
> efficient.

I don't think that these are applicable verbatim, but there may be some 
solution. Let's find it before assuming that we can have it.

In particular, 
- editing a WebArchive is not really feasible;
- jsfiddle is cool, but I can't just copy a WPT test to jsfiddle to share it.

> Let me explain why I think that WebKit tests are often more valuable as 
> regression tests than WPT tests are. We add tests as we fix bugs, so we know 
> that the tests are generally for problems that have a high impact on users 
> and developers - that's because someone actually discovered the problem, and 
> someone prioritized it highly enough to fix. We also know that our tests 
> cover code that is error prone, which is why we had bugs in the first place. 
> Of course, anything can be broken, but certain things are less likely to. 
> Compliance tests written for specs are also valuable, but at some point we 
> need to prioritize which tests to investigate and even to run.
>  
> I don't really see why we should prioritize the tests to run when all of them 
> provide clear value to some WebKit members.

We prioritize which tests to run all the time, there are whole configurations 
for which we don't run any tests. That's largely because it's not enough to 
simply run the tests - keeping them in a state where they produce actionable 
results requires a lot of attention.

The time it takes to run tests definitely matters a lot for WebKit. So far we 
haven't taken a route like the one Robert mentioned, with a huge amount of 
hardware running shards of tests. There are multiple reasons to that, one of 
those being that we are very interested in finding bugs below WebKit too, and 
having dedicated testers helps with that. When the machine gets into a weird 
state after a few weeks of uptime, it is much easier to start isolating the 
problem when we can see which specific queues are hitting it. It is also much 
easier to reason about data collected by centralized systems, such as crash log 
collection services.

- Alexey


> I agree that we need to prioritize tests we investigate. There can be a 
> solution inside WPT, like adding WebKit specific metadata so that:
> - WPT contributors would communicate with WebKit members whenever changing 
> such tests
> - WebKit contributors would prioritize WPT-WebKit-metadata failing tests
> 
> That said, if these tests are so beneficial to WebKit, they are potentially 
> very useful to other teams as well.
> And vice-versa, we might find really good WPT tests that show useful crashes 
> and failures in WebKit.
> I am experiencing that nowadays with WPT service worker tests.


___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

2017-11-28 Thread Frédéric WANG 
On 28/11/2017 16:24, Philip Jägenstedt wrote:
>
> To do the upstreaming from WebKit first would get around this problem,
> and is something I could see us doing. If a Chromium developer
> upstreamed tests from WebKit and got review from a WebKit reviewer, I
> think we would rubberstamp the corresponding removal in Chromium.
>
> But that would be a futile exercise if the WebKit reviewers don't want
> to delete the tests from WebKit. The only remaining options would then
> be not sharing the tests with EdgeHTML and Gecko, or creating more
> duplication for WebKit.
Note that we can also have duplication in WebKit but for each port
decide which version will be executed by adding "skip" tags in
TestExpectation. Not ideal in my opinion, but that would allow e.g.
GTK/WPE ports to run the latest WPT tests while macOS/iOS ports keep the
legacy LayoutTests/ versions / history.

-- 
Frédéric Wang - frederic-wang.fr



signature.asc
Description: OpenPGP digital signature
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

2017-11-28 Thread Philip Jägenstedt 
On Tue, Nov 28, 2017 at 3:37 PM Ryosuke Niwa  wrote:

> On Mon, Nov 27, 2017 at 5:58 PM, Philip Jägenstedt 
> wrote:
>
>> (From Nov 24, I used the wrong email, resending for the archives.)
>>
>> On Sat, Nov 18, 2017 at 8:02 AM, youenn fablet  wrote:
>> > Thanks for taking the time to share this additional information.
>> > I think this is helpful to make progress.
>> > Please see inline for some more comments related to the points you are
>> > bringing to the table.
>> >
>> > Stepping back from the WPT server specific issue, being optimistic and
>> > assuming that we are able to run WPT tests with good enough
>> performances.
>> > Migration potential downsides I heard so far:
>> > - Prioritization of tests to investigate might be harder.
>> > - Sharing tests with other teams might be harder if subresource links
>> are
>> > not relative.
>> > These two seem like solvable issues to me.
>> >
>> > Le ven. 17 nov. 2017 à 10:09, Alexey Proskuryakov  a
>> écrit :
>> >>
>> >>
>> >> 17 нояб. 2017 г., в 9:18, youenn fablet 
>> написал(а):
>> >>
>> >>
>> >> Chris recently noticed that some heavily used files (testharness*) were
>> >> cacheable through Apache but not WPT.
>> >> This is now fixed and should improve WPT performances.
>> >>
>> >>
>> >> This is part of ,
>> another
>> >> part is that the server is 10x slower than Apache.
>> >
>> >
>> > Other measurements showed 3x slower, which makes it still worthwhile to
>> > explore.
>> > We need to be cautious here though since optimization is all about
>> chasing
>> > where time is actually spent.
>> >
>> > If we can prove to ourselves that this is an important issue, we should
>> > discuss with the WPT community to see how to fix this issue.
>> > In addition to better caching, other optimization like
>> > https://github.com/w3c/wptserve/pull/86 may bring some additional
>> benefit.
>> >
>> > If we do not find any good solution at WPT server level, we still have
>> the
>> > option to run some tests as file-based.
>> > Ryosuke mentioned the possibility to classify tests at import time by
>> > checking their results when served through WPT server and as file URLs.
>>
>> Thanks for filing https://github.com/w3c/web-platform-tests/issues/8391,
>> Youenn!
>>
>> I've put a placeholder around this in our planning for next quarter.
>> At a minimum, I'd like to investigate this and see how much slower the
>> tests are in Chromium's infra than they would be without wptserve, and
>> where that extra time is spent. But the reasons may not be the same in
>> WebKit, so please don't count on any improvement :)
>>
>> >> I just tested on my MacBook Pro, and WPT tests took 23% of time while
>> >> being only 9% of the total count. Taking in mind that WebKit own tests
>> have
>> >> higher value due to the way we choose what to test (see below), that's
>> not a
>> >> great story in my opinion.
>> >
>> >
>> > These numbers are difficult to interpret.
>> > WPT authoring style is multiple tests in a single file, which is bad for
>> > stability but potentially good for performances.
>> > WebKit usually prefers one file/one test.
>> >
>> > If we want to talk to WPT community about this, we need to provide some
>> more
>> > tangible numbers.
>> > We could try to run a large subset of WPT tests that run the same
>> through
>> > Apache and WPT.
>> > I just did that on a small subset of IDB tests. This seems to show
>> something
>> > like a 25% slowdown.
>> >
>> >> One other thing that we discussed before was the operational
>> complexity of
>> >> running WPT tests. We frequently need to share tests with people who
>> don't
>> >> work on WebKit directly, but have the need to edit and run our tests.
>> >> Inability to drag and drop a local copy into a Safari window is a
>> deterrent
>> >> to addressing problems caught by the tests. I think that the response
>> we got
>> >> was that tests will continue to require a server to run.
>> >
>> >
>> > Tests are available at https://w3c-test.org which makes it easy to
>> share
>> > through any tool supporting hyperlinks.
>> > A webarchive can also be made so that it is easy to share and probably
>> edit
>> > such tests.
>> > Tools like jsfiddle are also a great way to create/share/edit tests.
>> > I received several bug reports on bugzilla using it and this proved to
>> be
>> > efficient.
>> >
>> >>
>> >> Let me explain why I think that WebKit tests are often more valuable as
>> >> regression tests than WPT tests are. We add tests as we fix bugs, so
>> we know
>> >> that the tests are generally for problems that have a high impact on
>> users
>> >> and developers - that's because someone actually discovered the
>> problem, and
>> >> someone prioritized it highly enough to fix. We also know that our
>> tests
>> >> cover code that is error prone, which is why we had bugs in the first
>> place.
>> >> Of course, anything

Re: [webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

2017-11-28 Thread Ryosuke Niwa 
On Mon, Nov 27, 2017 at 5:58 PM, Philip Jägenstedt 
wrote:

> (From Nov 24, I used the wrong email, resending for the archives.)
>
> On Sat, Nov 18, 2017 at 8:02 AM, youenn fablet  wrote:
> > Thanks for taking the time to share this additional information.
> > I think this is helpful to make progress.
> > Please see inline for some more comments related to the points you are
> > bringing to the table.
> >
> > Stepping back from the WPT server specific issue, being optimistic and
> > assuming that we are able to run WPT tests with good enough performances.
> > Migration potential downsides I heard so far:
> > - Prioritization of tests to investigate might be harder.
> > - Sharing tests with other teams might be harder if subresource links are
> > not relative.
> > These two seem like solvable issues to me.
> >
> > Le ven. 17 nov. 2017 à 10:09, Alexey Proskuryakov  a
> écrit :
> >>
> >>
> >> 17 нояб. 2017 г., в 9:18, youenn fablet  написал(а):
> >>
> >>
> >> Chris recently noticed that some heavily used files (testharness*) were
> >> cacheable through Apache but not WPT.
> >> This is now fixed and should improve WPT performances.
> >>
> >>
> >> This is part of ,
> another
> >> part is that the server is 10x slower than Apache.
> >
> >
> > Other measurements showed 3x slower, which makes it still worthwhile to
> > explore.
> > We need to be cautious here though since optimization is all about
> chasing
> > where time is actually spent.
> >
> > If we can prove to ourselves that this is an important issue, we should
> > discuss with the WPT community to see how to fix this issue.
> > In addition to better caching, other optimization like
> > https://github.com/w3c/wptserve/pull/86 may bring some additional
> benefit.
> >
> > If we do not find any good solution at WPT server level, we still have
> the
> > option to run some tests as file-based.
> > Ryosuke mentioned the possibility to classify tests at import time by
> > checking their results when served through WPT server and as file URLs.
>
> Thanks for filing https://github.com/w3c/web-platform-tests/issues/8391,
> Youenn!
>
> I've put a placeholder around this in our planning for next quarter.
> At a minimum, I'd like to investigate this and see how much slower the
> tests are in Chromium's infra than they would be without wptserve, and
> where that extra time is spent. But the reasons may not be the same in
> WebKit, so please don't count on any improvement :)
>
> >> I just tested on my MacBook Pro, and WPT tests took 23% of time while
> >> being only 9% of the total count. Taking in mind that WebKit own tests
> have
> >> higher value due to the way we choose what to test (see below), that's
> not a
> >> great story in my opinion.
> >
> >
> > These numbers are difficult to interpret.
> > WPT authoring style is multiple tests in a single file, which is bad for
> > stability but potentially good for performances.
> > WebKit usually prefers one file/one test.
> >
> > If we want to talk to WPT community about this, we need to provide some
> more
> > tangible numbers.
> > We could try to run a large subset of WPT tests that run the same through
> > Apache and WPT.
> > I just did that on a small subset of IDB tests. This seems to show
> something
> > like a 25% slowdown.
> >
> >> One other thing that we discussed before was the operational complexity
> of
> >> running WPT tests. We frequently need to share tests with people who
> don't
> >> work on WebKit directly, but have the need to edit and run our tests.
> >> Inability to drag and drop a local copy into a Safari window is a
> deterrent
> >> to addressing problems caught by the tests. I think that the response
> we got
> >> was that tests will continue to require a server to run.
> >
> >
> > Tests are available at https://w3c-test.org which makes it easy to share
> > through any tool supporting hyperlinks.
> > A webarchive can also be made so that it is easy to share and probably
> edit
> > such tests.
> > Tools like jsfiddle are also a great way to create/share/edit tests.
> > I received several bug reports on bugzilla using it and this proved to be
> > efficient.
> >
> >>
> >> Let me explain why I think that WebKit tests are often more valuable as
> >> regression tests than WPT tests are. We add tests as we fix bugs, so we
> know
> >> that the tests are generally for problems that have a high impact on
> users
> >> and developers - that's because someone actually discovered the
> problem, and
> >> someone prioritized it highly enough to fix. We also know that our tests
> >> cover code that is error prone, which is why we had bugs in the first
> place.
> >> Of course, anything can be broken, but certain things are less likely
> to.
> >> Compliance tests written for specs are also valuable, but at some point
> we
> >> need to prioritize which tests to investigate and even to run.
> >
>