Re: [webkit-dev] Some text about the B3 compiler

[Carlos Alberto Lopez Perez]

On 02/02/16 00:27, Ryosuke Niwa wrote:
> On Mon, Feb 1, 2016 at 7:32 AM, Carlos Alberto Lopez Perez
>  wrote:
>> On 31/01/16 05:16, Filip Pizlo wrote:
>>> As my original message said, I was wondering if there was any support
>>> for running some JavaScript tests *in browser*.  run-jsc-stress-tests
>>> doesn’t support that because it doesn’t know what a browser is.
>>>
>>> Some tests, like Kraken, Octane, JetStream, and Speedometer, either
>>> require a browser to run (like JetStream and Speedometer) or have
>>> significantly different behavior in the browser than in their
>>> command-line harnesses (like Kraken and Octane).  If you did have a
>>> bot that ran these tests in some GTK+ or EFL browser, you’d probably
>>> catch bugs that testing the JSC shell cannot catch.
>>>
>>> -Filip
>>
>> Some questions regarding this JSC in-browser benchmarks:
>>
>> How does the Apple port runs this? Do you use some script that is
>> currently available on the WebKit source tree? Are the buildbots running
>> this tests public?
> 
> We've been running those benchmarks with
> http://trac.webkit.org/browser/trunk/Tools/Scripts/run-benchmark
> 

I see.

But this script seems focused on comparing the performance between
different browsers (safari vs chrome vs firefox) rather than in testing
and comparing the performance between different revisions of WebKit.

Do you think it makes any difference (from the point of view of
detecting failures, not from the performance PoV) to run this tests in a
full-fledged browser like Safari rather than in WebKitTestRunner ?

We already have a performance test bot running tests inside WTR.
And I see that the current set of tests executed on this bot already
includes Speedometer, and that JetStream and Sunspider are skipped on
PerformanceTests/Skipped.

So I see some options going forward:

 - Fix the JetStream and Sunspider tests so they can be run as part of
the current run-perf-tests script that the performance bots execute.

 - Implement support on the script run-benchmark to run the tests inside
WTR, and create a new step running this script that will be executed on
the test bots.

 - Deploy a new bot that runs run-perf-tests on a full-fledged browser
like Safari or Epiphany.

I wonder what you think is the best option or if there is some option
not viable.

From my PoV, the option #1 has the advantage of reusing the current
infrastructure that collects and draws performance data at
https://perf.webkit.org

> It might be useful for Mac and other ports to have bots available on
> build.webkit.org as well.
> 

Indeed.



signature.asc
Description: OpenPGP digital signature
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Some text about the B3 compiler

[Ryosuke Niwa]

On Tue, Feb 2, 2016 at 10:42 AM, Carlos Alberto Lopez Perez
 wrote:
> On 02/02/16 00:27,
- R. Niwa
 wrote:
>> On Mon, Feb 1, 2016 at 7:32 AM, Carlos Alberto Lopez Perez
>>  wrote:
>>> On 31/01/16 05:16, Filip Pizlo wrote:
 As my original message said, I was wondering if there was any support
 for running some JavaScript tests *in browser*.  run-jsc-stress-tests
 doesn’t support that because it doesn’t know what a browser is.

 Some tests, like Kraken, Octane, JetStream, and Speedometer, either
 require a browser to run (like JetStream and Speedometer) or have
 significantly different behavior in the browser than in their
 command-line harnesses (like Kraken and Octane).  If you did have a
 bot that ran these tests in some GTK+ or EFL browser, you’d probably
 catch bugs that testing the JSC shell cannot catch.

 -Filip
>>>
>>> Some questions regarding this JSC in-browser benchmarks:
>>>
>>> How does the Apple port runs this? Do you use some script that is
>>> currently available on the WebKit source tree? Are the buildbots running
>>> this tests public?
>>
>> We've been running those benchmarks with
>> http://trac.webkit.org/browser/trunk/Tools/Scripts/run-benchmark
>>
>
> I see.
>
> But this script seems focused on comparing the performance between
> different browsers (safari vs chrome vs firefox) rather than in testing
> and comparing the performance between different revisions of WebKit.

Not at all.  It simply supports running benchmark in other browsers.

> Do you think it makes any difference (from the point of view of
> detecting failures, not from the performance PoV) to run this tests in a
> full-fledged browser like Safari rather than in WebKitTestRunner ?

Yes. There are many browser features that can significantly impact the
real world performance.

> We already have a performance test bot running tests inside WTR.
> And I see that the current set of tests executed on this bot already
> includes Speedometer, and that JetStream and Sunspider are skipped on
> PerformanceTests/Skipped.
>
> So I see some options going forward:
>
>  - Fix the JetStream and Sunspider tests so they can be run as part of
> the current run-perf-tests script that the performance bots execute.

We should use run-benchmark instead since run-benchmark spits out the
JSON file that's compatible with run-pref-tests.

>  - Implement support on the script run-benchmark to run the tests inside
> WTR, and create a new step running this script that will be executed on
> the test bots.

I don't see a point in doing this.   Why is it desirable to run these
benchmarks inside WebKitTestRunner?

>  - Deploy a new bot that runs run-perf-tests on a full-fledged browser
> like Safari or Epiphany.

We should just do this.

> I wonder what you think is the best option or if there is some option
> not viable.
>
> From my PoV, the option #1 has the advantage of reusing the current
> infrastructure that collects and draws performance data at
> https://perf.webkit.org

We have an internal instance of the same dashboard to which we're
reporting results of run-benchmark script.

- R. Niwa
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Some text about the B3 compiler

[Filip Pizlo]

> On Feb 2, 2016, at 4:56 PM, Carlos Alberto Lopez Perez  
> wrote:
> 
> On 02/02/16 19:58, Ryosuke Niwa wrote:
>> On Tue, Feb 2, 2016 at 10:42 AM, Carlos Alberto Lopez Perez
>>> But this script seems focused on comparing the performance between
>>> different browsers (safari vs chrome vs firefox) rather than in testing
>>> and comparing the performance between different revisions of WebKit.
>> 
>> Not at all.  It simply supports running benchmark in other browsers.
>> 
>>> Do you think it makes any difference (from the point of view of
>>> detecting failures, not from the performance PoV) to run this tests in a
>>> full-fledged browser like Safari rather than in WebKitTestRunner ?
>> 
>> Yes. There are many browser features that can significantly impact the
>> real world performance.
>> 
> 
> I'm specifically not asking about performance, but about correctness.
> 
> This discussion was started because Filip said that running JS tests on
> a browser catches many failures that are not cached when running the
> tests from the terminal.
> 
> So, I'm wondering if running the JS tests on WTR or Safari makes any
> difference when catching failures.

I suspect that the browser will catch more failures.

> 
>>> We already have a performance test bot running tests inside WTR.
>>> And I see that the current set of tests executed on this bot already
>>> includes Speedometer, and that JetStream and Sunspider are skipped on
>>> PerformanceTests/Skipped.
>>> 
>>> So I see some options going forward:
>>> 
>>> - Fix the JetStream and Sunspider tests so they can be run as part of
>>> the current run-perf-tests script that the performance bots execute.
>> 
>> We should use run-benchmark instead since run-benchmark spits out the
>> JSON file that's compatible with run-pref-tests.
>> 
> 
> I'm a bit lost here. Are you planning to deprecate run-perf-tests with
> run-benchmark? What is wrong with run-perf-tests?
> 
>>> - Implement support on the script run-benchmark to run the tests inside
>>> WTR, and create a new step running this script that will be executed on
>>> the test bots.
>> 
>> I don't see a point in doing this.   Why is it desirable to run these
>> benchmarks inside WebKitTestRunner?
>> 
> 
> Less dependencies: WTR (or the MiniBrowser) is something that is
> currently built by the bots on each build.
> If we want to use Epiphany (for example) for the performance tests, is
> another thing we have to take care of building before each run. Not a
> big deal, but I wonder if is really worth.
> 
>>> - Deploy a new bot that runs run-perf-tests on a full-fledged browser
>>> like Safari or Epiphany.
>> 
>> We should just do this.
>> 
>>> I wonder what you think is the best option or if there is some option
>>> not viable.
>>> 
>>> From my PoV, the option #1 has the advantage of reusing the current
>>> infrastructure that collects and draws performance data at
>>> https://perf.webkit.org
>> 
>> We have an internal instance of the same dashboard to which we're
>> reporting results of run-benchmark script.
>> 
> 
> What about making this public? We will happily contribute with a
> GTK+/Linux buildbot for it.
> 
> 
> Regards.
> 
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev

___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev

Re: [webkit-dev] Some text about the B3 compiler

[Carlos Alberto Lopez Perez]

On 02/02/16 19:58, Ryosuke Niwa wrote:
> On Tue, Feb 2, 2016 at 10:42 AM, Carlos Alberto Lopez Perez
>> But this script seems focused on comparing the performance between
>> different browsers (safari vs chrome vs firefox) rather than in testing
>> and comparing the performance between different revisions of WebKit.
> 
> Not at all.  It simply supports running benchmark in other browsers.
> 
>> Do you think it makes any difference (from the point of view of
>> detecting failures, not from the performance PoV) to run this tests in a
>> full-fledged browser like Safari rather than in WebKitTestRunner ?
> 
> Yes. There are many browser features that can significantly impact the
> real world performance.
> 

I'm specifically not asking about performance, but about correctness.

This discussion was started because Filip said that running JS tests on
a browser catches many failures that are not cached when running the
tests from the terminal.

So, I'm wondering if running the JS tests on WTR or Safari makes any
difference when catching failures.

>> We already have a performance test bot running tests inside WTR.
>> And I see that the current set of tests executed on this bot already
>> includes Speedometer, and that JetStream and Sunspider are skipped on
>> PerformanceTests/Skipped.
>>
>> So I see some options going forward:
>>
>>  - Fix the JetStream and Sunspider tests so they can be run as part of
>> the current run-perf-tests script that the performance bots execute.
> 
> We should use run-benchmark instead since run-benchmark spits out the
> JSON file that's compatible with run-pref-tests.
> 

I'm a bit lost here. Are you planning to deprecate run-perf-tests with
run-benchmark? What is wrong with run-perf-tests?

>>  - Implement support on the script run-benchmark to run the tests inside
>> WTR, and create a new step running this script that will be executed on
>> the test bots.
> 
> I don't see a point in doing this.   Why is it desirable to run these
> benchmarks inside WebKitTestRunner?
> 

Less dependencies: WTR (or the MiniBrowser) is something that is
currently built by the bots on each build.
If we want to use Epiphany (for example) for the performance tests, is
another thing we have to take care of building before each run. Not a
big deal, but I wonder if is really worth.

>>  - Deploy a new bot that runs run-perf-tests on a full-fledged browser
>> like Safari or Epiphany.
> 
> We should just do this.
> 
>> I wonder what you think is the best option or if there is some option
>> not viable.
>>
>> From my PoV, the option #1 has the advantage of reusing the current
>> infrastructure that collects and draws performance data at
>> https://perf.webkit.org
> 
> We have an internal instance of the same dashboard to which we're
> reporting results of run-benchmark script.
> 

What about making this public? We will happily contribute with a
GTK+/Linux buildbot for it.


Regards.



signature.asc
Description: OpenPGP digital signature
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev