[webkit-dev] Making performance tests in WebKit more useful
Hi, Three major problems with existing WebKit performance tests are: 1. There are too many tests to run 2. Some tests have variances that are too high to be of any use 3. Some tests are too specific to be of any general use To address them, I’m going to segregate the tests into 3 tiers: 1. Reliable tests that should be run on bots and locally when testing patches. 2. Supplemental tests that could be ran optionally. 3. Skipped tests. In addition, I’m going to add a forth category between 1 and 2 for new tests that have just been added since deciding whether a test is reliable or not is hard unless we have some data. This forth category is very important because while perf.webkit.org has an ability to aggregate results for each suite (e.g. for the entire DOM) by arbitary functions (e.g. arithmetic means, geometric means, etc…), letting everyone add arbitary tests to any suite will undermine our ability to monitor the results of reliable tests due to the added noise and means skewed by new tests. In an essense, we need a way to determine whether new tests can be added to tier 1 “test suites”. - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Making performance tests in WebKit more useful
On Fri, Mar 8, 2013 at 2:15 AM, Ryosuke Niwa rn...@webkit.org wrote: Three major problems with existing WebKit performance tests are: 1. There are too many tests to run 2. Some tests have variances that are too high to be of any use 3. Some tests are too specific to be of any general use If you find what differentiate the good tests from the bad tests, it would be good to create a wiki page explaining what to be careful about to make a good test. One thing that have bitten my ass in the past is the JIT being smart and removing some dead test code because it has no side effect. I am sure there are plenty of mistake like this to be avoided. Cheers, Benjamin ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Making performance tests in WebKit more useful
On Fri, Mar 8, 2013 at 2:24 AM, Benjamin Poulain benja...@webkit.orgwrote: On Fri, Mar 8, 2013 at 2:15 AM, Ryosuke Niwa rn...@webkit.org wrote: Three major problems with existing WebKit performance tests are: 1. There are too many tests to run 2. Some tests have variances that are too high to be of any use 3. Some tests are too specific to be of any general use If you find what differentiate the good tests from the bad tests, it would be good to create a wiki page explaining what to be careful about to make a good test. Yeah. We need to spend more time analysing tests in categories 2 3. One thing that have bitten my ass in the past is the JIT being smart and removing some dead test code because it has no side effect. I am sure there are plenty of mistake like this to be avoided. I was always curious about this. I wonder if there’s a good way to amortize the effect so that we can measure relialistic number. The last thing I want is to introduce a lot of measurement biases to lower the variance and end up measuring something unrealistic. - R. Niwa ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
