On Jul 7, 2009, at 4:01 PM, Mike Belshe wrote:
I'd like benchmarks to:
a) have meaning even as browsers change over time
b) evolve. as new areas of JS (or whatever) become important,
the benchmark should have facilities to include that.
Fair? Good? Bad?
I think we can't rule out the possibility of a benchmark becoming less
meaningful over time. I do think that we should eventually produce a
new and rebalanced set of test content. I think it's fair to say that
time is approaching for SunSpider.
In particular, I don't think geometric means are a magic bullet. When
SunSpider was first created, regexps were a small proportion of the
total execution in what were the fastest publicly available at the
time. Eventually, everything else got much faster. So at some point,
SunSpider said "it might be a good idea to quadruple the speed of
regexp matching now". But if it used a geometric mean, it would always
say it's a good idea to quadruple the speed of regexp matching, unless
it omitted regexp tests entirely. From any starting point, and
regardless of speed of other facilities, speeding up regexps by a
factor of N would always show the same improvement in your overall
score. SunSpider, on the other hand, was deliberately designed to
highlight the area where an engine most needs improvement.
I think the only real way to deal with this is to periodically revise
and rebalance the benchmark.
Regards,
Maciej
_______________________________________________
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
