Over the past few days I made some changes to SunSpider to address
some of the more serious issues reported. I focused on only changes
that seem to make a significant difference to fairness and validity,
so for example I did not remove accidental access to global variables.
I also made a small number of harness changes that do not affect
results but fix flaws in the harness.
We are hesitant to change the SunSpider content or harness much at
all, since it's been used for cross-version and cross-brwoser
comparisons for so long. But these problems (many originally
suggpointed out by Chrome or Mozilla folks) seemed important enough to
address. Also, in addition to the patched content set, the original
sunspider-0.9 content set is also available to run through the new
The most important harness change is greatly reducing the time between
tests (as sugested by Mike Belshe) to avoid the negative impact of
power management on many systems (both Mac and Windows), and which are
most apparent for very fast browsers.
I'm deliberately not posting this on the web site yet because I don't
want a flood of gawkers testing their browser before enough people
have had a chance to review and verify these changes.
In-browser SunSpider suffers excessive penalty under power management
Enable Web-hosted version of SunSpider to handle multiple versions
Use JSON.parse instead of eval for Web-hosted SunSpider results
Some Browser-hosted SunSpider files are not valid HTML5
Make sunspider-0.9.1 the default content set (both command-line and
Content changes (in sunspider-0.9.1 suite only; sunspider-0.9 is as
SunSpider/tests/string-base64.js does not compute a valid base64
sunspider regexp-dna is inaccurate on firefox
Further changes I'm considering but am unsure about:
- Add correctness checking to all tests that don't use random numbers.
- Stop using array-like indexing of strings in the base64 test since
that doesn't work in IE8 and lower; but it is a standard construct now
(ES5), future IE will support it, and it's a useful thing to test.
Changes that probably won't be considered until a 2.0 version:
- Adding new tests to cover other areas.
- Rebalancing the runtime of the existing tests.
- Considering different scoring methodology such as bigger-is-better
or geometric mean or the like.
- Removing use of random numbers from tests that do use them.
webkit-dev mailing list