Hello folks,

Over the past few days I made some changes to SunSpider to address some of the more serious issues reported. I focused on only changes that seem to make a significant difference to fairness and validity, so for example I did not remove accidental access to global variables. I also made a small number of harness changes that do not affect results but fix flaws in the harness.

We are hesitant to change the SunSpider content or harness much at all, since it's been used for cross-version and cross-brwoser comparisons for so long. But these problems (many originally suggpointed out by Chrome or Mozilla folks) seemed important enough to address. Also, in addition to the patched content set, the original sunspider-0.9 content set is also available to run through the new harness.

The most important harness change is greatly reducing the time between tests (as sugested by Mike Belshe) to avoid the negative impact of power management on many systems (both Mac and Windows), and which are most apparent for very fast browsers.

I'm deliberately not posting this on the web site yet because I don't want a flood of gawkers testing their browser before enough people have had a chance to review and verify these changes.


Harness changes:

In-browser SunSpider suffers excessive penalty under power management
https://bugs.webkit.org/show_bug.cgi?id=32505

Enable Web-hosted version of SunSpider to handle multiple versions
https://bugs.webkit.org/show_bug.cgi?id=32478

Use JSON.parse instead of eval for Web-hosted SunSpider results processing
https://bugs.webkit.org/show_bug.cgi?id=32490

Some Browser-hosted SunSpider files are not valid HTML5
https://bugs.webkit.org/show_bug.cgi?id=32536

Make sunspider-0.9.1 the default content set (both command-line and hosted)
https://bugs.webkit.org/show_bug.cgi?id=32537


Content changes (in sunspider-0.9.1 suite only; sunspider-0.9 is as originally posted):

SunSpider/tests/string-base64.js does not compute a valid base64 encoded string
https://bugs.webkit.org/show_bug.cgi?id=16806

sunspider regexp-dna is inaccurate on firefox
https://bugs.webkit.org/show_bug.cgi?id=18989



Further changes I'm considering but am unsure about:
- Add correctness checking to all tests that don't use random numbers.
- Stop using array-like indexing of strings in the base64 test since that doesn't work in IE8 and lower; but it is a standard construct now (ES5), future IE will support it, and it's a useful thing to test.

Changes that probably won't be considered until a 2.0 version:
- Adding new tests to cover other areas.
- Rebalancing the runtime of the existing tests.
- Considering different scoring methodology such as bigger-is-better or geometric mean or the like.
- Removing use of random numbers from tests that do use them.

Regards,
Maciej

_______________________________________________
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Reply via email to