On Dec 15, 2009, at 12:30 PM, Mike Belshe wrote:
[+cc John Resig since he's using this as part of dromaeo]
Overall, sounds like good progress.
A couple of ideas:
- can we make it so that if you try to cut-and-paste comparisons
of 0.9 to 0.9.1 results, it will say "these results are from a
different version"?
Good idea. Filed <https://bugs.webkit.org/show_bug.cgi?id=32573>
- can we make the version more prominent in the title?
I'll see if I can find a reasonable way to do so. <https://bugs.webkit.org/show_bug.cgi?id=32574
>
- what would you think of reducing the setTimeout(..., 500) to
something like setTimeout(..., 100)? This will cut the runtime of
the test by ~80% :-)
I don't know if you noticed in my comments below, but the gap between
individual tests is now 10ms, there is only a one-time initial delay
of 500ms to give browsers a chance to recover from the effects of
loading the driver page.
Or are you talking about the initial 500ms delay? On my MacBook Pro,
in 64-bit Safari, each run of the test takes 370ms measured time, plus
260ms for the gaps between tests, and there are a total of 5 cycles.
So actual test time is 3780. So reducing the initial pause from 500ms
to 100ms would be a 10% improvement on total runtime of the benchmark.
I don't think that would be a meaningful difference. But I can look at
whether this number can be reduced without distorting the results.
I'll volunteer to do any of these tasks this week if you want me to
look at it.
In general help is welcome, but I think I can take care of the two
bugs cited above.
Regards,
Maciej
Mike
On Mon, Dec 14, 2009 at 11:32 PM, Maciej Stachowiak <m...@apple.com>
wrote:
Hello folks,
Over the past few days I made some changes to SunSpider to address
some of the more serious issues reported. I focused on only changes
that seem to make a significant difference to fairness and validity,
so for example I did not remove accidental access to global
variables. I also made a small number of harness changes that do not
affect results but fix flaws in the harness.
We are hesitant to change the SunSpider content or harness much at
all, since it's been used for cross-version and cross-brwoser
comparisons for so long. But these problems (many originally
suggpointed out by Chrome or Mozilla folks) seemed important enough
to address. Also, in addition to the patched content set, the
original sunspider-0.9 content set is also available to run through
the new harness.
The most important harness change is greatly reducing the time
between tests (as sugested by Mike Belshe) to avoid the negative
impact of power management on many systems (both Mac and Windows),
and which are most apparent for very fast browsers.
I'm deliberately not posting this on the web site yet because I
don't want a flood of gawkers testing their browser before enough
people have had a chance to review and verify these changes.
Harness changes:
In-browser SunSpider suffers excessive penalty under power management
https://bugs.webkit.org/show_bug.cgi?id=32505
Enable Web-hosted version of SunSpider to handle multiple versions
https://bugs.webkit.org/show_bug.cgi?id=32478
Use JSON.parse instead of eval for Web-hosted SunSpider results
processing
https://bugs.webkit.org/show_bug.cgi?id=32490
Some Browser-hosted SunSpider files are not valid HTML5
https://bugs.webkit.org/show_bug.cgi?id=32536
Make sunspider-0.9.1 the default content set (both command-line and
hosted)
https://bugs.webkit.org/show_bug.cgi?id=32537
Content changes (in sunspider-0.9.1 suite only; sunspider-0.9 is as
originally posted):
SunSpider/tests/string-base64.js does not compute a valid base64
encoded string
https://bugs.webkit.org/show_bug.cgi?id=16806
sunspider regexp-dna is inaccurate on firefox
https://bugs.webkit.org/show_bug.cgi?id=18989
Further changes I'm considering but am unsure about:
- Add correctness checking to all tests that don't use random numbers.
- Stop using array-like indexing of strings in the base64 test since
that doesn't work in IE8 and lower; but it is a standard construct
now (ES5), future IE will support it, and it's a useful thing to test.
Changes that probably won't be considered until a 2.0 version:
- Adding new tests to cover other areas.
- Rebalancing the runtime of the existing tests.
- Considering different scoring methodology such as bigger-is-better
or geometric mean or the like.
- Removing use of random numbers from tests that do use them.
Regards,
Maciej
_______________________________________________
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
_______________________________________________
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev