On Tue, Dec 15, 2009 at 1:01 PM, Maciej Stachowiak <m...@apple.com> wrote:
> > On Dec 15, 2009, at 12:30 PM, Mike Belshe wrote: > > [+cc John Resig since he's using this as part of dromaeo] > > Overall, sounds like good progress. > > A couple of ideas: > - can we make it so that if you try to cut-and-paste comparisons of 0.9 > to 0.9.1 results, it will say "these results are from a different version"? > > > Good idea. Filed <https://bugs.webkit.org/show_bug.cgi?id=32573> > > - can we make the version more prominent in the title? > > > I'll see if I can find a reasonable way to do so. < > https://bugs.webkit.org/show_bug.cgi?id=32574> > > - what would you think of reducing the setTimeout(..., 500) to something > like setTimeout(..., 100)? This will cut the runtime of the test by ~80% > :-) > > > I don't know if you noticed in my comments below, but the gap between > individual tests is now 10ms, there is only a one-time initial delay of > 500ms to give browsers a chance to recover from the effects of loading the > driver page. > Ok - I had checked the code, saw the 500ms setTimeout there, and then griped :-) This sounds great! > > Or are you talking about the initial 500ms delay? On my MacBook Pro, in > 64-bit Safari, each run of the test takes 370ms measured time, plus 260ms > for the gaps between tests, and there are a total of 5 cycles. So actual > test time is 3780. So reducing the initial pause from 500ms to 100ms would > be a 10% improvement on total runtime of the benchmark. I don't think that > would be a meaningful difference. But I can look at whether this number can > be reduced without distorting the results. > Agree - I don't care about that. thanks, Mike > > > I'll volunteer to do any of these tasks this week if you want me to look at > it. > > > In general help is welcome, but I think I can take care of the two bugs > cited above. > > Regards, > Maciej > > > > Mike > > > On Mon, Dec 14, 2009 at 11:32 PM, Maciej Stachowiak <m...@apple.com> wrote: > >> >> Hello folks, >> >> Over the past few days I made some changes to SunSpider to address some of >> the more serious issues reported. I focused on only changes that seem to >> make a significant difference to fairness and validity, so for example I did >> not remove accidental access to global variables. I also made a small number >> of harness changes that do not affect results but fix flaws in the harness. >> >> We are hesitant to change the SunSpider content or harness much at all, >> since it's been used for cross-version and cross-brwoser comparisons for so >> long. But these problems (many originally suggpointed out by Chrome or >> Mozilla folks) seemed important enough to address. Also, in addition to the >> patched content set, the original sunspider-0.9 content set is also >> available to run through the new harness. >> >> The most important harness change is greatly reducing the time between >> tests (as sugested by Mike Belshe) to avoid the negative impact of power >> management on many systems (both Mac and Windows), and which are most >> apparent for very fast browsers. >> >> I'm deliberately not posting this on the web site yet because I don't want >> a flood of gawkers testing their browser before enough people have had a >> chance to review and verify these changes. >> >> >> Harness changes: >> >> In-browser SunSpider suffers excessive penalty under power management >> https://bugs.webkit.org/show_bug.cgi?id=32505 >> >> Enable Web-hosted version of SunSpider to handle multiple versions >> https://bugs.webkit.org/show_bug.cgi?id=32478 >> >> Use JSON.parse instead of eval for Web-hosted SunSpider results processing >> https://bugs.webkit.org/show_bug.cgi?id=32490 >> >> Some Browser-hosted SunSpider files are not valid HTML5 >> https://bugs.webkit.org/show_bug.cgi?id=32536 >> >> Make sunspider-0.9.1 the default content set (both command-line and >> hosted) >> https://bugs.webkit.org/show_bug.cgi?id=32537 >> >> >> Content changes (in sunspider-0.9.1 suite only; sunspider-0.9 is as >> originally posted): >> >> SunSpider/tests/string-base64.js does not compute a valid base64 encoded >> string >> https://bugs.webkit.org/show_bug.cgi?id=16806 >> >> sunspider regexp-dna is inaccurate on firefox >> https://bugs.webkit.org/show_bug.cgi?id=18989 >> >> >> >> Further changes I'm considering but am unsure about: >> - Add correctness checking to all tests that don't use random numbers. >> - Stop using array-like indexing of strings in the base64 test since that >> doesn't work in IE8 and lower; but it is a standard construct now (ES5), >> future IE will support it, and it's a useful thing to test. >> >> Changes that probably won't be considered until a 2.0 version: >> - Adding new tests to cover other areas. >> - Rebalancing the runtime of the existing tests. >> - Considering different scoring methodology such as bigger-is-better or >> geometric mean or the like. >> - Removing use of random numbers from tests that do use them. >> >> Regards, >> Maciej >> >> _______________________________________________ >> webkit-dev mailing list >> webkit-dev@lists.webkit.org >> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev >> > > >
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev