On 4 Dec, 2006, at 12:55, Heikki Toivonen wrote:
Now that Chandler is runnable with python 2.5 on the branch I did some
performance comparisons between trunk and that branch. I was expecting
overall improvement of around 10%, but unfortunately it does not
appear
that way.
The first column shows the test, second column the time (in seconds)
with Python 2.4, the third the time with python 2.5, and the last the
difference between the two. A negative value in the last column
means we
got that much faster with 2.5.
I run the tests just once, but I had shut down most applications and I
wasn't doing anything else with the machine, and this being dual
core I
would expect the normal variation between test results to be under 2%.
In other words, any difference less than +/-2% might not be a real
indication of change.
I'm somewhat skeptical of this 2% figure. Really, the only way to
measure "normal variation" is to run tests of the same code base
repeatedly, under as close to the same conditions as possible, and
look at the standard deviation of the mean.
When I do this with the performance tests, even with around 10 runs,
I usually end up with 5-10% variations in several of the test
results. (Like you, I was doing nothing else on the machine, and had
shut down all extraneous applications). What often happens is that
once in a while, a test runs anomalously slowly, probably due to some
random task being undertaken by the OS. This is perhaps not
surprising, but I wanted to point out that single measurements aren't
very trustworthy.
--Grant
The tests were run on Ubuntu Linux (Dapper Drake).
Test 2.4 2.5 %
=======================================================
#1 Startup 3.98 3.96 -0.50%
#2 New event (menu) 0.19 0.2 4.83%
#3 New event (double click) 0.24 0.22 -8.94%
#4 New calendar 0.17 0.16 -5.47%
#5 Import 3k calendar 37.08 37.52 1.18%
#6 Startup with 3k 4.5 4.61 2.44%
#7 New event (menu) with 3k 0.5 0.51 2.15%
#8 New event (double click) 3k 0.35 0.37 3.97%
#9 New calendar with 3k 0.68 0.64 -5.47%
Switch Views 0.23 0.25 8.01%
Stamp 0.19 0.21 11.30%
Switch views with 3k 0.89 0.89 0.61%
Stamp with 3k 0.27 0.28 2.31%
Scroll calendar with 3k 0.02 0.02 6.66%
Scroll table with 3k 0.26 0.26 1.27%
Jump calendar by week with 3k 0.29 0.29 0.28%
Overlay calendar with 3k 1.55 1.55 -0.10%
Switch calendar with 3k 1.27 1.25 -0.98%
Sharing 7.79 9.31 19.57%
--
Heikki Toivonen
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Open Source Applications Foundation "chandler-dev" mailing list
http://lists.osafoundation.org/mailman/listinfo/chandler-dev
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Open Source Applications Foundation "chandler-dev" mailing list
http://lists.osafoundation.org/mailman/listinfo/chandler-dev
