Hi Jim,
Your point regarding the performance difference between Mono and MS.NET is
an important one. It's one of those things that should have been a lot more
obvious than it was. None-the-less, thanks for bringing it to the surface!
One question: While I recognize the fact that for obvious reasons, ngen'ing
the IronPython assemblies as part of the distribution isn't an option (don't
worry, I'm not suggesting that it should or even could be an option ;-), has
anyone ran these same benchmarks against an ngen'd version of the assemblies
to compare the results? I realize that performance gains are no guarantee,
but I have found that more often than not the performance gains are fairly
significant, and worth the extra effort to ngen during the setup processes
in *most* cases. And in this case, given the fact that the CPython runtime
is compiled to native code and IronPython is still in all of its CIL glory
during the start-up process I wonder if its worth gaining a more
apples-to-apples comparison by running the same tests with ngen'd IP
assemblies?
I would suggest the same for Mono, but as of the last time I checked, there
is currently no support for AOT compiling .NET 2.0 assemblies (though I will
quickly check to verify, as its been a month or two since I last checked)
Worth pointing out: With these updated numbers, and given your statement,
IronPython just reached its 1.1 release and almost all of our recent work
has been about compatibility and completeness.
I must admit, I am certainly looking forward to the result when "speed and
optimization" is added to the mix. :)
Thanks for all of your hard work!
On 4/22/07, Jim Hugunin <[EMAIL PROTECTED]> wrote:
I'd like to point out one fact about these benchmarks that seems to have
been missed. Seo's numbers are for running IronPython 1.1 on Mono - not
on the Microsoft .NET implementation. The Mono team has done excellent work
with that project, but today the performance of the .NET implementation is
still significantly better and is a much better platform for benchmarking
IronPython's performance.
I reran pybench on my laptop tonight on the RTM version of .NET that ships
with Windows Vista - and has been available for XP for quite some time
now. I compared with CPython-2.5.1 (the latest version).
Running on .NET, I find that IronPython is faster on some tests and
CPython is faster on others. We know that we still have performance work to
do. After all, IronPython just reached its 1.1 release and almost all of
our recent work has been about compatibility and completeness. However, the
story is already quite interesting. Out of the 51 tests in pybench, CPython
is more than 2x faster on 10 of the tests and IronPython is more than 2x
faster on 9 of the tests. Depending on what your code does, either
implementation could run faster.
The most interesting cases to me are the 5 tests where CPython is more
than 3x faster than IronPython and the other 5 tests where IronPython is
more than 3x faster than CPython. CPython's strongest performance is in
dictionaries with integer and string keys, list slicing, small tuples and
code that actually throws and catches exceptions. IronPython's strongest
performance is in calling builtin functions, if/then/else blocks, calling
python functions, deep recursion, and try/except blocks that don't actually
catch an exception.
The places where IronPython is performing strongest are where it can use
the .NET code generation and JIT optimization most effectively. The places
where it is slowest are mainly in runtime library implementation of core
datatypes. CPython's list and dictionary datatypes are written in C and
have been hand-tuned for Python-style workloads over the past 10
years. IronPython's datatypes are much newer and clearly need more tuning
as the implementation matures.
The only two tests that really stand out as showing a deep performance
issue are the two exception handling ones. These are the only tests where
either implementation is more than 4x faster than the other. IronPython is
10x faster on the try/catch without an exception and CPython is 30x faster
when an exception is actually raised. This is a deliberate design decision
within .NET to make code that doesn't throw exceptions run faster - even if
that means slowing down code that does throw exceptions. I'm fairly
confident this was the right decision - and even remember the day long ago
when Guido was discussing Python's exception system and explained that he'd
accept almost any slow-down to the exceptional case in return for removing a
single instruction from the non-exceptional path.
Thanks - Jim
My results
-------------------------------------------------------------------------------
PYBENCH 2.0
-------------------------------------------------------------------------------
* using Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32
bit (Int
el)]
* disabled garbage collection
* system check interval set to maximum: 2147483647
* using timer: time.clock
-------------------------------------------------------------------------------
Benchmark: ipy11.pybench
-------------------------------------------------------------------------------
Rounds: 10
Warp: 10
Timer: time.time
Machine Details:
Platform ID: cli-32bit
Processor:
Python:
Implementation: Python
Executable: c:\ironpython-1.1\ipy.exe
Version: 1.1.0
Compiler: X
Bits: 32bit
Build: 0 0 (#0)
Unicode: UCS2
-------------------------------------------------------------------------------
Comparing with: py25.pybench
-------------------------------------------------------------------------------
Rounds: 10
Warp: 10
Timer: time.clock
Machine Details:
Platform ID: Microsoft-Windows-32bit-WindowsPE
Processor:
Python:
Implementation: Python
Executable: c:\python25\python.exe
Version: 2.5.1
Compiler: MSC v.1310 32 bit (Intel)
Bits: 32bit
Build: Apr 18 2007 08:51:08 (#r251:54863)
Unicode: UCS2
Test minimum run-time average run-time
this other diff this other
diff
-------------------------------------------------------------------------------
BuiltinFunctionCalls: 44ms 181ms -75.7% 48ms 182ms -
73.7%
BuiltinMethodLookup: 335ms 160ms +109.1% 344ms 162ms
+112.0%
CompareFloats: 99ms 111ms -11.3% 105ms 112ms
-6.0%
CompareFloatsIntegers: 63ms 125ms -49.7% 65ms 126ms -
48.7%
CompareIntegers: 102ms 114ms -10.4% 105ms 115ms
-8.9%
CompareInternedStrings: 175ms 126ms +39.0% 180ms
127ms +41.6%
CompareLongs: 111ms 105ms +5.3% 114ms 106ms
+7.2%
CompareStrings: 167ms 126ms +32.7% 172ms
127ms +35.1%
CompareUnicode: 121ms 125ms -3.4% 123ms 126ms
-2.0%
ConcatStrings: 482ms 272ms +77.2% 533ms
275ms +93.4%
ConcatUnicode: 305ms 216ms +41.0% 350ms
217ms +61.5%
CreateInstances: 102ms 148ms -31.1% 106ms 149ms -
28.9%
CreateNewInstances: 326ms 131ms +148.0% 335ms 133ms
+151.9%
CreateStringsWithConcat: 228ms 147ms +55.9% 238ms
148ms +61.0%
CreateUnicodeWithConcat: 91ms 154ms -41.2% 94ms 157ms -
40.4%
DictCreation: 137ms 105ms +30.0% 143ms
106ms +33.9%
DictWithFloatKeys: 301ms 230ms +30.8% 306ms
232ms +32.0%
DictWithIntegerKeys: 343ms 106ms +223.0% 351ms 108ms
+224.7%
DictWithStringKeys: 388ms 102ms +282.1% 395ms 102ms
+285.1%
ForLoops: 39ms 95ms -59.0% 40ms 97ms -
58.5%
IfThenElse: 35ms 111ms -68.9% 37ms 112ms -
66.8%
ListSlicing: 468ms 155ms +201.3% 477ms 157ms
+204.4%
NestedForLoops: 55ms 120ms -54.1% 57ms 122ms -
53.0%
NormalClassAttribute: 276ms 127ms +117.7% 282ms 129ms
+119.7%
NormalInstanceAttribute: 106ms 119ms -11.0% 109ms 120ms
-9.7%
PythonFunctionCalls: 37ms 130ms -71.7% 40ms 131ms -
69.5%
PythonMethodCalls: 127ms 158ms -19.6% 132ms 159ms -
16.8%
Recursion: 49ms 177ms -72.5% 50ms 179ms -
71.8%
SecondImport: 189ms 126ms +50.4% 195ms
127ms +53.6%
SecondPackageImport: 196ms 134ms +45.8% 202ms
136ms +48.5%
SecondSubmoduleImport: 266ms 177ms +50.6% 272ms
180ms +51.3%
SimpleComplexArithmetic: 97ms 148ms -34.4% 101ms 149ms -
32.0%
SimpleDictManipulation: 311ms 118ms +164.4% 318ms 118ms
+168.0%
SimpleFloatArithmetic: 64ms 119ms -46.6% 69ms 121ms -
43.2%
SimpleIntFloatArithmetic: 41ms 100ms -59.4% 43ms 102ms -
58.2%
SimpleIntegerArithmetic: 43ms 100ms -57.2% 44ms 101ms -
56.4%
SimpleListManipulation: 121ms 104ms +16.6% 124ms
105ms +18.3%
SimpleLongArithmetic: 126ms 113ms +11.2% 131ms
115ms +14.1%
SmallLists: 223ms 154ms +45.0% 229ms
156ms +47.0%
SmallTuples: 508ms 144ms +252.2% 523ms 145ms
+259.7%
SpecialClassAttribute: 275ms 124ms +121.2% 283ms 126ms
+124.5%
SpecialInstanceAttribute: 105ms 210ms -50.1% 109ms 212ms -
48.5%
StringMappings: 342ms 633ms -46.0% 351ms 637ms -
44.9%
StringPredicates: 222ms 217ms +2.3% 228ms 219ms
+4.4%
StringSlicing: 245ms 162ms +51.4% 260ms
163ms +59.4%
TryExcept: 7ms 100ms -93.4% 11ms 102ms -
89.4%
TryRaiseExcept: 3413ms 114ms +2896.2% 3453ms 115ms
+2910.6%
TupleSlicing: 227ms 150ms +51.6% 235ms
151ms +54.9%
UnicodeMappings: 227ms 126ms +79.6% 233ms
129ms +81.4%
UnicodePredicates: 218ms 129ms +68.8% 224ms
132ms +70.4%
UnicodeSlicing: 211ms 182ms +15.8% 223ms
184ms +21.3%
-------------------------------------------------------------------------------
Totals: 12781ms 7659ms +66.9%
13191ms 7741ms +70.4%
(this=ipy11.pybench, other=py25.pybench)
_______________________________________________
users mailing list
[email protected]
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
--
/M:D
M. David Peterson
http://mdavid.name | http://www.oreillynet.com/pub/au/2354 |
http://dev.aol.com/blog/3155
_______________________________________________
users mailing list
[email protected]
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
