Jim Hugunin wrote: > I'd like to point out one fact about these benchmarks that seems to have been > missed. Seo's numbers are for running IronPython 1.1 on Mono - not on the > Microsoft .NET implementation. The Mono team has done excellent work with > that project, but today the performance of the .NET implementation is still > significantly better and is a much better platform for benchmarking > IronPython's performance. > And on close examination the results are impressive. Nice one.
The anomaly with raising exceptions is very weird though (IronPython is around 30 times slower!) : TryExcept: 7ms 100ms -93.4% 11ms 102ms -89.4% TryRaiseExcept: 3413ms 114ms +2896.2% 3453ms 115ms +2910.6% All the best, Michael Foord > I reran pybench on my laptop tonight on the RTM version of .NET that ships > with Windows Vista - and has been available for XP for quite some time now. > I compared with CPython-2.5.1 (the latest version). > > Running on .NET, I find that IronPython is faster on some tests and CPython > is faster on others. We know that we still have performance work to do. > After all, IronPython just reached its 1.1 release and almost all of our > recent work has been about compatibility and completeness. However, the > story is already quite interesting. Out of the 51 tests in pybench, CPython > is more than 2x faster on 10 of the tests and IronPython is more than 2x > faster on 9 of the tests. Depending on what your code does, either > implementation could run faster. > > The most interesting cases to me are the 5 tests where CPython is more than > 3x faster than IronPython and the other 5 tests where IronPython is more than > 3x faster than CPython. CPython's strongest performance is in dictionaries > with integer and string keys, list slicing, small tuples and code that > actually throws and catches exceptions. IronPython's strongest performance > is in calling builtin functions, if/then/else blocks, calling python > functions, deep recursion, and try/except blocks that don't actually catch an > exception. > > The places where IronPython is performing strongest are where it can use the > .NET code generation and JIT optimization most effectively. The places where > it is slowest are mainly in runtime library implementation of core datatypes. > CPython's list and dictionary datatypes are written in C and have been > hand-tuned for Python-style workloads over the past 10 years. IronPython's > datatypes are much newer and clearly need more tuning as the implementation > matures. > > The only two tests that really stand out as showing a deep performance issue > are the two exception handling ones. These are the only tests where either > implementation is more than 4x faster than the other. IronPython is 10x > faster on the try/catch without an exception and CPython is 30x faster when > an exception is actually raised. This is a deliberate design decision within > .NET to make code that doesn't throw exceptions run faster - even if that > means slowing down code that does throw exceptions. I'm fairly confident > this was the right decision - and even remember the day long ago when Guido > was discussing Python's exception system and explained that he'd accept > almost any slow-down to the exceptional case in return for removing a single > instruction from the non-exceptional path. > > Thanks - Jim > > My results > ------------------------------------------------------------------------------- > PYBENCH 2.0 > ------------------------------------------------------------------------------- > * using Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit > (Int > el)] > * disabled garbage collection > * system check interval set to maximum: 2147483647 > * using timer: time.clock > > ------------------------------------------------------------------------------- > Benchmark: ipy11.pybench > ------------------------------------------------------------------------------- > > Rounds: 10 > Warp: 10 > Timer: time.time > > Machine Details: > Platform ID: cli-32bit > Processor: > > Python: > Implementation: Python > Executable: c:\ironpython-1.1\ipy.exe > Version: 1.1.0 > Compiler: X > Bits: 32bit > Build: 0 0 (#0) > Unicode: UCS2 > > > ------------------------------------------------------------------------------- > Comparing with: py25.pybench > ------------------------------------------------------------------------------- > > Rounds: 10 > Warp: 10 > Timer: time.clock > > Machine Details: > Platform ID: Microsoft-Windows-32bit-WindowsPE > Processor: > > Python: > Implementation: Python > Executable: c:\python25\python.exe > Version: 2.5.1 > Compiler: MSC v.1310 32 bit (Intel) > Bits: 32bit > Build: Apr 18 2007 08:51:08 (#r251:54863) > Unicode: UCS2 > > > Test minimum run-time average run-time > this other diff this other diff > ------------------------------------------------------------------------------- > BuiltinFunctionCalls: 44ms 181ms -75.7% 48ms 182ms > -73.7% > BuiltinMethodLookup: 335ms 160ms +109.1% 344ms 162ms > +112.0% > CompareFloats: 99ms 111ms -11.3% 105ms 112ms > -6.0% > CompareFloatsIntegers: 63ms 125ms -49.7% 65ms 126ms > -48.7% > CompareIntegers: 102ms 114ms -10.4% 105ms 115ms > -8.9% > CompareInternedStrings: 175ms 126ms +39.0% 180ms 127ms > +41.6% > CompareLongs: 111ms 105ms +5.3% 114ms 106ms > +7.2% > CompareStrings: 167ms 126ms +32.7% 172ms 127ms > +35.1% > CompareUnicode: 121ms 125ms -3.4% 123ms 126ms > -2.0% > ConcatStrings: 482ms 272ms +77.2% 533ms 275ms > +93.4% > ConcatUnicode: 305ms 216ms +41.0% 350ms 217ms > +61.5% > CreateInstances: 102ms 148ms -31.1% 106ms 149ms > -28.9% > CreateNewInstances: 326ms 131ms +148.0% 335ms 133ms > +151.9% > CreateStringsWithConcat: 228ms 147ms +55.9% 238ms 148ms > +61.0% > CreateUnicodeWithConcat: 91ms 154ms -41.2% 94ms 157ms > -40.4% > DictCreation: 137ms 105ms +30.0% 143ms 106ms > +33.9% > DictWithFloatKeys: 301ms 230ms +30.8% 306ms 232ms > +32.0% > DictWithIntegerKeys: 343ms 106ms +223.0% 351ms 108ms > +224.7% > DictWithStringKeys: 388ms 102ms +282.1% 395ms 102ms > +285.1% > ForLoops: 39ms 95ms -59.0% 40ms 97ms > -58.5% > IfThenElse: 35ms 111ms -68.9% 37ms 112ms > -66.8% > ListSlicing: 468ms 155ms +201.3% 477ms 157ms > +204.4% > NestedForLoops: 55ms 120ms -54.1% 57ms 122ms > -53.0% > NormalClassAttribute: 276ms 127ms +117.7% 282ms 129ms > +119.7% > NormalInstanceAttribute: 106ms 119ms -11.0% 109ms 120ms > -9.7% > PythonFunctionCalls: 37ms 130ms -71.7% 40ms 131ms > -69.5% > PythonMethodCalls: 127ms 158ms -19.6% 132ms 159ms > -16.8% > Recursion: 49ms 177ms -72.5% 50ms 179ms > -71.8% > SecondImport: 189ms 126ms +50.4% 195ms 127ms > +53.6% > SecondPackageImport: 196ms 134ms +45.8% 202ms 136ms > +48.5% > SecondSubmoduleImport: 266ms 177ms +50.6% 272ms 180ms > +51.3% > SimpleComplexArithmetic: 97ms 148ms -34.4% 101ms 149ms > -32.0% > SimpleDictManipulation: 311ms 118ms +164.4% 318ms 118ms > +168.0% > SimpleFloatArithmetic: 64ms 119ms -46.6% 69ms 121ms > -43.2% > SimpleIntFloatArithmetic: 41ms 100ms -59.4% 43ms 102ms > -58.2% > SimpleIntegerArithmetic: 43ms 100ms -57.2% 44ms 101ms > -56.4% > SimpleListManipulation: 121ms 104ms +16.6% 124ms 105ms > +18.3% > SimpleLongArithmetic: 126ms 113ms +11.2% 131ms 115ms > +14.1% > SmallLists: 223ms 154ms +45.0% 229ms 156ms > +47.0% > SmallTuples: 508ms 144ms +252.2% 523ms 145ms > +259.7% > SpecialClassAttribute: 275ms 124ms +121.2% 283ms 126ms > +124.5% > SpecialInstanceAttribute: 105ms 210ms -50.1% 109ms 212ms > -48.5% > StringMappings: 342ms 633ms -46.0% 351ms 637ms > -44.9% > StringPredicates: 222ms 217ms +2.3% 228ms 219ms > +4.4% > StringSlicing: 245ms 162ms +51.4% 260ms 163ms > +59.4% > TryExcept: 7ms 100ms -93.4% 11ms 102ms > -89.4% > TryRaiseExcept: 3413ms 114ms +2896.2% 3453ms 115ms > +2910.6% > TupleSlicing: 227ms 150ms +51.6% 235ms 151ms > +54.9% > UnicodeMappings: 227ms 126ms +79.6% 233ms 129ms > +81.4% > UnicodePredicates: 218ms 129ms +68.8% 224ms 132ms > +70.4% > UnicodeSlicing: 211ms 182ms +15.8% 223ms 184ms > +21.3% > ------------------------------------------------------------------------------- > Totals: 12781ms 7659ms +66.9% 13191ms 7741ms > +70.4% > > (this=ipy11.pybench, other=py25.pybench) > _______________________________________________ > users mailing list > [email protected] > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > _______________________________________________ users mailing list [email protected] http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
