To see if the bad iterative_count and threaded_count results were consistently bad, I ran the benchmark suite on my MacBook Pro to see how "reliable" the benchmarks were. The output is below.
Basically 6 benchmarks (regex_effbot, queens, startup_nosite, iterative_count, threaded_count, and telco) had a variance of more than 15% performance between my 2 computers, although queens, iterative_count, and threaded_count were the only ones that swung between neutral/good to bad depending on the machine (the rest either want from bad to very bad, or very good to more very good). And before Antoine asks, I added a ``sys.modules['markupsafe'] = None` line to the mako_v2 benchmark locally. =) Still need to either explicitly block it or emit a warning in the code in the repo. ######################################### Report on Darwin Darwin Kernel Version 12.2.0: Sat Aug 25 00:48:52 PDT 2012; root:xnu-2050.18.24~1/RELEASE_X86_64 x86_64 i386 Total CPU cores: 8 ### 2to3 ### 10.321463 -> 9.525119: 1.08x faster ### call_method ### Min: 0.466812 -> 0.417812: 1.12x faster Avg: 0.483324 -> 0.427158: 1.13x faster Significant (t=28.77) Stddev: 0.01876 -> 0.01483: 1.2644x smaller Timeline: b'http://tinyurl.com/8al5lmm' ### call_method_slots ### Min: 0.484923 -> 0.409452: 1.18x faster Avg: 0.487877 -> 0.413054: 1.18x faster Significant (t=131.11) Stddev: 0.00395 -> 0.00577: 1.4589x larger Timeline: b'http://tinyurl.com/9zhpg6z' ### call_method_unknown ### Min: 0.547050 -> 0.406866: 1.34x faster Avg: 0.550721 -> 0.409359: 1.35x faster Significant (t=328.32) Stddev: 0.00415 -> 0.00325: 1.2795x smaller Timeline: b'http://tinyurl.com/9wxoddz' ### call_simple ### Min: 0.391213 -> 0.332055: 1.18x faster Avg: 0.393563 -> 0.335362: 1.17x faster Significant (t=127.15) Stddev: 0.00363 -> 0.00427: 1.1764x larger Timeline: b'http://tinyurl.com/8mmepzw' ### chameleon ### Min: 0.078505 -> 0.070175: 1.12x faster Avg: 0.083754 -> 0.071500: 1.17x faster Significant (t=2.95) Stddev: 0.05086 -> 0.00119: 42.8425x smaller Timeline: b'http://tinyurl.com/8bz9hpl' ### chaos ### Min: 0.353739 -> 0.423587: 1.20x slower Avg: 0.356297 -> 0.428197: 1.20x slower Significant (t=-108.44) Stddev: 0.00200 -> 0.00424: 2.1147x larger Timeline: b'http://tinyurl.com/98e56le' ### django ### Min: 0.824149 -> 0.862750: 1.05x slower Avg: 0.831614 -> 0.869112: 1.05x slower Significant (t=-21.47) Stddev: 0.01020 -> 0.00697: 1.4634x smaller Timeline: b'http://tinyurl.com/8kz8owv' ### fannkuch ### Min: 1.776913 -> 1.832973: 1.03x slower Avg: 1.793116 -> 1.915348: 1.07x slower Significant (t=-11.57) Stddev: 0.01436 -> 0.07329: 5.1030x larger Timeline: b'http://tinyurl.com/9ptae4z' ### fastpickle ### Min: 0.810968 -> 0.739322: 1.10x faster Avg: 0.818099 -> 0.745148: 1.10x faster Significant (t=58.02) Stddev: 0.00577 -> 0.00677: 1.1731x larger Timeline: b'http://tinyurl.com/8l769dd' ### fastunpickle ### Min: 0.644198 -> 0.659345: 1.02x slower Avg: 0.647976 -> 0.666154: 1.03x slower Significant (t=-18.96) Stddev: 0.00343 -> 0.00584: 1.7020x larger Timeline: b'http://tinyurl.com/93xn7el' ### float ### Min: 0.420888 -> 0.363410: 1.16x faster Avg: 0.432285 -> 0.376179: 1.15x faster Significant (t=38.14) Stddev: 0.00762 -> 0.00708: 1.0766x smaller Timeline: b'http://tinyurl.com/8bjwka9' ### formatted_logging ### Min: 0.325707 -> 0.413196: 1.27x slower Avg: 0.329846 -> 0.418099: 1.27x slower Significant (t=-119.89) Stddev: 0.00397 -> 0.00337: 1.1787x smaller Timeline: b'http://tinyurl.com/8ktbs49' ### genshi ### Min: 0.254604 -> 0.269696: 1.06x slower Avg: 0.258585 -> 0.275615: 1.07x slower Significant (t=-33.39) Stddev: 0.00283 -> 0.00557: 1.9704x larger Timeline: b'http://tinyurl.com/8bqvcwl' ### go ### Min: 0.676453 -> 0.745504: 1.10x slower Avg: 0.681833 -> 0.752170: 1.10x slower Significant (t=-48.67) Stddev: 0.00520 -> 0.00880: 1.6917x larger Timeline: b'http://tinyurl.com/9d6qj3y' ### hexiom2 ### Min: 186.378727 -> 172.939507: 1.08x faster Avg: 186.679821 -> 173.103242: 1.08x faster Significant (t=39.61) Stddev: 0.42581 -> 0.23156: 1.8389x smaller Timeline: b'http://tinyurl.com/9mc3pmg' ### html5lib ### Min: 11.827770 -> 11.239556: 1.05x faster Avg: 11.858253 -> 11.370960: 1.04x faster Significant (t=6.93) Stddev: 0.02825 -> 0.15466: 5.4746x larger Timeline: b'http://tinyurl.com/8vl952y' ### iterative_count ### Min: 0.168182 -> 0.154105: 1.09x faster Avg: 0.169512 -> 0.155952: 1.09x faster Significant (t=50.77) Stddev: 0.00139 -> 0.00128: 1.0899x smaller Timeline: b'http://tinyurl.com/9eymjtf' ### json_dump_v2 ### Min: 3.350528 -> 3.795307: 1.13x slower Avg: 3.369661 -> 3.825400: 1.14x slower Significant (t=-125.93) Stddev: 0.01470 -> 0.02095: 1.4250x larger Timeline: b'http://tinyurl.com/8wyn9qa' ### json_load ### Min: 0.999717 -> 0.607549: 1.65x faster Avg: 1.007319 -> 0.613016: 1.64x faster Significant (t=289.24) Stddev: 0.00673 -> 0.00690: 1.0240x larger Timeline: b'http://tinyurl.com/8qxakdw' ### mako_v2 ### Min: 0.094817 -> 0.279593: 2.95x slower Avg: 0.096962 -> 0.286479: 2.95x slower Significant (t=-866.63) Stddev: 0.00182 -> 0.00454: 2.4945x larger Timeline: b'http://tinyurl.com/9lufgwz' ### meteor_contest ### Min: 0.276138 -> 0.243228: 1.14x faster Avg: 0.279559 -> 0.246018: 1.14x faster Significant (t=72.30) Stddev: 0.00298 -> 0.00136: 2.1943x smaller Timeline: b'http://tinyurl.com/8pj9dnc' ### nbody ### Min: 0.421698 -> 0.320496: 1.32x faster Avg: 0.425878 -> 0.323483: 1.32x faster Significant (t=158.15) Stddev: 0.00386 -> 0.00247: 1.5638x smaller Timeline: b'http://tinyurl.com/9fy8dfg' ### normal_startup ### Min: 0.612120 -> 0.876470: 1.43x slower Avg: 0.618945 -> 0.885492: 1.43x slower Significant (t=-280.36) Stddev: 0.00422 -> 0.00523: 1.2397x larger Timeline: b'http://tinyurl.com/98ap93d' ### nqueens ### Min: 0.402125 -> 0.410580: 1.02x slower Avg: 0.406403 -> 0.414676: 1.02x slower Significant (t=-12.06) Stddev: 0.00442 -> 0.00199: 2.2189x smaller Timeline: b'http://tinyurl.com/8wd3lez' ### pathlib ### Min: 0.132423 -> 0.164525: 1.24x slower Avg: 0.136298 -> 0.168843: 1.24x slower Significant (t=-49.05) Stddev: 0.00763 -> 0.00720: 1.0586x smaller Timeline: b'http://tinyurl.com/9o86dc5' ### pidigits ### Min: 0.387690 -> 0.367871: 1.05x faster Avg: 0.391308 -> 0.371194: 1.05x faster Significant (t=32.69) Stddev: 0.00369 -> 0.00230: 1.6066x smaller Timeline: b'http://tinyurl.com/9med7ko' ### raytrace ### Min: 1.650066 -> 1.808829: 1.10x slower Avg: 1.660110 -> 1.832654: 1.10x slower Significant (t=-25.26) Stddev: 0.01165 -> 0.04687: 4.0224x larger Timeline: b'http://tinyurl.com/8fmyhex' ### regex_compile ### Min: 0.559449 -> 0.571906: 1.02x slower Avg: 0.563738 -> 0.580054: 1.03x slower Significant (t=-8.38) Stddev: 0.00434 -> 0.01306: 3.0087x larger Timeline: b'http://tinyurl.com/8g6xcmd' ### regex_effbot ### Min: 0.074999 -> 0.097456: 1.30x slower Avg: 0.076343 -> 0.099435: 1.30x slower Significant (t=-39.79) Stddev: 0.00147 -> 0.00383: 2.5994x larger Timeline: b'http://tinyurl.com/9vfaeux' ### regex_v8 ### Min: 0.087433 -> 0.104053: 1.19x slower Avg: 0.088804 -> 0.105520: 1.19x slower Significant (t=-39.48) Stddev: 0.00115 -> 0.00277: 2.4122x larger Timeline: b'http://tinyurl.com/8un7vfr' ### richards ### Min: 0.247208 -> 0.222483: 1.11x faster Avg: 0.251661 -> 0.225276: 1.12x faster Significant (t=44.04) Stddev: 0.00392 -> 0.00161: 2.4275x smaller Timeline: b'http://tinyurl.com/8b2zv34' ### silent_logging ### Min: 0.099170 -> 0.095099: 1.04x faster Avg: 0.099713 -> 0.095892: 1.04x faster Significant (t=33.32) Stddev: 0.00045 -> 0.00068: 1.5062x larger Timeline: b'http://tinyurl.com/9arurw6' ### simple_logging ### Min: 0.316639 -> 0.392833: 1.24x slower Avg: 0.320059 -> 0.396853: 1.24x slower Significant (t=-120.31) Stddev: 0.00224 -> 0.00392: 1.7450x larger Timeline: b'http://tinyurl.com/95bfxu7' ### spectral_norm ### Min: 0.434691 -> 0.379294: 1.15x faster Avg: 0.437958 -> 0.383761: 1.14x faster Significant (t=67.75) Stddev: 0.00410 -> 0.00390: 1.0502x smaller Timeline: b'http://tinyurl.com/98s9c56' ### startup_nosite ### Min: 0.209685 -> 0.660867: 3.15x slower Avg: 0.218654 -> 0.673249: 3.08x slower Significant (t=-458.50) Stddev: 0.00646 -> 0.00752: 1.1645x larger Timeline: b'http://tinyurl.com/9zyerhn' ### telco ### Min: 0.840453 -> 0.018312: 45.90x faster Avg: 0.844250 -> 0.019255: 43.85x faster Significant (t=1088.45) Stddev: 0.00521 -> 0.00127: 4.0959x smaller Timeline: b'http://tinyurl.com/924mje7' ### threaded_count ### Min: 0.197525 -> 0.151649: 1.30x faster Avg: 0.213657 -> 0.153572: 1.39x faster Significant (t=52.58) Stddev: 0.00779 -> 0.00214: 3.6451x smaller Timeline: b'http://tinyurl.com/8mrrqla' ### unpack_sequence ### Min: 0.000060 -> 0.000052: 1.16x faster Avg: 0.000088 -> 0.000069: 1.29x faster Significant (t=1118.61) Stddev: 0.00000 -> 0.00000: 1.0022x larger Timeline: b'http://tinyurl.com/9ejrega'
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
