Another example on the same computer. It's interesting: * MAD and std dev is the half of result 1 * the benchmark is less unstable * median is very close to result 1 * mean changed much more than median
Benchmark result 1: Median +- MAD: 276 ns +- 10 ns Mean +- std dev: 371 ns +- 196 ns Benchmark result 2: Median +- MAD: 278 ns +- 5 ns Mean +- std dev: 303 ns +- 103 ns If the goal is to get reproductible results, Median +- MAD seems better. --- haypo@selma$ PYTHONPATH=~/prog/GIT/perf ./python -m perf show --hist --stats bench2.json.gz 250 ns: 75 ########################################################### 278 ns: 73 ######################################################### 306 ns: 3 ## 333 ns: 0 | 361 ns: 0 | 389 ns: 0 | 417 ns: 0 | 445 ns: 0 | 472 ns: 1 # 500 ns: 1 # 528 ns: 0 | 556 ns: 0 | 584 ns: 0 | 611 ns: 1 # 639 ns: 0 | 667 ns: 0 | 695 ns: 1 # 722 ns: 0 | 750 ns: 1 # 778 ns: 1 # 806 ns: 3 ## Total duration: 1 min 4.0 sec Start date: 2017-03-07 00:39:03 End date: 2017-03-07 00:41:05 Raw sample minimum: 140 ms Raw sample maximum: 431 ms Number of runs: 42 Total number of samples: 160 Number of samples per run: 4 Number of warmups per run: 2 Loop iterations per sample: 2^19 (128 outer-loops x 4096 inner-loops) Minimum: 266 ns (-4%) Median +- MAD: 278 ns +- 5 ns Mean +- std dev: 303 ns +- 103 ns Maximum: 822 ns (+195%) ERROR: the benchmark is very unstable, the standard deviation is very high (stdev/mean: 34%)! Try to rerun the benchmark with more runs, samples and/or loops Median +- MAD: 278 ns +- 5 ns ---
bench2.json.gz
Description: GNU Zip compressed data
_______________________________________________ Speed mailing list Speed@python.org https://mail.python.org/mailman/listinfo/speed
