>> Should I proceed to detect outliers? Since we do not get the same >> error rate consistently, I think we will not find the target we >> expected by outliers. > > Why do you think so? Please explain your reasoning. Just remember > that backup processes (like cleaning up the hard disk, running some > cron jobs, etc.) can pop up anytime, thus influencing the result. > Such spontaneous events have to be eliminated. > > Have you actually tried something along the method I suggested?
BTW, here is another repository that provides a framework for benchmarks: https://github.com/sharkdp/hyperfine It looks quite nice, AFAICS – maybe you can check its code for identifying outliers and the like. Werner
