>> What happens if you use, say, `-c 100000', just running the
>> `Get_Char_Index` test? Are the percental timing differences then
>> still that large?
> Actually Get_Char_Index, on the three pages I have sent in the
> prev. mail, is higher than 6% only 4 times out of 15 total. (which is
> seem on other tests as well).
Well, the thing is that IMHO the difference should be *much* smaller –
your HTML pages show the execution of identical code on an identical
machine, right?
> about outliers, i splitted every tests into chuncks that is sized
> 100. Made IQR calculations and calculated average time on valid
> chunks. you can find the result in the attachment also pushed to
> gitlab.
Thanks. Hin-Tak gave additional suggestions how to possibly improve
the removal of outliers.
> also, since statistics and benchmarking are a sciences their self, i
> am a bit struggling while approaching the problem as well as feels
> like out of the gsoc project scope.
Indeed, the focus lately shifted from a representational aspect to a
more thorough approach how to handle benchmarks in general. You are
done with the first part, more or less, and it looks fine. The
latter, however, is definitely part of the GSoC project, too, and I'm
surprised that you think this might not be so: What are benchmark
timings good for if the returned values are completely meaningless?
In most cases, a small performance optimization in FreeType might
yield, say, an improvement of 1%. Right now, such a change would not
be detectable at all if using the framework you are working on – it
would be completely hidden by noise.
To summarize: Benchmark comparisons only work if there is a sound
mathematical foundation to reduce the noise. I don't ask you to
reinvent the wheel, but please do some more internet research and
check existing code how to tackle such problems. I'm 100% sure that
such code already exists (for example, the Google benchmark stuff
mentioned in a previous e-mail, scientific papers on arXiv, etc.,
etc.) and can be easily used, adapted, and simplified for our
purposes.
Werner
