> What happens if you use, say, `-c 10', just running the
> `Get_Char_Index` test? Are the percental timing differences then
> still that large?
Actually Get_Char_Index, on the three pages I have sent in the prev. mail, is
higher than 6% only 4 times out of 15 total. (which is seem on other tests as
well).
> Why do you think so? Please explain your reasoning. Just remember
> that backup processes (like cleaning up the hard disk, running some
> cron jobs, etc.) can pop up anytime, thus influencing the result.
> Such spontaneous events have to be eliminated.
yes, right, didn't thought of the spontaneous events.
about outliers, i splitted every tests into chuncks that is sized 100. Made IQR
calculations and calculated average time on valid chunks. you can find the
result in the attachment also pushed to gitlab.
also, since statistics and benchmarking are a sciences their self, i am a bit
struggling while approaching the problem as well as feels like out of the gsoc
project scope. I would like to share this with your indulgence. yet, of course
I will move in accordance with your instructions.
Best,
Goksu
goksu.in
On 18 Aug 2023 00:02 +0300, Werner LEMBERG , wrote:
>
> > I have added the total table that you suggested.
>
> Thanks.
>
> > I think Get_Char_Index is not the problem, the results varies all
> > the time.
>
> As far as I can see, there is a direct relationship between the total
> cumulated time of a test and the timing variation: The smaller the
> cumulated time, the larger the variation.
>
> What happens if you use, say, `-c 10', just running the
> `Get_Char_Index` test? Are the percental timing differences then
> still that large?
>
> > Should I proceed to detect outliers? Since we do not get the same
> > error rate consistently, I think we will not find the target we
> > expected by outliers.
>
> Why do you think so? Please explain your reasoning. Just remember
> that backup processes (like cleaning up the hard disk, running some
> cron jobs, etc.) can pop up anytime, thus influencing the result.
> Such spontaneous events have to be eliminated.
>
> Have you actually tried something along the method I suggested?
>
>
> Werner
Freetype Benchmark Results
Warning: Baseline and Benchmark have the same commit ID!
Info
InfoBaselineBenchmark
Parameters-c 550 -w 50-c 550 -w 50
Commit ID3553148135531481
Commit Date2023-08-18 02:04:38 +03002023-08-18 02:04:38 +0300
BranchGSoC-2023-AhmetGSoC-2023-Ahmet
* Average time for all iterations. Smaller values are better.** N count in (x | y) format is for showing baseline and benchmark N counts seperately when they differs.Total Results
TestNBaseline (µs)Benchmark (µs)Difference (%)
Load54029304021700.2
Load_Advances (Normal)5327190357270-9.2
Load_Advances (Fast)513881554-12.0
Load_Advances (Unscaled)5153413988.9
Render5199593213029-6.7
Get_Glyph58504488477-4.0
Get_Char_Index4700013161365-3.8
Iterate CMap50010231039-1.6
New_Face5002582728613-10.8
Embolden5120237125197-4.1
Stroke515491651561868-0.8
Get_BBox55394854843-1.7
Get_CBox53899339678-1.8
New_Face & load glyph(s)56126663766-4.1
TOTAL59800028694532940267-2.5
Results for Roboto_subset.ttf
TestN* Baseline (µs)* Benchmark (µs)Difference (%)
Load1200069078678471.8
Load_Advances (Normal)120005681764449-13.4
Load_Advances (Fast)12000311358-15.4
Load_Advances (Unscaled)12000290340-17.4
Render120004814651436-6.8
Get_Glyph120001976020782-5.2
Get_Char_Index9400284306-7.9
Iterate CMap1002402323.5
New_Face10045146317-40.0
Embolden120002471328423-15.0
Stroke120003632683526572.9
Get_BBox1200011871114883.2
Get_CBox12000947688186.9
New_Face & load glyph(s)1200011584110974.2
TOTAL2832006203506245510.7
Results for Arial_subset.ttf
TestN* Baseline (µs)* Benchmark (µs)Difference (%)
Load95009213494812-2.9
Load_Advances (Normal)95007716390204-16.9
Load_Advances (Fast)9500285370-29.9
Load_Advances (Unscaled)950033329112.8
Render95004099043614-6.4
Get_Glyph95001753117640-0.6
Get_Char_Index94002622494.8
Iterate CMap100
