Re: Progress update on adjustment database

[Werner LEMBERG]

> If you're testing this yourself, keep in mind that the adjustment
> will only be applied to n with tilde.  Any other letter with a tilde
> can be used as a control group.

Right now I'm abroad (actually, in New York :-) – it will take some
time until I can do such tests, sorry.


Werner

Re: -warmup

[Werner LEMBERG]

>> What exactly means 'Baseline (ms)'? Is the shown number the time
>>  for one loop? For all loops together? Please clarify and mention
>>  this on the HTML page.
>
> Clarified that the times are milliseconds for the cumulative time
> for all iterations.

Thanks.  The sentence is not easily comprehensible.  Perhaps change it
to something like

```
Cumulative time for all iterations.  Smaller values means better.
```

BTW, in column 'N' I see stuff like '68160 | 65880'.  What does this
mean?  Please add an explanatory comment to the HTML page.

Another thing: Please mention on the HTML page the completion time for
each test, and the total execution time of all tests together.

>> Looking at the 'Load_Advances (Unscaled)' row, I think that 100%
>>  difference between 0.001 and 0.002 doesn't make any sense. How do
>>  you compute the percentage? Is this based on the cumulative time
>> of  all loops? If so, and you really get such small numbers, there
>> must  be some fine-tuning for high-speed tests (for example,
>> increasing N  for this particular test by a factor of 10, say) to
>> get meaningful  timing values.
>
> it was cumulative time in milliseconds but converted it microseconds
> as how it was and it seem got better.

We are getting nearer, again :-)

What worries me, though, is that we still have such enormous
differences.  For `Get_Char_Index` I think it's lack of precision.
Please try to fix this – if the ratio

   cumulative_time / N

is smaller than a given threshold, N must be increased a lot.  In
other words, for `Roboto_subset.ttf`, N should be set to, say, 10*N.

For the other large differences I think we need some statistical
analysis to get better results – simple cumulation is not good enough.
In particular, outliers should be removed (at least this is my
hypothesis).  Maybe you can look up the internet to find some simple
code to handle them.

An idea to identify outliers could be to split the cumulation time
into, say, 100 smaller intervals.  You can the discard the too-large
values and compute the mean of the remaining data.  My reasoning is
that other CPU activity happens in parallel, but only for short
amounts of time.

Have you actually done a statistical analysis of, say, 'Load_Advances
(Normal)' for `Arial_subset.ttf`?  For example, printing all timings
of the datapoints as histograms for runs A and B?  *Are* there
outliers?  Maybe there is another statistical mean value that gives
more meaningful results.


Werner

Re: -warmup

[Ahmet Göksu]

Hi!
I changed code to warmup with number of iterations.
> What exactly means 'Baseline (ms)'? Is the shown number the time
>  for one loop? For all loops together? Please clarify and mention
>  this on the HTML page.
Clarified that the times are milliseconds for the cumulative time for all 
iterations.
> There seems to be a fundamental math problem in calculating the
>  percentage numbers. For example, looking at the 'TOTAL' field, the
>  percental difference between 2.788 and 2.740 is not -6.1% but -1.7%!
it was average of the all percentages but you are right. I have changed it 
percentage of total time changes.
> Looking at the 'Load_Advances (Unscaled)' row, I think that 100%
>  difference between 0.001 and 0.002 doesn't make any sense. How do
>  you compute the percentage? Is this based on the cumulative time of
>  all loops? If so, and you really get such small numbers, there must
>  be some fine-tuning for high-speed tests (for example, increasing N
>  for this particular test by a factor of 10, say) to get meaningful
>  timing values.
it was cumulative time in milliseconds but converted it microseconds as how it 
was and it seem got better. If any fine-tuning needed since now, i will.

Looking for reply.


Best,
Goksu
goksu.in
On 3 Aug 2023 19:50 +0300, Werner LEMBERG , wrote:
>
> > It is warming up as the given number of seconds with -w flag before
> > every benchmark test.
> >
> > There are still differences like 100%.. Also, 1 sec warmup means
> > (test count)*(font count) 70 secs for the results.
>
> Mhmm, I'm not sure whether a warmup *time span* makes sense. I would
> rather have thought that every test would get a certain number of
> warmup *loops*. For example, '--warmup 100' means that for a value of
> N=5, the first 100 loops of each test are not taken into account
> for timing so that effects of the various processor and memory caches,
> the operating system's memory page swapping, etc., etc., doesn't have
> too much influence. This should be just a very small fraction of
> time, not 70s.
>
> > I am thinking of what else can be done and waiting for your test.
>
> Just looking at your most recent HTML page I see some peculiarities.
>
> * What exactly means 'Baseline (ms)'? Is the shown number the time
> for one loop? For all loops together? Please clarify and mention
> this on the HTML page.
>
> * There seems to be a fundamental math problem in calculating the
> percentage numbers. For example, looking at the 'TOTAL' field, the
> percental difference between 2.788 and 2.740 is not -6.1% but -1.7%!
> What am I missing?
>
> * Looking at the 'Load_Advances (Unscaled)' row, I think that 100%
> difference between 0.001 and 0.002 doesn't make any sense. How do
> you compute the percentage? Is this based on the cumulative time of
> all loops? If so, and you really get such small numbers, there must
> be some fine-tuning for high-speed tests (for example, increasing N
> for this particular test by a factor of 10, say) to get meaningful
> timing values.
>
>
> Werner



Freetype Benchmark Results
Warning: Baseline and Benchmark have the same commit ID!
Info

InfoBaselineBenchmark
Parameters-c 1000 -w 100-c 1000 -w 100
Commit IDd7371720d7371720
Commit Date2023-08-03 19:08:57 +03002023-08-03 19:08:57 +0300
BranchGSoC-2023-AhmetGSoC-2023-Ahmet
* Cumulative time for iterations which is better in smaller values
Results for Roboto_subset.ttf

TestN* Baseline (µs)* Benchmark (µs)Difference (%)
Load12544769548050-0.6
Load_Advances (Normal)12472392483467-2.3
Load_Advances (Fast)12281828040.5
Load_Advances (Unscaled)1227742875-3.6
Render12407268425227-4.4
Get_Glyph12160786166644-3.6
Get_Char_Index940002728231815.0
Iterate CMap1000177117183.0
New_Face100039404390151.0
Embolden122140852099871.9
Stroke68160 | 65880162217116184290.2
Get_BBox12101134101693-0.6
Get_CBox1281055792772.2
New_Face & load glyph(s)1297837100719-2.9
TOTAL2726040375099237822230.8

Results for Arial_subset.ttf

TestN* Baseline (µs)* Benchmark (µs)Difference (%)
Load95000696891751976-7.9
Load_Advances (Normal)95000614680740438-20.5
Load_Advances (Fast)9500022922519-9.9
Load_Advances (Unscaled)9500021682516-16.1
Render950003335063251032.5
Get_Glyph950001425961378083.4
Get_Char_Index9400022893078-34.5
Iterate CMap1000187517516.6
New_Face100049544469655.2
Embolden95000