--------
Martin Grigorov writes:
> Any feedback and ideas how to tweak it (VCL or even patches) are very
> welcome!
First you need to tweak your benchmark setup.
aarch64
Thread Stats Avg Stdev Max +/- Stdev
Latency 655.40us 798.70us 28.43ms 90.52%
Strictly speaking, you cannot rule out that the ARM machine
sends responses before it receives the request, because your
standard deviation is larger than your average.
In other words: Those numbers tell us nothing.
If you want to do this comparison, and I would love for you to do so,
you really need to take the time it takes, and get your "noise" down.
Here is how you should do it:
for machine in ARM, INTEL
Reboot machine
For i in (at least) 1-5:
Run test for 5 minutes
If the results from the first run on each machine is very different
from the other four runs, you can disrecard it, as a startup/bootup
artifact.
Report the numbers for all the runs for both machines.
Make a plot of all those numbers, where you plot the reported
average +/- stddev as a line, and the max value as a dot/cross/box.
If you want to get fancy, you can do a Student's T test to tell
you if there is any real difference. There's a program called
"ministat" which will do this for you.
Also: I can highly recommend this book:
http://www.larrygonick.com/titles/science/the-cartoon-guide-to-statistics/
--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
_______________________________________________
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
