On 01/30/2014 01:37 PM, Alex Rousskov wrote:
> On 01/30/2014 12:14 PM, Kinkie wrote:
>> Ok, here's some numbers (same testing methodology as before)
>>
>> Trunk: mean RPS (CPU time)
>> 10029.11 (996.661872)
>> 9786.60 (1021.695007)
>> 10116.93 (988.395665)
>> 9958.71 (1004.039956)
>>
>> stdvector: mean RPS (CPUtime)
>> 9732.57 (1027.426563)
>> 10388.38 (962.418333)
>> 10332.17 (967.824790)
>
> OK, so we are probably just looking at noise: The variation within each
> data set (1.4% or 3.6%) is about the same or more than the difference
> between the set averages (1.8%).
>
> We will test this in a Polygraph lab, but I would not be surprised to
> see no significant difference there as well.
Polygraph results are in. No significant difference is visible.
FWIW, here are a few basic response stats extracted from four tests
("rep" is reply and "rptm" is response time):
* trunk vs. branch, take1:
rep.rate: 0% 2199.97 = 2199.97
rep.rptm.count: 0% 11880004.00 = 11880004.00
rep.rptm.mean: 0% 501.12 < 501.16
rep.rptm.std_dev: 0% 485.77 = 485.77
rep.size.count: 0% 11880004.00 = 11880004.00
rep.size.mean: 0% 8738.48 < 8739.91
rep.size.std_dev: 0% 11446.90 > 11446.12
* trunk vs. branch, take2:
rep.rate: 0% 2199.97 = 2199.97
rep.rptm.count: 0% 11880008.00 = 11880002.00
rep.rptm.mean: 0% 501.14 < 501.16
rep.rptm.std_dev: 0% 485.77 = 485.77
rep.size.count: 0% 11880008.00 = 11880002.00
rep.size.mean: 0% 8738.49 < 8739.14
rep.size.std_dev: 0% 11446.42 < 11448.64
The results appear to be pretty stable, but we have not run
more/different tests to really verify that:
* take1 vs. take2, trunk:
rep.rate: 0% 2199.97 = 2199.97
rep.rptm.count: 0% 11880004.00 = 11880008.00
rep.rptm.mean: 0% 501.12 < 501.14
rep.rptm.std_dev: 0% 485.77 = 485.77
rep.size.count: 0% 11880004.00 = 11880008.00
rep.size.mean: 0% 8738.48 < 8738.49
rep.size.std_dev: 0% 11446.90 > 11446.42
* take1 vs. take2, branch:
rep.rate: 0% 2199.97 = 2199.97
rep.rptm.count: 0% 11880004.00 = 11880002.00
rep.rptm.mean: 0% 501.16 = 501.16
rep.rptm.std_dev: 0% 485.77 = 485.77
rep.size.count: 0% 11880004.00 = 11880002.00
rep.size.mean: 0% 8739.91 > 8739.14
rep.size.std_dev: 0% 11446.12 < 11448.64
Note that the comparison script uses 1.0e-6 epsilon to declare equality
so some slightly different numbers are printed with an equal sign.
I hope I did not screw up while pasting these numbers :-).
HTH,
Alex.
