Thank you for running this test.
I guess that with these results, it can make sense to go forward with
the project of replacing Vector with std::vector, does everyone agree?
Thanks!
On Fri, Feb 7, 2014 at 6:18 AM, Alex Rousskov
<rouss...@measurement-factory.com> wrote:
> On 01/30/2014 01:37 PM, Alex Rousskov wrote:
>> On 01/30/2014 12:14 PM, Kinkie wrote:
>>> Ok, here's some numbers (same testing methodology as before)
>>>
>>> Trunk: mean RPS (CPU time)
>>> 10029.11 (996.661872)
>>> 9786.60 (1021.695007)
>>> 10116.93 (988.395665)
>>> 9958.71 (1004.039956)
>>>
>>> stdvector: mean RPS (CPUtime)
>>> 9732.57 (1027.426563)
>>> 10388.38 (962.418333)
>>> 10332.17 (967.824790)
>>
>> OK, so we are probably just looking at noise: The variation within each
>> data set (1.4% or 3.6%) is about the same or more than the difference
>> between the set averages (1.8%).
>>
>> We will test this in a Polygraph lab, but I would not be surprised to
>> see no significant difference there as well.
>
>
> Polygraph results are in. No significant difference is visible.
>
>
> FWIW, here are a few basic response stats extracted from four tests
> ("rep" is reply and "rptm" is response time):
>
> * trunk vs. branch, take1:
> rep.rate: 0% 2199.97 = 2199.97
> rep.rptm.count: 0% 11880004.00 = 11880004.00
> rep.rptm.mean: 0% 501.12 < 501.16
> rep.rptm.std_dev: 0% 485.77 = 485.77
> rep.size.count: 0% 11880004.00 = 11880004.00
> rep.size.mean: 0% 8738.48 < 8739.91
> rep.size.std_dev: 0% 11446.90 > 11446.12
>
> * trunk vs. branch, take2:
> rep.rate: 0% 2199.97 = 2199.97
> rep.rptm.count: 0% 11880008.00 = 11880002.00
> rep.rptm.mean: 0% 501.14 < 501.16
> rep.rptm.std_dev: 0% 485.77 = 485.77
> rep.size.count: 0% 11880008.00 = 11880002.00
> rep.size.mean: 0% 8738.49 < 8739.14
> rep.size.std_dev: 0% 11446.42 < 11448.64
>
>
> The results appear to be pretty stable, but we have not run
> more/different tests to really verify that:
>
> * take1 vs. take2, trunk:
> rep.rate: 0% 2199.97 = 2199.97
> rep.rptm.count: 0% 11880004.00 = 11880008.00
> rep.rptm.mean: 0% 501.12 < 501.14
> rep.rptm.std_dev: 0% 485.77 = 485.77
> rep.size.count: 0% 11880004.00 = 11880008.00
> rep.size.mean: 0% 8738.48 < 8738.49
> rep.size.std_dev: 0% 11446.90 > 11446.42
>
> * take1 vs. take2, branch:
> rep.rate: 0% 2199.97 = 2199.97
> rep.rptm.count: 0% 11880004.00 = 11880002.00
> rep.rptm.mean: 0% 501.16 = 501.16
> rep.rptm.std_dev: 0% 485.77 = 485.77
> rep.size.count: 0% 11880004.00 = 11880002.00
> rep.size.mean: 0% 8739.91 > 8739.14
> rep.size.std_dev: 0% 11446.12 < 11448.64
>
>
> Note that the comparison script uses 1.0e-6 epsilon to declare equality
> so some slightly different numbers are printed with an equal sign.
>
> I hope I did not screw up while pasting these numbers :-).
>
>
> HTH,
>
> Alex.
>
--
Francesco
