The reason of removing the first round result is that we found that the first round result is much different from other rounds. It is usually slower to open the sample file at the first time. So, we could view the first round result as an outlier and removed it when computing the average and standard deviation. However, the first round is also important for user experience. Maybe, we could list the first round performance separately.
As for the second value, t-test is a good suggestion and I'll research on it and use it in the following test. Thanks! On Tue, Jul 3, 2012 at 7:41 PM, Rob Weir <[email protected]> wrote: > On Tue, Jul 3, 2012 at 6:39 AM, Linyi Li <[email protected]> wrote: > > From Xuan Xuan's introduction in the beginning, I think the first data is > > average value of the test results and the second data is the standard > > deviation. > > > > So why skip the numbers for the first round test? Isn't that what > real users see, the first round? Sure, it will be slower as code is > loaded into memory, files read from disk, etc. But the same thing > happens for users. > > Also, I think the interesting 2nd number is the "standard error of the > mean", which == std deviation / sqrt(count of measurements). That is > what gives the error bars (confidence interval) on the measurement. > For example, 95% confidence limits on a measurement would be: > > lower bound = mean - 1.96*standard_error > upper bound = mean + 1.96*standard_error > > And easy "rule of thumb" is to compare the "before" and "after" > measures and see if there is overlap in the intervals. > > For example: > > Before interval: (1.0, 2.0) > After interval: (1.5, 2.5) > > Because the intervals overlap, there might not be a significant > difference between the two. > > But: > > Before interval: (1.0, 2.0) > After interval: (2.5, 3.5) > > In this case there is a clear difference, because the confidence > intervals do not overlap. > > A t-test could also be used here, but the above approach works well in > Calc if you use the "stock 2" type chart. This has series for > high/low/close/open. So you could do something where the high and low > values are 95% confidence intervals. This makes it easy to tell what > is important from a glance. > > -Rob > > > > > On Tue, Jul 3, 2012 at 5:36 PM, Ji Yan <[email protected]> wrote: > > > >> I'm sure Yi Xuan will update her wiki page with detail test case and the > >> meaning of report data > >> > >> 2012/7/3 Andre Fischer <[email protected]> > >> > >> > On 03.07.2012 09:02, Herbert Duerr wrote: > >> > > >> >> |---------|-------------------**--------|---------------|-----** > >> >>> ------------| > >> >>> | Filter | odt Load Show | Plain | 0.72/ 0.03 | > >> >>> | | | Complex | 1.13/ 0.03 | > >> >>> |---------|-------------------**--------|---------------|-----** > >> >>> ------------| > >> >>> [...] > >> >>> Any comment is welcomed! > >> >>> > >> >> > >> >> Thanks for sharing this interesting data. > >> >> I haven't found an explanation what these numbers mean though, so I > have > >> >> to guess: The first number is the average value and the second is the > >> >> sigma value for running the test, right? > >> >> > >> > > >> > It would be good to put any explanation/documentation on the wiki > page or > >> > else the information about the test parameters from the first mail -- > (8 > >> > runs, average over 5), what is plain or complex -- would be lost. > >> > > >> > -Andre > >> > > >> > >> > >> > >> -- > >> > >> > >> Thanks & Best Regards, Yan Ji > >> > > > > > > > > -- > > Best wishes. > > Linyi Li >
