On Wed, Aug 26, 2009 at 10:13 PM, Brian Aker<[email protected]> wrote: > Hi! > > This looks like regression...
I'm having a tough time with these read/write numbers. I never know whether there is a regression or not due to the large variances in the numbers reported. For example, for the read/write workload at concurrency level 16, the numbers have been: 1121 2089 1122 2065 1124 2588 1125 2056 1126 2061 So it looks like there was a big spike in performance for the run on revision 1125. I've seen this happen before for branches I've ran through the benchmark system myself (I got up to 2600 one time on the read/write workload at concurrency level 16). In fact, 2 different runs of the same branch can produce variances in numbers like this. Basically, I'm not entirely convinced there is a regression here. But then again, its hard to make any sort of conclusion from these numbers. Is there an acceptable variance in the numbers that we are willing to live with? Is there a threshold in the variance over which we will consider a regression to have occurred? Could we record variances along with what we are already storing? -Padraig > > Cheers, > -Brian > > On Aug 26, 2009, at 5:10 PM, X4450 Machine wrote: > >> stage-1125 256 1442.806667 >> stage-1125 512 1433.936667 >> stage-1125 1024 843.546667 >> stage-1126 2 706.973333 >> stage-1126 4 1138.103333 >> stage-1126 8 2024.116667 >> stage-1126 16 2061.630000 >> stage-1126 32 2111.020000 >> stage-1126 64 1851.066667 >> stage-1126 128 1665.406667 >> stage-1126 256 1336.600000 >> stage-1126 512 729.270000 >> stage-1126 1024 714.686667 > > _______________________________________________ Mailing list: https://launchpad.net/~drizzle-benchmark Post to : [email protected] Unsubscribe : https://launchpad.net/~drizzle-benchmark More help : https://help.launchpad.net/ListHelp
