On Sat, May 02, 2009 at 01:06:02PM -0400, Jay Pipes wrote: > So, in running various benchmarks, I'm wondering what folks think of as > a regression? Is there a specific negative delta (percentage) which > folks consider significant? > > If I can get consensus, I will automate the process of notifying the ML > and the IRC channel when the automation detects a significant regression > at a given concurrency level in a revision...
I'd start at running (many) runs to find out the margin of error from one run to the next. any deviation from this should likely be reported. e.g. avg tps 2450 +/- 50. then a deviation of 60 should be looked into. assuming of course we get a decent enough sample set for what the +/- is. -- Stewart Smith _______________________________________________ Mailing list: https://launchpad.net/~drizzle-discuss Post to : [email protected] Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
