On Mon, 2014-10-13 at 20:09 -0400, Michael Goulish wrote:
> From now on until ebola gets me (and maybe long after that!)
> new proton-c code will be downloaded, built, performance-tested, and the 
> results posted in tasteful and attractive graphical form here:
> 
> 
>      
> http://people.apache.org/~mgoulish/protonics/performance/results/nightly.svg
> 
> 
> The testing is done with my proton-C engine-level clients.
> Each test consists of 50 trials, 5 million small messages each trial.
> 
> the graphics show you at a glance the mean value of the 50 tests for that
> day, as well as the plus-one sigma and minus-one sigma range.  (That is the 
> range in which about two-thirds of the test results will fall.)
> 
> The standard deviation (sigma) is important, because if you see that suddenly
> increasing -- even if the mean value remains relatively constant -- that 
> means 
> we have a problem.
> 
> A while ago we had a significant performance issue in qpidd that went 
> undetected
> for several months.   The goal here is to make sure that any significant 
> proton
> performance regression will become obvious within 24 hours.  (8.64e19 
> femtoseconds)
> 
> As always, I would be happy to hear any thoughts, questions, criticisms, 
> ideas,
> proposals, desires, hopes, dreams, or schemes relating to this system or 
> anything
> else.
> 

Excellent stuff. Now all we need is an automated process to watch for
significant performance regressions and send email listing all those
commits that might have been responsible.

Reply via email to