We’ve been working with Tibor and few other csit-dev and vpp-dev folks on the 
design to improve VPP performance trend tracking and automate anomaly detection 
(regression/progression). Result of this work is a design write-up published 
here: https://wiki.fd.io/view/CSIT/PerformanceTrendingAnalysis

We’re now working on the POC to put it in place two things:

1. PT - performance trending
  a. improved version of existing csit performance trending jobs.
  b. initially measuring only MRR (maximum received rate), with unlimited 
tolerance of packet loss, to measure vpp code efficiency (vpp vectors size at 
max), and cause it’s fast.
  c. optimized PDR and NDR searches will be added later.
  d. jobs should start running today/tomorrow, with daily cadence, 

2. PA - performance analysis
  a. automated trend analysis and anomaly detection.
  b. failed jobs signaling regressions.
  c. anomaly detection based on usual statistical metrics: trimmed moving avg, 
stdev, mean.
    - efficiency of detection to be tuned - see above link for description.
  c. funky graphs summarizing the trend and identifying performance regression 
and progression - daily, weekly, monthly views..
  d. concept code is already generating some graphs - can show on vpp-dev and 
csit-dev calls for feedback.

Hope folks find it useful - looking for comments and feedback..


Reply via email to