+1 On Sun, Feb 10, 2019 at 2:50 PM Vincent Massol <vinc...@massol.net> wrote: > > Hi devs, > > After TAKE 2 (see http://markmail.org/message/owtyhkmrz4tcbymn ), and after > analyzing several modules (I analyzed about 4-5 in total), I think we should > improve a bit the strategy to make it usable and applicable. > > Lessons learnt > ============ > > * It takes a lot of time to analyze a single global tpc drop (every time it > takes me around 2 hours) > * In general the results of the analysis are not that great. There are often > “good enough” reasons for the drop. It’s often a lack of unit tests and code > that is exercised by functional tests but the path has changed for various > reasons. > * I find the ratio of time to spend on the analysis vs result to be too low. > * In the end what’s important is that our global TPC continues to grow > > New Strategy > =========== > > * We run the Clover Jenkins pipeline every night (between 11PM-8AM) > * The pipeline sends an email whenever the new report has its global TPC > going down when compared with the baseline (vs one or more modules had their > TPC lowered in TAKE 2) > * The baseline report is the report generated just after each XS release. > This means that we keep the same baseline during a XS release > ** Technically it means that the pipeline will update the latest.txt file > (which contains the clover report timestamp) when it notices a version change > * We add a step in the Release Plan Template to have the report passing > before we can release. > * The RM is in charge of a release from day 1 to the release day (already the > case), and is also in charge of making sure that the global coverage job > failures get addressed before the release day so that we’re ready on the > release day. > * Implementation detail: don’t send a failure email when there are failing > tests in the build, to avoid false positives > > WDYT? > > Thanks > -Vincent
-- Thomas Mortagne
