+1 Thanks, Marius
On Sun, Feb 10, 2019 at 3:50 PM Vincent Massol <vinc...@massol.net> wrote: > Hi devs, > > After TAKE 2 (see http://markmail.org/message/owtyhkmrz4tcbymn ), and > after analyzing several modules (I analyzed about 4-5 in total), I think we > should improve a bit the strategy to make it usable and applicable. > > Lessons learnt > ============ > > * It takes a lot of time to analyze a single global tpc drop (every time > it takes me around 2 hours) > * In general the results of the analysis are not that great. There are > often “good enough” reasons for the drop. It’s often a lack of unit tests > and code that is exercised by functional tests but the path has changed for > various reasons. > * I find the ratio of time to spend on the analysis vs result to be too > low. > * In the end what’s important is that our global TPC continues to grow > > New Strategy > =========== > > * We run the Clover Jenkins pipeline every night (between 11PM-8AM) > * The pipeline sends an email whenever the new report has its global TPC > going down when compared with the baseline (vs one or more modules had > their TPC lowered in TAKE 2) > * The baseline report is the report generated just after each XS release. > This means that we keep the same baseline during a XS release > ** Technically it means that the pipeline will update the latest.txt file > (which contains the clover report timestamp) when it notices a version > change > * We add a step in the Release Plan Template to have the report passing > before we can release. > * The RM is in charge of a release from day 1 to the release day (already > the case), and is also in charge of making sure that the global coverage > job failures get addressed before the release day so that we’re ready on > the release day. > * Implementation detail: don’t send a failure email when there are failing > tests in the build, to avoid false positives > > WDYT? > > Thanks > -Vincent