Hi Simon, > On 11 Feb 2019, at 15:43, Simon Urli <simon.u...@xwiki.com> wrote: > > So if I sum up, we're moving the scope of global coverage from each modules > to the whole project, right?
Correct. > +1 for getting email/build failing only if the global TPC decrease > > Now I assume in case of decrease, we can get the whole report with info about > which modules had a change in their TPC, right? Correct again :) Actually it’s even my goal to improve it and also give details about package level (easy) and maybe even class level (harder), for “failing” modules. Thanks -Vincent > Simon > > On 11/02/2019 14:48, Thomas Mortagne wrote: >> +1 >> On Sun, Feb 10, 2019 at 2:50 PM Vincent Massol <vinc...@massol.net> wrote: >>> >>> Hi devs, >>> >>> After TAKE 2 (see http://markmail.org/message/owtyhkmrz4tcbymn ), and >>> after analyzing several modules (I analyzed about 4-5 in total), I think we >>> should improve a bit the strategy to make it usable and applicable. >>> >>> Lessons learnt >>> ============ >>> >>> * It takes a lot of time to analyze a single global tpc drop (every time it >>> takes me around 2 hours) >>> * In general the results of the analysis are not that great. There are >>> often “good enough” reasons for the drop. It’s often a lack of unit tests >>> and code that is exercised by functional tests but the path has changed for >>> various reasons. >>> * I find the ratio of time to spend on the analysis vs result to be too low. >>> * In the end what’s important is that our global TPC continues to grow >>> >>> New Strategy >>> =========== >>> >>> * We run the Clover Jenkins pipeline every night (between 11PM-8AM) >>> * The pipeline sends an email whenever the new report has its global TPC >>> going down when compared with the baseline (vs one or more modules had >>> their TPC lowered in TAKE 2) >>> * The baseline report is the report generated just after each XS release. >>> This means that we keep the same baseline during a XS release >>> ** Technically it means that the pipeline will update the latest.txt file >>> (which contains the clover report timestamp) when it notices a version >>> change >>> * We add a step in the Release Plan Template to have the report passing >>> before we can release. >>> * The RM is in charge of a release from day 1 to the release day (already >>> the case), and is also in charge of making sure that the global coverage >>> job failures get addressed before the release day so that we’re ready on >>> the release day. >>> * Implementation detail: don’t send a failure email when there are failing >>> tests in the build, to avoid false positives >>> >>> WDYT? >>> >>> Thanks >>> -Vincent > > -- > Simon Urli > Software Engineer at XWiki SAS > simon.u...@xwiki.com > More about us at http://www.xwiki.com
