+1

Thanks,
Marius

On Sun, Feb 10, 2019 at 3:50 PM Vincent Massol <vinc...@massol.net> wrote:

> Hi devs,
>
> After TAKE 2 (see http://markmail.org/message/owtyhkmrz4tcbymn ),  and
> after analyzing several modules (I analyzed about 4-5 in total), I think we
> should improve a bit the strategy to make it usable and applicable.
>
> Lessons learnt
> ============
>
> * It takes a lot of time to analyze a single global tpc drop (every time
> it takes me around 2 hours)
> * In general the results of the analysis are not that great. There are
> often “good enough” reasons for the drop. It’s often a lack of unit tests
> and code that is exercised by functional tests but the path has changed for
> various reasons.
> * I find the ratio of time to spend on the analysis vs result to be too
> low.
> * In the end what’s important is that our global TPC continues to grow
>
> New Strategy
> ===========
>
> * We run the Clover Jenkins pipeline every night (between 11PM-8AM)
> * The pipeline sends an email whenever the new report has its global TPC
> going down when compared with the baseline (vs one or more modules had
> their TPC lowered in TAKE 2)
> * The baseline report is the report generated just after each XS release.
> This means that we keep the same baseline during a XS release
> ** Technically it means that the pipeline will update the latest.txt file
> (which contains the clover report timestamp) when it notices a version
> change
> * We add a step in the Release Plan Template to have the report passing
> before we can release.
> * The RM is in charge of a release from day 1 to the release day (already
> the case), and is also in charge of making sure that the global coverage
> job failures get addressed before the release day so that we’re ready on
> the release day.
> * Implementation detail: don’t send a failure email when there are failing
> tests in the build, to avoid false positives
>
> WDYT?
>
> Thanks
> -Vincent

Reply via email to