> On 6 Jan 2019, at 12:03, Vincent Massol <vinc...@massol.net> wrote:
> 
> Hi devs,
> 
> I ran the modified Clover job yesterday and it generated the 2 reports:
> * Report - 20171222-1835 -> 20190106-0207 : 
> http://maven.xwiki.org/site/clover/20190106/XWikiReport-20171222-1835-20190106-0207.html
> * (New) Report Report - 20190101-2330 -> 20190106-0207 : 
> http://maven.xwiki.org/site/clover/20190106/XWikiReport-20190101-2330-20190106-0207.html
> 
> This is quite interesting. 
> 
> The second report shows the modules that made us loose TPC since the 1st of 
> Jan 2019, i.e. in only 6 days… 
> 
> Note: It also shows (that’s the positive aspect), that overall we won 0.1371 
> of global TPC. But we could have won more if we can understand why we lost 
> some TPC and find out if there are actions we can put in place to avoid that.
> 
> So I think it would be good if devs who have touched these modules recently 
> could analyze the cause of the decrease and verify they’re real and what we 
> want to do about them.
> 
> WDYT?

Two impressive results are:
* xwiki-platform-url-api: +11.9999%
* xwiki-rendering-syntax-wikimodel: +28.587%

Any idea why these 2 were increased so much in the past 6 days?

Thanks
-Vincent

> 
> Thanks!
> -Vincent
> 
>> On 4 Jan 2019, at 15:14, Vincent Massol <vinc...@massol.net> wrote:
>> 
>> Hi devs,
>> 
>> FYI I've modified the clover job to do the following today:
>> * run daily around midnight on a4
>> * generate 2 diff reports: one against 20171220 (that's the one we're 
>> currently getting) and one against 20190101
>> * the first diff report doesn’t send an email on failure while the second 
>> one does.
>> 
>> I also changed yesterday:
>> * Only mark in RED (ie failures) modules where the global diff TPC is 
>> negative (before it was in RED when the global contribution was < 0 but that 
>> wasn’t accounting for removed modules for example).
>> 
>> Consequences:
>> * When we get report failures from the CI we need to fix it since it means 
>> we’ve regressed since 20190101
>> * It would still be good that we continue to analyze the diff report since 
>> 20171220 to fully understand it and make sure we haven’t regressed in 
>> quality since then, or at least fix the big regressions.
>> 
>> TODO:
>> * Need to find a way to indicate false positives, i.e. modules in RED for 
>> which we consider it’s ok to have a negative diff TPC, and thus know what’s 
>> left to do for the release.
>> * Once we better understand and fix modules we’ll get to the second step 
>> which is to automate the baseline value (manual ATM) and to have the report 
>> passing as a constraint for releasing and thus indicated in the Release Plan.
>> 
>> Let me know if you have questions.
>> 
>> Thanks
>> -Vincent
>> 
>>> On 3 Oct 2018, at 10:56, Vincent Massol <vinc...@massol.net> wrote:
>>> 
>>> Hi devs,
>>> 
>>> We started discussing a first global test coverage strategy in
>>> https://markmail.org/message/grphwta63pp5p4l7
>>> 
>>> I’d like to propose some updates and tuning now that we have a Clover 
>>> Jenkins pipeline working (brainstormed with Simon):
>>> 
>>> Here’s the new proposal:
>>> 
>>> * We run the Clover Jenkins pipeline every night (between 11PM-8AM)
>>> * The pipeline sends an email whenever the new report has modules lowering 
>>> the global TPC score compared to the baseline report (negative contribution 
>>> per module)
>>> * The baseline report is the report generated just after each XS release. 
>>> This means that we keep the same baseline during a XS release
>>> ** Technically it means that the pipeline will update the latest.txt file 
>>> (which contains the clover report timestamp) when it notices a version 
>>> change
>>> * We add a step in the Release Plan Template to have the report passing 
>>> before we can release.
>>> * The RM is in charge of a release from day 1 to the release day (already 
>>> the case), and is also in charge of making sure that the global coverage 
>>> job failures get addressed before the release day so that we’re ready on 
>>> the release day.
>>> 
>>> Options:
>>> * Make it easier and only fail the pipeline job when the global TPC value 
>>> is lower than the baseline (vs failing whenever a module has negative 
>>> contribution)
>>> 
>>> WDYT?
>>> 
>>> Thanks
>>> -Vincent
>>> 
>> 
> 

Reply via email to