> On 6 Jan 2019, at 12:03, Vincent Massol <vinc...@massol.net> wrote:
>
> Hi devs,
>
> I ran the modified Clover job yesterday and it generated the 2 reports:
> * Report - 20171222-1835 -> 20190106-0207 :
> http://maven.xwiki.org/site/clover/20190106/XWikiReport-20171222-1835-20190106-0207.html
> * (New) Report Report - 20190101-2330 -> 20190106-0207 :
> http://maven.xwiki.org/site/clover/20190106/XWikiReport-20190101-2330-20190106-0207.html
>
> This is quite interesting.
>
> The second report shows the modules that made us loose TPC since the 1st of
> Jan 2019, i.e. in only 6 days…
>
> Note: It also shows (that’s the positive aspect), that overall we won 0.1371
> of global TPC. But we could have won more if we can understand why we lost
> some TPC and find out if there are actions we can put in place to avoid that.
>
> So I think it would be good if devs who have touched these modules recently
> could analyze the cause of the decrease and verify they’re real and what we
> want to do about them.
>
> WDYT?
Two impressive results are:
* xwiki-platform-url-api: +11.9999%
* xwiki-rendering-syntax-wikimodel: +28.587%
Any idea why these 2 were increased so much in the past 6 days?
Thanks
-Vincent
>
> Thanks!
> -Vincent
>
>> On 4 Jan 2019, at 15:14, Vincent Massol <vinc...@massol.net> wrote:
>>
>> Hi devs,
>>
>> FYI I've modified the clover job to do the following today:
>> * run daily around midnight on a4
>> * generate 2 diff reports: one against 20171220 (that's the one we're
>> currently getting) and one against 20190101
>> * the first diff report doesn’t send an email on failure while the second
>> one does.
>>
>> I also changed yesterday:
>> * Only mark in RED (ie failures) modules where the global diff TPC is
>> negative (before it was in RED when the global contribution was < 0 but that
>> wasn’t accounting for removed modules for example).
>>
>> Consequences:
>> * When we get report failures from the CI we need to fix it since it means
>> we’ve regressed since 20190101
>> * It would still be good that we continue to analyze the diff report since
>> 20171220 to fully understand it and make sure we haven’t regressed in
>> quality since then, or at least fix the big regressions.
>>
>> TODO:
>> * Need to find a way to indicate false positives, i.e. modules in RED for
>> which we consider it’s ok to have a negative diff TPC, and thus know what’s
>> left to do for the release.
>> * Once we better understand and fix modules we’ll get to the second step
>> which is to automate the baseline value (manual ATM) and to have the report
>> passing as a constraint for releasing and thus indicated in the Release Plan.
>>
>> Let me know if you have questions.
>>
>> Thanks
>> -Vincent
>>
>>> On 3 Oct 2018, at 10:56, Vincent Massol <vinc...@massol.net> wrote:
>>>
>>> Hi devs,
>>>
>>> We started discussing a first global test coverage strategy in
>>> https://markmail.org/message/grphwta63pp5p4l7
>>>
>>> I’d like to propose some updates and tuning now that we have a Clover
>>> Jenkins pipeline working (brainstormed with Simon):
>>>
>>> Here’s the new proposal:
>>>
>>> * We run the Clover Jenkins pipeline every night (between 11PM-8AM)
>>> * The pipeline sends an email whenever the new report has modules lowering
>>> the global TPC score compared to the baseline report (negative contribution
>>> per module)
>>> * The baseline report is the report generated just after each XS release.
>>> This means that we keep the same baseline during a XS release
>>> ** Technically it means that the pipeline will update the latest.txt file
>>> (which contains the clover report timestamp) when it notices a version
>>> change
>>> * We add a step in the Release Plan Template to have the report passing
>>> before we can release.
>>> * The RM is in charge of a release from day 1 to the release day (already
>>> the case), and is also in charge of making sure that the global coverage
>>> job failures get addressed before the release day so that we’re ready on
>>> the release day.
>>>
>>> Options:
>>> * Make it easier and only fail the pipeline job when the global TPC value
>>> is lower than the baseline (vs failing whenever a module has negative
>>> contribution)
>>>
>>> WDYT?
>>>
>>> Thanks
>>> -Vincent
>>>
>>
>
