> On 9 Jan 2019, at 16:05, Thomas Mortagne <thomas.morta...@xwiki.com> wrote:
>
> On Sun, Jan 6, 2019 at 12:06 PM Vincent Massol <vinc...@massol.net> wrote:
>>
>>
>>
>>> On 6 Jan 2019, at 12:03, Vincent Massol <vinc...@massol.net> wrote:
>>>
>>> Hi devs,
>>>
>>> I ran the modified Clover job yesterday and it generated the 2 reports:
>>> * Report - 20171222-1835 -> 20190106-0207 :
>>> http://maven.xwiki.org/site/clover/20190106/XWikiReport-20171222-1835-20190106-0207.html
>>> * (New) Report Report - 20190101-2330 -> 20190106-0207 :
>>> http://maven.xwiki.org/site/clover/20190106/XWikiReport-20190101-2330-20190106-0207.html
>>>
>>> This is quite interesting.
>>>
>>> The second report shows the modules that made us loose TPC since the 1st of
>>> Jan 2019, i.e. in only 6 days…
>>>
>>> Note: It also shows (that’s the positive aspect), that overall we won
>>> 0.1371 of global TPC. But we could have won more if we can understand why
>>> we lost some TPC and find out if there are actions we can put in place to
>>> avoid that.
>>>
>>> So I think it would be good if devs who have touched these modules recently
>>> could analyze the cause of the decrease and verify they’re real and what we
>>> want to do about them.
>>>
>>> WDYT?
>>
>> Two impressive results are:
>> * xwiki-platform-url-api: +11.9999%
>
> https://github.com/xwiki/xwiki-platform/commit/00d2a7af3e9d3a7c313738ef93843407f5c29ab0
> I think
Cool. It shows that this code being tested was large compared to what we had
before. So that’s cool that you fixed it.
Now we’re still negative compared to 2017 so we wouldn’t need a bit more tests
somewhere:
xwiki-platform-url-api 89.5652 86.909 -2.6561
>
>> * xwiki-rendering-syntax-wikimodel: +28.587%
>
> Probably because of
> https://github.com/xwiki/xwiki-rendering/commit/85887ab217df3a4edc88ee81ad8b796e377bbada
> which removed a pretty big untested class.
haha… indeed, I had forgotten :)
I removed code to fix the big regression in global TPC since 2017: from 76.5886
to 64.3207 (i.e. -12.2678%). Source: Report for 20171222-1835 -> 20190101-2330
(http://maven.xwiki.org/site/clover/20190101/XWikiReport-20171222-1835-20190101-2330.html).
So it makes sense that the report from 20190101 to 20190106 shows improvement
(wouldn’t have thought it would be as high as 28.587% though).
Just checked and the new global report since 2017 now shows (source:
http://maven.xwiki.org/site/clover/20190110/XWikiReport-20171222-1835-20190110-0224.html):
xwiki-rendering-syntax-wikimodel 76.5886 92.9078 16.3191
Thanks
-Vincent
>
>>
>> Any idea why these 2 were increased so much in the past 6 days?
>>
>> Thanks
>> -Vincent
>>
>>>
>>> Thanks!
>>> -Vincent
>>>
>>>> On 4 Jan 2019, at 15:14, Vincent Massol <vinc...@massol.net> wrote:
>>>>
>>>> Hi devs,
>>>>
>>>> FYI I've modified the clover job to do the following today:
>>>> * run daily around midnight on a4
>>>> * generate 2 diff reports: one against 20171220 (that's the one we're
>>>> currently getting) and one against 20190101
>>>> * the first diff report doesn’t send an email on failure while the second
>>>> one does.
>>>>
>>>> I also changed yesterday:
>>>> * Only mark in RED (ie failures) modules where the global diff TPC is
>>>> negative (before it was in RED when the global contribution was < 0 but
>>>> that wasn’t accounting for removed modules for example).
>>>>
>>>> Consequences:
>>>> * When we get report failures from the CI we need to fix it since it means
>>>> we’ve regressed since 20190101
>>>> * It would still be good that we continue to analyze the diff report since
>>>> 20171220 to fully understand it and make sure we haven’t regressed in
>>>> quality since then, or at least fix the big regressions.
>>>>
>>>> TODO:
>>>> * Need to find a way to indicate false positives, i.e. modules in RED for
>>>> which we consider it’s ok to have a negative diff TPC, and thus know
>>>> what’s left to do for the release.
>>>> * Once we better understand and fix modules we’ll get to the second step
>>>> which is to automate the baseline value (manual ATM) and to have the
>>>> report passing as a constraint for releasing and thus indicated in the
>>>> Release Plan.
>>>>
>>>> Let me know if you have questions.
>>>>
>>>> Thanks
>>>> -Vincent
>>>>
>>>>> On 3 Oct 2018, at 10:56, Vincent Massol <vinc...@massol.net> wrote:
>>>>>
>>>>> Hi devs,
>>>>>
>>>>> We started discussing a first global test coverage strategy in
>>>>> https://markmail.org/message/grphwta63pp5p4l7
>>>>>
>>>>> I’d like to propose some updates and tuning now that we have a Clover
>>>>> Jenkins pipeline working (brainstormed with Simon):
>>>>>
>>>>> Here’s the new proposal:
>>>>>
>>>>> * We run the Clover Jenkins pipeline every night (between 11PM-8AM)
>>>>> * The pipeline sends an email whenever the new report has modules
>>>>> lowering the global TPC score compared to the baseline report (negative
>>>>> contribution per module)
>>>>> * The baseline report is the report generated just after each XS release.
>>>>> This means that we keep the same baseline during a XS release
>>>>> ** Technically it means that the pipeline will update the latest.txt file
>>>>> (which contains the clover report timestamp) when it notices a version
>>>>> change
>>>>> * We add a step in the Release Plan Template to have the report passing
>>>>> before we can release.
>>>>> * The RM is in charge of a release from day 1 to the release day (already
>>>>> the case), and is also in charge of making sure that the global coverage
>>>>> job failures get addressed before the release day so that we’re ready on
>>>>> the release day.
>>>>>
>>>>> Options:
>>>>> * Make it easier and only fail the pipeline job when the global TPC value
>>>>> is lower than the baseline (vs failing whenever a module has negative
>>>>> contribution)
>>>>>
>>>>> WDYT?
>>>>>
>>>>> Thanks
>>>>> -Vincent
>>>>>
>>>>
>>>
>>
>
>
> --
> Thomas Mortagne
