On Sun, Jan 6, 2019 at 12:06 PM Vincent Massol <vinc...@massol.net> wrote: > > > > > On 6 Jan 2019, at 12:03, Vincent Massol <vinc...@massol.net> wrote: > > > > Hi devs, > > > > I ran the modified Clover job yesterday and it generated the 2 reports: > > * Report - 20171222-1835 -> 20190106-0207 : > > http://maven.xwiki.org/site/clover/20190106/XWikiReport-20171222-1835-20190106-0207.html > > * (New) Report Report - 20190101-2330 -> 20190106-0207 : > > http://maven.xwiki.org/site/clover/20190106/XWikiReport-20190101-2330-20190106-0207.html > > > > This is quite interesting. > > > > The second report shows the modules that made us loose TPC since the 1st of > > Jan 2019, i.e. in only 6 days… > > > > Note: It also shows (that’s the positive aspect), that overall we won > > 0.1371 of global TPC. But we could have won more if we can understand why > > we lost some TPC and find out if there are actions we can put in place to > > avoid that. > > > > So I think it would be good if devs who have touched these modules recently > > could analyze the cause of the decrease and verify they’re real and what we > > want to do about them. > > > > WDYT? > > Two impressive results are: > * xwiki-platform-url-api: +11.9999%
https://github.com/xwiki/xwiki-platform/commit/00d2a7af3e9d3a7c313738ef93843407f5c29ab0 I think > * xwiki-rendering-syntax-wikimodel: +28.587% Probably because of https://github.com/xwiki/xwiki-rendering/commit/85887ab217df3a4edc88ee81ad8b796e377bbada which removed a pretty big untested class. > > Any idea why these 2 were increased so much in the past 6 days? > > Thanks > -Vincent > > > > > Thanks! > > -Vincent > > > >> On 4 Jan 2019, at 15:14, Vincent Massol <vinc...@massol.net> wrote: > >> > >> Hi devs, > >> > >> FYI I've modified the clover job to do the following today: > >> * run daily around midnight on a4 > >> * generate 2 diff reports: one against 20171220 (that's the one we're > >> currently getting) and one against 20190101 > >> * the first diff report doesn’t send an email on failure while the second > >> one does. > >> > >> I also changed yesterday: > >> * Only mark in RED (ie failures) modules where the global diff TPC is > >> negative (before it was in RED when the global contribution was < 0 but > >> that wasn’t accounting for removed modules for example). > >> > >> Consequences: > >> * When we get report failures from the CI we need to fix it since it means > >> we’ve regressed since 20190101 > >> * It would still be good that we continue to analyze the diff report since > >> 20171220 to fully understand it and make sure we haven’t regressed in > >> quality since then, or at least fix the big regressions. > >> > >> TODO: > >> * Need to find a way to indicate false positives, i.e. modules in RED for > >> which we consider it’s ok to have a negative diff TPC, and thus know > >> what’s left to do for the release. > >> * Once we better understand and fix modules we’ll get to the second step > >> which is to automate the baseline value (manual ATM) and to have the > >> report passing as a constraint for releasing and thus indicated in the > >> Release Plan. > >> > >> Let me know if you have questions. > >> > >> Thanks > >> -Vincent > >> > >>> On 3 Oct 2018, at 10:56, Vincent Massol <vinc...@massol.net> wrote: > >>> > >>> Hi devs, > >>> > >>> We started discussing a first global test coverage strategy in > >>> https://markmail.org/message/grphwta63pp5p4l7 > >>> > >>> I’d like to propose some updates and tuning now that we have a Clover > >>> Jenkins pipeline working (brainstormed with Simon): > >>> > >>> Here’s the new proposal: > >>> > >>> * We run the Clover Jenkins pipeline every night (between 11PM-8AM) > >>> * The pipeline sends an email whenever the new report has modules > >>> lowering the global TPC score compared to the baseline report (negative > >>> contribution per module) > >>> * The baseline report is the report generated just after each XS release. > >>> This means that we keep the same baseline during a XS release > >>> ** Technically it means that the pipeline will update the latest.txt file > >>> (which contains the clover report timestamp) when it notices a version > >>> change > >>> * We add a step in the Release Plan Template to have the report passing > >>> before we can release. > >>> * The RM is in charge of a release from day 1 to the release day (already > >>> the case), and is also in charge of making sure that the global coverage > >>> job failures get addressed before the release day so that we’re ready on > >>> the release day. > >>> > >>> Options: > >>> * Make it easier and only fail the pipeline job when the global TPC value > >>> is lower than the baseline (vs failing whenever a module has negative > >>> contribution) > >>> > >>> WDYT? > >>> > >>> Thanks > >>> -Vincent > >>> > >> > > > -- Thomas Mortagne
