Re: [xwiki-devs] [Proposal] Improve our Test Coverage strategy (TAKE 3)
+1 Thanks, Marius On Thu, Mar 21, 2019 at 7:03 PM Thomas Mortagne wrote: > +1 > > On Thu, Mar 21, 2019 at 5:41 PM Vincent Massol wrote: > > > > Hi devs, > > > > I remember that we talked about this new global test coverage strategy > but I don’t recall where and I couldn’t find an email about it. So I’m > reposting the strategy that I’ve just finished implementing. > > > > * Every day the Clover job runs: > https://ci.xwiki.org/view/Tools/job/Clover/ > > * It generates a global coverage report and compares it automatically > with the first report generated for the current version > > * If the global coverage is reduced, then: > > ** an email is sent to notificati...@xwiki.org containing the report > > ** the job is marked as FAILING > > ** A failing badge is added to the job history > > ** A red large text is added to the job page and a link to the report is > sent > > * Developers MUST fix the global coverage before we can release a > version, so the it means the global coverage must not be reduced during a > version. > > * The RM should check the CI (already part of his chores) and the > “Recommended Failing” view on ci.xwiki.org must not have errors or > failures (so that includes pitest job and clover job). > > > > Any comments? Does that seem ok to you? > > > > If you confirm it’s ok, I’ll document it on > https://dev.xwiki.org/xwiki/bin/view/Community/Testing/TestCoverage/ > > > > Note: A few minutes ago an email has been sent since the global coverage > has been reduced compared to yesterday. > > > > See > http://maven.xwiki.org/site/clover/20190321/XWikiReport-20190320-0128-20190321-1059.html > > > > So we need to fix it now. > > > > Thanks > > -Vincent > > > > > -- > Thomas Mortagne >
Re: [xwiki-devs] [Proposal] Improve our Test Coverage strategy (TAKE 3)
+1 On Thu, Mar 21, 2019 at 5:41 PM Vincent Massol wrote: > > Hi devs, > > I remember that we talked about this new global test coverage strategy but I > don’t recall where and I couldn’t find an email about it. So I’m reposting > the strategy that I’ve just finished implementing. > > * Every day the Clover job runs: https://ci.xwiki.org/view/Tools/job/Clover/ > * It generates a global coverage report and compares it automatically with > the first report generated for the current version > * If the global coverage is reduced, then: > ** an email is sent to notificati...@xwiki.org containing the report > ** the job is marked as FAILING > ** A failing badge is added to the job history > ** A red large text is added to the job page and a link to the report is sent > * Developers MUST fix the global coverage before we can release a version, so > the it means the global coverage must not be reduced during a version. > * The RM should check the CI (already part of his chores) and the > “Recommended Failing” view on ci.xwiki.org must not have errors or failures > (so that includes pitest job and clover job). > > Any comments? Does that seem ok to you? > > If you confirm it’s ok, I’ll document it on > https://dev.xwiki.org/xwiki/bin/view/Community/Testing/TestCoverage/ > > Note: A few minutes ago an email has been sent since the global coverage has > been reduced compared to yesterday. > > See > http://maven.xwiki.org/site/clover/20190321/XWikiReport-20190320-0128-20190321-1059.html > > So we need to fix it now. > > Thanks > -Vincent > -- Thomas Mortagne
Re: [xwiki-devs] [Proposal] Improve our Test Coverage strategy (TAKE 3)
+1 Thanks, Marius On Sun, Feb 10, 2019 at 3:50 PM Vincent Massol wrote: > Hi devs, > > After TAKE 2 (see http://markmail.org/message/owtyhkmrz4tcbymn ), and > after analyzing several modules (I analyzed about 4-5 in total), I think we > should improve a bit the strategy to make it usable and applicable. > > Lessons learnt > > > * It takes a lot of time to analyze a single global tpc drop (every time > it takes me around 2 hours) > * In general the results of the analysis are not that great. There are > often “good enough” reasons for the drop. It’s often a lack of unit tests > and code that is exercised by functional tests but the path has changed for > various reasons. > * I find the ratio of time to spend on the analysis vs result to be too > low. > * In the end what’s important is that our global TPC continues to grow > > New Strategy > === > > * We run the Clover Jenkins pipeline every night (between 11PM-8AM) > * The pipeline sends an email whenever the new report has its global TPC > going down when compared with the baseline (vs one or more modules had > their TPC lowered in TAKE 2) > * The baseline report is the report generated just after each XS release. > This means that we keep the same baseline during a XS release > ** Technically it means that the pipeline will update the latest.txt file > (which contains the clover report timestamp) when it notices a version > change > * We add a step in the Release Plan Template to have the report passing > before we can release. > * The RM is in charge of a release from day 1 to the release day (already > the case), and is also in charge of making sure that the global coverage > job failures get addressed before the release day so that we’re ready on > the release day. > * Implementation detail: don’t send a failure email when there are failing > tests in the build, to avoid false positives > > WDYT? > > Thanks > -Vincent
Re: [xwiki-devs] [Proposal] Improve our Test Coverage strategy (TAKE 3)
Hi Simon, > On 11 Feb 2019, at 15:43, Simon Urli wrote: > > So if I sum up, we're moving the scope of global coverage from each modules > to the whole project, right? Correct. > +1 for getting email/build failing only if the global TPC decrease > > Now I assume in case of decrease, we can get the whole report with info about > which modules had a change in their TPC, right? Correct again :) Actually it’s even my goal to improve it and also give details about package level (easy) and maybe even class level (harder), for “failing” modules. Thanks -Vincent > Simon > > On 11/02/2019 14:48, Thomas Mortagne wrote: >> +1 >> On Sun, Feb 10, 2019 at 2:50 PM Vincent Massol wrote: >>> >>> Hi devs, >>> >>> After TAKE 2 (see http://markmail.org/message/owtyhkmrz4tcbymn ), and >>> after analyzing several modules (I analyzed about 4-5 in total), I think we >>> should improve a bit the strategy to make it usable and applicable. >>> >>> Lessons learnt >>> >>> >>> * It takes a lot of time to analyze a single global tpc drop (every time it >>> takes me around 2 hours) >>> * In general the results of the analysis are not that great. There are >>> often “good enough” reasons for the drop. It’s often a lack of unit tests >>> and code that is exercised by functional tests but the path has changed for >>> various reasons. >>> * I find the ratio of time to spend on the analysis vs result to be too low. >>> * In the end what’s important is that our global TPC continues to grow >>> >>> New Strategy >>> === >>> >>> * We run the Clover Jenkins pipeline every night (between 11PM-8AM) >>> * The pipeline sends an email whenever the new report has its global TPC >>> going down when compared with the baseline (vs one or more modules had >>> their TPC lowered in TAKE 2) >>> * The baseline report is the report generated just after each XS release. >>> This means that we keep the same baseline during a XS release >>> ** Technically it means that the pipeline will update the latest.txt file >>> (which contains the clover report timestamp) when it notices a version >>> change >>> * We add a step in the Release Plan Template to have the report passing >>> before we can release. >>> * The RM is in charge of a release from day 1 to the release day (already >>> the case), and is also in charge of making sure that the global coverage >>> job failures get addressed before the release day so that we’re ready on >>> the release day. >>> * Implementation detail: don’t send a failure email when there are failing >>> tests in the build, to avoid false positives >>> >>> WDYT? >>> >>> Thanks >>> -Vincent > > -- > Simon Urli > Software Engineer at XWiki SAS > simon.u...@xwiki.com > More about us at http://www.xwiki.com
Re: [xwiki-devs] [Proposal] Improve our Test Coverage strategy (TAKE 3)
So if I sum up, we're moving the scope of global coverage from each modules to the whole project, right? +1 for getting email/build failing only if the global TPC decrease Now I assume in case of decrease, we can get the whole report with info about which modules had a change in their TPC, right? Simon On 11/02/2019 14:48, Thomas Mortagne wrote: +1 On Sun, Feb 10, 2019 at 2:50 PM Vincent Massol wrote: Hi devs, After TAKE 2 (see http://markmail.org/message/owtyhkmrz4tcbymn ), and after analyzing several modules (I analyzed about 4-5 in total), I think we should improve a bit the strategy to make it usable and applicable. Lessons learnt * It takes a lot of time to analyze a single global tpc drop (every time it takes me around 2 hours) * In general the results of the analysis are not that great. There are often “good enough” reasons for the drop. It’s often a lack of unit tests and code that is exercised by functional tests but the path has changed for various reasons. * I find the ratio of time to spend on the analysis vs result to be too low. * In the end what’s important is that our global TPC continues to grow New Strategy === * We run the Clover Jenkins pipeline every night (between 11PM-8AM) * The pipeline sends an email whenever the new report has its global TPC going down when compared with the baseline (vs one or more modules had their TPC lowered in TAKE 2) * The baseline report is the report generated just after each XS release. This means that we keep the same baseline during a XS release ** Technically it means that the pipeline will update the latest.txt file (which contains the clover report timestamp) when it notices a version change * We add a step in the Release Plan Template to have the report passing before we can release. * The RM is in charge of a release from day 1 to the release day (already the case), and is also in charge of making sure that the global coverage job failures get addressed before the release day so that we’re ready on the release day. * Implementation detail: don’t send a failure email when there are failing tests in the build, to avoid false positives WDYT? Thanks -Vincent -- Simon Urli Software Engineer at XWiki SAS simon.u...@xwiki.com More about us at http://www.xwiki.com
Re: [xwiki-devs] [Proposal] Improve our Test Coverage strategy (TAKE 3)
+1 On Sun, Feb 10, 2019 at 2:50 PM Vincent Massol wrote: > > Hi devs, > > After TAKE 2 (see http://markmail.org/message/owtyhkmrz4tcbymn ), and after > analyzing several modules (I analyzed about 4-5 in total), I think we should > improve a bit the strategy to make it usable and applicable. > > Lessons learnt > > > * It takes a lot of time to analyze a single global tpc drop (every time it > takes me around 2 hours) > * In general the results of the analysis are not that great. There are often > “good enough” reasons for the drop. It’s often a lack of unit tests and code > that is exercised by functional tests but the path has changed for various > reasons. > * I find the ratio of time to spend on the analysis vs result to be too low. > * In the end what’s important is that our global TPC continues to grow > > New Strategy > === > > * We run the Clover Jenkins pipeline every night (between 11PM-8AM) > * The pipeline sends an email whenever the new report has its global TPC > going down when compared with the baseline (vs one or more modules had their > TPC lowered in TAKE 2) > * The baseline report is the report generated just after each XS release. > This means that we keep the same baseline during a XS release > ** Technically it means that the pipeline will update the latest.txt file > (which contains the clover report timestamp) when it notices a version change > * We add a step in the Release Plan Template to have the report passing > before we can release. > * The RM is in charge of a release from day 1 to the release day (already the > case), and is also in charge of making sure that the global coverage job > failures get addressed before the release day so that we’re ready on the > release day. > * Implementation detail: don’t send a failure email when there are failing > tests in the build, to avoid false positives > > WDYT? > > Thanks > -Vincent -- Thomas Mortagne
