Re: [xwiki-devs] Improving tests with Descartes
> On 25 Aug 2018, at 20:41, Vincent Massol wrote: > > Hi devs, > >> On 30 Jul 2018, at 12:39, Vincent Massol wrote: >> >> Hi devs, >> >> It would be great if you could help improve our unit tests using Descartes. >> This is needed for the STAMP research project >> (https://www.stamp-project.eu/view/main/) and will benefit XWiki by having 2 >> effects: >> * increasing the test coverage >> * improving the tests themselves (increasing their mutation score) >> >> Since 10.7 is 50% testing and 50% BFD, it would be great if you could spend >> all or a substantial part of your testing time working on this. > > Just thought about something. In order to know if Descartes (and other STAMP > tools) are interesting in the longer run, there’s one more parameter that I > think we should measure during our test sessions described below: the time it > took to improve by N% the coverage and P% the mutation score. > > Because, improving the TPC of a module is not an enough data by itself IMO. > Imagine that we have 1 free hour to work on testing. We could choose to do: > * Option 1: write a new test for some code that has not been tested and that > currently has 0% coverage and bring it to, say, 20% > * Option 2: use Descartes to help improve the quality of an existing tests > and improve the test coverage by, say, 2% > > If option 1 takes 2 hours and option 2 takes 5 minutes then option 2 might be > more interesting. If OTH option 1 takes 1 hour and option 2 takes 15mn then > option 1 is more interesting by far. > > Yes, in order to compare we would need to know also the time it takes to > write new tests. > > However I think that some order of magnitude for using Descartes could > already provide interesting info to us. > > Thus, if you agree if would be nice to also record this in the test data we > record at > https://github.com/STAMP-project/descartes-usecases-output/tree/master/xwiki > in each session. This point has now been discussed with the STAMP research project and apparently it was already the case that we need to measure the time for each test session… :) Thanks -Vincent > >> I propose the following strategy: >> * You find a module you want to work on. >> * In that module you run: mvn clean install -Pquality >> -Dxwiki.pitest.skip=false >> * Then you check target/pit-reports//issues/index.html and verify if >> there are "pseudo tested" methods listed (when we have finished fixing all >> of those we can move to “partially tested methods”). >> * If there are some, then please record the current jacoco threshold and the >> current mutation score. >> * You can get the jacoco threshold by running "mvn clean install -Pquality >> -Dxwiki.pitest.skip=false -Dxwiki.pitest.mutationThreshold=100” (or by >> checking target/pit-reports//index.html, I haven’t checked yet if they >> are the same). >> * You can get the current mutation score by checking >> target/pit-reports//index.html > > * Also record the current time > >> * Then fix the test so that Descartes doesn’t report any pseudo tested or >> partially tested methods > > * Record the time again > >> * Update the jacoco threshold and the mutation scores in the pom.xml >> * Send a PR on >> https://github.com/STAMP-project/descartes-usecases-output/tree/master/xwiki >> using the format already defined there. > > * And include the time in the PR. > > WDYT? > > Thanks > -Vincent > > PS: I’ll try to roughly estimate the times it took me to do the test sessions > already in > https://github.com/STAMP-project/descartes-usecases-output/tree/master/xwiki > >> WDYT? Doable? >> >> Thanks >> -Vincent
Re: [xwiki-devs] Improving tests with Descartes
Hi devs, > On 30 Jul 2018, at 12:39, Vincent Massol wrote: > > Hi devs, > > It would be great if you could help improve our unit tests using Descartes. > This is needed for the STAMP research project > (https://www.stamp-project.eu/view/main/) and will benefit XWiki by having 2 > effects: > * increasing the test coverage > * improving the tests themselves (increasing their mutation score) > > Since 10.7 is 50% testing and 50% BFD, it would be great if you could spend > all or a substantial part of your testing time working on this. Just thought about something. In order to know if Descartes (and other STAMP tools) are interesting in the longer run, there’s one more parameter that I think we should measure during our test sessions described below: the time it took to improve by N% the coverage and P% the mutation score. Because, improving the TPC of a module is not an enough data by itself IMO. Imagine that we have 1 free hour to work on testing. We could choose to do: * Option 1: write a new test for some code that has not been tested and that currently has 0% coverage and bring it to, say, 20% * Option 2: use Descartes to help improve the quality of an existing tests and improve the test coverage by, say, 2% If option 1 takes 2 hours and option 2 takes 5 minutes then option 2 might be more interesting. If OTH option 1 takes 1 hour and option 2 takes 15mn then option 1 is more interesting by far. Yes, in order to compare we would need to know also the time it takes to write new tests. However I think that some order of magnitude for using Descartes could already provide interesting info to us. Thus, if you agree if would be nice to also record this in the test data we record at https://github.com/STAMP-project/descartes-usecases-output/tree/master/xwiki in each session. > I propose the following strategy: > * You find a module you want to work on. > * In that module you run: mvn clean install -Pquality > -Dxwiki.pitest.skip=false > * Then you check target/pit-reports//issues/index.html and verify if > there are "pseudo tested" methods listed (when we have finished fixing all of > those we can move to “partially tested methods”). > * If there are some, then please record the current jacoco threshold and the > current mutation score. > * You can get the jacoco threshold by running "mvn clean install -Pquality > -Dxwiki.pitest.skip=false -Dxwiki.pitest.mutationThreshold=100” (or by > checking target/pit-reports//index.html, I haven’t checked yet if they > are the same). > * You can get the current mutation score by checking > target/pit-reports//index.html * Also record the current time > * Then fix the test so that Descartes doesn’t report any pseudo tested or > partially tested methods * Record the time again > * Update the jacoco threshold and the mutation scores in the pom.xml > * Send a PR on > https://github.com/STAMP-project/descartes-usecases-output/tree/master/xwiki > using the format already defined there. * And include the time in the PR. WDYT? Thanks -Vincent PS: I’ll try to roughly estimate the times it took me to do the test sessions already in https://github.com/STAMP-project/descartes-usecases-output/tree/master/xwiki > WDYT? Doable? > > Thanks > -Vincent >
Re: [xwiki-devs] Improving tests with Descartes
Note: Descartes (or Pitest) seems be a bit lost with tests located on
abstract classes. For example it pretty much react as if those 3
classes where not existing:
https://github.com/xwiki/xwiki-commons/tree/master/xwiki-commons-core/xwiki-commons-filter/xwiki-commons-filter-api/src/test/java/org/xwiki/filter/internal
(which makes the score way lower than it should I think).
On Thu, Aug 9, 2018 at 6:10 PM, Thomas Mortagne
wrote:
> Done: https://github.com/STAMP-project/descartes-usecases-output/pull/1
>
> On Thu, Aug 9, 2018 at 3:07 PM, Thomas Mortagne
> wrote:
>> Started looking at xwiki-commons-extension-api by the way.
>>
>> On Thu, Aug 9, 2018 at 11:50 AM, Thomas Mortagne
>> wrote:
>>> This is probably because xwiki-commons-extension-api contains a lot of
>>> what is closer to integration than unit tests.
>>>
>>> On Tue, Aug 7, 2018 at 6:07 PM, Vincent Massol wrote:
Note for Thomas (or whoever else is interested in improving the commons
extension tests) there are plenty in
./xwiki-commons-core/xwiki-commons-extension/xwiki-commons-extension-api/target/pit-reports/201808071749/methods.json:pseudo-tested
pseudo-tested
pseudo-tested
partially-tested
partially-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
partially-tested
pseudo-tested
pseudo-tested
partially-tested
pseudo-tested
partially-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
partially-tested
partially-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
partially-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
partially-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
partially-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
partially-tested
partially-tested
pseudo-tested
partially-tested
partially-tested
partially-tested
partially-tested
partially-tested
pseudo-tested
pseudo-tested
Fixing those should yield a good increase in both mutation score and test
coverage.
Thanks
-Vincent
> On 7 Aug 2018, at 18:05, Vincent Massol wrote:
>
>
>
>> On 30 Jul 2018, at 12:39, Vincent Massol wrote:
>>
>> Hi devs,
>>
>> It would be great if you could help improve our unit tests using
>> Descartes. This is needed for the STAMP research project
>> (https://www.stamp-project.eu/view/main/) and will benefit XWiki by
>> having 2 effects:
>> * increasing the test coverage
>> * improving the tests themselves (increasing their mutation score)
>>
>> Since 10.7 is 50% testing and 50% BFD, it would be great if you could
>> spend all or a substantial part of your testing time working on this.
>>
>> I propose the following strategy:
>> * You find a module you want to work on.
>> * In that module you run: mvn clean install -Pquality
>> -Dxwiki.pitest.skip=false
>
> You can also run that at the top of a multimodule project and then find
> pseudo/partially tested methods with:
>
> find . -name "methods.json" -exec egrep -oH
> "pseudo-tested|partially-tested" {} \;
>
> Thanks
> -Vincent
>
>> * Then you check target/pit-reports//issues/index.html and verify
>> if there are "pseudo tested" methods listed (when we have finished
>> fixing all of those we can move to “partially tested methods”).
>> * If there are some, then please record the current jacoco threshold and
>> the current mutation score.
>> * You can get the jacoco threshold by running "mvn clean install
>> -Pquality -Dxwiki.pitest.skip=false
>> -Dxwiki.pitest.mutationThreshold=100” (or by checking
>> target/pit-reports//index.html, I haven’t checked yet if they are
>> the same).
>> * You can get the current mutation score by checking
>> target/pit-reports//index.html
>> * Then fix the test so that Descartes doesn’t report any pseudo tested
>> or partially tested methods
>> * Update the jacoco threshold and the mutation scores in the pom.xml
>> * Send a PR on
>> https://github.com/STAMP-project/descartes-usecases-output/tree/master/xwiki
>> using the format already defined there.
>>
>> WDYT? Doable?
>>
>> Thanks
>> -Vincent
>>>
>>>
>>>
>>> --
>>> Thomas Mortagne
>>
>>
>>
>> --
>> Thomas Mortagne
>
>
>
> --
> Thomas Mortagne
--
Thomas Mortagne
Re: [xwiki-devs] Improving tests with Descartes
Done: https://github.com/STAMP-project/descartes-usecases-output/pull/1
On Thu, Aug 9, 2018 at 3:07 PM, Thomas Mortagne
wrote:
> Started looking at xwiki-commons-extension-api by the way.
>
> On Thu, Aug 9, 2018 at 11:50 AM, Thomas Mortagne
> wrote:
>> This is probably because xwiki-commons-extension-api contains a lot of
>> what is closer to integration than unit tests.
>>
>> On Tue, Aug 7, 2018 at 6:07 PM, Vincent Massol wrote:
>>> Note for Thomas (or whoever else is interested in improving the commons
>>> extension tests) there are plenty in
>>>
>>> ./xwiki-commons-core/xwiki-commons-extension/xwiki-commons-extension-api/target/pit-reports/201808071749/methods.json:pseudo-tested
>>> pseudo-tested
>>> pseudo-tested
>>> partially-tested
>>> partially-tested
>>> pseudo-tested
>>> pseudo-tested
>>> pseudo-tested
>>> pseudo-tested
>>> pseudo-tested
>>> partially-tested
>>> pseudo-tested
>>> pseudo-tested
>>> partially-tested
>>> pseudo-tested
>>> partially-tested
>>> pseudo-tested
>>> pseudo-tested
>>> pseudo-tested
>>> pseudo-tested
>>> pseudo-tested
>>> pseudo-tested
>>> pseudo-tested
>>> partially-tested
>>> partially-tested
>>> pseudo-tested
>>> pseudo-tested
>>> pseudo-tested
>>> pseudo-tested
>>> partially-tested
>>> pseudo-tested
>>> pseudo-tested
>>> pseudo-tested
>>> pseudo-tested
>>> pseudo-tested
>>> pseudo-tested
>>> pseudo-tested
>>> pseudo-tested
>>> partially-tested
>>> pseudo-tested
>>> pseudo-tested
>>> pseudo-tested
>>> pseudo-tested
>>> pseudo-tested
>>> pseudo-tested
>>> pseudo-tested
>>> pseudo-tested
>>> partially-tested
>>> pseudo-tested
>>> pseudo-tested
>>> pseudo-tested
>>> pseudo-tested
>>> pseudo-tested
>>> pseudo-tested
>>> pseudo-tested
>>> partially-tested
>>> partially-tested
>>> pseudo-tested
>>> partially-tested
>>> partially-tested
>>> partially-tested
>>> partially-tested
>>> partially-tested
>>> pseudo-tested
>>> pseudo-tested
>>>
>>> Fixing those should yield a good increase in both mutation score and test
>>> coverage.
>>>
>>> Thanks
>>> -Vincent
>>>
On 7 Aug 2018, at 18:05, Vincent Massol wrote:
> On 30 Jul 2018, at 12:39, Vincent Massol wrote:
>
> Hi devs,
>
> It would be great if you could help improve our unit tests using
> Descartes. This is needed for the STAMP research project
> (https://www.stamp-project.eu/view/main/) and will benefit XWiki by
> having 2 effects:
> * increasing the test coverage
> * improving the tests themselves (increasing their mutation score)
>
> Since 10.7 is 50% testing and 50% BFD, it would be great if you could
> spend all or a substantial part of your testing time working on this.
>
> I propose the following strategy:
> * You find a module you want to work on.
> * In that module you run: mvn clean install -Pquality
> -Dxwiki.pitest.skip=false
You can also run that at the top of a multimodule project and then find
pseudo/partially tested methods with:
find . -name "methods.json" -exec egrep -oH
"pseudo-tested|partially-tested" {} \;
Thanks
-Vincent
> * Then you check target/pit-reports//issues/index.html and verify
> if there are "pseudo tested" methods listed (when we have finished fixing
> all of those we can move to “partially tested methods”).
> * If there are some, then please record the current jacoco threshold and
> the current mutation score.
> * You can get the jacoco threshold by running "mvn clean install
> -Pquality -Dxwiki.pitest.skip=false -Dxwiki.pitest.mutationThreshold=100”
> (or by checking target/pit-reports//index.html, I haven’t checked
> yet if they are the same).
> * You can get the current mutation score by checking
> target/pit-reports//index.html
> * Then fix the test so that Descartes doesn’t report any pseudo tested or
> partially tested methods
> * Update the jacoco threshold and the mutation scores in the pom.xml
> * Send a PR on
> https://github.com/STAMP-project/descartes-usecases-output/tree/master/xwiki
> using the format already defined there.
>
> WDYT? Doable?
>
> Thanks
> -Vincent
>>>
>>
>>
>>
>> --
>> Thomas Mortagne
>
>
>
> --
> Thomas Mortagne
--
Thomas Mortagne
Re: [xwiki-devs] Improving tests with Descartes
Started looking at xwiki-commons-extension-api by the way.
On Thu, Aug 9, 2018 at 11:50 AM, Thomas Mortagne
wrote:
> This is probably because xwiki-commons-extension-api contains a lot of
> what is closer to integration than unit tests.
>
> On Tue, Aug 7, 2018 at 6:07 PM, Vincent Massol wrote:
>> Note for Thomas (or whoever else is interested in improving the commons
>> extension tests) there are plenty in
>>
>> ./xwiki-commons-core/xwiki-commons-extension/xwiki-commons-extension-api/target/pit-reports/201808071749/methods.json:pseudo-tested
>> pseudo-tested
>> pseudo-tested
>> partially-tested
>> partially-tested
>> pseudo-tested
>> pseudo-tested
>> pseudo-tested
>> pseudo-tested
>> pseudo-tested
>> partially-tested
>> pseudo-tested
>> pseudo-tested
>> partially-tested
>> pseudo-tested
>> partially-tested
>> pseudo-tested
>> pseudo-tested
>> pseudo-tested
>> pseudo-tested
>> pseudo-tested
>> pseudo-tested
>> pseudo-tested
>> partially-tested
>> partially-tested
>> pseudo-tested
>> pseudo-tested
>> pseudo-tested
>> pseudo-tested
>> partially-tested
>> pseudo-tested
>> pseudo-tested
>> pseudo-tested
>> pseudo-tested
>> pseudo-tested
>> pseudo-tested
>> pseudo-tested
>> pseudo-tested
>> partially-tested
>> pseudo-tested
>> pseudo-tested
>> pseudo-tested
>> pseudo-tested
>> pseudo-tested
>> pseudo-tested
>> pseudo-tested
>> pseudo-tested
>> partially-tested
>> pseudo-tested
>> pseudo-tested
>> pseudo-tested
>> pseudo-tested
>> pseudo-tested
>> pseudo-tested
>> pseudo-tested
>> partially-tested
>> partially-tested
>> pseudo-tested
>> partially-tested
>> partially-tested
>> partially-tested
>> partially-tested
>> partially-tested
>> pseudo-tested
>> pseudo-tested
>>
>> Fixing those should yield a good increase in both mutation score and test
>> coverage.
>>
>> Thanks
>> -Vincent
>>
>>> On 7 Aug 2018, at 18:05, Vincent Massol wrote:
>>>
>>>
>>>
On 30 Jul 2018, at 12:39, Vincent Massol wrote:
Hi devs,
It would be great if you could help improve our unit tests using
Descartes. This is needed for the STAMP research project
(https://www.stamp-project.eu/view/main/) and will benefit XWiki by having
2 effects:
* increasing the test coverage
* improving the tests themselves (increasing their mutation score)
Since 10.7 is 50% testing and 50% BFD, it would be great if you could
spend all or a substantial part of your testing time working on this.
I propose the following strategy:
* You find a module you want to work on.
* In that module you run: mvn clean install -Pquality
-Dxwiki.pitest.skip=false
>>>
>>> You can also run that at the top of a multimodule project and then find
>>> pseudo/partially tested methods with:
>>>
>>> find . -name "methods.json" -exec egrep -oH
>>> "pseudo-tested|partially-tested" {} \;
>>>
>>> Thanks
>>> -Vincent
>>>
* Then you check target/pit-reports//issues/index.html and verify if
there are "pseudo tested" methods listed (when we have finished fixing all
of those we can move to “partially tested methods”).
* If there are some, then please record the current jacoco threshold and
the current mutation score.
* You can get the jacoco threshold by running "mvn clean install -Pquality
-Dxwiki.pitest.skip=false -Dxwiki.pitest.mutationThreshold=100” (or by
checking target/pit-reports//index.html, I haven’t checked yet if
they are the same).
* You can get the current mutation score by checking
target/pit-reports//index.html
* Then fix the test so that Descartes doesn’t report any pseudo tested or
partially tested methods
* Update the jacoco threshold and the mutation scores in the pom.xml
* Send a PR on
https://github.com/STAMP-project/descartes-usecases-output/tree/master/xwiki
using the format already defined there.
WDYT? Doable?
Thanks
-Vincent
>>
>
>
>
> --
> Thomas Mortagne
--
Thomas Mortagne
Re: [xwiki-devs] Improving tests with Descartes
This is probably because xwiki-commons-extension-api contains a lot of
what is closer to integration than unit tests.
On Tue, Aug 7, 2018 at 6:07 PM, Vincent Massol wrote:
> Note for Thomas (or whoever else is interested in improving the commons
> extension tests) there are plenty in
>
> ./xwiki-commons-core/xwiki-commons-extension/xwiki-commons-extension-api/target/pit-reports/201808071749/methods.json:pseudo-tested
> pseudo-tested
> pseudo-tested
> partially-tested
> partially-tested
> pseudo-tested
> pseudo-tested
> pseudo-tested
> pseudo-tested
> pseudo-tested
> partially-tested
> pseudo-tested
> pseudo-tested
> partially-tested
> pseudo-tested
> partially-tested
> pseudo-tested
> pseudo-tested
> pseudo-tested
> pseudo-tested
> pseudo-tested
> pseudo-tested
> pseudo-tested
> partially-tested
> partially-tested
> pseudo-tested
> pseudo-tested
> pseudo-tested
> pseudo-tested
> partially-tested
> pseudo-tested
> pseudo-tested
> pseudo-tested
> pseudo-tested
> pseudo-tested
> pseudo-tested
> pseudo-tested
> pseudo-tested
> partially-tested
> pseudo-tested
> pseudo-tested
> pseudo-tested
> pseudo-tested
> pseudo-tested
> pseudo-tested
> pseudo-tested
> pseudo-tested
> partially-tested
> pseudo-tested
> pseudo-tested
> pseudo-tested
> pseudo-tested
> pseudo-tested
> pseudo-tested
> pseudo-tested
> partially-tested
> partially-tested
> pseudo-tested
> partially-tested
> partially-tested
> partially-tested
> partially-tested
> partially-tested
> pseudo-tested
> pseudo-tested
>
> Fixing those should yield a good increase in both mutation score and test
> coverage.
>
> Thanks
> -Vincent
>
>> On 7 Aug 2018, at 18:05, Vincent Massol wrote:
>>
>>
>>
>>> On 30 Jul 2018, at 12:39, Vincent Massol wrote:
>>>
>>> Hi devs,
>>>
>>> It would be great if you could help improve our unit tests using Descartes.
>>> This is needed for the STAMP research project
>>> (https://www.stamp-project.eu/view/main/) and will benefit XWiki by having
>>> 2 effects:
>>> * increasing the test coverage
>>> * improving the tests themselves (increasing their mutation score)
>>>
>>> Since 10.7 is 50% testing and 50% BFD, it would be great if you could spend
>>> all or a substantial part of your testing time working on this.
>>>
>>> I propose the following strategy:
>>> * You find a module you want to work on.
>>> * In that module you run: mvn clean install -Pquality
>>> -Dxwiki.pitest.skip=false
>>
>> You can also run that at the top of a multimodule project and then find
>> pseudo/partially tested methods with:
>>
>> find . -name "methods.json" -exec egrep -oH "pseudo-tested|partially-tested"
>> {} \;
>>
>> Thanks
>> -Vincent
>>
>>> * Then you check target/pit-reports//issues/index.html and verify if
>>> there are "pseudo tested" methods listed (when we have finished fixing all
>>> of those we can move to “partially tested methods”).
>>> * If there are some, then please record the current jacoco threshold and
>>> the current mutation score.
>>> * You can get the jacoco threshold by running "mvn clean install -Pquality
>>> -Dxwiki.pitest.skip=false -Dxwiki.pitest.mutationThreshold=100” (or by
>>> checking target/pit-reports//index.html, I haven’t checked yet if
>>> they are the same).
>>> * You can get the current mutation score by checking
>>> target/pit-reports//index.html
>>> * Then fix the test so that Descartes doesn’t report any pseudo tested or
>>> partially tested methods
>>> * Update the jacoco threshold and the mutation scores in the pom.xml
>>> * Send a PR on
>>> https://github.com/STAMP-project/descartes-usecases-output/tree/master/xwiki
>>> using the format already defined there.
>>>
>>> WDYT? Doable?
>>>
>>> Thanks
>>> -Vincent
>
--
Thomas Mortagne
Re: [xwiki-devs] Improving tests with Descartes
> On 30 Jul 2018, at 12:39, Vincent Massol wrote: > > Hi devs, > > It would be great if you could help improve our unit tests using Descartes. > This is needed for the STAMP research project > (https://www.stamp-project.eu/view/main/) and will benefit XWiki by having 2 > effects: > * increasing the test coverage > * improving the tests themselves (increasing their mutation score) > > Since 10.7 is 50% testing and 50% BFD, it would be great if you could spend > all or a substantial part of your testing time working on this. > > I propose the following strategy: > * You find a module you want to work on. > * In that module you run: mvn clean install -Pquality > -Dxwiki.pitest.skip=false > * Then you check target/pit-reports//issues/index.html and verify if > there are "pseudo tested" methods listed (when we have finished fixing all of > those we can move to “partially tested methods”). > * If there are some, then please record the current jacoco threshold and the > current mutation score. > * You can get the jacoco threshold by running "mvn clean install -Pquality > -Dxwiki.pitest.skip=false -Dxwiki.pitest.mutationThreshold=100” (or by > checking target/pit-reports//index.html, I haven’t checked yet if they > are the same). > * You can get the current mutation score by checking > target/pit-reports//index.html > * Then fix the test so that Descartes doesn’t report any pseudo tested or > partially tested methods > * Update the jacoco threshold and the mutation scores in the pom.xml > * Send a PR on > https://github.com/STAMP-project/descartes-usecases-output/tree/master/xwiki > using the format already defined there. > > WDYT? Doable? Just fixed all errors on the commons-component-api module and reported the score upgrade: https://github.com/STAMP-project/descartes-usecases-output/commit/bdc1ecb42b906b91fe5bdc09a2cae7aebdeb5146 Who’s next? :) Thanks -Vincent > > Thanks > -Vincent >
Re: [xwiki-devs] Improving tests with Descartes
> On 30 Jul 2018, at 12:39, Vincent Massol wrote: > > Hi devs, > > It would be great if you could help improve our unit tests using Descartes. > This is needed for the STAMP research project > (https://www.stamp-project.eu/view/main/) and will benefit XWiki by having 2 > effects: > * increasing the test coverage > * improving the tests themselves (increasing their mutation score) > > Since 10.7 is 50% testing and 50% BFD, it would be great if you could spend > all or a substantial part of your testing time working on this. > > I propose the following strategy: > * You find a module you want to work on. > * In that module you run: mvn clean install -Pquality > -Dxwiki.pitest.skip=false > * Then you check target/pit-reports//issues/index.html and verify if > there are "pseudo tested" methods listed (when we have finished fixing all of > those we can move to “partially tested methods”). > * If there are some, then please record the current jacoco threshold and the > current mutation score. > * You can get the jacoco threshold by running "mvn clean install -Pquality > -Dxwiki.pitest.skip=false -Dxwiki.pitest.mutationThreshold=100” (or by > checking target/pit-reports//index.html, I haven’t checked yet if they > are the same). Correction: jacoco threshold: mvn clean install -Pquality -Dxwiki.jacoco.instructionRatio=1.00 mutation threshold: mvn clean install -Pquality -Dxwiki.pitest.skip=false -Dxwiki.pitest.mutationThreshold=100 Thanks -Vincent > * You can get the current mutation score by checking > target/pit-reports//index.html > * Then fix the test so that Descartes doesn’t report any pseudo tested or > partially tested methods > * Update the jacoco threshold and the mutation scores in the pom.xml > * Send a PR on > https://github.com/STAMP-project/descartes-usecases-output/tree/master/xwiki > using the format already defined there. > > WDYT? Doable? > > Thanks > -Vincent >
Re: [xwiki-devs] Improving tests with Descartes
Note for Thomas (or whoever else is interested in improving the commons
extension tests) there are plenty in
./xwiki-commons-core/xwiki-commons-extension/xwiki-commons-extension-api/target/pit-reports/201808071749/methods.json:pseudo-tested
pseudo-tested
pseudo-tested
partially-tested
partially-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
partially-tested
pseudo-tested
pseudo-tested
partially-tested
pseudo-tested
partially-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
partially-tested
partially-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
partially-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
partially-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
partially-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
pseudo-tested
partially-tested
partially-tested
pseudo-tested
partially-tested
partially-tested
partially-tested
partially-tested
partially-tested
pseudo-tested
pseudo-tested
Fixing those should yield a good increase in both mutation score and test
coverage.
Thanks
-Vincent
> On 7 Aug 2018, at 18:05, Vincent Massol wrote:
>
>
>
>> On 30 Jul 2018, at 12:39, Vincent Massol wrote:
>>
>> Hi devs,
>>
>> It would be great if you could help improve our unit tests using Descartes.
>> This is needed for the STAMP research project
>> (https://www.stamp-project.eu/view/main/) and will benefit XWiki by having 2
>> effects:
>> * increasing the test coverage
>> * improving the tests themselves (increasing their mutation score)
>>
>> Since 10.7 is 50% testing and 50% BFD, it would be great if you could spend
>> all or a substantial part of your testing time working on this.
>>
>> I propose the following strategy:
>> * You find a module you want to work on.
>> * In that module you run: mvn clean install -Pquality
>> -Dxwiki.pitest.skip=false
>
> You can also run that at the top of a multimodule project and then find
> pseudo/partially tested methods with:
>
> find . -name "methods.json" -exec egrep -oH "pseudo-tested|partially-tested"
> {} \;
>
> Thanks
> -Vincent
>
>> * Then you check target/pit-reports//issues/index.html and verify if
>> there are "pseudo tested" methods listed (when we have finished fixing all
>> of those we can move to “partially tested methods”).
>> * If there are some, then please record the current jacoco threshold and the
>> current mutation score.
>> * You can get the jacoco threshold by running "mvn clean install -Pquality
>> -Dxwiki.pitest.skip=false -Dxwiki.pitest.mutationThreshold=100” (or by
>> checking target/pit-reports//index.html, I haven’t checked yet if they
>> are the same).
>> * You can get the current mutation score by checking
>> target/pit-reports//index.html
>> * Then fix the test so that Descartes doesn’t report any pseudo tested or
>> partially tested methods
>> * Update the jacoco threshold and the mutation scores in the pom.xml
>> * Send a PR on
>> https://github.com/STAMP-project/descartes-usecases-output/tree/master/xwiki
>> using the format already defined there.
>>
>> WDYT? Doable?
>>
>> Thanks
>> -Vincent
Re: [xwiki-devs] Improving tests with Descartes
> On 30 Jul 2018, at 12:39, Vincent Massol wrote:
>
> Hi devs,
>
> It would be great if you could help improve our unit tests using Descartes.
> This is needed for the STAMP research project
> (https://www.stamp-project.eu/view/main/) and will benefit XWiki by having 2
> effects:
> * increasing the test coverage
> * improving the tests themselves (increasing their mutation score)
>
> Since 10.7 is 50% testing and 50% BFD, it would be great if you could spend
> all or a substantial part of your testing time working on this.
>
> I propose the following strategy:
> * You find a module you want to work on.
> * In that module you run: mvn clean install -Pquality
> -Dxwiki.pitest.skip=false
You can also run that at the top of a multimodule project and then find
pseudo/partially tested methods with:
find . -name "methods.json" -exec egrep -oH "pseudo-tested|partially-tested" {}
\;
Thanks
-Vincent
> * Then you check target/pit-reports//issues/index.html and verify if
> there are "pseudo tested" methods listed (when we have finished fixing all of
> those we can move to “partially tested methods”).
> * If there are some, then please record the current jacoco threshold and the
> current mutation score.
> * You can get the jacoco threshold by running "mvn clean install -Pquality
> -Dxwiki.pitest.skip=false -Dxwiki.pitest.mutationThreshold=100” (or by
> checking target/pit-reports//index.html, I haven’t checked yet if they
> are the same).
> * You can get the current mutation score by checking
> target/pit-reports//index.html
> * Then fix the test so that Descartes doesn’t report any pseudo tested or
> partially tested methods
> * Update the jacoco threshold and the mutation scores in the pom.xml
> * Send a PR on
> https://github.com/STAMP-project/descartes-usecases-output/tree/master/xwiki
> using the format already defined there.
>
> WDYT? Doable?
>
> Thanks
> -Vincent
>
