Hi devs (and anyone else interested to improve the tests of XWiki),
History
======
It all started when I analyzed our global TPC and found that it was going down
globally even though we have the fail-build-on-jacoco-threshold strategy.
I sent several email threads:
- Loss of TPC: http://markmail.org/message/hqumkdiz7jm76ya6
- TPC evolution: http://markmail.org/message/up2gc2zzbbe4uqgn
- Improve our TPC strategy: http://markmail.org/message/grphwta63pp5p4l7
Note: As a consequence of this last thread, I implemented a Jenkins Pipeline to
send us a mail when the global TPC of an XWiki module goes down so that we fix
it ASAP. This is still a development in progress. A first version is done and
running at https://ci.xwiki.org/view/Tools/job/Clover/ but I need to debug it
and fix it (it’s not working ATM).
As a result of the global TPC going down/stagnating, I have proposed to have
10.7 focused on Tests + BFD.
- Initially I proposed to focus on increasing the global TPC by looking at the
reports from 1) above (http://markmail.org/message/qjemnip7hjva2rjd). See the
last report at https://up1.xwikisas.com/#mJ0loeB6nBrAgYeKA7MGGw (we need to fix
the red parts).
- Then with the STAMP mid-term review, a bigger urgency surfaced and I asked if
we could instead focus on fixing tests as reported by Descartes to increase
both coverage and mutation score (ie test quality), since those are 2
metrics/KPIs measured by STAMP and since XWiki participates to STAMP we need to
work on them and increase them substantially. See
http://markmail.org/message/ejmdkf3hx7drkj52
The results of XWiki 10.7 has been quite poor on test improvements (more focus
on BFD than tests, lots of devs on holidays, etc). This forces us to have a
different strategy.
Full Strategy proposal
=================
1) As many XWiki SAS devs as possible (and anyone else from the community who’s
interested ofc! :)) should spend 1 day per week working on improving STAMP
metrics
* Currently the agreement is that Thomas and myself will do this for the
foreseeable future till we get some good-enough metric progress
* Some other devs from XWiki SAS will help out for XWiki 10.8 only FTM (Marius,
Adel if he can, Simon in the future). The idea is to see where that could get
us by using substantial manpower.
2) All committers: More generally the global TPC failure is also already active
and dev need to modify modules that see their global TPC go down.
3) All committers: Of course, the jacoco strategy is also active at each module
level.
STAMP tools
==========
There are 4 tools developed by STAMP:
* Descartes: Improves quality of tests by increasing their mutation scores. See
http://markmail.org/message/bonb5f7f37omnnog and also
https://massol.myxwiki.org/xwiki/bin/view/Blog/MutationTestingDescartes
* DSpot: Automatically generate new tests, based on existing tests. See
https://massol.myxwiki.org/xwiki/bin/view/Blog/TestGenerationDspot
* CAMP: Takes a Dockerfile and generates mutations of it, then deploys and
execute tests on the software to see if the mutation works or not. Note this is
currently not fitting the need of XWiki and thus I’ve been developing another
tool as an experiment (which may go back in CAMP one day), based on
TestContainers, see
https://massol.myxwiki.org/xwiki/bin/view/Blog/EnvironmentTestingExperimentations
* EvoCrash: Takes a stack trace from production logs and generates a test that,
when executed, reproduces the crash. See
https://markmail.org/message/v74g3tsmflquqwra. See also
https://github.com/SERG-Delft/EvoCrash
Since XWiki is part of the STAMP research project, we need to use those 4 tools
to increase the KPIs associated with the tools. See below.
Objectives/KPIs/Metrics for STAMP
===========================
The STAMP project has defined 9 KPIs that all partners (and thus XWiki) need to
work on:
1) K01: Increase test coverage
* Global increase by reducing by 40% the non-covered code. For XWiki since
we’re at about 70%, this means reaching about 80% before the end of STAMP (ie.
before end of 2019)
* Increase the coverage contributions of each tool developed by STAMP.
Strategy:
* Primary goal:
** Increase coverage by executing Descartes and improving our tests. This is
http://markmail.org/message/ejmdkf3hx7drkj52
** Don’t do anything with DSpot. I’ll do that part. Note that the goal is to
write a Jenkins pipeline to automatically execute DSpot from time to time and
commit the generated tests in a separate test source and have our build execute
both src/test/java and this new test source.
** Don’t do anything with TestContainers FTM since I need to finish a first
working version. I may need help in the future to implement docker images for
more configurations (on Oracle, in a cluster, with LibreOffice, with an
external SOLR server, etc).
** For EvoCrash: We’ll count contributions of EvoCrash to coverage in K08.
* Secondary goal:
** Increase our global TPC as mentioned above by fixing the modules in red.
2) K02: Reduce flaky tests.
* Objective: reduce the number of flaky tests by 20%
Strategy:
* Record flaky tests in jira
* Fix the max number of them
3) K03: Better test quality
* Objective: increase mutation score by 20%
Strategy:
* Same strategy as K01.
4) K04: More configuration-related paths tested
* Objective: increase the code coverage of configuration-related paths in our
code by 20% (e.g. DB schema creation, cluster)related code, SOLR-related code,
LibreOffice-related code, etc).
Strategy:
* Leave it to FTM. The idea is to measure Clover TPC with the base
configuration, then execute all other configurations (with TestContainers) and
regenerate the Clover report to see how much the TPC has increased.
5) K05: Reduce system-specific bugs
* Objective: 30% improvement
Strategy:
* Run TestContainers, execute existing tests and find new bugs related to
configurations. Record them
6) K06: More configurations/Faster tests
* Objective: increase the number of automatically tested configurations by 50%
Strategy:
* Increase the # of configurations we test with TestContainers. I’ll do that
part initially.
* Reduce time it takes to deploy the software under a given configuration vs
time it used to take when done manually before STAMP. I’ll do this one. I’ve
already worked on it in the past year with the dockerization of XWiki.
7) K07: Pending, nothing to do FTM
8) K08: More crash replicating test cases
* Objective: increase the number of crash replicating test cases by at least 70%
Strategy:
* For all issues that are still open and that have stack traces and for all
issues closed but without tests, run EvoCrash on them to try to generate a test.
* Record and count the number of successful EvoCrash-generated test cases.
* Derive a regression test (which can be very different from the negative of
the test generated by evocrash!).
* Measure the new coverage increase
* Note that I haven’t experimented much with this yet myself.
9) K09: Pending, nothing to do FTM.
Conclusion
=========
Right now, I need your help for the following KPIs: K01, K02, K03, K08.
Since there’s a lot to understand in this email, I’m open to:
* Organizing a meeting on youtube live to discuss all this
* Answering any questions on this thread ofc
* Also feel free to ask on IRC/Matrix.
Here’s an extract from STAMP which has more details about the KPIs/metrics:
https://up1.xwikisas.com/#QJyxqspKXSzuWNOHUuAaEA
Thanks
-Vincent
