I propose to do this tomorrow Tuesday, starting with an intro from me, using youtube live.
WDYT? Thanks -Vincent > On 30 Aug 2018, at 12:27, Adel Atallah <[email protected]> wrote: > > Just to be clear, when I proposed "having a whole day dedicated on > using these tools", I didn't meant having to have it every week but > only once, so we can properly start improving the tests. It would be > some kind of training. > On my side I don't think I'll be able to have on a week one day > dedicated to tests and one for bug fixing, I won't have time left for > the roadmap as I will only work on the product 50% of the time. > > > On Thu, Aug 30, 2018 at 12:18 PM, Vincent Massol <[email protected]> wrote: >> Hi, >> >> I don’t remember discussing this with you Thomas. Actually I’m not convinced >> to have a fixed day: >> * we already have a fixed BFD and having a second one doesn’t leave much >> flexibility for working on roadmap items when it’s the best >> * test sessions can be short (0.5-1 hours) and it’s easy to do them between >> other tasks >> * it can be boring to spend a full day on them >> >> Now, I agree that not having a fixed day will make it hard to make sure that >> we work 20% on that topic. >> >> So if you prefer we can define a day, knowing that some won’t be able to >> always attend during that day and in this case they should do it on another >> day. What’s important is to have 20% done each week (i.e. enough work done >> on it). >> >> In term of day, if we have to choose one, I’d say Tuesday. That’s the most >> logical to me. >> >> WDYT? What do you prefer? >> >> Thanks >> -Vincent >> >>> On 30 Aug 2018, at 10:38, Thomas Mortagne <[email protected]> wrote: >>> >>> Indeed we discussed this but I don't see it in your mail Vincent. >>> >>> On Thu, Aug 30, 2018 at 10:33 AM, Adel Atallah <[email protected]> >>> wrote: >>>> Hello, >>>> >>>> Maybe we should agree on having a whole day dedicated on using these >>>> tools with a maximum number of developers. >>>> That way we will be able to help each other and maybe it will make the >>>> process easier to carry out in the future. >>>> >>>> WDYT? >>>> >>>> Thanks, >>>> Adel >>>> >>>> >>>> On Wed, Aug 29, 2018 at 11:20 AM, Vincent Massol <[email protected]> >>>> wrote: >>>>> Hi devs (and anyone else interested to improve the tests of XWiki), >>>>> >>>>> History >>>>> ====== >>>>> >>>>> It all started when I analyzed our global TPC and found that it was going >>>>> down globally even though we have the fail-build-on-jacoco-threshold >>>>> strategy. >>>>> >>>>> I sent several email threads: >>>>> >>>>> - Loss of TPC: http://markmail.org/message/hqumkdiz7jm76ya6 >>>>> - TPC evolution: http://markmail.org/message/up2gc2zzbbe4uqgn >>>>> - Improve our TPC strategy: http://markmail.org/message/grphwta63pp5p4l7 >>>>> >>>>> Note: As a consequence of this last thread, I implemented a Jenkins >>>>> Pipeline to send us a mail when the global TPC of an XWiki module goes >>>>> down so that we fix it ASAP. This is still a development in progress. A >>>>> first version is done and running at >>>>> https://ci.xwiki.org/view/Tools/job/Clover/ but I need to debug it and >>>>> fix it (it’s not working ATM). >>>>> >>>>> As a result of the global TPC going down/stagnating, I have proposed to >>>>> have 10.7 focused on Tests + BFD. >>>>> - Initially I proposed to focus on increasing the global TPC by looking >>>>> at the reports from 1) above >>>>> (http://markmail.org/message/qjemnip7hjva2rjd). See the last report at >>>>> https://up1.xwikisas.com/#mJ0loeB6nBrAgYeKA7MGGw (we need to fix the red >>>>> parts). >>>>> - Then with the STAMP mid-term review, a bigger urgency surfaced and I >>>>> asked if we could instead focus on fixing tests as reported by Descartes >>>>> to increase both coverage and mutation score (ie test quality), since >>>>> those are 2 metrics/KPIs measured by STAMP and since XWiki participates >>>>> to STAMP we need to work on them and increase them substantially. See >>>>> http://markmail.org/message/ejmdkf3hx7drkj52 >>>>> >>>>> The results of XWiki 10.7 has been quite poor on test improvements (more >>>>> focus on BFD than tests, lots of devs on holidays, etc). This forces us >>>>> to have a different strategy. >>>>> >>>>> Full Strategy proposal >>>>> ================= >>>>> >>>>> 1) As many XWiki SAS devs as possible (and anyone else from the community >>>>> who’s interested ofc! :)) should spend 1 day per week working on >>>>> improving STAMP metrics >>>>> * Currently the agreement is that Thomas and myself will do this for the >>>>> foreseeable future till we get some good-enough metric progress >>>>> * Some other devs from XWiki SAS will help out for XWiki 10.8 only FTM >>>>> (Marius, Adel if he can, Simon in the future). The idea is to see where >>>>> that could get us by using substantial manpower. >>>>> >>>>> 2) All committers: More generally the global TPC failure is also already >>>>> active and dev need to modify modules that see their global TPC go down. >>>>> >>>>> 3) All committers: Of course, the jacoco strategy is also active at each >>>>> module level. >>>>> >>>>> STAMP tools >>>>> ========== >>>>> >>>>> There are 4 tools developed by STAMP: >>>>> * Descartes: Improves quality of tests by increasing their mutation >>>>> scores. See http://markmail.org/message/bonb5f7f37omnnog and also >>>>> https://massol.myxwiki.org/xwiki/bin/view/Blog/MutationTestingDescartes >>>>> * DSpot: Automatically generate new tests, based on existing tests. See >>>>> https://massol.myxwiki.org/xwiki/bin/view/Blog/TestGenerationDspot >>>>> * CAMP: Takes a Dockerfile and generates mutations of it, then deploys >>>>> and execute tests on the software to see if the mutation works or not. >>>>> Note this is currently not fitting the need of XWiki and thus I’ve been >>>>> developing another tool as an experiment (which may go back in CAMP one >>>>> day), based on TestContainers, see >>>>> https://massol.myxwiki.org/xwiki/bin/view/Blog/EnvironmentTestingExperimentations >>>>> * EvoCrash: Takes a stack trace from production logs and generates a test >>>>> that, when executed, reproduces the crash. See >>>>> https://markmail.org/message/v74g3tsmflquqwra. See also >>>>> https://github.com/SERG-Delft/EvoCrash >>>>> >>>>> Since XWiki is part of the STAMP research project, we need to use those 4 >>>>> tools to increase the KPIs associated with the tools. See below. >>>>> >>>>> Objectives/KPIs/Metrics for STAMP >>>>> =========================== >>>>> >>>>> The STAMP project has defined 9 KPIs that all partners (and thus XWiki) >>>>> need to work on: >>>>> >>>>> 1) K01: Increase test coverage >>>>> * Global increase by reducing by 40% the non-covered code. For XWiki >>>>> since we’re at about 70%, this means reaching about 80% before the end of >>>>> STAMP (ie. before end of 2019) >>>>> * Increase the coverage contributions of each tool developed by STAMP. >>>>> >>>>> Strategy: >>>>> * Primary goal: >>>>> ** Increase coverage by executing Descartes and improving our tests. This >>>>> is http://markmail.org/message/ejmdkf3hx7drkj52 >>>>> ** Don’t do anything with DSpot. I’ll do that part. Note that the goal is >>>>> to write a Jenkins pipeline to automatically execute DSpot from time to >>>>> time and commit the generated tests in a separate test source and have >>>>> our build execute both src/test/java and this new test source. >>>>> ** Don’t do anything with TestContainers FTM since I need to finish a >>>>> first working version. I may need help in the future to implement docker >>>>> images for more configurations (on Oracle, in a cluster, with >>>>> LibreOffice, with an external SOLR server, etc). >>>>> ** For EvoCrash: We’ll count contributions of EvoCrash to coverage in K08. >>>>> * Secondary goal: >>>>> ** Increase our global TPC as mentioned above by fixing the modules in >>>>> red. >>>>> >>>>> 2) K02: Reduce flaky tests. >>>>> * Objective: reduce the number of flaky tests by 20% >>>>> >>>>> Strategy: >>>>> * Record flaky tests in jira >>>>> * Fix the max number of them >>>>> >>>>> 3) K03: Better test quality >>>>> * Objective: increase mutation score by 20% >>>>> >>>>> Strategy: >>>>> * Same strategy as K01. >>>>> >>>>> 4) K04: More configuration-related paths tested >>>>> * Objective: increase the code coverage of configuration-related paths in >>>>> our code by 20% (e.g. DB schema creation, cluster)related code, >>>>> SOLR-related code, LibreOffice-related code, etc). >>>>> >>>>> Strategy: >>>>> * Leave it to FTM. The idea is to measure Clover TPC with the base >>>>> configuration, then execute all other configurations (with >>>>> TestContainers) and regenerate the Clover report to see how much the TPC >>>>> has increased. >>>>> >>>>> 5) K05: Reduce system-specific bugs >>>>> * Objective: 30% improvement >>>>> >>>>> Strategy: >>>>> * Run TestContainers, execute existing tests and find new bugs related to >>>>> configurations. Record them >>>>> >>>>> 6) K06: More configurations/Faster tests >>>>> * Objective: increase the number of automatically tested configurations >>>>> by 50% >>>>> >>>>> Strategy: >>>>> * Increase the # of configurations we test with TestContainers. I’ll do >>>>> that part initially. >>>>> * Reduce time it takes to deploy the software under a given configuration >>>>> vs time it used to take when done manually before STAMP. I’ll do this >>>>> one. I’ve already worked on it in the past year with the dockerization of >>>>> XWiki. >>>>> >>>>> 7) K07: Pending, nothing to do FTM >>>>> >>>>> 8) K08: More crash replicating test cases >>>>> * Objective: increase the number of crash replicating test cases by at >>>>> least 70% >>>>> >>>>> Strategy: >>>>> * For all issues that are still open and that have stack traces and for >>>>> all issues closed but without tests, run EvoCrash on them to try to >>>>> generate a test. >>>>> * Record and count the number of successful EvoCrash-generated test cases. >>>>> * Derive a regression test (which can be very different from the negative >>>>> of the test generated by evocrash!). >>>>> * Measure the new coverage increase >>>>> * Note that I haven’t experimented much with this yet myself. >>>>> >>>>> 9) K09: Pending, nothing to do FTM. >>>>> >>>>> Conclusion >>>>> ========= >>>>> >>>>> Right now, I need your help for the following KPIs: K01, K02, K03, K08. >>>>> >>>>> Since there’s a lot to understand in this email, I’m open to: >>>>> * Organizing a meeting on youtube live to discuss all this >>>>> * Answering any questions on this thread ofc >>>>> * Also feel free to ask on IRC/Matrix. >>>>> >>>>> Here’s an extract from STAMP which has more details about the >>>>> KPIs/metrics: >>>>> https://up1.xwikisas.com/#QJyxqspKXSzuWNOHUuAaEA >>>>> >>>>> Thanks >>>>> -Vincent >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>> >>> >>> >>> -- >>> Thomas Mortagne >>
