>> Is anyone aware of existing research in this area? I have only skimmed the >> research papers your project has listed: >> https://www.jacoco.org/research/index.html >> <https://www.jacoco.org/research/index.html> I’m currently improving with one of my master's students the JaCoCo code coverage reports in Jenkins. We are focussing on a different goal but maybe our approach is helpful as well: we are trying to compute a delta report of two coverage analyses, on the one hand the coverage of a pull request and on the other hand the coverage of the target branch build. The goal is to give developers feedback about the quality of the pull request. Did the coverage increase or decrease? How is the coverage of the actual changes?
A sketch of our planned approach (we just startet the thesis): 1) read the coverage of the pull request build into a model (JaCoCo, Cobertura, etc. - the format actually does not matter) and map it to the source files 2) read the coverage of the target branch build into a model (JaCoCo, Cobertura, etc. - the format actually does not matter) and map it to the source files 3) read the changed lines and files from Git 4) combine the results of 1-3 Our model is based on the source code, so we do not use class files anymore. Just package, file, class, method. > Am 02.12.2021 um 15:30 schrieb Svante Schubert <svante.schub...@gmail.com>: > > Hello Marc, > > I must admit, I have some difficulties bringing the 4 steps you mentioned > below in synch with the JaCoCo Code and my upcoming user scenarios. ;-) > > One user scenario in mind is to establish a pre-commit hook that requests > from committers of new features to add also some feature regression test, > initially by testing the coverage-delta of a changed class. > Or the same scenario just slightly different (but less controversial) - the > usability feature to show automatically the statistic of changed coverage > along with every received pull request (PR). > > So, Initially, your 4 step solution recipe seemed quite easy as you pointed > me to the exact method ExecutionData.merge(..) and I thought I just have to > go "backwards" (compiling the project in my IDE and to a "used-by IDE query"). > But I have not found a way to load existing persistent coverage data > (XML/binary). This seemed helpful, as we try to bridge the usage of several > coverage tools and we realized that other none-Java coverage frameworks like > "Coverage.py for Python" and "GCOV for C++" are using the Cobertura XML as a > common denominator. At least we found already a transformation from Jacoco to > Cobertura XML (the other way around might exist somewhere and not that > difficult). Our attempt is to reuse some coverage tooling among various code > coverage tools. > > In addition, if a Java Source (and its class) was changed, how can we find > the related ExecutionData of the changed class, which are based on the class > hash? > I believe your design likely comes from the class-only-exist scenario, while > I would rather take the sources and even more some version control system as > a prerequisite for my subtype of scenarios. > The coverage identification could be done by file name (in Java package & > class name), while in addition Git information (as the commit and/or file > hash) could be added. > Especially helpful would be to receive to identify by git information any > class renaming/movements. > (I personally would initially focus on Git, but most version control systems > could be easily abstracted in an API.) > > Does this make sense to you, Marc? What would you suggest how to proceed? > > I really would like to invite you for some tea/beer and discuss this over > aside from a conference! But damn Corona, but if you like to share a remote > tea and just discuss/brainstorm a few things, just ping me directly. > This might remove some basic errors from my perspective as I am a newcomer in > this field - even the wording Agent & probe was difficult to understand until > I found out it references to JXM. Later I found probe very well (but a bit > hidden) explained in "Why can't JaCoCo simply use the class name to identify > classes?" at https://www.jacoco.org/jacoco/trunk/doc/classids.html > <https://www.jacoco.org/jacoco/trunk/doc/classids.html>, but regarding docu I > am in a glass house and not throwing any stone ;-) > > Looking forward to hearing from you, Marc! > > Hope you have a great week! > Svante > > Am Fr., 5. Nov. 2021 um 17:16 Uhr schrieb Marc Hoffmann > <hoffm...@mountainminds.com <mailto:hoffm...@mountainminds.com>>: > Hi Svante, > > regarding the second part of your email: This is referred to as “incremental > code coverage”, there was research on this based on JaCoCo/EclEmma (see > https://www.cs.ubc.ca/tr/2008/tr-2008-14 > <https://www.cs.ubc.ca/tr/2008/tr-2008-14>). > > I think you’re scenario is supported using the JaCoCo API. Here are the the > steps: > > 1) Collect execution data for loading plain “Hello World” (exec1) > 2) Collect execution data for loading bold “Hello World” (exec2) > 3) Subtract exec1 from exec2 (see substract flag in > https://www.jacoco.org/jacoco/trunk/doc/api/org/jacoco/core/data/ExecutionData.html#merge(org.jacoco.core.data.ExecutionData,%20boolean) > > <https://www.jacoco.org/jacoco/trunk/doc/api/org/jacoco/core/data/ExecutionData.html#merge(org.jacoco.core.data.ExecutionData,%20boolean)> > ) > 4) Generate Coverage report with exec data from step 3) > > This report should only show code which was executed in addition to render > the bold text. > > Cheers, > -marc > > >> On 5. Nov 2021, at 13:44, Svante Schubert <svante.schub...@gmail.com >> <mailto:svante.schub...@gmail.com>> wrote: >> >> Hi Marc, >> >> Thank you very much that does the trick! :-) >> >> I realized some "noise" on some tests being once executed and once not, but >> that might be as well a surefire problem: >> https://svanteschubert.github.io/odfdom-coverage-data/jacoco20211105-120509/org.odftoolkit.odfdom.dom.element.office/OfficeMetaElement.java.html#L129 >> >> <https://svanteschubert.github.io/odfdom-coverage-data/jacoco20211105-120509/org.odftoolkit.odfdom.dom.element.office/OfficeMetaElement.java.html#L129> >> https://svanteschubert.github.io/odfdom-coverage-data/jacoco20211104-101936/org.odftoolkit.odfdom.dom.element.office/OfficeMetaElement.java.html#L129 >> >> <https://svanteschubert.github.io/odfdom-coverage-data/jacoco20211104-101936/org.odftoolkit.odfdom.dom.element.office/OfficeMetaElement.java.html#L129> >> >> I will dive into the docu to see if I find more on how the coverage works... >> Any good pointers? >> >> In addition, does anyone have a pointer to existing open-source tooling >> (like Jenkins script) that pushes the coverage test results (like above) to >> some webserver (like GitHub pages)? >> My first aim would be to always align the coverage results with a Git commit. >> >> Allow me to paint the whole picture: >> As mentioned I am working on the OpenDocument format (I am co-chair of OASIS >> ODF TC <https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=office> >> and co-maintainer of the ODF Toolkit <https://github.com/tdf/odftoolkit>). >> In the ODF Toolkit and LibreOffice (LO) we have the problem that the tests >> are taking quite long (in LO even longer than the build). >> >> I like to evaluate a trick using coverage: >> First, I am loading a test document in an ODF application like LO - say only >> containing "Hello World" - and saving aside all the code coverage during the >> test. All the source code lines are required to load it (and layout it in >> LO). >> Second, I am loading a second test document - again "Hello World" but this >> time in bold - and saving aside again the code coverage data for the second >> test. >> >> I assume that the second document load test, which contains, in addition, >> the "bold" feature, will have higher test coverage. >> The subtraction of both test coverages should reveal the lines solely >> required for the "bold" feature. We like to align these feature lines with a >> very fast feature-smoke test in conjunction with a GIT pre-commit-hook. >> >> In addition, we would be able to "draw" some "feature map" of the source >> code for newcomers. >> >> Is anyone aware of existing research in this area? I have only skimmed the >> research papers your project has listed: >> https://www.jacoco.org/research/index.html >> <https://www.jacoco.org/research/index.html> >> Our work will be all open-source. We were able to receive a small grant from >> the German Ministry of Research >> <https://prototypefund.de/en/project/cover-rest-find-features-avoid-stress/>. >> We are Thorsten Behrens - taking over the C++ part, his son Linus Behrens >> (Python) and myself for Java. >> >> Last but not least, imagine we would be "the kings" of JCov, Jacoco, >> Cobertura... all Java coverage tools, wouldn't we think about reusing >> modules? Like front/backend as Clang is using? >> >> Have a great weekend & greetings from Berlin! >> Svante >> >> PS: One design aspect: The most revolutionary thing I observed in the past >> years in IDEs is the tendency to move from all-source test/compile to >> incremental testing/compilation. >> My favourite example is the Visual Studio Code (VSCode) extension (nice idea >> to split the language-dependent logic into a server part (language server >> <https://code.visualstudio.com/api/language-extensions/language-server-extension-guide> >> - quick intro <https://www.ncameron.org/blog/how-the-rls-works/>) and a >> stupid-only-layouting client part of the VSCode extension), where the >> official VSCode Rust extension <https://rust-analyzer.github.io/> switched >> to the incremental approach. >> >> >> Am Do., 4. Nov. 2021 um 08:50 Uhr schrieb Marc Hoffmann >> <hoffm...@mountainminds.com <mailto:hoffm...@mountainminds.com>>: >> Hi Svante, >> >> as you can see on the sessions page you only have execution data for a >> single library class: >> >> https://svanteschubert.github.io/odfdom-coverage-data/jacoco-sessions.html >> <https://svanteschubert.github.io/odfdom-coverage-data/jacoco-sessions.html> >> >> As this is the only (not excluded) IT test executed with >> maven-failsafe-plugin it looks like that there is simply no coverage for >> your unit tests executed with the maven-surefire-plugin plugin. >> >> The reason is that you overwrite the argLine which was prepared by JaCoCo >> before: >> >> https://github.com/tdf/odftoolkit/blob/master/odfdom/pom.xml#L257 >> <https://github.com/tdf/odftoolkit/blob/master/odfdom/pom.xml#L257> >> >> Please see JaCoCo documentation how to combine the JaCoCo agent >> configuration with your own VM arguments correctly: >> >> https://www.jacoco.org/jacoco/trunk/doc/prepare-agent-mojo.html >> <https://www.jacoco.org/jacoco/trunk/doc/prepare-agent-mojo.html> >> >> Probably the line should read: >> >> <argLine>@{argLine} >> -Dfile.encoding=${project.build.sourceEncoding}</argLine> >> >> Best refards, >> -marc >> >> >>> On 3. Nov 2021, at 23:45, Svante Schubert <svante.schub...@gmail.com >>> <mailto:svante.schub...@gmail.com>> wrote: >>> >>> Hi Marc, >>> >>> the tests are running and there are test output files (and debug output >>> from the uncovered files), but you are right the special JUnit class might >>> not be the problem. >>> Seems, there is only one test providing coverage: >>> https://svanteschubert.github.io/odfdom-coverage-data/ >>> <https://svanteschubert.github.io/odfdom-coverage-data/> <- I have uploaded >>> the JaCoCo report. >>> Only one file has coverage, being tested by an integration test: >>> https://github.com/tdf/odftoolkit/blob/coverage/odfdom/src/test/java/org/odftoolkit/odfdom/integrationtest/JarManifestIT.java >>> >>> <https://github.com/tdf/odftoolkit/blob/coverage/odfdom/src/test/java/org/odftoolkit/odfdom/integrationtest/JarManifestIT.java> >>> Now I suspect that I am doing the Maven pom.xml wrong as there are only >>> integration tests and JaCoCo in the reporting, see: >>> https://github.com/tdf/odftoolkit/blob/coverage/odfdom/pom.xml#L443 >>> <https://github.com/tdf/odftoolkit/blob/coverage/odfdom/pom.xml#L443> >>> >>> Likely I have overseen something... >>> Do you have any further hint? >>> >>> Thanks in advance, >>> Svante >>> >>> PS: It is not the small test you asked for, but you may via command line >>> clone git repo https://github.com/tdf/odftoolkit/ >>> <https://github.com/tdf/odftoolkit/> >>> switch to the coverage branch >>> change to subdirectory odfdom of odftoolkit >>> call 'mvn install' (using JDK >=9) >>> call 'mvn jacoco:report' >>> >>> >>> Am Mi., 3. Nov. 2021 um 22:38 Uhr schrieb Marc Hoffmann >>> <hoffm...@mountainminds.com <mailto:hoffm...@mountainminds.com>>: >>> Hi Svante, >>> >>> JaCoCo does not depend on a specific testing frameworks and I don’t think >>> there are technical limitations with @RunWith(Parameterized.class). Maybe >>> it does not work at all in your case and the code is in fact not executed? >>> >>> If you think there is a problem can you please extract a minimal reproducer >>> to demonstrate the problem? >>> >>> Thanks, >>> -marc >>> >>>> On 3. Nov 2021, at 21:34, Svante Schubert <svante.schub...@gmail.com >>>> <mailto:svante.schub...@gmail.com>> wrote: >>>> >>>> Hello coverage experts! >>>> >>>> Does anybody have success using JaCoCo and JUnit's >>>> @RunWith(Parameterized.class) >>>> <https://junit.org/junit4/javadoc/4.12/org/junit/runners/Parameterized.html>? >>>> >>>> I am working on the ODF file format and I am running a test transformation >>>> for every existing ODF test document in our test folder comparing it with >>>> the prior reference result. >>>> >>>> See >>>> https://github.com/tdf/odftoolkit/blob/coverage/odfdom/src/test/java/org/odftoolkit/odfdom/changes/RoundtripTest.java#L36 >>>> >>>> <https://github.com/tdf/odftoolkit/blob/coverage/odfdom/src/test/java/org/odftoolkit/odfdom/changes/RoundtripTest.java#L36> >>>> >>>> By using this generic test the coverage of the changes packages should be >>>> high, but the coverage of package org/odftoolkit/odfdom/changes is instead >>>> zero! >>>> >>>> Thanks in advance for any hints! >>>> Svante >>>> >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "JaCoCo and EclEmma Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to jacoco+unsubscr...@googlegroups.com >>>> <mailto:jacoco+unsubscr...@googlegroups.com>. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/jacoco/75f55ae9-9f24-4261-93bc-54c2e1c01367n%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/jacoco/75f55ae9-9f24-4261-93bc-54c2e1c01367n%40googlegroups.com?utm_medium=email&utm_source=footer>. >>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "JaCoCo and EclEmma Users" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/jacoco/dXbkWvvKj5Q/unsubscribe >>> <https://groups.google.com/d/topic/jacoco/dXbkWvvKj5Q/unsubscribe>. >>> To unsubscribe from this group and all its topics, send an email to >>> jacoco+unsubscr...@googlegroups.com >>> <mailto:jacoco+unsubscr...@googlegroups.com>. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/jacoco/3BBA4FEC-403A-46F2-BE9E-2F46E3C356CE%40mountainminds.com >>> >>> <https://groups.google.com/d/msgid/jacoco/3BBA4FEC-403A-46F2-BE9E-2F46E3C356CE%40mountainminds.com?utm_medium=email&utm_source=footer>. >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "JaCoCo and EclEmma Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to jacoco+unsubscr...@googlegroups.com >>> <mailto:jacoco+unsubscr...@googlegroups.com>. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/jacoco/CAHNbw8nSqxaAxyKRzP0Oa01mKD0awZN36ReBmgiPucQToVRqiQ%40mail.gmail.com >>> >>> <https://groups.google.com/d/msgid/jacoco/CAHNbw8nSqxaAxyKRzP0Oa01mKD0awZN36ReBmgiPucQToVRqiQ%40mail.gmail.com?utm_medium=email&utm_source=footer>. >> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "JaCoCo and EclEmma Users" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/jacoco/dXbkWvvKj5Q/unsubscribe >> <https://groups.google.com/d/topic/jacoco/dXbkWvvKj5Q/unsubscribe>. >> To unsubscribe from this group and all its topics, send an email to >> jacoco+unsubscr...@googlegroups.com >> <mailto:jacoco+unsubscr...@googlegroups.com>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/jacoco/AB47E08D-DE4D-4C16-8CB2-AA12DA0EF0AE%40mountainminds.com >> >> <https://groups.google.com/d/msgid/jacoco/AB47E08D-DE4D-4C16-8CB2-AA12DA0EF0AE%40mountainminds.com?utm_medium=email&utm_source=footer>. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "JaCoCo and EclEmma Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to jacoco+unsubscr...@googlegroups.com >> <mailto:jacoco+unsubscr...@googlegroups.com>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/jacoco/CAHNbw8kUx76m7Vr%2BbwMx2C%2B9SPbc-At91KOGAARm8MMwkK6NVQ%40mail.gmail.com >> >> <https://groups.google.com/d/msgid/jacoco/CAHNbw8kUx76m7Vr%2BbwMx2C%2B9SPbc-At91KOGAARm8MMwkK6NVQ%40mail.gmail.com?utm_medium=email&utm_source=footer>. > > > -- > You received this message because you are subscribed to a topic in the Google > Groups "JaCoCo and EclEmma Users" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/jacoco/dXbkWvvKj5Q/unsubscribe > <https://groups.google.com/d/topic/jacoco/dXbkWvvKj5Q/unsubscribe>. > To unsubscribe from this group and all its topics, send an email to > jacoco+unsubscr...@googlegroups.com > <mailto:jacoco+unsubscr...@googlegroups.com>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jacoco/41C398C6-FCF9-45E3-8C3D-914036AFDAED%40mountainminds.com > > <https://groups.google.com/d/msgid/jacoco/41C398C6-FCF9-45E3-8C3D-914036AFDAED%40mountainminds.com?utm_medium=email&utm_source=footer>. > > -- > You received this message because you are subscribed to the Google Groups > "JaCoCo and EclEmma Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to jacoco+unsubscr...@googlegroups.com > <mailto:jacoco+unsubscr...@googlegroups.com>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jacoco/CAHNbw8mQ1X1%3DYB0vzLo242%3Db74LNOYk5unZXCRoqCf6i0p8eCA%40mail.gmail.com > > <https://groups.google.com/d/msgid/jacoco/CAHNbw8mQ1X1%3DYB0vzLo242%3Db74LNOYk5unZXCRoqCf6i0p8eCA%40mail.gmail.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "JaCoCo and EclEmma Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jacoco+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jacoco/3FFA70B3-1E03-435B-96DC-3B6E895CD39A%40gmail.com.
