Hi Svante, regarding the second part of your email: This is referred to as “incremental code coverage”, there was research on this based on JaCoCo/EclEmma (see https://www.cs.ubc.ca/tr/2008/tr-2008-14 <https://www.cs.ubc.ca/tr/2008/tr-2008-14>).
I think you’re scenario is supported using the JaCoCo API. Here are the the steps: 1) Collect execution data for loading plain “Hello World” (exec1) 2) Collect execution data for loading bold “Hello World” (exec2) 3) Subtract exec1 from exec2 (see substract flag in https://www.jacoco.org/jacoco/trunk/doc/api/org/jacoco/core/data/ExecutionData.html#merge(org.jacoco.core.data.ExecutionData,%20boolean) <https://www.jacoco.org/jacoco/trunk/doc/api/org/jacoco/core/data/ExecutionData.html#merge(org.jacoco.core.data.ExecutionData, boolean)> ) 4) Generate Coverage report with exec data from step 3) This report should only show code which was executed in addition to render the bold text. Cheers, -marc > On 5. Nov 2021, at 13:44, Svante Schubert <svante.schub...@gmail.com> wrote: > > Hi Marc, > > Thank you very much that does the trick! :-) > > I realized some "noise" on some tests being once executed and once not, but > that might be as well a surefire problem: > https://svanteschubert.github.io/odfdom-coverage-data/jacoco20211105-120509/org.odftoolkit.odfdom.dom.element.office/OfficeMetaElement.java.html#L129 > > <https://svanteschubert.github.io/odfdom-coverage-data/jacoco20211105-120509/org.odftoolkit.odfdom.dom.element.office/OfficeMetaElement.java.html#L129> > https://svanteschubert.github.io/odfdom-coverage-data/jacoco20211104-101936/org.odftoolkit.odfdom.dom.element.office/OfficeMetaElement.java.html#L129 > > <https://svanteschubert.github.io/odfdom-coverage-data/jacoco20211104-101936/org.odftoolkit.odfdom.dom.element.office/OfficeMetaElement.java.html#L129> > > I will dive into the docu to see if I find more on how the coverage works... > Any good pointers? > > In addition, does anyone have a pointer to existing open-source tooling (like > Jenkins script) that pushes the coverage test results (like above) to some > webserver (like GitHub pages)? > My first aim would be to always align the coverage results with a Git commit. > > Allow me to paint the whole picture: > As mentioned I am working on the OpenDocument format (I am co-chair of OASIS > ODF TC <https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=office> > and co-maintainer of the ODF Toolkit <https://github.com/tdf/odftoolkit>). > In the ODF Toolkit and LibreOffice (LO) we have the problem that the tests > are taking quite long (in LO even longer than the build). > > I like to evaluate a trick using coverage: > First, I am loading a test document in an ODF application like LO - say only > containing "Hello World" - and saving aside all the code coverage during the > test. All the source code lines are required to load it (and layout it in LO). > Second, I am loading a second test document - again "Hello World" but this > time in bold - and saving aside again the code coverage data for the second > test. > > I assume that the second document load test, which contains, in addition, the > "bold" feature, will have higher test coverage. > The subtraction of both test coverages should reveal the lines solely > required for the "bold" feature. We like to align these feature lines with a > very fast feature-smoke test in conjunction with a GIT pre-commit-hook. > > In addition, we would be able to "draw" some "feature map" of the source code > for newcomers. > > Is anyone aware of existing research in this area? I have only skimmed the > research papers your project has listed: > https://www.jacoco.org/research/index.html > <https://www.jacoco.org/research/index.html> > Our work will be all open-source. We were able to receive a small grant from > the German Ministry of Research > <https://prototypefund.de/en/project/cover-rest-find-features-avoid-stress/>. > We are Thorsten Behrens - taking over the C++ part, his son Linus Behrens > (Python) and myself for Java. > > Last but not least, imagine we would be "the kings" of JCov, Jacoco, > Cobertura... all Java coverage tools, wouldn't we think about reusing > modules? Like front/backend as Clang is using? > > Have a great weekend & greetings from Berlin! > Svante > > PS: One design aspect: The most revolutionary thing I observed in the past > years in IDEs is the tendency to move from all-source test/compile to > incremental testing/compilation. > My favourite example is the Visual Studio Code (VSCode) extension (nice idea > to split the language-dependent logic into a server part (language server > <https://code.visualstudio.com/api/language-extensions/language-server-extension-guide> > - quick intro <https://www.ncameron.org/blog/how-the-rls-works/>) and a > stupid-only-layouting client part of the VSCode extension), where the > official VSCode Rust extension <https://rust-analyzer.github.io/> switched to > the incremental approach. > > > Am Do., 4. Nov. 2021 um 08:50 Uhr schrieb Marc Hoffmann > <hoffm...@mountainminds.com <mailto:hoffm...@mountainminds.com>>: > Hi Svante, > > as you can see on the sessions page you only have execution data for a single > library class: > > https://svanteschubert.github.io/odfdom-coverage-data/jacoco-sessions.html > <https://svanteschubert.github.io/odfdom-coverage-data/jacoco-sessions.html> > > As this is the only (not excluded) IT test executed with > maven-failsafe-plugin it looks like that there is simply no coverage for your > unit tests executed with the maven-surefire-plugin plugin. > > The reason is that you overwrite the argLine which was prepared by JaCoCo > before: > > https://github.com/tdf/odftoolkit/blob/master/odfdom/pom.xml#L257 > <https://github.com/tdf/odftoolkit/blob/master/odfdom/pom.xml#L257> > > Please see JaCoCo documentation how to combine the JaCoCo agent configuration > with your own VM arguments correctly: > > https://www.jacoco.org/jacoco/trunk/doc/prepare-agent-mojo.html > <https://www.jacoco.org/jacoco/trunk/doc/prepare-agent-mojo.html> > > Probably the line should read: > > <argLine>@{argLine} > -Dfile.encoding=${project.build.sourceEncoding}</argLine> > > Best refards, > -marc > > >> On 3. Nov 2021, at 23:45, Svante Schubert <svante.schub...@gmail.com >> <mailto:svante.schub...@gmail.com>> wrote: >> >> Hi Marc, >> >> the tests are running and there are test output files (and debug output from >> the uncovered files), but you are right the special JUnit class might not be >> the problem. >> Seems, there is only one test providing coverage: >> https://svanteschubert.github.io/odfdom-coverage-data/ >> <https://svanteschubert.github.io/odfdom-coverage-data/> <- I have uploaded >> the JaCoCo report. >> Only one file has coverage, being tested by an integration test: >> https://github.com/tdf/odftoolkit/blob/coverage/odfdom/src/test/java/org/odftoolkit/odfdom/integrationtest/JarManifestIT.java >> >> <https://github.com/tdf/odftoolkit/blob/coverage/odfdom/src/test/java/org/odftoolkit/odfdom/integrationtest/JarManifestIT.java> >> Now I suspect that I am doing the Maven pom.xml wrong as there are only >> integration tests and JaCoCo in the reporting, see: >> https://github.com/tdf/odftoolkit/blob/coverage/odfdom/pom.xml#L443 >> <https://github.com/tdf/odftoolkit/blob/coverage/odfdom/pom.xml#L443> >> >> Likely I have overseen something... >> Do you have any further hint? >> >> Thanks in advance, >> Svante >> >> PS: It is not the small test you asked for, but you may via command line >> clone git repo https://github.com/tdf/odftoolkit/ >> <https://github.com/tdf/odftoolkit/> >> switch to the coverage branch >> change to subdirectory odfdom of odftoolkit >> call 'mvn install' (using JDK >=9) >> call 'mvn jacoco:report' >> >> >> Am Mi., 3. Nov. 2021 um 22:38 Uhr schrieb Marc Hoffmann >> <hoffm...@mountainminds.com <mailto:hoffm...@mountainminds.com>>: >> Hi Svante, >> >> JaCoCo does not depend on a specific testing frameworks and I don’t think >> there are technical limitations with @RunWith(Parameterized.class). Maybe it >> does not work at all in your case and the code is in fact not executed? >> >> If you think there is a problem can you please extract a minimal reproducer >> to demonstrate the problem? >> >> Thanks, >> -marc >> >>> On 3. Nov 2021, at 21:34, Svante Schubert <svante.schub...@gmail.com >>> <mailto:svante.schub...@gmail.com>> wrote: >>> >>> Hello coverage experts! >>> >>> Does anybody have success using JaCoCo and JUnit's >>> @RunWith(Parameterized.class) >>> <https://junit.org/junit4/javadoc/4.12/org/junit/runners/Parameterized.html>? >>> >>> I am working on the ODF file format and I am running a test transformation >>> for every existing ODF test document in our test folder comparing it with >>> the prior reference result. >>> >>> See >>> https://github.com/tdf/odftoolkit/blob/coverage/odfdom/src/test/java/org/odftoolkit/odfdom/changes/RoundtripTest.java#L36 >>> >>> <https://github.com/tdf/odftoolkit/blob/coverage/odfdom/src/test/java/org/odftoolkit/odfdom/changes/RoundtripTest.java#L36> >>> >>> By using this generic test the coverage of the changes packages should be >>> high, but the coverage of package org/odftoolkit/odfdom/changes is instead >>> zero! >>> >>> Thanks in advance for any hints! >>> Svante >>> >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "JaCoCo and EclEmma Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to jacoco+unsubscr...@googlegroups.com >>> <mailto:jacoco+unsubscr...@googlegroups.com>. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/jacoco/75f55ae9-9f24-4261-93bc-54c2e1c01367n%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/jacoco/75f55ae9-9f24-4261-93bc-54c2e1c01367n%40googlegroups.com?utm_medium=email&utm_source=footer>. >> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "JaCoCo and EclEmma Users" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/jacoco/dXbkWvvKj5Q/unsubscribe >> <https://groups.google.com/d/topic/jacoco/dXbkWvvKj5Q/unsubscribe>. >> To unsubscribe from this group and all its topics, send an email to >> jacoco+unsubscr...@googlegroups.com >> <mailto:jacoco+unsubscr...@googlegroups.com>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/jacoco/3BBA4FEC-403A-46F2-BE9E-2F46E3C356CE%40mountainminds.com >> >> <https://groups.google.com/d/msgid/jacoco/3BBA4FEC-403A-46F2-BE9E-2F46E3C356CE%40mountainminds.com?utm_medium=email&utm_source=footer>. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "JaCoCo and EclEmma Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to jacoco+unsubscr...@googlegroups.com >> <mailto:jacoco+unsubscr...@googlegroups.com>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/jacoco/CAHNbw8nSqxaAxyKRzP0Oa01mKD0awZN36ReBmgiPucQToVRqiQ%40mail.gmail.com >> >> <https://groups.google.com/d/msgid/jacoco/CAHNbw8nSqxaAxyKRzP0Oa01mKD0awZN36ReBmgiPucQToVRqiQ%40mail.gmail.com?utm_medium=email&utm_source=footer>. > > > -- > You received this message because you are subscribed to a topic in the Google > Groups "JaCoCo and EclEmma Users" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/jacoco/dXbkWvvKj5Q/unsubscribe > <https://groups.google.com/d/topic/jacoco/dXbkWvvKj5Q/unsubscribe>. > To unsubscribe from this group and all its topics, send an email to > jacoco+unsubscr...@googlegroups.com > <mailto:jacoco+unsubscr...@googlegroups.com>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jacoco/AB47E08D-DE4D-4C16-8CB2-AA12DA0EF0AE%40mountainminds.com > > <https://groups.google.com/d/msgid/jacoco/AB47E08D-DE4D-4C16-8CB2-AA12DA0EF0AE%40mountainminds.com?utm_medium=email&utm_source=footer>. > > -- > You received this message because you are subscribed to the Google Groups > "JaCoCo and EclEmma Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to jacoco+unsubscr...@googlegroups.com > <mailto:jacoco+unsubscr...@googlegroups.com>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jacoco/CAHNbw8kUx76m7Vr%2BbwMx2C%2B9SPbc-At91KOGAARm8MMwkK6NVQ%40mail.gmail.com > > <https://groups.google.com/d/msgid/jacoco/CAHNbw8kUx76m7Vr%2BbwMx2C%2B9SPbc-At91KOGAARm8MMwkK6NVQ%40mail.gmail.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "JaCoCo and EclEmma Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jacoco+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jacoco/41C398C6-FCF9-45E3-8C3D-914036AFDAED%40mountainminds.com.
