Hi Svante, nice that Kaitlin responded to you!
Not sure about Garbage collection though: This should not have an impact on Code coverage unless you implemented finalize() methods (which are deprecated now) or ReferenceQueues. > I assume you suggest doing all steps from within the JaCoCo API, right? Correct, we don’t provide a tool integration with diff support. Cheers, -marc > On 15. Nov 2021, at 13:39, Svante Schubert <svante.schub...@gmail.com> wrote: > > Hi Marc > > Am Fr., 5. Nov. 2021 um 17:16 Uhr schrieb Marc Hoffmann > <hoffm...@mountainminds.com <mailto:hoffm...@mountainminds.com>>: > Hi Svante, > > regarding the second part of your email: This is referred to as “incremental > code coverage”, there was research on this based on JaCoCo/EclEmma (see > https://www.cs.ubc.ca/tr/2008/tr-2008-14 > <https://www.cs.ubc.ca/tr/2008/tr-2008-14>). > > That was the one paper I had already read, but now I took the opportunity to > contact Kaitlin and Gail. > Kaitlin was instantly answering and although their source code is available, > she strongly suggested building upon your JaCoco code. > > In addition, she wrote: > "For what it's worth, I now make a distinction between differential code > coverage (run two sessions, diff them) and incremental code coverage (one > session, where you get to a point in the program, turn on code coverage, run > something, and then turn code coverage collection off). I found differential > code coverage to be much more useful than incremental code coverage. > > The first issue is garbage collection: you are likely to get garbage > collection at some point in any quantum of code running, which incremental > code coverage will capture. Differential code coverage will likely have > garbage collection in both, so the user won't see it. > > I can't remember the second issue well enough to really describe it down > concretely, but my memory is that there was more "initialization" code which > got surfaced in incremental code coverage than in differential code coverage. > I think it was things like initializing draw routines for the UI -- > something which happens all the time and is not interesting -- which > differential code coverage hid for the user that incremental code coverage > still showed." > > I think you’re scenario is supported using the JaCoCo API. Here are the the > steps: > > 1) Collect execution data for loading plain “Hello World” (exec1) > 2) Collect execution data for loading bold “Hello World” (exec2) > 3) Subtract exec1 from exec2 (see substract flag in > https://www.jacoco.org/jacoco/trunk/doc/api/org/jacoco/core/data/ExecutionData.html#merge(org.jacoco.core.data.ExecutionData,%20boolean) > > <https://www.jacoco.org/jacoco/trunk/doc/api/org/jacoco/core/data/ExecutionData.html#merge(org.jacoco.core.data.ExecutionData,%20boolean)> > ) > 4) Generate Coverage report with exec data from step 3) > > I will see if I can bundle the above functionality from your given examples > https://github.com/jacoco/jacoco/tree/master/org.jacoco.examples/src/org/jacoco/examples > > <https://github.com/jacoco/jacoco/tree/master/org.jacoco.examples/src/org/jacoco/examples> > I assume you suggest doing all steps from within the JaCoCo API, right? > > This report should only show code which was executed in addition to render > the bold text. > > Great! Many thanks, Marc! > Svante > > Cheers, > -marc > > >> On 5. Nov 2021, at 13:44, Svante Schubert <svante.schub...@gmail.com >> <mailto:svante.schub...@gmail.com>> wrote: >> >> Hi Marc, >> >> Thank you very much that does the trick! :-) >> >> I realized some "noise" on some tests being once executed and once not, but >> that might be as well a surefire problem: >> https://svanteschubert.github.io/odfdom-coverage-data/jacoco20211105-120509/org.odftoolkit.odfdom.dom.element.office/OfficeMetaElement.java.html#L129 >> >> <https://svanteschubert.github.io/odfdom-coverage-data/jacoco20211105-120509/org.odftoolkit.odfdom.dom.element.office/OfficeMetaElement.java.html#L129> >> https://svanteschubert.github.io/odfdom-coverage-data/jacoco20211104-101936/org.odftoolkit.odfdom.dom.element.office/OfficeMetaElement.java.html#L129 >> >> <https://svanteschubert.github.io/odfdom-coverage-data/jacoco20211104-101936/org.odftoolkit.odfdom.dom.element.office/OfficeMetaElement.java.html#L129> >> >> I will dive into the docu to see if I find more on how the coverage works... >> Any good pointers? >> >> In addition, does anyone have a pointer to existing open-source tooling >> (like Jenkins script) that pushes the coverage test results (like above) to >> some webserver (like GitHub pages)? >> My first aim would be to always align the coverage results with a Git commit. >> >> Allow me to paint the whole picture: >> As mentioned I am working on the OpenDocument format (I am co-chair of OASIS >> ODF TC <https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=office> >> and co-maintainer of the ODF Toolkit <https://github.com/tdf/odftoolkit>). >> In the ODF Toolkit and LibreOffice (LO) we have the problem that the tests >> are taking quite long (in LO even longer than the build). >> >> I like to evaluate a trick using coverage: >> First, I am loading a test document in an ODF application like LO - say only >> containing "Hello World" - and saving aside all the code coverage during the >> test. All the source code lines are required to load it (and layout it in >> LO). >> Second, I am loading a second test document - again "Hello World" but this >> time in bold - and saving aside again the code coverage data for the second >> test. >> >> I assume that the second document load test, which contains, in addition, >> the "bold" feature, will have higher test coverage. >> The subtraction of both test coverages should reveal the lines solely >> required for the "bold" feature. We like to align these feature lines with a >> very fast feature-smoke test in conjunction with a GIT pre-commit-hook. >> >> In addition, we would be able to "draw" some "feature map" of the source >> code for newcomers. >> >> Is anyone aware of existing research in this area? I have only skimmed the >> research papers your project has listed: >> https://www.jacoco.org/research/index.html >> <https://www.jacoco.org/research/index.html> >> Our work will be all open-source. We were able to receive a small grant from >> the German Ministry of Research >> <https://prototypefund.de/en/project/cover-rest-find-features-avoid-stress/>. >> We are Thorsten Behrens - taking over the C++ part, his son Linus Behrens >> (Python) and myself for Java. >> >> Last but not least, imagine we would be "the kings" of JCov, Jacoco, >> Cobertura... all Java coverage tools, wouldn't we think about reusing >> modules? Like front/backend as Clang is using? >> >> Have a great weekend & greetings from Berlin! >> Svante >> >> PS: One design aspect: The most revolutionary thing I observed in the past >> years in IDEs is the tendency to move from all-source test/compile to >> incremental testing/compilation. >> My favourite example is the Visual Studio Code (VSCode) extension (nice idea >> to split the language-dependent logic into a server part (language server >> <https://code.visualstudio.com/api/language-extensions/language-server-extension-guide> >> - quick intro <https://www.ncameron.org/blog/how-the-rls-works/>) and a >> stupid-only-layouting client part of the VSCode extension), where the >> official VSCode Rust extension <https://rust-analyzer.github.io/> switched >> to the incremental approach. >> >> >> Am Do., 4. Nov. 2021 um 08:50 Uhr schrieb Marc Hoffmann >> <hoffm...@mountainminds.com <mailto:hoffm...@mountainminds.com>>: >> Hi Svante, >> >> as you can see on the sessions page you only have execution data for a >> single library class: >> >> https://svanteschubert.github.io/odfdom-coverage-data/jacoco-sessions.html >> <https://svanteschubert.github.io/odfdom-coverage-data/jacoco-sessions.html> >> >> As this is the only (not excluded) IT test executed with >> maven-failsafe-plugin it looks like that there is simply no coverage for >> your unit tests executed with the maven-surefire-plugin plugin. >> >> The reason is that you overwrite the argLine which was prepared by JaCoCo >> before: >> >> https://github.com/tdf/odftoolkit/blob/master/odfdom/pom.xml#L257 >> <https://github.com/tdf/odftoolkit/blob/master/odfdom/pom.xml#L257> >> >> Please see JaCoCo documentation how to combine the JaCoCo agent >> configuration with your own VM arguments correctly: >> >> https://www.jacoco.org/jacoco/trunk/doc/prepare-agent-mojo.html >> <https://www.jacoco.org/jacoco/trunk/doc/prepare-agent-mojo.html> >> >> Probably the line should read: >> >> <argLine>@{argLine} >> -Dfile.encoding=${project.build.sourceEncoding}</argLine> >> >> Best refards, >> -marc >> >> >>> On 3. Nov 2021, at 23:45, Svante Schubert <svante.schub...@gmail.com >>> <mailto:svante.schub...@gmail.com>> wrote: >>> >>> Hi Marc, >>> >>> the tests are running and there are test output files (and debug output >>> from the uncovered files), but you are right the special JUnit class might >>> not be the problem. >>> Seems, there is only one test providing coverage: >>> https://svanteschubert.github.io/odfdom-coverage-data/ >>> <https://svanteschubert.github.io/odfdom-coverage-data/> <- I have uploaded >>> the JaCoCo report. >>> Only one file has coverage, being tested by an integration test: >>> https://github.com/tdf/odftoolkit/blob/coverage/odfdom/src/test/java/org/odftoolkit/odfdom/integrationtest/JarManifestIT.java >>> >>> <https://github.com/tdf/odftoolkit/blob/coverage/odfdom/src/test/java/org/odftoolkit/odfdom/integrationtest/JarManifestIT.java> >>> Now I suspect that I am doing the Maven pom.xml wrong as there are only >>> integration tests and JaCoCo in the reporting, see: >>> https://github.com/tdf/odftoolkit/blob/coverage/odfdom/pom.xml#L443 >>> <https://github.com/tdf/odftoolkit/blob/coverage/odfdom/pom.xml#L443> >>> >>> Likely I have overseen something... >>> Do you have any further hint? >>> >>> Thanks in advance, >>> Svante >>> >>> PS: It is not the small test you asked for, but you may via command line >>> clone git repo https://github.com/tdf/odftoolkit/ >>> <https://github.com/tdf/odftoolkit/> >>> switch to the coverage branch >>> change to subdirectory odfdom of odftoolkit >>> call 'mvn install' (using JDK >=9) >>> call 'mvn jacoco:report' >>> >>> >>> Am Mi., 3. Nov. 2021 um 22:38 Uhr schrieb Marc Hoffmann >>> <hoffm...@mountainminds.com <mailto:hoffm...@mountainminds.com>>: >>> Hi Svante, >>> >>> JaCoCo does not depend on a specific testing frameworks and I don’t think >>> there are technical limitations with @RunWith(Parameterized.class). Maybe >>> it does not work at all in your case and the code is in fact not executed? >>> >>> If you think there is a problem can you please extract a minimal reproducer >>> to demonstrate the problem? >>> >>> Thanks, >>> -marc >>> >>>> On 3. Nov 2021, at 21:34, Svante Schubert <svante.schub...@gmail.com >>>> <mailto:svante.schub...@gmail.com>> wrote: >>>> >>>> Hello coverage experts! >>>> >>>> Does anybody have success using JaCoCo and JUnit's >>>> @RunWith(Parameterized.class) >>>> <https://junit.org/junit4/javadoc/4.12/org/junit/runners/Parameterized.html>? >>>> >>>> I am working on the ODF file format and I am running a test transformation >>>> for every existing ODF test document in our test folder comparing it with >>>> the prior reference result. >>>> >>>> See >>>> https://github.com/tdf/odftoolkit/blob/coverage/odfdom/src/test/java/org/odftoolkit/odfdom/changes/RoundtripTest.java#L36 >>>> >>>> <https://github.com/tdf/odftoolkit/blob/coverage/odfdom/src/test/java/org/odftoolkit/odfdom/changes/RoundtripTest.java#L36> >>>> >>>> By using this generic test the coverage of the changes packages should be >>>> high, but the coverage of package org/odftoolkit/odfdom/changes is instead >>>> zero! >>>> >>>> Thanks in advance for any hints! >>>> Svante >>>> >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "JaCoCo and EclEmma Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to jacoco+unsubscr...@googlegroups.com >>>> <mailto:jacoco+unsubscr...@googlegroups.com>. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/jacoco/75f55ae9-9f24-4261-93bc-54c2e1c01367n%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/jacoco/75f55ae9-9f24-4261-93bc-54c2e1c01367n%40googlegroups.com?utm_medium=email&utm_source=footer>. >>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "JaCoCo and EclEmma Users" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/jacoco/dXbkWvvKj5Q/unsubscribe >>> <https://groups.google.com/d/topic/jacoco/dXbkWvvKj5Q/unsubscribe>. >>> To unsubscribe from this group and all its topics, send an email to >>> jacoco+unsubscr...@googlegroups.com >>> <mailto:jacoco+unsubscr...@googlegroups.com>. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/jacoco/3BBA4FEC-403A-46F2-BE9E-2F46E3C356CE%40mountainminds.com >>> >>> <https://groups.google.com/d/msgid/jacoco/3BBA4FEC-403A-46F2-BE9E-2F46E3C356CE%40mountainminds.com?utm_medium=email&utm_source=footer>. >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "JaCoCo and EclEmma Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to jacoco+unsubscr...@googlegroups.com >>> <mailto:jacoco+unsubscr...@googlegroups.com>. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/jacoco/CAHNbw8nSqxaAxyKRzP0Oa01mKD0awZN36ReBmgiPucQToVRqiQ%40mail.gmail.com >>> >>> <https://groups.google.com/d/msgid/jacoco/CAHNbw8nSqxaAxyKRzP0Oa01mKD0awZN36ReBmgiPucQToVRqiQ%40mail.gmail.com?utm_medium=email&utm_source=footer>. >> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "JaCoCo and EclEmma Users" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/jacoco/dXbkWvvKj5Q/unsubscribe >> <https://groups.google.com/d/topic/jacoco/dXbkWvvKj5Q/unsubscribe>. >> To unsubscribe from this group and all its topics, send an email to >> jacoco+unsubscr...@googlegroups.com >> <mailto:jacoco+unsubscr...@googlegroups.com>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/jacoco/AB47E08D-DE4D-4C16-8CB2-AA12DA0EF0AE%40mountainminds.com >> >> <https://groups.google.com/d/msgid/jacoco/AB47E08D-DE4D-4C16-8CB2-AA12DA0EF0AE%40mountainminds.com?utm_medium=email&utm_source=footer>. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "JaCoCo and EclEmma Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to jacoco+unsubscr...@googlegroups.com >> <mailto:jacoco+unsubscr...@googlegroups.com>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/jacoco/CAHNbw8kUx76m7Vr%2BbwMx2C%2B9SPbc-At91KOGAARm8MMwkK6NVQ%40mail.gmail.com >> >> <https://groups.google.com/d/msgid/jacoco/CAHNbw8kUx76m7Vr%2BbwMx2C%2B9SPbc-At91KOGAARm8MMwkK6NVQ%40mail.gmail.com?utm_medium=email&utm_source=footer>. > > > -- > You received this message because you are subscribed to a topic in the Google > Groups "JaCoCo and EclEmma Users" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/jacoco/dXbkWvvKj5Q/unsubscribe > <https://groups.google.com/d/topic/jacoco/dXbkWvvKj5Q/unsubscribe>. > To unsubscribe from this group and all its topics, send an email to > jacoco+unsubscr...@googlegroups.com > <mailto:jacoco+unsubscr...@googlegroups.com>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jacoco/41C398C6-FCF9-45E3-8C3D-914036AFDAED%40mountainminds.com > > <https://groups.google.com/d/msgid/jacoco/41C398C6-FCF9-45E3-8C3D-914036AFDAED%40mountainminds.com?utm_medium=email&utm_source=footer>. > > -- > You received this message because you are subscribed to the Google Groups > "JaCoCo and EclEmma Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to jacoco+unsubscr...@googlegroups.com > <mailto:jacoco+unsubscr...@googlegroups.com>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jacoco/CAHNbw8nZWceHp44WwKK817LN%2B1-U7vNOOzmnFHLqG69S%2B5Jnrw%40mail.gmail.com > > <https://groups.google.com/d/msgid/jacoco/CAHNbw8nZWceHp44WwKK817LN%2B1-U7vNOOzmnFHLqG69S%2B5Jnrw%40mail.gmail.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "JaCoCo and EclEmma Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jacoco+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jacoco/BD44F030-764E-4B4A-AF6C-87D30B5D4C54%40mountainminds.com.
