Hi, I had troubles with several Apache Commons projects, but I managed to compile some interesting results nevertheless.
In short: - Adding probes for every INVOKE opcode results in about 2.4x as many probes. - The runtime ranges from 100% to 146% of the current JaCoCo release. - The number of instructions found to be covered increases by 0 to 2.5 percentage points (meaning, for example, the updated version finds 96% covered instructions while the current release finds 94%). I am satisfied with the results (finding higher coverage percentages). But I also see that, depending on the analyzed code, having more probes can dramatically increase the runtime. Do you need more numbers? (I'd need help then) Do you think changing JaCoCo in order to produce results outlined in this mail is a good idea? What do you think about having a separate version of JaCoCo which is more precise (and, sadly, slower)? Can you think of a way to speed up JaCoCo while still having better coverage results? Details: Apache Commons Beanutils: 4677 probes -> 12529 probes (2.7x) Runtime: 101% Coverage: +2.0% Apache Commons Collections: 19220 probes -> 59283 probes (3.1x) Runtime: 99% Coverage: +2.1% Apache Commons Math: 49340 probes -> 148996 probes (3.0x) Runtime: 146% Coverage: +2.3% Apache Commons Primitives: 11187 probes -> 28936 probes (2.6x) Runtime: 124% Coverage: +2.5% Jenkins Core: 101964 probes -> 188196 probes (1.8x) Runtime: 102% Coverage: +0.0% Fitnesse: 66503 probes -> 163509 probes (2.5x) Runtime: 100% Coverage: +0.3% Best regards, Carsten On Mon, Nov 3, 2014 at 9:31 PM, Marc R. Hoffmann <[email protected] > wrote: > Hi, > > we discussed this limitation many times before. Currently JaCoCo is > designed to add the smallest possible overhead to the application under > test as JaCoCo is mainly used for large scale projects. > > I like the idea of adding additional probes before every INVOKE* > instruction as this is the most likely place for implicit instructions > (beside NPE, IOOB, CCE, etc.). It would be interesting to see how this > increases > > 1) the total number of probes > 2) the execution time > > for a reasonable sized test set like e.g. Apache commons collections. > > BTW, I don't think this feature could be configurable: The way to > determine probes has to be absolutely deterministic at runtime and analysis > time. So it has to be ensured that the config setting must be the same. Or > the setting is written to the exec file, which makes merging impossible of > exec files with different settings. > > Cheers, > -marc > > > On 03.11.14 18:15, [email protected] wrote: > >> Dear all, >> >> JaCoCo does not detect that certain instructions are run (covered) if >> these lead to an implicit exception. In the following example, if c() >> throws an exception, the result indicate no coverage for b(). >> >> public void foo() { >> a(); >> try { >> b(); >> c(); >> } catch (Throwable t) {} >> } >> >> In the following example, no coverage at all is reported, assuming c() >> again throws an exception. >> >> public void bar() { >> a(); >> b(); >> c(); >> } >> >> Note that the last example is quite common when writing JUnit 4 tests >> using the "expected" keyword, as in the following example. >> >> @Test(expected=SomeException.class) >> public void testBar() { >> ... >> } >> >> I experimented with adding more probes inserted in front of opcodes that >> may thrown an (implicit) exception. With such probes more precise coverage >> results are obtained, at the cost of having more probes. This means a >> slower execution (analysis) speed, and bigger instrumented class files. >> >> In my _preliminary_ experiments I see about 1-2% of increased coverage, >> if coverage of the Test classes is included in the report. This comes at >> the cost of about 1% more runtime when compared to the current release of >> JaCoCo. >> >> I am now interested in your opinions on this. Do you think having a more >> precise analysis as described is worthwhile? Should the feature be in >> JaCoCo, or not? Should it be configurable, possibly disabled by default? >> >> On a more technical note, it is possible to only add probes for some of >> the opcodes which may thrown an exception. For example, one could add >> probes for INVOKE opcodes (which would help with the JUnit tests, based on >> my experience), and not add probes for PUTFIELD and array opcodes. >> >> Best regards, >> Carsten >> >> > -- > You received this message because you are subscribed to a topic in the > Google Groups "JaCoCo and EclEmma Users" group. > To unsubscribe from this topic, visit https://groups.google.com/d/ > topic/jacoco/s6Xbj69308Y/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- Carsten Otto [email protected] www.c-otto.de -- You received this message because you are subscribed to the Google Groups "JaCoCo and EclEmma Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
