Hi Carsten,
thanks for this detailed analysis! The increase in probes is quite
substantial. Interesting to see that the factor is that different for
the different products.
What does the increased coverage mean at the end? The tests itself are
not improved by using a different JaCoCo version. Now it is more correct
in terms of the amount of code that is covered by a given test suite.
Still the tests seem to lack to cover some of the non-exception
execution paths.
It would be interesting to see some examples of highlighted source code
for both versions of JaCoCo.
The modified version definitely shows more the code covered which has
actually been executed and gives a better picture in case of anticipated
exceptions. On the other hand since many years I try to teach people
that they should not care about tested code or percentage values. The
focus should be providing additional tests for yet untested code.
> What do you think about having a separate version of JaCoCo which is
more precise (and, sadly, slower)?
Speaking for the JaCoCo project we will stay with one version. We can't
handle multiple versions and this will also confuse users as many users
use JaCoCo in different tools.
> Can you think of a way to speed up JaCoCo while still having better
coverage results?
Not sure whether this can be implemented but I'm thinking about
installing exception handlers with the instrumented code since quite
some time. And as said before: The only reasonable way to get "better
coverage results" is to improve your tests. Anything else does not add
any value to your project.
Cheers,
-marc
On 05.11.14 15:18, Carsten Otto wrote:
Hi,
I had troubles with several Apache Commons projects, but I managed to
compile some interesting results nevertheless.
In short:
- Adding probes for every INVOKE opcode results in about 2.4x as many
probes.
- The runtime ranges from 100% to 146% of the current JaCoCo release.
- The number of instructions found to be covered increases by 0 to 2.5
percentage points (meaning, for example, the updated version finds 96%
covered instructions while the current release finds 94%).
I am satisfied with the results (finding higher coverage percentages).
But I also see that, depending on the analyzed code, having more
probes can dramatically increase the runtime.
Do you need more numbers? (I'd need help then)
Do you think changing JaCoCo in order to produce results outlined in
this mail is a good idea?
What do you think about having a separate version of JaCoCo which is
more precise (and, sadly, slower)?
Can you think of a way to speed up JaCoCo while still having better
coverage results?
Details:
Apache Commons Beanutils:
4677 probes -> 12529 probes (2.7x)
Runtime: 101%
Coverage: +2.0%
Apache Commons Collections:
19220 probes -> 59283 probes (3.1x)
Runtime: 99%
Coverage: +2.1%
Apache Commons Math:
49340 probes -> 148996 probes (3.0x)
Runtime: 146%
Coverage: +2.3%
Apache Commons Primitives:
11187 probes -> 28936 probes (2.6x)
Runtime: 124%
Coverage: +2.5%
Jenkins Core:
101964 probes -> 188196 probes (1.8x)
Runtime: 102%
Coverage: +0.0%
Fitnesse:
66503 probes -> 163509 probes (2.5x)
Runtime: 100%
Coverage: +0.3%
Best regards,
Carsten
On Mon, Nov 3, 2014 at 9:31 PM, Marc R. Hoffmann
<[email protected] <mailto:[email protected]>> wrote:
Hi,
we discussed this limitation many times before. Currently JaCoCo
is designed to add the smallest possible overhead to the
application under test as JaCoCo is mainly used for large scale
projects.
I like the idea of adding additional probes before every INVOKE*
instruction as this is the most likely place for implicit
instructions (beside NPE, IOOB, CCE, etc.). It would be
interesting to see how this increases
1) the total number of probes
2) the execution time
for a reasonable sized test set like e.g. Apache commons collections.
BTW, I don't think this feature could be configurable: The way to
determine probes has to be absolutely deterministic at runtime and
analysis time. So it has to be ensured that the config setting
must be the same. Or the setting is written to the exec file,
which makes merging impossible of exec files with different settings.
Cheers,
-marc
On 03.11.14 18:15, [email protected]
<mailto:[email protected]> wrote:
Dear all,
JaCoCo does not detect that certain instructions are run
(covered) if these lead to an implicit exception. In the
following example, if c() throws an exception, the result
indicate no coverage for b().
public void foo() {
a();
try {
b();
c();
} catch (Throwable t) {}
}
In the following example, no coverage at all is reported,
assuming c() again throws an exception.
public void bar() {
a();
b();
c();
}
Note that the last example is quite common when writing JUnit
4 tests using the "expected" keyword, as in the following example.
@Test(expected=SomeException.class)
public void testBar() {
...
}
I experimented with adding more probes inserted in front of
opcodes that may thrown an (implicit) exception. With such
probes more precise coverage results are obtained, at the cost
of having more probes. This means a slower execution
(analysis) speed, and bigger instrumented class files.
In my _preliminary_ experiments I see about 1-2% of increased
coverage, if coverage of the Test classes is included in the
report. This comes at the cost of about 1% more runtime when
compared to the current release of JaCoCo.
I am now interested in your opinions on this. Do you think
having a more precise analysis as described is worthwhile?
Should the feature be in JaCoCo, or not? Should it be
configurable, possibly disabled by default?
On a more technical note, it is possible to only add probes
for some of the opcodes which may thrown an exception. For
example, one could add probes for INVOKE opcodes (which would
help with the JUnit tests, based on my experience), and not
add probes for PUTFIELD and array opcodes.
Best regards,
Carsten
--
You received this message because you are subscribed to a topic in
the Google Groups "JaCoCo and EclEmma Users" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/jacoco/s6Xbj69308Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email
to [email protected]
<mailto:jacoco%[email protected]>.
For more options, visit https://groups.google.com/d/optout.
--
Carsten Otto
[email protected] <mailto:[email protected]>
www.c-otto.de <http://www.c-otto.de>
--
You received this message because you are subscribed to the Google
Groups "JaCoCo and EclEmma Users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to [email protected]
<mailto:[email protected]>.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "JaCoCo
and EclEmma Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
