We attached both the jacoco agent and our custom method-level-only agent
("funccover") to some sample codes and averaged their run execution time.1) https://github.com/google/google-java-format We attached “jacocoagent” and “funccover” to Google Java Formatter to test and compare “funccover”s performance. Results are the average of 200 runs. Normally formatter’s execution time is around 200 seconds, with “jacocoagent” it runs around 97 seconds (48% overhead) slower and for “funccover” it runs around 30 seconds slower (15% overhead). So our agent adds x3 less overhead for this case. 2) Merge Sort Algorithm In this test we implemented a program that sorts an array of size 10^7 with a merge sort algorithm. In each binary, we sort the array 50 times per run to be able to ignore the constant overhead of instrumentation since it is only done once per run. There is no I/O bottleneck and most of the time is spent on the CPU. Binary runs around 216 seconds without the instrumentation. Our agent "funccover" adds around 1.97% overhead whilst jacoco adds around 11.4%. Our instrumentation adds 5x less overhead. Results are average of 50x runs. 3) Simple Function Calls - Filtered We attached “jacocoagent” and “funccover” to a simple program that calls the following function 10^8 times with numbers from 1 to 10^8. static int f(int x) { if(x == 0) return 0; if(x % 2 == 0) return f(x / 2); return f(x / 2) + 1; } Results are average of 500x runs. Here jacoco adds 17% overhead whilst funccover adds less than 7% overhead. All this makes sense of course, you are updating less often the coverage data if you only add a single probe per method. On Fri, Oct 29, 2021 at 2:58 PM Marc Hoffmann <hoffm...@mountainminds.com> wrote: > Wow, can you share your test setup? Our measurements e.g. running test > suites of apache commons-collections show an overhead of about 20%. > > > On 29. Oct 2021, at 13:36, 'Antoine Baudoux' via JaCoCo and EclEmma Users < > jacoco@googlegroups.com> wrote: > > Do you have performance meassurements for your use case? > > > Yes, we did some benchmarks with a home-made agent that did function-level > instrumentation. > The instrumentation overhead was anywhere from 2x to 5x with > instruction-level coverage than function-level coverage. > > On Tue, Oct 19, 2021 at 5:56 PM Marc Hoffmann <hoffm...@mountainminds.com> > wrote: > >> Dear Antoine, >> >> adding different counters is actually not supported. >> >> Maintaining different instrumentation/analysis strategies is something >> we‘re actually not looking for. >> >> Do you have performance meassurements for your use case? >> >> Best regards, >> -marc >> >> On 19. Oct 2021, at 17:16, 'Antoine Baudoux' via JaCoCo and EclEmma Users >> <jacoco@googlegroups.com> wrote: >> >> >> By "method level coverage" I mean only adding one counter, one >> instruction at the start of each method to check if the method was called >> or not. And not adding any branch counter, so that there is less >> instrumentation overhead. >> I did not see a way to do that, if it is already supported I'm sorry >> about the confusion. >> >> Kind regards, >> >> Antoine >> >> >> On Tue, Oct 19, 2021 at 5:11 PM Marc Hoffmann <hoffm...@mountainminds.com> >> wrote: >> >>> Dear Antoine, >>> >>> JaCoCo already supports method level coverage. >>> >>> What exactly is the performance problem that requires a different >>> implementation? >>> >>> In general we’re very reluctant to such configuration options as they >>> require different implementations for the whole path: Instrumentation, exec >>> files and analysis. And of course causes more questions by users that get >>> these configuration options wrong... >>> >>> Regards, >>> -marc >>> >>> >>> >>> On 9. Oct 2021, at 12:33, 'Antoine Baudoux' via JaCoCo and EclEmma Users >>> <jacoco@googlegroups.com> wrote: >>> >>> Hello, >>> >>> Would you welcome a contribution to add method-level coverage? >>> Method-level coverage (as opposed to line-level, the current default >>> mode for JaCoCo) can be useful to track dead code, i.e. methods that never >>> get called. >>> >>> It has lower overhead and therefore could be enabled continuously for >>> code running in production, allowing for dead code detection and removal. >>> >>> If you think that it could be added to the JaCoCo codebase I would be >>> happy to contribute. Please let me know your thoughts. >>> >>> Kind regards, >>> >>> Antoine >>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "JaCoCo and EclEmma Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to jacoco+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/jacoco/e6da49e8-56a9-4c14-9e9d-fdea4c152349n%40googlegroups.com >>> <https://groups.google.com/d/msgid/jacoco/e6da49e8-56a9-4c14-9e9d-fdea4c152349n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "JaCoCo and EclEmma Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to jacoco+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/jacoco/11BDA5D5-887C-4937-BADE-3EE7DF9D84F9%40mountainminds.com >>> <https://groups.google.com/d/msgid/jacoco/11BDA5D5-887C-4937-BADE-3EE7DF9D84F9%40mountainminds.com?utm_medium=email&utm_source=footer> >>> . >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "JaCoCo and EclEmma Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to jacoco+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/jacoco/CADFGoLKzaShAcuPWF6fp_U9UYom93Zgw7PzacM37-UtvA7LBbg%40mail.gmail.com >> <https://groups.google.com/d/msgid/jacoco/CADFGoLKzaShAcuPWF6fp_U9UYom93Zgw7PzacM37-UtvA7LBbg%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "JaCoCo and EclEmma Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to jacoco+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/jacoco/5C499517-6BB3-40E0-AA57-0AB68F80D6D5%40mountainminds.com >> <https://groups.google.com/d/msgid/jacoco/5C499517-6BB3-40E0-AA57-0AB68F80D6D5%40mountainminds.com?utm_medium=email&utm_source=footer> >> . >> > > -- > You received this message because you are subscribed to the Google Groups > "JaCoCo and EclEmma Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to jacoco+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jacoco/CADFGoLLDcycVTEYBoTjeyin%3D0C6JK0ermBTuoKta3ZR91UrSqw%40mail.gmail.com > <https://groups.google.com/d/msgid/jacoco/CADFGoLLDcycVTEYBoTjeyin%3D0C6JK0ermBTuoKta3ZR91UrSqw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > > -- > You received this message because you are subscribed to the Google Groups > "JaCoCo and EclEmma Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to jacoco+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jacoco/F51ED86A-4112-42C9-A114-387A2ED1E43D%40mountainminds.com > <https://groups.google.com/d/msgid/jacoco/F51ED86A-4112-42C9-A114-387A2ED1E43D%40mountainminds.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "JaCoCo and EclEmma Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jacoco+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jacoco/CADFGoL%2BuOwjU9VKfDizDzBjWVjUtZB5sfGrR9wuxW1ZXMxJHTA%40mail.gmail.com.
