Re: Code coverage report for LibreSSL tests
On Wed, May 6, 2015 at 2:54 PM, Harri Porten por...@froglogic.com wrote: Hi! We've started to generate code coverage reports for test suites of some projects on a regular basis. You'll find on overview for LibreSSL (Portable) here: http://www.opencoverage.net/libressl/index_html/sources.html Out of 59877 conditions and decisions 13389, or 22.36% were covered during yesterdays run. OpenSSL currently has 67832 conditions/decisions, i.e. the project's goal of simplifying the code has been reached. The percentage covered through the test suite is lower however. Hence I wonder whether our use of the Linux port or our build configuration is to blame. Testing the latest LibreSSL 2.2.0 development branch, I now get the following result testing just the libraries (libtls, libcrypto, libssl) as it appears is reported at opencoverage.net: Overall coverage rate: lines..: 39.6% (28500 of 72029 lines) functions..: 42.0% (2299 of 5472 functions) branches...: 31.8% (13420 of 42223 branches) After adding the openssl(1) command itself, the overall coverage looks like this: Overall coverage rate: lines..: 33.5% (29595 of 88381 lines) functions..: 40.4% (2339 of 5795 functions) branches...: 26.4% (14017 of 53168 branches) The test script I used to generate this is here: https://github.com/libressl-portable/portable/blob/master/gen-coverage-report.sh Here is an example where these stats can be a little misleading: a compatibility wrapper that has largely been turned into a no-op on purpose. In fact, we probably want to make sure that _no_ functions in this file are called eventually: http://www.opencoverage.net/libressl/index_html/source_332.html
Code coverage report for LibreSSL tests
Hi! We've started to generate code coverage reports for test suites of some projects on a regular basis. You'll find on overview for LibreSSL (Portable) here: http://www.opencoverage.net/libressl/index_html/sources.html Out of 59877 conditions and decisions 13389, or 22.36% were covered during yesterdays run. OpenSSL currently has 67832 conditions/decisions, i.e. the project's goal of simplifying the code has been reached. The percentage covered through the test suite is lower however. Hence I wonder whether our use of the Linux port or our build configuration is to blame. You might be able to use the reports for your coding and testing efforts. Attempts to increase test coverage can sometimes result in spotting of buggy code paths. If you spot a problem with our setup or have any questions please drop me a note. Harri.
Re: Code coverage report for LibreSSL tests
On Wed, May 6, 2015 at 2:54 PM, Harri Porten por...@froglogic.com wrote: Hi! We've started to generate code coverage reports for test suites of some projects on a regular basis. You'll find on overview for LibreSSL (Portable) here: http://www.opencoverage.net/libressl/index_html/sources.html Out of 59877 conditions and decisions 13389, or 22.36% were covered during yesterdays run. OpenSSL currently has 67832 conditions/decisions, i.e. the project's goal of simplifying the code has been reached. The percentage covered through the test suite is lower however. Hence I wonder whether our use of the Linux port or our build configuration is to blame. You might be able to use the reports for your coding and testing efforts. Attempts to increase test coverage can sometimes result in spotting of buggy code paths. Hi Hari, Nice tool! What version are you testing, the latest release or builds straight from the source tree? There are a few more tests in the OpenBSD that are not in portable release yet, though it is not a huge amount. I just put in a PR to add a few more of the missing ones. I have looked at the output from the 'gen-coverage-report.sh' script that is in the libressl-portable source tree, and a large number of untested functions were ones that were either simple aliases for other functions, or empty stubs. Is there a way that lcov can be modified to display condition coverage as well, or can either of these tools segregate stub function coverage from real ones? - Brent
