Re: Varnish code coverage results online
In message
Re: Varnish code coverage results online
On Fri, Jan 6, 2017 at 12:36 PM, Poul-Henning Kampwrote: > > In message >
Re: Varnish code coverage results online
On Fri, Jan 6, 2017 at 12:05 PM, Poul-Henning Kampwrote: > > In message > , Dridi > Boukelmoune > writes: > >>> I have also, at the same time raised the official project goal to >>> "90%+ of *all* executable code tested automatically". >> >>Only line coverage? Nothing for branches? >> >>90% branch coverage is even a bigger challenge ;) > > That's probably impossible, given the number of asserts we have. Yes, my approach is to run the test suite with assertions, and disable them for coverage. >>Another thing: if we want proper code coverage reports, [...] > > Are my my l33t HTML skills not proper ?! > > Seriously: This is good enough to tell us which bits of our > code isn't covered by "make check", which is the important part. > > Merging in data from other runs would not be a problem, my scripts > already has support for merging because some sourcefiles drop .gcda Cool, I had no idea! > in multiple directories already. We'd just need to set up a client > to run "make check" and submit a condensed version of the raw data. And here comes tools/vcov.sh :p > If you feel like running a Linux (or other) client, let me know and > we can get it set up, I think my current plan is to run this only > once per night. If we come up with something, Travis CI could do that for us for each push (so some work needs to be done to match the nightly scheduling). We'd just need to set up a new job, and I think it would even run as root and skip less tests. Dridi ___ varnish-dev mailing list varnish-dev@varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
Re: Varnish code coverage results online
In message, Dridi Boukelmoune writes: >> I have also, at the same time raised the official project goal to >> "90%+ of *all* executable code tested automatically". > >Only line coverage? Nothing for branches? > >90% branch coverage is even a bigger challenge ;) That's probably impossible, given the number of asserts we have. >Another thing: if we want proper code coverage reports, [...] Are my my l33t HTML skills not proper ?! Seriously: This is good enough to tell us which bits of our code isn't covered by "make check", which is the important part. Merging in data from other runs would not be a problem, my scripts already has support for merging because some sourcefiles drop .gcda in multiple directories already. We'd just need to set up a client to run "make check" and submit a condensed version of the raw data. If you feel like running a Linux (or other) client, let me know and we can get it set up, I think my current plan is to run this only once per night. Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 p...@freebsd.org | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. ___ varnish-dev mailing list varnish-dev@varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
Re: Varnish code coverage results online
> I have also, at the same time raised the official project goal to > "90%+ of *all* executable code tested automatically". Only line coverage? Nothing for branches? 90% branch coverage is even a bigger challenge ;) Another thing: if we want proper code coverage reports, we need to aggregate from different platforms if we hope to cover platform-specific code. Does lcov(1) work on FreeBSD? I know the "l" stands for Linux but at least with lcov I _know_ we can merge reports. The problem would then be: how to collect the reports? I personally use codecov.io which can receive and aggregate coverage reports and pin them to commits. There are other similar services, I'm not trying to promote a particular solution. Dridi ___ varnish-dev mailing list varnish-dev@varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
Varnish code coverage results online
I have spent the day retooling the code-coverage stuff I have been running manually since day almost-one of Varnish. The results are now online: http://varnish-cache.org/gcov (There's a link in the navbar at the bottom of the project homepage) I have also, at the same time raised the official project goal to "90%+ of *all* executable code tested automatically". Your assistance reaching this goal will be appreciated. Right now none of the curses(3) code is tested automatically. Doing so is non-trivial, but doable. Maybe varnishtest::process should simply (ha!) grow a -pty option ? Another big chunk of non-tested code is in libvgz, and just on general principles I might go in there with some lobotomizing #ifdefs and WRONG()'s, since that is code that we don't want to ever execute in the first place. Poul-Henning PS: Needless to say: The instant we hit the 90% goal I'll raise it again. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 p...@freebsd.org | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. ___ varnish-dev mailing list varnish-dev@varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev