Re: Varnish code coverage results online

[Poul-Henning Kamp]

In message 

Re: Varnish code coverage results online

[Dridi Boukelmoune]

On Fri, Jan 6, 2017 at 12:36 PM, Poul-Henning Kamp  wrote:
> 
> In message 
> 

Re: Varnish code coverage results online

[Dridi Boukelmoune]

On Fri, Jan 6, 2017 at 12:05 PM, Poul-Henning Kamp  wrote:
> 
> In message 
> , Dridi 
> Boukelmoune
> writes:
>
>>> I have also, at the same time raised the official project goal to
>>> "90%+ of *all* executable code tested automatically".
>>
>>Only line coverage? Nothing for branches?
>>
>>90% branch coverage is even a bigger challenge ;)
>
> That's probably impossible, given the number of asserts we have.

Yes, my approach is to run the test suite with assertions, and disable
them for coverage.

>>Another thing: if we want proper code coverage reports, [...]
>
> Are my my l33t HTML skills not proper ?!
>
> Seriously:  This is good enough to tell us which bits of our
> code isn't covered by "make check", which is the important part.
>
> Merging in data from other runs would not be a problem, my scripts
> already has support for merging because some sourcefiles drop .gcda

Cool, I had no idea!

> in multiple directories already.  We'd just need to set up a client
> to run "make check" and submit a condensed version of the raw data.

And here comes tools/vcov.sh :p

> If you feel like running a Linux (or other) client, let me know and
> we can get it set up, I think my current plan is to run this only
> once per night.

If we come up with something, Travis CI could do that for us for each
push (so some work needs to be done to match the nightly scheduling).
We'd just need to set up a new job, and I think it would even run as
root and skip less tests.

Dridi

___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: Varnish code coverage results online

[Poul-Henning Kamp]

In message 
, Dridi 
Boukelmoune 
writes:

>> I have also, at the same time raised the official project goal to
>> "90%+ of *all* executable code tested automatically".
>
>Only line coverage? Nothing for branches?
>
>90% branch coverage is even a bigger challenge ;)

That's probably impossible, given the number of asserts we have.

>Another thing: if we want proper code coverage reports, [...]

Are my my l33t HTML skills not proper ?!

Seriously:  This is good enough to tell us which bits of our
code isn't covered by "make check", which is the important part.

Merging in data from other runs would not be a problem, my scripts
already has support for merging because some sourcefiles drop .gcda
in multiple directories already.  We'd just need to set up a client
to run "make check" and submit a condensed version of the raw data.

If you feel like running a Linux (or other) client, let me know and
we can get it set up, I think my current plan is to run this only
once per night.

Poul-Henning

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: Varnish code coverage results online

[Dridi Boukelmoune]

> I have also, at the same time raised the official project goal to
> "90%+ of *all* executable code tested automatically".

Only line coverage? Nothing for branches?

90% branch coverage is even a bigger challenge ;)

Another thing: if we want proper code coverage reports, we need to
aggregate from different platforms if we hope to cover platform-specific
code. Does lcov(1) work on FreeBSD? I know the "l" stands for Linux
but at least with lcov I _know_ we can merge reports.

The problem would then be: how to collect the reports?

I personally use codecov.io which can receive and aggregate coverage
reports and pin them to commits. There are other similar services, I'm
not trying to promote a particular solution.

Dridi

___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev