[chromium-dev] Re: Confusing test coverage numbers

[John Grabowski]

I'll take a look at this.
jrg


On Fri, Jul 17, 2009 at 11:40 AM, Andrew Scherkus wrote:

> I think something's busted.  If you look at the percent of test code
> executed:
> http://build.chromium.org/buildbot/perf/linux-debug/coverage/report.html?history=150&header=Linux&graph=PercentCoveredTest
>
>
> ...the
> graph goes down when people check in unit tests, and goes up when people
> clean up or modify existing tests.
>
> If you look to the far left of this graph, you'll notice our test code
> coverage jumped from 58% to 72%:
>
> http://build.chromium.org/buildbot/perf/mac-debug/coverage/report.html?history=300&header=Mac&graph=PercentCoveredTest
>
>
> There's
> no way the CLs checked in are responsible for that jump.  What appears to
> have happened was previously non-executing-but-accounted-for tests (like the
> ones I checked in) tests all started getting executed.  One of many
> examples:
>
> http://build.chromium.org/buildbot/coverage/mac-debug/20419/CHROMIUM/chrome/browser/tabs/index.html
> 
> http://build.chromium.org/buildbot/coverage/mac-debug/20426/CHROMIUM/chrome/browser/tabs/index.html
>
> Andrew
>
> On Thu, Jul 16, 2009 at 4:06 PM, John Grabowski  wrote:
>
>> For starters, thanks for caring about coverage
>>
>> It is possible your checkin was timed poorly (didn't land in time for the
>> bot
>> start), but looking at the coverage scripts I don't see how that could 
>> happen.
>> I noticed you landed, then reverted, this CL the day before.  My best
>> guess
>> is that such an action was enough info for croc to learn the filename and 
>> "stick", but since you reverted it didn't compile or get run.
>>
>> Let's wait one more run and see if things fix themselves.
>>
>> Ping again if the problem continues.
>>
>> jrg
>>
>>
>> On Thu, Jul 16, 2009 at 3:38 PM, Andrew Scherkus 
>> wrote:
>>
>>> I think we're getting confused by time zones here.
>>> I checked in at Wednesday at 18:26 PST (UTC-7), and assuming the coverage
>>> bot is also in my time zone that'd mean these results were generated ~20
>>> minutes afterwards.
>>>
>>>
>>> On Thu, Jul 16, 2009 at 3:22 PM, Nick Carter  wrote:
>>>
 The bottom of the link says that "coverage information was generated
 Wednesday".  Your checkin was Thursday.  Would that have something to do
 with it?

  - nick

 On Thu, Jul 16, 2009 at 12:43 PM, Andrew Scherkus <
 scher...@chromium.org> wrote:

> So I checked in a change that did some refactoring and added unit tests
> for two classes that were previously not tested:
> http://src.chromium.org/viewvc/chrome?view=rev&revision=20836
>
> According to the coverage graphs, that change actually brought both the
> source code and test code coverage *down*.  What's interesting is the
> coverage analysis reports my new tests aren't being executed at all:
>
> http://build.chromium.org/buildbot/coverage/linux-debug/20836/CHROMIUM/media/filters/index.html
>
> Now I have no clue who/what process is generating these numbers, but
> maybe it needs to be clobbered.
>
> Andrew
>
>
>

>>>
>>> >>>
>>>
>>
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Confusing test coverage numbers

[Andrew Scherkus]

I think something's busted.  If you look at the percent of test code
executed:
http://build.chromium.org/buildbot/perf/linux-debug/coverage/report.html?history=150&header=Linux&graph=PercentCoveredTest

...the
graph goes down when people check in unit tests, and goes up when people
clean up or modify existing tests.

If you look to the far left of this graph, you'll notice our test code
coverage jumped from 58% to 72%:
http://build.chromium.org/buildbot/perf/mac-debug/coverage/report.html?history=300&header=Mac&graph=PercentCoveredTest

There's
no way the CLs checked in are responsible for that jump.  What appears to
have happened was previously non-executing-but-accounted-for tests (like the
ones I checked in) tests all started getting executed.  One of many
examples:
http://build.chromium.org/buildbot/coverage/mac-debug/20419/CHROMIUM/chrome/browser/tabs/index.html

http://build.chromium.org/buildbot/coverage/mac-debug/20426/CHROMIUM/chrome/browser/tabs/index.html

Andrew

On Thu, Jul 16, 2009 at 4:06 PM, John Grabowski  wrote:

> For starters, thanks for caring about coverage
>
> It is possible your checkin was timed poorly (didn't land in time for the
> bot
> start), but looking at the coverage scripts I don't see how that could happen.
> I noticed you landed, then reverted, this CL the day before.  My best guess
> is that such an action was enough info for croc to learn the filename and 
> "stick", but since you reverted it didn't compile or get run.
>
> Let's wait one more run and see if things fix themselves.
>
> Ping again if the problem continues.
>
> jrg
>
>
> On Thu, Jul 16, 2009 at 3:38 PM, Andrew Scherkus wrote:
>
>> I think we're getting confused by time zones here.
>> I checked in at Wednesday at 18:26 PST (UTC-7), and assuming the coverage
>> bot is also in my time zone that'd mean these results were generated ~20
>> minutes afterwards.
>>
>>
>> On Thu, Jul 16, 2009 at 3:22 PM, Nick Carter  wrote:
>>
>>> The bottom of the link says that "coverage information was generated
>>> Wednesday".  Your checkin was Thursday.  Would that have something to do
>>> with it?
>>>
>>>  - nick
>>>
>>> On Thu, Jul 16, 2009 at 12:43 PM, Andrew Scherkus >> > wrote:
>>>
 So I checked in a change that did some refactoring and added unit tests
 for two classes that were previously not tested:
 http://src.chromium.org/viewvc/chrome?view=rev&revision=20836

 According to the coverage graphs, that change actually brought both the
 source code and test code coverage *down*.  What's interesting is the
 coverage analysis reports my new tests aren't being executed at all:

 http://build.chromium.org/buildbot/coverage/linux-debug/20836/CHROMIUM/media/filters/index.html

 Now I have no clue who/what process is generating these numbers, but
 maybe it needs to be clobbered.

 Andrew



>>>
>>
>> >>
>>
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Confusing test coverage numbers

[John Grabowski]

For starters, thanks for caring about coverage

It is possible your checkin was timed poorly (didn't land in time for the
bot
start), but looking at the coverage scripts I don't see how that could happen.
I noticed you landed, then reverted, this CL the day before.  My best guess
is that such an action was enough info for croc to learn the filename
and "stick", but since you reverted it didn't compile or get run.

Let's wait one more run and see if things fix themselves.

Ping again if the problem continues.

jrg


On Thu, Jul 16, 2009 at 3:38 PM, Andrew Scherkus wrote:

> I think we're getting confused by time zones here.
> I checked in at Wednesday at 18:26 PST (UTC-7), and assuming the coverage
> bot is also in my time zone that'd mean these results were generated ~20
> minutes afterwards.
>
>
> On Thu, Jul 16, 2009 at 3:22 PM, Nick Carter  wrote:
>
>> The bottom of the link says that "coverage information was generated
>> Wednesday".  Your checkin was Thursday.  Would that have something to do
>> with it?
>>
>>  - nick
>>
>> On Thu, Jul 16, 2009 at 12:43 PM, Andrew Scherkus 
>> wrote:
>>
>>> So I checked in a change that did some refactoring and added unit tests
>>> for two classes that were previously not tested:
>>> http://src.chromium.org/viewvc/chrome?view=rev&revision=20836
>>>
>>> According to the coverage graphs, that change actually brought both the
>>> source code and test code coverage *down*.  What's interesting is the
>>> coverage analysis reports my new tests aren't being executed at all:
>>>
>>> http://build.chromium.org/buildbot/coverage/linux-debug/20836/CHROMIUM/media/filters/index.html
>>>
>>> Now I have no clue who/what process is generating these numbers, but
>>> maybe it needs to be clobbered.
>>>
>>> Andrew
>>>
>>>
>>>
>>
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Confusing test coverage numbers

[Andrew Scherkus]

I think we're getting confused by time zones here.
I checked in at Wednesday at 18:26 PST (UTC-7), and assuming the coverage
bot is also in my time zone that'd mean these results were generated ~20
minutes afterwards.

On Thu, Jul 16, 2009 at 3:22 PM, Nick Carter  wrote:

> The bottom of the link says that "coverage information was generated
> Wednesday".  Your checkin was Thursday.  Would that have something to do
> with it?
>
>  - nick
>
> On Thu, Jul 16, 2009 at 12:43 PM, Andrew Scherkus 
> wrote:
>
>> So I checked in a change that did some refactoring and added unit tests
>> for two classes that were previously not tested:
>> http://src.chromium.org/viewvc/chrome?view=rev&revision=20836
>>
>> According to the coverage graphs, that change actually brought both the
>> source code and test code coverage *down*.  What's interesting is the
>> coverage analysis reports my new tests aren't being executed at all:
>>
>> http://build.chromium.org/buildbot/coverage/linux-debug/20836/CHROMIUM/media/filters/index.html
>>
>> Now I have no clue who/what process is generating these numbers, but maybe
>> it needs to be clobbered.
>>
>> Andrew
>>
>> >>
>>
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Confusing test coverage numbers

[Nick Carter]

The bottom of the link says that "coverage information was generated
Wednesday".  Your checkin was Thursday.  Would that have something to do
with it?

 - nick

On Thu, Jul 16, 2009 at 12:43 PM, Andrew Scherkus wrote:

> So I checked in a change that did some refactoring and added unit tests for
> two classes that were previously not tested:
> http://src.chromium.org/viewvc/chrome?view=rev&revision=20836
>
> According to the coverage graphs, that change actually brought both the
> source code and test code coverage *down*.  What's interesting is the
> coverage analysis reports my new tests aren't being executed at all:
>
> http://build.chromium.org/buildbot/coverage/linux-debug/20836/CHROMIUM/media/filters/index.html
>
> Now I have no clue who/what process is generating these numbers, but maybe
> it needs to be clobbered.
>
> Andrew
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---