CodeCoverage Monthly Update

2017-10-05 Thread Kyle Lahnakoski 

*# Q3 Codecoverage! update *

/If you //want to hear more about an//y of the//s//e items, please
contact me and I will get you more detailed information/*
*

*## Overview *

We can show the code coverage on new lines added by changeset!

  * *Link* - http://firefox-code-coverage.herokuapp.com/
  * *Warning* - This is alpha-quality; it is a work-in-progress, there
are still bugs, and it is not as responsive as we would like
  * *Visual walk through* -

https://github.com/armenzg/firefox-code-coverage-frontend/raw/7b67f2e948de94b7f837ba9a7c57a31e810d9fe7/docs/Prototype%20Instructions.pdf

Of course, we still have the regular coverage that we upload to
codecov.io [4] https://codecov.io/gh/marco-c/gecko-dev

*## Parts you don't see*

  * *Keep the tests passing* - Code coverage builds run a little
different, so we must constantly monitor and triage the unique test
failures.
  * *Building the backend for the UI* - The UI is supported by a backend
service that maps coverage to the changesets
  * *optimize jsvm lcov rewriter* - because everything is slow at our
scale of coverage
  * *merge coverage from test chunks* - because codecov.io can not
handle our raw data volume
  * *adding function info to grcov *- a different resolution of coverage
that is consistent with other coverage tools
  * *Rust LLVM build*  - We require a special build of Rust in order to
collect Rust coverage;

https://treeherder.mozilla.org/#/jobs?repo=try=014e361716000d0dee4427bf7195316a2aba7b21

*## Future Plans*

  * *Verify this UI prototype* - We want to ensure RelMan can use this
to inform their bug risk assessment
  * **Improve the UI* - *There are inevitable changes to be made when we
get our feedback from RelMan, plus the known bugs and issues we need
to work on.
  *

  * *Coverage for Windows with Clang* -
https://bugzilla.mozilla.org/show_bug.cgi?id=1381163
  * *Solve bugs* - Of course, there is a long tail of bugs 
  * *Show what tests cover a file* - Experimental - a couple UCOSP
students will attempt to show coverage at the per-test level; this
should prove useful to those that made a code change, and would like
to know what test suite (or test chunk) covers it.  If useful, then
we can use this same information to inform our automation.
  * *Merge coverage from different revisions* - Experimental - a UCOSP
student will attempt a pan-temporal representation of our source
code: If done, it will allow us transport code coverage forward, and
backward, through time and revisions.  This can be used to compare
coverage runs, merge runs from different revisions, and allow us to
collect coverage at a higher resolution with lower data storage costs.

*## Meetings*

We have weekly CodeCoverage meetings, and you are welcome to attend:

  * *When* - Held every Friday @ 11:30 EDT (08:30 PDT)
  * *Where* - Kyle's video room
https://v.mozilla.com/flex.html?roomdirect.html=huhL8WaTwCwC
  * *Etherpad* - https://public.etherpad-mozilla.org/p/code_coverage_Q1_17

*## Links*

[1] The frontend UI http://firefox-code-coverage.herokuapp.com/

[2] Frontend code https://github.com/armenzg/code_cov_experiments

[3] Backend server
https://github.com/mozilla-releng/services/tree/master/src/shipit_uplift/shipit_uplift

[4] Total Coverage https://codecov.io/gh/marco-c/gecko-dev

[5] UCOSP - http://ucosp.ca/
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: CodeCoverage! Monthly Update

2017-08-10 Thread Sylvestre Ledru 
The patch providing coverage in rust landed here:
https://github.com/rust-lang/rust/pull/42433

If I understood correctly, we are waiting for rust 1.20 to be able to
use it.

Sylvestre




Le 10/08/2017 à 22:03, Chris Peterson a écrit :
> Kyle, do you know if Rust code coverage is blocked on any remaining
> Rust toolchain issues?
>
> chris
>
>
> On 2017-08-10 11:31 AM, Kyle Lahnakoski wrote:
>>
>>
>> Did you have that sense you were missing something? Well, you were
>> right: You were missing your ...
>>
>> # *Monthly CodeCoverage! update!  \o/ *
>>
>> /If you //want to hear more about an//y of the//s//e items, please
>> contact me and I will get you more detailed information/*
>> *
>>
>> *## **Summary of July*
>>
>>   * *More hard work on the ETL pipeline*: Condensing the data to make
>> it manageable, and writing more efficient code for faster
>> processing. There is still more work to be done, but it's
>> working.  Marco summarized this work with an excellent blog post:
>> [1]
>> https://marco-c.github.io/2017/07/28/code-coverage-architecture.html
>>   * *Analyzing coverage variability* - If you recall from previous
>> months, I mentioned that different coverage runs show different
>> coverage numbers: We call this "coverage variability", and it
>> even exists when comparing two runs from the same revision.
>> gmierz has been looking into this variability to better
>> understand its size, and character. [2], [3]
>>   * *Finished detailed the plan for the MVP *[4] - This MVP is for
>> visualizing coverage of net-new lines: Relman is interested in
>> the coverage on the net-new lines of every changeset  This is
>> hard given we can only run coverage on every central push. We now
>> have a plan for how to get this done, as best as possible, with
>> the information given.
>>
>> *## Plans for August*
>>
>>   * *Build the MVP* - Visualize coverage of net new lines by
>> changeset: Now that we have a plan,  armenzg has already started
>> the frontend UI work [5], [6], and will be working with marco
>> working on the backend [7]
>>   * *Continue work on optimizing the ETL pipelines* - necessary work *
>> *
>>
>> *
>> **## Meetings*
>>
>> We have weekly CodeCoverage meetings, and you are welcome to attend:
>>
>>   * When: Held every Friday @ 11:30 EDT (08:30 PDT)
>>   * Where: Kyle's video room
>> https://v.mozilla.com/flex.html?roomdirect.html=huhL8WaTwCwC
>>   * Etherpad: https://public.etherpad-mozilla.org/p/code_coverage_Q1_17
>>
>>
>> *## Reference*
>>
>> [1] Blog post on coverage collection and aggregation:
>> https://marco-c.github.io/2017/07/28/code-coverage-architecture.html
>>
>> [2] Variability analysis code - 
>> https://github.com/gmierz/coco-variability-tools
>>
>> [3] Example variability output -
>> https://gmierz.github.io/variability/variability_index.html
>>
>> [4] Details for UI -
>> https://public.etherpad-mozilla.org/p/code_coverage_Q3_17
>>
>> [5] Code for UI - https://github.com/armenzg/code_cov_experiments
>>
>> [6] Example UI (very rough, but stay tuned!) -
>> https://armenzg.github.io/code_cov_experiments/
>>
>> [7] Code for backend -
>> https://github.com/mozilla-releng/services/tree/master/src/shipit_code_coverage
>>
>>
>>
>>
>> On 2017-07-06 21:37, Kyle Lahnakoski wrote:
>>>
>>>
>>>
>>> ## Summary of Past Quarter
>>>
>>> Coverage is enabled for nearly all tests, and scheduled every push
>>> [1]!!
>>>
>>>   * All c/c++ test suites have coverage enabled
>>>   * talos coverage is enabled
>>>   * jsvm[7] coverage is enabled, and running
>>>   * codecov.io [2] shows the results, broken down by directory
>>>
>>> ## Plans for Q3
>>>
>>> The overall plan for Q3 is laid out in the planning document [12]. 
>>> Put simply, a **coverage differential viewer**, operating at low
>>> resolution (per central push), has enough promise to justify Q3
>>> effort on CodeCoverage.
>>>
>>> ## The Complications
>>>
>>>   * Rust code coverage is still delayed [6] - maybe by mid quarter
>>>   * The data volume is very large; coveralls.io and codecov.io are
>>> having some difficulty dealing with the volume.
>>>   * There is instability in the coverage numbers due to variability
>>> in our test runs; think GC and telemetry logic.  Multiple
>>> coverage runs will be required to get a total coverage number
>>>   * Intermittents are impacting coverage reliability - we will
>>> require a coverage health monitor to know if coverage is "complete"
>>>
>>> ## Summary of this past June
>>>
>>>   * upgrading tests to use Ubuntu 16.04
>>>   * fixing blockers that stood in the way of rust coverage[6]
>>>   * enabling coverage to even more suites, like talos [10]
>>>   * adding JavaScript to the codecov/coveralls report
>>>   * getting a handle on the volume of data code coverage is producing
>>>
>>> ## Plans for July
>>>
>>>   * continued work on ETL pipeline
>>>   * enable coverage for spidermonkey [11]
>>>   * see the first

Re: CodeCoverage! Monthly Update

2017-08-10 Thread Chris Peterson 
Kyle, do you know if Rust code coverage is blocked on any remaining Rust 
toolchain issues?


chris


On 2017-08-10 11:31 AM, Kyle Lahnakoski wrote:



Did you have that sense you were missing something? Well, you were 
right: You were missing your ...


# *Monthly CodeCoverage! update!  \o/ *

/If you //want to hear more about an//y of the//s//e items, please 
contact me and I will get you more detailed information/*

*

*## **Summary of July*

  * *More hard work on the ETL pipeline*: Condensing the data to make
it manageable, and writing more efficient code for faster
processing. There is still more work to be done, but it's
working.  Marco summarized this work with an excellent blog post:
[1]
https://marco-c.github.io/2017/07/28/code-coverage-architecture.html
  * *Analyzing coverage variability* - If you recall from previous
months, I mentioned that different coverage runs show different
coverage numbers: We call this "coverage variability", and it even
exists when comparing two runs from the same revision. gmierz has
been looking into this variability to better understand its size,
and character. [2], [3]
  * *Finished detailed the plan for the MVP *[4] - This MVP is for
visualizing coverage of net-new lines: Relman is interested in the
coverage on the net-new lines of every changeset  This is hard
given we can only run coverage on every central push. We now have
a plan for how to get this done, as best as possible, with the
information given.

*## Plans for August*

  * *Build the MVP* - Visualize coverage of net new lines by
changeset: Now that we have a plan,  armenzg has already started
the frontend UI work [5], [6], and will be working with marco
working on the backend [7]
  * *Continue work on optimizing the ETL pipelines* - necessary work *
*

*
**## Meetings*

We have weekly CodeCoverage meetings, and you are welcome to attend:

  * When: Held every Friday @ 11:30 EDT (08:30 PDT)
  * Where: Kyle's video room
https://v.mozilla.com/flex.html?roomdirect.html=huhL8WaTwCwC
  * Etherpad: https://public.etherpad-mozilla.org/p/code_coverage_Q1_17


*## Reference*

[1] Blog post on coverage collection and aggregation: 
https://marco-c.github.io/2017/07/28/code-coverage-architecture.html


[2] Variability analysis code - 
https://github.com/gmierz/coco-variability-tools


[3] Example variability output - 
https://gmierz.github.io/variability/variability_index.html


[4] Details for UI - 
https://public.etherpad-mozilla.org/p/code_coverage_Q3_17


[5] Code for UI - https://github.com/armenzg/code_cov_experiments

[6] Example UI (very rough, but stay tuned!) - 
https://armenzg.github.io/code_cov_experiments/


[7] Code for backend - 
https://github.com/mozilla-releng/services/tree/master/src/shipit_code_coverage





On 2017-07-06 21:37, Kyle Lahnakoski wrote:




## Summary of Past Quarter

Coverage is enabled for nearly all tests, and scheduled every push [1]!!

  * All c/c++ test suites have coverage enabled
  * talos coverage is enabled
  * jsvm[7] coverage is enabled, and running
  * codecov.io [2] shows the results, broken down by directory

## Plans for Q3

The overall plan for Q3 is laid out in the planning document [12].  
Put simply, a **coverage differential viewer**, operating at low 
resolution (per central push), has enough promise to justify Q3 
effort on CodeCoverage.


## The Complications

  * Rust code coverage is still delayed [6] - maybe by mid quarter
  * The data volume is very large; coveralls.io and codecov.io are
having some difficulty dealing with the volume.
  * There is instability in the coverage numbers due to variability
in our test runs; think GC and telemetry logic.  Multiple
coverage runs will be required to get a total coverage number
  * Intermittents are impacting coverage reliability - we will
require a coverage health monitor to know if coverage is "complete"

## Summary of this past June

  * upgrading tests to use Ubuntu 16.04
  * fixing blockers that stood in the way of rust coverage[6]
  * enabling coverage to even more suites, like talos [10]
  * adding JavaScript to the codecov/coveralls report
  * getting a handle on the volume of data code coverage is producing

## Plans for July

  * continued work on ETL pipeline
  * enable coverage for spidermonkey [11]
  * see the first hints of Rust coverage
  * build a coverage health monitor to deal with "the complications"
(above)

## Meetings

We have weekly CodeCoverage meetings, and you are welcome to attend:

  * When: Held every Friday @ 11:30 EDT (08:30 PDT)
  * Where: Kyle's video room
https://v.mozilla.com/flex.html?roomdirect.html=huhL8WaTwCwC
  * Etherpad: https://public.etherpad-mozilla.org/p/code_coverage_Q1_17


## Reference

[1] See coverage on TH 
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central=ccov


[1b] Example on TH:

CodeCoverage! Monthly Update

2017-08-10 Thread Kyle Lahnakoski 

Did you have that sense you were missing something? Well, you were
right: You were missing your ...

# *Monthly CodeCoverage! update!  \o/ *

/If you //want to hear more about an//y of the//s//e items, please
contact me and I will get you more detailed information/*
*

*## **Summary of July*

  * *More hard work on the ETL pipeline*: Condensing the data to make it
manageable, and writing more efficient code for faster processing.
There is still more work to be done, but it's working.  Marco
summarized this work with an excellent blog post: [1]
https://marco-c.github.io/2017/07/28/code-coverage-architecture.html
  * *Analyzing coverage variability* - If you recall from previous
months, I mentioned that different coverage runs show different
coverage numbers: We call this "coverage variability", and it even
exists when comparing two runs from the same revision. gmierz has
been looking into this variability to better understand its size,
and character. [2], [3]
  * *Finished detailed the plan for the MVP *[4] - This MVP is for
visualizing coverage of net-new lines: Relman is interested in the
coverage on the net-new lines of every changeset  This is hard given
we can only run coverage on every central push. We now have a plan
for how to get this done, as best as possible, with the information
given.

*## Plans for August*

  * *Build the MVP* - Visualize coverage of net new lines by changeset:
Now that we have a plan,  armenzg has already started the frontend
UI work [5], [6], and will be working with marco working on the
backend [7]
  * *Continue work on optimizing the ETL pipelines* - necessary work *
*

*
**## Meetings*

We have weekly CodeCoverage meetings, and you are welcome to attend:

  * When: Held every Friday @ 11:30 EDT (08:30 PDT)
  * Where: Kyle's video room
https://v.mozilla.com/flex.html?roomdirect.html=huhL8WaTwCwC
  * Etherpad: https://public.etherpad-mozilla.org/p/code_coverage_Q1_17


*## Reference*

[1] Blog post on coverage collection and aggregation:
https://marco-c.github.io/2017/07/28/code-coverage-architecture.html

[2] Variability analysis code - 
https://github.com/gmierz/coco-variability-tools

[3] Example variability output -
https://gmierz.github.io/variability/variability_index.html

[4] Details for UI -
https://public.etherpad-mozilla.org/p/code_coverage_Q3_17

[5] Code for UI - https://github.com/armenzg/code_cov_experiments

[6] Example UI (very rough, but stay tuned!) -
https://armenzg.github.io/code_cov_experiments/

[7] Code for backend -
https://github.com/mozilla-releng/services/tree/master/src/shipit_code_coverage




On 2017-07-06 21:37, Kyle Lahnakoski wrote:
>
>
>
> ## Summary of Past Quarter
>
> Coverage is enabled for nearly all tests, and scheduled every push [1]!!
>
>   * All c/c++ test suites have coverage enabled
>   * talos coverage is enabled
>   * jsvm[7] coverage is enabled, and running
>   * codecov.io [2] shows the results, broken down by directory
>
> ## Plans for Q3
>
> The overall plan for Q3 is laid out in the planning document [12]. 
> Put simply, a **coverage differential viewer**, operating at low
> resolution (per central push), has enough promise to justify Q3 effort
> on CodeCoverage.
>
> ## The Complications
>
>   * Rust code coverage is still delayed [6] - maybe by mid quarter
>   * The data volume is very large; coveralls.io and codecov.io are
> having some difficulty dealing with the volume.
>   * There is instability in the coverage numbers due to variability in
> our test runs; think GC and telemetry logic.  Multiple coverage
> runs will be required to get a total coverage number
>   * Intermittents are impacting coverage reliability - we will require
> a coverage health monitor to know if coverage is "complete"
>
> ## Summary of this past June
>
>   * upgrading tests to use Ubuntu 16.04
>   * fixing blockers that stood in the way of rust coverage[6]
>   * enabling coverage to even more suites, like talos [10]
>   * adding JavaScript to the codecov/coveralls report
>   * getting a handle on the volume of data code coverage is producing
>
> ## Plans for July
>
>   * continued work on ETL pipeline
>   * enable coverage for spidermonkey [11]
>   * see the first hints of Rust coverage
>   * build a coverage health monitor to deal with "the complications"
> (above)
>
> ## Meetings
>
> We have weekly CodeCoverage meetings, and you are welcome to attend:
>
>   * When: Held every Friday @ 11:30 EDT (08:30 PDT)
>   * Where: Kyle's video room
> https://v.mozilla.com/flex.html?roomdirect.html=huhL8WaTwCwC
>   * Etherpad: https://public.etherpad-mozilla.org/p/code_coverage_Q1_17
>
>
> ## Reference
>
> [1] See coverage on TH
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-central=ccov
>
> [1b] Example on TH:
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-central=433c379d6e448dc64df25a6b4d8739c99e84d102=ccov
>
>
> [2] codecov.io

CodeCoverage Monthly Update

2017-07-06 Thread Kyle Lahnakoski 


## Summary of Past Quarter

Coverage is enabled for nearly all tests, and scheduled every push [1]!!

  * All c/c++ test suites have coverage enabled
  * talos coverage is enabled
  * jsvm[7] coverage is enabled, and running
  * codecov.io [2] shows the results, broken down by directory

## Plans for Q3

The overall plan for Q3 is laid out in the planning document [12].  Put
simply, a **coverage differential viewer**, operating at low resolution
(per central push), has enough promise to justify Q3 effort on CodeCoverage.

## The Complications

  * Rust code coverage is still delayed [6] - maybe by mid quarter
  * The data volume is very large; coveralls.io and codecov.io are
having some difficulty dealing with the volume.
  * There is instability in the coverage numbers due to variability in
our test runs; think GC and telemetry logic.  Multiple coverage runs
will be required to get a total coverage number
  * Intermittents are impacting coverage reliability - we will require a
coverage health monitor to know if coverage is "complete"

## Summary of this past June

  * upgrading tests to use Ubuntu 16.04
  * fixing blockers that stood in the way of rust coverage[6]
  * enabling coverage to even more suites, like talos [10]
  * adding JavaScript to the codecov/coveralls report
  * getting a handle on the volume of data code coverage is producing

## Plans for July

  * continued work on ETL pipeline
  * enable coverage for spidermonkey [11]
  * see the first hints of Rust coverage
  * build a coverage health monitor to deal with "the complications" (above)

## Meetings

We have weekly CodeCoverage meetings, and you are welcome to attend:

  * When: Held every Friday @ 11:30 EDT (08:30 PDT)
  * Where: Kyle's video room
https://v.mozilla.com/flex.html?roomdirect.html=huhL8WaTwCwC
  * Etherpad: https://public.etherpad-mozilla.org/p/code_coverage_Q1_17


## Reference

[1] See coverage on TH
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central=ccov

[1b] Example on TH:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central=433c379d6e448dc64df25a6b4d8739c99e84d102=ccov


[2] codecov.io https://codecov.io/gh/marco-c/gecko-dev

[3] Local Coverage
https://developer.mozilla.org/en-US/docs/Mozilla/Testing/Measuring_Code_Coverage_on_Firefox?document_saved=true#Generate_Code_Coverage_report_from_a_try_build_(or_any_other_treeherder_build
)

[4] iOS Coverage
https://codecov.io/gh/mozilla-mobile/firefox-ios/branch/master

[5] User Cases
https://docs.google.com/document/d/1JUEPS8Xdtx4y8fXA4Au_Ggme0fMyJQxZbby0X2e4yDI/edit#heading=h.ddwyjkvxus4-


[6] Rust coverage https://bugzilla.mozilla.org/show_bug.cgi?id=1335518
and https://github.com/rust-lang/rust/pull/42433

[7] JSVM coverage: https://bugzilla.mozilla.org/show_bug.cgi?id=1301174

[8] e10s coverage example:
https://treeherder.mozilla.org/#/jobs?repo=try=b6e9cefe95adc3dd281bf8e2a2f897e8f4839e51

[9] Everything:
https://bugzilla.mozilla.org/showdependencytree.cgi?id=1278393

[10] talos coverage: https://bugzilla.mozilla.org/show_bug.cgi?id=1372324

[11] spidermonkey coverage:
https://bugzilla.mozilla.org/show_bug.cgi?id=1362013


[12] Planning document:
https://docs.google.com/document/d/1dOWi18qrudwaOThNAYoCMS3e9LzhxGUiMLLrQ_WVR9w/edit?usp=sharing






___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: CodeCoverage Monthly Update (Rust)

2017-06-10 Thread Kyle Lahnakoski 

A couple of nice people noticed my link to the Rust coverage was wrong:

Here is the main tracking bug:
https://bugzilla.mozilla.org/show_bug.cgi?id=1335518

Plus Github issues tracking the specific work:
https://github.com/rust-lang/rust/pull/38608
https://github.com/rust-lang/rust/pull/42433


On 2017-06-09 15:14, Kyle Lahnakoski wrote:
>
> ## Summary of the past month
>
> Coverage is enabled for nearly all tests, and scheduled every night [1]!!
>
>   * All c/c++ test suites have coverage enabled and are running nightly
>   * jsvm[7] coverage is enabled, and running
>   * codecov.io [2] shows the results, broken down by directory
>
> ## Complications
>
>   * Rust code coverage is still delayed [6]
>   * There is instability in the coverage numbers due to variability in
> our test runs. This is minor compared to the large volume of
> coverage we are collecting, but it may impact discriminating analysis.
>
> ## Plans for remaining Quarter
>
>   * Release management requires a view into how coverage is changing
> over time, and relate that change to changesets and bugs. 
> mcastelluccio has a very rough prototype [10]
>   * Work on the long tail of test suite flavors, esoteric tests,
> intermittent tests, skipped tests, and operational details to keep
> this whole thing running smooth [9]
>   * Streamline the grcov ETL pipeline
>   * Explore what else we can do with this data.
>   * Put together a presentation deck, just in case someone is
> interested, for AllHands.
>
> ## Meetings
>
> We have weekly CodeCoverage meetings, and you are welcome to attend:
>
>   * When: Held every Friday @ 11:30 EDT (08:30 PDT)
>   * Where: Kyle's video room
> https://v.mozilla.com/flex.html?roomdirect.html=huhL8WaTwCwC
>   * Etherpad: https://public.etherpad-mozilla.org/p/code_coverage_Q1_17
>
>
> ## Reference
>
> [1] See coverage on TH
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-central=ccov
>
> [1b] Example on TH:
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-central=e61060be36424240058f8bef4c5597f401bc8b7e=ccov
>
> [2] codecov.io https://codecov.io/gh/marco-c/gecko-dev
>
> [3] Local Coverage
> https://developer.mozilla.org/en-US/docs/Mozilla/Testing/Measuring_Code_Coverage_on_Firefox?document_saved=true#Generate_Code_Coverage_report_from_a_try_build_(or_any_other_treeherder_build
> )
>
> [4] iOS Coverage
> https://codecov.io/gh/mozilla-mobile/firefox-ios/branch/master
>
> [5] User Cases
> https://docs.google.com/document/d/1JUEPS8Xdtx4y8fXA4Au_Ggme0fMyJQxZbby0X2e4yDI/edit#heading=h.ddwyjkvxus4-
>
>
> [6] Rust coverage
> https://docs.google.com/document/d/1JUEPS8Xdtx4y8fXA4Au_Ggme0fMyJQxZbby0X2e4yDI/edit#heading=h.ddwyjkvxus4
>
> [7] JSVM coverage: https://bugzilla.mozilla.org/show_bug.cgi?id=1301174
>
> [8] e10s coverage example:
> https://treeherder.mozilla.org/#/jobs?repo=try=b6e9cefe95adc3dd281bf8e2a2f897e8f4839e51
>
> [9] Everything:
> https://bugzilla.mozilla.org/showdependencytree.cgi?id=1278393
>
> [10] Show coverage differences, with bug numbers:
> https://marco-c.github.io/grcov-test/coverage_by_dir.html
>

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

CodeCoverage Monthly Update

2017-06-09 Thread Kyle Lahnakoski 

## Summary of the past month

Coverage is enabled for nearly all tests, and scheduled every night [1]!!

  * All c/c++ test suites have coverage enabled and are running nightly
  * jsvm[7] coverage is enabled, and running
  * codecov.io [2] shows the results, broken down by directory

## Complications

  * Rust code coverage is still delayed [6]
  * There is instability in the coverage numbers due to variability in
our test runs. This is minor compared to the large volume of
coverage we are collecting, but it may impact discriminating analysis.

## Plans for remaining Quarter

  * Release management requires a view into how coverage is changing
over time, and relate that change to changesets and bugs. 
mcastelluccio has a very rough prototype [10]
  * Work on the long tail of test suite flavors, esoteric tests,
intermittent tests, skipped tests, and operational details to keep
this whole thing running smooth [9]
  * Streamline the grcov ETL pipeline
  * Explore what else we can do with this data.
  * Put together a presentation deck, just in case someone is
interested, for AllHands.

## Meetings

We have weekly CodeCoverage meetings, and you are welcome to attend:

  * When: Held every Friday @ 11:30 EDT (08:30 PDT)
  * Where: Kyle's video room
https://v.mozilla.com/flex.html?roomdirect.html=huhL8WaTwCwC
  * Etherpad: https://public.etherpad-mozilla.org/p/code_coverage_Q1_17


## Reference

[1] See coverage on TH
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central=ccov

[1b] Example on TH:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central=e61060be36424240058f8bef4c5597f401bc8b7e=ccov

[2] codecov.io https://codecov.io/gh/marco-c/gecko-dev

[3] Local Coverage
https://developer.mozilla.org/en-US/docs/Mozilla/Testing/Measuring_Code_Coverage_on_Firefox?document_saved=true#Generate_Code_Coverage_report_from_a_try_build_(or_any_other_treeherder_build
)

[4] iOS Coverage
https://codecov.io/gh/mozilla-mobile/firefox-ios/branch/master

[5] User Cases
https://docs.google.com/document/d/1JUEPS8Xdtx4y8fXA4Au_Ggme0fMyJQxZbby0X2e4yDI/edit#heading=h.ddwyjkvxus4-


[6] Rust coverage
https://docs.google.com/document/d/1JUEPS8Xdtx4y8fXA4Au_Ggme0fMyJQxZbby0X2e4yDI/edit#heading=h.ddwyjkvxus4

[7] JSVM coverage: https://bugzilla.mozilla.org/show_bug.cgi?id=1301174

[8] e10s coverage example:
https://treeherder.mozilla.org/#/jobs?repo=try=b6e9cefe95adc3dd281bf8e2a2f897e8f4839e51

[9] Everything:
https://bugzilla.mozilla.org/showdependencytree.cgi?id=1278393

[10] Show coverage differences, with bug numbers:
https://marco-c.github.io/grcov-test/coverage_by_dir.html

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: CodeCoverage Monthly Update

2017-05-09 Thread Ted Mielczarek 
On Tue, May 9, 2017, at 05:48 AM, Henri Sivonen wrote:
> On Thu, Apr 6, 2017 at 6:26 AM, Kyle Lahnakoski 
> wrote:
> > * Getting Rust to emit coverage artifacts is important:
> > https://bugzilla.mozilla.org/show_bug.cgi?id=1335518
> 
> Is there a plan to factor "cargo test" of individual vendored crates
> into the coverage of Rust code? For example, for encoding_rs, I was
> thinking of testing mainly the C++ integration as an
> mozilla-central-specific gtest and leaving the testing of the crate
> internals to the crate's standalone "cargo test".

Note that we're not currently running `cargo test` tests for anything:
https://bugzilla.mozilla.org/show_bug.cgi?id=1331022

-Ted
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: CodeCoverage Monthly Update

2017-05-09 Thread Henri Sivonen 
On Thu, Apr 6, 2017 at 6:26 AM, Kyle Lahnakoski  wrote:
> * Getting Rust to emit coverage artifacts is important:
> https://bugzilla.mozilla.org/show_bug.cgi?id=1335518

Is there a plan to factor "cargo test" of individual vendored crates
into the coverage of Rust code? For example, for encoding_rs, I was
thinking of testing mainly the C++ integration as an
mozilla-central-specific gtest and leaving the testing of the crate
internals to the crate's standalone "cargo test".

-- 
Henri Sivonen
hsivo...@hsivonen.fi
https://hsivonen.fi/
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Re: CodeCoverage Monthly Update

2017-05-08 Thread Marco Castelluccio 
No, currently not, but we can build one ourselves using their API (which 
we might to do with codecov.io anyway, since we expect to have custom 
needs).


- Marco.


Il 04/05/17 08:22, Chris Peterson ha scritto:

On 2017-05-03 8:44 PM, Kyle Lahnakoski wrote:

  * Daily coverage reports on coveralls.io [1] and on codecov.io[2].
Which do you like?


Does coveralls.io have a top-down coverage view like codecov.io? That 
view seems more useful for both people that want a global view and 
developers that want to drill down into just their component directories.




___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: CodeCoverage Monthly Update

2017-05-04 Thread Chris Peterson 

On 2017-05-03 8:44 PM, Kyle Lahnakoski wrote:

  * Daily coverage reports on coveralls.io [1] and on codecov.io[2].
Which do you like?


Does coveralls.io have a top-down coverage view like codecov.io? That 
view seems more useful for both people that want a global view and 
developers that want to drill down into just their component directories.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

CodeCoverage Monthly Update

2017-05-03 Thread Kyle Lahnakoski 

It is the first Wednesday for the month!  Time to write an update!


## Summary of Activity

  * Daily coverage reports on coveralls.io [1] and on codecov.io[2].
Which do you like?
  * You can now use one-click loaners to generate coverage locally
(thanks maja and mcastelluccio!)
  * We met the nice people working on coverage for iOS [4] (thanks isabel!)
  * e10s coverage builds are working! (thanks mcastelluccio!) [8]
  * Listed the use cases we will be solving for in the short term, and
the long term [5]

## Some bad news

  * Rust code coverage will probably not be available until July, or
later.  Work is progressing [6], but the release schedule, CI
integration, and the unknowns, will add delay.
  * JSVM, despite it is working[7], has not been verified and scheduled

## Future Plans

  * e10s coverage will be compared to ensure there are no multiprocess
anomallies
  * More test suites will be enabled for nightly scheduled coverage
  * JSVM coverage work will resume
  * Start identifying the gaps between current visualizations and
Release Management's expectations so we have priority worklist for
end of the quarter

## Meetings

We have weekly CodeCoverage meetings, and you are welcome to attend:

  * When: Held every Friday @ 11:30 EDT (08:30 PDT)
  * Where: Kyle's video room
https://v.mozilla.com/flex.html?roomdirect.html=huhL8WaTwCwC
  * Etherpad: https://public.etherpad-mozilla.org/p/code_coverage_Q1_17


## Reference

[1] coveralls.io https://coveralls.io/github/marco-c/gecko-dev

[2] codecov.io https://codecov.io/gh/marco-c/gecko-dev

[3] Local Coverage
https://developer.mozilla.org/en-US/docs/Mozilla/Testing/Measuring_Code_Coverage_on_Firefox?document_saved=true#Generate_Code_Coverage_report_from_a_try_build_(or_any_other_treeherder_build
)

[4] iOS Coverage
https://codecov.io/gh/mozilla-mobile/firefox-ios/branch/master

[5] User Cases
https://docs.google.com/document/d/1JUEPS8Xdtx4y8fXA4Au_Ggme0fMyJQxZbby0X2e4yDI/edit#heading=h.ddwyjkvxus4-


[6] Rust coverage
https://docs.google.com/document/d/1JUEPS8Xdtx4y8fXA4Au_Ggme0fMyJQxZbby0X2e4yDI/edit#heading=h.ddwyjkvxus4

[7] JSVM coverage: https://bugzilla.mozilla.org/show_bug.cgi?id=1301174

[8] e10s coverage example:
https://treeherder.mozilla.org/#/jobs?repo=try=b6e9cefe95adc3dd281bf8e2a2f897e8f4839e51

[9] Everything:
https://bugzilla.mozilla.org/showdependencytree.cgi?id=1278393

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

CodeCoverage Monthly Update

2017-04-05 Thread Kyle Lahnakoski 

This is the second monthly update on our CodeCoverage effort:

## The Progress to Date

* Coverage is scheduled to run daily on central.
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central=cov
* The UCOSP[1] students have completed their term, and can show
aggregate statistics by folder with the code coverage ui [5]
https://ericdesj.github.io/moz-coco-w17-preview/
* c/c++ coverage is being ETL'ed into our unified coverage datastore,
but does not yet show in the UI
* `grcov`[4] was written mcastelluccio to process coverage artifacts
much faster; reducing processing time from hours to
seconds. `grcov` also outputs multiple formats, including for coveralls.io
* There is exploration into using coveralls.io:
https://coveralls.io/jobs/24085087, the main page is
https://coveralls.io/github/marco-c/gecko-dev.
* JSVM coverage got working today!
https://treeherder.mozilla.org/#/jobs?repo=try=ce2c56eeceb1bc252c6b25a50304f725c320d5e0


## The Blockers

These items are further behind than I would like.  If you can help,
please do.

* Getting Rust to emit coverage artifacts is important:
https://bugzilla.mozilla.org/show_bug.cgi?id=1335518
* Continuing the hard work of enabling C/C++ codecoverage on all our
tests: https://bugzilla.mozilla.org/show_bug.cgi?id=1301170

## The Biggest Problem

**Our test suites are unstable**. Many of you may already know that many
tests fail intermittently; these "intermittents" impact code coverage;
either by breaking the test run, and preventing coverage collection; or
by changing code paths. The challenge is automating the metatdata
collection and having a strategy to mitigate the loss of data. [8]
https://bugzilla.mozilla.org/show_bug.cgi?id=1337241

Even so, **test runs are not deterministic**, so we expect fluctuation
in the aggregate coverage statistics; How large this fluctuation is we
do not know.

Got ideas?  You want to work on a problem that no one will thank you
for?  ;)  Please talk to me! [3]

## The Current Plan

For the next month we will be continuing work on:

* validating the data and process, including adding Rust and more test
suites.
* Adding per-line coverage to the CoCo UI: The aggregate numbers are
good for identifying sections of code that are not tested, but we need
line-level details to inform action.
https://github.com/mozilla/moz-coco/issues/7
* Explore how coveralls.io can be used to make that job easier  
* Explore how to leverage Taskcluster-generated coverage artifacts with
local coverage tools [9]
https://bugzilla.mozilla.org/show_bug.cgi?id=1350446
* We are investigating if coverage tools behave differently when e10s is
enabled, then enabling it.
* JSVM ETL pipeline work will commence:
https://bugzilla.mozilla.org/show_bug.cgi?id=1301174


## Motivation

*Unchanged from last email* - Knowing code coverage statistics and code
coverage specifics can help evaluate risk in code: It can show what test
suites cover what lines, it can show if new code is covered by an
existing test.  There are many other exciting insights we can extract
from code coverage. But first, we must collect it.

## Reference

[1] UCOSP (Undergraduate Capstone Open Source Projects) http://ucosp.ca/

[2] The original plan:
https://docs.google.com/document/d/1dOWi18qrudwaOThNAYoCMS3e9LzhxGUiMLLrQ_WVR9w/edit#

[3] Kyle Lahnakoski email: klahnako...@mozilla.org  irc: ekyle on
#at...@irc.mozilla.org

[4] grcov https://github.com/marco-c/grcov

[5] (Co)de (Co)verage UI preview -
https://ericdesj.github.io/moz-coco-w17-preview/

[5b] CoCo is on Github - https://github.com/mozilla/moz-coco

[6] CoCo relay - https://github.com/ericdesj/moz-coco-relay

[7] Rust Coverage Tracking
https://bugzilla.mozilla.org/show_bug.cgi?id=1335518

[8] Knowing when the coverage is complete:
https://bugzilla.mozilla.org/show_bug.cgi?id=1337241

[9] Coverage with local tools:
https://bugzilla.mozilla.org/show_bug.cgi?id=1350446
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform