from:"Kyle Lahnakoski"

Code Coverage Summary

2018-06-22 Thread Kyle Lahnakoski

## Overview

  * **Coverage on recent changesets** - A list of recent changesets and
the percent of new lines covered by tests.
https://firefox-code-coverage.herokuapp.com
  * **Coverage addon** - Use it to show coverage on your patches in
Bugzilla -
https://addons.mozilla.org/en-US/firefox/addon/gecko-code-coverage/
  * **Daily coverage aggregates** - by directory and file -
https://codecov.io/gh/mozilla/gecko-dev
  * **Zero coverage reports** -
https://marco-c.github.io/code-coverage-reports/ with ore details here:
https://release.mozilla.org/tooling/codecoverage/2018/03/23/code-coverage.html
  * **Not-so-raw coverage artifacts** - If you are interested in the
details, you may want to see the artifacts produced by coverage builds:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central=ccov


## Contact Us

If you want to know more, or want to participate, stop by and say "hi":

  * **Mail list** - codecover...@mozilla.com
  * **IRC** - #codecoverage  


## Details 

This past 6 months have been spent working on minutia and hard problems
of the CodeCoverage system. There are many platforms, many suites, a
complex build, and a lot of data. Taming the complexity so we can
provide "clean" coverage artifacts is proving arduous. Here are some
examples:

 * **translating filenames** - Our build has generated source files, and
files copied to appropriate directories before compile. The file
reported in coverage artifacts must be mapped back to files we see in
the tree. Our coverage processing now does this work.
 * **chunk <-> file mapping** - Our coverage is collected by running
hundreds of tasks, each responsible for running some part of a test
suite; called a "chunk".  An interesting question is "if we have a
changeset, and we know what files it changed, can we schedule less
chunks?" The chunk<->file mapping allows use to answer that question.
Turns out the answer is "not really".
 * **windows coverage** - If you visit Treeherder you will see we are
collecting coverage on Windows (using clang-cl). This allows us to see
platform-specific lines.
 * **grcov improvements** - now faster than before, and can process llvm
coverage output
 * **jsdcov e10s** - We had to verify that coverage was getting
collected by all processes
 * **frontend refinements** - There has been continued work on the
frontend to meet Release Management's use cases.  Another cohort of
UCOSP students has helped fix bugs, and added functionality to the
frontend. 
 * **variability analysis** - Coverage at the Firefox-scale has
variability from many different sources. For instance, running the same
test on the same revision multiple times shows differences from one run
to the next. Plus, in some other cases, tests fail, which results in
different coverage. These situations were explored in more detail to
better understand what variability we are dealing with.
 * **jsdcov vs jsvm** - Both are collecting coverage on javascript; they
appear to be capturing different coverage; we must look into this more
 * **bugs!** - Many bugs, timing bugs in the tests, and bugs in the
coverage collection/reporting pipeline.
 * **android experiments** - Android will be be given more focus in the
coming year, so we should look at how coverage will help.
 * **backend improvements** - Work that must be done to deal with data
volume, but is barely visible to the end user. 
 * **backend moved to production** - CodeCoverage has moved to
production; and this took time.
 * **subtracting baselines** - We reduce our coverage artifact size by
subtracting no-test coverage from test coverage.  In theory, after
subtraction, the coverage represents what is uniquely used by the test
and removes all standard browser/test start and shutdown coverage.
However, we have found that coverage variability is affecting these
results as well and are now looking into mitigation strategies.
 * **per test coverage** - One of our goals is to collect coverage at
the test level; this can help guide what tests should be run when a file
changes. It may also help with understanding gaining a deeper
understanding of what our tests are testing.

## The Future

  * **move off codecov.io** - Our data volume and our use cases, do not
fit with codecov.io. We will use our own database to store coverage
artifacts, and enable us to do more. 
  * **per test coverage** - One of our goals is to collect coverage at
the test level; this can help guide what tests should be run when a file
changes. It might also help understanding which tests are important
(e.g. for evaluating whether to disable or fix intermittent tests)
  * **compare coverage across revisions** - We want to markup the
coverage so a reasonable comparison can be done between different
revisions. This should mitigate much of our coverage variability:
https://github.com/mozilla/TUID/blob/dev/docs/CodeCoverage%20TUID.md


## Meetings

We have weekly CodeCoverage meetings, and you are welcome to attend:

  * **When** - Held every Friday @ 11:30

CodeCoverage Monthly Update

2017-10-05 Thread Kyle Lahnakoski


*# Q3 Codecoverage! update *

/If you //want to hear more about an//y of the//s//e items, please
contact me and I will get you more detailed information/*
*

*## Overview *

We can show the code coverage on new lines added by changeset!

  * *Link* - http://firefox-code-coverage.herokuapp.com/
  * *Warning* - This is alpha-quality; it is a work-in-progress, there
are still bugs, and it is not as responsive as we would like
  * *Visual walk through* -

https://github.com/armenzg/firefox-code-coverage-frontend/raw/7b67f2e948de94b7f837ba9a7c57a31e810d9fe7/docs/Prototype%20Instructions.pdf

Of course, we still have the regular coverage that we upload to
codecov.io [4] https://codecov.io/gh/marco-c/gecko-dev

*## Parts you don't see*

  * *Keep the tests passing* - Code coverage builds run a little
different, so we must constantly monitor and triage the unique test
failures.
  * *Building the backend for the UI* - The UI is supported by a backend
service that maps coverage to the changesets
  * *optimize jsvm lcov rewriter* - because everything is slow at our
scale of coverage
  * *merge coverage from test chunks* - because codecov.io can not
handle our raw data volume
  * *adding function info to grcov *- a different resolution of coverage
that is consistent with other coverage tools
  * *Rust LLVM build*  - We require a special build of Rust in order to
collect Rust coverage;

https://treeherder.mozilla.org/#/jobs?repo=try=014e361716000d0dee4427bf7195316a2aba7b21

*## Future Plans*

  * *Verify this UI prototype* - We want to ensure RelMan can use this
to inform their bug risk assessment
  * **Improve the UI* - *There are inevitable changes to be made when we
get our feedback from RelMan, plus the known bugs and issues we need
to work on.
  *

  * *Coverage for Windows with Clang* -
https://bugzilla.mozilla.org/show_bug.cgi?id=1381163
  * *Solve bugs* - Of course, there is a long tail of bugs 
  * *Show what tests cover a file* - Experimental - a couple UCOSP
students will attempt to show coverage at the per-test level; this
should prove useful to those that made a code change, and would like
to know what test suite (or test chunk) covers it.  If useful, then
we can use this same information to inform our automation.
  * *Merge coverage from different revisions* - Experimental - a UCOSP
student will attempt a pan-temporal representation of our source
code: If done, it will allow us transport code coverage forward, and
backward, through time and revisions.  This can be used to compare
coverage runs, merge runs from different revisions, and allow us to
collect coverage at a higher resolution with lower data storage costs.

*## Meetings*

We have weekly CodeCoverage meetings, and you are welcome to attend:

  * *When* - Held every Friday @ 11:30 EDT (08:30 PDT)
  * *Where* - Kyle's video room
https://v.mozilla.com/flex.html?roomdirect.html=huhL8WaTwCwC
  * *Etherpad* - https://public.etherpad-mozilla.org/p/code_coverage_Q1_17

*## Links*

[1] The frontend UI http://firefox-code-coverage.herokuapp.com/

[2] Frontend code https://github.com/armenzg/code_cov_experiments

[3] Backend server
https://github.com/mozilla-releng/services/tree/master/src/shipit_uplift/shipit_uplift

[4] Total Coverage https://codecov.io/gh/marco-c/gecko-dev

[5] UCOSP - http://ucosp.ca/
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Reminder on Try usage and infrastructure resources

2017-09-14 Thread Kyle Lahnakoski


You can try ActiveData, which stores all test results from the past few
weeks.  Here is an example query that shows the chunk number for each
run/build combo in the past day.  ActiveData is sometimes more than a
day behind

https://activedata.allizom.org/tools/query.html#query_id=4HHuBgDu

{
    "from":"unittest",
    "select":[
        {"aggregate":"count"},
        {"value":"action.start_time","aggregate":"max"}
    ],
    "groupby":[
        "run.suite",
        "run.chunk",
        "result.test",
        "build.platform",
        "build.type",
        "run.type"
    ],
    "where":{"and":[
        {"eq":{"build.branch":"mozilla-inbound"}},
        {"prefix":{"run.suite":"moch"}},
        {"gt":{"action.start_time":{"date":"today-day"}}},
        {"regex":{"result.test":".*browser_623779.js.*"}}
    ]},
    "limit":1000
}



On 2017-09-14 11:49, Michael de Boer wrote:
>> On 14 Sep 2017, at 17:48, Marco Bonardo  wrote:
>>
>> When I need to retrigger a mochitest-browser test multiple times (to
>> investigate an intermittent), often I end up running all the
>> mochitest-browser tests, looking at every log until I find the chunk
>> where the test is, and retrigger just that chunk. The chunk number
>> changes based on the platform and debug/opt, so it's painful.
>> Is there a way to trigger only the chunk that will contain a given
>> test, so I can save running all of the other chunks?
> This! This! This! I’d love to be able to do this - would making testing 
> possible test failure fixes sooo much easier.
>
> Cheers,
>
> Mike.
>

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

CodeCoverage! Monthly Update

2017-08-10 Thread Kyle Lahnakoski


Did you have that sense you were missing something? Well, you were
right: You were missing your ...

# *Monthly CodeCoverage! update!  \o/ *

/If you //want to hear more about an//y of the//s//e items, please
contact me and I will get you more detailed information/*
*

*## **Summary of July*

  * *More hard work on the ETL pipeline*: Condensing the data to make it
manageable, and writing more efficient code for faster processing.
There is still more work to be done, but it's working.  Marco
summarized this work with an excellent blog post: [1]
https://marco-c.github.io/2017/07/28/code-coverage-architecture.html
  * *Analyzing coverage variability* - If you recall from previous
months, I mentioned that different coverage runs show different
coverage numbers: We call this "coverage variability", and it even
exists when comparing two runs from the same revision. gmierz has
been looking into this variability to better understand its size,
and character. [2], [3]
  * *Finished detailed the plan for the MVP *[4] - This MVP is for
visualizing coverage of net-new lines: Relman is interested in the
coverage on the net-new lines of every changeset  This is hard given
we can only run coverage on every central push. We now have a plan
for how to get this done, as best as possible, with the information
given.

*## Plans for August*

  * *Build the MVP* - Visualize coverage of net new lines by changeset:
Now that we have a plan,  armenzg has already started the frontend
UI work [5], [6], and will be working with marco working on the
backend [7]
  * *Continue work on optimizing the ETL pipelines* - necessary work *
*

*
**## Meetings*

We have weekly CodeCoverage meetings, and you are welcome to attend:

  * When: Held every Friday @ 11:30 EDT (08:30 PDT)
  * Where: Kyle's video room
https://v.mozilla.com/flex.html?roomdirect.html=huhL8WaTwCwC
  * Etherpad: https://public.etherpad-mozilla.org/p/code_coverage_Q1_17


*## Reference*

[1] Blog post on coverage collection and aggregation:
https://marco-c.github.io/2017/07/28/code-coverage-architecture.html

[2] Variability analysis code - 
https://github.com/gmierz/coco-variability-tools

[3] Example variability output -
https://gmierz.github.io/variability/variability_index.html

[4] Details for UI -
https://public.etherpad-mozilla.org/p/code_coverage_Q3_17

[5] Code for UI - https://github.com/armenzg/code_cov_experiments

[6] Example UI (very rough, but stay tuned!) -
https://armenzg.github.io/code_cov_experiments/

[7] Code for backend -
https://github.com/mozilla-releng/services/tree/master/src/shipit_code_coverage




On 2017-07-06 21:37, Kyle Lahnakoski wrote:
>
>
>
> ## Summary of Past Quarter
>
> Coverage is enabled for nearly all tests, and scheduled every push [1]!!
>
>   * All c/c++ test suites have coverage enabled
>   * talos coverage is enabled
>   * jsvm[7] coverage is enabled, and running
>   * codecov.io [2] shows the results, broken down by directory
>
> ## Plans for Q3
>
> The overall plan for Q3 is laid out in the planning document [12]. 
> Put simply, a **coverage differential viewer**, operating at low
> resolution (per central push), has enough promise to justify Q3 effort
> on CodeCoverage.
>
> ## The Complications
>
>   * Rust code coverage is still delayed [6] - maybe by mid quarter
>   * The data volume is very large; coveralls.io and codecov.io are
> having some difficulty dealing with the volume.
>   * There is instability in the coverage numbers due to variability in
> our test runs; think GC and telemetry logic.  Multiple coverage
> runs will be required to get a total coverage number
>   * Intermittents are impacting coverage reliability - we will require
> a coverage health monitor to know if coverage is "complete"
>
> ## Summary of this past June
>
>   * upgrading tests to use Ubuntu 16.04
>   * fixing blockers that stood in the way of rust coverage[6]
>   * enabling coverage to even more suites, like talos [10]
>   * adding JavaScript to the codecov/coveralls report
>   * getting a handle on the volume of data code coverage is producing
>
> ## Plans for July
>
>   * continued work on ETL pipeline
>   * enable coverage for spidermonkey [11]
>   * see the first hints of Rust coverage
>   * build a coverage health monitor to deal with "the complications"
> (above)
>
> ## Meetings
>
> We have weekly CodeCoverage meetings, and you are welcome to attend:
>
>   * When: Held every Friday @ 11:30 EDT (08:30 PDT)
>   * Where: Kyle's video room
> https://v.mozilla.com/flex.html?roomdirect.html=huhL8WaTwCwC
>   * Etherpad: https://public.etherpad-mozilla.org/p/code_coverage_Q1_17
>
>
> ## Reference
>
> [1] See coverage on TH
> https://tre

CodeCoverage Monthly Update

2017-07-06 Thread Kyle Lahnakoski



## Summary of Past Quarter

Coverage is enabled for nearly all tests, and scheduled every push [1]!!

  * All c/c++ test suites have coverage enabled
  * talos coverage is enabled
  * jsvm[7] coverage is enabled, and running
  * codecov.io [2] shows the results, broken down by directory

## Plans for Q3

The overall plan for Q3 is laid out in the planning document [12].  Put
simply, a **coverage differential viewer**, operating at low resolution
(per central push), has enough promise to justify Q3 effort on CodeCoverage.

## The Complications

  * Rust code coverage is still delayed [6] - maybe by mid quarter
  * The data volume is very large; coveralls.io and codecov.io are
having some difficulty dealing with the volume.
  * There is instability in the coverage numbers due to variability in
our test runs; think GC and telemetry logic.  Multiple coverage runs
will be required to get a total coverage number
  * Intermittents are impacting coverage reliability - we will require a
coverage health monitor to know if coverage is "complete"

## Summary of this past June

  * upgrading tests to use Ubuntu 16.04
  * fixing blockers that stood in the way of rust coverage[6]
  * enabling coverage to even more suites, like talos [10]
  * adding JavaScript to the codecov/coveralls report
  * getting a handle on the volume of data code coverage is producing

## Plans for July

  * continued work on ETL pipeline
  * enable coverage for spidermonkey [11]
  * see the first hints of Rust coverage
  * build a coverage health monitor to deal with "the complications" (above)

## Meetings

We have weekly CodeCoverage meetings, and you are welcome to attend:

  * When: Held every Friday @ 11:30 EDT (08:30 PDT)
  * Where: Kyle's video room
https://v.mozilla.com/flex.html?roomdirect.html=huhL8WaTwCwC
  * Etherpad: https://public.etherpad-mozilla.org/p/code_coverage_Q1_17


## Reference

[1] See coverage on TH
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central=ccov

[1b] Example on TH:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central=433c379d6e448dc64df25a6b4d8739c99e84d102=ccov


[2] codecov.io https://codecov.io/gh/marco-c/gecko-dev

[3] Local Coverage
https://developer.mozilla.org/en-US/docs/Mozilla/Testing/Measuring_Code_Coverage_on_Firefox?document_saved=true#Generate_Code_Coverage_report_from_a_try_build_(or_any_other_treeherder_build
)

[4] iOS Coverage
https://codecov.io/gh/mozilla-mobile/firefox-ios/branch/master

[5] User Cases
https://docs.google.com/document/d/1JUEPS8Xdtx4y8fXA4Au_Ggme0fMyJQxZbby0X2e4yDI/edit#heading=h.ddwyjkvxus4-


[6] Rust coverage https://bugzilla.mozilla.org/show_bug.cgi?id=1335518
and https://github.com/rust-lang/rust/pull/42433

[7] JSVM coverage: https://bugzilla.mozilla.org/show_bug.cgi?id=1301174

[8] e10s coverage example:
https://treeherder.mozilla.org/#/jobs?repo=try=b6e9cefe95adc3dd281bf8e2a2f897e8f4839e51

[9] Everything:
https://bugzilla.mozilla.org/showdependencytree.cgi?id=1278393

[10] talos coverage: https://bugzilla.mozilla.org/show_bug.cgi?id=1372324

[11] spidermonkey coverage:
https://bugzilla.mozilla.org/show_bug.cgi?id=1362013


[12] Planning document:
https://docs.google.com/document/d/1dOWi18qrudwaOThNAYoCMS3e9LzhxGUiMLLrQ_WVR9w/edit?usp=sharing






___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: CodeCoverage Monthly Update (Rust)

2017-06-10 Thread Kyle Lahnakoski


A couple of nice people noticed my link to the Rust coverage was wrong:

Here is the main tracking bug:
https://bugzilla.mozilla.org/show_bug.cgi?id=1335518

Plus Github issues tracking the specific work:
https://github.com/rust-lang/rust/pull/38608
https://github.com/rust-lang/rust/pull/42433


On 2017-06-09 15:14, Kyle Lahnakoski wrote:
>
> ## Summary of the past month
>
> Coverage is enabled for nearly all tests, and scheduled every night [1]!!
>
>   * All c/c++ test suites have coverage enabled and are running nightly
>   * jsvm[7] coverage is enabled, and running
>   * codecov.io [2] shows the results, broken down by directory
>
> ## Complications
>
>   * Rust code coverage is still delayed [6]
>   * There is instability in the coverage numbers due to variability in
> our test runs. This is minor compared to the large volume of
> coverage we are collecting, but it may impact discriminating analysis.
>
> ## Plans for remaining Quarter
>
>   * Release management requires a view into how coverage is changing
> over time, and relate that change to changesets and bugs. 
> mcastelluccio has a very rough prototype [10]
>   * Work on the long tail of test suite flavors, esoteric tests,
> intermittent tests, skipped tests, and operational details to keep
> this whole thing running smooth [9]
>   * Streamline the grcov ETL pipeline
>   * Explore what else we can do with this data.
>   * Put together a presentation deck, just in case someone is
> interested, for AllHands.
>
> ## Meetings
>
> We have weekly CodeCoverage meetings, and you are welcome to attend:
>
>   * When: Held every Friday @ 11:30 EDT (08:30 PDT)
>   * Where: Kyle's video room
> https://v.mozilla.com/flex.html?roomdirect.html=huhL8WaTwCwC
>   * Etherpad: https://public.etherpad-mozilla.org/p/code_coverage_Q1_17
>
>
> ## Reference
>
> [1] See coverage on TH
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-central=ccov
>
> [1b] Example on TH:
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-central=e61060be36424240058f8bef4c5597f401bc8b7e=ccov
>
> [2] codecov.io https://codecov.io/gh/marco-c/gecko-dev
>
> [3] Local Coverage
> https://developer.mozilla.org/en-US/docs/Mozilla/Testing/Measuring_Code_Coverage_on_Firefox?document_saved=true#Generate_Code_Coverage_report_from_a_try_build_(or_any_other_treeherder_build
> <https://developer.mozilla.org/en-US/docs/Mozilla/Testing/Measuring_Code_Coverage_on_Firefox?document_saved=true#Generate_Code_Coverage_report_from_a_try_build_%28or_any_other_treeherder_build>)
>
> [4] iOS Coverage
> https://codecov.io/gh/mozilla-mobile/firefox-ios/branch/master
>
> [5] User Cases
> https://docs.google.com/document/d/1JUEPS8Xdtx4y8fXA4Au_Ggme0fMyJQxZbby0X2e4yDI/edit#heading=h.ddwyjkvxus4-
>
>
> [6] Rust coverage
> https://docs.google.com/document/d/1JUEPS8Xdtx4y8fXA4Au_Ggme0fMyJQxZbby0X2e4yDI/edit#heading=h.ddwyjkvxus4
>
> [7] JSVM coverage: https://bugzilla.mozilla.org/show_bug.cgi?id=1301174
>
> [8] e10s coverage example:
> https://treeherder.mozilla.org/#/jobs?repo=try=b6e9cefe95adc3dd281bf8e2a2f897e8f4839e51
>
> [9] Everything:
> https://bugzilla.mozilla.org/showdependencytree.cgi?id=1278393
>
> [10] Show coverage differences, with bug numbers:
> https://marco-c.github.io/grcov-test/coverage_by_dir.html
>

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

CodeCoverage Monthly Update

2017-06-09 Thread Kyle Lahnakoski


## Summary of the past month

Coverage is enabled for nearly all tests, and scheduled every night [1]!!

  * All c/c++ test suites have coverage enabled and are running nightly
  * jsvm[7] coverage is enabled, and running
  * codecov.io [2] shows the results, broken down by directory

## Complications

  * Rust code coverage is still delayed [6]
  * There is instability in the coverage numbers due to variability in
our test runs. This is minor compared to the large volume of
coverage we are collecting, but it may impact discriminating analysis.

## Plans for remaining Quarter

  * Release management requires a view into how coverage is changing
over time, and relate that change to changesets and bugs. 
mcastelluccio has a very rough prototype [10]
  * Work on the long tail of test suite flavors, esoteric tests,
intermittent tests, skipped tests, and operational details to keep
this whole thing running smooth [9]
  * Streamline the grcov ETL pipeline
  * Explore what else we can do with this data.
  * Put together a presentation deck, just in case someone is
interested, for AllHands.

## Meetings

We have weekly CodeCoverage meetings, and you are welcome to attend:

  * When: Held every Friday @ 11:30 EDT (08:30 PDT)
  * Where: Kyle's video room
https://v.mozilla.com/flex.html?roomdirect.html=huhL8WaTwCwC
  * Etherpad: https://public.etherpad-mozilla.org/p/code_coverage_Q1_17


## Reference

[1] See coverage on TH
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central=ccov

[1b] Example on TH:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central=e61060be36424240058f8bef4c5597f401bc8b7e=ccov

[2] codecov.io https://codecov.io/gh/marco-c/gecko-dev

[3] Local Coverage
https://developer.mozilla.org/en-US/docs/Mozilla/Testing/Measuring_Code_Coverage_on_Firefox?document_saved=true#Generate_Code_Coverage_report_from_a_try_build_(or_any_other_treeherder_build
)

[4] iOS Coverage
https://codecov.io/gh/mozilla-mobile/firefox-ios/branch/master

[5] User Cases
https://docs.google.com/document/d/1JUEPS8Xdtx4y8fXA4Au_Ggme0fMyJQxZbby0X2e4yDI/edit#heading=h.ddwyjkvxus4-


[6] Rust coverage
https://docs.google.com/document/d/1JUEPS8Xdtx4y8fXA4Au_Ggme0fMyJQxZbby0X2e4yDI/edit#heading=h.ddwyjkvxus4

[7] JSVM coverage: https://bugzilla.mozilla.org/show_bug.cgi?id=1301174

[8] e10s coverage example:
https://treeherder.mozilla.org/#/jobs?repo=try=b6e9cefe95adc3dd281bf8e2a2f897e8f4839e51

[9] Everything:
https://bugzilla.mozilla.org/showdependencytree.cgi?id=1278393

[10] Show coverage differences, with bug numbers:
https://marco-c.github.io/grcov-test/coverage_by_dir.html

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

CodeCoverage Monthly Update

2017-05-03 Thread Kyle Lahnakoski


It is the first Wednesday for the month!  Time to write an update!


## Summary of Activity

  * Daily coverage reports on coveralls.io [1] and on codecov.io[2].
Which do you like?
  * You can now use one-click loaners to generate coverage locally
(thanks maja and mcastelluccio!)
  * We met the nice people working on coverage for iOS [4] (thanks isabel!)
  * e10s coverage builds are working! (thanks mcastelluccio!) [8]
  * Listed the use cases we will be solving for in the short term, and
the long term [5]

## Some bad news

  * Rust code coverage will probably not be available until July, or
later.  Work is progressing [6], but the release schedule, CI
integration, and the unknowns, will add delay.
  * JSVM, despite it is working[7], has not been verified and scheduled

## Future Plans

  * e10s coverage will be compared to ensure there are no multiprocess
anomallies
  * More test suites will be enabled for nightly scheduled coverage
  * JSVM coverage work will resume
  * Start identifying the gaps between current visualizations and
Release Management's expectations so we have priority worklist for
end of the quarter

## Meetings

We have weekly CodeCoverage meetings, and you are welcome to attend:

  * When: Held every Friday @ 11:30 EDT (08:30 PDT)
  * Where: Kyle's video room
https://v.mozilla.com/flex.html?roomdirect.html=huhL8WaTwCwC
  * Etherpad: https://public.etherpad-mozilla.org/p/code_coverage_Q1_17


## Reference

[1] coveralls.io https://coveralls.io/github/marco-c/gecko-dev

[2] codecov.io https://codecov.io/gh/marco-c/gecko-dev

[3] Local Coverage
https://developer.mozilla.org/en-US/docs/Mozilla/Testing/Measuring_Code_Coverage_on_Firefox?document_saved=true#Generate_Code_Coverage_report_from_a_try_build_(or_any_other_treeherder_build
)

[4] iOS Coverage
https://codecov.io/gh/mozilla-mobile/firefox-ios/branch/master

[5] User Cases
https://docs.google.com/document/d/1JUEPS8Xdtx4y8fXA4Au_Ggme0fMyJQxZbby0X2e4yDI/edit#heading=h.ddwyjkvxus4-


[6] Rust coverage
https://docs.google.com/document/d/1JUEPS8Xdtx4y8fXA4Au_Ggme0fMyJQxZbby0X2e4yDI/edit#heading=h.ddwyjkvxus4

[7] JSVM coverage: https://bugzilla.mozilla.org/show_bug.cgi?id=1301174

[8] e10s coverage example:
https://treeherder.mozilla.org/#/jobs?repo=try=b6e9cefe95adc3dd281bf8e2a2f897e8f4839e51

[9] Everything:
https://bugzilla.mozilla.org/showdependencytree.cgi?id=1278393

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

CodeCoverage Monthly Update

2017-04-05 Thread Kyle Lahnakoski


This is the second monthly update on our CodeCoverage effort:

## The Progress to Date

* Coverage is scheduled to run daily on central.
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central=cov
* The UCOSP[1] students have completed their term, and can show
aggregate statistics by folder with the code coverage ui [5]
https://ericdesj.github.io/moz-coco-w17-preview/
* c/c++ coverage is being ETL'ed into our unified coverage datastore,
but does not yet show in the UI
* `grcov`[4] was written mcastelluccio to process coverage artifacts
much faster; reducing processing time from hours to
seconds. `grcov` also outputs multiple formats, including for coveralls.io
* There is exploration into using coveralls.io:
https://coveralls.io/jobs/24085087, the main page is
https://coveralls.io/github/marco-c/gecko-dev.
* JSVM coverage got working today!
https://treeherder.mozilla.org/#/jobs?repo=try=ce2c56eeceb1bc252c6b25a50304f725c320d5e0


## The Blockers

These items are further behind than I would like.  If you can help,
please do.

* Getting Rust to emit coverage artifacts is important:
https://bugzilla.mozilla.org/show_bug.cgi?id=1335518
* Continuing the hard work of enabling C/C++ codecoverage on all our
tests: https://bugzilla.mozilla.org/show_bug.cgi?id=1301170

## The Biggest Problem

**Our test suites are unstable**. Many of you may already know that many
tests fail intermittently; these "intermittents" impact code coverage;
either by breaking the test run, and preventing coverage collection; or
by changing code paths. The challenge is automating the metatdata
collection and having a strategy to mitigate the loss of data. [8]
https://bugzilla.mozilla.org/show_bug.cgi?id=1337241

Even so, **test runs are not deterministic**, so we expect fluctuation
in the aggregate coverage statistics; How large this fluctuation is we
do not know.

Got ideas?  You want to work on a problem that no one will thank you
for?  ;)  Please talk to me! [3]

## The Current Plan

For the next month we will be continuing work on:

* validating the data and process, including adding Rust and more test
suites.
* Adding per-line coverage to the CoCo UI: The aggregate numbers are
good for identifying sections of code that are not tested, but we need
line-level details to inform action.
https://github.com/mozilla/moz-coco/issues/7
* Explore how coveralls.io can be used to make that job easier  
* Explore how to leverage Taskcluster-generated coverage artifacts with
local coverage tools [9]
https://bugzilla.mozilla.org/show_bug.cgi?id=1350446
* We are investigating if coverage tools behave differently when e10s is
enabled, then enabling it.
* JSVM ETL pipeline work will commence:
https://bugzilla.mozilla.org/show_bug.cgi?id=1301174


## Motivation

*Unchanged from last email* - Knowing code coverage statistics and code
coverage specifics can help evaluate risk in code: It can show what test
suites cover what lines, it can show if new code is covered by an
existing test.  There are many other exciting insights we can extract
from code coverage. But first, we must collect it.

## Reference

[1] UCOSP (Undergraduate Capstone Open Source Projects) http://ucosp.ca/

[2] The original plan:
https://docs.google.com/document/d/1dOWi18qrudwaOThNAYoCMS3e9LzhxGUiMLLrQ_WVR9w/edit#

[3] Kyle Lahnakoski email: klahnako...@mozilla.org  irc: ekyle on
#at...@irc.mozilla.org

[4] grcov https://github.com/marco-c/grcov

[5] (Co)de (Co)verage UI preview -
https://ericdesj.github.io/moz-coco-w17-preview/

[5b] CoCo is on Github - https://github.com/mozilla/moz-coco

[6] CoCo relay - https://github.com/ericdesj/moz-coco-relay

[7] Rust Coverage Tracking
https://bugzilla.mozilla.org/show_bug.cgi?id=1335518

[8] Knowing when the coverage is complete:
https://bugzilla.mozilla.org/show_bug.cgi?id=1337241

[9] Coverage with local tools:
https://bugzilla.mozilla.org/show_bug.cgi?id=1350446
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Fwd: CodeCoverage: First Monthly Update

2017-03-03 Thread Kyle Lahnakoski


It was pointed out that my email address did not have enough "L"s in
it.  Here is my real email address:

klahnako...@mozilla.org



 Forwarded Message 
Subject:CodeCoverage: First Monthly Update
Date:   Thu, 2 Mar 2017 21:29:10 -0500
From:   Kyle Lahnakoski <klahnako...@mozilla.com>
To: dev-platform@lists.mozilla.org




It is my pleasure to introduce you all to our project to collect code
coverage!


Motivation

Knowing code coverage statistics and code coverage specifics can help
evaluate risk in code: It can show what test suites cover what lines, it
can show if new code is covered by an existing test.  There are many
other exciting insights we can extract from code coverage. But first, we
must collect it.

The Problem

The biggest problem we face is scale:  Firefox has many LoC, it is
composed of multiple languages, it is run on multiple platforms, and the
test suites are split among hundreds of machines for a single build.

History

For just over a year, this CodeCoverage project has had number of
excellent contributors from various Universities across Canada[1].  They
did the exploratory work required to solve the problems of scaling
collection and aggregation so to handle the Mozilla-scale problem with
code coverage.  If any of those contributors are reading this: Thank you!

The Plan

For the next month we will be working on c++ coverage, Rust coverage and
validating the data and process.  There are many unknowns and I will
have more concrete deliverables in the next update.  The details of that
plan is in our freshly minted planning document [2].  There you can find
bugs, and code links.  If you are interested in more, please contact me [3]

I will send out monthly emails with our progress, some links where you
see our progress, and our future plans.


[1] UCOSP (Undergraduate Capstone Open Source Projects)http://ucosp.ca/

[2] The actual plan:
https://docs.google.com/document/d/1dOWi18qrudwaOThNAYoCMS3e9LzhxGUiMLLrQ_WVR9w/edit#

[3] Kyle Lahnakoski email: klahnako...@mozilla.org  irc: ekyle on
#at...@irc.mozilla.org

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

CodeCoverage: First Monthly Update

2017-03-02 Thread Kyle Lahnakoski


It is my pleasure to introduce you all to our project to collect code
coverage!


Motivation

Knowing code coverage statistics and code coverage specifics can help
evaluate risk in code: It can show what test suites cover what lines, it
can show if new code is covered by an existing test.  There are many
other exciting insights we can extract from code coverage. But first, we
must collect it.

The Problem

The biggest problem we face is scale:  Firefox has many LoC, it is
composed of multiple languages, it is run on multiple platforms, and the
test suites are split among hundreds of machines for a single build.

History

For just over a year, this CodeCoverage project has had number of
excellent contributors from various Universities across Canada[1].  They
did the exploratory work required to solve the problems of scaling
collection and aggregation so to handle the Mozilla-scale problem with
code coverage.  If any of those contributors are reading this: Thank you!

The Plan

For the next month we will be working on c++ coverage, Rust coverage and
validating the data and process.  There are many unknowns and I will
have more concrete deliverables in the next update.  The details of that
plan is in our freshly minted planning document [2].  There you can find
bugs, and code links.  If you are interested in more, please contact me [3]

I will send out monthly emails with our progress, some links where you
see our progress, and our future plans.


[1] UCOSP (Undergraduate Capstone Open Source Projects)http://ucosp.ca/

[2] The actual plan:
https://docs.google.com/document/d/1dOWi18qrudwaOThNAYoCMS3e9LzhxGUiMLLrQ_WVR9w/edit#

[3] Kyle Lahnakoski email: klahnako...@mozila.org  irc: ekyle on
#at...@irc.mozilla.org

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Should &&/|| really be at the end of lines?

2017-02-18 Thread Kyle Lahnakoski



On 2017-02-16 18:21, Jean-Yves Avenard wrote:
>
>
> On 17/02/17 00:17, Jeff Gilbert wrote:
>> If there are so many parens that we require an editor to disambiguate
>> them, it's too dense. Add whitespace or decompose the expression until
>> readable.
>>
> I'd love to know how adding any whitespaces could make any logical
> expressions clearer that what I posted above.
>
> That's the point, there's no more whitespace to add, nor a need to
> decompose further, it's already been fully expanded.
>

There is lots of whitespace to add: If the expression is over a line
long, then use indentation so the block structure follows the logical
structure:

return
(
(aCodecMask & VPXDecoder::VP8)
&&
aMimeType.EqualsLiteral("video/webm; codecs=vp8")
)
||
(
(aCodecMask & VPXDecoder::VP9)
&&
aMimeType.EqualsLiteral("video/webm; codecs=vp9")
)
||
(
(aCodecMask & VPXDecoder::VP9)
&&
aMimeType.EqualsLiteral("video/vp9")
)
;

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Now measuring Firefox size per-commit. What else should we be tracking?

2015-11-05 Thread Kyle Lahnakoski

Al,,

ActiveData[1] is warehouse of our test times, build times, and whatever
other properties or measures I can get my hands on.   It has a primitive
query tool[2] that a human can use to send queries to the public service
[3].   The `unittest` table has results from all tests that emit
structured logs, and is the largest at 4 billion records.

Getting a feel for what is in ActiveData will take a little time.  I
would first suggest a short list of some example documents.  `{"from":
"unittest"}` will give you some example records.  `{"from":"unittest",
"groupby":"build.platform"}` will give you the cardinality of the
platforms.   There are also some starter docs on the query tool page [2].

Please forgive the low speed, it is scaled down for minimal cost.

Some other tables:
* {"from":"unittest.result.subtests"}  a table of all nested unittest
documents: only has the failure messages, for now
* {"from":"jobs"} the buildbot job properties and timing (builds are not
properly categorized, he Treeherder code should be used for this),
* {"from":"jobs.action.timings"} each buildbot step, each mozharness
step, and their timing
* {"from":"orange_factor"} copy of the orangefactor data, plus `push_date`
* {"from":"perf"}   all the Talos results, including replicates

I have only started to ingest the TaskCluster information.

[1] https://wiki.mozilla.org/Auto-tools/Projects/ActiveData
[2] http://activedata.allizom.org/tools/query.html
[3] http://activedata.allizom.org/query


On 2015-11-05 09:18, L. David Baron wrote:
> On Wednesday 2015-11-04 12:46 -0500, William Lachance wrote:
>> On 2015-11-04 10:55 AM, William Lachance wrote:
>>> 1. Relatively deterministic.
>>> 2. Something people actually care about and are willing to act on, on a
>>> per-commit basis. If you're only going to look at it once a quarter or
>>> so, it doesn't need to be in Perfherder.
>>>
>>> Anyway, just thought I'd open the floor to brainstorming. I'd prefer to
>>> add stuff incrementally, to make sure Perfherder can handle the load,
>>> but I'd love to hear all your ideas.
>> Someone mentioned "test times" to me in private email.
> That was me.  (I didn't feel like sending a late-at-night
> one-sentence email to the whole list, and figured there was a decent
> chance that somebody else would mention it as well.)
>
> I think they're worth tracking because we've had substantial
> performance regressions (I think including as bad as a doubling in
> times) that weren't caught quickly, and led to substantially worse
> load on our testing infrastructure.
>
>> I do think test times are worth tracking, but probably not in Perfherder:
>> test times might not be deterministic depending on where / how they're
>> running (which makes it difficult to automatically detect regressions and
>> sheriff them on a per commit basis) and regardless there's too much data to
>> really be manageable by Perfherder's intended interface even if that problem
>> were magically solved.
> It seems like if we're running the same tests on different sorts of
> machines, we could track different perf numbers for the test run on
> different machine classes.
>
> We'd also want to measure the test time and *not* the time spent
> downloading the build.
>
> And we'd probably want to measure the total time across chunks so
> that we don't count redistribution between chunks as a set of
> regressions and improvements.
>
> So that does make it a bit difficult, but it does seem doable.
>
>> As a possible alternative, I believe Kyle Lahnakoski's ActiveData project
>> (https://wiki.mozilla.org/Auto-tools/Projects/ActiveData) already *does*
>> track this type of data but last I heard he was looking for more feedback on
>> how to alert/present it to the platform community. If you have any ideas on
>> this, please let him know (he's CC'ed). :)
> Perhaps, but I have no idea how to use it or what that would look
> like.  The wiki page explicitly says it's for automated clients and
> not by humans; it would be useful to see an example of such an
> automated client to have an idea of how this would work.
>
> -David
>

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Code Coverage Summary

CodeCoverage Monthly Update

Re: Reminder on Try usage and infrastructure resources

CodeCoverage! Monthly Update

CodeCoverage Monthly Update

Re: CodeCoverage Monthly Update (Rust)

CodeCoverage Monthly Update

CodeCoverage Monthly Update

CodeCoverage Monthly Update

Fwd: CodeCoverage: First Monthly Update

CodeCoverage: First Monthly Update

Re: Should &&/|| really be at the end of lines?

Re: Now measuring Firefox size per-commit. What else should we be tracking?

13 matches

Site Navigation

Mail list logo

Footer information