Re: Code Coverage on Phabricator
Of course, sorry for the weird formatting... Here is the right version: Hi everyone, We have recently implemented a solution to integrate code coverage results into Phabricator. Coverage information is uploaded either automatically for revisions after they are landed in mozilla-central (for example for release managers when looking at uplift requests), or on-demand for in-progress revisions. For revisions under review, in order to upload coverage you just need to trigger a try push containing code coverage builds and tests, e.g. by using: $ mach try fuzzy --full and selecting the relevant ccov builds and test suites. In the future, we will also likely automatically trigger coverage try builds for revisions we deem to be risky, alongside the on-demand option. For example you can see my try build at https://treeherder.mozilla.org/#/jobs?repo=try=38213b49dc00cd108dfa9a246045ed677c34de91 which produced the coverage information for my revision at https://phabricator.services.mozilla.com/D14758. Once the try build and linked tests finish, the coverage artifacts get parsed and uploaded to the Phabricator revisions corresponding to the commits in the try push. The analysis works on *all* commits in the try push that are linked to Phabricator revisions. Stacks of revisions are supported as well. The coverage information is shown on Phabricator both at a high-level view, in the "Revision Contents" section, and at a detailed view in the "Diff" section. The "Revision Contents" section contains a list of the files modified by the revision, showing both the coverage percentage of the whole file and the coverage percentage of touched lines. The "Diff" section instead shows the coverage line per line, coloring the bar on the right-hand side. Orange corresponds to "uncovered lines", light blue corresponds to "non-executable lines" (e.g. a comment, a definition, and so on), dark blue corresponds to "covered lines". When hovering the bar, the corresponding line is highlighted in the same color. You can find an alternative version of this email with fancy screenshots at https://marco-c.github.io/2019/01/21/code-coverage-phabricator.html or https://release.mozilla.org/tooling/codecoverage/2019/01/20/code-coverage-on-phabricator.html. Thanks, Marco. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Code Coverage on Phabricator
Hi everyone, We have recently implemented a solution to integrate code coverage results into Phabricator. Coverage information is uploaded either automatically for revisions after they are landed in mozilla-central (for example for release managers when looking at uplift requests), or on-demand for in-progress revisions. For revisions under review, in order to upload coverage you just need to trigger a try push containing code coverage builds and tests, e.g. by using: $ mach try fuzzy --full and selecting the relevant ccov builds and test suites. In the future, we will also likely automatically trigger coverage try builds for revisions we deem to be risky, alongside the on-demand option. For example you can see my [try build](https://treeherder.mozilla.org/#/jobs?repo=try=38213b49dc00cd108dfa9a246045ed677c34de91) which produced the coverage information for [my revision](https://phabricator.services.mozilla.com/D14758). Once the try build and linked tests finish, the coverage artifacts get parsed and uploaded to the Phabricator revisions corresponding to the commits in the try push. The analysis works on *all* commits in the try push that are linked to Phabricator revisions. Stacks of revisions are supported as well. The coverage information is shown on Phabricator both at a high-level view, in the *Revision Contents* section, and at a detailed view in the *Diff* section. The *Revision Contents* section contains a list of the files modified by the revision, showing both the coverage percentage of the whole file and the coverage percentage of touched lines. The *Diff* section instead shows the coverage line per line, coloring the bar on the right-hand side. **Orange** corresponds to **uncovered lines**, **light blue** corresponds to **non-executable lines** (e.g. a comment, a definition, and so on), **dark blue** corresponds to **covered lines**. When hovering the bar, the corresponding line is highlighted in the same color. You can find an alternative version of this email with fancy screenshots at https://marco-c.github.io/2019/01/21/code-coverage-phabricator.html or https://release.mozilla.org/tooling/codecoverage/2019/01/20/code-coverage-on-phabricator.html. Thanks, Marco. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Rust code coverage
Il 07/01/19 14:36, Henri Sivonen ha scritto: > On Fri, Jan 4, 2019 at 2:54 PM Marco Castelluccio > wrote: >> Hi everyone, >> we have recently enabled collecting code coverage for Rust code too, > Nice! > >> running Rust tests in coverage builds. > Does this mean running cargo test for each crate under > third_party/rust, running Firefox test suites or both? Both. We're running Rust specific tests (cargo test) like the Rusttests builds, and we're running the normal test suites. > > As for trying to make sense of what the numbers mean: > > Is the coverage ratio reported on lines attributed at all in ELF as > opposed to looking at the number of lines in the source files? We're using LLVM GCOV instrumentation, which uses debug info. > > What kind of expectations one should have on how the system measures > coverage for code that gets inlined? Inlining could cause inconsistencies in line counts, but it shouldn't cause inconsistencies in "is this line covered or not". We have thought of disabling inlining for coverage builds in the past, but there is a big runtime cost. Also, coverage instrumentation by itself increases the size of the functions, so fewer functions are inlined than in normal debug builds. - Marco. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal to adjust testing to run on PGO builds only and not test on OPT builds
On 21/01/2019 10:18, Jan de Mooij wrote: On Fri, Jan 18, 2019 at 10:36 PM Joel Maher wrote: Are there any concerns with this latest proposal? This proposal sounds great to me. Thank you! +1. This seems like the right first step to me. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposal to adjust testing to run on PGO builds only and not test on OPT builds
On Fri, Jan 18, 2019 at 10:36 PM Joel Maher wrote: > Are there any concerns with this latest proposal? > This proposal sounds great to me. Thank you! Jan On Thu, Jan 17, 2019 at 12:52 PM Jan de Mooij wrote: > >> Hi Joel, >> >> Can you say more about this point in your original email: "3) This will >> reduce the jobs (about 16%) we run which in turn reduces, cpu time, money >> spent, turnaround time, intermittents, complexity of the taskgraph." It >> seems to me that if we remove non-PGO opt builds even on Try, we might use >> more cpu time because there are so many Try pushes requesting opt builds. >> Do we have data on this? >> >> Thanks, >> Jan >> >> On Thu, Jan 17, 2019 at 5:45 PM jmaher wrote: >> >>> Following up on this, thanks to Chris we have fast artifact builds for >>> PGO, so the time to develop and use try server is in parity with current >>> opt solutions for many cases (front end development, most bisection cases). >>> >>> I have also looked in depth at what the impact on the integration >>> branches would be. In the data set from July-December (H2 2018) there were >>> 11 instances of tests that we originally only scheduled in the OPT config >>> and we didn't have PGO or Debug test jobs to point out the regression (this >>> is due to scheduling choices). Worse case scenario is finding the >>> regression on PGO up to 1 hour later 11 times or roughly 2x/month. >>> Backfilling to find the offending patch as we do now 24% of the time would >>> be similar time. In fact running the OPT jobs on Debug instead would >>> result in same time for all 11 instances (due to more chunks on debug and >>> similar runtimes). In short, little to no impact. >>> >>> Lastly there was a pending question about talos. There is an edge case >>> where we can see a regression on talos that is PGO, but it is unrelated to >>> the code and just a side effect of how PGO works. I looked into that in >>> https://bugzilla.mozilla.org/show_bug.cgi?id=1514829. I found that if >>> we didn't get opt alerts that we would not have missed any regressions. >>> Furthermore, for the regressions, for the ones that were pgo only >>> regressions (very rare) there were many other regressions at the same time >>> (say a build change, or test change, etc.) and usually these were accepted >>> changes, backed out, or investigated on a different test or platform. In >>> the past when we have determined a regression is a PGO artifact we have >>> resolved it as WONTFIX and moved on. >>> >>> Given this summary, I feel that most concerns around removing testing >>> for OPT are addressed. I would also like to extend the proposal to remove >>> the OPT builds since no unit or perf tests would run on there. >>> >>> As my original timeline is not realistic, I would like to see if there >>> are comments until next Wednesday- January 23rd, then I can follow up on >>> remaining issues or work towards ensuring we start the process of making >>> this happen and what the right timeline is. >>> ___ >>> dev-platform mailing list >>> dev-platform@lists.mozilla.org >>> https://lists.mozilla.org/listinfo/dev-platform >>> >> ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
