CodeCoverage Monthly Update

[Kyle Lahnakoski]

## Summary of the past month

Coverage is enabled for nearly all tests, and scheduled every night [1]!!

  * All c/c++ test suites have coverage enabled and are running nightly
  * jsvm[7] coverage is enabled, and running
  * codecov.io [2] shows the results, broken down by directory

## Complications

  * Rust code coverage is still delayed [6]
  * There is instability in the coverage numbers due to variability in
our test runs. This is minor compared to the large volume of
coverage we are collecting, but it may impact discriminating analysis.

## Plans for remaining Quarter

  * Release management requires a view into how coverage is changing
over time, and relate that change to changesets and bugs. 
mcastelluccio has a very rough prototype [10]
  * Work on the long tail of test suite flavors, esoteric tests,
intermittent tests, skipped tests, and operational details to keep
this whole thing running smooth [9]
  * Streamline the grcov ETL pipeline
  * Explore what else we can do with this data.
  * Put together a presentation deck, just in case someone is
interested, for AllHands.

## Meetings

We have weekly CodeCoverage meetings, and you are welcome to attend:

  * When: Held every Friday @ 11:30 EDT (08:30 PDT)
  * Where: Kyle's video room
https://v.mozilla.com/flex.html?roomdirect.html=huhL8WaTwCwC
  * Etherpad: https://public.etherpad-mozilla.org/p/code_coverage_Q1_17


## Reference

[1] See coverage on TH
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central=ccov

[1b] Example on TH:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central=e61060be36424240058f8bef4c5597f401bc8b7e=ccov

[2] codecov.io https://codecov.io/gh/marco-c/gecko-dev

[3] Local Coverage
https://developer.mozilla.org/en-US/docs/Mozilla/Testing/Measuring_Code_Coverage_on_Firefox?document_saved=true#Generate_Code_Coverage_report_from_a_try_build_(or_any_other_treeherder_build
)

[4] iOS Coverage
https://codecov.io/gh/mozilla-mobile/firefox-ios/branch/master

[5] User Cases
https://docs.google.com/document/d/1JUEPS8Xdtx4y8fXA4Au_Ggme0fMyJQxZbby0X2e4yDI/edit#heading=h.ddwyjkvxus4-


[6] Rust coverage
https://docs.google.com/document/d/1JUEPS8Xdtx4y8fXA4Au_Ggme0fMyJQxZbby0X2e4yDI/edit#heading=h.ddwyjkvxus4

[7] JSVM coverage: https://bugzilla.mozilla.org/show_bug.cgi?id=1301174

[8] e10s coverage example:
https://treeherder.mozilla.org/#/jobs?repo=try=b6e9cefe95adc3dd281bf8e2a2f897e8f4839e51

[9] Everything:
https://bugzilla.mozilla.org/showdependencytree.cgi?id=1278393

[10] Show coverage differences, with bug numbers:
https://marco-c.github.io/grcov-test/coverage_by_dir.html

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Quantum Flow Engineering Newsletter #12

[Ehsan Akhgari]

Hi everyone,

It has been a few weeks since I have given an update about our progress on
reducing the amount of slow synchronous IPC messages that we send across
our processes.  This hasn't been because there hasn't been a lot to talk
about, quite to the contrary, so much great work has happened here that for
a while I decided it may be better to highlight other ongoing work
instead.  But now as the development cycle of Firefox 55 comes to a closing
point, it's time to have another look at where we stand on this issue.

I've prepared a new Sync IPC Analysis

for today including data from both JS and C++ initiated sync IPCs.  First
bit of unfortunate news is that the historical data in the spreadsheet is
lost because the server hosting the data had a few hiccups and Google
Spreadsheets seems to not really not like that.  Second bit of unfortunate
news is that our hopes

for disabling the non-multiprocess compatible add-ons by default in Nightly
 helping with
reducing some of the noise in this data don't seem to have panned out.  The
data still shows a lot of synchronous IPC triggered from JS as before, and
the lion's share of it are messages that are clearly coming from add-ons
judging from their names.  My guess about why is that Nightly users have
probably turned these add-ons back on manually.  So we will have to live
with the noise in the data for now (this is an issue that we have to
struggle with when dealing with a lot of telemetry data unfortunately, here
is another recent example
 that wasted some
time and energy).

This time I won't give out a percentage based break-down because now after
many of these bugs have been fixed, the impact of really commonly occurring
IPC messages such as the one we have for document.cookie
 really makes the
earlier method of exploring the data pointless (you can explore the pie
chart to get a quick sense of why, I'll just say that message alone is now
55% of the chart and that plus the second one together form 75% of the
data.)  This is a great problem to have, of course, it means that we're now
starting to get to the "long tail" part of this issue.

The current top offenders, besides the mentioned bug (which BTW is still
being made great progress on!) are add-on/browser CPOW messages, two
graphics initialization
 messages
 that we send at
content process startup, NotifyIMEFocus
 that's in the
process of being fixed, and window.open()
 which I've spent
weeks on but have yet to fix all of our tests to be able to land my fixes
for (which I've also temporarily given up working on looking for something
that isn't this bug to work on for a little while!).  Besides those if you
look at the dependency list of the tracker bug, there are many other bugs
that are very close to being fixed.  Firefox 55 is going to be much better
from this perspective and I hope the future releases will improve on that!

The other effort that is moving ahead quite fast is optimizing for Speedometer
V2
.
See the chart of our progress on AreWeFastYet.com:



Last week, our score on this chart was about 84.  Now we are at about 91.
Not bad for a week worth a work!  If you're curious to follow along, see
our tracker bug .
Also, Speedometer is a very JS heavy benchmark, so a lot of the bugs that
are filed and fixed for it happen inside SpiderMonkey so watching the
SpiderMonkey
specific tracker bug 
is probably a good idea as well.

It's time for a short performance story!  This one is about technical
debt.  I've looked at many performance bugs over the past few months of the
Quantum Flow project, and in many cases the solutions have turned out to be
just deleting the slow code, that's it!  It turns out that in a large code
base as code ages, there is a lot of code that isn't really serving any
purpose any more but nobody discovers this because it's impractical to
audit every single line of code with scrutiny.  But then some of this
unnecessary code is bound to have severe performance issues, and when it
does, your software ends up carrying that cruft for years!  Here are a few
examples: a function call taking 2.7 seconds on a cold startup