Re: Try-based code coverage results
On Monday, July 7, 2014 4:44:50 PM UTC-7, Joshua Cranmer wrote: On 7/7/2014 5:25 PM, Jonathan Griffin wrote: Filed https://bugzilla.mozilla.org/show_bug.cgi?id=1035464 for those that would like to follow along. Perhaps bug 890116 is a better measure of tracking. -- Joshua Cranmer Thunderbird and DXR developer Source code archæologist Hi Joshua-- I work on the Treeherder dev team. Would you be up for writing a few bugs against Treeherder on what you found? Here’s a link to do so: https://bugzilla.mozilla.org/enter_bug.cgi?product=Tree%20Management We definitely want to get things like this fixed. Also, please feel free to ping us on the #treeherder channel on irc and we can work with you on issues that might block your progress. Thanks! -Cam ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Try-based code coverage results
On Friday, July 11, 2014 3:59:41 PM UTC-7, cda...@mozilla.com wrote: On Monday, July 7, 2014 4:44:50 PM UTC-7, Joshua Cranmer wrote: On 7/7/2014 5:25 PM, Jonathan Griffin wrote: Filed https://bugzilla.mozilla.org/show_bug.cgi?id=1035464 for those that would like to follow along. Perhaps bug 890116 is a better measure of tracking. -- Joshua Cranmer Thunderbird and DXR developer Source code archæologist Hi Joshua-- I work on the Treeherder dev team. Would you be up for writing a few bugs against Treeherder on what you found? Here’s a link to do so: https://bugzilla.mozilla.org/enter_bug.cgi?product=Tree%20Management We definitely want to get things like this fixed. Also, please feel free to ping us on the #treeherder channel on irc and we can work with you on issues that might block your progress. Thanks! -Cam Hi Joshua-- Actually I see the bug wrt the M-e10s jobs all looking the same. I wrote up https://bugzilla.mozilla.org/show_bug.cgi?id=1037719 and should have a fix for that by Monday. -Cam ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Try-based code coverage results
Hey Joshua, That's awesome! How long does the try run take that generated this data? We should consider scheduling a periodic job to collect this data and track it over time. Jonathan On 7/6/2014 10:02 PM, Joshua Cranmer wrote: I don't know how many people follow code-coverage updates in general, but I've produced relatively up-to-date code coverage results based on http://hg.mozilla.org/mozilla-central/rev/81691a55e60f, and they may be found here: http://www.tjhsst.edu/~jcranmer/m-ccov/. In contrast to earlier versions of my work, you can actually explore the coverage as delineated by specific tests, as identified by their TBPL identifier. Christian's persistent requests for me to limit the depth of the treemap view are still unresolved, because, well, at 2 AM in the morning, I just wanted to push a version that worked. The test data was generated by pushing modified configs to try and using blobber features to grab the resulting coverage data. Only Linux32/64 is used, and only opt builds are represented (it's a --disable-optimize --disable-debug kind of build), the latter because I wanted to push a version out tonight and the debug .gcda tarballs are taking way too long to finish downloading. Effectively, only xpcshell tests, and the M, M-e10s, and R groups are represented in the output data. M-e10s is slightly borked: only M-e10s(1) [I think] is shown, because, well, treeherder didn't distinguish between the five of them. A similar problem with the debug M(dt1/dt2/dt3) test suites will arise when I incorporate that data. C++ unit tests are not present because blobber doesn't run on C++ unit tests for some reason, and Jit-tests, jetpack tests, and Marionette tests await me hooking in the upload scripts to those testsuites (and Jit-tests would suffer a similar numbering problems). The individual testsuites within M-oth may be mislabeled because I can't sort names properly. There's a final, separate issue with treeherder not recording the blobber upload artifacts for a few of the runs (e.g., Linux32 opt X), even though it finished without errors and tbpl records those artifacts. So coverage data is missing for the affected run. It's also worth noting that a few test runs are mired with timeouts and excessive failures, the worst culprit being Linux32 debug where half the testsuites either had some failures or buildbot timeouts (and no data at all). If you want the underlying raw data (the .info files I prepare from every individual run's info), I can provide that on request, but the data is rather large (~2.5G uncompressed). In short: * I have up-to-date code-coverage on Linux 32-bit and Linux 64-bit. Opt is up right now; debug will be uploaded hopefully within 24 hours. * Per-test [TBPL run] level of detail is visible. * Treeherder seems to be having a bit of an ontology issue... ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Try-based code coverage results
On 7/7/2014 11:39 AM, Jonathan Griffin wrote: Hey Joshua, That's awesome! How long does the try run take that generated this data? We should consider scheduling a periodic job to collect this data and track it over time. Well, it depends on how overloaded try is at the moment. ^_^ The builds take an hour themselves, and the longest-running tests on debug builds can run long enough to encroach the hard (?) 2 hour limit for tests. Post-processing of the try data can take another several hours (a large part of which is limited by the time it takes to download ~3.5GB of data). -- Joshua Cranmer Thunderbird and DXR developer Source code archæologist ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Try-based code coverage results
On Mon, Jul 7, 2014 at 11:11 AM, Jonathan Griffin jgrif...@mozilla.com wrote: I guess a related question is, if we could run this periodically on TBPL, what would be the right frequency? We could potentially create a job in buidlbot that would handle the downloading/post-processing, which might be a bit faster than doing it on an external system. Ideally, you would be able to trigger it on a try run for specific test suites or even specific subsets of tests. For example, for certificate verification changes and SSL changes, it would be great for the reviewer to be able to insist on seeing code coverage reports on the try run that preceded the review request, for xpcshell, cppunit, and GTest, without doing coverage for all test suites. To minimize the performance impact of it further, ideally it would be possible to scope the try runs to cppunit, GTest, and xpcshell tests under the security/ directory in the tree. This would make code review more efficient, because the reviewers wouldn't have to spend as much time suggesting missing tests as part of the review. In PSM, and probably in Gecko generally, people are unlikely to write new tests for old code that they are not changing, so periodic full reports would be less helpful than reports for tryserver. Cheers, Brian ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Try-based code coverage results
On 7/7/2014 1:11 PM, Jonathan Griffin wrote: I guess a related question is, if we could run this periodically on TBPL, what would be the right frequency? Several years ago, I did a project where I ran code-coverage on roughly every nightly build of Thunderbird [1] (and I still have those results!). When I talked about this issue back then, people seemed to think that weekly was a good metric. I think Christian Holler was doing builds roughly monthly a few years ago based on an earlier version of my code-coverage-on-try technique until those builds fell apart [2]. [1] Brief aside: if you thought building mozilla code was hard, try building Mozilla code from two years ago (I was building 2008-era code in 2010)... [2] I used to dump the code coverage data to stdout and have scripts to extract them from the tbpl logs. That stopped working when mochitest-1 logs grew way too long, and it wasn't until blobber was up and running that anyone re-attempted the project. -- Joshua Cranmer Thunderbird and DXR developer Source code archæologist ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Try-based code coverage results
Filed https://bugzilla.mozilla.org/show_bug.cgi?id=1035464 for those that would like to follow along. Jonathan On 7/7/2014 3:22 PM, Jonathan Griffin wrote: So it sounds like it would be valuable to add try syntax to trigger this, as well as produce periodic reports. Most of the work needed is the same. I'll file a bug to track this; I don't have an ETA for starting work on it, but we want to get to it before things bitrot. Jonathan On 7/7/2014 12:49 PM, Joshua Cranmer wrote: On 7/7/2014 1:11 PM, Jonathan Griffin wrote: I guess a related question is, if we could run this periodically on TBPL, what would be the right frequency? Several years ago, I did a project where I ran code-coverage on roughly every nightly build of Thunderbird [1] (and I still have those results!). When I talked about this issue back then, people seemed to think that weekly was a good metric. I think Christian Holler was doing builds roughly monthly a few years ago based on an earlier version of my code-coverage-on-try technique until those builds fell apart [2]. On 7/7/2014 11:18 AM, Brian Smith wrote: Ideally, you would be able to trigger it on a try run for specific test suites or even specific subsets of tests. For example, for certificate verification changes and SSL changes, it would be great for the reviewer to be able to insist on seeing code coverage reports on the try run that preceded the review request, for xpcshell, cppunit, and GTest, without doing coverage for all test suites. To minimize the performance impact of it further, ideally it would be possible to scope the try runs to cppunit, GTest, and xpcshell tests under the security/ directory in the tree. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Try-based code coverage results
On 7/7/2014 5:25 PM, Jonathan Griffin wrote: Filed https://bugzilla.mozilla.org/show_bug.cgi?id=1035464 for those that would like to follow along. Perhaps bug 890116 is a better measure of tracking. -- Joshua Cranmer Thunderbird and DXR developer Source code archæologist ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Try-based code coverage results
I don't know how many people follow code-coverage updates in general, but I've produced relatively up-to-date code coverage results based on http://hg.mozilla.org/mozilla-central/rev/81691a55e60f, and they may be found here: http://www.tjhsst.edu/~jcranmer/m-ccov/. In contrast to earlier versions of my work, you can actually explore the coverage as delineated by specific tests, as identified by their TBPL identifier. Christian's persistent requests for me to limit the depth of the treemap view are still unresolved, because, well, at 2 AM in the morning, I just wanted to push a version that worked. The test data was generated by pushing modified configs to try and using blobber features to grab the resulting coverage data. Only Linux32/64 is used, and only opt builds are represented (it's a --disable-optimize --disable-debug kind of build), the latter because I wanted to push a version out tonight and the debug .gcda tarballs are taking way too long to finish downloading. Effectively, only xpcshell tests, and the M, M-e10s, and R groups are represented in the output data. M-e10s is slightly borked: only M-e10s(1) [I think] is shown, because, well, treeherder didn't distinguish between the five of them. A similar problem with the debug M(dt1/dt2/dt3) test suites will arise when I incorporate that data. C++ unit tests are not present because blobber doesn't run on C++ unit tests for some reason, and Jit-tests, jetpack tests, and Marionette tests await me hooking in the upload scripts to those testsuites (and Jit-tests would suffer a similar numbering problems). The individual testsuites within M-oth may be mislabeled because I can't sort names properly. There's a final, separate issue with treeherder not recording the blobber upload artifacts for a few of the runs (e.g., Linux32 opt X), even though it finished without errors and tbpl records those artifacts. So coverage data is missing for the affected run. It's also worth noting that a few test runs are mired with timeouts and excessive failures, the worst culprit being Linux32 debug where half the testsuites either had some failures or buildbot timeouts (and no data at all). If you want the underlying raw data (the .info files I prepare from every individual run's info), I can provide that on request, but the data is rather large (~2.5G uncompressed). In short: * I have up-to-date code-coverage on Linux 32-bit and Linux 64-bit. Opt is up right now; debug will be uploaded hopefully within 24 hours. * Per-test [TBPL run] level of detail is visible. * Treeherder seems to be having a bit of an ontology issue... -- Joshua Cranmer Thunderbird and DXR developer Source code archæologist ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
