On 17:37, Sat, 22 Feb, L. David Baron wrote:
On Saturday 2014-02-22 15:57 -0800, Gregory Szorc wrote:On Feb 22, 2014, at 8:18, Kyle Huey <[email protected]> wrote: > If you needed another reason to follow the style guide: > https://www.imperialviolet.org/2014/02/22/applebug.html >Code coverage would have caught this as well. The time investment into 100% line and branch coverage is debatable. But you can't argue that code coverage has its place, especially for high-importance code such as crypto. AFAIK, our automation currently does not collect code coverage from any test suite. Should that change?There was some automation running code coverage reports (with gcov, I think) for at least reftests + mochitests for an extended period of time; I found it useful for improving style system test coverage while it was running. I'm not sure how strong our commitment is to keeping such tests running, though; I frequently have to defend the tests against people who want to disable them because they take a long time (i.e., a long time for a single test file, which sometimes leads the tests to approach the per-file timeouts on slow VMs) or because they happen to exhibit the latest JIT crash frequently because they run a lot of code. I'm worried we're moving to a model where tests need to have active defenders to keep them running (even though that isn't how features on the Web platform work), because we blame the old test rather than the new regression.
Tests need owners just like any other piece of code.One of the big problems with the previous code coverage reports was that they were failing to run, and nobody was stepping up to fix them. When we're resource constrained, it's a big waste of resources to run things that are broken and nobody is working on fixing.
Slow tests are a slightly different issue. If you're adding a test that takes 60s to run, you're saying you think it's important enough to make all other developers wait another minute for their test runs to complete locally. You're saying it's worthwhile to spend an extra minute per push per platform, to delay all future landings and merges by an extra minute per push. At ~249 pushes per day and at least 18 test platforms, you're adding at least 74 hours of additional machine time per day.
I know the cost of *not* testing code is even higher than this! It's even more expensive and painful to track down regressions after the fact. This doesn't mean we can't put more effort into writing efficient tests.
Cheers, Chris
signature.asc
Description: Digital signature
_______________________________________________ dev-platform mailing list [email protected] https://lists.mozilla.org/listinfo/dev-platform
