Graphics in particular is plagued by non-cross-platform code. Debug coverage on Linux gives us no practical coverage for our windows, mac, android, or b2g code. Maybe this is better solved with reviving the Graphics branch, however.
-Jeff ----- Original Message ----- From: "Chris AtLee" <cat...@mozilla.com> To: "Ehsan Akhgari" <ehsan.akhg...@gmail.com> Cc: "Jonathan Griffin" <jgrif...@mozilla.com>, "Jeff Gilbert" <jgilb...@mozilla.com>, dev-platform@lists.mozilla.org Sent: Wednesday, August 20, 2014 9:02:14 AM Subject: Re: Experiment with running debug tests less often on mozilla-inbound the week of August 25 On 18:25, Tue, 19 Aug, Ehsan Akhgari wrote: >On 2014-08-19, 5:49 PM, Jonathan Griffin wrote: >>On 8/19/2014 2:41 PM, Ehsan Akhgari wrote: >>>On 2014-08-19, 3:57 PM, Jeff Gilbert wrote: >>>>I would actually say that debug tests are more important for >>>>continuous integration than opt tests. At least in code I deal with, >>>>we have a ton of asserts to guarantee behavior, and we really want >>>>test coverage with these via CI. If a test passes on debug, it should >>>>almost certainly pass on opt, just faster. The opposite is not true. >>>> >>>>"They take a long time and then break" is part of what I believe >>>>caused us to not bother with debug testing on much of Android and >>>>B2G, which we still haven't completely fixed. It should be >>>>unacceptable to ship without CI on debug tests, but here we are >>>>anyways. (This is finally nearly fixed, though there is still some >>>>work to do) >>>> >>>>I'm not saying running debug tests less often is on the same scale of >>>>bad, but I would like to express my concerns about heading in that >>>>direction. >>> >>>I second this. I'm curious to know why you picked debug tests for >>>this experiment. Would it not make more sense to run opt tests on >>>desktop platforms on every other run? >>> >>Just based on the fact that they take longer and thus running them less >>frequently would have a larger impact. If there's a broad consensus >>that debug runs are more valuable, we could switch to running opt tests >>less frequently instead. > >Yep, the debug tests indeed take more time, mostly because they run >more checks. :-) The checks in opt builds are not exactly a subset >of the ones in debug builds, but they are close. Based on that, I >think running opt tests on every other push is a more conservative >one, and I support it more. That being said, for this one week >limited trial, given that the sheriffs will help backfill the skipped >tests, I don't care very strongly about this, as long as it doesn't >set the precedence that we can ignore debug tests! I'd like to highlight that we're still planning on running debug linux64 tests for every build. This is based on the assumption that debug-specific failures are generally cross-platform failures as well. Does this help alleviate some concern? Or is that assumption just plain wrong? Cheers, Chris _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
