I completely agree with Jeff Gilbert on this one. I think we should try to coalesce -better-. I just checked the current state of mozilla-inbound and it doesn't feel any of the current patch really need their own set of tests because they're are not time sensitive or sufficiently complex. Right now developers are asked to create bugs for their own change with their own patch. This leads to a lot of little patches being landed by individual developers which seems to reflect the current state of mozilla-inbound.
Perhaps we should instead promote checkin-needed (or a similar simple) to coalesce simple changes together. Opting into this means that your patch may take significantly longer to get merged if it's landed with another bad patch and should only be used when that's acceptable. Right now developers with commit access are not encouraged to make use of checkin-needed AFAIK. If we started recommending against individual landings for simple changes, and improved the process, we could probably significantly cut the number of tests jobs by cutting the number of pushes. On Tue, Aug 19, 2014 at 3:57 PM, Jeff Gilbert <jgilb...@mozilla.com> wrote: > I would actually say that debug tests are more important for continuous > integration than opt tests. At least in code I deal with, we have a ton of > asserts to guarantee behavior, and we really want test coverage with these > via CI. If a test passes on debug, it should almost certainly pass on opt, > just faster. The opposite is not true. > > "They take a long time and then break" is part of what I believe caused us to > not bother with debug testing on much of Android and B2G, which we still > haven't completely fixed. It should be unacceptable to ship without CI on > debug tests, but here we are anyways. (This is finally nearly fixed, though > there is still some work to do) > > I'm not saying running debug tests less often is on the same scale of bad, > but I would like to express my concerns about heading in that direction. > > -Jeff > > ----- Original Message ----- > From: "Jonathan Griffin" <jgrif...@mozilla.com> > To: dev-platform@lists.mozilla.org > Sent: Tuesday, August 19, 2014 12:22:21 PM > Subject: Experiment with running debug tests less often on mozilla-inbound > the week of August 25 > > Our pools of test slaves are often at or over capacity, and this has the > effect of increasing job coalescing and test wait times. This, in turn, > can lead to longer tree closures caused by test bustage, and can cause > try runs to be very slow to complete. > > One of the easiest ways to mitigate this is to run tests less often. > > To assess the impact of doing this, we will be performing an experiment > the week of August 25, in which we will run debug tests on > mozilla-inbound on most desktop platforms every other run, instead of > every run as we do now. Debug tests on linux64 will continue to run > every time. Non-desktop platforms and trees other than mozilla-inbound > will not be affected. > > This approach is based on the premise that the number of debug-only > platform-specific failures on desktop is low enough to be manageable, > and that the extra burden this imposes on the sheriffs will be small > enough compared to the improvement in test slave metrics to justify the > cost. > > While this experiment is in progress, we will be monitoring job > coalescing and test wait times, as well as impacts on sheriffs and > developers. If the experiment causes sheriffs to be unable to perform > their job effectively, it can be terminated prematurely. > > We intend to use the data we collect during the experiment to inform > decisions about additional tooling we need to make this or a similar > plan permanent at some point in the future, as well as validating the > premise on which this experiment is based. > > After the conclusion of this experiment, a follow-up post will be made > which will discuss our findings. If you have any concerns, feel free to > reach out to me. > > Jonathan > > _______________________________________________ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > _______________________________________________ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
