On Fri, Nov 29, 2013 at 09:19:33PM +0000, Hausmann Simon wrote: > Once scenario I'm worried about is when a less active module (say qtscript) > has > a regression test failure due to a change in an active module (qtbase) and > it's > only visible in the qt5.git integration (because nobody submits a change to > qtscript). > > We will see the failure still, but we will also be very very tempted to still > cut a release from qt5.git because of schedules etc. , despite the failure.
That's not really different from severe performance regressions, or regressions that hit most user setups, but "accidentally" not the CI machines etc etc. As an example, for a bit more than two weeks now I have quite a few test cases failing locally which happily pass CI. _Of course_ this means that something is broken with my local setup, especially since this happened after a distro upgrade, but what system makes sure that the Real World is more similar to what the CI system sees to what my system sees? I can easily produce dozens of examples from the past where fundamentally wrong changes got CI approval, as well as examples of case where good changes got rejected. There are examples of unwanted behaviour changes that passed CI because the same change also changed the auto-test, etc. No CI system is infallible, and if we accept that, it's better to have a way to recover quickly, and not wait a full day, or two, to get "obvious" fixes in. > Currently we have a mechanism in place that makes it very hard to release in > such a situation. I am afraid this is a misconception. And even if it was not, the benefits would have to be weighed against the costs. Not every CI failure is fatal for the release, and CI checks only a small fraction of possible failures. This should be handled at face value: an advice, strong advice even, but not as a final judgement. > We are removing this protection and am replacing an automated > mechanism with a human gate keeper. Indeed. And that's a good thing, since it allows a balanced decision. > I'm not against trying this, but we have to be very careful I think, more > discipline will be required from the release side or else we will release with > known regressions. Not different from now. Andre' _______________________________________________ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
