Stepan Mishura wrote: > Sure this approach makes sense and I think we should accept and follow > it. I see only one issue here - it lets instabilities get accumulated > and present unnoticed (ignored?) in the code for some period of time. > This may result that minor update can have unintended consequences.
Apologies for not being so clear, we should continue to ensure that the automated tests and pre-commit tests are run as we do today to ensure day-to-day stability. Of course, as work progresses there may be regressions that require a commit to be backed-out (or better still, fixed). The stability period approaching a Milestone is a pact that only stability enhancing fixes are applied to ensure there is no last minute disruption during the enhanced testing and declaration of the release. > Currently if we identify a regression we try to find a guilty commit > and to fix it or do rollback. I think it is the right way – we keep > code base in a good shape and don't let a number of known problems to > grow. This approach showed its efficiency and the only thing I can do > here is only encourage all contributors to run all available tests > after doing any non-trivial change. Agreed. > But it seems for intermittent > failures the approach with running all testing scenarios doesn't work > well – usually they are not immediately detected. And it's hard to > find guilty update after a long time so we tend to put such tests to > exclude list. Right, and we should triage these and draw extra focus on them during the stability pass. > I'd like to propose the next approach that may help us to know about > instabilities: develop (or take existing one, for example, Eclipse > hello world) a scenario for testing stability and configure CC to run > it at all times. The stability scenario must be the only one scenario > for CC; it must be short (no longer then an hour), test JRE in stress > conditions and cover most of functionality. If the scenario fails then > all newly committed updates are subject for investigation and fix (or > rollback). None from me. Regards, Tim
