at the meetup yesterday we discussed/debated ways to make sure we have high quality releases.
we wanted to put the discussion on the mailing list to see if we should make a policy change. MOTIVATION: it is clear that to have high quality releases, we need quality testing. zookeeper is such a critical service that we can't afford to put things out that we don't have high confidence in. to get quality testing we need good test cases, something we agree we all need to continue working on. this proposal is not put forth as a substitute for quality test cases. another thing that some of us put forward is that we need good pre-release testing from actual users. these are system tests that zookeeper users run in their qa environments. the great thing about these tests is that users have such diverse uses and environments that sometimes they hit cases that sometimes are hard to come up with. they also have some rather creative uses of zookeeper that we don't always imagine. the current release voting time is 3-5 days. during this time it is usually the committers and pmc that run their various checks to make sure the release looks good. however, 3-5 days isn't really long enough for customers to try things out. actually in that time period it's possible for people to miss the voting period all together given that they have their normal work to do. also, designating a release as an alpha or beta release is a confusing. does everyone know that 3.4.x is currently a beta release? PROPOSAL: for non-bugfix releases we should extend the release candidate time to 2 weeks (more?). the committers and pmc should have all their votes in in the normal 3 day voting period, but the rest of the time should be given to allow our users to validate the release themselves before releasing. PRO: major bugs tend to crop up very early on in the release, so we can catch them before the release is out. if we don't have confidence in a release, we can just extend the candidacy period. CON: if everyone just waits until the actual release to try things out, then all we have done is extended the time it takes to do a release. so, the questions are: 1) what do people think in general? and 2) if we do have a release candidate that is out there for a while before the real release, would you run it through your test environment? thanx ben
