Mike, rest assured you and Marcus are not the only ones. More guarantee on a stable master is a general concern. Personally I don't feel we need more control on what is in the next release, if we make unit tests and automated integration tests a priority. That is kind of a claim I do have 'the' solution, though not well cooked ;) It's going to take a while (a colleague said four or five releases) before we have a good enough test set and a smoothly running continuous integration test engine. I think we at least need the distributed Jenkins setup where you can run your own integration tests to make sure your invested logic remains intact. This of course being only part of 'all the' answers.
regards, On Tue, Sep 24, 2013 at 9:09 PM, Mike Tutkowski < mike.tutkow...@solidfire.com> wrote: > I was a bit hesitant to keep pushing this because there doesn't seem to be > a lot of support for it, but - as Marcus pointed out - I was quite alarmed > by the number and criticality of bugs checked in right before we cut our > first RC for 4.2. We simply were not ready. > > To me, it felt like something one might do before one gets out a decent > beta release. > > I certainly don't claim to have all the answers for this, but I do think we > need to develop some kind of a process whereby very few changes are made > immediately prior (like a month) to the first cut of a RC. We might even > need to discuss such changes as a community before they get checked in > (after a certain point). > > As far as master not always being usable, this is a serious problem, as > well. > > For example, I've been having trouble getting KVM to work and - in the > meanwhile - my code has fallen out of date with master over the past week > or so. However, I'm always afraid if I update from master while in the > middle of solving one problem that I'll have more problems to deal with > before I can get back to the initial problem (because something didn't work > in master). > > Again, I don't claim to have any solution for this problem, but I am happy > to help brainstorm. > > > On Tue, Sep 24, 2013 at 10:00 AM, Marcus Sorensen <shadow...@gmail.com > >wrote: > > > On Mon, Sep 23, 2013 at 1:55 PM, Animesh Chaturvedi > > <animesh.chaturv...@citrix.com> wrote: > > > > > > > > >> -----Original Message----- > > >> From: Marcus Sorensen [mailto:shadow...@gmail.com] > > >> Sent: Monday, September 23, 2013 12:25 PM > > >> To: dev@cloudstack.apache.org > > >> Subject: RE: [PROPOSAL] move away from time-based releases and/or > revamp > > >> release process > > >> > > >> On Sep 23, 2013 1:03 PM, "Animesh Chaturvedi" > > >> <animesh.chaturv...@citrix.com> > > >> wrote: > > >> > > > >> > > > >> > > > >> > > -----Original Message----- > > >> > > From: Marcus Sorensen [mailto:shadow...@gmail.com] > > >> > > Sent: Monday, September 23, 2013 11:38 AM > > >> > > To: dev@cloudstack.apache.org > > >> > > Subject: [PROPOSAL] move away from time-based releases and/or > revamp > > >> > > release process > > >> > > > > >> > > Guys, I think we are not currently in a state to handle > time-based > > >> > > releases. Until we can cut master at any time and have it > > >> > > releasable, or at least at a reasonable RC-level matching minimum > > >> > > tested requirements, it's just going to continue to be an exercise > > >> > > in frustration to cut RCs simply because we hit a deadline. > > >> > [Animesh>] David is going to propose Release Criterion up for > > >> > discussion > > >> as per his thread [1] > > >> > > >> I see that thread more about defining what minimum bar we should > always > > >> have master at in order to meet time-based releases. Its where we want > > >> to go, but not what to do in the meantime. > > > [Animesh>] His proposal is not just for master, but also for deciding > > the release exit criterion and IMO is something we should follow for > 4.3.0 > > and onwards > > > > Yes, I know. What I meant was that it will be a step toward > > stabilizing master, until we do that I'm not convinced we can adhere > > to any time-based expectation). It still doesn't fix our issue if > > we're going to insist on time-based releases, it just (from my > > undertanding) sets a bar for what is acceptable and what isn't, for > > any release. It stops the argument of "should we release with this > > bug". > > > > >> > > >> > > > > >> > > Maybe we can get away with sticking to time-based if we revamp our > > >> > > schedule and procedures, I don't know, but in light of how 4.1 > > >> > > (dragged on so long that some were seriously considering > > >> > > skipping/not releasing it with 4.2 on its heels) and 4.2 (six > rounds > > >> > > of votes so > > >> > > far) have worked it's probably worth discussing. > > >> > > > > >> > > Any suggestions on what might be better? It's been mentioned in > the > > >> > > past that it's a chicken-egg thing, many really don't try it until > > >> > > we hit an RC, which causes multiple iterations. I do agree that > many > > >> > > don't take it seriously until we start cutting artifacts, but > maybe > > >> > > we do this in a more deliberate fashion instead of jumping right > to > > >> > > the vote. After feature/code freeze, cut some alpha artifacts, > wait > > >> > > a week, cut alpha2 or some beta artifacts, etc, and then at some > > >> > > point anyone can propose that certain artifacts (or a new set of > > >> > > artifacts) be put up for a vote as an RC. This gives us a way to > > >> > > signal that we're gearing up for release and gives plenty of time > > >> > > for people to test their components, or see the [PROPOSAL] and say > > >> > > 'oh crap, I had better test my stuff', prior to cutting an RC. > > >> > > Maybe this wouldn't help in practice, but I think right now we go > > >> > > from telling the community "code is frozen, don't check anything > in > > >> > > unless its a bug fix" to "here's our RC, try it out", without a > > >> formal testing window. > > >> > > I realize the whole thing should be a testing window, but I don't > > >> > > think it's conveyed well. > > >> > > > >> > [Animesh>] After the code freeze is all the stabilization and > > >> > integration > > >> testing phase and has been documented at [2]. No one should be > waiting > > >> until the RC to test their components for the first time. It should be > > >> happening after code freeze. > > >> > > >> > > > >> > [1] http://markmail.org/thread/wlaq4zg36xnpgsjm > > >> > [2] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Releases > > >> > > > >> > > >> Got it. As mentioned I realize that the whole time there is supposed > to > > >> be testing, but its not really working that way in practice. People > are > > >> volunteers, they forget where things are, or they dont want to mess > with > > >> it unless there is an indication that its semi-stable, and then > suddenly > > >> an RC is thrown over the fence and we go through iterations of RC. By > > >> the time the RC comes through we should be done testing and just > verify > > >> that someone's last minute bug fix didn't cause a regression or > > >> something. > > > [Animesh>] RC is not thrown in it is discussed as part of the release > > schedule. After the code-freeze date everyone is expected to complete > > their integration testing by RC date. In fact I had sent numerous > reminders > > prior to the first RC starting from 2 weeks before the proposed RC date. > > > > That's not the point. The code is changing at a rapid pace. Mike, for > > example, commented on tons of critical fixes going in right up until > > the RC is cut. Then we cut some artifacts and give people 72 hours to > > test and buy off. What I'm advocating is to lengthen the process, > > and not tie it to a timeline until we have better testing that > > stabilizes our master. At that time, when people can trust master > > remotely, then maybe individuals will take the time to poke at it > > prior to RC. Maybe that's a horrible idea, but let's at least talk > > about doing something until we're stable... or do we think we can > > accomplish that in a timely fashion? > > > > I think there are a few subgroups in our team here. 1) people whose > > job it is to develop on cloudstack, but don't really use it, 2) people > > who use cloudstack daily, and only do development to bugfix and/or add > > a pet feature. There may be some overlap for some individuals. This > > process might work great for individuals whose job it is to focus on > > cloudstack every single day and are tightly integrated with the > > massive changes, but the rest of us who consume cloudstack don't > > always have time to look at the big picture and focus on the unstable > > branches. We use the releases and focus on making the stable ones > > better and/or fixing/adding our pet features, until the next stable > > one comes around. Until the development branches stabilize I don't > > believe it will work for the users, they won't get involved until the > > end. > > > > For me, personally, it's a waste of time to even look at a branch that > > probably won't work due to sweeping changes that tend to occur between > > releases. Make your core changes, add spring, replace the storage > > subsystem, whatever it is, and then I'll go back and see what it broke > > after the bugs are worked out in all of that. That's how group #2 > > thinks, in general. And right now the only indicator that we're to > > that point is when we start talking RC, at which point I have a 3 day > > window that I hopefully catch and have time to play with it. > > > > > > > > > > > > My impression from your responses Animesh is that you feel everything > > is fine as-is. I don't know how anyone could think that given what > > we've seen over the last two releases, especially you who had to cut > > six RCs. We're blowing past our "time based releases", and trying to > > push through buggy releases (for some reason). My intent was to sum up > > and focus on some of the comments I've seen over the past few weeks > > about low/sporadic RC participation, major changes going on at the > > last minute, etc. I guess I'm in the minority though, since we're the > > only ones discussing it. > > > > > > -- > *Mike Tutkowski* > *Senior CloudStack Developer, SolidFire Inc.* > e: mike.tutkow...@solidfire.com > o: 303.746.7302 > Advancing the way the world uses the > cloud<http://solidfire.com/solution/overview/?video=play> > *™* >
