Accidentally just sent this to Daan. ---------- Forwarded message ---------- From: Mike Tutkowski <mike.tutkow...@solidfire.com> Date: Wed, Sep 25, 2013 at 8:34 AM Subject: Re: [PROPOSAL] move away from time-based releases and/or revamp release process To: Daan Hoogland <daan.hoogl...@gmail.com>
I think that would be a really good use of a Hackathon Day at CCC. On Wed, Sep 25, 2013 at 5:27 AM, Daan Hoogland <daan.hoogl...@gmail.com>wrote: > So you are willing to spend a hackathon on that in november in Amsterdam? > > @Prasanna: can we expect you with your invalued input on this subject > there? > > I would really feel a lot of people in the community and in Citrix > would sleep better if we have this rolling more smoothly. > > On Tue, Sep 24, 2013 at 11:20 PM, Mike Tutkowski > <mike.tutkow...@solidfire.com> wrote: > > I think a distributed Jenkins setup would be great. > > > > If we had really awesome test coverage, I would be less frightened of > > last-minute checkins, as well. :) > > > > > > On Tue, Sep 24, 2013 at 3:17 PM, Daan Hoogland <daan.hoogl...@gmail.com > >wrote: > > > >> Mike, rest assured you and Marcus are not the only ones. More guarantee > on > >> a stable master is a general concern. Personally I don't feel we need > more > >> control on what is in the next release, if we make unit tests and > automated > >> integration tests a priority. That is kind of a claim I do have 'the' > >> solution, though not well cooked ;) It's going to take a while (a > colleague > >> said four or five releases) before we have a good enough test set and a > >> smoothly running continuous integration test engine. I think we at least > >> need the distributed Jenkins setup where you can run your own > integration > >> tests to make sure your invested logic remains intact. This of course > being > >> only part of 'all the' answers. > >> > >> regards, > >> > >> > >> On Tue, Sep 24, 2013 at 9:09 PM, Mike Tutkowski < > >> mike.tutkow...@solidfire.com> wrote: > >> > >> > I was a bit hesitant to keep pushing this because there doesn't seem > to > >> be > >> > a lot of support for it, but - as Marcus pointed out - I was quite > >> alarmed > >> > by the number and criticality of bugs checked in right before we cut > our > >> > first RC for 4.2. We simply were not ready. > >> > > >> > To me, it felt like something one might do before one gets out a > decent > >> > beta release. > >> > > >> > I certainly don't claim to have all the answers for this, but I do > think > >> we > >> > need to develop some kind of a process whereby very few changes are > made > >> > immediately prior (like a month) to the first cut of a RC. We might > even > >> > need to discuss such changes as a community before they get checked in > >> > (after a certain point). > >> > > >> > As far as master not always being usable, this is a serious problem, > as > >> > well. > >> > > >> > For example, I've been having trouble getting KVM to work and - in the > >> > meanwhile - my code has fallen out of date with master over the past > week > >> > or so. However, I'm always afraid if I update from master while in the > >> > middle of solving one problem that I'll have more problems to deal > with > >> > before I can get back to the initial problem (because something didn't > >> work > >> > in master). > >> > > >> > Again, I don't claim to have any solution for this problem, but I am > >> happy > >> > to help brainstorm. > >> > > >> > > >> > On Tue, Sep 24, 2013 at 10:00 AM, Marcus Sorensen < > shadow...@gmail.com > >> > >wrote: > >> > > >> > > On Mon, Sep 23, 2013 at 1:55 PM, Animesh Chaturvedi > >> > > <animesh.chaturv...@citrix.com> wrote: > >> > > > > >> > > > > >> > > >> -----Original Message----- > >> > > >> From: Marcus Sorensen [mailto:shadow...@gmail.com] > >> > > >> Sent: Monday, September 23, 2013 12:25 PM > >> > > >> To: dev@cloudstack.apache.org > >> > > >> Subject: RE: [PROPOSAL] move away from time-based releases and/or > >> > revamp > >> > > >> release process > >> > > >> > >> > > >> On Sep 23, 2013 1:03 PM, "Animesh Chaturvedi" > >> > > >> <animesh.chaturv...@citrix.com> > >> > > >> wrote: > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > -----Original Message----- > >> > > >> > > From: Marcus Sorensen [mailto:shadow...@gmail.com] > >> > > >> > > Sent: Monday, September 23, 2013 11:38 AM > >> > > >> > > To: dev@cloudstack.apache.org > >> > > >> > > Subject: [PROPOSAL] move away from time-based releases and/or > >> > revamp > >> > > >> > > release process > >> > > >> > > > >> > > >> > > Guys, I think we are not currently in a state to handle > >> > time-based > >> > > >> > > releases. Until we can cut master at any time and have it > >> > > >> > > releasable, or at least at a reasonable RC-level matching > >> minimum > >> > > >> > > tested requirements, it's just going to continue to be an > >> exercise > >> > > >> > > in frustration to cut RCs simply because we hit a deadline. > >> > > >> > [Animesh>] David is going to propose Release Criterion up for > >> > > >> > discussion > >> > > >> as per his thread [1] > >> > > >> > >> > > >> I see that thread more about defining what minimum bar we should > >> > always > >> > > >> have master at in order to meet time-based releases. Its where we > >> want > >> > > >> to go, but not what to do in the meantime. > >> > > > [Animesh>] His proposal is not just for master, but also for > deciding > >> > > the release exit criterion and IMO is something we should follow for > >> > 4.3.0 > >> > > and onwards > >> > > > >> > > Yes, I know. What I meant was that it will be a step toward > >> > > stabilizing master, until we do that I'm not convinced we can adhere > >> > > to any time-based expectation). It still doesn't fix our issue if > >> > > we're going to insist on time-based releases, it just (from my > >> > > undertanding) sets a bar for what is acceptable and what isn't, for > >> > > any release. It stops the argument of "should we release with this > >> > > bug". > >> > > > >> > > >> > >> > > >> > > > >> > > >> > > Maybe we can get away with sticking to time-based if we > revamp > >> our > >> > > >> > > schedule and procedures, I don't know, but in light of how > 4.1 > >> > > >> > > (dragged on so long that some were seriously considering > >> > > >> > > skipping/not releasing it with 4.2 on its heels) and 4.2 (six > >> > rounds > >> > > >> > > of votes so > >> > > >> > > far) have worked it's probably worth discussing. > >> > > >> > > > >> > > >> > > Any suggestions on what might be better? It's been mentioned > in > >> > the > >> > > >> > > past that it's a chicken-egg thing, many really don't try it > >> until > >> > > >> > > we hit an RC, which causes multiple iterations. I do agree > that > >> > many > >> > > >> > > don't take it seriously until we start cutting artifacts, but > >> > maybe > >> > > >> > > we do this in a more deliberate fashion instead of jumping > right > >> > to > >> > > >> > > the vote. After feature/code freeze, cut some alpha > artifacts, > >> > wait > >> > > >> > > a week, cut alpha2 or some beta artifacts, etc, and then at > some > >> > > >> > > point anyone can propose that certain artifacts (or a new > set of > >> > > >> > > artifacts) be put up for a vote as an RC. This gives us a > way to > >> > > >> > > signal that we're gearing up for release and gives plenty of > >> time > >> > > >> > > for people to test their components, or see the [PROPOSAL] > and > >> say > >> > > >> > > 'oh crap, I had better test my stuff', prior to cutting an > RC. > >> > > >> > > Maybe this wouldn't help in practice, but I think right now > we > >> go > >> > > >> > > from telling the community "code is frozen, don't check > anything > >> > in > >> > > >> > > unless its a bug fix" to "here's our RC, try it out", > without a > >> > > >> formal testing window. > >> > > >> > > I realize the whole thing should be a testing window, but I > >> don't > >> > > >> > > think it's conveyed well. > >> > > >> > > >> > > >> > [Animesh>] After the code freeze is all the stabilization and > >> > > >> > integration > >> > > >> testing phase and has been documented at [2]. No one should be > >> > waiting > >> > > >> until the RC to test their components for the first time. It > should > >> be > >> > > >> happening after code freeze. > >> > > >> > >> > > >> > > >> > > >> > [1] http://markmail.org/thread/wlaq4zg36xnpgsjm > >> > > >> > [2] > >> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Releases > >> > > >> > > >> > > >> > >> > > >> Got it. As mentioned I realize that the whole time there is > supposed > >> > to > >> > > >> be testing, but its not really working that way in practice. > People > >> > are > >> > > >> volunteers, they forget where things are, or they dont want to > mess > >> > with > >> > > >> it unless there is an indication that its semi-stable, and then > >> > suddenly > >> > > >> an RC is thrown over the fence and we go through iterations of > RC. > >> By > >> > > >> the time the RC comes through we should be done testing and just > >> > verify > >> > > >> that someone's last minute bug fix didn't cause a regression or > >> > > >> something. > >> > > > [Animesh>] RC is not thrown in it is discussed as part of the > release > >> > > schedule. After the code-freeze date everyone is expected to > complete > >> > > their integration testing by RC date. In fact I had sent numerous > >> > reminders > >> > > prior to the first RC starting from 2 weeks before the proposed RC > >> date. > >> > > > >> > > That's not the point. The code is changing at a rapid pace. Mike, > for > >> > > example, commented on tons of critical fixes going in right up until > >> > > the RC is cut. Then we cut some artifacts and give people 72 hours > to > >> > > test and buy off. What I'm advocating is to lengthen the process, > >> > > and not tie it to a timeline until we have better testing that > >> > > stabilizes our master. At that time, when people can trust master > >> > > remotely, then maybe individuals will take the time to poke at it > >> > > prior to RC. Maybe that's a horrible idea, but let's at least talk > >> > > about doing something until we're stable... or do we think we can > >> > > accomplish that in a timely fashion? > >> > > > >> > > I think there are a few subgroups in our team here. 1) people whose > >> > > job it is to develop on cloudstack, but don't really use it, 2) > people > >> > > who use cloudstack daily, and only do development to bugfix and/or > add > >> > > a pet feature. There may be some overlap for some individuals. This > >> > > process might work great for individuals whose job it is to focus on > >> > > cloudstack every single day and are tightly integrated with the > >> > > massive changes, but the rest of us who consume cloudstack don't > >> > > always have time to look at the big picture and focus on the > unstable > >> > > branches. We use the releases and focus on making the stable ones > >> > > better and/or fixing/adding our pet features, until the next stable > >> > > one comes around. Until the development branches stabilize I don't > >> > > believe it will work for the users, they won't get involved until > the > >> > > end. > >> > > > >> > > For me, personally, it's a waste of time to even look at a branch > that > >> > > probably won't work due to sweeping changes that tend to occur > between > >> > > releases. Make your core changes, add spring, replace the storage > >> > > subsystem, whatever it is, and then I'll go back and see what it > broke > >> > > after the bugs are worked out in all of that. That's how group #2 > >> > > thinks, in general. And right now the only indicator that we're to > >> > > that point is when we start talking RC, at which point I have a 3 > day > >> > > window that I hopefully catch and have time to play with it. > >> > > > >> > > > > >> > > > > >> > > > >> > > My impression from your responses Animesh is that you feel > everything > >> > > is fine as-is. I don't know how anyone could think that given what > >> > > we've seen over the last two releases, especially you who had to cut > >> > > six RCs. We're blowing past our "time based releases", and trying to > >> > > push through buggy releases (for some reason). My intent was to sum > up > >> > > and focus on some of the comments I've seen over the past few weeks > >> > > about low/sporadic RC participation, major changes going on at the > >> > > last minute, etc. I guess I'm in the minority though, since we're > the > >> > > only ones discussing it. > >> > > > >> > > >> > > >> > > >> > -- > >> > *Mike Tutkowski* > >> > *Senior CloudStack Developer, SolidFire Inc.* > >> > e: mike.tutkow...@solidfire.com > >> > o: 303.746.7302 > >> > Advancing the way the world uses the > >> > cloud<http://solidfire.com/solution/overview/?video=play> > >> > *™* > >> > > >> > > > > > > > > -- > > *Mike Tutkowski* > > *Senior CloudStack Developer, SolidFire Inc.* > > e: mike.tutkow...@solidfire.com > > o: 303.746.7302 > > Advancing the way the world uses the > > cloud<http://solidfire.com/solution/overview/?video=play> > > *™* > -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloud<http://solidfire.com/solution/overview/?video=play> *™* -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloud<http://solidfire.com/solution/overview/?video=play> *™*
