"The historical trend with the Cassandra codebase has been to test
throw the code over the wall, and get feedback from people putting it in
prod who run into issues."
At the summit Brandon and a couple others were making fun over range
tombstones from thrift
I added the thrift support based on code already in trunk. But there was
something ugly bit in there
and far on down the line someone else stuck with an edge case and had to
fix it. Now, I actually added a number
of tests, unit test, and nosetests. I am sure the range tombstones also had
their own set of tests at the storage level.
So as Brandon was making fun of me, I was thinking to myself, "Well I did
not make the bug, I just made it possible for others to find it! So I am
The next time I submit a thrift patch I am going to write 5x the unit tests
On Fri, Sep 16, 2016 at 11:18 AM, Jonathan Haddad <j...@jonhaddad.com> wrote:
> I've worked on a few projects where we've had a branch that new stuff went
> in before merging to master / trunk. What you've described reminds me a
> lot of git-flow (http://nvie.com/posts/a-successful-git-branching-model/)
> although not quite the same. I'll be verbose in this email to minimize the
> reader's assumptions.
> The goals of the release cycle should be (in descending order of priority):
> 1. Minimize bugs introduced through change
> 2. Allow the codebase to iterate quickly
> 3. Not get caught up in a ton of back porting bug fixes
> There is significant benefit to having a releasable trunk. This is
> different from a trunk which is constantly released. A releasable trunk
> simply means all tests should *always* pass and PMC & committers should
> feel confident that they could actually put it in prod for a project that
> actually matters. Having it always be releasable (all tests pass, etc)
> means people can at least test the DB on sample data or evaluate it before
> the release happens, and get feedback to the team when there are bugs.
> This is a different mentality from having a "features" branch, where it's
> implied that at times it's acceptable that it not be stable. The
> historical trend with the Cassandra codebase has been to test minimally,
> throw the code over the wall, and get feedback from people putting it in
> prod who run into issues. In my experience I have found a general purpose
> "features" branch to result in poorly quality codebases. It's shares a lot
> of the same problems as the 1+ year release cycle did previously, with
> things getting merged in and then an attempt to stabilize later.
> Improving the state of testing in trunk will catch more bugs, satisfying
> #1, which naturally leads to #2, and by reducing bugs before they get
> released #3 will happen over time.
> My suggestion for a *supported* feature release every 3 months (could just
> as well be 4 or 6) mixed with Benedict's idea of frequent non-supported
> releases (tagged as alpha). Supported releases should get ~6 months worth
> of bug fixes, which if done right, will decrease over time due to a
> hopefully more stable codebase. I 100% agree with Mick that semver makes
> sense here, it's not just for frameworks. Major.Minor.Patch is well
> understood and is pretty standard throughout the world, I don't think we
> need to reinvent versioning.
> Release every 3 months
> Support for 6
> Keep a stable trunk
> New features get merged into trunk but the standard for code quality and
> testing needs to be property defined as something closer to "production
> ready" rather than "let the poor user figure it out"
> On Fri, Sep 16, 2016 at 3:05 AM Sylvain Lebresne <sylv...@datastax.com>
> > As probably pretty much everyone at this point, I agree the tick-tock
> > experiment
> > isn't working as well as it should and that it's probably worth course
> > correcting. I happen to have been thinking about this quite a bit already
> > as it
> > turns out so I'm going to share my reasoning and suggestion below, even
> > though
> > it's going to be pretty long, in the hope it can be useful (and if it
> > isn't, so
> > be it).
> > My current thinking is that a good cycle should accommodate 2 main
> > constraints:
> > 1) be useful for users
> > 2) be realistic/limit friction on the development side
> > and let me develop what I mean by both points slightly first.
> > I think users mostly want 2 things out of the release schedule: they
> want a
> > clearly labeled stable branch to know what they should run into
> > and
> > they want new features and improvements. Let me clarify that different
> > users
> > will want those 2 in different degrees and with variation over time, but
> > believe it's mainly some combination of those. On the development side, I
> > don't
> > think it's realistic to expect more than 2/3 branches/series to be
> > supported at
> > any one time (not going to argue that, let's call it a professional
> > opinion). I
> > also think accumulating new work for any meaningful length of time before
> > releasing, as we used to do, is bad as it pushes devs to rush things to
> > meet a
> > given release deadline as they don't want to wait for the next one. This
> > turn
> > impacts quality and creates unnecessary drama. It's also good imo to
> have a
> > clear policy regarding where a given work can go (having to debate on
> > ticket on which branch it should go is a waste of dev time).
> > With those "goals" in mind, I'll note that:
> > - the fixed _and_ short cadence of tick-tock is imo very good, in
> > particular in
> > (but not limited to) avoiding the 'accumulate unreleased stuffs'
> > - we have ample evidence that stuffs don't get truly stable until they
> > only
> > bug fixes for a few months. Which doesn't mean at all that we shouldn't
> > continue to make progress on increasing the quality of new code btw.
> > - Simple is also a great quality of a release cycle. I think we should
> > to
> > define what's truly important to achieve (my opinion on that is above)
> > and do
> > the simplest thing that achieve that. This does imply the release cycle
> > won't make the coffee, but that's alright, it probably shouldn't
> > In light of all this, my suggesting for a release cycle woud be:
> > - To have 3 branches: 'features', 'testing' and 'stable', with an X month
> > rotation: 'features' becomes 'testing' after X months and then 'stable'
> > after
> > X more, before getting EOL X months later.
> > - The feature branch gets everything. The testing branch only gets bug
> > fixes.
> > The stable branch only gets critical bug fixes. And imo, we should be
> > very
> > strict on this (I acknowledge that there is sometimes a bit of
> > subjectivity on
> > whether something is a bug or an improvement, and if it's critical or
> > not, but
> > I think it's not that hard to get consensus if we're all reasonable
> > (though it
> > might worth agreeing on some rough but written guideline upfront)).
> > - We release on a short and fixed cadence of Y month(s) for both the
> > feature and
> > testing branch. For the stable branch, given that it already had X
> > of
> > only bug fixes during the testing phase, one can hope critical fixes
> > be
> > fairly rare, less than 1 per Y period on average). Further, it's
> > to
> > be stable and fixes are supposed to be critical, so doing hot-fix
> > releases
> > probably makes the most sense (though it probably only work if we're
> > indeed
> > strict on what is considered critical).
> > And that's about it. I think it would believably achieve stability (with
> > clear
> > label on which releases are stable), but also provide new features and
> > improvements quickly for those that wants that. The testing phase is imo
> > necessary intermediate step to get the stable one.
> > On thing to define is X and Y. For Y (the cadence of feature/testing), I
> > think 1
> > or 2 months are the only options that make sense (less than 1 month is
> > fast,
> > and more than 2 months is imo starting to get too long). For X, that's
> > debatable but it's open-source and we should recognize volunteers
> > don't want to maintain things for too long either. My 2 is that 6 or 8
> > months
> > are probably the best options here.
> > We'd also have to put some numbering scheme on top of that, but that's
> > really the important part (the meaning is in the branch labels, not the
> > numbers). To give just one possible option (and assuming X=6, Y=1), in
> > January
> > 2017 we could cut 4.0 as the start of both 'feature' and 'testing'. We'd
> > then
> > have 4.1, 4.2, ... on the 'feature' branch, and 4.0.1, 4.0.2, ... on the
> > testing
> > branch for the next 6 months. In July, we'd switch from 4.5 to 5.0, with
> > that
> > becoming the new 'feature' and 'testing' base. At the same time, we'd cut
> > 4.0.6 from 4.0.5 as the new 'stable' branch. Hot-fix on that stable
> > would
> > be versioned 184.108.40.206, 220.127.116.11 and so on.
> > Of course, there can be variations on all this. We could have a
> different X
> > and/or Y for the different branches to name just one. I'm just trying to
> > abide
> > by the "simpler is better" rule though.
> > Lastly, I'll note that the versioning wouldn't use semver. I like semver
> > alright
> > but I think it's more suited to libraries than something like Cassandra.
> > Amongst
> > other things, it strongly rely on a fine grained notion of backward
> > compatibility which I think is ill-defined for Cassandra in general.
> > Sylvain
> > On Fri, Sep 16, 2016 at 4:33 AM, Mick Semb Wever <m...@thelastpickle.com
> > wrote:
> > > Totally agree with all the frustrations felt by Jon here.
> > >
> > >
> > > TL;DR
> > > Here's a proposal for 4.0 and beyond: that is puts together the
> > > from Benedict, Jon, Tyler, Jeremy, and Ed;
> > >
> > > - keep bimonthly feature releases,
> > > - revert from tick-tock to SemVer numbering scheme,
> > > - during the release vote also vote on the quality label (feature
> > branches
> > > start with a 'Alpha' and the first patch release as 'Beta'),
> > > - accept that every feature release isn't by default initially
> > supported,
> > > and its branch might never be,
> > > - maintain 3 'GA' branches at any one time,
> > > - accept that it's not going to be the oldest GA branches that
> > necessarily
> > > reach EOL first.
> > >
> > >
> > > Background and rationale…
> > >
> > > IMO the problem with Tick-Tock is that it introduces two separate
> > concepts:
> > > - incremental development, and
> > > - limiting patch releases.
> > >
> > > The first concept: having bimonthly tocks; made C* development more
> > > incremental. A needed improvement.
> > > No coincidence, at the same time as tick-tock was introduced, there was
> > > also a lot of effort being put into testing and a QA framework.
> > > From this we've seen a lot of fantastic features incrementally added to
> > C*!
> > >
> > > The second concept: having bimonthly ticks; limited C* to having only
> > > patch release per tock release.
> > > The only real benefit to this was to reduce the effort involved in
> > > maintenance, required because of the more frequent tock releases.
> > > The consequence is instability has gone bananas, as Jon clearly
> > > demonstrates. Someone went and let the monkey out.
> > >
> > > A quick comparison of before to tick-tock:
> > >
> > > * Before tick-tock: against 6-12 months of development it took a
> > > time-frame of 3-6 months and 6+ patch releases to stabilise C*.
> > >
> > > * After tick-tock: against 2 months of development we could have
> > > expected the same time-frame of 3-6 months (because adoption is
> > by
> > > users, not developers) and *over* this period 1-2 patch releases to
> > > stabilise. It seemed to have been a fools errand to force this to 1
> > > release after only one month. It seems that the notion of incremental
> > > development was applied for the developers where-as the waterfall model
> > was
> > > applied to QA in production for the users. (note: all this is not
> > > into account advantages of incremental development, an improved QA
> > > framework, and a move towards a stable-master.)
> > >
> > > The question remains to how many of these releases can the community
> > afford
> > > to support. And being realistic much of this effort relies upon the
> > > commercial entities around the community. For example having 1 year of
> > > support means having to support 6 feature releases, and there's
> > > not the people power to do that. It also means that in effect any
> > > is actually only supported for 6-9 months, since it took 3-6 for it to
> > get
> > > to production-ready.
> > >
> > > A typical Apache release process is that each new major release gets
> > voted
> > > on as only 'Alpha' or 'Beta'. As patch releases are made it is
> > ascertained
> > > whether enough people are using it (eg in production) and the quality
> > label
> > > appropriately raised to either 'Beta' or 'GA'. The quality label can
> > > proposed in the vote or left to be voted upon by everyone. The quality
> > > label is itself not part of the version number, so that the version
> > number
> > > can follow strict SemVer.
> > >
> > > Then the community can say, for example, it supports 3 'GA' branches.
> > This
> > > permits some major releases to never make it to GA, and others to hang
> > over
> > > for a bit longer. It's something that the community gets a feel for by
> > > appreciating the users and actors around it. The number of branches
> > > supported depends on what the community can sustain (including the new
> > > non-GA branches). The community also becomes a bit more honest about
> > > quality of x.y.0 releases.
> > >
> > > The proposal is an example that embraces incremental development and
> > > release-often mentality, while keeping a realistic and flexible
> > to
> > > how many branches can be supported. The cost of supporting branches is
> > > still very real, and pushing for a stable master means no feature
> > is
> > > cut without passing everything in the QA framework and 100% belief that
> > it
> > > can be put into a user's production. That is there's not a return to
> > > thinking about feature branches as a place for ongoing stabilisation
> > > efforts, just because they have a 'Alpha/Beta' label. The onus of work
> > > put upon the developer having to maintain branches for features
> > > for master, and not on the community having to stabilise and support
> > > feature branches.
> > >
> > > BTW has anyone figured out whether it's the tick or the tock that
> > > represents the feature release?? I probably got it wrong here :-)
> > >
> > >
> > > ~mck
> > >