Re: Cassandra 2.2, 3.0, and beyond

[Robert Coli]

On Thu, Jun 11, 2015 at 6:56 PM, Mohammed Guller 
wrote:

>  By that logic, 2.1.0  should have been somewhat as stable as 2.0.10 (the
> last release of 2.0.x branch before 2.1.0). However, we found out that it
> took almost 9 months for 2.1.x series to become stable and suitable for
> production. Going by past history, I am worried that it may take the same
> time for 2.2 to become stable.
>

The instability of initial point releases is a significant part of the
motivation for the new release cadence.[1] If new versions continued to
take just as long to be production ready, the new process would have failed
at one of its major goals...

For the record, I agree with the reasoning in the linked post and am
cautiously optimistic about the effect it will have on the stability of
released versions. :D

=Rob
[1]
http://mail-archives.apache.org/mod_mbox/cassandra-dev/201503.mbox/%3CCALdd-zjAyiTbZksMeq2LxGwLF5LPhoi_4vsjy8JBHBRnsxH=8...@mail.gmail.com%3E
"
Unfortunately, even after DataStax hired half a dozen full-time test
engineers, 2.1.0 continued the proud tradition of being unready for
production use, with "wait for .5 before upgrading" once again looking like
a good guideline.

I’m starting to think that the entire model of “write a bunch of new
features all at once and then try to stabilize it for release” is broken.
We’ve been trying that for years and empirically speaking the evidence is
that it just doesn’t work, either from a stability standpoint or even just
shipping on time.
...
So, I’d like to try something different.  I think we were on the right
track with shorter releases with more compatibility.  But I’d like to throw
in a twist.  Intel cuts down on risk with a “tick-tock” schedule for new
architectures and process shrinks instead of trying to do both at once. We
can do something similar here:

One month releases.  Period.  If it’s not done, it can wait.
*Every other release only accepts bug fixes.*

By itself, one-month releases are going to dramatically reduce the
complexity of testing and debugging new releases -- and bugs that do slip
past us will only affect a smaller percentage of users, avoiding the “big
release has a bunch of bugs no one has seen before and pretty much everyone
is hit by something” scenario.  ***But by adding in the second rule, I
think we have a real chance to make a quantum leap here: stable,
production-ready
releases every two months.***
"

(*** emphasis mine)

Re: Cassandra 2.2, 3.0, and beyond

[Jonathan Ellis]

We've started using the docs-impacting label

to make it easier for the technical writers to keep up, but otherwise we're
not planning any major changes.

On Thu, Jun 11, 2015 at 4:50 AM, Daniel Compton <
daniel.compton.li...@gmail.com> wrote:

> Hi Jonathan
>
> Does documentation fit into the new monthly releases and definition of
> done as well, or is that part of another process? I didn't see any mention
> of it in the docs, though I may have missed it.
>
> On Thu, 11 Jun 2015 at 9:10 pm Stefan Podkowinski <
> stefan.podkowin...@1und1.de> wrote:
>
>> > We are also extending our backwards compatibility policy to cover all
>> 3.x releases: you will be able to upgrade seamlessly from 3.1 to 3.7, for
>> instance, including cross-version repair.
>>
>> What will be the EOL policy for releases after 3.0? Given your example,
>> will 3.1 still see bugfixes at this point when I decide to upgrade to 3.7?
>>
> --
> --
> Daniel
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder, http://www.datastax.com
@spyced

RE: Cassandra 2.2, 3.0, and beyond

[Mohammed Guller]

By that logic, 2.1.0  should have been somewhat as stable as 2.0.10 (the last 
release of 2.0.x branch before 2.1.0). However, we found out that it took 
almost 9 months for 2.1.x series to become stable and suitable for production. 
Going by past history, I am worried that it may take the same time for 2.2 to 
become stable.

Mohammed

From: graham sanderson [mailto:gra...@vast.com]
Sent: Thursday, June 11, 2015 6:34 PM
To: user@cassandra.apache.org
Subject: Re: Cassandra 2.2, 3.0, and beyond

I think the point is that 2.2 will replace 2.1.x + (i.e. the done/safe bits of 
3.0 are included in 2.2).. so 2.2.x and 2.1.x are somewhat synonymous.

On Jun 11, 2015, at 8:14 PM, Mohammed Guller 
mailto:moham...@glassbeam.com>> wrote:

Considering that 2.1.6 was just released and it is the first “stable” release 
ready for production in the 2.1 series, won’t it be too soon to EOL 2.1.x when 
3.0 comes out in September?

Mohammed

From: Jonathan Ellis [mailto:jbel...@gmail.com]
Sent: Thursday, June 11, 2015 10:14 AM
To: user
Subject: Re: Cassandra 2.2, 3.0, and beyond

As soon as 8099 is done.

On Thu, Jun 11, 2015 at 11:53 AM, Pierre Devops 
mailto:pierredev...@gmail.com>> wrote:
Hi,

3.x beta release date ?

2015-06-11 16:21 GMT+02:00 Jonathan Ellis 
mailto:jbel...@gmail.com>>:
3.1 is EOL as soon as 3.3 (the next bug fix release) comes out.

On Thu, Jun 11, 2015 at 4:10 AM, Stefan Podkowinski 
mailto:stefan.podkowin...@1und1.de>> wrote:
> We are also extending our backwards compatibility policy to cover all 3.x 
> releases: you will be able to upgrade seamlessly from 3.1 to 3.7, for 
> instance, including cross-version repair.

What will be the EOL policy for releases after 3.0? Given your example, will 
3.1 still see bugfixes at this point when I decide to upgrade to 3.7?



--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder, http://www.datastax.com<http://www.datastax.com/>
@spyced




--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder, http://www.datastax.com<http://www.datastax.com/>
@spyced

Re: Cassandra 2.2, 3.0, and beyond

[graham sanderson]

I think the point is that 2.2 will replace 2.1.x + (i.e. the done/safe bits of 
3.0 are included in 2.2).. so 2.2.x and 2.1.x are somewhat synonymous.

> On Jun 11, 2015, at 8:14 PM, Mohammed Guller  wrote:
> 
> Considering that 2.1.6 was just released and it is the first “stable” release 
> ready for production in the 2.1 series, won’t it be too soon to EOL 2.1.x 
> when 3.0 comes out in September?
>  
> Mohammed
>  
> From: Jonathan Ellis [mailto:jbel...@gmail.com <mailto:jbel...@gmail.com>] 
> Sent: Thursday, June 11, 2015 10:14 AM
> To: user
> Subject: Re: Cassandra 2.2, 3.0, and beyond
>  
> As soon as 8099 is done.
>  
> On Thu, Jun 11, 2015 at 11:53 AM, Pierre Devops  <mailto:pierredev...@gmail.com>> wrote:
> Hi,
>  
> 3.x beta release date ?
>  
> 2015-06-11 16:21 GMT+02:00 Jonathan Ellis  <mailto:jbel...@gmail.com>>:
> 3.1 is EOL as soon as 3.3 (the next bug fix release) comes out.
>  
> On Thu, Jun 11, 2015 at 4:10 AM, Stefan Podkowinski 
> mailto:stefan.podkowin...@1und1.de>> wrote:
> > We are also extending our backwards compatibility policy to cover all 3.x 
> > releases: you will be able to upgrade seamlessly from 3.1 to 3.7, for 
> > instance, including cross-version repair.
> 
> What will be the EOL policy for releases after 3.0? Given your example, will 
> 3.1 still see bugfixes at this point when I decide to upgrade to 3.7?
> 
> 
> 
> -- 
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder, http://www.datastax.com <http://www.datastax.com/>
> @spyced
>  
> 
> 
> 
> -- 
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder, http://www.datastax.com <http://www.datastax.com/>
> @spyced



smime.p7s
Description: S/MIME cryptographic signature

RE: Cassandra 2.2, 3.0, and beyond

[Mohammed Guller]

Considering that 2.1.6 was just released and it is the first “stable” release 
ready for production in the 2.1 series, won’t it be too soon to EOL 2.1.x when 
3.0 comes out in September?

Mohammed

From: Jonathan Ellis [mailto:jbel...@gmail.com]
Sent: Thursday, June 11, 2015 10:14 AM
To: user
Subject: Re: Cassandra 2.2, 3.0, and beyond

As soon as 8099 is done.

On Thu, Jun 11, 2015 at 11:53 AM, Pierre Devops 
mailto:pierredev...@gmail.com>> wrote:
Hi,

3.x beta release date ?

2015-06-11 16:21 GMT+02:00 Jonathan Ellis 
mailto:jbel...@gmail.com>>:
3.1 is EOL as soon as 3.3 (the next bug fix release) comes out.

On Thu, Jun 11, 2015 at 4:10 AM, Stefan Podkowinski 
mailto:stefan.podkowin...@1und1.de>> wrote:
> We are also extending our backwards compatibility policy to cover all 3.x 
> releases: you will be able to upgrade seamlessly from 3.1 to 3.7, for 
> instance, including cross-version repair.

What will be the EOL policy for releases after 3.0? Given your example, will 
3.1 still see bugfixes at this point when I decide to upgrade to 3.7?



--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder, http://www.datastax.com
@spyced




--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder, http://www.datastax.com
@spyced

Re: Cassandra 2.2, 3.0, and beyond

[Jonathan Ellis]

As soon as 8099 is done.

On Thu, Jun 11, 2015 at 11:53 AM, Pierre Devops 
wrote:

> Hi,
>
> 3.x beta release date ?
>
> 2015-06-11 16:21 GMT+02:00 Jonathan Ellis :
>
>> 3.1 is EOL as soon as 3.3 (the next bug fix release) comes out.
>>
>> On Thu, Jun 11, 2015 at 4:10 AM, Stefan Podkowinski <
>> stefan.podkowin...@1und1.de> wrote:
>>
>>> > We are also extending our backwards compatibility policy to cover all
>>> 3.x releases: you will be able to upgrade seamlessly from 3.1 to 3.7, for
>>> instance, including cross-version repair.
>>>
>>> What will be the EOL policy for releases after 3.0? Given your example,
>>> will 3.1 still see bugfixes at this point when I decide to upgrade to 3.7?
>>>
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder, http://www.datastax.com
>> @spyced
>>
>
>


-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder, http://www.datastax.com
@spyced

Re: Cassandra 2.2, 3.0, and beyond

[Pierre Devops]

Hi,

3.x beta release date ?

2015-06-11 16:21 GMT+02:00 Jonathan Ellis :

> 3.1 is EOL as soon as 3.3 (the next bug fix release) comes out.
>
> On Thu, Jun 11, 2015 at 4:10 AM, Stefan Podkowinski <
> stefan.podkowin...@1und1.de> wrote:
>
>> > We are also extending our backwards compatibility policy to cover all
>> 3.x releases: you will be able to upgrade seamlessly from 3.1 to 3.7, for
>> instance, including cross-version repair.
>>
>> What will be the EOL policy for releases after 3.0? Given your example,
>> will 3.1 still see bugfixes at this point when I decide to upgrade to 3.7?
>>
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder, http://www.datastax.com
> @spyced
>

Re: Cassandra 2.2, 3.0, and beyond

[Jonathan Ellis]

3.1 is EOL as soon as 3.3 (the next bug fix release) comes out.

On Thu, Jun 11, 2015 at 4:10 AM, Stefan Podkowinski <
stefan.podkowin...@1und1.de> wrote:

> > We are also extending our backwards compatibility policy to cover all
> 3.x releases: you will be able to upgrade seamlessly from 3.1 to 3.7, for
> instance, including cross-version repair.
>
> What will be the EOL policy for releases after 3.0? Given your example,
> will 3.1 still see bugfixes at this point when I decide to upgrade to 3.7?
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder, http://www.datastax.com
@spyced

Re: Cassandra 2.2, 3.0, and beyond

[Daniel Compton]

Hi Jonathan

Does documentation fit into the new monthly releases and definition of done
as well, or is that part of another process? I didn't see any mention of it
in the docs, though I may have missed it.
On Thu, 11 Jun 2015 at 9:10 pm Stefan Podkowinski <
stefan.podkowin...@1und1.de> wrote:

> > We are also extending our backwards compatibility policy to cover all
> 3.x releases: you will be able to upgrade seamlessly from 3.1 to 3.7, for
> instance, including cross-version repair.
>
> What will be the EOL policy for releases after 3.0? Given your example,
> will 3.1 still see bugfixes at this point when I decide to upgrade to 3.7?
>
-- 
--
Daniel

Re: Cassandra 2.2, 3.0, and beyond

[Stefan Podkowinski]

> We are also extending our backwards compatibility policy to cover all 3.x 
> releases: you will be able to upgrade seamlessly from 3.1 to 3.7, for 
> instance, including cross-version repair.  

What will be the EOL policy for releases after 3.0? Given your example, will 
3.1 still see bugfixes at this point when I decide to upgrade to 3.7?

Re: Cassandra 2.2, 3.0, and beyond

[Tyler Hobbs]

On Wed, Jun 10, 2015 at 1:43 PM,  wrote:

> With 3.0, what happens to existing Thrift-based tables (with dynamic
> column names, etc.)?


Just like in Cassandra 2.x, they will show up as COMPACT STORAGE tables in
a format that CQL can work with.  We're making a few adjustments to how the
schema is presented in CQL, mostly to better deal with a mixture of defined
and undefined column names (mixed static and dynamic).  That mostly
involves treating defined columns as "static".

However, the storage format for COMPACT STORAGE tables will not be
(significantly) different from normal tables any more.  You can read a few
details about the new storage format here:
https://github.com/pcmanus/cassandra/blob/8099_engine_refactor/guide_8099.md#storage-format-on-disk-and-on-wire


-- 
Tyler Hobbs
DataStax 

RE: Cassandra 2.2, 3.0, and beyond

[SEAN_R_DURITY]

With 3.0, what happens to existing Thrift-based tables (with dynamic column 
names, etc.)?

Sean Durity

From: Jonathan Ellis [mailto:jbel...@gmail.com]
Sent: Wednesday, June 10, 2015 10:30 AM
To: user
Subject: Cassandra 2.2, 3.0, and beyond


As you know, we've split our post-2.1 release into two pieces, with 2.2 to be 
released in July (rc1 out Monday<http://cassandra.apache.org/download/>) and 
3.0 in September.


2.2 will include Windows support, commitlog 
compression<https://issues.apache.org/jira/browse/CASSANDRA-6809>, JSON 
support<https://issues.apache.org/jira/browse/CASSANDRA-7970>, role-based 
authorization<http://www.datastax.com/dev/blog/role-based-access-control-in-cassandra>,
 bootstrap-aware leveled 
compaction<https://issues.apache.org/jira/browse/CASSANDRA-7460>, and 
user-defined 
functions<http://christopher-batey.blogspot.com/2015/05/cassandra-aggregates-min-max-avg-group.html>.


3.0 will include a major storage engine 
rewrite<https://issues.apache.org/jira/browse/CASSANDRA-8099> and materialized 
views<https://issues.apache.org/jira/browse/CASSANDRA-6477>.


We're splitting things up this way because we don't want to block the features 
that are already complete while waiting for 8099 (the new storage engine).  
Releasing them now as 2.2 reduces the risk for users (2.2 has a lot in common 
with 2.1) and allows us to stabilize that independently of the upheaval from 
8099.


After 3.0, we'll take this even further: we will release 3.x versions monthly.  
Even releases will include both bugfixes and new features; odd releases will be 
bugfix-only.  You may have heard this referred to as "tick-tock" releases, 
after Intel's policy of changing process and architecture 
independently<http://www.intel.com/content/www/us/en/silicon-innovations/intel-tick-tock-model-general.html>.


The primary goal is to improve release quality.  Our current major "dot zero" 
releases require another five or six months to make them stable enough for 
production.  This is directly related to how we pile features in for 9 to 12 
months and release all at once.  The interactions between the new features are 
complex and not always obvious.  2.1 was no exception, despite DataStax hiring 
a full time test engineering team specifically for Apache Cassandra.


We need to try something different.  Tick-tock releases will dramatically 
reduce the number of features in each version, which will necessarily improve 
our ability to quickly track down any regressions.  And "pausing" every other 
month to focus on bug fixes will help ensure that we don't accumulate issues 
faster than we can fix them.


Tick-tock will also prevent situations like the one we are in now with 8099 
delaying everything else.  Users will get to test new features almost 
immediately.


To get there, we are investing significant effort in making trunk "always 
releasable," with the goal that each release, or at least each odd-numbered 
bugfix release, should be usable in production.  We’ve extended our continuous 
integration server to make it easy for contributors to run tests against 
feature 
branches<http://www.datastax.com/dev/blog/cassandra-testing-improvements-for-developer-convenience-and-confidence>
 before merging to trunk and we’re working on more test 
infrastructure<https://docs.google.com/document/d/1Seku0vPwChbnH3uYYxon0UO-b6LDtSqluZiH--sWWi0>
 and 
procedures<https://docs.google.com/document/d/1ptr47UQ56N80jqL_O6AlE67b0STyn_cVp2k5DTv-OMc>
 to improve release quality.  You can see how this is coming along in our May 
retrospective<https://docs.google.com/document/d/1GtuYRocdr9luNdwmm8wE84uC5Wr6TvewFbQtqoAFVeU/edit>.


We are also extending our backwards compatibility policy to cover all 3.x 
releases: you will be able to upgrade seamlessly from 3.1 to 3.7, for instance, 
including cross-version repair.  We will not introduce any extra upgrade 
requirements or remove deprecated features until 4.0, no sooner than a year 
after 3.0.


Under normal conditions, we will not release 3.x.y stability releases for x > 
0.  That is, we will have a traditional 3.0.y stability series, but the 
odd-numbered bugfix-only releases will fill that role for the tick-tock series 
-- recognizing that occasionally we will need to be flexible enough to release 
an emergency fix in the case of a critical bug or security vulnerability.


We do recognize that it will take some time for tick-tock releases to deliver 
production-level stability, which is why we will continue to deliver 2.2.y and 
3.0.y bugfix releases.  (But if we do demonstrate that tick-tock can deliver 
the stability we want, there will be no need for a 4.0.y bugfix series, only 
4.x tick-tock.)

After 2.2.0 is released, 2.0 will reach end-of-life as planned.  After 3.0.0 is 
released, 2.1 will also reach end of life.  This is earlier

Cassandra 2.2, 3.0, and beyond

[Jonathan Ellis]

*As you know, we've split our post-2.1 release into two pieces, with 2.2 to
be released in July (rc1 out Monday
) and 3.0 in September.2.2 will
include Windows support, commitlog compression
, JSON support
, role-based
authorization
,
bootstrap-aware leveled compaction
, and user-defined
functions
.
3.0 will include a major storage engine rewrite
 and materialized
views .We're
splitting things up this way because we don't want to block the features
that are already complete while waiting for 8099 (the new storage engine).
Releasing them now as 2.2 reduces the risk for users (2.2 has a lot in
common with 2.1) and allows us to stabilize that independently of the
upheaval from 8099.After 3.0, we'll take this even further: we will release
3.x versions monthly.  Even releases will include both bugfixes and new
features; odd releases will be bugfix-only.  You may have heard this
referred to as "tick-tock" releases, after Intel's policy of changing
process and architecture independently
.The
primary goal is to improve release quality.  Our current major "dot zero"
releases require another five or six months to make them stable enough for
production.  This is directly related to how we pile features in for 9 to
12 months and release all at once.  The interactions between the new
features are complex and not always obvious.  2.1 was no exception, despite
DataStax hiring a full time test engineering team specifically for Apache
Cassandra.We need to try something different.  Tick-tock releases will
dramatically reduce the number of features in each version, which will
necessarily improve our ability to quickly track down any regressions.  And
"pausing" every other month to focus on bug fixes will help ensure that we
don't accumulate issues faster than we can fix them.Tick-tock will also
prevent situations like the one we are in now with 8099 delaying everything
else.  Users will get to test new features almost immediately.To get there,
we are investing significant effort in making trunk "always releasable,"
with the goal that each release, or at least each odd-numbered bugfix
release, should be usable in production.  We’ve extended our continuous
integration server to make it easy for contributors to run tests against
feature branches

before merging to trunk and we’re working on more test infrastructure

and procedures

to improve release quality.  You can see how this is coming along in our
May retrospective
.We
are also extending our backwards compatibility policy to cover all 3.x
releases: you will be able to upgrade seamlessly from 3.1 to 3.7, for
instance, including cross-version repair.  We will not introduce any extra
upgrade requirements or remove deprecated features until 4.0, no sooner
than a year after 3.0.Under normal conditions, we will not release 3.x.y
stability releases for x > 0.  That is, we will have a traditional 3.0.y
stability series, but the odd-numbered bugfix-only releases will fill that
role for the tick-tock series -- recognizing that occasionally we will need
to be flexible enough to release an emergency fix in the case of a critical
bug or security vulnerability.We do recognize that it will take some time
for tick-tock releases to deliver production-level stability, which is why
we will continue to deliver 2.2.y and 3.0.y bugfix releases.  (But if we do
demonstrate that tick-tock can deliver the stability we want, there will be
no need for a 4.0.y bugfix series, only 4.x tick-tock.) After 2.2.0 is
released, 2.0 will reach end-of-life as planned.  After 3.0.0 is released,
2.1 will also reach end of life.  This is earlier than expected, but 2.2
will be very close to as stable as 2.1 and users will be well served by
upgrading.  We will maintain the 2.2 stability series until 4.0 is
released, and 3.0 for six months after that.Thanks for reading this far,
and I look forward to hearing how 2.2rc1 works for you!*
-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder, http://www.datastax.com
@spyced