Re: [VOTE] Release Apache Cassandra 3.6

2016-05-12 Thread Tyler Hobbs 
Based on CASSANDRA-11613 (and CASSANDRA-11760), I'm changing my vote to a
(non-binding) -1.  There is a legit regression in upgrading non-frozen UDTs
that needs to be fixed before releasing 3.6.

On Thu, May 12, 2016 at 12:44 PM, Philip Thompson <
philip.thomp...@datastax.com> wrote:

> I've updated the TE report with the results of the upgrade testing we did.
> We experienced a higher than expected number of test failures, which
> prompted the filing of:
>
> CASSANDRA-11760
> CASSANDRA-11763
> CASSANDRA-11765
> CASSANDRA-11767
>
> Two errors were related to handling the legacy hint format after upgrades
> to 3.6 from either the 2.1 or 2.2 series. This should not affect upgrades
> from 3.0.x, or new 3.6 clusters. 11760 is an issue handling UDTs in
> mixed-versions 2.2.5 / 3.6 clusters.
>
> These four bugs were accompanied by other existing, known upgrade failures.
> We do not suspect that these four issues are 3.6 regressions, as we have
> tested this release with a large number of new upgrade tests, that were not
> run on previous tick-tock releases. I am re-running these tests against
> 3.4, to confirm that suspicion.
>
> On Tue, May 10, 2016 at 9:54 PM, Jake Luciani  wrote:
>
> > I propose the following artifacts for release as 3.6.
> >
> > sha1: c17cbe1875a974a00822ffbfad716abde363c8da
> > Git:
> >
> >
> http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/3.6-tentative
> > Artifacts:
> >
> >
> https://repository.apache.org/content/repositories/orgapachecassandra-1112/org/apache/cassandra/apache-cassandra/3.6/
> > Staging repository:
> >
> https://repository.apache.org/content/repositories/orgapachecassandra-1112/
> >
> > The artifacts as well as the debian package are also available here:
> > http://people.apache.org/~jake
> >
> > The vote will be open for 72 hours (longer if needed).
> >
> > [1]: http://goo.gl/Yv15Qz (CHANGES.txt)
> > [2]: http://goo.gl/VyR9EG (NEWS.txt)
> > [3]: https://goo.gl/raz8ok (DataStax QA Report)
> >
>



-- 
Tyler Hobbs
DataStax

Re: [VOTE] Release Apache Cassandra 3.6

2016-05-12 Thread Philip Thompson 
I've updated the TE report with the results of the upgrade testing we did.
We experienced a higher than expected number of test failures, which
prompted the filing of:

CASSANDRA-11760
CASSANDRA-11763
CASSANDRA-11765
CASSANDRA-11767

Two errors were related to handling the legacy hint format after upgrades
to 3.6 from either the 2.1 or 2.2 series. This should not affect upgrades
from 3.0.x, or new 3.6 clusters. 11760 is an issue handling UDTs in
mixed-versions 2.2.5 / 3.6 clusters.

These four bugs were accompanied by other existing, known upgrade failures.
We do not suspect that these four issues are 3.6 regressions, as we have
tested this release with a large number of new upgrade tests, that were not
run on previous tick-tock releases. I am re-running these tests against
3.4, to confirm that suspicion.

On Tue, May 10, 2016 at 9:54 PM, Jake Luciani  wrote:

> I propose the following artifacts for release as 3.6.
>
> sha1: c17cbe1875a974a00822ffbfad716abde363c8da
> Git:
>
> http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/3.6-tentative
> Artifacts:
>
> https://repository.apache.org/content/repositories/orgapachecassandra-1112/org/apache/cassandra/apache-cassandra/3.6/
> Staging repository:
> https://repository.apache.org/content/repositories/orgapachecassandra-1112/
>
> The artifacts as well as the debian package are also available here:
> http://people.apache.org/~jake
>
> The vote will be open for 72 hours (longer if needed).
>
> [1]: http://goo.gl/Yv15Qz (CHANGES.txt)
> [2]: http://goo.gl/VyR9EG (NEWS.txt)
> [3]: https://goo.gl/raz8ok (DataStax QA Report)
>

Re: [VOTE] Release Apache Cassandra 3.6

2016-05-12 Thread Gary Dusbabek 
+1

On Tue, May 10, 2016 at 8:54 PM, Jake Luciani  wrote:

> I propose the following artifacts for release as 3.6.
>
> sha1: c17cbe1875a974a00822ffbfad716abde363c8da
> Git:
>
> http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/3.6-tentative
> Artifacts:
>
> https://repository.apache.org/content/repositories/orgapachecassandra-1112/org/apache/cassandra/apache-cassandra/3.6/
> Staging repository:
> https://repository.apache.org/content/repositories/orgapachecassandra-1112/
>
> The artifacts as well as the debian package are also available here:
> http://people.apache.org/~jake
>
> The vote will be open for 72 hours (longer if needed).
>
> [1]: http://goo.gl/Yv15Qz (CHANGES.txt)
> [2]: http://goo.gl/VyR9EG (NEWS.txt)
> [3]: https://goo.gl/raz8ok (DataStax QA Report)
>

Re: [VOTE] Release Apache Cassandra 3.0.6

2016-05-12 Thread Gary Dusbabek 
+1

On Tue, May 10, 2016 at 8:54 PM, Jake Luciani  wrote:

> I propose the following artifacts for release as 3.0.6.
>
> sha1: 52447873a361647a5e80c547adea9cf5ee85254a
> Git:
>
> http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/3.0.6-tentative
> Artifacts:
>
> https://repository.apache.org/content/repositories/orgapachecassandra-1110/org/apache/cassandra/apache-cassandra/3.0.6/
> Staging repository:
> https://repository.apache.org/content/repositories/orgapachecassandra-1110/
>
> The artifacts as well as the debian package are also available here:
> http://people.apache.org/~jake
>
> The vote will be open for 72 hours (longer if needed).
>
> [1]: http://goo.gl/IiNyVb (CHANGES.txt)
> [2]: http://goo.gl/ZAr03L (NEWS.txt)
> [3]: https://goo.gl/2jPtss (DataStax QA Report)
>

Re: [VOTE] Release Apache Cassandra 3.0.6

2016-05-12 Thread Codarren Velvindron 
+1
On 11 May 2016 05:55, "Jake Luciani"  wrote:

> I propose the following artifacts for release as 3.0.6.
>
> sha1: 52447873a361647a5e80c547adea9cf5ee85254a
> Git:
>
> http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/3.0.6-tentative
> Artifacts:
>
> https://repository.apache.org/content/repositories/orgapachecassandra-1110/org/apache/cassandra/apache-cassandra/3.0.6/
> Staging repository:
> https://repository.apache.org/content/repositories/orgapachecassandra-1110/
>
> The artifacts as well as the debian package are also available here:
> http://people.apache.org/~jake
>
> The vote will be open for 72 hours (longer if needed).
>
> [1]: http://goo.gl/IiNyVb (CHANGES.txt)
> [2]: http://goo.gl/ZAr03L (NEWS.txt)
> [3]: https://goo.gl/2jPtss (DataStax QA Report)
>

Re: [VOTE] Release Apache Cassandra 3.0.6

2016-05-12 Thread Tommy Stendahl 

+1


On 2016-05-11 03:54, Jake Luciani wrote:

I propose the following artifacts for release as 3.0.6.

sha1: 52447873a361647a5e80c547adea9cf5ee85254a
Git:
http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/3.0.6-tentative
Artifacts:
https://repository.apache.org/content/repositories/orgapachecassandra-1110/org/apache/cassandra/apache-cassandra/3.0.6/
Staging repository:
https://repository.apache.org/content/repositories/orgapachecassandra-1110/

The artifacts as well as the debian package are also available here:
http://people.apache.org/~jake

The vote will be open for 72 hours (longer if needed).

[1]: http://goo.gl/IiNyVb (CHANGES.txt)
[2]: http://goo.gl/ZAr03L (NEWS.txt)
[3]: https://goo.gl/2jPtss (DataStax QA Report)

RE: Cassandra 2.0.x OOM during startsup - schema version inconsistency after reboot

2016-05-12 Thread Michael Fong 
Hi Alain,

Thanks for your reply.

We understood that there is a chance that this would be left unresolved, since 
we are really way behind the official Cassandra releases.

Here is what have further found for the OOM issue, which seems to be related to 
# of gossip message accumulated on a live node waiting to connect to the 
rebooted node. Once that node is rebooted, all the gossip message floods into 
each other, triggers StorageService.onAlive() and schedules a schema pull on 
demand. In our case, schema version sometimes is different after reboot. When 
that happens, schema-exchange storm begins.

Also, thanks for your tip on sharing the SOP on stopping an ode, here is what 
we have for our stop procedure:
Disable thrift
Disable Binary
Wait 10s
Disable gossip
Drain
Kill 

Any thought on this to be further improved?

Thanks!

Sincerely,

Michael Fong


From: Alain RODRIGUEZ [mailto:arodr...@gmail.com]
Sent: Wednesday, May 11, 2016 10:01 PM
To: u...@cassandra.apache.org
Cc: dev@cassandra.apache.org
Subject: Re: Cassandra 2.0.x OOM during startsup - schema version inconsistency 
after reboot

Hi Michaels :-),

My guess is this ticket will be closed with a "Won't Fix" resolution.

Cassandra 2.0 is no longer supported and I have seen tickets being rejected 
like CASSANDRA-10510.

Would you like to upgrade to 2.1.last and see if you still have the issue?

About your issue, do you stop your node using a command like the following one?

nodetool disablethrift && nodetool disablebinary && sleep 5 && nodetool 
disablegossip && sleep 10 && nodetool drain && sleep 10 && sudo service 
cassandra stop

or even flushing:

nodetool disablethrift && nodetool disablebinary && sleep 5 && nodetool 
disablegossip && sleep 10 && nodetool flush && nodetool drain && sleep 10 && 
sudo service cassandra stop

Are commitlogs empty when you start cassandra?

C*heers,

---
Alain Rodriguez - al...@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2016-05-11 5:35 GMT+02:00 Michael Fong 
>:
Hi,

Thanks for your recommendation.
I also opened a ticket to keep track @ 
https://issues.apache.org/jira/browse/CASSANDRA-11748
Hope this could brought someone's attention to take a look. Thanks.

Sincerely,

Michael Fong

-Original Message-
From: Michael Kjellman 
[mailto:mkjell...@internalcircle.com]
Sent: Monday, May 09, 2016 11:57 AM
To: dev@cassandra.apache.org
Cc: u...@cassandra.apache.org
Subject: Re: Cassandra 2.0.x OOM during startsup - schema version inconsistency 
after reboot

I'd recommend you create a JIRA! That way you can get some traction on the 
issue. Obviously an OOM is never correct, even if your process is wrong in some 
way!

Best,
kjellman

Sent from my iPhone

> On May 8, 2016, at 8:48 PM, Michael Fong 
> > 
> wrote:
>
> Hi, all,
>
>
> Haven't heard any responses so far, and this isue has troubled us for quite 
> some time. Here is another update:
>
> We have noticed several times that The schema version may change after 
> migration and reboot:
>
> Here is the scenario:
>
> 1.   Two node cluster (1 & 2).
>
> 2.   There are some schema changes, i.e. create a few new columnfamily. 
> The cluster will wait until both nodes have schema version in sync (describe 
> cluster) before moving on.
>
> 3.   Right before node2 is rebooted, the schema version is consistent; 
> however, after ndoe2 reboots and starts servicing, the MigrationManager would 
> gossip different schema version.
>
> 4.   Afterwards, both nodes starts exchanging schema  message 
> indefinitely until one of the node dies.
>
> We currently suspect the change of schema is due to replying the old entry in 
> commit log. We wish to continue dig further, but need experts help on this.
>
> I don't know if anyone has seen this before, or if there is anything wrong 
> with our migration flow though..
>
> Thanks in advance.
>
> Best regards,
>
>
> Michael Fong
>
> From: Michael Fong 
> [mailto:michael.f...@ruckuswireless.com]
> Sent: Thursday, April 21, 2016 6:41 PM
> To: u...@cassandra.apache.org; 
> dev@cassandra.apache.org
> Subject: RE: Cassandra 2.0.x OOM during bootstrap
>
> Hi, all,
>
> Here is some more information on before the OOM happened on the rebooted node 
> in a 2-node test cluster:
>
>
> 1.   It seems the schema version has changed on the rebooted node after 
> reboot, i.e.
> Before reboot,
> Node 1: DEBUG [MigrationStage:1] 2016-04-19 11:09:42,326
> MigrationManager.java (line 328) Gossiping my schema version
>