[jira] [Commented] (CASSANDRA-9977) Support counter-columns for native aggregates (sum,avg,max,min)

[Sylvain Lebresne (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15058298#comment-15058298
 ] 

Sylvain Lebresne commented on CASSANDRA-9977:
-

bq. Want to say: you no longer have a counter but just a value which is then 
"independent" from its origin

That's pretty ill defined though. When you read a counter directly, you also 
get "just a value". For all intent and purposes, we could have return 
{{bigint}} for a raw select of a counter, but we don't, so let's be consistent 
and preserve the {{counter}} type by default. And as Benjamin mention, if you 
use the {{max}} function for instance, your "intuition" doesn't really hold 
anymore.

I'll note that if people really care about getting a {{bigint}}, they can 
always cast the return (haven't checked if we allow casting a counter to a 
bigint but if we don't, we should) so its not like we're losing anything. 
Chances are, almost no-one will notice however.


> Support counter-columns for native aggregates (sum,avg,max,min)
> ---
>
> Key: CASSANDRA-9977
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9977
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CQL
>Reporter: Noam Liran
>Assignee: Robert Stupp
> Fix For: 2.2.x
>
>
> When trying to SUM a column of type COUNTER, this error is returned:
> {noformat}
> InvalidRequest: code=2200 [Invalid query] message="Invalid call to function 
> sum, none of its type signatures match (known type signatures: system.sum : 
> (tinyint) -> tinyint, system.sum : (smallint) -> smallint, system.sum : (int) 
> -> int, system.sum : (bigint) -> bigint, system.sum : (float) -> float, 
> system.sum : (double) -> double, system.sum : (decimal) -> decimal, 
> system.sum : (varint) -> varint)"
> {noformat}
> This might be relevant for other agg. functions.
> CQL for reproduction:
> {noformat}
> CREATE TABLE test (
> key INT,
> ctr COUNTER,
> PRIMARY KEY (
> key
> )
> );
> UPDATE test SET ctr = ctr + 1 WHERE key = 1;
> SELECT SUM(ctr) FROM test;
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-6018) Add option to encrypt commitlog

[Sylvain Lebresne (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15058328#comment-15058328
 ] 

Sylvain Lebresne commented on CASSANDRA-6018:
-

[~jasobrown] Planning on updating this at some point or have you lose interest 
in it?

> Add option to encrypt commitlog 
> 
>
> Key: CASSANDRA-6018
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6018
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Jason Brown
>Assignee: Jason Brown
>  Labels: commit_log, encryption, security
> Fix For: 3.x
>
>
> We are going to start using cassandra for a billing system, and while I can 
> encrypt sstables at rest (via Datastax Enterprise), commit logs are more or 
> less plain text. Thus, an attacker would be able to easily read, for example, 
> credit card numbers in the clear text commit log (if the calling app does not 
> encrypt the data itself before sending it to cassandra).
> I want to allow the option of encrypting the commit logs, most likely 
> controlled by a property in the yaml.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10750) Minor code improvements

[Robert Stupp (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15058265#comment-15058265
 ] 

Robert Stupp commented on CASSANDRA-10750:
--

Your changes LGTM and I've cherry-picked them onto my branch.
Also removed the tail-recursion changes in stress - I'd like to leave the other 
changes as these are in "hot" code paths.

Triggered another CI run. Not sure whether we can get a clean run (without 
timeouts). It not, I can run these tests locally.

> Minor code improvements
> ---
>
> Key: CASSANDRA-10750
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10750
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Robert Stupp
>Assignee: Robert Stupp
>Priority: Minor
>
> Went though several IDE inspections and found some places in the code that 
> could be improved. These are just minor improvements and no bug fixes (except 
> one minor "theoretical" thing).
> The [branch on github against 
> trunk|https://github.com/snazy/cassandra/tree/10750-code-opts-trunk] contains 
> a series of commits:
> * simplify Mutation.apply to remove the casts
> * "minor code improvements" just replaces some expressions that are 
> effectively constant
> * remove unused assignments (probably just cosmetic)
> * collapse identical if-branches (probably just cosmetic)
> * empty array constants
> * fix printf usage (could potentially raise an exception in printf)
> * replace tail-recursion in some critical sections (as the JVM cannot 
> optimize that AFAIK)
> * remove methods identical to their super methods (probably just cosmetic)
> [cassci results 
> here|http://cassci.datastax.com/view/Dev/view/snazy/search/?q=snazy-10750-]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10837) Cluster/session should be closed in Cassandra Hadoop Input/Output classes

[Jeremiah Jordan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeremiah Jordan updated CASSANDRA-10837:

Fix Version/s: 3.0.2

> Cluster/session should be closed in Cassandra Hadoop Input/Output classes
> -
>
> Key: CASSANDRA-10837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10837
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Alex Liu
>Assignee: Alex Liu
> Fix For: 3.0.2
>
> Attachments: 10837-3.0-branch.txt, 10837-v2-3.0-branch.txt
>
>
> See a lot of following warnings during Hadoop job running
> {code}
> ERROR 11:37:45 LEAK: You are creating too many HashedWheelTimer instances.  
> HashedWheelTimer is a shared resource that must be reused across the JVM,so 
> that only a few instances are created.
> {code}
> Each cluster/session needs be closed and a shared HashedWheelTimer may reduce 
> the resource leakage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10822) SSTable data loss when upgrading with row tombstone present

[Russ Hatch (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15058350#comment-15058350
 ] 

Russ Hatch commented on CASSANDRA-10822:


dtest committed, and passing on 3.0.2. (test also passes on trunk but that's 
vetting an upgrade from 3.0.1 to trunk, which presumably wouldn't have the 
issue in the first place).

> SSTable data loss when upgrading with row tombstone present
> ---
>
> Key: CASSANDRA-10822
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10822
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Andy Tolbert
>Assignee: Branimir Lambov
>Priority: Critical
> Fix For: 3.0.2, 3.2
>
>
> I ran into an issue when upgrading between 2.1.11 to 3.0.0 (and also 
> cassandra-3.0 branch) where subsequent rows were lost within a partition 
> where there is a row tombstone present.
> Here's a scenario that reproduces the issue.
> Using ccm create a single node cluster at 2.1.11:
> {{ccm create -n 1 -v 2.1.11 -s financial}}
> Run the following queries to create schema, populate some data and then 
> delete some data for november:
> {noformat}
> drop keyspace if exists financial;
> create keyspace if not exists financial with replication = {'class': 
> 'SimpleStrategy', 'replication_factor' : 1 };
> create table if not exists financial.symbol_history (
>   symbol text,
>   name text static,
>   year int,
>   month int,
>   day int,
>   volume bigint,
>   close double,
>   open double,
>   low double,
>   high double,
>   primary key((symbol, year), month, day)
> ) with CLUSTERING ORDER BY (month desc, day desc);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) 
> values ('CORP', 'MegaCorp', 2004, 1, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) 
> values ('CORP', 'MegaCorp', 2004, 2, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) 
> values ('CORP', 'MegaCorp', 2004, 3, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) 
> values ('CORP', 'MegaCorp', 2004, 4, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) 
> values ('CORP', 'MegaCorp', 2004, 5, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) 
> values ('CORP', 'MegaCorp', 2004, 6, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) 
> values ('CORP', 'MegaCorp', 2004, 7, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) 
> values ('CORP', 'MegaCorp', 2004, 8, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) 
> values ('CORP', 'MegaCorp', 2004, 9, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) 
> values ('CORP', 'MegaCorp', 2004, 10, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) 
> values ('CORP', 'MegaCorp', 2004, 11, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) 
> values ('CORP', 'MegaCorp', 2004, 12, 1, 100);
> delete from financial.symbol_history where symbol='CORP' and year = 2004 and 
> month=11;
> {noformat}
> Flush and run sstable2json on the sole Data.db file:
> {noformat}
> ccm node1 flush
> sstable2json /path/to/file.db
> {noformat}
> The output should look like the following:
> {code}
> [
> {"key": "CORP:2004",
>  "cells": [["::name","MegaCorp",1449457517033030],
>["12:1:","",1449457517033030],
>["12:1:volume","100",1449457517033030],
>["11:_","11:!",1449457564983269,"t",1449457564],
>["10:1:","",1449457516313738],
>["10:1:volume","100",1449457516313738],
>["9:1:","",1449457516310205],
>["9:1:volume","100",1449457516310205],
>["8:1:","",1449457516235664],
>["8:1:volume","100",1449457516235664],
>["7:1:","",1449457516233535],
>["7:1:volume","100",1449457516233535],
>["6:1:","",1449457516231458],
>["6:1:volume","100",1449457516231458],
>["5:1:","",1449457516228307],
>["5:1:volume","100",1449457516228307],
>["4:1:","",1449457516225415],
>["4:1:volume","100",1449457516225415],
>["3:1:","",1449457516222811],
>["3:1:volume","100",1449457516222811],
>["2:1:","",1449457516220301],
>["2:1:volume","100",1449457516220301],
>["1:1:","",1449457516210758],
>["1:1:volume","100",1449457516210758]]}
> ]
> {code}
> Prepare for upgrade
> {noformat}
> ccm node1 nodetool snapshot financial
> ccm node1 nodetool drain
> ccm node1 stop
> {noformat}
> Upgrade to 

[jira] [Commented] (CASSANDRA-10700) 2.1 sstableloader will fail if there are collections in the schema tables

[T Jake Luciani (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15058360#comment-15058360
 ] 

T Jake Luciani commented on CASSANDRA-10700:


Removed duplicate test case and that import, will commit once next test run is 
done

> 2.1 sstableloader will fail if there are collections in the schema tables
> -
>
> Key: CASSANDRA-10700
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10700
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Tyler Hobbs
>Assignee: T Jake Luciani
> Fix For: 2.1.x
>
>
> In {{BulkLoader.ExternalClient}}, we use the Thrift {{execute_cql3_query()}} 
> method to read the system schema tables.  Because it's a Thrift connection, 
> we use the v2 protocol format for serializing data.  However, when we later 
> read the results with {{CFMetadata.fromThriftCqlRow()}}, we use the v3 
> protocol format to deserialize the results.  If there are any collections in 
> the results, such as entries in {{dropped_columns}}, the following error will 
> occur:
> {noformat}
> Caused by: java.lang.IllegalArgumentException: null
> at java.nio.Buffer.limit(Buffer.java:275) ~[na:1.8.0_45-internal]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.readBytes(ByteBufferUtil.java:543) 
> ~[cassandra-all-2.1.11.jar:2.1.11]
> at 
> org.apache.cassandra.serializers.CollectionSerializer.readValue(CollectionSerializer.java:124)
>  ~[cassandra-all-2.1.1
> 1.jar:2.1.11]
> at 
> org.apache.cassandra.serializers.MapSerializer.deserializeForNativeProtocol(MapSerializer.java:101)
>  ~[cassandra-all-
> 2.1.11.jar:2.1.11]
> at 
> org.apache.cassandra.serializers.MapSerializer.deserializeForNativeProtocol(MapSerializer.java:30)
>  ~[cassandra-all-2
> .1.11.jar:2.1.11]
> at 
> org.apache.cassandra.serializers.CollectionSerializer.deserialize(CollectionSerializer.java:50)
>  ~[cassandra-all-2.1.
> 11.jar:2.1.11]
> at 
> org.apache.cassandra.db.marshal.AbstractType.compose(AbstractType.java:68) 
> ~[cassandra-all-2.1.11.jar:2.1.11]
> at 
> org.apache.cassandra.cql3.UntypedResultSet$Row.getMap(UntypedResultSet.java:287)
>  ~[cassandra-all-2.1.11.jar:2.1.11]
> at 
> org.apache.cassandra.config.CFMetaData.fromSchemaNoTriggers(CFMetaData.java:1833)
>  ~[cassandra-all-2.1.11.jar:2.1.11]
> at 
> org.apache.cassandra.config.CFMetaData.fromThriftCqlRow(CFMetaData.java:1126) 
> ~[cassandra-all-2.1.11.jar:2.1.11]
> at 
> org.apache.cassandra.tools.BulkLoader$ExternalClient.init(BulkLoader.java:330)
>  ~[cassandra-all-2.1.11.jar:na]
> ... 7 common frames omitted
> {noformat}
> I believe this only affects 2.1 due to the re-working of 
> BulkLoader/sstableloader in 2.2 and 3.0, but I haven't confirmed that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

cassandra git commit: email typo

[jake]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0.2 f0d235f58 -> 9b655ac18


email typo


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9b655ac1
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9b655ac1
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9b655ac1

Branch: refs/heads/cassandra-3.0.2
Commit: 9b655ac181e732e2c489e102d6742cad6f7029e6
Parents: f0d235f
Author: T Jake Luciani 
Authored: Tue Dec 15 12:53:06 2015 -0500
Committer: T Jake Luciani 
Committed: Tue Dec 15 12:53:06 2015 -0500

--
 debian/changelog | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/9b655ac1/debian/changelog
--
diff --git a/debian/changelog b/debian/changelog
index 8add839..bad7a7b 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -2,7 +2,7 @@ cassandra (3.0.2) unstable; urgency=medium
 
   * New release 
 
- -- Jake Luciani   Mon, 14 Dec 2015 11:58:12 -0500
+ -- Jake Luciani   Mon, 14 Dec 2015 11:58:12 -0500
 
 cassandra (3.0.1) unstable; urgency=medium
 

[jira] [Commented] (CASSANDRA-9302) Optimize cqlsh COPY FROM, part 3

[Stefania (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15058299#comment-15058299
 ] 

Stefania commented on CASSANDRA-9302:
-

Thanks Adam. As discussed, here are two possible follow-ups:

* The ingest rate only works correctly if chunk size << ingest rate since we 
still send at least one chunk at a time.

* The 6 seconds improvement noted when I reverted to batching by primary key 
rather than by replica, is caused by a slow lookup in the token map (bisect 
right). The driver TAR only performs one lookup per batch whilst to batch by 
replica we must perform one lookup per record. In order to make batching by 
replica viable, which should be faster in theory, we must optimize the TM 
lookup but this is not easy to do. Provided we have at least one local replica 
this should not be worth it but we may want to revisit this for non local 
clusters if the need arises.

> Optimize cqlsh COPY FROM, part 3
> 
>
> Key: CASSANDRA-9302
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9302
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Jonathan Ellis
>Assignee: Stefania
>Priority: Critical
> Fix For: 2.1.x
>
>
> We've had some discussion moving to Spark CSV import for bulk load in 3.x, 
> but people need a good bulk load tool now.  One option is to add a separate 
> Java bulk load tool (CASSANDRA-9048), but if we can match that performance 
> from cqlsh I would prefer to leave COPY FROM as the preferred option to which 
> we point people, rather than adding more tools that need to be supported 
> indefinitely.
> Previous work on COPY FROM optimization was done in CASSANDRA-7405 and 
> CASSANDRA-8225.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-5902) Dealing with hints after a topology change

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-5902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15058454#comment-15058454
 ] 

Aleksey Yeschenko commented on CASSANDRA-5902:
--

This is fine, conceptually.

That said, if our working assumption is that this situation should be rare (and 
I agree with the assumption here), then we might go a slightly simpler route:
- for hints files for which the node is no longer part of the ring, instead of 
regular delivery, write the new hints locally instead, just to their new 
destinations - and have them replay later via the regular hints replay path
- on the receiving end, if the hint no longer belongs, write a hint on that 
node for all the replicas instead, and have them be delivered later via the 
regular replay path

We do pay a bit more price in file IO - in redundant writes, but writing a hint 
for multiple replicas is relatively cheap (serialised just once, using the same 
space in the buffer), and this should be rare. 

As a bonus, {{Hint::apply()}} remains local only, without calls to 
{{StorageProxy}}, plus we could avoid further extending {{AWRH}}.

Please let me know if I'm just being silly and none of this makes any sense - I 
haven't head enough tea today.

> Dealing with hints after a topology change
> --
>
> Key: CASSANDRA-5902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5902
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Coordination
>Reporter: Jonathan Ellis
>Assignee: Branimir Lambov
>Priority: Minor
> Fix For: 3.0.x, 3.x
>
>
> Hints are stored and delivered by destination node id.  This allows them to 
> survive IP changes in the target, while making "scan all the hints for a 
> given destination" an efficient operation.  However, we do not detect and 
> handle new node assuming responsibility for the hinted row via bootstrap 
> before it can be delivered.
> I think we have to take a performance hit in this case -- we need to deliver 
> such a hint to *all* replicas, since we don't know which is the "new" one.  
> This happens infrequently enough, however -- requiring first the target node 
> to be down to create the hint, then the hint owner to be down long enough for 
> the target to both recover and stream to a new node -- that this should be 
> okay.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9302) Optimize cqlsh COPY FROM, part 3

[Adam Holmberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15058288#comment-15058288
 ] 

Adam Holmberg commented on CASSANDRA-9302:
--

+1 review

> Optimize cqlsh COPY FROM, part 3
> 
>
> Key: CASSANDRA-9302
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9302
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Jonathan Ellis
>Assignee: Stefania
>Priority: Critical
> Fix For: 2.1.x
>
>
> We've had some discussion moving to Spark CSV import for bulk load in 3.x, 
> but people need a good bulk load tool now.  One option is to add a separate 
> Java bulk load tool (CASSANDRA-9048), but if we can match that performance 
> from cqlsh I would prefer to leave COPY FROM as the preferred option to which 
> we point people, rather than adding more tools that need to be supported 
> indefinitely.
> Previous work on COPY FROM optimization was done in CASSANDRA-7405 and 
> CASSANDRA-8225.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10700) 2.1 sstableloader will fail if there are collections in the schema tables

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15058322#comment-15058322
 ] 

Aleksey Yeschenko commented on CASSANDRA-10700:
---

{{AbstractType}} import does seem to be redundant, but +1 to the patch, so long 
as cassci is happy.

> 2.1 sstableloader will fail if there are collections in the schema tables
> -
>
> Key: CASSANDRA-10700
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10700
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Tyler Hobbs
>Assignee: T Jake Luciani
> Fix For: 2.1.x
>
>
> In {{BulkLoader.ExternalClient}}, we use the Thrift {{execute_cql3_query()}} 
> method to read the system schema tables.  Because it's a Thrift connection, 
> we use the v2 protocol format for serializing data.  However, when we later 
> read the results with {{CFMetadata.fromThriftCqlRow()}}, we use the v3 
> protocol format to deserialize the results.  If there are any collections in 
> the results, such as entries in {{dropped_columns}}, the following error will 
> occur:
> {noformat}
> Caused by: java.lang.IllegalArgumentException: null
> at java.nio.Buffer.limit(Buffer.java:275) ~[na:1.8.0_45-internal]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.readBytes(ByteBufferUtil.java:543) 
> ~[cassandra-all-2.1.11.jar:2.1.11]
> at 
> org.apache.cassandra.serializers.CollectionSerializer.readValue(CollectionSerializer.java:124)
>  ~[cassandra-all-2.1.1
> 1.jar:2.1.11]
> at 
> org.apache.cassandra.serializers.MapSerializer.deserializeForNativeProtocol(MapSerializer.java:101)
>  ~[cassandra-all-
> 2.1.11.jar:2.1.11]
> at 
> org.apache.cassandra.serializers.MapSerializer.deserializeForNativeProtocol(MapSerializer.java:30)
>  ~[cassandra-all-2
> .1.11.jar:2.1.11]
> at 
> org.apache.cassandra.serializers.CollectionSerializer.deserialize(CollectionSerializer.java:50)
>  ~[cassandra-all-2.1.
> 11.jar:2.1.11]
> at 
> org.apache.cassandra.db.marshal.AbstractType.compose(AbstractType.java:68) 
> ~[cassandra-all-2.1.11.jar:2.1.11]
> at 
> org.apache.cassandra.cql3.UntypedResultSet$Row.getMap(UntypedResultSet.java:287)
>  ~[cassandra-all-2.1.11.jar:2.1.11]
> at 
> org.apache.cassandra.config.CFMetaData.fromSchemaNoTriggers(CFMetaData.java:1833)
>  ~[cassandra-all-2.1.11.jar:2.1.11]
> at 
> org.apache.cassandra.config.CFMetaData.fromThriftCqlRow(CFMetaData.java:1126) 
> ~[cassandra-all-2.1.11.jar:2.1.11]
> at 
> org.apache.cassandra.tools.BulkLoader$ExternalClient.init(BulkLoader.java:330)
>  ~[cassandra-all-2.1.11.jar:na]
> ... 7 common frames omitted
> {noformat}
> I believe this only affects 2.1 due to the re-working of 
> BulkLoader/sstableloader in 2.2 and 3.0, but I haven't confirmed that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10750) Minor code improvements

[Stefania (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15058313#comment-15058313
 ] 

Stefania commented on CASSANDRA-10750:
--

bq. Also removed the tail-recursion changes in stress - I'd like to leave the 
other changes as these are in "hot" code paths.

WFM

+1 once CI is OK.

> Minor code improvements
> ---
>
> Key: CASSANDRA-10750
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10750
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Robert Stupp
>Assignee: Robert Stupp
>Priority: Minor
>
> Went though several IDE inspections and found some places in the code that 
> could be improved. These are just minor improvements and no bug fixes (except 
> one minor "theoretical" thing).
> The [branch on github against 
> trunk|https://github.com/snazy/cassandra/tree/10750-code-opts-trunk] contains 
> a series of commits:
> * simplify Mutation.apply to remove the casts
> * "minor code improvements" just replaces some expressions that are 
> effectively constant
> * remove unused assignments (probably just cosmetic)
> * collapse identical if-branches (probably just cosmetic)
> * empty array constants
> * fix printf usage (could potentially raise an exception in printf)
> * replace tail-recursion in some critical sections (as the JVM cannot 
> optimize that AFAIK)
> * remove methods identical to their super methods (probably just cosmetic)
> [cassci results 
> here|http://cassci.datastax.com/view/Dev/view/snazy/search/?q=snazy-10750-]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[cassandra] Git Push Summary

[jake]

Repository: cassandra
Updated Tags:  refs/tags/3.0.2-tentative [created] 9b655ac18

[jira] [Updated] (CASSANDRA-9258) Range movement causes CPU & performance impact

[Dikang Gu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dikang Gu updated CASSANDRA-9258:
-
Attachment: 0001-pending-ranges-map.patch

Address nits.

[~blambov] Sure, I'd love to invest more time. I think this version is already 
much better than the trunk, can you try to commit this? Thanks!

> Range movement causes CPU & performance impact
> --
>
> Key: CASSANDRA-9258
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9258
> Project: Cassandra
>  Issue Type: Bug
> Environment: Cassandra 2.1.4
>Reporter: Rick Branson
>Assignee: Dikang Gu
> Fix For: 2.1.x
>
> Attachments: 0001-pending-ranges-map.patch
>
>
> Observing big CPU & latency regressions when doing range movements on 
> clusters with many tens of thousands of vnodes. See CPU usage increase by 
> ~80% when a single node is being replaced.
> Top methods are:
> 1) Ljava/math/BigInteger;.compareTo in 
> Lorg/apache/cassandra/dht/ComparableObjectToken;.compareTo 
> 2) Lcom/google/common/collect/AbstractMapBasedMultimap;.wrapCollection in 
> Lcom/google/common/collect/AbstractMapBasedMultimap$AsMap$AsMapIterator;.next
> 3) Lorg/apache/cassandra/db/DecoratedKey;.compareTo in 
> Lorg/apache/cassandra/dht/Range;.contains
> Here's a sample stack from a thread dump:
> {code}
> "Thrift:50673" daemon prio=10 tid=0x7f2f20164800 nid=0x3a04af runnable 
> [0x7f2d878d]
>java.lang.Thread.State: RUNNABLE
>   at org.apache.cassandra.dht.Range.isWrapAround(Range.java:260)
>   at org.apache.cassandra.dht.Range.contains(Range.java:51)
>   at org.apache.cassandra.dht.Range.contains(Range.java:110)
>   at 
> org.apache.cassandra.locator.TokenMetadata.pendingEndpointsFor(TokenMetadata.java:916)
>   at 
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:775)
>   at 
> org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:541)
>   at 
> org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:616)
>   at 
> org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:1101)
>   at 
> org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:1083)
>   at 
> org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:976)
>   at 
> org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3996)
>   at 
> org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3980)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>   at 
> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:205)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745){code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10726) Read repair inserts should not be blocking

[sankalp kohli (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15058496#comment-15058496
 ] 

sankalp kohli commented on CASSANDRA-10726:
---

cc [~brandon.williams] [~slebresne] 
What do you think? 

> Read repair inserts should not be blocking
> --
>
> Key: CASSANDRA-10726
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10726
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Coordination
>Reporter: Richard Low
>
> Today, if there’s a digest mismatch in a foreground read repair, the insert 
> to update out of date replicas is blocking. This means, if it fails, the read 
> fails with a timeout. If a node is dropping writes (maybe it is overloaded or 
> the mutation stage is backed up for some other reason), all reads to a 
> replica set could fail. Further, replicas dropping writes get more out of 
> sync so will require more read repair.
> The comment on the code for why the writes are blocking is:
> {code}
> // wait for the repair writes to be acknowledged, to minimize impact on any 
> replica that's
> // behind on writes in case the out-of-sync row is read multiple times in 
> quick succession
> {code}
> but the bad side effect is that reads timeout. Either the writes should not 
> be blocking or we should return success for the read even if the write times 
> out.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10872) Debian Package does not prompt the user to review the config files; it just replaces them causing trouble (since the daemon starts by default)

[Michael Shuler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Shuler updated CASSANDRA-10872:
---
Fix Version/s: 2.1.x

> Debian Package does not prompt the user to review the config files; it just 
> replaces them causing trouble (since the daemon starts by default)
> --
>
> Key: CASSANDRA-10872
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10872
> Project: Cassandra
>  Issue Type: Bug
>  Components: Packaging
> Environment: Ubuntu 12.04, Cassandra 2.0.16 -> 2.0.17
>Reporter: Vasilis
>Assignee: Michael Shuler
>Priority: Critical
> Fix For: 2.1.x
>
>
> We run Cassandra 2.0.16 on Ubuntu 12.04 and were trying to upgrade to 2.0.17 
> due to CASSANDRA-9662. The problem is that during the upgrade the we were not 
> prompted how to handle cassandra-env.sh and cassandra-rackdc.properties. 
> The output from the upgrade:
> Setting up cassandra (2.0.17) ...
> Installing new version of config file /etc/cassandra/cassandra-env.sh ...
> Installing new version of config file 
> /etc/cassandra/cassandra-rackdc.properties ...
> This meant that the nodes started automatically after the install with the 
> wrong DC name. 
> I don't think that these config files should have been replaced without the 
> admin being asked; this doesn't appear to comply with standard Debian 
> packages.
> Secondly if CASSANDRA-2356 was implemented the problem would not be as 
> severe; i.e. it would be possible to workaround this issue. Whereas 
> currently, there is no way to prevent the node when upgraded from starting in 
> the wrong DC.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-9258) Range movement causes CPU & performance impact

[Dikang Gu (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15058431#comment-15058431
 ] 

Dikang Gu edited comment on CASSANDRA-9258 at 12/15/15 7:13 PM:


Address nits.

[~blambov] Sure, I'd love to invest more time. I think this version is already 
much better than the trunk, can we try to commit this? Thanks!


was (Author: dikanggu):
Address nits.

[~blambov] Sure, I'd love to invest more time. I think this version is already 
much better than the trunk, can you try to commit this? Thanks!

> Range movement causes CPU & performance impact
> --
>
> Key: CASSANDRA-9258
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9258
> Project: Cassandra
>  Issue Type: Bug
> Environment: Cassandra 2.1.4
>Reporter: Rick Branson
>Assignee: Dikang Gu
> Fix For: 2.1.x
>
> Attachments: 0001-pending-ranges-map.patch
>
>
> Observing big CPU & latency regressions when doing range movements on 
> clusters with many tens of thousands of vnodes. See CPU usage increase by 
> ~80% when a single node is being replaced.
> Top methods are:
> 1) Ljava/math/BigInteger;.compareTo in 
> Lorg/apache/cassandra/dht/ComparableObjectToken;.compareTo 
> 2) Lcom/google/common/collect/AbstractMapBasedMultimap;.wrapCollection in 
> Lcom/google/common/collect/AbstractMapBasedMultimap$AsMap$AsMapIterator;.next
> 3) Lorg/apache/cassandra/db/DecoratedKey;.compareTo in 
> Lorg/apache/cassandra/dht/Range;.contains
> Here's a sample stack from a thread dump:
> {code}
> "Thrift:50673" daemon prio=10 tid=0x7f2f20164800 nid=0x3a04af runnable 
> [0x7f2d878d]
>java.lang.Thread.State: RUNNABLE
>   at org.apache.cassandra.dht.Range.isWrapAround(Range.java:260)
>   at org.apache.cassandra.dht.Range.contains(Range.java:51)
>   at org.apache.cassandra.dht.Range.contains(Range.java:110)
>   at 
> org.apache.cassandra.locator.TokenMetadata.pendingEndpointsFor(TokenMetadata.java:916)
>   at 
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:775)
>   at 
> org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:541)
>   at 
> org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:616)
>   at 
> org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:1101)
>   at 
> org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:1083)
>   at 
> org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:976)
>   at 
> org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3996)
>   at 
> org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3980)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>   at 
> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:205)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745){code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10837) Cluster/session should be closed in Cassandra Hadoop Input/Output classes

[Benjamin Lerer (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15058713#comment-15058713
 ] 

Benjamin Lerer commented on CASSANDRA-10837:


[~alexliu68] Thanks for the patch.

If I am not mistaken Netty is using {{HashedWheelTimer}} to schedule task 
within the {{EventLoop}} and a new {{EventLoop}} will be created for each new 
{{Cluster}} instance.

I ran some simple tests using only the java driver. They confirmed that the 
ERROR is logged only when too many {{Cluster}} instances are created.

Regarding the patch I have the following comments:
* The changes to {{CqlRecordWriter}} seems wrong to me. 
## The {{Cluster}} and the {{Session}} instances were both properly managed by 
the 
[try-with-resources|https://docs.oracle.com/javase/tutorial/essential/exceptions/tryResourceClose.html]
 statement. In the patch version only the {{Cluster}} instance is closed. 
## Making the {{Cluster}} an instance variable in {{NativeRingCache}} will 
trigger an error when the {{write}} method will be called, as the {{Cluster}} 
has been closed at the end of the constructor.
* In {{CqlInputFormat}} could you use {{try-with-resources}} for both 
{{Cluster}} and {{Session}} instances. I think it is best to do things properly 
by closing both of them. 

 

> Cluster/session should be closed in Cassandra Hadoop Input/Output classes
> -
>
> Key: CASSANDRA-10837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10837
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Alex Liu
>Assignee: Alex Liu
> Fix For: 3.0.x
>
> Attachments: 10837-3.0-branch.txt, 10837-v2-3.0-branch.txt
>
>
> See a lot of following warnings during Hadoop job running
> {code}
> ERROR 11:37:45 LEAK: You are creating too many HashedWheelTimer instances.  
> HashedWheelTimer is a shared resource that must be reused across the JVM,so 
> that only a few instances are created.
> {code}
> Each cluster/session needs be closed and a shared HashedWheelTimer may reduce 
> the resource leakage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-5902) Dealing with hints after a topology change

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-5902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko updated CASSANDRA-5902:
-
Fix Version/s: 3.0.x

> Dealing with hints after a topology change
> --
>
> Key: CASSANDRA-5902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5902
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Coordination
>Reporter: Jonathan Ellis
>Assignee: Branimir Lambov
>Priority: Minor
> Fix For: 3.0.x, 3.x
>
>
> Hints are stored and delivered by destination node id.  This allows them to 
> survive IP changes in the target, while making "scan all the hints for a 
> given destination" an efficient operation.  However, we do not detect and 
> handle new node assuming responsibility for the hinted row via bootstrap 
> before it can be delivered.
> I think we have to take a performance hit in this case -- we need to deliver 
> such a hint to *all* replicas, since we don't know which is the "new" one.  
> This happens infrequently enough, however -- requiring first the target node 
> to be down to create the hint, then the hint owner to be down long enough for 
> the target to both recover and stream to a new node -- that this should be 
> okay.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10850) v4 spec has tons of grammatical mistakes

[Sandeep Tamhankar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sandeep Tamhankar updated CASSANDRA-10850:
--
Attachment: v4-protocol.patch

Here's my patch file. Let me know if you need anything else.

> v4 spec has tons of grammatical mistakes
> 
>
> Key: CASSANDRA-10850
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10850
> Project: Cassandra
>  Issue Type: Bug
>  Components: Documentation and Website
>Reporter: Sandeep Tamhankar
> Fix For: 3.0.2
>
> Attachments: v4-protocol.patch
>
>
> https://github.com/apache/cassandra/blob/cassandra-3.0/doc/native_protocol_v4.spec
> I notice the following in the first section of the spec and then gave up:
> "The list of allowed opcode is defined Section 2.3" => "The list of allowed 
> opcode*s* is defined in Section 2.3"
> "the details of each corresponding message is described Section 4" => "the 
> details of each corresponding message are described in Section 4" since the 
> subject is details, not message.
> "Requests are those frame sent by" => "Requests are those frame*s* sent by"
> I think someone should go through the whole spec and fix all the mistakes 
> rather than me pointing out the ones I notice piece-meal. I found the grammar 
> errors to be rather distracting.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10837) Cluster/session should be closed in Cassandra Hadoop Input/Output classes

[Benjamin Lerer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Lerer updated CASSANDRA-10837:
---
Component/s: CQL

> Cluster/session should be closed in Cassandra Hadoop Input/Output classes
> -
>
> Key: CASSANDRA-10837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10837
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Alex Liu
>Assignee: Alex Liu
> Fix For: 3.0.x
>
> Attachments: 10837-3.0-branch.txt, 10837-v2-3.0-branch.txt
>
>
> See a lot of following warnings during Hadoop job running
> {code}
> ERROR 11:37:45 LEAK: You are creating too many HashedWheelTimer instances.  
> HashedWheelTimer is a shared resource that must be reused across the JVM,so 
> that only a few instances are created.
> {code}
> Each cluster/session needs be closed and a shared HashedWheelTimer may reduce 
> the resource leakage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (CASSANDRA-10872) Debian Package does not prompt the user to review the config files; it just replaces them causing trouble (since the daemon starts by default)

[Michael Shuler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Shuler reassigned CASSANDRA-10872:
--

Assignee: Michael Shuler

> Debian Package does not prompt the user to review the config files; it just 
> replaces them causing trouble (since the daemon starts by default)
> --
>
> Key: CASSANDRA-10872
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10872
> Project: Cassandra
>  Issue Type: Bug
>  Components: Packaging
> Environment: Ubuntu 12.04, Cassandra 2.0.16 -> 2.0.17
>Reporter: Vasilis
>Assignee: Michael Shuler
>Priority: Critical
>
> We run Cassandra 2.0.16 on Ubuntu 12.04 and were trying to upgrade to 2.0.17 
> due to CASSANDRA-9662. The problem is that during the upgrade the we were not 
> prompted how to handle cassandra-env.sh and cassandra-rackdc.properties. 
> The output from the upgrade:
> Setting up cassandra (2.0.17) ...
> Installing new version of config file /etc/cassandra/cassandra-env.sh ...
> Installing new version of config file 
> /etc/cassandra/cassandra-rackdc.properties ...
> This meant that the nodes started automatically after the install with the 
> wrong DC name. 
> I don't think that these config files should have been replaced without the 
> admin being asked; this doesn't appear to comply with standard Debian 
> packages.
> Secondly if CASSANDRA-2356 was implemented the problem would not be as 
> severe; i.e. it would be possible to workaround this issue. Whereas 
> currently, there is no way to prevent the node when upgraded from starting in 
> the wrong DC.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-10822) SSTable data loss when upgrading with row tombstone present

[Russ Hatch (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15058628#comment-15058628
 ] 

Russ Hatch edited comment on CASSANDRA-10822 at 12/15/15 7:36 PM:
--

dtest passes on 3.1.1 as well. (though again, this would be an upgrade from 
3.0.1 to 3.1.1)


was (Author: rhatch):
dtest passes on 3.1.1 as well.

> SSTable data loss when upgrading with row tombstone present
> ---
>
> Key: CASSANDRA-10822
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10822
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Andy Tolbert
>Assignee: Branimir Lambov
>Priority: Critical
> Fix For: 3.0.2, 3.2
>
>
> I ran into an issue when upgrading between 2.1.11 to 3.0.0 (and also 
> cassandra-3.0 branch) where subsequent rows were lost within a partition 
> where there is a row tombstone present.
> Here's a scenario that reproduces the issue.
> Using ccm create a single node cluster at 2.1.11:
> {{ccm create -n 1 -v 2.1.11 -s financial}}
> Run the following queries to create schema, populate some data and then 
> delete some data for november:
> {noformat}
> drop keyspace if exists financial;
> create keyspace if not exists financial with replication = {'class': 
> 'SimpleStrategy', 'replication_factor' : 1 };
> create table if not exists financial.symbol_history (
>   symbol text,
>   name text static,
>   year int,
>   month int,
>   day int,
>   volume bigint,
>   close double,
>   open double,
>   low double,
>   high double,
>   primary key((symbol, year), month, day)
> ) with CLUSTERING ORDER BY (month desc, day desc);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) 
> values ('CORP', 'MegaCorp', 2004, 1, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) 
> values ('CORP', 'MegaCorp', 2004, 2, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) 
> values ('CORP', 'MegaCorp', 2004, 3, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) 
> values ('CORP', 'MegaCorp', 2004, 4, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) 
> values ('CORP', 'MegaCorp', 2004, 5, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) 
> values ('CORP', 'MegaCorp', 2004, 6, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) 
> values ('CORP', 'MegaCorp', 2004, 7, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) 
> values ('CORP', 'MegaCorp', 2004, 8, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) 
> values ('CORP', 'MegaCorp', 2004, 9, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) 
> values ('CORP', 'MegaCorp', 2004, 10, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) 
> values ('CORP', 'MegaCorp', 2004, 11, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) 
> values ('CORP', 'MegaCorp', 2004, 12, 1, 100);
> delete from financial.symbol_history where symbol='CORP' and year = 2004 and 
> month=11;
> {noformat}
> Flush and run sstable2json on the sole Data.db file:
> {noformat}
> ccm node1 flush
> sstable2json /path/to/file.db
> {noformat}
> The output should look like the following:
> {code}
> [
> {"key": "CORP:2004",
>  "cells": [["::name","MegaCorp",1449457517033030],
>["12:1:","",1449457517033030],
>["12:1:volume","100",1449457517033030],
>["11:_","11:!",1449457564983269,"t",1449457564],
>["10:1:","",1449457516313738],
>["10:1:volume","100",1449457516313738],
>["9:1:","",1449457516310205],
>["9:1:volume","100",1449457516310205],
>["8:1:","",1449457516235664],
>["8:1:volume","100",1449457516235664],
>["7:1:","",1449457516233535],
>["7:1:volume","100",1449457516233535],
>["6:1:","",1449457516231458],
>["6:1:volume","100",1449457516231458],
>["5:1:","",1449457516228307],
>["5:1:volume","100",1449457516228307],
>["4:1:","",1449457516225415],
>["4:1:volume","100",1449457516225415],
>["3:1:","",1449457516222811],
>["3:1:volume","100",1449457516222811],
>["2:1:","",1449457516220301],
>["2:1:volume","100",1449457516220301],
>["1:1:","",1449457516210758],
>["1:1:volume","100",1449457516210758]]}
> ]
> {code}
> Prepare for upgrade
> {noformat}
> ccm node1 nodetool snapshot financial
> ccm node1 nodetool drain
> ccm node1 stop
> 

[jira] [Commented] (CASSANDRA-10822) SSTable data loss when upgrading with row tombstone present

[Russ Hatch (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15058628#comment-15058628
 ] 

Russ Hatch commented on CASSANDRA-10822:


dtest passes on 3.1.1 as well.

> SSTable data loss when upgrading with row tombstone present
> ---
>
> Key: CASSANDRA-10822
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10822
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Andy Tolbert
>Assignee: Branimir Lambov
>Priority: Critical
> Fix For: 3.0.2, 3.2
>
>
> I ran into an issue when upgrading between 2.1.11 to 3.0.0 (and also 
> cassandra-3.0 branch) where subsequent rows were lost within a partition 
> where there is a row tombstone present.
> Here's a scenario that reproduces the issue.
> Using ccm create a single node cluster at 2.1.11:
> {{ccm create -n 1 -v 2.1.11 -s financial}}
> Run the following queries to create schema, populate some data and then 
> delete some data for november:
> {noformat}
> drop keyspace if exists financial;
> create keyspace if not exists financial with replication = {'class': 
> 'SimpleStrategy', 'replication_factor' : 1 };
> create table if not exists financial.symbol_history (
>   symbol text,
>   name text static,
>   year int,
>   month int,
>   day int,
>   volume bigint,
>   close double,
>   open double,
>   low double,
>   high double,
>   primary key((symbol, year), month, day)
> ) with CLUSTERING ORDER BY (month desc, day desc);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) 
> values ('CORP', 'MegaCorp', 2004, 1, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) 
> values ('CORP', 'MegaCorp', 2004, 2, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) 
> values ('CORP', 'MegaCorp', 2004, 3, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) 
> values ('CORP', 'MegaCorp', 2004, 4, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) 
> values ('CORP', 'MegaCorp', 2004, 5, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) 
> values ('CORP', 'MegaCorp', 2004, 6, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) 
> values ('CORP', 'MegaCorp', 2004, 7, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) 
> values ('CORP', 'MegaCorp', 2004, 8, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) 
> values ('CORP', 'MegaCorp', 2004, 9, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) 
> values ('CORP', 'MegaCorp', 2004, 10, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) 
> values ('CORP', 'MegaCorp', 2004, 11, 1, 100);
> insert into financial.symbol_history (symbol, name, year, month, day, volume) 
> values ('CORP', 'MegaCorp', 2004, 12, 1, 100);
> delete from financial.symbol_history where symbol='CORP' and year = 2004 and 
> month=11;
> {noformat}
> Flush and run sstable2json on the sole Data.db file:
> {noformat}
> ccm node1 flush
> sstable2json /path/to/file.db
> {noformat}
> The output should look like the following:
> {code}
> [
> {"key": "CORP:2004",
>  "cells": [["::name","MegaCorp",1449457517033030],
>["12:1:","",1449457517033030],
>["12:1:volume","100",1449457517033030],
>["11:_","11:!",1449457564983269,"t",1449457564],
>["10:1:","",1449457516313738],
>["10:1:volume","100",1449457516313738],
>["9:1:","",1449457516310205],
>["9:1:volume","100",1449457516310205],
>["8:1:","",1449457516235664],
>["8:1:volume","100",1449457516235664],
>["7:1:","",1449457516233535],
>["7:1:volume","100",1449457516233535],
>["6:1:","",1449457516231458],
>["6:1:volume","100",1449457516231458],
>["5:1:","",1449457516228307],
>["5:1:volume","100",1449457516228307],
>["4:1:","",1449457516225415],
>["4:1:volume","100",1449457516225415],
>["3:1:","",1449457516222811],
>["3:1:volume","100",1449457516222811],
>["2:1:","",1449457516220301],
>["2:1:volume","100",1449457516220301],
>["1:1:","",1449457516210758],
>["1:1:volume","100",1449457516210758]]}
> ]
> {code}
> Prepare for upgrade
> {noformat}
> ccm node1 nodetool snapshot financial
> ccm node1 nodetool drain
> ccm node1 stop
> {noformat}
> Upgrade to cassandra-3.0 and start the node
> {noformat}
> ccm node1 setdir -v git:cassandra-3.0
> ccm node1 start
> {noformat}
> Run command in cqlsh and 

[jira] [Commented] (CASSANDRA-10726) Read repair inserts should not be blocking

[Brandon Williams (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15058502#comment-15058502
 ] 

Brandon Williams commented on CASSANDRA-10726:
--

Blocking foreground repair basically means a quorum or all consistency level.  
If we return before completing the write, that violates consistency.

> Read repair inserts should not be blocking
> --
>
> Key: CASSANDRA-10726
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10726
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Coordination
>Reporter: Richard Low
>
> Today, if there’s a digest mismatch in a foreground read repair, the insert 
> to update out of date replicas is blocking. This means, if it fails, the read 
> fails with a timeout. If a node is dropping writes (maybe it is overloaded or 
> the mutation stage is backed up for some other reason), all reads to a 
> replica set could fail. Further, replicas dropping writes get more out of 
> sync so will require more read repair.
> The comment on the code for why the writes are blocking is:
> {code}
> // wait for the repair writes to be acknowledged, to minimize impact on any 
> replica that's
> // behind on writes in case the out-of-sync row is read multiple times in 
> quick succession
> {code}
> but the bad side effect is that reads timeout. Either the writes should not 
> be blocking or we should return success for the read even if the write times 
> out.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-9813) cqlsh column header can be incorrect when no rows are returned

[Ariel Weisberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ariel Weisberg updated CASSANDRA-9813:
--
Fix Version/s: (was: 2.1.x)

> cqlsh column header can be incorrect when no rows are returned
> --
>
> Key: CASSANDRA-9813
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9813
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Aleksey Yeschenko
>Assignee: Adam Holmberg
>  Labels: cqlsh
> Fix For: 2.2.x, 3.x
>
> Attachments: 9813-2.1.txt, Test-for-9813.txt
>
>
> Upon migration, we internally create a pair of surrogate clustering/regular 
> columns for compact static tables. These shouldn't be exposed to the user.
> That is, for the table
> {code}
> CREATE TABLE bar (k int, c int, PRIMARY KEY (k)) WITH COMPACT STORAGE;
> {code}
> {{SELECT * FROM bar}} should not be returning this result set:
> {code}
> cqlsh:test> select * from bar;
>  c | column1 | k | value
> ---+-+---+---
> (0 rows)
> {code}
> Should only contain the defined {{c}} and {{k}} columns.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

cassandra git commit: stableloader will fail if there are collections in the schema tables

[jake]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.1 cee35e42d -> 994250c8d


stableloader will fail if there are collections in the schema tables

Fix and new testcase

Patch by tjake; reviewed by Aleksey Yeschenko for CASSANDRA-10700


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/994250c8
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/994250c8
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/994250c8

Branch: refs/heads/cassandra-2.1
Commit: 994250c8d38b3b4299f2e33ebe405ff601b5ae85
Parents: cee35e4
Author: T Jake Luciani 
Authored: Mon Dec 14 11:40:53 2015 -0500
Committer: T Jake Luciani 
Committed: Tue Dec 15 14:15:12 2015 -0500

--
 CHANGES.txt |   4 +
 .../org/apache/cassandra/tools/BulkLoader.java  |  31 +++-
 .../apache/cassandra/tools/BulkLoaderTest.java  | 172 +++
 3 files changed, 202 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/994250c8/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 7f1d66b..8e58703 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,8 +1,12 @@
 2.1.13
+<<< HEAD
  * Allow CREATE TABLE WITH ID (CASSANDRA-9179)
  * Make Stress compiles within eclipse (CASSANDRA-10807)
  * Cassandra Daemon should print JVM arguments (CASSANDRA-10764)
  * Allow cancellation of index summary redistribution (CASSANDRA-8805)
+===
+ * sstableloader will fail if there are collections in the schema tables 
(CASSANDRA-10700)
+>>> 5377183... stableloader will fail if there are collections in the 
schema tables
  * Disable reloading of GossipingPropertyFileSnitch (CASSANDRA-9474)
  * Fix Stress profile parsing on Windows (CASSANDRA-10808)
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/994250c8/src/java/org/apache/cassandra/tools/BulkLoader.java
--
diff --git a/src/java/org/apache/cassandra/tools/BulkLoader.java 
b/src/java/org/apache/cassandra/tools/BulkLoader.java
index f4b30cb..96e826d 100644
--- a/src/java/org/apache/cassandra/tools/BulkLoader.java
+++ b/src/java/org/apache/cassandra/tools/BulkLoader.java
@@ -24,9 +24,10 @@ import java.util.*;
 import com.google.common.base.Joiner;
 import com.google.common.collect.HashMultimap;
 import com.google.common.collect.Multimap;
-import com.google.common.collect.Sets;
 
 import org.apache.commons.cli.*;
+import org.apache.commons.lang3.StringUtils;
+
 import org.apache.thrift.protocol.TBinaryProtocol;
 import org.apache.thrift.protocol.TProtocol;
 import org.apache.thrift.transport.TTransport;
@@ -310,10 +311,11 @@ public class BulkLoader
 }
 }
 
-String cfQuery = String.format("SELECT * FROM %s.%s WHERE 
keyspace_name = '%s'",
- Keyspace.SYSTEM_KS,
- 
SystemKeyspace.SCHEMA_COLUMNFAMILIES_CF,
- keyspace);
+String cfQuery = String.format("SELECT %s FROM %s.%s WHERE 
keyspace_name = '%s'",
+   
StringUtils.join(getCFColumnsWithoutCollections(), ","),
+   Keyspace.SYSTEM_KS,
+   
SystemKeyspace.SCHEMA_COLUMNFAMILIES_CF,
+   keyspace);
 CqlResult cfRes = 
client.execute_cql3_query(ByteBufferUtil.bytes(cfQuery), Compression.NONE, 
ConsistencyLevel.ONE);
 
 
@@ -340,6 +342,25 @@ public class BulkLoader
 }
 }
 
+//Remove dropped_columns since we can't parse collections in v2 which 
is used by thrift
+//See CASSANDRA-10700
+List getCFColumnsWithoutCollections()
+{
+
+Iterator allColumns = 
CFMetaData.SchemaColumnFamiliesCf.allColumnsInSelectOrder();
+List selectedColumns = new ArrayList<>();
+
+while (allColumns.hasNext())
+{
+ColumnDefinition def = allColumns.next();
+
+if (!def.type.isCollection())
+
selectedColumns.add(UTF8Type.instance.getString(def.name.bytes));
+}
+
+return selectedColumns;
+}
+
 @Override
 public StreamConnectionFactory getConnectionFactory()
 {

http://git-wip-us.apache.org/repos/asf/cassandra/blob/994250c8/test/unit/org/apache/cassandra/tools/BulkLoaderTest.java
--
diff --git 

[jira] [Commented] (CASSANDRA-10872) Debian Package does not prompt the user to review the config files; it just replaces them causing trouble (since the daemon starts by default)

[Michael Shuler (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15058577#comment-15058577
 ] 

Michael Shuler commented on CASSANDRA-10872:


Thanks for the report, [~synbit].

> Debian Package does not prompt the user to review the config files; it just 
> replaces them causing trouble (since the daemon starts by default)
> --
>
> Key: CASSANDRA-10872
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10872
> Project: Cassandra
>  Issue Type: Bug
>  Components: Packaging
> Environment: Ubuntu 12.04, Cassandra 2.0.16 -> 2.0.17
>Reporter: Vasilis
>Assignee: Michael Shuler
>Priority: Critical
>
> We run Cassandra 2.0.16 on Ubuntu 12.04 and were trying to upgrade to 2.0.17 
> due to CASSANDRA-9662. The problem is that during the upgrade the we were not 
> prompted how to handle cassandra-env.sh and cassandra-rackdc.properties. 
> The output from the upgrade:
> Setting up cassandra (2.0.17) ...
> Installing new version of config file /etc/cassandra/cassandra-env.sh ...
> Installing new version of config file 
> /etc/cassandra/cassandra-rackdc.properties ...
> This meant that the nodes started automatically after the install with the 
> wrong DC name. 
> I don't think that these config files should have been replaced without the 
> admin being asked; this doesn't appear to comply with standard Debian 
> packages.
> Secondly if CASSANDRA-2356 was implemented the problem would not be as 
> severe; i.e. it would be possible to workaround this issue. Whereas 
> currently, there is no way to prevent the node when upgraded from starting in 
> the wrong DC.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10700) 2.1 sstableloader will fail if there are collections in the schema tables

[T Jake Luciani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

T Jake Luciani updated CASSANDRA-10700:
---
Fix Version/s: (was: 2.1.x)
   2.1.13

> 2.1 sstableloader will fail if there are collections in the schema tables
> -
>
> Key: CASSANDRA-10700
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10700
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Tyler Hobbs
>Assignee: T Jake Luciani
> Fix For: 2.1.13
>
>
> In {{BulkLoader.ExternalClient}}, we use the Thrift {{execute_cql3_query()}} 
> method to read the system schema tables.  Because it's a Thrift connection, 
> we use the v2 protocol format for serializing data.  However, when we later 
> read the results with {{CFMetadata.fromThriftCqlRow()}}, we use the v3 
> protocol format to deserialize the results.  If there are any collections in 
> the results, such as entries in {{dropped_columns}}, the following error will 
> occur:
> {noformat}
> Caused by: java.lang.IllegalArgumentException: null
> at java.nio.Buffer.limit(Buffer.java:275) ~[na:1.8.0_45-internal]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.readBytes(ByteBufferUtil.java:543) 
> ~[cassandra-all-2.1.11.jar:2.1.11]
> at 
> org.apache.cassandra.serializers.CollectionSerializer.readValue(CollectionSerializer.java:124)
>  ~[cassandra-all-2.1.1
> 1.jar:2.1.11]
> at 
> org.apache.cassandra.serializers.MapSerializer.deserializeForNativeProtocol(MapSerializer.java:101)
>  ~[cassandra-all-
> 2.1.11.jar:2.1.11]
> at 
> org.apache.cassandra.serializers.MapSerializer.deserializeForNativeProtocol(MapSerializer.java:30)
>  ~[cassandra-all-2
> .1.11.jar:2.1.11]
> at 
> org.apache.cassandra.serializers.CollectionSerializer.deserialize(CollectionSerializer.java:50)
>  ~[cassandra-all-2.1.
> 11.jar:2.1.11]
> at 
> org.apache.cassandra.db.marshal.AbstractType.compose(AbstractType.java:68) 
> ~[cassandra-all-2.1.11.jar:2.1.11]
> at 
> org.apache.cassandra.cql3.UntypedResultSet$Row.getMap(UntypedResultSet.java:287)
>  ~[cassandra-all-2.1.11.jar:2.1.11]
> at 
> org.apache.cassandra.config.CFMetaData.fromSchemaNoTriggers(CFMetaData.java:1833)
>  ~[cassandra-all-2.1.11.jar:2.1.11]
> at 
> org.apache.cassandra.config.CFMetaData.fromThriftCqlRow(CFMetaData.java:1126) 
> ~[cassandra-all-2.1.11.jar:2.1.11]
> at 
> org.apache.cassandra.tools.BulkLoader$ExternalClient.init(BulkLoader.java:330)
>  ~[cassandra-all-2.1.11.jar:na]
> ... 7 common frames omitted
> {noformat}
> I believe this only affects 2.1 due to the re-working of 
> BulkLoader/sstableloader in 2.2 and 3.0, but I haven't confirmed that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2

[jake]

Merge branch 'cassandra-2.1' into cassandra-2.2


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/de55c39c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/de55c39c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/de55c39c

Branch: refs/heads/cassandra-2.2
Commit: de55c39cd2efeb66a1adace391bbe456213559e8
Parents: 85c7241 994250c
Author: T Jake Luciani 
Authored: Tue Dec 15 14:16:55 2015 -0500
Committer: T Jake Luciani 
Committed: Tue Dec 15 14:16:55 2015 -0500

--

--

[1/3] cassandra git commit: stableloader will fail if there are collections in the schema tables

[jake]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 db40ef839 -> 942e5e599


stableloader will fail if there are collections in the schema tables

Fix and new testcase

Patch by tjake; reviewed by Aleksey Yeschenko for CASSANDRA-10700


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/994250c8
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/994250c8
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/994250c8

Branch: refs/heads/cassandra-3.0
Commit: 994250c8d38b3b4299f2e33ebe405ff601b5ae85
Parents: cee35e4
Author: T Jake Luciani 
Authored: Mon Dec 14 11:40:53 2015 -0500
Committer: T Jake Luciani 
Committed: Tue Dec 15 14:15:12 2015 -0500

--
 CHANGES.txt |   4 +
 .../org/apache/cassandra/tools/BulkLoader.java  |  31 +++-
 .../apache/cassandra/tools/BulkLoaderTest.java  | 172 +++
 3 files changed, 202 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/994250c8/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 7f1d66b..8e58703 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,8 +1,12 @@
 2.1.13
+<<< HEAD
  * Allow CREATE TABLE WITH ID (CASSANDRA-9179)
  * Make Stress compiles within eclipse (CASSANDRA-10807)
  * Cassandra Daemon should print JVM arguments (CASSANDRA-10764)
  * Allow cancellation of index summary redistribution (CASSANDRA-8805)
+===
+ * sstableloader will fail if there are collections in the schema tables 
(CASSANDRA-10700)
+>>> 5377183... stableloader will fail if there are collections in the 
schema tables
  * Disable reloading of GossipingPropertyFileSnitch (CASSANDRA-9474)
  * Fix Stress profile parsing on Windows (CASSANDRA-10808)
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/994250c8/src/java/org/apache/cassandra/tools/BulkLoader.java
--
diff --git a/src/java/org/apache/cassandra/tools/BulkLoader.java 
b/src/java/org/apache/cassandra/tools/BulkLoader.java
index f4b30cb..96e826d 100644
--- a/src/java/org/apache/cassandra/tools/BulkLoader.java
+++ b/src/java/org/apache/cassandra/tools/BulkLoader.java
@@ -24,9 +24,10 @@ import java.util.*;
 import com.google.common.base.Joiner;
 import com.google.common.collect.HashMultimap;
 import com.google.common.collect.Multimap;
-import com.google.common.collect.Sets;
 
 import org.apache.commons.cli.*;
+import org.apache.commons.lang3.StringUtils;
+
 import org.apache.thrift.protocol.TBinaryProtocol;
 import org.apache.thrift.protocol.TProtocol;
 import org.apache.thrift.transport.TTransport;
@@ -310,10 +311,11 @@ public class BulkLoader
 }
 }
 
-String cfQuery = String.format("SELECT * FROM %s.%s WHERE 
keyspace_name = '%s'",
- Keyspace.SYSTEM_KS,
- 
SystemKeyspace.SCHEMA_COLUMNFAMILIES_CF,
- keyspace);
+String cfQuery = String.format("SELECT %s FROM %s.%s WHERE 
keyspace_name = '%s'",
+   
StringUtils.join(getCFColumnsWithoutCollections(), ","),
+   Keyspace.SYSTEM_KS,
+   
SystemKeyspace.SCHEMA_COLUMNFAMILIES_CF,
+   keyspace);
 CqlResult cfRes = 
client.execute_cql3_query(ByteBufferUtil.bytes(cfQuery), Compression.NONE, 
ConsistencyLevel.ONE);
 
 
@@ -340,6 +342,25 @@ public class BulkLoader
 }
 }
 
+//Remove dropped_columns since we can't parse collections in v2 which 
is used by thrift
+//See CASSANDRA-10700
+List getCFColumnsWithoutCollections()
+{
+
+Iterator allColumns = 
CFMetaData.SchemaColumnFamiliesCf.allColumnsInSelectOrder();
+List selectedColumns = new ArrayList<>();
+
+while (allColumns.hasNext())
+{
+ColumnDefinition def = allColumns.next();
+
+if (!def.type.isCollection())
+
selectedColumns.add(UTF8Type.instance.getString(def.name.bytes));
+}
+
+return selectedColumns;
+}
+
 @Override
 public StreamConnectionFactory getConnectionFactory()
 {

http://git-wip-us.apache.org/repos/asf/cassandra/blob/994250c8/test/unit/org/apache/cassandra/tools/BulkLoaderTest.java
--
diff --git 

[4/4] cassandra git commit: Merge branch 'cassandra-3.0' into trunk

[jake]

Merge branch 'cassandra-3.0' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7139f959
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7139f959
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7139f959

Branch: refs/heads/trunk
Commit: 7139f959d1dfe3121f0284dc9dcc9904d020ef74
Parents: 1c62850 942e5e5
Author: T Jake Luciani 
Authored: Tue Dec 15 14:25:43 2015 -0500
Committer: T Jake Luciani 
Committed: Tue Dec 15 14:25:43 2015 -0500

--

--

[2/4] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2

[jake]

Merge branch 'cassandra-2.1' into cassandra-2.2


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/de55c39c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/de55c39c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/de55c39c

Branch: refs/heads/trunk
Commit: de55c39cd2efeb66a1adace391bbe456213559e8
Parents: 85c7241 994250c
Author: T Jake Luciani 
Authored: Tue Dec 15 14:16:55 2015 -0500
Committer: T Jake Luciani 
Committed: Tue Dec 15 14:16:55 2015 -0500

--

--

[jira] [Updated] (CASSANDRA-10837) Cluster/session should be closed in Cassandra Hadoop Input/Output classes

[Jeremiah Jordan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeremiah Jordan updated CASSANDRA-10837:

Fix Version/s: (was: 3.0.2)
   3.0.x

> Cluster/session should be closed in Cassandra Hadoop Input/Output classes
> -
>
> Key: CASSANDRA-10837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10837
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Alex Liu
>Assignee: Alex Liu
> Fix For: 3.0.x
>
> Attachments: 10837-3.0-branch.txt, 10837-v2-3.0-branch.txt
>
>
> See a lot of following warnings during Hadoop job running
> {code}
> ERROR 11:37:45 LEAK: You are creating too many HashedWheelTimer instances.  
> HashedWheelTimer is a shared resource that must be reused across the JVM,so 
> that only a few instances are created.
> {code}
> Each cluster/session needs be closed and a shared HashedWheelTimer may reduce 
> the resource leakage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/3] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2

[jake]

Merge branch 'cassandra-2.1' into cassandra-2.2


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/de55c39c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/de55c39c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/de55c39c

Branch: refs/heads/cassandra-3.0
Commit: de55c39cd2efeb66a1adace391bbe456213559e8
Parents: 85c7241 994250c
Author: T Jake Luciani 
Authored: Tue Dec 15 14:16:55 2015 -0500
Committer: T Jake Luciani 
Committed: Tue Dec 15 14:16:55 2015 -0500

--

--

[1/4] cassandra git commit: stableloader will fail if there are collections in the schema tables

[jake]

Repository: cassandra
Updated Branches:
  refs/heads/trunk 1c62850c9 -> 7139f959d


stableloader will fail if there are collections in the schema tables

Fix and new testcase

Patch by tjake; reviewed by Aleksey Yeschenko for CASSANDRA-10700


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/994250c8
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/994250c8
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/994250c8

Branch: refs/heads/trunk
Commit: 994250c8d38b3b4299f2e33ebe405ff601b5ae85
Parents: cee35e4
Author: T Jake Luciani 
Authored: Mon Dec 14 11:40:53 2015 -0500
Committer: T Jake Luciani 
Committed: Tue Dec 15 14:15:12 2015 -0500

--
 CHANGES.txt |   4 +
 .../org/apache/cassandra/tools/BulkLoader.java  |  31 +++-
 .../apache/cassandra/tools/BulkLoaderTest.java  | 172 +++
 3 files changed, 202 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/994250c8/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 7f1d66b..8e58703 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,8 +1,12 @@
 2.1.13
+<<< HEAD
  * Allow CREATE TABLE WITH ID (CASSANDRA-9179)
  * Make Stress compiles within eclipse (CASSANDRA-10807)
  * Cassandra Daemon should print JVM arguments (CASSANDRA-10764)
  * Allow cancellation of index summary redistribution (CASSANDRA-8805)
+===
+ * sstableloader will fail if there are collections in the schema tables 
(CASSANDRA-10700)
+>>> 5377183... stableloader will fail if there are collections in the 
schema tables
  * Disable reloading of GossipingPropertyFileSnitch (CASSANDRA-9474)
  * Fix Stress profile parsing on Windows (CASSANDRA-10808)
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/994250c8/src/java/org/apache/cassandra/tools/BulkLoader.java
--
diff --git a/src/java/org/apache/cassandra/tools/BulkLoader.java 
b/src/java/org/apache/cassandra/tools/BulkLoader.java
index f4b30cb..96e826d 100644
--- a/src/java/org/apache/cassandra/tools/BulkLoader.java
+++ b/src/java/org/apache/cassandra/tools/BulkLoader.java
@@ -24,9 +24,10 @@ import java.util.*;
 import com.google.common.base.Joiner;
 import com.google.common.collect.HashMultimap;
 import com.google.common.collect.Multimap;
-import com.google.common.collect.Sets;
 
 import org.apache.commons.cli.*;
+import org.apache.commons.lang3.StringUtils;
+
 import org.apache.thrift.protocol.TBinaryProtocol;
 import org.apache.thrift.protocol.TProtocol;
 import org.apache.thrift.transport.TTransport;
@@ -310,10 +311,11 @@ public class BulkLoader
 }
 }
 
-String cfQuery = String.format("SELECT * FROM %s.%s WHERE 
keyspace_name = '%s'",
- Keyspace.SYSTEM_KS,
- 
SystemKeyspace.SCHEMA_COLUMNFAMILIES_CF,
- keyspace);
+String cfQuery = String.format("SELECT %s FROM %s.%s WHERE 
keyspace_name = '%s'",
+   
StringUtils.join(getCFColumnsWithoutCollections(), ","),
+   Keyspace.SYSTEM_KS,
+   
SystemKeyspace.SCHEMA_COLUMNFAMILIES_CF,
+   keyspace);
 CqlResult cfRes = 
client.execute_cql3_query(ByteBufferUtil.bytes(cfQuery), Compression.NONE, 
ConsistencyLevel.ONE);
 
 
@@ -340,6 +342,25 @@ public class BulkLoader
 }
 }
 
+//Remove dropped_columns since we can't parse collections in v2 which 
is used by thrift
+//See CASSANDRA-10700
+List getCFColumnsWithoutCollections()
+{
+
+Iterator allColumns = 
CFMetaData.SchemaColumnFamiliesCf.allColumnsInSelectOrder();
+List selectedColumns = new ArrayList<>();
+
+while (allColumns.hasNext())
+{
+ColumnDefinition def = allColumns.next();
+
+if (!def.type.isCollection())
+
selectedColumns.add(UTF8Type.instance.getString(def.name.bytes));
+}
+
+return selectedColumns;
+}
+
 @Override
 public StreamConnectionFactory getConnectionFactory()
 {

http://git-wip-us.apache.org/repos/asf/cassandra/blob/994250c8/test/unit/org/apache/cassandra/tools/BulkLoaderTest.java
--
diff --git 

[3/4] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[jake]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/942e5e59
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/942e5e59
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/942e5e59

Branch: refs/heads/trunk
Commit: 942e5e599d5bb3184213419ff8a4fffd63eaca20
Parents: db40ef8 de55c39
Author: T Jake Luciani 
Authored: Tue Dec 15 14:17:26 2015 -0500
Committer: T Jake Luciani 
Committed: Tue Dec 15 14:17:26 2015 -0500

--

--

[jira] [Updated] (CASSANDRA-10700) 2.1 sstableloader will fail if there are collections in the schema tables

[Tyler Hobbs (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tyler Hobbs updated CASSANDRA-10700:

Reviewer: Aleksey Yeschenko

> 2.1 sstableloader will fail if there are collections in the schema tables
> -
>
> Key: CASSANDRA-10700
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10700
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Tyler Hobbs
>Assignee: T Jake Luciani
> Fix For: 2.1.x
>
>
> In {{BulkLoader.ExternalClient}}, we use the Thrift {{execute_cql3_query()}} 
> method to read the system schema tables.  Because it's a Thrift connection, 
> we use the v2 protocol format for serializing data.  However, when we later 
> read the results with {{CFMetadata.fromThriftCqlRow()}}, we use the v3 
> protocol format to deserialize the results.  If there are any collections in 
> the results, such as entries in {{dropped_columns}}, the following error will 
> occur:
> {noformat}
> Caused by: java.lang.IllegalArgumentException: null
> at java.nio.Buffer.limit(Buffer.java:275) ~[na:1.8.0_45-internal]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.readBytes(ByteBufferUtil.java:543) 
> ~[cassandra-all-2.1.11.jar:2.1.11]
> at 
> org.apache.cassandra.serializers.CollectionSerializer.readValue(CollectionSerializer.java:124)
>  ~[cassandra-all-2.1.1
> 1.jar:2.1.11]
> at 
> org.apache.cassandra.serializers.MapSerializer.deserializeForNativeProtocol(MapSerializer.java:101)
>  ~[cassandra-all-
> 2.1.11.jar:2.1.11]
> at 
> org.apache.cassandra.serializers.MapSerializer.deserializeForNativeProtocol(MapSerializer.java:30)
>  ~[cassandra-all-2
> .1.11.jar:2.1.11]
> at 
> org.apache.cassandra.serializers.CollectionSerializer.deserialize(CollectionSerializer.java:50)
>  ~[cassandra-all-2.1.
> 11.jar:2.1.11]
> at 
> org.apache.cassandra.db.marshal.AbstractType.compose(AbstractType.java:68) 
> ~[cassandra-all-2.1.11.jar:2.1.11]
> at 
> org.apache.cassandra.cql3.UntypedResultSet$Row.getMap(UntypedResultSet.java:287)
>  ~[cassandra-all-2.1.11.jar:2.1.11]
> at 
> org.apache.cassandra.config.CFMetaData.fromSchemaNoTriggers(CFMetaData.java:1833)
>  ~[cassandra-all-2.1.11.jar:2.1.11]
> at 
> org.apache.cassandra.config.CFMetaData.fromThriftCqlRow(CFMetaData.java:1126) 
> ~[cassandra-all-2.1.11.jar:2.1.11]
> at 
> org.apache.cassandra.tools.BulkLoader$ExternalClient.init(BulkLoader.java:330)
>  ~[cassandra-all-2.1.11.jar:na]
> ... 7 common frames omitted
> {noformat}
> I believe this only affects 2.1 due to the re-working of 
> BulkLoader/sstableloader in 2.2 and 3.0, but I haven't confirmed that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-9258) Range movement causes CPU & performance impact

[Dikang Gu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dikang Gu updated CASSANDRA-9258:
-
Attachment: (was: 0001-pending-ranges-map.patch)

> Range movement causes CPU & performance impact
> --
>
> Key: CASSANDRA-9258
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9258
> Project: Cassandra
>  Issue Type: Bug
> Environment: Cassandra 2.1.4
>Reporter: Rick Branson
>Assignee: Dikang Gu
> Fix For: 2.1.x
>
>
> Observing big CPU & latency regressions when doing range movements on 
> clusters with many tens of thousands of vnodes. See CPU usage increase by 
> ~80% when a single node is being replaced.
> Top methods are:
> 1) Ljava/math/BigInteger;.compareTo in 
> Lorg/apache/cassandra/dht/ComparableObjectToken;.compareTo 
> 2) Lcom/google/common/collect/AbstractMapBasedMultimap;.wrapCollection in 
> Lcom/google/common/collect/AbstractMapBasedMultimap$AsMap$AsMapIterator;.next
> 3) Lorg/apache/cassandra/db/DecoratedKey;.compareTo in 
> Lorg/apache/cassandra/dht/Range;.contains
> Here's a sample stack from a thread dump:
> {code}
> "Thrift:50673" daemon prio=10 tid=0x7f2f20164800 nid=0x3a04af runnable 
> [0x7f2d878d]
>java.lang.Thread.State: RUNNABLE
>   at org.apache.cassandra.dht.Range.isWrapAround(Range.java:260)
>   at org.apache.cassandra.dht.Range.contains(Range.java:51)
>   at org.apache.cassandra.dht.Range.contains(Range.java:110)
>   at 
> org.apache.cassandra.locator.TokenMetadata.pendingEndpointsFor(TokenMetadata.java:916)
>   at 
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:775)
>   at 
> org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:541)
>   at 
> org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:616)
>   at 
> org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:1101)
>   at 
> org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:1083)
>   at 
> org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:976)
>   at 
> org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3996)
>   at 
> org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3980)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>   at 
> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:205)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745){code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9813) cqlsh column header can be incorrect when no rows are returned

[Ariel Weisberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15058569#comment-15058569
 ] 

Ariel Weisberg commented on CASSANDRA-9813:
---

I tested this manually in 2.2 and 3.0 and it worked. Reviewed the tests and 
they are passing.

Tests for trunk, once the complete I'll mark it ready to commit.
|trunk|[utests|http://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-CASSANDRA-9813-trunk-testall/]|[dtests|http://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-CASSANDRA-9813-trunk-dtest/]|

> cqlsh column header can be incorrect when no rows are returned
> --
>
> Key: CASSANDRA-9813
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9813
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Aleksey Yeschenko
>Assignee: Adam Holmberg
>  Labels: cqlsh
> Fix For: 2.1.x, 2.2.x, 3.x
>
> Attachments: 9813-2.1.txt, Test-for-9813.txt
>
>
> Upon migration, we internally create a pair of surrogate clustering/regular 
> columns for compact static tables. These shouldn't be exposed to the user.
> That is, for the table
> {code}
> CREATE TABLE bar (k int, c int, PRIMARY KEY (k)) WITH COMPACT STORAGE;
> {code}
> {{SELECT * FROM bar}} should not be returning this result set:
> {code}
> cqlsh:test> select * from bar;
>  c | column1 | k | value
> ---+-+---+---
> (0 rows)
> {code}
> Should only contain the defined {{c}} and {{k}} columns.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] cassandra git commit: stableloader will fail if there are collections in the schema tables

[jake]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.2 85c724183 -> de55c39cd


stableloader will fail if there are collections in the schema tables

Fix and new testcase

Patch by tjake; reviewed by Aleksey Yeschenko for CASSANDRA-10700


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/994250c8
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/994250c8
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/994250c8

Branch: refs/heads/cassandra-2.2
Commit: 994250c8d38b3b4299f2e33ebe405ff601b5ae85
Parents: cee35e4
Author: T Jake Luciani 
Authored: Mon Dec 14 11:40:53 2015 -0500
Committer: T Jake Luciani 
Committed: Tue Dec 15 14:15:12 2015 -0500

--
 CHANGES.txt |   4 +
 .../org/apache/cassandra/tools/BulkLoader.java  |  31 +++-
 .../apache/cassandra/tools/BulkLoaderTest.java  | 172 +++
 3 files changed, 202 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/994250c8/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 7f1d66b..8e58703 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,8 +1,12 @@
 2.1.13
+<<< HEAD
  * Allow CREATE TABLE WITH ID (CASSANDRA-9179)
  * Make Stress compiles within eclipse (CASSANDRA-10807)
  * Cassandra Daemon should print JVM arguments (CASSANDRA-10764)
  * Allow cancellation of index summary redistribution (CASSANDRA-8805)
+===
+ * sstableloader will fail if there are collections in the schema tables 
(CASSANDRA-10700)
+>>> 5377183... stableloader will fail if there are collections in the 
schema tables
  * Disable reloading of GossipingPropertyFileSnitch (CASSANDRA-9474)
  * Fix Stress profile parsing on Windows (CASSANDRA-10808)
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/994250c8/src/java/org/apache/cassandra/tools/BulkLoader.java
--
diff --git a/src/java/org/apache/cassandra/tools/BulkLoader.java 
b/src/java/org/apache/cassandra/tools/BulkLoader.java
index f4b30cb..96e826d 100644
--- a/src/java/org/apache/cassandra/tools/BulkLoader.java
+++ b/src/java/org/apache/cassandra/tools/BulkLoader.java
@@ -24,9 +24,10 @@ import java.util.*;
 import com.google.common.base.Joiner;
 import com.google.common.collect.HashMultimap;
 import com.google.common.collect.Multimap;
-import com.google.common.collect.Sets;
 
 import org.apache.commons.cli.*;
+import org.apache.commons.lang3.StringUtils;
+
 import org.apache.thrift.protocol.TBinaryProtocol;
 import org.apache.thrift.protocol.TProtocol;
 import org.apache.thrift.transport.TTransport;
@@ -310,10 +311,11 @@ public class BulkLoader
 }
 }
 
-String cfQuery = String.format("SELECT * FROM %s.%s WHERE 
keyspace_name = '%s'",
- Keyspace.SYSTEM_KS,
- 
SystemKeyspace.SCHEMA_COLUMNFAMILIES_CF,
- keyspace);
+String cfQuery = String.format("SELECT %s FROM %s.%s WHERE 
keyspace_name = '%s'",
+   
StringUtils.join(getCFColumnsWithoutCollections(), ","),
+   Keyspace.SYSTEM_KS,
+   
SystemKeyspace.SCHEMA_COLUMNFAMILIES_CF,
+   keyspace);
 CqlResult cfRes = 
client.execute_cql3_query(ByteBufferUtil.bytes(cfQuery), Compression.NONE, 
ConsistencyLevel.ONE);
 
 
@@ -340,6 +342,25 @@ public class BulkLoader
 }
 }
 
+//Remove dropped_columns since we can't parse collections in v2 which 
is used by thrift
+//See CASSANDRA-10700
+List getCFColumnsWithoutCollections()
+{
+
+Iterator allColumns = 
CFMetaData.SchemaColumnFamiliesCf.allColumnsInSelectOrder();
+List selectedColumns = new ArrayList<>();
+
+while (allColumns.hasNext())
+{
+ColumnDefinition def = allColumns.next();
+
+if (!def.type.isCollection())
+
selectedColumns.add(UTF8Type.instance.getString(def.name.bytes));
+}
+
+return selectedColumns;
+}
+
 @Override
 public StreamConnectionFactory getConnectionFactory()
 {

http://git-wip-us.apache.org/repos/asf/cassandra/blob/994250c8/test/unit/org/apache/cassandra/tools/BulkLoaderTest.java
--
diff --git 

[3/3] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[jake]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/942e5e59
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/942e5e59
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/942e5e59

Branch: refs/heads/cassandra-3.0
Commit: 942e5e599d5bb3184213419ff8a4fffd63eaca20
Parents: db40ef8 de55c39
Author: T Jake Luciani 
Authored: Tue Dec 15 14:17:26 2015 -0500
Committer: T Jake Luciani 
Committed: Tue Dec 15 14:17:26 2015 -0500

--

--

[jira] [Commented] (CASSANDRA-10726) Read repair inserts should not be blocking

[Richard Low (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15058576#comment-15058576
 ] 

Richard Low commented on CASSANDRA-10726:
-

How does it violate consistency? The replica was already inconsistent to 
require a read repair insert so returning before completing the write can't 
make it any worse.

> Read repair inserts should not be blocking
> --
>
> Key: CASSANDRA-10726
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10726
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Coordination
>Reporter: Richard Low
>
> Today, if there’s a digest mismatch in a foreground read repair, the insert 
> to update out of date replicas is blocking. This means, if it fails, the read 
> fails with a timeout. If a node is dropping writes (maybe it is overloaded or 
> the mutation stage is backed up for some other reason), all reads to a 
> replica set could fail. Further, replicas dropping writes get more out of 
> sync so will require more read repair.
> The comment on the code for why the writes are blocking is:
> {code}
> // wait for the repair writes to be acknowledged, to minimize impact on any 
> replica that's
> // behind on writes in case the out-of-sync row is read multiple times in 
> quick succession
> {code}
> but the bad side effect is that reads timeout. Either the writes should not 
> be blocking or we should return success for the read even if the write times 
> out.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/3] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2

[marcuse]

Merge branch 'cassandra-2.1' into cassandra-2.2


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/85c72418
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/85c72418
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/85c72418

Branch: refs/heads/cassandra-3.0
Commit: 85c724183a2caee3a9bb73441d27b3cbce7207c4
Parents: 263763a cee35e4
Author: Marcus Eriksson 
Authored: Tue Dec 15 10:01:20 2015 +0100
Committer: Marcus Eriksson 
Committed: Tue Dec 15 10:06:44 2015 +0100

--
 CHANGES.txt |   1 +
 .../org/apache/cassandra/config/CFMetaData.java |   4 +-
 .../cql3/statements/AlterTableStatement.java|   2 +
 .../cassandra/cql3/statements/CFPropDefs.java   |  24 -
 .../cql3/statements/CreateTableStatement.java   |   9 +-
 .../operations/DropRecreateAndRestoreTest.java  | 105 +++
 6 files changed, 135 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/85c72418/CHANGES.txt
--
diff --cc CHANGES.txt
index 592ba0a,7f1d66b..c9074fc
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,10 -1,5 +1,11 @@@
 -2.1.13
 +2.2.5
 + * Add property to allow listening on broadcast interface (CASSANDRA-9748)
 + * Fix regression in split size on CqlInputFormat (CASSANDRA-10835)
 + * Better handling of SSL connection errors inter-node (CASSANDRA-10816)
 + * Disable reloading of GossipingPropertyFileSnitch (CASSANDRA-9474)
 + * Verify tables in pseudo-system keyspaces at startup (CASSANDRA-10761)
 +Merged from 2.1:
+  * Allow CREATE TABLE WITH ID (CASSANDRA-9179)
   * Make Stress compiles within eclipse (CASSANDRA-10807)
   * Cassandra Daemon should print JVM arguments (CASSANDRA-10764)
   * Allow cancellation of index summary redistribution (CASSANDRA-8805)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/85c72418/src/java/org/apache/cassandra/config/CFMetaData.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/85c72418/src/java/org/apache/cassandra/cql3/statements/AlterTableStatement.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/85c72418/src/java/org/apache/cassandra/cql3/statements/CFPropDefs.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/85c72418/src/java/org/apache/cassandra/cql3/statements/CreateTableStatement.java
--
diff --cc 
src/java/org/apache/cassandra/cql3/statements/CreateTableStatement.java
index 7810a8a,2c4b23b..1b3665c
--- a/src/java/org/apache/cassandra/cql3/statements/CreateTableStatement.java
+++ b/src/java/org/apache/cassandra/cql3/statements/CreateTableStatement.java
@@@ -71,13 -72,21 +72,14 @@@ public class CreateTableStatement exten
  this.properties = properties;
  this.ifNotExists = ifNotExists;
  this.staticColumns = staticColumns;
+ this.id = id;
  
 -try
 -{
 -if (!this.properties.hasProperty(CFPropDefs.KW_COMPRESSION) && 
CFMetaData.DEFAULT_COMPRESSOR != null)
 -this.properties.addProperty(CFPropDefs.KW_COMPRESSION,
 -new HashMap()
 -{{
 -
put(CompressionParameters.SSTABLE_COMPRESSION, CFMetaData.DEFAULT_COMPRESSOR);
 -}});
 -}
 -catch (SyntaxException e)
 -{
 -throw new AssertionError(e);
 -}
 +if (!this.properties.hasProperty(CFPropDefs.KW_COMPRESSION) && 
CFMetaData.DEFAULT_COMPRESSOR != null)
 +this.properties.addProperty(CFPropDefs.KW_COMPRESSION,
 +new HashMap()
 +{{
 +
put(CompressionParameters.SSTABLE_COMPRESSION, CFMetaData.DEFAULT_COMPRESSOR);
 +}});
  }
  
  public void checkAccess(ClientState state) throws UnauthorizedException, 
InvalidRequestException

http://git-wip-us.apache.org/repos/asf/cassandra/blob/85c72418/test/unit/org/apache/cassandra/cql3/validation/operations/DropRecreateAndRestoreTest.java
--
diff --cc 
test/unit/org/apache/cassandra/cql3/validation/operations/DropRecreateAndRestoreTest.java
index 000,385ed3d..4a3a51d
mode 00,100644..100644
--- 

[3/3] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[marcuse]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/db40ef83
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/db40ef83
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/db40ef83

Branch: refs/heads/cassandra-3.0
Commit: db40ef8398e34618086184487ffcc3310830cec7
Parents: 71bac92 85c7241
Author: Marcus Eriksson 
Authored: Tue Dec 15 10:25:33 2015 +0100
Committer: Marcus Eriksson 
Committed: Tue Dec 15 10:25:33 2015 +0100

--
 CHANGES.txt |   1 +
 .../cql3/statements/CreateTableStatement.java   |   7 +-
 .../cql3/statements/CreateViewStatement.java|   1 +
 .../cql3/statements/TableAttributes.java|  19 
 .../operations/DropRecreateAndRestoreTest.java  | 104 +++
 5 files changed, 130 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/db40ef83/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/db40ef83/src/java/org/apache/cassandra/cql3/statements/CreateTableStatement.java
--
diff --cc 
src/java/org/apache/cassandra/cql3/statements/CreateTableStatement.java
index 1363bee,1b3665c..c19f970
--- a/src/java/org/apache/cassandra/cql3/statements/CreateTableStatement.java
+++ b/src/java/org/apache/cassandra/cql3/statements/CreateTableStatement.java
@@@ -37,35 -38,48 +37,37 @@@ import org.apache.cassandra.service.Cli
  import org.apache.cassandra.service.MigrationManager;
  import org.apache.cassandra.service.QueryState;
  import org.apache.cassandra.transport.Event;
 -import org.apache.cassandra.utils.ByteBufferUtil;
  
 -/** A CREATE TABLE parsed from a CQL query statement. */
 +/** A {@code CREATE TABLE} parsed from a CQL query statement. */
  public class CreateTableStatement extends SchemaAlteringStatement
  {
 -public CellNameType comparator;
 -private AbstractType defaultValidator;
 -private AbstractType keyValidator;
 +private List keyTypes;
 +private List clusteringTypes;
  
 -private final List keyAliases = new ArrayList();
 -private final List columnAliases = new 
ArrayList();
 -private ByteBuffer valueAlias;
 +private final Map collections = new 
HashMap<>();
 +
 +private final List keyAliases = new ArrayList<>();
 +private final List columnAliases = new ArrayList<>();
  
  private boolean isDense;
 +private boolean isCompound;
 +private boolean hasCounters;
  
  // use a TreeMap to preserve ordering across JDK versions (see 
CASSANDRA-9492)
 -private final Map columns = new 
TreeMap<>(new Comparator()
 -{
 -public int compare(ColumnIdentifier o1, ColumnIdentifier o2)
 -{
 -return o1.bytes.compareTo(o2.bytes);
 -}
 -});
 +private final Map columns = new 
TreeMap<>((o1, o2) -> o1.bytes.compareTo(o2.bytes));
 +
  private final Set staticColumns;
 -private final CFPropDefs properties;
 +private final TableParams params;
  private final boolean ifNotExists;
+ private final UUID id;
  
- public CreateTableStatement(CFName name, TableParams params, boolean 
ifNotExists, Set staticColumns)
 -public CreateTableStatement(CFName name, CFPropDefs properties, boolean 
ifNotExists, Set staticColumns, UUID id)
++public CreateTableStatement(CFName name, TableParams params, boolean 
ifNotExists, Set staticColumns, UUID id)
  {
  super(name);
 -this.properties = properties;
 +this.params = params;
  this.ifNotExists = ifNotExists;
  this.staticColumns = staticColumns;
+ this.id = id;
 -
 -if (!this.properties.hasProperty(CFPropDefs.KW_COMPRESSION) && 
CFMetaData.DEFAULT_COMPRESSOR != null)
 -this.properties.addProperty(CFPropDefs.KW_COMPRESSION,
 -new HashMap()
 -{{
 -
put(CompressionParameters.SSTABLE_COMPRESSION, CFMetaData.DEFAULT_COMPRESSOR);
 -}});
  }
  
  public void checkAccess(ClientState state) throws UnauthorizedException, 
InvalidRequestException
@@@ -109,49 -144,9 +111,50 @@@
  }
  }
  
 +public CFMetaData.Builder metadataBuilder()
 +{
 +CFMetaData.Builder builder = CFMetaData.Builder.create(keyspace(), 
columnFamily(), isDense, isCompound, hasCounters);
++builder.withId(id);
 +for 

[1/3] cassandra git commit: Allow CREATE TABLE WITH ID

[marcuse]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 71bac92cf -> db40ef839


Allow CREATE TABLE WITH ID

Patch by Branimir Lambov; reviewed by marcuse for CASSANDRA-9179


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/cee35e42
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/cee35e42
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/cee35e42

Branch: refs/heads/cassandra-3.0
Commit: cee35e42d05f125bf274b9cefb7653c21ca4b676
Parents: 9b30d65
Author: Branimir Lambov 
Authored: Tue Dec 8 11:59:09 2015 +0200
Committer: Marcus Eriksson 
Committed: Tue Dec 15 09:58:42 2015 +0100

--
 CHANGES.txt |   1 +
 .../org/apache/cassandra/config/CFMetaData.java |   6 +-
 .../cql3/statements/AlterTableStatement.java|   2 +
 .../cassandra/cql3/statements/CFPropDefs.java   |  23 +++-
 .../cql3/statements/CreateTableStatement.java   |   9 +-
 .../operations/DropRecreateAndRestoreTest.java  | 105 +++
 6 files changed, 136 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/cee35e42/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 091ac52..7f1d66b 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.1.13
+ * Allow CREATE TABLE WITH ID (CASSANDRA-9179)
  * Make Stress compiles within eclipse (CASSANDRA-10807)
  * Cassandra Daemon should print JVM arguments (CASSANDRA-10764)
  * Allow cancellation of index summary redistribution (CASSANDRA-8805)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/cee35e42/src/java/org/apache/cassandra/config/CFMetaData.java
--
diff --git a/src/java/org/apache/cassandra/config/CFMetaData.java 
b/src/java/org/apache/cassandra/config/CFMetaData.java
index 2939f09..bce5c72 100644
--- a/src/java/org/apache/cassandra/config/CFMetaData.java
+++ b/src/java/org/apache/cassandra/config/CFMetaData.java
@@ -470,12 +470,12 @@ public final class CFMetaData
  */
 public CFMetaData(String keyspace, String name, ColumnFamilyType type, 
CellNameType comp)
 {
-this(keyspace, name, type, comp, UUIDGen.getTimeUUID());
+this(keyspace, name, type, comp, null);
 }
 
-private CFMetaData(String keyspace, String name, ColumnFamilyType type, 
CellNameType comp, UUID id)
+public CFMetaData(String keyspace, String name, ColumnFamilyType type, 
CellNameType comp, UUID id)
 {
-cfId = id;
+cfId = id != null ? id : UUIDGen.getTimeUUID();
 ksName = keyspace;
 cfName = name;
 ksAndCFName = Pair.create(keyspace, name);

http://git-wip-us.apache.org/repos/asf/cassandra/blob/cee35e42/src/java/org/apache/cassandra/cql3/statements/AlterTableStatement.java
--
diff --git 
a/src/java/org/apache/cassandra/cql3/statements/AlterTableStatement.java 
b/src/java/org/apache/cassandra/cql3/statements/AlterTableStatement.java
index 44bc2bc..42b7f7f 100644
--- a/src/java/org/apache/cassandra/cql3/statements/AlterTableStatement.java
+++ b/src/java/org/apache/cassandra/cql3/statements/AlterTableStatement.java
@@ -87,6 +87,8 @@ public class AlterTableStatement extends 
SchemaAlteringStatement
 columnName = rawColumnName.prepare(cfm);
 def = cfm.getColumnDefinition(columnName);
 }
+if (cfProps.getId() != null)
+throw new ConfigurationException("Cannot alter table id.");
 
 switch (oType)
 {

http://git-wip-us.apache.org/repos/asf/cassandra/blob/cee35e42/src/java/org/apache/cassandra/cql3/statements/CFPropDefs.java
--
diff --git a/src/java/org/apache/cassandra/cql3/statements/CFPropDefs.java 
b/src/java/org/apache/cassandra/cql3/statements/CFPropDefs.java
index 343c0c0..17edd6d 100644
--- a/src/java/org/apache/cassandra/cql3/statements/CFPropDefs.java
+++ b/src/java/org/apache/cassandra/cql3/statements/CFPropDefs.java
@@ -17,10 +17,7 @@
  */
 package org.apache.cassandra.cql3.statements;
 
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.Map;
-import java.util.Set;
+import java.util.*;
 
 import org.apache.cassandra.cache.CachingOptions;
 import org.apache.cassandra.config.CFMetaData;
@@ -49,6 +46,8 @@ public class CFPropDefs extends PropertyDefinitions
 public static final String KW_COMPACTION = "compaction";
 public static final String KW_COMPRESSION = "compression";
 
+public static final String KW_ID = "id";
+
 public static final String COMPACTION_STRATEGY_CLASS_KEY = "class";
 
 

cassandra git commit: Allow CREATE TABLE WITH ID

[marcuse]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.1 9b30d6572 -> cee35e42d


Allow CREATE TABLE WITH ID

Patch by Branimir Lambov; reviewed by marcuse for CASSANDRA-9179


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/cee35e42
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/cee35e42
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/cee35e42

Branch: refs/heads/cassandra-2.1
Commit: cee35e42d05f125bf274b9cefb7653c21ca4b676
Parents: 9b30d65
Author: Branimir Lambov 
Authored: Tue Dec 8 11:59:09 2015 +0200
Committer: Marcus Eriksson 
Committed: Tue Dec 15 09:58:42 2015 +0100

--
 CHANGES.txt |   1 +
 .../org/apache/cassandra/config/CFMetaData.java |   6 +-
 .../cql3/statements/AlterTableStatement.java|   2 +
 .../cassandra/cql3/statements/CFPropDefs.java   |  23 +++-
 .../cql3/statements/CreateTableStatement.java   |   9 +-
 .../operations/DropRecreateAndRestoreTest.java  | 105 +++
 6 files changed, 136 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/cee35e42/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 091ac52..7f1d66b 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.1.13
+ * Allow CREATE TABLE WITH ID (CASSANDRA-9179)
  * Make Stress compiles within eclipse (CASSANDRA-10807)
  * Cassandra Daemon should print JVM arguments (CASSANDRA-10764)
  * Allow cancellation of index summary redistribution (CASSANDRA-8805)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/cee35e42/src/java/org/apache/cassandra/config/CFMetaData.java
--
diff --git a/src/java/org/apache/cassandra/config/CFMetaData.java 
b/src/java/org/apache/cassandra/config/CFMetaData.java
index 2939f09..bce5c72 100644
--- a/src/java/org/apache/cassandra/config/CFMetaData.java
+++ b/src/java/org/apache/cassandra/config/CFMetaData.java
@@ -470,12 +470,12 @@ public final class CFMetaData
  */
 public CFMetaData(String keyspace, String name, ColumnFamilyType type, 
CellNameType comp)
 {
-this(keyspace, name, type, comp, UUIDGen.getTimeUUID());
+this(keyspace, name, type, comp, null);
 }
 
-private CFMetaData(String keyspace, String name, ColumnFamilyType type, 
CellNameType comp, UUID id)
+public CFMetaData(String keyspace, String name, ColumnFamilyType type, 
CellNameType comp, UUID id)
 {
-cfId = id;
+cfId = id != null ? id : UUIDGen.getTimeUUID();
 ksName = keyspace;
 cfName = name;
 ksAndCFName = Pair.create(keyspace, name);

http://git-wip-us.apache.org/repos/asf/cassandra/blob/cee35e42/src/java/org/apache/cassandra/cql3/statements/AlterTableStatement.java
--
diff --git 
a/src/java/org/apache/cassandra/cql3/statements/AlterTableStatement.java 
b/src/java/org/apache/cassandra/cql3/statements/AlterTableStatement.java
index 44bc2bc..42b7f7f 100644
--- a/src/java/org/apache/cassandra/cql3/statements/AlterTableStatement.java
+++ b/src/java/org/apache/cassandra/cql3/statements/AlterTableStatement.java
@@ -87,6 +87,8 @@ public class AlterTableStatement extends 
SchemaAlteringStatement
 columnName = rawColumnName.prepare(cfm);
 def = cfm.getColumnDefinition(columnName);
 }
+if (cfProps.getId() != null)
+throw new ConfigurationException("Cannot alter table id.");
 
 switch (oType)
 {

http://git-wip-us.apache.org/repos/asf/cassandra/blob/cee35e42/src/java/org/apache/cassandra/cql3/statements/CFPropDefs.java
--
diff --git a/src/java/org/apache/cassandra/cql3/statements/CFPropDefs.java 
b/src/java/org/apache/cassandra/cql3/statements/CFPropDefs.java
index 343c0c0..17edd6d 100644
--- a/src/java/org/apache/cassandra/cql3/statements/CFPropDefs.java
+++ b/src/java/org/apache/cassandra/cql3/statements/CFPropDefs.java
@@ -17,10 +17,7 @@
  */
 package org.apache.cassandra.cql3.statements;
 
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.Map;
-import java.util.Set;
+import java.util.*;
 
 import org.apache.cassandra.cache.CachingOptions;
 import org.apache.cassandra.config.CFMetaData;
@@ -49,6 +46,8 @@ public class CFPropDefs extends PropertyDefinitions
 public static final String KW_COMPACTION = "compaction";
 public static final String KW_COMPRESSION = "compression";
 
+public static final String KW_ID = "id";
+
 public static final String COMPACTION_STRATEGY_CLASS_KEY = "class";
 
 

[2/2] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2

[marcuse]

Merge branch 'cassandra-2.1' into cassandra-2.2


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/85c72418
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/85c72418
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/85c72418

Branch: refs/heads/cassandra-2.2
Commit: 85c724183a2caee3a9bb73441d27b3cbce7207c4
Parents: 263763a cee35e4
Author: Marcus Eriksson 
Authored: Tue Dec 15 10:01:20 2015 +0100
Committer: Marcus Eriksson 
Committed: Tue Dec 15 10:06:44 2015 +0100

--
 CHANGES.txt |   1 +
 .../org/apache/cassandra/config/CFMetaData.java |   4 +-
 .../cql3/statements/AlterTableStatement.java|   2 +
 .../cassandra/cql3/statements/CFPropDefs.java   |  24 -
 .../cql3/statements/CreateTableStatement.java   |   9 +-
 .../operations/DropRecreateAndRestoreTest.java  | 105 +++
 6 files changed, 135 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/85c72418/CHANGES.txt
--
diff --cc CHANGES.txt
index 592ba0a,7f1d66b..c9074fc
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,10 -1,5 +1,11 @@@
 -2.1.13
 +2.2.5
 + * Add property to allow listening on broadcast interface (CASSANDRA-9748)
 + * Fix regression in split size on CqlInputFormat (CASSANDRA-10835)
 + * Better handling of SSL connection errors inter-node (CASSANDRA-10816)
 + * Disable reloading of GossipingPropertyFileSnitch (CASSANDRA-9474)
 + * Verify tables in pseudo-system keyspaces at startup (CASSANDRA-10761)
 +Merged from 2.1:
+  * Allow CREATE TABLE WITH ID (CASSANDRA-9179)
   * Make Stress compiles within eclipse (CASSANDRA-10807)
   * Cassandra Daemon should print JVM arguments (CASSANDRA-10764)
   * Allow cancellation of index summary redistribution (CASSANDRA-8805)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/85c72418/src/java/org/apache/cassandra/config/CFMetaData.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/85c72418/src/java/org/apache/cassandra/cql3/statements/AlterTableStatement.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/85c72418/src/java/org/apache/cassandra/cql3/statements/CFPropDefs.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/85c72418/src/java/org/apache/cassandra/cql3/statements/CreateTableStatement.java
--
diff --cc 
src/java/org/apache/cassandra/cql3/statements/CreateTableStatement.java
index 7810a8a,2c4b23b..1b3665c
--- a/src/java/org/apache/cassandra/cql3/statements/CreateTableStatement.java
+++ b/src/java/org/apache/cassandra/cql3/statements/CreateTableStatement.java
@@@ -71,13 -72,21 +72,14 @@@ public class CreateTableStatement exten
  this.properties = properties;
  this.ifNotExists = ifNotExists;
  this.staticColumns = staticColumns;
+ this.id = id;
  
 -try
 -{
 -if (!this.properties.hasProperty(CFPropDefs.KW_COMPRESSION) && 
CFMetaData.DEFAULT_COMPRESSOR != null)
 -this.properties.addProperty(CFPropDefs.KW_COMPRESSION,
 -new HashMap()
 -{{
 -
put(CompressionParameters.SSTABLE_COMPRESSION, CFMetaData.DEFAULT_COMPRESSOR);
 -}});
 -}
 -catch (SyntaxException e)
 -{
 -throw new AssertionError(e);
 -}
 +if (!this.properties.hasProperty(CFPropDefs.KW_COMPRESSION) && 
CFMetaData.DEFAULT_COMPRESSOR != null)
 +this.properties.addProperty(CFPropDefs.KW_COMPRESSION,
 +new HashMap()
 +{{
 +
put(CompressionParameters.SSTABLE_COMPRESSION, CFMetaData.DEFAULT_COMPRESSOR);
 +}});
  }
  
  public void checkAccess(ClientState state) throws UnauthorizedException, 
InvalidRequestException

http://git-wip-us.apache.org/repos/asf/cassandra/blob/85c72418/test/unit/org/apache/cassandra/cql3/validation/operations/DropRecreateAndRestoreTest.java
--
diff --cc 
test/unit/org/apache/cassandra/cql3/validation/operations/DropRecreateAndRestoreTest.java
index 000,385ed3d..4a3a51d
mode 00,100644..100644
--- 

[1/2] cassandra git commit: Allow CREATE TABLE WITH ID

[marcuse]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.2 263763ab9 -> 85c724183


Allow CREATE TABLE WITH ID

Patch by Branimir Lambov; reviewed by marcuse for CASSANDRA-9179


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/cee35e42
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/cee35e42
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/cee35e42

Branch: refs/heads/cassandra-2.2
Commit: cee35e42d05f125bf274b9cefb7653c21ca4b676
Parents: 9b30d65
Author: Branimir Lambov 
Authored: Tue Dec 8 11:59:09 2015 +0200
Committer: Marcus Eriksson 
Committed: Tue Dec 15 09:58:42 2015 +0100

--
 CHANGES.txt |   1 +
 .../org/apache/cassandra/config/CFMetaData.java |   6 +-
 .../cql3/statements/AlterTableStatement.java|   2 +
 .../cassandra/cql3/statements/CFPropDefs.java   |  23 +++-
 .../cql3/statements/CreateTableStatement.java   |   9 +-
 .../operations/DropRecreateAndRestoreTest.java  | 105 +++
 6 files changed, 136 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/cee35e42/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 091ac52..7f1d66b 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.1.13
+ * Allow CREATE TABLE WITH ID (CASSANDRA-9179)
  * Make Stress compiles within eclipse (CASSANDRA-10807)
  * Cassandra Daemon should print JVM arguments (CASSANDRA-10764)
  * Allow cancellation of index summary redistribution (CASSANDRA-8805)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/cee35e42/src/java/org/apache/cassandra/config/CFMetaData.java
--
diff --git a/src/java/org/apache/cassandra/config/CFMetaData.java 
b/src/java/org/apache/cassandra/config/CFMetaData.java
index 2939f09..bce5c72 100644
--- a/src/java/org/apache/cassandra/config/CFMetaData.java
+++ b/src/java/org/apache/cassandra/config/CFMetaData.java
@@ -470,12 +470,12 @@ public final class CFMetaData
  */
 public CFMetaData(String keyspace, String name, ColumnFamilyType type, 
CellNameType comp)
 {
-this(keyspace, name, type, comp, UUIDGen.getTimeUUID());
+this(keyspace, name, type, comp, null);
 }
 
-private CFMetaData(String keyspace, String name, ColumnFamilyType type, 
CellNameType comp, UUID id)
+public CFMetaData(String keyspace, String name, ColumnFamilyType type, 
CellNameType comp, UUID id)
 {
-cfId = id;
+cfId = id != null ? id : UUIDGen.getTimeUUID();
 ksName = keyspace;
 cfName = name;
 ksAndCFName = Pair.create(keyspace, name);

http://git-wip-us.apache.org/repos/asf/cassandra/blob/cee35e42/src/java/org/apache/cassandra/cql3/statements/AlterTableStatement.java
--
diff --git 
a/src/java/org/apache/cassandra/cql3/statements/AlterTableStatement.java 
b/src/java/org/apache/cassandra/cql3/statements/AlterTableStatement.java
index 44bc2bc..42b7f7f 100644
--- a/src/java/org/apache/cassandra/cql3/statements/AlterTableStatement.java
+++ b/src/java/org/apache/cassandra/cql3/statements/AlterTableStatement.java
@@ -87,6 +87,8 @@ public class AlterTableStatement extends 
SchemaAlteringStatement
 columnName = rawColumnName.prepare(cfm);
 def = cfm.getColumnDefinition(columnName);
 }
+if (cfProps.getId() != null)
+throw new ConfigurationException("Cannot alter table id.");
 
 switch (oType)
 {

http://git-wip-us.apache.org/repos/asf/cassandra/blob/cee35e42/src/java/org/apache/cassandra/cql3/statements/CFPropDefs.java
--
diff --git a/src/java/org/apache/cassandra/cql3/statements/CFPropDefs.java 
b/src/java/org/apache/cassandra/cql3/statements/CFPropDefs.java
index 343c0c0..17edd6d 100644
--- a/src/java/org/apache/cassandra/cql3/statements/CFPropDefs.java
+++ b/src/java/org/apache/cassandra/cql3/statements/CFPropDefs.java
@@ -17,10 +17,7 @@
  */
 package org.apache.cassandra.cql3.statements;
 
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.Map;
-import java.util.Set;
+import java.util.*;
 
 import org.apache.cassandra.cache.CachingOptions;
 import org.apache.cassandra.config.CFMetaData;
@@ -49,6 +46,8 @@ public class CFPropDefs extends PropertyDefinitions
 public static final String KW_COMPACTION = "compaction";
 public static final String KW_COMPRESSION = "compression";
 
+public static final String KW_ID = "id";
+
 public static final String COMPACTION_STRATEGY_CLASS_KEY = "class";
 
 

[jira] [Created] (CASSANDRA-10872) Debian Package does not prompt the user to review the config files; it just replaces them causing trouble (since the daemon starts by default)

[Vasilis (JIRA)]

Vasilis created CASSANDRA-10872:
---

 Summary: Debian Package does not prompt the user to review the 
config files; it just replaces them causing trouble (since the daemon starts by 
default)
 Key: CASSANDRA-10872
 URL: https://issues.apache.org/jira/browse/CASSANDRA-10872
 Project: Cassandra
  Issue Type: Bug
  Components: Packaging
 Environment: Ubuntu 12.04, Cassandra 2.0.16 -> 2.0.17
Reporter: Vasilis
Priority: Critical


We run Cassandra 2.0.16 on Ubuntu 12.04 and were trying to upgrade to 2.0.17 
due to CASSANDRA-9662. The problem is that during the upgrade the we were not 
prompted how to handle cassandra-env.sh and cassandra-rackdc.properties. 

The output from the upgrade:

Setting up cassandra (2.0.17) ...
Installing new version of config file /etc/cassandra/cassandra-env.sh ...
Installing new version of config file 
/etc/cassandra/cassandra-rackdc.properties ...

This meant that the nodes started automatically after the install with the 
wrong DC name. 

I don't think that these config files should have been replaced without the 
admin being asked; this doesn't appear to comply with standard Debian packages.

Secondly if CASSANDRA-2356 was implemented the problem would not be as severe; 
i.e. it would be possible to workaround this issue. Whereas currently, there is 
no way to prevent the node when upgraded from starting in the wrong DC.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-6696) Partition sstables by token range

[Marcus Eriksson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15057746#comment-15057746
 ] 

Marcus Eriksson commented on CASSANDRA-6696:


fixed the rest of the comments and pushed to 
https://github.com/krummas/cassandra/commits/marcuse/6696-11

renamed rebalance do redistribute, but if anyone has a better suggestion for 
this, let me know

> Partition sstables by token range
> -
>
> Key: CASSANDRA-6696
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6696
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: sankalp kohli
>Assignee: Marcus Eriksson
>  Labels: compaction, correctness, dense-storage, 
> jbod-aware-compaction, performance
> Fix For: 3.2
>
>
> In JBOD, when someone gets a bad drive, the bad drive is replaced with a new 
> empty one and repair is run. 
> This can cause deleted data to come back in some cases. Also this is true for 
> corrupt stables in which we delete the corrupt stable and run repair. 
> Here is an example:
> Say we have 3 nodes A,B and C and RF=3 and GC grace=10days. 
> row=sankalp col=sankalp is written 20 days back and successfully went to all 
> three nodes. 
> Then a delete/tombstone was written successfully for the same row column 15 
> days back. 
> Since this tombstone is more than gc grace, it got compacted in Nodes A and B 
> since it got compacted with the actual data. So there is no trace of this row 
> column in node A and B.
> Now in node C, say the original data is in drive1 and tombstone is in drive2. 
> Compaction has not yet reclaimed the data and tombstone.  
> Drive2 becomes corrupt and was replaced with new empty drive. 
> Due to the replacement, the tombstone in now gone and row=sankalp col=sankalp 
> has come back to life. 
> Now after replacing the drive we run repair. This data will be propagated to 
> all nodes. 
> Note: This is still a problem even if we run repair every gc grace. 
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10839) cqlsh failed to format value bytearray

[Severin Leonhardt (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15057685#comment-15057685
 ] 

Severin Leonhardt commented on CASSANDRA-10839:
---

You are correct, we are on Python 2.6. This is what the RPM package specifies 
as the minimum required version:

{noformat}
# repoquery --requires cassandra21
/bin/sh
python(abi) >= 2.6
shadow-utils
{noformat}

I was unable to find any information about Cassandra requiring Python 2.7. 
Please let me know where such requirements are documented.

> cqlsh failed to format value bytearray
> --
>
> Key: CASSANDRA-10839
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10839
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Severin Leonhardt
>Priority: Minor
>
> Execute the following in cqlsh (5.0.1):
> {noformat}
> > create table test(column blob, primary key(column));
> > insert into test (column) VALUES(0x00);
> > select * from test;
>  column
> 
>  bytearray(b'\x00')
> (1 rows)
> Failed to format value bytearray(b'\x00') : b2a_hex() argument 1 must be 
> string or read-only buffer, not bytearray
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9302) Optimize cqlsh COPY FROM, part 3

[Stefania (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15057710#comment-15057710
 ] 

Stefania commented on CASSANDRA-9302:
-

bq. Now that we're not choosing session based on replica host, we might further 
simplify split_batches to just group by partition key (i.e., no need for 
get_replica). Alternatively, if you want to send to a specific host other than 
one that load balancing would choose, we would need to borrow a connection and 
send directly on that (I don't think that's worth doing).

We need to batch by replica rather than just by partition key as the scope is 
much wider. Initially I was batching only by primary key but that gave very bad 
results for workloads with unique primary keys, like the one we normally use to 
benchmark these tools, _cassandra-stress_. If the current approach does not 
guarantee we contact the same host then we must borrow a connection to ensure 
that's the case or revert back to individual sessions, since we do have a cap 
of max_requests, we would have to ensure sessions are closed when we are 
finished with them rather than at the very end.

INGESTRATE is used to throttle sending more work but it cannot be smaller than 
a single workload unit (chunk_size * max_requests * num_processes). I'll update 
the documentation at a minimum, or see if this can be simplified.

I'll fix the other two minor points as well, so moving back to in progress.

> Optimize cqlsh COPY FROM, part 3
> 
>
> Key: CASSANDRA-9302
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9302
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Jonathan Ellis
>Assignee: Stefania
>Priority: Critical
> Fix For: 2.1.x
>
>
> We've had some discussion moving to Spark CSV import for bulk load in 3.x, 
> but people need a good bulk load tool now.  One option is to add a separate 
> Java bulk load tool (CASSANDRA-9048), but if we can match that performance 
> from cqlsh I would prefer to leave COPY FROM as the preferred option to which 
> we point people, rather than adding more tools that need to be supported 
> indefinitely.
> Previous work on COPY FROM optimization was done in CASSANDRA-7405 and 
> CASSANDRA-8225.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10750) Minor code improvements

[Robert Stupp (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15057618#comment-15057618
 ] 

Robert Stupp commented on CASSANDRA-10750:
--

Yea - {{Stack.java}} produced NPEs because the order of the constants (static 
final fields) was wrong.
After that fix, both utests and dtests look ok - i.e. there are "just" timeouts 
in the dtests similar to current trunk.

> Minor code improvements
> ---
>
> Key: CASSANDRA-10750
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10750
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Robert Stupp
>Assignee: Robert Stupp
>Priority: Minor
>
> Went though several IDE inspections and found some places in the code that 
> could be improved. These are just minor improvements and no bug fixes (except 
> one minor "theoretical" thing).
> The [branch on github against 
> trunk|https://github.com/snazy/cassandra/tree/10750-code-opts-trunk] contains 
> a series of commits:
> * simplify Mutation.apply to remove the casts
> * "minor code improvements" just replaces some expressions that are 
> effectively constant
> * remove unused assignments (probably just cosmetic)
> * collapse identical if-branches (probably just cosmetic)
> * empty array constants
> * fix printf usage (could potentially raise an exception in printf)
> * replace tail-recursion in some critical sections (as the JVM cannot 
> optimize that AFAIK)
> * remove methods identical to their super methods (probably just cosmetic)
> [cassci results 
> here|http://cassci.datastax.com/view/Dev/view/snazy/search/?q=snazy-10750-]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10750) Minor code improvements

[Stefania (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15057691#comment-15057691
 ] 

Stefania commented on CASSANDRA-10750:
--

Well spotted.

I've finished the review, I've put my changes in [this 
commit|https://github.com/stef1927/cassandra/commit/fd56e6a8361e0c6a6bdd90fc486dd482a2245c01]
 of [this branch|https://github.com/stef1927/cassandra/commits/10750]. I've 
only changed slightly {{PasswordAuthenticator}} and {{MerkleTree}}, the rest 
should be nits. 

The replacement of recursive code with iterative code, whilst it's true that it 
is more efficient, it is also true that the recursive code is more elegant. I 
am thinking of the 2 changes in the stress tool specifically, they don't seem 
to recurse much and the original code was more elegant. I leave that to your 
decision however.

If we can get one clean CI run without timeouts and with my changes cross 
reviewed by you and applied, then this can be committed.

> Minor code improvements
> ---
>
> Key: CASSANDRA-10750
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10750
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Robert Stupp
>Assignee: Robert Stupp
>Priority: Minor
>
> Went though several IDE inspections and found some places in the code that 
> could be improved. These are just minor improvements and no bug fixes (except 
> one minor "theoretical" thing).
> The [branch on github against 
> trunk|https://github.com/snazy/cassandra/tree/10750-code-opts-trunk] contains 
> a series of commits:
> * simplify Mutation.apply to remove the casts
> * "minor code improvements" just replaces some expressions that are 
> effectively constant
> * remove unused assignments (probably just cosmetic)
> * collapse identical if-branches (probably just cosmetic)
> * empty array constants
> * fix printf usage (could potentially raise an exception in printf)
> * replace tail-recursion in some critical sections (as the JVM cannot 
> optimize that AFAIK)
> * remove methods identical to their super methods (probably just cosmetic)
> [cassci results 
> here|http://cassci.datastax.com/view/Dev/view/snazy/search/?q=snazy-10750-]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/4] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2

[marcuse]

Merge branch 'cassandra-2.1' into cassandra-2.2


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/85c72418
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/85c72418
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/85c72418

Branch: refs/heads/trunk
Commit: 85c724183a2caee3a9bb73441d27b3cbce7207c4
Parents: 263763a cee35e4
Author: Marcus Eriksson 
Authored: Tue Dec 15 10:01:20 2015 +0100
Committer: Marcus Eriksson 
Committed: Tue Dec 15 10:06:44 2015 +0100

--
 CHANGES.txt |   1 +
 .../org/apache/cassandra/config/CFMetaData.java |   4 +-
 .../cql3/statements/AlterTableStatement.java|   2 +
 .../cassandra/cql3/statements/CFPropDefs.java   |  24 -
 .../cql3/statements/CreateTableStatement.java   |   9 +-
 .../operations/DropRecreateAndRestoreTest.java  | 105 +++
 6 files changed, 135 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/85c72418/CHANGES.txt
--
diff --cc CHANGES.txt
index 592ba0a,7f1d66b..c9074fc
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,10 -1,5 +1,11 @@@
 -2.1.13
 +2.2.5
 + * Add property to allow listening on broadcast interface (CASSANDRA-9748)
 + * Fix regression in split size on CqlInputFormat (CASSANDRA-10835)
 + * Better handling of SSL connection errors inter-node (CASSANDRA-10816)
 + * Disable reloading of GossipingPropertyFileSnitch (CASSANDRA-9474)
 + * Verify tables in pseudo-system keyspaces at startup (CASSANDRA-10761)
 +Merged from 2.1:
+  * Allow CREATE TABLE WITH ID (CASSANDRA-9179)
   * Make Stress compiles within eclipse (CASSANDRA-10807)
   * Cassandra Daemon should print JVM arguments (CASSANDRA-10764)
   * Allow cancellation of index summary redistribution (CASSANDRA-8805)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/85c72418/src/java/org/apache/cassandra/config/CFMetaData.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/85c72418/src/java/org/apache/cassandra/cql3/statements/AlterTableStatement.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/85c72418/src/java/org/apache/cassandra/cql3/statements/CFPropDefs.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/85c72418/src/java/org/apache/cassandra/cql3/statements/CreateTableStatement.java
--
diff --cc 
src/java/org/apache/cassandra/cql3/statements/CreateTableStatement.java
index 7810a8a,2c4b23b..1b3665c
--- a/src/java/org/apache/cassandra/cql3/statements/CreateTableStatement.java
+++ b/src/java/org/apache/cassandra/cql3/statements/CreateTableStatement.java
@@@ -71,13 -72,21 +72,14 @@@ public class CreateTableStatement exten
  this.properties = properties;
  this.ifNotExists = ifNotExists;
  this.staticColumns = staticColumns;
+ this.id = id;
  
 -try
 -{
 -if (!this.properties.hasProperty(CFPropDefs.KW_COMPRESSION) && 
CFMetaData.DEFAULT_COMPRESSOR != null)
 -this.properties.addProperty(CFPropDefs.KW_COMPRESSION,
 -new HashMap()
 -{{
 -
put(CompressionParameters.SSTABLE_COMPRESSION, CFMetaData.DEFAULT_COMPRESSOR);
 -}});
 -}
 -catch (SyntaxException e)
 -{
 -throw new AssertionError(e);
 -}
 +if (!this.properties.hasProperty(CFPropDefs.KW_COMPRESSION) && 
CFMetaData.DEFAULT_COMPRESSOR != null)
 +this.properties.addProperty(CFPropDefs.KW_COMPRESSION,
 +new HashMap()
 +{{
 +
put(CompressionParameters.SSTABLE_COMPRESSION, CFMetaData.DEFAULT_COMPRESSOR);
 +}});
  }
  
  public void checkAccess(ClientState state) throws UnauthorizedException, 
InvalidRequestException

http://git-wip-us.apache.org/repos/asf/cassandra/blob/85c72418/test/unit/org/apache/cassandra/cql3/validation/operations/DropRecreateAndRestoreTest.java
--
diff --cc 
test/unit/org/apache/cassandra/cql3/validation/operations/DropRecreateAndRestoreTest.java
index 000,385ed3d..4a3a51d
mode 00,100644..100644
--- 

[3/4] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[marcuse]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/db40ef83
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/db40ef83
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/db40ef83

Branch: refs/heads/trunk
Commit: db40ef8398e34618086184487ffcc3310830cec7
Parents: 71bac92 85c7241
Author: Marcus Eriksson 
Authored: Tue Dec 15 10:25:33 2015 +0100
Committer: Marcus Eriksson 
Committed: Tue Dec 15 10:25:33 2015 +0100

--
 CHANGES.txt |   1 +
 .../cql3/statements/CreateTableStatement.java   |   7 +-
 .../cql3/statements/CreateViewStatement.java|   1 +
 .../cql3/statements/TableAttributes.java|  19 
 .../operations/DropRecreateAndRestoreTest.java  | 104 +++
 5 files changed, 130 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/db40ef83/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/db40ef83/src/java/org/apache/cassandra/cql3/statements/CreateTableStatement.java
--
diff --cc 
src/java/org/apache/cassandra/cql3/statements/CreateTableStatement.java
index 1363bee,1b3665c..c19f970
--- a/src/java/org/apache/cassandra/cql3/statements/CreateTableStatement.java
+++ b/src/java/org/apache/cassandra/cql3/statements/CreateTableStatement.java
@@@ -37,35 -38,48 +37,37 @@@ import org.apache.cassandra.service.Cli
  import org.apache.cassandra.service.MigrationManager;
  import org.apache.cassandra.service.QueryState;
  import org.apache.cassandra.transport.Event;
 -import org.apache.cassandra.utils.ByteBufferUtil;
  
 -/** A CREATE TABLE parsed from a CQL query statement. */
 +/** A {@code CREATE TABLE} parsed from a CQL query statement. */
  public class CreateTableStatement extends SchemaAlteringStatement
  {
 -public CellNameType comparator;
 -private AbstractType defaultValidator;
 -private AbstractType keyValidator;
 +private List keyTypes;
 +private List clusteringTypes;
  
 -private final List keyAliases = new ArrayList();
 -private final List columnAliases = new 
ArrayList();
 -private ByteBuffer valueAlias;
 +private final Map collections = new 
HashMap<>();
 +
 +private final List keyAliases = new ArrayList<>();
 +private final List columnAliases = new ArrayList<>();
  
  private boolean isDense;
 +private boolean isCompound;
 +private boolean hasCounters;
  
  // use a TreeMap to preserve ordering across JDK versions (see 
CASSANDRA-9492)
 -private final Map columns = new 
TreeMap<>(new Comparator()
 -{
 -public int compare(ColumnIdentifier o1, ColumnIdentifier o2)
 -{
 -return o1.bytes.compareTo(o2.bytes);
 -}
 -});
 +private final Map columns = new 
TreeMap<>((o1, o2) -> o1.bytes.compareTo(o2.bytes));
 +
  private final Set staticColumns;
 -private final CFPropDefs properties;
 +private final TableParams params;
  private final boolean ifNotExists;
+ private final UUID id;
  
- public CreateTableStatement(CFName name, TableParams params, boolean 
ifNotExists, Set staticColumns)
 -public CreateTableStatement(CFName name, CFPropDefs properties, boolean 
ifNotExists, Set staticColumns, UUID id)
++public CreateTableStatement(CFName name, TableParams params, boolean 
ifNotExists, Set staticColumns, UUID id)
  {
  super(name);
 -this.properties = properties;
 +this.params = params;
  this.ifNotExists = ifNotExists;
  this.staticColumns = staticColumns;
+ this.id = id;
 -
 -if (!this.properties.hasProperty(CFPropDefs.KW_COMPRESSION) && 
CFMetaData.DEFAULT_COMPRESSOR != null)
 -this.properties.addProperty(CFPropDefs.KW_COMPRESSION,
 -new HashMap()
 -{{
 -
put(CompressionParameters.SSTABLE_COMPRESSION, CFMetaData.DEFAULT_COMPRESSOR);
 -}});
  }
  
  public void checkAccess(ClientState state) throws UnauthorizedException, 
InvalidRequestException
@@@ -109,49 -144,9 +111,50 @@@
  }
  }
  
 +public CFMetaData.Builder metadataBuilder()
 +{
 +CFMetaData.Builder builder = CFMetaData.Builder.create(keyspace(), 
columnFamily(), isDense, isCompound, hasCounters);
++builder.withId(id);
 +for (int i = 

[1/4] cassandra git commit: Allow CREATE TABLE WITH ID

[marcuse]

Repository: cassandra
Updated Branches:
  refs/heads/trunk e5db3f2b1 -> d5f1175f3


Allow CREATE TABLE WITH ID

Patch by Branimir Lambov; reviewed by marcuse for CASSANDRA-9179


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/cee35e42
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/cee35e42
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/cee35e42

Branch: refs/heads/trunk
Commit: cee35e42d05f125bf274b9cefb7653c21ca4b676
Parents: 9b30d65
Author: Branimir Lambov 
Authored: Tue Dec 8 11:59:09 2015 +0200
Committer: Marcus Eriksson 
Committed: Tue Dec 15 09:58:42 2015 +0100

--
 CHANGES.txt |   1 +
 .../org/apache/cassandra/config/CFMetaData.java |   6 +-
 .../cql3/statements/AlterTableStatement.java|   2 +
 .../cassandra/cql3/statements/CFPropDefs.java   |  23 +++-
 .../cql3/statements/CreateTableStatement.java   |   9 +-
 .../operations/DropRecreateAndRestoreTest.java  | 105 +++
 6 files changed, 136 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/cee35e42/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 091ac52..7f1d66b 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.1.13
+ * Allow CREATE TABLE WITH ID (CASSANDRA-9179)
  * Make Stress compiles within eclipse (CASSANDRA-10807)
  * Cassandra Daemon should print JVM arguments (CASSANDRA-10764)
  * Allow cancellation of index summary redistribution (CASSANDRA-8805)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/cee35e42/src/java/org/apache/cassandra/config/CFMetaData.java
--
diff --git a/src/java/org/apache/cassandra/config/CFMetaData.java 
b/src/java/org/apache/cassandra/config/CFMetaData.java
index 2939f09..bce5c72 100644
--- a/src/java/org/apache/cassandra/config/CFMetaData.java
+++ b/src/java/org/apache/cassandra/config/CFMetaData.java
@@ -470,12 +470,12 @@ public final class CFMetaData
  */
 public CFMetaData(String keyspace, String name, ColumnFamilyType type, 
CellNameType comp)
 {
-this(keyspace, name, type, comp, UUIDGen.getTimeUUID());
+this(keyspace, name, type, comp, null);
 }
 
-private CFMetaData(String keyspace, String name, ColumnFamilyType type, 
CellNameType comp, UUID id)
+public CFMetaData(String keyspace, String name, ColumnFamilyType type, 
CellNameType comp, UUID id)
 {
-cfId = id;
+cfId = id != null ? id : UUIDGen.getTimeUUID();
 ksName = keyspace;
 cfName = name;
 ksAndCFName = Pair.create(keyspace, name);

http://git-wip-us.apache.org/repos/asf/cassandra/blob/cee35e42/src/java/org/apache/cassandra/cql3/statements/AlterTableStatement.java
--
diff --git 
a/src/java/org/apache/cassandra/cql3/statements/AlterTableStatement.java 
b/src/java/org/apache/cassandra/cql3/statements/AlterTableStatement.java
index 44bc2bc..42b7f7f 100644
--- a/src/java/org/apache/cassandra/cql3/statements/AlterTableStatement.java
+++ b/src/java/org/apache/cassandra/cql3/statements/AlterTableStatement.java
@@ -87,6 +87,8 @@ public class AlterTableStatement extends 
SchemaAlteringStatement
 columnName = rawColumnName.prepare(cfm);
 def = cfm.getColumnDefinition(columnName);
 }
+if (cfProps.getId() != null)
+throw new ConfigurationException("Cannot alter table id.");
 
 switch (oType)
 {

http://git-wip-us.apache.org/repos/asf/cassandra/blob/cee35e42/src/java/org/apache/cassandra/cql3/statements/CFPropDefs.java
--
diff --git a/src/java/org/apache/cassandra/cql3/statements/CFPropDefs.java 
b/src/java/org/apache/cassandra/cql3/statements/CFPropDefs.java
index 343c0c0..17edd6d 100644
--- a/src/java/org/apache/cassandra/cql3/statements/CFPropDefs.java
+++ b/src/java/org/apache/cassandra/cql3/statements/CFPropDefs.java
@@ -17,10 +17,7 @@
  */
 package org.apache.cassandra.cql3.statements;
 
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.Map;
-import java.util.Set;
+import java.util.*;
 
 import org.apache.cassandra.cache.CachingOptions;
 import org.apache.cassandra.config.CFMetaData;
@@ -49,6 +46,8 @@ public class CFPropDefs extends PropertyDefinitions
 public static final String KW_COMPACTION = "compaction";
 public static final String KW_COMPRESSION = "compression";
 
+public static final String KW_ID = "id";
+
 public static final String COMPACTION_STRATEGY_CLASS_KEY = "class";
 
 public static 

[4/4] cassandra git commit: Merge branch 'cassandra-3.0' into trunk

[marcuse]

Merge branch 'cassandra-3.0' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d5f1175f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d5f1175f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d5f1175f

Branch: refs/heads/trunk
Commit: d5f1175f3cbb99c2d73d5318b838b46b8e2e1c8b
Parents: e5db3f2 db40ef8
Author: Marcus Eriksson 
Authored: Tue Dec 15 10:25:51 2015 +0100
Committer: Marcus Eriksson 
Committed: Tue Dec 15 10:25:51 2015 +0100

--
 CHANGES.txt |   1 +
 .../cql3/statements/CreateTableStatement.java   |   7 +-
 .../cql3/statements/CreateViewStatement.java|   1 +
 .../cql3/statements/TableAttributes.java|  19 
 .../operations/DropRecreateAndRestoreTest.java  | 104 +++
 5 files changed, 130 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/d5f1175f/CHANGES.txt
--
diff --cc CHANGES.txt
index eadd387,4af7573..0294efc
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -21,7 -3,12 +21,8 @@@ Merged from 3.0
 (CASSANDRA-10822)
  Merged from 2.2
   * Add property to allow listening on broadcast interface (CASSANDRA-9748)
 - * Fix regression in split size on CqlInputFormat (CASSANDRA-10835)
 - * Better handling of SSL connection errors inter-node (CASSANDRA-10816)
 - * Disable reloading of GossipingPropertyFileSnitch (CASSANDRA-9474)
 - * Verify tables in pseudo-system keyspaces at startup (CASSANDRA-10761)
  Merged from 2.1:
+  * Allow CREATE TABLE WITH ID (CASSANDRA-9179)
   * Make Stress compiles within eclipse (CASSANDRA-10807)
   * Cassandra Daemon should print JVM arguments (CASSANDRA-10764)
   * Allow cancellation of index summary redistribution (CASSANDRA-8805)

[jira] [Commented] (CASSANDRA-10871) MemtableFlushWriter blocks and no flushing happens

[Rafael Harutyunyan (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15057936#comment-15057936
 ] 

Rafael Harutyunyan commented on CASSANDRA-10871:


Incremental repairs are off. I was doing partial repair on one other node when 
this started. 

> MemtableFlushWriter blocks and no flushing happens
> --
>
> Key: CASSANDRA-10871
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10871
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction, Local Write-Read Paths
> Environment: Linux cassandra1 2.6.32-573.3.1.el6.x86_64 #1 SMP Thu 
> Aug 13 22:55:16 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux; Java(TM) SE Runtime 
> Environment (build 1.7.0_67-b01)
>Reporter: Rafael Harutyunyan
>Priority: Critical
> Fix For: 2.1.11
>
> Attachments: full_thread_dump.txt
>
>
> After some time MemtableFlushWriter thread blocks, resulting first full 
> filling of the FlushWriterQueue, than full filling of MutationStage queue. 
> After this 2 things might happen - Cassandra might drop the queued mutations 
> and everything becomes normal or it shuts down with insufficient HeapSpace.
> Here is the thread dump.
> {noformat}
> "MemtableFlushWriter:3" - Thread t@2610
>java.lang.Thread.State: BLOCKED
>   at 
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.handleNotification(WrappingCompactionStrategy.java:250)
>   - waiting to lock  (a 
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy) owned by 
> "CompactionExecutor:51" t@2638
>   at org.apache.cassandra.db.DataTracker.notifyAdded(DataTracker.java:518)
>   at 
> org.apache.cassandra.db.DataTracker.replaceFlushed(DataTracker.java:178)
>   at 
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.replaceFlushed(AbstractCompactionStrategy.java:234)
>   at 
> org.apache.cassandra.db.ColumnFamilyStore.replaceFlushed(ColumnFamilyStore.java:1502)
>   at 
> org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:336)
>   at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>   at 
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
>   at 
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1115)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <7ef8cd1b> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "MemtableFlushWriter:4" - Thread t@2616
>java.lang.Thread.State: BLOCKED
>   at 
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.handleNotification(WrappingCompactionStrategy.java:250)
>   - waiting to lock  (a 
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy) owned by 
> "CompactionExecutor:51" t@2638
>   at org.apache.cassandra.db.DataTracker.notifyAdded(DataTracker.java:518)
>   at 
> org.apache.cassandra.db.DataTracker.replaceFlushed(DataTracker.java:178)
>   at 
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.replaceFlushed(AbstractCompactionStrategy.java:234)
>   at 
> org.apache.cassandra.db.ColumnFamilyStore.replaceFlushed(ColumnFamilyStore.java:1502)
>   at 
> org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:336)
>   at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>   at 
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
>   at 
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1115)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <2f842d9b> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> {noformat}
> and here are the tpsats
> {noformat}
> Pool NameActive   Pending  Completed   Blocked  All 
> time blocked
> CounterMutationStage  0 0  0 0
>  0
> ReadStage 0 0 28 0
>  0
> RequestResponseStage  0 02020253 0
>  0
> MutationStage32 63221   27858588 0
>  0
> ReadRepairStage   0 0  0 0
>  0
> 

[jira] [Commented] (CASSANDRA-10871) MemtableFlushWriter blocks and no flushing happens

[Rafael Harutyunyan (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15057948#comment-15057948
 ] 

Rafael Harutyunyan commented on CASSANDRA-10871:


Yes, I have 14 servers. 12 with 64 vnodes, 2 with 256 vnodes (they are on a 
proportionally bigger hardware)

> MemtableFlushWriter blocks and no flushing happens
> --
>
> Key: CASSANDRA-10871
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10871
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction, Local Write-Read Paths
> Environment: Linux cassandra1 2.6.32-573.3.1.el6.x86_64 #1 SMP Thu 
> Aug 13 22:55:16 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux; Java(TM) SE Runtime 
> Environment (build 1.7.0_67-b01)
>Reporter: Rafael Harutyunyan
>Priority: Critical
> Fix For: 2.1.11
>
> Attachments: full_thread_dump.txt
>
>
> After some time MemtableFlushWriter thread blocks, resulting first full 
> filling of the FlushWriterQueue, than full filling of MutationStage queue. 
> After this 2 things might happen - Cassandra might drop the queued mutations 
> and everything becomes normal or it shuts down with insufficient HeapSpace.
> Here is the thread dump.
> {noformat}
> "MemtableFlushWriter:3" - Thread t@2610
>java.lang.Thread.State: BLOCKED
>   at 
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.handleNotification(WrappingCompactionStrategy.java:250)
>   - waiting to lock  (a 
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy) owned by 
> "CompactionExecutor:51" t@2638
>   at org.apache.cassandra.db.DataTracker.notifyAdded(DataTracker.java:518)
>   at 
> org.apache.cassandra.db.DataTracker.replaceFlushed(DataTracker.java:178)
>   at 
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.replaceFlushed(AbstractCompactionStrategy.java:234)
>   at 
> org.apache.cassandra.db.ColumnFamilyStore.replaceFlushed(ColumnFamilyStore.java:1502)
>   at 
> org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:336)
>   at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>   at 
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
>   at 
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1115)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <7ef8cd1b> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "MemtableFlushWriter:4" - Thread t@2616
>java.lang.Thread.State: BLOCKED
>   at 
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.handleNotification(WrappingCompactionStrategy.java:250)
>   - waiting to lock  (a 
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy) owned by 
> "CompactionExecutor:51" t@2638
>   at org.apache.cassandra.db.DataTracker.notifyAdded(DataTracker.java:518)
>   at 
> org.apache.cassandra.db.DataTracker.replaceFlushed(DataTracker.java:178)
>   at 
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.replaceFlushed(AbstractCompactionStrategy.java:234)
>   at 
> org.apache.cassandra.db.ColumnFamilyStore.replaceFlushed(ColumnFamilyStore.java:1502)
>   at 
> org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:336)
>   at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>   at 
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
>   at 
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1115)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <2f842d9b> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> {noformat}
> and here are the tpsats
> {noformat}
> Pool NameActive   Pending  Completed   Blocked  All 
> time blocked
> CounterMutationStage  0 0  0 0
>  0
> ReadStage 0 0 28 0
>  0
> RequestResponseStage  0 02020253 0
>  0
> MutationStage32 63221   27858588 0
>  0
> ReadRepairStage   0 0  0 0
>   

[jira] [Comment Edited] (CASSANDRA-10871) MemtableFlushWriter blocks and no flushing happens

[Rafael Harutyunyan (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15057948#comment-15057948
 ] 

Rafael Harutyunyan edited comment on CASSANDRA-10871 at 12/15/15 12:14 PM:
---

Yes, I have 14 servers. 12 with 64 vnodes, 2 with 256 vnodes (they are on a 
proportionally bigger hardware). The two nodes with 256 vnodes are the ones 
where I see this issue. The rest are fine. 


was (Author: rahar):
Yes, I have 14 servers. 12 with 64 vnodes, 2 with 256 vnodes (they are on a 
proportionally bigger hardware)

> MemtableFlushWriter blocks and no flushing happens
> --
>
> Key: CASSANDRA-10871
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10871
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction, Local Write-Read Paths
> Environment: Linux cassandra1 2.6.32-573.3.1.el6.x86_64 #1 SMP Thu 
> Aug 13 22:55:16 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux; Java(TM) SE Runtime 
> Environment (build 1.7.0_67-b01)
>Reporter: Rafael Harutyunyan
>Priority: Critical
> Attachments: full_thread_dump.txt
>
>
> After some time MemtableFlushWriter thread blocks, resulting first full 
> filling of the FlushWriterQueue, than full filling of MutationStage queue. 
> After this 2 things might happen - Cassandra might drop the queued mutations 
> and everything becomes normal or it shuts down with insufficient HeapSpace.
> Here is the thread dump.
> {noformat}
> "MemtableFlushWriter:3" - Thread t@2610
>java.lang.Thread.State: BLOCKED
>   at 
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.handleNotification(WrappingCompactionStrategy.java:250)
>   - waiting to lock  (a 
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy) owned by 
> "CompactionExecutor:51" t@2638
>   at org.apache.cassandra.db.DataTracker.notifyAdded(DataTracker.java:518)
>   at 
> org.apache.cassandra.db.DataTracker.replaceFlushed(DataTracker.java:178)
>   at 
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.replaceFlushed(AbstractCompactionStrategy.java:234)
>   at 
> org.apache.cassandra.db.ColumnFamilyStore.replaceFlushed(ColumnFamilyStore.java:1502)
>   at 
> org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:336)
>   at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>   at 
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
>   at 
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1115)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <7ef8cd1b> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "MemtableFlushWriter:4" - Thread t@2616
>java.lang.Thread.State: BLOCKED
>   at 
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.handleNotification(WrappingCompactionStrategy.java:250)
>   - waiting to lock  (a 
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy) owned by 
> "CompactionExecutor:51" t@2638
>   at org.apache.cassandra.db.DataTracker.notifyAdded(DataTracker.java:518)
>   at 
> org.apache.cassandra.db.DataTracker.replaceFlushed(DataTracker.java:178)
>   at 
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.replaceFlushed(AbstractCompactionStrategy.java:234)
>   at 
> org.apache.cassandra.db.ColumnFamilyStore.replaceFlushed(ColumnFamilyStore.java:1502)
>   at 
> org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:336)
>   at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>   at 
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
>   at 
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1115)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <2f842d9b> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> {noformat}
> and here are the tpsats
> {noformat}
> Pool NameActive   Pending  Completed   Blocked  All 
> time blocked
> CounterMutationStage  0 0  0 0
>  0
> ReadStage 0 0 28 0
>  0
> RequestResponseStage 

[jira] [Commented] (CASSANDRA-10871) MemtableFlushWriter blocks and no flushing happens

[Marcus Eriksson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15057943#comment-15057943
 ] 

Marcus Eriksson commented on CASSANDRA-10871:
-

vnodes?

> MemtableFlushWriter blocks and no flushing happens
> --
>
> Key: CASSANDRA-10871
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10871
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction, Local Write-Read Paths
> Environment: Linux cassandra1 2.6.32-573.3.1.el6.x86_64 #1 SMP Thu 
> Aug 13 22:55:16 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux; Java(TM) SE Runtime 
> Environment (build 1.7.0_67-b01)
>Reporter: Rafael Harutyunyan
>Priority: Critical
> Fix For: 2.1.11
>
> Attachments: full_thread_dump.txt
>
>
> After some time MemtableFlushWriter thread blocks, resulting first full 
> filling of the FlushWriterQueue, than full filling of MutationStage queue. 
> After this 2 things might happen - Cassandra might drop the queued mutations 
> and everything becomes normal or it shuts down with insufficient HeapSpace.
> Here is the thread dump.
> {noformat}
> "MemtableFlushWriter:3" - Thread t@2610
>java.lang.Thread.State: BLOCKED
>   at 
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.handleNotification(WrappingCompactionStrategy.java:250)
>   - waiting to lock  (a 
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy) owned by 
> "CompactionExecutor:51" t@2638
>   at org.apache.cassandra.db.DataTracker.notifyAdded(DataTracker.java:518)
>   at 
> org.apache.cassandra.db.DataTracker.replaceFlushed(DataTracker.java:178)
>   at 
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.replaceFlushed(AbstractCompactionStrategy.java:234)
>   at 
> org.apache.cassandra.db.ColumnFamilyStore.replaceFlushed(ColumnFamilyStore.java:1502)
>   at 
> org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:336)
>   at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>   at 
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
>   at 
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1115)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <7ef8cd1b> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "MemtableFlushWriter:4" - Thread t@2616
>java.lang.Thread.State: BLOCKED
>   at 
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.handleNotification(WrappingCompactionStrategy.java:250)
>   - waiting to lock  (a 
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy) owned by 
> "CompactionExecutor:51" t@2638
>   at org.apache.cassandra.db.DataTracker.notifyAdded(DataTracker.java:518)
>   at 
> org.apache.cassandra.db.DataTracker.replaceFlushed(DataTracker.java:178)
>   at 
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.replaceFlushed(AbstractCompactionStrategy.java:234)
>   at 
> org.apache.cassandra.db.ColumnFamilyStore.replaceFlushed(ColumnFamilyStore.java:1502)
>   at 
> org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:336)
>   at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>   at 
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
>   at 
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1115)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <2f842d9b> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> {noformat}
> and here are the tpsats
> {noformat}
> Pool NameActive   Pending  Completed   Blocked  All 
> time blocked
> CounterMutationStage  0 0  0 0
>  0
> ReadStage 0 0 28 0
>  0
> RequestResponseStage  0 02020253 0
>  0
> MutationStage32 63221   27858588 0
>  0
> ReadRepairStage   0 0  0 0
>  0
> GossipStage   0 0  16430 0
>  0
> 

[jira] [Commented] (CASSANDRA-10871) MemtableFlushWriter blocks and no flushing happens

[Rafael Harutyunyan (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15057966#comment-15057966
 ] 

Rafael Harutyunyan commented on CASSANDRA-10871:


Do you have any suggestion how can I eliminate this for now? The two nodes that 
have 256 vnodes are the ones that face this issue. 

> MemtableFlushWriter blocks and no flushing happens
> --
>
> Key: CASSANDRA-10871
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10871
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction, Local Write-Read Paths
> Environment: Linux cassandra1 2.6.32-573.3.1.el6.x86_64 #1 SMP Thu 
> Aug 13 22:55:16 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux; Java(TM) SE Runtime 
> Environment (build 1.7.0_67-b01)
>Reporter: Rafael Harutyunyan
>Priority: Critical
> Attachments: full_thread_dump.txt
>
>
> After some time MemtableFlushWriter thread blocks, resulting first full 
> filling of the FlushWriterQueue, than full filling of MutationStage queue. 
> After this 2 things might happen - Cassandra might drop the queued mutations 
> and everything becomes normal or it shuts down with insufficient HeapSpace.
> Here is the thread dump.
> {noformat}
> "MemtableFlushWriter:3" - Thread t@2610
>java.lang.Thread.State: BLOCKED
>   at 
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.handleNotification(WrappingCompactionStrategy.java:250)
>   - waiting to lock  (a 
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy) owned by 
> "CompactionExecutor:51" t@2638
>   at org.apache.cassandra.db.DataTracker.notifyAdded(DataTracker.java:518)
>   at 
> org.apache.cassandra.db.DataTracker.replaceFlushed(DataTracker.java:178)
>   at 
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.replaceFlushed(AbstractCompactionStrategy.java:234)
>   at 
> org.apache.cassandra.db.ColumnFamilyStore.replaceFlushed(ColumnFamilyStore.java:1502)
>   at 
> org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:336)
>   at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>   at 
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
>   at 
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1115)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <7ef8cd1b> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "MemtableFlushWriter:4" - Thread t@2616
>java.lang.Thread.State: BLOCKED
>   at 
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.handleNotification(WrappingCompactionStrategy.java:250)
>   - waiting to lock  (a 
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy) owned by 
> "CompactionExecutor:51" t@2638
>   at org.apache.cassandra.db.DataTracker.notifyAdded(DataTracker.java:518)
>   at 
> org.apache.cassandra.db.DataTracker.replaceFlushed(DataTracker.java:178)
>   at 
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.replaceFlushed(AbstractCompactionStrategy.java:234)
>   at 
> org.apache.cassandra.db.ColumnFamilyStore.replaceFlushed(ColumnFamilyStore.java:1502)
>   at 
> org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:336)
>   at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>   at 
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
>   at 
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1115)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <2f842d9b> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> {noformat}
> and here are the tpsats
> {noformat}
> Pool NameActive   Pending  Completed   Blocked  All 
> time blocked
> CounterMutationStage  0 0  0 0
>  0
> ReadStage 0 0 28 0
>  0
> RequestResponseStage  0 02020253 0
>  0
> MutationStage32 63221   27858588 0
>  0
> ReadRepairStage   0 0  0 0
>  

[jira] [Commented] (CASSANDRA-10871) MemtableFlushWriter blocks and no flushing happens

[Marcus Eriksson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15057964#comment-15057964
 ] 

Marcus Eriksson commented on CASSANDRA-10871:
-

but, for the record, you should probably investigate why your hardware can't 
keep up with compactions so that it ends up with that many sstables

> MemtableFlushWriter blocks and no flushing happens
> --
>
> Key: CASSANDRA-10871
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10871
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction, Local Write-Read Paths
> Environment: Linux cassandra1 2.6.32-573.3.1.el6.x86_64 #1 SMP Thu 
> Aug 13 22:55:16 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux; Java(TM) SE Runtime 
> Environment (build 1.7.0_67-b01)
>Reporter: Rafael Harutyunyan
>Priority: Critical
> Attachments: full_thread_dump.txt
>
>
> After some time MemtableFlushWriter thread blocks, resulting first full 
> filling of the FlushWriterQueue, than full filling of MutationStage queue. 
> After this 2 things might happen - Cassandra might drop the queued mutations 
> and everything becomes normal or it shuts down with insufficient HeapSpace.
> Here is the thread dump.
> {noformat}
> "MemtableFlushWriter:3" - Thread t@2610
>java.lang.Thread.State: BLOCKED
>   at 
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.handleNotification(WrappingCompactionStrategy.java:250)
>   - waiting to lock  (a 
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy) owned by 
> "CompactionExecutor:51" t@2638
>   at org.apache.cassandra.db.DataTracker.notifyAdded(DataTracker.java:518)
>   at 
> org.apache.cassandra.db.DataTracker.replaceFlushed(DataTracker.java:178)
>   at 
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.replaceFlushed(AbstractCompactionStrategy.java:234)
>   at 
> org.apache.cassandra.db.ColumnFamilyStore.replaceFlushed(ColumnFamilyStore.java:1502)
>   at 
> org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:336)
>   at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>   at 
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
>   at 
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1115)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <7ef8cd1b> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "MemtableFlushWriter:4" - Thread t@2616
>java.lang.Thread.State: BLOCKED
>   at 
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.handleNotification(WrappingCompactionStrategy.java:250)
>   - waiting to lock  (a 
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy) owned by 
> "CompactionExecutor:51" t@2638
>   at org.apache.cassandra.db.DataTracker.notifyAdded(DataTracker.java:518)
>   at 
> org.apache.cassandra.db.DataTracker.replaceFlushed(DataTracker.java:178)
>   at 
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.replaceFlushed(AbstractCompactionStrategy.java:234)
>   at 
> org.apache.cassandra.db.ColumnFamilyStore.replaceFlushed(ColumnFamilyStore.java:1502)
>   at 
> org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:336)
>   at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>   at 
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
>   at 
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1115)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <2f842d9b> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> {noformat}
> and here are the tpsats
> {noformat}
> Pool NameActive   Pending  Completed   Blocked  All 
> time blocked
> CounterMutationStage  0 0  0 0
>  0
> ReadStage 0 0 28 0
>  0
> RequestResponseStage  0 02020253 0
>  0
> MutationStage32 63221   27858588 0
>  0
> ReadRepairStage   0 0  0 0
>

[jira] [Commented] (CASSANDRA-10845) jmxmetrics_test.TestJMXMetrics.begin_test is failing

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15057945#comment-15057945
 ] 

Stefan Podkowinski commented on CASSANDRA-10845:


The stress-tool will exit with an error in case it can't create a table that 
already exists (counter1). You can use a custom yaml profile as workaround, see 
patch. 

> jmxmetrics_test.TestJMXMetrics.begin_test is failing
> 
>
> Key: CASSANDRA-10845
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10845
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Testing
>Reporter: Philip Thompson
>Assignee: Philip Thompson
>
> This test is failing on 2.1-head. There appear to be structural issues with 
> the test, and no C* bug to be fixed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-10871) MemtableFlushWriter blocks and no flushing happens

[Marcus Eriksson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson resolved CASSANDRA-10871.
-
   Resolution: Duplicate
Fix Version/s: (was: 2.1.11)

ok, this is a duplicate of CASSANDRA-10099

> MemtableFlushWriter blocks and no flushing happens
> --
>
> Key: CASSANDRA-10871
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10871
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction, Local Write-Read Paths
> Environment: Linux cassandra1 2.6.32-573.3.1.el6.x86_64 #1 SMP Thu 
> Aug 13 22:55:16 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux; Java(TM) SE Runtime 
> Environment (build 1.7.0_67-b01)
>Reporter: Rafael Harutyunyan
>Priority: Critical
> Attachments: full_thread_dump.txt
>
>
> After some time MemtableFlushWriter thread blocks, resulting first full 
> filling of the FlushWriterQueue, than full filling of MutationStage queue. 
> After this 2 things might happen - Cassandra might drop the queued mutations 
> and everything becomes normal or it shuts down with insufficient HeapSpace.
> Here is the thread dump.
> {noformat}
> "MemtableFlushWriter:3" - Thread t@2610
>java.lang.Thread.State: BLOCKED
>   at 
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.handleNotification(WrappingCompactionStrategy.java:250)
>   - waiting to lock  (a 
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy) owned by 
> "CompactionExecutor:51" t@2638
>   at org.apache.cassandra.db.DataTracker.notifyAdded(DataTracker.java:518)
>   at 
> org.apache.cassandra.db.DataTracker.replaceFlushed(DataTracker.java:178)
>   at 
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.replaceFlushed(AbstractCompactionStrategy.java:234)
>   at 
> org.apache.cassandra.db.ColumnFamilyStore.replaceFlushed(ColumnFamilyStore.java:1502)
>   at 
> org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:336)
>   at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>   at 
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
>   at 
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1115)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <7ef8cd1b> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "MemtableFlushWriter:4" - Thread t@2616
>java.lang.Thread.State: BLOCKED
>   at 
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.handleNotification(WrappingCompactionStrategy.java:250)
>   - waiting to lock  (a 
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy) owned by 
> "CompactionExecutor:51" t@2638
>   at org.apache.cassandra.db.DataTracker.notifyAdded(DataTracker.java:518)
>   at 
> org.apache.cassandra.db.DataTracker.replaceFlushed(DataTracker.java:178)
>   at 
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.replaceFlushed(AbstractCompactionStrategy.java:234)
>   at 
> org.apache.cassandra.db.ColumnFamilyStore.replaceFlushed(ColumnFamilyStore.java:1502)
>   at 
> org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:336)
>   at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>   at 
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
>   at 
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1115)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - locked <2f842d9b> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> {noformat}
> and here are the tpsats
> {noformat}
> Pool NameActive   Pending  Completed   Blocked  All 
> time blocked
> CounterMutationStage  0 0  0 0
>  0
> ReadStage 0 0 28 0
>  0
> RequestResponseStage  0 02020253 0
>  0
> MutationStage32 63221   27858588 0
>  0
> ReadRepairStage   0 0  0 0
>  0
> GossipStage   0 0  16430 

[jira] [Commented] (CASSANDRA-9258) Range movement causes CPU & performance impact

[Branimir Lambov (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15057958#comment-15057958
 ] 

Branimir Lambov commented on CASSANDRA-9258:


The tests, as well as {{PendingRangesMap.iterator}} don't follow Cassandra's 
opening bracket convention.

Nit: the repetition in {{PendingRangesMap.addPendingRange}} could be removed by 
extracting the action in a private static method.
Smaller nit: the bigger/smaller selection would also look better if the 
intersection is done in a method call.

Pushed to [github 
branch|https://github.com/blambov/cassandra/tree/dikang85/9258] to have cassci 
run [unit 
tests|http://cassci.datastax.com/view/Dev/view/blambov/job/blambov-dikang85-9258-testall/]
 and 
[dtests|http://cassci.datastax.com/view/Dev/view/blambov/job/blambov-dikang85-9258-dtest/].

Note: If you want to invest more time in this, it could be made even more 
efficient, and perhaps cleaner, if you use two levels of indexing and 
unwrapping of the ranges.


> Range movement causes CPU & performance impact
> --
>
> Key: CASSANDRA-9258
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9258
> Project: Cassandra
>  Issue Type: Bug
> Environment: Cassandra 2.1.4
>Reporter: Rick Branson
>Assignee: Dikang Gu
> Fix For: 2.1.x
>
> Attachments: 0001-pending-ranges-map.patch
>
>
> Observing big CPU & latency regressions when doing range movements on 
> clusters with many tens of thousands of vnodes. See CPU usage increase by 
> ~80% when a single node is being replaced.
> Top methods are:
> 1) Ljava/math/BigInteger;.compareTo in 
> Lorg/apache/cassandra/dht/ComparableObjectToken;.compareTo 
> 2) Lcom/google/common/collect/AbstractMapBasedMultimap;.wrapCollection in 
> Lcom/google/common/collect/AbstractMapBasedMultimap$AsMap$AsMapIterator;.next
> 3) Lorg/apache/cassandra/db/DecoratedKey;.compareTo in 
> Lorg/apache/cassandra/dht/Range;.contains
> Here's a sample stack from a thread dump:
> {code}
> "Thrift:50673" daemon prio=10 tid=0x7f2f20164800 nid=0x3a04af runnable 
> [0x7f2d878d]
>java.lang.Thread.State: RUNNABLE
>   at org.apache.cassandra.dht.Range.isWrapAround(Range.java:260)
>   at org.apache.cassandra.dht.Range.contains(Range.java:51)
>   at org.apache.cassandra.dht.Range.contains(Range.java:110)
>   at 
> org.apache.cassandra.locator.TokenMetadata.pendingEndpointsFor(TokenMetadata.java:916)
>   at 
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:775)
>   at 
> org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:541)
>   at 
> org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:616)
>   at 
> org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:1101)
>   at 
> org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:1083)
>   at 
> org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:976)
>   at 
> org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3996)
>   at 
> org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3980)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>   at 
> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:205)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745){code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

cassandra git commit: Sort compactionhistory output by timestamp

[yukim]

Repository: cassandra
Updated Branches:
  refs/heads/trunk 7139f959d -> 07c6a36cc


Sort compactionhistory output by timestamp

patch by Michael Edge; reviewed by yukim for CASSANDRA-10464


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/07c6a36c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/07c6a36c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/07c6a36c

Branch: refs/heads/trunk
Commit: 07c6a36cc3c9661348688256dff69ef1b795f30a
Parents: 7139f95
Author: michael 
Authored: Tue Nov 17 14:22:21 2015 +0800
Committer: Yuki Morishita 
Committed: Tue Dec 15 14:29:40 2015 -0600

--
 CHANGES.txt |  1 +
 .../tools/nodetool/CompactionHistory.java   | 74 ++--
 2 files changed, 70 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/07c6a36c/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 4149d91..991d42a 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.2
+ * Sort compactionhistory output by timestamp (CASSANDRA-10464)
  * More efficient BTree removal (CASSANDRA-9991)
  * Make tablehistograms accept the same syntax as tablestats (CASSANDRA-10149)
  * Group pending compactions based on table (CASSANDRA-10718)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/07c6a36c/src/java/org/apache/cassandra/tools/nodetool/CompactionHistory.java
--
diff --git 
a/src/java/org/apache/cassandra/tools/nodetool/CompactionHistory.java 
b/src/java/org/apache/cassandra/tools/nodetool/CompactionHistory.java
index cbb054a..1348d05 100644
--- a/src/java/org/apache/cassandra/tools/nodetool/CompactionHistory.java
+++ b/src/java/org/apache/cassandra/tools/nodetool/CompactionHistory.java
@@ -17,17 +17,22 @@
  */
 package org.apache.cassandra.tools.nodetool;
 
-import static com.google.common.collect.Iterables.toArray;
-import io.airlift.command.Command;
-
+import java.time.Instant;
+import java.time.LocalDateTime;
+import java.time.ZoneId;
+import java.util.ArrayList;
+import java.util.Collections;
 import java.util.List;
 import java.util.Set;
-
 import javax.management.openmbean.TabularData;
 
+import io.airlift.command.Command;
+
 import org.apache.cassandra.tools.NodeProbe;
 import org.apache.cassandra.tools.NodeTool.NodeToolCmd;
 
+import static com.google.common.collect.Iterables.toArray;
+
 @Command(name = "compactionhistory", description = "Print history of 
compaction")
 public class CompactionHistory extends NodeToolCmd
 {
@@ -48,10 +53,69 @@ public class CompactionHistory extends NodeToolCmd
 System.out.printf(format, toArray(indexNames, Object.class));
 
 Set values = tabularData.keySet();
+List chr = new ArrayList<>();
 for (Object eachValue : values)
 {
 List value = (List) eachValue;
-System.out.printf(format, toArray(value, Object.class));
+CompactionHistoryRow chc = new 
CompactionHistoryRow((String)value.get(0),
+
(String)value.get(1),
+
(String)value.get(2),
+
(Long)value.get(3),
+
(Long)value.get(4),
+
(Long)value.get(5),
+
(String)value.get(6));
+chr.add(chc);
 }
+Collections.sort(chr);
+for (CompactionHistoryRow eachChc : chr)
+{
+System.out.printf(format, eachChc.getAllAsArray());
+}
+}
+}
+
+/**
+ * Allows the Compaction History output to be ordered by 'compactedAt' - that 
is the
+ * time at which compaction finished.
+ */
+class CompactionHistoryRow implements Comparable
+{
+private final String id;
+private final String ksName;
+private final String cfName;
+private final long compactedAt;
+private final long bytesIn;
+private final long bytesOut;
+private final String rowMerged;
+
+CompactionHistoryRow(String id, String ksName, String cfName, long 
compactedAt, long bytesIn, long bytesOut, String rowMerged)
+{
+this.id = id;
+this.ksName = ksName;
+this.cfName = cfName;
+this.compactedAt = compactedAt;
+this.bytesIn = bytesIn;
+this.bytesOut = bytesOut;
+this.rowMerged = rowMerged;
+}
+
+public int compareTo(CompactionHistoryRow chc)
+{
+return 

[jira] [Commented] (CASSANDRA-10850) v4 spec has tons of grammatical mistakes

[Sandeep Tamhankar (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15058992#comment-15058992
 ] 

Sandeep Tamhankar commented on CASSANDRA-10850:
---

Yeah, I didn't make any assumptions regarding the answers to the outstanding 
questions, so I didn't change the text for those.

Also, I didn't try this, but I presumed the process was that I was supposed to 
submit a patch, as opposed to trying to commit/push a branch and then 
submitting a PR. I'd be happy to go the push/PR route, assuming I have the 
appropriate permissions.

> v4 spec has tons of grammatical mistakes
> 
>
> Key: CASSANDRA-10850
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10850
> Project: Cassandra
>  Issue Type: Bug
>  Components: Documentation and Website
>Reporter: Sandeep Tamhankar
> Fix For: 3.0.2
>
> Attachments: v4-protocol.patch
>
>
> https://github.com/apache/cassandra/blob/cassandra-3.0/doc/native_protocol_v4.spec
> I notice the following in the first section of the spec and then gave up:
> "The list of allowed opcode is defined Section 2.3" => "The list of allowed 
> opcode*s* is defined in Section 2.3"
> "the details of each corresponding message is described Section 4" => "the 
> details of each corresponding message are described in Section 4" since the 
> subject is details, not message.
> "Requests are those frame sent by" => "Requests are those frame*s* sent by"
> I think someone should go through the whole spec and fix all the mistakes 
> rather than me pointing out the ones I notice piece-meal. I found the grammar 
> errors to be rather distracting.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-10874) running stress with compaction strategy and replication factor fails on read after write

[Andrew Hust (JIRA)]

Andrew Hust created CASSANDRA-10874:
---

 Summary: running stress with compaction strategy and replication 
factor fails on read after write
 Key: CASSANDRA-10874
 URL: https://issues.apache.org/jira/browse/CASSANDRA-10874
 Project: Cassandra
  Issue Type: Bug
  Components: Tools
Reporter: Andrew Hust


When running a read stress after write stress with a compaction strategy and 
replication factor matching the node count will fail with an exception.  
{code}
Operation x0 on key(s) [38343433384b34364c30]: Data returned was not validated
{code}

Example run:
{code}
ccm create stress -v git:cassandra-3.0 -n 3 -s
ccm node1 stress write n=10M -rate threads=300 -schema replication\(factor=3\) 
compaction\(strategy=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy\)
ccm node1 nodetool flush
ccm node1 nodetool compactionstats # check until quiet
ccm node1 stress read n=10M -rate threads=300
{code}
- This will fail with/out vnodes but will occasionally pass without vnodes. 
- Changing the read phase to be CL=QUORUM will make it pass.  
- Removing the replication factor on write will make it pass.
- Happens on all compaction strategies

So with that in mind I attempted to add a repair after the write phase.  This 
leads to 1 of 2 outcomes.

1: a repair that has a greater than 100% completion, usually stalls after a 
bit, but have seen it get to >400% progress:
{code}
  id   compaction typekeyspace   
table completed totalunit   progress
2d5344c0-9dc8-11e5-9d5f-4fdec8d76c27Validation   keyspace1   
standard1   94722609949   44035292145   bytes215.11%
{code}

2: a repair that has a greatly inflated completed/total value, it will crunch 
for a bit then lockup:
{code}
 id   compaction typekeyspace   
table   completed  totalunit   progress
   8c4cf7f0-a34a-11e5-a321-777be88c58aeValidation   keyspace1   
standard1   0   874811100900   bytes  0.00%

❯ du -sh ~/.ccm/stress/node1/
2.4G  ~/.ccm/stress/node1/
❯ du -sh ~/.ccm/stress
7.1G  ~/.ccm/stress
{code}

This has been reproduced on cassandra-3.0 and cassandra-2.2 both locally and 
using cstar_perf (links below).  
A big twist is that cassandra-2.2 will pass the majority of the time.  It will 
complete successfully without the repair 8 out of 10 runs.  This can be seen in 
the cstar_perf links below.

cstar_perf runs:
http://cstar.datastax.com/tests/id/a8b6af02-a2ce-11e5-bb72-0256e416528f
http://cstar.datastax.com/tests/id/a254c572-a2ce-11e5-a8b9-0256e416528f



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10874) running stress with compaction strategy and replication factor fails on read after write

[Andrew Hust (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Hust updated CASSANDRA-10874:

Description: 
When running a read stress after write stress with a compaction strategy and 
replication factor matching the node count will fail with an exception.  
{code}
Operation x0 on key(s) [38343433384b34364c30]: Data returned was not validated
{code}

Example run:
{code}
ccm create stress -v git:cassandra-3.0 -n 3 -s
ccm node1 stress write n=10M -rate threads=300 -schema replication\(factor=3\) 
compaction\(strategy=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy\)
ccm node1 nodetool flush
ccm node1 nodetool compactionstats # check until quiet
ccm node1 stress read n=10M -rate threads=300
{code}
- This will fail with/out vnodes but will occasionally pass without vnodes. 
- Changing the read phase to be CL=QUORUM will make it pass.  
- Removing the replication factor on write will make it pass.
- Happens on all compaction strategies

So with that in mind I attempted to add a repair after the write phase.  This 
leads to 1 of 2 outcomes.

1: a repair that has a greater than 100% completion, usually stalls after a 
bit, but have seen it get to >400% progress:
{code}
  id   compaction typekeyspace   
table completed totalunit   progress
2d5344c0-9dc8-11e5-9d5f-4fdec8d76c27Validation   keyspace1   
standard1   94722609949   44035292145   bytes215.11%
{code}

2: a repair that has a greatly inflated completed/total value, it will crunch 
for a bit then lockup:
{code}
 id   compaction typekeyspace   
table   completed  totalunit   progress
   8c4cf7f0-a34a-11e5-a321-777be88c58aeValidation   keyspace1   
standard1   0   874811100900   bytes  0.00%

❯ du -sh ~/.ccm/stress/node1/
2.4G  ~/.ccm/stress/node1/
❯ du -sh ~/.ccm/stress
7.1G  ~/.ccm/stress
{code}

This has been reproduced on cassandra-3.0 and cassandra-2.2 both locally and 
using cstar_perf (links below).  
A big twist is that cassandra-2.2 will pass the majority of the time.  It will 
complete successfully without the repair 8 out of 10 runs.  This can be seen in 
the cstar_perf links below.

cstar_perf runs:
http://cstar.datastax.com/tests/id/c8fa27a4-a205-11e5-8fbc-0256e416528f
http://cstar.datastax.com/tests/id/a254c572-a2ce-11e5-a8b9-0256e416528f

  was:
When running a read stress after write stress with a compaction strategy and 
replication factor matching the node count will fail with an exception.  
{code}
Operation x0 on key(s) [38343433384b34364c30]: Data returned was not validated
{code}

Example run:
{code}
ccm create stress -v git:cassandra-3.0 -n 3 -s
ccm node1 stress write n=10M -rate threads=300 -schema replication\(factor=3\) 
compaction\(strategy=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy\)
ccm node1 nodetool flush
ccm node1 nodetool compactionstats # check until quiet
ccm node1 stress read n=10M -rate threads=300
{code}
- This will fail with/out vnodes but will occasionally pass without vnodes. 
- Changing the read phase to be CL=QUORUM will make it pass.  
- Removing the replication factor on write will make it pass.
- Happens on all compaction strategies

So with that in mind I attempted to add a repair after the write phase.  This 
leads to 1 of 2 outcomes.

1: a repair that has a greater than 100% completion, usually stalls after a 
bit, but have seen it get to >400% progress:
{code}
  id   compaction typekeyspace   
table completed totalunit   progress
2d5344c0-9dc8-11e5-9d5f-4fdec8d76c27Validation   keyspace1   
standard1   94722609949   44035292145   bytes215.11%
{code}

2: a repair that has a greatly inflated completed/total value, it will crunch 
for a bit then lockup:
{code}
 id   compaction typekeyspace   
table   completed  totalunit   progress
   8c4cf7f0-a34a-11e5-a321-777be88c58aeValidation   keyspace1   
standard1   0   874811100900   bytes  0.00%

❯ du -sh ~/.ccm/stress/node1/
2.4G  ~/.ccm/stress/node1/
❯ du -sh ~/.ccm/stress
7.1G  ~/.ccm/stress
{code}

This has been reproduced on cassandra-3.0 and cassandra-2.2 both locally and 
using cstar_perf (links below).  
A big twist is that cassandra-2.2 will pass the majority of the time.  It will 
complete successfully without the repair 8 out of 10 runs.  This can be seen in 
the cstar_perf links below.

cstar_perf runs:
http://cstar.datastax.com/tests/id/a8b6af02-a2ce-11e5-bb72-0256e416528f
http://cstar.datastax.com/tests/id/a254c572-a2ce-11e5-a8b9-0256e416528f


> running stress with compaction strategy and replication factor fails on read 
> after write
> 

[jira] [Commented] (CASSANDRA-10850) v4 spec has tons of grammatical mistakes

[Michael Shuler (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15058831#comment-15058831
 ] 

Michael Shuler commented on CASSANDRA-10850:


[~stamhankar999] does your patch include updates for your previous outstanding 
questions? (4.1.4 - For 0x01 flag..., etc.) Do those still need to be looked at 
by someone?

> v4 spec has tons of grammatical mistakes
> 
>
> Key: CASSANDRA-10850
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10850
> Project: Cassandra
>  Issue Type: Bug
>  Components: Documentation and Website
>Reporter: Sandeep Tamhankar
> Fix For: 3.0.2
>
> Attachments: v4-protocol.patch
>
>
> https://github.com/apache/cassandra/blob/cassandra-3.0/doc/native_protocol_v4.spec
> I notice the following in the first section of the spec and then gave up:
> "The list of allowed opcode is defined Section 2.3" => "The list of allowed 
> opcode*s* is defined in Section 2.3"
> "the details of each corresponding message is described Section 4" => "the 
> details of each corresponding message are described in Section 4" since the 
> subject is details, not message.
> "Requests are those frame sent by" => "Requests are those frame*s* sent by"
> I think someone should go through the whole spec and fix all the mistakes 
> rather than me pointing out the ones I notice piece-meal. I found the grammar 
> errors to be rather distracting.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[3/6] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2

[aleksey]

http://git-wip-us.apache.org/repos/asf/cassandra/blob/57d558fc/bin/cqlsh.py
--
diff --cc bin/cqlsh.py
index a3fa666,000..42c2923
mode 100644,00..100644
--- a/bin/cqlsh.py
+++ b/bin/cqlsh.py
@@@ -1,2763 -1,0 +1,2486 @@@
 +#!/bin/sh
 +# -*- mode: Python -*-
 +
 +# Licensed to the Apache Software Foundation (ASF) under one
 +# or more contributor license agreements.  See the NOTICE file
 +# distributed with this work for additional information
 +# regarding copyright ownership.  The ASF licenses this file
 +# to you under the Apache License, Version 2.0 (the
 +# "License"); you may not use this file except in compliance
 +# with the License.  You may obtain a copy of the License at
 +#
 +# http://www.apache.org/licenses/LICENSE-2.0
 +#
 +# Unless required by applicable law or agreed to in writing, software
 +# distributed under the License is distributed on an "AS IS" BASIS,
 +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 +# See the License for the specific language governing permissions and
 +# limitations under the License.
 +
 +""":"
 +# bash code here; finds a suitable python interpreter and execs this file.
 +# prefer unqualified "python" if suitable:
 +python -c 'import sys; sys.exit(not (0x020500b0 < sys.hexversion < 
0x0300))' 2>/dev/null \
 +&& exec python "$0" "$@"
 +for pyver in 2.6 2.7 2.5; do
 +which python$pyver > /dev/null 2>&1 && exec python$pyver "$0" "$@"
 +done
 +echo "No appropriate python interpreter found." >&2
 +exit 1
 +":"""
 +
 +from __future__ import with_statement
 +
 +import cmd
 +import codecs
 +import ConfigParser
 +import csv
 +import getpass
 +import locale
- import multiprocessing as mp
 +import optparse
 +import os
 +import platform
 +import sys
 +import time
 +import traceback
 +import warnings
 +import webbrowser
++from StringIO import StringIO
 +from contextlib import contextmanager
- from functools import partial
 +from glob import glob
- from StringIO import StringIO
 +from uuid import UUID
 +
 +if sys.version_info[0] != 2 or sys.version_info[1] != 7:
 +sys.exit("\nCQL Shell supports only Python 2.7\n")
 +
 +description = "CQL Shell for Apache Cassandra"
 +version = "5.0.1"
 +
 +readline = None
 +try:
 +# check if tty first, cause readline doesn't check, and only cares
 +# about $TERM. we don't want the funky escape code stuff to be
 +# output if not a tty.
 +if sys.stdin.isatty():
 +import readline
 +except ImportError:
 +pass
 +
 +CQL_LIB_PREFIX = 'cassandra-driver-internal-only-'
 +
 +CASSANDRA_PATH = os.path.join(os.path.dirname(os.path.realpath(__file__)), 
'..')
 +CASSANDRA_CQL_HTML_FALLBACK = 
'https://cassandra.apache.org/doc/cql3/CQL-2.2.html'
 +
 +if os.path.exists(CASSANDRA_PATH + '/doc/cql3/CQL.html'):
 +# default location of local CQL.html
 +CASSANDRA_CQL_HTML = 'file://' + CASSANDRA_PATH + '/doc/cql3/CQL.html'
 +elif os.path.exists('/usr/share/doc/cassandra/CQL.html'):
 +# fallback to package file
 +CASSANDRA_CQL_HTML = 'file:///usr/share/doc/cassandra/CQL.html'
 +else:
 +# fallback to online version
 +CASSANDRA_CQL_HTML = CASSANDRA_CQL_HTML_FALLBACK
 +
 +# On Linux, the Python webbrowser module uses the 'xdg-open' executable
 +# to open a file/URL. But that only works, if the current session has been
 +# opened from _within_ a desktop environment. I.e. 'xdg-open' will fail,
 +# if the session's been opened via ssh to a remote box.
 +#
 +# Use 'python' to get some information about the detected browsers.
 +# >>> import webbrowser
 +# >>> webbrowser._tryorder
 +# >>> webbrowser._browser
 +#
 +if len(webbrowser._tryorder) == 0:
 +CASSANDRA_CQL_HTML = CASSANDRA_CQL_HTML_FALLBACK
 +elif webbrowser._tryorder[0] == 'xdg-open' and 
os.environ.get('XDG_DATA_DIRS', '') == '':
 +# only on Linux (some OS with xdg-open)
 +webbrowser._tryorder.remove('xdg-open')
 +webbrowser._tryorder.append('xdg-open')
 +
 +# use bundled libs for python-cql and thrift, if available. if there
 +# is a ../lib dir, use bundled libs there preferentially.
 +ZIPLIB_DIRS = [os.path.join(CASSANDRA_PATH, 'lib')]
 +myplatform = platform.system()
 +if myplatform == 'Linux':
 +ZIPLIB_DIRS.append('/usr/share/cassandra/lib')
 +
 +if os.environ.get('CQLSH_NO_BUNDLED', ''):
 +ZIPLIB_DIRS = ()
 +
 +
 +def find_zip(libprefix):
 +for ziplibdir in ZIPLIB_DIRS:
 +zips = glob(os.path.join(ziplibdir, libprefix + '*.zip'))
 +if zips:
 +return max(zips)   # probably the highest version, if multiple
 +
 +cql_zip = find_zip(CQL_LIB_PREFIX)
 +if cql_zip:
 +ver = os.path.splitext(os.path.basename(cql_zip))[0][len(CQL_LIB_PREFIX):]
 +sys.path.insert(0, os.path.join(cql_zip, 'cassandra-driver-' + ver))
 +
 +third_parties = ('futures-', 'six-')
 +
 +for lib in third_parties:
 +lib_zip = find_zip(lib)
 +if lib_zip:
 +sys.path.insert(0, lib_zip)
 +
 

[1/6] cassandra git commit: (cqlsh) further optimise COPY FROM

[aleksey]

Repository: cassandra
Updated Branches:
  refs/heads/trunk a018bcb7d -> bab66dd1a


(cqlsh) further optimise COPY FROM

patch by Stefania Alborghetti; reviewed by Adam Holmberg for
CASSANDRA-9302


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/124f1bd2
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/124f1bd2
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/124f1bd2

Branch: refs/heads/trunk
Commit: 124f1bd2613e400f69f8369ada0ad15c28738530
Parents: 994250c
Author: Stefania Alborghetti 
Authored: Thu Oct 22 17:16:50 2015 +0800
Committer: Aleksey Yeschenko 
Committed: Tue Dec 15 21:03:31 2015 +

--
 CHANGES.txt|   4 +-
 bin/cqlsh  | 285 ++---
 pylib/cqlshlib/copyutil.py | 910 ++--
 pylib/cqlshlib/util.py |  19 +
 4 files changed, 838 insertions(+), 380 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/124f1bd2/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 8e58703..90f1bca 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,12 +1,10 @@
 2.1.13
-<<< HEAD
+ * (cqlsh) further optimise COPY FROM (CASSANDRA-9302)
  * Allow CREATE TABLE WITH ID (CASSANDRA-9179)
  * Make Stress compiles within eclipse (CASSANDRA-10807)
  * Cassandra Daemon should print JVM arguments (CASSANDRA-10764)
  * Allow cancellation of index summary redistribution (CASSANDRA-8805)
-===
  * sstableloader will fail if there are collections in the schema tables 
(CASSANDRA-10700)
->>> 5377183... stableloader will fail if there are collections in the 
schema tables
  * Disable reloading of GossipingPropertyFileSnitch (CASSANDRA-9474)
  * Fix Stress profile parsing on Windows (CASSANDRA-10808)
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/124f1bd2/bin/cqlsh
--
diff --git a/bin/cqlsh b/bin/cqlsh
index e72624a..651420d 100755
--- a/bin/cqlsh
+++ b/bin/cqlsh
@@ -37,7 +37,6 @@ import ConfigParser
 import csv
 import getpass
 import locale
-import multiprocessing as mp
 import optparse
 import os
 import platform
@@ -48,7 +47,6 @@ import warnings
 
 from StringIO import StringIO
 from contextlib import contextmanager
-from functools import partial
 from glob import glob
 from uuid import UUID
 
@@ -110,10 +108,10 @@ except ImportError, e:
 
 from cassandra.auth import PlainTextAuthProvider
 from cassandra.cluster import Cluster, PagedResult
-from cassandra.metadata import protect_name, protect_names, protect_value
+from cassandra.metadata import protect_name, protect_names
 from cassandra.policies import WhiteListRoundRobinPolicy
-from cassandra.protocol import QueryMessage, ResultMessage
-from cassandra.query import SimpleStatement, ordered_dict_factory
+from cassandra.protocol import ResultMessage
+from cassandra.query import SimpleStatement, ordered_dict_factory, 
tuple_factory
 
 # cqlsh should run correctly when run out of a Cassandra source tree,
 # out of an unpacked Cassandra tarball, and after a proper package install.
@@ -334,7 +332,7 @@ cqlsh_extra_syntax_rules = r'''
 
  ::= 
   | 
-  | 
+  | 
   ;
 
 # avoiding just "DEBUG" so that this rule doesn't get treated as a terminal
@@ -412,17 +410,20 @@ def complete_copy_column_names(ctxt, cqlsh):
 return set(colnames[1:]) - set(existcols)
 
 
-COPY_OPTIONS = ['DELIMITER', 'QUOTE', 'ESCAPE', 'HEADER', 'NULL', 'ENCODING',
-'TIMEFORMAT', 'JOBS', 'PAGESIZE', 'PAGETIMEOUT', 'MAXATTEMPTS']
+COPY_COMMON_OPTIONS = ['DELIMITER', 'QUOTE', 'ESCAPE', 'HEADER', 'NULL',
+   'MAXATTEMPTS', 'REPORTFREQUENCY']
+COPY_FROM_OPTIONS = ['CHUNKSIZE', 'INGESTRATE', 'MAXBATCHSIZE', 'MINBATCHSIZE']
+COPY_TO_OPTIONS = ['ENCODING', 'TIMEFORMAT', 'PAGESIZE', 'PAGETIMEOUT', 
'MAXREQUESTS']
 
 
 @cqlsh_syntax_completer('copyOption', 'optnames')
 def complete_copy_options(ctxt, cqlsh):
 optnames = map(str.upper, ctxt.get_binding('optnames', ()))
 direction = ctxt.get_binding('dir').upper()
-opts = set(COPY_OPTIONS) - set(optnames)
 if direction == 'FROM':
-opts -= set(['ENCODING', 'TIMEFORMAT', 'JOBS', 'PAGESIZE', 
'PAGETIMEOUT', 'MAXATTEMPTS'])
+opts = set(COPY_COMMON_OPTIONS + COPY_FROM_OPTIONS) - set(optnames)
+elif direction == 'TO':
+opts = set(COPY_COMMON_OPTIONS + COPY_TO_OPTIONS) - set(optnames)
 return opts
 
 
@@ -1520,12 +1521,18 @@ class Shell(cmd.Cmd):
   ESCAPE='\'  - character to appear before the QUOTE char 
when quoted
   HEADER=false- whether to ignore the 

[4/6] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2

[aleksey]

Merge branch 'cassandra-2.1' into cassandra-2.2


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/57d558fc
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/57d558fc
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/57d558fc

Branch: refs/heads/trunk
Commit: 57d558fc1ef8117c41567541b967e2b782d04a50
Parents: de55c39 124f1bd
Author: Aleksey Yeschenko 
Authored: Tue Dec 15 21:37:09 2015 +
Committer: Aleksey Yeschenko 
Committed: Tue Dec 15 21:37:09 2015 +

--
 CHANGES.txt|   4 +
 bin/cqlsh.py   | 329 ++-
 pylib/cqlshlib/copyutil.py | 912 ++--
 pylib/cqlshlib/util.py |  19 +
 4 files changed, 843 insertions(+), 421 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/57d558fc/CHANGES.txt
--
diff --cc CHANGES.txt
index c9074fc,90f1bca..c969a4d
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,36 -1,15 +1,40 @@@
 -2.1.13
 +2.2.5
 + * Add property to allow listening on broadcast interface (CASSANDRA-9748)
 + * Fix regression in split size on CqlInputFormat (CASSANDRA-10835)
 + * Better handling of SSL connection errors inter-node (CASSANDRA-10816)
 + * Disable reloading of GossipingPropertyFileSnitch (CASSANDRA-9474)
 + * Verify tables in pseudo-system keyspaces at startup (CASSANDRA-10761)
 +Merged from 2.1:
+  * (cqlsh) further optimise COPY FROM (CASSANDRA-9302)
   * Allow CREATE TABLE WITH ID (CASSANDRA-9179)
   * Make Stress compiles within eclipse (CASSANDRA-10807)
   * Cassandra Daemon should print JVM arguments (CASSANDRA-10764)
   * Allow cancellation of index summary redistribution (CASSANDRA-8805)
+  * sstableloader will fail if there are collections in the schema tables 
(CASSANDRA-10700)
+  * Disable reloading of GossipingPropertyFileSnitch (CASSANDRA-9474)
   * Fix Stress profile parsing on Windows (CASSANDRA-10808)
  
+ 
 -2.1.12
 +2.2.4
 + * Show CQL help in cqlsh in web browser (CASSANDRA-7225)
 + * Serialize on disk the proper SSTable compression ratio (CASSANDRA-10775)
 + * Reject index queries while the index is building (CASSANDRA-8505)
 + * CQL.textile syntax incorrectly includes optional keyspace for aggregate 
SFUNC and FINALFUNC (CASSANDRA-10747)
 + * Fix JSON update with prepared statements (CASSANDRA-10631)
 + * Don't do anticompaction after subrange repair (CASSANDRA-10422)
 + * Fix SimpleDateType type compatibility (CASSANDRA-10027)
 + * (Hadoop) fix splits calculation (CASSANDRA-10640)
 + * (Hadoop) ensure that Cluster instances are always closed (CASSANDRA-10058)
 + * (cqlsh) show partial trace if incomplete after max_trace_wait 
(CASSANDRA-7645)
 + * Use most up-to-date version of schema for system tables (CASSANDRA-10652)
 + * Deprecate memory_allocator in cassandra.yaml (CASSANDRA-10581,10628)
 + * Expose phi values from failure detector via JMX and tweak debug
 +   and trace logging (CASSANDRA-9526)
 + * Fix RangeNamesQueryPager (CASSANDRA-10509)
 + * Deprecate Pig support (CASSANDRA-10542)
 + * Reduce contention getting instances of CompositeType (CASSANDRA-10433)
 + * Fix IllegalArgumentException in DataOutputBuffer.reallocate for large 
buffers (CASSANDRA-10592)
 +Merged from 2.1:
   * Fix incremental repair hang when replica is down (CASSANDRA-10288)
   * Avoid writing range tombstones after END_OF_ROW marker (CASSANDRA-10791)
   * Optimize the way we check if a token is repaired in anticompaction 
(CASSANDRA-10768)

[6/6] cassandra git commit: Merge branch 'cassandra-3.0' into trunk

[aleksey]

Merge branch 'cassandra-3.0' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bab66dd1
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bab66dd1
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bab66dd1

Branch: refs/heads/trunk
Commit: bab66dd1af377bec8aa3a144f19e925d4ecfa9cb
Parents: a018bcb 1deb044
Author: Aleksey Yeschenko 
Authored: Tue Dec 15 21:40:32 2015 +
Committer: Aleksey Yeschenko 
Committed: Tue Dec 15 21:40:32 2015 +

--
 CHANGES.txt|   1 +
 bin/cqlsh.py   | 329 ++-
 pylib/cqlshlib/copyutil.py | 912 ++--
 pylib/cqlshlib/util.py |  19 +
 4 files changed, 840 insertions(+), 421 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/bab66dd1/CHANGES.txt
--
diff --cc CHANGES.txt
index fb6f7e9,e167126..b3dc35b
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -24,7 -3,12 +24,8 @@@ Merged from 3.0
 (CASSANDRA-10822)
  Merged from 2.2
   * Add property to allow listening on broadcast interface (CASSANDRA-9748)
 - * Fix regression in split size on CqlInputFormat (CASSANDRA-10835)
 - * Better handling of SSL connection errors inter-node (CASSANDRA-10816)
 - * Disable reloading of GossipingPropertyFileSnitch (CASSANDRA-9474)
 - * Verify tables in pseudo-system keyspaces at startup (CASSANDRA-10761)
  Merged from 2.1:
+  * (cqlsh) further optimise COPY FROM (CASSANDRA-9302)
   * Allow CREATE TABLE WITH ID (CASSANDRA-9179)
   * Make Stress compiles within eclipse (CASSANDRA-10807)
   * Cassandra Daemon should print JVM arguments (CASSANDRA-10764)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/bab66dd1/bin/cqlsh.py
--

[5/6] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[aleksey]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1deb0443
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1deb0443
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1deb0443

Branch: refs/heads/trunk
Commit: 1deb044321f1e729d99e731f94ead760deaf1898
Parents: 942e5e5 57d558f
Author: Aleksey Yeschenko 
Authored: Tue Dec 15 21:39:25 2015 +
Committer: Aleksey Yeschenko 
Committed: Tue Dec 15 21:39:25 2015 +

--
 CHANGES.txt|   2 +
 bin/cqlsh.py   | 329 ++-
 pylib/cqlshlib/copyutil.py | 912 ++--
 pylib/cqlshlib/util.py |  19 +
 4 files changed, 841 insertions(+), 421 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/1deb0443/CHANGES.txt
--
diff --cc CHANGES.txt
index 4af7573,c969a4d..e167126
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -12,38 -10,12 +13,39 @@@ Merged from 2.1
   * Make Stress compiles within eclipse (CASSANDRA-10807)
   * Cassandra Daemon should print JVM arguments (CASSANDRA-10764)
   * Allow cancellation of index summary redistribution (CASSANDRA-8805)
 - * sstableloader will fail if there are collections in the schema tables 
(CASSANDRA-10700)
 - * Disable reloading of GossipingPropertyFileSnitch (CASSANDRA-9474)
 - * Fix Stress profile parsing on Windows (CASSANDRA-10808)
  
+ 
 -2.2.4
 +3.0.1
 + * Avoid MV race during node decommission (CASSANDRA-10674)
 + * Disable reloading of GossipingPropertyFileSnitch (CASSANDRA-9474)
 + * Handle single-column deletions correction in materialized views
 +   when the column is part of the view primary key (CASSANDRA-10796)
 + * Fix issue with datadir migration on upgrade (CASSANDRA-10788)
 + * Fix bug with range tombstones on reverse queries and test coverage for
 +   AbstractBTreePartition (CASSANDRA-10059)
 + * Remove 64k limit on collection elements (CASSANDRA-10374)
 + * Remove unclear Indexer.indexes() method (CASSANDRA-10690)
 + * Fix NPE on stream read error (CASSANDRA-10771)
 + * Normalize cqlsh DESC output (CASSANDRA-10431)
 + * Rejects partition range deletions when columns are specified 
(CASSANDRA-10739)
 + * Fix error when saving cached key for old format sstable (CASSANDRA-10778)
 + * Invalidate prepared statements on DROP INDEX (CASSANDRA-10758)
 + * Fix SELECT statement with IN restrictions on partition key,
 +   ORDER BY and LIMIT (CASSANDRA-10729)
 + * Improve stress performance over 1k threads (CASSANDRA-7217)
 + * Wait for migration responses to complete before bootstrapping 
(CASSANDRA-10731)
 + * Unable to create a function with argument of type Inet (CASSANDRA-10741)
 + * Fix backward incompatibiliy in CqlInputFormat (CASSANDRA-10717)
 + * Correctly preserve deletion info on updated rows when notifying indexers
 +   of single-row deletions (CASSANDRA-10694)
 + * Notify indexers of partition delete during cleanup (CASSANDRA-10685)
 + * Keep the file open in trySkipCache (CASSANDRA-10669)
 + * Updated trigger example (CASSANDRA-10257)
 +Merged from 2.2:
 + * Fix regression on split size in CqlInputFormat (CASSANDRA-10835)
 + * Better handling of SSL connection errors inter-node (CASSANDRA-10816)
 + * Verify tables in pseudo-system keyspaces at startup (CASSANDRA-10761)
 + * Fix IllegalArgumentException in DataOutputBuffer.reallocate for large 
buffers (CASSANDRA-10592)
   * Show CQL help in cqlsh in web browser (CASSANDRA-7225)
   * Serialize on disk the proper SSTable compression ratio (CASSANDRA-10775)
   * Reject index queries while the index is building (CASSANDRA-8505)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/1deb0443/bin/cqlsh.py
--

[jira] [Commented] (CASSANDRA-6018) Add option to encrypt commitlog

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15058906#comment-15058906
 ] 

Jason Brown commented on CASSANDRA-6018:


I can probably rustle up a final version over the holidays. Was waiting for 3.0 
to be released, but then got swept away in other tasks after that happened.

> Add option to encrypt commitlog 
> 
>
> Key: CASSANDRA-6018
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6018
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Jason Brown
>Assignee: Jason Brown
>  Labels: commit_log, encryption, security
> Fix For: 3.x
>
>
> We are going to start using cassandra for a billing system, and while I can 
> encrypt sstables at rest (via Datastax Enterprise), commit logs are more or 
> less plain text. Thus, an attacker would be able to easily read, for example, 
> credit card numbers in the clear text commit log (if the calling app does not 
> encrypt the data itself before sending it to cassandra).
> I want to allow the option of encrypting the commit logs, most likely 
> controlled by a property in the yaml.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9303) Match cassandra-loader options in COPY FROM

[Paulo Motta (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059004#comment-15059004
 ] 

Paulo Motta commented on CASSANDRA-9303:


Impressive work [~Stefania]. I tested it locally and most options work as 
expected. Overall I'm very satisfied with code and tests, except for some minor 
details listed below:
* I didn't really get the purpose of the {{CONFIGSECTIONS}} option and I think 
it complicates more than bring us benefits. Is there any particular case you 
want to achieve with this option? Can't we just have a general {{\[copy\]}} 
section, in addition specific {{\[ks.table\]}} sections for custom options 
per-table?
** We could also support those sections on {{cqlshrc}} as well, and maybe add 
example to {{conf/cqlshrc.sample}}. But if it's too much additional work just 
leave it as is.
* The {{ERRFILE}} option is not present in {{COPY_FROM_OPTIONS}} so it does not 
show up in the auto completer.
** Also, it seems the default {{ERRFILE}} is not being written if one is not 
explicitly specified.
** We could extend this error message on {{ImportTask.process_records}} to 
print the errfile name so the user will know where to look if he didn't specify 
one:
*** {{"Failed to process 10 rows (failed rows written to bla.err)"}}
* In {{copyutil.py:maybe_read_config_file}} can you replace 
{noformat}ret.update(dict([(k, v,) for k, v in opts.iteritems() if k not in 
['configfile', 'configsections']])){noformat} with 
{noformat}ret.update(opts){noformat} since you already popped {{'configfile'}} 
and {{'configsections'}} from {{opts}} before? (or maybe there's something I'm 
missing).
* The name {{ExportTask.check_processes}} is a bit misleading, since it sends 
work and monitors progress, maybe rename to {{schedule_and_monitor}}, or 
{{coordinate}} or {{start_work}} or even {{monitor_processes}} ? I can't find a 
good name as well as you can see :P
* Minor typo in {{ExportTask.get_ranges}} method description: {{rage}} -> 
{{range}}
* On this snippet in {{ExportTask.get_ranges}}:
{code}
#  For the last ring interval we query the same replicas that hold the 
last token in the ring
if previous_range and (not end_token or previous < end_token):
ranges[(previous, end_token)] = ranges[previous_range].copy()
{code}
for the last ring interval aren't we supposed to query the replicas that hold 
the first token in the ring instead (wrap-around)?
* On {{ImportProcess.run_normal}}, did you find out the reason why the 
commented snippet below slows down the query? Did you try it again after the 
review changes of CASSANDRA-9302?
{noformat}
# not sure if this is required but it does slow things down three fold
# query_statement.consistency_level = self.consistency_level
{noformat} If it still holds that's quite bizarre, as the same consistency is 
used later in the batch statement. I wonder how the prepared statement CL 
interacts with the batch CL, if it does at all.
* On {{ImportReader.get_source}} you forgot a debug print: {{print "Returning 
source {}".format(ret)}}. You should probably remove it or print only on debug 
mode.
* On {{formatting.py}} you can probably replace the 
{{format_integer_with_thousands_sep}} with a simpler implementation taking 
advantage of python support to thousand separator formatting (only available 
with "," though, that's why the replace afterwards):
{code}
def format_integer_with_thousands_sep(val, thousands_sep=','):
return "{:,}".format(val).replace(',', thousands_sep)
{code}
* Suggestion: modify the following messages to include the number of files 
written/read:
{noformat}
1000 rows exported to N files in 1.257 seconds.
130 rows imported from N files in 0.154 seconds.
{noformat}
* I found two situations where one corrupted row may fail importing of all the 
other rows, so you should probably cover these in your dtests:
** when there is a parse error in the primary key (stress-generated blob in 
this case):
{noformat}
Failed to import 1000 rows: ParseError - non-hexadecimal number found in 
fromhex() arg at position 0 -  given up without retries
Exceeded maximum number of parse errors 10
Failed to process 1000 rows
{noformat}
** when there is a row with fewer number of columns in the CSV:
{noformat}
Failed to import 20 rows: InvalidRequest - code=2200 [Invalid query] 
message="There were 6 markers(?) in CQL but 5 bound variables" -  will retry 
later, attempt 1 of 5
Failed to import 20 rows: InvalidRequest - code=2200 [Invalid query] 
message="There were 6 markers(?) in CQL but 5 bound variables" -  will retry 
later, attempt 2 of 5
Failed to import 20 rows: InvalidRequest - code=2200 [Invalid query] 
message="There were 6 markers(?) in CQL but 5 bound variables" -  will retry 
later, attempt 3 of 5
Failed to import 20 rows: InvalidRequest - code=2200 [Invalid query] 
message="There were 6 markers(?) in CQL but 5 bound 

cassandra git commit: Establish bootstrap stream sessions sequentially

[yukim]

Repository: cassandra
Updated Branches:
  refs/heads/trunk 07c6a36cc -> a018bcb7d


Establish bootstrap stream sessions sequentially

patch by Paulo Motta; reviewed by yukim for CASSANDRA-6992


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a018bcb7
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a018bcb7
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a018bcb7

Branch: refs/heads/trunk
Commit: a018bcb7d7ae74f9fa3f33cce2f0cc0deed6a442
Parents: 07c6a36
Author: Paulo Motta 
Authored: Wed Nov 18 16:00:36 2015 -0800
Committer: Yuki Morishita 
Committed: Tue Dec 15 15:24:22 2015 -0600

--
 CHANGES.txt |  1 +
 .../org/apache/cassandra/dht/BootStrapper.java  |  3 +-
 .../org/apache/cassandra/dht/RangeStreamer.java |  5 +-
 .../cassandra/io/sstable/SSTableLoader.java |  2 +-
 .../apache/cassandra/repair/LocalSyncTask.java  |  2 +-
 .../cassandra/repair/StreamingRepairTask.java   |  2 +-
 .../cassandra/service/StorageService.java   |  3 +-
 .../cassandra/streaming/StreamCoordinator.java  | 57 +++-
 .../apache/cassandra/streaming/StreamPlan.java  | 12 +++--
 .../cassandra/streaming/StreamResultFuture.java |  9 ++--
 .../apache/cassandra/dht/BootStrapperTest.java  |  2 +-
 11 files changed, 80 insertions(+), 18 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/a018bcb7/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 991d42a..fb6f7e9 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.2
+ * Establish bootstrap stream sessions sequentially (CASSANDRA-6992)
  * Sort compactionhistory output by timestamp (CASSANDRA-10464)
  * More efficient BTree removal (CASSANDRA-9991)
  * Make tablehistograms accept the same syntax as tablestats (CASSANDRA-10149)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/a018bcb7/src/java/org/apache/cassandra/dht/BootStrapper.java
--
diff --git a/src/java/org/apache/cassandra/dht/BootStrapper.java 
b/src/java/org/apache/cassandra/dht/BootStrapper.java
index 8d8f5c7..d10aa3b 100644
--- a/src/java/org/apache/cassandra/dht/BootStrapper.java
+++ b/src/java/org/apache/cassandra/dht/BootStrapper.java
@@ -74,7 +74,8 @@ public class BootStrapper extends ProgressEventNotifierSupport
"Bootstrap",
useStrictConsistency,

DatabaseDescriptor.getEndpointSnitch(),
-   stateStore);
+   stateStore,
+   true);
 streamer.addSourceFilter(new 
RangeStreamer.FailureDetectorSourceFilter(FailureDetector.instance));
 
 for (String keyspaceName : Schema.instance.getNonSystemKeyspaces())

http://git-wip-us.apache.org/repos/asf/cassandra/blob/a018bcb7/src/java/org/apache/cassandra/dht/RangeStreamer.java
--
diff --git a/src/java/org/apache/cassandra/dht/RangeStreamer.java 
b/src/java/org/apache/cassandra/dht/RangeStreamer.java
index 3da6bc8..47e0c15 100644
--- a/src/java/org/apache/cassandra/dht/RangeStreamer.java
+++ b/src/java/org/apache/cassandra/dht/RangeStreamer.java
@@ -117,13 +117,14 @@ public class RangeStreamer
  String description,
  boolean useStrictConsistency,
  IEndpointSnitch snitch,
- StreamStateStore stateStore)
+ StreamStateStore stateStore,
+ boolean connectSequentially)
 {
 this.metadata = metadata;
 this.tokens = tokens;
 this.address = address;
 this.description = description;
-this.streamPlan = new StreamPlan(description, true);
+this.streamPlan = new StreamPlan(description, true, 
connectSequentially);
 this.useStrictConsistency = useStrictConsistency;
 this.snitch = snitch;
 this.stateStore = stateStore;

http://git-wip-us.apache.org/repos/asf/cassandra/blob/a018bcb7/src/java/org/apache/cassandra/io/sstable/SSTableLoader.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/SSTableLoader.java 
b/src/java/org/apache/cassandra/io/sstable/SSTableLoader.java
index 3286522..043f6fa 100644
--- a/src/java/org/apache/cassandra/io/sstable/SSTableLoader.java
+++ 

[jira] [Comment Edited] (CASSANDRA-10837) Cluster/session should be closed in Cassandra Hadoop Input/Output classes

[Alex Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15058870#comment-15058870
 ] 

Alex Liu edited comment on CASSANDRA-10837 at 12/15/15 9:32 PM:


The changes to CqlRecordWriter seems wrong to me.
The Cluster and the Session instances were both properly managed by the 
try-with-resources statement. In the patch version only the Cluster instance is 
closed.

https://github.com/datastax/java-driver/blob/2.1/driver-core/src/main/java/com/datastax/driver/core/Cluster.java
Cluster doesn't implement AutoClosable interface, so it has to be closed 
manually. Closing cluster will close all session objects associated with it. 

Making the Cluster an instance variable in NativeRingCache will trigger an 
error when the write method will be called, as the Cluster has been closed at 
the end of the constructor.
=
NativeRingCache uses the cluster to get the metadata in the constructor when 
cluster object is still open. once the initialization of constructor is done, 
the cluster object is not used by NativeRingCache anymore.

In CqlInputFormat could you use try-with-resources for both Cluster and Session 
instances. I think it is best to do things properly by closing both of them.

Similarly closing cluster object auto-close all sessions objects.


was (Author: alexliu68):
The changes to CqlRecordWriter seems wrong to me.
The Cluster and the Session instances were both properly managed by the 
try-with-resources statement. In the patch version only the Cluster instance is 
closed.

https://github.com/datastax/java-driver/blob/2.1/driver-core/src/main/java/com/datastax/driver/core/Cluster.java
Cluster doesn't implements AutoClosable interface, so it has to be closes 
manually. Closing cluster will close all session objects associated with it. 

Making the Cluster an instance variable in NativeRingCache will trigger an 
error when the write method will be called, as the Cluster has been closed at 
the end of the constructor.
=
NativeRingCache uses the cluster to get the metadata in the constructor when 
cluster object is still open. once the initialization of constructor is done, 
the cluster object is not used by NativeRingCache anymore.

In CqlInputFormat could you use try-with-resources for both Cluster and Session 
instances. I think it is best to do things properly by closing both of them.

Similarly closing cluster object auto-close all sessions objects.

> Cluster/session should be closed in Cassandra Hadoop Input/Output classes
> -
>
> Key: CASSANDRA-10837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10837
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Alex Liu
>Assignee: Alex Liu
> Fix For: 3.0.x
>
> Attachments: 10837-3.0-branch.txt, 10837-v2-3.0-branch.txt
>
>
> See a lot of following warnings during Hadoop job running
> {code}
> ERROR 11:37:45 LEAK: You are creating too many HashedWheelTimer instances.  
> HashedWheelTimer is a shared resource that must be reused across the JVM,so 
> that only a few instances are created.
> {code}
> Each cluster/session needs be closed and a shared HashedWheelTimer may reduce 
> the resource leakage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10837) Cluster/session should be closed in Cassandra Hadoop Input/Output classes

[Alex Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15058870#comment-15058870
 ] 

Alex Liu commented on CASSANDRA-10837:
--

The changes to CqlRecordWriter seems wrong to me.
The Cluster and the Session instances were both properly managed by the 
try-with-resources statement. In the patch version only the Cluster instance is 
closed.

https://github.com/datastax/java-driver/blob/2.1/driver-core/src/main/java/com/datastax/driver/core/Cluster.java
Cluster doesn't implements AutoClosable interface, so it has to be closes 
manually. Closing cluster will close all session objects associated with it. 

Making the Cluster an instance variable in NativeRingCache will trigger an 
error when the write method will be called, as the Cluster has been closed at 
the end of the constructor.
=
NativeRingCache uses the cluster to get the metadata in the constructor when 
cluster object is still open. once the initialization of constructor is done, 
the cluster object is not used by NativeRingCache anymore.

In CqlInputFormat could you use try-with-resources for both Cluster and Session 
instances. I think it is best to do things properly by closing both of them.

Similarly closing cluster object auto-close all sessions objects.

> Cluster/session should be closed in Cassandra Hadoop Input/Output classes
> -
>
> Key: CASSANDRA-10837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10837
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Alex Liu
>Assignee: Alex Liu
> Fix For: 3.0.x
>
> Attachments: 10837-3.0-branch.txt, 10837-v2-3.0-branch.txt
>
>
> See a lot of following warnings during Hadoop job running
> {code}
> ERROR 11:37:45 LEAK: You are creating too many HashedWheelTimer instances.  
> HashedWheelTimer is a shared resource that must be reused across the JVM,so 
> that only a few instances are created.
> {code}
> Each cluster/session needs be closed and a shared HashedWheelTimer may reduce 
> the resource leakage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/5] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2

[aleksey]

http://git-wip-us.apache.org/repos/asf/cassandra/blob/57d558fc/pylib/cqlshlib/copyutil.py
--
diff --cc pylib/cqlshlib/copyutil.py
index a2fab00,f699e64..a117ec3
--- a/pylib/cqlshlib/copyutil.py
+++ b/pylib/cqlshlib/copyutil.py
@@@ -23,19 -26,25 +26,25 @@@ import sy
  import time
  import traceback
  
- from StringIO import StringIO
+ from calendar import timegm
+ from collections import defaultdict, deque, namedtuple
+ from decimal import Decimal
  from random import randrange
+ from StringIO import StringIO
  from threading import Lock
+ from uuid import UUID
  
  from cassandra.cluster import Cluster
+ from cassandra.cqltypes import ReversedType, UserType
  from cassandra.metadata import protect_name, protect_names
- from cassandra.policies import RetryPolicy, WhiteListRoundRobinPolicy, 
TokenAwarePolicy
- from cassandra.query import tuple_factory
+ from cassandra.policies import RetryPolicy, WhiteListRoundRobinPolicy, 
TokenAwarePolicy, DCAwareRoundRobinPolicy
+ from cassandra.query import BatchStatement, BatchType, SimpleStatement, 
tuple_factory
+ from cassandra.util import Date, Time
  
- 
- import sslhandling
+ from cql3handling import CqlRuleSet
  from displaying import NO_COLOR_MAP
 -from formatting import format_value_default, EMPTY, get_formatter
 +from formatting import format_value_default, DateTimeFormat, EMPTY, 
get_formatter
+ from sslhandling import ssl_settings
  
  
  def parse_options(shell, opts):
@@@ -65,10 -74,13 +74,15 @@@
  # by default the page timeout is 10 seconds per 1000 entries in the page 
size or 10 seconds if pagesize is smaller
  csv_options['pagetimeout'] = int(opts.pop('pagetimeout', max(10, 10 * 
(csv_options['pagesize'] / 1000
  csv_options['maxattempts'] = int(opts.pop('maxattempts', 5))
- csv_options['float_precision'] = shell.display_float_precision
 -csv_options['dtformats'] = opts.pop('timeformat', 
shell.display_time_format)
 +csv_options['dtformats'] = DateTimeFormat(opts.pop('timeformat', 
shell.display_timestamp_format),
 +  shell.display_date_format,
 +  shell.display_nanotime_format)
+ csv_options['float_precision'] = shell.display_float_precision
+ csv_options['chunksize'] = int(opts.pop('chunksize', 1000))
+ csv_options['ingestrate'] = int(opts.pop('ingestrate', 10))
+ csv_options['maxbatchsize'] = int(opts.pop('maxbatchsize', 20))
+ csv_options['minbatchsize'] = int(opts.pop('minbatchsize', 2))
+ csv_options['reportfrequency'] = float(opts.pop('reportfrequency', 0.25))
  
  return csv_options, dialect_options, opts
  
@@@ -371,30 -648,18 +650,18 @@@ class ExportProcess(ChildProcess)
  An child worker process for the export task, ExportTask.
  """
  
- def __init__(self, inmsg, outmsg, ks, cf, columns, dialect_options, 
csv_options,
-  debug, port, cql_version, auth_provider, ssl, 
protocol_version, config_file):
- mp.Process.__init__(self, target=self.run)
- self.inmsg = inmsg
- self.outmsg = outmsg
- self.ks = ks
- self.cf = cf
- self.columns = columns
- self.dialect_options = dialect_options
+ def __init__(self, params):
+ ChildProcess.__init__(self, params=params, target=self.run)
+ self.dialect_options = params['dialect_options']
  self.hosts_to_sessions = dict()
  
- self.debug = debug
- self.port = port
- self.cql_version = cql_version
- self.auth_provider = auth_provider
- self.ssl = ssl
- self.protocol_version = protocol_version
- self.config_file = config_file
- 
+ csv_options = params['csv_options']
  self.encoding = csv_options['encoding']
 -self.time_format = csv_options['dtformats']
 +self.date_time_format = csv_options['dtformats']
  self.float_precision = csv_options['float_precision']
  self.nullval = csv_options['nullval']
- self.maxjobs = csv_options['jobs']
+ self.max_attempts = csv_options['maxattempts']
+ self.max_requests = csv_options['maxrequests']
  self.csv_options = csv_options
  self.formatters = dict()
  
@@@ -600,13 -851,424 +853,424 @@@
  return query
  
  
+ class ImportConversion(object):
+ """
+ A class for converting strings to values when importing from csv, used by 
ImportProcess,
+ the parent.
+ """
+ def __init__(self, parent, table_meta, statement):
+ self.ks = parent.ks
+ self.cf = parent.cf
+ self.columns = parent.columns
+ self.nullval = parent.nullval
+ self.printmsg = parent.printmsg
+ self.table_meta = table_meta
+ self.primary_key_indexes = [self.columns.index(col.name) for col in 
self.table_meta.primary_key]
+ self.partition_key_indexes = 

[5/5] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[aleksey]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1deb0443
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1deb0443
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1deb0443

Branch: refs/heads/cassandra-3.0
Commit: 1deb044321f1e729d99e731f94ead760deaf1898
Parents: 942e5e5 57d558f
Author: Aleksey Yeschenko 
Authored: Tue Dec 15 21:39:25 2015 +
Committer: Aleksey Yeschenko 
Committed: Tue Dec 15 21:39:25 2015 +

--
 CHANGES.txt|   2 +
 bin/cqlsh.py   | 329 ++-
 pylib/cqlshlib/copyutil.py | 912 ++--
 pylib/cqlshlib/util.py |  19 +
 4 files changed, 841 insertions(+), 421 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/1deb0443/CHANGES.txt
--
diff --cc CHANGES.txt
index 4af7573,c969a4d..e167126
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -12,38 -10,12 +13,39 @@@ Merged from 2.1
   * Make Stress compiles within eclipse (CASSANDRA-10807)
   * Cassandra Daemon should print JVM arguments (CASSANDRA-10764)
   * Allow cancellation of index summary redistribution (CASSANDRA-8805)
 - * sstableloader will fail if there are collections in the schema tables 
(CASSANDRA-10700)
 - * Disable reloading of GossipingPropertyFileSnitch (CASSANDRA-9474)
 - * Fix Stress profile parsing on Windows (CASSANDRA-10808)
  
+ 
 -2.2.4
 +3.0.1
 + * Avoid MV race during node decommission (CASSANDRA-10674)
 + * Disable reloading of GossipingPropertyFileSnitch (CASSANDRA-9474)
 + * Handle single-column deletions correction in materialized views
 +   when the column is part of the view primary key (CASSANDRA-10796)
 + * Fix issue with datadir migration on upgrade (CASSANDRA-10788)
 + * Fix bug with range tombstones on reverse queries and test coverage for
 +   AbstractBTreePartition (CASSANDRA-10059)
 + * Remove 64k limit on collection elements (CASSANDRA-10374)
 + * Remove unclear Indexer.indexes() method (CASSANDRA-10690)
 + * Fix NPE on stream read error (CASSANDRA-10771)
 + * Normalize cqlsh DESC output (CASSANDRA-10431)
 + * Rejects partition range deletions when columns are specified 
(CASSANDRA-10739)
 + * Fix error when saving cached key for old format sstable (CASSANDRA-10778)
 + * Invalidate prepared statements on DROP INDEX (CASSANDRA-10758)
 + * Fix SELECT statement with IN restrictions on partition key,
 +   ORDER BY and LIMIT (CASSANDRA-10729)
 + * Improve stress performance over 1k threads (CASSANDRA-7217)
 + * Wait for migration responses to complete before bootstrapping 
(CASSANDRA-10731)
 + * Unable to create a function with argument of type Inet (CASSANDRA-10741)
 + * Fix backward incompatibiliy in CqlInputFormat (CASSANDRA-10717)
 + * Correctly preserve deletion info on updated rows when notifying indexers
 +   of single-row deletions (CASSANDRA-10694)
 + * Notify indexers of partition delete during cleanup (CASSANDRA-10685)
 + * Keep the file open in trySkipCache (CASSANDRA-10669)
 + * Updated trigger example (CASSANDRA-10257)
 +Merged from 2.2:
 + * Fix regression on split size in CqlInputFormat (CASSANDRA-10835)
 + * Better handling of SSL connection errors inter-node (CASSANDRA-10816)
 + * Verify tables in pseudo-system keyspaces at startup (CASSANDRA-10761)
 + * Fix IllegalArgumentException in DataOutputBuffer.reallocate for large 
buffers (CASSANDRA-10592)
   * Show CQL help in cqlsh in web browser (CASSANDRA-7225)
   * Serialize on disk the proper SSTable compression ratio (CASSANDRA-10775)
   * Reject index queries while the index is building (CASSANDRA-8505)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/1deb0443/bin/cqlsh.py
--

[1/5] cassandra git commit: (cqlsh) further optimise COPY FROM

[aleksey]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 942e5e599 -> 1deb04432


(cqlsh) further optimise COPY FROM

patch by Stefania Alborghetti; reviewed by Adam Holmberg for
CASSANDRA-9302


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/124f1bd2
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/124f1bd2
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/124f1bd2

Branch: refs/heads/cassandra-3.0
Commit: 124f1bd2613e400f69f8369ada0ad15c28738530
Parents: 994250c
Author: Stefania Alborghetti 
Authored: Thu Oct 22 17:16:50 2015 +0800
Committer: Aleksey Yeschenko 
Committed: Tue Dec 15 21:03:31 2015 +

--
 CHANGES.txt|   4 +-
 bin/cqlsh  | 285 ++---
 pylib/cqlshlib/copyutil.py | 910 ++--
 pylib/cqlshlib/util.py |  19 +
 4 files changed, 838 insertions(+), 380 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/124f1bd2/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 8e58703..90f1bca 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,12 +1,10 @@
 2.1.13
-<<< HEAD
+ * (cqlsh) further optimise COPY FROM (CASSANDRA-9302)
  * Allow CREATE TABLE WITH ID (CASSANDRA-9179)
  * Make Stress compiles within eclipse (CASSANDRA-10807)
  * Cassandra Daemon should print JVM arguments (CASSANDRA-10764)
  * Allow cancellation of index summary redistribution (CASSANDRA-8805)
-===
  * sstableloader will fail if there are collections in the schema tables 
(CASSANDRA-10700)
->>> 5377183... stableloader will fail if there are collections in the 
schema tables
  * Disable reloading of GossipingPropertyFileSnitch (CASSANDRA-9474)
  * Fix Stress profile parsing on Windows (CASSANDRA-10808)
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/124f1bd2/bin/cqlsh
--
diff --git a/bin/cqlsh b/bin/cqlsh
index e72624a..651420d 100755
--- a/bin/cqlsh
+++ b/bin/cqlsh
@@ -37,7 +37,6 @@ import ConfigParser
 import csv
 import getpass
 import locale
-import multiprocessing as mp
 import optparse
 import os
 import platform
@@ -48,7 +47,6 @@ import warnings
 
 from StringIO import StringIO
 from contextlib import contextmanager
-from functools import partial
 from glob import glob
 from uuid import UUID
 
@@ -110,10 +108,10 @@ except ImportError, e:
 
 from cassandra.auth import PlainTextAuthProvider
 from cassandra.cluster import Cluster, PagedResult
-from cassandra.metadata import protect_name, protect_names, protect_value
+from cassandra.metadata import protect_name, protect_names
 from cassandra.policies import WhiteListRoundRobinPolicy
-from cassandra.protocol import QueryMessage, ResultMessage
-from cassandra.query import SimpleStatement, ordered_dict_factory
+from cassandra.protocol import ResultMessage
+from cassandra.query import SimpleStatement, ordered_dict_factory, 
tuple_factory
 
 # cqlsh should run correctly when run out of a Cassandra source tree,
 # out of an unpacked Cassandra tarball, and after a proper package install.
@@ -334,7 +332,7 @@ cqlsh_extra_syntax_rules = r'''
 
  ::= 
   | 
-  | 
+  | 
   ;
 
 # avoiding just "DEBUG" so that this rule doesn't get treated as a terminal
@@ -412,17 +410,20 @@ def complete_copy_column_names(ctxt, cqlsh):
 return set(colnames[1:]) - set(existcols)
 
 
-COPY_OPTIONS = ['DELIMITER', 'QUOTE', 'ESCAPE', 'HEADER', 'NULL', 'ENCODING',
-'TIMEFORMAT', 'JOBS', 'PAGESIZE', 'PAGETIMEOUT', 'MAXATTEMPTS']
+COPY_COMMON_OPTIONS = ['DELIMITER', 'QUOTE', 'ESCAPE', 'HEADER', 'NULL',
+   'MAXATTEMPTS', 'REPORTFREQUENCY']
+COPY_FROM_OPTIONS = ['CHUNKSIZE', 'INGESTRATE', 'MAXBATCHSIZE', 'MINBATCHSIZE']
+COPY_TO_OPTIONS = ['ENCODING', 'TIMEFORMAT', 'PAGESIZE', 'PAGETIMEOUT', 
'MAXREQUESTS']
 
 
 @cqlsh_syntax_completer('copyOption', 'optnames')
 def complete_copy_options(ctxt, cqlsh):
 optnames = map(str.upper, ctxt.get_binding('optnames', ()))
 direction = ctxt.get_binding('dir').upper()
-opts = set(COPY_OPTIONS) - set(optnames)
 if direction == 'FROM':
-opts -= set(['ENCODING', 'TIMEFORMAT', 'JOBS', 'PAGESIZE', 
'PAGETIMEOUT', 'MAXATTEMPTS'])
+opts = set(COPY_COMMON_OPTIONS + COPY_FROM_OPTIONS) - set(optnames)
+elif direction == 'TO':
+opts = set(COPY_COMMON_OPTIONS + COPY_TO_OPTIONS) - set(optnames)
 return opts
 
 
@@ -1520,12 +1521,18 @@ class Shell(cmd.Cmd):
   ESCAPE='\'  - character to appear before the QUOTE char 
when quoted
   HEADER=false- whether to 

[4/5] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2

[aleksey]

Merge branch 'cassandra-2.1' into cassandra-2.2


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/57d558fc
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/57d558fc
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/57d558fc

Branch: refs/heads/cassandra-3.0
Commit: 57d558fc1ef8117c41567541b967e2b782d04a50
Parents: de55c39 124f1bd
Author: Aleksey Yeschenko 
Authored: Tue Dec 15 21:37:09 2015 +
Committer: Aleksey Yeschenko 
Committed: Tue Dec 15 21:37:09 2015 +

--
 CHANGES.txt|   4 +
 bin/cqlsh.py   | 329 ++-
 pylib/cqlshlib/copyutil.py | 912 ++--
 pylib/cqlshlib/util.py |  19 +
 4 files changed, 843 insertions(+), 421 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/57d558fc/CHANGES.txt
--
diff --cc CHANGES.txt
index c9074fc,90f1bca..c969a4d
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,36 -1,15 +1,40 @@@
 -2.1.13
 +2.2.5
 + * Add property to allow listening on broadcast interface (CASSANDRA-9748)
 + * Fix regression in split size on CqlInputFormat (CASSANDRA-10835)
 + * Better handling of SSL connection errors inter-node (CASSANDRA-10816)
 + * Disable reloading of GossipingPropertyFileSnitch (CASSANDRA-9474)
 + * Verify tables in pseudo-system keyspaces at startup (CASSANDRA-10761)
 +Merged from 2.1:
+  * (cqlsh) further optimise COPY FROM (CASSANDRA-9302)
   * Allow CREATE TABLE WITH ID (CASSANDRA-9179)
   * Make Stress compiles within eclipse (CASSANDRA-10807)
   * Cassandra Daemon should print JVM arguments (CASSANDRA-10764)
   * Allow cancellation of index summary redistribution (CASSANDRA-8805)
+  * sstableloader will fail if there are collections in the schema tables 
(CASSANDRA-10700)
+  * Disable reloading of GossipingPropertyFileSnitch (CASSANDRA-9474)
   * Fix Stress profile parsing on Windows (CASSANDRA-10808)
  
+ 
 -2.1.12
 +2.2.4
 + * Show CQL help in cqlsh in web browser (CASSANDRA-7225)
 + * Serialize on disk the proper SSTable compression ratio (CASSANDRA-10775)
 + * Reject index queries while the index is building (CASSANDRA-8505)
 + * CQL.textile syntax incorrectly includes optional keyspace for aggregate 
SFUNC and FINALFUNC (CASSANDRA-10747)
 + * Fix JSON update with prepared statements (CASSANDRA-10631)
 + * Don't do anticompaction after subrange repair (CASSANDRA-10422)
 + * Fix SimpleDateType type compatibility (CASSANDRA-10027)
 + * (Hadoop) fix splits calculation (CASSANDRA-10640)
 + * (Hadoop) ensure that Cluster instances are always closed (CASSANDRA-10058)
 + * (cqlsh) show partial trace if incomplete after max_trace_wait 
(CASSANDRA-7645)
 + * Use most up-to-date version of schema for system tables (CASSANDRA-10652)
 + * Deprecate memory_allocator in cassandra.yaml (CASSANDRA-10581,10628)
 + * Expose phi values from failure detector via JMX and tweak debug
 +   and trace logging (CASSANDRA-9526)
 + * Fix RangeNamesQueryPager (CASSANDRA-10509)
 + * Deprecate Pig support (CASSANDRA-10542)
 + * Reduce contention getting instances of CompositeType (CASSANDRA-10433)
 + * Fix IllegalArgumentException in DataOutputBuffer.reallocate for large 
buffers (CASSANDRA-10592)
 +Merged from 2.1:
   * Fix incremental repair hang when replica is down (CASSANDRA-10288)
   * Avoid writing range tombstones after END_OF_ROW marker (CASSANDRA-10791)
   * Optimize the way we check if a token is repaired in anticompaction 
(CASSANDRA-10768)

cassandra git commit: Fix CHANGES.txt for 3.0.3

[aleksey]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 1deb04432 -> 98178f53a


Fix CHANGES.txt for 3.0.3


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/98178f53
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/98178f53
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/98178f53

Branch: refs/heads/cassandra-3.0
Commit: 98178f53ac5d0525b63631b9fb359f43a14cde24
Parents: 1deb044
Author: Aleksey Yeschenko 
Authored: Tue Dec 15 21:46:58 2015 +
Committer: Aleksey Yeschenko 
Committed: Tue Dec 15 21:47:18 2015 +

--
 CHANGES.txt | 11 +++
 1 file changed, 7 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/98178f53/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index e167126..dd1896c 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,7 +1,5 @@
-3.0.2
- * Fix upgrade data loss due to range tombstone deleting more data than then 
should
-   (CASSANDRA-10822)
-Merged from 2.2
+3.0.3
+Merged from 2.2:
  * Add property to allow listening on broadcast interface (CASSANDRA-9748)
  * Fix regression in split size on CqlInputFormat (CASSANDRA-10835)
  * Better handling of SSL connection errors inter-node (CASSANDRA-10816)
@@ -15,6 +13,11 @@ Merged from 2.1:
  * Allow cancellation of index summary redistribution (CASSANDRA-8805)
 
 
+3.0.2
+ * Fix upgrade data loss due to range tombstone deleting more data than then 
should
+   (CASSANDRA-10822)
+
+
 3.0.1
  * Avoid MV race during node decommission (CASSANDRA-10674)
  * Disable reloading of GossipingPropertyFileSnitch (CASSANDRA-9474)

[jira] [Commented] (CASSANDRA-10837) Cluster/session should be closed in Cassandra Hadoop Input/Output classes

[Benjamin Lerer (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15058991#comment-15058991
 ] 

Benjamin Lerer commented on CASSANDRA-10837:


{quote} Cluster doesn't implement AutoClosable interface, so it has to be 
closed manually. Closing cluster will close all session objects associated with 
it. 
{quote}

{{Closeable}} extends {{AutoCloseable}} so you can use {{close-with-resources}} 
We have no guaranties on the {{Cluster.close()}} implementation. So let's do 
things properly: first closing the {{Sessions}} then closing the {{Cluster}}. 
It is the best way to avoid bad surprises.

{quote}NativeRingCache uses the cluster to get the metadata in the constructor 
when cluster object is still open. once the initialization of constructor is 
done, the cluster object is not used by NativeRingCache anymore.
{quote}

If the cluster is only used in the constructor there is not need to store it as 
an instance variable. Sooner or later somebody will try to use it and burn 
his/her fingers. 
The {{Cluster}}, or even better the {{Metadata}}, should be passed as an 
argument to the {{refreshEndpointMap}} method.
There is also no need to open a {{Session}} to get the {{Metadata}} 

> Cluster/session should be closed in Cassandra Hadoop Input/Output classes
> -
>
> Key: CASSANDRA-10837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10837
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Alex Liu
>Assignee: Alex Liu
> Fix For: 3.0.x
>
> Attachments: 10837-3.0-branch.txt, 10837-v2-3.0-branch.txt
>
>
> See a lot of following warnings during Hadoop job running
> {code}
> ERROR 11:37:45 LEAK: You are creating too many HashedWheelTimer instances.  
> HashedWheelTimer is a shared resource that must be reused across the JVM,so 
> that only a few instances are created.
> {code}
> Each cluster/session needs be closed and a shared HashedWheelTimer may reduce 
> the resource leakage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9303) Match cassandra-loader options in COPY FROM

[Paulo Motta (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059027#comment-15059027
 ] 

Paulo Motta commented on CASSANDRA-9303:


Also, there seems to be a new failure with 
[cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_round_trip_with_rate_file|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-9303-2.1-dtest/lastCompletedBuild/testReport/cqlsh_tests.cqlsh_copy_tests/CqlshCopyTest/test_round_trip_with_rate_file/]
 probably due to the review changes of CASSANDRA-9302.

> Match cassandra-loader options in COPY FROM
> ---
>
> Key: CASSANDRA-9303
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9303
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Tools
>Reporter: Jonathan Ellis
>Assignee: Stefania
>Priority: Critical
> Fix For: 2.1.x
>
>
> https://github.com/brianmhess/cassandra-loader added a bunch of options to 
> handle real world requirements, we should match those.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[3/5] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2

[aleksey]

http://git-wip-us.apache.org/repos/asf/cassandra/blob/57d558fc/bin/cqlsh.py
--
diff --cc bin/cqlsh.py
index a3fa666,000..42c2923
mode 100644,00..100644
--- a/bin/cqlsh.py
+++ b/bin/cqlsh.py
@@@ -1,2763 -1,0 +1,2486 @@@
 +#!/bin/sh
 +# -*- mode: Python -*-
 +
 +# Licensed to the Apache Software Foundation (ASF) under one
 +# or more contributor license agreements.  See the NOTICE file
 +# distributed with this work for additional information
 +# regarding copyright ownership.  The ASF licenses this file
 +# to you under the Apache License, Version 2.0 (the
 +# "License"); you may not use this file except in compliance
 +# with the License.  You may obtain a copy of the License at
 +#
 +# http://www.apache.org/licenses/LICENSE-2.0
 +#
 +# Unless required by applicable law or agreed to in writing, software
 +# distributed under the License is distributed on an "AS IS" BASIS,
 +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 +# See the License for the specific language governing permissions and
 +# limitations under the License.
 +
 +""":"
 +# bash code here; finds a suitable python interpreter and execs this file.
 +# prefer unqualified "python" if suitable:
 +python -c 'import sys; sys.exit(not (0x020500b0 < sys.hexversion < 
0x0300))' 2>/dev/null \
 +&& exec python "$0" "$@"
 +for pyver in 2.6 2.7 2.5; do
 +which python$pyver > /dev/null 2>&1 && exec python$pyver "$0" "$@"
 +done
 +echo "No appropriate python interpreter found." >&2
 +exit 1
 +":"""
 +
 +from __future__ import with_statement
 +
 +import cmd
 +import codecs
 +import ConfigParser
 +import csv
 +import getpass
 +import locale
- import multiprocessing as mp
 +import optparse
 +import os
 +import platform
 +import sys
 +import time
 +import traceback
 +import warnings
 +import webbrowser
++from StringIO import StringIO
 +from contextlib import contextmanager
- from functools import partial
 +from glob import glob
- from StringIO import StringIO
 +from uuid import UUID
 +
 +if sys.version_info[0] != 2 or sys.version_info[1] != 7:
 +sys.exit("\nCQL Shell supports only Python 2.7\n")
 +
 +description = "CQL Shell for Apache Cassandra"
 +version = "5.0.1"
 +
 +readline = None
 +try:
 +# check if tty first, cause readline doesn't check, and only cares
 +# about $TERM. we don't want the funky escape code stuff to be
 +# output if not a tty.
 +if sys.stdin.isatty():
 +import readline
 +except ImportError:
 +pass
 +
 +CQL_LIB_PREFIX = 'cassandra-driver-internal-only-'
 +
 +CASSANDRA_PATH = os.path.join(os.path.dirname(os.path.realpath(__file__)), 
'..')
 +CASSANDRA_CQL_HTML_FALLBACK = 
'https://cassandra.apache.org/doc/cql3/CQL-2.2.html'
 +
 +if os.path.exists(CASSANDRA_PATH + '/doc/cql3/CQL.html'):
 +# default location of local CQL.html
 +CASSANDRA_CQL_HTML = 'file://' + CASSANDRA_PATH + '/doc/cql3/CQL.html'
 +elif os.path.exists('/usr/share/doc/cassandra/CQL.html'):
 +# fallback to package file
 +CASSANDRA_CQL_HTML = 'file:///usr/share/doc/cassandra/CQL.html'
 +else:
 +# fallback to online version
 +CASSANDRA_CQL_HTML = CASSANDRA_CQL_HTML_FALLBACK
 +
 +# On Linux, the Python webbrowser module uses the 'xdg-open' executable
 +# to open a file/URL. But that only works, if the current session has been
 +# opened from _within_ a desktop environment. I.e. 'xdg-open' will fail,
 +# if the session's been opened via ssh to a remote box.
 +#
 +# Use 'python' to get some information about the detected browsers.
 +# >>> import webbrowser
 +# >>> webbrowser._tryorder
 +# >>> webbrowser._browser
 +#
 +if len(webbrowser._tryorder) == 0:
 +CASSANDRA_CQL_HTML = CASSANDRA_CQL_HTML_FALLBACK
 +elif webbrowser._tryorder[0] == 'xdg-open' and 
os.environ.get('XDG_DATA_DIRS', '') == '':
 +# only on Linux (some OS with xdg-open)
 +webbrowser._tryorder.remove('xdg-open')
 +webbrowser._tryorder.append('xdg-open')
 +
 +# use bundled libs for python-cql and thrift, if available. if there
 +# is a ../lib dir, use bundled libs there preferentially.
 +ZIPLIB_DIRS = [os.path.join(CASSANDRA_PATH, 'lib')]
 +myplatform = platform.system()
 +if myplatform == 'Linux':
 +ZIPLIB_DIRS.append('/usr/share/cassandra/lib')
 +
 +if os.environ.get('CQLSH_NO_BUNDLED', ''):
 +ZIPLIB_DIRS = ()
 +
 +
 +def find_zip(libprefix):
 +for ziplibdir in ZIPLIB_DIRS:
 +zips = glob(os.path.join(ziplibdir, libprefix + '*.zip'))
 +if zips:
 +return max(zips)   # probably the highest version, if multiple
 +
 +cql_zip = find_zip(CQL_LIB_PREFIX)
 +if cql_zip:
 +ver = os.path.splitext(os.path.basename(cql_zip))[0][len(CQL_LIB_PREFIX):]
 +sys.path.insert(0, os.path.join(cql_zip, 'cassandra-driver-' + ver))
 +
 +third_parties = ('futures-', 'six-')
 +
 +for lib in third_parties:
 +lib_zip = find_zip(lib)
 +if lib_zip:
 +sys.path.insert(0, lib_zip)
 +
 

[2/6] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2

[aleksey]

http://git-wip-us.apache.org/repos/asf/cassandra/blob/57d558fc/pylib/cqlshlib/copyutil.py
--
diff --cc pylib/cqlshlib/copyutil.py
index a2fab00,f699e64..a117ec3
--- a/pylib/cqlshlib/copyutil.py
+++ b/pylib/cqlshlib/copyutil.py
@@@ -23,19 -26,25 +26,25 @@@ import sy
  import time
  import traceback
  
- from StringIO import StringIO
+ from calendar import timegm
+ from collections import defaultdict, deque, namedtuple
+ from decimal import Decimal
  from random import randrange
+ from StringIO import StringIO
  from threading import Lock
+ from uuid import UUID
  
  from cassandra.cluster import Cluster
+ from cassandra.cqltypes import ReversedType, UserType
  from cassandra.metadata import protect_name, protect_names
- from cassandra.policies import RetryPolicy, WhiteListRoundRobinPolicy, 
TokenAwarePolicy
- from cassandra.query import tuple_factory
+ from cassandra.policies import RetryPolicy, WhiteListRoundRobinPolicy, 
TokenAwarePolicy, DCAwareRoundRobinPolicy
+ from cassandra.query import BatchStatement, BatchType, SimpleStatement, 
tuple_factory
+ from cassandra.util import Date, Time
  
- 
- import sslhandling
+ from cql3handling import CqlRuleSet
  from displaying import NO_COLOR_MAP
 -from formatting import format_value_default, EMPTY, get_formatter
 +from formatting import format_value_default, DateTimeFormat, EMPTY, 
get_formatter
+ from sslhandling import ssl_settings
  
  
  def parse_options(shell, opts):
@@@ -65,10 -74,13 +74,15 @@@
  # by default the page timeout is 10 seconds per 1000 entries in the page 
size or 10 seconds if pagesize is smaller
  csv_options['pagetimeout'] = int(opts.pop('pagetimeout', max(10, 10 * 
(csv_options['pagesize'] / 1000
  csv_options['maxattempts'] = int(opts.pop('maxattempts', 5))
- csv_options['float_precision'] = shell.display_float_precision
 -csv_options['dtformats'] = opts.pop('timeformat', 
shell.display_time_format)
 +csv_options['dtformats'] = DateTimeFormat(opts.pop('timeformat', 
shell.display_timestamp_format),
 +  shell.display_date_format,
 +  shell.display_nanotime_format)
+ csv_options['float_precision'] = shell.display_float_precision
+ csv_options['chunksize'] = int(opts.pop('chunksize', 1000))
+ csv_options['ingestrate'] = int(opts.pop('ingestrate', 10))
+ csv_options['maxbatchsize'] = int(opts.pop('maxbatchsize', 20))
+ csv_options['minbatchsize'] = int(opts.pop('minbatchsize', 2))
+ csv_options['reportfrequency'] = float(opts.pop('reportfrequency', 0.25))
  
  return csv_options, dialect_options, opts
  
@@@ -371,30 -648,18 +650,18 @@@ class ExportProcess(ChildProcess)
  An child worker process for the export task, ExportTask.
  """
  
- def __init__(self, inmsg, outmsg, ks, cf, columns, dialect_options, 
csv_options,
-  debug, port, cql_version, auth_provider, ssl, 
protocol_version, config_file):
- mp.Process.__init__(self, target=self.run)
- self.inmsg = inmsg
- self.outmsg = outmsg
- self.ks = ks
- self.cf = cf
- self.columns = columns
- self.dialect_options = dialect_options
+ def __init__(self, params):
+ ChildProcess.__init__(self, params=params, target=self.run)
+ self.dialect_options = params['dialect_options']
  self.hosts_to_sessions = dict()
  
- self.debug = debug
- self.port = port
- self.cql_version = cql_version
- self.auth_provider = auth_provider
- self.ssl = ssl
- self.protocol_version = protocol_version
- self.config_file = config_file
- 
+ csv_options = params['csv_options']
  self.encoding = csv_options['encoding']
 -self.time_format = csv_options['dtformats']
 +self.date_time_format = csv_options['dtformats']
  self.float_precision = csv_options['float_precision']
  self.nullval = csv_options['nullval']
- self.maxjobs = csv_options['jobs']
+ self.max_attempts = csv_options['maxattempts']
+ self.max_requests = csv_options['maxrequests']
  self.csv_options = csv_options
  self.formatters = dict()
  
@@@ -600,13 -851,424 +853,424 @@@
  return query
  
  
+ class ImportConversion(object):
+ """
+ A class for converting strings to values when importing from csv, used by 
ImportProcess,
+ the parent.
+ """
+ def __init__(self, parent, table_meta, statement):
+ self.ks = parent.ks
+ self.cf = parent.cf
+ self.columns = parent.columns
+ self.nullval = parent.nullval
+ self.printmsg = parent.printmsg
+ self.table_meta = table_meta
+ self.primary_key_indexes = [self.columns.index(col.name) for col in 
self.table_meta.primary_key]
+ self.partition_key_indexes = 

[2/2] cassandra git commit: Merge branch 'cassandra-3.0' into trunk

[aleksey]

Merge branch 'cassandra-3.0' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/80250aa0
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/80250aa0
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/80250aa0

Branch: refs/heads/trunk
Commit: 80250aa015adfd17791a78dc23ba1b1067f02104
Parents: bab66dd 98178f5
Author: Aleksey Yeschenko 
Authored: Tue Dec 15 21:51:05 2015 +
Committer: Aleksey Yeschenko 
Committed: Tue Dec 15 21:51:05 2015 +

--
 CHANGES.txt | 11 +++
 1 file changed, 7 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/80250aa0/CHANGES.txt
--
diff --cc CHANGES.txt
index b3dc35b,dd1896c..062ab10
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,29 -1,10 +1,26 @@@
 -3.0.3
 +3.2
 + * Establish bootstrap stream sessions sequentially (CASSANDRA-6992)
 + * Sort compactionhistory output by timestamp (CASSANDRA-10464)
 + * More efficient BTree removal (CASSANDRA-9991)
 + * Make tablehistograms accept the same syntax as tablestats (CASSANDRA-10149)
 + * Group pending compactions based on table (CASSANDRA-10718)
 + * Add compressor name in sstablemetadata output (CASSANDRA-9879)
 + * Fix type casting for counter columns (CASSANDRA-10824)
 + * Prevent running Cassandra as root (CASSANDRA-8142)
 + * bound maximum in-flight commit log replay mutation bytes to 64 megabytes 
(CASSANDRA-8639)
 + * Normalize all scripts (CASSANDRA-10679)
 + * Make compression ratio much more accurate (CASSANDRA-10225)
 + * Optimize building of Clustering object when only one is created 
(CASSANDRA-10409)
 + * Make index building pluggable (CASSANDRA-10681)
 + * Add sstable flush observer (CASSANDRA-10678)
 + * Improve NTS endpoints calculation (CASSANDRA-10200)
 + * Improve performance of the folderSize function (CASSANDRA-10677)
 + * Add support for type casting in selection clause (CASSANDRA-10310)
 + * Added graphing option to cassandra-stress (CASSANDRA-7918)
 + * Abort in-progress queries that time out (CASSANDRA-7392)
 + * Add transparent data encryption core classes (CASSANDRA-9945)
- Merged from 3.0:
-  * Fix upgrade data loss due to range tombstone deleting more data than then 
should
-(CASSANDRA-10822)
- Merged from 2.2
+ Merged from 2.2:
   * Add property to allow listening on broadcast interface (CASSANDRA-9748)
 - * Fix regression in split size on CqlInputFormat (CASSANDRA-10835)
 - * Better handling of SSL connection errors inter-node (CASSANDRA-10816)
 - * Disable reloading of GossipingPropertyFileSnitch (CASSANDRA-9474)
 - * Verify tables in pseudo-system keyspaces at startup (CASSANDRA-10761)
  Merged from 2.1:
   * (cqlsh) further optimise COPY FROM (CASSANDRA-9302)
   * Allow CREATE TABLE WITH ID (CASSANDRA-9179)
@@@ -32,8 -13,12 +29,14 @@@
   * Allow cancellation of index summary redistribution (CASSANDRA-8805)
  
  
 -3.0.2
 - * Fix upgrade data loss due to range tombstone deleting more data than then 
should
 -   (CASSANDRA-10822)
++3.1.1
++Merged from 3.0:
++  * Fix upgrade data loss due to range tombstone deleting more data than then 
should
++(CASSANDRA-10822)
+ 
+ 
 -3.0.1
 +3.1
 +Merged from 3.0:
   * Avoid MV race during node decommission (CASSANDRA-10674)
   * Disable reloading of GossipingPropertyFileSnitch (CASSANDRA-9474)
   * Handle single-column deletions correction in materialized views

[1/2] cassandra git commit: Fix CHANGES.txt for 3.0.3

[aleksey]

Repository: cassandra
Updated Branches:
  refs/heads/trunk bab66dd1a -> 80250aa01


Fix CHANGES.txt for 3.0.3


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/98178f53
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/98178f53
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/98178f53

Branch: refs/heads/trunk
Commit: 98178f53ac5d0525b63631b9fb359f43a14cde24
Parents: 1deb044
Author: Aleksey Yeschenko 
Authored: Tue Dec 15 21:46:58 2015 +
Committer: Aleksey Yeschenko 
Committed: Tue Dec 15 21:47:18 2015 +

--
 CHANGES.txt | 11 +++
 1 file changed, 7 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/98178f53/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index e167126..dd1896c 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,7 +1,5 @@
-3.0.2
- * Fix upgrade data loss due to range tombstone deleting more data than then 
should
-   (CASSANDRA-10822)
-Merged from 2.2
+3.0.3
+Merged from 2.2:
  * Add property to allow listening on broadcast interface (CASSANDRA-9748)
  * Fix regression in split size on CqlInputFormat (CASSANDRA-10835)
  * Better handling of SSL connection errors inter-node (CASSANDRA-10816)
@@ -15,6 +13,11 @@ Merged from 2.1:
  * Allow cancellation of index summary redistribution (CASSANDRA-8805)
 
 
+3.0.2
+ * Fix upgrade data loss due to range tombstone deleting more data than then 
should
+   (CASSANDRA-10822)
+
+
 3.0.1
  * Avoid MV race during node decommission (CASSANDRA-10674)
  * Disable reloading of GossipingPropertyFileSnitch (CASSANDRA-9474)

[jira] [Commented] (CASSANDRA-10850) v4 spec has tons of grammatical mistakes

[Michael Shuler (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059043#comment-15059043
 ] 

Michael Shuler commented on CASSANDRA-10850:


Patches are preferred, so this is perfect. :)

> v4 spec has tons of grammatical mistakes
> 
>
> Key: CASSANDRA-10850
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10850
> Project: Cassandra
>  Issue Type: Bug
>  Components: Documentation and Website
>Reporter: Sandeep Tamhankar
> Fix For: 3.0.2
>
> Attachments: v4-protocol.patch
>
>
> https://github.com/apache/cassandra/blob/cassandra-3.0/doc/native_protocol_v4.spec
> I notice the following in the first section of the spec and then gave up:
> "The list of allowed opcode is defined Section 2.3" => "The list of allowed 
> opcode*s* is defined in Section 2.3"
> "the details of each corresponding message is described Section 4" => "the 
> details of each corresponding message are described in Section 4" since the 
> subject is details, not message.
> "Requests are those frame sent by" => "Requests are those frame*s* sent by"
> I think someone should go through the whole spec and fix all the mistakes 
> rather than me pointing out the ones I notice piece-meal. I found the grammar 
> errors to be rather distracting.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10837) Cluster/session should be closed in Cassandra Hadoop Input/Output classes

[Alex Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059195#comment-15059195
 ] 

Alex Liu commented on CASSANDRA-10837:
--

Attach v3 path to use close-with-resources and pass cluster to NativeRingCache 
to minimize creating new cluster objects

> Cluster/session should be closed in Cassandra Hadoop Input/Output classes
> -
>
> Key: CASSANDRA-10837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10837
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Alex Liu
>Assignee: Alex Liu
> Fix For: 3.0.x
>
> Attachments: 10837-3.0-branch.txt, 10837-v2-3.0-branch.txt, 
> 10837-v3-3.0-branch.txt
>
>
> See a lot of following warnings during Hadoop job running
> {code}
> ERROR 11:37:45 LEAK: You are creating too many HashedWheelTimer instances.  
> HashedWheelTimer is a shared resource that must be reused across the JVM,so 
> that only a few instances are created.
> {code}
> Each cluster/session needs be closed and a shared HashedWheelTimer may reduce 
> the resource leakage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10866) Column Family should expose latency metrics for dropped mutations.

[Paulo Motta (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059207#comment-15059207
 ] 

Paulo Motta commented on CASSANDRA-10866:
-

I don't see much sense in exposing latency metrics for dropped mutations on a 
per-cf basis, since this won't vary much on a per-cf basis, but rather on a 
per-node basis (also internal vs cross node), so the global dropped mutation 
latency metric should suffice to tune node timeouts accordingly. Keeping a 
count for dropped mutations per CF may make sense though, as you may want to 
monitor the distribution of dropped mutations across different CFs.

> Column Family should expose latency metrics for dropped mutations.
> --
>
> Key: CASSANDRA-10866
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10866
> Project: Cassandra
>  Issue Type: Improvement
> Environment: PROD
>Reporter: Anubhav Kale
>Priority: Minor
>
> Please take a look at the discussion in CASSANDRA-10580. This is opened so 
> that the latency on dropped mutations is exposed as a metric on column 
> families.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10580) On dropped mutations, more details should be logged.

[Anubhav Kale (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059191#comment-15059191
 ] 

Anubhav Kale commented on CASSANDRA-10580:
--

Can someone review this so we don't have to deal with too many merge conflicts 
later ? Thanks a lot !

> On dropped mutations, more details should be logged.
> 
>
> Key: CASSANDRA-10580
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10580
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Coordination
> Environment: Production
>Reporter: Anubhav Kale
>Assignee: Anubhav Kale
>Priority: Minor
> Fix For: 3.2, 2.2.x
>
> Attachments: 0001-Metrics.patch, 10580-Metrics.patch, 10580.patch, 
> CASSANDRA-10580-Head.patch, Trunk.patch
>
>
> In our production cluster, we are seeing a large number of dropped mutations. 
> At a minimum, we should print the time the thread took to get scheduled 
> thereby dropping the mutation (We should also print the Message / Mutation so 
> it helps in figuring out which column family was affected). This will help 
> find the right tuning parameter for write_timeout_in_ms. 
> The change is small and is in StorageProxy.java and MessagingTask.java. I 
> will submit a patch shortly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10837) Cluster/session should be closed in Cassandra Hadoop Input/Output classes

[Alex Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Liu updated CASSANDRA-10837:
-
Attachment: 10837-v3-3.0-branch.txt

> Cluster/session should be closed in Cassandra Hadoop Input/Output classes
> -
>
> Key: CASSANDRA-10837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10837
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Alex Liu
>Assignee: Alex Liu
> Fix For: 3.0.x
>
> Attachments: 10837-3.0-branch.txt, 10837-v2-3.0-branch.txt, 
> 10837-v3-3.0-branch.txt
>
>
> See a lot of following warnings during Hadoop job running
> {code}
> ERROR 11:37:45 LEAK: You are creating too many HashedWheelTimer instances.  
> HashedWheelTimer is a shared resource that must be reused across the JVM,so 
> that only a few instances are created.
> {code}
> Each cluster/session needs be closed and a shared HashedWheelTimer may reduce 
> the resource leakage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10806) sstableloader can't handle upper case keyspace

[Yuki Morishita (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059153#comment-15059153
 ] 

Yuki Morishita commented on CASSANDRA-10806:


Patch looks good to me.
Kicked test with the patch.

||branch||testall||dtest||
|[10806-3.0|https://github.com/yukim/cassandra/tree/10806-3.0]|[testall|http://cassci.datastax.com/view/Dev/view/yukim/job/yukim-10806-3.0-testall/lastCompletedBuild/testReport/]|[dtest|http://cassci.datastax.com/view/Dev/view/yukim/job/yukim-10806-3.0-dtest/lastCompletedBuild/testReport/]|
|[10806-3.2|https://github.com/yukim/cassandra/tree/10806-3.2]|[testall|http://cassci.datastax.com/view/Dev/view/yukim/job/yukim-10806-3.2-testall/lastCompletedBuild/testReport/]|[dtest|http://cassci.datastax.com/view/Dev/view/yukim/job/yukim-10806-3.2-dtest/lastCompletedBuild/testReport/]|

Also, I created new dtest to cover this case.
https://github.com/riptano/cassandra-dtest/pull/713

> sstableloader can't handle upper case keyspace
> --
>
> Key: CASSANDRA-10806
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10806
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Alex Liu
>Assignee: Alex Liu
>Priority: Minor
> Fix For: 3.0.x, 3.x
>
> Attachments: CASSANDRA-10806-3.0-branch.txt
>
>
> sstableloader can't handle upper case keyspace. The following shows the 
> endpoint is missing
> {code}
> cassandra/bin/sstableloader 
> /var/folders/zz/zyxvpxvq6csfxvn_n0/T/bulk-write-to-Test1-Words-a9343a5f-62f3-4901-a9c8-ab7dc42a458e/Test1/Words-5
>   -d 127.0.0.1
> objc[7818]: Class JavaLaunchHelper is implemented in both 
> /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/bin/java and 
> /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/jre/lib/libinstrument.dylib.
>  One of the two will be used. Which one is undefined.
> Established connection to initial hosts
> Opening sstables and calculating sections to stream
> Streaming relevant part of 
> /var/folders/zz/zyxvpxvq6csfxvn_n0/T/bulk-write-to-Test1-Words-a9343a5f-62f3-4901-a9c8-ab7dc42a458e/Test1/Words-5/ma-1-big-Data.db
>  to []
> Summary statistics: 
>   Connections per host:: 1
>   Total files transferred:  : 0
>   Total bytes transferred:  : 0
>   Total duration (ms):  : 923  
>   Average transfer rate (MB/s): : 0
>   Peak transfer rate (MB/s):: 0 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)