[jira] [Commented] (CASSANDRA-7281) SELECT on tuple relations are broken for mixed ASC/DESC clustering order
[ https://issues.apache.org/jira/browse/CASSANDRA-7281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14550010#comment-14550010 ] Marcin Szymaniuk commented on CASSANDRA-7281: - I will provide another patch by the end of this week SELECT on tuple relations are broken for mixed ASC/DESC clustering order Key: CASSANDRA-7281 URL: https://issues.apache.org/jira/browse/CASSANDRA-7281 Project: Cassandra Issue Type: Bug Reporter: Sylvain Lebresne Assignee: Marcin Szymaniuk Fix For: 2.1.x Attachments: 0001-CASSANDRA-7281-SELECT-on-tuple-relations-are-broken-.patch, 0001-CASSANDRA-7281-SELECT-on-tuple-relations-are-broken-v2.patch, 0001-CASSANDRA-7281-SELECT-on-tuple-relations-are-broken-v3.patch As noted on [CASSANDRA-6875|https://issues.apache.org/jira/browse/CASSANDRA-6875?focusedCommentId=13992153page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13992153], the tuple notation is broken when the clustering order mixes ASC and DESC directives because the range of data they describe don't correspond to a single continuous slice internally. To copy the example from CASSANDRA-6875: {noformat} cqlsh:ks create table foo (a int, b int, c int, PRIMARY KEY (a, b, c)) WITH CLUSTERING ORDER BY (b DESC, c ASC); cqlsh:ks INSERT INTO foo (a, b, c) VALUES (0, 2, 0); cqlsh:ks INSERT INTO foo (a, b, c) VALUES (0, 1, 0); cqlsh:ks INSERT INTO foo (a, b, c) VALUES (0, 1, 1); cqlsh:ks INSERT INTO foo (a, b, c) VALUES (0, 0, 0); cqlsh:ks SELECT * FROM foo WHERE a=0; a | b | c ---+---+--- 0 | 2 | 0 0 | 1 | 0 0 | 1 | 1 0 | 0 | 0 (4 rows) cqlsh:ks SELECT * FROM foo WHERE a=0 AND (b, c) (1, 0); a | b | c ---+---+--- 0 | 2 | 0 (1 rows) {noformat} The last query should really return {{(0, 2, 0)}} and {{(0, 1, 1)}}. For that specific example we should generate 2 internal slices, but I believe that with more clustering columns we may have more slices. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9413) Add a default limit size (in bytes) for requests
[ https://issues.apache.org/jira/browse/CASSANDRA-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549953#comment-14549953 ] Antoine Blanchet commented on CASSANDRA-9413: - Got it. Thank you. Add a default limit size (in bytes) for requests Key: CASSANDRA-9413 URL: https://issues.apache.org/jira/browse/CASSANDRA-9413 Project: Cassandra Issue Type: Improvement Components: Core Environment: Cassandra 2.0.10, requested using Thrift Reporter: Antoine Blanchet We experienced a crash on our production cluster following a massive wide row read using Thrift A client tried to read a wide row (~4GB of raw data) without specifying any slice condition, which resulted in the crash of multiple nodes (as many as the replication factor) after long garbage collections. We know that wide rows should not be that big, but it is not the topic here. My question is the following: Is it possible to prevent Cassandra from OOM'ing when a client does this kind of requests? I'd rather have an error thrown to the client than a multi-server crash. The issue has already been discussed on the user mailing list, the thread is here : https://www.mail-archive.com/user@cassandra.apache.org/msg42340.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9186) AbstractType vs. CQL3Type loses frozen keyword
[ https://issues.apache.org/jira/browse/CASSANDRA-9186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549920#comment-14549920 ] Sylvain Lebresne commented on CASSANDRA-9186: - bq. should we really make a frozen/non-frozen distinction at UDF level at all? Good question. I suppose we don't have too and not doing it would be more flexible (typically it could be annoying to have to write twice the same function just because it's sometimes applied to a frozen type and sometimes not). So I like the idea of ignoring frozenness for functions but that probably mean we want to complain if a function is defined with a frozen type instead of just ignoring it silently as the later would probably be a bit confusing. AbstractType vs. CQL3Type loses frozen keyword -- Key: CASSANDRA-9186 URL: https://issues.apache.org/jira/browse/CASSANDRA-9186 Project: Cassandra Issue Type: Bug Components: Core Reporter: Robert Stupp Assignee: Robert Stupp Priority: Minor Fix For: 3.x {code} create keyspace functionmetadatatest with replication = {'class':'SimpleStrategy', 'replication_factor':1}; use functionmetadatatest ; CREATE TYPE udtx (x int); CREATE FUNCTION u_frozen(u frozenudtx)RETURNS int LANGUAGE java AS 'return new Integer(0);'; SELECT function_name, signature, argument_types FROM system.schema_functions WHERE keyspace_name='functionmetadatatest'; function_name | signature| argument_types ---+--+-- u_frozen | ['udtx'] | ['org.apache.cassandra.db.marshal.UserType(functionmetadatatest,75647478,78:org.apache.cassandra.db.marshal.Int32Type)'] {code} Problem is that {{UserType}} and {{TupleType}} do not have a _frozen_ (or _isMultiCell_) attribute. A conversion from type to cql3type therefore loses the _frozen_ status. EDIT: The issue has been discovered by [~aholmber] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9420) Table option for promising that you will never touch a column twice
[ https://issues.apache.org/jira/browse/CASSANDRA-9420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549993#comment-14549993 ] Sylvain Lebresne commented on CASSANDRA-9420: - bq. The problem stems from the fact that Cassandra plays safe when a TTL has expired, and turns it into a tombstone, rather than getting rid of it on the spot. If {{gcGrace == 0}}, which you seem fine with, then we'll get rid of expired column on the spot. The fact that expired column are turned into tombstones is a technicality that only exists so we keep expired columns for gcGrace seconds after expiration, but if the later is 0, it's removed right away. If you've observed differently in practice, then that would sound like a bug and if you provided reproduction steps we certainly can look into it. But, and unless I've misunderstood what you're saying, we don't need yet another option for this. Table option for promising that you will never touch a column twice --- Key: CASSANDRA-9420 URL: https://issues.apache.org/jira/browse/CASSANDRA-9420 Project: Cassandra Issue Type: New Feature Reporter: Björn Hegerfors There are time series use cases where you write all values with various TTLs, have GC grace = 0 and never ever update or delete a column after insertion. In the case where all TTLs are the same, DTCS with recent patches works great. But when there is lots of variations in TTLs, you are forced to choose between splitting your table into multiple TTL tiers or having your SSTables filled to the majority with tombstones. Or running frequent major compactions. The problem stems from the fact that Cassandra plays safe when a TTL has expired, and turns it into a tombstone, rather than getting rid of it on the spot. The reason is that this TTL _may_ have been in a column which has had an earlier write without (or with a higher) TTL. And then that one should now be deleted too. I propose that there should be table level setting to say I guarantee that there will never be any updates to any columns. The effect of enabling that option is that all tombstones and expired TTLs should always be immediately removed during compaction. And the check for dropping entirely expired SSTables can be very loosened for these tables. This option should probably require gc_grace_seconds to be set to zero. It's also questionable if writes without TTL should be allowed to such a table, since those would become constants. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9399) cqlsh support for native protocol v4 features
[ https://issues.apache.org/jira/browse/CASSANDRA-9399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14550022#comment-14550022 ] Stefania commented on CASSANDRA-9399: - tinyint and smallint have been added to cqlsh along with dtests to check: - tracing - date and time types - tinyint and smallint. Still todo in cqlsh: - client warnings - tests for CASSANDRA-8975 cqlsh support for native protocol v4 features - Key: CASSANDRA-9399 URL: https://issues.apache.org/jira/browse/CASSANDRA-9399 Project: Cassandra Issue Type: Bug Reporter: Aleksey Yeschenko Assignee: Stefania Labels: cqlsh Fix For: 2.2 rc1 cqlsh/python-driver need to add support for all the new 2.2 CQL features: - {{date}} and {{time}} types - CASSANDRA-7523 - not in cqlsh yet - {{smallint}} and {{tinyint}} types - CASSANDRA-8951 - not in the driver yet - client warnings - CASSANDRA-8930 - tracing improvements - CASSANDRA-7807 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
cassandra git commit: Log error when closing streaming connection at debug
Repository: cassandra Updated Branches: refs/heads/cassandra-2.1 7acd0e69d - 59b8e171e Log error when closing streaming connection at debug (this is good hygiene and remove a findBugs warning in particular) Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/59b8e171 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/59b8e171 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/59b8e171 Branch: refs/heads/cassandra-2.1 Commit: 59b8e171e91e4d714acac5ec195f40158e2e2971 Parents: 7acd0e6 Author: Sylvain Lebresne sylv...@datastax.com Authored: Tue May 19 10:31:48 2015 +0200 Committer: Sylvain Lebresne sylv...@datastax.com Committed: Tue May 19 10:31:52 2015 +0200 -- .../org/apache/cassandra/streaming/ConnectionHandler.java | 7 ++- 1 file changed, 6 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/59b8e171/src/java/org/apache/cassandra/streaming/ConnectionHandler.java -- diff --git a/src/java/org/apache/cassandra/streaming/ConnectionHandler.java b/src/java/org/apache/cassandra/streaming/ConnectionHandler.java index 6092046..aa3504a 100644 --- a/src/java/org/apache/cassandra/streaming/ConnectionHandler.java +++ b/src/java/org/apache/cassandra/streaming/ConnectionHandler.java @@ -215,7 +215,12 @@ public class ConnectionHandler { socket.close(); } -catch (IOException ignore) {} +catch (IOException e) +{ +// Erroring out while closing shouldn't happen but is not really a big deal, so just log +// it at DEBUG and ignore otherwise. +logger.debug(Unexpected error while closing streaming connection, e); +} } }
[1/3] cassandra git commit: Log error when closing streaming connection at debug
Repository: cassandra Updated Branches: refs/heads/trunk 6f5542792 - c844bfeea Log error when closing streaming connection at debug (this is good hygiene and remove a findBugs warning in particular) Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/59b8e171 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/59b8e171 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/59b8e171 Branch: refs/heads/trunk Commit: 59b8e171e91e4d714acac5ec195f40158e2e2971 Parents: 7acd0e6 Author: Sylvain Lebresne sylv...@datastax.com Authored: Tue May 19 10:31:48 2015 +0200 Committer: Sylvain Lebresne sylv...@datastax.com Committed: Tue May 19 10:31:52 2015 +0200 -- .../org/apache/cassandra/streaming/ConnectionHandler.java | 7 ++- 1 file changed, 6 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/59b8e171/src/java/org/apache/cassandra/streaming/ConnectionHandler.java -- diff --git a/src/java/org/apache/cassandra/streaming/ConnectionHandler.java b/src/java/org/apache/cassandra/streaming/ConnectionHandler.java index 6092046..aa3504a 100644 --- a/src/java/org/apache/cassandra/streaming/ConnectionHandler.java +++ b/src/java/org/apache/cassandra/streaming/ConnectionHandler.java @@ -215,7 +215,12 @@ public class ConnectionHandler { socket.close(); } -catch (IOException ignore) {} +catch (IOException e) +{ +// Erroring out while closing shouldn't happen but is not really a big deal, so just log +// it at DEBUG and ignore otherwise. +logger.debug(Unexpected error while closing streaming connection, e); +} } }
[2/3] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2
Merge branch 'cassandra-2.1' into cassandra-2.2 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/816c2b3f Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/816c2b3f Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/816c2b3f Branch: refs/heads/trunk Commit: 816c2b3f777843279295e67b0906f3df92ccc84a Parents: aa133b7 59b8e17 Author: Sylvain Lebresne sylv...@datastax.com Authored: Tue May 19 10:33:06 2015 +0200 Committer: Sylvain Lebresne sylv...@datastax.com Committed: Tue May 19 10:33:06 2015 +0200 -- .../org/apache/cassandra/streaming/ConnectionHandler.java | 7 ++- 1 file changed, 6 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/816c2b3f/src/java/org/apache/cassandra/streaming/ConnectionHandler.java --
[3/3] cassandra git commit: Merge branch 'cassandra-2.2' into trunk
Merge branch 'cassandra-2.2' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c844bfee Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c844bfee Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c844bfee Branch: refs/heads/trunk Commit: c844bfeeabd252f1b9c871964bfcf3abfaf17d35 Parents: 6f55427 816c2b3 Author: Sylvain Lebresne sylv...@datastax.com Authored: Tue May 19 10:33:25 2015 +0200 Committer: Sylvain Lebresne sylv...@datastax.com Committed: Tue May 19 10:33:25 2015 +0200 -- .../org/apache/cassandra/streaming/ConnectionHandler.java | 7 ++- 1 file changed, 6 insertions(+), 1 deletion(-) --
[1/2] cassandra git commit: Log error when closing streaming connection at debug
Repository: cassandra Updated Branches: refs/heads/cassandra-2.2 aa133b70f - 816c2b3f7 Log error when closing streaming connection at debug (this is good hygiene and remove a findBugs warning in particular) Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/59b8e171 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/59b8e171 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/59b8e171 Branch: refs/heads/cassandra-2.2 Commit: 59b8e171e91e4d714acac5ec195f40158e2e2971 Parents: 7acd0e6 Author: Sylvain Lebresne sylv...@datastax.com Authored: Tue May 19 10:31:48 2015 +0200 Committer: Sylvain Lebresne sylv...@datastax.com Committed: Tue May 19 10:31:52 2015 +0200 -- .../org/apache/cassandra/streaming/ConnectionHandler.java | 7 ++- 1 file changed, 6 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/59b8e171/src/java/org/apache/cassandra/streaming/ConnectionHandler.java -- diff --git a/src/java/org/apache/cassandra/streaming/ConnectionHandler.java b/src/java/org/apache/cassandra/streaming/ConnectionHandler.java index 6092046..aa3504a 100644 --- a/src/java/org/apache/cassandra/streaming/ConnectionHandler.java +++ b/src/java/org/apache/cassandra/streaming/ConnectionHandler.java @@ -215,7 +215,12 @@ public class ConnectionHandler { socket.close(); } -catch (IOException ignore) {} +catch (IOException e) +{ +// Erroring out while closing shouldn't happen but is not really a big deal, so just log +// it at DEBUG and ignore otherwise. +logger.debug(Unexpected error while closing streaming connection, e); +} } }
[2/2] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2
Merge branch 'cassandra-2.1' into cassandra-2.2 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/816c2b3f Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/816c2b3f Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/816c2b3f Branch: refs/heads/cassandra-2.2 Commit: 816c2b3f777843279295e67b0906f3df92ccc84a Parents: aa133b7 59b8e17 Author: Sylvain Lebresne sylv...@datastax.com Authored: Tue May 19 10:33:06 2015 +0200 Committer: Sylvain Lebresne sylv...@datastax.com Committed: Tue May 19 10:33:06 2015 +0200 -- .../org/apache/cassandra/streaming/ConnectionHandler.java | 7 ++- 1 file changed, 6 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/816c2b3f/src/java/org/apache/cassandra/streaming/ConnectionHandler.java --
[jira] [Commented] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces return java.lang.RuntimeException: Could not create snapshot
[ https://issues.apache.org/jira/browse/CASSANDRA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14550024#comment-14550024 ] Rafał Furmański commented on CASSANDRA-8696: I see no errors after increasing this timeout to very high value. I hope repair will end now.. nodetool repair on cassandra 2.1.2 keyspaces return java.lang.RuntimeException: Could not create snapshot - Key: CASSANDRA-8696 URL: https://issues.apache.org/jira/browse/CASSANDRA-8696 Project: Cassandra Issue Type: Bug Reporter: Jeff Liu Assignee: Yuki Morishita Fix For: 2.1.x When trying to run nodetool repair -pr on cassandra node ( 2.1.2), cassandra throw java exceptions: cannot create snapshot. the error log from system.log: {noformat} INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:28,815 StreamResultFuture.java:166 - [Stream #692c1450-a692-11e4-9973-070e938df227 ID#0] Prepare completed. Receiving 2 files(221187 bytes), sending 5 files(632105 bytes) INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046 StreamResultFuture.java:180 - [Stream #692c1450-a692-11e4-9973-070e938df227] Session with /10.97.9.110 is complete INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046 StreamResultFuture.java:212 - [Stream #692c1450-a692-11e4-9973-070e938df227] All sessions completed INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,047 StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227] streaming task succeed, returning response to /10.98.194.68 INFO [RepairJobTask:1] 2015-01-28 02:07:29,065 StreamResultFuture.java:86 - [Stream #692c6270-a692-11e4-9973-070e938df227] Executing streaming plan for Repair INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,065 StreamSession.java:213 - [Stream #692c6270-a692-11e4-9973-070e938df227] Starting streaming to /10.66.187.201 INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,070 StreamCoordinator.java:209 - [Stream #692c6270-a692-11e4-9973-070e938df227, ID#0] Beginning stream session with /10.66.187.201 INFO [STREAM-IN-/10.66.187.201] 2015-01-28 02:07:29,465 StreamResultFuture.java:166 - [Stream #692c6270-a692-11e4-9973-070e938df227 ID#0] Prepare completed. Receiving 5 files(627994 bytes), sending 5 files(632105 bytes) INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,971 StreamResultFuture.java:180 - [Stream #692c6270-a692-11e4-9973-070e938df227] Session with /10.66.187.201 is complete INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972 StreamResultFuture.java:212 - [Stream #692c6270-a692-11e4-9973-070e938df227] All sessions completed INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972 StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227] streaming task succeed, returning response to /10.98.194.68 ERROR [RepairJobTask:1] 2015-01-28 02:07:39,444 RepairJob.java:127 - Error occurred during snapshot phase java.lang.RuntimeException: Could not create snapshot at /10.97.9.110 at org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:347) ~[apache-cassandra-2.1.2.jar:2.1.2] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_45] at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_45] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45] at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45] INFO [AntiEntropySessions:6] 2015-01-28 02:07:39,445 RepairSession.java:260 - [repair #6f85e740-a692-11e4-9973-070e938df227] new session: will sync /10.98.194.68, /10.66.187.201, /10.226.218.135 on range (12817179804668051873746972069086 2638799,12863540308359254031520865977436165] for events.[bigint0text, bigint0boolean, bigint0int, dataset_catalog, column_categories, bigint0double, bigint0bigint] ERROR [AntiEntropySessions:5] 2015-01-28 02:07:39,445 RepairSession.java:303 - [repair #685e3d00-a692-11e4-9973-070e938df227] session completed with the following error java.io.IOException: Failed during snapshot creation. at org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:128) ~[apache-cassandra-2.1.2.jar:2.1.2] at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)
[jira] [Commented] (CASSANDRA-7281) SELECT on tuple relations are broken for mixed ASC/DESC clustering order
[ https://issues.apache.org/jira/browse/CASSANDRA-7281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14550027#comment-14550027 ] Benjamin Lerer commented on CASSANDRA-7281: --- [~marcinszymaniuk] Great! Thanks SELECT on tuple relations are broken for mixed ASC/DESC clustering order Key: CASSANDRA-7281 URL: https://issues.apache.org/jira/browse/CASSANDRA-7281 Project: Cassandra Issue Type: Bug Reporter: Sylvain Lebresne Assignee: Marcin Szymaniuk Fix For: 2.1.x Attachments: 0001-CASSANDRA-7281-SELECT-on-tuple-relations-are-broken-.patch, 0001-CASSANDRA-7281-SELECT-on-tuple-relations-are-broken-v2.patch, 0001-CASSANDRA-7281-SELECT-on-tuple-relations-are-broken-v3.patch As noted on [CASSANDRA-6875|https://issues.apache.org/jira/browse/CASSANDRA-6875?focusedCommentId=13992153page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13992153], the tuple notation is broken when the clustering order mixes ASC and DESC directives because the range of data they describe don't correspond to a single continuous slice internally. To copy the example from CASSANDRA-6875: {noformat} cqlsh:ks create table foo (a int, b int, c int, PRIMARY KEY (a, b, c)) WITH CLUSTERING ORDER BY (b DESC, c ASC); cqlsh:ks INSERT INTO foo (a, b, c) VALUES (0, 2, 0); cqlsh:ks INSERT INTO foo (a, b, c) VALUES (0, 1, 0); cqlsh:ks INSERT INTO foo (a, b, c) VALUES (0, 1, 1); cqlsh:ks INSERT INTO foo (a, b, c) VALUES (0, 0, 0); cqlsh:ks SELECT * FROM foo WHERE a=0; a | b | c ---+---+--- 0 | 2 | 0 0 | 1 | 0 0 | 1 | 1 0 | 0 | 0 (4 rows) cqlsh:ks SELECT * FROM foo WHERE a=0 AND (b, c) (1, 0); a | b | c ---+---+--- 0 | 2 | 0 (1 rows) {noformat} The last query should really return {{(0, 2, 0)}} and {{(0, 1, 1)}}. For that specific example we should generate 2 internal slices, but I believe that with more clustering columns we may have more slices. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[3/9] cassandra git commit: Improve javadoc around trigger execution
Improve javadoc around trigger execution Patch by Sam Tunnicliffe; reviewed by brandonwilliams for CASSANDRA-9334 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/116de2be Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/116de2be Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/116de2be Branch: refs/heads/cassandra-2.2 Commit: 116de2bee5920bc6842300a3f686d88f443b281d Parents: 59b8e17 Author: Sam Tunnicliffe s...@beobal.com Authored: Tue May 12 11:49:03 2015 +0100 Committer: Sam Tunnicliffe s...@beobal.com Committed: Tue May 19 11:59:44 2015 +0100 -- .../org/apache/cassandra/triggers/ITrigger.java | 4 +-- .../cassandra/triggers/TriggerExecutor.java | 32 2 files changed, 34 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/116de2be/src/java/org/apache/cassandra/triggers/ITrigger.java -- diff --git a/src/java/org/apache/cassandra/triggers/ITrigger.java b/src/java/org/apache/cassandra/triggers/ITrigger.java index 4701b61..21aba05 100644 --- a/src/java/org/apache/cassandra/triggers/ITrigger.java +++ b/src/java/org/apache/cassandra/triggers/ITrigger.java @@ -36,7 +36,7 @@ import org.apache.cassandra.db.Mutation; * 2) ITrigger implementation can be instantiated multiple times during the server life time. * (Depends on the number of times trigger folder is updated.)br * 3) ITrigger implementation should be state-less (avoid dependency on instance variables).br - * + * * brbThe API is still beta and can change./b */ public interface ITrigger @@ -46,7 +46,7 @@ public interface ITrigger * * @param partitionKey - partition Key for the update. * @param update - update received for the CF - * @return modifications to be applied, null if no action to be performed. + * @return additional modifications to be applied along with the supplied update */ public CollectionMutation augment(ByteBuffer partitionKey, ColumnFamily update); } http://git-wip-us.apache.org/repos/asf/cassandra/blob/116de2be/src/java/org/apache/cassandra/triggers/TriggerExecutor.java -- diff --git a/src/java/org/apache/cassandra/triggers/TriggerExecutor.java b/src/java/org/apache/cassandra/triggers/TriggerExecutor.java index 677daad..bc6f46c 100644 --- a/src/java/org/apache/cassandra/triggers/TriggerExecutor.java +++ b/src/java/org/apache/cassandra/triggers/TriggerExecutor.java @@ -60,6 +60,23 @@ public class TriggerExecutor cachedTriggers.clear(); } +/** + * Augment a partition update by executing triggers to generate an intermediate + * set of mutations, then merging the ColumnFamily from each mutation with those + * supplied. This is called from @{link org.apache.cassandra.service.StorageProxy#cas} + * which is scoped for a single partition. For that reason, any mutations generated + * by triggers are checked to ensure that they are for the same table and partition + * key as the primary update; if not, InvalidRequestException is thrown. If no + * additional mutations are generated, the original updates are returned unmodified. + * + * @param key partition key for the update + * @param updates partition update to be applied, contains the merge of the original + *update and any generated mutations + * @return the final update to be applied, the original update merged with any + * additional mutations generated by configured triggers + * @throws InvalidRequestException if any mutation generated by a trigger does not + * apply to the exact same partition as the initial update + */ public ColumnFamily execute(ByteBuffer key, ColumnFamily updates) throws InvalidRequestException { ListMutation intermediate = executeInternal(key, updates); @@ -78,6 +95,21 @@ public class TriggerExecutor return updates; } +/** + * Takes a collection of mutations and possibly augments it by adding extra mutations + * generated by configured triggers. If no additional mutations are created + * this returns null, signalling to the caller that only the initial set of + * mutations should be applied. If additional mutations iare/i generated, + * the total set (i.e. the original plus the additional mutations) are applied + * together in a logged batch. Should this not be possible because the initial + * mutations contain counter updates, InvalidRequestException is thrown. + * + * @param mutations initial collection of mutations + * @return augmented mutations. Either the union of the initial
[7/9] cassandra git commit: Add test for triggers which throw IRE
Add test for triggers which throw IRE Patch by Sam Tunnicliffe; reviewed by brandonwilliams for CASSANDRA-9334 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/86c9c00e Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/86c9c00e Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/86c9c00e Branch: refs/heads/cassandra-2.2 Commit: 86c9c00e98a1867deff3b8acc883d224b532197b Parents: e2723a4 Author: Sam Tunnicliffe s...@beobal.com Authored: Tue May 12 16:26:24 2015 +0100 Committer: Sam Tunnicliffe s...@beobal.com Committed: Tue May 19 12:02:40 2015 +0100 -- .../apache/cassandra/triggers/TriggersTest.java | 49 +++- 1 file changed, 38 insertions(+), 11 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/86c9c00e/test/unit/org/apache/cassandra/triggers/TriggersTest.java -- diff --git a/test/unit/org/apache/cassandra/triggers/TriggersTest.java b/test/unit/org/apache/cassandra/triggers/TriggersTest.java index 41d4bb8..b0a5aca 100644 --- a/test/unit/org/apache/cassandra/triggers/TriggersTest.java +++ b/test/unit/org/apache/cassandra/triggers/TriggersTest.java @@ -22,18 +22,13 @@ import java.nio.ByteBuffer; import java.util.Collection; import java.util.Collections; -import org.junit.AfterClass; -import org.junit.Before; -import org.junit.BeforeClass; -import org.junit.Test; +import org.junit.*; import org.apache.cassandra.SchemaLoader; import org.apache.cassandra.config.Schema; import org.apache.cassandra.cql3.QueryProcessor; import org.apache.cassandra.cql3.UntypedResultSet; -import org.apache.cassandra.db.ArrayBackedSortedColumns; -import org.apache.cassandra.db.BufferCell; -import org.apache.cassandra.db.ColumnFamily; +import org.apache.cassandra.db.*; import org.apache.cassandra.db.ConsistencyLevel; import org.apache.cassandra.db.Mutation; import org.apache.cassandra.exceptions.ConfigurationException; @@ -42,11 +37,10 @@ import org.apache.cassandra.service.StorageService; import org.apache.cassandra.thrift.*; import org.apache.thrift.protocol.TBinaryProtocol; -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertTrue; - import static org.apache.cassandra.utils.ByteBufferUtil.bytes; import static org.apache.cassandra.utils.ByteBufferUtil.toInt; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; public class TriggersTest { @@ -61,12 +55,12 @@ public class TriggersTest public static void beforeTest() throws ConfigurationException { SchemaLoader.loadSchema(); +StorageService.instance.initServer(0); } @Before public void setup() throws Exception { -StorageService.instance.initServer(0); if (thriftServer == null || ! thriftServer.isRunning()) { thriftServer = new ThriftServer(InetAddress.getLocalHost(), 9170, 50); @@ -283,6 +277,29 @@ public class TriggersTest } } +@Test(expected=RuntimeException.class) +public void ifTriggerThrowsErrorNoMutationsAreApplied() throws Exception +{ +String cf = cf + System.nanoTime(); +try +{ +setupTableWithTrigger(cf, ErrorTrigger.class); +String cql = String.format(INSERT INTO %s.%s (k, v1) VALUES (11, 11), ksName, cf); +QueryProcessor.process(cql, ConsistencyLevel.ONE); +} +catch (Exception e) +{ +Throwable cause = e.getCause(); +assertTrue((cause instanceof org.apache.cassandra.exceptions.InvalidRequestException)); +assertTrue(cause.getMessage().equals(ErrorTrigger.MESSAGE)); +throw e; +} +finally +{ +assertUpdateNotExecuted(cf, 11); +} +} + private void setupTableWithTrigger(String cf, Class? extends ITrigger triggerImpl) throws RequestExecutionException { @@ -350,4 +367,14 @@ public class TriggersTest return Collections.singletonList(new Mutation(ksName, key, extraUpdate)); } } + +public static class ErrorTrigger implements ITrigger +{ +public static final String MESSAGE = Thrown by ErrorTrigger; +public CollectionMutation augment(ByteBuffer partitionKey, ColumnFamily update) +{ +throw new org.apache.cassandra.exceptions.InvalidRequestException(MESSAGE); +} +} + }
[9/9] cassandra git commit: Merge branch 'cassandra-2.2' into trunk
Merge branch 'cassandra-2.2' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3a6e9c91 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3a6e9c91 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3a6e9c91 Branch: refs/heads/trunk Commit: 3a6e9c91b83ad0dc9b2423b51975277c790af66d Parents: c844bfe 86c9c00 Author: Sam Tunnicliffe s...@beobal.com Authored: Tue May 19 12:05:24 2015 +0100 Committer: Sam Tunnicliffe s...@beobal.com Committed: Tue May 19 12:05:24 2015 +0100 -- .../org/apache/cassandra/triggers/ITrigger.java | 4 +- .../cassandra/triggers/TriggerExecutor.java | 32 + .../apache/cassandra/triggers/TriggersTest.java | 49 +++- 3 files changed, 72 insertions(+), 13 deletions(-) --
[4/9] cassandra git commit: Improve javadoc around trigger execution
Improve javadoc around trigger execution Patch by Sam Tunnicliffe; reviewed by brandonwilliams for CASSANDRA-9334 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/116de2be Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/116de2be Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/116de2be Branch: refs/heads/trunk Commit: 116de2bee5920bc6842300a3f686d88f443b281d Parents: 59b8e17 Author: Sam Tunnicliffe s...@beobal.com Authored: Tue May 12 11:49:03 2015 +0100 Committer: Sam Tunnicliffe s...@beobal.com Committed: Tue May 19 11:59:44 2015 +0100 -- .../org/apache/cassandra/triggers/ITrigger.java | 4 +-- .../cassandra/triggers/TriggerExecutor.java | 32 2 files changed, 34 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/116de2be/src/java/org/apache/cassandra/triggers/ITrigger.java -- diff --git a/src/java/org/apache/cassandra/triggers/ITrigger.java b/src/java/org/apache/cassandra/triggers/ITrigger.java index 4701b61..21aba05 100644 --- a/src/java/org/apache/cassandra/triggers/ITrigger.java +++ b/src/java/org/apache/cassandra/triggers/ITrigger.java @@ -36,7 +36,7 @@ import org.apache.cassandra.db.Mutation; * 2) ITrigger implementation can be instantiated multiple times during the server life time. * (Depends on the number of times trigger folder is updated.)br * 3) ITrigger implementation should be state-less (avoid dependency on instance variables).br - * + * * brbThe API is still beta and can change./b */ public interface ITrigger @@ -46,7 +46,7 @@ public interface ITrigger * * @param partitionKey - partition Key for the update. * @param update - update received for the CF - * @return modifications to be applied, null if no action to be performed. + * @return additional modifications to be applied along with the supplied update */ public CollectionMutation augment(ByteBuffer partitionKey, ColumnFamily update); } http://git-wip-us.apache.org/repos/asf/cassandra/blob/116de2be/src/java/org/apache/cassandra/triggers/TriggerExecutor.java -- diff --git a/src/java/org/apache/cassandra/triggers/TriggerExecutor.java b/src/java/org/apache/cassandra/triggers/TriggerExecutor.java index 677daad..bc6f46c 100644 --- a/src/java/org/apache/cassandra/triggers/TriggerExecutor.java +++ b/src/java/org/apache/cassandra/triggers/TriggerExecutor.java @@ -60,6 +60,23 @@ public class TriggerExecutor cachedTriggers.clear(); } +/** + * Augment a partition update by executing triggers to generate an intermediate + * set of mutations, then merging the ColumnFamily from each mutation with those + * supplied. This is called from @{link org.apache.cassandra.service.StorageProxy#cas} + * which is scoped for a single partition. For that reason, any mutations generated + * by triggers are checked to ensure that they are for the same table and partition + * key as the primary update; if not, InvalidRequestException is thrown. If no + * additional mutations are generated, the original updates are returned unmodified. + * + * @param key partition key for the update + * @param updates partition update to be applied, contains the merge of the original + *update and any generated mutations + * @return the final update to be applied, the original update merged with any + * additional mutations generated by configured triggers + * @throws InvalidRequestException if any mutation generated by a trigger does not + * apply to the exact same partition as the initial update + */ public ColumnFamily execute(ByteBuffer key, ColumnFamily updates) throws InvalidRequestException { ListMutation intermediate = executeInternal(key, updates); @@ -78,6 +95,21 @@ public class TriggerExecutor return updates; } +/** + * Takes a collection of mutations and possibly augments it by adding extra mutations + * generated by configured triggers. If no additional mutations are created + * this returns null, signalling to the caller that only the initial set of + * mutations should be applied. If additional mutations iare/i generated, + * the total set (i.e. the original plus the additional mutations) are applied + * together in a logged batch. Should this not be possible because the initial + * mutations contain counter updates, InvalidRequestException is thrown. + * + * @param mutations initial collection of mutations + * @return augmented mutations. Either the union of the initial and
[1/9] cassandra git commit: Improve javadoc around trigger execution
Repository: cassandra Updated Branches: refs/heads/cassandra-2.0 0b92967d3 - 37d81b9dc refs/heads/cassandra-2.1 59b8e171e - 116de2bee refs/heads/cassandra-2.2 816c2b3f7 - 86c9c00e9 refs/heads/trunk c844bfeea - 3a6e9c91b Improve javadoc around trigger execution Patch by Sam Tunnicliffe; reviewed by brandonwilliams for CASSANDRA-9334 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/37d81b9d Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/37d81b9d Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/37d81b9d Branch: refs/heads/cassandra-2.0 Commit: 37d81b9dcb08dd8c16a560591f46481b5d316841 Parents: 0b92967 Author: Sam Tunnicliffe s...@beobal.com Authored: Tue May 12 11:49:03 2015 +0100 Committer: Sam Tunnicliffe s...@beobal.com Committed: Tue May 19 09:39:56 2015 +0100 -- .../org/apache/cassandra/triggers/ITrigger.java | 4 +-- .../cassandra/triggers/TriggerExecutor.java | 32 2 files changed, 34 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/37d81b9d/src/java/org/apache/cassandra/triggers/ITrigger.java -- diff --git a/src/java/org/apache/cassandra/triggers/ITrigger.java b/src/java/org/apache/cassandra/triggers/ITrigger.java index 15ed7ba..fe2a9e6 100644 --- a/src/java/org/apache/cassandra/triggers/ITrigger.java +++ b/src/java/org/apache/cassandra/triggers/ITrigger.java @@ -36,7 +36,7 @@ import org.apache.cassandra.db.RowMutation; * 2) ITrigger implementation can be instantiated multiple times during the server life time. * (Depends on the number of times trigger folder is updated.)br * 3) ITrigger implementation should be state-less (avoid dependency on instance variables).br - * + * * brbThe API is still beta and can change./b */ public interface ITrigger @@ -46,7 +46,7 @@ public interface ITrigger * * @param key - Row Key for the update. * @param update - Update received for the CF - * @return modifications to be applied, null if no action to be performed. + * @return additional modifications to be applied along with the supplied update */ public CollectionRowMutation augment(ByteBuffer key, ColumnFamily update); } http://git-wip-us.apache.org/repos/asf/cassandra/blob/37d81b9d/src/java/org/apache/cassandra/triggers/TriggerExecutor.java -- diff --git a/src/java/org/apache/cassandra/triggers/TriggerExecutor.java b/src/java/org/apache/cassandra/triggers/TriggerExecutor.java index 988c6a7..bae0da6 100644 --- a/src/java/org/apache/cassandra/triggers/TriggerExecutor.java +++ b/src/java/org/apache/cassandra/triggers/TriggerExecutor.java @@ -64,6 +64,23 @@ public class TriggerExecutor cachedTriggers.clear(); } +/** + * Augment a partition update by executing triggers to generate an intermediate + * set of mutations, then merging the ColumnFamily from each mutation with those + * supplied. This is called from @{link org.apache.cassandra.service.StorageProxy#cas} + * which is scoped for a single partition. For that reason, any mutations generated + * by triggers are checked to ensure that they are for the same table and partition + * key as the primary update; if not, InvalidRequestException is thrown. If no + * additional mutations are generated, the original updates are returned unmodified. + * + * @param key partition key for the update + * @param updates partition update to be applied, contains the merge of the original + *update and any generated mutations + * @return the final update to be applied, the original update merged with any + * additional mutations generated by configured triggers + * @throws InvalidRequestException if any mutation generated by a trigger does not + * apply to the exact same partition as the initial update + */ public ColumnFamily execute(ByteBuffer key, ColumnFamily updates) throws InvalidRequestException { ListRowMutation intermediate = executeInternal(key, updates); @@ -79,6 +96,21 @@ public class TriggerExecutor return updates; } +/** + * Takes a collection of mutations and possibly augments it by adding extra mutations + * generated by configured triggers. If no additional mutations are created + * this returns null, signalling to the caller that only the initial set of + * mutations should be applied. If additional mutations iare/i generated, + * the total set (i.e. the original plus the additional mutations) are applied + * together in a logged batch. Should this not be possible because the
[6/9] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2
Merge branch 'cassandra-2.1' into cassandra-2.2 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e2723a40 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e2723a40 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e2723a40 Branch: refs/heads/trunk Commit: e2723a400d1fec823d853374d3a64c6d061c55a5 Parents: 816c2b3 116de2b Author: Sam Tunnicliffe s...@beobal.com Authored: Tue May 19 12:02:33 2015 +0100 Committer: Sam Tunnicliffe s...@beobal.com Committed: Tue May 19 12:02:33 2015 +0100 -- .../org/apache/cassandra/triggers/ITrigger.java | 4 +-- .../cassandra/triggers/TriggerExecutor.java | 32 2 files changed, 34 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/e2723a40/src/java/org/apache/cassandra/triggers/TriggerExecutor.java --
[5/9] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2
Merge branch 'cassandra-2.1' into cassandra-2.2 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e2723a40 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e2723a40 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e2723a40 Branch: refs/heads/cassandra-2.2 Commit: e2723a400d1fec823d853374d3a64c6d061c55a5 Parents: 816c2b3 116de2b Author: Sam Tunnicliffe s...@beobal.com Authored: Tue May 19 12:02:33 2015 +0100 Committer: Sam Tunnicliffe s...@beobal.com Committed: Tue May 19 12:02:33 2015 +0100 -- .../org/apache/cassandra/triggers/ITrigger.java | 4 +-- .../cassandra/triggers/TriggerExecutor.java | 32 2 files changed, 34 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/e2723a40/src/java/org/apache/cassandra/triggers/TriggerExecutor.java --
[2/9] cassandra git commit: Improve javadoc around trigger execution
Improve javadoc around trigger execution Patch by Sam Tunnicliffe; reviewed by brandonwilliams for CASSANDRA-9334 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/116de2be Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/116de2be Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/116de2be Branch: refs/heads/cassandra-2.1 Commit: 116de2bee5920bc6842300a3f686d88f443b281d Parents: 59b8e17 Author: Sam Tunnicliffe s...@beobal.com Authored: Tue May 12 11:49:03 2015 +0100 Committer: Sam Tunnicliffe s...@beobal.com Committed: Tue May 19 11:59:44 2015 +0100 -- .../org/apache/cassandra/triggers/ITrigger.java | 4 +-- .../cassandra/triggers/TriggerExecutor.java | 32 2 files changed, 34 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/116de2be/src/java/org/apache/cassandra/triggers/ITrigger.java -- diff --git a/src/java/org/apache/cassandra/triggers/ITrigger.java b/src/java/org/apache/cassandra/triggers/ITrigger.java index 4701b61..21aba05 100644 --- a/src/java/org/apache/cassandra/triggers/ITrigger.java +++ b/src/java/org/apache/cassandra/triggers/ITrigger.java @@ -36,7 +36,7 @@ import org.apache.cassandra.db.Mutation; * 2) ITrigger implementation can be instantiated multiple times during the server life time. * (Depends on the number of times trigger folder is updated.)br * 3) ITrigger implementation should be state-less (avoid dependency on instance variables).br - * + * * brbThe API is still beta and can change./b */ public interface ITrigger @@ -46,7 +46,7 @@ public interface ITrigger * * @param partitionKey - partition Key for the update. * @param update - update received for the CF - * @return modifications to be applied, null if no action to be performed. + * @return additional modifications to be applied along with the supplied update */ public CollectionMutation augment(ByteBuffer partitionKey, ColumnFamily update); } http://git-wip-us.apache.org/repos/asf/cassandra/blob/116de2be/src/java/org/apache/cassandra/triggers/TriggerExecutor.java -- diff --git a/src/java/org/apache/cassandra/triggers/TriggerExecutor.java b/src/java/org/apache/cassandra/triggers/TriggerExecutor.java index 677daad..bc6f46c 100644 --- a/src/java/org/apache/cassandra/triggers/TriggerExecutor.java +++ b/src/java/org/apache/cassandra/triggers/TriggerExecutor.java @@ -60,6 +60,23 @@ public class TriggerExecutor cachedTriggers.clear(); } +/** + * Augment a partition update by executing triggers to generate an intermediate + * set of mutations, then merging the ColumnFamily from each mutation with those + * supplied. This is called from @{link org.apache.cassandra.service.StorageProxy#cas} + * which is scoped for a single partition. For that reason, any mutations generated + * by triggers are checked to ensure that they are for the same table and partition + * key as the primary update; if not, InvalidRequestException is thrown. If no + * additional mutations are generated, the original updates are returned unmodified. + * + * @param key partition key for the update + * @param updates partition update to be applied, contains the merge of the original + *update and any generated mutations + * @return the final update to be applied, the original update merged with any + * additional mutations generated by configured triggers + * @throws InvalidRequestException if any mutation generated by a trigger does not + * apply to the exact same partition as the initial update + */ public ColumnFamily execute(ByteBuffer key, ColumnFamily updates) throws InvalidRequestException { ListMutation intermediate = executeInternal(key, updates); @@ -78,6 +95,21 @@ public class TriggerExecutor return updates; } +/** + * Takes a collection of mutations and possibly augments it by adding extra mutations + * generated by configured triggers. If no additional mutations are created + * this returns null, signalling to the caller that only the initial set of + * mutations should be applied. If additional mutations iare/i generated, + * the total set (i.e. the original plus the additional mutations) are applied + * together in a logged batch. Should this not be possible because the initial + * mutations contain counter updates, InvalidRequestException is thrown. + * + * @param mutations initial collection of mutations + * @return augmented mutations. Either the union of the initial
[8/9] cassandra git commit: Add test for triggers which throw IRE
Add test for triggers which throw IRE Patch by Sam Tunnicliffe; reviewed by brandonwilliams for CASSANDRA-9334 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/86c9c00e Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/86c9c00e Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/86c9c00e Branch: refs/heads/trunk Commit: 86c9c00e98a1867deff3b8acc883d224b532197b Parents: e2723a4 Author: Sam Tunnicliffe s...@beobal.com Authored: Tue May 12 16:26:24 2015 +0100 Committer: Sam Tunnicliffe s...@beobal.com Committed: Tue May 19 12:02:40 2015 +0100 -- .../apache/cassandra/triggers/TriggersTest.java | 49 +++- 1 file changed, 38 insertions(+), 11 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/86c9c00e/test/unit/org/apache/cassandra/triggers/TriggersTest.java -- diff --git a/test/unit/org/apache/cassandra/triggers/TriggersTest.java b/test/unit/org/apache/cassandra/triggers/TriggersTest.java index 41d4bb8..b0a5aca 100644 --- a/test/unit/org/apache/cassandra/triggers/TriggersTest.java +++ b/test/unit/org/apache/cassandra/triggers/TriggersTest.java @@ -22,18 +22,13 @@ import java.nio.ByteBuffer; import java.util.Collection; import java.util.Collections; -import org.junit.AfterClass; -import org.junit.Before; -import org.junit.BeforeClass; -import org.junit.Test; +import org.junit.*; import org.apache.cassandra.SchemaLoader; import org.apache.cassandra.config.Schema; import org.apache.cassandra.cql3.QueryProcessor; import org.apache.cassandra.cql3.UntypedResultSet; -import org.apache.cassandra.db.ArrayBackedSortedColumns; -import org.apache.cassandra.db.BufferCell; -import org.apache.cassandra.db.ColumnFamily; +import org.apache.cassandra.db.*; import org.apache.cassandra.db.ConsistencyLevel; import org.apache.cassandra.db.Mutation; import org.apache.cassandra.exceptions.ConfigurationException; @@ -42,11 +37,10 @@ import org.apache.cassandra.service.StorageService; import org.apache.cassandra.thrift.*; import org.apache.thrift.protocol.TBinaryProtocol; -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertTrue; - import static org.apache.cassandra.utils.ByteBufferUtil.bytes; import static org.apache.cassandra.utils.ByteBufferUtil.toInt; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; public class TriggersTest { @@ -61,12 +55,12 @@ public class TriggersTest public static void beforeTest() throws ConfigurationException { SchemaLoader.loadSchema(); +StorageService.instance.initServer(0); } @Before public void setup() throws Exception { -StorageService.instance.initServer(0); if (thriftServer == null || ! thriftServer.isRunning()) { thriftServer = new ThriftServer(InetAddress.getLocalHost(), 9170, 50); @@ -283,6 +277,29 @@ public class TriggersTest } } +@Test(expected=RuntimeException.class) +public void ifTriggerThrowsErrorNoMutationsAreApplied() throws Exception +{ +String cf = cf + System.nanoTime(); +try +{ +setupTableWithTrigger(cf, ErrorTrigger.class); +String cql = String.format(INSERT INTO %s.%s (k, v1) VALUES (11, 11), ksName, cf); +QueryProcessor.process(cql, ConsistencyLevel.ONE); +} +catch (Exception e) +{ +Throwable cause = e.getCause(); +assertTrue((cause instanceof org.apache.cassandra.exceptions.InvalidRequestException)); +assertTrue(cause.getMessage().equals(ErrorTrigger.MESSAGE)); +throw e; +} +finally +{ +assertUpdateNotExecuted(cf, 11); +} +} + private void setupTableWithTrigger(String cf, Class? extends ITrigger triggerImpl) throws RequestExecutionException { @@ -350,4 +367,14 @@ public class TriggersTest return Collections.singletonList(new Mutation(ksName, key, extraUpdate)); } } + +public static class ErrorTrigger implements ITrigger +{ +public static final String MESSAGE = Thrown by ErrorTrigger; +public CollectionMutation augment(ByteBuffer partitionKey, ColumnFamily update) +{ +throw new org.apache.cassandra.exceptions.InvalidRequestException(MESSAGE); +} +} + }
[jira] [Updated] (CASSANDRA-9421) TriggerExecutor should not wrap all execution exceptions in a RuntimeException
[ https://issues.apache.org/jira/browse/CASSANDRA-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-9421: --- Priority: Minor (was: Major) Fix Version/s: 2.2.x Issue Type: Improvement (was: Bug) TriggerExecutor should not wrap all execution exceptions in a RuntimeException -- Key: CASSANDRA-9421 URL: https://issues.apache.org/jira/browse/CASSANDRA-9421 Project: Cassandra Issue Type: Improvement Reporter: Sam Tunnicliffe Priority: Minor Fix For: 2.2.x It may be desirable to have implementations of {{ITrigger#augment}} throw {{RequestValidationException}} or {{RequestExecutionException}} when conditions for applying the trigger are not met (see CASSANDRA-9334 for more discussion). {{TriggerExecutor#executeInternal}} currently catches any exception thrown by a trigger, wraps it in a {{RuntimeException}} and rethrows. It would be better to allow {{CassandraExceptions}} to propagate back to clients directly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-9421) TriggerExecutor should not wrap all execution exceptions in a RuntimeException
Sam Tunnicliffe created CASSANDRA-9421: -- Summary: TriggerExecutor should not wrap all execution exceptions in a RuntimeException Key: CASSANDRA-9421 URL: https://issues.apache.org/jira/browse/CASSANDRA-9421 Project: Cassandra Issue Type: Bug Reporter: Sam Tunnicliffe It may be desirable to have implementations of {{ITrigger#augment}} throw {{RequestValidationException}} or {{RequestExecutionException}} when conditions for applying the trigger are not met (see CASSANDRA-9334 for more discussion). {{TriggerExecutor#executeInternal}} currently catches any exception thrown by a trigger, wraps it in a {{RuntimeException}} and rethrows. It would be better to allow {{CassandraExceptions}} to propagate back to clients directly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9419) CqlRecordWriter does not adequately determine if driver failed to connect due to interrupt
[ https://issues.apache.org/jira/browse/CASSANDRA-9419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-9419: --- Reviewer: Sam Tunnicliffe (was: Aleksey Yeschenko) CqlRecordWriter does not adequately determine if driver failed to connect due to interrupt -- Key: CASSANDRA-9419 URL: https://issues.apache.org/jira/browse/CASSANDRA-9419 Project: Cassandra Issue Type: Bug Reporter: Philip Thompson Assignee: Philip Thompson Fix For: 2.2 rc1 Attachments: 9419.txt Until https://datastax-oss.atlassian.net/browse/JAVA-769 is implemented, I have had to implement a workaround. Java's thread interrupt status flag can be cleared for any number of reasons. After discussion, I've changed to parsing the driver's DriverException to tell if we are safe to retry the failed connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9186) AbstractType vs. CQL3Type loses frozen keyword
[ https://issues.apache.org/jira/browse/CASSANDRA-9186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14550285#comment-14550285 ] Aleksey Yeschenko commented on CASSANDRA-9186: -- bq. but that probably mean we want to complain if a function is defined with a frozen type instead of just ignoring it silently as the later would probably be a bit confusing. Right. +1 AbstractType vs. CQL3Type loses frozen keyword -- Key: CASSANDRA-9186 URL: https://issues.apache.org/jira/browse/CASSANDRA-9186 Project: Cassandra Issue Type: Bug Components: Core Reporter: Robert Stupp Assignee: Robert Stupp Priority: Minor Fix For: 3.x {code} create keyspace functionmetadatatest with replication = {'class':'SimpleStrategy', 'replication_factor':1}; use functionmetadatatest ; CREATE TYPE udtx (x int); CREATE FUNCTION u_frozen(u frozenudtx)RETURNS int LANGUAGE java AS 'return new Integer(0);'; SELECT function_name, signature, argument_types FROM system.schema_functions WHERE keyspace_name='functionmetadatatest'; function_name | signature| argument_types ---+--+-- u_frozen | ['udtx'] | ['org.apache.cassandra.db.marshal.UserType(functionmetadatatest,75647478,78:org.apache.cassandra.db.marshal.Int32Type)'] {code} Problem is that {{UserType}} and {{TupleType}} do not have a _frozen_ (or _isMultiCell_) attribute. A conversion from type to cql3type therefore loses the _frozen_ status. EDIT: The issue has been discovered by [~aholmber] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-9422) SystemKeyspaceTest fails with non-release version string
Sam Tunnicliffe created CASSANDRA-9422: -- Summary: SystemKeyspaceTest fails with non-release version string Key: CASSANDRA-9422 URL: https://issues.apache.org/jira/browse/CASSANDRA-9422 Project: Cassandra Issue Type: Bug Reporter: Sam Tunnicliffe Assignee: Sam Tunnicliffe Fix For: 2.2 beta 1 {{SystemKeyspaceTest#snapshotSystemKeyspaceIfUpgrading}} is currently failing on the 2.2 branch because the build version {{2.2.0-beta1-SNAPSHOT}} cannot be parsed as a {{SemanticVersion}}. This is only a problem with the test, as the code under test doesn't actually use {{SemanticVersion}}, it just checks whether the old new version strings are different. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8163) Re-introduce DESCRIBE permission
[ https://issues.apache.org/jira/browse/CASSANDRA-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14550333#comment-14550333 ] Aleksey Yeschenko commented on CASSANDRA-8163: -- Can we, for 2.2, just introduce the permission to the enum, and have it be a no-op? I don't want us to break the API again in 3.0. Re-introduce DESCRIBE permission Key: CASSANDRA-8163 URL: https://issues.apache.org/jira/browse/CASSANDRA-8163 Project: Cassandra Issue Type: Improvement Reporter: Vishy Kasar Priority: Minor Fix For: 3.x We have a cluster like this: project1_keyspace table101 table102 project2_keyspace table201 table202 We have set up following users and grants: project1_user has all access to project1_keyspace project2_user has all access to project2_keyspace However project1_user can still do a 'describe schema' and get the schema for project2_keyspace as well. We do not want project1_user to have any knowledge for project2 in any way (cqlsh/java-driver etc) . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9399) cqlsh support for native protocol v4 features
[ https://issues.apache.org/jira/browse/CASSANDRA-9399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14551818#comment-14551818 ] Stefania commented on CASSANDRA-9399: - Implemented client warnings in cqlsh and fixed formatting issue for dates, CASSANDRA-8975. cqlsh support for native protocol v4 features - Key: CASSANDRA-9399 URL: https://issues.apache.org/jira/browse/CASSANDRA-9399 Project: Cassandra Issue Type: Bug Reporter: Aleksey Yeschenko Assignee: Stefania Labels: cqlsh Fix For: 2.2.0 rc1 cqlsh/python-driver need to add support for all the new 2.2 CQL features: - {{date}} and {{time}} types - CASSANDRA-7523 - not in cqlsh yet - {{smallint}} and {{tinyint}} types - CASSANDRA-8951 - not in the driver yet - client warnings - CASSANDRA-8930 - tracing improvements - CASSANDRA-7807 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-9403) Experiment with skipping file syncs during unit tests to reduce test time
[ https://issues.apache.org/jira/browse/CASSANDRA-9403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14551589#comment-14551589 ] Stefania edited comment on CASSANDRA-9403 at 5/20/15 1:09 AM: -- Regarding this utest failure, http://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-C-9403-testall/lastCompletedBuild/testReport/org.apache.cassandra.io.sstable/SSTableRewriterTest/testNumberOfFilesAndSizes/, I would simply store {{s.bytesOnDisk()}} in a variable just like {{startStorageMetricsLoad}}. As far as I understand the code, once the sstable is compacted and replaced, at line 285, the reference is released and there is nothing stopping the deleting task from removing the files, see DescriptorTypeTidy in SSTableReader and SSTableDeletingTask. SSTableDeletingTask runs asynchronously. I can reliably fail that assertion by calling SSTableDeletingTask.waitForDeletions(); just before it. Alternatively you can try increasing the reference count for s, {{s.selfRef().tryRef()}}, and this should keep the files but you also need to release it at the end of the test, {{s.selfRef().release()}}. was (Author: stefania): Regarding this utest failure, http://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-C-9403-testall/lastCompletedBuild/testReport/org.apache.cassandra.io.sstable/SSTableRewriterTest/testNumberOfFilesAndSizes/, I would simply store {{s.bytesOnDisk()}} in a variable just like {{startStorageMetricsLoad}}. As far as I understand the code, once the sstable is compacted and replaced, at line 285, the reference is released and there is nothing stopping the deleting task from removing the files, see DescriptorTypeTidy in SSTableReader and SSTableDeletingTask. The SSTableDeletingTask runs asynchronously. I can reliably fail that assertion by calling SSTableDeletingTask.waitForDeletions(); just before it. Alternatively you can try increasing the reference count for s, {{s.selfRef().tryRef()}}, and this should keep the files but you also need to release it at the end of the test, {{s.selfRef().release()}}. Experiment with skipping file syncs during unit tests to reduce test time - Key: CASSANDRA-9403 URL: https://issues.apache.org/jira/browse/CASSANDRA-9403 Project: Cassandra Issue Type: Test Reporter: Ariel Weisberg Assignee: Ariel Weisberg Some environments have ridiculous outliers for disk syncing. 20 seconds ridiculous. Unit tests aren't testing crash safety so it is a pointless exercise. Instead we could intercept calls to sync files and check whether it looks like the sync would succeed. Check that the things are not null, mapped, closed etc. Outside of units tests it can go straight to the regular sync call. I would also like to have the disks for unit and dtests mounted with barrier=0,noatime,nodiratime to further reduce susceptibility to outliers. We aren't going to recover these nodes if they crash/restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-9434) If a node loses schema_columns SSTables it could delete all secondary indexes from the schema
Richard Low created CASSANDRA-9434: -- Summary: If a node loses schema_columns SSTables it could delete all secondary indexes from the schema Key: CASSANDRA-9434 URL: https://issues.apache.org/jira/browse/CASSANDRA-9434 Project: Cassandra Issue Type: Bug Reporter: Richard Low It is possible that a single bad node can delete all secondary indexes if it restarts and cannot read its schema_columns SSTables. Here's a reproduction: * Create a 2 node cluster (we saw it on 2.0.11) * Create the schema: create keyspace myks with replication = {'class':'SimpleStrategy', 'replication_factor':1}; use myks; create table mytable (a text, b text, c text, PRIMARY KEY (a, b) ); create index myindex on mytable(b); NB index must be on clustering column to repro * Kill one node * Wipe its commitlog and system/schema_columns sstables. * Start it again * Run on this node select index_name from system.schema_columns where keyspace_name = 'myks' and columnfamily_name = 'mytable' and column_name = 'b'; and you'll see the index is null. * Run 'describe schema' on the other node. Sometimes it will not show the index, but you might need to bounce for it to disappear. I think the culprit is SystemKeyspace.copyAllAliasesToColumnsProper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9206) Remove seed gossip probability
[ https://issues.apache.org/jira/browse/CASSANDRA-9206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14551604#comment-14551604 ] sankalp kohli commented on CASSANDRA-9206: -- Instead of removing this completely, why can't we make the default 1 and make it tunable. We have seen Gossip backing up many times due to heavy load. With this, the point at which things go south will come much earlier. Remove seed gossip probability -- Key: CASSANDRA-9206 URL: https://issues.apache.org/jira/browse/CASSANDRA-9206 Project: Cassandra Issue Type: Improvement Reporter: Brandon Williams Assignee: Brandon Williams Fix For: 3.x Attachments: 9206.txt Currently, we use probability to determine whether a node will gossip with a seed: {noformat} double probability = seeds.size() / (double) (liveEndpoints.size() + unreachableEndpoints.size()); double randDbl = random.nextDouble(); if (randDbl = probability) sendGossip(prod, seeds); {noformat} I propose that we remove this probability, and instead *always* gossip with a seed. This of course means increased traffic and processing on the seed(s), but even a 1000 node cluster with a single seed will only put ~1000 messages per second on the seed, which is virtually nothing. Should it become a problem, the solution is simple: add more seeds. Since seeds will also always gossip with each other, this effectively gives us a poor man's spanning tree, with the only cost being removing a few lines of code, and should greatly improve our gossip convergence time, especially in large clusters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9422) SystemKeyspaceTest fails with non-release version string
[ https://issues.apache.org/jira/browse/CASSANDRA-9422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14551657#comment-14551657 ] Jeremiah Jordan commented on CASSANDRA-9422: Better to fix the SemanticVersion code to parse the version string? We have other code that sends the version string to SemanticVersion to gate messages during rolling upgrades. Will the beta-snapshot version mess those up? SystemKeyspaceTest fails with non-release version string Key: CASSANDRA-9422 URL: https://issues.apache.org/jira/browse/CASSANDRA-9422 Project: Cassandra Issue Type: Bug Reporter: Sam Tunnicliffe Assignee: Sam Tunnicliffe Fix For: 2.2.0 rc1 {{SystemKeyspaceTest#snapshotSystemKeyspaceIfUpgrading}} is currently failing on the 2.2 branch because the build version {{2.2.0-beta1-SNAPSHOT}} cannot be parsed as a {{SemanticVersion}}. This is only a problem with the test, as the code under test doesn't actually use {{SemanticVersion}}, it just checks whether the old new version strings are different. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-9403) Experiment with skipping file syncs during unit tests to reduce test time
[ https://issues.apache.org/jira/browse/CASSANDRA-9403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14551589#comment-14551589 ] Stefania edited comment on CASSANDRA-9403 at 5/20/15 1:10 AM: -- Regarding this utest failure, http://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-C-9403-testall/lastCompletedBuild/testReport/org.apache.cassandra.io.sstable/SSTableRewriterTest/testNumberOfFilesAndSizes/, I would simply store {{s.bytesOnDisk()}} in a variable just like {{startStorageMetricsLoad}}. As far as I understand the code, once the sstable is compacted and replaced, at line 285, the reference is released and there is nothing stopping the deleting task from removing the files, see DescriptorTypeTidy in SSTableReader and SSTableDeletingTask. SSTableDeletingTask runs asynchronously. I can reliably fail that assertion by calling SSTableDeletingTask.waitForDeletions(); just before it. Alternatively we can try increasing the reference count for s, {{s.selfRef().tryRef()}}, and this should keep the files but we also need to release it at the end of the test, {{s.selfRef().release()}}. Odd that this test is only failing on this branch, else this would merit its own ticket. was (Author: stefania): Regarding this utest failure, http://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-C-9403-testall/lastCompletedBuild/testReport/org.apache.cassandra.io.sstable/SSTableRewriterTest/testNumberOfFilesAndSizes/, I would simply store {{s.bytesOnDisk()}} in a variable just like {{startStorageMetricsLoad}}. As far as I understand the code, once the sstable is compacted and replaced, at line 285, the reference is released and there is nothing stopping the deleting task from removing the files, see DescriptorTypeTidy in SSTableReader and SSTableDeletingTask. SSTableDeletingTask runs asynchronously. I can reliably fail that assertion by calling SSTableDeletingTask.waitForDeletions(); just before it. Alternatively you can try increasing the reference count for s, {{s.selfRef().tryRef()}}, and this should keep the files but you also need to release it at the end of the test, {{s.selfRef().release()}}. Experiment with skipping file syncs during unit tests to reduce test time - Key: CASSANDRA-9403 URL: https://issues.apache.org/jira/browse/CASSANDRA-9403 Project: Cassandra Issue Type: Test Reporter: Ariel Weisberg Assignee: Ariel Weisberg Some environments have ridiculous outliers for disk syncing. 20 seconds ridiculous. Unit tests aren't testing crash safety so it is a pointless exercise. Instead we could intercept calls to sync files and check whether it looks like the sync would succeed. Check that the things are not null, mapped, closed etc. Outside of units tests it can go straight to the regular sync call. I would also like to have the disks for unit and dtests mounted with barrier=0,noatime,nodiratime to further reduce susceptibility to outliers. We aren't going to recover these nodes if they crash/restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9434) If a node loses schema_columns SSTables it could delete all secondary indexes from the schema
[ https://issues.apache.org/jira/browse/CASSANDRA-9434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14551601#comment-14551601 ] Richard Low commented on CASSANDRA-9434: cc [~iamaleksey] If a node loses schema_columns SSTables it could delete all secondary indexes from the schema - Key: CASSANDRA-9434 URL: https://issues.apache.org/jira/browse/CASSANDRA-9434 Project: Cassandra Issue Type: Bug Reporter: Richard Low It is possible that a single bad node can delete all secondary indexes if it restarts and cannot read its schema_columns SSTables. Here's a reproduction: * Create a 2 node cluster (we saw it on 2.0.11) * Create the schema: create keyspace myks with replication = {'class':'SimpleStrategy', 'replication_factor':1}; use myks; create table mytable (a text, b text, c text, PRIMARY KEY (a, b) ); create index myindex on mytable(b); NB index must be on clustering column to repro * Kill one node * Wipe its commitlog and system/schema_columns sstables. * Start it again * Run on this node select index_name from system.schema_columns where keyspace_name = 'myks' and columnfamily_name = 'mytable' and column_name = 'b'; and you'll see the index is null. * Run 'describe schema' on the other node. Sometimes it will not show the index, but you might need to bounce for it to disappear. I think the culprit is SystemKeyspace.copyAllAliasesToColumnsProper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9432) SliceQueryFilterWithTombstonesTest.testExpiredTombstones is failing infrequently
[ https://issues.apache.org/jira/browse/CASSANDRA-9432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14551489#comment-14551489 ] Aleksey Yeschenko commented on CASSANDRA-9432: -- +1 SliceQueryFilterWithTombstonesTest.testExpiredTombstones is failing infrequently Key: CASSANDRA-9432 URL: https://issues.apache.org/jira/browse/CASSANDRA-9432 Project: Cassandra Issue Type: Test Reporter: Ariel Weisberg Assignee: Ariel Weisberg I have only seen one instance of this, but haven't looked hard. http://cassci.datastax.com/view/trunk/job/trunk_testall/98/testReport/junit/org.apache.cassandra.cql3/SliceQueryFilterWithTombstonesTest/testExpiredTombstones/ I have it running in a loop and it's not reproducing. For now I just want to log the exception that is causing the failure. Need the ticket to to tie the commit to. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9399) cqlsh support for native protocol v4 features
[ https://issues.apache.org/jira/browse/CASSANDRA-9399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14551488#comment-14551488 ] Stefania commented on CASSANDRA-9399: - Noted. I will update the tracing tests accordingly, since there is no longer any need to test multiple protocol versions. I will also revert the part of my patch in cqlsh where I added support for specifying the protocol version via a command line argument, {{--protocolversion}} unless you think this might be useful for other reasons or tests. cqlsh support for native protocol v4 features - Key: CASSANDRA-9399 URL: https://issues.apache.org/jira/browse/CASSANDRA-9399 Project: Cassandra Issue Type: Bug Reporter: Aleksey Yeschenko Assignee: Stefania Labels: cqlsh Fix For: 2.2.0 rc1 cqlsh/python-driver need to add support for all the new 2.2 CQL features: - {{date}} and {{time}} types - CASSANDRA-7523 - not in cqlsh yet - {{smallint}} and {{tinyint}} types - CASSANDRA-8951 - not in the driver yet - client warnings - CASSANDRA-8930 - tracing improvements - CASSANDRA-7807 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9403) Experiment with skipping file syncs during unit tests to reduce test time
[ https://issues.apache.org/jira/browse/CASSANDRA-9403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14551589#comment-14551589 ] Stefania commented on CASSANDRA-9403: - Regarding this utest failure, http://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-C-9403-testall/lastCompletedBuild/testReport/org.apache.cassandra.io.sstable/SSTableRewriterTest/testNumberOfFilesAndSizes/, I would simply store {{s.bytesOnDisk()}} in a variable just like {{startStorageMetricsLoad}}. As far as I understand the code, once the sstable is compacted and replaced, at line 285, the reference is released and there is nothing stopping the deleting task from removing the files, see DescriptorTypeTidy in SSTableReader and SSTableDeletingTask. The SSTableDeletingTask runs asynchronously. I can reliably fail that assertion by calling SSTableDeletingTask.waitForDeletions(); just before it. Alternatively you can try increasing the reference count for s, {{s.selfRef().tryRef()}}, and this should keep the files but you also need to release it at the end of the test, {{s.selfRef().release()}}. Experiment with skipping file syncs during unit tests to reduce test time - Key: CASSANDRA-9403 URL: https://issues.apache.org/jira/browse/CASSANDRA-9403 Project: Cassandra Issue Type: Test Reporter: Ariel Weisberg Assignee: Ariel Weisberg Some environments have ridiculous outliers for disk syncing. 20 seconds ridiculous. Unit tests aren't testing crash safety so it is a pointless exercise. Instead we could intercept calls to sync files and check whether it looks like the sync would succeed. Check that the things are not null, mapped, closed etc. Outside of units tests it can go straight to the regular sync call. I would also like to have the disks for unit and dtests mounted with barrier=0,noatime,nodiratime to further reduce susceptibility to outliers. We aren't going to recover these nodes if they crash/restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9434) If a node loses schema_columns SSTables it could delete all secondary indexes from the schema
[ https://issues.apache.org/jira/browse/CASSANDRA-9434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Low updated CASSANDRA-9434: --- Description: It is possible that a single bad node can delete all secondary indexes if it restarts and cannot read its schema_columns SSTables. Here's a reproduction: * Create a 2 node cluster (we saw it on 2.0.11) * Create the schema: {code} create keyspace myks with replication = {'class':'SimpleStrategy', 'replication_factor':1}; use myks; create table mytable (a text, b text, c text, PRIMARY KEY (a, b) ); create index myindex on mytable(b); {code} NB index must be on clustering column to repro * Kill one node * Wipe its commitlog and system/schema_columns sstables. * Start it again * Run on this node select index_name from system.schema_columns where keyspace_name = 'myks' and columnfamily_name = 'mytable' and column_name = 'b'; and you'll see the index is null. * Run 'describe schema' on the other node. Sometimes it will not show the index, but you might need to bounce for it to disappear. I think the culprit is SystemKeyspace.copyAllAliasesToColumnsProper. was: It is possible that a single bad node can delete all secondary indexes if it restarts and cannot read its schema_columns SSTables. Here's a reproduction: * Create a 2 node cluster (we saw it on 2.0.11) * Create the schema: create keyspace myks with replication = {'class':'SimpleStrategy', 'replication_factor':1}; use myks; create table mytable (a text, b text, c text, PRIMARY KEY (a, b) ); create index myindex on mytable(b); NB index must be on clustering column to repro * Kill one node * Wipe its commitlog and system/schema_columns sstables. * Start it again * Run on this node select index_name from system.schema_columns where keyspace_name = 'myks' and columnfamily_name = 'mytable' and column_name = 'b'; and you'll see the index is null. * Run 'describe schema' on the other node. Sometimes it will not show the index, but you might need to bounce for it to disappear. I think the culprit is SystemKeyspace.copyAllAliasesToColumnsProper. If a node loses schema_columns SSTables it could delete all secondary indexes from the schema - Key: CASSANDRA-9434 URL: https://issues.apache.org/jira/browse/CASSANDRA-9434 Project: Cassandra Issue Type: Bug Reporter: Richard Low It is possible that a single bad node can delete all secondary indexes if it restarts and cannot read its schema_columns SSTables. Here's a reproduction: * Create a 2 node cluster (we saw it on 2.0.11) * Create the schema: {code} create keyspace myks with replication = {'class':'SimpleStrategy', 'replication_factor':1}; use myks; create table mytable (a text, b text, c text, PRIMARY KEY (a, b) ); create index myindex on mytable(b); {code} NB index must be on clustering column to repro * Kill one node * Wipe its commitlog and system/schema_columns sstables. * Start it again * Run on this node select index_name from system.schema_columns where keyspace_name = 'myks' and columnfamily_name = 'mytable' and column_name = 'b'; and you'll see the index is null. * Run 'describe schema' on the other node. Sometimes it will not show the index, but you might need to bounce for it to disappear. I think the culprit is SystemKeyspace.copyAllAliasesToColumnsProper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9432) SliceQueryFilterWithTombstonesTest.testExpiredTombstones is failing infrequently
[ https://issues.apache.org/jira/browse/CASSANDRA-9432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Yeschenko updated CASSANDRA-9432: - Reviewer: Aleksey Yeschenko SliceQueryFilterWithTombstonesTest.testExpiredTombstones is failing infrequently Key: CASSANDRA-9432 URL: https://issues.apache.org/jira/browse/CASSANDRA-9432 Project: Cassandra Issue Type: Test Reporter: Ariel Weisberg Assignee: Ariel Weisberg I have only seen one instance of this, but haven't looked hard. http://cassci.datastax.com/view/trunk/job/trunk_testall/98/testReport/junit/org.apache.cassandra.cql3/SliceQueryFilterWithTombstonesTest/testExpiredTombstones/ I have it running in a loop and it's not reproducing. For now I just want to log the exception that is causing the failure. Need the ticket to to tie the commit to. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8917) Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions
[ https://issues.apache.org/jira/browse/CASSANDRA-8917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Ogden updated CASSANDRA-8917: -- Attachment: Screen Shot 2015-05-19 at 10.50.23 AM.png Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions - Key: CASSANDRA-8917 URL: https://issues.apache.org/jira/browse/CASSANDRA-8917 Project: Cassandra Issue Type: Bug Environment: C* 2.0.9, Centos 6.5, Java 1.7.0_72, spring data cassandra 1.1.1, cassandra java driver 2.0.9 Reporter: Gary Ogden Fix For: 2.1.x Attachments: Screen Shot 2015-05-19 at 10.50.23 AM.png, b_output.log, jersey_error.log, node1-cassandra.yaml, node1-system.log, node2-cassandra.yaml, node2-system.log, node3-cassandra.yaml, node3-system.log We have java apps running on glassfish that read/write to our 3 node cluster running on 2.0.9. we have the CL set to quorum for all reads and writes. When we started to upgrade the first node and did the sstable upgrade on that node, we started getting this error on reads and writes: com.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency QUORUM (2 required but only 1 alive) How is that possible when we have 3 nodes total, and there was 2 that were up and it's saying we can't get the required CL? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8917) Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions
[ https://issues.apache.org/jira/browse/CASSANDRA-8917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14550468#comment-14550468 ] Gary Ogden commented on CASSANDRA-8917: --- I'm actually able to reproduce this same error under a different scenario in a test cluster using devcenter. One of our nodes got into a funny state but is still actually running. Here's the log output: ERROR 13:29:32 JVM state determined to be unstable. Exiting forcefully due to: java.lang.OutOfMemoryError: Java heap space at org.apache.cassandra.io.util.RandomAccessReader.init(RandomAccessReader.java:69) ~[apache-cassandra-2.1.3.jar:2.1.3] at org.apache.cassandra.io.compress.CompressedRandomAccessReader.init(CompressedRandomAccessReader.java:77) ~[apache-cassandra-2.1.3.jar:2.1.3] at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:56) ~[apache-cassandra-2.1.3.jar:2.1.3] at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1832) ~[apache-cassandra-2.1.3.jar:2.1.3] at org.apache.cassandra.io.sstable.SSTableScanner.init(SSTableScanner.java:78) ~[apache-cassandra-2.1.3.jar:2.1.3] at org.apache.cassandra.io.sstable.SSTableScanner.getScanner(SSTableScanner.java:57) ~[apache-cassandra-2.1.3.jar:2.1.3] at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1602) ~[apache-cassandra-2.1.3.jar:2.1.3] at org.apache.cassandra.db.RowIteratorFactory.getIterator(RowIteratorFactory.java:67) ~[apache-cassandra-2.1.3.jar:2.1.3] at org.apache.cassandra.db.ColumnFamilyStore.getSequentialIterator(ColumnFamilyStore.java:1988) ~[apache-cassandra-2.1.3.jar:2.1.3] at org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:2105) ~[apache-cassandra-2.1.3.jar:2.1.3] at org.apache.cassandra.db.RangeSliceCommand.executeLocally(RangeSliceCommand.java:132) ~[apache-cassandra-2.1.3.jar:2.1.3] at org.apache.cassandra.service.RangeSliceVerbHandler.doVerb(RangeSliceVerbHandler.java:39) ~[apache-cassandra-2.1.3.jar:2.1.3] at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) ~[apache-cassandra-2.1.3.jar:2.1.3] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_75] at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[apache-cassandra-2 .1.3.jar:2.1.3] at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$TraceSessionFutureTask.run(AbstractTracingAwareExecutorService.java:136) [apache- cassandra-2.1.3.jar:2.1.3] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.3.jar:2.1.3] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] CompilerOracle: inline org/apache/cassandra/db/AbstractNativeCell.compareTo (Lorg/apache/cassandra/db/composites/Composite;)I CompilerOracle: inline org/apache/cassandra/db/composites/AbstractSimpleCellNameType.compareUnsigned (Lorg/apache/cassandra/db/composites/Composite;Lorg/apache/ cassandra/db/composites/Composite;)I CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare (Ljava/nio/ByteBuffer;[B)I CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare ([BLjava/nio/ByteBuffer;)I CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/lang/Object;JI)I CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/nio/ByteBuffer;)I CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I CompilerOracle: inline org/apache/cassandra/db/AbstractNativeCell.compareTo (Lorg/apache/cassandra/db/composites/Composite;)I CompilerOracle: inline org/apache/cassandra/db/composites/AbstractSimpleCellNameType.compareUnsigned (Lorg/apache/cassandra/db/composites/Composite;Lorg/apache/ cassandra/db/composites/Composite;)I CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare (Ljava/nio/ByteBuffer;[B)I CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare ([BLjava/nio/ByteBuffer;)I CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/lang/Object;JI)I CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo
[12/16] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
http://git-wip-us.apache.org/repos/asf/cassandra/blob/d693ca12/src/java/org/apache/cassandra/tools/NodeTool.java -- diff --cc src/java/org/apache/cassandra/tools/NodeTool.java index 2c913e0,000..5156479 mode 100644,00..100644 --- a/src/java/org/apache/cassandra/tools/NodeTool.java +++ b/src/java/org/apache/cassandra/tools/NodeTool.java @@@ -1,2685 -1,0 +1,2690 @@@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * License); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an AS IS BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.cassandra.tools; + +import static com.google.common.base.Preconditions.checkArgument; +import static com.google.common.base.Preconditions.checkState; +import static com.google.common.base.Throwables.getStackTraceAsString; +import static com.google.common.collect.Iterables.toArray; +import static com.google.common.collect.Lists.newArrayList; +import static java.lang.Integer.parseInt; +import static java.lang.String.format; +import static org.apache.commons.lang3.ArrayUtils.EMPTY_STRING_ARRAY; +import static org.apache.commons.lang3.StringUtils.EMPTY; +import static org.apache.commons.lang3.StringUtils.isEmpty; +import static org.apache.commons.lang3.StringUtils.isNotEmpty; +import static org.apache.commons.lang3.StringUtils.join; +import io.airlift.command.Arguments; +import io.airlift.command.Cli; +import io.airlift.command.Command; +import io.airlift.command.Help; +import io.airlift.command.Option; +import io.airlift.command.OptionType; +import io.airlift.command.ParseArgumentsMissingException; +import io.airlift.command.ParseArgumentsUnexpectedException; +import io.airlift.command.ParseCommandMissingException; +import io.airlift.command.ParseCommandUnrecognizedException; +import io.airlift.command.ParseOptionConversionException; +import io.airlift.command.ParseOptionMissingException; +import io.airlift.command.ParseOptionMissingValueException; + +import java.io.Console; +import java.io.File; +import java.io.FileNotFoundException; +import java.io.FileWriter; +import java.io.IOError; +import java.io.IOException; +import java.lang.management.MemoryUsage; +import java.net.InetAddress; +import java.net.UnknownHostException; +import java.text.DecimalFormat; +import java.text.SimpleDateFormat; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.Comparator; +import java.util.Date; +import java.util.HashMap; +import java.util.Iterator; +import java.util.List; +import java.util.Map; +import java.util.Map.Entry; +import java.util.Scanner; +import java.util.Set; +import java.util.concurrent.ExecutionException; + +import javax.management.InstanceNotFoundException; +import javax.management.openmbean.CompositeData; +import javax.management.openmbean.OpenDataException; +import javax.management.openmbean.TabularData; +import javax.management.openmbean.TabularDataSupport; + +import org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutorMBean; +import org.apache.cassandra.db.ColumnFamilyStoreMBean; +import org.apache.cassandra.db.Keyspace; +import org.apache.cassandra.db.compaction.CompactionManagerMBean; +import org.apache.cassandra.db.compaction.OperationType; +import org.apache.cassandra.io.util.FileUtils; +import org.apache.cassandra.locator.EndpointSnitchInfoMBean; +import org.apache.cassandra.metrics.ColumnFamilyMetrics.Sampler; +import org.apache.cassandra.net.MessagingServiceMBean; +import org.apache.cassandra.repair.RepairParallelism; +import org.apache.cassandra.service.CacheServiceMBean; +import org.apache.cassandra.service.StorageProxyMBean; +import org.apache.cassandra.streaming.ProgressInfo; +import org.apache.cassandra.streaming.SessionInfo; +import org.apache.cassandra.streaming.StreamState; +import org.apache.cassandra.utils.FBUtilities; +import org.apache.cassandra.utils.JVMStabilityInspector; +import org.apache.commons.lang3.ArrayUtils; + +import com.google.common.base.Joiner; +import com.google.common.base.Throwables; +import com.google.common.collect.ArrayListMultimap; +import com.google.common.collect.LinkedHashMultimap; +import com.google.common.collect.Lists; +import
[04/16] cassandra git commit: Add option to not validate atoms during scrub
Add option to not validate atoms during scrub patch by Jordan West; reviewed by yukim for CASSANDRA-9406 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/be9eff57 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/be9eff57 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/be9eff57 Branch: refs/heads/trunk Commit: be9eff573342cd4993eb4fc8ee694dc885e8a8e4 Parents: 37d81b9 Author: Jordan West jorda...@gmail.com Authored: Tue May 19 08:26:27 2015 -0500 Committer: Yuki Morishita yu...@apache.org Committed: Tue May 19 08:26:27 2015 -0500 -- CHANGES.txt | 1 + .../apache/cassandra/db/ColumnFamilyStore.java | 4 +-- .../db/compaction/CompactionManager.java| 12 +++ .../cassandra/db/compaction/Scrubber.java | 12 --- .../cassandra/service/StorageService.java | 7 +++- .../cassandra/service/StorageServiceMBean.java | 2 ++ .../org/apache/cassandra/tools/NodeCmd.java | 5 ++- .../org/apache/cassandra/tools/NodeProbe.java | 4 +-- .../cassandra/tools/StandaloneScrubber.java | 6 +++- .../unit/org/apache/cassandra/db/ScrubTest.java | 37 +--- 10 files changed, 60 insertions(+), 30 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/be9eff57/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index a9d04d6..cf124b4 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -3,6 +3,7 @@ * Clone SliceQueryFilter in AbstractReadCommand implementations (CASSANDRA-8940) * Push correct protocol notification for DROP INDEX (CASSANDRA-9310) * token-generator - generated tokens too long (CASSANDRA-9300) + * Add option not to validate atoms during scrub (CASSANDRA-9406) 2.0.15: http://git-wip-us.apache.org/repos/asf/cassandra/blob/be9eff57/src/java/org/apache/cassandra/db/ColumnFamilyStore.java -- diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java index 5ea1287..eec4044 100644 --- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java +++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java @@ -1122,12 +1122,12 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean CompactionManager.instance.performCleanup(ColumnFamilyStore.this, renewer); } -public void scrub(boolean disableSnapshot, boolean skipCorrupted) throws ExecutionException, InterruptedException +public void scrub(boolean disableSnapshot, boolean skipCorrupted, boolean checkData) throws ExecutionException, InterruptedException { // skip snapshot creation during scrub, SEE JIRA 5891 if(!disableSnapshot) snapshotWithoutFlush(pre-scrub- + System.currentTimeMillis()); -CompactionManager.instance.performScrub(ColumnFamilyStore.this, skipCorrupted); +CompactionManager.instance.performScrub(ColumnFamilyStore.this, skipCorrupted, checkData); } public void sstablesRewrite(boolean excludeCurrentVersion) throws ExecutionException, InterruptedException http://git-wip-us.apache.org/repos/asf/cassandra/blob/be9eff57/src/java/org/apache/cassandra/db/compaction/CompactionManager.java -- diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java index 0978ae6..207b90d 100644 --- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java +++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java @@ -234,13 +234,13 @@ public class CompactionManager implements CompactionManagerMBean executor.submit(runnable).get(); } -public void performScrub(ColumnFamilyStore cfStore, final boolean skipCorrupted) throws InterruptedException, ExecutionException +public void performScrub(ColumnFamilyStore cfStore, final boolean skipCorrupted, final boolean checkData) throws InterruptedException, ExecutionException { performAllSSTableOperation(cfStore, new AllSSTablesOperation() { public void perform(ColumnFamilyStore store, IterableSSTableReader sstables) throws IOException { -doScrub(store, sstables, skipCorrupted); +doScrub(store, sstables, skipCorrupted, checkData); } }); } @@ -432,16 +432,16 @@ public class CompactionManager implements CompactionManagerMBean * * @throws IOException */ -private void doScrub(ColumnFamilyStore cfs, IterableSSTableReader sstables, boolean skipCorrupted) throws
[15/16] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2
Merge branch 'cassandra-2.1' into cassandra-2.2 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ff1a1562 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ff1a1562 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ff1a1562 Branch: refs/heads/trunk Commit: ff1a156290440934826cf2e6a7f080d160683a16 Parents: 678291a d693ca1 Author: Yuki Morishita yu...@apache.org Authored: Tue May 19 09:00:08 2015 -0500 Committer: Yuki Morishita yu...@apache.org Committed: Tue May 19 09:00:08 2015 -0500 -- CHANGES.txt | 1 + .../apache/cassandra/db/ColumnFamilyStore.java | 8 ++-- .../db/compaction/CompactionManager.java| 8 ++-- .../cassandra/db/compaction/Scrubber.java | 8 ++-- .../cassandra/service/StorageService.java | 7 +++- .../cassandra/service/StorageServiceMBean.java | 2 + .../org/apache/cassandra/tools/NodeProbe.java | 8 ++-- .../cassandra/tools/StandaloneScrubber.java | 6 ++- .../apache/cassandra/tools/nodetool/Scrub.java | 9 - .../unit/org/apache/cassandra/db/ScrubTest.java | 42 ++-- 10 files changed, 66 insertions(+), 33 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/ff1a1562/CHANGES.txt -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/ff1a1562/src/java/org/apache/cassandra/db/ColumnFamilyStore.java -- diff --cc src/java/org/apache/cassandra/db/ColumnFamilyStore.java index 062efe8,0951c01..738e9eb --- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java +++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java @@@ -1387,64 -1399,12 +1387,64 @@@ public class ColumnFamilyStore implemen return CompactionManager.instance.performCleanup(ColumnFamilyStore.this); } - public CompactionManager.AllSSTableOpStatus scrub(boolean disableSnapshot, boolean skipCorrupted) throws ExecutionException, InterruptedException + public CompactionManager.AllSSTableOpStatus scrub(boolean disableSnapshot, boolean skipCorrupted, boolean checkData) throws ExecutionException, InterruptedException { - return scrub(disableSnapshot, skipCorrupted, false); ++return scrub(disableSnapshot, skipCorrupted, false, checkData); +} + +@VisibleForTesting - public CompactionManager.AllSSTableOpStatus scrub(boolean disableSnapshot, boolean skipCorrupted, boolean alwaysFail) throws ExecutionException, InterruptedException ++public CompactionManager.AllSSTableOpStatus scrub(boolean disableSnapshot, boolean skipCorrupted, boolean alwaysFail, boolean checkData) throws ExecutionException, InterruptedException +{ // skip snapshot creation during scrub, SEE JIRA 5891 if(!disableSnapshot) snapshotWithoutFlush(pre-scrub- + System.currentTimeMillis()); -return CompactionManager.instance.performScrub(ColumnFamilyStore.this, skipCorrupted, checkData); + +try +{ - return CompactionManager.instance.performScrub(ColumnFamilyStore.this, skipCorrupted); ++return CompactionManager.instance.performScrub(ColumnFamilyStore.this, skipCorrupted, checkData); +} +catch(Throwable t) +{ +if (!rebuildOnFailedScrub(t)) +throw t; + +return alwaysFail ? CompactionManager.AllSSTableOpStatus.ABORTED : CompactionManager.AllSSTableOpStatus.SUCCESSFUL; +} +} + +/** + * CASSANDRA-5174 : For an index cfs we may be able to discard everything and just rebuild + * the index when a scrub fails. + * + * @return true if we are an index cfs and we successfully rebuilt the index + */ +public boolean rebuildOnFailedScrub(Throwable failure) +{ +if (!isIndex()) +return false; + +SecondaryIndex index = null; +if (metadata.cfName.contains(Directories.SECONDARY_INDEX_NAME_SEPARATOR)) +{ +String[] parts = metadata.cfName.split(\\ + Directories.SECONDARY_INDEX_NAME_SEPARATOR, 2); +ColumnFamilyStore parentCfs = keyspace.getColumnFamilyStore(parts[0]); +index = parentCfs.indexManager.getIndexByName(metadata.cfName); +assert index != null; +} + +if (index == null) +return false; + +truncateBlocking(); + +logger.warn(Rebuilding index for {} because of {}, name, failure.getMessage()); +index.getBaseCfs().rebuildSecondaryIndex(index.getIndexName()); +return true; +} + +public CompactionManager.AllSSTableOpStatus verify(boolean
[14/16] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2
Merge branch 'cassandra-2.1' into cassandra-2.2 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ff1a1562 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ff1a1562 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ff1a1562 Branch: refs/heads/cassandra-2.2 Commit: ff1a156290440934826cf2e6a7f080d160683a16 Parents: 678291a d693ca1 Author: Yuki Morishita yu...@apache.org Authored: Tue May 19 09:00:08 2015 -0500 Committer: Yuki Morishita yu...@apache.org Committed: Tue May 19 09:00:08 2015 -0500 -- CHANGES.txt | 1 + .../apache/cassandra/db/ColumnFamilyStore.java | 8 ++-- .../db/compaction/CompactionManager.java| 8 ++-- .../cassandra/db/compaction/Scrubber.java | 8 ++-- .../cassandra/service/StorageService.java | 7 +++- .../cassandra/service/StorageServiceMBean.java | 2 + .../org/apache/cassandra/tools/NodeProbe.java | 8 ++-- .../cassandra/tools/StandaloneScrubber.java | 6 ++- .../apache/cassandra/tools/nodetool/Scrub.java | 9 - .../unit/org/apache/cassandra/db/ScrubTest.java | 42 ++-- 10 files changed, 66 insertions(+), 33 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/ff1a1562/CHANGES.txt -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/ff1a1562/src/java/org/apache/cassandra/db/ColumnFamilyStore.java -- diff --cc src/java/org/apache/cassandra/db/ColumnFamilyStore.java index 062efe8,0951c01..738e9eb --- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java +++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java @@@ -1387,64 -1399,12 +1387,64 @@@ public class ColumnFamilyStore implemen return CompactionManager.instance.performCleanup(ColumnFamilyStore.this); } - public CompactionManager.AllSSTableOpStatus scrub(boolean disableSnapshot, boolean skipCorrupted) throws ExecutionException, InterruptedException + public CompactionManager.AllSSTableOpStatus scrub(boolean disableSnapshot, boolean skipCorrupted, boolean checkData) throws ExecutionException, InterruptedException { - return scrub(disableSnapshot, skipCorrupted, false); ++return scrub(disableSnapshot, skipCorrupted, false, checkData); +} + +@VisibleForTesting - public CompactionManager.AllSSTableOpStatus scrub(boolean disableSnapshot, boolean skipCorrupted, boolean alwaysFail) throws ExecutionException, InterruptedException ++public CompactionManager.AllSSTableOpStatus scrub(boolean disableSnapshot, boolean skipCorrupted, boolean alwaysFail, boolean checkData) throws ExecutionException, InterruptedException +{ // skip snapshot creation during scrub, SEE JIRA 5891 if(!disableSnapshot) snapshotWithoutFlush(pre-scrub- + System.currentTimeMillis()); -return CompactionManager.instance.performScrub(ColumnFamilyStore.this, skipCorrupted, checkData); + +try +{ - return CompactionManager.instance.performScrub(ColumnFamilyStore.this, skipCorrupted); ++return CompactionManager.instance.performScrub(ColumnFamilyStore.this, skipCorrupted, checkData); +} +catch(Throwable t) +{ +if (!rebuildOnFailedScrub(t)) +throw t; + +return alwaysFail ? CompactionManager.AllSSTableOpStatus.ABORTED : CompactionManager.AllSSTableOpStatus.SUCCESSFUL; +} +} + +/** + * CASSANDRA-5174 : For an index cfs we may be able to discard everything and just rebuild + * the index when a scrub fails. + * + * @return true if we are an index cfs and we successfully rebuilt the index + */ +public boolean rebuildOnFailedScrub(Throwable failure) +{ +if (!isIndex()) +return false; + +SecondaryIndex index = null; +if (metadata.cfName.contains(Directories.SECONDARY_INDEX_NAME_SEPARATOR)) +{ +String[] parts = metadata.cfName.split(\\ + Directories.SECONDARY_INDEX_NAME_SEPARATOR, 2); +ColumnFamilyStore parentCfs = keyspace.getColumnFamilyStore(parts[0]); +index = parentCfs.indexManager.getIndexByName(metadata.cfName); +assert index != null; +} + +if (index == null) +return false; + +truncateBlocking(); + +logger.warn(Rebuilding index for {} because of {}, name, failure.getMessage()); +index.getBaseCfs().rebuildSecondaryIndex(index.getIndexName()); +return true; +} + +public CompactionManager.AllSSTableOpStatus
[08/16] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
http://git-wip-us.apache.org/repos/asf/cassandra/blob/d693ca12/src/java/org/apache/cassandra/tools/StandaloneScrubber.java -- diff --cc src/java/org/apache/cassandra/tools/StandaloneScrubber.java index d420218,26768f3..459df98 --- a/src/java/org/apache/cassandra/tools/StandaloneScrubber.java +++ b/src/java/org/apache/cassandra/tools/StandaloneScrubber.java @@@ -109,7 -121,7 +110,7 @@@ public class StandaloneScrubbe { try { - Scrubber scrubber = new Scrubber(cfs, sstable, options.skipCorrupted, handler, true); -Scrubber scrubber = new Scrubber(cfs, sstable, options.skipCorrupted, !options.noValidate, handler, true); ++Scrubber scrubber = new Scrubber(cfs, sstable, options.skipCorrupted, handler, true, !options.noValidate); try { scrubber.scrub(); http://git-wip-us.apache.org/repos/asf/cassandra/blob/d693ca12/test/unit/org/apache/cassandra/db/ScrubTest.java -- diff --cc test/unit/org/apache/cassandra/db/ScrubTest.java index a19c76d,94c7e34..028cf6c --- a/test/unit/org/apache/cassandra/db/ScrubTest.java +++ b/test/unit/org/apache/cassandra/db/ScrubTest.java @@@ -30,7 -30,10 +30,11 @@@ import java.util.Set import java.util.concurrent.ExecutionException; import org.apache.cassandra.cql3.QueryProcessor; -import org.apache.cassandra.db.compaction.OperationType; ++import org.apache.cassandra.db.composites.CellNameType; +import org.apache.cassandra.exceptions.ConfigurationException; + import org.apache.cassandra.db.marshal.CompositeType; + import org.apache.cassandra.db.marshal.LongType; + import org.apache.cassandra.db.marshal.UTF8Type; import org.apache.cassandra.exceptions.RequestExecutionException; import org.apache.cassandra.io.compress.CompressionMetadata; import org.apache.cassandra.utils.UUIDGen; @@@ -124,7 -123,7 +128,7 @@@ public class ScrubTest extends SchemaLo overrideWithGarbage(sstable, ByteBufferUtil.bytes(0), ByteBufferUtil.bytes(1)); // with skipCorrupted == false, the scrub is expected to fail - Scrubber scrubber = new Scrubber(cfs, sstable, false, false); -Scrubber scrubber = new Scrubber(cfs, sstable, false, true); ++Scrubber scrubber = new Scrubber(cfs, sstable, false, false, true); try { scrubber.scrub(); @@@ -134,9 -133,10 +138,9 @@@ // with skipCorrupted == true, the corrupt row will be skipped Scrubber.ScrubResult scrubResult; - scrubber = new Scrubber(cfs, sstable, true, false); -scrubber = new Scrubber(cfs, sstable, true, true); ++scrubber = new Scrubber(cfs, sstable, true, false, true); scrubResult = scrubber.scrubWithResult(); scrubber.close(); -cfs.replaceCompactedSSTables(Collections.singletonList(sstable), Collections.singletonList(scrubber.getNewSSTable()), OperationType.SCRUB); assertNotNull(scrubResult); @@@ -182,7 -181,7 +186,7 @@@ overrideWithGarbage(sstable, ByteBufferUtil.bytes(0), ByteBufferUtil.bytes(1)); // with skipCorrupted == false, the scrub is expected to fail - Scrubber scrubber = new Scrubber(cfs, sstable, false, false); -Scrubber scrubber = new Scrubber(cfs, sstable, false, true); ++Scrubber scrubber = new Scrubber(cfs, sstable, false, false, true); try { scrubber.scrub(); @@@ -191,9 -190,10 +195,9 @@@ catch (IOError err) {} // with skipCorrupted == true, the corrupt row will be skipped - scrubber = new Scrubber(cfs, sstable, true, false); -scrubber = new Scrubber(cfs, sstable, true, true); ++scrubber = new Scrubber(cfs, sstable, true, false, true); scrubber.scrub(); scrubber.close(); -cfs.replaceCompactedSSTables(Collections.singletonList(sstable), Collections.singletonList(scrubber.getNewSSTable()), OperationType.SCRUB); assertEquals(1, cfs.getSSTables().size()); // verify that we can read all of the rows, and there is now one less row @@@ -324,7 -324,7 +328,7 @@@ components.add(Component.TOC); SSTableReader sstable = SSTableReader.openNoValidation(desc, components, metadata); --Scrubber scrubber = new Scrubber(cfs, sstable, false, true); ++Scrubber scrubber = new Scrubber(cfs, sstable, false, true, true); scrubber.scrub(); cfs.loadNewSSTables(); @@@ -420,9 -420,19 +424,19 @@@ Keyspace keyspace = Keyspace.open(Keyspace1); ColumnFamilyStore cfs = keyspace.getColumnFamilyStore(test_compact_static_columns); -QueryProcessor.processInternal(INSERT INTO
[16/16] cassandra git commit: Merge branch 'cassandra-2.2' into trunk
Merge branch 'cassandra-2.2' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5c413ef7 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5c413ef7 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5c413ef7 Branch: refs/heads/trunk Commit: 5c413ef76062da188ff661064ef45a02b5e5beda Parents: bade0fb ff1a156 Author: Yuki Morishita yu...@apache.org Authored: Tue May 19 09:00:17 2015 -0500 Committer: Yuki Morishita yu...@apache.org Committed: Tue May 19 09:00:17 2015 -0500 -- CHANGES.txt | 1 + .../apache/cassandra/db/ColumnFamilyStore.java | 8 ++-- .../db/compaction/CompactionManager.java| 8 ++-- .../cassandra/db/compaction/Scrubber.java | 8 ++-- .../cassandra/service/StorageService.java | 7 +++- .../cassandra/service/StorageServiceMBean.java | 2 + .../org/apache/cassandra/tools/NodeProbe.java | 8 ++-- .../cassandra/tools/StandaloneScrubber.java | 6 ++- .../apache/cassandra/tools/nodetool/Scrub.java | 9 - .../unit/org/apache/cassandra/db/ScrubTest.java | 42 ++-- 10 files changed, 66 insertions(+), 33 deletions(-) --
[02/16] cassandra git commit: Add option to not validate atoms during scrub
Add option to not validate atoms during scrub patch by Jordan West; reviewed by yukim for CASSANDRA-9406 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/be9eff57 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/be9eff57 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/be9eff57 Branch: refs/heads/cassandra-2.1 Commit: be9eff573342cd4993eb4fc8ee694dc885e8a8e4 Parents: 37d81b9 Author: Jordan West jorda...@gmail.com Authored: Tue May 19 08:26:27 2015 -0500 Committer: Yuki Morishita yu...@apache.org Committed: Tue May 19 08:26:27 2015 -0500 -- CHANGES.txt | 1 + .../apache/cassandra/db/ColumnFamilyStore.java | 4 +-- .../db/compaction/CompactionManager.java| 12 +++ .../cassandra/db/compaction/Scrubber.java | 12 --- .../cassandra/service/StorageService.java | 7 +++- .../cassandra/service/StorageServiceMBean.java | 2 ++ .../org/apache/cassandra/tools/NodeCmd.java | 5 ++- .../org/apache/cassandra/tools/NodeProbe.java | 4 +-- .../cassandra/tools/StandaloneScrubber.java | 6 +++- .../unit/org/apache/cassandra/db/ScrubTest.java | 37 +--- 10 files changed, 60 insertions(+), 30 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/be9eff57/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index a9d04d6..cf124b4 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -3,6 +3,7 @@ * Clone SliceQueryFilter in AbstractReadCommand implementations (CASSANDRA-8940) * Push correct protocol notification for DROP INDEX (CASSANDRA-9310) * token-generator - generated tokens too long (CASSANDRA-9300) + * Add option not to validate atoms during scrub (CASSANDRA-9406) 2.0.15: http://git-wip-us.apache.org/repos/asf/cassandra/blob/be9eff57/src/java/org/apache/cassandra/db/ColumnFamilyStore.java -- diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java index 5ea1287..eec4044 100644 --- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java +++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java @@ -1122,12 +1122,12 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean CompactionManager.instance.performCleanup(ColumnFamilyStore.this, renewer); } -public void scrub(boolean disableSnapshot, boolean skipCorrupted) throws ExecutionException, InterruptedException +public void scrub(boolean disableSnapshot, boolean skipCorrupted, boolean checkData) throws ExecutionException, InterruptedException { // skip snapshot creation during scrub, SEE JIRA 5891 if(!disableSnapshot) snapshotWithoutFlush(pre-scrub- + System.currentTimeMillis()); -CompactionManager.instance.performScrub(ColumnFamilyStore.this, skipCorrupted); +CompactionManager.instance.performScrub(ColumnFamilyStore.this, skipCorrupted, checkData); } public void sstablesRewrite(boolean excludeCurrentVersion) throws ExecutionException, InterruptedException http://git-wip-us.apache.org/repos/asf/cassandra/blob/be9eff57/src/java/org/apache/cassandra/db/compaction/CompactionManager.java -- diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java index 0978ae6..207b90d 100644 --- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java +++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java @@ -234,13 +234,13 @@ public class CompactionManager implements CompactionManagerMBean executor.submit(runnable).get(); } -public void performScrub(ColumnFamilyStore cfStore, final boolean skipCorrupted) throws InterruptedException, ExecutionException +public void performScrub(ColumnFamilyStore cfStore, final boolean skipCorrupted, final boolean checkData) throws InterruptedException, ExecutionException { performAllSSTableOperation(cfStore, new AllSSTablesOperation() { public void perform(ColumnFamilyStore store, IterableSSTableReader sstables) throws IOException { -doScrub(store, sstables, skipCorrupted); +doScrub(store, sstables, skipCorrupted, checkData); } }); } @@ -432,16 +432,16 @@ public class CompactionManager implements CompactionManagerMBean * * @throws IOException */ -private void doScrub(ColumnFamilyStore cfs, IterableSSTableReader sstables, boolean
[13/16] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
Merge branch 'cassandra-2.0' into cassandra-2.1 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d693ca12 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d693ca12 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d693ca12 Branch: refs/heads/trunk Commit: d693ca12c76c2651df1769e137a94b954174e061 Parents: adb8831 be9eff5 Author: Yuki Morishita yu...@apache.org Authored: Tue May 19 08:50:28 2015 -0500 Committer: Yuki Morishita yu...@apache.org Committed: Tue May 19 08:50:28 2015 -0500 -- CHANGES.txt | 1 + .../apache/cassandra/db/ColumnFamilyStore.java | 4 +-- .../db/compaction/CompactionManager.java| 8 ++--- .../cassandra/db/compaction/Scrubber.java | 12 --- .../cassandra/service/StorageService.java | 7 +++- .../cassandra/service/StorageServiceMBean.java | 2 ++ .../org/apache/cassandra/tools/NodeProbe.java | 8 ++--- .../org/apache/cassandra/tools/NodeTool.java| 7 +++- .../cassandra/tools/StandaloneScrubber.java | 6 +++- .../unit/org/apache/cassandra/db/ScrubTest.java | 38 +--- 10 files changed, 63 insertions(+), 30 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/d693ca12/CHANGES.txt -- diff --cc CHANGES.txt index 198935b,cf124b4..6fc1c9c --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -30,6 -3,10 +30,7 @@@ Merged from 2.0 * Clone SliceQueryFilter in AbstractReadCommand implementations (CASSANDRA-8940) * Push correct protocol notification for DROP INDEX (CASSANDRA-9310) * token-generator - generated tokens too long (CASSANDRA-9300) + * Add option not to validate atoms during scrub (CASSANDRA-9406) - - -2.0.15: * Fix counting of tombstones for TombstoneOverwhelmingException (CASSANDRA-9299) * Fix ReconnectableSnitch reconnecting to peers during upgrade (CASSANDRA-6702) * Include keyspace and table name in error log for collections over the size http://git-wip-us.apache.org/repos/asf/cassandra/blob/d693ca12/src/java/org/apache/cassandra/db/ColumnFamilyStore.java -- diff --cc src/java/org/apache/cassandra/db/ColumnFamilyStore.java index bdc2d8b,eec4044..0951c01 --- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java +++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java @@@ -1394,22 -1117,22 +1394,22 @@@ public class ColumnFamilyStore implemen return maxFile; } -public void forceCleanup(CounterId.OneShotRenewer renewer) throws ExecutionException, InterruptedException +public CompactionManager.AllSSTableOpStatus forceCleanup() throws ExecutionException, InterruptedException { -CompactionManager.instance.performCleanup(ColumnFamilyStore.this, renewer); +return CompactionManager.instance.performCleanup(ColumnFamilyStore.this); } - public CompactionManager.AllSSTableOpStatus scrub(boolean disableSnapshot, boolean skipCorrupted) throws ExecutionException, InterruptedException -public void scrub(boolean disableSnapshot, boolean skipCorrupted, boolean checkData) throws ExecutionException, InterruptedException ++public CompactionManager.AllSSTableOpStatus scrub(boolean disableSnapshot, boolean skipCorrupted, boolean checkData) throws ExecutionException, InterruptedException { // skip snapshot creation during scrub, SEE JIRA 5891 if(!disableSnapshot) snapshotWithoutFlush(pre-scrub- + System.currentTimeMillis()); - return CompactionManager.instance.performScrub(ColumnFamilyStore.this, skipCorrupted); -CompactionManager.instance.performScrub(ColumnFamilyStore.this, skipCorrupted, checkData); ++return CompactionManager.instance.performScrub(ColumnFamilyStore.this, skipCorrupted, checkData); } -public void sstablesRewrite(boolean excludeCurrentVersion) throws ExecutionException, InterruptedException +public CompactionManager.AllSSTableOpStatus sstablesRewrite(boolean excludeCurrentVersion) throws ExecutionException, InterruptedException { - CompactionManager.instance.performSSTableRewrite(ColumnFamilyStore.this, excludeCurrentVersion); +return CompactionManager.instance.performSSTableRewrite(ColumnFamilyStore.this, excludeCurrentVersion); } public void markObsolete(CollectionSSTableReader sstables, OperationType compactionType) http://git-wip-us.apache.org/repos/asf/cassandra/blob/d693ca12/src/java/org/apache/cassandra/db/compaction/CompactionManager.java -- diff --cc
[jira] [Commented] (CASSANDRA-9407) SSTableRewriterTest is hard failing after Fix canonical view returning early opened SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-9407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14550494#comment-14550494 ] Benedict commented on CASSANDRA-9407: - bq. Separate tickets kind of maybe aid in searchability and tracking. I am fine with reopening instead. I think the opposite is true, but I'm not wed to any approach. JIRA is messy, and slow. The less mess we need to filter through the better in my book. Historically we have generally reopened unless the problem made it into a release. Hopefully before long that will in itself make this point a bit moot, so probably not worth agonizing over. SSTableRewriterTest is hard failing after Fix canonical view returning early opened SSTables -- Key: CASSANDRA-9407 URL: https://issues.apache.org/jira/browse/CASSANDRA-9407 Project: Cassandra Issue Type: Test Reporter: Ariel Weisberg Assignee: Benedict Fix For: 2.2.0 rc1 When I bisected it seems to start failing after https://github.com/apache/cassandra/commit/7d6a60ddd0e156f6028c05d97fa9179f86fe02ee -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[1/4] cassandra git commit: Improve javadoc around trigger execution
Repository: cassandra Updated Branches: refs/heads/trunk 3a6e9c91b - f7d15d6af Improve javadoc around trigger execution Patch by Sam Tunnicliffe; reviewed by brandonwilliams for CASSANDRA-9334 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/37d81b9d Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/37d81b9d Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/37d81b9d Branch: refs/heads/trunk Commit: 37d81b9dcb08dd8c16a560591f46481b5d316841 Parents: 0b92967 Author: Sam Tunnicliffe s...@beobal.com Authored: Tue May 12 11:49:03 2015 +0100 Committer: Sam Tunnicliffe s...@beobal.com Committed: Tue May 19 09:39:56 2015 +0100 -- .../org/apache/cassandra/triggers/ITrigger.java | 4 +-- .../cassandra/triggers/TriggerExecutor.java | 32 2 files changed, 34 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/37d81b9d/src/java/org/apache/cassandra/triggers/ITrigger.java -- diff --git a/src/java/org/apache/cassandra/triggers/ITrigger.java b/src/java/org/apache/cassandra/triggers/ITrigger.java index 15ed7ba..fe2a9e6 100644 --- a/src/java/org/apache/cassandra/triggers/ITrigger.java +++ b/src/java/org/apache/cassandra/triggers/ITrigger.java @@ -36,7 +36,7 @@ import org.apache.cassandra.db.RowMutation; * 2) ITrigger implementation can be instantiated multiple times during the server life time. * (Depends on the number of times trigger folder is updated.)br * 3) ITrigger implementation should be state-less (avoid dependency on instance variables).br - * + * * brbThe API is still beta and can change./b */ public interface ITrigger @@ -46,7 +46,7 @@ public interface ITrigger * * @param key - Row Key for the update. * @param update - Update received for the CF - * @return modifications to be applied, null if no action to be performed. + * @return additional modifications to be applied along with the supplied update */ public CollectionRowMutation augment(ByteBuffer key, ColumnFamily update); } http://git-wip-us.apache.org/repos/asf/cassandra/blob/37d81b9d/src/java/org/apache/cassandra/triggers/TriggerExecutor.java -- diff --git a/src/java/org/apache/cassandra/triggers/TriggerExecutor.java b/src/java/org/apache/cassandra/triggers/TriggerExecutor.java index 988c6a7..bae0da6 100644 --- a/src/java/org/apache/cassandra/triggers/TriggerExecutor.java +++ b/src/java/org/apache/cassandra/triggers/TriggerExecutor.java @@ -64,6 +64,23 @@ public class TriggerExecutor cachedTriggers.clear(); } +/** + * Augment a partition update by executing triggers to generate an intermediate + * set of mutations, then merging the ColumnFamily from each mutation with those + * supplied. This is called from @{link org.apache.cassandra.service.StorageProxy#cas} + * which is scoped for a single partition. For that reason, any mutations generated + * by triggers are checked to ensure that they are for the same table and partition + * key as the primary update; if not, InvalidRequestException is thrown. If no + * additional mutations are generated, the original updates are returned unmodified. + * + * @param key partition key for the update + * @param updates partition update to be applied, contains the merge of the original + *update and any generated mutations + * @return the final update to be applied, the original update merged with any + * additional mutations generated by configured triggers + * @throws InvalidRequestException if any mutation generated by a trigger does not + * apply to the exact same partition as the initial update + */ public ColumnFamily execute(ByteBuffer key, ColumnFamily updates) throws InvalidRequestException { ListRowMutation intermediate = executeInternal(key, updates); @@ -79,6 +96,21 @@ public class TriggerExecutor return updates; } +/** + * Takes a collection of mutations and possibly augments it by adding extra mutations + * generated by configured triggers. If no additional mutations are created + * this returns null, signalling to the caller that only the initial set of + * mutations should be applied. If additional mutations iare/i generated, + * the total set (i.e. the original plus the additional mutations) are applied + * together in a logged batch. Should this not be possible because the initial + * mutations contain counter updates, InvalidRequestException is thrown. + * + * @param mutations initial collection of mutations + *
cassandra git commit: Fix 2.2 CHANGES.txt
Repository: cassandra Updated Branches: refs/heads/cassandra-2.2 889c1872e - 678291a75 Fix 2.2 CHANGES.txt Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/678291a7 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/678291a7 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/678291a7 Branch: refs/heads/cassandra-2.2 Commit: 678291a750e9a0c412957fbb87ce9aeb98e7e673 Parents: 889c187 Author: Aleksey Yeschenko alek...@apache.org Authored: Tue May 19 16:48:56 2015 +0300 Committer: Aleksey Yeschenko alek...@apache.org Committed: Tue May 19 16:48:56 2015 +0300 -- CHANGES.txt | 6 +- 1 file changed, 5 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/678291a7/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 7885e0e..2ffd6ed 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,3 +1,8 @@ +2.2 +Merged from 2.1: + * Use configured gcgs in anticompaction (CASSANDRA-9397) + + 2.2.0-beta1 * Introduce Transactional API for internal state changes (CASSANDRA-8984) * Add a flag in cassandra.yaml to enable UDFs (CASSANDRA-9404) @@ -114,7 +119,6 @@ 2.1.6 - * Use configured gcgs in anticompaction (CASSANDRA-9397) * Warn on misuse of unlogged batches (CASSANDRA-9282) * Failure detector detects and ignores local pauses (CASSANDRA-9183) * Add utility class to support for rate limiting a given log statement (CASSANDRA-9029)
[jira] [Updated] (CASSANDRA-9397) Wrong gc_grace_seconds used in anticompaction
[ https://issues.apache.org/jira/browse/CASSANDRA-9397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Yeschenko updated CASSANDRA-9397: - Fix Version/s: (was: 2.2.0 beta 1) 2.2.0 rc1 Wrong gc_grace_seconds used in anticompaction - Key: CASSANDRA-9397 URL: https://issues.apache.org/jira/browse/CASSANDRA-9397 Project: Cassandra Issue Type: Improvement Reporter: Marcus Eriksson Assignee: Marcus Eriksson Fix For: 2.1.6, 3.0 beta 1, 2.2.0 rc1 Attachments: 0001-fix.patch looks like we use CFMetaData.DEFAULT_GC_GRACE_SECONDS instead of the configured one during anticompaction attached patch fixes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9407) SSTableRewriterTest is hard failing after Fix canonical view returning early opened SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-9407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14550515#comment-14550515 ] Ariel Weisberg commented on CASSANDRA-9407: --- Thinking on it more. If you use JIRA and you are all in on the agile features you don't really open tickets from previous iterations and that is what the habit stems from. The history I am thinking of is the history of what was completed in previous iterations. The way we use JIRA I think the policy of reopening when something hasn't shipped makes more sense. SSTableRewriterTest is hard failing after Fix canonical view returning early opened SSTables -- Key: CASSANDRA-9407 URL: https://issues.apache.org/jira/browse/CASSANDRA-9407 Project: Cassandra Issue Type: Test Reporter: Ariel Weisberg Assignee: Benedict Fix For: 2.2.0 rc1 When I bisected it seems to start failing after https://github.com/apache/cassandra/commit/7d6a60ddd0e156f6028c05d97fa9179f86fe02ee -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9422) SystemKeyspaceTest fails with non-release version string
[ https://issues.apache.org/jira/browse/CASSANDRA-9422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] T Jake Luciani updated CASSANDRA-9422: -- Fix Version/s: (was: 2.2.0 beta 1) 2.2.0 rc1 SystemKeyspaceTest fails with non-release version string Key: CASSANDRA-9422 URL: https://issues.apache.org/jira/browse/CASSANDRA-9422 Project: Cassandra Issue Type: Bug Reporter: Sam Tunnicliffe Assignee: Sam Tunnicliffe Fix For: 2.2.0 rc1 {{SystemKeyspaceTest#snapshotSystemKeyspaceIfUpgrading}} is currently failing on the 2.2 branch because the build version {{2.2.0-beta1-SNAPSHOT}} cannot be parsed as a {{SemanticVersion}}. This is only a problem with the test, as the code under test doesn't actually use {{SemanticVersion}}, it just checks whether the old new version strings are different. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9369) HSHA dtest for closing connections is failing on trunk
[ https://issues.apache.org/jira/browse/CASSANDRA-9369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14550458#comment-14550458 ] Michael Shuler commented on CASSANDRA-9369: --- Why are we disabling tests that happen to fail? Disabling failing tests from the dtest suite only hides the failure from the people that want to see *all* failures, disrupts viewing of test history in CI, and does not seem to have a clear positive advantage. In my opinion, there are only 2 reasons to disable a test: 1) The test completely hangs the test server. 2) A feature test is committed before the feature itself has been committed to Cassandra. HSHA dtest for closing connections is failing on trunk -- Key: CASSANDRA-9369 URL: https://issues.apache.org/jira/browse/CASSANDRA-9369 Project: Cassandra Issue Type: Test Components: Tests Reporter: Tyler Hobbs Assignee: Ariel Weisberg Fix For: 2.2.x The {{thrift_hsha_test.ThriftHSHATest.test_closing_connections}} dtest is failing against trunk (but not against 2.1). Here's an example failure: {noformat} Error Message There are non-closed connections: COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java28202 automaton7u IPv4 60767 0t0 TCP localhost:58436-localhost:9160 (CLOSE_WAIT) begin captured logging dtest: DEBUG: cluster ccm directory: /tmp/dtest-U0pP7H dtest: DEBUG: Creating connection pools.. dtest: DEBUG: Disabling/Enabling thrift iteration #0 dtest: DEBUG: Closing connections from the client side.. pycassa.pool: INFO: Pool 13972199784 was disposed pycassa.pool: INFO: Pool 139721997714512 was disposed pycassa.pool: INFO: Pool 139721997749904 was disposed - end captured logging - Stacktrace File /usr/lib/python2.7/unittest/case.py, line 331, in run testMethod() File /home/automaton/cassandra-dtest/thrift_hsha_test.py, line 59, in test_closing_connections self.assertEqual(len(lines), 0, There are non-closed connections: %s % stdout) File /usr/lib/python2.7/unittest/case.py, line 515, in assertEqual assertion_func(first, second, msg=msg) File /usr/lib/python2.7/unittest/case.py, line 508, in _baseAssertEqual raise self.failureException(msg) 'There are non-closed connections: COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME\njava28202 automaton7u IPv4 60767 0t0 TCP localhost:58436-localhost:9160 (CLOSE_WAIT)\n\n begin captured logging \ndtest: DEBUG: cluster ccm directory: /tmp/dtest-U0pP7H\ndtest: DEBUG: Creating connection pools..\ndtest: DEBUG: Disabling/Enabling thrift iteration #0\ndtest: DEBUG: Closing connections from the client side..\npycassa.pool: INFO: Pool 13972199784 was disposed\npycassa.pool: INFO: Pool 139721997714512 was disposed\npycassa.pool: INFO: Pool 139721997749904 was disposed\n- end captured logging -' {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8163) Re-introduce DESCRIBE permission
[ https://issues.apache.org/jira/browse/CASSANDRA-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14550333#comment-14550333 ] Aleksey Yeschenko edited comment on CASSANDRA-8163 at 5/19/15 1:58 PM: --- -Can we, for 2.2, just introduce the permission to the enum, and have it be a no-op?- -I don't want us to break the API again in 3.0.- Scratch that, we already have it in the enum, for roles. Sorry for the confusion. was (Author: iamaleksey): Can we, for 2.2, just introduce the permission to the enum, and have it be a no-op? I don't want us to break the API again in 3.0. Re-introduce DESCRIBE permission Key: CASSANDRA-8163 URL: https://issues.apache.org/jira/browse/CASSANDRA-8163 Project: Cassandra Issue Type: Improvement Reporter: Vishy Kasar Priority: Minor Fix For: 3.x We have a cluster like this: project1_keyspace table101 table102 project2_keyspace table201 table202 We have set up following users and grants: project1_user has all access to project1_keyspace project2_user has all access to project2_keyspace However project1_user can still do a 'describe schema' and get the schema for project2_keyspace as well. We do not want project1_user to have any knowledge for project2 in any way (cqlsh/java-driver etc) . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Git Push Summary
Repository: cassandra Updated Tags: refs/tags/cassandra-2.2.0-beta1 [created] f7e42e94e
[2/3] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2
Merge branch 'cassandra-2.1' into cassandra-2.2 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e8766e53 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e8766e53 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e8766e53 Branch: refs/heads/trunk Commit: e8766e53d11645b41e25d904d5bdbdb2b1c0c9a9 Parents: ff1a156 7af9c6a Author: T Jake Luciani j...@apache.org Authored: Tue May 19 10:13:14 2015 -0400 Committer: T Jake Luciani j...@apache.org Committed: Tue May 19 10:13:14 2015 -0400 -- .../DebuggableScheduledThreadPoolExecutor.java | 9 ++ ...buggableScheduledThreadPoolExecutorTest.java | 115 +++ 2 files changed, 124 insertions(+) --
[1/3] cassandra git commit: cancel any rejected tasks on shutdown so callers who care can progress
Repository: cassandra Updated Branches: refs/heads/trunk 5c413ef76 - 77f0d99fa cancel any rejected tasks on shutdown so callers who care can progress patch by tjake; reviewed by Sergio Bossa for CASSANDRA-9417 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7af9c6ac Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7af9c6ac Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7af9c6ac Branch: refs/heads/trunk Commit: 7af9c6ac88278d34202fea922606905b326f7811 Parents: d693ca1 Author: T Jake Luciani j...@apache.org Authored: Mon May 18 15:18:24 2015 -0400 Committer: T Jake Luciani j...@apache.org Committed: Tue May 19 10:12:07 2015 -0400 -- .../DebuggableScheduledThreadPoolExecutor.java | 9 ++ ...buggableScheduledThreadPoolExecutorTest.java | 115 +++ 2 files changed, 124 insertions(+) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/7af9c6ac/src/java/org/apache/cassandra/concurrent/DebuggableScheduledThreadPoolExecutor.java -- diff --git a/src/java/org/apache/cassandra/concurrent/DebuggableScheduledThreadPoolExecutor.java b/src/java/org/apache/cassandra/concurrent/DebuggableScheduledThreadPoolExecutor.java index 7226db8..ea0715c 100644 --- a/src/java/org/apache/cassandra/concurrent/DebuggableScheduledThreadPoolExecutor.java +++ b/src/java/org/apache/cassandra/concurrent/DebuggableScheduledThreadPoolExecutor.java @@ -31,6 +31,11 @@ import org.apache.cassandra.utils.JVMStabilityInspector; * * DebuggableScheduledThreadPoolExecutor also catches exceptions during Task execution * so that they don't supress subsequent invocations of the task. + * + * Finally, there is a special rejected execution handler for tasks rejected during the shutdown hook. + * + * For fire and forget tasks (like ref tidy) we can safely ignore the exceptions. + * For any callers that care to know their task was rejected we cancel passed task. */ public class DebuggableScheduledThreadPoolExecutor extends ScheduledThreadPoolExecutor { @@ -45,6 +50,10 @@ public class DebuggableScheduledThreadPoolExecutor extends ScheduledThreadPoolEx if (!StorageService.instance.isInShutdownHook()) throw new RejectedExecutionException(ScheduledThreadPoolExecutor has shut down.); +//Give some notification to the caller the task isn't going to run +if (task instanceof Future) +((Future) task).cancel(false); + logger.debug(ScheduledThreadPoolExecutor has shut down as part of C* shutdown); } else http://git-wip-us.apache.org/repos/asf/cassandra/blob/7af9c6ac/test/unit/org/apache/cassandra/concurrent/DebuggableScheduledThreadPoolExecutorTest.java -- diff --git a/test/unit/org/apache/cassandra/concurrent/DebuggableScheduledThreadPoolExecutorTest.java b/test/unit/org/apache/cassandra/concurrent/DebuggableScheduledThreadPoolExecutorTest.java new file mode 100644 index 000..46b2764 --- /dev/null +++ b/test/unit/org/apache/cassandra/concurrent/DebuggableScheduledThreadPoolExecutorTest.java @@ -0,0 +1,115 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * License); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an AS IS BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.cassandra.concurrent; + +import java.io.IOException; +import java.util.concurrent.CancellationException; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.ScheduledFuture; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.TimeoutException; +import java.util.concurrent.atomic.AtomicInteger; + +import org.junit.BeforeClass; +import org.junit.Test; + +import junit.framework.Assert; +import org.apache.cassandra.service.EmbeddedCassandraService; +import org.apache.cassandra.service.StorageService; + +public class DebuggableScheduledThreadPoolExecutorTest +{ + +static
[3/3] cassandra git commit: Merge branch 'cassandra-2.2' into trunk
Merge branch 'cassandra-2.2' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/77f0d99f Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/77f0d99f Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/77f0d99f Branch: refs/heads/trunk Commit: 77f0d99facbfdd744fe289ec5c1edae3abf7f4d9 Parents: 5c413ef e8766e5 Author: T Jake Luciani j...@apache.org Authored: Tue May 19 10:13:54 2015 -0400 Committer: T Jake Luciani j...@apache.org Committed: Tue May 19 10:13:54 2015 -0400 -- .../DebuggableScheduledThreadPoolExecutor.java | 9 ++ ...buggableScheduledThreadPoolExecutorTest.java | 115 +++ 2 files changed, 124 insertions(+) --
[jira] [Commented] (CASSANDRA-9369) HSHA dtest for closing connections is failing on trunk
[ https://issues.apache.org/jira/browse/CASSANDRA-9369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14550496#comment-14550496 ] Ariel Weisberg commented on CASSANDRA-9369: --- If we were caught up and all tests were passing then yes disabling failing tests would be a no-no. Right now when we are far behind and trying to catch up it makes sense to disable tests we know aren't passing. See CASSANDRA-8997 and CASSANDRA-9002 for why (and feel free to comment and disagree). Discussing this might be good retrospective fodder. I want to bring up the fact that it's been a few months and utests and dtests still aren't passing, and people are still doing feature work. What's the point of doing feature work for a system that you can't ship anyways because the tests don't pass? I wouldn't be surprised if the tests don't pass by the time we want to ship 2.2. By branching 2.2 we increased test load an complexity because now we have two places to merge to and two sets of tests to monitor. I don't agree with the thinking that addressing test issues delays feature work. It's already delayed you are just in denial. HSHA dtest for closing connections is failing on trunk -- Key: CASSANDRA-9369 URL: https://issues.apache.org/jira/browse/CASSANDRA-9369 Project: Cassandra Issue Type: Test Components: Tests Reporter: Tyler Hobbs Assignee: Ariel Weisberg Fix For: 2.2.x The {{thrift_hsha_test.ThriftHSHATest.test_closing_connections}} dtest is failing against trunk (but not against 2.1). Here's an example failure: {noformat} Error Message There are non-closed connections: COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java28202 automaton7u IPv4 60767 0t0 TCP localhost:58436-localhost:9160 (CLOSE_WAIT) begin captured logging dtest: DEBUG: cluster ccm directory: /tmp/dtest-U0pP7H dtest: DEBUG: Creating connection pools.. dtest: DEBUG: Disabling/Enabling thrift iteration #0 dtest: DEBUG: Closing connections from the client side.. pycassa.pool: INFO: Pool 13972199784 was disposed pycassa.pool: INFO: Pool 139721997714512 was disposed pycassa.pool: INFO: Pool 139721997749904 was disposed - end captured logging - Stacktrace File /usr/lib/python2.7/unittest/case.py, line 331, in run testMethod() File /home/automaton/cassandra-dtest/thrift_hsha_test.py, line 59, in test_closing_connections self.assertEqual(len(lines), 0, There are non-closed connections: %s % stdout) File /usr/lib/python2.7/unittest/case.py, line 515, in assertEqual assertion_func(first, second, msg=msg) File /usr/lib/python2.7/unittest/case.py, line 508, in _baseAssertEqual raise self.failureException(msg) 'There are non-closed connections: COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME\njava28202 automaton7u IPv4 60767 0t0 TCP localhost:58436-localhost:9160 (CLOSE_WAIT)\n\n begin captured logging \ndtest: DEBUG: cluster ccm directory: /tmp/dtest-U0pP7H\ndtest: DEBUG: Creating connection pools..\ndtest: DEBUG: Disabling/Enabling thrift iteration #0\ndtest: DEBUG: Closing connections from the client side..\npycassa.pool: INFO: Pool 13972199784 was disposed\npycassa.pool: INFO: Pool 139721997714512 was disposed\npycassa.pool: INFO: Pool 139721997749904 was disposed\n- end captured logging -' {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9422) SystemKeyspaceTest fails with non-release version string
[ https://issues.apache.org/jira/browse/CASSANDRA-9422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14550346#comment-14550346 ] Sam Tunnicliffe commented on CASSANDRA-9422: Test fix pushed [here|https://github.com/beobal/cassandra/tree/9422] SystemKeyspaceTest fails with non-release version string Key: CASSANDRA-9422 URL: https://issues.apache.org/jira/browse/CASSANDRA-9422 Project: Cassandra Issue Type: Bug Reporter: Sam Tunnicliffe Assignee: Sam Tunnicliffe Fix For: 2.2 beta 1 {{SystemKeyspaceTest#snapshotSystemKeyspaceIfUpgrading}} is currently failing on the 2.2 branch because the build version {{2.2.0-beta1-SNAPSHOT}} cannot be parsed as a {{SemanticVersion}}. This is only a problem with the test, as the code under test doesn't actually use {{SemanticVersion}}, it just checks whether the old new version strings are different. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Git Push Summary
Repository: cassandra Updated Tags: refs/tags/2.2.0-beta1-tentative [deleted] 1735249eb
[3/4] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2
Merge branch 'cassandra-2.1' into cassandra-2.2 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/889c1872 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/889c1872 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/889c1872 Branch: refs/heads/trunk Commit: 889c1872e036362636ef0be5419006f19c35a796 Parents: 86c9c00 adb8831 Author: Aleksey Yeschenko alek...@apache.org Authored: Tue May 19 16:43:11 2015 +0300 Committer: Aleksey Yeschenko alek...@apache.org Committed: Tue May 19 16:43:11 2015 +0300 -- --
[3/3] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2
Merge branch 'cassandra-2.1' into cassandra-2.2 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/889c1872 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/889c1872 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/889c1872 Branch: refs/heads/cassandra-2.2 Commit: 889c1872e036362636ef0be5419006f19c35a796 Parents: 86c9c00 adb8831 Author: Aleksey Yeschenko alek...@apache.org Authored: Tue May 19 16:43:11 2015 +0300 Committer: Aleksey Yeschenko alek...@apache.org Committed: Tue May 19 16:43:11 2015 +0300 -- --
[1/3] cassandra git commit: Improve javadoc around trigger execution
Repository: cassandra Updated Branches: refs/heads/cassandra-2.2 86c9c00e9 - 889c1872e Improve javadoc around trigger execution Patch by Sam Tunnicliffe; reviewed by brandonwilliams for CASSANDRA-9334 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/37d81b9d Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/37d81b9d Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/37d81b9d Branch: refs/heads/cassandra-2.2 Commit: 37d81b9dcb08dd8c16a560591f46481b5d316841 Parents: 0b92967 Author: Sam Tunnicliffe s...@beobal.com Authored: Tue May 12 11:49:03 2015 +0100 Committer: Sam Tunnicliffe s...@beobal.com Committed: Tue May 19 09:39:56 2015 +0100 -- .../org/apache/cassandra/triggers/ITrigger.java | 4 +-- .../cassandra/triggers/TriggerExecutor.java | 32 2 files changed, 34 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/37d81b9d/src/java/org/apache/cassandra/triggers/ITrigger.java -- diff --git a/src/java/org/apache/cassandra/triggers/ITrigger.java b/src/java/org/apache/cassandra/triggers/ITrigger.java index 15ed7ba..fe2a9e6 100644 --- a/src/java/org/apache/cassandra/triggers/ITrigger.java +++ b/src/java/org/apache/cassandra/triggers/ITrigger.java @@ -36,7 +36,7 @@ import org.apache.cassandra.db.RowMutation; * 2) ITrigger implementation can be instantiated multiple times during the server life time. * (Depends on the number of times trigger folder is updated.)br * 3) ITrigger implementation should be state-less (avoid dependency on instance variables).br - * + * * brbThe API is still beta and can change./b */ public interface ITrigger @@ -46,7 +46,7 @@ public interface ITrigger * * @param key - Row Key for the update. * @param update - Update received for the CF - * @return modifications to be applied, null if no action to be performed. + * @return additional modifications to be applied along with the supplied update */ public CollectionRowMutation augment(ByteBuffer key, ColumnFamily update); } http://git-wip-us.apache.org/repos/asf/cassandra/blob/37d81b9d/src/java/org/apache/cassandra/triggers/TriggerExecutor.java -- diff --git a/src/java/org/apache/cassandra/triggers/TriggerExecutor.java b/src/java/org/apache/cassandra/triggers/TriggerExecutor.java index 988c6a7..bae0da6 100644 --- a/src/java/org/apache/cassandra/triggers/TriggerExecutor.java +++ b/src/java/org/apache/cassandra/triggers/TriggerExecutor.java @@ -64,6 +64,23 @@ public class TriggerExecutor cachedTriggers.clear(); } +/** + * Augment a partition update by executing triggers to generate an intermediate + * set of mutations, then merging the ColumnFamily from each mutation with those + * supplied. This is called from @{link org.apache.cassandra.service.StorageProxy#cas} + * which is scoped for a single partition. For that reason, any mutations generated + * by triggers are checked to ensure that they are for the same table and partition + * key as the primary update; if not, InvalidRequestException is thrown. If no + * additional mutations are generated, the original updates are returned unmodified. + * + * @param key partition key for the update + * @param updates partition update to be applied, contains the merge of the original + *update and any generated mutations + * @return the final update to be applied, the original update merged with any + * additional mutations generated by configured triggers + * @throws InvalidRequestException if any mutation generated by a trigger does not + * apply to the exact same partition as the initial update + */ public ColumnFamily execute(ByteBuffer key, ColumnFamily updates) throws InvalidRequestException { ListRowMutation intermediate = executeInternal(key, updates); @@ -79,6 +96,21 @@ public class TriggerExecutor return updates; } +/** + * Takes a collection of mutations and possibly augments it by adding extra mutations + * generated by configured triggers. If no additional mutations are created + * this returns null, signalling to the caller that only the initial set of + * mutations should be applied. If additional mutations iare/i generated, + * the total set (i.e. the original plus the additional mutations) are applied + * together in a logged batch. Should this not be possible because the initial + * mutations contain counter updates, InvalidRequestException is thrown. + * + * @param mutations initial collection of
[2/4] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
Merge branch 'cassandra-2.0' into cassandra-2.1 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/adb88316 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/adb88316 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/adb88316 Branch: refs/heads/trunk Commit: adb88316ac7ec26be96751062615c3ad0c22aaff Parents: 116de2b 37d81b9 Author: Aleksey Yeschenko alek...@apache.org Authored: Tue May 19 16:41:41 2015 +0300 Committer: Aleksey Yeschenko alek...@apache.org Committed: Tue May 19 16:41:41 2015 +0300 -- --
[06/16] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
http://git-wip-us.apache.org/repos/asf/cassandra/blob/d693ca12/src/java/org/apache/cassandra/tools/NodeTool.java -- diff --cc src/java/org/apache/cassandra/tools/NodeTool.java index 2c913e0,000..5156479 mode 100644,00..100644 --- a/src/java/org/apache/cassandra/tools/NodeTool.java +++ b/src/java/org/apache/cassandra/tools/NodeTool.java @@@ -1,2685 -1,0 +1,2690 @@@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * License); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an AS IS BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.cassandra.tools; + +import static com.google.common.base.Preconditions.checkArgument; +import static com.google.common.base.Preconditions.checkState; +import static com.google.common.base.Throwables.getStackTraceAsString; +import static com.google.common.collect.Iterables.toArray; +import static com.google.common.collect.Lists.newArrayList; +import static java.lang.Integer.parseInt; +import static java.lang.String.format; +import static org.apache.commons.lang3.ArrayUtils.EMPTY_STRING_ARRAY; +import static org.apache.commons.lang3.StringUtils.EMPTY; +import static org.apache.commons.lang3.StringUtils.isEmpty; +import static org.apache.commons.lang3.StringUtils.isNotEmpty; +import static org.apache.commons.lang3.StringUtils.join; +import io.airlift.command.Arguments; +import io.airlift.command.Cli; +import io.airlift.command.Command; +import io.airlift.command.Help; +import io.airlift.command.Option; +import io.airlift.command.OptionType; +import io.airlift.command.ParseArgumentsMissingException; +import io.airlift.command.ParseArgumentsUnexpectedException; +import io.airlift.command.ParseCommandMissingException; +import io.airlift.command.ParseCommandUnrecognizedException; +import io.airlift.command.ParseOptionConversionException; +import io.airlift.command.ParseOptionMissingException; +import io.airlift.command.ParseOptionMissingValueException; + +import java.io.Console; +import java.io.File; +import java.io.FileNotFoundException; +import java.io.FileWriter; +import java.io.IOError; +import java.io.IOException; +import java.lang.management.MemoryUsage; +import java.net.InetAddress; +import java.net.UnknownHostException; +import java.text.DecimalFormat; +import java.text.SimpleDateFormat; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.Comparator; +import java.util.Date; +import java.util.HashMap; +import java.util.Iterator; +import java.util.List; +import java.util.Map; +import java.util.Map.Entry; +import java.util.Scanner; +import java.util.Set; +import java.util.concurrent.ExecutionException; + +import javax.management.InstanceNotFoundException; +import javax.management.openmbean.CompositeData; +import javax.management.openmbean.OpenDataException; +import javax.management.openmbean.TabularData; +import javax.management.openmbean.TabularDataSupport; + +import org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutorMBean; +import org.apache.cassandra.db.ColumnFamilyStoreMBean; +import org.apache.cassandra.db.Keyspace; +import org.apache.cassandra.db.compaction.CompactionManagerMBean; +import org.apache.cassandra.db.compaction.OperationType; +import org.apache.cassandra.io.util.FileUtils; +import org.apache.cassandra.locator.EndpointSnitchInfoMBean; +import org.apache.cassandra.metrics.ColumnFamilyMetrics.Sampler; +import org.apache.cassandra.net.MessagingServiceMBean; +import org.apache.cassandra.repair.RepairParallelism; +import org.apache.cassandra.service.CacheServiceMBean; +import org.apache.cassandra.service.StorageProxyMBean; +import org.apache.cassandra.streaming.ProgressInfo; +import org.apache.cassandra.streaming.SessionInfo; +import org.apache.cassandra.streaming.StreamState; +import org.apache.cassandra.utils.FBUtilities; +import org.apache.cassandra.utils.JVMStabilityInspector; +import org.apache.commons.lang3.ArrayUtils; + +import com.google.common.base.Joiner; +import com.google.common.base.Throwables; +import com.google.common.collect.ArrayListMultimap; +import com.google.common.collect.LinkedHashMultimap; +import com.google.common.collect.Lists; +import
[07/16] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
Merge branch 'cassandra-2.0' into cassandra-2.1 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d693ca12 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d693ca12 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d693ca12 Branch: refs/heads/cassandra-2.2 Commit: d693ca12c76c2651df1769e137a94b954174e061 Parents: adb8831 be9eff5 Author: Yuki Morishita yu...@apache.org Authored: Tue May 19 08:50:28 2015 -0500 Committer: Yuki Morishita yu...@apache.org Committed: Tue May 19 08:50:28 2015 -0500 -- CHANGES.txt | 1 + .../apache/cassandra/db/ColumnFamilyStore.java | 4 +-- .../db/compaction/CompactionManager.java| 8 ++--- .../cassandra/db/compaction/Scrubber.java | 12 --- .../cassandra/service/StorageService.java | 7 +++- .../cassandra/service/StorageServiceMBean.java | 2 ++ .../org/apache/cassandra/tools/NodeProbe.java | 8 ++--- .../org/apache/cassandra/tools/NodeTool.java| 7 +++- .../cassandra/tools/StandaloneScrubber.java | 6 +++- .../unit/org/apache/cassandra/db/ScrubTest.java | 38 +--- 10 files changed, 63 insertions(+), 30 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/d693ca12/CHANGES.txt -- diff --cc CHANGES.txt index 198935b,cf124b4..6fc1c9c --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -30,6 -3,10 +30,7 @@@ Merged from 2.0 * Clone SliceQueryFilter in AbstractReadCommand implementations (CASSANDRA-8940) * Push correct protocol notification for DROP INDEX (CASSANDRA-9310) * token-generator - generated tokens too long (CASSANDRA-9300) + * Add option not to validate atoms during scrub (CASSANDRA-9406) - - -2.0.15: * Fix counting of tombstones for TombstoneOverwhelmingException (CASSANDRA-9299) * Fix ReconnectableSnitch reconnecting to peers during upgrade (CASSANDRA-6702) * Include keyspace and table name in error log for collections over the size http://git-wip-us.apache.org/repos/asf/cassandra/blob/d693ca12/src/java/org/apache/cassandra/db/ColumnFamilyStore.java -- diff --cc src/java/org/apache/cassandra/db/ColumnFamilyStore.java index bdc2d8b,eec4044..0951c01 --- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java +++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java @@@ -1394,22 -1117,22 +1394,22 @@@ public class ColumnFamilyStore implemen return maxFile; } -public void forceCleanup(CounterId.OneShotRenewer renewer) throws ExecutionException, InterruptedException +public CompactionManager.AllSSTableOpStatus forceCleanup() throws ExecutionException, InterruptedException { -CompactionManager.instance.performCleanup(ColumnFamilyStore.this, renewer); +return CompactionManager.instance.performCleanup(ColumnFamilyStore.this); } - public CompactionManager.AllSSTableOpStatus scrub(boolean disableSnapshot, boolean skipCorrupted) throws ExecutionException, InterruptedException -public void scrub(boolean disableSnapshot, boolean skipCorrupted, boolean checkData) throws ExecutionException, InterruptedException ++public CompactionManager.AllSSTableOpStatus scrub(boolean disableSnapshot, boolean skipCorrupted, boolean checkData) throws ExecutionException, InterruptedException { // skip snapshot creation during scrub, SEE JIRA 5891 if(!disableSnapshot) snapshotWithoutFlush(pre-scrub- + System.currentTimeMillis()); - return CompactionManager.instance.performScrub(ColumnFamilyStore.this, skipCorrupted); -CompactionManager.instance.performScrub(ColumnFamilyStore.this, skipCorrupted, checkData); ++return CompactionManager.instance.performScrub(ColumnFamilyStore.this, skipCorrupted, checkData); } -public void sstablesRewrite(boolean excludeCurrentVersion) throws ExecutionException, InterruptedException +public CompactionManager.AllSSTableOpStatus sstablesRewrite(boolean excludeCurrentVersion) throws ExecutionException, InterruptedException { - CompactionManager.instance.performSSTableRewrite(ColumnFamilyStore.this, excludeCurrentVersion); +return CompactionManager.instance.performSSTableRewrite(ColumnFamilyStore.this, excludeCurrentVersion); } public void markObsolete(CollectionSSTableReader sstables, OperationType compactionType) http://git-wip-us.apache.org/repos/asf/cassandra/blob/d693ca12/src/java/org/apache/cassandra/db/compaction/CompactionManager.java -- diff --cc
[09/16] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
http://git-wip-us.apache.org/repos/asf/cassandra/blob/d693ca12/src/java/org/apache/cassandra/tools/NodeTool.java -- diff --cc src/java/org/apache/cassandra/tools/NodeTool.java index 2c913e0,000..5156479 mode 100644,00..100644 --- a/src/java/org/apache/cassandra/tools/NodeTool.java +++ b/src/java/org/apache/cassandra/tools/NodeTool.java @@@ -1,2685 -1,0 +1,2690 @@@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * License); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an AS IS BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.cassandra.tools; + +import static com.google.common.base.Preconditions.checkArgument; +import static com.google.common.base.Preconditions.checkState; +import static com.google.common.base.Throwables.getStackTraceAsString; +import static com.google.common.collect.Iterables.toArray; +import static com.google.common.collect.Lists.newArrayList; +import static java.lang.Integer.parseInt; +import static java.lang.String.format; +import static org.apache.commons.lang3.ArrayUtils.EMPTY_STRING_ARRAY; +import static org.apache.commons.lang3.StringUtils.EMPTY; +import static org.apache.commons.lang3.StringUtils.isEmpty; +import static org.apache.commons.lang3.StringUtils.isNotEmpty; +import static org.apache.commons.lang3.StringUtils.join; +import io.airlift.command.Arguments; +import io.airlift.command.Cli; +import io.airlift.command.Command; +import io.airlift.command.Help; +import io.airlift.command.Option; +import io.airlift.command.OptionType; +import io.airlift.command.ParseArgumentsMissingException; +import io.airlift.command.ParseArgumentsUnexpectedException; +import io.airlift.command.ParseCommandMissingException; +import io.airlift.command.ParseCommandUnrecognizedException; +import io.airlift.command.ParseOptionConversionException; +import io.airlift.command.ParseOptionMissingException; +import io.airlift.command.ParseOptionMissingValueException; + +import java.io.Console; +import java.io.File; +import java.io.FileNotFoundException; +import java.io.FileWriter; +import java.io.IOError; +import java.io.IOException; +import java.lang.management.MemoryUsage; +import java.net.InetAddress; +import java.net.UnknownHostException; +import java.text.DecimalFormat; +import java.text.SimpleDateFormat; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.Comparator; +import java.util.Date; +import java.util.HashMap; +import java.util.Iterator; +import java.util.List; +import java.util.Map; +import java.util.Map.Entry; +import java.util.Scanner; +import java.util.Set; +import java.util.concurrent.ExecutionException; + +import javax.management.InstanceNotFoundException; +import javax.management.openmbean.CompositeData; +import javax.management.openmbean.OpenDataException; +import javax.management.openmbean.TabularData; +import javax.management.openmbean.TabularDataSupport; + +import org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutorMBean; +import org.apache.cassandra.db.ColumnFamilyStoreMBean; +import org.apache.cassandra.db.Keyspace; +import org.apache.cassandra.db.compaction.CompactionManagerMBean; +import org.apache.cassandra.db.compaction.OperationType; +import org.apache.cassandra.io.util.FileUtils; +import org.apache.cassandra.locator.EndpointSnitchInfoMBean; +import org.apache.cassandra.metrics.ColumnFamilyMetrics.Sampler; +import org.apache.cassandra.net.MessagingServiceMBean; +import org.apache.cassandra.repair.RepairParallelism; +import org.apache.cassandra.service.CacheServiceMBean; +import org.apache.cassandra.service.StorageProxyMBean; +import org.apache.cassandra.streaming.ProgressInfo; +import org.apache.cassandra.streaming.SessionInfo; +import org.apache.cassandra.streaming.StreamState; +import org.apache.cassandra.utils.FBUtilities; +import org.apache.cassandra.utils.JVMStabilityInspector; +import org.apache.commons.lang3.ArrayUtils; + +import com.google.common.base.Joiner; +import com.google.common.base.Throwables; +import com.google.common.collect.ArrayListMultimap; +import com.google.common.collect.LinkedHashMultimap; +import com.google.common.collect.Lists; +import
[11/16] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
http://git-wip-us.apache.org/repos/asf/cassandra/blob/d693ca12/src/java/org/apache/cassandra/tools/StandaloneScrubber.java -- diff --cc src/java/org/apache/cassandra/tools/StandaloneScrubber.java index d420218,26768f3..459df98 --- a/src/java/org/apache/cassandra/tools/StandaloneScrubber.java +++ b/src/java/org/apache/cassandra/tools/StandaloneScrubber.java @@@ -109,7 -121,7 +110,7 @@@ public class StandaloneScrubbe { try { - Scrubber scrubber = new Scrubber(cfs, sstable, options.skipCorrupted, handler, true); -Scrubber scrubber = new Scrubber(cfs, sstable, options.skipCorrupted, !options.noValidate, handler, true); ++Scrubber scrubber = new Scrubber(cfs, sstable, options.skipCorrupted, handler, true, !options.noValidate); try { scrubber.scrub(); http://git-wip-us.apache.org/repos/asf/cassandra/blob/d693ca12/test/unit/org/apache/cassandra/db/ScrubTest.java -- diff --cc test/unit/org/apache/cassandra/db/ScrubTest.java index a19c76d,94c7e34..028cf6c --- a/test/unit/org/apache/cassandra/db/ScrubTest.java +++ b/test/unit/org/apache/cassandra/db/ScrubTest.java @@@ -30,7 -30,10 +30,11 @@@ import java.util.Set import java.util.concurrent.ExecutionException; import org.apache.cassandra.cql3.QueryProcessor; -import org.apache.cassandra.db.compaction.OperationType; ++import org.apache.cassandra.db.composites.CellNameType; +import org.apache.cassandra.exceptions.ConfigurationException; + import org.apache.cassandra.db.marshal.CompositeType; + import org.apache.cassandra.db.marshal.LongType; + import org.apache.cassandra.db.marshal.UTF8Type; import org.apache.cassandra.exceptions.RequestExecutionException; import org.apache.cassandra.io.compress.CompressionMetadata; import org.apache.cassandra.utils.UUIDGen; @@@ -124,7 -123,7 +128,7 @@@ public class ScrubTest extends SchemaLo overrideWithGarbage(sstable, ByteBufferUtil.bytes(0), ByteBufferUtil.bytes(1)); // with skipCorrupted == false, the scrub is expected to fail - Scrubber scrubber = new Scrubber(cfs, sstable, false, false); -Scrubber scrubber = new Scrubber(cfs, sstable, false, true); ++Scrubber scrubber = new Scrubber(cfs, sstable, false, false, true); try { scrubber.scrub(); @@@ -134,9 -133,10 +138,9 @@@ // with skipCorrupted == true, the corrupt row will be skipped Scrubber.ScrubResult scrubResult; - scrubber = new Scrubber(cfs, sstable, true, false); -scrubber = new Scrubber(cfs, sstable, true, true); ++scrubber = new Scrubber(cfs, sstable, true, false, true); scrubResult = scrubber.scrubWithResult(); scrubber.close(); -cfs.replaceCompactedSSTables(Collections.singletonList(sstable), Collections.singletonList(scrubber.getNewSSTable()), OperationType.SCRUB); assertNotNull(scrubResult); @@@ -182,7 -181,7 +186,7 @@@ overrideWithGarbage(sstable, ByteBufferUtil.bytes(0), ByteBufferUtil.bytes(1)); // with skipCorrupted == false, the scrub is expected to fail - Scrubber scrubber = new Scrubber(cfs, sstable, false, false); -Scrubber scrubber = new Scrubber(cfs, sstable, false, true); ++Scrubber scrubber = new Scrubber(cfs, sstable, false, false, true); try { scrubber.scrub(); @@@ -191,9 -190,10 +195,9 @@@ catch (IOError err) {} // with skipCorrupted == true, the corrupt row will be skipped - scrubber = new Scrubber(cfs, sstable, true, false); -scrubber = new Scrubber(cfs, sstable, true, true); ++scrubber = new Scrubber(cfs, sstable, true, false, true); scrubber.scrub(); scrubber.close(); -cfs.replaceCompactedSSTables(Collections.singletonList(sstable), Collections.singletonList(scrubber.getNewSSTable()), OperationType.SCRUB); assertEquals(1, cfs.getSSTables().size()); // verify that we can read all of the rows, and there is now one less row @@@ -324,7 -324,7 +328,7 @@@ components.add(Component.TOC); SSTableReader sstable = SSTableReader.openNoValidation(desc, components, metadata); --Scrubber scrubber = new Scrubber(cfs, sstable, false, true); ++Scrubber scrubber = new Scrubber(cfs, sstable, false, true, true); scrubber.scrub(); cfs.loadNewSSTables(); @@@ -420,9 -420,19 +424,19 @@@ Keyspace keyspace = Keyspace.open(Keyspace1); ColumnFamilyStore cfs = keyspace.getColumnFamilyStore(test_compact_static_columns); -QueryProcessor.processInternal(INSERT INTO
[10/16] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
Merge branch 'cassandra-2.0' into cassandra-2.1 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d693ca12 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d693ca12 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d693ca12 Branch: refs/heads/cassandra-2.1 Commit: d693ca12c76c2651df1769e137a94b954174e061 Parents: adb8831 be9eff5 Author: Yuki Morishita yu...@apache.org Authored: Tue May 19 08:50:28 2015 -0500 Committer: Yuki Morishita yu...@apache.org Committed: Tue May 19 08:50:28 2015 -0500 -- CHANGES.txt | 1 + .../apache/cassandra/db/ColumnFamilyStore.java | 4 +-- .../db/compaction/CompactionManager.java| 8 ++--- .../cassandra/db/compaction/Scrubber.java | 12 --- .../cassandra/service/StorageService.java | 7 +++- .../cassandra/service/StorageServiceMBean.java | 2 ++ .../org/apache/cassandra/tools/NodeProbe.java | 8 ++--- .../org/apache/cassandra/tools/NodeTool.java| 7 +++- .../cassandra/tools/StandaloneScrubber.java | 6 +++- .../unit/org/apache/cassandra/db/ScrubTest.java | 38 +--- 10 files changed, 63 insertions(+), 30 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/d693ca12/CHANGES.txt -- diff --cc CHANGES.txt index 198935b,cf124b4..6fc1c9c --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -30,6 -3,10 +30,7 @@@ Merged from 2.0 * Clone SliceQueryFilter in AbstractReadCommand implementations (CASSANDRA-8940) * Push correct protocol notification for DROP INDEX (CASSANDRA-9310) * token-generator - generated tokens too long (CASSANDRA-9300) + * Add option not to validate atoms during scrub (CASSANDRA-9406) - - -2.0.15: * Fix counting of tombstones for TombstoneOverwhelmingException (CASSANDRA-9299) * Fix ReconnectableSnitch reconnecting to peers during upgrade (CASSANDRA-6702) * Include keyspace and table name in error log for collections over the size http://git-wip-us.apache.org/repos/asf/cassandra/blob/d693ca12/src/java/org/apache/cassandra/db/ColumnFamilyStore.java -- diff --cc src/java/org/apache/cassandra/db/ColumnFamilyStore.java index bdc2d8b,eec4044..0951c01 --- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java +++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java @@@ -1394,22 -1117,22 +1394,22 @@@ public class ColumnFamilyStore implemen return maxFile; } -public void forceCleanup(CounterId.OneShotRenewer renewer) throws ExecutionException, InterruptedException +public CompactionManager.AllSSTableOpStatus forceCleanup() throws ExecutionException, InterruptedException { -CompactionManager.instance.performCleanup(ColumnFamilyStore.this, renewer); +return CompactionManager.instance.performCleanup(ColumnFamilyStore.this); } - public CompactionManager.AllSSTableOpStatus scrub(boolean disableSnapshot, boolean skipCorrupted) throws ExecutionException, InterruptedException -public void scrub(boolean disableSnapshot, boolean skipCorrupted, boolean checkData) throws ExecutionException, InterruptedException ++public CompactionManager.AllSSTableOpStatus scrub(boolean disableSnapshot, boolean skipCorrupted, boolean checkData) throws ExecutionException, InterruptedException { // skip snapshot creation during scrub, SEE JIRA 5891 if(!disableSnapshot) snapshotWithoutFlush(pre-scrub- + System.currentTimeMillis()); - return CompactionManager.instance.performScrub(ColumnFamilyStore.this, skipCorrupted); -CompactionManager.instance.performScrub(ColumnFamilyStore.this, skipCorrupted, checkData); ++return CompactionManager.instance.performScrub(ColumnFamilyStore.this, skipCorrupted, checkData); } -public void sstablesRewrite(boolean excludeCurrentVersion) throws ExecutionException, InterruptedException +public CompactionManager.AllSSTableOpStatus sstablesRewrite(boolean excludeCurrentVersion) throws ExecutionException, InterruptedException { - CompactionManager.instance.performSSTableRewrite(ColumnFamilyStore.this, excludeCurrentVersion); +return CompactionManager.instance.performSSTableRewrite(ColumnFamilyStore.this, excludeCurrentVersion); } public void markObsolete(CollectionSSTableReader sstables, OperationType compactionType) http://git-wip-us.apache.org/repos/asf/cassandra/blob/d693ca12/src/java/org/apache/cassandra/db/compaction/CompactionManager.java -- diff --cc
[03/16] cassandra git commit: Add option to not validate atoms during scrub
Add option to not validate atoms during scrub patch by Jordan West; reviewed by yukim for CASSANDRA-9406 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/be9eff57 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/be9eff57 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/be9eff57 Branch: refs/heads/cassandra-2.2 Commit: be9eff573342cd4993eb4fc8ee694dc885e8a8e4 Parents: 37d81b9 Author: Jordan West jorda...@gmail.com Authored: Tue May 19 08:26:27 2015 -0500 Committer: Yuki Morishita yu...@apache.org Committed: Tue May 19 08:26:27 2015 -0500 -- CHANGES.txt | 1 + .../apache/cassandra/db/ColumnFamilyStore.java | 4 +-- .../db/compaction/CompactionManager.java| 12 +++ .../cassandra/db/compaction/Scrubber.java | 12 --- .../cassandra/service/StorageService.java | 7 +++- .../cassandra/service/StorageServiceMBean.java | 2 ++ .../org/apache/cassandra/tools/NodeCmd.java | 5 ++- .../org/apache/cassandra/tools/NodeProbe.java | 4 +-- .../cassandra/tools/StandaloneScrubber.java | 6 +++- .../unit/org/apache/cassandra/db/ScrubTest.java | 37 +--- 10 files changed, 60 insertions(+), 30 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/be9eff57/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index a9d04d6..cf124b4 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -3,6 +3,7 @@ * Clone SliceQueryFilter in AbstractReadCommand implementations (CASSANDRA-8940) * Push correct protocol notification for DROP INDEX (CASSANDRA-9310) * token-generator - generated tokens too long (CASSANDRA-9300) + * Add option not to validate atoms during scrub (CASSANDRA-9406) 2.0.15: http://git-wip-us.apache.org/repos/asf/cassandra/blob/be9eff57/src/java/org/apache/cassandra/db/ColumnFamilyStore.java -- diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java index 5ea1287..eec4044 100644 --- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java +++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java @@ -1122,12 +1122,12 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean CompactionManager.instance.performCleanup(ColumnFamilyStore.this, renewer); } -public void scrub(boolean disableSnapshot, boolean skipCorrupted) throws ExecutionException, InterruptedException +public void scrub(boolean disableSnapshot, boolean skipCorrupted, boolean checkData) throws ExecutionException, InterruptedException { // skip snapshot creation during scrub, SEE JIRA 5891 if(!disableSnapshot) snapshotWithoutFlush(pre-scrub- + System.currentTimeMillis()); -CompactionManager.instance.performScrub(ColumnFamilyStore.this, skipCorrupted); +CompactionManager.instance.performScrub(ColumnFamilyStore.this, skipCorrupted, checkData); } public void sstablesRewrite(boolean excludeCurrentVersion) throws ExecutionException, InterruptedException http://git-wip-us.apache.org/repos/asf/cassandra/blob/be9eff57/src/java/org/apache/cassandra/db/compaction/CompactionManager.java -- diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java index 0978ae6..207b90d 100644 --- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java +++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java @@ -234,13 +234,13 @@ public class CompactionManager implements CompactionManagerMBean executor.submit(runnable).get(); } -public void performScrub(ColumnFamilyStore cfStore, final boolean skipCorrupted) throws InterruptedException, ExecutionException +public void performScrub(ColumnFamilyStore cfStore, final boolean skipCorrupted, final boolean checkData) throws InterruptedException, ExecutionException { performAllSSTableOperation(cfStore, new AllSSTablesOperation() { public void perform(ColumnFamilyStore store, IterableSSTableReader sstables) throws IOException { -doScrub(store, sstables, skipCorrupted); +doScrub(store, sstables, skipCorrupted, checkData); } }); } @@ -432,16 +432,16 @@ public class CompactionManager implements CompactionManagerMBean * * @throws IOException */ -private void doScrub(ColumnFamilyStore cfs, IterableSSTableReader sstables, boolean
[01/16] cassandra git commit: Add option to not validate atoms during scrub
Repository: cassandra Updated Branches: refs/heads/cassandra-2.0 37d81b9dc - be9eff573 refs/heads/cassandra-2.1 adb88316a - d693ca12c refs/heads/cassandra-2.2 678291a75 - ff1a15629 refs/heads/trunk bade0fb1a - 5c413ef76 Add option to not validate atoms during scrub patch by Jordan West; reviewed by yukim for CASSANDRA-9406 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/be9eff57 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/be9eff57 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/be9eff57 Branch: refs/heads/cassandra-2.0 Commit: be9eff573342cd4993eb4fc8ee694dc885e8a8e4 Parents: 37d81b9 Author: Jordan West jorda...@gmail.com Authored: Tue May 19 08:26:27 2015 -0500 Committer: Yuki Morishita yu...@apache.org Committed: Tue May 19 08:26:27 2015 -0500 -- CHANGES.txt | 1 + .../apache/cassandra/db/ColumnFamilyStore.java | 4 +-- .../db/compaction/CompactionManager.java| 12 +++ .../cassandra/db/compaction/Scrubber.java | 12 --- .../cassandra/service/StorageService.java | 7 +++- .../cassandra/service/StorageServiceMBean.java | 2 ++ .../org/apache/cassandra/tools/NodeCmd.java | 5 ++- .../org/apache/cassandra/tools/NodeProbe.java | 4 +-- .../cassandra/tools/StandaloneScrubber.java | 6 +++- .../unit/org/apache/cassandra/db/ScrubTest.java | 37 +--- 10 files changed, 60 insertions(+), 30 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/be9eff57/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index a9d04d6..cf124b4 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -3,6 +3,7 @@ * Clone SliceQueryFilter in AbstractReadCommand implementations (CASSANDRA-8940) * Push correct protocol notification for DROP INDEX (CASSANDRA-9310) * token-generator - generated tokens too long (CASSANDRA-9300) + * Add option not to validate atoms during scrub (CASSANDRA-9406) 2.0.15: http://git-wip-us.apache.org/repos/asf/cassandra/blob/be9eff57/src/java/org/apache/cassandra/db/ColumnFamilyStore.java -- diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java index 5ea1287..eec4044 100644 --- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java +++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java @@ -1122,12 +1122,12 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean CompactionManager.instance.performCleanup(ColumnFamilyStore.this, renewer); } -public void scrub(boolean disableSnapshot, boolean skipCorrupted) throws ExecutionException, InterruptedException +public void scrub(boolean disableSnapshot, boolean skipCorrupted, boolean checkData) throws ExecutionException, InterruptedException { // skip snapshot creation during scrub, SEE JIRA 5891 if(!disableSnapshot) snapshotWithoutFlush(pre-scrub- + System.currentTimeMillis()); -CompactionManager.instance.performScrub(ColumnFamilyStore.this, skipCorrupted); +CompactionManager.instance.performScrub(ColumnFamilyStore.this, skipCorrupted, checkData); } public void sstablesRewrite(boolean excludeCurrentVersion) throws ExecutionException, InterruptedException http://git-wip-us.apache.org/repos/asf/cassandra/blob/be9eff57/src/java/org/apache/cassandra/db/compaction/CompactionManager.java -- diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java index 0978ae6..207b90d 100644 --- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java +++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java @@ -234,13 +234,13 @@ public class CompactionManager implements CompactionManagerMBean executor.submit(runnable).get(); } -public void performScrub(ColumnFamilyStore cfStore, final boolean skipCorrupted) throws InterruptedException, ExecutionException +public void performScrub(ColumnFamilyStore cfStore, final boolean skipCorrupted, final boolean checkData) throws InterruptedException, ExecutionException { performAllSSTableOperation(cfStore, new AllSSTablesOperation() { public void perform(ColumnFamilyStore store, IterableSSTableReader sstables) throws IOException { -doScrub(store, sstables, skipCorrupted); +doScrub(store, sstables, skipCorrupted, checkData); } }); } @@
[05/16] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
http://git-wip-us.apache.org/repos/asf/cassandra/blob/d693ca12/src/java/org/apache/cassandra/tools/StandaloneScrubber.java -- diff --cc src/java/org/apache/cassandra/tools/StandaloneScrubber.java index d420218,26768f3..459df98 --- a/src/java/org/apache/cassandra/tools/StandaloneScrubber.java +++ b/src/java/org/apache/cassandra/tools/StandaloneScrubber.java @@@ -109,7 -121,7 +110,7 @@@ public class StandaloneScrubbe { try { - Scrubber scrubber = new Scrubber(cfs, sstable, options.skipCorrupted, handler, true); -Scrubber scrubber = new Scrubber(cfs, sstable, options.skipCorrupted, !options.noValidate, handler, true); ++Scrubber scrubber = new Scrubber(cfs, sstable, options.skipCorrupted, handler, true, !options.noValidate); try { scrubber.scrub(); http://git-wip-us.apache.org/repos/asf/cassandra/blob/d693ca12/test/unit/org/apache/cassandra/db/ScrubTest.java -- diff --cc test/unit/org/apache/cassandra/db/ScrubTest.java index a19c76d,94c7e34..028cf6c --- a/test/unit/org/apache/cassandra/db/ScrubTest.java +++ b/test/unit/org/apache/cassandra/db/ScrubTest.java @@@ -30,7 -30,10 +30,11 @@@ import java.util.Set import java.util.concurrent.ExecutionException; import org.apache.cassandra.cql3.QueryProcessor; -import org.apache.cassandra.db.compaction.OperationType; ++import org.apache.cassandra.db.composites.CellNameType; +import org.apache.cassandra.exceptions.ConfigurationException; + import org.apache.cassandra.db.marshal.CompositeType; + import org.apache.cassandra.db.marshal.LongType; + import org.apache.cassandra.db.marshal.UTF8Type; import org.apache.cassandra.exceptions.RequestExecutionException; import org.apache.cassandra.io.compress.CompressionMetadata; import org.apache.cassandra.utils.UUIDGen; @@@ -124,7 -123,7 +128,7 @@@ public class ScrubTest extends SchemaLo overrideWithGarbage(sstable, ByteBufferUtil.bytes(0), ByteBufferUtil.bytes(1)); // with skipCorrupted == false, the scrub is expected to fail - Scrubber scrubber = new Scrubber(cfs, sstable, false, false); -Scrubber scrubber = new Scrubber(cfs, sstable, false, true); ++Scrubber scrubber = new Scrubber(cfs, sstable, false, false, true); try { scrubber.scrub(); @@@ -134,9 -133,10 +138,9 @@@ // with skipCorrupted == true, the corrupt row will be skipped Scrubber.ScrubResult scrubResult; - scrubber = new Scrubber(cfs, sstable, true, false); -scrubber = new Scrubber(cfs, sstable, true, true); ++scrubber = new Scrubber(cfs, sstable, true, false, true); scrubResult = scrubber.scrubWithResult(); scrubber.close(); -cfs.replaceCompactedSSTables(Collections.singletonList(sstable), Collections.singletonList(scrubber.getNewSSTable()), OperationType.SCRUB); assertNotNull(scrubResult); @@@ -182,7 -181,7 +186,7 @@@ overrideWithGarbage(sstable, ByteBufferUtil.bytes(0), ByteBufferUtil.bytes(1)); // with skipCorrupted == false, the scrub is expected to fail - Scrubber scrubber = new Scrubber(cfs, sstable, false, false); -Scrubber scrubber = new Scrubber(cfs, sstable, false, true); ++Scrubber scrubber = new Scrubber(cfs, sstable, false, false, true); try { scrubber.scrub(); @@@ -191,9 -190,10 +195,9 @@@ catch (IOError err) {} // with skipCorrupted == true, the corrupt row will be skipped - scrubber = new Scrubber(cfs, sstable, true, false); -scrubber = new Scrubber(cfs, sstable, true, true); ++scrubber = new Scrubber(cfs, sstable, true, false, true); scrubber.scrub(); scrubber.close(); -cfs.replaceCompactedSSTables(Collections.singletonList(sstable), Collections.singletonList(scrubber.getNewSSTable()), OperationType.SCRUB); assertEquals(1, cfs.getSSTables().size()); // verify that we can read all of the rows, and there is now one less row @@@ -324,7 -324,7 +328,7 @@@ components.add(Component.TOC); SSTableReader sstable = SSTableReader.openNoValidation(desc, components, metadata); --Scrubber scrubber = new Scrubber(cfs, sstable, false, true); ++Scrubber scrubber = new Scrubber(cfs, sstable, false, true, true); scrubber.scrub(); cfs.loadNewSSTables(); @@@ -420,9 -420,19 +424,19 @@@ Keyspace keyspace = Keyspace.open(Keyspace1); ColumnFamilyStore cfs = keyspace.getColumnFamilyStore(test_compact_static_columns); -QueryProcessor.processInternal(INSERT INTO
cassandra git commit: cancel any rejected tasks on shutdown so callers who care can progress
Repository: cassandra Updated Branches: refs/heads/cassandra-2.1 d693ca12c - 7af9c6ac8 cancel any rejected tasks on shutdown so callers who care can progress patch by tjake; reviewed by Sergio Bossa for CASSANDRA-9417 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7af9c6ac Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7af9c6ac Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7af9c6ac Branch: refs/heads/cassandra-2.1 Commit: 7af9c6ac88278d34202fea922606905b326f7811 Parents: d693ca1 Author: T Jake Luciani j...@apache.org Authored: Mon May 18 15:18:24 2015 -0400 Committer: T Jake Luciani j...@apache.org Committed: Tue May 19 10:12:07 2015 -0400 -- .../DebuggableScheduledThreadPoolExecutor.java | 9 ++ ...buggableScheduledThreadPoolExecutorTest.java | 115 +++ 2 files changed, 124 insertions(+) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/7af9c6ac/src/java/org/apache/cassandra/concurrent/DebuggableScheduledThreadPoolExecutor.java -- diff --git a/src/java/org/apache/cassandra/concurrent/DebuggableScheduledThreadPoolExecutor.java b/src/java/org/apache/cassandra/concurrent/DebuggableScheduledThreadPoolExecutor.java index 7226db8..ea0715c 100644 --- a/src/java/org/apache/cassandra/concurrent/DebuggableScheduledThreadPoolExecutor.java +++ b/src/java/org/apache/cassandra/concurrent/DebuggableScheduledThreadPoolExecutor.java @@ -31,6 +31,11 @@ import org.apache.cassandra.utils.JVMStabilityInspector; * * DebuggableScheduledThreadPoolExecutor also catches exceptions during Task execution * so that they don't supress subsequent invocations of the task. + * + * Finally, there is a special rejected execution handler for tasks rejected during the shutdown hook. + * + * For fire and forget tasks (like ref tidy) we can safely ignore the exceptions. + * For any callers that care to know their task was rejected we cancel passed task. */ public class DebuggableScheduledThreadPoolExecutor extends ScheduledThreadPoolExecutor { @@ -45,6 +50,10 @@ public class DebuggableScheduledThreadPoolExecutor extends ScheduledThreadPoolEx if (!StorageService.instance.isInShutdownHook()) throw new RejectedExecutionException(ScheduledThreadPoolExecutor has shut down.); +//Give some notification to the caller the task isn't going to run +if (task instanceof Future) +((Future) task).cancel(false); + logger.debug(ScheduledThreadPoolExecutor has shut down as part of C* shutdown); } else http://git-wip-us.apache.org/repos/asf/cassandra/blob/7af9c6ac/test/unit/org/apache/cassandra/concurrent/DebuggableScheduledThreadPoolExecutorTest.java -- diff --git a/test/unit/org/apache/cassandra/concurrent/DebuggableScheduledThreadPoolExecutorTest.java b/test/unit/org/apache/cassandra/concurrent/DebuggableScheduledThreadPoolExecutorTest.java new file mode 100644 index 000..46b2764 --- /dev/null +++ b/test/unit/org/apache/cassandra/concurrent/DebuggableScheduledThreadPoolExecutorTest.java @@ -0,0 +1,115 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * License); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an AS IS BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.cassandra.concurrent; + +import java.io.IOException; +import java.util.concurrent.CancellationException; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.ScheduledFuture; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.TimeoutException; +import java.util.concurrent.atomic.AtomicInteger; + +import org.junit.BeforeClass; +import org.junit.Test; + +import junit.framework.Assert; +import org.apache.cassandra.service.EmbeddedCassandraService; +import org.apache.cassandra.service.StorageService; + +public class DebuggableScheduledThreadPoolExecutorTest +{ + +static
[2/2] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2
Merge branch 'cassandra-2.1' into cassandra-2.2 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e8766e53 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e8766e53 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e8766e53 Branch: refs/heads/cassandra-2.2 Commit: e8766e53d11645b41e25d904d5bdbdb2b1c0c9a9 Parents: ff1a156 7af9c6a Author: T Jake Luciani j...@apache.org Authored: Tue May 19 10:13:14 2015 -0400 Committer: T Jake Luciani j...@apache.org Committed: Tue May 19 10:13:14 2015 -0400 -- .../DebuggableScheduledThreadPoolExecutor.java | 9 ++ ...buggableScheduledThreadPoolExecutorTest.java | 115 +++ 2 files changed, 124 insertions(+) --
[jira] [Commented] (CASSANDRA-9407) SSTableRewriterTest is hard failing after Fix canonical view returning early opened SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-9407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14550490#comment-14550490 ] Ariel Weisberg commented on CASSANDRA-9407: --- I am in the habit of being pretty generous with opening tickets for commits because JIRA has a really nice feature we don't have hooked up where all the commits for a ticket are shown in JIRA under one of the tabs at the bottom if they mention the ticket in the commit message. That is why I always include the ticket name in every commit. Separate tickets kind of maybe aid in searchability and tracking. I am fine with reopening instead. SSTableRewriterTest is hard failing after Fix canonical view returning early opened SSTables -- Key: CASSANDRA-9407 URL: https://issues.apache.org/jira/browse/CASSANDRA-9407 Project: Cassandra Issue Type: Test Reporter: Ariel Weisberg Assignee: Benedict Fix For: 2.2.0 rc1 When I bisected it seems to start failing after https://github.com/apache/cassandra/commit/7d6a60ddd0e156f6028c05d97fa9179f86fe02ee -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9419) CqlRecordWriter does not adequately determine if driver failed to connect due to interrupt
[ https://issues.apache.org/jira/browse/CASSANDRA-9419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14550493#comment-14550493 ] Sam Tunnicliffe commented on CASSANDRA-9419: I'm still seeing failures locally, where the log output looks like this (with some of the WARNs elided): {noformat} [junit] Testsuite: org.apache.cassandra.pig.CqlTableTest [junit] Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 47.007 sec [junit] [junit] - Standard Output --- ... [junit] WARN 13:27:58 job_local_0003 [junit] java.io.IOException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: localhost/127.0.0.1:9042 (com.datastax.driver.core.exceptions.DriverException: Connection thread interrupted)) [junit] at org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.run(CqlRecordWriter.java:329) ~[main/:na] [junit] Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: localhost/127.0.0.1:9042 (com.datastax.driver.core.exceptions.DriverException: Connection thread interrupted)) [junit] at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:223) ~[cassandra-driver-core-2.1.5-shaded.jar:na] [junit] at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:78) ~[cassandra-driver-core-2.1.5-shaded.jar:na] [junit] at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1230) ~[cassandra-driver-core-2.1.5-shaded.jar:na] [junit] at com.datastax.driver.core.Cluster.init(Cluster.java:157) ~[cassandra-driver-core-2.1.5-shaded.jar:na] [junit] at com.datastax.driver.core.Cluster.connect(Cluster.java:245) ~[cassandra-driver-core-2.1.5-shaded.jar:na] [junit] at org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.run(CqlRecordWriter.java:320) ~[main/:na] [junit] WARN 13:27:58 job_local_0003 [junit] java.io.IOException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: localhost/127.0.0.1:9042 (com.datastax.driver.core.exceptions.DriverException: Connection thread interrupted)) [junit] at org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.run(CqlRecordWriter.java:329) ~[main/:na] [junit] Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: localhost/127.0.0.1:9042 (com.datastax.driver.core.exceptions.DriverException: Connection thread interrupted)) [junit] at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:223) ~[cassandra-driver-core-2.1.5-shaded.jar:na] [junit] at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:78) ~[cassandra-driver-core-2.1.5-shaded.jar:na] [junit] at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1230) ~[cassandra-driver-core-2.1.5-shaded.jar:na] [junit] at com.datastax.driver.core.Cluster.init(Cluster.java:157) ~[cassandra-driver-core-2.1.5-shaded.jar:na] [junit] at com.datastax.driver.core.Cluster.connect(Cluster.java:245) ~[cassandra-driver-core-2.1.5-shaded.jar:na] [junit] at org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.run(CqlRecordWriter.java:320) ~[main/:na] [junit] WARN 13:27:58 Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately on failure. [junit] WARN 13:27:58 Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately on failure. [junit] ERROR 13:27:58 1 map reduce job(s) failed! [junit] ERROR 13:27:58 1 map reduce job(s) failed! [junit] WARN 13:27:59 Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. [junit] WARN 13:27:59 Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. [junit] WARN 13:27:59 No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String). [junit] WARN 13:27:59 No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String). [junit] WARN 13:28:02 Failed to get RunningJob for job job_local_0004 [junit] WARN 13:28:02 Failed to get RunningJob for job job_local_0004 [junit] WARN 13:28:02 SchemaTupleBackend has already been initialized [junit] WARN 13:28:02 SchemaTupleBackend has already been initialized [junit] WARN 13:28:03 Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. [junit] WARN 13:28:03 Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. [junit] WARN 13:28:03
[jira] [Updated] (CASSANDRA-8603) Cut tombstone memory footprint in half for cql deletes
[ https://issues.apache.org/jira/browse/CASSANDRA-8603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Lerer updated CASSANDRA-8603: -- Attachment: 8603-2.1-V3.txt Patch for 2.1 that perform the check during deserialization Cut tombstone memory footprint in half for cql deletes -- Key: CASSANDRA-8603 URL: https://issues.apache.org/jira/browse/CASSANDRA-8603 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Dominic Letz Assignee: Dominic Letz Labels: tombstone Attachments: 8603-2.1-V3.txt, cassandra-2.0.11-8603.txt, cassandra-2.1-8603.txt, cassandra-2.1-8603_v2.txt, system.log As CQL does not yet support range deletes every delete from CQL results in a Semi-RangeTombstone which actually has the same start and end values - but until today they are copies. Effectively doubling the required heap memory to store the RangeTombstone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8163) Re-introduce DESCRIBE permission
[ https://issues.apache.org/jira/browse/CASSANDRA-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14550527#comment-14550527 ] Jonathan Ellis commented on CASSANDRA-8163: --- Since this is a breaking API change, we'll have to do it in 4.0 if it doesn't make 3.0. (But I'm not tagging it for 3.0 b1 since I'd prefer to reserve that for must-have tickets.) Re-introduce DESCRIBE permission Key: CASSANDRA-8163 URL: https://issues.apache.org/jira/browse/CASSANDRA-8163 Project: Cassandra Issue Type: Improvement Reporter: Vishy Kasar Priority: Minor Fix For: 3.x We have a cluster like this: project1_keyspace table101 table102 project2_keyspace table201 table202 We have set up following users and grants: project1_user has all access to project1_keyspace project2_user has all access to project2_keyspace However project1_user can still do a 'describe schema' and get the schema for project2_keyspace as well. We do not want project1_user to have any knowledge for project2 in any way (cqlsh/java-driver etc) . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8163) Re-introduce DESCRIBE permission
[ https://issues.apache.org/jira/browse/CASSANDRA-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14550527#comment-14550527 ] Jonathan Ellis edited comment on CASSANDRA-8163 at 5/19/15 2:44 PM: If this is a breaking API change, we'll have to do it in 4.0 if it doesn't make 3.0. (But I'm not tagging it for 3.0 b1 since I'd prefer to reserve that for must-have tickets.) was (Author: jbellis): Since this is a breaking API change, we'll have to do it in 4.0 if it doesn't make 3.0. (But I'm not tagging it for 3.0 b1 since I'd prefer to reserve that for must-have tickets.) Re-introduce DESCRIBE permission Key: CASSANDRA-8163 URL: https://issues.apache.org/jira/browse/CASSANDRA-8163 Project: Cassandra Issue Type: Improvement Reporter: Vishy Kasar Priority: Minor Fix For: 3.x We have a cluster like this: project1_keyspace table101 table102 project2_keyspace table201 table202 We have set up following users and grants: project1_user has all access to project1_keyspace project2_user has all access to project2_keyspace However project1_user can still do a 'describe schema' and get the schema for project2_keyspace as well. We do not want project1_user to have any knowledge for project2 in any way (cqlsh/java-driver etc) . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CASSANDRA-9414) CommitLogTest failing intermittently on Windows
[ https://issues.apache.org/jira/browse/CASSANDRA-9414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joshua McKenzie resolved CASSANDRA-9414. Resolution: Not A Problem No longer occurring either locally or on CI box on 15 runs (was about 90% failure rate when I opened this ticket). Closing as Not a Problem as there's enough other things to fix (see CASSANDRA-9418) that it's not worth chasing a transient utest failure right now. CommitLogTest failing intermittently on Windows --- Key: CASSANDRA-9414 URL: https://issues.apache.org/jira/browse/CASSANDRA-9414 Project: Cassandra Issue Type: Bug Reporter: Joshua McKenzie Assignee: Joshua McKenzie Priority: Minor Labels: Windows Fix For: 2.2.0 rc1 Failure is intermittent enough that bisect is proving to be more hassle than it's worth. Seems pretty consistent in CI. {noformat} [junit] Testcase: testDeleteIfNotDirty(org.apache.cassandra.db.CommitLogTest): Caused an ERROR [junit] java.nio.file.AccessDeniedException: build\test\cassandra\commitlog;0\CommitLog-5-1431965988394.log [junit] FSWriteError in build\test\cassandra\commitlog;0\CommitLog-5-1431965988394.log [junit] at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:131) [junit] at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:148) [junit] at org.apache.cassandra.db.commitlog.CommitLogSegmentManager.recycleSegment(CommitLogSegmentManager.java:360) [junit] at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:166) [junit] at org.apache.cassandra.db.commitlog.CommitLog.startUnsafe(CommitLog.java:416) [junit] at org.apache.cassandra.db.commitlog.CommitLog.resetUnsafe(CommitLog.java:389) [junit] at org.apache.cassandra.db.CommitLogTest.testDeleteIfNotDirty(CommitLogTest.java:178) [junit] Caused by: java.nio.file.AccessDeniedException: build\test\cassandra\commitlog;0\CommitLog-5-1431965988394.log [junit] at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83) [junit] at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) [junit] at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102) [junit] at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269) [junit] at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103) [junit] at java.nio.file.Files.delete(Files.java:1126) [junit] at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:125) {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8603) Cut tombstone memory footprint in half for cql deletes
[ https://issues.apache.org/jira/browse/CASSANDRA-8603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14550559#comment-14550559 ] Aleksey Yeschenko commented on CASSANDRA-8603: -- v3 LGTM. We don't need new tests here, but I'd like to see cassci results before committing. Cut tombstone memory footprint in half for cql deletes -- Key: CASSANDRA-8603 URL: https://issues.apache.org/jira/browse/CASSANDRA-8603 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Dominic Letz Assignee: Benjamin Lerer Labels: tombstone Attachments: 8603-2.1-V3.txt, cassandra-2.0.11-8603.txt, cassandra-2.1-8603.txt, cassandra-2.1-8603_v2.txt, system.log As CQL does not yet support range deletes every delete from CQL results in a Semi-RangeTombstone which actually has the same start and end values - but until today they are copies. Effectively doubling the required heap memory to store the RangeTombstone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9369) HSHA dtest for closing connections is failing on trunk
[ https://issues.apache.org/jira/browse/CASSANDRA-9369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14550567#comment-14550567 ] Michael Shuler commented on CASSANDRA-9369: --- Speaking for myself and my own workflow, the only way to see what tests are failing is to run them continually in CI. CI is the tool I am using to track the tests that are failing repeatedly, intermittently failing, or blowing up everything. If they are hidden from my view and I don't see them fail in CI, then they don't get brought to my attention. I volunteered to track the testing to try to get to the goal of 100% passing, and I found this particular ticket from a git commit message that caught my attention - not from the tool I am using to track the state of testing, which is where I would like it. As an example of the problem I have with disabling tests with {{@require}}, I just now removed this disabling for CASSANDRA-8049 CASSANDRA-9300 and CASSANDRA-9294 - the committers of those may or may not have known these dtests were disabled, I don't know. So I understand your point, but I would kindly request that you leave failing tests to fail in CI. It is much easier for me to help with tracking the state of tests. HSHA dtest for closing connections is failing on trunk -- Key: CASSANDRA-9369 URL: https://issues.apache.org/jira/browse/CASSANDRA-9369 Project: Cassandra Issue Type: Test Components: Tests Reporter: Tyler Hobbs Assignee: Ariel Weisberg Fix For: 2.2.x The {{thrift_hsha_test.ThriftHSHATest.test_closing_connections}} dtest is failing against trunk (but not against 2.1). Here's an example failure: {noformat} Error Message There are non-closed connections: COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java28202 automaton7u IPv4 60767 0t0 TCP localhost:58436-localhost:9160 (CLOSE_WAIT) begin captured logging dtest: DEBUG: cluster ccm directory: /tmp/dtest-U0pP7H dtest: DEBUG: Creating connection pools.. dtest: DEBUG: Disabling/Enabling thrift iteration #0 dtest: DEBUG: Closing connections from the client side.. pycassa.pool: INFO: Pool 13972199784 was disposed pycassa.pool: INFO: Pool 139721997714512 was disposed pycassa.pool: INFO: Pool 139721997749904 was disposed - end captured logging - Stacktrace File /usr/lib/python2.7/unittest/case.py, line 331, in run testMethod() File /home/automaton/cassandra-dtest/thrift_hsha_test.py, line 59, in test_closing_connections self.assertEqual(len(lines), 0, There are non-closed connections: %s % stdout) File /usr/lib/python2.7/unittest/case.py, line 515, in assertEqual assertion_func(first, second, msg=msg) File /usr/lib/python2.7/unittest/case.py, line 508, in _baseAssertEqual raise self.failureException(msg) 'There are non-closed connections: COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME\njava28202 automaton7u IPv4 60767 0t0 TCP localhost:58436-localhost:9160 (CLOSE_WAIT)\n\n begin captured logging \ndtest: DEBUG: cluster ccm directory: /tmp/dtest-U0pP7H\ndtest: DEBUG: Creating connection pools..\ndtest: DEBUG: Disabling/Enabling thrift iteration #0\ndtest: DEBUG: Closing connections from the client side..\npycassa.pool: INFO: Pool 13972199784 was disposed\npycassa.pool: INFO: Pool 139721997714512 was disposed\npycassa.pool: INFO: Pool 139721997749904 was disposed\n- end captured logging -' {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-9369) HSHA dtest for closing connections is failing on trunk
[ https://issues.apache.org/jira/browse/CASSANDRA-9369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14550567#comment-14550567 ] Michael Shuler edited comment on CASSANDRA-9369 at 5/19/15 2:56 PM: Speaking for myself and my own workflow, the only way to see what tests are failing is to run them continually in CI. CI is the tool I am using to track the tests that are failing repeatedly, intermittently failing, or blowing up everything. If they are hidden from my view and I don't see them fail in CI, then they don't get brought to my attention. I volunteered to track the testing to try to get to the goal of 100% passing, and I found this particular ticket from a git commit message that caught my attention - not from the tool I am using to track the state of testing, which is where I would like it. As an example of the problem I have with disabling tests with {{@require}}, I just now removed this disabling for CASSANDRA-8049 CASSANDRA-9300 and CASSANDRA-9194 - the committers of those may or may not have known these dtests were disabled, I don't know. So I understand your point, but I would kindly request that you leave failing tests to fail in CI. It is much easier for me to help with tracking the state of tests. was (Author: mshuler): Speaking for myself and my own workflow, the only way to see what tests are failing is to run them continually in CI. CI is the tool I am using to track the tests that are failing repeatedly, intermittently failing, or blowing up everything. If they are hidden from my view and I don't see them fail in CI, then they don't get brought to my attention. I volunteered to track the testing to try to get to the goal of 100% passing, and I found this particular ticket from a git commit message that caught my attention - not from the tool I am using to track the state of testing, which is where I would like it. As an example of the problem I have with disabling tests with {{@require}}, I just now removed this disabling for CASSANDRA-8049 CASSANDRA-9300 and CASSANDRA-9294 - the committers of those may or may not have known these dtests were disabled, I don't know. So I understand your point, but I would kindly request that you leave failing tests to fail in CI. It is much easier for me to help with tracking the state of tests. HSHA dtest for closing connections is failing on trunk -- Key: CASSANDRA-9369 URL: https://issues.apache.org/jira/browse/CASSANDRA-9369 Project: Cassandra Issue Type: Test Components: Tests Reporter: Tyler Hobbs Assignee: Ariel Weisberg Fix For: 2.2.x The {{thrift_hsha_test.ThriftHSHATest.test_closing_connections}} dtest is failing against trunk (but not against 2.1). Here's an example failure: {noformat} Error Message There are non-closed connections: COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java28202 automaton7u IPv4 60767 0t0 TCP localhost:58436-localhost:9160 (CLOSE_WAIT) begin captured logging dtest: DEBUG: cluster ccm directory: /tmp/dtest-U0pP7H dtest: DEBUG: Creating connection pools.. dtest: DEBUG: Disabling/Enabling thrift iteration #0 dtest: DEBUG: Closing connections from the client side.. pycassa.pool: INFO: Pool 13972199784 was disposed pycassa.pool: INFO: Pool 139721997714512 was disposed pycassa.pool: INFO: Pool 139721997749904 was disposed - end captured logging - Stacktrace File /usr/lib/python2.7/unittest/case.py, line 331, in run testMethod() File /home/automaton/cassandra-dtest/thrift_hsha_test.py, line 59, in test_closing_connections self.assertEqual(len(lines), 0, There are non-closed connections: %s % stdout) File /usr/lib/python2.7/unittest/case.py, line 515, in assertEqual assertion_func(first, second, msg=msg) File /usr/lib/python2.7/unittest/case.py, line 508, in _baseAssertEqual raise self.failureException(msg) 'There are non-closed connections: COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME\njava28202 automaton7u IPv4 60767 0t0 TCP localhost:58436-localhost:9160 (CLOSE_WAIT)\n\n begin captured logging \ndtest: DEBUG: cluster ccm directory: /tmp/dtest-U0pP7H\ndtest: DEBUG: Creating connection pools..\ndtest: DEBUG: Disabling/Enabling thrift iteration #0\ndtest: DEBUG: Closing connections from the client side..\npycassa.pool: INFO: Pool 13972199784 was disposed\npycassa.pool: INFO: Pool 139721997714512 was disposed\npycassa.pool: INFO: Pool 139721997749904 was disposed\n- end captured logging -' {noformat}
[jira] [Updated] (CASSANDRA-8603) Cut tombstone memory footprint in half for cql deletes
[ https://issues.apache.org/jira/browse/CASSANDRA-8603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Yeschenko updated CASSANDRA-8603: - Assignee: Benjamin Lerer (was: Dominic Letz) Cut tombstone memory footprint in half for cql deletes -- Key: CASSANDRA-8603 URL: https://issues.apache.org/jira/browse/CASSANDRA-8603 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Dominic Letz Assignee: Benjamin Lerer Labels: tombstone Attachments: 8603-2.1-V3.txt, cassandra-2.0.11-8603.txt, cassandra-2.1-8603.txt, cassandra-2.1-8603_v2.txt, system.log As CQL does not yet support range deletes every delete from CQL results in a Semi-RangeTombstone which actually has the same start and end values - but until today they are copies. Effectively doubling the required heap memory to store the RangeTombstone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7304) Ability to distinguish between NULL and UNSET values in Prepared Statements
[ https://issues.apache.org/jira/browse/CASSANDRA-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Holmberg updated CASSANDRA-7304: - Labels: client-impacting cql protocolv4 (was: cql protocolv4) Ability to distinguish between NULL and UNSET values in Prepared Statements --- Key: CASSANDRA-7304 URL: https://issues.apache.org/jira/browse/CASSANDRA-7304 Project: Cassandra Issue Type: Sub-task Reporter: Drew Kutcharian Assignee: Oded Peer Labels: client-impacting, cql, protocolv4 Fix For: 2.2.0 beta 1 Attachments: 7304-03.patch, 7304-04.patch, 7304-05.patch, 7304-06.patch, 7304-07.patch, 7304-2.patch, 7304-V8.txt, 7304.patch Currently Cassandra inserts tombstones when a value of a column is bound to NULL in a prepared statement. At higher insert rates managing all these tombstones becomes an unnecessary overhead. This limits the usefulness of the prepared statements since developers have to either create multiple prepared statements (each with a different combination of column names, which at times is just unfeasible because of the sheer number of possible combinations) or fall back to using regular (non-prepared) statements. This JIRA is here to explore the possibility of either: A. Have a flag on prepared statements that once set, tells Cassandra to ignore null columns or B. Have an UNSET value which makes Cassandra skip the null columns and not tombstone them Basically, in the context of a prepared statement, a null value means delete, but we don’t have anything that means ignore (besides creating a new prepared statement without the ignored column). Please refer to the original conversation on DataStax Java Driver mailing list for more background: https://groups.google.com/a/lists.datastax.com/d/topic/java-driver-user/cHE3OOSIXBU/discussion *EDIT 18/12/14 - [~odpeer] Implementation Notes:* The motivation hasn't changed. Protocol version 4 specifies that bind variables do not require having a value when executing a statement. Bind variables without a value are called 'unset'. The 'unset' bind variable is serialized as the int value '-2' without following bytes. \\ \\ * An unset bind variable in an EXECUTE or BATCH request ** On a {{value}} does not modify the value and does not create a tombstone ** On the {{ttl}} clause is treated as 'unlimited' ** On the {{timestamp}} clause is treated as 'now' ** On a map key or a list index throws {{InvalidRequestException}} ** On a {{counter}} increment or decrement operation does not change the counter value, e.g. {{UPDATE my_tab SET c = c - ? WHERE k = 1}} does change the value of counter {{c}} ** On a tuple field or UDT field throws {{InvalidRequestException}} * An unset bind variable in a QUERY request ** On a partition column, clustering column or index column in the {{WHERE}} clause throws {{InvalidRequestException}} ** On the {{limit}} clause is treated as 'unlimited' -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8603) Cut tombstone memory footprint in half for cql deletes
[ https://issues.apache.org/jira/browse/CASSANDRA-8603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Yeschenko updated CASSANDRA-8603: - Reviewer: Aleksey Yeschenko (was: Benjamin Lerer) Cut tombstone memory footprint in half for cql deletes -- Key: CASSANDRA-8603 URL: https://issues.apache.org/jira/browse/CASSANDRA-8603 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Dominic Letz Assignee: Benjamin Lerer Labels: tombstone Attachments: 8603-2.1-V3.txt, cassandra-2.0.11-8603.txt, cassandra-2.1-8603.txt, cassandra-2.1-8603_v2.txt, system.log As CQL does not yet support range deletes every delete from CQL results in a Semi-RangeTombstone which actually has the same start and end values - but until today they are copies. Effectively doubling the required heap memory to store the RangeTombstone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
svn commit: r1680337 - /cassandra/site/publish/doc/cql3/CQL-2.2.html
Author: tylerhobbs Date: Tue May 19 16:37:26 2015 New Revision: 1680337 URL: http://svn.apache.org/r1680337 Log: Add CQL3 docs for Cassandra 2.2 Added: cassandra/site/publish/doc/cql3/CQL-2.2.html Added: cassandra/site/publish/doc/cql3/CQL-2.2.html URL: http://svn.apache.org/viewvc/cassandra/site/publish/doc/cql3/CQL-2.2.html?rev=1680337view=auto == --- cassandra/site/publish/doc/cql3/CQL-2.2.html (added) +++ cassandra/site/publish/doc/cql3/CQL-2.2.html Tue May 19 16:37:26 2015 @@ -0,0 +1,516 @@ +?xml version='1.0' encoding='utf-8' ?!DOCTYPE html PUBLIC -//W3C//DTD XHTML 1.0 Transitional//EN http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd;html xmlns=http://www.w3.org/1999/xhtml;headmeta http-equiv=Content-Type content=text/html; charset=utf-8/titleCQL/title/headbodyplink rel=StyleSheet href=CQL.css type=text/css media=screen/ph1 id=CassandraQueryLanguageCQLv3.3.0Cassandra Query Language (CQL) v3.3.0/h1span id=tableOfContentsol style=list-style: none;lia href=CQL.html#CassandraQueryLanguageCQLv3.3.0Cassandra Query Language (CQL) v3.3.0/aol style=list-style: none;lia href=CQL.html#CQLSyntaxCQL Syntax/aol style=list-style: none;lia href=CQL.html#PreamblePreamble/a/lilia href=CQL.html#ConventionsConventions/a/lilia href=CQL.html#identifiersIdentifiers and keywords/a/lilia href=CQL.html#constantsConstants/a/lilia href=CQL.html#CommentsComments/a/l ilia href=CQL.html#statementsStatements/a/lilia href=CQL.html#preparedStatementPrepared Statement/a/li/ol/lilia href=CQL.html#dataDefinitionData Definition/aol style=list-style: none;lia href=CQL.html#createKeyspaceStmtCREATE KEYSPACE/a/lilia href=CQL.html#useStmtUSE/a/lilia href=CQL.html#alterKeyspaceStmtALTER KEYSPACE/a/lilia href=CQL.html#dropKeyspaceStmtDROP KEYSPACE/a/lilia href=CQL.html#createTableStmtCREATE TABLE/a/lilia href=CQL.html#alterTableStmtALTER TABLE/a/lilia href=CQL.html#dropTableStmtDROP TABLE/a/lilia href=CQL.html#truncateStmtTRUNCATE/a/lilia href=CQL.html#createIndexStmtCREATE INDEX/a/lilia href=CQL.html#dropIndexStmtDROP INDEX/a/lilia href=CQL.html#createTypeStmtCREATE TYPE/a/lilia href=CQL.html#alterTypeStmtALTER TYPE/a/lilia href=CQL.html#dropTypeStmtDROP TYPE/a/lilia href=CQL.html#createTri ggerStmtCREATE TRIGGER/a/lilia href=CQL.html#dropTriggerStmtDROP TRIGGER/a/lilia href=CQL.html#createFunctionStmtCREATE FUNCTION/a/lilia href=CQL.html#dropFunctionStmtDROP FUNCTION/a/lilia href=CQL.html#createAggregateStmtCREATE AGGREGATE/a/lilia href=CQL.html#dropAggregateStmtDROP AGGREGATE/a/li/ol/lilia href=CQL.html#dataManipulationData Manipulation/aol style=list-style: none;lia href=CQL.html#insertStmtINSERT/a/lilia href=CQL.html#updateStmtUPDATE/a/lilia href=CQL.html#deleteStmtDELETE/a/lilia href=CQL.html#batchStmtBATCH/a/li/ol/lilia href=CQL.html#queriesQueries/aol style=list-style: none;lia href=CQL.html#selectStmtSELECT/a/li/ol/lilia href=CQL.html#typesData Types/aol style=list-style: none;lia href=CQL.html#usingtimestampsWorking with timestamps/a/lilia href=CQL.html#usingdatesWorking with dates/a/lili a href=CQL.html#usingtimeWorking with time/a/lilia href=CQL.html#countersCounters/a/lilia href=CQL.html#collectionsWorking with collections/a/li/ol/lilia href=CQL.html#functionsFunctions/aol style=list-style: none;lia href=CQL.html#tokenFunToken/a/lilia href=CQL.html#uuidFunUuid/a/lilia href=CQL.html#timeuuidFunTimeuuid functions/a/lilia href=CQL.html#blobFunBlob conversion functions/a/li/ol/lilia href=CQL.html#udfsUser-Defined Functions/a/lilia href=CQL.html#udasUser-Defined Aggregates/a/lilia href=CQL.html#jsonJSON Support/aol style=list-style: none;lia href=CQL.html#selectJsonSELECT JSON/a/lilia href=CQL.html#insertJsonINSERT JSON/a/lilia href=CQL.html#jsonEncodingJSON Encoding of Cassandra Data Types/a/lilia href=CQL.html#fromJsonThe fromJson() Function/a/lilia href=CQL.html#toJsonThe toJson() Function/a/li/ol /lilia href=CQL.html#appendixAAppendix A: CQL Keywords/a/lilia href=CQL.html#appendixBAppendix B: CQL Reserved Types/a/lilia href=CQL.html#changesChanges/aol style=list-style: none;lia href=CQL.html#a3.3.03.3.0/a/lilia href=CQL.html#a3.2.03.2.0/a/lilia href=CQL.html#a3.1.73.1.7/a/lilia href=CQL.html#a3.1.63.1.6/a/lilia href=CQL.html#a3.1.53.1.5/a/lilia href=CQL.html#a3.1.43.1.4/a/lilia href=CQL.html#a3.1.33.1.3/a/lilia href=CQL.html#a3.1.23.1.2/a/lilia href=CQL.html#a3.1.13.1.1/a/lilia href=CQL.html#a3.1.03.1.0/a/lilia href=CQL.html#a3.0.53.0.5/a/lilia href=CQL.html#a3.0.43.0.4/a/lilia href=CQL.html#a3.0.33.0.3/a/lilia href=CQL.html#a3.0.23.0.2/a/lilia href=CQL.html#a3.0.13.0.1/a/li/ol/lilia href=CQL.html#VersioningVersioning/a/li/ol/li/ol/spanh2 id=CQLSyntaxCQL Syntax/h2h 3 id=PreamblePreamble/h3pThis document describes the Cassandra Query Language (CQL) version 3. CQL v3 is not backward compatible with CQL v2 and differs from
[jira] [Updated] (CASSANDRA-9425) Make node-local schema fully immutable
[ https://issues.apache.org/jira/browse/CASSANDRA-9425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Yeschenko updated CASSANDRA-9425: - Issue Type: Sub-task (was: Improvement) Parent: CASSANDRA-9424 Make node-local schema fully immutable -- Key: CASSANDRA-9425 URL: https://issues.apache.org/jira/browse/CASSANDRA-9425 Project: Cassandra Issue Type: Sub-task Reporter: Aleksey Yeschenko Assignee: Aleksey Yeschenko Fix For: 3.x The way we handle schema changes currently is inherently racy. All of our {{SchemaAlteringStatement}} s perform validation on a schema state that's won't necessarily be there when the statement gets executed and mutates schema. We should make all the *Metadata classes ({{KeyspaceMetadata, TableMetadata}}, {{ColumnMetadata}}, immutable, and local schema persistently snapshottable, with a single top-level {{AtomicReference}} to the current snapshot. Have DDL statements perform validation and transformation on the same state. In pseudo-code, think {code} public interface DDLStatement { /** * Validates that the DDL statement can be applied to the provided schema snapshot. * * @param schema snapshot of schema before executing CREATE KEYSPACE */ void validate(SchemaSnapshot schema); /** * Applies the DDL statement to the provided schema snapshot. * Implies that validate() has already been called on the provided snapshot. * * @param schema snapshot of schema before executing the statement * @return snapshot of schema as it would be after executing the statement */ SchemaSnapshot transform(SchemaSnapshot schema); } {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
svn commit: r1680340 - /cassandra/site/publish/doc/cql3/CQL-2.2.html
Author: tylerhobbs Date: Tue May 19 16:42:12 2015 New Revision: 1680340 URL: http://svn.apache.org/r1680340 Log: Fix links in CQL-2.2.html Modified: cassandra/site/publish/doc/cql3/CQL-2.2.html Modified: cassandra/site/publish/doc/cql3/CQL-2.2.html URL: http://svn.apache.org/viewvc/cassandra/site/publish/doc/cql3/CQL-2.2.html?rev=1680340r1=1680339r2=1680340view=diff == --- cassandra/site/publish/doc/cql3/CQL-2.2.html (original) +++ cassandra/site/publish/doc/cql3/CQL-2.2.html Tue May 19 16:42:12 2015 @@ -1,4 +1,4 @@ -?xml version='1.0' encoding='utf-8' ?!DOCTYPE html PUBLIC -//W3C//DTD XHTML 1.0 Transitional//EN http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd;html xmlns=http://www.w3.org/1999/xhtml;headmeta http-equiv=Content-Type content=text/html; charset=utf-8/titleCQL/title/headbodyplink rel=StyleSheet href=CQL.css type=text/css media=screen/ph1 id=CassandraQueryLanguageCQLv3.3.0Cassandra Query Language (CQL) v3.3.0/h1span id=tableOfContentsol style=list-style: none;lia href=CQL.html#CassandraQueryLanguageCQLv3.3.0Cassandra Query Language (CQL) v3.3.0/aol style=list-style: none;lia href=CQL.html#CQLSyntaxCQL Syntax/aol style=list-style: none;lia href=CQL.html#PreamblePreamble/a/lilia href=CQL.html#ConventionsConventions/a/lilia href=CQL.html#identifiersIdentifiers and keywords/a/lilia href=CQL.html#constantsConstants/a/lilia href=CQL.html#CommentsComments/a/l ilia href=CQL.html#statementsStatements/a/lilia href=CQL.html#preparedStatementPrepared Statement/a/li/ol/lilia href=CQL.html#dataDefinitionData Definition/aol style=list-style: none;lia href=CQL.html#createKeyspaceStmtCREATE KEYSPACE/a/lilia href=CQL.html#useStmtUSE/a/lilia href=CQL.html#alterKeyspaceStmtALTER KEYSPACE/a/lilia href=CQL.html#dropKeyspaceStmtDROP KEYSPACE/a/lilia href=CQL.html#createTableStmtCREATE TABLE/a/lilia href=CQL.html#alterTableStmtALTER TABLE/a/lilia href=CQL.html#dropTableStmtDROP TABLE/a/lilia href=CQL.html#truncateStmtTRUNCATE/a/lilia href=CQL.html#createIndexStmtCREATE INDEX/a/lilia href=CQL.html#dropIndexStmtDROP INDEX/a/lilia href=CQL.html#createTypeStmtCREATE TYPE/a/lilia href=CQL.html#alterTypeStmtALTER TYPE/a/lilia href=CQL.html#dropTypeStmtDROP TYPE/a/lilia href=CQL.html#createTri ggerStmtCREATE TRIGGER/a/lilia href=CQL.html#dropTriggerStmtDROP TRIGGER/a/lilia href=CQL.html#createFunctionStmtCREATE FUNCTION/a/lilia href=CQL.html#dropFunctionStmtDROP FUNCTION/a/lilia href=CQL.html#createAggregateStmtCREATE AGGREGATE/a/lilia href=CQL.html#dropAggregateStmtDROP AGGREGATE/a/li/ol/lilia href=CQL.html#dataManipulationData Manipulation/aol style=list-style: none;lia href=CQL.html#insertStmtINSERT/a/lilia href=CQL.html#updateStmtUPDATE/a/lilia href=CQL.html#deleteStmtDELETE/a/lilia href=CQL.html#batchStmtBATCH/a/li/ol/lilia href=CQL.html#queriesQueries/aol style=list-style: none;lia href=CQL.html#selectStmtSELECT/a/li/ol/lilia href=CQL.html#typesData Types/aol style=list-style: none;lia href=CQL.html#usingtimestampsWorking with timestamps/a/lilia href=CQL.html#usingdatesWorking with dates/a/lili a href=CQL.html#usingtimeWorking with time/a/lilia href=CQL.html#countersCounters/a/lilia href=CQL.html#collectionsWorking with collections/a/li/ol/lilia href=CQL.html#functionsFunctions/aol style=list-style: none;lia href=CQL.html#tokenFunToken/a/lilia href=CQL.html#uuidFunUuid/a/lilia href=CQL.html#timeuuidFunTimeuuid functions/a/lilia href=CQL.html#blobFunBlob conversion functions/a/li/ol/lilia href=CQL.html#udfsUser-Defined Functions/a/lilia href=CQL.html#udasUser-Defined Aggregates/a/lilia href=CQL.html#jsonJSON Support/aol style=list-style: none;lia href=CQL.html#selectJsonSELECT JSON/a/lilia href=CQL.html#insertJsonINSERT JSON/a/lilia href=CQL.html#jsonEncodingJSON Encoding of Cassandra Data Types/a/lilia href=CQL.html#fromJsonThe fromJson() Function/a/lilia href=CQL.html#toJsonThe toJson() Function/a/li/ol /lilia href=CQL.html#appendixAAppendix A: CQL Keywords/a/lilia href=CQL.html#appendixBAppendix B: CQL Reserved Types/a/lilia href=CQL.html#changesChanges/aol style=list-style: none;lia href=CQL.html#a3.3.03.3.0/a/lilia href=CQL.html#a3.2.03.2.0/a/lilia href=CQL.html#a3.1.73.1.7/a/lilia href=CQL.html#a3.1.63.1.6/a/lilia href=CQL.html#a3.1.53.1.5/a/lilia href=CQL.html#a3.1.43.1.4/a/lilia href=CQL.html#a3.1.33.1.3/a/lilia href=CQL.html#a3.1.23.1.2/a/lilia href=CQL.html#a3.1.13.1.1/a/lilia href=CQL.html#a3.1.03.1.0/a/lilia href=CQL.html#a3.0.53.0.5/a/lilia href=CQL.html#a3.0.43.0.4/a/lilia href=CQL.html#a3.0.33.0.3/a/lilia href=CQL.html#a3.0.23.0.2/a/lilia href=CQL.html#a3.0.13.0.1/a/li/ol/lilia href=CQL.html#VersioningVersioning/a/li/ol/li/ol/spanh2 id=CQLSyntaxCQL Syntax/h2h 3 id=PreamblePreamble/h3pThis document describes the Cassandra Query Language (CQL) version 3. CQL v3 is not backward compatible with CQL
[jira] [Commented] (CASSANDRA-9425) Make node-local schema fully immutable
[ https://issues.apache.org/jira/browse/CASSANDRA-9425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14550746#comment-14550746 ] Aleksey Yeschenko commented on CASSANDRA-9425: -- FWIW I have most of the work done, originally as part of CASSANDRA-6717. That said, it's not correct to bundle the two, given that they aren't, strictly speaking, related, and one of them is blocking 3.0beta1. Make node-local schema fully immutable -- Key: CASSANDRA-9425 URL: https://issues.apache.org/jira/browse/CASSANDRA-9425 Project: Cassandra Issue Type: Sub-task Reporter: Aleksey Yeschenko Assignee: Aleksey Yeschenko Fix For: 3.x The way we handle schema changes currently is inherently racy. All of our {{SchemaAlteringStatement}} s perform validation on a schema state that's won't necessarily be there when the statement gets executed and mutates schema. We should make all the *Metadata classes ({{KeyspaceMetadata, TableMetadata}}, {{ColumnMetadata}}, immutable, and local schema persistently snapshottable, with a single top-level {{AtomicReference}} to the current snapshot. Have DDL statements perform validation and transformation on the same state. In pseudo-code, think {code} public interface DDLStatement { /** * Validates that the DDL statement can be applied to the provided schema snapshot. * * @param schema snapshot of schema before executing CREATE KEYSPACE */ void validate(SchemaSnapshot schema); /** * Applies the DDL statement to the provided schema snapshot. * Implies that validate() has already been called on the provided snapshot. * * @param schema snapshot of schema before executing the statement * @return snapshot of schema as it would be after executing the statement */ SchemaSnapshot transform(SchemaSnapshot schema); } {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-6038) Support schema changes in mixed version clusters
[ https://issues.apache.org/jira/browse/CASSANDRA-6038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Yeschenko updated CASSANDRA-6038: - Assignee: (was: Aleksey Yeschenko) Support schema changes in mixed version clusters Key: CASSANDRA-6038 URL: https://issues.apache.org/jira/browse/CASSANDRA-6038 Project: Cassandra Issue Type: Sub-task Reporter: Aleksey Yeschenko Labels: client-impacting Fix For: 3.x CASSANDRA-5845 made it so that schema changes in a major-mixed cluster are not propagated to the minorer-major nodes. This lets us perform backwards-incompatible schema changes in major releases safely - like adding the schema_triggers table, moving all the aliases to schema_columns, getting rid of the deprecated schema columns, etc. Even this limitation might be too strict, however, and should be avoided if possible (starting with at least versioning schema separately from messaging service versioning, and resorting to major-minor schema block only when truly necessary and not for every x-y pair). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (CASSANDRA-9367) Builtin CQL functions should be added to the schema_functions table
[ https://issues.apache.org/jira/browse/CASSANDRA-9367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Yeschenko reopened CASSANDRA-9367: -- Builtin CQL functions should be added to the schema_functions table --- Key: CASSANDRA-9367 URL: https://issues.apache.org/jira/browse/CASSANDRA-9367 Project: Cassandra Issue Type: Improvement Reporter: Michaël Figuière Priority: Minor As CASSANDRA-6717 will not be part of Cassandra 2.2, my comment there got to be considered for the current {{system.schema_functions}} table, to avoid creating a special case just for 2.2: {quote} Builtin CQL functions are not described in the {{system}} keyspace in its current representation, this new schema should include them next to the user defined ones as: * The {{system}} keyspace and its tables are described in the {{system}} keyspace, therefore it would be consistent. * Having builtin CQL functions described there would allow external tools to manipulate all the functions in a similar way. * This would document the available builtin function for users that don't remember the ones available in their current version of Cassandra. {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9367) Expose builtin-in CQL functions via schema data dictionary
[ https://issues.apache.org/jira/browse/CASSANDRA-9367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Yeschenko updated CASSANDRA-9367: - Summary: Expose builtin-in CQL functions via schema data dictionary (was: Builtin CQL functions should be added to the schema_functions table) Expose builtin-in CQL functions via schema data dictionary -- Key: CASSANDRA-9367 URL: https://issues.apache.org/jira/browse/CASSANDRA-9367 Project: Cassandra Issue Type: Improvement Reporter: Michaël Figuière Priority: Minor As CASSANDRA-6717 will not be part of Cassandra 2.2, my comment there got to be considered for the current {{system.schema_functions}} table, to avoid creating a special case just for 2.2: {quote} Builtin CQL functions are not described in the {{system}} keyspace in its current representation, this new schema should include them next to the user defined ones as: * The {{system}} keyspace and its tables are described in the {{system}} keyspace, therefore it would be consistent. * Having builtin CQL functions described there would allow external tools to manipulate all the functions in a similar way. * This would document the available builtin function for users that don't remember the ones available in their current version of Cassandra. {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
svn commit: r1680344 - /cassandra/site/publish/doc/cql3/CQL-2.2.html
Author: tylerhobbs Date: Tue May 19 16:51:55 2015 New Revision: 1680344 URL: http://svn.apache.org/r1680344 Log: Replace Cassandra 3.0 reference with 2.2 Modified: cassandra/site/publish/doc/cql3/CQL-2.2.html Modified: cassandra/site/publish/doc/cql3/CQL-2.2.html URL: http://svn.apache.org/viewvc/cassandra/site/publish/doc/cql3/CQL-2.2.html?rev=1680344r1=1680343r2=1680344view=diff == --- cassandra/site/publish/doc/cql3/CQL-2.2.html (original) +++ cassandra/site/publish/doc/cql3/CQL-2.2.html Tue May 19 16:51:55 2015 @@ -512,5 +512,5 @@ INSERT INTO atable (pk, val) VALUES (2,2 INSERT INTO atable (pk, val) VALUES (3,3); INSERT INTO atable (pk, val) VALUES (4,4); SELECT average(val) FROM atable; -/pre/prep/ppSee a href=#createAggregateStmtcodeCREATE AGGREGATE/code/a and a href=#dropAggregateStmtcodeDROP AGGREGATE/code/a./ph2 id=jsonJSON Support/h2pCassandra 3.0 introduces JSON support to a href=#selectStmtcodeSELECT/code/a and a href=#insertStmtcodeINSERT/code/a statements. This support does not fundamentally alter the CQL API (for example, the schema is still enforced), it simply provides a convenient way to work with JSON documents./ph3 id=selectJsonSELECT JSON/h3pWith codeSELECT/code statements, the new codeJSON/code keyword can be used to return each row as a single codeJSON/code encoded map. The remainder of the codeSELECT/code statment behavior is the same./ppThe result map keys are the same as the column names in a normal result set. For example, a statement like codeSELECT JSON a, ttl(b) FROM .../code would result in a map with keys codea/code and codettl(b)/ code. However, this is one notable exception: for symmetry with codeINSERT JSON/code behavior, case-sensitive column names with upper-case letters will be surrounded with double quotes. For example, codeSELECT JSON myColumn FROM .../code would result in a map key code\myColumn\/code (note the escaped quotes)./ppThe map values will codeJSON/code-encoded representations (as described below) of the result set values./ph3 id=insertJsonINSERT JSON/h3pWith codeINSERT/code statements, the new codeJSON/code keyword can be used to enable inserting a codeJSON/code encoded map as a single row. The format of the codeJSON/code map should generally match that returned by a codeSELECT JSON/code statement on the same table. In particular, case-sensitive column names should be surrounded with double quotes. For example, to insert into a table with two columns named #8220;myKey#8221; and #8220;value#8221;, you would do the following:/ ppre class=samplepreINSERT INTO mytable JSON '{\myKey\: 0, value: 0}' -/pre/prepAny columns which are ommitted from the codeJSON/code map will be defaulted to a codeNULL/code value (which will result in a tombstone being created)./ph3 id=jsonEncodingJSON Encoding of Cassandra Data Types/h3pWhere possible, Cassandra will represent and accept data types in their native codeJSON/code representation. Cassandra will also accept string representations matching the CQL literal format for all data types. The following table describes the encodings that Cassandra will accept in codeINSERT JSON/code values (and codefromJson()/code arguments) as well as the format Cassandra will use when returning data for codeSELECT JSON/code statements (and codefromJson()/code):/ptabletrthtype /ththformats accepted /ththreturn format /ththnotes/th/trtrtdcodeascii/code/tdtdstring /tdtdstring /tdtdUses JSON#8217;s code\u/code character escape/td/tr trtdcodebigint/code /tdtdinteger, string /tdtdinteger /tdtdString must be valid 64 bit integer/td/trtrtdcodeblob/code /tdtdstring /tdtdstring /tdtdString should be 0x followed by an even number of hex digits/td/trtrtdcodeboolean/code /tdtdboolean, string /tdtdboolean /tdtdString must be #8220;true#8221; or false/td/trtrtdcodedate/code /tdtdstring/tdtdstring /tdtdDate in format code-MM-DD/code, timezone UTC/td/trtrtdcodedecimal/code /tdtdinteger, float, string/tdtdfloat/tdtdMay exceed 32 or 64-bit IEEE-754 floating point precision in client-side decoder/td/trtrtdcodedouble/code /tdtdinteger, float, string/tdtdfloat/tdtdString must be valid integer or float/td/trtrtdcodefloat/code/tdtdinteger, float, string/tdtdfloat/tdtdString must be valid integer or float/td/trtrtdcodeinet/code /tdtdstring /tdtdstring /tdtdIPv4 or IPv6 address/td/trtrtdcodeint/code /tdtdinteger, string /tdtdinteger /tdtdString must be valid 32 bit integer/td/trtrtdcodetext/code /tdtdstring /tdtdstring /tdtdUses JSON#8217;s code\u/code character escape/td/trtrtdcodetime/code /tdtdstring /tdtdstring /tdtdTime of day
[jira] [Updated] (CASSANDRA-9367) Expose builtin-in CQL functions via schema data dictionary
[ https://issues.apache.org/jira/browse/CASSANDRA-9367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Yeschenko updated CASSANDRA-9367: - Issue Type: Sub-task (was: Improvement) Parent: CASSANDRA-9424 Expose builtin-in CQL functions via schema data dictionary -- Key: CASSANDRA-9367 URL: https://issues.apache.org/jira/browse/CASSANDRA-9367 Project: Cassandra Issue Type: Sub-task Reporter: Michaël Figuière Priority: Minor Fix For: 3.x As CASSANDRA-6717 will not be part of Cassandra 2.2, my comment there got to be considered for the current {{system.schema_functions}} table, to avoid creating a special case just for 2.2: {quote} Builtin CQL functions are not described in the {{system}} keyspace in its current representation, this new schema should include them next to the user defined ones as: * The {{system}} keyspace and its tables are described in the {{system}} keyspace, therefore it would be consistent. * Having builtin CQL functions described there would allow external tools to manipulate all the functions in a similar way. * This would document the available builtin function for users that don't remember the ones available in their current version of Cassandra. {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)