[jira] [Updated] (CASSANDRA-13959) Add yaml flag for disabling MVs, log warnings on creation

2017-10-26 Thread Blake Eggleston (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Blake Eggleston updated CASSANDRA-13959:

Resolution: Fixed
Status: Resolved  (was: Ready to Commit)

committed as {{b8697441d7a051e7ff68def6aa9cf14bd92ace9e}}

> Add yaml flag for disabling MVs, log warnings on creation
> -
>
> Key: CASSANDRA-13959
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13959
> Project: Cassandra
>  Issue Type: Bug
>  Components: Materialized Views
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Minor
> Fix For: 4.0, 3.0.x, 3.11.x
>
>
> As discussed on dev@, we should give operators the option to disable 
> materialized view creation, and log warnings when they're created.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[1/6] cassandra git commit: Add flag to disable materialized views, and warnings on creation

2017-10-26 Thread bdeggleston 
Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 8fad4cd59 -> b8697441d
  refs/heads/cassandra-3.11 7d26815c5 -> ea443dfe3
  refs/heads/trunk af2c78459 -> 569c6abcf


Add flag to disable materialized views, and warnings on creation

Patch by Blake Eggleston; Reviewed by Aleksey Yeschenko for CASSANDRA-13959


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b8697441
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b8697441
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b8697441

Branch: refs/heads/cassandra-3.0
Commit: b8697441d7a051e7ff68def6aa9cf14bd92ace9e
Parents: 8fad4cd
Author: Blake Eggleston 
Authored: Mon Oct 9 16:40:04 2017 -0700
Committer: Blake Eggleston 
Committed: Thu Oct 26 13:54:37 2017 -0700

--
 CHANGES.txt |  2 +-
 NEWS.txt|  8 
 conf/cassandra.yaml |  4 
 src/java/org/apache/cassandra/config/Config.java|  3 +++
 .../apache/cassandra/config/DatabaseDescriptor.java |  5 +
 .../cql3/statements/CreateViewStatement.java| 16 
 6 files changed, 37 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/b8697441/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 75f2c29..935931c 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,10 +1,10 @@
 3.0.16
+ * Add flag to disable materialized views, and warnings on creation 
(CASSANDRA-13959)
  * Don't let user drop or generally break tables in system_distributed 
(CASSANDRA-13813)
  * Provide a JMX call to sync schema with local storage (CASSANDRA-13954)
  * Mishandling of cells for removed/dropped columns when reading legacy files 
(CASSANDRA-13939)
  * Deserialise sstable metadata in nodetool verify (CASSANDRA-13922)
 
-
 3.0.15
  * Improve TRUNCATE performance (CASSANDRA-13909)
  * Implement short read protection on partition boundaries (CASSANDRA-13595)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/b8697441/NEWS.txt
--
diff --git a/NEWS.txt b/NEWS.txt
index 944857b..60cf77c 100644
--- a/NEWS.txt
+++ b/NEWS.txt
@@ -21,6 +21,14 @@ Upgrading
- Nothing specific to this release, but please see previous upgrading 
sections,
  especially if you are upgrading from 2.2.
 
+Materialized Views
+---
+   - Following a discussion regarding concerns about the design and safety of 
Materialized Views, the C* development
+ community no longer recommends them for production use, and considers 
them experimental. Warnings messages will
+ now be logged when they are created. (See 
https://www.mail-archive.com/dev@cassandra.apache.org/msg11511.html)
+   - An 'enable_materialized_views' flag has been added to cassandra.yaml to 
allow operators to prevent creation of
+ views
+
 3.0.15
 =
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/b8697441/conf/cassandra.yaml
--
diff --git a/conf/cassandra.yaml b/conf/cassandra.yaml
index d77d27a..b783090 100644
--- a/conf/cassandra.yaml
+++ b/conf/cassandra.yaml
@@ -952,6 +952,10 @@ enable_user_defined_functions: false
 # This option has no effect, if enable_user_defined_functions is false.
 enable_scripted_user_defined_functions: false
 
+# Enables materialized view creation on this node.
+# Materialized views are considered experimental and are not recommended for 
production use.
+enable_materialized_views: true
+
 # The default Windows kernel timer and scheduling resolution is 15.6ms for 
power conservation.
 # Lowering this value on Windows can provide much tighter latency and better 
throughput, however
 # some virtualized environments may see a negative performance impact from 
changing this setting

http://git-wip-us.apache.org/repos/asf/cassandra/blob/b8697441/src/java/org/apache/cassandra/config/Config.java
--
diff --git a/src/java/org/apache/cassandra/config/Config.java 
b/src/java/org/apache/cassandra/config/Config.java
index 6a99cd3..64d41bb 100644
--- a/src/java/org/apache/cassandra/config/Config.java
+++ b/src/java/org/apache/cassandra/config/Config.java
@@ -308,6 +308,9 @@ public class Config
 
 public boolean enable_user_defined_functions = false;
 public boolean enable_scripted_user_defined_functions = false;
+
+public boolean enable_materialized_views = true;
+
 /**
  * Optionally disable asynchronous UDF execution.
  * Disabling asynchronous UDF execution also implicitly

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2017-10-26 Thread bdeggleston 
Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ea443dfe
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ea443dfe
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ea443dfe

Branch: refs/heads/cassandra-3.11
Commit: ea443dfe3ed8d154c11856bb9f5189afc3f7236f
Parents: 7d26815 b869744
Author: Blake Eggleston 
Authored: Thu Oct 26 14:03:33 2017 -0700
Committer: Blake Eggleston 
Committed: Thu Oct 26 14:03:33 2017 -0700

--
 CHANGES.txt|  1 +
 NEWS.txt   |  8 
 conf/cassandra.yaml|  4 
 src/java/org/apache/cassandra/config/Config.java   |  3 +++
 .../cassandra/config/DatabaseDescriptor.java   |  5 +
 .../cql3/statements/CreateViewStatement.java   | 17 +
 6 files changed, 38 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/ea443dfe/CHANGES.txt
--
diff --cc CHANGES.txt
index 532c958,935931c..275294f
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,8 -1,5 +1,9 @@@
 -3.0.16
 +3.11.2
 + * Round buffer size to powers of 2 for the chunk cache (CASSANDRA-13897)
 + * Update jackson JSON jars (CASSANDRA-13949)
 + * Avoid locks when checking LCS fanout and if we should defrag 
(CASSANDRA-13930)
 +Merged from 3.0:
+  * Add flag to disable materialized views, and warnings on creation 
(CASSANDRA-13959)
   * Don't let user drop or generally break tables in system_distributed 
(CASSANDRA-13813)
   * Provide a JMX call to sync schema with local storage (CASSANDRA-13954)
   * Mishandling of cells for removed/dropped columns when reading legacy files 
(CASSANDRA-13939)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ea443dfe/NEWS.txt
--
diff --cc NEWS.txt
index 96285c7,60cf77c..f8dd93b
--- a/NEWS.txt
+++ b/NEWS.txt
@@@ -18,10 -18,19 +18,18 @@@ using the provided 'sstableupgrade' too
  
  Upgrading
  -
 -   - Nothing specific to this release, but please see previous upgrading 
sections,
 - especially if you are upgrading from 2.2.
 +- Nothing specific to this release, but please see previous upgrading 
sections.
  
+ Materialized Views
+ ---
+- Following a discussion regarding concerns about the design and safety of 
Materialized Views, the C* development
+  community no longer recommends them for production use, and considers 
them experimental. Warnings messages will
+  now be logged when they are created. (See 
https://www.mail-archive.com/dev@cassandra.apache.org/msg11511.html)
+- An 'enable_materialized_views' flag has been added to cassandra.yaml to 
allow operators to prevent creation of
+  views
+ 
 -3.0.15
 -=
 +3.11.1
 +==
  
  Upgrading
  -

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ea443dfe/conf/cassandra.yaml
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ea443dfe/src/java/org/apache/cassandra/config/Config.java
--
diff --cc src/java/org/apache/cassandra/config/Config.java
index 9489af9,64d41bb..a01203c
--- a/src/java/org/apache/cassandra/config/Config.java
+++ b/src/java/org/apache/cassandra/config/Config.java
@@@ -323,22 -303,14 +323,25 @@@ public class Confi
   */
  public static final int otc_backlog_expiration_interval_ms_default = 200;
  public volatile int otc_backlog_expiration_interval_ms = 
otc_backlog_expiration_interval_ms_default;
 - 
 +
  public int windows_timer_interval = 0;
  
 +/**
 + * Size of the CQL prepared statements cache in MB.
 + * Defaults to 1/256th of the heap size or 10MB, whichever is greater.
 + */
 +public Long prepared_statements_cache_size_mb = null;
 +/**
 + * Size of the Thrift prepared statements cache in MB.
 + * Defaults to 1/256th of the heap size or 10MB, whichever is greater.
 + */
 +public Long thrift_prepared_statements_cache_size_mb = null;
 +
  public boolean enable_user_defined_functions = false;
  public boolean enable_scripted_user_defined_functions = false;
+ 
+ public boolean enable_materialized_views = true;
+ 
  /**
   * Optionally disable asynchronous UDF execution.
   * Disabling asynchronous UDF execution also implicitly disables the 
security-manager!

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ea443dfe/src/java/org/apache/cassandra/config/DatabaseDescriptor.java

[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

2017-10-26 Thread bdeggleston 
Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/569c6abc
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/569c6abc
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/569c6abc

Branch: refs/heads/trunk
Commit: 569c6abcfa50901fdf73d0b6e89cc1562e698c43
Parents: af2c784 ea443df
Author: Blake Eggleston 
Authored: Thu Oct 26 14:06:24 2017 -0700
Committer: Blake Eggleston 
Committed: Thu Oct 26 14:06:24 2017 -0700

--
 CHANGES.txt|  1 +
 NEWS.txt   |  8 
 conf/cassandra.yaml|  4 
 src/java/org/apache/cassandra/config/Config.java   |  3 +++
 .../cassandra/config/DatabaseDescriptor.java   |  5 +
 .../cql3/statements/CreateViewStatement.java   | 17 +
 6 files changed, 38 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/569c6abc/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/569c6abc/NEWS.txt
--
diff --cc NEWS.txt
index 02a4117,f8dd93b..7a133b8
--- a/NEWS.txt
+++ b/NEWS.txt
@@@ -33,41 -18,16 +33,49 @@@ New feature
  
  Upgrading
  -
 -- Nothing specific to this release, but please see previous upgrading 
sections.
 +- Support for legacy auth tables in the system_auth keyspace (users,
 +  permissions, credentials) and the migration code has been removed. 
Migration
 +  of these legacy auth tables must have been completed before the upgrade 
to
 +  4.0 and the legacy tables must have been removed. See the 'Upgrading' 
section
 +  for version 2.2 for migration instructions.
 +- Cassandra 4.0 removed support for the deprecated Thrift interface. 
Amongst
 +  Tother things, this imply the removal of all yaml option related to 
thrift
 +  ('start_rpc', rpc_port, ...).
 +- Cassandra 4.0 removed support for any pre-3.0 format. This means you
 +  cannot upgrade from a 2.x version to 4.0 directly, you have to upgrade 
to
 +  a 3.0.x/3.x version first (and run upgradesstable). In particular, this
 +  mean Cassandra 4.0 cannot load or read pre-3.0 sstables in any way: you
 +  will need to upgrade those sstable in 3.0.x/3.x first.
 +- Upgrades from 3.0.x or 3.x are supported since 3.0.13 or 3.11.0, 
previous
 +  versions will causes issues during rolling upgrades (CASSANDRA-13274).
 +- Cassandra will no longer allow invalid keyspace replication options, 
such
 +  as invalid datacenter names for NetworkTopologyStrategy. Operators MUST
 +  add new nodes to a datacenter before they can set set ALTER or CREATE
 +  keyspace replication policies using that datacenter. Existing keyspaces
 +  will continue to operate, but CREATE and ALTER will validate that all
 +  datacenters specified exist in the cluster.
 +- Cassandra 4.0 fixes a problem with incremental repair which caused 
repaired
 +  data to be inconsistent between nodes. The fix changes the behavior of 
both
 +  full and incremental repairs. For full repairs, data is no longer marked
 +  repaired. For incremental repairs, anticompaction is run at the 
beginning
 +  of the repair, instead of at the end. If incremental repair was being 
used
 +  prior to upgrading, a full repair should be run after upgrading to 
resolve
 +  any inconsistencies.
 +- Config option index_interval has been removed (it was deprecated since 
2.0)
 +- Deprecated repair JMX APIs are removed.
 +- The version of snappy-java has been upgraded to 1.1.2.6
 +  - the miniumum value for internode message timeouts is 10ms. 
Previously, any
 +positive value was allowed. See cassandra.yaml entries like
 +read_request_timeout_in_ms for more details.
  
+ Materialized Views
+ ---
+- Following a discussion regarding concerns about the design and safety of 
Materialized Views, the C* development
+  community no longer recommends them for production use, and considers 
them experimental. Warnings messages will
+  now be logged when they are created. (See 
https://www.mail-archive.com/dev@cassandra.apache.org/msg11511.html)
+- An 'enable_materialized_views' flag has been added to cassandra.yaml to 
allow operators to prevent creation of
+  views
+ 
  3.11.1
  ==
  

http://git-wip-us.apache.org/repos/asf/cassandra/blob/569c6abc/conf/cassandra.yaml
--

[2/6] cassandra git commit: Add flag to disable materialized views, and warnings on creation

2017-10-26 Thread bdeggleston 
Add flag to disable materialized views, and warnings on creation

Patch by Blake Eggleston; Reviewed by Aleksey Yeschenko for CASSANDRA-13959


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b8697441
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b8697441
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b8697441

Branch: refs/heads/cassandra-3.11
Commit: b8697441d7a051e7ff68def6aa9cf14bd92ace9e
Parents: 8fad4cd
Author: Blake Eggleston 
Authored: Mon Oct 9 16:40:04 2017 -0700
Committer: Blake Eggleston 
Committed: Thu Oct 26 13:54:37 2017 -0700

--
 CHANGES.txt |  2 +-
 NEWS.txt|  8 
 conf/cassandra.yaml |  4 
 src/java/org/apache/cassandra/config/Config.java|  3 +++
 .../apache/cassandra/config/DatabaseDescriptor.java |  5 +
 .../cql3/statements/CreateViewStatement.java| 16 
 6 files changed, 37 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/b8697441/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 75f2c29..935931c 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,10 +1,10 @@
 3.0.16
+ * Add flag to disable materialized views, and warnings on creation 
(CASSANDRA-13959)
  * Don't let user drop or generally break tables in system_distributed 
(CASSANDRA-13813)
  * Provide a JMX call to sync schema with local storage (CASSANDRA-13954)
  * Mishandling of cells for removed/dropped columns when reading legacy files 
(CASSANDRA-13939)
  * Deserialise sstable metadata in nodetool verify (CASSANDRA-13922)
 
-
 3.0.15
  * Improve TRUNCATE performance (CASSANDRA-13909)
  * Implement short read protection on partition boundaries (CASSANDRA-13595)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/b8697441/NEWS.txt
--
diff --git a/NEWS.txt b/NEWS.txt
index 944857b..60cf77c 100644
--- a/NEWS.txt
+++ b/NEWS.txt
@@ -21,6 +21,14 @@ Upgrading
- Nothing specific to this release, but please see previous upgrading 
sections,
  especially if you are upgrading from 2.2.
 
+Materialized Views
+---
+   - Following a discussion regarding concerns about the design and safety of 
Materialized Views, the C* development
+ community no longer recommends them for production use, and considers 
them experimental. Warnings messages will
+ now be logged when they are created. (See 
https://www.mail-archive.com/dev@cassandra.apache.org/msg11511.html)
+   - An 'enable_materialized_views' flag has been added to cassandra.yaml to 
allow operators to prevent creation of
+ views
+
 3.0.15
 =
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/b8697441/conf/cassandra.yaml
--
diff --git a/conf/cassandra.yaml b/conf/cassandra.yaml
index d77d27a..b783090 100644
--- a/conf/cassandra.yaml
+++ b/conf/cassandra.yaml
@@ -952,6 +952,10 @@ enable_user_defined_functions: false
 # This option has no effect, if enable_user_defined_functions is false.
 enable_scripted_user_defined_functions: false
 
+# Enables materialized view creation on this node.
+# Materialized views are considered experimental and are not recommended for 
production use.
+enable_materialized_views: true
+
 # The default Windows kernel timer and scheduling resolution is 15.6ms for 
power conservation.
 # Lowering this value on Windows can provide much tighter latency and better 
throughput, however
 # some virtualized environments may see a negative performance impact from 
changing this setting

http://git-wip-us.apache.org/repos/asf/cassandra/blob/b8697441/src/java/org/apache/cassandra/config/Config.java
--
diff --git a/src/java/org/apache/cassandra/config/Config.java 
b/src/java/org/apache/cassandra/config/Config.java
index 6a99cd3..64d41bb 100644
--- a/src/java/org/apache/cassandra/config/Config.java
+++ b/src/java/org/apache/cassandra/config/Config.java
@@ -308,6 +308,9 @@ public class Config
 
 public boolean enable_user_defined_functions = false;
 public boolean enable_scripted_user_defined_functions = false;
+
+public boolean enable_materialized_views = true;
+
 /**
  * Optionally disable asynchronous UDF execution.
  * Disabling asynchronous UDF execution also implicitly disables the 
security-manager!

http://git-wip-us.apache.org/repos/asf/cassandra/blob/b8697441/src/java/org/apache/cassandra/config/DatabaseDescriptor.java

[3/6] cassandra git commit: Add flag to disable materialized views, and warnings on creation

2017-10-26 Thread bdeggleston 
Add flag to disable materialized views, and warnings on creation

Patch by Blake Eggleston; Reviewed by Aleksey Yeschenko for CASSANDRA-13959


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b8697441
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b8697441
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b8697441

Branch: refs/heads/trunk
Commit: b8697441d7a051e7ff68def6aa9cf14bd92ace9e
Parents: 8fad4cd
Author: Blake Eggleston 
Authored: Mon Oct 9 16:40:04 2017 -0700
Committer: Blake Eggleston 
Committed: Thu Oct 26 13:54:37 2017 -0700

--
 CHANGES.txt |  2 +-
 NEWS.txt|  8 
 conf/cassandra.yaml |  4 
 src/java/org/apache/cassandra/config/Config.java|  3 +++
 .../apache/cassandra/config/DatabaseDescriptor.java |  5 +
 .../cql3/statements/CreateViewStatement.java| 16 
 6 files changed, 37 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/b8697441/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 75f2c29..935931c 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,10 +1,10 @@
 3.0.16
+ * Add flag to disable materialized views, and warnings on creation 
(CASSANDRA-13959)
  * Don't let user drop or generally break tables in system_distributed 
(CASSANDRA-13813)
  * Provide a JMX call to sync schema with local storage (CASSANDRA-13954)
  * Mishandling of cells for removed/dropped columns when reading legacy files 
(CASSANDRA-13939)
  * Deserialise sstable metadata in nodetool verify (CASSANDRA-13922)
 
-
 3.0.15
  * Improve TRUNCATE performance (CASSANDRA-13909)
  * Implement short read protection on partition boundaries (CASSANDRA-13595)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/b8697441/NEWS.txt
--
diff --git a/NEWS.txt b/NEWS.txt
index 944857b..60cf77c 100644
--- a/NEWS.txt
+++ b/NEWS.txt
@@ -21,6 +21,14 @@ Upgrading
- Nothing specific to this release, but please see previous upgrading 
sections,
  especially if you are upgrading from 2.2.
 
+Materialized Views
+---
+   - Following a discussion regarding concerns about the design and safety of 
Materialized Views, the C* development
+ community no longer recommends them for production use, and considers 
them experimental. Warnings messages will
+ now be logged when they are created. (See 
https://www.mail-archive.com/dev@cassandra.apache.org/msg11511.html)
+   - An 'enable_materialized_views' flag has been added to cassandra.yaml to 
allow operators to prevent creation of
+ views
+
 3.0.15
 =
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/b8697441/conf/cassandra.yaml
--
diff --git a/conf/cassandra.yaml b/conf/cassandra.yaml
index d77d27a..b783090 100644
--- a/conf/cassandra.yaml
+++ b/conf/cassandra.yaml
@@ -952,6 +952,10 @@ enable_user_defined_functions: false
 # This option has no effect, if enable_user_defined_functions is false.
 enable_scripted_user_defined_functions: false
 
+# Enables materialized view creation on this node.
+# Materialized views are considered experimental and are not recommended for 
production use.
+enable_materialized_views: true
+
 # The default Windows kernel timer and scheduling resolution is 15.6ms for 
power conservation.
 # Lowering this value on Windows can provide much tighter latency and better 
throughput, however
 # some virtualized environments may see a negative performance impact from 
changing this setting

http://git-wip-us.apache.org/repos/asf/cassandra/blob/b8697441/src/java/org/apache/cassandra/config/Config.java
--
diff --git a/src/java/org/apache/cassandra/config/Config.java 
b/src/java/org/apache/cassandra/config/Config.java
index 6a99cd3..64d41bb 100644
--- a/src/java/org/apache/cassandra/config/Config.java
+++ b/src/java/org/apache/cassandra/config/Config.java
@@ -308,6 +308,9 @@ public class Config
 
 public boolean enable_user_defined_functions = false;
 public boolean enable_scripted_user_defined_functions = false;
+
+public boolean enable_materialized_views = true;
+
 /**
  * Optionally disable asynchronous UDF execution.
  * Disabling asynchronous UDF execution also implicitly disables the 
security-manager!

http://git-wip-us.apache.org/repos/asf/cassandra/blob/b8697441/src/java/org/apache/cassandra/config/DatabaseDescriptor.java

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2017-10-26 Thread bdeggleston 
Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ea443dfe
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ea443dfe
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ea443dfe

Branch: refs/heads/trunk
Commit: ea443dfe3ed8d154c11856bb9f5189afc3f7236f
Parents: 7d26815 b869744
Author: Blake Eggleston 
Authored: Thu Oct 26 14:03:33 2017 -0700
Committer: Blake Eggleston 
Committed: Thu Oct 26 14:03:33 2017 -0700

--
 CHANGES.txt|  1 +
 NEWS.txt   |  8 
 conf/cassandra.yaml|  4 
 src/java/org/apache/cassandra/config/Config.java   |  3 +++
 .../cassandra/config/DatabaseDescriptor.java   |  5 +
 .../cql3/statements/CreateViewStatement.java   | 17 +
 6 files changed, 38 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/ea443dfe/CHANGES.txt
--
diff --cc CHANGES.txt
index 532c958,935931c..275294f
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,8 -1,5 +1,9 @@@
 -3.0.16
 +3.11.2
 + * Round buffer size to powers of 2 for the chunk cache (CASSANDRA-13897)
 + * Update jackson JSON jars (CASSANDRA-13949)
 + * Avoid locks when checking LCS fanout and if we should defrag 
(CASSANDRA-13930)
 +Merged from 3.0:
+  * Add flag to disable materialized views, and warnings on creation 
(CASSANDRA-13959)
   * Don't let user drop or generally break tables in system_distributed 
(CASSANDRA-13813)
   * Provide a JMX call to sync schema with local storage (CASSANDRA-13954)
   * Mishandling of cells for removed/dropped columns when reading legacy files 
(CASSANDRA-13939)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ea443dfe/NEWS.txt
--
diff --cc NEWS.txt
index 96285c7,60cf77c..f8dd93b
--- a/NEWS.txt
+++ b/NEWS.txt
@@@ -18,10 -18,19 +18,18 @@@ using the provided 'sstableupgrade' too
  
  Upgrading
  -
 -   - Nothing specific to this release, but please see previous upgrading 
sections,
 - especially if you are upgrading from 2.2.
 +- Nothing specific to this release, but please see previous upgrading 
sections.
  
+ Materialized Views
+ ---
+- Following a discussion regarding concerns about the design and safety of 
Materialized Views, the C* development
+  community no longer recommends them for production use, and considers 
them experimental. Warnings messages will
+  now be logged when they are created. (See 
https://www.mail-archive.com/dev@cassandra.apache.org/msg11511.html)
+- An 'enable_materialized_views' flag has been added to cassandra.yaml to 
allow operators to prevent creation of
+  views
+ 
 -3.0.15
 -=
 +3.11.1
 +==
  
  Upgrading
  -

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ea443dfe/conf/cassandra.yaml
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ea443dfe/src/java/org/apache/cassandra/config/Config.java
--
diff --cc src/java/org/apache/cassandra/config/Config.java
index 9489af9,64d41bb..a01203c
--- a/src/java/org/apache/cassandra/config/Config.java
+++ b/src/java/org/apache/cassandra/config/Config.java
@@@ -323,22 -303,14 +323,25 @@@ public class Confi
   */
  public static final int otc_backlog_expiration_interval_ms_default = 200;
  public volatile int otc_backlog_expiration_interval_ms = 
otc_backlog_expiration_interval_ms_default;
 - 
 +
  public int windows_timer_interval = 0;
  
 +/**
 + * Size of the CQL prepared statements cache in MB.
 + * Defaults to 1/256th of the heap size or 10MB, whichever is greater.
 + */
 +public Long prepared_statements_cache_size_mb = null;
 +/**
 + * Size of the Thrift prepared statements cache in MB.
 + * Defaults to 1/256th of the heap size or 10MB, whichever is greater.
 + */
 +public Long thrift_prepared_statements_cache_size_mb = null;
 +
  public boolean enable_user_defined_functions = false;
  public boolean enable_scripted_user_defined_functions = false;
+ 
+ public boolean enable_materialized_views = true;
+ 
  /**
   * Optionally disable asynchronous UDF execution.
   * Disabling asynchronous UDF execution also implicitly disables the 
security-manager!

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ea443dfe/src/java/org/apache/cassandra/config/DatabaseDescriptor.java

[3/6] cassandra git commit: fixing CASSANDRA-13123 test

2017-10-26 Thread bdeggleston 
fixing CASSANDRA-13123 test


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/8fad4cd5
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/8fad4cd5
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/8fad4cd5

Branch: refs/heads/trunk
Commit: 8fad4cd59ea5ef6d111cfe67d9a4c0345c4d7fd7
Parents: 9e37967
Author: Blake Eggleston 
Authored: Thu Oct 26 11:05:02 2017 -0700
Committer: Blake Eggleston 
Committed: Thu Oct 26 13:33:35 2017 -0700

--
 .../CommitLogSegmentBackpressureTest.java   | 140 
 .../commitlog/CommitLogSegmentManagerTest.java  | 167 ---
 .../db/commitlog/CommitlogShutdownTest.java |  98 +++
 3 files changed, 238 insertions(+), 167 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/8fad4cd5/test/unit/org/apache/cassandra/db/commitlog/CommitLogSegmentBackpressureTest.java
--
diff --git 
a/test/unit/org/apache/cassandra/db/commitlog/CommitLogSegmentBackpressureTest.java
 
b/test/unit/org/apache/cassandra/db/commitlog/CommitLogSegmentBackpressureTest.java
new file mode 100644
index 000..b651098
--- /dev/null
+++ 
b/test/unit/org/apache/cassandra/db/commitlog/CommitLogSegmentBackpressureTest.java
@@ -0,0 +1,140 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+package org.apache.cassandra.db.commitlog;
+
+import java.nio.ByteBuffer;
+import java.util.Random;
+import java.util.concurrent.Semaphore;
+
+import com.google.common.collect.ImmutableMap;
+
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+
+import org.apache.cassandra.SchemaLoader;
+import org.apache.cassandra.Util;
+import org.apache.cassandra.config.Config.CommitLogSync;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.config.ParameterizedClass;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.db.Mutation;
+import org.apache.cassandra.db.RowUpdateBuilder;
+import org.apache.cassandra.db.compaction.CompactionManager;
+import org.apache.cassandra.db.marshal.AsciiType;
+import org.apache.cassandra.db.marshal.BytesType;
+import org.apache.cassandra.schema.KeyspaceParams;
+import org.jboss.byteman.contrib.bmunit.BMRule;
+import org.jboss.byteman.contrib.bmunit.BMRules;
+import org.jboss.byteman.contrib.bmunit.BMUnitRunner;
+
+/**
+ * Since this test depends on byteman rules being setup during initialization, 
you shouldn't add tests to this class
+ */
+@RunWith(BMUnitRunner.class)
+public class CommitLogSegmentBackpressureTest
+{
+//Block commit log service from syncing
+private static final Semaphore allowSync = new Semaphore(1);
+
+private static final String KEYSPACE1 = "CommitLogTest";
+private static final String STANDARD1 = "Standard1";
+private static final String STANDARD2 = "Standard2";
+
+private final static byte[] entropy = new byte[1024 * 256];
+
+@Test
+@BMRules(rules = {@BMRule(name = "Acquire Semaphore before sync",
+  targetClass = "AbstractCommitLogService$1",
+  targetMethod = "run",
+  targetLocation = "AT INVOKE 
org.apache.cassandra.db.commitlog.CommitLog.sync",
+  action = 
"org.apache.cassandra.db.commitlog.CommitLogSegmentBackpressureTest.allowSync.acquire()"),
+  @BMRule(name = "Release Semaphore after sync",
+  targetClass = "AbstractCommitLogService$1",
+  targetMethod = "run",
+  targetLocation = "AFTER INVOKE 
org.apache.cassandra.db.commitlog.CommitLog.sync",
+  action = 
"org.apache.cassandra.db.commitlog.CommitLogSegmentBackpressureTest.allowSync.release()")})
+public void testCompressedCommitLogBackpressure()

[jira] [Resolved] (CASSANDRA-13123) Draining a node might fail to delete all inactive commitlogs

2017-10-26 Thread Blake Eggleston (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Blake Eggleston resolved CASSANDRA-13123.
-
   Resolution: Fixed
Reproduced In: 3.0.10, 2.2.8  (was: 2.2.8, 3.0.10)

committed as {{8fad4cd59ea5ef6d111cfe67d9a4c0345c4d7fd7}}

> Draining a node might fail to delete all inactive commitlogs
> 
>
> Key: CASSANDRA-13123
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13123
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths
>Reporter: Jan Urbański
>Assignee: Jan Urbański
> Fix For: 3.0.16, 3.11.2, 4.0
>
> Attachments: 13123-2.2.8.txt, 13123-3.0.10.txt, 13123-3.9.txt, 
> 13123-trunk.txt
>
>
> After issuing a drain command, it's possible that not all of the inactive 
> commitlogs are removed.
> The drain command shuts down the CommitLog instance, which in turn shuts down 
> the CommitLogSegmentManager. This has the effect of discarding any pending 
> management tasks it might have, like the removal of inactive commitlogs.
> This in turn leads to an excessive amount of commitlogs being left behind 
> after a drain and a lengthy recovery after a restart. With a fleet of dozens 
> of nodes, each of them leaving several GB of commitlogs after a drain and 
> taking up to two minutes to recover them on restart, the additional time 
> required to restart the entire fleet becomes noticeable.
> This problem is not present in 3.x or trunk because of the CLSM rewrite done 
> in CASSANDRA-8844.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[2/6] cassandra git commit: fixing CASSANDRA-13123 test

2017-10-26 Thread bdeggleston 
fixing CASSANDRA-13123 test


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/8fad4cd5
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/8fad4cd5
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/8fad4cd5

Branch: refs/heads/cassandra-3.11
Commit: 8fad4cd59ea5ef6d111cfe67d9a4c0345c4d7fd7
Parents: 9e37967
Author: Blake Eggleston 
Authored: Thu Oct 26 11:05:02 2017 -0700
Committer: Blake Eggleston 
Committed: Thu Oct 26 13:33:35 2017 -0700

--
 .../CommitLogSegmentBackpressureTest.java   | 140 
 .../commitlog/CommitLogSegmentManagerTest.java  | 167 ---
 .../db/commitlog/CommitlogShutdownTest.java |  98 +++
 3 files changed, 238 insertions(+), 167 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/8fad4cd5/test/unit/org/apache/cassandra/db/commitlog/CommitLogSegmentBackpressureTest.java
--
diff --git 
a/test/unit/org/apache/cassandra/db/commitlog/CommitLogSegmentBackpressureTest.java
 
b/test/unit/org/apache/cassandra/db/commitlog/CommitLogSegmentBackpressureTest.java
new file mode 100644
index 000..b651098
--- /dev/null
+++ 
b/test/unit/org/apache/cassandra/db/commitlog/CommitLogSegmentBackpressureTest.java
@@ -0,0 +1,140 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+package org.apache.cassandra.db.commitlog;
+
+import java.nio.ByteBuffer;
+import java.util.Random;
+import java.util.concurrent.Semaphore;
+
+import com.google.common.collect.ImmutableMap;
+
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+
+import org.apache.cassandra.SchemaLoader;
+import org.apache.cassandra.Util;
+import org.apache.cassandra.config.Config.CommitLogSync;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.config.ParameterizedClass;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.db.Mutation;
+import org.apache.cassandra.db.RowUpdateBuilder;
+import org.apache.cassandra.db.compaction.CompactionManager;
+import org.apache.cassandra.db.marshal.AsciiType;
+import org.apache.cassandra.db.marshal.BytesType;
+import org.apache.cassandra.schema.KeyspaceParams;
+import org.jboss.byteman.contrib.bmunit.BMRule;
+import org.jboss.byteman.contrib.bmunit.BMRules;
+import org.jboss.byteman.contrib.bmunit.BMUnitRunner;
+
+/**
+ * Since this test depends on byteman rules being setup during initialization, 
you shouldn't add tests to this class
+ */
+@RunWith(BMUnitRunner.class)
+public class CommitLogSegmentBackpressureTest
+{
+//Block commit log service from syncing
+private static final Semaphore allowSync = new Semaphore(1);
+
+private static final String KEYSPACE1 = "CommitLogTest";
+private static final String STANDARD1 = "Standard1";
+private static final String STANDARD2 = "Standard2";
+
+private final static byte[] entropy = new byte[1024 * 256];
+
+@Test
+@BMRules(rules = {@BMRule(name = "Acquire Semaphore before sync",
+  targetClass = "AbstractCommitLogService$1",
+  targetMethod = "run",
+  targetLocation = "AT INVOKE 
org.apache.cassandra.db.commitlog.CommitLog.sync",
+  action = 
"org.apache.cassandra.db.commitlog.CommitLogSegmentBackpressureTest.allowSync.acquire()"),
+  @BMRule(name = "Release Semaphore after sync",
+  targetClass = "AbstractCommitLogService$1",
+  targetMethod = "run",
+  targetLocation = "AFTER INVOKE 
org.apache.cassandra.db.commitlog.CommitLog.sync",
+  action = 
"org.apache.cassandra.db.commitlog.CommitLogSegmentBackpressureTest.allowSync.release()")})
+public void

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2017-10-26 Thread bdeggleston 
Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7d26815c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7d26815c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7d26815c

Branch: refs/heads/cassandra-3.11
Commit: 7d26815c5e270c4e142df868ab8e9c44be022595
Parents: 5b23054 8fad4cd
Author: Blake Eggleston 
Authored: Thu Oct 26 13:35:21 2017 -0700
Committer: Blake Eggleston 
Committed: Thu Oct 26 13:37:00 2017 -0700

--
 .../CommitLogSegmentBackpressureTest.java   | 145 
 .../commitlog/CommitLogSegmentManagerTest.java  | 172 ---
 .../db/commitlog/CommitlogShutdownTest.java |  99 +++
 3 files changed, 244 insertions(+), 172 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/7d26815c/test/unit/org/apache/cassandra/db/commitlog/CommitLogSegmentBackpressureTest.java
--
diff --cc 
test/unit/org/apache/cassandra/db/commitlog/CommitLogSegmentBackpressureTest.java
index 000,b651098..3956de5
mode 00,100644..100644
--- 
a/test/unit/org/apache/cassandra/db/commitlog/CommitLogSegmentBackpressureTest.java
+++ 
b/test/unit/org/apache/cassandra/db/commitlog/CommitLogSegmentBackpressureTest.java
@@@ -1,0 -1,140 +1,145 @@@
+ /*
+  *
+  * Licensed to the Apache Software Foundation (ASF) under one
+  * or more contributor license agreements.  See the NOTICE file
+  * distributed with this work for additional information
+  * regarding copyright ownership.  The ASF licenses this file
+  * to you under the Apache License, Version 2.0 (the
+  * "License"); you may not use this file except in compliance
+  * with the License.  You may obtain a copy of the License at
+  *
+  *   http://www.apache.org/licenses/LICENSE-2.0
+  *
+  * Unless required by applicable law or agreed to in writing,
+  * software distributed under the License is distributed on an
+  * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  * KIND, either express or implied.  See the License for the
+  * specific language governing permissions and limitations
+  * under the License.
+  *
+  */
+ package org.apache.cassandra.db.commitlog;
+ 
+ import java.nio.ByteBuffer;
++import java.util.ArrayList;
+ import java.util.Random;
+ import java.util.concurrent.Semaphore;
+ 
+ import com.google.common.collect.ImmutableMap;
+ 
+ import org.junit.Assert;
+ import org.junit.Test;
+ import org.junit.runner.RunWith;
+ 
+ import org.apache.cassandra.SchemaLoader;
+ import org.apache.cassandra.Util;
+ import org.apache.cassandra.config.Config.CommitLogSync;
+ import org.apache.cassandra.config.DatabaseDescriptor;
+ import org.apache.cassandra.config.ParameterizedClass;
+ import org.apache.cassandra.db.ColumnFamilyStore;
+ import org.apache.cassandra.db.Keyspace;
+ import org.apache.cassandra.db.Mutation;
+ import org.apache.cassandra.db.RowUpdateBuilder;
+ import org.apache.cassandra.db.compaction.CompactionManager;
+ import org.apache.cassandra.db.marshal.AsciiType;
+ import org.apache.cassandra.db.marshal.BytesType;
+ import org.apache.cassandra.schema.KeyspaceParams;
+ import org.jboss.byteman.contrib.bmunit.BMRule;
+ import org.jboss.byteman.contrib.bmunit.BMRules;
+ import org.jboss.byteman.contrib.bmunit.BMUnitRunner;
+ 
+ /**
+  * Since this test depends on byteman rules being setup during 
initialization, you shouldn't add tests to this class
+  */
+ @RunWith(BMUnitRunner.class)
+ public class CommitLogSegmentBackpressureTest
+ {
+ //Block commit log service from syncing
+ private static final Semaphore allowSync = new Semaphore(1);
+ 
+ private static final String KEYSPACE1 = "CommitLogTest";
+ private static final String STANDARD1 = "Standard1";
+ private static final String STANDARD2 = "Standard2";
+ 
+ private final static byte[] entropy = new byte[1024 * 256];
+ 
+ @Test
+ @BMRules(rules = {@BMRule(name = "Acquire Semaphore before sync",
+   targetClass = "AbstractCommitLogService$1",
+   targetMethod = "run",
+   targetLocation = "AT INVOKE 
org.apache.cassandra.db.commitlog.CommitLog.sync",
+   action = 
"org.apache.cassandra.db.commitlog.CommitLogSegmentBackpressureTest.allowSync.acquire()"),
+   @BMRule(name = "Release Semaphore after sync",
+   targetClass = "AbstractCommitLogService$1",
+   targetMethod = "run",
+   targetLocation = "AFTER INVOKE 
org.apache.cassandra.db.commitlog.CommitLog.sync",
+   action =

[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

2017-10-26 Thread bdeggleston 
Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/af2c7845
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/af2c7845
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/af2c7845

Branch: refs/heads/trunk
Commit: af2c78459ad69b3263620fbb5da484f1186133d1
Parents: 2b507c0 7d26815
Author: Blake Eggleston 
Authored: Thu Oct 26 13:37:50 2017 -0700
Committer: Blake Eggleston 
Committed: Thu Oct 26 13:38:18 2017 -0700

--
 .../CommitLogSegmentBackpressureTest.java   | 146 
 .../commitlog/CommitLogSegmentManagerTest.java  | 173 ---
 .../db/commitlog/CommitlogShutdownTest.java |  99 +++
 3 files changed, 245 insertions(+), 173 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/af2c7845/test/unit/org/apache/cassandra/db/commitlog/CommitLogSegmentBackpressureTest.java
--
diff --cc 
test/unit/org/apache/cassandra/db/commitlog/CommitLogSegmentBackpressureTest.java
index 000,3956de5..f4a63a9
mode 00,100644..100644
--- 
a/test/unit/org/apache/cassandra/db/commitlog/CommitLogSegmentBackpressureTest.java
+++ 
b/test/unit/org/apache/cassandra/db/commitlog/CommitLogSegmentBackpressureTest.java
@@@ -1,0 -1,145 +1,146 @@@
+ /*
+  *
+  * Licensed to the Apache Software Foundation (ASF) under one
+  * or more contributor license agreements.  See the NOTICE file
+  * distributed with this work for additional information
+  * regarding copyright ownership.  The ASF licenses this file
+  * to you under the Apache License, Version 2.0 (the
+  * "License"); you may not use this file except in compliance
+  * with the License.  You may obtain a copy of the License at
+  *
+  *   http://www.apache.org/licenses/LICENSE-2.0
+  *
+  * Unless required by applicable law or agreed to in writing,
+  * software distributed under the License is distributed on an
+  * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  * KIND, either express or implied.  See the License for the
+  * specific language governing permissions and limitations
+  * under the License.
+  *
+  */
+ package org.apache.cassandra.db.commitlog;
+ 
+ import java.nio.ByteBuffer;
+ import java.util.ArrayList;
+ import java.util.Random;
+ import java.util.concurrent.Semaphore;
+ 
+ import com.google.common.collect.ImmutableMap;
+ 
+ import org.junit.Assert;
+ import org.junit.Test;
+ import org.junit.runner.RunWith;
+ 
+ import org.apache.cassandra.SchemaLoader;
+ import org.apache.cassandra.Util;
+ import org.apache.cassandra.config.Config.CommitLogSync;
+ import org.apache.cassandra.config.DatabaseDescriptor;
+ import org.apache.cassandra.config.ParameterizedClass;
+ import org.apache.cassandra.db.ColumnFamilyStore;
+ import org.apache.cassandra.db.Keyspace;
+ import org.apache.cassandra.db.Mutation;
+ import org.apache.cassandra.db.RowUpdateBuilder;
+ import org.apache.cassandra.db.compaction.CompactionManager;
+ import org.apache.cassandra.db.marshal.AsciiType;
+ import org.apache.cassandra.db.marshal.BytesType;
+ import org.apache.cassandra.schema.KeyspaceParams;
++import org.apache.cassandra.schema.TableId;
+ import org.jboss.byteman.contrib.bmunit.BMRule;
+ import org.jboss.byteman.contrib.bmunit.BMRules;
+ import org.jboss.byteman.contrib.bmunit.BMUnitRunner;
+ 
+ /**
+  * Since this test depends on byteman rules being setup during 
initialization, you shouldn't add tests to this class
+  */
+ @RunWith(BMUnitRunner.class)
+ public class CommitLogSegmentBackpressureTest
+ {
+ //Block commit log service from syncing
+ private static final Semaphore allowSync = new Semaphore(1);
+ 
+ private static final String KEYSPACE1 = "CommitLogTest";
+ private static final String STANDARD1 = "Standard1";
+ private static final String STANDARD2 = "Standard2";
+ 
+ private final static byte[] entropy = new byte[1024 * 256];
+ 
+ @Test
+ @BMRules(rules = {@BMRule(name = "Acquire Semaphore before sync",
+   targetClass = "AbstractCommitLogService$1",
+   targetMethod = "run",
+   targetLocation = "AT INVOKE 
org.apache.cassandra.db.commitlog.CommitLog.sync",
+   action = 
"org.apache.cassandra.db.commitlog.CommitLogSegmentBackpressureTest.allowSync.acquire()"),
+   @BMRule(name = "Release Semaphore after sync",
+   targetClass = "AbstractCommitLogService$1",
+   targetMethod = "run",
+   targetLocation = "AFTER INVOKE 
org.apache.cassandra.db.commitlog.CommitLog.sync",
+

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2017-10-26 Thread bdeggleston 
Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7d26815c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7d26815c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7d26815c

Branch: refs/heads/trunk
Commit: 7d26815c5e270c4e142df868ab8e9c44be022595
Parents: 5b23054 8fad4cd
Author: Blake Eggleston 
Authored: Thu Oct 26 13:35:21 2017 -0700
Committer: Blake Eggleston 
Committed: Thu Oct 26 13:37:00 2017 -0700

--
 .../CommitLogSegmentBackpressureTest.java   | 145 
 .../commitlog/CommitLogSegmentManagerTest.java  | 172 ---
 .../db/commitlog/CommitlogShutdownTest.java |  99 +++
 3 files changed, 244 insertions(+), 172 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/7d26815c/test/unit/org/apache/cassandra/db/commitlog/CommitLogSegmentBackpressureTest.java
--
diff --cc 
test/unit/org/apache/cassandra/db/commitlog/CommitLogSegmentBackpressureTest.java
index 000,b651098..3956de5
mode 00,100644..100644
--- 
a/test/unit/org/apache/cassandra/db/commitlog/CommitLogSegmentBackpressureTest.java
+++ 
b/test/unit/org/apache/cassandra/db/commitlog/CommitLogSegmentBackpressureTest.java
@@@ -1,0 -1,140 +1,145 @@@
+ /*
+  *
+  * Licensed to the Apache Software Foundation (ASF) under one
+  * or more contributor license agreements.  See the NOTICE file
+  * distributed with this work for additional information
+  * regarding copyright ownership.  The ASF licenses this file
+  * to you under the Apache License, Version 2.0 (the
+  * "License"); you may not use this file except in compliance
+  * with the License.  You may obtain a copy of the License at
+  *
+  *   http://www.apache.org/licenses/LICENSE-2.0
+  *
+  * Unless required by applicable law or agreed to in writing,
+  * software distributed under the License is distributed on an
+  * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  * KIND, either express or implied.  See the License for the
+  * specific language governing permissions and limitations
+  * under the License.
+  *
+  */
+ package org.apache.cassandra.db.commitlog;
+ 
+ import java.nio.ByteBuffer;
++import java.util.ArrayList;
+ import java.util.Random;
+ import java.util.concurrent.Semaphore;
+ 
+ import com.google.common.collect.ImmutableMap;
+ 
+ import org.junit.Assert;
+ import org.junit.Test;
+ import org.junit.runner.RunWith;
+ 
+ import org.apache.cassandra.SchemaLoader;
+ import org.apache.cassandra.Util;
+ import org.apache.cassandra.config.Config.CommitLogSync;
+ import org.apache.cassandra.config.DatabaseDescriptor;
+ import org.apache.cassandra.config.ParameterizedClass;
+ import org.apache.cassandra.db.ColumnFamilyStore;
+ import org.apache.cassandra.db.Keyspace;
+ import org.apache.cassandra.db.Mutation;
+ import org.apache.cassandra.db.RowUpdateBuilder;
+ import org.apache.cassandra.db.compaction.CompactionManager;
+ import org.apache.cassandra.db.marshal.AsciiType;
+ import org.apache.cassandra.db.marshal.BytesType;
+ import org.apache.cassandra.schema.KeyspaceParams;
+ import org.jboss.byteman.contrib.bmunit.BMRule;
+ import org.jboss.byteman.contrib.bmunit.BMRules;
+ import org.jboss.byteman.contrib.bmunit.BMUnitRunner;
+ 
+ /**
+  * Since this test depends on byteman rules being setup during 
initialization, you shouldn't add tests to this class
+  */
+ @RunWith(BMUnitRunner.class)
+ public class CommitLogSegmentBackpressureTest
+ {
+ //Block commit log service from syncing
+ private static final Semaphore allowSync = new Semaphore(1);
+ 
+ private static final String KEYSPACE1 = "CommitLogTest";
+ private static final String STANDARD1 = "Standard1";
+ private static final String STANDARD2 = "Standard2";
+ 
+ private final static byte[] entropy = new byte[1024 * 256];
+ 
+ @Test
+ @BMRules(rules = {@BMRule(name = "Acquire Semaphore before sync",
+   targetClass = "AbstractCommitLogService$1",
+   targetMethod = "run",
+   targetLocation = "AT INVOKE 
org.apache.cassandra.db.commitlog.CommitLog.sync",
+   action = 
"org.apache.cassandra.db.commitlog.CommitLogSegmentBackpressureTest.allowSync.acquire()"),
+   @BMRule(name = "Release Semaphore after sync",
+   targetClass = "AbstractCommitLogService$1",
+   targetMethod = "run",
+   targetLocation = "AFTER INVOKE 
org.apache.cassandra.db.commitlog.CommitLog.sync",
+   action =

[1/6] cassandra git commit: fixing CASSANDRA-13123 test

2017-10-26 Thread bdeggleston 
Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 9e37967e1 -> 8fad4cd59
  refs/heads/cassandra-3.11 5b23054f1 -> 7d26815c5
  refs/heads/trunk 2b507c03c -> af2c78459


fixing CASSANDRA-13123 test


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/8fad4cd5
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/8fad4cd5
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/8fad4cd5

Branch: refs/heads/cassandra-3.0
Commit: 8fad4cd59ea5ef6d111cfe67d9a4c0345c4d7fd7
Parents: 9e37967
Author: Blake Eggleston 
Authored: Thu Oct 26 11:05:02 2017 -0700
Committer: Blake Eggleston 
Committed: Thu Oct 26 13:33:35 2017 -0700

--
 .../CommitLogSegmentBackpressureTest.java   | 140 
 .../commitlog/CommitLogSegmentManagerTest.java  | 167 ---
 .../db/commitlog/CommitlogShutdownTest.java |  98 +++
 3 files changed, 238 insertions(+), 167 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/8fad4cd5/test/unit/org/apache/cassandra/db/commitlog/CommitLogSegmentBackpressureTest.java
--
diff --git 
a/test/unit/org/apache/cassandra/db/commitlog/CommitLogSegmentBackpressureTest.java
 
b/test/unit/org/apache/cassandra/db/commitlog/CommitLogSegmentBackpressureTest.java
new file mode 100644
index 000..b651098
--- /dev/null
+++ 
b/test/unit/org/apache/cassandra/db/commitlog/CommitLogSegmentBackpressureTest.java
@@ -0,0 +1,140 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+package org.apache.cassandra.db.commitlog;
+
+import java.nio.ByteBuffer;
+import java.util.Random;
+import java.util.concurrent.Semaphore;
+
+import com.google.common.collect.ImmutableMap;
+
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+
+import org.apache.cassandra.SchemaLoader;
+import org.apache.cassandra.Util;
+import org.apache.cassandra.config.Config.CommitLogSync;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.config.ParameterizedClass;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.db.Mutation;
+import org.apache.cassandra.db.RowUpdateBuilder;
+import org.apache.cassandra.db.compaction.CompactionManager;
+import org.apache.cassandra.db.marshal.AsciiType;
+import org.apache.cassandra.db.marshal.BytesType;
+import org.apache.cassandra.schema.KeyspaceParams;
+import org.jboss.byteman.contrib.bmunit.BMRule;
+import org.jboss.byteman.contrib.bmunit.BMRules;
+import org.jboss.byteman.contrib.bmunit.BMUnitRunner;
+
+/**
+ * Since this test depends on byteman rules being setup during initialization, 
you shouldn't add tests to this class
+ */
+@RunWith(BMUnitRunner.class)
+public class CommitLogSegmentBackpressureTest
+{
+//Block commit log service from syncing
+private static final Semaphore allowSync = new Semaphore(1);
+
+private static final String KEYSPACE1 = "CommitLogTest";
+private static final String STANDARD1 = "Standard1";
+private static final String STANDARD2 = "Standard2";
+
+private final static byte[] entropy = new byte[1024 * 256];
+
+@Test
+@BMRules(rules = {@BMRule(name = "Acquire Semaphore before sync",
+  targetClass = "AbstractCommitLogService$1",
+  targetMethod = "run",
+  targetLocation = "AT INVOKE 
org.apache.cassandra.db.commitlog.CommitLog.sync",
+  action = 
"org.apache.cassandra.db.commitlog.CommitLogSegmentBackpressureTest.allowSync.acquire()"),
+  @BMRule(name = "Release Semaphore after sync",
+  targetClass = "AbstractCommitLogService$1",
+  targetMethod = "run",
+  targetLocation = "AFTER INVOKE 
org.apache.cassandra.db.commitlog.CommitLog.sync",

[jira] [Commented] (CASSANDRA-13123) Draining a node might fail to delete all inactive commitlogs

2017-10-26 Thread Jeff Jirsa (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220925#comment-16220925
 ] 

Jeff Jirsa commented on CASSANDRA-13123:


+1 (on splitting, and on your patch)




> Draining a node might fail to delete all inactive commitlogs
> 
>
> Key: CASSANDRA-13123
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13123
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths
>Reporter: Jan Urbański
>Assignee: Jan Urbański
> Fix For: 3.0.16, 3.11.2, 4.0
>
> Attachments: 13123-2.2.8.txt, 13123-3.0.10.txt, 13123-3.9.txt, 
> 13123-trunk.txt
>
>
> After issuing a drain command, it's possible that not all of the inactive 
> commitlogs are removed.
> The drain command shuts down the CommitLog instance, which in turn shuts down 
> the CommitLogSegmentManager. This has the effect of discarding any pending 
> management tasks it might have, like the removal of inactive commitlogs.
> This in turn leads to an excessive amount of commitlogs being left behind 
> after a drain and a lengthy recovery after a restart. With a fleet of dozens 
> of nodes, each of them leaving several GB of commitlogs after a drain and 
> taking up to two minutes to recover them on restart, the additional time 
> required to restart the entire fleet becomes noticeable.
> This problem is not present in 3.x or trunk because of the CLSM rewrite done 
> in CASSANDRA-8844.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13123) Draining a node might fail to delete all inactive commitlogs

2017-10-26 Thread Blake Eggleston (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220919#comment-16220919
 ] 

Blake Eggleston commented on CASSANDRA-13123:
-

I don't think these 2 tests can be in the same test class without being run in 
a specific order. {{testCompressedCommitLogBackpressure}} needs it's byteman 
rules setup before the commit log is started. So if 
{{testShutdownWithPendingTasks}} sets up it's schema and successfully runs 
first, the other will hang.

I have a branch where each test is in it's own class 
[here|https://github.com/bdeggleston/cassandra/tree/13123-fix-3.0], let me know 
if there are any objections

> Draining a node might fail to delete all inactive commitlogs
> 
>
> Key: CASSANDRA-13123
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13123
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths
>Reporter: Jan Urbański
>Assignee: Jan Urbański
> Fix For: 3.0.16, 3.11.2, 4.0
>
> Attachments: 13123-2.2.8.txt, 13123-3.0.10.txt, 13123-3.9.txt, 
> 13123-trunk.txt
>
>
> After issuing a drain command, it's possible that not all of the inactive 
> commitlogs are removed.
> The drain command shuts down the CommitLog instance, which in turn shuts down 
> the CommitLogSegmentManager. This has the effect of discarding any pending 
> management tasks it might have, like the removal of inactive commitlogs.
> This in turn leads to an excessive amount of commitlogs being left behind 
> after a drain and a lengthy recovery after a restart. With a fleet of dozens 
> of nodes, each of them leaving several GB of commitlogs after a drain and 
> taking up to two minutes to recover them on restart, the additional time 
> required to restart the entire fleet becomes noticeable.
> This problem is not present in 3.x or trunk because of the CLSM rewrite done 
> in CASSANDRA-8844.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13830) Simplify MerkleTree.difference/differenceHelper

2017-10-26 Thread Marcus Eriksson (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220414#comment-16220414
 ] 

Marcus Eriksson commented on CASSANDRA-13830:
-

the refactor looks good to me, just a few small nits

* we can remove everything but the constructor from TreeDifference - looks like 
nothing uses the size/rowsOnRight/Left etc. Keeping the class around at all 
could be nice to make it clear we are tracking differences
* {{detectDifferences}} could be private and have the {{@VisibleForTesting}} 
removed

It is a bit scary that the test coverage for {{MerkleTree.difference}} is 
minimal though, we should probably improve that before committing this.

> Simplify MerkleTree.difference/differenceHelper
> ---
>
> Key: CASSANDRA-13830
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13830
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Minor
> Fix For: 4.x
>
>
> As brought up in CASSANDRA-13603, {{MerkleTree.differenceHelper}} is overly 
> complex and difficult to follow for what it's doing. It also shares some of 
> it's responsibilities with {{difference}}, and assumes that the trees it's 
> given have differences, which makes it a potential source of future bugs.
> Since we're just trying to recursively compare these trees and record the 
> largest contiguous out of sync ranges, I think this could be simplified a 
> bit. I propose that we refactor {{difference}} / {{differenceHelper}} so that 
> {{difference}} is only concerned with supplying the range, and dealing with 
> the {{FULLY_INCONSISTENT}} case, and move everything else into a recursable 
> helper method.
> I put together an alternate implementation 
> [here|https://github.com/bdeggleston/cassandra/tree/differencer-cleanup].



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-10726) Read repair inserts should not be blocking

2017-10-26 Thread Aleksey Yeschenko (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220351#comment-16220351
 ] 

Aleksey Yeschenko commented on CASSANDRA-10726:
---

I don't agree that it's *required*, but I'm not talking here about changing the 
default behaviour. I'm talking about providing extra options, while retaining 
the default blocking behaviour.

> Read repair inserts should not be blocking
> --
>
> Key: CASSANDRA-10726
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10726
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Coordination
>Reporter: Richard Low
>Assignee: Xiaolong Jiang
> Fix For: 4.x
>
>
> Today, if there’s a digest mismatch in a foreground read repair, the insert 
> to update out of date replicas is blocking. This means, if it fails, the read 
> fails with a timeout. If a node is dropping writes (maybe it is overloaded or 
> the mutation stage is backed up for some other reason), all reads to a 
> replica set could fail. Further, replicas dropping writes get more out of 
> sync so will require more read repair.
> The comment on the code for why the writes are blocking is:
> {code}
> // wait for the repair writes to be acknowledged, to minimize impact on any 
> replica that's
> // behind on writes in case the out-of-sync row is read multiple times in 
> quick succession
> {code}
> but the bad side effect is that reads timeout. Either the writes should not 
> be blocking or we should return success for the read even if the write times 
> out.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org