[jira] [Updated] (CASSANDRA-12681) Reject empty options and invalid DC names in replication configuration while creating or altering a keyspace.

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-12681:
---
Fix Version/s: (was: 3.0.10)
   (was: 3.10)
   4.0

> Reject empty options and invalid DC names in replication configuration while 
> creating or altering a keyspace.
> -
>
> Key: CASSANDRA-12681
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12681
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Distributed Metadata
>Reporter: Nachiket Patil
>Assignee: Nachiket Patil
>Priority: Minor
> Fix For: 4.0
>
> Attachments: trunkpatch.diff, v3.0patch.diff
>
>
> Add some restrictions around create / alter keyspace with 
> NetworkTopologyStrategy:
> 1. Do not accept empty replication configuration (no DC options after class). 
> Cassandra checks that SimpleStrategy must have replication_factor option but 
> does not check that at least one DC should be present in the options for 
> NetworkTopologyStrategy.
> 2. Cassandra accepts any random string as DC name replication option for 
> NetworkTopologyStrategy while creating or altering keyspaces. Add a 
> restriction that the options specified is valid datacenter name. Using 
> incorrect value or simple mistake in typing the DC name can cause outage in 
> production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

cassandra git commit: reduce list allocations/resizes in loop

[dbrosius]

Repository: cassandra
Updated Branches:
  refs/heads/trunk 1be5cc0db -> 56e01770b


reduce list allocations/resizes in loop


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/56e01770
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/56e01770
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/56e01770

Branch: refs/heads/trunk
Commit: 56e01770bf556f3cfde84539da880235243cabef
Parents: 1be5cc0
Author: Dave Brosius 
Authored: Fri Sep 30 00:46:06 2016 -0400
Committer: Dave Brosius 
Committed: Fri Sep 30 00:46:06 2016 -0400

--
 .../index/sasi/utils/RangeIntersectionIterator.java | 9 ++---
 1 file changed, 6 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/56e01770/src/java/org/apache/cassandra/index/sasi/utils/RangeIntersectionIterator.java
--
diff --git 
a/src/java/org/apache/cassandra/index/sasi/utils/RangeIntersectionIterator.java 
b/src/java/org/apache/cassandra/index/sasi/utils/RangeIntersectionIterator.java
index 02d9527..7babf04 100644
--- 
a/src/java/org/apache/cassandra/index/sasi/utils/RangeIntersectionIterator.java
+++ 
b/src/java/org/apache/cassandra/index/sasi/utils/RangeIntersectionIterator.java
@@ -124,6 +124,8 @@ public class RangeIntersectionIterator
 
 protected D computeNext()
 {
+List> processed = null;
+
 while (!ranges.isEmpty())
 {
 RangeIterator head = ranges.poll();
@@ -139,7 +141,8 @@ public class RangeIntersectionIterator
 return endOfData();
 }
 
-List> processed = new ArrayList<>();
+if (processed == null)
+processed = new ArrayList<>();
 
 boolean intersectsAll = true, exhausted = false;
 while (!ranges.isEmpty())
@@ -180,8 +183,8 @@ public class RangeIntersectionIterator
 
 ranges.add(head);
 
-for (RangeIterator range : processed)
-ranges.add(range);
+ranges.addAll(processed);
+processed.clear();
 
 if (exhausted)
 return endOfData();

cassandra git commit: make constructor private to respect factory pattern

[dbrosius]

Repository: cassandra
Updated Branches:
  refs/heads/trunk 7efe1ccc5 -> 1be5cc0db


make constructor private to respect factory pattern


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1be5cc0d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1be5cc0d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1be5cc0d

Branch: refs/heads/trunk
Commit: 1be5cc0dbdba3792a11ea22c63f83d3b2b76cb7b
Parents: 7efe1cc
Author: Dave Brosius 
Authored: Fri Sep 30 00:35:59 2016 -0400
Committer: Dave Brosius 
Committed: Fri Sep 30 00:35:59 2016 -0400

--
 src/java/org/apache/cassandra/cache/CounterCacheKey.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/1be5cc0d/src/java/org/apache/cassandra/cache/CounterCacheKey.java
--
diff --git a/src/java/org/apache/cassandra/cache/CounterCacheKey.java 
b/src/java/org/apache/cassandra/cache/CounterCacheKey.java
index 8b173bf..bee53d0 100644
--- a/src/java/org/apache/cassandra/cache/CounterCacheKey.java
+++ b/src/java/org/apache/cassandra/cache/CounterCacheKey.java
@@ -33,7 +33,7 @@ public final class CounterCacheKey extends CacheKey
 public final byte[] partitionKey;
 public final byte[] cellName;
 
-public CounterCacheKey(Pair ksAndCFName, ByteBuffer 
partitionKey, ByteBuffer cellName)
+private CounterCacheKey(Pair ksAndCFName, ByteBuffer 
partitionKey, ByteBuffer cellName)
 {
 super(ksAndCFName);
 this.partitionKey = ByteBufferUtil.getArray(partitionKey);

[jira] [Commented] (CASSANDRA-12705) Add column definition kind to system schema dropped columns

[Stefania (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534987#comment-15534987
 ] 

Stefania commented on CASSANDRA-12705:
--

Thanks, rebased, CI running:

|[patch|https://github.com/stef1927/cassandra/commits/12705]|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12705-testall/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12705-dtest/]|


> Add column definition kind to system schema dropped columns
> ---
>
> Key: CASSANDRA-12705
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12705
> Project: Cassandra
>  Issue Type: Bug
>  Components: Distributed Metadata
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 4.0
>
>
> Both regular and static columns can currently be dropped by users, but this 
> information is currently not stored in {{SchemaKeyspace.DroppedColumns}}. As 
> a consequence, {{CFMetadata.getDroppedColumnDefinition}} returns a regular 
> column and this has caused problems such as CASSANDRA-12582.
> We should add the column kind to {{SchemaKeyspace.DroppedColumns}} so that 
> {{CFMetadata.getDroppedColumnDefinition}} can create the correct column 
> definition. However, altering schema tables would cause inter-node 
> communication failures during a rolling upgrade, see CASSANDRA-12236. 
> Therefore we should wait for a full schema migration when upgrading to the 
> next major version.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

cassandra git commit: remove dead field 'windowSize'

[dbrosius]

Repository: cassandra
Updated Branches:
  refs/heads/trunk a052dbfe6 -> 7efe1ccc5


remove dead field 'windowSize'


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7efe1ccc
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7efe1ccc
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7efe1ccc

Branch: refs/heads/trunk
Commit: 7efe1ccc5b9fda790b469ac96bc8c86599b08357
Parents: a052dbf
Author: Dave Brosius 
Authored: Fri Sep 30 00:25:18 2016 -0400
Committer: Dave Brosius 
Committed: Fri Sep 30 00:25:59 2016 -0400

--
 .../org/apache/cassandra/net/RateBasedBackPressureState.java   | 6 ++
 1 file changed, 2 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/7efe1ccc/src/java/org/apache/cassandra/net/RateBasedBackPressureState.java
--
diff --git a/src/java/org/apache/cassandra/net/RateBasedBackPressureState.java 
b/src/java/org/apache/cassandra/net/RateBasedBackPressureState.java
index c19f277..7bcfd6d 100644
--- a/src/java/org/apache/cassandra/net/RateBasedBackPressureState.java
+++ b/src/java/org/apache/cassandra/net/RateBasedBackPressureState.java
@@ -48,7 +48,6 @@ import org.apache.cassandra.utils.concurrent.IntervalLock;
 class RateBasedBackPressureState extends IntervalLock implements 
BackPressureState
 {
 private final InetAddress host;
-private final long windowSize;
 final SlidingTimeRate incomingRate;
 final SlidingTimeRate outgoingRate;
 final RateLimiter rateLimiter;
@@ -57,9 +56,8 @@ class RateBasedBackPressureState extends IntervalLock 
implements BackPressureSta
 {
 super(timeSource);
 this.host = host;
-this.windowSize = windowSize;
-this.incomingRate = new SlidingTimeRate(timeSource, this.windowSize, 
this.windowSize / 10, TimeUnit.MILLISECONDS);
-this.outgoingRate = new SlidingTimeRate(timeSource, this.windowSize, 
this.windowSize / 10, TimeUnit.MILLISECONDS);
+this.incomingRate = new SlidingTimeRate(timeSource, windowSize, 
windowSize / 10, TimeUnit.MILLISECONDS);
+this.outgoingRate = new SlidingTimeRate(timeSource, windowSize, 
windowSize / 10, TimeUnit.MILLISECONDS);
 this.rateLimiter = RateLimiter.create(Double.POSITIVE_INFINITY);
 }
 

[jira] [Commented] (CASSANDRA-11811) dtest failure in snapshot_test.TestArchiveCommitlog.test_archive_commitlog

[Stefania (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534966#comment-15534966
 ] 

Stefania commented on CASSANDRA-11811:
--

Please review this pull request: 
https://github.com/riptano/cassandra-dtest/pull/1354. 

This failure is blocking CASSANDRA-12509, which drains a node on shutdown. With 
the 12509 patch applied, this failure happens every single time.

> dtest failure in snapshot_test.TestArchiveCommitlog.test_archive_commitlog
> --
>
> Key: CASSANDRA-11811
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11811
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Philip Thompson
>Assignee: Jim Witschey
>  Labels: dtest
> Fix For: 3.x
>
>
> example failure:
> http://cassci.datastax.com/job/trunk_dtest_win32/416/testReport/snapshot_test/TestArchiveCommitlog/test_archive_commitlog
> Failed on CassCI build trunk_dtest_win32 #416
> Relevant error is pasted. This is clearly a test problem. No idea why it only 
> happens on windows, as of yet. Affecting most tests in the 
> TestArchiveCommitlog suite
> {code}
> WARN: Failed to flush node: node1 on shutdown.
> Unexpected error in node1 log, error: 
> ERROR [main] 2016-05-13 21:15:02,701 CassandraDaemon.java:729 - Fatal 
> configuration error
> org.apache.cassandra.exceptions.ConfigurationException: Cannot change the 
> number of tokens from 64 to 32
>   at 
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1043)
>  ~[main/:na]
>   at 
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:740)
>  ~[main/:na]
>   at 
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:625)
>  ~[main/:na]
>   at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:368) 
> [main/:na]
>   at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:583)
>  [main/:na]
>   at 
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:712) 
> [main/:na]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12509) Shutdown process triggered twice during if the node is drained

[Stefania (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534961#comment-15534961
 ] 

Stefania commented on CASSANDRA-12509:
--

Unfortunately the 3.0 patch has broken 
[test_archive_commitlog_with_active_commitlog|http://cassci.datastax.com/job/ifesdjeen-12509-3.0-dtest/lastCompletedBuild/testReport/snapshot_test/TestArchiveCommitlog/test_archive_commitlog_with_active_commitlog_2/]
 in 3.0. It's a known failure, CASSANDRA-11811, but whilst it currently fails 
rarely, this patch makes it fail every single time, even locally.

The problem is that an archived commitlog segment from a destroyed cluster is 
replayed, which adds the bootstrap tokens of the old cluster, but subsequently 
new bootstrap tokens are generated and the node fails to start because it 
thinks it has twice as many tokens.

I've proposed this [pull 
request|https://github.com/riptano/cassandra-dtest/pull/1354] to fix the 
problem, which prevents the new cluster from generating new tokens, let's see 
if it gets accepted.

> Shutdown process triggered twice during if the node is drained
> --
>
> Key: CASSANDRA-12509
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12509
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Alex Petrov
>Assignee: Alex Petrov
>
> If the node is drained, the {{StorageService#drain}} 
> [method|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/StorageService.java#L4212]
>  is called, which triggers shutdown of mutation stage, messaging service, 
> compaction, batchlog etc. In the end of this process, the node is moved to 
> {{DRAINED}} status with the process still running. 
> When JVM is shutdown, the JVM shutdown hooks are ran, which are subscribed 
> during the server initialisation: 
> {{Runtime.getRuntime().addShutdownHook(drainOnShutdown);}} 
> [here|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/StorageService.java#L575-L636].
>  
> I noticed this behaviour while reviewing [CASSANDRA-12461], as if we'd like 
> add custom pre and post-shutdown hooks, most likely it makes sense to run 
> them once (or user might expect such behaviour). 
> Is this behaviour correct? Should we run whole shutdown process twice or just 
> once in "drain" and no-op during JVM shutdown?  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-12681) Reject empty options and invalid DC names in replication configuration while creating or altering a keyspace.

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534866#comment-15534866
 ] 

Jeff Jirsa edited comment on CASSANDRA-12681 at 9/30/16 3:12 AM:
-

Reverted in 3.0 ( {{cd8a98a2dd2d7e4993cd1aa3c282602b48be73c8}} ) and 3.X ( 
{{02404115e0db1762473556f985c8e5388db92da6}} ), updated NEWS.txt to be more 
descriptive in trunk/4.0:

{code}
- Cassandra will no longer allow invalid keyspace replication options, such 
as invalid datacenter names for
  NetworkTopologyStrategy. Operators MUST add new nodes to a datacenter 
before they can set set ALTER or 
  CREATE keyspace replication policies using that datacenter. Existing 
keyspaces will continue to operate, 
  but CREATE and ALTER will validate that all datacenters specified exist 
in the cluster. 
{code}

Apologies to you folks for the inconvenience. Having been on the wrong side of 
API changes in minor versions, I should have been more sensitive to this.


was (Author: jjirsa):
Reverted in 3.0 ( {{cd8a98a2dd2d7e4993cd1aa3c282602b48be73c8}} ) and 3.X ( 
{{02404115e0db1762473556f985c8e5388db92da6}} ), updated NEWS.txt to be more 
descriptive in trunk/4.0:

{code}
- Cassandra will no longer allow invalid keyspace replication options, such 
as invalid datacenter names for
  NetworkTopologyStrategy. Operators MUST add new nodes to a datacenter 
before they can set set ALTER or 
  CREATE keyspace replication policies using that datacenter. Existing 
keyspaces will continue to operate, 
  but CREATE and ALTER will validate that all datacenters specified exist 
in the cluster. 
{code}

Apologies to you DSE folks for the inconvenience. Having been on the wrong side 
of API changes in minor versions, I should have been more sensitive to this.

> Reject empty options and invalid DC names in replication configuration while 
> creating or altering a keyspace.
> -
>
> Key: CASSANDRA-12681
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12681
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Distributed Metadata
>Reporter: Nachiket Patil
>Assignee: Nachiket Patil
>Priority: Minor
> Fix For: 3.0.10, 3.10
>
> Attachments: trunkpatch.diff, v3.0patch.diff
>
>
> Add some restrictions around create / alter keyspace with 
> NetworkTopologyStrategy:
> 1. Do not accept empty replication configuration (no DC options after class). 
> Cassandra checks that SimpleStrategy must have replication_factor option but 
> does not check that at least one DC should be present in the options for 
> NetworkTopologyStrategy.
> 2. Cassandra accepts any random string as DC name replication option for 
> NetworkTopologyStrategy while creating or altering keyspaces. Add a 
> restriction that the options specified is valid datacenter name. Using 
> incorrect value or simple mistake in typing the DC name can cause outage in 
> production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12681) Reject empty options and invalid DC names in replication configuration while creating or altering a keyspace.

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534866#comment-15534866
 ] 

Jeff Jirsa commented on CASSANDRA-12681:


Reverted in 3.0 ( {{cd8a98a2dd2d7e4993cd1aa3c282602b48be73c8}} ) and 3.X ( 
{{02404115e0db1762473556f985c8e5388db92da6}} ), updated NEWS.txt to be more 
descriptive in trunk/4.0:

{code}
- Cassandra will no longer allow invalid keyspace replication options, such 
as invalid datacenter names for
  NetworkTopologyStrategy. Operators MUST add new nodes to a datacenter 
before they can set set ALTER or 
  CREATE keyspace replication policies using that datacenter. Existing 
keyspaces will continue to operate, 
  but CREATE and ALTER will validate that all datacenters specified exist 
in the cluster. 
{code}

Apologies to you DSE folks for the inconvenience. Having been on the wrong side 
of API changes in minor versions, I should have been more sensitive to this.

> Reject empty options and invalid DC names in replication configuration while 
> creating or altering a keyspace.
> -
>
> Key: CASSANDRA-12681
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12681
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Distributed Metadata
>Reporter: Nachiket Patil
>Assignee: Nachiket Patil
>Priority: Minor
> Fix For: 3.0.10, 3.10
>
> Attachments: trunkpatch.diff, v3.0patch.diff
>
>
> Add some restrictions around create / alter keyspace with 
> NetworkTopologyStrategy:
> 1. Do not accept empty replication configuration (no DC options after class). 
> Cassandra checks that SimpleStrategy must have replication_factor option but 
> does not check that at least one DC should be present in the options for 
> NetworkTopologyStrategy.
> 2. Cassandra accepts any random string as DC name replication option for 
> NetworkTopologyStrategy while creating or altering keyspaces. Add a 
> restriction that the options specified is valid datacenter name. Using 
> incorrect value or simple mistake in typing the DC name can cause outage in 
> production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/4] cassandra git commit: Revert "Reject invalid DC names as option while creating or altering NetworkTopologyStrategy"

[jjirsa]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 413e48e65 -> cd8a98a2d
  refs/heads/cassandra-3.X e3b34dc85 -> 02404115e
  refs/heads/trunk 87825f820 -> a052dbfe6


Revert "Reject invalid DC names as option while creating or altering 
NetworkTopologyStrategy"

This reverts commit f2c5ad743933498e60e7eef55e8daaa6ce338a03.


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/cd8a98a2
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/cd8a98a2
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/cd8a98a2

Branch: refs/heads/cassandra-3.0
Commit: cd8a98a2dd2d7e4993cd1aa3c282602b48be73c8
Parents: 413e48e
Author: Jeff Jirsa 
Authored: Thu Sep 29 19:50:04 2016 -0700
Committer: Jeff Jirsa 
Committed: Thu Sep 29 19:50:04 2016 -0700

--
 CHANGES.txt |  1 -
 NEWS.txt| 11 
 .../locator/AbstractReplicationStrategy.java|  2 +-
 .../locator/NetworkTopologyStrategy.java| 39 +
 .../org/apache/cassandra/cql3/CQLTester.java| 11 
 .../validation/entities/SecondaryIndexTest.java | 10 
 .../cql3/validation/operations/AlterTest.java   | 47 +---
 .../cql3/validation/operations/CreateTest.java  | 59 
 .../apache/cassandra/dht/BootStrapperTest.java  | 10 +---
 .../org/apache/cassandra/service/MoveTest.java  |  9 +--
 10 files changed, 18 insertions(+), 181 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/cd8a98a2/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 9076e7a..d12a8f8 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -7,7 +7,6 @@
  * Extend ColumnIdentifier.internedInstances key to include the type that 
generated the byte buffer (CASSANDRA-12516)
  * Backport CASSANDRA-10756 (race condition in NativeTransportService 
shutdown) (CASSANDRA-12472)
  * If CF has no clustering columns, any row cache is full partition cache 
(CASSANDRA-12499)
- * Reject invalid replication settings when creating or altering a keyspace 
(CASSANDRA-12681)
 Merged from 2.2:
  * Fix merkle tree depth calculation (CASSANDRA-12580)
  * Make Collections deserialization more robust (CASSANDRA-12618)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/cd8a98a2/NEWS.txt
--
diff --git a/NEWS.txt b/NEWS.txt
index b97a420..0bd3920 100644
--- a/NEWS.txt
+++ b/NEWS.txt
@@ -13,17 +13,6 @@ restore snapshots created with the previous major version 
using the
 'sstableloader' tool. You can upgrade the file format of your snapshots
 using the provided 'sstableupgrade' tool.
 
-3.0.10
-=
-
-Upgrading
--
-   - To protect against accidental data loss, cassandra no longer allows 
- users to set arbitrary datacenter names for NetworkTopologyStrategy. 
- Cassandra will allow users to continue using existing keyspaces
- with invalid datacenter names, but will validat DC names on CREATE and
- ALTER
-
 3.0.9
 =
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/cd8a98a2/src/java/org/apache/cassandra/locator/AbstractReplicationStrategy.java
--
diff --git 
a/src/java/org/apache/cassandra/locator/AbstractReplicationStrategy.java 
b/src/java/org/apache/cassandra/locator/AbstractReplicationStrategy.java
index d72c0c2..c90c6a1 100644
--- a/src/java/org/apache/cassandra/locator/AbstractReplicationStrategy.java
+++ b/src/java/org/apache/cassandra/locator/AbstractReplicationStrategy.java
@@ -319,7 +319,7 @@ public abstract class AbstractReplicationStrategy
 }
 }
 
-protected void validateExpectedOptions() throws ConfigurationException
+private void validateExpectedOptions() throws ConfigurationException
 {
 Collection expectedOptions = recognizedOptions();
 if (expectedOptions == null)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/cd8a98a2/src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java
--
diff --git a/src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java 
b/src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java
index 78f5b06..7c8d95e 100644
--- a/src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java
+++ b/src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java
@@ -24,11 +24,9 @@ import java.util.Map.Entry;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import org.apache.cassandra.config.DatabaseDescriptor;
 import org.apache.cassandra.exceptions.ConfigurationException;
 

[3/4] cassandra git commit: Revert "Merge branch 'cassandra-3.0' into trunk"

[jjirsa]

Revert "Merge branch 'cassandra-3.0' into trunk"

This reverts commit d45f323eb972c6fec146e5cfa84fdc47eb8aa5eb, reversing
changes made to b80ef9b2580c123da90879b4456606ef5b01b6f2.


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/02404115
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/02404115
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/02404115

Branch: refs/heads/trunk
Commit: 02404115e0db1762473556f985c8e5388db92da6
Parents: e3b34dc
Author: Jeff Jirsa 
Authored: Thu Sep 29 20:00:41 2016 -0700
Committer: Jeff Jirsa 
Committed: Thu Sep 29 20:00:41 2016 -0700

--
 CHANGES.txt |  1 -
 NEWS.txt|  3 -
 .../locator/AbstractReplicationStrategy.java|  2 +-
 .../locator/NetworkTopologyStrategy.java| 41 --
 .../org/apache/cassandra/cql3/CQLTester.java| 11 
 .../validation/entities/SecondaryIndexTest.java | 10 
 .../cql3/validation/operations/AlterTest.java   | 47 +---
 .../cql3/validation/operations/CreateTest.java  | 59 
 .../apache/cassandra/dht/BootStrapperTest.java  |  8 ---
 .../org/apache/cassandra/service/MoveTest.java  |  9 +--
 10 files changed, 15 insertions(+), 176 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/02404115/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index ba08745..ca95411 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -92,7 +92,6 @@ Merged from 3.0:
  * Calculate last compacted key on startup (CASSANDRA-6216)
  * Add schema to snapshot manifest, add USING TIMESTAMP clause to ALTER TABLE 
statements (CASSANDRA-7190)
  * If CF has no clustering columns, any row cache is full partition cache 
(CASSANDRA-12499)
- * Reject invalid replication settings when creating or altering a keyspace 
(CASSANDRA-12681)
 Merged from 2.2:
  * Fix merkle tree depth calculation (CASSANDRA-12580)
  * Make Collections deserialization more robust (CASSANDRA-12618)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/02404115/NEWS.txt
--
diff --git a/NEWS.txt b/NEWS.txt
index 9ab7c26..ad0f2be 100644
--- a/NEWS.txt
+++ b/NEWS.txt
@@ -98,9 +98,6 @@ Upgrading
 - Application layer keep-alives were added to the streaming protocol to 
prevent idle incoming connections from
   timing out and failing the stream session (CASSANDRA-11839). This 
effectively deprecates the streaming_socket_timeout_in_ms
   property in favor of streaming_keep_alive_period_in_secs. See 
cassandra.yaml for more details about this property.
-- Cassandra will no longer allow invalid keyspace replication options, 
such as invalid datacenter names for
-  NetworkTopologyStrategy. Existing keyspaces will continue to operate, 
but CREATE and ALTER will validate that
-  all datacenters specified exist in the cluster. 
 
 3.8
 ===

http://git-wip-us.apache.org/repos/asf/cassandra/blob/02404115/src/java/org/apache/cassandra/locator/AbstractReplicationStrategy.java
--
diff --git 
a/src/java/org/apache/cassandra/locator/AbstractReplicationStrategy.java 
b/src/java/org/apache/cassandra/locator/AbstractReplicationStrategy.java
index 4ba1da8..038ac9f 100644
--- a/src/java/org/apache/cassandra/locator/AbstractReplicationStrategy.java
+++ b/src/java/org/apache/cassandra/locator/AbstractReplicationStrategy.java
@@ -320,7 +320,7 @@ public abstract class AbstractReplicationStrategy
 }
 }
 
-protected void validateExpectedOptions() throws ConfigurationException
+private void validateExpectedOptions() throws ConfigurationException
 {
 Collection expectedOptions = recognizedOptions();
 if (expectedOptions == null)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/02404115/src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java
--
diff --git a/src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java 
b/src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java
index 442e6cf..756b689 100644
--- a/src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java
+++ b/src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java
@@ -24,11 +24,9 @@ import java.util.Map.Entry;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import org.apache.cassandra.config.DatabaseDescriptor;
 import org.apache.cassandra.exceptions.ConfigurationException;
 import org.apache.cassandra.dht.Token;
 import 

[2/4] cassandra git commit: Revert "Merge branch 'cassandra-3.0' into trunk"

[jjirsa]

Revert "Merge branch 'cassandra-3.0' into trunk"

This reverts commit d45f323eb972c6fec146e5cfa84fdc47eb8aa5eb, reversing
changes made to b80ef9b2580c123da90879b4456606ef5b01b6f2.


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/02404115
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/02404115
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/02404115

Branch: refs/heads/cassandra-3.X
Commit: 02404115e0db1762473556f985c8e5388db92da6
Parents: e3b34dc
Author: Jeff Jirsa 
Authored: Thu Sep 29 20:00:41 2016 -0700
Committer: Jeff Jirsa 
Committed: Thu Sep 29 20:00:41 2016 -0700

--
 CHANGES.txt |  1 -
 NEWS.txt|  3 -
 .../locator/AbstractReplicationStrategy.java|  2 +-
 .../locator/NetworkTopologyStrategy.java| 41 --
 .../org/apache/cassandra/cql3/CQLTester.java| 11 
 .../validation/entities/SecondaryIndexTest.java | 10 
 .../cql3/validation/operations/AlterTest.java   | 47 +---
 .../cql3/validation/operations/CreateTest.java  | 59 
 .../apache/cassandra/dht/BootStrapperTest.java  |  8 ---
 .../org/apache/cassandra/service/MoveTest.java  |  9 +--
 10 files changed, 15 insertions(+), 176 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/02404115/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index ba08745..ca95411 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -92,7 +92,6 @@ Merged from 3.0:
  * Calculate last compacted key on startup (CASSANDRA-6216)
  * Add schema to snapshot manifest, add USING TIMESTAMP clause to ALTER TABLE 
statements (CASSANDRA-7190)
  * If CF has no clustering columns, any row cache is full partition cache 
(CASSANDRA-12499)
- * Reject invalid replication settings when creating or altering a keyspace 
(CASSANDRA-12681)
 Merged from 2.2:
  * Fix merkle tree depth calculation (CASSANDRA-12580)
  * Make Collections deserialization more robust (CASSANDRA-12618)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/02404115/NEWS.txt
--
diff --git a/NEWS.txt b/NEWS.txt
index 9ab7c26..ad0f2be 100644
--- a/NEWS.txt
+++ b/NEWS.txt
@@ -98,9 +98,6 @@ Upgrading
 - Application layer keep-alives were added to the streaming protocol to 
prevent idle incoming connections from
   timing out and failing the stream session (CASSANDRA-11839). This 
effectively deprecates the streaming_socket_timeout_in_ms
   property in favor of streaming_keep_alive_period_in_secs. See 
cassandra.yaml for more details about this property.
-- Cassandra will no longer allow invalid keyspace replication options, 
such as invalid datacenter names for
-  NetworkTopologyStrategy. Existing keyspaces will continue to operate, 
but CREATE and ALTER will validate that
-  all datacenters specified exist in the cluster. 
 
 3.8
 ===

http://git-wip-us.apache.org/repos/asf/cassandra/blob/02404115/src/java/org/apache/cassandra/locator/AbstractReplicationStrategy.java
--
diff --git 
a/src/java/org/apache/cassandra/locator/AbstractReplicationStrategy.java 
b/src/java/org/apache/cassandra/locator/AbstractReplicationStrategy.java
index 4ba1da8..038ac9f 100644
--- a/src/java/org/apache/cassandra/locator/AbstractReplicationStrategy.java
+++ b/src/java/org/apache/cassandra/locator/AbstractReplicationStrategy.java
@@ -320,7 +320,7 @@ public abstract class AbstractReplicationStrategy
 }
 }
 
-protected void validateExpectedOptions() throws ConfigurationException
+private void validateExpectedOptions() throws ConfigurationException
 {
 Collection expectedOptions = recognizedOptions();
 if (expectedOptions == null)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/02404115/src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java
--
diff --git a/src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java 
b/src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java
index 442e6cf..756b689 100644
--- a/src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java
+++ b/src/java/org/apache/cassandra/locator/NetworkTopologyStrategy.java
@@ -24,11 +24,9 @@ import java.util.Map.Entry;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import org.apache.cassandra.config.DatabaseDescriptor;
 import org.apache.cassandra.exceptions.ConfigurationException;
 import org.apache.cassandra.dht.Token;
 import 

[4/4] cassandra git commit: Merge branch 'cassandra-3.X' into trunk

[jjirsa]

Merge branch 'cassandra-3.X' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a052dbfe
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a052dbfe
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a052dbfe

Branch: refs/heads/trunk
Commit: a052dbfe6fe6d513f3898216ee42325e6ca5e8a4
Parents: 87825f8 0240411
Author: Jeff Jirsa 
Authored: Thu Sep 29 20:02:31 2016 -0700
Committer: Jeff Jirsa 
Committed: Thu Sep 29 20:05:38 2016 -0700

--
 CHANGES.txt | 3 +--
 NEWS.txt| 8 
 2 files changed, 5 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/a052dbfe/CHANGES.txt
--
diff --cc CHANGES.txt
index 727ad64,ca95411..4472411
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,6 -1,3 +1,6 @@@
 +4.0
- 
++ * Reject invalid replication settings when creating or altering a keyspace 
(CASSANDRA-12681)
 +
  3.10
   * cdc column addition still breaks schema migration tasks (CASSANDRA-12697)
   * Upgrade metrics-reporter dependencies (CASSANDRA-12089)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/a052dbfe/NEWS.txt
--
diff --cc NEWS.txt
index e1e952b,ad0f2be..9db425a
--- a/NEWS.txt
+++ b/NEWS.txt
@@@ -13,16 -13,6 +13,19 @@@ restore snapshots created with the prev
  'sstableloader' tool. You can upgrade the file format of your snapshots
  using the provided 'sstableupgrade' tool.
  
 +4.0
 +===
 +
 +New features
 +
 +
 +Upgrading
 +-
- 
++- Cassandra will no longer allow invalid keyspace replication options, 
such as invalid datacenter names for
++  NetworkTopologyStrategy. Operators MUST add new nodes to a datacenter 
before they can set set ALTER or 
++  CREATE keyspace replication policies using that datacenter. Existing 
keyspaces will continue to operate, 
++  but CREATE and ALTER will validate that all datacenters specified exist 
in the cluster. 
 +
  3.10
  
  

[jira] [Commented] (CASSANDRA-12462) NullPointerException in CompactionInfo.getId(CompactionInfo.java:65)

[Yuki Morishita (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534513#comment-15534513
 ] 

Yuki Morishita commented on CASSANDRA-12462:


The only case that {{cfm}} in {{CompactionInfo}} be {{null}} is index summary 
distribution (see 
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/io/sstable/IndexSummaryRedistribution.java#L309
 and 
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/compaction/CompactionInfo.java#L47).

Index summary distibution runs across tables, so you cannot pass one table 
metadata.

[~szhou] your patch only takes care of {{getId()}}. You should patch all 
{{cfm}} access in {{CompactionInfo}} class and add null check.

> NullPointerException in CompactionInfo.getId(CompactionInfo.java:65)
> 
>
> Key: CASSANDRA-12462
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12462
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Jonathan DePrizio
> Attachments: 0001-Fix-NPE-when-running-nodetool-compactionstats.patch
>
>
> Note: The same trace is cited in the last comment of 
> https://issues.apache.org/jira/browse/CASSANDRA-11961
> I've noticed that some of my nodes in my 2.1 cluster have fallen way behind 
> on compactions, and have huge numbers (thousands) of uncompacted, tiny 
> SSTables (~30MB or so).
> In diagnosing the issue, I've found that "nodetool compactionstats" returns 
> the exception below.  Restarting cassandra on the node here causes the 
> pending tasks count to jump to ~2000.  Compactions run properly for about an 
> hour, until this exception occurs again.  Once it occurs, I see the pending 
> tasks value rapidly drop towards zero, but without any compactions actually 
> running (the logs show no compactions finishing).  It would seem that this is 
> causing compactions to fail on this node, which is leading to it running out 
> of space, etc.
> [redacted]# nodetool compactionstats
> xss =  -ea -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar 
> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms12G -Xmx12G 
> -Xmn1000M -Xss255k
> pending tasks: 5
> error: null
> -- StackTrace --
> java.lang.NullPointerException
>   at 
> org.apache.cassandra.db.compaction.CompactionInfo.getId(CompactionInfo.java:65)
>   at 
> org.apache.cassandra.db.compaction.CompactionInfo.asMap(CompactionInfo.java:118)
>   at 
> org.apache.cassandra.db.compaction.CompactionManager.getCompactions(CompactionManager.java:1405)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
>   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>   at java.lang.reflect.Method.invoke(Unknown Source)
>   at sun.reflect.misc.Trampoline.invoke(Unknown Source)
>   at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>   at java.lang.reflect.Method.invoke(Unknown Source)
>   at sun.reflect.misc.MethodUtil.invoke(Unknown Source)
>   at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown 
> Source)
>   at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown 
> Source)
>   at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
>   at com.sun.jmx.mbeanserver.PerInterface.getAttribute(Unknown Source)
>   at com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(Unknown Source)
>   at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(Unknown 
> Source)
>   at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(Unknown Source)
>   at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown 
> Source)
>   at javax.management.remote.rmi.RMIConnectionImpl.access$300(Unknown 
> Source)
>   at 
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown 
> Source)
>   at 
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown 
> Source)
>   at javax.management.remote.rmi.RMIConnectionImpl.getAttribute(Unknown 
> Source)
>   at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>   at java.lang.reflect.Method.invoke(Unknown Source)
>   at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
>   at sun.rmi.transport.Transport$1.run(Unknown Source)
>   at sun.rmi.transport.Transport$1.run(Unknown Source)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at sun.rmi.transport.Transport.serviceCall(Unknown Source)
>   at 

[jira] [Updated] (CASSANDRA-12478) cassandra stress still uses CFMetaData.compile()

[Denis Ranger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denis Ranger updated CASSANDRA-12478:
-
Attachment: 0002-Used-client-instead-of-tool-init.patch

v2 + 0002 patch seems to do the trick.

> cassandra stress still uses CFMetaData.compile()
> 
>
> Key: CASSANDRA-12478
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12478
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Denis Ranger
>  Labels: stress
> Fix For: 3.0.x
>
> Attachments: 
> 0001-Replaced-using-CFMetaData.compile-in-cassandra-stres.patch, 
> 0002-Used-client-instead-of-tool-init.patch, cassandra-stress-trunk.patch, 
> cassandra-stress-v2.patch
>
>
> Using CFMetaData.compile() on a client tool causes permission problems. To 
> reproduce:
> * Start cassandra under user _cassandra_
> * Run {{chmod -R go-rwx /var/lib/cassandra}} to deny access to other users.
> * Use a non-root user to run {{cassandra-stress}} 
> This produces an access denied message on {{/var/lib/cassandra/commitlog}}.
> The attached fix uses client-mode functionality.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-12462) NullPointerException in CompactionInfo.getId(CompactionInfo.java:65)

[Simon Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Zhou updated CASSANDRA-12462:
---
Reviewer: Jonathan Ellis
  Status: Patch Available  (was: Open)

patch attached.

> NullPointerException in CompactionInfo.getId(CompactionInfo.java:65)
> 
>
> Key: CASSANDRA-12462
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12462
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Jonathan DePrizio
> Attachments: 0001-Fix-NPE-when-running-nodetool-compactionstats.patch
>
>
> Note: The same trace is cited in the last comment of 
> https://issues.apache.org/jira/browse/CASSANDRA-11961
> I've noticed that some of my nodes in my 2.1 cluster have fallen way behind 
> on compactions, and have huge numbers (thousands) of uncompacted, tiny 
> SSTables (~30MB or so).
> In diagnosing the issue, I've found that "nodetool compactionstats" returns 
> the exception below.  Restarting cassandra on the node here causes the 
> pending tasks count to jump to ~2000.  Compactions run properly for about an 
> hour, until this exception occurs again.  Once it occurs, I see the pending 
> tasks value rapidly drop towards zero, but without any compactions actually 
> running (the logs show no compactions finishing).  It would seem that this is 
> causing compactions to fail on this node, which is leading to it running out 
> of space, etc.
> [redacted]# nodetool compactionstats
> xss =  -ea -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar 
> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms12G -Xmx12G 
> -Xmn1000M -Xss255k
> pending tasks: 5
> error: null
> -- StackTrace --
> java.lang.NullPointerException
>   at 
> org.apache.cassandra.db.compaction.CompactionInfo.getId(CompactionInfo.java:65)
>   at 
> org.apache.cassandra.db.compaction.CompactionInfo.asMap(CompactionInfo.java:118)
>   at 
> org.apache.cassandra.db.compaction.CompactionManager.getCompactions(CompactionManager.java:1405)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
>   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>   at java.lang.reflect.Method.invoke(Unknown Source)
>   at sun.reflect.misc.Trampoline.invoke(Unknown Source)
>   at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>   at java.lang.reflect.Method.invoke(Unknown Source)
>   at sun.reflect.misc.MethodUtil.invoke(Unknown Source)
>   at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown 
> Source)
>   at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown 
> Source)
>   at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
>   at com.sun.jmx.mbeanserver.PerInterface.getAttribute(Unknown Source)
>   at com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(Unknown Source)
>   at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(Unknown 
> Source)
>   at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(Unknown Source)
>   at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown 
> Source)
>   at javax.management.remote.rmi.RMIConnectionImpl.access$300(Unknown 
> Source)
>   at 
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown 
> Source)
>   at 
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown 
> Source)
>   at javax.management.remote.rmi.RMIConnectionImpl.getAttribute(Unknown 
> Source)
>   at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>   at java.lang.reflect.Method.invoke(Unknown Source)
>   at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
>   at sun.rmi.transport.Transport$1.run(Unknown Source)
>   at sun.rmi.transport.Transport$1.run(Unknown Source)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at sun.rmi.transport.Transport.serviceCall(Unknown Source)
>   at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
>   at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown 
> Source)
>   at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown 
> Source)
>   at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>   at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>   at java.lang.Thread.run(Unknown Source)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12681) Reject empty options and invalid DC names in replication configuration while creating or altering a keyspace.

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534243#comment-15534243
 ] 

Jeff Jirsa commented on CASSANDRA-12681:


Will revert it when I get home. Will re-land in 4.0 only. Feel free to revert 
on my behalf if you're in a hurry.

> Reject empty options and invalid DC names in replication configuration while 
> creating or altering a keyspace.
> -
>
> Key: CASSANDRA-12681
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12681
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Distributed Metadata
>Reporter: Nachiket Patil
>Assignee: Nachiket Patil
>Priority: Minor
> Fix For: 3.0.10, 3.10
>
> Attachments: trunkpatch.diff, v3.0patch.diff
>
>
> Add some restrictions around create / alter keyspace with 
> NetworkTopologyStrategy:
> 1. Do not accept empty replication configuration (no DC options after class). 
> Cassandra checks that SimpleStrategy must have replication_factor option but 
> does not check that at least one DC should be present in the options for 
> NetworkTopologyStrategy.
> 2. Cassandra accepts any random string as DC name replication option for 
> NetworkTopologyStrategy while creating or altering keyspaces. Add a 
> restriction that the options specified is valid datacenter name. Using 
> incorrect value or simple mistake in typing the DC name can cause outage in 
> production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12681) Reject empty options and invalid DC names in replication configuration while creating or altering a keyspace.

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534174#comment-15534174
 ] 

Aleksey Yeschenko commented on CASSANDRA-12681:
---

It's breaking the existing contract by not accepting previously accepted 
values. It's broken a couple strategy implementations that inherit from NTS in 
DSE, for one.

But, really, we have rules, and one of them is that only bug fixes go to 3.0.x 
- and this isn't a bug fix.

> Reject empty options and invalid DC names in replication configuration while 
> creating or altering a keyspace.
> -
>
> Key: CASSANDRA-12681
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12681
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Distributed Metadata
>Reporter: Nachiket Patil
>Assignee: Nachiket Patil
>Priority: Minor
> Fix For: 3.0.10, 3.10
>
> Attachments: trunkpatch.diff, v3.0patch.diff
>
>
> Add some restrictions around create / alter keyspace with 
> NetworkTopologyStrategy:
> 1. Do not accept empty replication configuration (no DC options after class). 
> Cassandra checks that SimpleStrategy must have replication_factor option but 
> does not check that at least one DC should be present in the options for 
> NetworkTopologyStrategy.
> 2. Cassandra accepts any random string as DC name replication option for 
> NetworkTopologyStrategy while creating or altering keyspaces. Add a 
> restriction that the options specified is valid datacenter name. Using 
> incorrect value or simple mistake in typing the DC name can cause outage in 
> production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11138) cassandra-stress tool - clustering key values not distributed

[Yap Sok Ann (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534169#comment-15534169
 ] 

Yap Sok Ann commented on CASSANDRA-11138:
-

I didn't set the population for {{col2}}, so it should be the default 
{{uniform(1..100B)}}.

Yes, so far I am able to replicate this consistently. May I know if you get the 
same results as mine without the patch?

> cassandra-stress tool - clustering key values not distributed
> -
>
> Key: CASSANDRA-11138
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11138
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
> Environment: Cassandra 2.2.4, Centos 6.5, Java 8
>Reporter: Ralf Steppacher
>  Labels: stress
> Attachments: 11138-trunk.patch
>
>
> I am trying to get the stress tool to generate random values for three 
> clustering keys. I am trying to simulate collecting events per user id (text, 
> partition key). Events have a session type (text), event type (text), and 
> creation time (timestamp) (clustering keys, in that order). For testing 
> purposes I ended up with the following column spec:
> {noformat}
> columnspec:
> - name: created_at
>   cluster: uniform(10..10)
> - name: event_type
>   size: uniform(5..10)
>   population: uniform(1..30)
>   cluster: uniform(1..30)
> - name: session_type
>   size: fixed(5)
>   population: uniform(1..4)
>   cluster: uniform(1..4)
> - name: user_id
>   size: fixed(15)
>   population: uniform(1..100)
> - name: message
>   size: uniform(10..100)
>   population: uniform(1..100B)
> {noformat}
> My expectation was that this would lead to anywhere between 10 and 1200 rows 
> to be created per partition key. But it seems that exactly 10 rows are being 
> created, with the {{created_at}} timestamp being the only variable that is 
> assigned variable values (per partition key). The {{session_type}} and 
> {{event_type}} variables are assigned fixed values. This is even the case if 
> I set the cluster distribution to uniform(30..30) and uniform(4..4) 
> respectively. With this setting I expected 1200 rows per partition key to be 
> created, as announced when running the stress tool, but it is still 10.
> {noformat}
> [rsteppac@centos bin]$ ./cassandra-stress user 
> profile=../batch_too_large.yaml ops\(insert=1\) -log level=verbose 
> file=~/centos_eventy_patient_session_event_timestamp_insert_only.log -node 
> 10.211.55.8
> …
> Created schema. Sleeping 1s for propagation.
> Generating batches with [1..1] partitions and [1..1] rows (of [1200..1200] 
> total rows in the partitions)
> Improvement over 4 threadCount: 19%
> ...
> {noformat}
> Sample of generated data:
> {noformat}
> cqlsh> select user_id, event_type, session_type, created_at from 
> stresscql.batch_too_large LIMIT 30 ;
> user_id | event_type   | session_type | created_at
> -+--+--+--
>   %\x7f\x03/.d29 08:14:11+
>   %\x7f\x03/.d29 04:04:56+
>   %\x7f\x03/.d29 00:39:23+
>   %\x7f\x03/.d29 19:56:30+
>   %\x7f\x03/.d29 20:46:26+
>   %\x7f\x03/.d29 03:27:17+
>   %\x7f\x03/.d29 23:30:34+
>   %\x7f\x03/.d29 02:41:28+
>   %\x7f\x03/.d29 07:23:48+
>   %\x7f\x03/.d29 23:23:04+
>  N!\x0eUA7^r7d\x06J 17:48:51+
>  N!\x0eUA7^r7d\x06J 06:21:13+
>  N!\x0eUA7^r7d\x06J 03:34:41+
>  N!\x0eUA7^r7d\x06J 05:26:21+
>  N!\x0eUA7^r7d\x06J 01:31:24+
>  N!\x0eUA7^r7d\x06J 14:22:43+
>  N!\x0eUA7^r7d\x06J 14:54:29+
>  N!\x0eUA7^r7d\x06J 13:31:54+
>  N!\x0eUA7^r7d\x06J 06:38:40+
>  N!\x0eUA7^r7d\x06J

[jira] [Commented] (CASSANDRA-12681) Reject empty options and invalid DC names in replication configuration while creating or altering a keyspace.

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534157#comment-15534157
 ] 

Aleksey Yeschenko commented on CASSANDRA-12681:
---

+1, this is not acceptable for 3.0, not at this stage (almost a year in). 
Borderline acceptable for 3.X, but safest to commit to 4.0 only.

> Reject empty options and invalid DC names in replication configuration while 
> creating or altering a keyspace.
> -
>
> Key: CASSANDRA-12681
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12681
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Distributed Metadata
>Reporter: Nachiket Patil
>Assignee: Nachiket Patil
>Priority: Minor
> Fix For: 3.0.10, 3.10
>
> Attachments: trunkpatch.diff, v3.0patch.diff
>
>
> Add some restrictions around create / alter keyspace with 
> NetworkTopologyStrategy:
> 1. Do not accept empty replication configuration (no DC options after class). 
> Cassandra checks that SimpleStrategy must have replication_factor option but 
> does not check that at least one DC should be present in the options for 
> NetworkTopologyStrategy.
> 2. Cassandra accepts any random string as DC name replication option for 
> NetworkTopologyStrategy while creating or altering keyspaces. Add a 
> restriction that the options specified is valid datacenter name. Using 
> incorrect value or simple mistake in typing the DC name can cause outage in 
> production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12681) Reject empty options and invalid DC names in replication configuration while creating or altering a keyspace.

[sankalp kohli (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534159#comment-15534159
 ] 

sankalp kohli commented on CASSANDRA-12681:
---

What API is it breaking? It stops users from accidental outages right. 

> Reject empty options and invalid DC names in replication configuration while 
> creating or altering a keyspace.
> -
>
> Key: CASSANDRA-12681
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12681
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Distributed Metadata
>Reporter: Nachiket Patil
>Assignee: Nachiket Patil
>Priority: Minor
> Fix For: 3.0.10, 3.10
>
> Attachments: trunkpatch.diff, v3.0patch.diff
>
>
> Add some restrictions around create / alter keyspace with 
> NetworkTopologyStrategy:
> 1. Do not accept empty replication configuration (no DC options after class). 
> Cassandra checks that SimpleStrategy must have replication_factor option but 
> does not check that at least one DC should be present in the options for 
> NetworkTopologyStrategy.
> 2. Cassandra accepts any random string as DC name replication option for 
> NetworkTopologyStrategy while creating or altering keyspaces. Add a 
> restriction that the options specified is valid datacenter name. Using 
> incorrect value or simple mistake in typing the DC name can cause outage in 
> production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12681) Reject empty options and invalid DC names in replication configuration while creating or altering a keyspace.

[Jeremiah Jordan (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534124#comment-15534124
 ] 

Jeremiah Jordan commented on CASSANDRA-12681:
-

[~jjirsa] since this is not a bug fix, and it breaks anything doing stuff like 
mentioned in CASSANDRA-4795 I really think we should revert this from 3.0.10 
and possibly only commit it to 4.0.  This is a breaking "API" change/behavior.

> Reject empty options and invalid DC names in replication configuration while 
> creating or altering a keyspace.
> -
>
> Key: CASSANDRA-12681
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12681
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Distributed Metadata
>Reporter: Nachiket Patil
>Assignee: Nachiket Patil
>Priority: Minor
> Fix For: 3.0.10, 3.10
>
> Attachments: trunkpatch.diff, v3.0patch.diff
>
>
> Add some restrictions around create / alter keyspace with 
> NetworkTopologyStrategy:
> 1. Do not accept empty replication configuration (no DC options after class). 
> Cassandra checks that SimpleStrategy must have replication_factor option but 
> does not check that at least one DC should be present in the options for 
> NetworkTopologyStrategy.
> 2. Cassandra accepts any random string as DC name replication option for 
> NetworkTopologyStrategy while creating or altering keyspaces. Add a 
> restriction that the options specified is valid datacenter name. Using 
> incorrect value or simple mistake in typing the DC name can cause outage in 
> production environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-7190) Add schema to snapshot manifest

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534072#comment-15534072
 ] 

Aleksey Yeschenko commented on CASSANDRA-7190:
--

bq. Should I create a new ticket?

Please do.

> Add schema to snapshot manifest
> ---
>
> Key: CASSANDRA-7190
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7190
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Jonathan Ellis
>Assignee: Alex Petrov
>Priority: Minor
>  Labels: client-impacting, doc-impacting, lhf
> Fix For: 3.0.9, 3.10
>
>
> followup from CASSANDRA-6326



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

svn commit: r1762840 - /cassandra/site/publish/index.html

[mshuler]

Author: mshuler
Date: Thu Sep 29 20:53:40 2016
New Revision: 1762840

URL: http://svn.apache.org/viewvc?rev=1762840=rev
Log:
3.9 release on index

Modified:
cassandra/site/publish/index.html

Modified: cassandra/site/publish/index.html
URL: 
http://svn.apache.org/viewvc/cassandra/site/publish/index.html?rev=1762840=1762839=1762840=diff
==
--- cassandra/site/publish/index.html (original)
+++ cassandra/site/publish/index.html Thu Sep 29 20:53:40 2016
@@ -89,7 +89,7 @@
 
 
   http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/cassandra-3.7;>Cassandra
 3.7 Changelog
+ 
href="http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/cassandra-3.9;>Cassandra
 3.9 Changelog
   
 
   

svn commit: r1762839 - in /cassandra/site: publish/download/index.html src/_data/releases.yaml

[mshuler]

Author: mshuler
Date: Thu Sep 29 20:51:14 2016
New Revision: 1762839

URL: http://svn.apache.org/viewvc?rev=1762839=rev
Log:
3.9 release

Modified:
cassandra/site/publish/download/index.html
cassandra/site/src/_data/releases.yaml

Modified: cassandra/site/publish/download/index.html
URL: 
http://svn.apache.org/viewvc/cassandra/site/publish/download/index.html?rev=1762839=1762838=1762839=diff
==
--- cassandra/site/publish/download/index.html (original)
+++ cassandra/site/publish/download/index.html Thu Sep 29 20:51:14 2016
@@ -97,7 +97,7 @@
 features; odd-numbered releases (e.g. 3.3) contain bug fixes 
only. If a critical bug is found, a patch will be
 released against the most recent bug fix release. Read more about http://www.planetcassandra.org/blog/cassandra-2-2-3-0-and-beyond/;>tick-tock
 here.
 
-Download the latest Cassandra release: http://www.apache.org/dyn/closer.lua/cassandra/3.7/apache-cassandra-3.7-bin.tar.gz;>3.7
 (http://www.apache.org/dist/cassandra/3.7/apache-cassandra-3.7-bin.tar.gz.asc;>pgp,
 http://www.apache.org/dist/cassandra/3.7/apache-cassandra-3.7-bin.tar.gz.md5;>md5
 and http://www.apache.org/dist/cassandra/3.7/apache-cassandra-3.7-bin.tar.gz.sha1;>sha1),
 released on 2016-06-13.
+Download the latest Cassandra release: http://www.apache.org/dyn/closer.lua/cassandra/3.9/apache-cassandra-3.9-bin.tar.gz;>3.9
 (http://www.apache.org/dist/cassandra/3.9/apache-cassandra-3.9-bin.tar.gz.asc;>pgp,
 http://www.apache.org/dist/cassandra/3.9/apache-cassandra-3.9-bin.tar.gz.md5;>md5
 and http://www.apache.org/dist/cassandra/3.9/apache-cassandra-3.9-bin.tar.gz.sha1;>sha1),
 released on 2016-09-29.
 
 Older supported releases
 

Modified: cassandra/site/src/_data/releases.yaml
URL: 
http://svn.apache.org/viewvc/cassandra/site/src/_data/releases.yaml?rev=1762839=1762838=1762839=diff
==
--- cassandra/site/src/_data/releases.yaml (original)
+++ cassandra/site/src/_data/releases.yaml Thu Sep 29 20:51:14 2016
@@ -1,6 +1,6 @@
 latest:
-  name: 3.7
-  date: 2016-06-13
+  name: 3.9
+  date: 2016-09-29
 
 #latest_bugfix:
 #  name: 3.5

svn commit: r15999 - in /release/cassandra: 3.9/ debian/dists/39x/ debian/dists/39x/main/ debian/dists/39x/main/binary-amd64/ debian/dists/39x/main/binary-i386/ debian/dists/39x/main/source/ debian/po

[mshuler]

Author: mshuler
Date: Thu Sep 29 20:24:53 2016
New Revision: 15999

Log:
Apache Cassandra 3.9 Release

Added:
release/cassandra/3.9/
release/cassandra/3.9/apache-cassandra-3.9-bin.tar.gz   (with props)
release/cassandra/3.9/apache-cassandra-3.9-bin.tar.gz.asc
release/cassandra/3.9/apache-cassandra-3.9-bin.tar.gz.asc.md5
release/cassandra/3.9/apache-cassandra-3.9-bin.tar.gz.asc.sha1
release/cassandra/3.9/apache-cassandra-3.9-bin.tar.gz.md5
release/cassandra/3.9/apache-cassandra-3.9-bin.tar.gz.sha1
release/cassandra/3.9/apache-cassandra-3.9-src.tar.gz   (with props)
release/cassandra/3.9/apache-cassandra-3.9-src.tar.gz.asc
release/cassandra/3.9/apache-cassandra-3.9-src.tar.gz.asc.md5
release/cassandra/3.9/apache-cassandra-3.9-src.tar.gz.asc.sha1
release/cassandra/3.9/apache-cassandra-3.9-src.tar.gz.md5
release/cassandra/3.9/apache-cassandra-3.9-src.tar.gz.sha1
release/cassandra/debian/dists/39x/
release/cassandra/debian/dists/39x/InRelease
release/cassandra/debian/dists/39x/Release
release/cassandra/debian/dists/39x/Release.gpg
release/cassandra/debian/dists/39x/main/
release/cassandra/debian/dists/39x/main/binary-amd64/
release/cassandra/debian/dists/39x/main/binary-amd64/Packages
release/cassandra/debian/dists/39x/main/binary-amd64/Packages.gz   (with 
props)
release/cassandra/debian/dists/39x/main/binary-amd64/Release
release/cassandra/debian/dists/39x/main/binary-i386/
release/cassandra/debian/dists/39x/main/binary-i386/Packages
release/cassandra/debian/dists/39x/main/binary-i386/Packages.gz   (with 
props)
release/cassandra/debian/dists/39x/main/binary-i386/Release
release/cassandra/debian/dists/39x/main/source/
release/cassandra/debian/dists/39x/main/source/Release
release/cassandra/debian/dists/39x/main/source/Sources.gz   (with props)
release/cassandra/debian/pool/main/c/cassandra/cassandra-tools_3.9_all.deb  
 (with props)
release/cassandra/debian/pool/main/c/cassandra/cassandra_3.9.diff.gz   
(with props)
release/cassandra/debian/pool/main/c/cassandra/cassandra_3.9.dsc
release/cassandra/debian/pool/main/c/cassandra/cassandra_3.9.orig.tar.gz   
(with props)
release/cassandra/debian/pool/main/c/cassandra/cassandra_3.9.orig.tar.gz.asc
release/cassandra/debian/pool/main/c/cassandra/cassandra_3.9_all.deb   
(with props)

Added: release/cassandra/3.9/apache-cassandra-3.9-bin.tar.gz
==
Binary file - no diff available.

Propchange: release/cassandra/3.9/apache-cassandra-3.9-bin.tar.gz
--
svn:mime-type = application/octet-stream

Added: release/cassandra/3.9/apache-cassandra-3.9-bin.tar.gz.asc
==
--- release/cassandra/3.9/apache-cassandra-3.9-bin.tar.gz.asc (added)
+++ release/cassandra/3.9/apache-cassandra-3.9-bin.tar.gz.asc Thu Sep 29 
20:24:53 2016
@@ -0,0 +1,17 @@
+-BEGIN PGP SIGNATURE-
+Version: GnuPG v1
+
+iQIcBAABCAAGBQJX6TeiAAoJEKJ4t4H+Syvarp0P/2q+sKmuVB0DZz1+rsW2k+Lt
+XxTNHTw3tn1dO5dIcVfE458pn8d7gK08PGELrz5J/uPPRqq2JIRJmwvtErn+rrYV
+AMIxJ+YfL5OGdY1N90KmLhTYlt/h19stwNPzUZif0JXHEvrU2SPybSzg9kKUAh9p
+7G5MFY1njBGguMM72SFyOHBIuBdai8x0efKeuiJWNlzRoAhs31A6fOWHDyKScFqB
+sGa2Hx2fS2WTWfH1RTLMipmJEj/u/izcZbCDd3uLfWmDFBxke0igpoxkH+gjfvpA
+wDfPMxsME8DF/nRkhirVFkSNoSxG9bb/nl1N2DRDjg91NS1pIcHk8IACl/yy7IXg
+yA6D6da4BMyrzmebxb3jEjOQXUuJvVBqA1oy4tMPAtWbzQ093Xh9Vafu9/vPerjK
+zfc+cU85Cn+KcRllH8KpXYhBTcY3TyPBuA9rTMLjfWxobgFfxfKzK5hMuj8tSWRW
+5nYQMppuZPHpU7TI90NYpJvZNTeuj1feDT6d2sPtYjdJJAfm4wqHfNCd6XAzckdE
+eSWdkHC2iPj0cqxiI8ke7H1UNVe1+fn+1F+efFFNiikEIl30BRK210QAu2OXgwXt
+P/1Pq5xaZDZAgQ7MzddJXSB3qYGMvhEj4qQuotNguL40yEIqW6KDlOqQGRITxe2D
++gGtQMybYxP7j9PtDRR2
+=b72o
+-END PGP SIGNATURE-

Added: release/cassandra/3.9/apache-cassandra-3.9-bin.tar.gz.asc.md5
==
--- release/cassandra/3.9/apache-cassandra-3.9-bin.tar.gz.asc.md5 (added)
+++ release/cassandra/3.9/apache-cassandra-3.9-bin.tar.gz.asc.md5 Thu Sep 29 
20:24:53 2016
@@ -0,0 +1 @@
+a3fc292f2f04db2deaebfe84528a1920
\ No newline at end of file

Added: release/cassandra/3.9/apache-cassandra-3.9-bin.tar.gz.asc.sha1
==
--- release/cassandra/3.9/apache-cassandra-3.9-bin.tar.gz.asc.sha1 (added)
+++ release/cassandra/3.9/apache-cassandra-3.9-bin.tar.gz.asc.sha1 Thu Sep 29 
20:24:53 2016
@@ -0,0 +1 @@
+47172292d9d8fd7ec568cfb8b4c6d4c81a8ae0d7
\ No newline at end of file

Added: release/cassandra/3.9/apache-cassandra-3.9-bin.tar.gz.md5
==
--- release/cassandra/3.9/apache-cassandra-3.9-bin.tar.gz.md5 (added)
+++ release/cassandra/3.9/apache-cassandra-3.9-bin.tar.gz.md5 Thu 

[cassandra] Git Push Summary

[mshuler]

Repository: cassandra
Updated Tags:  refs/tags/cassandra-3.9 [created] 18bafa8b8

[cassandra] Git Push Summary

[mshuler]

Repository: cassandra
Updated Tags:  refs/tags/3.9-tentative [deleted] c1fa21458

[jira] [Updated] (CASSANDRA-12580) Fix merkle tree size calculation

[Yuki Morishita (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuki Morishita updated CASSANDRA-12580:
---
   Resolution: Fixed
Fix Version/s: (was: 3.0.x)
   (was: 2.2.x)
   (was: 2.1.x)
   (was: 3.x)
   3.10
   3.0.10
   2.2.9
   Status: Resolved  (was: Ready to Commit)

Committed to 2.2+ as {{c70ce6307da824529762ff40673642b6f86972aa}}.
(Skipped 2.1 because it is not critical at this point.)

> Fix merkle tree size calculation
> 
>
> Key: CASSANDRA-12580
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12580
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Paulo Motta
>Assignee: Paulo Motta
> Fix For: 2.2.9, 3.0.10, 3.10
>
>
> On CASSANDRA-5263 it was introduced dynamic merkle tree sizing based on 
> estimated number of partitions as {{estimatedDepth = lg(numPartitions)}}, but 
> on 
> [CompactionManager.doValidationCompaction|https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/db/compaction/CompactionManager.java#L1052]
>  this is being calculated as:
> {{int depth = numPartitions > 0 ? (int) 
> Math.min(Math.floor(Math.log(numPartitions)), 20) : 0;}}
> This is actually calculating {{ln(numPartitions)}} (base-e) instead of 
> {{lg(numPartitions)}} (base-2), which causes merkle trees to lose resolution, 
> what may result in overstreaming.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[09/10] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.X

[yukim]

Merge branch 'cassandra-3.0' into cassandra-3.X


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e3b34dc8
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e3b34dc8
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e3b34dc8

Branch: refs/heads/trunk
Commit: e3b34dc8584373c9f503e830ff3241e0865ab994
Parents: 57e9a83 413e48e
Author: Yuki Morishita 
Authored: Thu Sep 29 15:05:12 2016 -0500
Committer: Yuki Morishita 
Committed: Thu Sep 29 15:05:12 2016 -0500

--
 CHANGES.txt |   1 +
 .../db/compaction/CompactionManager.java|   6 +-
 .../org/apache/cassandra/repair/Validator.java  |   9 +-
 .../org/apache/cassandra/utils/MerkleTree.java  |  10 ++
 .../org/apache/cassandra/utils/MerkleTrees.java |  10 ++
 .../db/compaction/CompactionsTest.java  |   2 +-
 .../apache/cassandra/repair/ValidatorTest.java  | 167 ---
 7 files changed, 145 insertions(+), 60 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/e3b34dc8/CHANGES.txt
--
diff --cc CHANGES.txt
index c33b1d3,9076e7a..ba08745
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -91,59 -35,12 +91,60 @@@ Merged from 3.0
   * Disk failure policy should not be invoked on out of space (CASSANDRA-12385)
   * Calculate last compacted key on startup (CASSANDRA-6216)
   * Add schema to snapshot manifest, add USING TIMESTAMP clause to ALTER TABLE 
statements (CASSANDRA-7190)
 + * If CF has no clustering columns, any row cache is full partition cache 
(CASSANDRA-12499)
 + * Reject invalid replication settings when creating or altering a keyspace 
(CASSANDRA-12681)
 +Merged from 2.2:
++ * Fix merkle tree depth calculation (CASSANDRA-12580)
 + * Make Collections deserialization more robust (CASSANDRA-12618)
 + * Fix exceptions when enabling gossip on nodes that haven't joined the ring 
(CASSANDRA-12253)
 + * Fix authentication problem when invoking clqsh copy from a SOURCE command 
(CASSANDRA-12642)
 + * Decrement pending range calculator jobs counter in finally block
 + * cqlshlib tests: increase default execute timeout (CASSANDRA-12481)
 + * Forward writes to replacement node when replace_address != 
broadcast_address (CASSANDRA-8523)
 + * Fail repair on non-existing table (CASSANDRA-12279)
 + * Enable repair -pr and -local together (fix regression of CASSANDRA-7450) 
(CASSANDRA-12522)
 +
 +
 +3.8, 3.9
 + * Fix value skipping with counter columns (CASSANDRA-11726)
 + * Fix nodetool tablestats miss SSTable count (CASSANDRA-12205)
 + * Fixed flacky SSTablesIteratedTest (CASSANDRA-12282)
 + * Fixed flacky SSTableRewriterTest: check file counts before calling 
validateCFS (CASSANDRA-12348)
 + * cqlsh: Fix handling of $$-escaped strings (CASSANDRA-12189)
 + * Fix SSL JMX requiring truststore containing server cert (CASSANDRA-12109)
 + * RTE from new CDC column breaks in flight queries (CASSANDRA-12236)
 + * Fix hdr logging for single operation workloads (CASSANDRA-12145)
 + * Fix SASI PREFIX search in CONTAINS mode with partial terms 
(CASSANDRA-12073)
 + * Increase size of flushExecutor thread pool (CASSANDRA-12071)
 + * Partial revert of CASSANDRA-11971, cannot recycle buffer in 
SP.sendMessagesToNonlocalDC (CASSANDRA-11950)
 + * Upgrade netty to 4.0.39 (CASSANDRA-12032, CASSANDRA-12034)
 + * Improve details in compaction log message (CASSANDRA-12080)
 + * Allow unset values in CQLSSTableWriter (CASSANDRA-11911)
 + * Chunk cache to request compressor-compatible buffers if pool space is 
exhausted (CASSANDRA-11993)
 + * Remove DatabaseDescriptor dependencies from SequentialWriter 
(CASSANDRA-11579)
 + * Move skip_stop_words filter before stemming (CASSANDRA-12078)
 + * Support seek() in EncryptedFileSegmentInputStream (CASSANDRA-11957)
 + * SSTable tools mishandling LocalPartitioner (CASSANDRA-12002)
 + * When SEPWorker assigned work, set thread name to match pool 
(CASSANDRA-11966)
 + * Add cross-DC latency metrics (CASSANDRA-11596)
 + * Allow terms in selection clause (CASSANDRA-10783)
 + * Add bind variables to trace (CASSANDRA-11719)
 + * Switch counter shards' clock to timestamps (CASSANDRA-9811)
 + * Introduce HdrHistogram and response/service/wait separation to stress tool 
(CASSANDRA-11853)
 + * entry-weighers in QueryProcessor should respect partitionKeyBindIndexes 
field (CASSANDRA-11718)
 + * Support older ant versions (CASSANDRA-11807)
 + * Estimate compressed on disk size when deciding if sstable size limit 
reached (CASSANDRA-11623)
 + * cassandra-stress profiles should support case sensitive schemas 
(CASSANDRA-11546)
 + * Remove DatabaseDescriptor dependency from FileUtils (CASSANDRA-11578)
 + * Faster streaming (CASSANDRA-9766)
 + * Add prepared query 

[02/10] cassandra git commit: Fix merkle tree depth calculation

[yukim]

Fix merkle tree depth calculation

Patch by Paulo Motta; Reviewed by Yuki Morishita for CASSANDRA-12580


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c70ce630
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c70ce630
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c70ce630

Branch: refs/heads/cassandra-3.0
Commit: c70ce6307da824529762ff40673642b6f86972aa
Parents: 2383935
Author: Paulo Motta 
Authored: Tue Aug 30 21:06:39 2016 -0300
Committer: Yuki Morishita 
Committed: Thu Sep 29 14:26:53 2016 -0500

--
 CHANGES.txt |   2 +-
 .../db/compaction/CompactionManager.java|   4 +-
 .../org/apache/cassandra/repair/Validator.java  |   9 +-
 .../org/apache/cassandra/utils/MerkleTree.java  |  10 ++
 .../db/compaction/CompactionsTest.java  |   2 +-
 .../apache/cassandra/repair/ValidatorTest.java  | 159 ---
 6 files changed, 125 insertions(+), 61 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/c70ce630/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 998849e..97bc70a 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.9
+ * Fix merkle tree depth calculation (CASSANDRA-12580)
  * Make Collections deserialization more robust (CASSANDRA-12618)
  
  
@@ -36,7 +37,6 @@
  * Don't write shadowed range tombstone (CASSANDRA-12030)
 Merged from 2.1:
  * Add system property to set the max number of native transport requests in 
queue (CASSANDRA-11363)
- * Fix queries with empty ByteBuffer values in clustering column restrictions 
(CASSANDRA-12127) 
  * Disable passing control to post-flush after flush failure to prevent data 
loss (CASSANDRA-11828)
  * Allow STCS-in-L0 compactions to reduce scope with LCS (CASSANDRA-12040)
  * cannot use cql since upgrading python to 2.7.11+ (CASSANDRA-11850)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c70ce630/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java 
b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
index cf82498..78fa23c 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
@@ -1112,8 +1112,8 @@ public class CompactionManager implements 
CompactionManagerMBean
 {
 numPartitions += 
sstable.estimatedKeysForRanges(singleton(validator.desc.range));
 }
-// determine tree depth from number of partitions, but cap at 20 
to prevent large tree.
-int depth = numPartitions > 0 ? (int) 
Math.min(Math.floor(Math.log(numPartitions)), 20) : 0;
+// determine tree depth from number of partitions, but cap at 20 
to prevent large tree (CASSANDRA-5263)
+int depth = numPartitions > 0 ? (int) 
Math.min(Math.ceil(Math.log(numPartitions) / Math.log(2)), 20) : 0;
 MerkleTree tree = new MerkleTree(cfs.partitioner, 
validator.desc.range, MerkleTree.RECOMMENDED_DEPTH, (int) Math.pow(2, depth));
 
 long start = System.nanoTime();

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c70ce630/src/java/org/apache/cassandra/repair/Validator.java
--
diff --git a/src/java/org/apache/cassandra/repair/Validator.java 
b/src/java/org/apache/cassandra/repair/Validator.java
index 4db1cfb..8dbb4cf 100644
--- a/src/java/org/apache/cassandra/repair/Validator.java
+++ b/src/java/org/apache/cassandra/repair/Validator.java
@@ -54,6 +54,7 @@ public class Validator implements Runnable
 public final RepairJobDesc desc;
 public final InetAddress initiator;
 public final int gcBefore;
+private final boolean evenTreeDistribution;
 
 // null when all rows with the min token have been consumed
 private long validated;
@@ -67,19 +68,25 @@ public class Validator implements Runnable
 
 public Validator(RepairJobDesc desc, InetAddress initiator, int gcBefore)
 {
+this(desc, initiator, gcBefore, false);
+}
+
+public Validator(RepairJobDesc desc, InetAddress initiator, int gcBefore, 
boolean evenTreeDistribution)
+{
 this.desc = desc;
 this.initiator = initiator;
 this.gcBefore = gcBefore;
 validated = 0;
 range = null;
 ranges = null;
+this.evenTreeDistribution = evenTreeDistribution;
 }
 
 public void prepare(ColumnFamilyStore cfs, MerkleTree tree)
 {
 this.tree = tree;
 
-if 

[07/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[yukim]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/413e48e6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/413e48e6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/413e48e6

Branch: refs/heads/trunk
Commit: 413e48e6571e3c23362d5053e0c7fcdd99bd1e7d
Parents: 5cebd1f c70ce63
Author: Yuki Morishita 
Authored: Thu Sep 29 14:32:49 2016 -0500
Committer: Yuki Morishita 
Committed: Thu Sep 29 14:32:49 2016 -0500

--
 CHANGES.txt |   3 +-
 .../db/compaction/CompactionManager.java|   6 +-
 .../org/apache/cassandra/repair/Validator.java  |   9 +-
 .../org/apache/cassandra/utils/MerkleTree.java  |  10 ++
 .../org/apache/cassandra/utils/MerkleTrees.java |  10 ++
 .../db/compaction/CompactionsTest.java  |   2 +-
 .../apache/cassandra/repair/ValidatorTest.java  | 167 ---
 7 files changed, 146 insertions(+), 61 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/413e48e6/CHANGES.txt
--
diff --cc CHANGES.txt
index f2f8dac,97bc70a..9076e7a
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,77 -1,13 +1,78 @@@
 -2.2.9
 +3.0.10
 + * Fix failure in LogTransactionTest (CASSANDRA-12632)
 + * Fix potentially incomplete non-frozen UDT values when querying with the
 +   full primary key specified (CASSANDRA-12605)
 + * Skip writing MV mutations to commitlog on mutation.applyUnsafe() 
(CASSANDRA-11670)
 + * Establish consistent distinction between non-existing partition and NULL 
value for LWTs on static columns (CASSANDRA-12060)
 + * Extend ColumnIdentifier.internedInstances key to include the type that 
generated the byte buffer (CASSANDRA-12516)
 + * Backport CASSANDRA-10756 (race condition in NativeTransportService 
shutdown) (CASSANDRA-12472)
 + * If CF has no clustering columns, any row cache is full partition cache 
(CASSANDRA-12499)
 + * Reject invalid replication settings when creating or altering a keyspace 
(CASSANDRA-12681)
 +Merged from 2.2:
+  * Fix merkle tree depth calculation (CASSANDRA-12580)
+  * Make Collections deserialization more robust (CASSANDRA-12618)
 - 
 - 
 -2.2.8
   * Fix exceptions when enabling gossip on nodes that haven't joined the ring 
(CASSANDRA-12253)
   * Fix authentication problem when invoking clqsh copy from a SOURCE command 
(CASSANDRA-12642)
   * Decrement pending range calculator jobs counter in finally block
(CASSANDRA-12554)
 +Merged from 2.1:
-  * Make Collections deserialization more robust (CASSANDRA-12618)
 + * Add system property to set the max number of native transport requests in 
queue (CASSANDRA-11363)
 +
 +
 +3.0.9
 + * Handle composite prefixes with final EOC=0 as in 2.x and refactor 
LegacyLayout.decodeBound (CASSANDRA-12423)
 + * Fix paging for 2.x to 3.x upgrades (CASSANDRA-11195)
 + * select_distinct_with_deletions_test failing on non-vnode environments 
(CASSANDRA-11126)
 + * Stack Overflow returned to queries while upgrading (CASSANDRA-12527)
 + * Fix legacy regex for temporary files from 2.2 (CASSANDRA-12565)
 + * Add option to state current gc_grace_seconds to tools/bin/sstablemetadata 
(CASSANDRA-12208)
 + * Fix file system race condition that may cause LogAwareFileLister to fail 
to classify files (CASSANDRA-11889)
 + * Fix file handle leaks due to simultaneous compaction/repair and
 +   listing snapshots, calculating snapshot sizes, or making schema
 +   changes (CASSANDRA-11594)
 + * Fix nodetool repair exits with 0 for some errors (CASSANDRA-12508)
 + * Do not shut down BatchlogManager twice during drain (CASSANDRA-12504)
 + * Disk failure policy should not be invoked on out of space (CASSANDRA-12385)
 + * Calculate last compacted key on startup (CASSANDRA-6216)
 + * Add schema to snapshot manifest, add USING TIMESTAMP clause to ALTER TABLE 
statements (CASSANDRA-7190)
 + * Fix clean interval not sent to commit log for empty memtable flush 
(CASSANDRA-12436)
 + * Fix potential resource leak in RMIServerSocketFactoryImpl (CASSANDRA-12331)
 + * Backport CASSANDRA-12002 (CASSANDRA-12177)
 + * Make sure compaction stats are updated when compaction is interrupted 
(CASSANDRA-12100)
 + * Fix potential bad messaging service message for paged range reads
 +   within mixed-version 3.x clusters (CASSANDRA-12249)
 + * Change commitlog and sstables to track dirty and clean intervals 
(CASSANDRA-11828)
 + * NullPointerException during compaction on table with static columns 
(CASSANDRA-12336)
 + * Fixed ConcurrentModificationException when reading metrics in 
GraphiteReporter (CASSANDRA-11823)
 + * Fix upgrade of super columns on thrift (CASSANDRA-12335)
 + * Fixed flacky BlacklistingCompactionsTest, switched 

[01/10] cassandra git commit: Fix merkle tree depth calculation

[yukim]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.2 238393520 -> c70ce6307
  refs/heads/cassandra-3.0 5cebd1fb0 -> 413e48e65
  refs/heads/cassandra-3.X 57e9a83b2 -> e3b34dc85
  refs/heads/trunk c18968b1b -> 87825f820


Fix merkle tree depth calculation

Patch by Paulo Motta; Reviewed by Yuki Morishita for CASSANDRA-12580


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c70ce630
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c70ce630
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c70ce630

Branch: refs/heads/cassandra-2.2
Commit: c70ce6307da824529762ff40673642b6f86972aa
Parents: 2383935
Author: Paulo Motta 
Authored: Tue Aug 30 21:06:39 2016 -0300
Committer: Yuki Morishita 
Committed: Thu Sep 29 14:26:53 2016 -0500

--
 CHANGES.txt |   2 +-
 .../db/compaction/CompactionManager.java|   4 +-
 .../org/apache/cassandra/repair/Validator.java  |   9 +-
 .../org/apache/cassandra/utils/MerkleTree.java  |  10 ++
 .../db/compaction/CompactionsTest.java  |   2 +-
 .../apache/cassandra/repair/ValidatorTest.java  | 159 ---
 6 files changed, 125 insertions(+), 61 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/c70ce630/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 998849e..97bc70a 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.9
+ * Fix merkle tree depth calculation (CASSANDRA-12580)
  * Make Collections deserialization more robust (CASSANDRA-12618)
  
  
@@ -36,7 +37,6 @@
  * Don't write shadowed range tombstone (CASSANDRA-12030)
 Merged from 2.1:
  * Add system property to set the max number of native transport requests in 
queue (CASSANDRA-11363)
- * Fix queries with empty ByteBuffer values in clustering column restrictions 
(CASSANDRA-12127) 
  * Disable passing control to post-flush after flush failure to prevent data 
loss (CASSANDRA-11828)
  * Allow STCS-in-L0 compactions to reduce scope with LCS (CASSANDRA-12040)
  * cannot use cql since upgrading python to 2.7.11+ (CASSANDRA-11850)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c70ce630/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java 
b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
index cf82498..78fa23c 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
@@ -1112,8 +1112,8 @@ public class CompactionManager implements 
CompactionManagerMBean
 {
 numPartitions += 
sstable.estimatedKeysForRanges(singleton(validator.desc.range));
 }
-// determine tree depth from number of partitions, but cap at 20 
to prevent large tree.
-int depth = numPartitions > 0 ? (int) 
Math.min(Math.floor(Math.log(numPartitions)), 20) : 0;
+// determine tree depth from number of partitions, but cap at 20 
to prevent large tree (CASSANDRA-5263)
+int depth = numPartitions > 0 ? (int) 
Math.min(Math.ceil(Math.log(numPartitions) / Math.log(2)), 20) : 0;
 MerkleTree tree = new MerkleTree(cfs.partitioner, 
validator.desc.range, MerkleTree.RECOMMENDED_DEPTH, (int) Math.pow(2, depth));
 
 long start = System.nanoTime();

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c70ce630/src/java/org/apache/cassandra/repair/Validator.java
--
diff --git a/src/java/org/apache/cassandra/repair/Validator.java 
b/src/java/org/apache/cassandra/repair/Validator.java
index 4db1cfb..8dbb4cf 100644
--- a/src/java/org/apache/cassandra/repair/Validator.java
+++ b/src/java/org/apache/cassandra/repair/Validator.java
@@ -54,6 +54,7 @@ public class Validator implements Runnable
 public final RepairJobDesc desc;
 public final InetAddress initiator;
 public final int gcBefore;
+private final boolean evenTreeDistribution;
 
 // null when all rows with the min token have been consumed
 private long validated;
@@ -67,19 +68,25 @@ public class Validator implements Runnable
 
 public Validator(RepairJobDesc desc, InetAddress initiator, int gcBefore)
 {
+this(desc, initiator, gcBefore, false);
+}
+
+public Validator(RepairJobDesc desc, InetAddress initiator, int gcBefore, 
boolean evenTreeDistribution)
+{
 this.desc = desc;
 this.initiator = initiator;
 this.gcBefore = gcBefore;
 validated 

[08/10] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.X

[yukim]

Merge branch 'cassandra-3.0' into cassandra-3.X


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e3b34dc8
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e3b34dc8
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e3b34dc8

Branch: refs/heads/cassandra-3.X
Commit: e3b34dc8584373c9f503e830ff3241e0865ab994
Parents: 57e9a83 413e48e
Author: Yuki Morishita 
Authored: Thu Sep 29 15:05:12 2016 -0500
Committer: Yuki Morishita 
Committed: Thu Sep 29 15:05:12 2016 -0500

--
 CHANGES.txt |   1 +
 .../db/compaction/CompactionManager.java|   6 +-
 .../org/apache/cassandra/repair/Validator.java  |   9 +-
 .../org/apache/cassandra/utils/MerkleTree.java  |  10 ++
 .../org/apache/cassandra/utils/MerkleTrees.java |  10 ++
 .../db/compaction/CompactionsTest.java  |   2 +-
 .../apache/cassandra/repair/ValidatorTest.java  | 167 ---
 7 files changed, 145 insertions(+), 60 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/e3b34dc8/CHANGES.txt
--
diff --cc CHANGES.txt
index c33b1d3,9076e7a..ba08745
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -91,59 -35,12 +91,60 @@@ Merged from 3.0
   * Disk failure policy should not be invoked on out of space (CASSANDRA-12385)
   * Calculate last compacted key on startup (CASSANDRA-6216)
   * Add schema to snapshot manifest, add USING TIMESTAMP clause to ALTER TABLE 
statements (CASSANDRA-7190)
 + * If CF has no clustering columns, any row cache is full partition cache 
(CASSANDRA-12499)
 + * Reject invalid replication settings when creating or altering a keyspace 
(CASSANDRA-12681)
 +Merged from 2.2:
++ * Fix merkle tree depth calculation (CASSANDRA-12580)
 + * Make Collections deserialization more robust (CASSANDRA-12618)
 + * Fix exceptions when enabling gossip on nodes that haven't joined the ring 
(CASSANDRA-12253)
 + * Fix authentication problem when invoking clqsh copy from a SOURCE command 
(CASSANDRA-12642)
 + * Decrement pending range calculator jobs counter in finally block
 + * cqlshlib tests: increase default execute timeout (CASSANDRA-12481)
 + * Forward writes to replacement node when replace_address != 
broadcast_address (CASSANDRA-8523)
 + * Fail repair on non-existing table (CASSANDRA-12279)
 + * Enable repair -pr and -local together (fix regression of CASSANDRA-7450) 
(CASSANDRA-12522)
 +
 +
 +3.8, 3.9
 + * Fix value skipping with counter columns (CASSANDRA-11726)
 + * Fix nodetool tablestats miss SSTable count (CASSANDRA-12205)
 + * Fixed flacky SSTablesIteratedTest (CASSANDRA-12282)
 + * Fixed flacky SSTableRewriterTest: check file counts before calling 
validateCFS (CASSANDRA-12348)
 + * cqlsh: Fix handling of $$-escaped strings (CASSANDRA-12189)
 + * Fix SSL JMX requiring truststore containing server cert (CASSANDRA-12109)
 + * RTE from new CDC column breaks in flight queries (CASSANDRA-12236)
 + * Fix hdr logging for single operation workloads (CASSANDRA-12145)
 + * Fix SASI PREFIX search in CONTAINS mode with partial terms 
(CASSANDRA-12073)
 + * Increase size of flushExecutor thread pool (CASSANDRA-12071)
 + * Partial revert of CASSANDRA-11971, cannot recycle buffer in 
SP.sendMessagesToNonlocalDC (CASSANDRA-11950)
 + * Upgrade netty to 4.0.39 (CASSANDRA-12032, CASSANDRA-12034)
 + * Improve details in compaction log message (CASSANDRA-12080)
 + * Allow unset values in CQLSSTableWriter (CASSANDRA-11911)
 + * Chunk cache to request compressor-compatible buffers if pool space is 
exhausted (CASSANDRA-11993)
 + * Remove DatabaseDescriptor dependencies from SequentialWriter 
(CASSANDRA-11579)
 + * Move skip_stop_words filter before stemming (CASSANDRA-12078)
 + * Support seek() in EncryptedFileSegmentInputStream (CASSANDRA-11957)
 + * SSTable tools mishandling LocalPartitioner (CASSANDRA-12002)
 + * When SEPWorker assigned work, set thread name to match pool 
(CASSANDRA-11966)
 + * Add cross-DC latency metrics (CASSANDRA-11596)
 + * Allow terms in selection clause (CASSANDRA-10783)
 + * Add bind variables to trace (CASSANDRA-11719)
 + * Switch counter shards' clock to timestamps (CASSANDRA-9811)
 + * Introduce HdrHistogram and response/service/wait separation to stress tool 
(CASSANDRA-11853)
 + * entry-weighers in QueryProcessor should respect partitionKeyBindIndexes 
field (CASSANDRA-11718)
 + * Support older ant versions (CASSANDRA-11807)
 + * Estimate compressed on disk size when deciding if sstable size limit 
reached (CASSANDRA-11623)
 + * cassandra-stress profiles should support case sensitive schemas 
(CASSANDRA-11546)
 + * Remove DatabaseDescriptor dependency from FileUtils (CASSANDRA-11578)
 + * Faster streaming (CASSANDRA-9766)
 + * Add 

[06/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[yukim]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/413e48e6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/413e48e6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/413e48e6

Branch: refs/heads/cassandra-3.0
Commit: 413e48e6571e3c23362d5053e0c7fcdd99bd1e7d
Parents: 5cebd1f c70ce63
Author: Yuki Morishita 
Authored: Thu Sep 29 14:32:49 2016 -0500
Committer: Yuki Morishita 
Committed: Thu Sep 29 14:32:49 2016 -0500

--
 CHANGES.txt |   3 +-
 .../db/compaction/CompactionManager.java|   6 +-
 .../org/apache/cassandra/repair/Validator.java  |   9 +-
 .../org/apache/cassandra/utils/MerkleTree.java  |  10 ++
 .../org/apache/cassandra/utils/MerkleTrees.java |  10 ++
 .../db/compaction/CompactionsTest.java  |   2 +-
 .../apache/cassandra/repair/ValidatorTest.java  | 167 ---
 7 files changed, 146 insertions(+), 61 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/413e48e6/CHANGES.txt
--
diff --cc CHANGES.txt
index f2f8dac,97bc70a..9076e7a
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,77 -1,13 +1,78 @@@
 -2.2.9
 +3.0.10
 + * Fix failure in LogTransactionTest (CASSANDRA-12632)
 + * Fix potentially incomplete non-frozen UDT values when querying with the
 +   full primary key specified (CASSANDRA-12605)
 + * Skip writing MV mutations to commitlog on mutation.applyUnsafe() 
(CASSANDRA-11670)
 + * Establish consistent distinction between non-existing partition and NULL 
value for LWTs on static columns (CASSANDRA-12060)
 + * Extend ColumnIdentifier.internedInstances key to include the type that 
generated the byte buffer (CASSANDRA-12516)
 + * Backport CASSANDRA-10756 (race condition in NativeTransportService 
shutdown) (CASSANDRA-12472)
 + * If CF has no clustering columns, any row cache is full partition cache 
(CASSANDRA-12499)
 + * Reject invalid replication settings when creating or altering a keyspace 
(CASSANDRA-12681)
 +Merged from 2.2:
+  * Fix merkle tree depth calculation (CASSANDRA-12580)
+  * Make Collections deserialization more robust (CASSANDRA-12618)
 - 
 - 
 -2.2.8
   * Fix exceptions when enabling gossip on nodes that haven't joined the ring 
(CASSANDRA-12253)
   * Fix authentication problem when invoking clqsh copy from a SOURCE command 
(CASSANDRA-12642)
   * Decrement pending range calculator jobs counter in finally block
(CASSANDRA-12554)
 +Merged from 2.1:
-  * Make Collections deserialization more robust (CASSANDRA-12618)
 + * Add system property to set the max number of native transport requests in 
queue (CASSANDRA-11363)
 +
 +
 +3.0.9
 + * Handle composite prefixes with final EOC=0 as in 2.x and refactor 
LegacyLayout.decodeBound (CASSANDRA-12423)
 + * Fix paging for 2.x to 3.x upgrades (CASSANDRA-11195)
 + * select_distinct_with_deletions_test failing on non-vnode environments 
(CASSANDRA-11126)
 + * Stack Overflow returned to queries while upgrading (CASSANDRA-12527)
 + * Fix legacy regex for temporary files from 2.2 (CASSANDRA-12565)
 + * Add option to state current gc_grace_seconds to tools/bin/sstablemetadata 
(CASSANDRA-12208)
 + * Fix file system race condition that may cause LogAwareFileLister to fail 
to classify files (CASSANDRA-11889)
 + * Fix file handle leaks due to simultaneous compaction/repair and
 +   listing snapshots, calculating snapshot sizes, or making schema
 +   changes (CASSANDRA-11594)
 + * Fix nodetool repair exits with 0 for some errors (CASSANDRA-12508)
 + * Do not shut down BatchlogManager twice during drain (CASSANDRA-12504)
 + * Disk failure policy should not be invoked on out of space (CASSANDRA-12385)
 + * Calculate last compacted key on startup (CASSANDRA-6216)
 + * Add schema to snapshot manifest, add USING TIMESTAMP clause to ALTER TABLE 
statements (CASSANDRA-7190)
 + * Fix clean interval not sent to commit log for empty memtable flush 
(CASSANDRA-12436)
 + * Fix potential resource leak in RMIServerSocketFactoryImpl (CASSANDRA-12331)
 + * Backport CASSANDRA-12002 (CASSANDRA-12177)
 + * Make sure compaction stats are updated when compaction is interrupted 
(CASSANDRA-12100)
 + * Fix potential bad messaging service message for paged range reads
 +   within mixed-version 3.x clusters (CASSANDRA-12249)
 + * Change commitlog and sstables to track dirty and clean intervals 
(CASSANDRA-11828)
 + * NullPointerException during compaction on table with static columns 
(CASSANDRA-12336)
 + * Fixed ConcurrentModificationException when reading metrics in 
GraphiteReporter (CASSANDRA-11823)
 + * Fix upgrade of super columns on thrift (CASSANDRA-12335)
 + * Fixed flacky BlacklistingCompactionsTest, 

[04/10] cassandra git commit: Fix merkle tree depth calculation

[yukim]

Fix merkle tree depth calculation

Patch by Paulo Motta; Reviewed by Yuki Morishita for CASSANDRA-12580


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c70ce630
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c70ce630
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c70ce630

Branch: refs/heads/trunk
Commit: c70ce6307da824529762ff40673642b6f86972aa
Parents: 2383935
Author: Paulo Motta 
Authored: Tue Aug 30 21:06:39 2016 -0300
Committer: Yuki Morishita 
Committed: Thu Sep 29 14:26:53 2016 -0500

--
 CHANGES.txt |   2 +-
 .../db/compaction/CompactionManager.java|   4 +-
 .../org/apache/cassandra/repair/Validator.java  |   9 +-
 .../org/apache/cassandra/utils/MerkleTree.java  |  10 ++
 .../db/compaction/CompactionsTest.java  |   2 +-
 .../apache/cassandra/repair/ValidatorTest.java  | 159 ---
 6 files changed, 125 insertions(+), 61 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/c70ce630/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 998849e..97bc70a 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.9
+ * Fix merkle tree depth calculation (CASSANDRA-12580)
  * Make Collections deserialization more robust (CASSANDRA-12618)
  
  
@@ -36,7 +37,6 @@
  * Don't write shadowed range tombstone (CASSANDRA-12030)
 Merged from 2.1:
  * Add system property to set the max number of native transport requests in 
queue (CASSANDRA-11363)
- * Fix queries with empty ByteBuffer values in clustering column restrictions 
(CASSANDRA-12127) 
  * Disable passing control to post-flush after flush failure to prevent data 
loss (CASSANDRA-11828)
  * Allow STCS-in-L0 compactions to reduce scope with LCS (CASSANDRA-12040)
  * cannot use cql since upgrading python to 2.7.11+ (CASSANDRA-11850)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c70ce630/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java 
b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
index cf82498..78fa23c 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
@@ -1112,8 +1112,8 @@ public class CompactionManager implements 
CompactionManagerMBean
 {
 numPartitions += 
sstable.estimatedKeysForRanges(singleton(validator.desc.range));
 }
-// determine tree depth from number of partitions, but cap at 20 
to prevent large tree.
-int depth = numPartitions > 0 ? (int) 
Math.min(Math.floor(Math.log(numPartitions)), 20) : 0;
+// determine tree depth from number of partitions, but cap at 20 
to prevent large tree (CASSANDRA-5263)
+int depth = numPartitions > 0 ? (int) 
Math.min(Math.ceil(Math.log(numPartitions) / Math.log(2)), 20) : 0;
 MerkleTree tree = new MerkleTree(cfs.partitioner, 
validator.desc.range, MerkleTree.RECOMMENDED_DEPTH, (int) Math.pow(2, depth));
 
 long start = System.nanoTime();

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c70ce630/src/java/org/apache/cassandra/repair/Validator.java
--
diff --git a/src/java/org/apache/cassandra/repair/Validator.java 
b/src/java/org/apache/cassandra/repair/Validator.java
index 4db1cfb..8dbb4cf 100644
--- a/src/java/org/apache/cassandra/repair/Validator.java
+++ b/src/java/org/apache/cassandra/repair/Validator.java
@@ -54,6 +54,7 @@ public class Validator implements Runnable
 public final RepairJobDesc desc;
 public final InetAddress initiator;
 public final int gcBefore;
+private final boolean evenTreeDistribution;
 
 // null when all rows with the min token have been consumed
 private long validated;
@@ -67,19 +68,25 @@ public class Validator implements Runnable
 
 public Validator(RepairJobDesc desc, InetAddress initiator, int gcBefore)
 {
+this(desc, initiator, gcBefore, false);
+}
+
+public Validator(RepairJobDesc desc, InetAddress initiator, int gcBefore, 
boolean evenTreeDistribution)
+{
 this.desc = desc;
 this.initiator = initiator;
 this.gcBefore = gcBefore;
 validated = 0;
 range = null;
 ranges = null;
+this.evenTreeDistribution = evenTreeDistribution;
 }
 
 public void prepare(ColumnFamilyStore cfs, MerkleTree tree)
 {
 this.tree = tree;
 
-if 

[03/10] cassandra git commit: Fix merkle tree depth calculation

[yukim]

Fix merkle tree depth calculation

Patch by Paulo Motta; Reviewed by Yuki Morishita for CASSANDRA-12580


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c70ce630
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c70ce630
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c70ce630

Branch: refs/heads/cassandra-3.X
Commit: c70ce6307da824529762ff40673642b6f86972aa
Parents: 2383935
Author: Paulo Motta 
Authored: Tue Aug 30 21:06:39 2016 -0300
Committer: Yuki Morishita 
Committed: Thu Sep 29 14:26:53 2016 -0500

--
 CHANGES.txt |   2 +-
 .../db/compaction/CompactionManager.java|   4 +-
 .../org/apache/cassandra/repair/Validator.java  |   9 +-
 .../org/apache/cassandra/utils/MerkleTree.java  |  10 ++
 .../db/compaction/CompactionsTest.java  |   2 +-
 .../apache/cassandra/repair/ValidatorTest.java  | 159 ---
 6 files changed, 125 insertions(+), 61 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/c70ce630/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 998849e..97bc70a 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.9
+ * Fix merkle tree depth calculation (CASSANDRA-12580)
  * Make Collections deserialization more robust (CASSANDRA-12618)
  
  
@@ -36,7 +37,6 @@
  * Don't write shadowed range tombstone (CASSANDRA-12030)
 Merged from 2.1:
  * Add system property to set the max number of native transport requests in 
queue (CASSANDRA-11363)
- * Fix queries with empty ByteBuffer values in clustering column restrictions 
(CASSANDRA-12127) 
  * Disable passing control to post-flush after flush failure to prevent data 
loss (CASSANDRA-11828)
  * Allow STCS-in-L0 compactions to reduce scope with LCS (CASSANDRA-12040)
  * cannot use cql since upgrading python to 2.7.11+ (CASSANDRA-11850)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c70ce630/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java 
b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
index cf82498..78fa23c 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
@@ -1112,8 +1112,8 @@ public class CompactionManager implements 
CompactionManagerMBean
 {
 numPartitions += 
sstable.estimatedKeysForRanges(singleton(validator.desc.range));
 }
-// determine tree depth from number of partitions, but cap at 20 
to prevent large tree.
-int depth = numPartitions > 0 ? (int) 
Math.min(Math.floor(Math.log(numPartitions)), 20) : 0;
+// determine tree depth from number of partitions, but cap at 20 
to prevent large tree (CASSANDRA-5263)
+int depth = numPartitions > 0 ? (int) 
Math.min(Math.ceil(Math.log(numPartitions) / Math.log(2)), 20) : 0;
 MerkleTree tree = new MerkleTree(cfs.partitioner, 
validator.desc.range, MerkleTree.RECOMMENDED_DEPTH, (int) Math.pow(2, depth));
 
 long start = System.nanoTime();

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c70ce630/src/java/org/apache/cassandra/repair/Validator.java
--
diff --git a/src/java/org/apache/cassandra/repair/Validator.java 
b/src/java/org/apache/cassandra/repair/Validator.java
index 4db1cfb..8dbb4cf 100644
--- a/src/java/org/apache/cassandra/repair/Validator.java
+++ b/src/java/org/apache/cassandra/repair/Validator.java
@@ -54,6 +54,7 @@ public class Validator implements Runnable
 public final RepairJobDesc desc;
 public final InetAddress initiator;
 public final int gcBefore;
+private final boolean evenTreeDistribution;
 
 // null when all rows with the min token have been consumed
 private long validated;
@@ -67,19 +68,25 @@ public class Validator implements Runnable
 
 public Validator(RepairJobDesc desc, InetAddress initiator, int gcBefore)
 {
+this(desc, initiator, gcBefore, false);
+}
+
+public Validator(RepairJobDesc desc, InetAddress initiator, int gcBefore, 
boolean evenTreeDistribution)
+{
 this.desc = desc;
 this.initiator = initiator;
 this.gcBefore = gcBefore;
 validated = 0;
 range = null;
 ranges = null;
+this.evenTreeDistribution = evenTreeDistribution;
 }
 
 public void prepare(ColumnFamilyStore cfs, MerkleTree tree)
 {
 this.tree = tree;
 
-if 

[05/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[yukim]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/413e48e6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/413e48e6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/413e48e6

Branch: refs/heads/cassandra-3.X
Commit: 413e48e6571e3c23362d5053e0c7fcdd99bd1e7d
Parents: 5cebd1f c70ce63
Author: Yuki Morishita 
Authored: Thu Sep 29 14:32:49 2016 -0500
Committer: Yuki Morishita 
Committed: Thu Sep 29 14:32:49 2016 -0500

--
 CHANGES.txt |   3 +-
 .../db/compaction/CompactionManager.java|   6 +-
 .../org/apache/cassandra/repair/Validator.java  |   9 +-
 .../org/apache/cassandra/utils/MerkleTree.java  |  10 ++
 .../org/apache/cassandra/utils/MerkleTrees.java |  10 ++
 .../db/compaction/CompactionsTest.java  |   2 +-
 .../apache/cassandra/repair/ValidatorTest.java  | 167 ---
 7 files changed, 146 insertions(+), 61 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/413e48e6/CHANGES.txt
--
diff --cc CHANGES.txt
index f2f8dac,97bc70a..9076e7a
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,77 -1,13 +1,78 @@@
 -2.2.9
 +3.0.10
 + * Fix failure in LogTransactionTest (CASSANDRA-12632)
 + * Fix potentially incomplete non-frozen UDT values when querying with the
 +   full primary key specified (CASSANDRA-12605)
 + * Skip writing MV mutations to commitlog on mutation.applyUnsafe() 
(CASSANDRA-11670)
 + * Establish consistent distinction between non-existing partition and NULL 
value for LWTs on static columns (CASSANDRA-12060)
 + * Extend ColumnIdentifier.internedInstances key to include the type that 
generated the byte buffer (CASSANDRA-12516)
 + * Backport CASSANDRA-10756 (race condition in NativeTransportService 
shutdown) (CASSANDRA-12472)
 + * If CF has no clustering columns, any row cache is full partition cache 
(CASSANDRA-12499)
 + * Reject invalid replication settings when creating or altering a keyspace 
(CASSANDRA-12681)
 +Merged from 2.2:
+  * Fix merkle tree depth calculation (CASSANDRA-12580)
+  * Make Collections deserialization more robust (CASSANDRA-12618)
 - 
 - 
 -2.2.8
   * Fix exceptions when enabling gossip on nodes that haven't joined the ring 
(CASSANDRA-12253)
   * Fix authentication problem when invoking clqsh copy from a SOURCE command 
(CASSANDRA-12642)
   * Decrement pending range calculator jobs counter in finally block
(CASSANDRA-12554)
 +Merged from 2.1:
-  * Make Collections deserialization more robust (CASSANDRA-12618)
 + * Add system property to set the max number of native transport requests in 
queue (CASSANDRA-11363)
 +
 +
 +3.0.9
 + * Handle composite prefixes with final EOC=0 as in 2.x and refactor 
LegacyLayout.decodeBound (CASSANDRA-12423)
 + * Fix paging for 2.x to 3.x upgrades (CASSANDRA-11195)
 + * select_distinct_with_deletions_test failing on non-vnode environments 
(CASSANDRA-11126)
 + * Stack Overflow returned to queries while upgrading (CASSANDRA-12527)
 + * Fix legacy regex for temporary files from 2.2 (CASSANDRA-12565)
 + * Add option to state current gc_grace_seconds to tools/bin/sstablemetadata 
(CASSANDRA-12208)
 + * Fix file system race condition that may cause LogAwareFileLister to fail 
to classify files (CASSANDRA-11889)
 + * Fix file handle leaks due to simultaneous compaction/repair and
 +   listing snapshots, calculating snapshot sizes, or making schema
 +   changes (CASSANDRA-11594)
 + * Fix nodetool repair exits with 0 for some errors (CASSANDRA-12508)
 + * Do not shut down BatchlogManager twice during drain (CASSANDRA-12504)
 + * Disk failure policy should not be invoked on out of space (CASSANDRA-12385)
 + * Calculate last compacted key on startup (CASSANDRA-6216)
 + * Add schema to snapshot manifest, add USING TIMESTAMP clause to ALTER TABLE 
statements (CASSANDRA-7190)
 + * Fix clean interval not sent to commit log for empty memtable flush 
(CASSANDRA-12436)
 + * Fix potential resource leak in RMIServerSocketFactoryImpl (CASSANDRA-12331)
 + * Backport CASSANDRA-12002 (CASSANDRA-12177)
 + * Make sure compaction stats are updated when compaction is interrupted 
(CASSANDRA-12100)
 + * Fix potential bad messaging service message for paged range reads
 +   within mixed-version 3.x clusters (CASSANDRA-12249)
 + * Change commitlog and sstables to track dirty and clean intervals 
(CASSANDRA-11828)
 + * NullPointerException during compaction on table with static columns 
(CASSANDRA-12336)
 + * Fixed ConcurrentModificationException when reading metrics in 
GraphiteReporter (CASSANDRA-11823)
 + * Fix upgrade of super columns on thrift (CASSANDRA-12335)
 + * Fixed flacky BlacklistingCompactionsTest, 

[10/10] cassandra git commit: Merge branch 'cassandra-3.X' into trunk

[yukim]

Merge branch 'cassandra-3.X' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/87825f82
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/87825f82
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/87825f82

Branch: refs/heads/trunk
Commit: 87825f8208fc21000a841f724c7f89f3702d8e70
Parents: c18968b e3b34dc
Author: Yuki Morishita 
Authored: Thu Sep 29 15:05:25 2016 -0500
Committer: Yuki Morishita 
Committed: Thu Sep 29 15:05:25 2016 -0500

--
 CHANGES.txt |   1 +
 .../db/compaction/CompactionManager.java|   6 +-
 .../org/apache/cassandra/repair/Validator.java  |   9 +-
 .../org/apache/cassandra/utils/MerkleTree.java  |  10 ++
 .../org/apache/cassandra/utils/MerkleTrees.java |  10 ++
 .../db/compaction/CompactionsTest.java  |   2 +-
 .../apache/cassandra/repair/ValidatorTest.java  | 167 ---
 7 files changed, 145 insertions(+), 60 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/87825f82/CHANGES.txt
--

svn commit: r15993 - in /release/cassandra: 3.8/ debian/dists/38x/ debian/dists/38x/main/ debian/dists/38x/main/binary-amd64/ debian/dists/38x/main/binary-i386/ debian/dists/38x/main/source/ debian/po

[mshuler]

Author: mshuler
Date: Thu Sep 29 19:38:38 2016
New Revision: 15993

Log:
Apache Cassandra 3.8 Release

Added:
release/cassandra/3.8/
release/cassandra/3.8/apache-cassandra-3.8-bin.tar.gz   (with props)
release/cassandra/3.8/apache-cassandra-3.8-bin.tar.gz.asc
release/cassandra/3.8/apache-cassandra-3.8-bin.tar.gz.asc.md5
release/cassandra/3.8/apache-cassandra-3.8-bin.tar.gz.asc.sha1
release/cassandra/3.8/apache-cassandra-3.8-bin.tar.gz.md5
release/cassandra/3.8/apache-cassandra-3.8-bin.tar.gz.sha1
release/cassandra/3.8/apache-cassandra-3.8-src.tar.gz   (with props)
release/cassandra/3.8/apache-cassandra-3.8-src.tar.gz.asc
release/cassandra/3.8/apache-cassandra-3.8-src.tar.gz.asc.md5
release/cassandra/3.8/apache-cassandra-3.8-src.tar.gz.asc.sha1
release/cassandra/3.8/apache-cassandra-3.8-src.tar.gz.md5
release/cassandra/3.8/apache-cassandra-3.8-src.tar.gz.sha1
release/cassandra/debian/dists/38x/
release/cassandra/debian/dists/38x/InRelease
release/cassandra/debian/dists/38x/Release
release/cassandra/debian/dists/38x/Release.gpg
release/cassandra/debian/dists/38x/main/
release/cassandra/debian/dists/38x/main/binary-amd64/
release/cassandra/debian/dists/38x/main/binary-amd64/Packages
release/cassandra/debian/dists/38x/main/binary-amd64/Packages.gz   (with 
props)
release/cassandra/debian/dists/38x/main/binary-amd64/Release
release/cassandra/debian/dists/38x/main/binary-i386/
release/cassandra/debian/dists/38x/main/binary-i386/Packages
release/cassandra/debian/dists/38x/main/binary-i386/Packages.gz   (with 
props)
release/cassandra/debian/dists/38x/main/binary-i386/Release
release/cassandra/debian/dists/38x/main/source/
release/cassandra/debian/dists/38x/main/source/Release
release/cassandra/debian/dists/38x/main/source/Sources.gz   (with props)
release/cassandra/debian/pool/main/c/cassandra/cassandra-tools_3.8_all.deb  
 (with props)
release/cassandra/debian/pool/main/c/cassandra/cassandra_3.8.diff.gz   
(with props)
release/cassandra/debian/pool/main/c/cassandra/cassandra_3.8.dsc
release/cassandra/debian/pool/main/c/cassandra/cassandra_3.8.orig.tar.gz   
(with props)
release/cassandra/debian/pool/main/c/cassandra/cassandra_3.8.orig.tar.gz.asc
release/cassandra/debian/pool/main/c/cassandra/cassandra_3.8_all.deb   
(with props)

Added: release/cassandra/3.8/apache-cassandra-3.8-bin.tar.gz
==
Binary file - no diff available.

Propchange: release/cassandra/3.8/apache-cassandra-3.8-bin.tar.gz
--
svn:mime-type = application/octet-stream

Added: release/cassandra/3.8/apache-cassandra-3.8-bin.tar.gz.asc
==
--- release/cassandra/3.8/apache-cassandra-3.8-bin.tar.gz.asc (added)
+++ release/cassandra/3.8/apache-cassandra-3.8-bin.tar.gz.asc Thu Sep 29 
19:38:38 2016
@@ -0,0 +1,17 @@
+-BEGIN PGP SIGNATURE-
+Version: GnuPG v1
+
+iQIcBAABCAAGBQJX6TNJAAoJEKJ4t4H+SyvaR5gP/j9zPX4H6he3kehQG/TjUDbQ
+youq2M4ipk+ZyWP19veSBkb78sKh5Ak2kgSlnyjqpTfvY5Z9hEVThPJTldP2mXyg
+/+hiSUL0tx/wNyeNy3XeH//S6P0WnDZ62dTX96HMO6BRdqP1X5wZsK8attbj67jm
+i4DkOh+KcLraXgjcsByZKrg0mGJ68f2V60p8qHwH2/rCfvSNDr46QtiEq7U0O2Jq
+wDbHpEhNe6DcdJhH9OEqBmvBT12N1MRADQUz+R1rWEI2lb889cqLiK2YG8w9LFMK
++EW17EWHBIBBHpR4B0wQdle8qMCmgzNliiOhm1btqxh5YsIm7fAvxJhAFva/jG8W
+vQyPVB4xU+SCLJi+A6qxK1qBTH4bikcitab2w/zNPxUE0A35AJmbmD4pCaRsqH6I
+EMoH2syYUWxdMX7V5lmF+1PamCYHrglasGhl5qW+9GBQf3M32xikcdj2SDcpbpZN
+FbcwanNbiDSahTC+qr0v+NDji2Xa6wAJShqK6b8xAsimiSKXmuKkH97X8Ur8b/qk
+k/g/FmH7tRi7Hym3jU4j3I7tXKSBRpm5aQA3Kjob2lLmcY3dAc9xP3tys6r4U6OX
+NlUGTwik0OPWgpu/SDBM0LrkUdnE8PrC5wgPcZqf/TaeTs/RQK87X2Z/rOkiX/qQ
+W9s7uep3UaQVAJrfnLuu
+=Z4Uy
+-END PGP SIGNATURE-

Added: release/cassandra/3.8/apache-cassandra-3.8-bin.tar.gz.asc.md5
==
--- release/cassandra/3.8/apache-cassandra-3.8-bin.tar.gz.asc.md5 (added)
+++ release/cassandra/3.8/apache-cassandra-3.8-bin.tar.gz.asc.md5 Thu Sep 29 
19:38:38 2016
@@ -0,0 +1 @@
+a30ad57aec8cbd65b459e3d87f987306
\ No newline at end of file

Added: release/cassandra/3.8/apache-cassandra-3.8-bin.tar.gz.asc.sha1
==
--- release/cassandra/3.8/apache-cassandra-3.8-bin.tar.gz.asc.sha1 (added)
+++ release/cassandra/3.8/apache-cassandra-3.8-bin.tar.gz.asc.sha1 Thu Sep 29 
19:38:38 2016
@@ -0,0 +1 @@
+7e37a2ce26cd3a0214487ac9b1ce19957139bf9a
\ No newline at end of file

Added: release/cassandra/3.8/apache-cassandra-3.8-bin.tar.gz.md5
==
--- release/cassandra/3.8/apache-cassandra-3.8-bin.tar.gz.md5 (added)
+++ release/cassandra/3.8/apache-cassandra-3.8-bin.tar.gz.md5 Thu 

[cassandra] Git Push Summary

[mshuler]

Repository: cassandra
Updated Tags:  refs/tags/3.8-tentative [deleted] ce609d19f

[cassandra] Git Push Summary

[mshuler]

Repository: cassandra
Updated Tags:  refs/tags/cassandra-3.8 [created] d414b4e4b

[jira] [Updated] (CASSANDRA-12733) Throw an exception if there is a prepared statement id hash conflict.

[Jeremiah Jordan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeremiah Jordan updated CASSANDRA-12733:

Status: Patch Available  (was: Open)

> Throw an exception if there is a prepared statement id hash conflict.
> -
>
> Key: CASSANDRA-12733
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12733
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CQL
>Reporter: Jeremiah Jordan
>Assignee: Jeremiah Jordan
>Priority: Minor
>
> I seriously doubt there is any chance of actually getting two prepared 
> statement strings that have the same MD5.  But there should probably be 
> checks in QueryProcessor.getStoredPreparedStatement that the query string of 
> the statement being prepared matches the query string of the ID returned from 
> the cache when one already exists there.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-12733) Throw an exception if there is a prepared statement id hash conflict.

[Jeremiah Jordan (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533728#comment-15533728
 ] 

Jeremiah Jordan edited comment on CASSANDRA-12733 at 9/29/16 7:17 PM:
--

||branch||testall||dtest||
|[CASSANDRA-12733-3X|https://github.com/JeremiahDJordan/cassandra/tree/CASSANDRA-12733-3X]|[testall|http://cassci.datastax.com/job/JeremiahDJordan-CASSANDRA-12733-3X-testall/]|[dtest|http://cassci.datastax.com/job/JeremiahDJordan-CASSANDRA-12733-3X-dtest/]|
|[CASSANDRA-12733-trunk|https://github.com/JeremiahDJordan/cassandra/tree/CASSANDRA-12733-trunk]|[testall|http://cassci.datastax.com/job/JeremiahDJordan-CASSANDRA-12733-trunk-testall/]|[dtest|http://cassci.datastax.com/job/JeremiahDJordan-CASSANDRA-12733-trunk-dtest/]|



was (Author: jjordan):
||branch||testall||dtest||
|[CASSANDRA-12733-22|https://github.com/JeremiahDJordan/cassandra/tree/CASSANDRA-12733-22]|[testall|http://cassci.datastax.com/job/JeremiahDJordan-CASSANDRA-12733-22-testall/]|[dtest|http://cassci.datastax.com/job/JeremiahDJordan-CASSANDRA-12733-22-dtest/]|
|[CASSANDRA-12733-30|https://github.com/JeremiahDJordan/cassandra/tree/CASSANDRA-12733-30]|[testall|http://cassci.datastax.com/job/JeremiahDJordan-CASSANDRA-12733-30-testall/]|[dtest|http://cassci.datastax.com/job/JeremiahDJordan-CASSANDRA-12733-30-dtest/]|
|[CASSANDRA-12733-3X|https://github.com/JeremiahDJordan/cassandra/tree/CASSANDRA-12733-3X]|[testall|http://cassci.datastax.com/job/JeremiahDJordan-CASSANDRA-12733-3X-testall/]|[dtest|http://cassci.datastax.com/job/JeremiahDJordan-CASSANDRA-12733-3X-dtest/]|
|[CASSANDRA-12733-trunk|https://github.com/JeremiahDJordan/cassandra/tree/CASSANDRA-12733-trunk]|[testall|http://cassci.datastax.com/job/JeremiahDJordan-CASSANDRA-12733-trunk-testall/]|[dtest|http://cassci.datastax.com/job/JeremiahDJordan-CASSANDRA-12733-trunk-dtest/]|

2.2 branch merges cleanly up to trunk.

> Throw an exception if there is a prepared statement id hash conflict.
> -
>
> Key: CASSANDRA-12733
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12733
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CQL
>Reporter: Jeremiah Jordan
>Assignee: Jeremiah Jordan
>Priority: Minor
>
> I seriously doubt there is any chance of actually getting two prepared 
> statement strings that have the same MD5.  But there should probably be 
> checks in QueryProcessor.getStoredPreparedStatement that the query string of 
> the statement being prepared matches the query string of the ID returned from 
> the cache when one already exists there.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-12733) Throw an exception if there is a prepared statement id hash conflict.

[Jeremiah Jordan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeremiah Jordan updated CASSANDRA-12733:

Status: Open  (was: Patch Available)

> Throw an exception if there is a prepared statement id hash conflict.
> -
>
> Key: CASSANDRA-12733
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12733
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CQL
>Reporter: Jeremiah Jordan
>Assignee: Jeremiah Jordan
>Priority: Minor
>
> I seriously doubt there is any chance of actually getting two prepared 
> statement strings that have the same MD5.  But there should probably be 
> checks in QueryProcessor.getStoredPreparedStatement that the query string of 
> the statement being prepared matches the query string of the ID returned from 
> the cache when one already exists there.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-12733) Throw an exception if there is a prepared statement id hash conflict.

[Jeremiah Jordan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeremiah Jordan updated CASSANDRA-12733:

Status: Patch Available  (was: Open)

> Throw an exception if there is a prepared statement id hash conflict.
> -
>
> Key: CASSANDRA-12733
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12733
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CQL
>Reporter: Jeremiah Jordan
>Assignee: Jeremiah Jordan
>Priority: Minor
>
> I seriously doubt there is any chance of actually getting two prepared 
> statement strings that have the same MD5.  But there should probably be 
> checks in QueryProcessor.getStoredPreparedStatement that the query string of 
> the statement being prepared matches the query string of the ID returned from 
> the cache when one already exists there.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-12733) Throw an exception if there is a prepared statement id hash conflict.

[Jeremiah Jordan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeremiah Jordan updated CASSANDRA-12733:

Priority: Minor  (was: Major)

> Throw an exception if there is a prepared statement id hash conflict.
> -
>
> Key: CASSANDRA-12733
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12733
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CQL
>Reporter: Jeremiah Jordan
>Assignee: Jeremiah Jordan
>Priority: Minor
>
> I seriously doubt there is any chance of actually getting two prepared 
> statement strings that have the same MD5.  But there should probably be 
> checks in QueryProcessor.getStoredPreparedStatement that the query string of 
> the statement being prepared matches the query string of the ID returned from 
> the cache when one already exists there.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-12733) Throw an exception if there is a prepared statement id hash conflict.

[Jeremiah Jordan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeremiah Jordan updated CASSANDRA-12733:

Component/s: CQL

> Throw an exception if there is a prepared statement id hash conflict.
> -
>
> Key: CASSANDRA-12733
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12733
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CQL
>Reporter: Jeremiah Jordan
>Assignee: Jeremiah Jordan
>Priority: Minor
>
> I seriously doubt there is any chance of actually getting two prepared 
> statement strings that have the same MD5.  But there should probably be 
> checks in QueryProcessor.getStoredPreparedStatement that the query string of 
> the statement being prepared matches the query string of the ID returned from 
> the cache when one already exists there.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12733) Throw an exception if there is a prepared statement id hash conflict.

[Jeremiah Jordan (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533728#comment-15533728
 ] 

Jeremiah Jordan commented on CASSANDRA-12733:
-

||branch||testall||dtest||
|[CASSANDRA-12733-22|https://github.com/JeremiahDJordan/cassandra/tree/CASSANDRA-12733-22]|[testall|http://cassci.datastax.com/job/JeremiahDJordan-CASSANDRA-12733-22-testall/]|[dtest|http://cassci.datastax.com/job/JeremiahDJordan-CASSANDRA-12733-22-dtest/]|
|[CASSANDRA-12733-30|https://github.com/JeremiahDJordan/cassandra/tree/CASSANDRA-12733-30]|[testall|http://cassci.datastax.com/job/JeremiahDJordan-CASSANDRA-12733-30-testall/]|[dtest|http://cassci.datastax.com/job/JeremiahDJordan-CASSANDRA-12733-30-dtest/]|
|[CASSANDRA-12733-3X|https://github.com/JeremiahDJordan/cassandra/tree/CASSANDRA-12733-3X]|[testall|http://cassci.datastax.com/job/JeremiahDJordan-CASSANDRA-12733-3X-testall/]|[dtest|http://cassci.datastax.com/job/JeremiahDJordan-CASSANDRA-12733-3X-dtest/]|
|[CASSANDRA-12733-trunk|https://github.com/JeremiahDJordan/cassandra/tree/CASSANDRA-12733-trunk]|[testall|http://cassci.datastax.com/job/JeremiahDJordan-CASSANDRA-12733-trunk-testall/]|[dtest|http://cassci.datastax.com/job/JeremiahDJordan-CASSANDRA-12733-trunk-dtest/]|

2.2 branch merges cleanly up to trunk.

> Throw an exception if there is a prepared statement id hash conflict.
> -
>
> Key: CASSANDRA-12733
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12733
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Jeremiah Jordan
>Assignee: Jeremiah Jordan
>
> I seriously doubt there is any chance of actually getting two prepared 
> statement strings that have the same MD5.  But there should probably be 
> checks in QueryProcessor.getStoredPreparedStatement that the query string of 
> the statement being prepared matches the query string of the ID returned from 
> the cache when one already exists there.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (CASSANDRA-12733) Throw an exception if there is a prepared statement id hash conflict.

[Jeremiah Jordan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeremiah Jordan reassigned CASSANDRA-12733:
---

Assignee: Jeremiah Jordan

> Throw an exception if there is a prepared statement id hash conflict.
> -
>
> Key: CASSANDRA-12733
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12733
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Jeremiah Jordan
>Assignee: Jeremiah Jordan
>
> I seriously doubt there is any chance of actually getting two prepared 
> statement strings that have the same MD5.  But there should probably be 
> checks in QueryProcessor.getStoredPreparedStatement that the query string of 
> the statement being prepared matches the query string of the ID returned from 
> the cache when one already exists there.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-7190) Add schema to snapshot manifest

[Hau Phan (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533480#comment-15533480
 ] 

Hau Phan edited comment on CASSANDRA-7190 at 9/29/16 6:46 PM:
--

I noticed that the schema.cql file that gets generated with snapshots for 
materialized views are being treated as a table instead of a materialized view. 

Also to note, the mv schema is not included with the table schema.cql from 
which it's linked to.

Should I create a new ticket?


was (Author: nothau):
I noticed that the schema.cql file that gets generated with snapshots for 
materialized views are being treated as a table instead of a materialized view. 

Should I create a new ticket?

> Add schema to snapshot manifest
> ---
>
> Key: CASSANDRA-7190
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7190
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Jonathan Ellis
>Assignee: Alex Petrov
>Priority: Minor
>  Labels: client-impacting, doc-impacting, lhf
> Fix For: 3.0.9, 3.10
>
>
> followup from CASSANDRA-6326



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-12733) Throw an exception if there is a prepared statement id hash conflict.

[Jeremiah Jordan (JIRA)]

Jeremiah Jordan created CASSANDRA-12733:
---

 Summary: Throw an exception if there is a prepared statement id 
hash conflict.
 Key: CASSANDRA-12733
 URL: https://issues.apache.org/jira/browse/CASSANDRA-12733
 Project: Cassandra
  Issue Type: Improvement
Reporter: Jeremiah Jordan


I seriously doubt there is any chance of actually getting two prepared 
statement strings that have the same MD5.  But there should probably be checks 
in QueryProcessor.getStoredPreparedStatement that the query string of the 
statement being prepared matches the query string of the ID returned from the 
cache when one already exists there.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-12732) sstableofflinerelevel missing in tools deb build

[Zach Aller (JIRA)]

Zach Aller created CASSANDRA-12732:
--

 Summary: sstableofflinerelevel missing in tools deb build
 Key: CASSANDRA-12732
 URL: https://issues.apache.org/jira/browse/CASSANDRA-12732
 Project: Cassandra
  Issue Type: Bug
Reporter: Zach Aller
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-12732) sstableofflinerelevel missing in tools deb package

[Zach Aller (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zach Aller updated CASSANDRA-12732:
---
Summary: sstableofflinerelevel missing in tools deb package  (was: 
sstableofflinerelevel missing in tools deb build)

> sstableofflinerelevel missing in tools deb package
> --
>
> Key: CASSANDRA-12732
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12732
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Zach Aller
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-7190) Add schema to snapshot manifest

[Hau Phan (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533480#comment-15533480
 ] 

Hau Phan commented on CASSANDRA-7190:
-

I noticed that the schema.cql file that gets generated with snapshots for 
materialized views are being treated as a table instead of a materialized view. 

Should I create a new ticket?

> Add schema to snapshot manifest
> ---
>
> Key: CASSANDRA-7190
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7190
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Jonathan Ellis
>Assignee: Alex Petrov
>Priority: Minor
>  Labels: client-impacting, doc-impacting, lhf
> Fix For: 3.0.9, 3.10
>
>
> followup from CASSANDRA-6326



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12715) Fix exceptions with the new vnode allocation.

[Dikang Gu (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533315#comment-15533315
 ] 

Dikang Gu commented on CASSANDRA-12715:
---

yes, the test will fail without my patch.

> Fix exceptions with the new vnode allocation.
> -
>
> Key: CASSANDRA-12715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12715
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Dikang Gu
>Assignee: Dikang Gu
> Fix For: 3.0.x, 3.x
>
>
> Problem: see exceptions when bootstrapping nodes using the new vnode 
> allocation algorithm. I'm able to reproduce it in trunk as well:
> {code}
> INFO  [main] 2016-09-26 15:36:54,978 StorageService.java:1437 - JOINING: 
> calculation complete, ready to bootstrap
> INFO  [main] 2016-09-26 15:36:54,978 StorageService.java:1437 - JOINING: 
> getting bootstrap token
> ERROR [main] 2016-09-26 15:36:54,989 CassandraDaemon.java:752 - Exception 
> encountered during startup
> java.lang.AssertionError: null
> at 
> org.apache.cassandra.locator.TokenMetadata.getTopology(TokenMetadata.java:1209)
>  ~[main/:na]
> at 
> org.apache.cassandra.dht.tokenallocator.TokenAllocation.getStrategy(TokenAllocation.java:201)
>  ~[main/:na]
> at 
> org.apache.cassandra.dht.tokenallocator.TokenAllocation.getStrategy(TokenAllocation.java:164)
>  ~[main/:na]
> at 
> org.apache.cassandra.dht.tokenallocator.TokenAllocation.allocateTokens(TokenAllocation.java:54)
>  ~[main/:na]
> at 
> org.apache.cassandra.dht.BootStrapper.allocateTokens(BootStrapper.java:207) 
> ~[main/:na]
> at 
> org.apache.cassandra.dht.BootStrapper.getBootstrapTokens(BootStrapper.java:174)
>  ~[main/:na]
> at 
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:929)
>  ~[main/:na]
> at 
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:697)
>  ~[main/:na]
> at 
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:582)
>  ~[main/:na]
> at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:392) 
> [main/:na]
> at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:601)
>  [main/:na]
> at 
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:735) 
> [main/:na]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-12731) Remove IndexInfo cache from FileIndexInfoRetriever.

[Yasuharu Goto (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yasuharu Goto updated CASSANDRA-12731:
--
Description: 
Hi guys.
In the patch of CASSANDRA-11206 , I found that FileIndexInfoRetriever allocates 
a (potentially very large) IndexInfo array (up to the number of IndexInfo in 
the RowIndexEntry has) as a cache in every single read path.

After some experiments using LargePartitionTest on my MacBook, I got results 
that show that removing FileIndexInfoRetriever improves the performance for 
large partitions like below (latencies reduced by 41% and by 45%).

{noformat}
// LargePartitionsTest.test_13_4G with cache by array
INFO  [main] 2016-09-29 23:11:25,763 ?:? - SELECTs 1 for part=4194304k 
total=16384M took 94197 ms
INFO  [main] 2016-09-29 23:12:50,914 ?:? - SELECTs 2 for part=4194304k 
total=16384M took 85151 ms

// LargePartitionsTest.test_13_4G without cache
INFO  [main] 2016-09-30 00:13:26,050 ?:? - SELECTs 1 for part=4194304k 
total=16384M took 55112 ms
INFO  [main] 2016-09-30 00:14:12,132 ?:? - SELECTs 2 for part=4194304k 
total=16384M took 46082 ms
{noformat}

Code is 
[here|https://github.com/matope/cassandra/commit/86fb910a0e38f7520e1be40fb42f74a692f2ebce]
 (based on trunk)

Heap memory usage during running LargePartitionsTest (except for 8G test) with 
array cache(original)
!screenshot-1.png!
Heap memory usage during running LargePartitionsTest (except for 8G test) 
without cache
!screenshot-2.png!



Of course, I have attempted to use some collection containers instead of a 
plain array. But I could not recognize great improvement enough to justify 
using these cache mechanism by them. (Unless I did some mistake or overlook 
about this test)

|| LargePartitionsTest.test_12_2G || SELECTs 1 (ms) || SELECTs 2 (ms) || Scan 
(ms) ||
|Original (array) | 62736 | 48562 | 41540 |
|ConcurrentHashMap | 47597 | 30854 | 18271 |
|ConcurrentHashMap 2nd trial |44036|26895|17443|
|LinkedHashCache (capacity=16, limit=10, fifo) 1st|42668|32165|17323|
|LinkedHashCache (capacity=16, limit=10, fifo) 2nd|48863|28066|18053|
|LinkedHashCache (capacity=16, limit=16, fifo) | 46979 | 29810 | 18620 |
|LinkedHashCache (capacity=16, limit=10, lru) | 46456 | 29749 | 20311 |
|No Cache | 47579 | 32480 | 18337 |
|No Cache 2nd trial | 46534 | 27670 | 18700 |

Code that I used for this comparison is 
[here|https://github.com/matope/cassandra/commit/e12fcac77f0f46bdf4104ef21c6454bfb2bb92d0].
 LinkedHashCache is a simple fifo/lru cache that is extended by LinkedHashMap.
Scan is a execution time to iterate through the large partition.

So, in this issue, I'd like to propose to remove IndexInfo cache from 
FileIndexInfoRetriever to improve the performance on large partitions.

  was:
Hi guys.
In the patch of CASSANDRA-11206 , I found that FileIndexInfoRetriever allocates 
a (potentially very large) IndexInfo array (up to the number of IndexInfo in 
the RowIndexEntry has) as a cache in every single read path.

After some experiments using LargePartitionTest on my MacBook, I got results 
that show that removing FileIndexInfoRetriever improves the performance for 
large partitions like below (latencies reduced by 41% and by 45%).

{noformat}
// LargePartitionsTest.test_13_4G with cache by array
INFO  [main] 2016-09-29 23:11:25,763 ?:? - SELECTs 1 for part=4194304k 
total=16384M took 94197 ms
INFO  [main] 2016-09-29 23:12:50,914 ?:? - SELECTs 2 for part=4194304k 
total=16384M took 85151 ms

// LargePartitionsTest.test_13_4G without cache
INFO  [main] 2016-09-30 00:13:26,050 ?:? - SELECTs 1 for part=4194304k 
total=16384M took 55112 ms
INFO  [main] 2016-09-30 00:14:12,132 ?:? - SELECTs 2 for part=4194304k 
total=16384M took 46082 ms
{noformat}

Code is 
[here|https://github.com/matope/cassandra/commit/86fb910a0e38f7520e1be40fb42f74a692f2ebce]
 (based on trunk)

Heap memory usage during running LargePartitionsTest (except for 8G test) with 
array cache(original)
!screenshot-1.png!
Heap memory usage during running LargePartitionsTest (except for 8G test) 
without cache
!screenshot-2.png!



Of course, I have attempted to use some collection containers instead of a 
plain array. But I could not recognize great improvement enough to justify 
using these cache mechanism by them. (Unless I did some mistake or overlook 
about this test)

|| LargePartitionsTest.test_12_2G || SELECTs 1 (ms) || SELECTs 2 (ms) || Scan 
(ms) ||
|Original (array) | 62736 | 48562 | 41540 |
|ConcurrentHashMap | 47597 | 30854 | 18271 |
|ConcurrentHashMap 2nd trial |44036|26895|17443|
|LinkedHashCache (capacity=16, limit=10, fifo) 1st|42668|32165|17323|
|LinkedHashCache (capacity=16, limit=10, fifo) 2nd|48863|28066|18053|
|LinkedHashCache (capacity=16, limit=16, fifo) | 46979 | 29810 | 18620 |
|LinkedHashCache (capacity=16, limit=10, lru) | 46456 | 29749 | 20311 |
|No Cache | 47579 | 32480 | 18337 |
|No Cache 2nd trial | 46534 | 27670 | 18700 |

Code that 

[jira] [Commented] (CASSANDRA-12731) Remove IndexInfo cache from FileIndexInfoRetriever.

[Yasuharu Goto (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533273#comment-15533273
 ] 

Yasuharu Goto commented on CASSANDRA-12731:
---

Patch is here.
https://github.com/matope/cassandra/commit/86fb910a0e38f7520e1be40fb42f74a692f2ebce

> Remove IndexInfo cache from FileIndexInfoRetriever.
> ---
>
> Key: CASSANDRA-12731
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12731
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Yasuharu Goto
> Attachments: screenshot-1.png, screenshot-2.png
>
>
> Hi guys.
> In the patch of CASSANDRA-11206 , I found that FileIndexInfoRetriever 
> allocates a (potentially very large) IndexInfo array (up to the number of 
> IndexInfo in the RowIndexEntry has) as a cache in every single read path.
> After some experiments using LargePartitionTest on my MacBook, I got results 
> that show that removing FileIndexInfoRetriever improves the performance for 
> large partitions like below (latencies reduced by 41% and by 45%).
> {noformat}
> // LargePartitionsTest.test_13_4G with cache by array
> INFO  [main] 2016-09-29 23:11:25,763 ?:? - SELECTs 1 for part=4194304k 
> total=16384M took 94197 ms
> INFO  [main] 2016-09-29 23:12:50,914 ?:? - SELECTs 2 for part=4194304k 
> total=16384M took 85151 ms
> // LargePartitionsTest.test_13_4G without cache
> INFO  [main] 2016-09-30 00:13:26,050 ?:? - SELECTs 1 for part=4194304k 
> total=16384M took 55112 ms
> INFO  [main] 2016-09-30 00:14:12,132 ?:? - SELECTs 2 for part=4194304k 
> total=16384M took 46082 ms
> {noformat}
> Code is 
> [here|https://github.com/matope/cassandra/commit/86fb910a0e38f7520e1be40fb42f74a692f2ebce]
>  (based on trunk)
> Heap memory usage during running LargePartitionsTest (except for 8G test) 
> with array cache(original)
> !screenshot-1.png!
> Heap memory usage during running LargePartitionsTest (except for 8G test) 
> without cache
> !screenshot-2.png!
> Of course, I have attempted to use some collection containers instead of a 
> plain array. But I could not recognize great improvement enough to justify 
> using these cache mechanism by them. (Unless I did some mistake or overlook 
> about this test)
> || LargePartitionsTest.test_12_2G || SELECTs 1 (ms) || SELECTs 2 (ms) || Scan 
> (ms) ||
> |Original (array) | 62736 | 48562 | 41540 |
> |ConcurrentHashMap | 47597 | 30854 | 18271 |
> |ConcurrentHashMap 2nd trial |44036|26895|17443|
> |LinkedHashCache (capacity=16, limit=10, fifo) 1st|42668|32165|17323|
> |LinkedHashCache (capacity=16, limit=10, fifo) 2nd|48863|28066|18053|
> |LinkedHashCache (capacity=16, limit=16, fifo) | 46979 | 29810 | 18620 |
> |LinkedHashCache (capacity=16, limit=10, lru) | 46456 | 29749 | 20311 |
> |No Cache | 47579 | 32480 | 18337 |
> |No Cache 2nd trial | 46534 | 27670 | 18700 |
> Code that I used for this comparison is 
> [here|https://github.com/matope/cassandra/commit/e12fcac77f0f46bdf4104ef21c6454bfb2bb92d0].
>  LinkedHashCache is a simple fifo/lru cache that is extended by LinkedHashMap.
> Scan is a execution time to iterate through the large partition.
> So, In this issue, I'd like to propose to remove IndexInfo cache from 
> FileIndexInfoRetriever to improve the performance on large partition.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-12731) Remove IndexInfo cache from FileIndexInfoRetriever.

[Yasuharu Goto (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yasuharu Goto updated CASSANDRA-12731:
--
Description: 
Hi guys.
In the patch of CASSANDRA-11206 , I found that FileIndexInfoRetriever allocates 
a (potentially very large) IndexInfo array (up to the number of IndexInfo in 
the RowIndexEntry has) as a cache in every single read path.

After some experiments using LargePartitionTest on my MacBook, I got results 
that show that removing FileIndexInfoRetriever improves the performance for 
large partitions like below (latencies reduced by 41% and by 45%).

{noformat}
// LargePartitionsTest.test_13_4G with cache by array
INFO  [main] 2016-09-29 23:11:25,763 ?:? - SELECTs 1 for part=4194304k 
total=16384M took 94197 ms
INFO  [main] 2016-09-29 23:12:50,914 ?:? - SELECTs 2 for part=4194304k 
total=16384M took 85151 ms

// LargePartitionsTest.test_13_4G without cache
INFO  [main] 2016-09-30 00:13:26,050 ?:? - SELECTs 1 for part=4194304k 
total=16384M took 55112 ms
INFO  [main] 2016-09-30 00:14:12,132 ?:? - SELECTs 2 for part=4194304k 
total=16384M took 46082 ms
{noformat}

Code is 
[here|https://github.com/matope/cassandra/commit/86fb910a0e38f7520e1be40fb42f74a692f2ebce]
 (based on trunk)

Heap memory usage during running LargePartitionsTest (except for 8G test) with 
array cache(original)
!screenshot-1.png!
Heap memory usage during running LargePartitionsTest (except for 8G test) 
without cache
!screenshot-2.png!



Of course, I have attempted to use some collection containers instead of a 
plain array. But I could not recognize great improvement enough to justify 
using these cache mechanism by them. (Unless I did some mistake or overlook 
about this test)

|| LargePartitionsTest.test_12_2G || SELECTs 1 (ms) || SELECTs 2 (ms) || Scan 
(ms) ||
|Original (array) | 62736 | 48562 | 41540 |
|ConcurrentHashMap | 47597 | 30854 | 18271 |
|ConcurrentHashMap 2nd trial |44036|26895|17443|
|LinkedHashCache (capacity=16, limit=10, fifo) 1st|42668|32165|17323|
|LinkedHashCache (capacity=16, limit=10, fifo) 2nd|48863|28066|18053|
|LinkedHashCache (capacity=16, limit=16, fifo) | 46979 | 29810 | 18620 |
|LinkedHashCache (capacity=16, limit=10, lru) | 46456 | 29749 | 20311 |
|No Cache | 47579 | 32480 | 18337 |
|No Cache 2nd trial | 46534 | 27670 | 18700 |

Code that I used for this comparison is 
[here|https://github.com/matope/cassandra/commit/e12fcac77f0f46bdf4104ef21c6454bfb2bb92d0].
 LinkedHashCache is a simple fifo/lru cache that is extended by LinkedHashMap.
Scan is a execution time to iterate through the large partition.

So, In this issue, I'd like to propose to remove IndexInfo cache from 
FileIndexInfoRetriever to improve the performance on large partition.

  was:
Hi guys.
In the patch of CASSANDRA-11206 , I found that FileIndexInfoRetriever allocates 
a very large IndexInfo array (up to the number of IndexInfo in the 
RowIndexEntry has) as a cache in every single read path.

After some experiments using LargePartitionTest on my MacBook, I got results 
that show that removing FileIndexInfoRetriever improves the performance for 
large partitions like below (latencies reduced by 41% and by 45%).

{noformat}
// LargePartitionsTest.test_13_4G with cache by array
INFO  [main] 2016-09-29 23:11:25,763 ?:? - SELECTs 1 for part=4194304k 
total=16384M took 94197 ms
INFO  [main] 2016-09-29 23:12:50,914 ?:? - SELECTs 2 for part=4194304k 
total=16384M took 85151 ms

// LargePartitionsTest.test_13_4G without cache
INFO  [main] 2016-09-30 00:13:26,050 ?:? - SELECTs 1 for part=4194304k 
total=16384M took 55112 ms
INFO  [main] 2016-09-30 00:14:12,132 ?:? - SELECTs 2 for part=4194304k 
total=16384M took 46082 ms
{noformat}

Code is 
[here|https://github.com/matope/cassandra/commit/86fb910a0e38f7520e1be40fb42f74a692f2ebce]
 (based on trunk)

Heap memory usage during running LargePartitionsTest (except for 8G test) with 
array cache(original)
!screenshot-1.png!
Heap memory usage during running LargePartitionsTest (except for 8G test) 
without cache
!screenshot-2.png!



Of course, I have attempted to use some collection containers instead of a 
plain array. But I could not recognize great improvement enough to justify 
using these cache mechanism by them. (Unless I did some mistake or overlook 
about this test)

|| LargePartitionsTest.test_12_2G || SELECTs 1 (ms) || SELECTs 2 (ms) || Scan 
(ms) ||
|Original (array) | 62736 | 48562 | 41540 |
|ConcurrentHashMap | 47597 | 30854 | 18271 |
|ConcurrentHashMap 2nd trial |44036|26895|17443|
|LinkedHashCache (capacity=16, limit=10, fifo) 1st|42668|32165|17323|
|LinkedHashCache (capacity=16, limit=10, fifo) 2nd|48863|28066|18053|
|LinkedHashCache (capacity=16, limit=16, fifo) | 46979 | 29810 | 18620 |
|LinkedHashCache (capacity=16, limit=10, lru) | 46456 | 29749 | 20311 |
|No Cache | 47579 | 32480 | 18337 |
|No Cache 2nd trial | 46534 | 27670 | 18700 |

Code that I used for 

[jira] [Updated] (CASSANDRA-12731) Remove IndexInfo cache from FileIndexInfoRetriever.

[Yasuharu Goto (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yasuharu Goto updated CASSANDRA-12731:
--
Description: 
Hi guys.
In the patch of CASSANDRA-11206 , I found that FileIndexInfoRetriever allocates 
a very large IndexInfo array (up to the number of IndexInfo in the 
RowIndexEntry has) as a cache in every single read path.

After some experiments using LargePartitionTest on my MacBook, I got results 
that show that removing FileIndexInfoRetriever improves the performance for 
large partitions like below (latencies reduced by 41% and by 45%).

{noformat}
// LargePartitionsTest.test_13_4G with cache by array
INFO  [main] 2016-09-29 23:11:25,763 ?:? - SELECTs 1 for part=4194304k 
total=16384M took 94197 ms
INFO  [main] 2016-09-29 23:12:50,914 ?:? - SELECTs 2 for part=4194304k 
total=16384M took 85151 ms

// LargePartitionsTest.test_13_4G without cache
INFO  [main] 2016-09-30 00:13:26,050 ?:? - SELECTs 1 for part=4194304k 
total=16384M took 55112 ms
INFO  [main] 2016-09-30 00:14:12,132 ?:? - SELECTs 2 for part=4194304k 
total=16384M took 46082 ms
{noformat}

Code is 
[here|https://github.com/matope/cassandra/commit/86fb910a0e38f7520e1be40fb42f74a692f2ebce]
 (based on trunk)

Heap memory usage during running LargePartitionsTest (except for 8G test) with 
array cache(original)
!screenshot-1.png!
Heap memory usage during running LargePartitionsTest (except for 8G test) 
without cache
!screenshot-2.png!



Of course, I have attempted to use some collection containers instead of a 
plain array. But I could not recognize great improvement enough to justify 
using these cache mechanism by them. (Unless I did some mistake or overlook 
about this test)

|| LargePartitionsTest.test_12_2G || SELECTs 1 (ms) || SELECTs 2 (ms) || Scan 
(ms) ||
|Original (array) | 62736 | 48562 | 41540 |
|ConcurrentHashMap | 47597 | 30854 | 18271 |
|ConcurrentHashMap 2nd trial |44036|26895|17443|
|LinkedHashCache (capacity=16, limit=10, fifo) 1st|42668|32165|17323|
|LinkedHashCache (capacity=16, limit=10, fifo) 2nd|48863|28066|18053|
|LinkedHashCache (capacity=16, limit=16, fifo) | 46979 | 29810 | 18620 |
|LinkedHashCache (capacity=16, limit=10, lru) | 46456 | 29749 | 20311 |
|No Cache | 47579 | 32480 | 18337 |
|No Cache 2nd trial | 46534 | 27670 | 18700 |

Code that I used for this comparison is 
[here|https://github.com/matope/cassandra/commit/e12fcac77f0f46bdf4104ef21c6454bfb2bb92d0].
 LinkedHashCache is a simple fifo/lru cache that is extended by LinkedHashMap.
Scan is a execution time to iterate through the large partition.

So, In this issue, I'd like to propose to remove IndexInfo cache from 
FileIndexInfoRetriever to improve the performance on large partition.

  was:
Hi guys.
In the patch of CASSANDRA-11206 , I found that FileIndexInfoRetriever allocates 
a very large IndexInfo array (up to the number of IndexInfo in the 
RowIndexEntry has) as a cache in every single read path.

After some experiments using LargePartitionTest on my MacBook, I got results 
that show that removing FileIndexInfoRetriever improves the performance for 
large partitions like below (latencies reduced by 41% and by 45%).

{noformat}
// LargePartitionsTest.test_13_4G with cache by array
INFO  [main] 2016-09-29 23:11:25,763 ?:? - SELECTs 1 for part=4194304k 
total=16384M took 94197 ms
INFO  [main] 2016-09-29 23:12:50,914 ?:? - SELECTs 2 for part=4194304k 
total=16384M took 85151 ms

// LargePartitionsTest.test_13_4G without cache
INFO  [main] 2016-09-30 00:13:26,050 ?:? - SELECTs 1 for part=4194304k 
total=16384M took 55112 ms
INFO  [main] 2016-09-30 00:14:12,132 ?:? - SELECTs 2 for part=4194304k 
total=16384M took 46082 ms
{noformat}

Code is 
[here|https://github.com/matope/cassandra/commit/86fb910a0e38f7520e1be40fb42f74a692f2ebce]
 (based on trunk)

Heap memory usage during running LargePartitionsTest (except for 8G test) with 
array cache(original)
!screenshot-1.png!
Heap memory usage during running LargePartitionsTest (except for 8G test) 
without cache
!screenshot-2.png!



Of course, I have attempted to use some collection containers instead of a 
plain array. But I could not recognize great improvement enough to justify 
using these cache mechanism by them. (Unless I did some mistake or overlook 
about this test)

|| LargePartitionsTest.test_12_2G || SELECTs 1 (ms) || SELECTs 2 (ms) || Scan 
(ms) ||
|Original (array) | 62736 | 48562 | 41540 |
|ConcurrentHashMap 1st| 47597 | 30854 | 18271 |
|ConcurrentHashMap 2nd|44036|26895|17443|
|LinkedHashCache (capacity=16, limit=10, fifo) 1st|42668|32165|17323|
|LinkedHashCache (capacity=16, limit=10, fifo) 2nd|48863|28066|18053|
|LinkedHashCache (capacity=16, limit=16, fifo) | 46979 | 29810 | 18620 |
|LinkedHashCache (capacity=16, limit=10, lru) | 46456 | 29749 | 20311 |
|No Cache 1st | 47579 | 32480 | 18337 |
|No Cache 2nd | 46534 | 27670 | 18700 |

Code that I used for this comparison is 

[jira] [Updated] (CASSANDRA-12731) Remove IndexInfo cache from FileIndexInfoRetriever.

[Yasuharu Goto (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yasuharu Goto updated CASSANDRA-12731:
--
Description: 
Hi guys.
In the patch of CASSANDRA-11206 , I found that FileIndexInfoRetriever allocates 
a very large IndexInfo array (up to the number of IndexInfo in the 
RowIndexEntry has) as a cache in every single read path.

After some experiments using LargePartitionTest on my MacBook, I got results 
that show that removing FileIndexInfoRetriever improves the performance for 
large partitions like below (latencies reduced by 41% and by 45%).

{noformat}
// LargePartitionsTest.test_13_4G with cache by array
INFO  [main] 2016-09-29 23:11:25,763 ?:? - SELECTs 1 for part=4194304k 
total=16384M took 94197 ms
INFO  [main] 2016-09-29 23:12:50,914 ?:? - SELECTs 2 for part=4194304k 
total=16384M took 85151 ms

// LargePartitionsTest.test_13_4G without cache
INFO  [main] 2016-09-30 00:13:26,050 ?:? - SELECTs 1 for part=4194304k 
total=16384M took 55112 ms
INFO  [main] 2016-09-30 00:14:12,132 ?:? - SELECTs 2 for part=4194304k 
total=16384M took 46082 ms
{noformat}

Code is 
[here|https://github.com/matope/cassandra/commit/86fb910a0e38f7520e1be40fb42f74a692f2ebce]
 (based on trunk)

Heap memory usage during running LargePartitionsTest (except for 8G test) with 
array cache(original)
!screenshot-1.png!
Heap memory usage during running LargePartitionsTest (except for 8G test) 
without cache
!screenshot-2.png!



Of course, I have attempted to use some collection containers instead of a 
plain array. But I could not recognize great improvement enough to justify 
using these cache mechanism by them. (Unless I did some mistake or overlook 
about this test)

|| LargePartitionsTest.test_12_2G || SELECTs 1 (ms) || SELECTs 2 (ms) || Scan 
(ms) ||
|Original (array) | 62736 | 48562 | 41540 |
|ConcurrentHashMap 1st| 47597 | 30854 | 18271 |
|ConcurrentHashMap 2nd|44036|26895|17443|
|LinkedHashCache (capacity=16, limit=10, fifo) 1st|42668|32165|17323|
|LinkedHashCache (capacity=16, limit=10, fifo) 2nd|48863|28066|18053|
|LinkedHashCache (capacity=16, limit=16, fifo) | 46979 | 29810 | 18620 |
|LinkedHashCache (capacity=16, limit=10, lru) | 46456 | 29749 | 20311 |
|No Cache 1st | 47579 | 32480 | 18337 |
|No Cache 2nd | 46534 | 27670 | 18700 |

Code that I used for this comparison is 
[here|https://github.com/matope/cassandra/commit/e12fcac77f0f46bdf4104ef21c6454bfb2bb92d0].
 LinkedHashCache is a simple fifo/lru cache that is extended by LinkedHashMap.
Scan is a execution time to iterate through the large partition.

So, In this issue, I'd like to propose to remove IndexInfo cache from 
FileIndexInfoRetriever to improve the performance on large partition.

  was:
Hi guys.
In the patch of CASSANDRA-11206 , I found that FileIndexInfoRetriever allocates 
a very large IndexInfo array (up to the number of IndexInfo in the 
RowIndexEntry has) as a cache in every single read path.

After some experiments using LargePartitionTest on my MacBook, I got results 
that show that removing FileIndexInfoRetriever improves the performance for 
large partitions like below (latencies reduced by 41% and by 45%).

{noformat}
// LargePartitionsTest.test_13_4G with cache by array
INFO  [main] 2016-09-29 23:11:25,763 ?:? - SELECTs 1 for part=4194304k 
total=16384M took 94197 ms
INFO  [main] 2016-09-29 23:12:50,914 ?:? - SELECTs 2 for part=4194304k 
total=16384M took 85151 ms

// LargePartitionsTest.test_13_4G without cache
INFO  [main] 2016-09-30 00:13:26,050 ?:? - SELECTs 1 for part=4194304k 
total=16384M took 55112 ms
INFO  [main] 2016-09-30 00:14:12,132 ?:? - SELECTs 2 for part=4194304k 
total=16384M took 46082 ms
{noformat}

Code is 
[here|https://github.com/matope/cassandra/commit/86fb910a0e38f7520e1be40fb42f74a692f2ebce]
 (based on trunk)

Of course, I have attempted to use some collection containers instead of a 
plain array. But I could not recognize great improvement enough to justify 
using these cache mechanism by them. (Unless I did some mistake or overlook 
about this test)

|| LargePartitionsTest.test_12_2G || SELECTs 1 (ms) || SELECTs 2 (ms) || Scan 
(ms) ||
|Original (array) | 62736 | 48562 | 41540 |
|ConcurrentHashMap 1st| 47597 | 30854 | 18271 |
|ConcurrentHashMap 2nd|44036|26895|17443|
|LinkedHashCache (capacity=16, limit=10, fifo) 1st|42668|32165|17323|
|LinkedHashCache (capacity=16, limit=10, fifo) 2nd|48863|28066|18053|
|LinkedHashCache (capacity=16, limit=16, fifo) | 46979 | 29810 | 18620 |
|LinkedHashCache (capacity=16, limit=10, lru) | 46456 | 29749 | 20311 |
|No Cache 1st | 47579 | 32480 | 18337 |
|No Cache 2nd | 46534 | 27670 | 18700 |

Code that I used for this comparison is 
[here|https://github.com/matope/cassandra/commit/e12fcac77f0f46bdf4104ef21c6454bfb2bb92d0].
 LinkedHashCache is a simple fifo/lru cache that is extended by LinkedHashMap.
Scan is a execution time to iterate through the large partition.

So, 

[jira] [Updated] (CASSANDRA-12731) Remove IndexInfo cache from FileIndexInfoRetriever.

[Yasuharu Goto (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yasuharu Goto updated CASSANDRA-12731:
--
Attachment: screenshot-2.png

> Remove IndexInfo cache from FileIndexInfoRetriever.
> ---
>
> Key: CASSANDRA-12731
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12731
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Yasuharu Goto
> Attachments: screenshot-1.png, screenshot-2.png
>
>
> Hi guys.
> In the patch of CASSANDRA-11206 , I found that FileIndexInfoRetriever 
> allocates a very large IndexInfo array (up to the number of IndexInfo in the 
> RowIndexEntry has) as a cache in every single read path.
> After some experiments using LargePartitionTest on my MacBook, I got results 
> that show that removing FileIndexInfoRetriever improves the performance for 
> large partitions like below (latencies reduced by 41% and by 45%).
> {noformat}
> // LargePartitionsTest.test_13_4G with cache by array
> INFO  [main] 2016-09-29 23:11:25,763 ?:? - SELECTs 1 for part=4194304k 
> total=16384M took 94197 ms
> INFO  [main] 2016-09-29 23:12:50,914 ?:? - SELECTs 2 for part=4194304k 
> total=16384M took 85151 ms
> // LargePartitionsTest.test_13_4G without cache
> INFO  [main] 2016-09-30 00:13:26,050 ?:? - SELECTs 1 for part=4194304k 
> total=16384M took 55112 ms
> INFO  [main] 2016-09-30 00:14:12,132 ?:? - SELECTs 2 for part=4194304k 
> total=16384M took 46082 ms
> {noformat}
> Code is 
> [here|https://github.com/matope/cassandra/commit/86fb910a0e38f7520e1be40fb42f74a692f2ebce]
>  (based on trunk)
> Of course, I have attempted to use some collection containers instead of a 
> plain array. But I could not recognize great improvement enough to justify 
> using these cache mechanism by them. (Unless I did some mistake or overlook 
> about this test)
> || LargePartitionsTest.test_12_2G || SELECTs 1 (ms) || SELECTs 2 (ms) || Scan 
> (ms) ||
> |Original (array) | 62736 | 48562 | 41540 |
> |ConcurrentHashMap 1st| 47597 | 30854 | 18271 |
> |ConcurrentHashMap 2nd|44036|26895|17443|
> |LinkedHashCache (capacity=16, limit=10, fifo) 1st|42668|32165|17323|
> |LinkedHashCache (capacity=16, limit=10, fifo) 2nd|48863|28066|18053|
> |LinkedHashCache (capacity=16, limit=16, fifo) | 46979 | 29810 | 18620 |
> |LinkedHashCache (capacity=16, limit=10, lru) | 46456 | 29749 | 20311 |
> |No Cache 1st | 47579 | 32480 | 18337 |
> |No Cache 2nd | 46534 | 27670 | 18700 |
> Code that I used for this comparison is 
> [here|https://github.com/matope/cassandra/commit/e12fcac77f0f46bdf4104ef21c6454bfb2bb92d0].
>  LinkedHashCache is a simple fifo/lru cache that is extended by LinkedHashMap.
> Scan is a execution time to iterate through the large partition.
> So, In this issue, I'd like to propose to remove IndexInfo cache from 
> FileIndexInfoRetriever to improve the performance on large partition.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-12731) Remove IndexInfo cache from FileIndexInfoRetriever.

[Yasuharu Goto (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yasuharu Goto updated CASSANDRA-12731:
--
Attachment: screenshot-1.png

> Remove IndexInfo cache from FileIndexInfoRetriever.
> ---
>
> Key: CASSANDRA-12731
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12731
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Yasuharu Goto
> Attachments: screenshot-1.png, screenshot-2.png
>
>
> Hi guys.
> In the patch of CASSANDRA-11206 , I found that FileIndexInfoRetriever 
> allocates a very large IndexInfo array (up to the number of IndexInfo in the 
> RowIndexEntry has) as a cache in every single read path.
> After some experiments using LargePartitionTest on my MacBook, I got results 
> that show that removing FileIndexInfoRetriever improves the performance for 
> large partitions like below (latencies reduced by 41% and by 45%).
> {noformat}
> // LargePartitionsTest.test_13_4G with cache by array
> INFO  [main] 2016-09-29 23:11:25,763 ?:? - SELECTs 1 for part=4194304k 
> total=16384M took 94197 ms
> INFO  [main] 2016-09-29 23:12:50,914 ?:? - SELECTs 2 for part=4194304k 
> total=16384M took 85151 ms
> // LargePartitionsTest.test_13_4G without cache
> INFO  [main] 2016-09-30 00:13:26,050 ?:? - SELECTs 1 for part=4194304k 
> total=16384M took 55112 ms
> INFO  [main] 2016-09-30 00:14:12,132 ?:? - SELECTs 2 for part=4194304k 
> total=16384M took 46082 ms
> {noformat}
> Code is 
> [here|https://github.com/matope/cassandra/commit/86fb910a0e38f7520e1be40fb42f74a692f2ebce]
>  (based on trunk)
> Of course, I have attempted to use some collection containers instead of a 
> plain array. But I could not recognize great improvement enough to justify 
> using these cache mechanism by them. (Unless I did some mistake or overlook 
> about this test)
> || LargePartitionsTest.test_12_2G || SELECTs 1 (ms) || SELECTs 2 (ms) || Scan 
> (ms) ||
> |Original (array) | 62736 | 48562 | 41540 |
> |ConcurrentHashMap 1st| 47597 | 30854 | 18271 |
> |ConcurrentHashMap 2nd|44036|26895|17443|
> |LinkedHashCache (capacity=16, limit=10, fifo) 1st|42668|32165|17323|
> |LinkedHashCache (capacity=16, limit=10, fifo) 2nd|48863|28066|18053|
> |LinkedHashCache (capacity=16, limit=16, fifo) | 46979 | 29810 | 18620 |
> |LinkedHashCache (capacity=16, limit=10, lru) | 46456 | 29749 | 20311 |
> |No Cache 1st | 47579 | 32480 | 18337 |
> |No Cache 2nd | 46534 | 27670 | 18700 |
> Code that I used for this comparison is 
> [here|https://github.com/matope/cassandra/commit/e12fcac77f0f46bdf4104ef21c6454bfb2bb92d0].
>  LinkedHashCache is a simple fifo/lru cache that is extended by LinkedHashMap.
> Scan is a execution time to iterate through the large partition.
> So, In this issue, I'd like to propose to remove IndexInfo cache from 
> FileIndexInfoRetriever to improve the performance on large partition.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-12731) Remove IndexInfo cache from FileIndexInfoRetriever.

[Yasuharu Goto (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yasuharu Goto updated CASSANDRA-12731:
--
Description: 
Hi guys.
In the patch of CASSANDRA-11206 , I found that FileIndexInfoRetriever allocates 
a very large IndexInfo array (up to the number of IndexInfo in the 
RowIndexEntry has) as a cache in every single read path.

After some experiments using LargePartitionTest on my MacBook, I got results 
that show that removing FileIndexInfoRetriever improves the performance for 
large partitions like below (latencies reduced by 41% and by 45%).

{noformat}
// LargePartitionsTest.test_13_4G with cache by array
INFO  [main] 2016-09-29 23:11:25,763 ?:? - SELECTs 1 for part=4194304k 
total=16384M took 94197 ms
INFO  [main] 2016-09-29 23:12:50,914 ?:? - SELECTs 2 for part=4194304k 
total=16384M took 85151 ms

// LargePartitionsTest.test_13_4G without cache
INFO  [main] 2016-09-30 00:13:26,050 ?:? - SELECTs 1 for part=4194304k 
total=16384M took 55112 ms
INFO  [main] 2016-09-30 00:14:12,132 ?:? - SELECTs 2 for part=4194304k 
total=16384M took 46082 ms
{noformat}

Code is 
[here|https://github.com/matope/cassandra/commit/86fb910a0e38f7520e1be40fb42f74a692f2ebce]
 (based on trunk)

Of course, I have attempted to use some collection containers instead of a 
plain array. But I could not recognize great improvement enough to justify 
using these cache mechanism by them. (Unless I did some mistake or overlook 
about this test)

|| LargePartitionsTest.test_12_2G || SELECTs 1 (ms) || SELECTs 2 (ms) || Scan 
(ms) ||
|Original (array) | 62736 | 48562 | 41540 |
|ConcurrentHashMap 1st| 47597 | 30854 | 18271 |
|ConcurrentHashMap 2nd|44036|26895|17443|
|LinkedHashCache (capacity=16, limit=10, fifo) 1st|42668|32165|17323|
|LinkedHashCache (capacity=16, limit=10, fifo) 2nd|48863|28066|18053|
|LinkedHashCache (capacity=16, limit=16, fifo) | 46979 | 29810 | 18620 |
|LinkedHashCache (capacity=16, limit=10, lru) | 46456 | 29749 | 20311 |
|No Cache 1st | 47579 | 32480 | 18337 |
|No Cache 2nd | 46534 | 27670 | 18700 |

Code that I used for this comparison is 
[here|https://github.com/matope/cassandra/commit/e12fcac77f0f46bdf4104ef21c6454bfb2bb92d0].
 LinkedHashCache is a simple fifo/lru cache that is extended by LinkedHashMap.
Scan is a execution time to iterate through the large partition.

So, In this issue, I'd like to propose to remove IndexInfo cache from 
FileIndexInfoRetriever to improve the performance on large partition.

  was:
Hi guys.
In the patch of CASSANDRA-11206 , I found that FileIndexInfoRetriever allocates 
a very large IndexInfo array (up to the number of IndexInfo in the 
RowIndexEntry has) as a cache in every single read path.

After some experiments using LargePartitionTest , I got results that show that 
removing FileIndexInfoRetriever improves the performance for large partitions 
like below (latencies reduced by 41% and by 45%).

{noformat}
// LargePartitionsTest.test_13_4G with cache by array
INFO  [main] 2016-09-29 23:11:25,763 ?:? - SELECTs 1 for part=4194304k 
total=16384M took 94197 ms
INFO  [main] 2016-09-29 23:12:50,914 ?:? - SELECTs 2 for part=4194304k 
total=16384M took 85151 ms

// LargePartitionsTest.test_13_4G without cache
INFO  [main] 2016-09-30 00:13:26,050 ?:? - SELECTs 1 for part=4194304k 
total=16384M took 55112 ms
INFO  [main] 2016-09-30 00:14:12,132 ?:? - SELECTs 2 for part=4194304k 
total=16384M took 46082 ms
{noformat}

Code is 
[here|https://github.com/matope/cassandra/commit/86fb910a0e38f7520e1be40fb42f74a692f2ebce]
 (based on trunk)

Of course, I have attempted to use some collection containers instead of a 
plain array. But I could not recognize great improvement enough to justify 
using these cache mechanism by them. (Unless I did some mistake or overlook 
about this test)

|| LargePartitionsTest.test_12_2G || SELECTs 1 (ms) || SELECTs 2 (ms) || Scan 
(ms) ||
|Original (array) | 62736 | 48562 | 41540 |
|ConcurrentHashMap 1st| 47597 | 30854 | 18271 |
|ConcurrentHashMap 2nd|44036|26895|17443|
|LinkedHashCache (capacity=16, limit=10, fifo) 1st|42668|32165|17323|
|LinkedHashCache (capacity=16, limit=10, fifo) 2nd|48863|28066|18053|
|LinkedHashCache (capacity=16, limit=16, fifo) | 46979 | 29810 | 18620 |
|LinkedHashCache (capacity=16, limit=10, lru) | 46456 | 29749 | 20311 |
|No Cache 1st | 47579 | 32480 | 18337 |
|No Cache 2nd | 46534 | 27670 | 18700 |

Code that I used for this comparison is 
[here|https://github.com/matope/cassandra/commit/e12fcac77f0f46bdf4104ef21c6454bfb2bb92d0].
 LinkedHashCache is a simple fifo/lru cache that is extended by LinkedHashMap.
Scan is a execution time to iterate through the large partition.

So, In this issue, I'd like to propose to remove IndexInfo cache from 
FileIndexInfoRetriever to improve the performance on large partition.


> Remove IndexInfo cache from FileIndexInfoRetriever.
> 

[jira] [Created] (CASSANDRA-12731) Remove IndexInfo cache from FileIndexInfoRetriever.

[Yasuharu Goto (JIRA)]

Yasuharu Goto created CASSANDRA-12731:
-

 Summary: Remove IndexInfo cache from FileIndexInfoRetriever.
 Key: CASSANDRA-12731
 URL: https://issues.apache.org/jira/browse/CASSANDRA-12731
 Project: Cassandra
  Issue Type: Improvement
Reporter: Yasuharu Goto


Hi guys.
In the patch of CASSANDRA-11206 , I found that FileIndexInfoRetriever allocates 
a very large IndexInfo array (up to the number of IndexInfo in the 
RowIndexEntry has) as a cache in every single read path.

After some experiments using LargePartitionTest , I got results that show that 
removing FileIndexInfoRetriever improves the performance for large partitions 
like below (latencies reduced by 41% and by 45%).

{noformat}
// LargePartitionsTest.test_13_4G with cache by array
INFO  [main] 2016-09-29 23:11:25,763 ?:? - SELECTs 1 for part=4194304k 
total=16384M took 94197 ms
INFO  [main] 2016-09-29 23:12:50,914 ?:? - SELECTs 2 for part=4194304k 
total=16384M took 85151 ms

// LargePartitionsTest.test_13_4G without cache
INFO  [main] 2016-09-30 00:13:26,050 ?:? - SELECTs 1 for part=4194304k 
total=16384M took 55112 ms
INFO  [main] 2016-09-30 00:14:12,132 ?:? - SELECTs 2 for part=4194304k 
total=16384M took 46082 ms
{noformat}

Code is 
[here|https://github.com/matope/cassandra/commit/86fb910a0e38f7520e1be40fb42f74a692f2ebce]
 (based on trunk)

Of course, I have attempted to use some collection containers instead of a 
plain array. But I could not recognize great improvement enough to justify 
using these cache mechanism by them. (Unless I did some mistake or overlook 
about this test)

|| LargePartitionsTest.test_12_2G || SELECTs 1 (ms) || SELECTs 2 (ms) || Scan 
(ms) ||
|Original (array) | 62736 | 48562 | 41540 |
|ConcurrentHashMap 1st| 47597 | 30854 | 18271 |
|ConcurrentHashMap 2nd|44036|26895|17443|
|LinkedHashCache (capacity=16, limit=10, fifo) 1st|42668|32165|17323|
|LinkedHashCache (capacity=16, limit=10, fifo) 2nd|48863|28066|18053|
|LinkedHashCache (capacity=16, limit=16, fifo) | 46979 | 29810 | 18620 |
|LinkedHashCache (capacity=16, limit=10, lru) | 46456 | 29749 | 20311 |
|No Cache 1st | 47579 | 32480 | 18337 |
|No Cache 2nd | 46534 | 27670 | 18700 |

Code that I used for this comparison is 
[here|https://github.com/matope/cassandra/commit/e12fcac77f0f46bdf4104ef21c6454bfb2bb92d0].
 LinkedHashCache is a simple fifo/lru cache that is extended by LinkedHashMap.
Scan is a execution time to iterate through the large partition.

So, In this issue, I'd like to propose to remove IndexInfo cache from 
FileIndexInfoRetriever to improve the performance on large partition.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12420) Duplicated Key in IN clause with a small fetch size will run forever

[ZhaoYang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533182#comment-15533182
 ] 

ZhaoYang commented on CASSANDRA-12420:
--

Thanks for the fix.

> Duplicated Key in IN clause with a small fetch size will run forever
> 
>
> Key: CASSANDRA-12420
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12420
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
> Environment: cassandra 2.1.14, driver 2.1.7.1
>Reporter: ZhaoYang
>Assignee: Tyler Hobbs
>  Labels: doc-impacting
> Fix For: 2.1.16
>
> Attachments: CASSANDRA-12420.patch
>
>
> This can be easily reproduced and fetch size is smaller than the correct 
> number of rows.
> A table has 2 partition key, 1 clustering key, 1 column.
> >Select select = QueryBuilder.select().from("ks", "cf");
> >select.where().and(QueryBuilder.eq("a", 1));
> >select.where().and(QueryBuilder.in("b", Arrays.asList(1, 1, 1)));
> >select.setFetchSize(5);
> Now we put a distinct method in client side to eliminate the duplicated key, 
> but it's better to fix inside Cassandra.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11846) Invalid QueryBuilder.insert is not invalidated which causes OOM

[ZhaoYang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533179#comment-15533179
 ] 

ZhaoYang commented on CASSANDRA-11846:
--

thanks

> Invalid QueryBuilder.insert is not invalidated which causes OOM
> ---
>
> Key: CASSANDRA-11846
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11846
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
> Environment: cassandra-2.1.14
>Reporter: ZhaoYang
>Priority: Minor
> Fix For: 2.1.x
>
>
> create table test(  key text primary key, value list );
> When using QueryBuilder.Insert() to bind column `value` with a blob, 
> Cassandra didn't consider it to be an invalid query and then lead to OOM and 
> crashed.
> the same plain query(String) can be invalidated by Cassandra and C* responds 
> InvalidQuery.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12700) During writing data into Cassandra 3.7.0 using Python driver 3.7 sometimes Connection get lost, because of Server NullPointerException

[Jeremiah Jordan (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533132#comment-15533132
 ] 

Jeremiah Jordan commented on CASSANDRA-12700:
-

I don't think this is any different for 3.0, so what ever we do to fix the bug 
here, we should probably apply it to cassandra-3.0 and maybe even cassandra-2.2.

> During writing data into Cassandra 3.7.0 using Python driver 3.7 sometimes 
> Connection get lost, because of Server NullPointerException
> --
>
> Key: CASSANDRA-12700
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12700
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: Cassandra cluster with two nodes running C* version 
> 3.7.0 and Python Driver 3.7 using Python 2.7.11. 
> OS: Red Hat Enterprise Linux 6.x x64, 
> RAM :8GB
> DISK :210GB
> Cores: 2
> Java 1.8.0_73 JRE
>Reporter: Rajesh Radhakrishnan
>Assignee: Jeff Jirsa
> Fix For: 3.x
>
>
> In our C* cluster we are using the latest Cassandra 3.7.0 (datastax-ddc.3.70) 
> with Python driver 3.7. Trying to insert 2 million row or more data into the 
> database, but sometimes we are getting "Null pointer Exception". 
> We are using Python 2.7.11 and Java 1.8.0_73 in the Cassandra nodes and in 
> the client its Python 2.7.12.
> {code:title=cassandra server log}
> ERROR [SharedPool-Worker-6] 2016-09-23 09:42:55,002 Message.java:611 - 
> Unexpected exception during request; channel = [id: 0xc208da86, 
> L:/IP1.IP2.IP3.IP4:9042 - R:/IP5.IP6.IP7.IP8:58418]
> java.lang.NullPointerException: null
> at 
> org.apache.cassandra.serializers.BooleanSerializer.deserialize(BooleanSerializer.java:33)
>  ~[apache-cassandra-3.7.0.jar:3.7.0]
> at 
> org.apache.cassandra.serializers.BooleanSerializer.deserialize(BooleanSerializer.java:24)
>  ~[apache-cassandra-3.7.0.jar:3.7.0]
> at 
> org.apache.cassandra.db.marshal.AbstractType.compose(AbstractType.java:113) 
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at 
> org.apache.cassandra.cql3.UntypedResultSet$Row.getBoolean(UntypedResultSet.java:273)
>  ~[apache-cassandra-3.7.0.jar:3.7.0]
> at 
> org.apache.cassandra.auth.CassandraRoleManager$1.apply(CassandraRoleManager.java:85)
>  ~[apache-cassandra-3.7.0.jar:3.7.0]
> at 
> org.apache.cassandra.auth.CassandraRoleManager$1.apply(CassandraRoleManager.java:81)
>  ~[apache-cassandra-3.7.0.jar:3.7.0]
> at 
> org.apache.cassandra.auth.CassandraRoleManager.getRoleFromTable(CassandraRoleManager.java:503)
>  ~[apache-cassandra-3.7.0.jar:3.7.0]
> at 
> org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:485)
>  ~[apache-cassandra-3.7.0.jar:3.7.0]
> at 
> org.apache.cassandra.auth.CassandraRoleManager.canLogin(CassandraRoleManager.java:298)
>  ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.service.ClientState.login(ClientState.java:227) 
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at 
> org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:79)
>  ~[apache-cassandra-3.7.0.jar:3.7.0]
> at 
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507)
>  [apache-cassandra-3.7.0.jar:3.7.0]
> at 
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401)
>  [apache-cassandra-3.7.0.jar:3.7.0]
> at 
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
>  [netty-all-4.0.36.Final.jar:4.0.36.Final]
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:292)
>  [netty-all-4.0.36.Final.jar:4.0.36.Final]
> at 
> io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:32)
>  [netty-all-4.0.36.Final.jar:4.0.36.Final]
> at 
> io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:283)
>  [netty-all-4.0.36.Final.jar:4.0.36.Final]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_73]
> at 
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
>  [apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) 
> [apache-cassandra-3.7.0.jar:3.7.0]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_73]
> ERROR [SharedPool-Worker-1] 2016-09-23 09:42:56,238 Message.java:611 - 
> Unexpected exception during request; channel = [id: 0x8e2eae00, 
> L:/IP1.IP2.IP3.IP4:9042 - R:/IP5.IP6.IP7.IP8:58421]
> java.lang.NullPointerException: null
> at 
> org.apache.cassandra.serializers.BooleanSerializer.deserialize(BooleanSerializer.java:33)

[jira] [Commented] (CASSANDRA-11995) Commitlog replaced with all NULs

[James Howe (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533075#comment-15533075
 ] 

James Howe commented on CASSANDRA-11995:


Just happened to me again.
[~jogarcia], are you also running on Windows?

> Commitlog replaced with all NULs
> 
>
> Key: CASSANDRA-11995
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11995
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: Windows 10 Enterprise 1511
> DataStax Cassandra Community Server 2.2.3
>Reporter: James Howe
>
> I noticed this morning that Cassandra was failing to start, after being shut 
> down on Friday.
> {code}
> ERROR 09:13:37 Exiting due to error while processing commit log during 
> initialization.
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: 
> Could not read commit log descriptor in file C:\Program Files\DataStax 
> Community\data\commitlog\CommitLog-5-1465571056722.log
>   at 
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:622)
>  [apache-cassandra-2.2.3.jar:2.2.3]
>   at 
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:302)
>  [apache-cassandra-2.2.3.jar:2.2.3]
>   at 
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:147)
>  [apache-cassandra-2.2.3.jar:2.2.3]
>   at 
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189) 
> [apache-cassandra-2.2.3.jar:2.2.3]
>   at 
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169) 
> [apache-cassandra-2.2.3.jar:2.2.3]
>   at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:273) 
> [apache-cassandra-2.2.3.jar:2.2.3]
>   at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:513)
>  [apache-cassandra-2.2.3.jar:2.2.3]
>   at 
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:622) 
> [apache-cassandra-2.2.3.jar:2.2.3]
> {code}
> Checking the referenced file reveals it comprises 33,554,432 NUL bytes.
> No logs (stdout, stderr, prunsrv) from the shutdown show any other issues and 
> appear exactly as normal.
> Is installed as a service via DataStax's distribution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-12694) PAXOS Update Corrupted empty row exception

[Joshua McKenzie (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua McKenzie updated CASSANDRA-12694:

Assignee: Alex Petrov

> PAXOS Update Corrupted empty row exception
> --
>
> Key: CASSANDRA-12694
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12694
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths
> Environment: 3 node cluster using RF=3 running on cassandra 3.7
>Reporter: Cameron Zemek
>Assignee: Alex Petrov
>
> {noformat}
> cqlsh> create table test.test (test_id TEXT, last_updated TIMESTAMP, 
> message_id TEXT, PRIMARY KEY(test_id));
> update test.test set last_updated = 1474494363669 where test_id = 'test1' if 
> message_id = null;
> {noformat}
> Then nodetool flush on the all 3 nodes.
> {noformat}
> cqlsh> update test.test set last_updated = 1474494363669 where test_id = 
> 'test1' if message_id = null;
> ServerError: 
> {noformat}
> From cassandra log
> {noformat}
> ERROR [SharedPool-Worker-1] 2016-09-23 12:09:13,179 Message.java:611 - 
> Unexpected exception during request; channel = [id: 0x7a22599e, 
> L:/127.0.0.1:9042 - R:/127.0.0.1:58297]
> java.io.IOError: java.io.IOException: Corrupt empty row found in unfiltered 
> partition
> at 
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:224)
>  ~[main/:na]
> at 
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:212)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
> ~[main/:na]
> at 
> org.apache.cassandra.db.rows.UnfilteredRowIterators.digest(UnfilteredRowIterators.java:125)
>  ~[main/:na]
> at 
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators.digest(UnfilteredPartitionIterators.java:249)
>  ~[main/:na]
> at 
> org.apache.cassandra.db.ReadResponse.makeDigest(ReadResponse.java:87) 
> ~[main/:na]
> at 
> org.apache.cassandra.db.ReadResponse$DataResponse.digest(ReadResponse.java:192)
>  ~[main/:na]
> at 
> org.apache.cassandra.service.DigestResolver.resolve(DigestResolver.java:80) 
> ~[main/:na]
> at 
> org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:139) 
> ~[main/:na]
> at 
> org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:145)
>  ~[main/:na]
> at 
> org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.awaitResultsAndRetryOnDigestMismatch(StorageProxy.java:1714)
>  ~[main/:na]
> at 
> org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1663) 
> ~[main/:na]
> at 
> org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1604) 
> ~[main/:na]
> at 
> org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1523) 
> ~[main/:na]
> at 
> org.apache.cassandra.service.StorageProxy.readOne(StorageProxy.java:1497) 
> ~[main/:na]
> at 
> org.apache.cassandra.service.StorageProxy.readOne(StorageProxy.java:1491) 
> ~[main/:na]
> at 
> org.apache.cassandra.service.StorageProxy.cas(StorageProxy.java:249) 
> ~[main/:na]
> at 
> org.apache.cassandra.cql3.statements.ModificationStatement.executeWithCondition(ModificationStatement.java:441)
>  ~[main/:na]
> at 
> org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:416)
>  ~[main/:na]
> at 
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:208)
>  ~[main/:na]
> at 
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:239) 
> ~[main/:na]
> at 
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:224) 
> ~[main/:na]
> at 
> org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:115)
>  ~[main/:na]
> at 
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507)
>  [main/:na]
> at 
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401)
>  [main/:na]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-12697) cdc column addition still breaks schema migration tasks

[Sylvain Lebresne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-12697:
-
   Resolution: Fixed
Fix Version/s: (was: 3.x)
   3.10
   Status: Resolved  (was: Ready to Commit)

Committed, thanks.

> cdc column addition still breaks schema migration tasks
> ---
>
> Key: CASSANDRA-12697
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12697
> Project: Cassandra
>  Issue Type: Bug
>  Components: Distributed Metadata
>Reporter: Sylvain Lebresne
>Assignee: Sylvain Lebresne
> Fix For: 3.10
>
>
> This is a followup of CASSANDRA-12236, which basically didn't fully fix the 
> problem. Namely, the fix from CASSANDRA-12236 skipped the {{cdc}} column in 
> {{SchemaKeyspace.addTableParamsToRowBuilder()}}, but that method isn't used 
> by schema "migration tasks" ({{MigrationRequestVerbHandler}}, which instead 
> directly send the content of the full schema table it reads from disk. So we 
> still end up with a RTE like CASSANDRA-12236. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/3] cassandra git commit: cdc column addition still breaks schema migration tasks

[slebresne]

cdc column addition still breaks schema migration tasks

patch by Sylvain Lebresne; reviewed by Aleksey Yeschenko for CASSANDRA-12697


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/57e9a83b
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/57e9a83b
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/57e9a83b

Branch: refs/heads/trunk
Commit: 57e9a83b2abf08d7a1261e8f7a9f435d221a1f81
Parents: 25d4c7b
Author: Sylvain Lebresne 
Authored: Fri Sep 23 11:26:22 2016 +0200
Committer: Sylvain Lebresne 
Committed: Thu Sep 29 17:00:07 2016 +0200

--
 CHANGES.txt |  1 +
 .../cassandra/db/rows/WithOnlyQueriedData.java  |  7 +
 .../cassandra/db/transform/Transformation.java  | 12 +
 .../cassandra/db/transform/UnfilteredRows.java  | 11 
 .../apache/cassandra/schema/SchemaKeyspace.java | 28 +++-
 5 files changed, 58 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/57e9a83b/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index a9e46f7..c33b1d3 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.10
+ * cdc column addition still breaks schema migration tasks (CASSANDRA-12697)
  * Upgrade metrics-reporter dependencies (CASSANDRA-12089)
  * Tune compaction thread count via nodetool (CASSANDRA-12248)
  * Add +=/-= shortcut syntax for update queries (CASSANDRA-12232)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/57e9a83b/src/java/org/apache/cassandra/db/rows/WithOnlyQueriedData.java
--
diff --git a/src/java/org/apache/cassandra/db/rows/WithOnlyQueriedData.java 
b/src/java/org/apache/cassandra/db/rows/WithOnlyQueriedData.java
index 0b407f2..dcf0891 100644
--- a/src/java/org/apache/cassandra/db/rows/WithOnlyQueriedData.java
+++ b/src/java/org/apache/cassandra/db/rows/WithOnlyQueriedData.java
@@ -17,6 +17,7 @@
  */
 package org.apache.cassandra.db.rows;
 
+import org.apache.cassandra.db.PartitionColumns;
 import org.apache.cassandra.db.filter.ColumnFilter;
 import org.apache.cassandra.db.transform.Transformation;
 
@@ -35,6 +36,12 @@ public class WithOnlyQueriedData> extends Transform
 }
 
 @Override
+protected PartitionColumns applyToPartitionColumns(PartitionColumns 
columns)
+{
+return filter.queriedColumns();
+}
+
+@Override
 protected Row applyToStatic(Row row)
 {
 return row.withOnlyQueriedData(filter);

http://git-wip-us.apache.org/repos/asf/cassandra/blob/57e9a83b/src/java/org/apache/cassandra/db/transform/Transformation.java
--
diff --git a/src/java/org/apache/cassandra/db/transform/Transformation.java 
b/src/java/org/apache/cassandra/db/transform/Transformation.java
index 3134725..33c1fe7 100644
--- a/src/java/org/apache/cassandra/db/transform/Transformation.java
+++ b/src/java/org/apache/cassandra/db/transform/Transformation.java
@@ -22,6 +22,7 @@ package org.apache.cassandra.db.transform;
 
 import org.apache.cassandra.db.DecoratedKey;
 import org.apache.cassandra.db.DeletionTime;
+import org.apache.cassandra.db.PartitionColumns;
 import org.apache.cassandra.db.partitions.PartitionIterator;
 import org.apache.cassandra.db.partitions.UnfilteredPartitionIterator;
 import org.apache.cassandra.db.rows.*;
@@ -109,6 +110,17 @@ public abstract class Transformation>
 return deletionTime;
 }
 
+/**
+ * Applied to the {@code PartitionColumns} of any rows iterator.
+ *
+ * NOTE: same remark than for applyToDeletion: it is only applied to the 
first iterator in a sequence of iterators
+ * filled by MoreContents.
+ */
+protected PartitionColumns applyToPartitionColumns(PartitionColumns 
columns)
+{
+return columns;
+}
+
 
 //**
 //  Static Application Methods

http://git-wip-us.apache.org/repos/asf/cassandra/blob/57e9a83b/src/java/org/apache/cassandra/db/transform/UnfilteredRows.java
--
diff --git a/src/java/org/apache/cassandra/db/transform/UnfilteredRows.java 
b/src/java/org/apache/cassandra/db/transform/UnfilteredRows.java
index f000fcf..ba86066 100644
--- a/src/java/org/apache/cassandra/db/transform/UnfilteredRows.java
+++ b/src/java/org/apache/cassandra/db/transform/UnfilteredRows.java
@@ -21,17 +21,20 @@
 package org.apache.cassandra.db.transform;
 
 import org.apache.cassandra.db.DeletionTime;
+import org.apache.cassandra.db.PartitionColumns;
 import 

[1/3] cassandra git commit: cdc column addition still breaks schema migration tasks

[slebresne]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.X 25d4c7baa -> 57e9a83b2
  refs/heads/trunk 78ff37707 -> c18968b1b


cdc column addition still breaks schema migration tasks

patch by Sylvain Lebresne; reviewed by Aleksey Yeschenko for CASSANDRA-12697


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/57e9a83b
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/57e9a83b
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/57e9a83b

Branch: refs/heads/cassandra-3.X
Commit: 57e9a83b2abf08d7a1261e8f7a9f435d221a1f81
Parents: 25d4c7b
Author: Sylvain Lebresne 
Authored: Fri Sep 23 11:26:22 2016 +0200
Committer: Sylvain Lebresne 
Committed: Thu Sep 29 17:00:07 2016 +0200

--
 CHANGES.txt |  1 +
 .../cassandra/db/rows/WithOnlyQueriedData.java  |  7 +
 .../cassandra/db/transform/Transformation.java  | 12 +
 .../cassandra/db/transform/UnfilteredRows.java  | 11 
 .../apache/cassandra/schema/SchemaKeyspace.java | 28 +++-
 5 files changed, 58 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/57e9a83b/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index a9e46f7..c33b1d3 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.10
+ * cdc column addition still breaks schema migration tasks (CASSANDRA-12697)
  * Upgrade metrics-reporter dependencies (CASSANDRA-12089)
  * Tune compaction thread count via nodetool (CASSANDRA-12248)
  * Add +=/-= shortcut syntax for update queries (CASSANDRA-12232)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/57e9a83b/src/java/org/apache/cassandra/db/rows/WithOnlyQueriedData.java
--
diff --git a/src/java/org/apache/cassandra/db/rows/WithOnlyQueriedData.java 
b/src/java/org/apache/cassandra/db/rows/WithOnlyQueriedData.java
index 0b407f2..dcf0891 100644
--- a/src/java/org/apache/cassandra/db/rows/WithOnlyQueriedData.java
+++ b/src/java/org/apache/cassandra/db/rows/WithOnlyQueriedData.java
@@ -17,6 +17,7 @@
  */
 package org.apache.cassandra.db.rows;
 
+import org.apache.cassandra.db.PartitionColumns;
 import org.apache.cassandra.db.filter.ColumnFilter;
 import org.apache.cassandra.db.transform.Transformation;
 
@@ -35,6 +36,12 @@ public class WithOnlyQueriedData> extends Transform
 }
 
 @Override
+protected PartitionColumns applyToPartitionColumns(PartitionColumns 
columns)
+{
+return filter.queriedColumns();
+}
+
+@Override
 protected Row applyToStatic(Row row)
 {
 return row.withOnlyQueriedData(filter);

http://git-wip-us.apache.org/repos/asf/cassandra/blob/57e9a83b/src/java/org/apache/cassandra/db/transform/Transformation.java
--
diff --git a/src/java/org/apache/cassandra/db/transform/Transformation.java 
b/src/java/org/apache/cassandra/db/transform/Transformation.java
index 3134725..33c1fe7 100644
--- a/src/java/org/apache/cassandra/db/transform/Transformation.java
+++ b/src/java/org/apache/cassandra/db/transform/Transformation.java
@@ -22,6 +22,7 @@ package org.apache.cassandra.db.transform;
 
 import org.apache.cassandra.db.DecoratedKey;
 import org.apache.cassandra.db.DeletionTime;
+import org.apache.cassandra.db.PartitionColumns;
 import org.apache.cassandra.db.partitions.PartitionIterator;
 import org.apache.cassandra.db.partitions.UnfilteredPartitionIterator;
 import org.apache.cassandra.db.rows.*;
@@ -109,6 +110,17 @@ public abstract class Transformation>
 return deletionTime;
 }
 
+/**
+ * Applied to the {@code PartitionColumns} of any rows iterator.
+ *
+ * NOTE: same remark than for applyToDeletion: it is only applied to the 
first iterator in a sequence of iterators
+ * filled by MoreContents.
+ */
+protected PartitionColumns applyToPartitionColumns(PartitionColumns 
columns)
+{
+return columns;
+}
+
 
 //**
 //  Static Application Methods

http://git-wip-us.apache.org/repos/asf/cassandra/blob/57e9a83b/src/java/org/apache/cassandra/db/transform/UnfilteredRows.java
--
diff --git a/src/java/org/apache/cassandra/db/transform/UnfilteredRows.java 
b/src/java/org/apache/cassandra/db/transform/UnfilteredRows.java
index f000fcf..ba86066 100644
--- a/src/java/org/apache/cassandra/db/transform/UnfilteredRows.java
+++ b/src/java/org/apache/cassandra/db/transform/UnfilteredRows.java
@@ -21,17 +21,20 @@
 package 

[jira] [Assigned] (CASSANDRA-12674) [SASI] Confusing AND/OR semantics for StandardAnalyzer

[Alex Petrov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Petrov reassigned CASSANDRA-12674:
---

Assignee: Alex Petrov

> [SASI] Confusing AND/OR semantics for StandardAnalyzer 
> ---
>
> Key: CASSANDRA-12674
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12674
> Project: Cassandra
>  Issue Type: Bug
>  Components: sasi
> Environment: Cassandra 3.7
>Reporter: DOAN DuyHai
>Assignee: Alex Petrov
>
> {code:sql}
> Connected to Test Cluster at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 3.7 | CQL spec 3.4.2 | Native protocol v4]
> Use HELP for help.
> cqlsh> use test;
> cqlsh:test> CREATE TABLE sasi_bug(id int, clustering int, val text, PRIMARY 
> KEY((id), clustering));
> cqlsh:test> CREATE CUSTOM INDEX ON sasi_bug(val) USING 
> 'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS = {
> 'mode': 'CONTAINS',
>  'analyzer_class': 
> 'org.apache.cassandra.index.sasi.analyzer.StandardAnalyzer',
> 'analyzed': 'true'};
> //1st example SAME PARTITION KEY
> cqlsh:test> INSERT INTO sasi_bug(id, clustering , val ) VALUES(1, 1, 
> 'homeworker');
> cqlsh:test> INSERT INTO sasi_bug(id, clustering , val ) VALUES(1, 2, 
> 'hardworker');
> cqlsh:test> SELECT * FROM sasi_bug WHERE val LIKE '%work home%';
>  id | clustering | val
> ++
>   1 |  1 | homeworker
>   1 |  2 | hardworker
> (2 rows)
> //2nd example DIFFERENT PARTITION KEY
> cqlsh:test> INSERT INTO sasi_bug(id, clustering, val) VALUES(10, 1, 
> 'speedrun');
> cqlsh:test> INSERT INTO sasi_bug(id, clustering, val) VALUES(11, 1, 
> 'longrun');
> cqlsh:test> SELECT * FROM sasi_bug WHERE val LIKE '%long run%';
>  id | clustering | val
> ++-
>  11 |  1 | longrun
> (1 rows)
> {code}
> In the 1st example, both rows belong to the same partition so SASI returns 
> both values. Indeed {{LIKE '%work home%'}} means {{contains 'work' OR 
> 'home'}} so the result makes sense
> In the 2nd example, only one row is returned whereas we expect 2 rows because 
> {{LIKE '%long run%'}} means {{contains 'long' OR 'run'}} so *speedrun* should 
> be returned too.
> So where is the problem ? Explanation:
> When there is only 1 predicate, the root operation type is an *AND*:
> {code:java|title=QueryPlan}
> private Operation analyze()
> {
> try
> {
> Operation.Builder and = new Operation.Builder(OperationType.AND, 
> controller);
> controller.getExpressions().forEach(and::add);
> return and.complete();
> }
>...
> }
> {code}
> During the parsing of {{LIKE '%long run%'}}, SASI creates 2 expressions for 
> the searched term: {{long}} and {{run}}, which corresponds to an *OR* logic. 
> However, this piece of code just ruins the *OR* logic:
> {code:java|title=Operation}
> public Operation complete()
> {
> if (!expressions.isEmpty())
> {
> ListMultimap 
> analyzedExpressions = analyzeGroup(controller, op, expressions);
> RangeIterator.Builder range = 
> controller.getIndexes(op, analyzedExpressions.values());
>  ...
> }
> {code}
> As you can see, we blindly take all the *values* of the MultiMap (which 
> contains a single entry for the {{val}} column with 2 expressions) and pass 
> it to {{controller.getIndexes(...)}}
> {code:java|title=QueryController}
> public RangeIterator.Builder getIndexes(OperationType op, 
> Collection expressions)
> {
> if (resources.containsKey(expressions))
> throw new IllegalArgumentException("Can't process the same 
> expressions multiple times.");
> RangeIterator.Builder builder = op == OperationType.OR
> ? RangeUnionIterator. Token>builder()
> : 
> RangeIntersectionIterator.builder();
> ...
> }
> {code}
> And because the root operation has *AND* type, the 
> {{RangeIntersectionIterator}} will be used on both expressions {{long}} and 
> {{run}}.
> So when data belong to different partitions, we have the *AND* logic that 
> applies and eliminates _speedrun_
> When data belong to the same partition but different row, the 
> {{RangeIntersectionIterator}} returns a single partition and then the rows 
> are filtered further by {{operationTree.satisfiedBy}} and the results are 
> correct
> {code:java|title=QueryPlan}
> while (currentKeys.hasNext())
> {
> DecoratedKey key = currentKeys.next();
> if (!keyRange.right.isMinimum() && 
> keyRange.right.compareTo(key) < 0)
>  

[3/3] cassandra git commit: Merge branch 'cassandra-3.X' into trunk

[slebresne]

Merge branch 'cassandra-3.X' into trunk

* cassandra-3.X:
  cdc column addition still breaks schema migration tasks


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c18968b1
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c18968b1
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c18968b1

Branch: refs/heads/trunk
Commit: c18968b1bd54b182f8d65126ccc03586001b62ed
Parents: 78ff377 57e9a83
Author: Sylvain Lebresne 
Authored: Thu Sep 29 17:00:57 2016 +0200
Committer: Sylvain Lebresne 
Committed: Thu Sep 29 17:00:57 2016 +0200

--
 CHANGES.txt |  1 +
 .../cassandra/db/rows/WithOnlyQueriedData.java  |  7 +
 .../cassandra/db/transform/Transformation.java  | 12 +
 .../cassandra/db/transform/UnfilteredRows.java  | 11 
 .../apache/cassandra/schema/SchemaKeyspace.java | 28 +++-
 5 files changed, 58 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/c18968b1/CHANGES.txt
--
diff --cc CHANGES.txt
index 03456a0,c33b1d3..8a1fd1d
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,7 -1,5 +1,8 @@@
 +4.0
 +
 +
  3.10
+  * cdc column addition still breaks schema migration tasks (CASSANDRA-12697)
   * Upgrade metrics-reporter dependencies (CASSANDRA-12089)
   * Tune compaction thread count via nodetool (CASSANDRA-12248)
   * Add +=/-= shortcut syntax for update queries (CASSANDRA-12232)

[jira] [Updated] (CASSANDRA-12662) OOM when using SASI index

[Alex Petrov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Petrov updated CASSANDRA-12662:

Assignee: (was: Alex Petrov)

> OOM when using SASI index
> -
>
> Key: CASSANDRA-12662
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12662
> Project: Cassandra
>  Issue Type: Bug
> Environment: Linux, 4 CPU cores, 16Gb RAM, Cassandra process utilizes 
> ~8Gb, of which ~4Gb is Java heap
>Reporter: Maxim Podkolzine
>Priority: Critical
> Fix For: 3.x
>
> Attachments: memory-dump.png
>
>
> 2.8Gb of the heap is taken by the index data, pending for flush (see the 
> screenshot). As a result the node fails with OOM.
> Questions:
> - Why can't Cassandra keep up with the inserted data and flush it?
> - What resources/configuration should be changed to improve the performance?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12662) OOM when using SASI index

[Alex Petrov (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533008#comment-15533008
 ] 

Alex Petrov commented on CASSANDRA-12662:
-

20Gb per node should be quite fine. Did you try configuring 
{{memtable_*heap_space_in_mb}} (which is 1/4 heap by default) and 
{{memtable_cleanup_threshold}}. So technically 4Gb partition updates is still 
"under" the threshold. Problem is that there's some memory (also, CPU) involved 
into the flush process, so making it lower should help.

> OOM when using SASI index
> -
>
> Key: CASSANDRA-12662
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12662
> Project: Cassandra
>  Issue Type: Bug
> Environment: Linux, 4 CPU cores, 16Gb RAM, Cassandra process utilizes 
> ~8Gb, of which ~4Gb is Java heap
>Reporter: Maxim Podkolzine
>Assignee: Alex Petrov
>Priority: Critical
> Fix For: 3.x
>
> Attachments: memory-dump.png
>
>
> 2.8Gb of the heap is taken by the index data, pending for flush (see the 
> screenshot). As a result the node fails with OOM.
> Questions:
> - Why can't Cassandra keep up with the inserted data and flush it?
> - What resources/configuration should be changed to improve the performance?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-12399) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x.multi_collection_test

[Benjamin Lerer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Lerer resolved CASSANDRA-12399.

Resolution: Cannot Reproduce

> dtest failure in 
> upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x.multi_collection_test
> 
>
> Key: CASSANDRA-12399
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12399
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Sean McCarthy
>Assignee: Benjamin Lerer
>  Labels: dtest
>
> example failure:
> http://cassci.datastax.com/job/trunk_dtest_upgrade/17/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x/multi_collection_test
> {code}
> Stacktrace
>   File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
>   File "/home/automaton/cassandra-dtest/upgrade_tests/cql_tests.py", line 
> 1419, in multi_collection_test
> sortedset([1, 3, 5, 7, 11, 13])
>   File "/home/automaton/cassandra-dtest/assertions.py", line 163, in 
> assert_all
> assert list_res == expected, "Expected {} from {}, but got 
> {}".format(expected, query, list_res)
> "Expected [[[1, 3, 5, 7, 11, 13], OrderedDict([('bar', 3), ('foo', 1), 
> ('foobar', 4)]), SortedSet([1, 3, 5, 7, 11, 13])]] from SELECT L, M, S FROM 
> foo WHERE k = b017f48f-ae67-11e1-9096-005056c8, but got [[[1, 3, 5, 1, 3, 
> 5, 7, 11, 13], OrderedMapSerializedKey([(u'bar', 3), (u'foo', 1), (u'foobar', 
> 4)]), SortedSet([1, 3, 5, 7, 11, 13])]]
> {code}
> Related failure:
> http://cassci.datastax.com/job/trunk_dtest_upgrade/17/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x/multi_collection_test/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-12401) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x.multi_list_set_test

[Benjamin Lerer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Lerer resolved CASSANDRA-12401.

Resolution: Cannot Reproduce

> dtest failure in 
> upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x.multi_list_set_test
> 
>
> Key: CASSANDRA-12401
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12401
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Sean McCarthy
>Assignee: Benjamin Lerer
>  Labels: dtest
>
> example failure:
> http://cassci.datastax.com/job/trunk_dtest_upgrade/17/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x/multi_list_set_test
> {code}
> Stacktrace
>   File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
>   File "/home/automaton/cassandra-dtest/upgrade_tests/cql_tests.py", line 
> 2289, in multi_list_set_test
> assert_one(cursor, "SELECT l1, l2 FROM test WHERE k = 0", [[1, 24, 3], 
> [4, 42, 6]])
>   File "/home/automaton/cassandra-dtest/assertions.py", line 124, in 
> assert_one
> assert list_res == [expected], "Expected {} from {}, but got 
> {}".format([expected], query, list_res)
> "Expected [[[1, 24, 3], [4, 42, 6]]] from SELECT l1, l2 FROM test WHERE k = 
> 0, but got [[[1, 24, 3, 1, 2, 3], [4, 42, 6, 4, 5, 6]]]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (CASSANDRA-12662) OOM when using SASI index

[Alex Petrov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Petrov reassigned CASSANDRA-12662:
---

Assignee: Alex Petrov

> OOM when using SASI index
> -
>
> Key: CASSANDRA-12662
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12662
> Project: Cassandra
>  Issue Type: Bug
> Environment: Linux, 4 CPU cores, 16Gb RAM, Cassandra process utilizes 
> ~8Gb, of which ~4Gb is Java heap
>Reporter: Maxim Podkolzine
>Assignee: Alex Petrov
>Priority: Critical
> Fix For: 3.x
>
> Attachments: memory-dump.png
>
>
> 2.8Gb of the heap is taken by the index data, pending for flush (see the 
> screenshot). As a result the node fails with OOM.
> Questions:
> - Why can't Cassandra keep up with the inserted data and flush it?
> - What resources/configuration should be changed to improve the performance?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (CASSANDRA-12654) Query Validation Error : CQL IN operator over last partitioning|clustering column (valid) is rejected if a query fetches collection columns

[Alex Petrov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Petrov reassigned CASSANDRA-12654:
---

Assignee: Alex Petrov

> Query Validation Error : CQL IN operator over last partitioning|clustering 
> column (valid) is rejected if a query fetches collection columns
> ---
>
> Key: CASSANDRA-12654
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12654
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Samba Siva Rao Kolusu
>Assignee: Alex Petrov
>Priority: Minor
>  Labels: easyfix
> Fix For: 3.x
>
>
> although IN operator is allowed over the last clustering or partitioning 
> columns, the CQL Query Validator is rejecting queries when they attempt to 
> fetch collection columns in their result set.
> It seems a similar bug (CASSANDRA-5376) was raised some time ago, and a fix 
> (rather mask) was provided to give a better error message to such queries in 
> 1.2.4. 
> Considering that Cassandra & CQL has evolved a great deal from that period, 
> it now seems possible to provide an actual fix to this problem, i.e. allowing 
> queries to fetch collection columns even when IN operator is used.
> please read the following mail thread to understand the context : 
> https://lists.apache.org/thread.html/8e1765d14bd9798bf9c0938a793f1dbc9c9349062a8705db2e28d291@%3Cuser.cassandra.apache.org%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-11846) Invalid QueryBuilder.insert is not invalidated which causes OOM

[Alex Petrov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Petrov resolved CASSANDRA-11846.
-
Resolution: Duplicate

> Invalid QueryBuilder.insert is not invalidated which causes OOM
> ---
>
> Key: CASSANDRA-11846
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11846
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
> Environment: cassandra-2.1.14
>Reporter: ZhaoYang
>Priority: Minor
> Fix For: 2.1.x
>
>
> create table test(  key text primary key, value list );
> When using QueryBuilder.Insert() to bind column `value` with a blob, 
> Cassandra didn't consider it to be an invalid query and then lead to OOM and 
> crashed.
> the same plain query(String) can be invalidated by Cassandra and C* responds 
> InvalidQuery.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-11846) Invalid QueryBuilder.insert is not invalidated which causes OOM

[Alex Petrov (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15532942#comment-15532942
 ] 

Alex Petrov edited comment on CASSANDRA-11846 at 9/29/16 2:32 PM:
--

Fixed in [CASSANDRA-12618]


was (Author: ifesdjeen):
Fixed in [https://issues.apache.org/jira/browse/CASSANDRA-12618]

> Invalid QueryBuilder.insert is not invalidated which causes OOM
> ---
>
> Key: CASSANDRA-11846
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11846
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
> Environment: cassandra-2.1.14
>Reporter: ZhaoYang
>Priority: Minor
> Fix For: 2.1.x
>
>
> create table test(  key text primary key, value list );
> When using QueryBuilder.Insert() to bind column `value` with a blob, 
> Cassandra didn't consider it to be an invalid query and then lead to OOM and 
> crashed.
> the same plain query(String) can be invalidated by Cassandra and C* responds 
> InvalidQuery.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11846) Invalid QueryBuilder.insert is not invalidated which causes OOM

[Alex Petrov (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15532942#comment-15532942
 ] 

Alex Petrov commented on CASSANDRA-11846:
-

Fixed in [https://issues.apache.org/jira/browse/CASSANDRA-12618]

> Invalid QueryBuilder.insert is not invalidated which causes OOM
> ---
>
> Key: CASSANDRA-11846
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11846
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
> Environment: cassandra-2.1.14
>Reporter: ZhaoYang
>Priority: Minor
> Fix For: 2.1.x
>
>
> create table test(  key text primary key, value list );
> When using QueryBuilder.Insert() to bind column `value` with a blob, 
> Cassandra didn't consider it to be an invalid query and then lead to OOM and 
> crashed.
> the same plain query(String) can be invalidated by Cassandra and C* responds 
> InvalidQuery.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12461) Add hooks to StorageService shutdown

[Alex Petrov (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15532870#comment-15532870
 ] 

Alex Petrov commented on CASSANDRA-12461:
-

Right, [CASSANDRA-12011] is contained. I've already created patch in 
[CASSANDRA-12509], but anything would work for me)

> Add hooks to StorageService shutdown
> 
>
> Key: CASSANDRA-12461
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12461
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Anthony Cozzie
>Assignee: Anthony Cozzie
> Fix For: 3.x
>
> Attachments: 
> 0001-CASSANDRA-12461-add-C-support-for-shutdown-runnables.patch
>
>
> The JVM will usually run shutdown hooks in parallel.  This can lead to 
> synchronization problems between Cassandra, services that depend on it, and 
> services it depends on.  This patch adds some simple support for shutdown 
> hooks to StorageService.
> This should nearly solve CASSANDRA-12011



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12461) Add hooks to StorageService shutdown

[Jeremiah Jordan (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15532856#comment-15532856
 ] 

Jeremiah Jordan commented on CASSANDRA-12461:
-

If we put the hooks in 3.0 we can use them to fix CASSANDRA-12011. 

> Add hooks to StorageService shutdown
> 
>
> Key: CASSANDRA-12461
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12461
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Anthony Cozzie
>Assignee: Anthony Cozzie
> Fix For: 3.x
>
> Attachments: 
> 0001-CASSANDRA-12461-add-C-support-for-shutdown-runnables.patch
>
>
> The JVM will usually run shutdown hooks in parallel.  This can lead to 
> synchronization problems between Cassandra, services that depend on it, and 
> services it depends on.  This patch adds some simple support for shutdown 
> hooks to StorageService.
> This should nearly solve CASSANDRA-12011



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-12461) Add hooks to StorageService shutdown

[Jeremiah Jordan (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15532856#comment-15532856
 ] 

Jeremiah Jordan edited comment on CASSANDRA-12461 at 9/29/16 2:01 PM:
--

If we put the hooks in 3.0 we can use them to fix CASSANDRA-12011, if we want 
to fix that there.


was (Author: jjordan):
If we put the hooks in 3.0 we can use them to fix CASSANDRA-12011. 

> Add hooks to StorageService shutdown
> 
>
> Key: CASSANDRA-12461
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12461
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Anthony Cozzie
>Assignee: Anthony Cozzie
> Fix For: 3.x
>
> Attachments: 
> 0001-CASSANDRA-12461-add-C-support-for-shutdown-runnables.patch
>
>
> The JVM will usually run shutdown hooks in parallel.  This can lead to 
> synchronization problems between Cassandra, services that depend on it, and 
> services it depends on.  This patch adds some simple support for shutdown 
> hooks to StorageService.
> This should nearly solve CASSANDRA-12011



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12571) cqlsh lost the ability to have a request wait indefinitely

[Nate Sanders (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15532847#comment-15532847
 ] 

Nate Sanders commented on CASSANDRA-12571:
--

Sounds good to me.  Thanks.

> cqlsh lost the ability to have a request wait indefinitely
> --
>
> Key: CASSANDRA-12571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12571
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
> Environment: Cassandra 3.7
>Reporter: Nate Sanders
>Assignee: Stefania
>Priority: Minor
>  Labels: cqlsh, doc-impacting, lhf
>
> In commit c7f0032912798b5e53b64d8391e3e3d7e4121165, when client_timeout 
> became request_timeout, the logic was changed so that you can no longer use a 
> timeout of None, despite the docs saying that you can:
> https://docs.datastax.com/en/cql/3.3/cql/cql_reference/cqlshUsingCqlshrc.html#cqlshUsingCqlshrc__request-timeout



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-12509) Shutdown process triggered twice during if the node is drained

[Alex Petrov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Petrov updated CASSANDRA-12509:

Assignee: Alex Petrov
Reviewer: Stefania
  Status: Patch Available  (was: Open)

Moving the bugfix here from [CASSANDRA-12461], re-triggering CI:

|[12509-trunk|https://github.com/ifesdjeen/cassandra/tree/12509-trunk]|[dtest|http://cassci.datastax.com/job/ifesdjeen-12509-trunk-dtest/]|[utest|http://cassci.datastax.com/job/ifesdjeen-12509-trunk-testall/]|
|[12509-3.0|https://github.com/ifesdjeen/cassandra/tree/12509-3.0]|[dtest|http://cassci.datastax.com/job/ifesdjeen-12509-3.0-dtest/]|[utest|http://cassci.datastax.com/job/ifesdjeen-12509-3.0-testall/]|

> Shutdown process triggered twice during if the node is drained
> --
>
> Key: CASSANDRA-12509
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12509
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Alex Petrov
>Assignee: Alex Petrov
>
> If the node is drained, the {{StorageService#drain}} 
> [method|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/StorageService.java#L4212]
>  is called, which triggers shutdown of mutation stage, messaging service, 
> compaction, batchlog etc. In the end of this process, the node is moved to 
> {{DRAINED}} status with the process still running. 
> When JVM is shutdown, the JVM shutdown hooks are ran, which are subscribed 
> during the server initialisation: 
> {{Runtime.getRuntime().addShutdownHook(drainOnShutdown);}} 
> [here|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/StorageService.java#L575-L636].
>  
> I noticed this behaviour while reviewing [CASSANDRA-12461], as if we'd like 
> add custom pre and post-shutdown hooks, most likely it makes sense to run 
> them once (or user might expect such behaviour). 
> Is this behaviour correct? Should we run whole shutdown process twice or just 
> once in "drain" and no-op during JVM shutdown?  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12461) Add hooks to StorageService shutdown

[Alex Petrov (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15532753#comment-15532753
 ] 

Alex Petrov commented on CASSANDRA-12461:
-

Moved the bugfix to [CASSANDRA-12509], will have to rebase after it's 
committed. 

> Add hooks to StorageService shutdown
> 
>
> Key: CASSANDRA-12461
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12461
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Anthony Cozzie
>Assignee: Anthony Cozzie
> Fix For: 3.x
>
> Attachments: 
> 0001-CASSANDRA-12461-add-C-support-for-shutdown-runnables.patch
>
>
> The JVM will usually run shutdown hooks in parallel.  This can lead to 
> synchronization problems between Cassandra, services that depend on it, and 
> services it depends on.  This patch adds some simple support for shutdown 
> hooks to StorageService.
> This should nearly solve CASSANDRA-12011



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[cassandra] Git Push Summary

[slebresne]

Repository: cassandra
Updated Branches:
  refs/heads/3.X [deleted] 25d4c7baa

[cassandra] Git Push Summary

[slebresne]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.X [created] 25d4c7baa

[jira] [Comment Edited] (CASSANDRA-12724) dtest failure in cql_tests.LWTTester.conditional_updates_on_static_columns_with_null_values_test

[Alex Petrov (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15532715#comment-15532715
 ] 

Alex Petrov edited comment on CASSANDRA-12724 at 9/29/16 1:00 PM:
--

[CASSANDRA-12060] is only for 3.0.10 and 3.10 and up. 3.0.9 and 3.9 would still 
fail. This patch fixes it.

[dtest branch|https://github.com/riptano/cassandra-dtest/pull/1353] 


was (Author: ifesdjeen):
[dtest branch|https://github.com/riptano/cassandra-dtest/pull/1353] 

> dtest failure in 
> cql_tests.LWTTester.conditional_updates_on_static_columns_with_null_values_test
> 
>
> Key: CASSANDRA-12724
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12724
> Project: Cassandra
>  Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Alex Petrov
>  Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-3.9_dtest/57/testReport/cql_tests/LWTTester/conditional_updates_on_static_columns_with_null_values_test
> {code}
> Error Message
> Expected [[False, None]] from DELETE s1 FROM 
> conditional_deletes_on_static_with_null WHERE a = 2 IF s2 IN (10,20,30), but 
> got [[False]]
> {code}
> {code}
> Stacktrace
>   File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
>   File "/home/automaton/cassandra-dtest/cql_tests.py", line 1289, in 
> conditional_deletes_on_static_columns_with_null_values_test
> assert_one(session, "DELETE s1 FROM {} WHERE a = 2 IF s2 IN 
> (10,20,30)".format(table_name), [False, None])
>   File "/home/automaton/cassandra-dtest/tools/assertions.py", line 130, in 
> assert_one
> assert list_res == [expected], "Expected {} from {}, but got 
> {}".format([expected], query, list_res)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12724) dtest failure in cql_tests.LWTTester.conditional_updates_on_static_columns_with_null_values_test

[Alex Petrov (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15532715#comment-15532715
 ] 

Alex Petrov commented on CASSANDRA-12724:
-

[dtest branch|https://github.com/riptano/cassandra-dtest/pull/1353] 

> dtest failure in 
> cql_tests.LWTTester.conditional_updates_on_static_columns_with_null_values_test
> 
>
> Key: CASSANDRA-12724
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12724
> Project: Cassandra
>  Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Alex Petrov
>  Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-3.9_dtest/57/testReport/cql_tests/LWTTester/conditional_updates_on_static_columns_with_null_values_test
> {code}
> Error Message
> Expected [[False, None]] from DELETE s1 FROM 
> conditional_deletes_on_static_with_null WHERE a = 2 IF s2 IN (10,20,30), but 
> got [[False]]
> {code}
> {code}
> Stacktrace
>   File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
>   File "/home/automaton/cassandra-dtest/cql_tests.py", line 1289, in 
> conditional_deletes_on_static_columns_with_null_values_test
> assert_one(session, "DELETE s1 FROM {} WHERE a = 2 IF s2 IN 
> (10,20,30)".format(table_name), [False, None])
>   File "/home/automaton/cassandra-dtest/tools/assertions.py", line 130, in 
> assert_one
> assert list_res == [expected], "Expected {} from {}, but got 
> {}".format([expected], query, list_res)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-12717) IllegalArgumentException in CompactionTask

[Yasuharu Goto (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yasuharu Goto updated CASSANDRA-12717:
--
Summary: IllegalArgumentException in CompactionTask  (was: Fix 
IllegalArgumentException in CompactionTask)

> IllegalArgumentException in CompactionTask
> --
>
> Key: CASSANDRA-12717
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12717
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Yasuharu Goto
>Assignee: Yasuharu Goto
>
> When I was ran LargePartitionsTest.test_11_1G at trunk, I found that this 
> test fails due to a java.lang.IllegalArgumentException during compaction.
> This exception apparently happens when the compaction merges a large (>2GB) 
> partition.
> {noformat}
> DEBUG [COMMIT-LOG-ALLOCATOR] 2016-09-28 00:32:48,074 ?:? - No segments in 
> reserve; creating a fresh one
> DEBUG [COMMIT-LOG-ALLOCATOR] 2016-09-28 00:32:48,437 ?:? - No segments in 
> reserve; creating a fresh one
> WARN  [CompactionExecutor:14] 2016-09-28 00:32:48,463 ?:? - Writing large 
> partition cql_test_keyspace/table_4:10 (1.004GiB)
> ERROR [CompactionExecutor:14] 2016-09-28 00:32:49,734 ?:? - Fatal exception 
> in thread Thread[CompactionExecutor:14,1,main]
> java.lang.IllegalArgumentException: Out of range: 2234434614
> at com.google.common.primitives.Ints.checkedCast(Ints.java:91) 
> ~[guava-18.0.jar:na]
> at 
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:206)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) 
> ~[main/:na]
> at 
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
>  ~[main/:na]
> at 
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
>  ~[main/:na]
> at 
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:267)
>  ~[main/:na]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> ~[na:1.8.0_77]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> ~[na:1.8.0_77]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  ~[na:1.8.0_77]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_77]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
> DEBUG [COMMIT-LOG-ALLOCATOR] 2016-09-28 00:32:49,909 ?:? - No segments in 
> reserve; creating a fresh one
> DEBUG [COMMIT-LOG-ALLOCATOR] 2016-09-28 00:32:50,148 ?:? - No segments in 
> reserve; creating a fresh one
> DEBUG [COMMIT-LOG-ALLOCATOR] 2016-09-28 00:32:50,385 ?:? - No segments in 
> reserve; creating a fresh one
> DEBUG [COMMIT-LOG-ALLOCATOR] 2016-09-28 00:32:50,620 ?:? - No segments in 
> reserve; creating a fresh one
> {noformat}
> {noformat}
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: 
> java.lang.IllegalArgumentException: Out of range: 2540348821
> at org.apache.cassandra.utils.Throwables.maybeFail(Throwables.java:51)
> at 
> org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:393)
> at 
> org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:695)
> at 
> org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:2066)
> at 
> org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:2061)
> at org.apache.cassandra.cql3.CQLTester.compact(CQLTester.java:426)
> at 
> org.apache.cassandra.io.sstable.LargePartitionsTest.lambda$withPartitionSize$2(LargePartitionsTest.java:92)
> at 
> org.apache.cassandra.io.sstable.LargePartitionsTest.measured(LargePartitionsTest.java:50)
> at 
> org.apache.cassandra.io.sstable.LargePartitionsTest.withPartitionSize(LargePartitionsTest.java:90)
> at 
> org.apache.cassandra.io.sstable.LargePartitionsTest.test_11_1G(LargePartitionsTest.java:198)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at 
> 

[jira] [Commented] (CASSANDRA-12693) Add the JMX metrics about the total number of hints we have delivered

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15532423#comment-15532423
 ] 

Aleksey Yeschenko commented on CASSANDRA-12693:
---

||branch||testall||dtest||
|[12693|https://github.com/iamaleksey/cassandra/tree/12693]|[testall|http://cassci.datastax.com/view/Dev/view/iamaleksey/job/iamaleksey-12693-testall]|[dtest|http://cassci.datastax.com/view/Dev/view/iamaleksey/job/iamaleksey-12693-dtest]|

> Add the JMX metrics about the total number of hints we have delivered
> -
>
> Key: CASSANDRA-12693
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12693
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Dikang Gu
>Assignee: Dikang Gu
>Priority: Minor
> Fix For: 3.x
>
> Attachments: 
> 0001-Add-the-metrics-about-total-number-of-hints-we-have-.patch
>
>
> I find there are no metrics about the number of hints we have delivered, I 
> think it would be great to have the metrics, so that we have better 
> estimation about the progress of hints replay.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12461) Add hooks to StorageService shutdown

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15532397#comment-15532397
 ] 

Aleksey Yeschenko commented on CASSANDRA-12461:
---

I'd only leave the bug fix for 3.0. To make it less confusing, pull them all 
into a separate ticket, this one only adding the hooks.

> Add hooks to StorageService shutdown
> 
>
> Key: CASSANDRA-12461
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12461
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Anthony Cozzie
>Assignee: Anthony Cozzie
> Fix For: 3.x
>
> Attachments: 
> 0001-CASSANDRA-12461-add-C-support-for-shutdown-runnables.patch
>
>
> The JVM will usually run shutdown hooks in parallel.  This can lead to 
> synchronization problems between Cassandra, services that depend on it, and 
> services it depends on.  This patch adds some simple support for shutdown 
> hooks to StorageService.
> This should nearly solve CASSANDRA-12011



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12397) Altering a column's type breaks commitlog replay

[Stefania (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15532302#comment-15532302
 ] 

Stefania commented on CASSANDRA-12397:
--

Thanks, I'll put this on hold for now then. 

> Altering a column's type breaks commitlog replay
> 
>
> Key: CASSANDRA-12397
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12397
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Carl Yeksigian
>Assignee: Stefania
>
> When switching from a fixed-length column to a variable-length column, 
> replaying the commitlog on restart will have the same issue as 
> CASSANDRA-11820. Seems like it is related to the schema being flushed and 
> used when restarted, but commitlogs having been written in the old format.
> {noformat}
> org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException:
>  Unexpected error deserializing mutation; saved to 
> /tmp/mutation4816372620457789996dat.  This may be caused by replaying a 
> mutation against a table with the same name but incompatible schema.  
> Exception follows: java.io.IOError: java.io.EOFException: EOF after 259 bytes 
> out of 3336
> at 
> org.apache.cassandra.db.commitlog.CommitLogReader.readMutation(CommitLogReader.java:409)
>  [main/:na]
> at 
> org.apache.cassandra.db.commitlog.CommitLogReader.readSection(CommitLogReader.java:342)
>  [main/:na]
> at 
> org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:201)
>  [main/:na]
> at 
> org.apache.cassandra.db.commitlog.CommitLogReader.readAllFiles(CommitLogReader.java:84)
>  [main/:na]
> at 
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:139)
>  [main/:na]
> at 
> org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:177) 
> [main/:na]
> at 
> org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:158)
>  [main/:na]
> at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:316) 
> [main/:na]
> at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:591)
>  [main/:na]
> at 
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:720) 
> [main/:na]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12728) Handling partially written hint files

[Sharvanath Pathak (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15532285#comment-15532285
 ] 

Sharvanath Pathak commented on CASSANDRA-12728:
---

yeah

> Handling partially written hint files
> -
>
> Key: CASSANDRA-12728
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12728
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Sharvanath Pathak
>Assignee: Aleksey Yeschenko
>
> {noformat}
> ERROR [HintsDispatcher:1] 2016-09-28 17:44:43,397 
> HintsDispatchExecutor.java:225 - Failed to dispatch hints file 
> d5d7257c-9f81-49b2-8633-6f9bda6e3dea-1474892654160-1.hints: file is corrupted 
> ({})
> org.apache.cassandra.io.FSReadError: java.io.EOFException
> at 
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:282)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:252)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
> ~[apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.hints.HintsDispatcher.sendHints(HintsDispatcher.java:156)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:137)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:119) 
> ~[apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:91) 
> ~[apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.deliver(HintsDispatchExecutor.java:259)
>  [apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:242)
>  [apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:220)
>  [apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:199)
>  [apache-cassandra-3.0.6.jar:3.0.6]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_77]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [na:1.8.0_77]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_77]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_77]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
> Caused by: java.io.EOFException: null
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:68)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.hints.ChecksummedDataInput.readFully(ChecksummedDataInput.java:126)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:402) 
> ~[apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.hints.HintsReader$BuffersIterator.readBuffer(HintsReader.java:310)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNextInternal(HintsReader.java:301)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:278)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
> ... 15 common frames omitted
> {noformat}
> We've found out that the hint file was truncated because there was a hard 
> reboot around the time of last write to the file. I think we basically need 
> to handle partially written hint files. Also, the CRC file does not exist in 
> this case (probably because it crashed while writing the hints file). May be 
> ignoring and cleaning up such partially written hint files can be a way to 
> fix this?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12728) Handling partially written hint files

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15532281#comment-15532281
 ] 

Aleksey Yeschenko commented on CASSANDRA-12728:
---

This is without compression enabled?

> Handling partially written hint files
> -
>
> Key: CASSANDRA-12728
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12728
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Sharvanath Pathak
>
> {noformat}
> ERROR [HintsDispatcher:1] 2016-09-28 17:44:43,397 
> HintsDispatchExecutor.java:225 - Failed to dispatch hints file 
> d5d7257c-9f81-49b2-8633-6f9bda6e3dea-1474892654160-1.hints: file is corrupted 
> ({})
> org.apache.cassandra.io.FSReadError: java.io.EOFException
> at 
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:282)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:252)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
> ~[apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.hints.HintsDispatcher.sendHints(HintsDispatcher.java:156)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:137)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:119) 
> ~[apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:91) 
> ~[apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.deliver(HintsDispatchExecutor.java:259)
>  [apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:242)
>  [apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:220)
>  [apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:199)
>  [apache-cassandra-3.0.6.jar:3.0.6]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_77]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [na:1.8.0_77]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_77]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_77]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
> Caused by: java.io.EOFException: null
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:68)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.hints.ChecksummedDataInput.readFully(ChecksummedDataInput.java:126)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:402) 
> ~[apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.hints.HintsReader$BuffersIterator.readBuffer(HintsReader.java:310)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNextInternal(HintsReader.java:301)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
> at 
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:278)
>  ~[apache-cassandra-3.0.6.jar:3.0.6]
> ... 15 common frames omitted
> {noformat}
> We've found out that the hint file was truncated because there was a hard 
> reboot around the time of last write to the file. I think we basically need 
> to handle partially written hint files. Also, the CRC file does not exist in 
> this case (probably because it crashed while writing the hints file). May be 
> ignoring and cleaning up such partially written hint files can be a way to 
> fix this?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)