[jira] [Created] (CASSANDRA-11382) Support modifications to schema tables in minor releases

[Sam Tunnicliffe (JIRA)]

Sam Tunnicliffe created CASSANDRA-11382:
---

 Summary: Support modifications to schema tables in minor releases
 Key: CASSANDRA-11382
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11382
 Project: Cassandra
  Issue Type: Improvement
  Components: Distributed Metadata
Reporter: Sam Tunnicliffe
 Fix For: 4.x


At the moment, any changes to the {{system_schema}} tables requires a major 
version bump to avoid schema disagreement problems and related failures during 
schema migrations. 

Recent examples where this has been problematic include CASSANDRA-11050 and 
CASSANDRA-8094, both of which require temporary {{TODO - remove this in 4.0}} 
type workarounds.

Diminishing the significance of major releases changes is a long term goal, so 
coming up with a general solution for this would be useful.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11195) static_columns_paging_test upgrade dtest flapping

[Russ Hatch (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15200341#comment-15200341
 ] 

Russ Hatch commented on CASSANDRA-11195:


this is starting to look more like a paging bug (driver or c*) to me. If I 
attach a python debugger to the test, I can continue to query with page size 2, 
and only get partial results. if I open up a cqlsh session I can see all the 
rows in fact exist.

> static_columns_paging_test upgrade dtest flapping
> -
>
> Key: CASSANDRA-11195
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11195
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Jim Witschey
>Assignee: Russ Hatch
>  Labels: dtest
> Attachments: node1.log, node1_debug.log, node2.log, node2_debug.log
>
>
> On some upgrade paths, {{static_columns_paging_test}} is flapping:
> http://cassci.datastax.com/job/upgrade_tests-all/9/testReport/upgrade_tests.paging_test/TestPagingDataNodes2RF1_2_2_HEAD_UpTo_Trunk/static_columns_paging_test/history/
> http://cassci.datastax.com/job/upgrade_tests-all/8/testReport/upgrade_tests.paging_test/TestPagingDataNodes2RF1_2_2_UpTo_Trunk/static_columns_paging_test/history/
> http://cassci.datastax.com/job/upgrade_tests-all/8/testReport/upgrade_tests.paging_test/TestPagingDataNodes2RF1_2_2_UpTo_3_3_HEAD/static_columns_paging_test/history
> The failures indicate there is missing data. I have not reproduced the 
> failure locally. I've only seen the failures on 2-node clusters with RF=1, 
> not on the 3-node runs with RF=3.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] cassandra git commit: Merge branch cassandra-3.0 into cassandra-3.5

[blerer]

Merge branch cassandra-3.0 into cassandra-3.5


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4651ac73
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4651ac73
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4651ac73

Branch: refs/heads/cassandra-3.5
Commit: 4651ac7352357ec61bc4940d7f8e9e976fa6a1c1
Parents: e36d233 9995521
Author: Benjamin Lerer 
Authored: Wed Mar 16 18:17:41 2016 +0100
Committer: Benjamin Lerer 
Committed: Wed Mar 16 18:18:26 2016 +0100

--

--

[jira] [Updated] (CASSANDRA-9259) Bulk Reading from Cassandra

[Stefania (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefania updated CASSANDRA-9259:

Attachment: bulk-read-benchmark.1.html
bulk-read-jfr-profiles.1.tar.gz
bulk-read-jfr-profiles.2.tar.gz

> Bulk Reading from Cassandra
> ---
>
> Key: CASSANDRA-9259
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9259
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Compaction, CQL, Local Write-Read Paths, Streaming and 
> Messaging, Testing
>Reporter:  Brian Hess
>Assignee: Stefania
>Priority: Critical
> Fix For: 3.x
>
> Attachments: bulk-read-benchmark.1.html, 
> bulk-read-jfr-profiles.1.tar.gz, bulk-read-jfr-profiles.2.tar.gz
>
>
> This ticket is following on from the 2015 NGCC.  This ticket is designed to 
> be a place for discussing and designing an approach to bulk reading.
> The goal is to have a bulk reading path for Cassandra.  That is, a path 
> optimized to grab a large portion of the data for a table (potentially all of 
> it).  This is a core element in the Spark integration with Cassandra, and the 
> speed at which Cassandra can deliver bulk data to Spark is limiting the 
> performance of Spark-plus-Cassandra operations.  This is especially of 
> importance as Cassandra will (likely) leverage Spark for internal operations 
> (for example CASSANDRA-8234).
> The core CQL to consider is the following:
> SELECT a, b, c FROM myKs.myTable WHERE Token(partitionKey) > X AND 
> Token(partitionKey) <= Y
> Here, we choose X and Y to be contained within one token range (perhaps 
> considering the primary range of a node without vnodes, for example).  This 
> query pushes 50K-100K rows/sec, which is not very fast if we are doing bulk 
> operations via Spark (or other processing frameworks - ETL, etc).  There are 
> a few causes (e.g., inefficient paging).
> There are a few approaches that could be considered.  First, we consider a 
> new "Streaming Compaction" approach.  The key observation here is that a bulk 
> read from Cassandra is a lot like a major compaction, though instead of 
> outputting a new SSTable we would output CQL rows to a stream/socket/etc.  
> This would be similar to a CompactionTask, but would strip out some 
> unnecessary things in there (e.g., some of the indexing, etc). Predicates and 
> projections could also be encapsulated in this new "StreamingCompactionTask", 
> for example.
> Another approach would be an alternate storage format.  For example, we might 
> employ Parquet (just as an example) to store the same data as in the primary 
> Cassandra storage (aka SSTables).  This is akin to Global Indexes (an 
> alternate storage of the same data optimized for a particular query).  Then, 
> Cassandra can choose to leverage this alternate storage for particular CQL 
> queries (e.g., range scans).
> These are just 2 suggestions to get the conversation going.
> One thing to note is that it will be useful to have this storage segregated 
> by token range so that when you extract via these mechanisms you do not get 
> replications-factor numbers of copies of the data.  That will certainly be an 
> issue for some Spark operations (e.g., counting).  Thus, we will want 
> per-token-range storage (even for single disks), so this will likely leverage 
> CASSANDRA-6696 (though, we'll want to also consider the single disk case).
> It is also worth discussing what the success criteria is here.  It is 
> unlikely to be as fast as EDW or HDFS performance (though, that is still a 
> good goal), but being within some percentage of that performance should be 
> set as success.  For example, 2x as long as doing bulk operations on HDFS 
> with similar node count/size/etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (CASSANDRA-11096) Upgrade netty to >= 4.0.34

[Benjamin Lerer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Lerer reassigned CASSANDRA-11096:
--

Assignee: Benjamin Lerer

> Upgrade netty to >= 4.0.34
> --
>
> Key: CASSANDRA-11096
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11096
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CQL
>Reporter: Brandon Williams
>Assignee: Benjamin Lerer
> Fix For: 3.x
>
>
> Amongst other things, the native protocol will not bind ipv6 easily (see 
> CASSANDRA-11047) until we upgrade.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11379) Fix nodetool join does not transition from write survey to normal mode

[Yuki Morishita (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15200714#comment-15200714
 ] 

Yuki Morishita commented on CASSANDRA-11379:


Isn't it sufficient to add {{ && !isSurveyMode}} to current {{isJoined()}} 
method as before? (Though adding {{isSurveyMode()}} may be useful, but I 
consider it as a new feature).

Also, adding survey mode -> join dtest would be nice, it seems we don't have 
one right now.


> Fix nodetool join does not transition from write survey to normal mode
> --
>
> Key: CASSANDRA-11379
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11379
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Paulo Motta
>Assignee: Paulo Motta
>Priority: Minor
> Fix For: 2.1.x, 2.2.x, 3.0.x, 3.x
>
>
> CASSANDRA-11007 broke CASSANDRA-9740, so this patch restores the ability of 
> leaving write survey mode with nodetool join.
> Currently if you try to use nodetool join when on write survey mode  you get 
> a {{This node has already joined the ring}} message.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-11338) dtest failure in consistency_test.TestConsistency.quorum_available_during_failure_test

[Philip Thompson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Philip Thompson resolved CASSANDRA-11338.
-
Resolution: Fixed

> dtest failure in 
> consistency_test.TestConsistency.quorum_available_during_failure_test
> --
>
> Key: CASSANDRA-11338
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11338
> Project: Cassandra
>  Issue Type: Test
>Reporter: Jim Witschey
>Assignee: Philip Thompson
>  Labels: dtest
>
> example failure:
> http://cassci.datastax.com/job/trunk_offheap_dtest/68/testReport/consistency_test/TestConsistency/quorum_available_during_failure_test
> Failed on CassCI build trunk_offheap_dtest #68
> This seems to be failing after merging this CCM PR:
> https://github.com/pcmanus/ccm/pull/461
> I'm not sure why it would fail with that error-checking code but not without 
> it. On this test run, it ran after these two tests:
> {code}
> test_simple_strategy_each_quorum (consistency_test.TestAvailability) ... ok
> test_simple_strategy_each_quorum_counters (consistency_test.TestAccuracy) ... 
> ok
> {code}
> so, maybe processes are hanging around after one of those tests. 
> [~philipthompson] can you have a first look here?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-5200) Add a way to cancel TTL on a column (without changing value)

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-5200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko resolved CASSANDRA-5200.
--
Resolution: Won't Fix

The raciness you mention is inherent to this approach. If you move the read 
before write to the server, C* being a distributed system, you just move the 
race to C* itself. This cannot - easily - be done without imposing 
coordination/locking, which is an anti-pattern in C*.

Closing as Won't Fix.

> Add a way to cancel TTL on a column (without changing value)
> 
>
> Key: CASSANDRA-5200
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5200
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Anentropic
>Priority: Minor
>  Labels: ponies
>
> Currently you can set TTL on a column value:
> UPDATE table USING TTL 30 SET col1='test' WHERE id=123;
> The only way to cancel the TTL, so that col1='test' doesn't expire, is to set 
> col1 to a new value.
> If you want col1 to keep its current value but stop expiring I don't see a 
> reliable way. You (understandably) can't do:
> UPDATE table SET col1=col1 WHERE id=123;
> And obviously if you have to SELECT the value first you have a race condition.
> I understand the reasons for not allowing SET col1=col1 but I'm guessing it 
> wouldn't be too hard to allow some way to cancel the TTL without having to 
> set a value?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9830) Option to disable bloom filter in highest level of LCS sstables

[Marcus Eriksson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15197557#comment-15197557
 ] 

Marcus Eriksson commented on CASSANDRA-9830:


Haven't read the patch, but after incremental repair sstables move from 
unrepaired leveling to repaired leveling *if possible* - this means that a top 
level sstable in the unrepaired leveling might get dropped to level 0 in the 
repaired leveling without recompaction (we only mutate the repaired at field in 
sstable metadata)

> Option to disable bloom filter in highest level of LCS sstables
> ---
>
> Key: CASSANDRA-9830
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9830
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Compaction
>Reporter: Jonathan Ellis
>Assignee: Paulo Motta
>Priority: Minor
>  Labels: performance
> Fix For: 3.x
>
>
> We expect about 90% of data to be in the highest level of LCS in a fully 
> populated series.  (See also CASSANDRA-9829.)
> Thus if the user is primarily asking for data (partitions) that has actually 
> been inserted, the bloom filter on the highest level only helps reject 
> sstables about 10% of the time.
> We should add an option that suppresses bloom filter creation on top-level 
> sstables.  This will dramatically reduce memory usage for LCS and may even 
> improve performance as we no longer check a low-value filter.
> (This is also an idea from RocksDB.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

cassandra git commit: Add backpressure to compressed or encrypted commit log

[blerer]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.5 4651ac735 -> ee40e3b45


Add backpressure to compressed or encrypted commit log

patch by Ariel Weisberg; reviewed by Benjamin Lerer for CASSANDRA-10971


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ee40e3b4
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ee40e3b4
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ee40e3b4

Branch: refs/heads/cassandra-3.5
Commit: ee40e3b4529aa77d4d83fc3e7073902402cb3753
Parents: 4651ac7
Author: Ariel Weisberg 
Authored: Wed Mar 16 18:20:29 2016 +0100
Committer: Benjamin Lerer 
Committed: Wed Mar 16 18:20:29 2016 +0100

--
 CHANGES.txt |   1 +
 .../db/commitlog/AbstractCommitLogService.java  |   1 +
 .../db/commitlog/CommitLogSegment.java  |  23 +++--
 .../db/commitlog/CommitLogSegmentManager.java   |  11 +-
 .../db/commitlog/CompressedSegment.java |   4 +-
 .../db/commitlog/EncryptedSegment.java  |   4 +-
 .../db/commitlog/FileDirectSegment.java |  42 ++--
 .../commitlog/CommitLogSegmentManagerTest.java  | 100 ++
 .../cassandra/db/commitlog/CommitLogTest.java   | 101 ++-
 9 files changed, 220 insertions(+), 67 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/ee40e3b4/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 1ff4e6d..ffd0b24 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,5 +1,6 @@
 3.5
 Merged from 3.0:
+ * Add backpressure to compressed or encrypted commit log (CASSANDRA-10971)
  * SSTableExport supports secondary index tables (CASSANDRA-11330)
  * Fix sstabledump to include missing info in debug output (CASSANDRA-11321)
  * Establish and implement canonical bulk reading workload(s) (CASSANDRA-10331)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ee40e3b4/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java
--
diff --git 
a/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java 
b/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java
index 557bf50..113d1ba 100644
--- a/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java
+++ b/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java
@@ -89,6 +89,7 @@ public abstract class AbstractCommitLogService
 
 // sync and signal
 long syncStarted = System.currentTimeMillis();
+//This is a target for Byteman in 
CommitLogSegmentManagerTest
 commitLog.sync(shutdown);
 lastSyncedAt = syncStarted;
 syncComplete.signalAll();

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ee40e3b4/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java
--
diff --git a/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java 
b/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java
index 5e99a07..f2d8f92 100644
--- a/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java
+++ b/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java
@@ -119,15 +119,26 @@ public abstract class CommitLogSegment
 final CommitLog commitLog;
 public final CommitLogDescriptor descriptor;
 
-static CommitLogSegment createSegment(CommitLog commitLog)
+static CommitLogSegment createSegment(CommitLog commitLog, Runnable 
onClose)
 {
-CommitLogSegment segment = commitLog.encryptionContext.isEnabled() ? 
new EncryptedSegment(commitLog, commitLog.encryptionContext) :
-   commitLog.compressor != null ? new CompressedSegment(commitLog) 
:
-   new MemoryMappedSegment(commitLog);
+CommitLogSegment segment = commitLog.encryptionContext.isEnabled() ? 
new EncryptedSegment(commitLog, commitLog.encryptionContext, onClose) :
+   commitLog.compressor != null ? new CompressedSegment(commitLog, 
onClose) :
+  new 
MemoryMappedSegment(commitLog);
 segment.writeLogHeader();
 return segment;
 }
 
+/**
+ * Checks if the segments use a buffer pool.
+ *
+ * @param commitLog the commit log
+ * @return true if the segments use a buffer pool, 
false otherwise.
+ */
+static boolean usesBufferPool(CommitLog commitLog)
+{
+return commitLog.encryptionContext.isEnabled() || commitLog.compressor 
!= null;
+}
+
 static long getNextId()
 {

[jira] [Commented] (CASSANDRA-11356) EC2MRS ignores broadcast_rpc_address setting in cassandra.yaml

[Paulo Motta (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15197379#comment-15197379
 ] 

Paulo Motta commented on CASSANDRA-11356:
-

I'm not sure we should even be setting {{broadcast_rpc_address}} on EC2MRS. 
While setting {{broadcast_address}} is useful to the snitch's objective of 
making nodes from different dcs connect over the public address for internal 
communication, {{broadcast_rpc_address}} is a deployment option independent of 
EC2MRS choice. I'd be in favor of not setting this at all on EC2MRS and adding 
an upgrade notice for users who've been relying on this behavior.

> EC2MRS ignores broadcast_rpc_address setting in cassandra.yaml
> --
>
> Key: CASSANDRA-11356
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11356
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Thanh
> Fix For: 2.2.x, 3.x
>
>
> EC2MRS ignores broadcast_rpc_address setting in cassandra.yaml.  This is 
> problematic for those users who were using EC2MRS with an internal 
> rpc_address before the change introduced in 
> [CASSANDRA-5899|https://issues.apache.org/jira/browse/CASSANDRA-5899], 
> because the change results in EC2MRS always using the public ip regardless of 
> what the user has set for broadcast_rpc_address.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8958) Add client to cqlsh SHOW_SESSION

[Stefania (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15199382#comment-15199382
 ] 

Stefania commented on CASSANDRA-8958:
-

The patch and dtest results for trunk:

|[patch|https://github.com/stef1927/cassandra/commits/8958]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-8958-dtest/]|


> Add client to cqlsh SHOW_SESSION
> 
>
> Key: CASSANDRA-8958
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8958
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Stefania
>Assignee: Stefania
>Priority: Minor
>  Labels: lhf
> Fix For: 3.x
>
>
> Once the python driver supports it, 
> https://datastax-oss.atlassian.net/browse/PYTHON-235, add the client to cqlsh 
> {{SHOW_SESSION}} as done in this commit:
> https://github.com/apache/cassandra/commit/249f79d3718fa05347d60e09f9d3fa15059bd3d3
> Also, update the bundled python driver.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/4] cassandra git commit: Add backpressure to compressed commit log

[blerer]

Repository: cassandra
Updated Branches:
  refs/heads/trunk 4ecd8542d -> e5394f192


Add backpressure to compressed commit log

patch by Ariel Weisberg; reviewed by Benjamin Lerer for CASSANDRA-10971


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9995521f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9995521f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9995521f

Branch: refs/heads/trunk
Commit: 9995521fb9b3f510ee9c7012d75e6970ec7d5fb7
Parents: 0a5e220
Author: Ariel Weisberg 
Authored: Wed Mar 16 18:14:52 2016 +0100
Committer: Benjamin Lerer 
Committed: Wed Mar 16 18:14:52 2016 +0100

--
 CHANGES.txt |   1 +
 .../db/commitlog/AbstractCommitLogService.java  |   1 +
 .../db/commitlog/CommitLogSegment.java  |  19 +++-
 .../db/commitlog/CommitLogSegmentManager.java   |  11 +-
 .../db/commitlog/CompressedSegment.java |  39 ++--
 .../commitlog/CommitLogSegmentManagerTest.java  | 100 +++
 6 files changed, 159 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/9995521f/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 28de247..b264609 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.5
+ * Add backpressure to compressed commit log (CASSANDRA-10971)
  * SSTableExport supports secondary index tables (CASSANDRA-11330)
  * Fix sstabledump to include missing info in debug output (CASSANDRA-11321)
  * Establish and implement canonical bulk reading workload(s) (CASSANDRA-10331)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/9995521f/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java
--
diff --git 
a/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java 
b/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java
index 557bf50..113d1ba 100644
--- a/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java
+++ b/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogService.java
@@ -89,6 +89,7 @@ public abstract class AbstractCommitLogService
 
 // sync and signal
 long syncStarted = System.currentTimeMillis();
+//This is a target for Byteman in 
CommitLogSegmentManagerTest
 commitLog.sync(shutdown);
 lastSyncedAt = syncStarted;
 syncComplete.signalAll();

http://git-wip-us.apache.org/repos/asf/cassandra/blob/9995521f/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java
--
diff --git a/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java 
b/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java
index 5dd7c9f..0e9f502 100644
--- a/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java
+++ b/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java
@@ -117,9 +117,20 @@ public abstract class CommitLogSegment
 final CommitLog commitLog;
 public final CommitLogDescriptor descriptor;
 
-static CommitLogSegment createSegment(CommitLog commitLog)
+static CommitLogSegment createSegment(CommitLog commitLog, Runnable 
onClose)
 {
-return commitLog.compressor != null ? new CompressedSegment(commitLog) 
: new MemoryMappedSegment(commitLog);
+return commitLog.compressor != null ? new CompressedSegment(commitLog, 
onClose) : new MemoryMappedSegment(commitLog);
+}
+
+/**
+ * Checks if the segments use a buffer pool.
+ *
+ * @param commitLog the commit log
+ * @return true if the segments use a buffer pool, 
false otherwise.
+ */
+static boolean usesBufferPool(CommitLog commitLog)
+{
+return commitLog.compressor != null;
 }
 
 static long getNextId()
@@ -148,7 +159,7 @@ public abstract class CommitLogSegment
 {
 throw new FSWriteError(e, logFile);
 }
-
+
 buffer = createBuffer(commitLog);
 // write the header
 CommitLogDescriptor.writeHeader(buffer, descriptor);
@@ -255,7 +266,7 @@ public abstract class CommitLogSegment
 // Note: Even if the very first allocation of this sync section 
failed, we still want to enter this
 // to ensure the segment is closed. As allocatePosition is set to 1 
beyond the capacity of the buffer,
 // this will always be entered when a mutation allocation has been 
attempted after the marker allocation
-// succeeded in the previous sync. 
+   

[jira] [Updated] (CASSANDRA-11381) Node running with join_ring=false and authentication can not serve requests

[mck (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mck updated CASSANDRA-11381:

Attachment: (was: 11381-2.2.txt)

> Node running with join_ring=false and authentication can not serve requests
> ---
>
> Key: CASSANDRA-11381
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11381
> Project: Cassandra
>  Issue Type: Bug
>Reporter: mck
>Assignee: mck
> Attachments: 11381-2.0.txt, 11381-2.1.txt, 11381-2.2.txt
>
>
> Starting up a node with {{-Dcassandra.join_ring=false}} in a cluster that has 
> authentication configured, eg PasswordAuthenticator, won't be able to serve 
> requests. This is because {{Auth.setup()}} never gets called during the 
> startup.
> Without {{Auth.setup()}} having been called in {{StorageService}} clients 
> connecting to the node fail with the node throwing
> {noformat}
> java.lang.NullPointerException
> at 
> org.apache.cassandra.auth.PasswordAuthenticator.authenticate(PasswordAuthenticator.java:119)
> at 
> org.apache.cassandra.thrift.CassandraServer.login(CassandraServer.java:1471)
> at 
> org.apache.cassandra.thrift.Cassandra$Processor$login.getResult(Cassandra.java:3505)
> at 
> org.apache.cassandra.thrift.Cassandra$Processor$login.getResult(Cassandra.java:3489)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at com.thinkaurelius.thrift.Message.invoke(Message.java:314)
> at 
> com.thinkaurelius.thrift.Message$Invocation.execute(Message.java:90)
> at 
> com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:695)
> at 
> com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:689)
> at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:112)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> The exception thrown from the 
> [code|https://github.com/apache/cassandra/blob/cassandra-2.0.16/src/java/org/apache/cassandra/auth/PasswordAuthenticator.java#L119]
> {code}
> ResultMessage.Rows rows = 
> authenticateStatement.execute(QueryState.forInternalCalls(), new 
> QueryOptions(consistencyForUser(username),
>   
>Lists.newArrayList(ByteBufferUtil.bytes(username;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11290) dtest failure in materialized_views_test.TestMaterializedViewsConsistency.single_partition_consistent_reads_after_write_test

[Philip Thompson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Philip Thompson updated CASSANDRA-11290:

Fix Version/s: 3.x

> dtest failure in 
> materialized_views_test.TestMaterializedViewsConsistency.single_partition_consistent_reads_after_write_test
> 
>
> Key: CASSANDRA-11290
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11290
> Project: Cassandra
>  Issue Type: Test
>Reporter: Russ Hatch
>Assignee: Philip Thompson
>  Labels: dtest
> Fix For: 3.x
>
>
> example failure:
> http://cassci.datastax.com/job/trunk_offheap_dtest/26/testReport/materialized_views_test/TestMaterializedViewsConsistency/single_partition_consistent_reads_after_write_test
> Failed on CassCI build trunk_offheap_dtest #26
> Failing infrequently but same error on at least two of those infrequent flaps:
> {noformat}
> Connection to 127.0.0.2 was closed
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/4] cassandra git commit: Fix UTF8Validator.validate() for modified UTF-8

[blerer]

Repository: cassandra
Updated Branches:
  refs/heads/trunk 6a8c747c6 -> 645829214


Fix UTF8Validator.validate() for modified UTF-8

patch by Benjamin Lerer; reviewed by Sylvain Lebresne for CASSANDRA-10748


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/42cd66a1
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/42cd66a1
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/42cd66a1

Branch: refs/heads/trunk
Commit: 42cd66a1f91dcd0cf9238e7915d5efe5779b9933
Parents: 5e2d3d2
Author: Benjamin Lerer 
Authored: Thu Mar 17 14:27:27 2016 +0100
Committer: Benjamin Lerer 
Committed: Thu Mar 17 14:27:27 2016 +0100

--
 CHANGES.txt   | 1 +
 src/java/org/apache/cassandra/serializers/UTF8Serializer.java | 5 +++--
 2 files changed, 4 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/42cd66a1/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index fdb5807..fa4a726 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.6
+ * Fix UTF8Validator.validate() for modified UTF-8 (CASSANDRA-10748)
  * Clarify that now() function is calculated on the coordinator node in CQL 
documentation (CASSANDRA-10900)
  * Fix bloom filter sizing with LCS (CASSANDRA-11344)
  * (cqlsh) Fix error when result is 0 rows with EXPAND ON (CASSANDRA-11092)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/42cd66a1/src/java/org/apache/cassandra/serializers/UTF8Serializer.java
--
diff --git a/src/java/org/apache/cassandra/serializers/UTF8Serializer.java 
b/src/java/org/apache/cassandra/serializers/UTF8Serializer.java
index be5d2d6..e3ea2d5 100644
--- a/src/java/org/apache/cassandra/serializers/UTF8Serializer.java
+++ b/src/java/org/apache/cassandra/serializers/UTF8Serializer.java
@@ -71,11 +71,12 @@ public class UTF8Serializer extends AbstractTextSerializer
 {
 // validate first byte of 2-byte char, 0xc2-0xdf
 if (b == (byte) 0xc0)
-// speical case: modified utf8 null is 0xc080.
+// special case: modified utf8 null is 0xc080.
 state = State.TWO_80;
 else if ((b & 0x1e) == 0)
 return false;
-state = State.TWO;
+else
+state = State.TWO;
 }
 else if ((b >> 4) == -2)
 {

[jira] [Updated] (CASSANDRA-10809) Create a -D option to prevent gossip startup

[Sylvain Lebresne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-10809:
-
Attachment: 10809.txt

I believe restoring the previous behavior is as simple as aborting 
{{initServer}} before {{prepareToJoin}}, which the attached patch does. The 
patch doesn't add a way to start gossip after the fact, but I assume that if 
you're debugging a schema problem or something like that, it's ok to restart 
(without the flag) once you've fixed your problem.

> Create a -D option to prevent gossip startup
> 
>
> Key: CASSANDRA-10809
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10809
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Distributed Metadata
>Reporter: Brandon Williams
> Fix For: 2.1.x
>
> Attachments: 10809.txt
>
>
> In CASSANDRA-6961 we changed how join_ring=false works, to great benefit.  
> However, sometimes you need to a node to come up, but not interact with other 
> nodes whatsoever - for example if you have a schema problem, it will still 
> pull the schema from another node because they still gossip even though we're 
> in a dead state.
> We can add a way to restore the previous behavior by simply adding something 
> like -Dcassandra.start_gossip=false.
> In the meantime we can workaround by setting listen_address to localhost, but 
> that's kind of a pain.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-4839) Online toggle for node write-only status

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-4839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko resolved CASSANDRA-4839.
--
Resolution: Won't Fix

> Online toggle for node write-only status
> 
>
> Key: CASSANDRA-4839
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4839
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Rick Branson
>Priority: Minor
> Attachments: 4839.txt, 4839_revised.txt
>
>
> It would be really great if users could disable/enable reads on a given node, 
> while still allowing write operations to take place. This would be similar to 
> how we enable/disable thrift and gossip using JMX.
> The scenario for using this is that often a node needs to be brought down for 
> maintenance for a few minutes, and while the node is catching up from hints, 
> which can take 10-30 minutes depending on write load, it will serve stale 
> data. Do the math for a rolling restart of a large cluster and you have 
> potential windows of hours or days where a large amount of inconsistency is 
> surfacing.
> Avoiding this large time gap of inconsistency during regular maintenance 
> alleviates concerns about inconsistent data surfaced to users during normal, 
> planned activities. While a read consistency >ONE can indeed be used to 
> prevent any inconsistency from the scenario above, it seems ridiculous to 
> always incur the cost to cover the 0.1% case.
> In addition, it would open up the ability for a node to (optionally) 
> automatically "go dark" for reads while it's receiving hints after joining 
> the cluster or perhaps during repair. These obviously have their own 
> complications and justify separate tickets.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[4/6] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[slebresne]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/db10cb25
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/db10cb25
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/db10cb25

Branch: refs/heads/cassandra-3.5
Commit: db10cb25c036175ac51474ea1e1169524d0e98a6
Parents: 1dbfa4b b06bcf7
Author: Sylvain Lebresne 
Authored: Fri Mar 18 10:54:16 2016 +0100
Committer: Sylvain Lebresne 
Committed: Fri Mar 18 10:54:16 2016 +0100

--

--

[jira] [Commented] (CASSANDRA-11296) Run dtests with -Dcassandra.debugrefcount=true and increase checking frequency

[Marcus Eriksson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15199869#comment-15199869
 ] 

Marcus Eriksson commented on CASSANDRA-11296:
-

In SSTableTidier we keep a reference to parent which keeps a reference to 
Tracker to be able to notify subscribers of any changes

> Run dtests with -Dcassandra.debugrefcount=true and increase checking frequency
> --
>
> Key: CASSANDRA-11296
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11296
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>
> We should run dtests with refcount debugging and check every second instead 
> of every 15 minutes 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[5/9] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[slebresne]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1dbfa4b5
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1dbfa4b5
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1dbfa4b5

Branch: refs/heads/cassandra-3.5
Commit: 1dbfa4b5430726ff0f5598e845498acd9083dea0
Parents: e8651b6 01d8636
Author: Sylvain Lebresne 
Authored: Fri Mar 18 10:52:00 2016 +0100
Committer: Sylvain Lebresne 
Committed: Fri Mar 18 10:52:00 2016 +0100

--
 CHANGES.txt   | 1 +
 src/java/org/apache/cassandra/service/StorageService.java | 7 ++-
 2 files changed, 7 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/1dbfa4b5/CHANGES.txt
--
diff --cc CHANGES.txt
index 51cfc16,ee58fed..a077551
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,15 -1,5 +1,16 @@@
 -2.2.6
 +3.0.5
 + * Support streaming pre-3.0 sstables (CASSANDRA-10990)
 + * Add backpressure to compressed commit log (CASSANDRA-10971)
 + * SSTableExport supports secondary index tables (CASSANDRA-11330)
 + * Fix sstabledump to include missing info in debug output (CASSANDRA-11321)
 + * Establish and implement canonical bulk reading workload(s) 
(CASSANDRA-10331)
 + * Fix paging for IN queries on tables without clustering columns 
(CASSANDRA-11208)
 + * Remove recursive call from CompositesSearcher (CASSANDRA-11304)
 + * Fix filtering on non-primary key columns for queries without index 
(CASSANDRA-6377)
 + * Fix sstableloader fail when using materialized view (CASSANDRA-11275)
 +Merged from 2.2:
+  * Add -Dcassandra.start_gossip startup option (CASSANDRA-10809)
 + * Fix UTF8Validator.validate() for modified UTF-8 (CASSANDRA-10748)
   * Clarify that now() function is calculated on the coordinator node in CQL 
documentation (CASSANDRA-10900)
   * Fix bloom filter sizing with LCS (CASSANDRA-11344)
   * (cqlsh) Fix error when result is 0 rows with EXPAND ON (CASSANDRA-11092)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/1dbfa4b5/src/java/org/apache/cassandra/service/StorageService.java
--

[jira] [Updated] (CASSANDRA-11337) Add --hex-format option to nodetool getsstables

[Yuki Morishita (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuki Morishita updated CASSANDRA-11337:
---
   Resolution: Fixed
Fix Version/s: 3.6
   Status: Resolved  (was: Patch Available)

Committed only to trunk for improvements.
Commit sha is e24da8f7faea0e764ab4653343085adf6edcd75f.

Thanks!

> Add --hex-format option to nodetool getsstables
> ---
>
> Key: CASSANDRA-11337
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11337
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Paulo Motta
>Assignee: Paulo Motta
>Priority: Minor
>  Labels: lhf
> Fix For: 3.6
>
>
> Sometimes it's useful to retrieve an sstable from the hex string 
> representation of its key, for instance, when you get an exception like this 
> and you want to find out which sstable owns the faulty key:
> {noformat}
> java.lang.AssertionError: row DecoratedKey(2769066505137675224, 
> 0004002e080153441a3ef000) received out of order wrt 
> DecoratedKey(2774747040849866654, 0004019b08015348847eb200)
> {noformat}
> In this case,  {noformat}nodetool getsstables ks cf 
> 0004002e080153441a3ef000{noformat} will only work if {{ks.cf}} 
> has a blob primary key.
> It's straightforward to retrieve a {{DecoratedKey}} from the hexstr 
> representation of the key, so we should add a {{--hex-key}} option to allow 
> for that.
> {noformat}nodetool getsstables ks cf --hex-key 
> 0004002e080153441a3ef000{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-5645) Display PK values along the header when using EXPAND in cqlsh

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-5645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko resolved CASSANDRA-5645.
--
Resolution: Not A Problem
  Assignee: (was: Michał Michalski)

We opted for highlighting PK columns in different colours instead.

> Display PK values along the header when using EXPAND in cqlsh
> -
>
> Key: CASSANDRA-5645
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5645
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Michał Michalski
>Priority: Minor
>  Labels: lhf
>
> Follow-up to CASSANDRA-5597 proposed by [~jjordan].
> Currently cqlsh run in vertical mode prints a header like this:
> {noformat}cqlsh> EXPAND on;
> Now printing expanded output
> cqlsh> SELECT * FROM system.schema_columnfamilies limit 1;
> @ Row 1
> -+-
>  keyspace_name   | system_auth
>  columnfamily_name   | users
>  bloom_filter_fp_chance  | 0.01
>  caching | KEYS_ONLY
>  column_aliases  | []
> (...){noformat}
> The idea is to make it print header this way:
> {noformat}cqlsh> EXPAND on;
> Now printing expanded output
> cqlsh> SELECT * FROM system.schema_columnfamilies limit 1;
> @ Row 1: system_auth, users
> -+-
>  keyspace_name   | system_auth
>  columnfamily_name   | users
>  bloom_filter_fp_chance  | 0.01
>  caching | KEYS_ONLY
>  column_aliases  | []
> (...){noformat}
> [~jjordan], please verify if it's what you requested for.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11363) Blocked NTR When Connecting Causing Excessive Load

[Russell Bradberry (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15199929#comment-15199929
 ] 

Russell Bradberry commented on CASSANDRA-11363:
---

For troubleshooting we set up a coordinator-only node and pointed one app 
server at it.  This resulted in roughly 90 connections to the node.  We 
witnessed many timeouts of requests from the app server's perspective. We 
downgraded the coordinator-only node to 2.1.9 and upgraded point-release by 
point-release (DSE point releases) until we saw the same behavior in DSE 4.8.4 
(Cassandra 2.1.12).

I'm not certain this has to do with connection count anymore. 

We have several different types of workloads going on, but we found that only 
the workloads that use batches were timing out. Additionally this is only 
happening when the node is used as a coordinator.  We are not seeing this issue 
when we disable the binary protocol, effectively making the node no-longer a 
coordinator.

> Blocked NTR When Connecting Causing Excessive Load
> --
>
> Key: CASSANDRA-11363
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11363
> Project: Cassandra
>  Issue Type: Bug
>  Components: Coordination
>Reporter: Russell Bradberry
> Attachments: cassandra-102-cms.stack, cassandra-102-g1gc.stack
>
>
> When upgrading from 2.1.9 to 2.1.13, we are witnessing an issue where the 
> machine load increases to very high levels (> 120 on an 8 core machine) and 
> native transport requests get blocked in tpstats.
> I was able to reproduce this in both CMS and G1GC as well as on JVM 7 and 8.
> The issue does not seem to affect the nodes running 2.1.9.
> The issue seems to coincide with the number of connections OR the number of 
> total requests being processed at a given time (as the latter increases with 
> the former in our system)
> Currently there is between 600 and 800 client connections on each machine and 
> each machine is handling roughly 2000-3000 client requests per second.
> Disabling the binary protocol fixes the issue for this node but isn't a 
> viable option cluster-wide.
> Here is the output from tpstats:
> {code}
> Pool NameActive   Pending  Completed   Blocked  All 
> time blocked
> MutationStage 0 88387821 0
>  0
> ReadStage 0 0 355860 0
>  0
> RequestResponseStage  0 72532457 0
>  0
> ReadRepairStage   0 0150 0
>  0
> CounterMutationStage 32   104 897560 0
>  0
> MiscStage 0 0  0 0
>  0
> HintedHandoff 0 0 65 0
>  0
> GossipStage   0 0   2338 0
>  0
> CacheCleanupExecutor  0 0  0 0
>  0
> InternalResponseStage 0 0  0 0
>  0
> CommitLogArchiver 0 0  0 0
>  0
> CompactionExecutor2   190474 0
>  0
> ValidationExecutor0 0  0 0
>  0
> MigrationStage0 0 10 0
>  0
> AntiEntropyStage  0 0  0 0
>  0
> PendingRangeCalculator0 0310 0
>  0
> Sampler   0 0  0 0
>  0
> MemtableFlushWriter   110 94 0
>  0
> MemtablePostFlush 134257 0
>  0
> MemtableReclaimMemory 0 0 94 0
>  0
> Native-Transport-Requests   128   156 38795716
> 278451
> Message type   Dropped
> READ 0
> RANGE_SLICE  0
> _TRACE   0
> MUTATION 0
> COUNTER_MUTATION 0
> BINARY   0
> REQUEST_RESPONSE 0
> PAGED_RANGE  0
> READ_REPAIR  0
> {code}
> Attached is the jstack output for both CMS and G1GC.
> Flight recordings are here:
> https://s3.amazonaws.com/simple-logs/cassandra-102-cms.jfr
> https://s3.amazonaws.com/simple-logs/cassandra-102-g1gc.jfr
> It is interesting to note that while the flight 

[jira] [Updated] (CASSANDRA-11203) Improve 'nothing to repair' message when RF=1

[Marcus Eriksson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-11203:

Reviewer: Marcus Eriksson

> Improve 'nothing to repair' message when RF=1
> -
>
> Key: CASSANDRA-11203
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11203
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
> Environment: debian jesse up to date content
>Reporter: Jason Kania
>Assignee: Branimir Lambov
>Priority: Trivial
>  Labels: lhf
>
> When nodetool repair is run, it indicates that no repair is needed on some 
> keyspaces but on others it attempts repair. However, when run multiple times, 
> the output seems to indicate that the same triggering conditions still 
> persists that indicate a problem. Alternatively, the output could indicate 
> that the underlying condition has not been resolved.
> root@marble:/var/lib/cassandra/data/sensordb/periodicReading# nodetool repair
> [2016-02-21 23:33:10,356] Nothing to repair for keyspace 'sensordb'
> [2016-02-21 23:33:10,364] Nothing to repair for keyspace 'system_auth'
> [2016-02-21 23:33:10,402] Starting repair command #1, repairing keyspace 
> system_traces with repair options (parallelism: parallel, primary range: 
> false, incremental: true, job threads: 1, ColumnFamilies: [], dataCenters: 
> [], hosts: [], # of ranges: 256)
> [2016-02-21 23:33:12,144] Repair completed successfully
> [2016-02-21 23:33:12,157] Repair command #1 finished in 1 second
> root@marble:/var/lib/cassandra/data/sensordb/periodicReading# nodetool repair
> [2016-02-21 23:33:31,683] Nothing to repair for keyspace 'sensordb'
> [2016-02-21 23:33:31,689] Nothing to repair for keyspace 'system_auth'
> [2016-02-21 23:33:31,713] Starting repair command #2, repairing keyspace 
> system_traces with repair options (parallelism: parallel, primary range: 
> false, incremental: true, job threads: 1, ColumnFamilies: [], dataCenters: 
> [], hosts: [], # of ranges: 256)
> [2016-02-21 23:33:33,324] Repair completed successfully
> [2016-02-21 23:33:33,334] Repair command #2 finished in 1 second



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-7159) sstablemetadata command should print some more stuff

[Marcus Eriksson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-7159:
---
Status: Patch Available  (was: Open)

> sstablemetadata command should print some more stuff
> 
>
> Key: CASSANDRA-7159
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7159
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Jeremiah Jordan
>Assignee: Marcus Eriksson
>Priority: Trivial
>  Labels: lhf
> Fix For: 3.x
>
> Attachments: 
> CASSANDRA-7159_-_sstablemetadata_command_should_print_some_more_stuff.patch
>
>
> It would be nice if the sstablemetadata command printed out some more of the 
> stuff we track.  Like the Min/Max column names and the min/max token in the 
> file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11370) Display sstable count per level according to repair status on nodetool tablestats

[Marcus Eriksson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-11370:

Assignee: Paulo Motta

> Display sstable count per level according to repair status on nodetool 
> tablestats 
> --
>
> Key: CASSANDRA-11370
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11370
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Paulo Motta
>Assignee: Paulo Motta
>Priority: Minor
>  Labels: lhf
>
> After CASSANDRA-8004 we still display sstables in each level on nodetool 
> tablestats as if we had a single compaction strategy, while we have one 
> strategy for repaired and another for unrepaired data. 
> We should split display into repaired and unrepaired set, so this:
> SSTables in each level: [2, 20/10, 15, 0, 0, 0, 0, 0, 0]
> Would become:
> SSTables in each level (repaired): [1, 10, 0, 0, 0, 0, 0, 0, 0]
> SSTables in each level (unrepaired): [1, 10, 15, 0, 0, 0, 0, 0, 0]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-5399) Offer pluggable security for inter-node communication

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-5399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko resolved CASSANDRA-5399.
--
Resolution: Duplicate

bq. Out of curiosity, what else would you like to use? It seems that the 
transport would have to be encrypted anyway, if it's the auth part you want 
pluggable.

True. And we have that part pluggable already as CASSANDRA-5401. Closing as 
Duplicate.

> Offer pluggable security for inter-node communication 
> --
>
> Key: CASSANDRA-5399
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5399
> Project: Cassandra
>  Issue Type: Improvement
>Affects Versions: 1.1.10
> Environment: Production
>Reporter: Ahmed Bashir
>  Labels: security
>
> Inter-node communication can be only encrypted using TLS/SSL; it would be 
> good to allow this piece to be pluggable, as is the case with 
> authentication/authorization of Thrift requests and endpoint snitch 
> implementations.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (CASSANDRA-11228) dtest failure in counter_tests.TestCounters.upgrade_test

[Russ Hatch (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Russ Hatch reassigned CASSANDRA-11228:
--

Assignee: Russ Hatch  (was: DS Test Eng)

> dtest failure in counter_tests.TestCounters.upgrade_test
> 
>
> Key: CASSANDRA-11228
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11228
> Project: Cassandra
>  Issue Type: Test
>Reporter: Russ Hatch
>Assignee: Russ Hatch
>  Labels: dtest
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_offheap_dtest/301/testReport/counter_tests/TestCounters/upgrade_test
> Failed on CassCI build cassandra-2.1_offheap_dtest #301
> Looks like perhaps a test timing issue, 2 failures in the past with variants 
> of:
> {noformat}
> [Unavailable exception] message="Cannot achieve consistency level QUORUM"
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-11096) Upgrade netty to >= 4.0.34

[Benjamin Lerer (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15201301#comment-15201301
 ] 

Benjamin Lerer edited comment on CASSANDRA-11096 at 3/18/16 10:46 AM:
--

|[branch|https://github.com/apache/cassandra/compare/trunk...blerer:11096-trunk]|[utests|http://cassci.datastax.com/view/Dev/view/blerer/job/blerer-11096-trunk-testall/lastCompletedBuild/testReport/]|[dtests|http://cassci.datastax.com/view/Dev/view/blerer/job/blerer-11096-trunk-dtest/lastCompletedBuild/testReport/]|

The upgrade does not seems to have introduced any problem.


was (Author: blerer):
|[branch|https://github.com/apache/cassandra/compare/trunk...blerer:11096-trunk]|[utests|http://cassci.datastax.com/view/Dev/view/blerer/job/blerer-11096-trunk-testall/lastCompletedBuild/testReport/]|[utests|http://cassci.datastax.com/view/Dev/view/blerer/job/blerer-11096-trunk-dtest/lastCompletedBuild/testReport/]|

The upgrade does not seems to have introduced any problem.

> Upgrade netty to >= 4.0.34
> --
>
> Key: CASSANDRA-11096
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11096
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CQL
>Reporter: Brandon Williams
>Assignee: Benjamin Lerer
> Fix For: 3.x
>
>
> Amongst other things, the native protocol will not bind ipv6 easily (see 
> CASSANDRA-11047) until we upgrade.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (CASSANDRA-11279) dtest failure in disk_balance_test.TestDiskBalance.disk_balance_bootstrap_test

[Philip Thompson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Philip Thompson reassigned CASSANDRA-11279:
---

Assignee: Philip Thompson  (was: DS Test Eng)

> dtest failure in disk_balance_test.TestDiskBalance.disk_balance_bootstrap_test
> --
>
> Key: CASSANDRA-11279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11279
> Project: Cassandra
>  Issue Type: Test
>Reporter: Russ Hatch
>Assignee: Philip Thompson
>  Labels: dtest
>
> example failure:
> http://cassci.datastax.com/job/trunk_dtest/1011/testReport/disk_balance_test/TestDiskBalance/disk_balance_bootstrap_test
> Failed on CassCI build trunk_dtest #1011
> This looks likely to be a test issue, perhaps we need to relax the assertion 
> here a bit:
> {noformat}
> values not within 20.00% of the max: (474650, 382235, 513385) (node1)
> {noformat}
> This is flaky with several flaps in the last few weeks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-11383) SASI index build leads to massive OOM

[DOAN DuyHai (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202088#comment-15202088
 ] 

DOAN DuyHai edited comment on CASSANDRA-11383 at 3/18/16 8:31 PM:
--

[~jkrupan] 

1. Not that large, see below the Spark script to generate randomized data:

{noformat}
import java.util.UUID
import com.datastax.spark.connector._
case class Resource(dsrId:UUID, relSeq:Long, seq:Long, 
dspReleaseCode:String,
commercialOfferCode:String, transferCode:String, 
mediaCode:String,
modelCode:String, unicWork:String,
title:String, status:String, 
contributorsName:List[String],
periodEndMonthInt:Int, dspCode:String, 
territoryCode:String,
payingNetQty:Long, authorizedSocietiesTxt: String, 
relType:String)

val allDsps = List("youtube", "itunes", "spotify", "deezer", "vevo", 
"google-play", "7digital", "spotify", "youtube", "spotify", "youtube", 
"youtube", "youtube")
val allCountries = List("FR", "UK", "BE", "IT", "NL", "ES", "FR", "FR")
val allPeriodsEndMonths:Seq[Int] = for(year <- 2013 to 2015; month <- 1 to 
12) yield (year.toString + f"$month%02d").toInt
val allModelCodes = List("PayAsYouGo", "AdFunded", "Subscription")
val allMediaCodes = List("Music","Ringtone")
val allTransferCodes = List("Streaming","Download")
val allCommercialOffers = List("Premium","Free")
val status = "Declared"
val authorizedSocietiesTxt: String="sacem sgae"
val relType = "whatever"
val titlesAndContributors: Array[(String, String)] = 
sc.textFile("/tmp/top_100.csv").map(line => {val split = line.split(";"); 
(split(1),split(2))}).distinct.collect

for(i<- 1 to 100) {
sc.parallelize((1 to 4000).map(i => UUID.randomUUID)).
  map(dsrId => {
val r = new java.util.Random(System.currentTimeMillis())

val relSeq = r.nextLong()
val seq = r.nextLong()
val dspReleaseCode = seq.toString
val dspCode = allDsps(r.nextInt(allDsps.size))
val periodEndMonth = 
allPeriodsEndMonths(r.nextInt(allPeriodsEndMonths.size))
val territoryCode = allCountries(r.nextInt(allCountries.size))
val modelCode = allModelCodes(r.nextInt(allModelCodes.size))
val mediaCode = allMediaCodes(r.nextInt(allMediaCodes.size))
val transferCode = 
allTransferCodes(r.nextInt(allTransferCodes.size))
val commercialOffer = 
allCommercialOffers(r.nextInt(allCommercialOffers.size))
val titleAndContributor: (String, String) = 
titlesAndContributors(r.nextInt(titlesAndContributors.size))
val title = titleAndContributor._1
val contributorsName = titleAndContributor._2.split(",").toList
val unicWork = title + "|" + titleAndContributor._2
val payingNetQty = r.nextInt(100).toLong
Resource(dsrId, relSeq, seq, dspReleaseCode, commercialOffer, 
transferCode, mediaCode, modelCode,
  unicWork, title, status, contributorsName, periodEndMonth, 
dspCode, territoryCode, payingNetQty,
  authorizedSocietiesTxt, relType)

  }).
  saveToCassandra("keyspace", "resource")

Thread.sleep(500)
}
{noformat}

The indices

{noformat}
CREATE CUSTOM INDEX resource_territory_code_idx ON sharon.resource_bench 
(territory_code) USING 'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS 
= {'analyzer_class': 
'org.apache.cassandra.index.sasi.analyzer.NonTokenizingAnalyzer', 
'case_sensitive': 'false'};

CREATE CUSTOM INDEX resource_dsp_code_idx ON sharon.resource_bench (dsp_code) 
USING 'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS = 
{'analyzer_class': 
'org.apache.cassandra.index.sasi.analyzer.NonTokenizingAnalyzer', 
'case_sensitive': 'false'};

CREATE CUSTOM INDEX resource_commercial_offer_code_idx ON sharon.resource_bench 
(commercial_offer_code) USING 'org.apache.cassandra.index.sasi.SASIIndex' WITH 
OPTIONS = {'analyzer_class': 
'org.apache.cassandra.index.sasi.analyzer.NonTokenizingAnalyzer', 
'case_sensitive': 'false'};

CREATE CUSTOM INDEX resource_authorized_societies_txt_idx ON 
sharon.resource_bench (authorized_societies_txt) USING 
'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS = {'analyzer_class': 
'org.apache.cassandra.index.sasi.analyzer.StandardAnalyzer', 
'tokenization_normalize_lowercase': 'true', 'mode': 'PREFIX', 'analyzed': 
'true', 'tokenization_enable_stemming': 'true'};

CREATE CUSTOM INDEX resource_transfer_code_idx ON sharon.resource_bench 
(transfer_code) USING 'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS 
= {'analyzer_class': 
'org.apache.cassandra.index.sasi.analyzer.NonTokenizingAnalyzer', 
'case_sensitive': 'false'};

CREATE CUSTOM INDEX resource_rel_type_idx ON 

[jira] [Commented] (CASSANDRA-11255) COPY TO should have higher double precision

[Stefania (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15198815#comment-15198815
 ] 

Stefania commented on CASSANDRA-11255:
--

Patch details (trunk only):

|[patch|https://github.com/stef1927/cassandra/commits/11255]|[dtest 
results|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-11255-dtest/]|[dtest
 changes|https://github.com/stef1927/cassandra-dtest/tree/11255]|

CI still running.

> COPY TO should have higher double precision
> ---
>
> Key: CASSANDRA-11255
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11255
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Stefania
>Assignee: Stefania
>  Labels: doc-impacting, lhf
> Fix For: 3.x
>
>
> At the moment COPY TO uses the same float precision as cqlsh, which by 
> default is 5 but it can be changed in cqlshrc. However, typically people want 
> to preserve precision when exporting data and so this default is too low for 
> COPY TO.
> I suggest adding a new COPY FROM option to specify floating point precision 
> with a much higher default value, for example 12.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[14/15] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.5

[slebresne]

Merge branch 'cassandra-3.0' into cassandra-3.5


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d526ad79
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d526ad79
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d526ad79

Branch: refs/heads/trunk
Commit: d526ad79d7971a5d9ad730c232e56343e2b3b377
Parents: 807597f 8d101fd
Author: Sylvain Lebresne 
Authored: Thu Mar 17 10:21:58 2016 +0100
Committer: Sylvain Lebresne 
Committed: Thu Mar 17 10:21:58 2016 +0100

--
 doc/cql3/CQL.textile | 26 +-
 1 file changed, 17 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/d526ad79/doc/cql3/CQL.textile
--

[jira] [Updated] (CASSANDRA-10752) CQL.textile wasn't updated for CASSANDRA-6839

[Sylvain Lebresne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-10752:
-
   Resolution: Fixed
Fix Version/s: (was: 3.0.x)
   (was: 2.2.x)
   (was: 2.1.x)
   (was: 3.x)
   3.5
   3.0.5
   2.2.6
   2.1.14
   Status: Resolved  (was: Patch Available)

Committed and changed pushed online. Thanks.

> CQL.textile wasn't updated for CASSANDRA-6839
> -
>
> Key: CASSANDRA-10752
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10752
> Project: Cassandra
>  Issue Type: Bug
>  Components: Documentation and Website
>Reporter: Jeremiah Jordan
>Assignee: Sylvain Lebresne
> Fix For: 2.1.14, 2.2.6, 3.0.5, 3.5
>
>
> CQL.textile wasn't updated after CASSANDRA-6839 added inequalities for LWT's.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10809) Create a -D option to prevent gossip startup

[Sylvain Lebresne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-10809:
-
Assignee: Sylvain Lebresne
Reviewer: Brandon Williams
  Status: Patch Available  (was: Open)

[~brandon.williams] do you a few minutes to check if that's what you had in 
mind?

> Create a -D option to prevent gossip startup
> 
>
> Key: CASSANDRA-10809
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10809
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Distributed Metadata
>Reporter: Brandon Williams
>Assignee: Sylvain Lebresne
> Fix For: 2.1.x
>
> Attachments: 10809.txt
>
>
> In CASSANDRA-6961 we changed how join_ring=false works, to great benefit.  
> However, sometimes you need to a node to come up, but not interact with other 
> nodes whatsoever - for example if you have a schema problem, it will still 
> pull the schema from another node because they still gossip even though we're 
> in a dead state.
> We can add a way to restore the previous behavior by simply adding something 
> like -Dcassandra.start_gossip=false.
> In the meantime we can workaround by setting listen_address to localhost, but 
> that's kind of a pain.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-8969) Add indication in cassandra.yaml that rpc timeouts going too high will cause memory build up

[Sylvain Lebresne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-8969:

Fix Version/s: 3.x
   Status: Patch Available  (was: Open)

> Add indication in cassandra.yaml that rpc timeouts going too high will cause 
> memory build up
> 
>
> Key: CASSANDRA-8969
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8969
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Configuration
>Reporter: Jeremy Hanna
>Assignee: Jeremy Hanna
>Priority: Minor
>  Labels: lhf
> Fix For: 3.x
>
> Attachments: 8969.txt
>
>
> It would be helpful to communicate that setting the rpc timeouts too high may 
> cause memory problems on the server as it can become overloaded and has to 
> retain the in flight requests in memory.  I'll get this done but just adding 
> the ticket as a placeholder for memory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9259) Bulk Reading from Cassandra

[Stefania (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15197232#comment-15197232
 ] 

Stefania commented on CASSANDRA-9259:
-


Below are the results of a set of benchmarking measurements that were performed 
using the _cassandra-stress_ token range queries as delivered by 
CASSANDRA-10331. A token range query is a query as outlined in the description 
of this ticket: {{SELECT a, b, c FROM myKs.myTable WHERE Token(partitionKey) > 
X AND Token(partitionKey) <= Y}}.

These are suggested code enhancements that follow from the analysis of Java 
Flight Recorder profiles taken during these measurements.

h5. Optimizations to the current read path

* Avoid using heap buffers in {{DataOutputBuffer}}, ideally we should use the 
buffer pool and the buffer pool should be extended to support sizes larger than 
64 kbytes (point 4 of CASSANDRA-9468).

* Avoid creating a local response in {{LocalReadRunnable}}, which serializes 
the partition iterator in the READ stage thread only for it to be deserialized 
again in the storage proxy thread. I realize this is problematic because we 
need to guard sstables and memtables via a {{ReadExecutionController}}, as well 
as guarantee iterator safety, but removing this redundant step should help 
significantly for large analytics workloads where the client queries data 
locally. Perhaps this should be done together with a new output format 
optimized for this workflow, as described further below. Any thoughts 
[~slebresne]?

* Improve access to sstables for local token ranges in order to avoid the 
binary search on the index summary, and scanning of index entries until the 
range is found.

* Increase parallelism by splitting local token ranges; at the moment there is 
parallelism in storage proxy {{getRangeSlice()}} but ranges are only split by 
replica, a local range is not divided into smaller ranges. Splitting a local 
range into sub-ranges as stored in different sstables should increase 
performance.

h5. Extensive re-alignment of format and transfer mechanism

In addition to optimizing the current read path, to avoid the cost of encoding 
CQL rows, we could replace them with a format that is more analytics friendly. 
This would be similar to a special compaction task outputting a columnar format 
optimized for the query as suggested above. This format should allow efficient 
decoding in the client and its data should be transferred from Cassandra to the 
client as quickly as possible. If the client runs locally, for example we could 
use one of the following mechanisms:

* shared memory; 
* Unix domain pipes;
* another [fast IPC mechanisms|https://lwn.net/Articles/466304/]. 

We should probably leverage either [Apache Parquet|https://parquet.apache.org] 
for the format or [Apache arrow|https://arrow.apache.org] for both format and 
transfer mechanism. I think the latter is more aligned to what we are trying to 
do, but unfortunately this project is just starting. As far as I understand, 
eventually Spark will read Arrow memory and so if we wrote Arrow memory the 
transfer should become extremely efficient.

h5. Benchmark setup

Each _cassandra-stress_ operation corresponds to the retrieval of one page 
using a token range query where all columns are retrieved. Each token range 
corresponds to a VNODE range and the Cassandra host was configured with 256 
ranges. Different partition sizes were tested: 100 bytes, 1 kbyte, 10 kbytes, 
100 kbytes and 1 MB. Each partition had a single CQL row; for the 100 kbytes 
partition size however, two more tests were added, with 100 and 1000 CQL rows 
respectively. The size of the rows was adjusted so that the partition size 
stayed at approximately 100 kbytes. 

The page size for each test was chosen so that the amount of data downloaded in 
each operation was roughly 1 MB. So we can view the _cassandra-stress_ ops / 
second as roughly MB / seconds. A single _cassandra-stress_ thread was used.

The tests were run on a GCE *n1-standard-8* VM: 4 Intel Xeon @ 2.60GHz physical 
cores with 2 hyper-threads per core - all cores located on a single socket, 29 
GB of Memory and a 1073GB SCSI hard disk. Both _cassandra-stress_ and the 
Cassandra process were running on the same VM, as I understand this is normally 
how Spark or other analytics tools are deployed.

h5. Benchmark results

The following is a summary of the results in ops / second, 
[^bulk-read-benchmark.1.html] attached also contains diagrams, 
partitions/second and rows/second.

||100-bytes||1-kbyte||10-kbytes||100-kbytes||100-kbytes-100-rows||100-kbytes-1000-rows||1-MB||
|28|59|101|80|50|10|73|

The performance is variable and very much dependent on partition size and 
number of CQL rows in a partition. Java Flight Recorder (JFR) files for all 
cases, albeit measured with a smaller sample, have also been attached, see 
[^bulk-read-jfr-profiles.1.tar.gz] and 

[jira] [Updated] (CASSANDRA-10748) UTF8Validator.validate() wrong ??

[Benjamin Lerer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Lerer updated CASSANDRA-10748:
---
Component/s: Core

> UTF8Validator.validate() wrong ??
> -
>
> Key: CASSANDRA-10748
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10748
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Robert Stupp
>Assignee: Benjamin Lerer
>Priority: Minor
> Fix For: 2.2.6, 3.0.5, 3.5
>
>
> The switch-statement in {{UTF8Validator.validate()}} can never go into {{case 
> TWO_80}} as the assignment {{state = State.TWO_80;}} in line 75 is dead.
> I assume that the {{TWO_80}} case is completely superfluous - but would like 
> to have a 2nd opinion on this.
> /cc [~carlyeks] (CASSANDRA-4495)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10752) CQL.textile wasn't updated for CASSANDRA-6839

[Benjamin Lerer (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15199179#comment-15199179
 ] 

Benjamin Lerer commented on CASSANDRA-10752:


+1

> CQL.textile wasn't updated for CASSANDRA-6839
> -
>
> Key: CASSANDRA-10752
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10752
> Project: Cassandra
>  Issue Type: Bug
>  Components: Documentation and Website
>Reporter: Jeremiah Jordan
>Assignee: Sylvain Lebresne
> Fix For: 2.1.x, 2.2.x, 3.0.x, 3.x
>
>
> CQL.textile wasn't updated after CASSANDRA-6839 added inequalities for LWT's.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-11224) dtest failure in bootstrap_test.TestBootstrap.failed_bootstrap_wiped_node_can_join_test

[Russ Hatch (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Russ Hatch resolved CASSANDRA-11224.

Resolution: Cannot Reproduce

> dtest failure in 
> bootstrap_test.TestBootstrap.failed_bootstrap_wiped_node_can_join_test
> ---
>
> Key: CASSANDRA-11224
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11224
> Project: Cassandra
>  Issue Type: Test
>Reporter: Russ Hatch
>Assignee: Russ Hatch
>  Labels: dtest
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_novnode_dtest/209/testReport/bootstrap_test/TestBootstrap/failed_bootstrap_wiped_node_can_join_test
> Failed on CassCI build cassandra-2.1_novnode_dtest #209
> Happening only intermittently.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[4/4] cassandra git commit: Merge branch cassandra-3.5 into trunk

[blerer]

Merge branch cassandra-3.5 into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e5394f19
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e5394f19
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e5394f19

Branch: refs/heads/trunk
Commit: e5394f192e882a5f23527041a9a93f82e80c0a49
Parents: 4ecd854 ee40e3b
Author: Benjamin Lerer 
Authored: Wed Mar 16 18:26:48 2016 +0100
Committer: Benjamin Lerer 
Committed: Wed Mar 16 18:26:57 2016 +0100

--
 CHANGES.txt |   1 +
 .../db/commitlog/AbstractCommitLogService.java  |   1 +
 .../db/commitlog/CommitLogSegment.java  |  23 +++--
 .../db/commitlog/CommitLogSegmentManager.java   |  11 +-
 .../db/commitlog/CompressedSegment.java |   4 +-
 .../db/commitlog/EncryptedSegment.java  |   4 +-
 .../db/commitlog/FileDirectSegment.java |  42 ++--
 .../commitlog/CommitLogSegmentManagerTest.java  | 100 ++
 .../cassandra/db/commitlog/CommitLogTest.java   | 101 ++-
 9 files changed, 220 insertions(+), 67 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/e5394f19/CHANGES.txt
--
diff --cc CHANGES.txt
index 6897d61,ffd0b24..8eb0c1f
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,15 -1,6 +1,16 @@@
 +3.6
 + * Improve concurrency in CompactionStrategyManager (CASSANDRA-10099)
 + * (cqlsh) interpret CQL type for formatting blobs (CASSANDRA-11274)
 + * Refuse to start and print txn log information in case of disk
 +   corruption (CASSANDRA-10112)
 + * Resolve some eclipse-warnings (CASSANDRA-11086)
 + * (cqlsh) Show static columns in a different color (CASSANDRA-11059)
 + * Allow to remove TTLs on table with default_time_to_live (CASSANDRA-11207)
 +
 +
  3.5
  Merged from 3.0:
+  * Add backpressure to compressed or encrypted commit log (CASSANDRA-10971)
   * SSTableExport supports secondary index tables (CASSANDRA-11330)
   * Fix sstabledump to include missing info in debug output (CASSANDRA-11321)
   * Establish and implement canonical bulk reading workload(s) 
(CASSANDRA-10331)

[13/15] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.5

[slebresne]

Merge branch 'cassandra-3.0' into cassandra-3.5


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d526ad79
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d526ad79
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d526ad79

Branch: refs/heads/cassandra-3.5
Commit: d526ad79d7971a5d9ad730c232e56343e2b3b377
Parents: 807597f 8d101fd
Author: Sylvain Lebresne 
Authored: Thu Mar 17 10:21:58 2016 +0100
Committer: Sylvain Lebresne 
Committed: Thu Mar 17 10:21:58 2016 +0100

--
 doc/cql3/CQL.textile | 26 +-
 1 file changed, 17 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/d526ad79/doc/cql3/CQL.textile
--

[1/3] cassandra git commit: Fix UTF8Validator.validate() for modified UTF-8

[blerer]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.5 d526ad79d -> e3716ee36


Fix UTF8Validator.validate() for modified UTF-8

patch by Benjamin Lerer; reviewed by Sylvain Lebresne for CASSANDRA-10748


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/42cd66a1
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/42cd66a1
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/42cd66a1

Branch: refs/heads/cassandra-3.5
Commit: 42cd66a1f91dcd0cf9238e7915d5efe5779b9933
Parents: 5e2d3d2
Author: Benjamin Lerer 
Authored: Thu Mar 17 14:27:27 2016 +0100
Committer: Benjamin Lerer 
Committed: Thu Mar 17 14:27:27 2016 +0100

--
 CHANGES.txt   | 1 +
 src/java/org/apache/cassandra/serializers/UTF8Serializer.java | 5 +++--
 2 files changed, 4 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/42cd66a1/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index fdb5807..fa4a726 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.6
+ * Fix UTF8Validator.validate() for modified UTF-8 (CASSANDRA-10748)
  * Clarify that now() function is calculated on the coordinator node in CQL 
documentation (CASSANDRA-10900)
  * Fix bloom filter sizing with LCS (CASSANDRA-11344)
  * (cqlsh) Fix error when result is 0 rows with EXPAND ON (CASSANDRA-11092)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/42cd66a1/src/java/org/apache/cassandra/serializers/UTF8Serializer.java
--
diff --git a/src/java/org/apache/cassandra/serializers/UTF8Serializer.java 
b/src/java/org/apache/cassandra/serializers/UTF8Serializer.java
index be5d2d6..e3ea2d5 100644
--- a/src/java/org/apache/cassandra/serializers/UTF8Serializer.java
+++ b/src/java/org/apache/cassandra/serializers/UTF8Serializer.java
@@ -71,11 +71,12 @@ public class UTF8Serializer extends AbstractTextSerializer
 {
 // validate first byte of 2-byte char, 0xc2-0xdf
 if (b == (byte) 0xc0)
-// speical case: modified utf8 null is 0xc080.
+// special case: modified utf8 null is 0xc080.
 state = State.TWO_80;
 else if ((b & 0x1e) == 0)
 return false;
-state = State.TWO;
+else
+state = State.TWO;
 }
 else if ((b >> 4) == -2)
 {

[jira] [Commented] (CASSANDRA-11296) Run dtests with -Dcassandra.debugrefcount=true and increase checking frequency

[Marcus Eriksson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15197529#comment-15197529
 ] 

Marcus Eriksson commented on CASSANDRA-11296:
-

that seems to fix it, had to move subscription handling into its own class and 
just reference the totalDiskSpace Counter in SSTableTidier: 
https://github.com/krummas/cassandra/commits/marcuse/11296-trunk-fix

I'll clean it up and run a few more tests before patch available

> Run dtests with -Dcassandra.debugrefcount=true and increase checking frequency
> --
>
> Key: CASSANDRA-11296
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11296
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>
> We should run dtests with refcount debugging and check every second instead 
> of every 15 minutes 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9588) Make sstableofflinerelevel print stats before relevel

[Sylvain Lebresne (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15199250#comment-15199250
 ] 

Sylvain Lebresne commented on CASSANDRA-9588:
-

+1

> Make sstableofflinerelevel print stats before relevel
> -
>
> Key: CASSANDRA-9588
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9588
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Jens Rantil
>Assignee: Marcus Eriksson
>Priority: Trivial
>  Labels: lhf
> Fix For: 3.x
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> The current version of sstableofflinerelevel prints the new level hierarchy. 
> While "nodetool cfstats ..." will tell the current hierarchy it would be nice 
> to have "sstableofflinerelevel" output the current level histograms for easy 
> comparison of what changes will be made. Especially since 
> sstableofflinerelevel needs to run when node isn't running and "nodetool 
> cfstats ..." doesn't work because of that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[3/6] cassandra git commit: Add -Dcassandra.start_gossip option to skip starting gossip

[slebresne]

Add -Dcassandra.start_gossip option to skip starting gossip

patch by slebresne; reviewed by brandon.williams for CASSANDRA-10809


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b06bcf73
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b06bcf73
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b06bcf73

Branch: refs/heads/cassandra-3.5
Commit: b06bcf73fb50c8e887d4469042543a9d78f7848c
Parents: 42cd66a
Author: Sylvain Lebresne 
Authored: Thu Mar 17 17:21:06 2016 +0100
Committer: Sylvain Lebresne 
Committed: Fri Mar 18 10:53:33 2016 +0100

--
 CHANGES.txt   | 1 +
 src/java/org/apache/cassandra/service/StorageService.java | 7 ++-
 2 files changed, 7 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/b06bcf73/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index fa4a726..bcdf189 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.6
+ * Add -Dcassandra.start_gossip startup option (CASSANDRA-10809)
  * Fix UTF8Validator.validate() for modified UTF-8 (CASSANDRA-10748)
  * Clarify that now() function is calculated on the coordinator node in CQL 
documentation (CASSANDRA-10900)
  * Fix bloom filter sizing with LCS (CASSANDRA-11344)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/b06bcf73/src/java/org/apache/cassandra/service/StorageService.java
--
diff --git a/src/java/org/apache/cassandra/service/StorageService.java 
b/src/java/org/apache/cassandra/service/StorageService.java
index 0e716e8..9a0ba5d 100644
--- a/src/java/org/apache/cassandra/service/StorageService.java
+++ b/src/java/org/apache/cassandra/service/StorageService.java
@@ -690,6 +690,12 @@ public class StorageService extends 
NotificationBroadcasterSupport implements IE
 
 replacing = DatabaseDescriptor.isReplacing();
 
+if (!Boolean.parseBoolean(System.getProperty("cassandra.start_gossip", 
"true")))
+{
+logger.info("Not starting gossip as requested.");
+return;
+}
+
 prepareToJoin();
 
 // Has to be called after the host id has potentially changed in 
prepareToJoin().
@@ -703,7 +709,6 @@ public class StorageService extends 
NotificationBroadcasterSupport implements IE
 logger.warn("Error loading counter cache", t);
 }
 
-
 if (Boolean.parseBoolean(System.getProperty("cassandra.join_ring", 
"true")))
 {
 joinTokenRing(delay);

[jira] [Commented] (CASSANDRA-11374) LEAK DETECTED during repair

[Marcus Eriksson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15201664#comment-15201664
 ] 

Marcus Eriksson commented on CASSANDRA-11374:
-

[~jfgosselin] how exactly do you trigger repair?

> LEAK DETECTED during repair
> ---
>
> Key: CASSANDRA-11374
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11374
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Jean-Francois Gosselin
>Assignee: Marcus Eriksson
>
> When running a range repair we are seeing the following LEAK DETECTED errors:
> {noformat}
> ERROR [Reference-Reaper:1] 2016-03-17 06:58:52,261 Ref.java:179 - LEAK 
> DETECTED: a reference 
> (org.apache.cassandra.utils.concurrent.Ref$State@5ee90b43) to class 
> org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$1@367168611:[[OffHeapBitSet]]
>  was not released before the reference was garbage collected
> ERROR [Reference-Reaper:1] 2016-03-17 06:58:52,262 Ref.java:179 - LEAK 
> DETECTED: a reference 
> (org.apache.cassandra.utils.concurrent.Ref$State@4ea9d4a7) to class 
> org.apache.cassandra.io.util.SafeMemory$MemoryTidy@1875396681:Memory@[7f34b905fd10..7f34b9060b7a)
>  was not released before the reference was garbage collected
> ERROR [Reference-Reaper:1] 2016-03-17 06:58:52,262 Ref.java:179 - LEAK 
> DETECTED: a reference 
> (org.apache.cassandra.utils.concurrent.Ref$State@27a6b614) to class 
> org.apache.cassandra.io.util.SafeMemory$MemoryTidy@838594402:Memory@[7f34bae11ce0..7f34bae11d84)
>  was not released before the reference was garbage collected
> ERROR [Reference-Reaper:1] 2016-03-17 06:58:52,263 Ref.java:179 - LEAK 
> DETECTED: a reference 
> (org.apache.cassandra.utils.concurrent.Ref$State@64e7b566) to class 
> org.apache.cassandra.io.util.SafeMemory$MemoryTidy@674656075:Memory@[7f342deab4e0..7f342deb7ce0)
>  was not released before the reference was garbage collected
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10748) UTF8Validator.validate() wrong ??

[Benjamin Lerer (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15199474#comment-15199474
 ] 

Benjamin Lerer commented on CASSANDRA-10748:


||utests||dtests||
|[2.2|http://cassci.datastax.com/view/Dev/view/blerer/job/blerer-10748-2.2-testall/lastCompletedBuild/testReport/]|[2.2|http://cassci.datastax.com/view/Dev/view/blerer/job/blerer-10748-2.2-dtest/lastCompletedBuild/testReport/]|
|[3.0|http://cassci.datastax.com/view/Dev/view/blerer/job/blerer-10748-3.0-testall/lastCompletedBuild/testReport/]|[3.0|http://cassci.datastax.com/view/Dev/view/blerer/job/blerer-10748-3.0-dtest/lastCompletedBuild/testReport/]|
The tests looks good.

> UTF8Validator.validate() wrong ??
> -
>
> Key: CASSANDRA-10748
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10748
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Robert Stupp
>Assignee: Benjamin Lerer
>Priority: Minor
> Fix For: 2.2.6, 3.0.5, 3.5
>
>
> The switch-statement in {{UTF8Validator.validate()}} can never go into {{case 
> TWO_80}} as the assignment {{state = State.TWO_80;}} in line 75 is dead.
> I assume that the {{TWO_80}} case is completely superfluous - but would like 
> to have a 2nd opinion on this.
> /cc [~carlyeks] (CASSANDRA-4495)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-7642) Adaptive Consistency

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-7642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko resolved CASSANDRA-7642.
--
   Resolution: Not A Problem
Fix Version/s: (was: 3.x)

I still believe strongly that you should just use the lowest CL your 
application can tolerate.

That said, the proposed functionality does exist today in the official drivers, 
in the form of downgrading CL policy (though I'd again argue that it shouldn't 
exist, but, alas).

Closing as Not A Problem as 1) it isn't a problem and 2) we have a client-based 
solution for it.

> Adaptive Consistency
> 
>
> Key: CASSANDRA-7642
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7642
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Rustam Aliyev
>
> h4. Problem
> At minimum, application requires consistency level of X, which must be fault 
> tolerant CL. However, when there is no failure it would be advantageous to 
> use stronger consistency Y (Y>X).
> h4. Suggestion
> Application defines minimum (X) and maximum (Y) consistency levels. C* can 
> apply adaptive consistency logic to use Y whenever possible and downgrade to 
> X when failure occurs.
> Implementation should not negatively impact performance. Therefore, state has 
> to be maintained globally (not per request).
> h4. Example
> {{MIN_CL=LOCAL_QUORUM}}
> {{MAX_CL=EACH_QUORUM}}
> h4. Use Case
> Consider a case where user wants to maximize their uptime and consistency. 
> They designing a system using C* where transactions are read/written with 
> LOCAL_QUORUM and distributed across 2 DCs. Occasional inconsistencies between 
> DCs can be tolerated. R/W with LOCAL_QUORUM is satisfactory in most of the 
> cases.
> Application requires new transactions to be read back right after they were 
> generated. Write and read could be done through different DCs (no 
> stickiness). In some cases when user writes into DC1 and reads immediately 
> from DC2, replication delay may cause problems. Transaction won't show up on 
> read in DC2, user will retry and create duplicate transaction. Occasional 
> duplicates are fine and the goal is to minimize number of dups.
> Therefore, we want to perform writes with stronger consistency (EACH_QUORUM) 
> whenever possible without compromising on availability. Using adaptive 
> consistency they should be able to define:
>{{Read CL = LOCAL_QUORUM}}
>{{Write CL = ADAPTIVE (MIN:LOCAL_QUORUM, MAX:EACH_QUORUM)}}
> Similar scenario can be described for {{Write CL = ADAPTIVE (MIN:QUORUM, 
> MAX:ALL)}} case.
> h4. Criticism
> # This functionality can/should be implemented by user himself.
> bq. It will be hard for an average user to implement topology monitoring and 
> state machine. Moreover, this is a pattern which repeats.
> # Transparent downgrading violates the CL contract, and that contract 
> considered be just about the most important element of Cassandra's runtime 
> behavior.
> bq.Fully transparent downgrading without any contract is dangerous. However, 
> would it be problem if we specify explicitly only two discrete CL levels - 
> MIN_CL and MAX_CL?
> # If you have split brain DCs (partitioned in CAP), you have to sacrifice 
> either consistency or availability, and auto downgrading sacrifices the 
> consistency in dangerous ways if the application isn't designed to handle it. 
> And if the application is designed to handle it, then it should be able to 
> handle it in normal circumstances, not just degraded/extraordinary ones.
> bq. Agreed. Application should be designed for MIN_CL. In that case, MAX_CL 
> will not be causing much harm, only adding flexibility.
> # It might be a better idea to loudly downgrade, instead of silently 
> downgrading, meaning that the client code does an explicit retry with lower 
> consistency on failure and takes some other kind of action to attempt to 
> inform either users or operators of the problem. The silent part of the 
> downgrading which could be dangerous.
> bq. There are certainly cases where user should be informed when consistency 
> changes in order to perform custom action. For this purpose we could 
> allow/require user to register callback function which will be triggered when 
> consistency level changes. Best practices could be enforced by requiring 
> callback.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11383) SASI index build leads to massive OOM

[DOAN DuyHai (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202259#comment-15202259
 ] 

DOAN DuyHai commented on CASSANDRA-11383:
-

[~xedin] 

OK, I'm trying to fetch the sstable with its data files

In the meantime, I just re-create one index as shown above using * 
max_compaction_flush_memory_in_mb * = 128 and SASI flushes thousands of index 
files and eventually the server dies maybe because too many file handles

{noformat}
INFO  [SASI-General:1] 2016-03-18 22:45:37,480 PerSSTableIndexWriter.java:258 - 
Flushed index segment 
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-SI_resource_period_end_month_int_idx.db_130370,
 took 0 ms.
INFO  [SASI-General:1] 2016-03-18 22:45:37,581 PerSSTableIndexWriter.java:258 - 
Flushed index segment 
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-SI_resource_period_end_month_int_idx.db_130371,
 took 101 ms.
ERROR [SASI-General:1] 2016-03-18 22:45:37,582 CassandraDaemon.java:195 - 
Exception in thread Thread[SASI-General:1,5,main]
org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
{noformat}

> SASI index build leads to massive OOM
> -
>
> Key: CASSANDRA-11383
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11383
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
> Environment: C* 3.4
>Reporter: DOAN DuyHai
> Attachments: CASSANDRA-11383.patch, new_system_log_CMS_8GB_OOM.log, 
> system.log_sasi_build_oom
>
>
> 13 bare metal machines
> - 6 cores CPU (12 HT)
> - 64Gb RAM
> - 4 SSD in RAID0
>  JVM settings:
> - G1 GC
> - Xms32G, Xmx32G
> Data set:
>  - ≈ 100Gb/per node
>  - 1.3 Tb cluster-wide
>  - ≈ 20Gb for all SASI indices
> C* settings:
> - concurrent_compactors: 1
> - compaction_throughput_mb_per_sec: 256
> - memtable_heap_space_in_mb: 2048
> - memtable_offheap_space_in_mb: 2048
> I created 9 SASI indices
>  - 8 indices with text field, NonTokenizingAnalyser,  PREFIX mode, 
> case-insensitive
>  - 1 index with numeric field, SPARSE mode
>  After a while, the nodes just gone OOM.
>  I attach log files. You can see a lot of GC happening while index segments 
> are flush to disk. At some point the node OOM ...
> /cc [~xedin]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-11383) SASI index build leads to massive OOM

[DOAN DuyHai (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202259#comment-15202259
 ] 

DOAN DuyHai edited comment on CASSANDRA-11383 at 3/18/16 10:16 PM:
---

[~xedin] 

OK, I'm trying to fetch the sstable with its data files

In the meantime, I just re-create one index as shown above using 
*max_compaction_flush_memory_in_mb* = 128 and SASI flushes thousands of index 
files (of ~16kb each) and eventually the server dies maybe because too many 
file handles

{noformat}
INFO  [SASI-General:1] 2016-03-18 22:45:37,480 PerSSTableIndexWriter.java:258 - 
Flushed index segment 
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-SI_resource_period_end_month_int_idx.db_130370,
 took 0 ms.
INFO  [SASI-General:1] 2016-03-18 22:45:37,581 PerSSTableIndexWriter.java:258 - 
Flushed index segment 
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-SI_resource_period_end_month_int_idx.db_130371,
 took 101 ms.
ERROR [SASI-General:1] 2016-03-18 22:45:37,582 CassandraDaemon.java:195 - 
Exception in thread Thread[SASI-General:1,5,main]
org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
   at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:156) 
~[apache-cassandra-3.4.jar:3.4]
at 
org.apache.cassandra.index.sasi.utils.MappedBuffer.(MappedBuffer.java:78) 
~[apache-cassandra-3.4.jar:3.4]
at 
org.apache.cassandra.index.sasi.utils.MappedBuffer.(MappedBuffer.java:57) 
~[apache-cassandra-3.4.jar:3.4]
at 
org.apache.cassandra.index.sasi.disk.OnDiskIndex.(OnDiskIndex.java:142) 
~[apache-cassandra-3.4.jar:3.4]
at 
org.apache.cassandra.index.sasi.disk.PerSSTableIndexWriter$Index.lambda$scheduleSegmentFlush$260(PerSSTableIndexWriter.java:253)
 ~[apache-cassandra-3.4.jar:3.4]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
~[na:1.8.0_74]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
~[na:1.8.0_74]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940) 
~[na:1.8.0_74]
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152) 
~[apache-cassandra-3.4.jar:3.4]
... 8 common frames omitted
Caused by: java.lang.OutOfMemoryError: Map failed
{noformat}


was (Author: doanduyhai):
[~xedin] 

OK, I'm trying to fetch the sstable with its data files

In the meantime, I just re-create one index as shown above using 
*max_compaction_flush_memory_in_mb* = 128 and SASI flushes thousands of index 
files (of ~16kb each) and eventually the server dies maybe because too many 
file handles

{noformat}
INFO  [SASI-General:1] 2016-03-18 22:45:37,480 PerSSTableIndexWriter.java:258 - 
Flushed index segment 
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-SI_resource_period_end_month_int_idx.db_130370,
 took 0 ms.
INFO  [SASI-General:1] 2016-03-18 22:45:37,581 PerSSTableIndexWriter.java:258 - 
Flushed index segment 
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-SI_resource_period_end_month_int_idx.db_130371,
 took 101 ms.
ERROR [SASI-General:1] 2016-03-18 22:45:37,582 CassandraDaemon.java:195 - 
Exception in thread Thread[SASI-General:1,5,main]
org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
{noformat}

> SASI index build leads to massive OOM
> -
>
> Key: CASSANDRA-11383
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11383
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
> Environment: C* 3.4
>Reporter: DOAN DuyHai
> Attachments: CASSANDRA-11383.patch, new_system_log_CMS_8GB_OOM.log, 
> system.log_sasi_build_oom
>
>
> 13 bare metal machines
> - 6 cores CPU (12 HT)
> - 64Gb RAM
> - 4 SSD in RAID0
>  JVM settings:
> - G1 GC
> - Xms32G, Xmx32G
> Data set:
>  - ≈ 100Gb/per node
>  - 1.3 Tb cluster-wide
>  - ≈ 20Gb for all SASI indices
> C* settings:
> - concurrent_compactors: 1
> - compaction_throughput_mb_per_sec: 256
> - memtable_heap_space_in_mb: 2048
> - memtable_offheap_space_in_mb: 2048
> I created 9 SASI indices
>  - 8 indices with text field, NonTokenizingAnalyser,  PREFIX mode, 
> case-insensitive
>  - 1 index with numeric field, SPARSE mode
>  After a while, the nodes just gone OOM.
>  I attach log files. You can see a lot of GC happening while index segments 
> are flush to disk. At some point the node OOM ...
> /cc [~xedin]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-11384) dtest failure in materialized_views_test.TestMaterializedViews.complex_mv_select_statements_test

[Jim Witschey (JIRA)]

Jim Witschey created CASSANDRA-11384:


 Summary: dtest failure in 
materialized_views_test.TestMaterializedViews.complex_mv_select_statements_test
 Key: CASSANDRA-11384
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11384
 Project: Cassandra
  Issue Type: Test
Reporter: Jim Witschey
Assignee: DS Test Eng


We've got a single flap on the 3.0 novnode job:

http://cassci.datastax.com/job/cassandra-3.0_novnode_dtest/189/testReport/materialized_views_test/TestMaterializedViews/complex_mv_select_statements_test

{code}

Error Message

Expected [[1, 0, 1, 0], [1, 1, 1, 0], [1, 2, 1, 0]] from SELECT a, b, c, d FROM 
mv, but got [[1, 0, 1, 0], [1, 1, 1, 0]]
 >> begin captured logging << 
dtest: DEBUG: cluster ccm directory: /mnt/tmp/dtest-d0ZZ9_
dtest: DEBUG: Custom init_config not found. Setting defaults.
dtest: DEBUG: Done setting configuration options:
{   'num_tokens': None,
'phi_convict_threshold': 5,
'range_request_timeout_in_ms': 1,
'read_request_timeout_in_ms': 1,
'request_timeout_in_ms': 1,
'truncate_request_timeout_in_ms': 1,
'write_request_timeout_in_ms': 1}
dtest: DEBUG: Creating keyspace
dtest: DEBUG: Testing MV primary key: ((a, b), c)
dtest: DEBUG: Testing MV primary key: ((b, a), c)
- >> end captured logging << -
Stacktrace

  File "/usr/lib/python2.7/unittest/case.py", line 329, in run
testMethod()
  File "/home/automaton/cassandra-dtest/materialized_views_test.py", line 1193, 
in complex_mv_select_statements_test
cl=ConsistencyLevel.QUORUM
  File "/home/automaton/cassandra-dtest/assertions.py", line 67, in assert_all
assert list_res == expected, "Expected %s from %s, but got %s" % (expected, 
query, list_res)
"Expected [[1, 0, 1, 0], [1, 1, 1, 0], [1, 2, 1, 0]] from SELECT a, b, c, d 
FROM mv, but got [[1, 0, 1, 0], [1, 1, 1, 0]]\n >> begin 
captured logging << \ndtest: DEBUG: cluster ccm directory: 
/mnt/tmp/dtest-d0ZZ9_\ndtest: DEBUG: Custom init_config not found. Setting 
defaults.\ndtest: DEBUG: Done setting configuration options:\n{   'num_tokens': 
None,\n'phi_convict_threshold': 5,\n'range_request_timeout_in_ms': 
1,\n'read_request_timeout_in_ms': 1,\n'request_timeout_in_ms': 
1,\n'truncate_request_timeout_in_ms': 1,\n
'write_request_timeout_in_ms': 1}\ndtest: DEBUG: Creating keyspace\ndtest: 
DEBUG: Testing MV primary key: ((a, b), c)\ndtest: DEBUG: Testing MV primary 
key: ((b, a), c)\n- >> end captured logging << 
-"
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11383) SASI index build leads to massive OOM

[Pavel Yaskevich (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202281#comment-15202281
 ] 

Pavel Yaskevich commented on CASSANDRA-11383:
-

[~doanduyhai] Ok yeah, it looks like there are too many index components. I 
suspect that there is something wrong with index builder there because we have 
sstable files which are over 100G in size and have about 20 indexes attached to 
them without a problem, merging 24M sized segments should never be a problem, 
so it would be very helpful if you could share that ma-big-1831 sstable somehow 
so I can run couple of experiments and see where the things are.

> SASI index build leads to massive OOM
> -
>
> Key: CASSANDRA-11383
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11383
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
> Environment: C* 3.4
>Reporter: DOAN DuyHai
> Attachments: CASSANDRA-11383.patch, new_system_log_CMS_8GB_OOM.log, 
> system.log_sasi_build_oom
>
>
> 13 bare metal machines
> - 6 cores CPU (12 HT)
> - 64Gb RAM
> - 4 SSD in RAID0
>  JVM settings:
> - G1 GC
> - Xms32G, Xmx32G
> Data set:
>  - ≈ 100Gb/per node
>  - 1.3 Tb cluster-wide
>  - ≈ 20Gb for all SASI indices
> C* settings:
> - concurrent_compactors: 1
> - compaction_throughput_mb_per_sec: 256
> - memtable_heap_space_in_mb: 2048
> - memtable_offheap_space_in_mb: 2048
> I created 9 SASI indices
>  - 8 indices with text field, NonTokenizingAnalyser,  PREFIX mode, 
> case-insensitive
>  - 1 index with numeric field, SPARSE mode
>  After a while, the nodes just gone OOM.
>  I attach log files. You can see a lot of GC happening while index segments 
> are flush to disk. At some point the node OOM ...
> /cc [~xedin]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10734) Support openssl for internode encryption.

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202335#comment-15202335
 ] 

Jason Brown commented on CASSANDRA-10734:
-

If we can wait for CASSANDRA-8457 (where we'll switch internode messaging to 
netty), I can do this reasonably easily with netty's openssl support.

> Support openssl for internode encryption.
> -
>
> Key: CASSANDRA-10734
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10734
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Andy Tolbert
>Assignee: Jason Brown
> Fix For: 3.x
>
>
> It could be a nice improvement to support using openssl for SSL internode 
> encryption.
> This should not be made the default, but rather be an option that one can use.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (CASSANDRA-10404) Node to Node encryption transitional mode

[Jason Brown (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Brown reassigned CASSANDRA-10404:
---

Assignee: Jason Brown

> Node to Node encryption transitional mode
> -
>
> Key: CASSANDRA-10404
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10404
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Tom Lewis
>Assignee: Jason Brown
>
> Create a transitional mode for encryption that allows encrypted and 
> unencrypted traffic node-to-node during a change over to encryption from 
> unencrypted. This alleviates downtime during the switch.
>  This is similar to https://issues.apache.org/jira/browse/CASSANDRA-8803 
> which is intended for client-to-node



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10091) Integrated JMX authn & authz

[Sam Tunnicliffe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sam Tunnicliffe updated CASSANDRA-10091:

Summary: Integrated JMX authn & authz  (was: Align JMX authentication with 
internal authentication)

> Integrated JMX authn & authz
> 
>
> Key: CASSANDRA-10091
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10091
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Jan Karlsson
>Assignee: Sam Tunnicliffe
>Priority: Minor
> Fix For: 3.x
>
>
> It would be useful to authenticate with JMX through Cassandra's internal 
> authentication. This would reduce the overhead of keeping passwords in files 
> on the machine and would consolidate passwords to one location. It would also 
> allow the possibility to handle JMX permissions in Cassandra.
> It could be done by creating our own JMX server and setting custom classes 
> for the authenticator and authorizer. We could then add some parameters where 
> the user could specify what authenticator and authorizer to use in case they 
> want to make their own.
> This could also be done by creating a premain method which creates a jmx 
> server. This would give us the feature without changing the Cassandra code 
> itself. However I believe this would be a good feature to have in Cassandra.
> I am currently working on a solution which creates a JMX server and uses a 
> custom authenticator and authorizer. It is currently build as a premain, 
> however it would be great if we could put this in Cassandra instead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (CASSANDRA-11378) dtest failure in topology_test.TestTopology.decommissioned_node_cant_rejoin_test

[Philip Thompson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Philip Thompson reassigned CASSANDRA-11378:
---

Assignee: Philip Thompson  (was: DS Test Eng)

> dtest failure in 
> topology_test.TestTopology.decommissioned_node_cant_rejoin_test
> 
>
> Key: CASSANDRA-11378
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11378
> Project: Cassandra
>  Issue Type: Test
>Reporter: Jim Witschey
>Assignee: Philip Thompson
>  Labels: dtest
>
> This fails hard on 3.0:
> http://cassci.datastax.com/job/cassandra-3.0_novnode_dtest/186/testReport/topology_test/TestTopology/decommissioned_node_cant_rejoin_test
> Here's the changeset from the first failure on that job:
> http://cassci.datastax.com/job/cassandra-3.0_novnode_dtest/186/changes
> (just this commit: 
> https://github.com/apache/cassandra/commit/719caa67649bf6f27cdd99dd7d6055d2aa8546ae)
> It has failed on trunk as well:
> http://cassci.datastax.com/view/trunk/job/trunk_novnode_dtest/lastCompletedBuild/testReport/topology_test/TestTopology/decommissioned_node_cant_rejoin_test/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11383) SASI index build leads to massive OOM

[DOAN DuyHai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

DOAN DuyHai updated CASSANDRA-11383:

Description: 
13 bare metal machines
- 6 cores CPU (12 HT)
- 64Gb RAM
- 4 SSD in RAID0

 JVM settings:
- G1 GC
- Xms32G, Xmx32G

Data set:
 - ≈ 100Gb/per node
 - 1.3 Tb cluster-wide
 - ≈ 20Gb for all SASI indices

C* settings:
- concurrent_compactors: 1
- compaction_throughput_mb_per_sec: 256
- memtable_heap_space_in_mb: 2048
- memtable_offheap_space_in_mb: 2048

I created 9 SASI indices
 - 8 indices with text field, NonTokenizingAnalyser,  PREFIX mode, 
case-insensitive
 - 1 index with numeric field, SPARSE mode

 After a while, the nodes just gone OOM.

 I attach log files. You can see a lot of GC happening while index segments are 
flush to disk. At some point the node OOM ...

/cc [~xedin]

  was:
13 bare metal machines
- 6 cores CPU (12 HT)
- 64Gb RAM
- 4 SSD in RAID0

 JVM settings:
- G1 GC
- Xms32G, Xmx32G

C* settings:
- concurrent_compactors: 1
- compaction_throughput_mb_per_sec: 256
- memtable_heap_space_in_mb: 2048
- memtable_offheap_space_in_mb: 2048

I created 9 SASI indices
 - 8 indices with text field, NonTokenizingAnalyser,  PREFIX mode, 
case-insensitive
 - 1 index with numeric field, SPARSE mode

 After a while, the nodes just gone OOM.

 I attach log files. You can see a lot of GC happening while index segments are 
flush to disk. At some point the node OOM ...

/cc [~xedin]


> SASI index build leads to massive OOM
> -
>
> Key: CASSANDRA-11383
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11383
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
> Environment: C* 3.4
>Reporter: DOAN DuyHai
> Attachments: system.log_sasi_build_oom
>
>
> 13 bare metal machines
> - 6 cores CPU (12 HT)
> - 64Gb RAM
> - 4 SSD in RAID0
>  JVM settings:
> - G1 GC
> - Xms32G, Xmx32G
> Data set:
>  - ≈ 100Gb/per node
>  - 1.3 Tb cluster-wide
>  - ≈ 20Gb for all SASI indices
> C* settings:
> - concurrent_compactors: 1
> - compaction_throughput_mb_per_sec: 256
> - memtable_heap_space_in_mb: 2048
> - memtable_offheap_space_in_mb: 2048
> I created 9 SASI indices
>  - 8 indices with text field, NonTokenizingAnalyser,  PREFIX mode, 
> case-insensitive
>  - 1 index with numeric field, SPARSE mode
>  After a while, the nodes just gone OOM.
>  I attach log files. You can see a lot of GC happening while index segments 
> are flush to disk. At some point the node OOM ...
> /cc [~xedin]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-2986) Fix short reads in range (and index?) scans

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko resolved CASSANDRA-2986.
--
   Resolution: Duplicate
 Assignee: (was: Jason Brown)
Fix Version/s: (was: 3.x)

> Fix short reads in range (and index?) scans
> ---
>
> Key: CASSANDRA-2986
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2986
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Jonathan Ellis
>Priority: Minor
>
> See CASSANDRA-2643 for the [multi]get fix.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-4467) insufficient space for compaction when upgrade compaction strategy from SizeTiered to Leveled

[Marcus Eriksson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-4467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15201613#comment-15201613
 ] 

Marcus Eriksson commented on CASSANDRA-4467:


this was fixed in CASSANDRA-8329

> insufficient space for compaction when upgrade compaction strategy from 
> SizeTiered to Leveled
> -
>
> Key: CASSANDRA-4467
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4467
> Project: Cassandra
>  Issue Type: Improvement
>Affects Versions: 1.0.0
> Environment: Ubuntu,Oracle Java 1.7,Cassandra 1.0.10
>Reporter: Cheng Zhang
>Priority: Minor
>  Labels: compaction
>
> Cassandra has two data directories as follow:
> cassandra-disk0 use about 500G, about 250G free space
> cassandra-disk1 use about 500G, about 250G free space
> The max data file is about 400G. When I upgrade from 
> SizeTieredCompactionStrategy to LeveledCompactionStrategy, there is no space 
> to do this, for the free space of every data directory is small than the 
> largest data file. But the total free space is enough for compaction.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11383) SASI index build leads to massive OOM

[Pavel Yaskevich (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202377#comment-15202377
 ] 

Pavel Yaskevich commented on CASSANDRA-11383:
-

[~doanduyhai] Thanks! I've already started downloading everything. Meanwhile 
what you can also try is to switch to LCS with max sstable size of 1G or lower 
and try to re-create the data with all of the indexes defined. 

> SASI index build leads to massive OOM
> -
>
> Key: CASSANDRA-11383
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11383
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
> Environment: C* 3.4
>Reporter: DOAN DuyHai
> Attachments: CASSANDRA-11383.patch, new_system_log_CMS_8GB_OOM.log, 
> system.log_sasi_build_oom
>
>
> 13 bare metal machines
> - 6 cores CPU (12 HT)
> - 64Gb RAM
> - 4 SSD in RAID0
>  JVM settings:
> - G1 GC
> - Xms32G, Xmx32G
> Data set:
>  - ≈ 100Gb/per node
>  - 1.3 Tb cluster-wide
>  - ≈ 20Gb for all SASI indices
> C* settings:
> - concurrent_compactors: 1
> - compaction_throughput_mb_per_sec: 256
> - memtable_heap_space_in_mb: 2048
> - memtable_offheap_space_in_mb: 2048
> I created 9 SASI indices
>  - 8 indices with text field, NonTokenizingAnalyser,  PREFIX mode, 
> case-insensitive
>  - 1 index with numeric field, SPARSE mode
>  After a while, the nodes just gone OOM.
>  I attach log files. You can see a lot of GC happening while index segments 
> are flush to disk. At some point the node OOM ...
> /cc [~xedin]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-11228) dtest failure in counter_tests.TestCounters.upgrade_test

[Russ Hatch (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Russ Hatch resolved CASSANDRA-11228.

Resolution: Cannot Reproduce

> dtest failure in counter_tests.TestCounters.upgrade_test
> 
>
> Key: CASSANDRA-11228
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11228
> Project: Cassandra
>  Issue Type: Test
>Reporter: Russ Hatch
>Assignee: Russ Hatch
>  Labels: dtest
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_offheap_dtest/301/testReport/counter_tests/TestCounters/upgrade_test
> Failed on CassCI build cassandra-2.1_offheap_dtest #301
> Looks like perhaps a test timing issue, 2 failures in the past with variants 
> of:
> {noformat}
> [Unavailable exception] message="Cannot achieve consistency level QUORUM"
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11206) Support large partitions on the 3.0 sstable format

[Robert Stupp (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15198225#comment-15198225
 ] 

Robert Stupp commented on CASSANDRA-11206:
--

Note: utests and dtests are fine now (did nothing than a rebase and re-run).

bq. partition should be added to the key cache if not already present

Yes and no. This ticket will add a _shallow_ version of {{IndexedEntry}} to the 
key cache (without the IndexInfo objects as these cause a lot of heap 
pressure). So, when the {{IndexInfo}} objects are actually needed, these will 
be read from disk. My understanding of 
{{UnfilteredRowIteratorWithLowerBound#getPartitionIndexLowerBound}} is, that it 
uses the IndexInfo objects that are already in the key-cache and will go to 
disk if there is a key-cache miss. If we would re-read the IndexInfo objects in 
{{UnfilteredRowIteratorWithLowerBound#getPartitionIndexLowerBound}}, this would 
add overhead. Or did I get it wrong and 
{{UnfilteredRowIteratorWithLowerBound#getPartitionIndexLowerBound}} accesses 
the same partition as {{IndexState}} does? If that's the case, we can maybe 
pass the current, "fully accessible" {{IndexedEntry}} to 
{{UnfilteredRowInteratorWithLowerBound}} (not checked that yet).

We could (in theory) add stuff to the partition summary or change the 
serialized index - but unfortunately not in 3.x.

> Support large partitions on the 3.0 sstable format
> --
>
> Key: CASSANDRA-11206
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11206
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Jonathan Ellis
>Assignee: Robert Stupp
> Fix For: 3.x
>
>
> Cassandra saves a sample of IndexInfo objects that store the offset within 
> each partition of every 64KB (by default) range of rows.  To find a row, we 
> binary search this sample, then scan the partition of the appropriate range.
> The problem is that this scales poorly as partitions grow: on a cache miss, 
> we deserialize the entire set of IndexInfo, which both creates a lot of GC 
> overhead (as noted in CASSANDRA-9754) but is also non-negligible i/o activity 
> (relative to reading a single 64KB row range) as partitions get truly large.
> We introduced an "offset map" in CASSANDRA-10314 that allows us to perform 
> the IndexInfo bsearch while only deserializing IndexInfo that we need to 
> compare against, i.e. log(N) deserializations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11363) Blocked NTR When Connecting Causing Excessive Load

[Carlos Rolo (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15199944#comment-15199944
 ] 

Carlos Rolo commented on CASSANDRA-11363:
-

I can confirm that in both my clusters batches are in use.

> Blocked NTR When Connecting Causing Excessive Load
> --
>
> Key: CASSANDRA-11363
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11363
> Project: Cassandra
>  Issue Type: Bug
>  Components: Coordination
>Reporter: Russell Bradberry
> Attachments: cassandra-102-cms.stack, cassandra-102-g1gc.stack
>
>
> When upgrading from 2.1.9 to 2.1.13, we are witnessing an issue where the 
> machine load increases to very high levels (> 120 on an 8 core machine) and 
> native transport requests get blocked in tpstats.
> I was able to reproduce this in both CMS and G1GC as well as on JVM 7 and 8.
> The issue does not seem to affect the nodes running 2.1.9.
> The issue seems to coincide with the number of connections OR the number of 
> total requests being processed at a given time (as the latter increases with 
> the former in our system)
> Currently there is between 600 and 800 client connections on each machine and 
> each machine is handling roughly 2000-3000 client requests per second.
> Disabling the binary protocol fixes the issue for this node but isn't a 
> viable option cluster-wide.
> Here is the output from tpstats:
> {code}
> Pool NameActive   Pending  Completed   Blocked  All 
> time blocked
> MutationStage 0 88387821 0
>  0
> ReadStage 0 0 355860 0
>  0
> RequestResponseStage  0 72532457 0
>  0
> ReadRepairStage   0 0150 0
>  0
> CounterMutationStage 32   104 897560 0
>  0
> MiscStage 0 0  0 0
>  0
> HintedHandoff 0 0 65 0
>  0
> GossipStage   0 0   2338 0
>  0
> CacheCleanupExecutor  0 0  0 0
>  0
> InternalResponseStage 0 0  0 0
>  0
> CommitLogArchiver 0 0  0 0
>  0
> CompactionExecutor2   190474 0
>  0
> ValidationExecutor0 0  0 0
>  0
> MigrationStage0 0 10 0
>  0
> AntiEntropyStage  0 0  0 0
>  0
> PendingRangeCalculator0 0310 0
>  0
> Sampler   0 0  0 0
>  0
> MemtableFlushWriter   110 94 0
>  0
> MemtablePostFlush 134257 0
>  0
> MemtableReclaimMemory 0 0 94 0
>  0
> Native-Transport-Requests   128   156 38795716
> 278451
> Message type   Dropped
> READ 0
> RANGE_SLICE  0
> _TRACE   0
> MUTATION 0
> COUNTER_MUTATION 0
> BINARY   0
> REQUEST_RESPONSE 0
> PAGED_RANGE  0
> READ_REPAIR  0
> {code}
> Attached is the jstack output for both CMS and G1GC.
> Flight recordings are here:
> https://s3.amazonaws.com/simple-logs/cassandra-102-cms.jfr
> https://s3.amazonaws.com/simple-logs/cassandra-102-g1gc.jfr
> It is interesting to note that while the flight recording was taking place, 
> the load on the machine went back to healthy, and when the flight recording 
> finished the load went back to > 100.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

cassandra git commit: Fix ClientWarningsTest following CASSANDRA-10876

[samt]

Repository: cassandra
Updated Branches:
  refs/heads/trunk 39d5e6aae -> 9e9161b79


Fix ClientWarningsTest following CASSANDRA-10876

Patch by Joel Knighton; reviewed by Sam Tunnicliffe for CASSANDRA-11357


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9e9161b7
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9e9161b7
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9e9161b7

Branch: refs/heads/trunk
Commit: 9e9161b79c033ea4d4c63e6a1ecec1660eb567b5
Parents: 39d5e6a
Author: Joel Knighton 
Authored: Tue Mar 15 18:51:23 2016 -0500
Committer: Sam Tunnicliffe 
Committed: Wed Mar 16 11:30:20 2016 +

--
 test/unit/org/apache/cassandra/service/ClientWarningsTest.java | 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/9e9161b7/test/unit/org/apache/cassandra/service/ClientWarningsTest.java
--
diff --git a/test/unit/org/apache/cassandra/service/ClientWarningsTest.java 
b/test/unit/org/apache/cassandra/service/ClientWarningsTest.java
index cf14d55..78b1c88 100644
--- a/test/unit/org/apache/cassandra/service/ClientWarningsTest.java
+++ b/test/unit/org/apache/cassandra/service/ClientWarningsTest.java
@@ -72,9 +72,13 @@ public class ClientWarningsTest extends CQLTester
 {
 client.connect(false);
 
-QueryMessage query = new 
QueryMessage(createBatchStatement(DatabaseDescriptor.getBatchSizeWarnThreshold()),
 QueryOptions.DEFAULT);
+QueryMessage query = new 
QueryMessage(createBatchStatement2(DatabaseDescriptor.getBatchSizeWarnThreshold()
 / 2 + 1), QueryOptions.DEFAULT);
 Message.Response resp = client.execute(query);
 assertEquals(1, resp.getWarnings().size());
+
+query = new 
QueryMessage(createBatchStatement(DatabaseDescriptor.getBatchSizeWarnThreshold()),
 QueryOptions.DEFAULT);
+resp = client.execute(query);
+assertNull(resp.getWarnings());
 }
 }
 

[jira] [Resolved] (CASSANDRA-4623) Revise contents of clientutil.jar to reflect needs of new native protocol on client side

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-4623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko resolved CASSANDRA-4623.
--
Resolution: Won't Fix

> Revise contents of clientutil.jar to reflect needs of new native protocol on 
> client side
> 
>
> Key: CASSANDRA-4623
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4623
> Project: Cassandra
>  Issue Type: Improvement
>Affects Versions: 1.2.0
>Reporter: Rick Shaw
>Priority: Minor
>
> The introduction of the native transport protocol in 1.2 will introduce the 
> need for many additional classes and methods to be visible from the Server 
> side jars in the form of the {{clientutil.jar}} in order to not require the 
> full Cassandra server dependancy tree.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10752) CQL.textile wasn't updated for CASSANDRA-6839

[Sylvain Lebresne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-10752:
-
Status: Patch Available  (was: Open)

Patch is [here|https://github.com/pcmanus/cassandra/commits/10752]. It's 
against 2.1 since why not, but I'll just merge up if we're good with the 
changes.

> CQL.textile wasn't updated for CASSANDRA-6839
> -
>
> Key: CASSANDRA-10752
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10752
> Project: Cassandra
>  Issue Type: Bug
>  Components: Documentation and Website
>Reporter: Jeremiah Jordan
>Assignee: Sylvain Lebresne
> Fix For: 2.1.x, 2.2.x, 3.0.x, 3.x
>
>
> CQL.textile wasn't updated after CASSANDRA-6839 added inequalities for LWT's.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11352) Include units of metrics in the cassandra-stress tool

[Giampaolo (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15197672#comment-15197672
 ] 

Giampaolo commented on CASSANDRA-11352:
---

I would like to take this one. I just need more details. 
On latency mean, is the following correct?
{code}
latency mean : 0.7ms [READ:0.7ms, WRITE:0.7ms] 
{code}
and in the case that 0.7 are seconds
{code}
latency mean : 0.7s [READ:0.7s, WRITE:0.7s] 
{code}

Does
{code}
Total partitions : 105612753 [READ:52805915, WRITE:52806838]
{code}
need a three digit separation?
{code}
Total partitions : 105.612.753 [READ:52.805.915, WRITE:52.806.838]
{code}



> Include units of metrics in the cassandra-stress tool 
> --
>
> Key: CASSANDRA-11352
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11352
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Rajath Subramanyam
>Priority: Minor
>  Labels: lhf
> Fix For: 2.1.0
>
>
> cassandra-stress in the Results section can have units for the metrics as an 
> improvement to make the tool more usable. 
> Results:
> op rate   : 14668 [READ:7334, WRITE:7334]
> partition rate: 14668 [READ:7334, WRITE:7334]
> row rate  : 14668 [READ:7334, WRITE:7334]
> latency mean  : 0.7 [READ:0.7, WRITE:0.7]
> latency median: 0.6 [READ:0.6, WRITE:0.6]
> latency 95th percentile   : 0.8 [READ:0.8, WRITE:0.8]
> latency 99th percentile   : 1.2 [READ:1.2, WRITE:1.2]
> latency 99.9th percentile : 8.8 [READ:8.9, WRITE:9.0]
> latency max   : 448.7 [READ:162.3, WRITE:448.7]
> Total partitions  : 105612753 [READ:52805915, WRITE:52806838]
> Total errors  : 0 [READ:0, WRITE:0]
> total gc count: 0
> total gc mb   : 0
> total gc time (s) : 0
> avg gc time(ms)   : NaN
> stdev gc time(ms) : 0
> Total operation time  : 02:00:00
> END



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11354) PrimaryKeyRestrictionSet should be refactored

[Benjamin Lerer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Lerer updated CASSANDRA-11354:
---
Reviewer: Sylvain Lebresne

> PrimaryKeyRestrictionSet should be refactored
> -
>
> Key: CASSANDRA-11354
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11354
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CQL
>Reporter: Benjamin Lerer
>Assignee: Benjamin Lerer
>
> While reviewing CASSANDRA-11310 I realized that the code of 
> {{PrimaryKeyRestrictionSet}} was really confusing.
> The main 2 issues are:
> * the fact that it is used for partition keys and clustering columns 
> restrictions whereas those types of column required different processing
> * the {{isEQ}}, {{isSlice}}, {{isIN}} and {{isContains}} methods should not 
> be there as the set of restrictions might not match any of those categories 
> when secondary indexes are used.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9692) Print sensible units for all log messages

[Giampaolo (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15197769#comment-15197769
 ] 

Giampaolo commented on CASSANDRA-9692:
--

I'm not familiar with Python, so this could be a good opportunity to learn it. 
I will take care of the dtests. Do not worry about giving me even the most 
trivial information, I prefer to get bored than wrong :). As you said, this is 
my very first ticket and I need to learn many things.

> Print sensible units for all log messages
> -
>
> Key: CASSANDRA-9692
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9692
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Benedict
>Assignee: Giampaolo
>Priority: Minor
>  Labels: lhf
> Fix For: 3.x
>
> Attachments: 
> Cassandra9692-Rev1-trunk-giampaolo-trapasso-at-radicalbit-io.diff, 
> Cassandra9692-Rev2-trunk-giampaolo.trapasso-at-radicalbit-io.diff, 
> Cassandra9692-trunk-giampaolo-trapasso-at-radicalbit-io.diff
>
>
> Like CASSANDRA-9691, this has bugged me too long. it also adversely impacts 
> log analysis. I've introduced some improvements to the bits I touched for 
> CASSANDRA-9681, but we should do this across the codebase. It's a small 
> investment for a lot of long term clarity in the logs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11370) Display sstable count per level according to repair status on nodetool tablestats

[Paulo Motta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paulo Motta updated CASSANDRA-11370:

Status: Patch Available  (was: Open)

> Display sstable count per level according to repair status on nodetool 
> tablestats 
> --
>
> Key: CASSANDRA-11370
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11370
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Paulo Motta
>Priority: Minor
>  Labels: lhf
>
> After CASSANDRA-8004 we still display sstables in each level on nodetool 
> tablestats as if we had a single compaction strategy, while we have one 
> strategy for repaired and another for unrepaired data. 
> We should split display into repaired and unrepaired set, so this:
> SSTables in each level: [2, 20/10, 15, 0, 0, 0, 0, 0, 0]
> Would become:
> SSTables in each level (repaired): [1, 10, 0, 0, 0, 0, 0, 0, 0]
> SSTables in each level (unrepaired): [1, 10, 15, 0, 0, 0, 0, 0, 0]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11383) SASI index build leads to massive OOM

[Jack Krupansky (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202241#comment-15202241
 ] 

Jack Krupansky commented on CASSANDRA-11383:


What's the table schema? Is period_end_month_int text or int?

period_end_month_int has 3 years times 12 months = 36 unique values, so 3.4 
billion / 36 = 94.44 million rows for each indexed unique value.

> SASI index build leads to massive OOM
> -
>
> Key: CASSANDRA-11383
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11383
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
> Environment: C* 3.4
>Reporter: DOAN DuyHai
> Attachments: CASSANDRA-11383.patch, new_system_log_CMS_8GB_OOM.log, 
> system.log_sasi_build_oom
>
>
> 13 bare metal machines
> - 6 cores CPU (12 HT)
> - 64Gb RAM
> - 4 SSD in RAID0
>  JVM settings:
> - G1 GC
> - Xms32G, Xmx32G
> Data set:
>  - ≈ 100Gb/per node
>  - 1.3 Tb cluster-wide
>  - ≈ 20Gb for all SASI indices
> C* settings:
> - concurrent_compactors: 1
> - compaction_throughput_mb_per_sec: 256
> - memtable_heap_space_in_mb: 2048
> - memtable_offheap_space_in_mb: 2048
> I created 9 SASI indices
>  - 8 indices with text field, NonTokenizingAnalyser,  PREFIX mode, 
> case-insensitive
>  - 1 index with numeric field, SPARSE mode
>  After a while, the nodes just gone OOM.
>  I attach log files. You can see a lot of GC happening while index segments 
> are flush to disk. At some point the node OOM ...
> /cc [~xedin]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11290) dtest failure in materialized_views_test.TestMaterializedViewsConsistency.single_partition_consistent_reads_after_write_test

[Philip Thompson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Philip Thompson updated CASSANDRA-11290:

Attachment: node3.log
node3_debug.log
node2.log
node2_debug.log
node1.log
node1_debug.log

I'm uploading the logs from this run. There are a few exceptions in there.

{code}
WARN  [SharedPool-Worker-16] 2016-02-11 19:15:19,072 
AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread 
Thread[SharedPool-Worker-16,10,main]: {}
java.lang.NullPointerException: null
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:543) ~[main/:na]
at org.apache.cassandra.db.Keyspace.lambda$apply$62(Keyspace.java:469) 
~[main/:na]
at org.apache.cassandra.db.Keyspace$$Lambda$239/1334525713.run(Unknown 
Source) ~[na:na]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[na:1.8.0_45]
at 
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
 ~[main/:na]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) 
[main/:na]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
WARN  [SharedPool-Worker-10] 2016-02-11 19:15:19,107 
AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread 
Thread[SharedPool-Worker-10,10,main]: {}
java.lang.NullPointerException: null
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:543) ~[main/:na]
at org.apache.cassandra.db.Keyspace.lambda$apply$62(Keyspace.java:469) 
~[main/:na]
at org.apache.cassandra.db.Keyspace$$Lambda$239/1334525713.run(Unknown 
Source) ~[na:na]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[na:1.8.0_45]
at 
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
 ~[main/:na]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) 
[main/:na]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
{code}

> dtest failure in 
> materialized_views_test.TestMaterializedViewsConsistency.single_partition_consistent_reads_after_write_test
> 
>
> Key: CASSANDRA-11290
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11290
> Project: Cassandra
>  Issue Type: Test
>Reporter: Russ Hatch
>Assignee: Philip Thompson
>  Labels: dtest
> Fix For: 3.x
>
> Attachments: node1.log, node1_debug.log, node2.log, node2_debug.log, 
> node3.log, node3_debug.log
>
>
> example failure:
> http://cassci.datastax.com/job/trunk_offheap_dtest/26/testReport/materialized_views_test/TestMaterializedViewsConsistency/single_partition_consistent_reads_after_write_test
> Failed on CassCI build trunk_offheap_dtest #26
> Failing infrequently but same error on at least two of those infrequent flaps:
> {noformat}
> Connection to 127.0.0.2 was closed
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[03/12] cassandra git commit: Support streaming of older version sstables in 3.0

[yukim]

Support streaming of older version sstables in 3.0

patch by Paulo Motta; reviewed by Yuki Morishita for CASSANDRA-10990


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e8651b66
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e8651b66
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e8651b66

Branch: refs/heads/cassandra-3.0
Commit: e8651b6625c7f6260852f2a9c45fb189c63ab528
Parents: 7f1339c
Author: Paulo Motta 
Authored: Fri Feb 5 12:38:39 2016 -0300
Committer: Yuki Morishita 
Committed: Thu Mar 17 10:04:44 2016 -0500

--
 CHANGES.txt |   1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  |   2 +
 .../org/apache/cassandra/db/Directories.java|  30 +
 .../cassandra/db/SerializationHeader.java   |   5 +
 .../org/apache/cassandra/db/Serializers.java| 114 ++--
 .../columniterator/AbstractSSTableIterator.java |   4 +-
 .../cassandra/hints/ChecksummedDataInput.java   |   8 +-
 .../org/apache/cassandra/hints/HintMessage.java |   4 +-
 .../io/compress/CompressedSequentialWriter.java |   8 +-
 .../io/sstable/SSTableSimpleIterator.java   |  11 +-
 .../io/sstable/format/SSTableReader.java|   2 +-
 .../io/sstable/format/SSTableWriter.java|   2 +-
 .../io/sstable/format/big/BigTableWriter.java   |   4 +-
 .../cassandra/io/util/BytesReadTracker.java |  30 +
 .../apache/cassandra/io/util/DataPosition.java  |  21 +
 .../apache/cassandra/io/util/FileDataInput.java |   8 +-
 .../org/apache/cassandra/io/util/FileMark.java  |  20 -
 .../io/util/FileSegmentInputStream.java |   6 +-
 .../cassandra/io/util/RandomAccessReader.java   |   8 +-
 .../cassandra/io/util/RewindableDataInput.java  |  30 +
 .../io/util/RewindableDataInputStreamPlus.java  | 569 +++
 .../cassandra/io/util/SequentialWriter.java |   6 +-
 .../cassandra/io/util/TrackedDataInputPlus.java | 150 +
 .../cassandra/io/util/TrackedInputStream.java   |  76 +++
 .../cassandra/service/StorageService.java   |   1 +
 .../cassandra/streaming/StreamReader.java   |  84 ++-
 .../compress/CompressedStreamReader.java|  18 +-
 .../streaming/messages/FileMessageHeader.java   |   4 +-
 .../apache/cassandra/tools/nodetool/Repair.java |   2 +-
 .../cassandra/utils/BytesReadTracker.java   | 153 -
 .../cassandra/utils/CloseableIterator.java  |   1 -
 ...acy_jb_clust_compact-jb-1-CompressionInfo.db | Bin 0 -> 83 bytes
 ..._tables-legacy_jb_clust_compact-jb-1-Data.db | Bin 0 -> 5270 bytes
 ...ables-legacy_jb_clust_compact-jb-1-Filter.db | Bin 0 -> 24 bytes
 ...tables-legacy_jb_clust_compact-jb-1-Index.db | Bin 0 -> 157685 bytes
 ...s-legacy_jb_clust_compact-jb-1-Statistics.db | Bin 0 -> 6791 bytes
 ...bles-legacy_jb_clust_compact-jb-1-Summary.db | Bin 0 -> 71 bytes
 ..._tables-legacy_jb_clust_compact-jb-1-TOC.txt |   7 +
 ...lust_counter_compact-jb-1-CompressionInfo.db | Bin 0 -> 75 bytes
 ...legacy_jb_clust_counter_compact-jb-1-Data.db | Bin 0 -> 4228 bytes
 ...gacy_jb_clust_counter_compact-jb-1-Filter.db | Bin 0 -> 24 bytes
 ...egacy_jb_clust_counter_compact-jb-1-Index.db | Bin 0 -> 157685 bytes
 ..._jb_clust_counter_compact-jb-1-Statistics.db | Bin 0 -> 6791 bytes
 ...acy_jb_clust_counter_compact-jb-1-Summary.db | Bin 0 -> 71 bytes
 ...legacy_jb_clust_counter_compact-jb-1-TOC.txt |   7 +
 ...cy_jb_simple_compact-jb-1-CompressionInfo.db | Bin 0 -> 43 bytes
 ...tables-legacy_jb_simple_compact-jb-1-Data.db | Bin 0 -> 108 bytes
 ...bles-legacy_jb_simple_compact-jb-1-Filter.db | Bin 0 -> 24 bytes
 ...ables-legacy_jb_simple_compact-jb-1-Index.db | Bin 0 -> 75 bytes
 ...-legacy_jb_simple_compact-jb-1-Statistics.db | Bin 0 -> 4395 bytes
 ...les-legacy_jb_simple_compact-jb-1-Summary.db | Bin 0 -> 71 bytes
 ...tables-legacy_jb_simple_compact-jb-1-TOC.txt |   7 +
 ...mple_counter_compact-jb-1-CompressionInfo.db | Bin 0 -> 43 bytes
 ...egacy_jb_simple_counter_compact-jb-1-Data.db | Bin 0 -> 118 bytes
 ...acy_jb_simple_counter_compact-jb-1-Filter.db | Bin 0 -> 24 bytes
 ...gacy_jb_simple_counter_compact-jb-1-Index.db | Bin 0 -> 75 bytes
 ...jb_simple_counter_compact-jb-1-Statistics.db | Bin 0 -> 4395 bytes
 ...cy_jb_simple_counter_compact-jb-1-Summary.db | Bin 0 -> 71 bytes
 ...egacy_jb_simple_counter_compact-jb-1-TOC.txt |   7 +
 ...acy_ka_clust_compact-ka-1-CompressionInfo.db | Bin 0 -> 83 bytes
 ..._tables-legacy_ka_clust_compact-ka-1-Data.db | Bin 0 -> 5277 bytes
 ...les-legacy_ka_clust_compact-ka-1-Digest.sha1 |   1 +
 ...ables-legacy_ka_clust_compact-ka-1-Filter.db | Bin 0 -> 24 bytes
 ...tables-legacy_ka_clust_compact-ka-1-Index.db | Bin 0 -> 157685 bytes
 ...s-legacy_ka_clust_compact-ka-1-Statistics.db | Bin 0 -> 6859 bytes
 ...bles-legacy_ka_clust_compact-ka-1-Summary.db | Bin 0 -> 83 bytes
 ..._tables-legacy_ka_clust_compact-ka-1-TOC.txt |   8 

[jira] [Created] (CASSANDRA-11367) sstableofflinerelevel should only add the correct amount of data to each level

[Marcus Eriksson (JIRA)]

Marcus Eriksson created CASSANDRA-11367:
---

 Summary: sstableofflinerelevel should only add the correct amount 
of data to each level
 Key: CASSANDRA-11367
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11367
 Project: Cassandra
  Issue Type: Improvement
Reporter: Marcus Eriksson
Priority: Minor


Currently sstableofflinerelevel keeps adding sstables to a level until it can't 
add anymore due to creating overlap or running out of sstables.

It should just add the correct amount of data to each level to avoid doing a 
lot of compaction on startup



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11375) COPY FROM fails when importing blob

[Stefania (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15201208#comment-15201208
 ] 

Stefania commented on CASSANDRA-11375:
--

The issue is only reproducible with Python 2.6, Python 2.7 is fine.

Python 2.6 is only supported in cassandra 2.1, so this patch should be 
committed to 2.1 only:

|[patch|https://github.com/stef1927/cassandra/commits/11375-2.1]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-11375-2.1-dtest/]|

[~pauloricardomg] would you mind reviewing?


> COPY FROM fails when importing blob
> ---
>
> Key: CASSANDRA-11375
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11375
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Philippe Thibaudeau
>Assignee: Stefania
> Fix For: 2.1.x
>
>
> When we try to COPY TO a table containing a blob, we get this error 
> COPY test.blobTable FROM '/tmp/test1.csv' WITH NULL='null' AND DELIMITER=',' 
> AND QUOTE='"';
> /opt/apache-cassandra-2.1.13.4/bin/../pylib/cqlshlib/copyutil.py:1602: 
> DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
> /opt/apache-cassandra-2.1.13.4/bin/../pylib/cqlshlib/copyutil.py:1850: 
> DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
> Failed to import 5 rows: ParseError - fromhex() argument 1 must be unicode, 
> not str -  given up without retries
> Failed to process 5 rows; failed rows written to import_test_blobTable.err
> Same COPY TO function worked fine with 2.1.9
> The csv is generated by doing a COPY FROM on the same table.
> Is there any work around this issue?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11375) COPY FROM fails when importing blob

[Paulo Motta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paulo Motta updated CASSANDRA-11375:

Reviewer: Paulo Motta

> COPY FROM fails when importing blob
> ---
>
> Key: CASSANDRA-11375
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11375
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Philippe Thibaudeau
>Assignee: Stefania
> Fix For: 2.1.x
>
>
> When we try to COPY TO a table containing a blob, we get this error 
> COPY test.blobTable FROM '/tmp/test1.csv' WITH NULL='null' AND DELIMITER=',' 
> AND QUOTE='"';
> /opt/apache-cassandra-2.1.13.4/bin/../pylib/cqlshlib/copyutil.py:1602: 
> DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
> /opt/apache-cassandra-2.1.13.4/bin/../pylib/cqlshlib/copyutil.py:1850: 
> DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
> Failed to import 5 rows: ParseError - fromhex() argument 1 must be unicode, 
> not str -  given up without retries
> Failed to process 5 rows; failed rows written to import_test_blobTable.err
> Same COPY TO function worked fine with 2.1.9
> The csv is generated by doing a COPY FROM on the same table.
> Is there any work around this issue?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11383) SASI index build leads to massive OOM

[Pavel Yaskevich (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202245#comment-15202245
 ] 

Pavel Yaskevich commented on CASSANDRA-11383:
-

[~doanduyhai] So it actually happens when existing sstable being indexed, it's 
size is 16G and segment sizes are about 30MB which looks totally fine to me, 
there might be a bug I introduced while porting SASI to 3.x. I don't actually 
need whole dataset in this case since it's OOM indexing a single sstable, can 
you please just share that one sstable instead which is just 16G data?

> SASI index build leads to massive OOM
> -
>
> Key: CASSANDRA-11383
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11383
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
> Environment: C* 3.4
>Reporter: DOAN DuyHai
> Attachments: CASSANDRA-11383.patch, new_system_log_CMS_8GB_OOM.log, 
> system.log_sasi_build_oom
>
>
> 13 bare metal machines
> - 6 cores CPU (12 HT)
> - 64Gb RAM
> - 4 SSD in RAID0
>  JVM settings:
> - G1 GC
> - Xms32G, Xmx32G
> Data set:
>  - ≈ 100Gb/per node
>  - 1.3 Tb cluster-wide
>  - ≈ 20Gb for all SASI indices
> C* settings:
> - concurrent_compactors: 1
> - compaction_throughput_mb_per_sec: 256
> - memtable_heap_space_in_mb: 2048
> - memtable_offheap_space_in_mb: 2048
> I created 9 SASI indices
>  - 8 indices with text field, NonTokenizingAnalyser,  PREFIX mode, 
> case-insensitive
>  - 1 index with numeric field, SPARSE mode
>  After a while, the nodes just gone OOM.
>  I attach log files. You can see a lot of GC happening while index segments 
> are flush to disk. At some point the node OOM ...
> /cc [~xedin]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11377) dtest failure in topology_test.TestTopology.crash_during_decommission_test

[Philip Thompson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Philip Thompson updated CASSANDRA-11377:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> dtest failure in topology_test.TestTopology.crash_during_decommission_test
> --
>
> Key: CASSANDRA-11377
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11377
> Project: Cassandra
>  Issue Type: Test
>Reporter: Jim Witschey
>Assignee: Philip Thompson
>  Labels: dtest
>
> This has flapped once on 3.0:
> http://cassci.datastax.com/job/cassandra-3.0_dtest/614/testReport/topology_test/TestTopology/crash_during_decommission_test
> and more frequently on trunk:
> http://cassci.datastax.com/job/trunk_dtest/1057/testReport/junit/topology_test/TestTopology/crash_during_decommission_test/
> Message and trace:
> {code}
> Error Message
> 15 Mar 2016 10:48:55 [node1] Missing: ['127.0.0.2.* now UP']:
> .
> See system.log for remainder
>  >> begin captured logging << 
> dtest: DEBUG: cluster ccm directory: /mnt/tmp/dtest-EJfTo3
> dtest: DEBUG: Custom init_config not found. Setting defaults.
> dtest: DEBUG: Done setting configuration options:
> {   'initial_token': None,
> 'num_tokens': '32',
> 'phi_convict_threshold': 5,
> 'range_request_timeout_in_ms': 1,
> 'read_request_timeout_in_ms': 1,
> 'request_timeout_in_ms': 1,
> 'truncate_request_timeout_in_ms': 1,
> 'write_request_timeout_in_ms': 1}
> dtest: DEBUG: Status as reported by node 127.0.0.2
> dtest: DEBUG: Datacenter: datacenter1
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  AddressLoad   Tokens   Owns (effective)  Host ID  
>  Rack
> UN  127.0.0.1  112.85 KB  32   64.1% 
> d7d836bd-a0ff-474c-b04c-2c00bd2fc636  rack1
> UN  127.0.0.2  127.25 KB  32   61.7% 
> 68c5a8c4-8986-40df-afb2-ee0849814618  rack1
> UN  127.0.0.3  112.68 KB  32   74.2% 
> 6dd52315-53f4-444d-bace-a8a0d7c2b34e  rack1
> dtest: DEBUG: Restarting node2
> dtest: DEBUG: 
> dtest: DEBUG: 
> - >> end captured logging << -
> Stacktrace
>   File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
>   File "/home/automaton/cassandra-dtest/tools.py", line 253, in wrapped
> f(obj)
>   File "/home/automaton/cassandra-dtest/topology_test.py", line 230, in 
> crash_during_decommission_test
> node2.start(wait_for_binary_proto=True)
>   File "/home/automaton/ccm/ccmlib/node.py", line 597, in start
> node.watch_log_for_alive(self, from_mark=mark)
>   File "/home/automaton/ccm/ccmlib/node.py", line 449, in watch_log_for_alive
> self.watch_log_for(tofind, from_mark=from_mark, timeout=timeout, 
> filename=filename)
>   File "/home/automaton/ccm/ccmlib/node.py", line 417, in watch_log_for
> raise TimeoutError(time.strftime("%d %b %Y %H:%M:%S", time.gmtime()) + " 
> [" + self.name + "] Missing: " + str([e.pattern for e in tofind]) + ":\n" + 
> reads[:50] + ".\nSee {} for remainder".format(filename))
> "15 Mar 2016 10:48:55 [node1] Missing: ['127.0.0.2.* now UP']:\n.\nSee 
> system.log for remainder\n >> begin captured logging << 
> \ndtest: DEBUG: cluster ccm directory: 
> /mnt/tmp/dtest-EJfTo3\ndtest: DEBUG: Custom init_config not found. Setting 
> defaults.\ndtest: DEBUG: Done setting configuration options:\n{   
> 'initial_token': None,\n'num_tokens': '32',\n'phi_convict_threshold': 
> 5,\n'range_request_timeout_in_ms': 1,\n
> 'read_request_timeout_in_ms': 1,\n'request_timeout_in_ms': 1,\n   
>  'truncate_request_timeout_in_ms': 1,\n'write_request_timeout_in_ms': 
> 1}\ndtest: DEBUG: Status as reported by node 127.0.0.2\ndtest: DEBUG: 
> Datacenter: datacenter1\n===\nStatus=Up/Down\n|/ 
> State=Normal/Leaving/Joining/Moving\n--  AddressLoad   Tokens   
> Owns (effective)  Host ID   Rack\nUN  127.0.0.1  
> 112.85 KB  32   64.1% 
> d7d836bd-a0ff-474c-b04c-2c00bd2fc636  rack1\nUN  127.0.0.2  127.25 KB  32 
>   61.7% 68c5a8c4-8986-40df-afb2-ee0849814618  rack1\nUN  
> 127.0.0.3  112.68 KB  32   74.2% 
> 6dd52315-53f4-444d-bace-a8a0d7c2b34e  rack1\n\n\ndtest: DEBUG: Restarting 
> node2\ndtest: DEBUG: \ndtest: DEBUG: \n- >> end captured 
> logging << -"
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11376) dtest failure in replace_address_test.TestReplaceAddress.resumable_replace_test

[Philip Thompson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Philip Thompson updated CASSANDRA-11376:

Status: Patch Available  (was: Open)

Fixed by https://github.com/riptano/cassandra-dtest/pull/870

> dtest failure in 
> replace_address_test.TestReplaceAddress.resumable_replace_test
> ---
>
> Key: CASSANDRA-11376
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11376
> Project: Cassandra
>  Issue Type: Test
>Reporter: Jim Witschey
>Assignee: Philip Thompson
>  Labels: dtest
>
> example failures:
> http://cassci.datastax.com/job/cassandra-2.2_dtest/lastCompletedBuild/testReport/replace_address_test/TestReplaceAddress/resumable_replace_test/history/
> http://cassci.datastax.com/job/cassandra-3.0_dtest/612/testReport/replace_address_test/TestReplaceAddress/resumable_replace_test/
> Seems to fail hard with the following error:
> {code}
> 16 Mar 2016 18:42:00 [node1] Missing: ['127.0.0.4.* now UP']:
> INFO  [HANDSHAKE-/127.0.0.4] 2016-03-16 18:40:03,1.
> See system.log for remainder
>  >> begin captured logging << 
> dtest: DEBUG: cluster ccm directory: /mnt/tmp/dtest-JdjudW
> dtest: DEBUG: Custom init_config not found. Setting defaults.
> dtest: DEBUG: Done setting configuration options:
> {   'initial_token': None,
> 'num_tokens': '32',
> 'phi_convict_threshold': 5,
> 'start_rpc': 'true'}
> dtest: DEBUG: Starting node 4 to replace node 3
> - >> end captured logging << -
> Stacktrace
>   File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
>   File "/home/automaton/cassandra-dtest/tools.py", line 253, in wrapped
> f(obj)
>   File "/home/automaton/cassandra-dtest/replace_address_test.py", line 253, 
> in resumable_replace_test
> node4.start(jvm_args=["-Dcassandra.replace_address_first_boot=127.0.0.3"])
>   File "/home/automaton/ccm/ccmlib/node.py", line 597, in start
> node.watch_log_for_alive(self, from_mark=mark)
>   File "/home/automaton/ccm/ccmlib/node.py", line 449, in watch_log_for_alive
> self.watch_log_for(tofind, from_mark=from_mark, timeout=timeout, 
> filename=filename)
>   File "/home/automaton/ccm/ccmlib/node.py", line 417, in watch_log_for
> raise TimeoutError(time.strftime("%d %b %Y %H:%M:%S", time.gmtime()) + " 
> [" + self.name + "] Missing: " + str([e.pattern for e in tofind]) + ":\n" + 
> reads[:50] + ".\nSee {} for remainder".format(filename))
> "16 Mar 2016 18:42:00 [node1] Missing: ['127.0.0.4.* now UP']:\nINFO  
> [HANDSHAKE-/127.0.0.4] 2016-03-16 18:40:03,1.\nSee system.log for 
> remainder\n >> begin captured logging << 
> \ndtest: DEBUG: cluster ccm directory: 
> /mnt/tmp/dtest-JdjudW\ndtest: DEBUG: Custom init_config not found. Setting 
> defaults.\ndtest: DEBUG: Done setting configuration options:\n{   
> 'initial_token': None,\n'num_tokens': '32',\n'phi_convict_threshold': 
> 5,\n'start_rpc': 'true'}\ndtest: DEBUG: Starting node 4 to replace node 
> 3\n- >> end captured logging << -"
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-2527) Add ability to snapshot data as input to hadoop jobs

[Brian Hess (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202252#comment-15202252
 ] 

 Brian Hess commented on CASSANDRA-2527:


[~iamaleksey] - this ticket isn't only applicable to Hadoop, but to any 
analytics engine, including Spark.  I think we should either reopen this ticket 
and retarget to Spark (though the same mechanism would be interesting to 
Hadoop, Flink, etc) or open a new one with the word "Spark" replacing the word 
"Hadoop", but otherwise the exact same idea.

> Add ability to snapshot data as input to hadoop jobs
> 
>
> Key: CASSANDRA-2527
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2527
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Jeremy Hanna
>Priority: Minor
>  Labels: hadoop
>
> It is desirable to have immutable inputs to hadoop jobs for the duration of 
> the job.  That way re-execution of individual tasks do not alter the output.  
> One way to accomplish this would be to snapshot the data that is used as 
> input to a job.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-6909) A way to expire columns without converting to tombstones

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-6909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko resolved CASSANDRA-6909.
--
Resolution: Duplicate

CASSANDRA-5546 is going to mostly address the problem from a different angle - 
closing the ticket as a dup of that.

> A way to expire columns without converting to tombstones
> 
>
> Key: CASSANDRA-6909
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6909
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Bartłomiej Romański
>
> Imagine the following scenario. 
> - You need to store some data knowing that you will need them only for 
> limited time (say 7 days).
> - After that you just don't care. You don't need them to be returned in the 
> queries, but if they are returned that's not a problem at all - you won't 
> look at them anyway.
> - You records are small. Row keys and column names are even longer than the 
> actual values (e.g. ints vs strings).
> - You reuse rows. You add some new columns to most of the rows every day or 
> two. This means that columns expire often, rows usually not.
> - You generate a lot of data and want to make sure that expired records do 
> not consume disk space for too long.
> Current TTL feature do not handle that situation well. When compaction 
> finally decides that it's worth to compact the given sstable it won't simply 
> get rid of expired columns. Instead it will transform them into tombstones. 
> In case of small values that's not a saving at all.
> Even if you set grace period to 0 tombstones cannot be removed too early 
> because some other sstable can still have values that should be "covered" by 
> this tombstone. 
> You can get rid of tombstone only in two cases:
> - it's a major compaction (never happens with LCS, requires a lot of space in 
> STCS)
> - bloom filters tell you that there are no others sstable with this row key
> The second option is not common if you usually have multiple columns in a 
> single row that was written not at once. It's a great chance you'll have your 
> row spread across multiple sstables. And from time to time a new ones are 
> generated. There's very little chance they'll all meet in one compaction at 
> some point. 
> What's funny, bloom filters returns true if there's a tombstone for the given 
> row in the given sstable. So you won't remove tombstones during compaction, 
> because there's some other tombstone in another sstable for that row :/
> After a while, you end up with a lot of tombstones (majority of your data) 
> and can do nothing about that.
> Now image that Cassandra knows that we just don't care about data older than 
> 7 days. 
> Firstly, it can simply drop such columns during compactions (without 
> converting them to tombstones or anything like that).
> Secondly, if it detects an sstable older than 7 days it can safely remove it 
> at all (it cannot contain any active data).
> These two *guarantee* that you data will be removed after 14 days (2xTTL). If 
> we do compaction after 7 days, expired data will be removed. If we not, whole 
> sstable will be removed after another 7 days.
> That's what I expected from CASSANDRA-3974, but it turned out to be a just 
> trivial, frontend feature. 
> I suggest to rethink this mechanism. I don't believe that it's a common 
> scenario that someone who sets TTL for whole CF need all this strong 
> guarantees that data will not reappear in the future in case of some issues 
> with consistency (that's why we need this whole mess with tombstones). 
> I believe common case with per-CF TTL is that you just want an efficient way 
> of recover you disk space (and improve reads performance by having less 
> sstables and less data in general).
> To work around this we currently periodically stop Cassandra, simply remove 
> too old sstables, and start it back. Works OK, but does not solve problem 
> fully (if tombstone is rewritten by compactions often, we will never remove 
> it).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-11383) SASI index build leads to massive OOM

[DOAN DuyHai (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202259#comment-15202259
 ] 

DOAN DuyHai edited comment on CASSANDRA-11383 at 3/18/16 10:11 PM:
---

[~xedin] 

OK, I'm trying to fetch the sstable with its data files

In the meantime, I just re-create one index as shown above using 
*max_compaction_flush_memory_in_mb* = 128 and SASI flushes thousands of index 
files (of ~16kb each) and eventually the server dies maybe because too many 
file handles

{noformat}
INFO  [SASI-General:1] 2016-03-18 22:45:37,480 PerSSTableIndexWriter.java:258 - 
Flushed index segment 
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-SI_resource_period_end_month_int_idx.db_130370,
 took 0 ms.
INFO  [SASI-General:1] 2016-03-18 22:45:37,581 PerSSTableIndexWriter.java:258 - 
Flushed index segment 
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-SI_resource_period_end_month_int_idx.db_130371,
 took 101 ms.
ERROR [SASI-General:1] 2016-03-18 22:45:37,582 CassandraDaemon.java:195 - 
Exception in thread Thread[SASI-General:1,5,main]
org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
{noformat}


was (Author: doanduyhai):
[~xedin] 

OK, I'm trying to fetch the sstable with its data files

In the meantime, I just re-create one index as shown above using * 
max_compaction_flush_memory_in_mb * = 128 and SASI flushes thousands of index 
files and eventually the server dies maybe because too many file handles

{noformat}
INFO  [SASI-General:1] 2016-03-18 22:45:37,480 PerSSTableIndexWriter.java:258 - 
Flushed index segment 
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-SI_resource_period_end_month_int_idx.db_130370,
 took 0 ms.
INFO  [SASI-General:1] 2016-03-18 22:45:37,581 PerSSTableIndexWriter.java:258 - 
Flushed index segment 
/home/cassandra/data/sharon/resource_bench-4d065db0ebbc11e5995bd129cfce5717/ma-1831-big-SI_resource_period_end_month_int_idx.db_130371,
 took 101 ms.
ERROR [SASI-General:1] 2016-03-18 22:45:37,582 CassandraDaemon.java:195 - 
Exception in thread Thread[SASI-General:1,5,main]
org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
{noformat}

> SASI index build leads to massive OOM
> -
>
> Key: CASSANDRA-11383
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11383
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
> Environment: C* 3.4
>Reporter: DOAN DuyHai
> Attachments: CASSANDRA-11383.patch, new_system_log_CMS_8GB_OOM.log, 
> system.log_sasi_build_oom
>
>
> 13 bare metal machines
> - 6 cores CPU (12 HT)
> - 64Gb RAM
> - 4 SSD in RAID0
>  JVM settings:
> - G1 GC
> - Xms32G, Xmx32G
> Data set:
>  - ≈ 100Gb/per node
>  - 1.3 Tb cluster-wide
>  - ≈ 20Gb for all SASI indices
> C* settings:
> - concurrent_compactors: 1
> - compaction_throughput_mb_per_sec: 256
> - memtable_heap_space_in_mb: 2048
> - memtable_offheap_space_in_mb: 2048
> I created 9 SASI indices
>  - 8 indices with text field, NonTokenizingAnalyser,  PREFIX mode, 
> case-insensitive
>  - 1 index with numeric field, SPARSE mode
>  After a while, the nodes just gone OOM.
>  I attach log files. You can see a lot of GC happening while index segments 
> are flush to disk. At some point the node OOM ...
> /cc [~xedin]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11383) SASI index build leads to massive OOM

[DOAN DuyHai (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202353#comment-15202353
 ] 

DOAN DuyHai commented on CASSANDRA-11383:
-

[~xedin] 

Upload is on the way, here is the Google Drive folder for the data + schema + 
C* config: 
https://drive.google.com/folderview?id=0B6wR2aj4Cb6wdm03TFZtcXllX2M=sharing

 The big *Data* file (23Gb) is being uploaded, it will be available in ≈ 
45mins. I hope you have fiber optic to download it ...

> SASI index build leads to massive OOM
> -
>
> Key: CASSANDRA-11383
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11383
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
> Environment: C* 3.4
>Reporter: DOAN DuyHai
> Attachments: CASSANDRA-11383.patch, new_system_log_CMS_8GB_OOM.log, 
> system.log_sasi_build_oom
>
>
> 13 bare metal machines
> - 6 cores CPU (12 HT)
> - 64Gb RAM
> - 4 SSD in RAID0
>  JVM settings:
> - G1 GC
> - Xms32G, Xmx32G
> Data set:
>  - ≈ 100Gb/per node
>  - 1.3 Tb cluster-wide
>  - ≈ 20Gb for all SASI indices
> C* settings:
> - concurrent_compactors: 1
> - compaction_throughput_mb_per_sec: 256
> - memtable_heap_space_in_mb: 2048
> - memtable_offheap_space_in_mb: 2048
> I created 9 SASI indices
>  - 8 indices with text field, NonTokenizingAnalyser,  PREFIX mode, 
> case-insensitive
>  - 1 index with numeric field, SPARSE mode
>  After a while, the nodes just gone OOM.
>  I attach log files. You can see a lot of GC happening while index segments 
> are flush to disk. At some point the node OOM ...
> /cc [~xedin]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11332) nodes connect to themselves when NTS is used

[Branimir Lambov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Branimir Lambov updated CASSANDRA-11332:

Status: Patch Available  (was: Open)

Patch adding local address to {{PropertyFileSnitch}}:

|[code trunk|https://github.com/blambov/cassandra/tree/11332]|[utest 
|http://cassci.datastax.com/job/blambov-11332-testall/]|[dtest|http://cassci.datastax.com/job/blambov-9421-dtest/]|

This is an LHF-hackathon fix to the concrete problem, I think we should 
consider a more thorough investigation of where and how we use local addresses 
in token/rack/dc/etc. lookups.

> nodes connect to themselves when NTS is used
> 
>
> Key: CASSANDRA-11332
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11332
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Brandon Williams
>Assignee: Branimir Lambov
> Fix For: 2.1.x
>
>
> I tested this with both the simple snitch and PFS.  It's quite easy to repro, 
> setup a cluster, start it.  Mine looks like this:
> {noformat}
> tcp0  0 10.208.8.123:48003  10.208.8.63:7000
> ESTABLISHED 26254/java
> tcp0  0 10.208.8.123:7000   10.208.8.63:40215   
> ESTABLISHED 26254/java
> tcp0  0 10.208.8.123:9  10.208.35.225:7000  
> ESTABLISHED 26254/java
> tcp0  0 10.208.8.123:33498  10.208.8.63:7000
> ESTABLISHED 26254/java
> tcp0  0 10.208.8.123:7000   10.208.35.225:52530 
> ESTABLISHED 26254/java
> tcp0  0 10.208.8.123:7000   10.208.35.225:53674 
> ESTABLISHED 26254/java
> tcp0  0 10.208.8.123:40846  10.208.35.225:7000  
> ESTABLISHED 26254/java
> tcp0  0 10.208.8.123:7000   10.208.8.63:48880   
> ESTABLISHED 26254/java
> {noformat}
> No problems so far.  Now create a keyspace using NTS with an rf of 3, and 
> perform some writes.  Now it looks like this:
> {noformat}
> tcp0  0 10.208.8.123:48003  10.208.8.63:7000
> ESTABLISHED 26254/java  
> tcp0  0 10.208.8.123:7000   10.208.8.123:35024  
> ESTABLISHED 26254/java  
> tcp0  0 10.208.8.123:35024  10.208.8.123:7000   
> ESTABLISHED 26254/java  
> tcp0  0 10.208.8.123:47212  10.208.8.123:7000   
> ESTABLISHED 26254/java  
> tcp0  0 10.208.8.123:7000   10.208.8.63:40215   
> ESTABLISHED 26254/java  
> tcp0  0 10.208.8.123:9  10.208.35.225:7000  
> ESTABLISHED 26254/java  
> tcp0  0 10.208.8.123:33498  10.208.8.63:7000
> ESTABLISHED 26254/java  
> tcp0  0 10.208.8.123:7000   10.208.35.225:52530 
> ESTABLISHED 26254/java  
> tcp0  0 10.208.8.123:7000   10.208.35.225:53674 
> ESTABLISHED 26254/java  
> tcp0  0 10.208.8.123:7000   10.208.8.123:47212  
> ESTABLISHED 26254/java  
> tcp0  0 10.208.8.123:40846  10.208.35.225:7000  
> ESTABLISHED 26254/java  
> tcp0  0 10.208.8.123:7000   10.208.8.63:48880   
> ESTABLISHED 26254/java  
> {noformat}
> I can't think of any reason for a node to connect to itself, and this can 
> cause problems with PFS where you might only define the broadcast addresses, 
> but now you need the internal addresses too because the node will need to 
> look itself up when connecting to itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11383) SASI index build leads to massive OOM

[DOAN DuyHai (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202262#comment-15202262
 ] 

DOAN DuyHai commented on CASSANDRA-11383:
-

[~jkrupan]

 Below is the schema:

{code:sql}
create table if not exists sharon.resource_bench ( 
 dsr_id uuid,
 rel_seq bigint,
 seq bigint,
 dsp_code varchar,
 model_code varchar,
 media_code varchar,
 transfer_code varchar,
 commercial_offer_code varchar,
 territory_code varchar,
 period_end_month_int int,
 authorized_societies_txt text,
 rel_type text,
 status text,
 dsp_release_code text,
 title text,
 contributors_name list,
 unic_work text,
 paying_net_qty bigint,
PRIMARY KEY ((dsr_id, rel_seq), seq)
) WITH CLUSTERING ORDER BY (seq ASC); 
{code:sql}

> SASI index build leads to massive OOM
> -
>
> Key: CASSANDRA-11383
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11383
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
> Environment: C* 3.4
>Reporter: DOAN DuyHai
> Attachments: CASSANDRA-11383.patch, new_system_log_CMS_8GB_OOM.log, 
> system.log_sasi_build_oom
>
>
> 13 bare metal machines
> - 6 cores CPU (12 HT)
> - 64Gb RAM
> - 4 SSD in RAID0
>  JVM settings:
> - G1 GC
> - Xms32G, Xmx32G
> Data set:
>  - ≈ 100Gb/per node
>  - 1.3 Tb cluster-wide
>  - ≈ 20Gb for all SASI indices
> C* settings:
> - concurrent_compactors: 1
> - compaction_throughput_mb_per_sec: 256
> - memtable_heap_space_in_mb: 2048
> - memtable_offheap_space_in_mb: 2048
> I created 9 SASI indices
>  - 8 indices with text field, NonTokenizingAnalyser,  PREFIX mode, 
> case-insensitive
>  - 1 index with numeric field, SPARSE mode
>  After a while, the nodes just gone OOM.
>  I attach log files. You can see a lot of GC happening while index segments 
> are flush to disk. At some point the node OOM ...
> /cc [~xedin]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-11383) SASI index build leads to massive OOM

[DOAN DuyHai (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202262#comment-15202262
 ] 

DOAN DuyHai edited comment on CASSANDRA-11383 at 3/18/16 10:13 PM:
---

[~jkrupan]

 Below is the schema:

{noformat}
create table if not exists sharon.resource_bench ( 
 dsr_id uuid,
 rel_seq bigint,
 seq bigint,
 dsp_code varchar,
 model_code varchar,
 media_code varchar,
 transfer_code varchar,
 commercial_offer_code varchar,
 territory_code varchar,
 period_end_month_int int,
 authorized_societies_txt text,
 rel_type text,
 status text,
 dsp_release_code text,
 title text,
 contributors_name list,
 unic_work text,
 paying_net_qty bigint,
PRIMARY KEY ((dsr_id, rel_seq), seq)
) WITH CLUSTERING ORDER BY (seq ASC); 
{noformat}


was (Author: doanduyhai):
[~jkrupan]

 Below is the schema:

{code:sql}
create table if not exists sharon.resource_bench ( 
 dsr_id uuid,
 rel_seq bigint,
 seq bigint,
 dsp_code varchar,
 model_code varchar,
 media_code varchar,
 transfer_code varchar,
 commercial_offer_code varchar,
 territory_code varchar,
 period_end_month_int int,
 authorized_societies_txt text,
 rel_type text,
 status text,
 dsp_release_code text,
 title text,
 contributors_name list,
 unic_work text,
 paying_net_qty bigint,
PRIMARY KEY ((dsr_id, rel_seq), seq)
) WITH CLUSTERING ORDER BY (seq ASC); 
{code:sql}

> SASI index build leads to massive OOM
> -
>
> Key: CASSANDRA-11383
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11383
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
> Environment: C* 3.4
>Reporter: DOAN DuyHai
> Attachments: CASSANDRA-11383.patch, new_system_log_CMS_8GB_OOM.log, 
> system.log_sasi_build_oom
>
>
> 13 bare metal machines
> - 6 cores CPU (12 HT)
> - 64Gb RAM
> - 4 SSD in RAID0
>  JVM settings:
> - G1 GC
> - Xms32G, Xmx32G
> Data set:
>  - ≈ 100Gb/per node
>  - 1.3 Tb cluster-wide
>  - ≈ 20Gb for all SASI indices
> C* settings:
> - concurrent_compactors: 1
> - compaction_throughput_mb_per_sec: 256
> - memtable_heap_space_in_mb: 2048
> - memtable_offheap_space_in_mb: 2048
> I created 9 SASI indices
>  - 8 indices with text field, NonTokenizingAnalyser,  PREFIX mode, 
> case-insensitive
>  - 1 index with numeric field, SPARSE mode
>  After a while, the nodes just gone OOM.
>  I attach log files. You can see a lot of GC happening while index segments 
> are flush to disk. At some point the node OOM ...
> /cc [~xedin]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11383) SASI index build leads to massive OOM

[DOAN DuyHai (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202070#comment-15202070
 ] 

DOAN DuyHai commented on CASSANDRA-11383:
-

I try to create indices ONE by ONE, but same result.

new JVM settings:
 - CMS
 - Xms8G, Xmx8G

after some time, the node goes long Old GC and then crashes:

{noformat}
632936; Par Survivor Space: 125829120 -> 120349752
WARN  [Service Thread] 2016-03-18 21:02:45,237 GCInspector.java:282 - 
ConcurrentMarkSweep GC in 10417ms.  Par Eden Space: 1006632952 -> 1006632928; 
Par Survivor Space: 125829120 -> 120921320
WARN  [Service Thread] 2016-03-18 21:02:56,969 GCInspector.java:282 - 
ConcurrentMarkSweep GC in 11675ms.  Par Survivor Space: 125829112 -> 121748224
WARN  [Service Thread] 2016-03-18 21:03:07,359 GCInspector.java:282 - 
ConcurrentMarkSweep GC in 10327ms.  CMS Old Gen: 7331643344 -> 7331643392; Par 
Eden Space: 1006632960 -> 1006632864; Par Survivor Space: 125828928 -> 122147720
WARN  [Service Thread] 2016-03-18 21:03:50,019 GCInspector.java:282 - 
ConcurrentMarkSweep GC in 42574ms.  CMS Old Gen: 7331643392 -> 7331643368; Par 
Eden Space: 1006632960 -> 1006632824; Par Survivor Space: 125829120 -> 122651640
WARN  [Service Thread] 2016-03-18 21:05:01,704 GCInspector.java:282 - 
ConcurrentMarkSweep GC in 71592ms.  Par Eden Space: 1006632960 -> 1006632928; 
Par Survivor Space: 125829112 -> 123069400
{noformat}

> SASI index build leads to massive OOM
> -
>
> Key: CASSANDRA-11383
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11383
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
> Environment: C* 3.4
>Reporter: DOAN DuyHai
> Attachments: system.log_sasi_build_oom
>
>
> 13 bare metal machines
> - 6 cores CPU (12 HT)
> - 64Gb RAM
> - 4 SSD in RAID0
>  JVM settings:
> - G1 GC
> - Xms32G, Xmx32G
> Data set:
>  - ≈ 100Gb/per node
>  - 1.3 Tb cluster-wide
>  - ≈ 20Gb for all SASI indices
> C* settings:
> - concurrent_compactors: 1
> - compaction_throughput_mb_per_sec: 256
> - memtable_heap_space_in_mb: 2048
> - memtable_offheap_space_in_mb: 2048
> I created 9 SASI indices
>  - 8 indices with text field, NonTokenizingAnalyser,  PREFIX mode, 
> case-insensitive
>  - 1 index with numeric field, SPARSE mode
>  After a while, the nodes just gone OOM.
>  I attach log files. You can see a lot of GC happening while index segments 
> are flush to disk. At some point the node OOM ...
> /cc [~xedin]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11230) dtest failure in consistency_test.TestConsistency.short_read_reversed_test

[Russ Hatch (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202523#comment-15202523
 ] 

Russ Hatch commented on CASSANDRA-11230:


trying to repro on 100 iterations here: 
http://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/24/

> dtest failure in consistency_test.TestConsistency.short_read_reversed_test
> --
>
> Key: CASSANDRA-11230
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11230
> Project: Cassandra
>  Issue Type: Test
>Reporter: Russ Hatch
>Assignee: Russ Hatch
>  Labels: dtest
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.2_dtest/524/testReport/consistency_test/TestConsistency/short_read_reversed_test
> Failed on CassCI build cassandra-2.2_dtest #524
> flaps intermittently on earlier builds, most recent symptom:
> {noformat}
> [Unavailable exception] message="Cannot achieve consistency level QUORUM" 
> info={'required_replicas': 2, 'alive_replicas': 1, 'consistency': 'QUORUM'}
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-7159) sstablemetadata command should print some more stuff

[Yuki Morishita (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15201583#comment-15201583
 ] 

Yuki Morishita commented on CASSANDRA-7159:
---

+1

> sstablemetadata command should print some more stuff
> 
>
> Key: CASSANDRA-7159
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7159
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Jeremiah Jordan
>Assignee: Marcus Eriksson
>Priority: Trivial
>  Labels: lhf
> Fix For: 3.x
>
> Attachments: 
> CASSANDRA-7159_-_sstablemetadata_command_should_print_some_more_stuff.patch
>
>
> It would be nice if the sstablemetadata command printed out some more of the 
> stuff we track.  Like the Min/Max column names and the min/max token in the 
> file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)