[jira] [Commented] (CASSANDRA-13038) 33% of compaction time spent in StreamingHistogram.update()

[Corentin Chary (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865188#comment-15865188
 ] 

Corentin Chary commented on CASSANDRA-13038:


The code and the remaining property looks good to me.
The code of the benchmark could probably be slightly refactored but that's not 
really a big deal.

> 33% of compaction time spent in StreamingHistogram.update()
> ---
>
> Key: CASSANDRA-13038
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13038
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Corentin Chary
>Assignee: Jeff Jirsa
> Attachments: compaction-speedup.patch, 
> compaction-streaminghistrogram.png, profiler-snapshot.nps
>
>
> With the following table, that contains a *lot* of cells: 
> {code}
> CREATE TABLE biggraphite.datapoints_11520p_60s (
> metric uuid,
> time_start_ms bigint,
> offset smallint,
> count int,
> value double,
> PRIMARY KEY ((metric, time_start_ms), offset)
> ) WITH CLUSTERING ORDER BY (offset DESC);
> AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy', 
> 'compaction_window_size': '6', 'compaction_window_unit': 'HOURS', 
> 'max_threshold': '32', 'min_threshold': '6'}
> Keyspace : biggraphite
> Read Count: 1822
> Read Latency: 1.8870054884742042 ms.
> Write Count: 2212271647
> Write Latency: 0.027705127678653473 ms.
> Pending Flushes: 0
> Table: datapoints_11520p_60s
> SSTable count: 47
> Space used (live): 300417555945
> Space used (total): 303147395017
> Space used by snapshots (total): 0
> Off heap memory used (total): 207453042
> SSTable Compression Ratio: 0.4955200053039823
> Number of keys (estimate): 16343723
> Memtable cell count: 220576
> Memtable data size: 17115128
> Memtable off heap memory used: 0
> Memtable switch count: 2872
> Local read count: 0
> Local read latency: NaN ms
> Local write count: 1103167888
> Local write latency: 0.025 ms
> Pending flushes: 0
> Percent repaired: 0.0
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.0
> Bloom filter space used: 105118296
> Bloom filter off heap memory used: 106547192
> Index summary off heap memory used: 27730962
> Compression metadata off heap memory used: 73174888
> Compacted partition minimum bytes: 61
> Compacted partition maximum bytes: 51012
> Compacted partition mean bytes: 7899
> Average live cells per slice (last five minutes): NaN
> Maximum live cells per slice (last five minutes): 0
> Average tombstones per slice (last five minutes): NaN
> Maximum tombstones per slice (last five minutes): 0
> Dropped Mutations: 0
> {code}
> It looks like a good chunk of the compaction time is lost in 
> StreamingHistogram.update() (which is used to store the estimated tombstone 
> drop times).
> This could be caused by a huge number of different deletion times which would 
> makes the bin huge but it this histogram should be capped to 100 keys. It's 
> more likely caused by the huge number of cells.
> A simple solutions could be to only take into accounts part of the cells, the 
> fact the this table has a TWCS also gives us an additional hint that sampling 
> deletion times would be fine.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (CASSANDRA-13038) 33% of compaction time spent in StreamingHistogram.update()

[Corentin Chary (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865188#comment-15865188
 ] 

Corentin Chary edited comment on CASSANDRA-13038 at 2/14/17 6:41 AM:
-

The code and the remaining property looks good to me.
The code of the benchmark could probably be slightly refactored but that's not 
really a big deal.
Thanks for doing it !


was (Author: iksaif):
The code and the remaining property looks good to me.
The code of the benchmark could probably be slightly refactored but that's not 
really a big deal.

> 33% of compaction time spent in StreamingHistogram.update()
> ---
>
> Key: CASSANDRA-13038
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13038
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Corentin Chary
>Assignee: Jeff Jirsa
> Attachments: compaction-speedup.patch, 
> compaction-streaminghistrogram.png, profiler-snapshot.nps
>
>
> With the following table, that contains a *lot* of cells: 
> {code}
> CREATE TABLE biggraphite.datapoints_11520p_60s (
> metric uuid,
> time_start_ms bigint,
> offset smallint,
> count int,
> value double,
> PRIMARY KEY ((metric, time_start_ms), offset)
> ) WITH CLUSTERING ORDER BY (offset DESC);
> AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy', 
> 'compaction_window_size': '6', 'compaction_window_unit': 'HOURS', 
> 'max_threshold': '32', 'min_threshold': '6'}
> Keyspace : biggraphite
> Read Count: 1822
> Read Latency: 1.8870054884742042 ms.
> Write Count: 2212271647
> Write Latency: 0.027705127678653473 ms.
> Pending Flushes: 0
> Table: datapoints_11520p_60s
> SSTable count: 47
> Space used (live): 300417555945
> Space used (total): 303147395017
> Space used by snapshots (total): 0
> Off heap memory used (total): 207453042
> SSTable Compression Ratio: 0.4955200053039823
> Number of keys (estimate): 16343723
> Memtable cell count: 220576
> Memtable data size: 17115128
> Memtable off heap memory used: 0
> Memtable switch count: 2872
> Local read count: 0
> Local read latency: NaN ms
> Local write count: 1103167888
> Local write latency: 0.025 ms
> Pending flushes: 0
> Percent repaired: 0.0
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.0
> Bloom filter space used: 105118296
> Bloom filter off heap memory used: 106547192
> Index summary off heap memory used: 27730962
> Compression metadata off heap memory used: 73174888
> Compacted partition minimum bytes: 61
> Compacted partition maximum bytes: 51012
> Compacted partition mean bytes: 7899
> Average live cells per slice (last five minutes): NaN
> Maximum live cells per slice (last five minutes): 0
> Average tombstones per slice (last five minutes): NaN
> Maximum tombstones per slice (last five minutes): 0
> Dropped Mutations: 0
> {code}
> It looks like a good chunk of the compaction time is lost in 
> StreamingHistogram.update() (which is used to store the estimated tombstone 
> drop times).
> This could be caused by a huge number of different deletion times which would 
> makes the bin huge but it this histogram should be capped to 100 keys. It's 
> more likely caused by the huge number of cells.
> A simple solutions could be to only take into accounts part of the cells, the 
> fact the this table has a TWCS also gives us an additional hint that sampling 
> deletion times would be fine.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13038) 33% of compaction time spent in StreamingHistogram.update()

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865012#comment-15865012
 ] 

Jeff Jirsa commented on CASSANDRA-13038:


No, I didn't mean to override Object.finalize, and I've renamed that to avoid 
doing so. That was just carelessness on my part, and I can't believe I did it.

I've added spool.hashCode() to StreamingHistogram.hashCode().

I removed 2 of the 3 system properties. The third I left, primarily because the 
compaction tests assume the old behavior. I don't disagree with Sylvain about 
config bloat, though. I've force pushed fixes to the other two things while I 
think about how I want to handle this last system property.

> 33% of compaction time spent in StreamingHistogram.update()
> ---
>
> Key: CASSANDRA-13038
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13038
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Corentin Chary
>Assignee: Jeff Jirsa
> Attachments: compaction-speedup.patch, 
> compaction-streaminghistrogram.png, profiler-snapshot.nps
>
>
> With the following table, that contains a *lot* of cells: 
> {code}
> CREATE TABLE biggraphite.datapoints_11520p_60s (
> metric uuid,
> time_start_ms bigint,
> offset smallint,
> count int,
> value double,
> PRIMARY KEY ((metric, time_start_ms), offset)
> ) WITH CLUSTERING ORDER BY (offset DESC);
> AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy', 
> 'compaction_window_size': '6', 'compaction_window_unit': 'HOURS', 
> 'max_threshold': '32', 'min_threshold': '6'}
> Keyspace : biggraphite
> Read Count: 1822
> Read Latency: 1.8870054884742042 ms.
> Write Count: 2212271647
> Write Latency: 0.027705127678653473 ms.
> Pending Flushes: 0
> Table: datapoints_11520p_60s
> SSTable count: 47
> Space used (live): 300417555945
> Space used (total): 303147395017
> Space used by snapshots (total): 0
> Off heap memory used (total): 207453042
> SSTable Compression Ratio: 0.4955200053039823
> Number of keys (estimate): 16343723
> Memtable cell count: 220576
> Memtable data size: 17115128
> Memtable off heap memory used: 0
> Memtable switch count: 2872
> Local read count: 0
> Local read latency: NaN ms
> Local write count: 1103167888
> Local write latency: 0.025 ms
> Pending flushes: 0
> Percent repaired: 0.0
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.0
> Bloom filter space used: 105118296
> Bloom filter off heap memory used: 106547192
> Index summary off heap memory used: 27730962
> Compression metadata off heap memory used: 73174888
> Compacted partition minimum bytes: 61
> Compacted partition maximum bytes: 51012
> Compacted partition mean bytes: 7899
> Average live cells per slice (last five minutes): NaN
> Maximum live cells per slice (last five minutes): 0
> Average tombstones per slice (last five minutes): NaN
> Maximum tombstones per slice (last five minutes): 0
> Dropped Mutations: 0
> {code}
> It looks like a good chunk of the compaction time is lost in 
> StreamingHistogram.update() (which is used to store the estimated tombstone 
> drop times).
> This could be caused by a huge number of different deletion times which would 
> makes the bin huge but it this histogram should be capped to 100 keys. It's 
> more likely caused by the huge number of cells.
> A simple solutions could be to only take into accounts part of the cells, the 
> fact the this table has a TWCS also gives us an additional hint that sampling 
> deletion times would be fine.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-8457) nio MessagingService

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864943#comment-15864943
 ] 

Jason Brown commented on CASSANDRA-8457:


One other thing [~aweisberg] and I discussed was large messages and how they 
would be handled. In the current 8457 implementation, we will simply allocate a 
buffer of the {{serializedSize}}. If a message is supposed to be 50Mb, we'll 
allocate that and roll on. With enough large messages, sent to enough peers, we 
could OOM or get into some serious memory pressure problems. For comparison, 
the existing {{OutboundTcpConnection}} uses a {{BufferedOutputStream}}, which 
is defaulted to 64k, which we constantly reuse and never need to realloc.

Thus, I propose to bring back the {{SwappingByteBufDataOutputStreamPlus}} that 
I had in an earlier commit. To recap, the basic idea is provide a 
{{DataOutputPlus}} that has a backing {{ByteBuffer}} that is written to, and 
when it is filled, it is written to the netty context and flushed, then 
allocate a new buffer for more writes - kinda similar to a 
{{BufferedOutputStream}}, but replacing the backing buffer when full. Bringing 
this idea back is also what underpins one of the major performance things I 
wanted to address: buffering up smaller messages into one buffer to avoid going 
back to the netty allocator for every tiny buffer we might need - think 
Mutation acks.

We definitely need to address the large buffer issue, and I wouldn't mind 
knocking out the "buffering small messages" as it's really the same code (that 
I've written before). wdyt [~slebresne] and [~aweisberg]?


> nio MessagingService
> 
>
> Key: CASSANDRA-8457
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8457
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Jonathan Ellis
>Assignee: Jason Brown
>Priority: Minor
>  Labels: netty, performance
> Fix For: 4.x
>
>
> Thread-per-peer (actually two each incoming and outbound) is a big 
> contributor to context switching, especially for larger clusters.  Let's look 
> at switching to nio, possibly via Netty.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13217) minor bugs related to CASSANDRA-9143

[Blake Eggleston (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Blake Eggleston updated CASSANDRA-13217:

   Resolution: Fixed
Fix Version/s: 4.0
   Status: Resolved  (was: Patch Available)

> minor bugs related to CASSANDRA-9143
> 
>
> Key: CASSANDRA-13217
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13217
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Minor
> Fix For: 4.0
>
>
> We found a few minor bugs found in an internal review:
> * -incorrect log argument order 
> [here|https://github.com/apache/cassandra/blob/edcbef3e343778b4d5affe019f64c89da2a13aa2/src/java/org/apache/cassandra/streaming/compress/CompressedStreamReader.java#L75]-
> * {{SSTableReader#intersects}} should use Bounds, not Range 
> ([here|https://github.com/apache/cassandra/blob/edcbef3e343778b4d5affe019f64c89da2a13aa2/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L1761])
> * {{CompactionStrategyManager#validateForCompaction}} doesn't prevent 
> sstables from different repair session from being compacted together 
> [here|https://github.com/apache/cassandra/blob/edcbef3e343778b4d5affe019f64c89da2a13aa2/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L1761]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

cassandra git commit: fix minor bugs related to CASSANDRA-9143 patch by Blake Eggleston; reviewed by Jeff Jirsa for CASSANDRA-13217

[bdeggleston]

Repository: cassandra
Updated Branches:
  refs/heads/trunk edcbef3e3 -> 3f3db2d40


fix minor bugs related to CASSANDRA-9143
patch by Blake Eggleston; reviewed by Jeff Jirsa for CASSANDRA-13217


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3f3db2d4
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3f3db2d4
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3f3db2d4

Branch: refs/heads/trunk
Commit: 3f3db2d40d6b5edbf079b917953a30bcc1209d25
Parents: edcbef3
Author: Blake Eggleston 
Authored: Mon Feb 13 15:06:17 2017 -0800
Committer: Blake Eggleston 
Committed: Mon Feb 13 18:16:41 2017 -0800

--
 .../cassandra/db/compaction/CompactionStrategyManager.java  | 5 +++--
 .../org/apache/cassandra/io/sstable/format/SSTableReader.java   | 3 ++-
 2 files changed, 5 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/3f3db2d4/src/java/org/apache/cassandra/db/compaction/CompactionStrategyManager.java
--
diff --git 
a/src/java/org/apache/cassandra/db/compaction/CompactionStrategyManager.java 
b/src/java/org/apache/cassandra/db/compaction/CompactionStrategyManager.java
index 0ccdb49..9397708 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionStrategyManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionStrategyManager.java
@@ -840,14 +840,15 @@ public class CompactionStrategyManager implements 
INotificationConsumer
 boolean repaired = firstSSTable.isRepaired();
 int firstIndex = getCompactionStrategyIndex(cfs, directories, 
firstSSTable);
 boolean isPending = firstSSTable.isPendingRepair();
+UUID pendingRepair = firstSSTable.getSSTableMetadata().pendingRepair;
 for (SSTableReader sstable : input)
 {
 if (sstable.isRepaired() != repaired)
 throw new UnsupportedOperationException("You can't mix 
repaired and unrepaired data in a compaction");
 if (firstIndex != getCompactionStrategyIndex(cfs, directories, 
sstable))
 throw new UnsupportedOperationException("You can't mix 
sstables from different directories in a compaction");
-if (isPending != sstable.isPendingRepair())
-throw new UnsupportedOperationException("You can't compact 
sstables pending for repair with non-pending ones");
+if (isPending && 
!pendingRepair.equals(sstable.getSSTableMetadata().pendingRepair))
+throw new UnsupportedOperationException("You can't compact 
sstables from different pending repair sessions");
 }
 }
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/3f3db2d4/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java 
b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
index 716b27d..f9f6f5f 100644
--- a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
+++ b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
@@ -50,6 +50,7 @@ import org.apache.cassandra.db.filter.ColumnFilter;
 import org.apache.cassandra.db.rows.EncodingStats;
 import org.apache.cassandra.db.rows.UnfilteredRowIterator;
 import org.apache.cassandra.dht.AbstractBounds;
+import org.apache.cassandra.dht.Bounds;
 import org.apache.cassandra.dht.Range;
 import org.apache.cassandra.dht.Token;
 import org.apache.cassandra.io.FSError;
@@ -1758,7 +1759,7 @@ public abstract class SSTableReader extends SSTable 
implements SelfRefCounted> ranges)
 {
-Range range = new Range<>(first.getToken(), last.getToken());
+Bounds range = new Bounds<>(first.getToken(), last.getToken());
 return Iterables.any(ranges, r -> r.intersects(range));
 }
 

[jira] [Commented] (CASSANDRA-8457) nio MessagingService

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864888#comment-15864888
 ] 

Jason Brown commented on CASSANDRA-8457:


Thanks to [~aweisberg], caught another race condition in 
{{OutboundMessagingHandler}} and pushed. Needs to be updated when 
CASSANDRA-13090 is committed (probably today).

> nio MessagingService
> 
>
> Key: CASSANDRA-8457
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8457
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Jonathan Ellis
>Assignee: Jason Brown
>Priority: Minor
>  Labels: netty, performance
> Fix For: 4.x
>
>
> Thread-per-peer (actually two each incoming and outbound) is a big 
> contributor to context switching, especially for larger clusters.  Let's look 
> at switching to nio, possibly via Netty.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (CASSANDRA-13038) 33% of compaction time spent in StreamingHistogram.update()

[Nate McCall (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864782#comment-15864782
 ] 

Nate McCall edited comment on CASSANDRA-13038 at 2/14/17 1:08 AM:
--

[~jjirsa] Let's remove the {{System.setProperty}} invocations in the tests 
since you've removed those props and the subsequent commit. 

Should add {{spool.hashCode()}} to {{StreamingHistogram.hashCode()}} since we 
are using it in the {{equals()}} method. 

Did you intend to override {{Object.finalize()}}? We can't really rely on that 
for lifecycle. 


was (Author: zznate):
[~jjirsa] Do you need the {{System.setProperty}} invocations in the tests if 
you've removed those props? 

> 33% of compaction time spent in StreamingHistogram.update()
> ---
>
> Key: CASSANDRA-13038
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13038
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Corentin Chary
>Assignee: Jeff Jirsa
> Attachments: compaction-speedup.patch, 
> compaction-streaminghistrogram.png, profiler-snapshot.nps
>
>
> With the following table, that contains a *lot* of cells: 
> {code}
> CREATE TABLE biggraphite.datapoints_11520p_60s (
> metric uuid,
> time_start_ms bigint,
> offset smallint,
> count int,
> value double,
> PRIMARY KEY ((metric, time_start_ms), offset)
> ) WITH CLUSTERING ORDER BY (offset DESC);
> AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy', 
> 'compaction_window_size': '6', 'compaction_window_unit': 'HOURS', 
> 'max_threshold': '32', 'min_threshold': '6'}
> Keyspace : biggraphite
> Read Count: 1822
> Read Latency: 1.8870054884742042 ms.
> Write Count: 2212271647
> Write Latency: 0.027705127678653473 ms.
> Pending Flushes: 0
> Table: datapoints_11520p_60s
> SSTable count: 47
> Space used (live): 300417555945
> Space used (total): 303147395017
> Space used by snapshots (total): 0
> Off heap memory used (total): 207453042
> SSTable Compression Ratio: 0.4955200053039823
> Number of keys (estimate): 16343723
> Memtable cell count: 220576
> Memtable data size: 17115128
> Memtable off heap memory used: 0
> Memtable switch count: 2872
> Local read count: 0
> Local read latency: NaN ms
> Local write count: 1103167888
> Local write latency: 0.025 ms
> Pending flushes: 0
> Percent repaired: 0.0
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.0
> Bloom filter space used: 105118296
> Bloom filter off heap memory used: 106547192
> Index summary off heap memory used: 27730962
> Compression metadata off heap memory used: 73174888
> Compacted partition minimum bytes: 61
> Compacted partition maximum bytes: 51012
> Compacted partition mean bytes: 7899
> Average live cells per slice (last five minutes): NaN
> Maximum live cells per slice (last five minutes): 0
> Average tombstones per slice (last five minutes): NaN
> Maximum tombstones per slice (last five minutes): 0
> Dropped Mutations: 0
> {code}
> It looks like a good chunk of the compaction time is lost in 
> StreamingHistogram.update() (which is used to store the estimated tombstone 
> drop times).
> This could be caused by a huge number of different deletion times which would 
> makes the bin huge but it this histogram should be capped to 100 keys. It's 
> more likely caused by the huge number of cells.
> A simple solutions could be to only take into accounts part of the cells, the 
> fact the this table has a TWCS also gives us an additional hint that sampling 
> deletion times would be fine.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-12762) Cassandra 3.0.9 Fails both compact and repair without even debug logs

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-12762:
---
Description: 
After upgrading from 3.0.7 to 3.0.9, the following exception occurs when trying 
to run compaction (previous to the upgrade compaction worked fine):

{code}
error: 
(/home/circuitwatch/cassandra/data/circuitwatch/edgeTransitionByCircuitId-f5d33310024b11e5bb310d2316086bf7/mb-12063-big-Data.db):
 corruption detected, chunk at 345885546 of length 62024.
-- StackTrace --
org.apache.cassandra.io.compress.CorruptBlockException: 
(/home/circuitwatch/cassandra/data/circuitwatch/edgeTransitionByCircuitId-f5d33310024b11e5bb310d2316086bf7/mb-12063-big-Data.db):
 corruption detected, chunk at 345885546 of length 62024.
at 
org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap(CompressedRandomAccessReader.java:202)
at 
org.apache.cassandra.io.util.RandomAccessReader.reBuffer(RandomAccessReader.java:111)
at 
org.apache.cassandra.io.util.RebufferingInputStream.read(RebufferingInputStream.java:88)
at 
org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:66)
at 
org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
at 
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:404)
at 
org.apache.cassandra.db.marshal.AbstractType.readValue(AbstractType.java:406)
at 
org.apache.cassandra.db.rows.BufferCell$Serializer.deserialize(BufferCell.java:302)
at 
org.apache.cassandra.db.rows.UnfilteredSerializer.readSimpleColumn(UnfilteredSerializer.java:476)
at 
org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:454)
at 
org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:377)
at 
org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:87)
at 
org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:65)
at 
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.doCompute(SSTableIdentityIterator.java:123)
at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100)
at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30)
at 
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at 
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
at 
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
at 
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at 
org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:369)
at 
org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:189)
at 
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:158)
at 
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at 
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:509)
at 
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:369)
at 
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129)
at 
org.apache.cassandra.db.ColumnIndex$Builder.build(ColumnIndex.java:111)
at 
org.apache.cassandra.db.ColumnIndex.writeAndBuildIndex(ColumnIndex.java:52)
at 
org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:149)
at 
org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125)
at 
org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:57)
at 
org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109)
at 
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:183)
at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at 
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
at 
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
at 

[jira] [Updated] (CASSANDRA-12762) Cassandra 3.0.9 Fails both compact and repair without even debug logs

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-12762:
---
Description: 
After upgrading from 3.0.7 to 3.0.9, the following exception occurs when trying 
to run compaction (previous to the upgrade compaction worked fine):

{code}
error: 
(/home/circuitwatch/cassandra/data/circuitwatch/edgeTransitionByCircuitId-f5d33310024b11e5bb310d2316086bf7/mb-12063-big-Data.db):
 corruption detected, chunk at 345885546 of length 62024.
-- StackTrace --
org.apache.cassandra.io.compress.CorruptBlockException: 
(/home/circuitwatch/cassandra/data/circuitwatch/edgeTransitionByCircuitId-f5d33310024b11e5bb310d2316086bf7/mb-12063-big-Data.db):
 corruption detected, chunk at 345885546 of length 62024.
at 
org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap(CompressedRandomAccessReader.java:202)
at 
org.apache.cassandra.io.util.RandomAccessReader.reBuffer(RandomAccessReader.java:111)
at 
org.apache.cassandra.io.util.RebufferingInputStream.read(RebufferingInputStream.java:88)
at 
org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:66)
at 
org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
at 
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:404)
at 
org.apache.cassandra.db.marshal.AbstractType.readValue(AbstractType.java:406)
at 
org.apache.cassandra.db.rows.BufferCell$Serializer.deserialize(BufferCell.java:302)
at 
org.apache.cassandra.db.rows.UnfilteredSerializer.readSimpleColumn(UnfilteredSerializer.java:476)
at 
org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:454)
at 
org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:377)
at 
org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:87)
at 
org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:65)
at 
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.doCompute(SSTableIdentityIterator.java:123)
at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100)
at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30)
at 
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at 
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
at 
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
at 
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at 
org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:369)
at 
org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:189)
at 
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:158)
at 
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at 
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:509)
at 
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:369)
at 
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129)
at 
org.apache.cassandra.db.ColumnIndex$Builder.build(ColumnIndex.java:111)
at 
org.apache.cassandra.db.ColumnIndex.writeAndBuildIndex(ColumnIndex.java:52)
at 
org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:149)
at 
org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125)
at 
org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:57)
at 
org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109)
at 
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:183)
at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at 
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
at 
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
at 

[jira] [Commented] (CASSANDRA-13038) 33% of compaction time spent in StreamingHistogram.update()

[Nate McCall (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864782#comment-15864782
 ] 

Nate McCall commented on CASSANDRA-13038:
-

[~jjirsa] Do you need the {{System.setProperty}} invocations in the tests if 
you've removed those props? 

> 33% of compaction time spent in StreamingHistogram.update()
> ---
>
> Key: CASSANDRA-13038
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13038
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Corentin Chary
>Assignee: Jeff Jirsa
> Attachments: compaction-speedup.patch, 
> compaction-streaminghistrogram.png, profiler-snapshot.nps
>
>
> With the following table, that contains a *lot* of cells: 
> {code}
> CREATE TABLE biggraphite.datapoints_11520p_60s (
> metric uuid,
> time_start_ms bigint,
> offset smallint,
> count int,
> value double,
> PRIMARY KEY ((metric, time_start_ms), offset)
> ) WITH CLUSTERING ORDER BY (offset DESC);
> AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy', 
> 'compaction_window_size': '6', 'compaction_window_unit': 'HOURS', 
> 'max_threshold': '32', 'min_threshold': '6'}
> Keyspace : biggraphite
> Read Count: 1822
> Read Latency: 1.8870054884742042 ms.
> Write Count: 2212271647
> Write Latency: 0.027705127678653473 ms.
> Pending Flushes: 0
> Table: datapoints_11520p_60s
> SSTable count: 47
> Space used (live): 300417555945
> Space used (total): 303147395017
> Space used by snapshots (total): 0
> Off heap memory used (total): 207453042
> SSTable Compression Ratio: 0.4955200053039823
> Number of keys (estimate): 16343723
> Memtable cell count: 220576
> Memtable data size: 17115128
> Memtable off heap memory used: 0
> Memtable switch count: 2872
> Local read count: 0
> Local read latency: NaN ms
> Local write count: 1103167888
> Local write latency: 0.025 ms
> Pending flushes: 0
> Percent repaired: 0.0
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.0
> Bloom filter space used: 105118296
> Bloom filter off heap memory used: 106547192
> Index summary off heap memory used: 27730962
> Compression metadata off heap memory used: 73174888
> Compacted partition minimum bytes: 61
> Compacted partition maximum bytes: 51012
> Compacted partition mean bytes: 7899
> Average live cells per slice (last five minutes): NaN
> Maximum live cells per slice (last five minutes): 0
> Average tombstones per slice (last five minutes): NaN
> Maximum tombstones per slice (last five minutes): 0
> Dropped Mutations: 0
> {code}
> It looks like a good chunk of the compaction time is lost in 
> StreamingHistogram.update() (which is used to store the estimated tombstone 
> drop times).
> This could be caused by a huge number of different deletion times which would 
> makes the bin huge but it this histogram should be capped to 100 keys. It's 
> more likely caused by the huge number of cells.
> A simple solutions could be to only take into accounts part of the cells, the 
> fact the this table has a TWCS also gives us an additional hint that sampling 
> deletion times would be fine.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13217) minor bugs related to CASSANDRA-9143

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864694#comment-15864694
 ] 

Jeff Jirsa commented on CASSANDRA-13217:


The other 2 commits look good to me, +1


> minor bugs related to CASSANDRA-9143
> 
>
> Key: CASSANDRA-13217
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13217
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Minor
>
> We found a few minor bugs found in an internal review:
> * -incorrect log argument order 
> [here|https://github.com/apache/cassandra/blob/edcbef3e343778b4d5affe019f64c89da2a13aa2/src/java/org/apache/cassandra/streaming/compress/CompressedStreamReader.java#L75]-
> * {{SSTableReader#intersects}} should use Bounds, not Range 
> ([here|https://github.com/apache/cassandra/blob/edcbef3e343778b4d5affe019f64c89da2a13aa2/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L1761])
> * {{CompactionStrategyManager#validateForCompaction}} doesn't prevent 
> sstables from different repair session from being compacted together 
> [here|https://github.com/apache/cassandra/blob/edcbef3e343778b4d5affe019f64c89da2a13aa2/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L1761]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13217) minor bugs related to CASSANDRA-9143

[Blake Eggleston (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864683#comment-15864683
 ] 

Blake Eggleston commented on CASSANDRA-13217:
-

bq. Looks like e1dda71 ninja fixed the log ordering, so your flipping is wrong 
in this patch (because Dave already flipped it for you).

Removed the log order unfixing commit

> minor bugs related to CASSANDRA-9143
> 
>
> Key: CASSANDRA-13217
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13217
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Minor
>
> We found a few minor bugs found in an internal review:
> * -incorrect log argument order 
> [here|https://github.com/apache/cassandra/blob/edcbef3e343778b4d5affe019f64c89da2a13aa2/src/java/org/apache/cassandra/streaming/compress/CompressedStreamReader.java#L75]-
> * {{SSTableReader#intersects}} should use Bounds, not Range 
> ([here|https://github.com/apache/cassandra/blob/edcbef3e343778b4d5affe019f64c89da2a13aa2/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L1761])
> * {{CompactionStrategyManager#validateForCompaction}} doesn't prevent 
> sstables from different repair session from being compacted together 
> [here|https://github.com/apache/cassandra/blob/edcbef3e343778b4d5affe019f64c89da2a13aa2/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L1761]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (CASSANDRA-13217) minor bugs related to CASSANDRA-9143

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864681#comment-15864681
 ] 

Jeff Jirsa edited comment on CASSANDRA-13217 at 2/13/17 11:37 PM:
--

Looks like {{e1dda71}} ninja fixed the log ordering, so your log order fix 
isn't needed (because Dave already flipped it for you). 


was (Author: jjirsa):
Looks like {{e1dda71}} ninja fixed the log ordering, so your flipping is wrong 
in this patch (because Dave already flipped it for you). 

> minor bugs related to CASSANDRA-9143
> 
>
> Key: CASSANDRA-13217
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13217
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Minor
>
> We found a few minor bugs found in an internal review:
> * -incorrect log argument order 
> [here|https://github.com/apache/cassandra/blob/edcbef3e343778b4d5affe019f64c89da2a13aa2/src/java/org/apache/cassandra/streaming/compress/CompressedStreamReader.java#L75]-
> * {{SSTableReader#intersects}} should use Bounds, not Range 
> ([here|https://github.com/apache/cassandra/blob/edcbef3e343778b4d5affe019f64c89da2a13aa2/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L1761])
> * {{CompactionStrategyManager#validateForCompaction}} doesn't prevent 
> sstables from different repair session from being compacted together 
> [here|https://github.com/apache/cassandra/blob/edcbef3e343778b4d5affe019f64c89da2a13aa2/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L1761]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13217) minor bugs related to CASSANDRA-9143

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864681#comment-15864681
 ] 

Jeff Jirsa commented on CASSANDRA-13217:


Looks like {{e1dda71}} ninja fixed the log ordering, so your flipping is wrong 
in this patch (because Dave already flipped it for you). 

> minor bugs related to CASSANDRA-9143
> 
>
> Key: CASSANDRA-13217
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13217
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Minor
>
> We found a few minor bugs found in an internal review:
> * -incorrect log argument order 
> [here|https://github.com/apache/cassandra/blob/edcbef3e343778b4d5affe019f64c89da2a13aa2/src/java/org/apache/cassandra/streaming/compress/CompressedStreamReader.java#L75]-
> * {{SSTableReader#intersects}} should use Bounds, not Range 
> ([here|https://github.com/apache/cassandra/blob/edcbef3e343778b4d5affe019f64c89da2a13aa2/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L1761])
> * {{CompactionStrategyManager#validateForCompaction}} doesn't prevent 
> sstables from different repair session from being compacted together 
> [here|https://github.com/apache/cassandra/blob/edcbef3e343778b4d5affe019f64c89da2a13aa2/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L1761]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13217) minor bugs related to CASSANDRA-9143

[Blake Eggleston (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Blake Eggleston updated CASSANDRA-13217:

Description: 
We found a few minor bugs found in an internal review:
* -incorrect log argument order 
[here|https://github.com/apache/cassandra/blob/edcbef3e343778b4d5affe019f64c89da2a13aa2/src/java/org/apache/cassandra/streaming/compress/CompressedStreamReader.java#L75]-
* {{SSTableReader#intersects}} should use Bounds, not Range 
([here|https://github.com/apache/cassandra/blob/edcbef3e343778b4d5affe019f64c89da2a13aa2/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L1761])
* {{CompactionStrategyManager#validateForCompaction}} doesn't prevent sstables 
from different repair session from being compacted together 
[here|https://github.com/apache/cassandra/blob/edcbef3e343778b4d5affe019f64c89da2a13aa2/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L1761]

  was:
We found a few minor bugs found in an internal review:
* incorrect log argument order 
[here|https://github.com/apache/cassandra/blob/edcbef3e343778b4d5affe019f64c89da2a13aa2/src/java/org/apache/cassandra/streaming/compress/CompressedStreamReader.java#L75]
* {{SSTableReader#intersects}} should use Bounds, not Range 
([here|https://github.com/apache/cassandra/blob/edcbef3e343778b4d5affe019f64c89da2a13aa2/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L1761])
* {{CompactionStrategyManager#validateForCompaction}} doesn't prevent sstables 
from different repair session from being compacted together 
[here|https://github.com/apache/cassandra/blob/edcbef3e343778b4d5affe019f64c89da2a13aa2/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L1761]


> minor bugs related to CASSANDRA-9143
> 
>
> Key: CASSANDRA-13217
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13217
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Minor
>
> We found a few minor bugs found in an internal review:
> * -incorrect log argument order 
> [here|https://github.com/apache/cassandra/blob/edcbef3e343778b4d5affe019f64c89da2a13aa2/src/java/org/apache/cassandra/streaming/compress/CompressedStreamReader.java#L75]-
> * {{SSTableReader#intersects}} should use Bounds, not Range 
> ([here|https://github.com/apache/cassandra/blob/edcbef3e343778b4d5affe019f64c89da2a13aa2/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L1761])
> * {{CompactionStrategyManager#validateForCompaction}} doesn't prevent 
> sstables from different repair session from being compacted together 
> [here|https://github.com/apache/cassandra/blob/edcbef3e343778b4d5affe019f64c89da2a13aa2/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L1761]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13217) minor bugs related to CASSANDRA-9143

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-13217:
---
Reviewer: Jeff Jirsa

> minor bugs related to CASSANDRA-9143
> 
>
> Key: CASSANDRA-13217
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13217
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Minor
>
> We found a few minor bugs found in an internal review:
> * incorrect log argument order 
> [here|https://github.com/apache/cassandra/blob/edcbef3e343778b4d5affe019f64c89da2a13aa2/src/java/org/apache/cassandra/streaming/compress/CompressedStreamReader.java#L75]
> * {{SSTableReader#intersects}} should use Bounds, not Range 
> ([here|https://github.com/apache/cassandra/blob/edcbef3e343778b4d5affe019f64c89da2a13aa2/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L1761])
> * {{CompactionStrategyManager#validateForCompaction}} doesn't prevent 
> sstables from different repair session from being compacted together 
> [here|https://github.com/apache/cassandra/blob/edcbef3e343778b4d5affe019f64c89da2a13aa2/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L1761]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13217) minor bugs related to CASSANDRA-9143

[Blake Eggleston (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Blake Eggleston updated CASSANDRA-13217:

Status: Patch Available  (was: Open)

|[trunk|https://github.com/bdeggleston/cassandra/tree/13217] | 
[dtests|http://cassci.datastax.com/view/Dev/view/bdeggleston/job/bdeggleston-13217-dtest/]
 | 
[testall|http://cassci.datastax.com/view/Dev/view/bdeggleston/job/bdeggleston-13217-testall/]
 |

> minor bugs related to CASSANDRA-9143
> 
>
> Key: CASSANDRA-13217
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13217
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Minor
>
> We found a few minor bugs found in an internal review:
> * incorrect log argument order 
> [here|https://github.com/apache/cassandra/blob/edcbef3e343778b4d5affe019f64c89da2a13aa2/src/java/org/apache/cassandra/streaming/compress/CompressedStreamReader.java#L75]
> * {{SSTableReader#intersects}} should use Bounds, not Range 
> ([here|https://github.com/apache/cassandra/blob/edcbef3e343778b4d5affe019f64c89da2a13aa2/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L1761])
> * {{CompactionStrategyManager#validateForCompaction}} doesn't prevent 
> sstables from different repair session from being compacted together 
> [here|https://github.com/apache/cassandra/blob/edcbef3e343778b4d5affe019f64c89da2a13aa2/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L1761]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (CASSANDRA-13217) minor bugs related to CASSANDRA-9143

[Blake Eggleston (JIRA)]

Blake Eggleston created CASSANDRA-13217:
---

 Summary: minor bugs related to CASSANDRA-9143
 Key: CASSANDRA-13217
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13217
 Project: Cassandra
  Issue Type: Bug
Reporter: Blake Eggleston
Assignee: Blake Eggleston
Priority: Minor


We found a few minor bugs found in an internal review:
* incorrect log argument order 
[here|https://github.com/apache/cassandra/blob/edcbef3e343778b4d5affe019f64c89da2a13aa2/src/java/org/apache/cassandra/streaming/compress/CompressedStreamReader.java#L75]
* {{SSTableReader#intersects}} should use Bounds, not Range 
([here|https://github.com/apache/cassandra/blob/edcbef3e343778b4d5affe019f64c89da2a13aa2/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L1761])
* {{CompactionStrategyManager#validateForCompaction}} doesn't prevent sstables 
from different repair session from being compacted together 
[here|https://github.com/apache/cassandra/blob/edcbef3e343778b4d5affe019f64c89da2a13aa2/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L1761]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-10786) Include hash of result set metadata in prepared statement id

[Olivier Michallat (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Olivier Michallat updated CASSANDRA-10786:
--
Description: 
*_Initial description:_*
This is a follow-up to CASSANDRA-7910, which was about invalidating a prepared 
statement when the table is altered, to force clients to update their local 
copy of the metadata.

There's still an issue if multiple clients are connected to the same host. The 
first client to execute the query after the cache was invalidated will receive 
an UNPREPARED response, re-prepare, and update its local metadata. But other 
clients might miss it entirely (the MD5 hasn't changed), and they will keep 
using their old metadata. For example:
# {{SELECT * ...}} statement is prepared in Cassandra with md5 abc123, clientA 
and clientB both have a cache of the metadata (columns b and c) locally
# column a gets added to the table, C* invalidates its cache entry
# clientA sends an EXECUTE request for md5 abc123, gets UNPREPARED response, 
re-prepares on the fly and updates its local metadata to (a, b, c)
# prepared statement is now in C*’s cache again, with the same md5 abc123
# clientB sends an EXECUTE request for id abc123. Because the cache has been 
populated again, the query succeeds. But clientB still has not updated its 
metadata, it’s still (b,c)

One solution that was suggested is to include a hash of the result set metadata 
in the md5. This way the md5 would change at step 3, and any client using the 
old md5 would get an UNPREPARED, regardless of whether another client already 
reprepared.
-
*_Resolution (2017/02/13):_*
The following changes were made to native protocol v5:
- the PREPARED response includes {{result_metadata_id}}, a hash of the result 
set metadata.
- every EXECUTE message must provide {{result_metadata_id}} in addition to the 
prepared statement id. If it doesn't match the current one on the server, it 
means the client is operating on a stale schema.
- to notify the client, the server returns a ROWS response with a new 
{{Metadata_changed}} flag, the new {{result_metadata_id}} and the updated 
result metadata (this overrides the {{No_metadata}} flag, even if the client 
had requested it)
- the client updates its copy of the result metadata before it decodes the 
results.

So the scenario above would now look like:
# {{SELECT * ...}} statement is prepared in Cassandra with md5 abc123, and 
result set (b, c) that hashes to cde456
# column a gets added to the table, C* does not invalidate its cache entry, but 
only updates the result set to (a, b, c) which hashes to fff789
# client sends an EXECUTE request for (statementId=abc123, resultId=cde456) and 
skip_metadata flag
# cde456!=fff789, so C* responds with ROWS(..., no_metadata=false, 
metadata_changed=true, new_metadata_id=fff789,col specs for (a,b,c))
# client updates its column specifications, and will send the next execute 
queries with (statementId=abc123, resultId=fff789)

This works the same with multiple clients.

  was:
*_Initial description:_*
This is a follow-up to CASSANDRA-7910, which was about invalidating a prepared 
statement when the table is altered, to force clients to update their local 
copy of the metadata.

There's still an issue if multiple clients are connected to the same host. The 
first client to execute the query after the cache was invalidated will receive 
an UNPREPARED response, re-prepare, and update its local metadata. But other 
clients might miss it entirely (the MD5 hasn't changed), and they will keep 
using their old metadata. For example:
# {{SELECT * ...}} statement is prepared in Cassandra with md5 abc123, clientA 
and clientB both have a cache of the metadata (columns b and c) locally
# column a gets added to the table, C* invalidates its cache entry
# clientA sends an EXECUTE request for md5 abc123, gets UNPREPARED response, 
re-prepares on the fly and updates its local metadata to (a, b, c)
# prepared statement is now in C*’s cache again, with the same md5 abc123
# clientB sends an EXECUTE request for id abc123. Because the cache has been 
populated again, the query succeeds. But clientB still has not updated its 
metadata, it’s still (b,c)

One solution that was suggested is to include a hash of the result set metadata 
in the md5. This way the md5 would change at step 3, and any client using the 
old md5 would get an UNPREPARED, regardless of whether another client already 
reprepared.
-
*_Resolution:_*
The following changes were made to native protocol v5:
- the PREPARED response includes {{result_metadata_id}}, a hash of the result 
set metadata.
- every EXECUTE message must provide {{result_metadata_id}} in addition to the 
prepared statement id. If it doesn't match the current one on the server, it 
means the client is operating on a stale schema.
- to notify the client, the server returns a ROWS response with a new 

[jira] [Updated] (CASSANDRA-10786) Include hash of result set metadata in prepared statement id

[Olivier Michallat (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Olivier Michallat updated CASSANDRA-10786:
--
Description: 
*_Initial description:_*
This is a follow-up to CASSANDRA-7910, which was about invalidating a prepared 
statement when the table is altered, to force clients to update their local 
copy of the metadata.

There's still an issue if multiple clients are connected to the same host. The 
first client to execute the query after the cache was invalidated will receive 
an UNPREPARED response, re-prepare, and update its local metadata. But other 
clients might miss it entirely (the MD5 hasn't changed), and they will keep 
using their old metadata. For example:
# {{SELECT * ...}} statement is prepared in Cassandra with md5 abc123, clientA 
and clientB both have a cache of the metadata (columns b and c) locally
# column a gets added to the table, C* invalidates its cache entry
# clientA sends an EXECUTE request for md5 abc123, gets UNPREPARED response, 
re-prepares on the fly and updates its local metadata to (a, b, c)
# prepared statement is now in C*’s cache again, with the same md5 abc123
# clientB sends an EXECUTE request for id abc123. Because the cache has been 
populated again, the query succeeds. But clientB still has not updated its 
metadata, it’s still (b,c)

One solution that was suggested is to include a hash of the result set metadata 
in the md5. This way the md5 would change at step 3, and any client using the 
old md5 would get an UNPREPARED, regardless of whether another client already 
reprepared.
-
*_Resolution:_*
The following changes were made to native protocol v5:
- the PREPARED response includes {{result_metadata_id}}, a hash of the result 
set metadata.
- every EXECUTE message must provide {{result_metadata_id}} in addition to the 
prepared statement id. If it doesn't match the current one on the server, it 
means the client is operating on a stale schema.
- to notify the client, the server returns a ROWS response with a new 
{{Metadata_changed}} flag, the new {{result_metadata_id}} and the updated 
result metadata (this overrides the {{No_metadata}} flag, even if the client 
had requested it)
- the client updates its copy of the result metadata before it decodes the 
results.

So the scenario above would now look like:
# {{SELECT * ...}} statement is prepared in Cassandra with md5 abc123, and 
result set (b, c) that hashes to cde456
# column a gets added to the table, C* does not invalidate its cache entry, but 
only updates the result set to (a, b, c) which hashes to fff789
# client sends an EXECUTE request for (statementId=abc123, resultId=cde456) and 
skip_metadata flag
# cde456!=fff789, so C* responds with ROWS(..., no_metadata=false, 
metadata_changed=true, new_metadata_id=fff789,col specs for (a,b,c))
# client updates its column specifications, and will send the next execute 
queries with (statementId=abc123, resultId=fff789)

This works the same with multiple clients.

  was:
This is a follow-up to CASSANDRA-7910, which was about invalidating a prepared 
statement when the table is altered, to force clients to update their local 
copy of the metadata.

There's still an issue if multiple clients are connected to the same host. The 
first client to execute the query after the cache was invalidated will receive 
an UNPREPARED response, re-prepare, and update its local metadata. But other 
clients might miss it entirely (the MD5 hasn't changed), and they will keep 
using their old metadata. For example:
# {{SELECT * ...}} statement is prepared in Cassandra with md5 abc123, clientA 
and clientB both have a cache of the metadata (columns b and c) locally
# column a gets added to the table, C* invalidates its cache entry
# clientA sends an EXECUTE request for md5 abc123, gets UNPREPARED response, 
re-prepares on the fly and updates its local metadata to (a, b, c)
# prepared statement is now in C*’s cache again, with the same md5 abc123
# clientB sends an EXECUTE request for id abc123. Because the cache has been 
populated again, the query succeeds. But clientB still has not updated its 
metadata, it’s still (b,c)

One solution that was suggested is to include a hash of the result set metadata 
in the md5. This way the md5 would change at step 3, and any client using the 
old md5 would get an UNPREPARED, regardless of whether another client already 
reprepared.


> Include hash of result set metadata in prepared statement id
> 
>
> Key: CASSANDRA-10786
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10786
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: CQL
>Reporter: Olivier Michallat
>Assignee: Alex Petrov
>Priority: Minor
>  Labels: client-impacting, doc-impacting, 

[jira] [Commented] (CASSANDRA-13114) Upgrade netty to 4.0.44 to fix memory leak with client encryption

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864543#comment-15864543
 ] 

Jeff Jirsa commented on CASSANDRA-13114:


Was this dire enough to go into 2.1 ?  Seems like a pretty high bar for a .17 
release. 



> Upgrade netty to 4.0.44 to fix memory leak with client encryption
> -
>
> Key: CASSANDRA-13114
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13114
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Tom van der Woerdt
>Assignee: Stefan Podkowinski
>Priority: Blocker
> Fix For: 2.1.17, 2.2.9, 3.0.11, 3.11.0, 4.0
>
> Attachments: 13114_netty-4.0.44_2.x-3.0.patch, 
> 13114_netty-4.0.44_3.11.patch
>
>
> https://issues.apache.org/jira/browse/CASSANDRA-12032 updated netty for 
> Cassandra 3.8, but this wasn't backported. Netty 4.0.23, which ships with 
> Cassandra 3.0.x, has some serious bugs around memory handling for SSL 
> connections.
> It would be nice if both were updated to 4.0.42, a version released this year.
> 4.0.23 makes it impossible for me to run SSL, because nodes run out of memory 
> every ~30 minutes. This was fixed in 4.0.27.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13210) test failure in repair_tests.incremental_repair_test.TestIncRepair.sstable_marking_test_not_intersecting_all_ranges

[Blake Eggleston (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864532#comment-15864532
 ] 

Blake Eggleston commented on CASSANDRA-13210:
-

filed https://github.com/riptano/cassandra-dtest/pull/1442 to fix

> test failure in 
> repair_tests.incremental_repair_test.TestIncRepair.sstable_marking_test_not_intersecting_all_ranges
> ---
>
> Key: CASSANDRA-13210
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13210
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Michael Shuler
>Assignee: Blake Eggleston
>  Labels: dtest, test-failure
> Attachments: node1_debug.log, node1_gc.log, node1.log, 
> node2_debug.log, node2_gc.log, node2.log, node3_debug.log, node3_gc.log, 
> node3.log, node4_debug.log, node4_gc.log, node4.log
>
>
> example failure:
> http://cassci.datastax.com/job/trunk_large_dtest/53/testReport/repair_tests.incremental_repair_test/TestIncRepair/sstable_marking_test_not_intersecting_all_ranges
> {noformat}
> Error Message
> 'Repaired at: 0' unexpectedly found in 'SSTable: 
> /tmp/dtest-N7zjo6/test/node1/data0/keyspace1/standard1-a79a0c50efa211e6bf211330662f36ef/md-6-big\nPartitioner:
>  org.apache.cassandra.dht.Murmur3Partitioner\nBloom Filter FP chance: 
> 0.01\nMinimum timestamp: 148673926323\nMaximum timestamp: 
> 148673926323\nSSTable min local deletion time: 2147483647\nSSTable max 
> local deletion time: 2147483647\nCompressor: -\nTTL min: 0\nTTL max: 0\nFirst 
> token: 296988783704308703 (key=30503337373039503231)\nLast token: 
> 296988783704308703 (key=30503337373039503231)\nEstimated droppable 
> tombstones: 0.0\nSSTable Level: 0\nRepaired at: 0\nPending repair: 
> b099d1f0-efa2-11e6-89ec-d14624f1e47e\nReplay positions covered: 
> {CommitLogPosition(segmentId=1486739234777, 
> position=46796)=CommitLogPosition(segmentId=1486739234777, 
> position=50819)}\ntotalColumnsSet: 5\ntotalRows: 1\nEstimated tombstone drop 
> times:\nCount   Row SizeCell Count\n1 
>  0 0\n2  0 0\n3   
>0 0\n4  0  
>0\n5  0 2\n6   
>0 0\n7  0 
> 0\n8  0 0\n10 
> 0 0\n12 0 0\n14   
>   0 0\n17 0   
>   0\n20 0 0\n24   
>   0 0\n29 0 0\n35 
> 0 0\n42 0 
> 0\n50 0 0\n60 
> 0 0\n72 0 
> 0\n86 0 0\n103
> 0 0\n1240 0\n149  
>   0 0\n1790   
>   0\n2151 0\n258  
>   0 0\n3100 
> 0\n3720 0\n446
> 0 0\n5350 0\n642  
>   0 0\n7700   
>   0\n9240 0\n1109 
>   0 0\n1331   0 
> 0\n1597   0 0\n1916   
> 0 0\n2299   0 0\n2759 
>   0 0\n3311   0   
>   0\n3973   0 0\n4768 
>   0 0\n5722   0 
> 0\n6866   0 0\n8239   
> 0 0\n9887   0 0\n11864
>   0 0\n14237  0   
>   0\n17084  0 0\n20501
>   0 0\n24601  0 
> 0\n29521  0 0\n35425  
> 0 0\n42510  0   

[jira] [Commented] (CASSANDRA-12653) In-flight shadow round requests

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864426#comment-15864426
 ] 

Jeff Jirsa commented on CASSANDRA-12653:


[~spo...@gmail.com] - is there some deeper purpose of moving the 
{{FD.instance.isAlive()}} check higher in {{MigrationTask#runMayThrow()}} 
method beyond "check to see if it's dead before we bother checking to see if 
it's worth sending a migration task"? Is there a reason we don't let 
{{MM#shouldPullSchemaFrom}} return false if FD says the instance is dead? 

Given that the shadow round is meant to just get ring state without changing 
anything, should we add an explicit check to 
{{MigrationManager#scheduleSchemaPull()}} to ensure that 
{{Gossiper.instance.isInShadowRound()}} is false before scheduling?


> In-flight shadow round requests
> ---
>
> Key: CASSANDRA-12653
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12653
> Project: Cassandra
>  Issue Type: Bug
>  Components: Distributed Metadata
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Minor
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x
>
> Attachments: 12653-2.2.patch, 12653-3.0.patch, 12653-trunk.patch
>
>
> Bootstrapping or replacing a node in the cluster requires to gather and check 
> some host IDs or tokens by doing a gossip "shadow round" once before joining 
> the cluster. This is done by sending a gossip SYN to all seeds until we 
> receive a response with the cluster state, from where we can move on in the 
> bootstrap process. Receiving a response will call the shadow round done and 
> calls {{Gossiper.resetEndpointStateMap}} for cleaning up the received state 
> again.
> The issue here is that at this point there might be other in-flight requests 
> and it's very likely that shadow round responses from other seeds will be 
> received afterwards, while the current state of the bootstrap process doesn't 
> expect this to happen (e.g. gossiper may or may not be enabled). 
> One side effect will be that MigrationTasks are spawned for each shadow round 
> reply except the first. Tasks might or might not execute based on whether at 
> execution time {{Gossiper.resetEndpointStateMap}} had been called, which 
> effects the outcome of {{FailureDetector.instance.isAlive(endpoint))}} at 
> start of the task. You'll see error log messages such as follows when this 
> happend:
> {noformat}
> INFO  [SharedPool-Worker-1] 2016-09-08 08:36:39,255 Gossiper.java:993 - 
> InetAddress /xx.xx.xx.xx is now UP
> ERROR [MigrationStage:1]2016-09-08 08:36:39,255 FailureDetector.java:223 
> - unknown endpoint /xx.xx.xx.xx
> {noformat}
> Although is isn't pretty, I currently don't see any serious harm from this, 
> but it would be good to get a second opinion (feel free to close as "wont 
> fix").
> /cc [~Stefania] [~thobbs]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (CASSANDRA-8457) nio MessagingService

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863758#comment-15863758
 ] 

Jason Brown edited comment on CASSANDRA-8457 at 2/13/17 9:02 PM:
-

OK, so I've been performance load testing the snot of this code for the last 
several weeks, with help from netty committers, flight recorder, and flame 
graphs. As a result, I've made some major and some minor tweaks, and now I'm 
slightly faster than trunk with slightly better throughput. I have some 
optimizations in my back pocket that will increase even more, but as 
[~slebresne] and I have agreed before, we'll save those for follow up tickets.

trunk
{code}
 id, type   total ops,op/s,pk/s,   row/s,mean, 
med, .95, .99,.999, max,   time,   stderr, errors,  gc: #,  max 
ms,  sum ms,  sdv ms,  mb
  4 threadCount, total,233344,3889,3889,3889, 1.0, 
1.0, 1.2, 1.3, 1.5,68.2,   60.0,  0.01549,  0,  9, 
538, 538,   4,5381
  8 threadCount, total,544637,9076,9076,9076, 0.8, 
0.8, 1.0, 1.1, 1.4,73.8,   60.0,  0.00978,  0, 20,
1267,1267,   5,   11848
 16 threadCount, total,   1126627,   18774,   18774,   18774, 0.8, 
0.8, 0.9, 1.0, 5.5,78.2,   60.0,  0.01882,  0, 40,
2665,2665,   6,   23666
 24 threadCount, total,   1562460,   26036,   26036,   26036, 0.9, 
0.8, 1.0, 1.1, 9.1,81.3,   60.0,  0.00837,  0, 55,
3543,3543,   9,   32619
 36 threadCount, total,   2098097,   34962,   34962,   34962, 1.0, 
0.9, 1.1, 1.3,60.9,83.0,   60.0,  0.00793,  0, 73,
4665,4665,   7,   43144
 54 threadCount, total,   2741814,   45686,   45686,   45686, 1.1, 
1.0, 1.4, 1.7,62.2,   131.7,   60.0,  0.01321,  0, 93,
5748,5748,   7,   55097
 81 threadCount, total,   3851131,   64166,   64166,   64166, 1.2, 
1.0, 1.6, 2.6,62.3,   151.7,   60.0,  0.01152,  0,159,
8190,8521,  14,  106805
121 threadCount, total,   4798169,   79947,   79947,   79947, 1.5, 
1.1, 2.0, 3.0,63.5,   117.8,   60.0,  0.05689,  0,165,
9323,9439,   5,   97536
181 threadCount, total,   5647043,   94088,   94088,   94088, 1.9, 
1.4, 2.6, 4.9,68.5,   169.2,   60.0,  0.01639,  0,195,   
10106,   11011,  11,  126422
271 threadCount, total,   6450510,  107461,  107461,  107461, 2.5, 
1.8, 3.7,12.0,75.4,   155.8,   60.0,  0.01542,  0,228,   
10304,   12789,   9,  143857
406 threadCount, total,   6700764,  111635,  111635,  111635, 3.6, 
2.5, 5.3,55.8,75.6,   196.5,   60.0,  0.01800,  0,243,
9995,   13170,   7,  144166
609 threadCount, total,   7119535,  118477,  118477,  118477, 5.1, 
3.5, 7.9,62.8,85.1,   170.0,   60.1,  0.01775,  0,250,   
10149,   13781,   7,  148118
913 threadCount, total,   7093347,  117981,  117981,  117981, 7.7, 
4.9,15.7,71.3,   101.1,   173.4,   60.1,  0.02780,  0,246,   
10327,   13859,   8,  155896
{code}

8457
{code}
 id, type   total ops,op/s,pk/s,   row/s,mean, 
med, .95, .99,.999, max,   time,   stderr, errors,  gc: #,  max 
ms,  sum ms,  sdv ms,  mb
  4 threadCount, total,161668,2694,2694,2694, 1.4, 
1.4, 1.6, 1.7, 3.2,68.2,   60.0,  0.01264,  0,  6, 
363, 363,   4,3631
  8 threadCount, total,498139,8301,8301,8301, 0.9, 
0.9, 1.1, 1.3, 1.8,73.5,   60.0,  0.00446,  0, 19,
1164,1164,   6,   11266
 16 threadCount, total,765437,   12756,   12756,   12756, 1.2, 
1.2, 1.4, 1.5, 5.7,74.8,   60.0,  0.01251,  0, 29,
1819,1819,   5,   17238
 24 threadCount, total,   1122768,   18710,   18710,   18710, 1.2, 
1.2, 1.4, 1.5, 8.5,   127.7,   60.0,  0.00871,  0, 42,
2538,2538,   5,   25054
 36 threadCount, total,   1649658,   27489,   27489,   27489, 1.3, 
1.2, 1.4, 1.6,60.1,77.7,   60.0,  0.00627,  0, 57,
3652,3652,   7,   33743
 54 threadCount, total,   2258999,   37641,   37641,   37641, 1.4, 
1.3, 1.6, 1.8,62.5,81.7,   60.0,  0.00771,  0, 79,
4908,4908,   6,   46789
 81 threadCount, total,   3255005,   54220,   54220,   54220, 1.5, 
1.2, 1.7, 2.2,63.8,   133.4,   60.0,  0.02030,  0,117,
6953,7008,   9,   72208
121 

[jira] [Updated] (CASSANDRA-13090) Coalescing strategy sleeps too much

[Ariel Weisberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ariel Weisberg updated CASSANDRA-13090:
---
Fix Version/s: (was: 3.11.x)
   (was: 4.x)
   (was: 3.0.x)
   (was: 2.2.x)
   4.0
   3.11.0
   3.0.11
   2.2.9

> Coalescing strategy sleeps too much
> ---
>
> Key: CASSANDRA-13090
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13090
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and Messaging
>Reporter: Corentin Chary
>Assignee: Corentin Chary
> Fix For: 2.2.9, 3.0.11, 3.11.0, 4.0
>
> Attachments: 0001-Fix-wait-time-coalescing-CASSANDRA-13090-2.patch, 
> 0001-Fix-wait-time-coalescing-CASSANDRA-13090.patch
>
>
> With the current code maybeSleep is called even if we managed to take 
> maxItems out of the backlog. In this case we should really avoid sleeping 
> because it means that backlog is building up.
> I'll send a patch shortly.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13090) Coalescing strategy sleeps too much

[Ariel Weisberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ariel Weisberg updated CASSANDRA-13090:
---
Resolution: Fixed
Status: Resolved  (was: Ready to Commit)

Committed as 
[5725e2c422d21d8efe5ae3bc4389842939553650|https://github.com/apache/cassandra/commit/5725e2c422d21d8efe5ae3bc4389842939553650]

> Coalescing strategy sleeps too much
> ---
>
> Key: CASSANDRA-13090
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13090
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and Messaging
>Reporter: Corentin Chary
>Assignee: Corentin Chary
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x
>
> Attachments: 0001-Fix-wait-time-coalescing-CASSANDRA-13090-2.patch, 
> 0001-Fix-wait-time-coalescing-CASSANDRA-13090.patch
>
>
> With the current code maybeSleep is called even if we managed to take 
> maxItems out of the backlog. In this case we should really avoid sleeping 
> because it means that backlog is building up.
> I'll send a patch shortly.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-11961) Nonfatal NPE in CompactionMetrics

[Sotirios Delimanolis (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864334#comment-15864334
 ] 

Sotirios Delimanolis commented on CASSANDRA-11961:
--

Why is it possible for {{CFMetaData}} to be null there?

> Nonfatal NPE in CompactionMetrics
> -
>
> Key: CASSANDRA-11961
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11961
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Robert Stupp
>Assignee: Achal Shah
>Priority: Minor
>  Labels: lhf
> Fix For: 3.8
>
>
> Just saw the following NPE on trunk. Means, that {{metaData}} from 
> {{CFMetaData metaData = compaction.getCompactionInfo().getCFMetaData();}} is 
> {{null}}. A simple {{if (metaData == null) continue;}} should fix this.
> {code}
> Caused by: java.lang.NullPointerException: null
>   at 
> org.apache.cassandra.metrics.CompactionMetrics$2.getValue(CompactionMetrics.java:103)
>  ~[main/:na]
>   at 
> org.apache.cassandra.metrics.CompactionMetrics$2.getValue(CompactionMetrics.java:78)
>  ~[main/:na]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13006) Disable automatic heap dumps on OOM error

[anmols (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

anmols updated CASSANDRA-13006:
---
Status: Patch Available  (was: Open)

> Disable automatic heap dumps on OOM error
> -
>
> Key: CASSANDRA-13006
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13006
> Project: Cassandra
>  Issue Type: Bug
>  Components: Configuration
>Reporter: anmols
>Assignee: Benjamin Lerer
>Priority: Minor
> Fix For: 3.0.9
>
> Attachments: 13006-3.0.9.txt
>
>
> With CASSANDRA-9861, a change was added to enable collecting heap dumps by 
> default if the process encountered an OOM error. These heap dumps are stored 
> in the Apache Cassandra home directory unless configured otherwise (see 
> [Cassandra Support 
> Document|https://support.datastax.com/hc/en-us/articles/204225959-Generating-and-Analyzing-Heap-Dumps]
>  for this feature).
>  
> The creation and storage of heap dumps aides debugging and investigative 
> workflows, but is not be desirable for a production environment where these 
> heap dumps may occupy a large amount of disk space and require manual 
> intervention for cleanups. 
>  
> Managing heap dumps on out of memory errors and configuring the paths for 
> these heap dumps are available as JVM options in JVM. The current behavior 
> conflicts with the Boolean JVM flag HeapDumpOnOutOfMemoryError. 
>  
> A patch can be proposed here that would make the heap dump on OOM error honor 
> the HeapDumpOnOutOfMemoryError flag. Users who would want to still generate 
> heap dumps on OOM errors can set the -XX:+HeapDumpOnOutOfMemoryError JVM 
> option.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13196) test failure in snitch_test.TestGossipingPropertyFileSnitch.test_prefer_local_reconnect_on_listen_address

[Alex Petrov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Petrov updated CASSANDRA-13196:

Description: 
example failure:

http://cassci.datastax.com/job/trunk_dtest/1487/testReport/snitch_test/TestGossipingPropertyFileSnitch/test_prefer_local_reconnect_on_listen_address

{code}
{novnode}
Error Message

Error from server: code=2200 [Invalid query] message="keyspace keyspace1 does 
not exist"
 >> begin captured logging << 
dtest: DEBUG: cluster ccm directory: /tmp/dtest-k6b0iF
dtest: DEBUG: Done setting configuration options:
{   'initial_token': None,
'num_tokens': '32',
'phi_convict_threshold': 5,
'range_request_timeout_in_ms': 1,
'read_request_timeout_in_ms': 1,
'request_timeout_in_ms': 1,
'truncate_request_timeout_in_ms': 1,
'write_request_timeout_in_ms': 1}
cassandra.policies: INFO: Using datacenter 'dc1' for DCAwareRoundRobinPolicy 
(via host '127.0.0.1'); if incorrect, please specify a local_dc to the 
constructor, or limit contact points to local cluster nodes
cassandra.cluster: INFO: New Cassandra host  discovered
- >> end captured logging << -
Stacktrace

  File "/usr/lib/python2.7/unittest/case.py", line 329, in run
testMethod()
  File "/home/automaton/cassandra-dtest/snitch_test.py", line 87, in 
test_prefer_local_reconnect_on_listen_address
new_rows = list(session.execute("SELECT * FROM {}".format(stress_table)))
  File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line 1998, 
in execute
return self.execute_async(query, parameters, trace, custom_payload, 
timeout, execution_profile, paging_state).result()
  File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line 3784, 
in result
raise self._final_exception
'Error from server: code=2200 [Invalid query] message="keyspace keyspace1 does 
not exist"\n >> begin captured logging << 
\ndtest: DEBUG: cluster ccm directory: 
/tmp/dtest-k6b0iF\ndtest: DEBUG: Done setting configuration options:\n{   
\'initial_token\': None,\n\'num_tokens\': \'32\',\n
\'phi_convict_threshold\': 5,\n\'range_request_timeout_in_ms\': 1,\n
\'read_request_timeout_in_ms\': 1,\n\'request_timeout_in_ms\': 1,\n 
   \'truncate_request_timeout_in_ms\': 1,\n
\'write_request_timeout_in_ms\': 1}\ncassandra.policies: INFO: Using 
datacenter \'dc1\' for DCAwareRoundRobinPolicy (via host \'127.0.0.1\'); if 
incorrect, please specify a local_dc to the constructor, or limit contact 
points to local cluster nodes\ncassandra.cluster: INFO: New Cassandra host 
 discovered\n- >> end captured logging 
<< -'
{novnode}
{code}

  was:
example failure:

http://cassci.datastax.com/job/trunk_dtest/1487/testReport/snitch_test/TestGossipingPropertyFileSnitch/test_prefer_local_reconnect_on_listen_address

{novnode}
Error Message

Error from server: code=2200 [Invalid query] message="keyspace keyspace1 does 
not exist"
 >> begin captured logging << 
dtest: DEBUG: cluster ccm directory: /tmp/dtest-k6b0iF
dtest: DEBUG: Done setting configuration options:
{   'initial_token': None,
'num_tokens': '32',
'phi_convict_threshold': 5,
'range_request_timeout_in_ms': 1,
'read_request_timeout_in_ms': 1,
'request_timeout_in_ms': 1,
'truncate_request_timeout_in_ms': 1,
'write_request_timeout_in_ms': 1}
cassandra.policies: INFO: Using datacenter 'dc1' for DCAwareRoundRobinPolicy 
(via host '127.0.0.1'); if incorrect, please specify a local_dc to the 
constructor, or limit contact points to local cluster nodes
cassandra.cluster: INFO: New Cassandra host  discovered
- >> end captured logging << -
Stacktrace

  File "/usr/lib/python2.7/unittest/case.py", line 329, in run
testMethod()
  File "/home/automaton/cassandra-dtest/snitch_test.py", line 87, in 
test_prefer_local_reconnect_on_listen_address
new_rows = list(session.execute("SELECT * FROM {}".format(stress_table)))
  File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line 1998, 
in execute
return self.execute_async(query, parameters, trace, custom_payload, 
timeout, execution_profile, paging_state).result()
  File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line 3784, 
in result
raise self._final_exception
'Error from server: code=2200 [Invalid query] message="keyspace keyspace1 does 
not exist"\n >> begin captured logging << 
\ndtest: DEBUG: cluster ccm directory: 
/tmp/dtest-k6b0iF\ndtest: DEBUG: Done setting configuration options:\n{   
\'initial_token\': None,\n\'num_tokens\': \'32\',\n
\'phi_convict_threshold\': 5,\n

[jira] [Commented] (CASSANDRA-9384) Update jBCrypt dependency to version 0.4

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864265#comment-15864265
 ] 

ASF GitHub Bot commented on CASSANDRA-9384:
---

GitHub user dysmento opened a pull request:

https://github.com/apache/cassandra/pull/93

CASSANDRA-9384 bump version of jbcrypt to 0.4

org.mindrot:jbcrypt:0.4 just landed in Maven Central. Here's a version bump 
for Cassandra.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dysmento/cassandra trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/cassandra/pull/93.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #93






> Update jBCrypt dependency to version 0.4
> 
>
> Key: CASSANDRA-9384
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9384
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Sam Tunnicliffe
> Fix For: 2.1.x, 2.2.x, 3.0.x, 3.11.x
>
>
> https://bugzilla.mindrot.org/show_bug.cgi?id=2097
> Although the bug tracker lists it as NEW/OPEN, the release notes for 0.4 
> indicate that this is now fixed, so we should update.
> Thanks to [~Bereng] for identifying the issue.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13211) Use portable stderr for java error in startup

[Michael Shuler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Shuler updated CASSANDRA-13211:
---
Status: Patch Available  (was: Open)

> Use portable stderr for java error in startup
> -
>
> Key: CASSANDRA-13211
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13211
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Max Bowsher
>Assignee: Michael Shuler
> Fix For: 2.1.x
>
> Attachments: 13211_use-portable-stderr.patch
>
>
> The cassandra startup shell script contains this line:
> echo Unable to find java executable. Check JAVA_HOME and PATH environment 
> variables. > /dev/stderr
> The problem here is the construct "> /dev/stderr". If the user invoking 
> Cassandra has changed user (for example, by SSHing in as a personal user, and 
> then sudo-ing to an application user responsible for executing the Cassandra 
> daemon), then the attempt to open /dev/stderr will fail, because it will 
> point to a PTY node under /dev/pts/ owned by the original user.
> Ultimately this leads to the real problem being masked by the confusing error 
> message "bash: /dev/stderr: Permission denied".
> The correct technique is to replace "> /dev/stderr" with ">&2" which will 
> write to the already open stderr file descriptor, instead of resolving the 
> chain of symlinks starting at /dev/stderr, and attempting to reopen the 
> target by name.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13211) Use portable stderr for java error in startup

[Michael Shuler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Shuler updated CASSANDRA-13211:
---
Summary: Use portable stderr for java error in startup  (was: cassandra 
shell script uses bad approach to write "Unable to find java" error to stderr, 
causing real issue to be masked by a permission error if user has changed user 
since logging in)

> Use portable stderr for java error in startup
> -
>
> Key: CASSANDRA-13211
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13211
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Max Bowsher
>Assignee: Michael Shuler
> Fix For: 2.1.x
>
> Attachments: 13211_use-portable-stderr.patch
>
>
> The cassandra startup shell script contains this line:
> echo Unable to find java executable. Check JAVA_HOME and PATH environment 
> variables. > /dev/stderr
> The problem here is the construct "> /dev/stderr". If the user invoking 
> Cassandra has changed user (for example, by SSHing in as a personal user, and 
> then sudo-ing to an application user responsible for executing the Cassandra 
> daemon), then the attempt to open /dev/stderr will fail, because it will 
> point to a PTY node under /dev/pts/ owned by the original user.
> Ultimately this leads to the real problem being masked by the confusing error 
> message "bash: /dev/stderr: Permission denied".
> The correct technique is to replace "> /dev/stderr" with ">&2" which will 
> write to the already open stderr file descriptor, instead of resolving the 
> chain of symlinks starting at /dev/stderr, and attempting to reopen the 
> target by name.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13211) cassandra shell script uses bad approach to write "Unable to find java" error to stderr, causing real issue to be masked by a permission error if user has changed us

[Michael Shuler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Shuler updated CASSANDRA-13211:
---
Fix Version/s: 2.1.x

> cassandra shell script uses bad approach to write "Unable to find java" error 
> to stderr, causing real issue to be masked by a permission error if user has 
> changed user since logging in
> 
>
> Key: CASSANDRA-13211
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13211
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Max Bowsher
>Assignee: Michael Shuler
> Fix For: 2.1.x
>
> Attachments: 13211_use-portable-stderr.patch
>
>
> The cassandra startup shell script contains this line:
> echo Unable to find java executable. Check JAVA_HOME and PATH environment 
> variables. > /dev/stderr
> The problem here is the construct "> /dev/stderr". If the user invoking 
> Cassandra has changed user (for example, by SSHing in as a personal user, and 
> then sudo-ing to an application user responsible for executing the Cassandra 
> daemon), then the attempt to open /dev/stderr will fail, because it will 
> point to a PTY node under /dev/pts/ owned by the original user.
> Ultimately this leads to the real problem being masked by the confusing error 
> message "bash: /dev/stderr: Permission denied".
> The correct technique is to replace "> /dev/stderr" with ">&2" which will 
> write to the already open stderr file descriptor, instead of resolving the 
> chain of symlinks starting at /dev/stderr, and attempting to reopen the 
> target by name.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[09/10] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[aweisberg]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/702ec088
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/702ec088
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/702ec088

Branch: refs/heads/trunk
Commit: 702ec088f5f61106b41e128f3fb8f109da8cbe1c
Parents: 01b91cc 3d01e90
Author: Ariel Weisberg 
Authored: Mon Feb 13 13:31:23 2017 -0500
Committer: Ariel Weisberg 
Committed: Mon Feb 13 13:32:30 2017 -0500

--
 CHANGES.txt |  1 +
 conf/cassandra.yaml | 27 +
 .../org/apache/cassandra/config/Config.java |  3 +-
 .../cassandra/config/DatabaseDescriptor.java| 16 
 .../cassandra/net/OutboundTcpConnection.java|  4 +-
 .../cassandra/utils/CoalescingStrategies.java   | 32 
 .../utils/CoalescingStrategiesTest.java | 40 
 7 files changed, 114 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/702ec088/CHANGES.txt
--
diff --cc CHANGES.txt
index 67c45e8,b19550a..8164a52
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -215,20 -100,6 +215,21 @@@ Merged from 3.0
   * Correct log message for statistics of offheap memtable flush 
(CASSANDRA-12776)
   * Explicitly set locale for string validation 
(CASSANDRA-12541,CASSANDRA-12542,CASSANDRA-12543,CASSANDRA-12545)
  Merged from 2.2:
++ * Coalescing strategy sleeps too much and shouldn't be enabled by default 
(CASSANDRA-13090)
 + * Fix speculative retry bugs (CASSANDRA-13009)
 + * Fix handling of nulls and unsets in IN conditions (CASSANDRA-12981)
 + * Fix race causing infinite loop if Thrift server is stopped before it 
starts listening (CASSANDRA-12856)
 + * CompactionTasks now correctly drops sstables out of compaction when not 
enough disk space is available (CASSANDRA-12979)
 + * Remove support for non-JavaScript UDFs (CASSANDRA-12883)
 + * Fix DynamicEndpointSnitch noop in multi-datacenter situations 
(CASSANDRA-13074)
 + * cqlsh copy-from: encode column names to avoid primary key parsing errors 
(CASSANDRA-12909)
 + * Temporarily fix bug that creates commit log when running offline tools 
(CASSANDRA-8616)
 + * Reduce granuality of OpOrder.Group during index build (CASSANDRA-12796)
 + * Test bind parameters and unset parameters in InsertUpdateIfConditionTest 
(CASSANDRA-12980)
 + * Use saved tokens when setting local tokens on StorageService.joinRing 
(CASSANDRA-12935)
 + * cqlsh: fix DESC TYPES errors (CASSANDRA-12914)
 + * Fix leak on skipped SSTables in sstableupgrade (CASSANDRA-12899)
 + * Avoid blocking gossip during pending range calculation (CASSANDRA-12281)
   * Fix purgeability of tombstones with max timestamp (CASSANDRA-12792)
   * Fail repair if participant dies during sync or anticompaction 
(CASSANDRA-12901)
   * cqlsh COPY: unprotected pk values before converting them if not using 
prepared statements (CASSANDRA-12863)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/702ec088/conf/cassandra.yaml
--
diff --cc conf/cassandra.yaml
index 4436a02,a9d4c01..063a0b7
--- a/conf/cassandra.yaml
+++ b/conf/cassandra.yaml
@@@ -1173,24 -959,29 +1173,51 @@@ gc_warn_threshold_in_ms: 100
  # as corrupted.
  # max_value_size_in_mb: 256
  
 +# Back-pressure settings #
 +# If enabled, the coordinator will apply the back-pressure strategy specified 
below to each mutation
 +# sent to replicas, with the aim of reducing pressure on overloaded replicas.
 +back_pressure_enabled: false
 +# The back-pressure strategy applied.
 +# The default implementation, RateBasedBackPressure, takes three arguments:
 +# high ratio, factor, and flow type, and uses the ratio between incoming 
mutation responses and outgoing mutation requests.
 +# If below high ratio, outgoing mutations are rate limited according to the 
incoming rate decreased by the given factor;
 +# if above high ratio, the rate limiting is increased by the given factor;
 +# such factor is usually best configured between 1 and 10, use larger values 
for a faster recovery
 +# at the expense of potentially more dropped mutations;
 +# the rate limiting is applied according to the flow type: if FAST, it's rate 
limited at the speed of the fastest replica,
 +# if SLOW at the speed of the slowest one.
 +# New strategies can be added. Implementors need to implement 
org.apache.cassandra.net.BackpressureStrategy and
 +# provide a public constructor accepting a Map.
 +back_pressure_strategy:
 +- class_name: org.apache.cassandra.net.RateBasedBackPressure
 +  parameters:
 +- 

[05/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[aweisberg]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3d01e906
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3d01e906
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3d01e906

Branch: refs/heads/cassandra-3.11
Commit: 3d01e906152f02f826a9663afb4a1232060ad584
Parents: 3a570d7 5725e2c
Author: Ariel Weisberg 
Authored: Mon Feb 13 13:00:20 2017 -0500
Committer: Ariel Weisberg 
Committed: Mon Feb 13 13:23:39 2017 -0500

--
 CHANGES.txt |  1 +
 conf/cassandra.yaml | 27 +
 .../org/apache/cassandra/config/Config.java |  3 +-
 .../cassandra/config/DatabaseDescriptor.java| 16 
 .../cassandra/net/OutboundTcpConnection.java|  4 +-
 .../cassandra/utils/CoalescingStrategies.java   | 32 
 .../utils/CoalescingStrategiesTest.java | 40 
 7 files changed, 114 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/3d01e906/CHANGES.txt
--
diff --cc CHANGES.txt
index ac3d1ed,4052b0f..b19550a
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,43 -1,5 +1,44 @@@
 -2.2.9
 +3.0.11
 + * Add vm.max_map_count StartupCheck (CASSANDRA-13008)
 + * Hint related logging should include the IP address of the destination in 
addition to 
 +   host ID (CASSANDRA-13205)
 + * Reloading logback.xml does not work (CASSANDRA-13173)
 + * Lightweight transactions temporarily fail after upgrade from 2.1 to 3.0 
(CASSANDRA-13109)
 + * Duplicate rows after upgrading from 2.1.16 to 3.0.10/3.9 (CASSANDRA-13125)
 + * Fix UPDATE queries with empty IN restrictions (CASSANDRA-13152)
 + * Abort or retry on failed hints delivery (CASSANDRA-13124)
 + * Fix handling of partition with partition-level deletion plus
 +   live rows in sstabledump (CASSANDRA-13177)
 + * Provide user workaround when system_schema.columns does not contain entries
 +   for a table that's in system_schema.tables (CASSANDRA-13180)
 + * Dump threads when unit tests time out (CASSANDRA-13117)
 + * Better error when modifying function permissions without explicit keyspace 
(CASSANDRA-12925)
 + * Indexer is not correctly invoked when building indexes over sstables 
(CASSANDRA-13075)
 + * Read repair is not blocking repair to finish in foreground repair 
(CASSANDRA-13115)
 + * Stress daemon help is incorrect (CASSANDRA-12563)
 + * Remove ALTER TYPE support (CASSANDRA-12443)
 + * Fix assertion for certain legacy range tombstone pattern (CASSANDRA-12203)
 + * Set javac encoding to utf-8 (CASSANDRA-11077)
 + * Replace empty strings with null values if they cannot be converted 
(CASSANDRA-12794)
 + * Fixed flacky SSTableRewriterTest: check file counts before calling 
validateCFS (CASSANDRA-12348)
 + * Fix deserialization of 2.x DeletedCells (CASSANDRA-12620)
 + * Add parent repair session id to anticompaction log message 
(CASSANDRA-12186)
 + * Improve contention handling on failure to acquire MV lock for streaming 
and hints (CASSANDRA-12905)
 + * Fix DELETE and UPDATE queries with empty IN restrictions (CASSANDRA-12829)
 + * Mark MVs as built after successful bootstrap (CASSANDRA-12984)
 + * Estimated TS drop-time histogram updated with Cell.NO_DELETION_TIME 
(CASSANDRA-13040)
 + * Nodetool compactionstats fails with NullPointerException (CASSANDRA-13021)
 + * Thread local pools never cleaned up (CASSANDRA-13033)
 + * Set RPC_READY to false when draining or if a node is marked as shutdown 
(CASSANDRA-12781)
 + * Make sure sstables only get committed when it's safe to discard commit log 
records (CASSANDRA-12956)
 + * Reject default_time_to_live option when creating or altering MVs 
(CASSANDRA-12868)
 + * Nodetool should use a more sane max heap size (CASSANDRA-12739)
 + * LocalToken ensures token values are cloned on heap (CASSANDRA-12651)
 + * AnticompactionRequestSerializer serializedSize is incorrect 
(CASSANDRA-12934)
 + * Prevent reloading of logback.xml from UDF sandbox (CASSANDRA-12535)
 + * Reenable HeapPool (CASSANDRA-12900)
 +Merged from 2.2:
+  * Coalescing strategy sleeps too much and shouldn't be enabled by default 
(CASSANDRA-13090)
   * Fix negative mean latency metric (CASSANDRA-12876)
   * Use only one file pointer when creating commitlog segments 
(CASSANDRA-12539)
   * Fix speculative retry bugs (CASSANDRA-13009)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/3d01e906/conf/cassandra.yaml
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/3d01e906/src/java/org/apache/cassandra/config/Config.java

[02/10] cassandra git commit: Coalescing strategies improvements CASSANDRA-13090

[aweisberg]

Coalescing strategies improvements CASSANDRA-13090

With the previous code TIMEHORIZON and MOVINGAVERAGE
coalesing strategy would wait even when the backlog
still contains data which would make it grow even more.

Also:
- cleanups parkLoop()
- add otc_coalescing_max_coalesced_messages
- add otc_coalescing_enough_coalesced_messages
- add other otc_* settings to cassandra.yaml

patch by Corentin Chary  reviewed by Ariel Weisberg 
 for CASSANDRA-13090


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5725e2c4
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5725e2c4
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5725e2c4

Branch: refs/heads/cassandra-3.0
Commit: 5725e2c422d21d8efe5ae3bc4389842939553650
Parents: 7e05f39
Author: Corentin Chary 
Authored: Mon Jan 9 12:06:56 2017 -0500
Committer: Ariel Weisberg 
Committed: Mon Feb 13 12:58:29 2017 -0500

--
 CHANGES.txt |  1 +
 conf/cassandra.yaml | 32 
 .../org/apache/cassandra/config/Config.java |  3 +-
 .../cassandra/config/DatabaseDescriptor.java| 16 
 .../cassandra/net/OutboundTcpConnection.java|  4 +-
 .../cassandra/utils/CoalescingStrategies.java   | 32 
 .../utils/CoalescingStrategiesTest.java | 40 
 7 files changed, 119 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/5725e2c4/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 214fe97..4052b0f 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.9
+ * Coalescing strategy sleeps too much and shouldn't be enabled by default 
(CASSANDRA-13090)
  * Fix negative mean latency metric (CASSANDRA-12876)
  * Use only one file pointer when creating commitlog segments (CASSANDRA-12539)
  * Fix speculative retry bugs (CASSANDRA-13009)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/5725e2c4/conf/cassandra.yaml
--
diff --git a/conf/cassandra.yaml b/conf/cassandra.yaml
index ab2aa14..ddb7927 100644
--- a/conf/cassandra.yaml
+++ b/conf/cassandra.yaml
@@ -884,3 +884,35 @@ enable_user_defined_functions: false
 # below their system default. The sysinternals 'clockres' tool can confirm 
your system's default
 # setting.
 windows_timer_interval: 1
+
+# Maximum size of any value in SSTables. Safety measure to detect SSTable 
corruption
+# early. Any value size larger than this threshold will result into marking an 
SSTable
+# as corrupted.
+# max_value_size_in_mb: 256
+
+# Coalescing Strategies #
+# Coalescing multiples messages turns out to significantly boost message 
processing throughput (think doubling or more).
+# On bare metal, the floor for packet processing throughput is high enough 
that many applications won’t notice, but in
+# virtualized environments, the point at which an application can be bound by 
network packet processing can be
+# surprisingly low compared to the throughput of task processing that is 
possible inside a VM. It’s not that bare metal
+# doesn’t benefit from coalescing messages, it’s that the number of 
packets a bare metal network interface can process
+# is sufficient for many applications such that no load starvation is 
experienced even without coalescing.
+# There are other benefits to coalescing network messages that are harder to 
isolate with a simple metric like messages
+# per second. By coalescing multiple tasks together, a network thread can 
process multiple messages for the cost of one
+# trip to read from a socket, and all the task submission work can be done at 
the same time reducing context switching
+# and increasing cache friendliness of network message processing.
+# See CASSANDRA-8692 for details.
+
+# Strategy to use for coalescing messages in OutboundTcpConnection.
+# Can be fixed, movingaverage, timehorizon (default), disabled.
+# You can also specify a subclass of CoalescingStrategies.CoalescingStrategy 
by name.
+# otc_coalescing_strategy: TIMEHORIZON
+
+# How many microseconds to wait for coalescing. For fixed strategy this is the 
amount of time after the first
+# message is received before it will be sent with any accompanying messages. 
For moving average this is the
+# maximum amount of time that will be waited as well as the interval at which 
messages must arrive on average
+# for coalescing to be enabled.
+# otc_coalescing_window_us: 200
+
+# Do not try to coalesce messages if we already got that many messages. This 
should be more than 2 and less than 128.
+# 

[10/10] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

[aweisberg]

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/edcbef3e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/edcbef3e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/edcbef3e

Branch: refs/heads/trunk
Commit: edcbef3e343778b4d5affe019f64c89da2a13aa2
Parents: ffc82b1 702ec08
Author: Ariel Weisberg 
Authored: Mon Feb 13 13:33:11 2017 -0500
Committer: Ariel Weisberg 
Committed: Mon Feb 13 13:34:31 2017 -0500

--
 CHANGES.txt |  1 +
 conf/cassandra.yaml | 27 +
 .../org/apache/cassandra/config/Config.java |  3 +-
 .../cassandra/config/DatabaseDescriptor.java| 16 
 .../cassandra/net/OutboundTcpConnection.java|  4 +-
 .../cassandra/utils/CoalescingStrategies.java   | 32 
 .../utils/CoalescingStrategiesTest.java | 40 
 7 files changed, 114 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/edcbef3e/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/edcbef3e/conf/cassandra.yaml
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/edcbef3e/src/java/org/apache/cassandra/config/Config.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/edcbef3e/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/edcbef3e/src/java/org/apache/cassandra/net/OutboundTcpConnection.java
--
diff --cc src/java/org/apache/cassandra/net/OutboundTcpConnection.java
index 574e9e7,ea64ef4..69480df
--- a/src/java/org/apache/cassandra/net/OutboundTcpConnection.java
+++ b/src/java/org/apache/cassandra/net/OutboundTcpConnection.java
@@@ -82,9 -83,8 +82,11 @@@ public class OutboundTcpConnection exte
  private static final String BUFFER_SIZE_PROPERTY = PREFIX + 
"otc_buffer_size";
  private static final int BUFFER_SIZE = 
Integer.getInteger(BUFFER_SIZE_PROPERTY, 1024 * 64);
  
 +//Size of 3 elements added to every message
 +private static final int PROTOCOL_MAGIC_ID_TIMESTAMP_SIZE = 12;
 +
+ public static final int MAX_COALESCED_MESSAGES = 128;
+ 
  private static CoalescingStrategy newCoalescingStrategy(String 
displayName)
  {
  return 
CoalescingStrategies.newCoalescingStrategy(DatabaseDescriptor.getOtcCoalescingStrategy(),

[03/10] cassandra git commit: Coalescing strategies improvements CASSANDRA-13090

[aweisberg]

Coalescing strategies improvements CASSANDRA-13090

With the previous code TIMEHORIZON and MOVINGAVERAGE
coalesing strategy would wait even when the backlog
still contains data which would make it grow even more.

Also:
- cleanups parkLoop()
- add otc_coalescing_max_coalesced_messages
- add otc_coalescing_enough_coalesced_messages
- add other otc_* settings to cassandra.yaml

patch by Corentin Chary  reviewed by Ariel Weisberg 
 for CASSANDRA-13090


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5725e2c4
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5725e2c4
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5725e2c4

Branch: refs/heads/cassandra-3.11
Commit: 5725e2c422d21d8efe5ae3bc4389842939553650
Parents: 7e05f39
Author: Corentin Chary 
Authored: Mon Jan 9 12:06:56 2017 -0500
Committer: Ariel Weisberg 
Committed: Mon Feb 13 12:58:29 2017 -0500

--
 CHANGES.txt |  1 +
 conf/cassandra.yaml | 32 
 .../org/apache/cassandra/config/Config.java |  3 +-
 .../cassandra/config/DatabaseDescriptor.java| 16 
 .../cassandra/net/OutboundTcpConnection.java|  4 +-
 .../cassandra/utils/CoalescingStrategies.java   | 32 
 .../utils/CoalescingStrategiesTest.java | 40 
 7 files changed, 119 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/5725e2c4/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 214fe97..4052b0f 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.9
+ * Coalescing strategy sleeps too much and shouldn't be enabled by default 
(CASSANDRA-13090)
  * Fix negative mean latency metric (CASSANDRA-12876)
  * Use only one file pointer when creating commitlog segments (CASSANDRA-12539)
  * Fix speculative retry bugs (CASSANDRA-13009)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/5725e2c4/conf/cassandra.yaml
--
diff --git a/conf/cassandra.yaml b/conf/cassandra.yaml
index ab2aa14..ddb7927 100644
--- a/conf/cassandra.yaml
+++ b/conf/cassandra.yaml
@@ -884,3 +884,35 @@ enable_user_defined_functions: false
 # below their system default. The sysinternals 'clockres' tool can confirm 
your system's default
 # setting.
 windows_timer_interval: 1
+
+# Maximum size of any value in SSTables. Safety measure to detect SSTable 
corruption
+# early. Any value size larger than this threshold will result into marking an 
SSTable
+# as corrupted.
+# max_value_size_in_mb: 256
+
+# Coalescing Strategies #
+# Coalescing multiples messages turns out to significantly boost message 
processing throughput (think doubling or more).
+# On bare metal, the floor for packet processing throughput is high enough 
that many applications won’t notice, but in
+# virtualized environments, the point at which an application can be bound by 
network packet processing can be
+# surprisingly low compared to the throughput of task processing that is 
possible inside a VM. It’s not that bare metal
+# doesn’t benefit from coalescing messages, it’s that the number of 
packets a bare metal network interface can process
+# is sufficient for many applications such that no load starvation is 
experienced even without coalescing.
+# There are other benefits to coalescing network messages that are harder to 
isolate with a simple metric like messages
+# per second. By coalescing multiple tasks together, a network thread can 
process multiple messages for the cost of one
+# trip to read from a socket, and all the task submission work can be done at 
the same time reducing context switching
+# and increasing cache friendliness of network message processing.
+# See CASSANDRA-8692 for details.
+
+# Strategy to use for coalescing messages in OutboundTcpConnection.
+# Can be fixed, movingaverage, timehorizon (default), disabled.
+# You can also specify a subclass of CoalescingStrategies.CoalescingStrategy 
by name.
+# otc_coalescing_strategy: TIMEHORIZON
+
+# How many microseconds to wait for coalescing. For fixed strategy this is the 
amount of time after the first
+# message is received before it will be sent with any accompanying messages. 
For moving average this is the
+# maximum amount of time that will be waited as well as the interval at which 
messages must arrive on average
+# for coalescing to be enabled.
+# otc_coalescing_window_us: 200
+
+# Do not try to coalesce messages if we already got that many messages. This 
should be more than 2 and less than 128.
+# 

[07/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[aweisberg]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3d01e906
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3d01e906
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3d01e906

Branch: refs/heads/trunk
Commit: 3d01e906152f02f826a9663afb4a1232060ad584
Parents: 3a570d7 5725e2c
Author: Ariel Weisberg 
Authored: Mon Feb 13 13:00:20 2017 -0500
Committer: Ariel Weisberg 
Committed: Mon Feb 13 13:23:39 2017 -0500

--
 CHANGES.txt |  1 +
 conf/cassandra.yaml | 27 +
 .../org/apache/cassandra/config/Config.java |  3 +-
 .../cassandra/config/DatabaseDescriptor.java| 16 
 .../cassandra/net/OutboundTcpConnection.java|  4 +-
 .../cassandra/utils/CoalescingStrategies.java   | 32 
 .../utils/CoalescingStrategiesTest.java | 40 
 7 files changed, 114 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/3d01e906/CHANGES.txt
--
diff --cc CHANGES.txt
index ac3d1ed,4052b0f..b19550a
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,43 -1,5 +1,44 @@@
 -2.2.9
 +3.0.11
 + * Add vm.max_map_count StartupCheck (CASSANDRA-13008)
 + * Hint related logging should include the IP address of the destination in 
addition to 
 +   host ID (CASSANDRA-13205)
 + * Reloading logback.xml does not work (CASSANDRA-13173)
 + * Lightweight transactions temporarily fail after upgrade from 2.1 to 3.0 
(CASSANDRA-13109)
 + * Duplicate rows after upgrading from 2.1.16 to 3.0.10/3.9 (CASSANDRA-13125)
 + * Fix UPDATE queries with empty IN restrictions (CASSANDRA-13152)
 + * Abort or retry on failed hints delivery (CASSANDRA-13124)
 + * Fix handling of partition with partition-level deletion plus
 +   live rows in sstabledump (CASSANDRA-13177)
 + * Provide user workaround when system_schema.columns does not contain entries
 +   for a table that's in system_schema.tables (CASSANDRA-13180)
 + * Dump threads when unit tests time out (CASSANDRA-13117)
 + * Better error when modifying function permissions without explicit keyspace 
(CASSANDRA-12925)
 + * Indexer is not correctly invoked when building indexes over sstables 
(CASSANDRA-13075)
 + * Read repair is not blocking repair to finish in foreground repair 
(CASSANDRA-13115)
 + * Stress daemon help is incorrect (CASSANDRA-12563)
 + * Remove ALTER TYPE support (CASSANDRA-12443)
 + * Fix assertion for certain legacy range tombstone pattern (CASSANDRA-12203)
 + * Set javac encoding to utf-8 (CASSANDRA-11077)
 + * Replace empty strings with null values if they cannot be converted 
(CASSANDRA-12794)
 + * Fixed flacky SSTableRewriterTest: check file counts before calling 
validateCFS (CASSANDRA-12348)
 + * Fix deserialization of 2.x DeletedCells (CASSANDRA-12620)
 + * Add parent repair session id to anticompaction log message 
(CASSANDRA-12186)
 + * Improve contention handling on failure to acquire MV lock for streaming 
and hints (CASSANDRA-12905)
 + * Fix DELETE and UPDATE queries with empty IN restrictions (CASSANDRA-12829)
 + * Mark MVs as built after successful bootstrap (CASSANDRA-12984)
 + * Estimated TS drop-time histogram updated with Cell.NO_DELETION_TIME 
(CASSANDRA-13040)
 + * Nodetool compactionstats fails with NullPointerException (CASSANDRA-13021)
 + * Thread local pools never cleaned up (CASSANDRA-13033)
 + * Set RPC_READY to false when draining or if a node is marked as shutdown 
(CASSANDRA-12781)
 + * Make sure sstables only get committed when it's safe to discard commit log 
records (CASSANDRA-12956)
 + * Reject default_time_to_live option when creating or altering MVs 
(CASSANDRA-12868)
 + * Nodetool should use a more sane max heap size (CASSANDRA-12739)
 + * LocalToken ensures token values are cloned on heap (CASSANDRA-12651)
 + * AnticompactionRequestSerializer serializedSize is incorrect 
(CASSANDRA-12934)
 + * Prevent reloading of logback.xml from UDF sandbox (CASSANDRA-12535)
 + * Reenable HeapPool (CASSANDRA-12900)
 +Merged from 2.2:
+  * Coalescing strategy sleeps too much and shouldn't be enabled by default 
(CASSANDRA-13090)
   * Fix negative mean latency metric (CASSANDRA-12876)
   * Use only one file pointer when creating commitlog segments 
(CASSANDRA-12539)
   * Fix speculative retry bugs (CASSANDRA-13009)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/3d01e906/conf/cassandra.yaml
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/3d01e906/src/java/org/apache/cassandra/config/Config.java

[04/10] cassandra git commit: Coalescing strategies improvements CASSANDRA-13090

[aweisberg]

Coalescing strategies improvements CASSANDRA-13090

With the previous code TIMEHORIZON and MOVINGAVERAGE
coalesing strategy would wait even when the backlog
still contains data which would make it grow even more.

Also:
- cleanups parkLoop()
- add otc_coalescing_max_coalesced_messages
- add otc_coalescing_enough_coalesced_messages
- add other otc_* settings to cassandra.yaml

patch by Corentin Chary  reviewed by Ariel Weisberg 
 for CASSANDRA-13090


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5725e2c4
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5725e2c4
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5725e2c4

Branch: refs/heads/trunk
Commit: 5725e2c422d21d8efe5ae3bc4389842939553650
Parents: 7e05f39
Author: Corentin Chary 
Authored: Mon Jan 9 12:06:56 2017 -0500
Committer: Ariel Weisberg 
Committed: Mon Feb 13 12:58:29 2017 -0500

--
 CHANGES.txt |  1 +
 conf/cassandra.yaml | 32 
 .../org/apache/cassandra/config/Config.java |  3 +-
 .../cassandra/config/DatabaseDescriptor.java| 16 
 .../cassandra/net/OutboundTcpConnection.java|  4 +-
 .../cassandra/utils/CoalescingStrategies.java   | 32 
 .../utils/CoalescingStrategiesTest.java | 40 
 7 files changed, 119 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/5725e2c4/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 214fe97..4052b0f 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.9
+ * Coalescing strategy sleeps too much and shouldn't be enabled by default 
(CASSANDRA-13090)
  * Fix negative mean latency metric (CASSANDRA-12876)
  * Use only one file pointer when creating commitlog segments (CASSANDRA-12539)
  * Fix speculative retry bugs (CASSANDRA-13009)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/5725e2c4/conf/cassandra.yaml
--
diff --git a/conf/cassandra.yaml b/conf/cassandra.yaml
index ab2aa14..ddb7927 100644
--- a/conf/cassandra.yaml
+++ b/conf/cassandra.yaml
@@ -884,3 +884,35 @@ enable_user_defined_functions: false
 # below their system default. The sysinternals 'clockres' tool can confirm 
your system's default
 # setting.
 windows_timer_interval: 1
+
+# Maximum size of any value in SSTables. Safety measure to detect SSTable 
corruption
+# early. Any value size larger than this threshold will result into marking an 
SSTable
+# as corrupted.
+# max_value_size_in_mb: 256
+
+# Coalescing Strategies #
+# Coalescing multiples messages turns out to significantly boost message 
processing throughput (think doubling or more).
+# On bare metal, the floor for packet processing throughput is high enough 
that many applications won’t notice, but in
+# virtualized environments, the point at which an application can be bound by 
network packet processing can be
+# surprisingly low compared to the throughput of task processing that is 
possible inside a VM. It’s not that bare metal
+# doesn’t benefit from coalescing messages, it’s that the number of 
packets a bare metal network interface can process
+# is sufficient for many applications such that no load starvation is 
experienced even without coalescing.
+# There are other benefits to coalescing network messages that are harder to 
isolate with a simple metric like messages
+# per second. By coalescing multiple tasks together, a network thread can 
process multiple messages for the cost of one
+# trip to read from a socket, and all the task submission work can be done at 
the same time reducing context switching
+# and increasing cache friendliness of network message processing.
+# See CASSANDRA-8692 for details.
+
+# Strategy to use for coalescing messages in OutboundTcpConnection.
+# Can be fixed, movingaverage, timehorizon (default), disabled.
+# You can also specify a subclass of CoalescingStrategies.CoalescingStrategy 
by name.
+# otc_coalescing_strategy: TIMEHORIZON
+
+# How many microseconds to wait for coalescing. For fixed strategy this is the 
amount of time after the first
+# message is received before it will be sent with any accompanying messages. 
For moving average this is the
+# maximum amount of time that will be waited as well as the interval at which 
messages must arrive on average
+# for coalescing to be enabled.
+# otc_coalescing_window_us: 200
+
+# Do not try to coalesce messages if we already got that many messages. This 
should be more than 2 and less than 128.
+# otc_coalescing_enough_coalesced_messages: 8

[08/10] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[aweisberg]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/702ec088
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/702ec088
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/702ec088

Branch: refs/heads/cassandra-3.11
Commit: 702ec088f5f61106b41e128f3fb8f109da8cbe1c
Parents: 01b91cc 3d01e90
Author: Ariel Weisberg 
Authored: Mon Feb 13 13:31:23 2017 -0500
Committer: Ariel Weisberg 
Committed: Mon Feb 13 13:32:30 2017 -0500

--
 CHANGES.txt |  1 +
 conf/cassandra.yaml | 27 +
 .../org/apache/cassandra/config/Config.java |  3 +-
 .../cassandra/config/DatabaseDescriptor.java| 16 
 .../cassandra/net/OutboundTcpConnection.java|  4 +-
 .../cassandra/utils/CoalescingStrategies.java   | 32 
 .../utils/CoalescingStrategiesTest.java | 40 
 7 files changed, 114 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/702ec088/CHANGES.txt
--
diff --cc CHANGES.txt
index 67c45e8,b19550a..8164a52
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -215,20 -100,6 +215,21 @@@ Merged from 3.0
   * Correct log message for statistics of offheap memtable flush 
(CASSANDRA-12776)
   * Explicitly set locale for string validation 
(CASSANDRA-12541,CASSANDRA-12542,CASSANDRA-12543,CASSANDRA-12545)
  Merged from 2.2:
++ * Coalescing strategy sleeps too much and shouldn't be enabled by default 
(CASSANDRA-13090)
 + * Fix speculative retry bugs (CASSANDRA-13009)
 + * Fix handling of nulls and unsets in IN conditions (CASSANDRA-12981)
 + * Fix race causing infinite loop if Thrift server is stopped before it 
starts listening (CASSANDRA-12856)
 + * CompactionTasks now correctly drops sstables out of compaction when not 
enough disk space is available (CASSANDRA-12979)
 + * Remove support for non-JavaScript UDFs (CASSANDRA-12883)
 + * Fix DynamicEndpointSnitch noop in multi-datacenter situations 
(CASSANDRA-13074)
 + * cqlsh copy-from: encode column names to avoid primary key parsing errors 
(CASSANDRA-12909)
 + * Temporarily fix bug that creates commit log when running offline tools 
(CASSANDRA-8616)
 + * Reduce granuality of OpOrder.Group during index build (CASSANDRA-12796)
 + * Test bind parameters and unset parameters in InsertUpdateIfConditionTest 
(CASSANDRA-12980)
 + * Use saved tokens when setting local tokens on StorageService.joinRing 
(CASSANDRA-12935)
 + * cqlsh: fix DESC TYPES errors (CASSANDRA-12914)
 + * Fix leak on skipped SSTables in sstableupgrade (CASSANDRA-12899)
 + * Avoid blocking gossip during pending range calculation (CASSANDRA-12281)
   * Fix purgeability of tombstones with max timestamp (CASSANDRA-12792)
   * Fail repair if participant dies during sync or anticompaction 
(CASSANDRA-12901)
   * cqlsh COPY: unprotected pk values before converting them if not using 
prepared statements (CASSANDRA-12863)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/702ec088/conf/cassandra.yaml
--
diff --cc conf/cassandra.yaml
index 4436a02,a9d4c01..063a0b7
--- a/conf/cassandra.yaml
+++ b/conf/cassandra.yaml
@@@ -1173,24 -959,29 +1173,51 @@@ gc_warn_threshold_in_ms: 100
  # as corrupted.
  # max_value_size_in_mb: 256
  
 +# Back-pressure settings #
 +# If enabled, the coordinator will apply the back-pressure strategy specified 
below to each mutation
 +# sent to replicas, with the aim of reducing pressure on overloaded replicas.
 +back_pressure_enabled: false
 +# The back-pressure strategy applied.
 +# The default implementation, RateBasedBackPressure, takes three arguments:
 +# high ratio, factor, and flow type, and uses the ratio between incoming 
mutation responses and outgoing mutation requests.
 +# If below high ratio, outgoing mutations are rate limited according to the 
incoming rate decreased by the given factor;
 +# if above high ratio, the rate limiting is increased by the given factor;
 +# such factor is usually best configured between 1 and 10, use larger values 
for a faster recovery
 +# at the expense of potentially more dropped mutations;
 +# the rate limiting is applied according to the flow type: if FAST, it's rate 
limited at the speed of the fastest replica,
 +# if SLOW at the speed of the slowest one.
 +# New strategies can be added. Implementors need to implement 
org.apache.cassandra.net.BackpressureStrategy and
 +# provide a public constructor accepting a Map.
 +back_pressure_strategy:
 +- class_name: org.apache.cassandra.net.RateBasedBackPressure
 +  parameters:
 +

[01/10] cassandra git commit: Coalescing strategies improvements CASSANDRA-13090

[aweisberg]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.2 7e05f393f -> 5725e2c42
  refs/heads/cassandra-3.0 3a570d744 -> 3d01e9061
  refs/heads/cassandra-3.11 01b91cce2 -> 702ec088f
  refs/heads/trunk ffc82b1ee -> edcbef3e3


Coalescing strategies improvements CASSANDRA-13090

With the previous code TIMEHORIZON and MOVINGAVERAGE
coalesing strategy would wait even when the backlog
still contains data which would make it grow even more.

Also:
- cleanups parkLoop()
- add otc_coalescing_max_coalesced_messages
- add otc_coalescing_enough_coalesced_messages
- add other otc_* settings to cassandra.yaml

patch by Corentin Chary  reviewed by Ariel Weisberg 
 for CASSANDRA-13090


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5725e2c4
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5725e2c4
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5725e2c4

Branch: refs/heads/cassandra-2.2
Commit: 5725e2c422d21d8efe5ae3bc4389842939553650
Parents: 7e05f39
Author: Corentin Chary 
Authored: Mon Jan 9 12:06:56 2017 -0500
Committer: Ariel Weisberg 
Committed: Mon Feb 13 12:58:29 2017 -0500

--
 CHANGES.txt |  1 +
 conf/cassandra.yaml | 32 
 .../org/apache/cassandra/config/Config.java |  3 +-
 .../cassandra/config/DatabaseDescriptor.java| 16 
 .../cassandra/net/OutboundTcpConnection.java|  4 +-
 .../cassandra/utils/CoalescingStrategies.java   | 32 
 .../utils/CoalescingStrategiesTest.java | 40 
 7 files changed, 119 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/5725e2c4/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 214fe97..4052b0f 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.9
+ * Coalescing strategy sleeps too much and shouldn't be enabled by default 
(CASSANDRA-13090)
  * Fix negative mean latency metric (CASSANDRA-12876)
  * Use only one file pointer when creating commitlog segments (CASSANDRA-12539)
  * Fix speculative retry bugs (CASSANDRA-13009)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/5725e2c4/conf/cassandra.yaml
--
diff --git a/conf/cassandra.yaml b/conf/cassandra.yaml
index ab2aa14..ddb7927 100644
--- a/conf/cassandra.yaml
+++ b/conf/cassandra.yaml
@@ -884,3 +884,35 @@ enable_user_defined_functions: false
 # below their system default. The sysinternals 'clockres' tool can confirm 
your system's default
 # setting.
 windows_timer_interval: 1
+
+# Maximum size of any value in SSTables. Safety measure to detect SSTable 
corruption
+# early. Any value size larger than this threshold will result into marking an 
SSTable
+# as corrupted.
+# max_value_size_in_mb: 256
+
+# Coalescing Strategies #
+# Coalescing multiples messages turns out to significantly boost message 
processing throughput (think doubling or more).
+# On bare metal, the floor for packet processing throughput is high enough 
that many applications won’t notice, but in
+# virtualized environments, the point at which an application can be bound by 
network packet processing can be
+# surprisingly low compared to the throughput of task processing that is 
possible inside a VM. It’s not that bare metal
+# doesn’t benefit from coalescing messages, it’s that the number of 
packets a bare metal network interface can process
+# is sufficient for many applications such that no load starvation is 
experienced even without coalescing.
+# There are other benefits to coalescing network messages that are harder to 
isolate with a simple metric like messages
+# per second. By coalescing multiple tasks together, a network thread can 
process multiple messages for the cost of one
+# trip to read from a socket, and all the task submission work can be done at 
the same time reducing context switching
+# and increasing cache friendliness of network message processing.
+# See CASSANDRA-8692 for details.
+
+# Strategy to use for coalescing messages in OutboundTcpConnection.
+# Can be fixed, movingaverage, timehorizon (default), disabled.
+# You can also specify a subclass of CoalescingStrategies.CoalescingStrategy 
by name.
+# otc_coalescing_strategy: TIMEHORIZON
+
+# How many microseconds to wait for coalescing. For fixed strategy this is the 
amount of time after the first
+# message is received before it will be sent with any accompanying messages. 
For moving average this is the
+# maximum amount of time that will be waited as well as the interval at which 
messages must arrive on 

[06/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[aweisberg]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3d01e906
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3d01e906
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3d01e906

Branch: refs/heads/cassandra-3.0
Commit: 3d01e906152f02f826a9663afb4a1232060ad584
Parents: 3a570d7 5725e2c
Author: Ariel Weisberg 
Authored: Mon Feb 13 13:00:20 2017 -0500
Committer: Ariel Weisberg 
Committed: Mon Feb 13 13:23:39 2017 -0500

--
 CHANGES.txt |  1 +
 conf/cassandra.yaml | 27 +
 .../org/apache/cassandra/config/Config.java |  3 +-
 .../cassandra/config/DatabaseDescriptor.java| 16 
 .../cassandra/net/OutboundTcpConnection.java|  4 +-
 .../cassandra/utils/CoalescingStrategies.java   | 32 
 .../utils/CoalescingStrategiesTest.java | 40 
 7 files changed, 114 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/3d01e906/CHANGES.txt
--
diff --cc CHANGES.txt
index ac3d1ed,4052b0f..b19550a
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,43 -1,5 +1,44 @@@
 -2.2.9
 +3.0.11
 + * Add vm.max_map_count StartupCheck (CASSANDRA-13008)
 + * Hint related logging should include the IP address of the destination in 
addition to 
 +   host ID (CASSANDRA-13205)
 + * Reloading logback.xml does not work (CASSANDRA-13173)
 + * Lightweight transactions temporarily fail after upgrade from 2.1 to 3.0 
(CASSANDRA-13109)
 + * Duplicate rows after upgrading from 2.1.16 to 3.0.10/3.9 (CASSANDRA-13125)
 + * Fix UPDATE queries with empty IN restrictions (CASSANDRA-13152)
 + * Abort or retry on failed hints delivery (CASSANDRA-13124)
 + * Fix handling of partition with partition-level deletion plus
 +   live rows in sstabledump (CASSANDRA-13177)
 + * Provide user workaround when system_schema.columns does not contain entries
 +   for a table that's in system_schema.tables (CASSANDRA-13180)
 + * Dump threads when unit tests time out (CASSANDRA-13117)
 + * Better error when modifying function permissions without explicit keyspace 
(CASSANDRA-12925)
 + * Indexer is not correctly invoked when building indexes over sstables 
(CASSANDRA-13075)
 + * Read repair is not blocking repair to finish in foreground repair 
(CASSANDRA-13115)
 + * Stress daemon help is incorrect (CASSANDRA-12563)
 + * Remove ALTER TYPE support (CASSANDRA-12443)
 + * Fix assertion for certain legacy range tombstone pattern (CASSANDRA-12203)
 + * Set javac encoding to utf-8 (CASSANDRA-11077)
 + * Replace empty strings with null values if they cannot be converted 
(CASSANDRA-12794)
 + * Fixed flacky SSTableRewriterTest: check file counts before calling 
validateCFS (CASSANDRA-12348)
 + * Fix deserialization of 2.x DeletedCells (CASSANDRA-12620)
 + * Add parent repair session id to anticompaction log message 
(CASSANDRA-12186)
 + * Improve contention handling on failure to acquire MV lock for streaming 
and hints (CASSANDRA-12905)
 + * Fix DELETE and UPDATE queries with empty IN restrictions (CASSANDRA-12829)
 + * Mark MVs as built after successful bootstrap (CASSANDRA-12984)
 + * Estimated TS drop-time histogram updated with Cell.NO_DELETION_TIME 
(CASSANDRA-13040)
 + * Nodetool compactionstats fails with NullPointerException (CASSANDRA-13021)
 + * Thread local pools never cleaned up (CASSANDRA-13033)
 + * Set RPC_READY to false when draining or if a node is marked as shutdown 
(CASSANDRA-12781)
 + * Make sure sstables only get committed when it's safe to discard commit log 
records (CASSANDRA-12956)
 + * Reject default_time_to_live option when creating or altering MVs 
(CASSANDRA-12868)
 + * Nodetool should use a more sane max heap size (CASSANDRA-12739)
 + * LocalToken ensures token values are cloned on heap (CASSANDRA-12651)
 + * AnticompactionRequestSerializer serializedSize is incorrect 
(CASSANDRA-12934)
 + * Prevent reloading of logback.xml from UDF sandbox (CASSANDRA-12535)
 + * Reenable HeapPool (CASSANDRA-12900)
 +Merged from 2.2:
+  * Coalescing strategy sleeps too much and shouldn't be enabled by default 
(CASSANDRA-13090)
   * Fix negative mean latency metric (CASSANDRA-12876)
   * Use only one file pointer when creating commitlog segments 
(CASSANDRA-12539)
   * Fix speculative retry bugs (CASSANDRA-13009)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/3d01e906/conf/cassandra.yaml
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/3d01e906/src/java/org/apache/cassandra/config/Config.java

[jira] [Updated] (CASSANDRA-13211) cassandra shell script uses bad approach to write "Unable to find java" error to stderr, causing real issue to be masked by a permission error if user has changed us

[Michael Shuler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Shuler updated CASSANDRA-13211:
---
Attachment: 13211_use-portable-stderr.patch

cassandra-2.1-based patch attached. A test job will never hit this, since we 
set up java properly on test servers before doing any testing.

> cassandra shell script uses bad approach to write "Unable to find java" error 
> to stderr, causing real issue to be masked by a permission error if user has 
> changed user since logging in
> 
>
> Key: CASSANDRA-13211
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13211
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Max Bowsher
>Assignee: Michael Shuler
> Attachments: 13211_use-portable-stderr.patch
>
>
> The cassandra startup shell script contains this line:
> echo Unable to find java executable. Check JAVA_HOME and PATH environment 
> variables. > /dev/stderr
> The problem here is the construct "> /dev/stderr". If the user invoking 
> Cassandra has changed user (for example, by SSHing in as a personal user, and 
> then sudo-ing to an application user responsible for executing the Cassandra 
> daemon), then the attempt to open /dev/stderr will fail, because it will 
> point to a PTY node under /dev/pts/ owned by the original user.
> Ultimately this leads to the real problem being masked by the confusing error 
> message "bash: /dev/stderr: Permission denied".
> The correct technique is to replace "> /dev/stderr" with ">&2" which will 
> write to the already open stderr file descriptor, instead of resolving the 
> chain of symlinks starting at /dev/stderr, and attempting to reopen the 
> target by name.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (CASSANDRA-13211) cassandra shell script uses bad approach to write "Unable to find java" error to stderr, causing real issue to be masked by a permission error if user has changed u

[Michael Shuler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Shuler reassigned CASSANDRA-13211:
--

Assignee: Michael Shuler

> cassandra shell script uses bad approach to write "Unable to find java" error 
> to stderr, causing real issue to be masked by a permission error if user has 
> changed user since logging in
> 
>
> Key: CASSANDRA-13211
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13211
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Max Bowsher
>Assignee: Michael Shuler
>
> The cassandra startup shell script contains this line:
> echo Unable to find java executable. Check JAVA_HOME and PATH environment 
> variables. > /dev/stderr
> The problem here is the construct "> /dev/stderr". If the user invoking 
> Cassandra has changed user (for example, by SSHing in as a personal user, and 
> then sudo-ing to an application user responsible for executing the Cassandra 
> daemon), then the attempt to open /dev/stderr will fail, because it will 
> point to a PTY node under /dev/pts/ owned by the original user.
> Ultimately this leads to the real problem being masked by the confusing error 
> message "bash: /dev/stderr: Permission denied".
> The correct technique is to replace "> /dev/stderr" with ">&2" which will 
> write to the already open stderr file descriptor, instead of resolving the 
> chain of symlinks starting at /dev/stderr, and attempting to reopen the 
> target by name.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (CASSANDRA-13211) cassandra shell script uses bad approach to write "Unable to find java" error to stderr, causing real issue to be masked by a permission error if user has cha

[Michael Shuler (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864167#comment-15864167
 ] 

Michael Shuler edited comment on CASSANDRA-13211 at 2/13/17 6:36 PM:
-

I think this change to {{echo ... >&2}} would be fine and accomplishes the 
same. (When I move my /usr/bin/java out of the way, I get trapped earlier in 
the version check in cassandra-env.sh, since no java can be found)


was (Author: mshuler):
I think this change to {{ echo ... >&2}} would be fine and accomplishes the 
same. (When I move my /usr/bin/java out of the way, I get trapped earlier in 
the version check in cassandra-env.sh, since no java can be found)

> cassandra shell script uses bad approach to write "Unable to find java" error 
> to stderr, causing real issue to be masked by a permission error if user has 
> changed user since logging in
> 
>
> Key: CASSANDRA-13211
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13211
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Max Bowsher
>
> The cassandra startup shell script contains this line:
> echo Unable to find java executable. Check JAVA_HOME and PATH environment 
> variables. > /dev/stderr
> The problem here is the construct "> /dev/stderr". If the user invoking 
> Cassandra has changed user (for example, by SSHing in as a personal user, and 
> then sudo-ing to an application user responsible for executing the Cassandra 
> daemon), then the attempt to open /dev/stderr will fail, because it will 
> point to a PTY node under /dev/pts/ owned by the original user.
> Ultimately this leads to the real problem being masked by the confusing error 
> message "bash: /dev/stderr: Permission denied".
> The correct technique is to replace "> /dev/stderr" with ">&2" which will 
> write to the already open stderr file descriptor, instead of resolving the 
> chain of symlinks starting at /dev/stderr, and attempting to reopen the 
> target by name.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13211) cassandra shell script uses bad approach to write "Unable to find java" error to stderr, causing real issue to be masked by a permission error if user has changed

[Michael Shuler (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864167#comment-15864167
 ] 

Michael Shuler commented on CASSANDRA-13211:


I think this change to {{ echo ... >&2}} would be fine and accomplishes the 
same. (When I move my /usr/bin/java out of the way, I get trapped earlier in 
the version check in cassandra-env.sh, since no java can be found)

> cassandra shell script uses bad approach to write "Unable to find java" error 
> to stderr, causing real issue to be masked by a permission error if user has 
> changed user since logging in
> 
>
> Key: CASSANDRA-13211
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13211
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Max Bowsher
>
> The cassandra startup shell script contains this line:
> echo Unable to find java executable. Check JAVA_HOME and PATH environment 
> variables. > /dev/stderr
> The problem here is the construct "> /dev/stderr". If the user invoking 
> Cassandra has changed user (for example, by SSHing in as a personal user, and 
> then sudo-ing to an application user responsible for executing the Cassandra 
> daemon), then the attempt to open /dev/stderr will fail, because it will 
> point to a PTY node under /dev/pts/ owned by the original user.
> Ultimately this leads to the real problem being masked by the confusing error 
> message "bash: /dev/stderr: Permission denied".
> The correct technique is to replace "> /dev/stderr" with ">&2" which will 
> write to the already open stderr file descriptor, instead of resolving the 
> chain of symlinks starting at /dev/stderr, and attempting to reopen the 
> target by name.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13213) compactionstats not available, node eventually OOMs due to pending mutations

[Zachary Girouard (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864147#comment-15864147
 ] 

Zachary Girouard commented on CASSANDRA-13213:
--

Correct. I was not running any operations on sstables when this issue was 
happening. 

After turning off autocompaction I went through this weekend and ran 
relocatesstables on each node (one at a time). No issues while the relocate was 
running.

> compactionstats not available, node eventually OOMs due to pending mutations
> 
>
> Key: CASSANDRA-13213
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13213
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction, Local Write-Read Paths
>Reporter: Zachary Girouard
>
> I'm seeing semi-frequent instances of nodetool compactionstats hanging 
> forever. While this is occurring none of the compaction metrics are available 
> via jmx/jconsole.
> Sometimes the node will eventually recover, but I'm seeing a pattern where a 
> node will exhibit this behavior, and then eventually pending mutations start 
> piling up and the node dies due to OOM. Sometimes pending gossip operations 
> starting piling up too, but I think this is due to the impending OOM causing 
> everything to bog down.
> As an experiment I turned auto compaction off on all the nodes and I haven't 
> seen this issue occur since I did that. Additionally, I'm running 
> relocatesstables on some nodes with unthrottled compaction and so far none of 
> them have had any issues handling it.
> I managed to get some stack traces from a dying node:
> All MutationStage threads look similar to this:
> {noformat}
> Name: MutationStage-10
>   
> State: WAITING
> Total blocked: 9  Total waited: 1,959,850
> Stack trace: 
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(Unknown Source)
> org.apache.cassandra.utils.concurrent.WaitQueue$AbstractSignal.awaitUninterruptibly(WaitQueue.java:279)
> org.apache.cassandra.utils.memory.MemtableAllocator$SubAllocator.allocate(MemtableAllocator.java:162)
> org.apache.cassandra.utils.memory.SlabAllocator.allocate(SlabAllocator.java:89)
> org.apache.cassandra.utils.memory.ContextAllocator.allocate(ContextAllocator.java:57)
> org.apache.cassandra.utils.memory.ContextAllocator.clone(ContextAllocator.java:47)
> org.apache.cassandra.db.rows.BufferCell.copy(BufferCell.java:122)
> org.apache.cassandra.utils.memory.AbstractAllocator$CloningBTreeRowBuilder.addCell(AbstractAllocator.java:72)
> org.apache.cassandra.db.rows.Rows.copy(Rows.java:51)
> org.apache.cassandra.db.partitions.AtomicBTreePartition$RowUpdater.apply(AtomicBTreePartition.java:332)
> org.apache.cassandra.db.partitions.AtomicBTreePartition$RowUpdater.apply(AtomicBTreePartition.java:295)
> org.apache.cassandra.utils.btree.NodeBuilder.addNewKey(NodeBuilder.java:323)
> org.apache.cassandra.utils.btree.NodeBuilder.update(NodeBuilder.java:184)
> org.apache.cassandra.utils.btree.TreeBuilder.update(TreeBuilder.java:95)
> org.apache.cassandra.utils.btree.BTree.update(BTree.java:182)
> org.apache.cassandra.db.partitions.AtomicBTreePartition.addAllWithSizeDelta(AtomicBTreePartition.java:156)
> org.apache.cassandra.db.Memtable.put(Memtable.java:284)org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1316)
> org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:618)
> org.apache.cassandra.db.Keyspace.applyFuture(Keyspace.java:425)
> org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:222)
> org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:68)
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
> java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
> java.lang.Thread.run(Unknown Source)
> {noformat}
> The Compaction threads
> {noformat}
> Name: CompactionExecutor:4
> State: RUNNABLE
> Total blocked: 32,781,277  Total waited: 549
> Stack trace: 
> org.apache.cassandra.io.sstable.format.SSTableReader.getTotalBytes(SSTableReader.java:661)
> org.apache.cassandra.db.compaction.LeveledManifest.getCandidatesFor(LeveledManifest.java:669)
> org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:385)
>- locked org.apache.cassandra.db.compaction.LeveledManifest@55c79600
>
> 

[jira] [Commented] (CASSANDRA-13211) cassandra shell script uses bad approach to write "Unable to find java" error to stderr, causing real issue to be masked by a permission error if user has changed

[Brandon Williams (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864079#comment-15864079
 ] 

Brandon Williams commented on CASSANDRA-13211:
--

Ping [~mshuler]

> cassandra shell script uses bad approach to write "Unable to find java" error 
> to stderr, causing real issue to be masked by a permission error if user has 
> changed user since logging in
> 
>
> Key: CASSANDRA-13211
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13211
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Max Bowsher
>
> The cassandra startup shell script contains this line:
> echo Unable to find java executable. Check JAVA_HOME and PATH environment 
> variables. > /dev/stderr
> The problem here is the construct "> /dev/stderr". If the user invoking 
> Cassandra has changed user (for example, by SSHing in as a personal user, and 
> then sudo-ing to an application user responsible for executing the Cassandra 
> daemon), then the attempt to open /dev/stderr will fail, because it will 
> point to a PTY node under /dev/pts/ owned by the original user.
> Ultimately this leads to the real problem being masked by the confusing error 
> message "bash: /dev/stderr: Permission denied".
> The correct technique is to replace "> /dev/stderr" with ">&2" which will 
> write to the already open stderr file descriptor, instead of resolving the 
> chain of symlinks starting at /dev/stderr, and attempting to reopen the 
> target by name.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-9778) CQL support for time series aggregation

[Benjamin Lerer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Lerer updated CASSANDRA-9778:
--
Fix Version/s: (was: 3.11.x)

> CQL support for time series aggregation
> ---
>
> Key: CASSANDRA-9778
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9778
> Project: Cassandra
>  Issue Type: New Feature
>  Components: CQL
>Reporter: Jonathan Ellis
>Assignee: Benjamin Lerer
>
> Along with MV (CASSANDRA-6477), time series aggregation or "rollups" are a 
> common design pattern in cassandra applications.  I'd like to add CQL support 
> for this along these lines:
> {code}
> CREATE MATERIALIZED VIEW stocks_by_hour AS
> SELECT exchange, day, day_time(1h) AS hour, symbol, avg(price), sum(volume)
> FROM stocks
> GROUP BY exchange, day, symbol, hour
> PRIMARY KEY  ((exchange, day), hour, symbol);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13088) Update Cassandra Configuration File page with undocumented options

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863851#comment-15863851
 ] 

Stefan Podkowinski commented on CASSANDRA-13088:


The mentioned document is generated based on the stock {{cassandra.yaml}} using 
the {{convert_yaml_to_rst.py}} script. We'd have to either add the mentioned 
options to the yaml file (which is already +1k lines) or modify the script to 
include additional content. 

> Update Cassandra Configuration File page with undocumented options
> --
>
> Key: CASSANDRA-13088
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13088
> Project: Cassandra
>  Issue Type: Bug
>  Components: Documentation and Website
>Reporter: Alwyn Davis
>Priority: Trivial
>
> The documentation site doesn't cover all configuration options 
> (http://cassandra.apache.org/doc/latest/configuration/cassandra_config_file.html),
>  including (based on trunk and excluding deprecated options):
> * auto_bootstrap
> * commitlog_max_compression_buffers_in_pool
> * credentials_cache_max_entries
> * disk_access_mode
> * disk_optimization_estimate_percentile
> * disk_optimization_page_cross_chance
> * dynamic_snitch
> * enable_user_defined_functions_threads
> * max_mutation_size_in_kb
> * otc_coalescing_strategy
> * otc_coalescing_window_us
> * otc_coalescing_window_us_default
> * outboundBindAny
> * permissions_cache_max_entries
> * roles_cache_max_entries
> * rpc_listen_backlog
> * user_defined_function_fail_timeout
> * user_defined_function_warn_timeout
> * user_function_timeout_policy
> I'm not sure if some are intentionally omitted, but others would appear to be 
> useful e.g. {{max_mutation_size_in_kb}}. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (CASSANDRA-13150) Error in DDL documentation clustering columns example

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski resolved CASSANDRA-13150.

Resolution: Duplicate

> Error in DDL documentation clustering columns example
> -
>
> Key: CASSANDRA-13150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13150
> Project: Cassandra
>  Issue Type: Bug
>  Components: Documentation and Website
>Reporter: Adrian Hempel
>
> The following line:
> https://git1-us-west.apache.org/repos/asf?p=cassandra.git;a=blob;f=doc/source/cql/ddl.rst;h=fb97e54388b4cea5b84b9eb533904e43f0b18ec7;hb=HEAD#l362
> Should read:
> {{PRIMARY KEY (a, b, c)}}
> rather than:
> {{PRIMARY KEY (a, c, d)}}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (CASSANDRA-12727) "nodetool disablebinary/thrift"—the documentation does not clearly say whether this works as a drain or an immediate cut off

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski resolved CASSANDRA-12727.

Resolution: Invalid

Please 
[contact|http://docs.datastax.com/en/landing_page/doc/landing_page/contact.html]
 DataStax directly regarding any issues with the DataStax documentation.

> "nodetool disablebinary/thrift"—the documentation does not clearly say 
> whether this works as a drain or an immediate cut off
> 
>
> Key: CASSANDRA-12727
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12727
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Documentation and Website
>Reporter: Alexis Wilke
>Priority: Minor
>
> The documentation, as found here:
> https://docs.datastax.com/en/cassandra/3.x/cassandra/tools/toolsDisableBinary.html
> Says:
> bq. Description 
> bq. Disables the binary protocol, also known as the native transport.
> It is not clear to me whether this means existing connections will be drained 
> before they get closed, or whether those connections all get closed 
> immediately, whatever their state.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (CASSANDRA-9236) Document FULL keyword in CREATE INDEX statements

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski resolved CASSANDRA-9236.
---
Resolution: Won't Fix

Wrong Jira? Please 
[contact|http://docs.datastax.com/en/landing_page/doc/landing_page/contact.html]
 DataStax directly regarding any issues with the DataStax documentation.

> Document FULL keyword in CREATE INDEX statements
> 
>
> Key: CASSANDRA-9236
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9236
> Project: Cassandra
>  Issue Type: Task
>  Components: Documentation and Website
>Reporter: Jim Witschey
>Priority: Minor
> Fix For: 2.1.x
>
>
> [The docs page for CREATE INDEX| 
> http://docs.datastax.com/en/cql/3.1/cql/cql_reference/create_index_r.html] 
> should cover statements like {{CREATE INDEX my_idx ON my_table 
> (FULL(my_col))}}, which (according to [~thobbs] on IRC) is/can be used when 
> {{my_col}} is frozen.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-12365) Document cassandra stress

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863793#comment-15863793
 ] 

Stefan Podkowinski commented on CASSANDRA-12365:


Looks good, let's get this finally committed. Can you attach a patch with the 
standard message footer ("patch by ..") as described in the 
[docs|http://cassandra.apache.org/doc/latest/development/patches.html#creating-a-patch]?

> Document cassandra stress
> -
>
> Key: CASSANDRA-12365
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12365
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Documentation and Website
>Reporter: Christopher Batey
>Assignee: Christopher Batey
>Priority: Minor
>  Labels: stress
> Fix For: 3.11.x
>
>
> I've started on this so creating a ticket to avoid duplicate work.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (CASSANDRA-13216) testall failure in org.apache.cassandra.net.MessagingServiceTest.testDroppedMessages

[Sean McCarthy (JIRA)]

Sean McCarthy created CASSANDRA-13216:
-

 Summary: testall failure in 
org.apache.cassandra.net.MessagingServiceTest.testDroppedMessages
 Key: CASSANDRA-13216
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13216
 Project: Cassandra
  Issue Type: Bug
  Components: Testing
Reporter: Sean McCarthy
 Attachments: TEST-org.apache.cassandra.net.MessagingServiceTest.log

example failure:

http://cassci.datastax.com/job/cassandra-3.11_testall/81/testReport/org.apache.cassandra.net/MessagingServiceTest/testDroppedMessages

{code}
Error Message

expected:<... dropped latency: 27[30 ms and Mean cross-node dropped latency: 
2731] ms> but was:<... dropped latency: 27[28 ms and Mean cross-node dropped 
latency: 2730] ms>
{code}{code}
Stacktrace

junit.framework.AssertionFailedError: expected:<... dropped latency: 27[30 ms 
and Mean cross-node dropped latency: 2731] ms> but was:<... dropped latency: 
27[28 ms and Mean cross-node dropped latency: 2730] ms>
at 
org.apache.cassandra.net.MessagingServiceTest.testDroppedMessages(MessagingServiceTest.java:83)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-8457) nio MessagingService

[Norman Maurer (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863760#comment-15863760
 ] 

Norman Maurer commented on CASSANDRA-8457:
--

[~jasobrown] whooot :) I will do another review as well, just to double-check 
again.

> nio MessagingService
> 
>
> Key: CASSANDRA-8457
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8457
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Jonathan Ellis
>Assignee: Jason Brown
>Priority: Minor
>  Labels: netty, performance
> Fix For: 4.x
>
>
> Thread-per-peer (actually two each incoming and outbound) is a big 
> contributor to context switching, especially for larger clusters.  Let's look 
> at switching to nio, possibly via Netty.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-8457) nio MessagingService

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863758#comment-15863758
 ] 

Jason Brown commented on CASSANDRA-8457:


OK, so I've been performance load testing the snot of this code for the last 
several weeks, with help from netty committers, flight recorder, and flame 
graphs. As a result, I've made some major and some minor tweaks, and now I'm 
slightly faster than trunk with slightly better throughput. I have some 
optimizations in my back pocket that will increase even more, but as Sylvain 
has stated before, we'll save those for follow up tickets.

trunk
{code}
 id, type   total ops,op/s,pk/s,   row/s,mean, 
med, .95, .99,.999, max,   time,   stderr, errors,  gc: #,  max 
ms,  sum ms,  sdv ms,  mb
  4 threadCount, total,233344,3889,3889,3889, 1.0, 
1.0, 1.2, 1.3, 1.5,68.2,   60.0,  0.01549,  0,  9, 
538, 538,   4,5381
  8 threadCount, total,544637,9076,9076,9076, 0.8, 
0.8, 1.0, 1.1, 1.4,73.8,   60.0,  0.00978,  0, 20,
1267,1267,   5,   11848
 16 threadCount, total,   1126627,   18774,   18774,   18774, 0.8, 
0.8, 0.9, 1.0, 5.5,78.2,   60.0,  0.01882,  0, 40,
2665,2665,   6,   23666
 24 threadCount, total,   1562460,   26036,   26036,   26036, 0.9, 
0.8, 1.0, 1.1, 9.1,81.3,   60.0,  0.00837,  0, 55,
3543,3543,   9,   32619
 36 threadCount, total,   2098097,   34962,   34962,   34962, 1.0, 
0.9, 1.1, 1.3,60.9,83.0,   60.0,  0.00793,  0, 73,
4665,4665,   7,   43144
 54 threadCount, total,   2741814,   45686,   45686,   45686, 1.1, 
1.0, 1.4, 1.7,62.2,   131.7,   60.0,  0.01321,  0, 93,
5748,5748,   7,   55097
 81 threadCount, total,   3851131,   64166,   64166,   64166, 1.2, 
1.0, 1.6, 2.6,62.3,   151.7,   60.0,  0.01152,  0,159,
8190,8521,  14,  106805
121 threadCount, total,   4798169,   79947,   79947,   79947, 1.5, 
1.1, 2.0, 3.0,63.5,   117.8,   60.0,  0.05689,  0,165,
9323,9439,   5,   97536
181 threadCount, total,   5647043,   94088,   94088,   94088, 1.9, 
1.4, 2.6, 4.9,68.5,   169.2,   60.0,  0.01639,  0,195,   
10106,   11011,  11,  126422
271 threadCount, total,   6450510,  107461,  107461,  107461, 2.5, 
1.8, 3.7,12.0,75.4,   155.8,   60.0,  0.01542,  0,228,   
10304,   12789,   9,  143857
406 threadCount, total,   6700764,  111635,  111635,  111635, 3.6, 
2.5, 5.3,55.8,75.6,   196.5,   60.0,  0.01800,  0,243,
9995,   13170,   7,  144166
609 threadCount, total,   7119535,  118477,  118477,  118477, 5.1, 
3.5, 7.9,62.8,85.1,   170.0,   60.1,  0.01775,  0,250,   
10149,   13781,   7,  148118
913 threadCount, total,   7093347,  117981,  117981,  117981, 7.7, 
4.9,15.7,71.3,   101.1,   173.4,   60.1,  0.02780,  0,246,   
10327,   13859,   8,  155896
{code}

8457
{code}
 id, type   total ops,op/s,pk/s,   row/s,mean, 
med, .95, .99,.999, max,   time,   stderr, errors,  gc: #,  max 
ms,  sum ms,  sdv ms,  mb
  4 threadCount, total,161668,2694,2694,2694, 1.4, 
1.4, 1.6, 1.7, 3.2,68.2,   60.0,  0.01264,  0,  6, 
363, 363,   4,3631
  8 threadCount, total,498139,8301,8301,8301, 0.9, 
0.9, 1.1, 1.3, 1.8,73.5,   60.0,  0.00446,  0, 19,
1164,1164,   6,   11266
 16 threadCount, total,765437,   12756,   12756,   12756, 1.2, 
1.2, 1.4, 1.5, 5.7,74.8,   60.0,  0.01251,  0, 29,
1819,1819,   5,   17238
 24 threadCount, total,   1122768,   18710,   18710,   18710, 1.2, 
1.2, 1.4, 1.5, 8.5,   127.7,   60.0,  0.00871,  0, 42,
2538,2538,   5,   25054
 36 threadCount, total,   1649658,   27489,   27489,   27489, 1.3, 
1.2, 1.4, 1.6,60.1,77.7,   60.0,  0.00627,  0, 57,
3652,3652,   7,   33743
 54 threadCount, total,   2258999,   37641,   37641,   37641, 1.4, 
1.3, 1.6, 1.8,62.5,81.7,   60.0,  0.00771,  0, 79,
4908,4908,   6,   46789
 81 threadCount, total,   3255005,   54220,   54220,   54220, 1.5, 
1.2, 1.7, 2.2,63.8,   133.4,   60.0,  0.02030,  0,117,
6953,7008,   9,   72208
121 threadCount, total,   4643184,   77293,   77293,   77293, 1.5,  

[jira] [Updated] (CASSANDRA-12917) Increase error margin in SplitterTest

[Branimir Lambov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Branimir Lambov updated CASSANDRA-12917:

Status: Ready to Commit  (was: Patch Available)

> Increase error margin in SplitterTest
> -
>
> Key: CASSANDRA-12917
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12917
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>  Labels: lhf
> Fix For: 3.11.x
>
>
> SplitterTest is a randomized test - it generates random tokens and splits the 
> ranges in equal parts. Since it is random we sometimes get very big vnodes 
> right where we want a split and that makes the split unbalanced
> Bumping the error margin a bit will avoid these false positives.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-12917) Increase error margin in SplitterTest

[Branimir Lambov (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863702#comment-15863702
 ] 

Branimir Lambov commented on CASSANDRA-12917:
-

Code looks good.

However, raising the margin decreases the flakiness, but I'm not sure it 
eliminates it. In such cases I personally prefer to have a large number of 
points in the individual test (which this already has), a fixed seed, and a 
lower error margin.

> Increase error margin in SplitterTest
> -
>
> Key: CASSANDRA-12917
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12917
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>  Labels: lhf
> Fix For: 3.11.x
>
>
> SplitterTest is a randomized test - it generates random tokens and splits the 
> ranges in equal parts. Since it is random we sometimes get very big vnodes 
> right where we want a split and that makes the split unbalanced
> Bumping the error margin a bit will avoid these false positives.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-12831) OutOfMemoryError with Cassandra 3.0.9

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863654#comment-15863654
 ] 

Stefan Podkowinski commented on CASSANDRA-12831:


Looks like many netty related allocations in the heap dump. 3.9 is using an 
updated netty version, maybe that's why you're not seeing this kind of GC 
issues there. We've updated netty in 3.0.11 as well, but it's not released yet. 
But you can already give it a try and see if this is still an issue creating 
another ccm cluster from git, e.g. {{ccm create 3.0-latest -v git:cassandra-3.0 
-n 1}}.

> OutOfMemoryError with Cassandra 3.0.9
> -
>
> Key: CASSANDRA-12831
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12831
> Project: Cassandra
>  Issue Type: Bug
> Environment: Fedora 24  / Java  1.8.0_91 / Cassandra 3.0.9
> Mac OS X 10.11.6 / Java 1.8.0_102 / Cassandra 3.0.9
>Reporter: John Sanda
> Attachments: system.log
>
>
> I have running some tests on a monitoring system I work on and Cassandra is 
> consistently crashing with OOMEs, and the JVM exists. This is happening in a 
> dev environment with a single node created with ccm. 
> The monitoring server is ingesting 4,000 data points every 10 seconds. Every 
> two hours a job runs which fetches all raw data from the past two hours. The 
> raw data is compressed, written to another table, and then deleted. After 3 
> or 4 runs of the job Cassandra crashes. Initially I thought that the problem 
> was in my application code, but I am no longer of that opinion because I set 
> up the same test environment with Cassandra 3.9, and it has been running for 
> almost 48 hours without error. And I actually increased the load on the 3.9 
> environment.
> The schema for the raw data which is queried looks like:
> {noformat}
> CREATE TABLE hawkular_metrics.data (
> tenant_id text,
> type tinyint,
> metric text,
> dpart bigint,
> time timeuuid,
> n_value double,
> tags map,
> PRIMARY KEY ((tenant_id, type, metric, dpart), time)
> ) WITH CLUSTERING ORDER BY (time DESC)
> {noformat}
> And the schema for the table that is written to:
> {noformat}
> CREATE TABLE hawkular_metrics.data_compressed (
> tenant_id text,
> type tinyint,
> metric text,
> dpart bigint,
> time timestamp,
> c_value blob,
> tags blob,
> ts_value blob,
> PRIMARY KEY ((tenant_id, type, metric, dpart), time)
> ) WITH CLUSTERING ORDER BY (time DESC)
> {noformat}
> I am using version 3.0.1 of the DataStax Java driver. Last night I changed 
> the driver's page size from the default to 1000, and so far I have not yet 
> seen any errors.
> I have attached the log file. I was going to attach one of the heap dumps, 
> but it looks like they are too big.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (CASSANDRA-13215) Cassandra nodes startup time 20x more after upgarding to 3.x

[Viktor Kuzmin (JIRA)]

Viktor Kuzmin created CASSANDRA-13215:
-

 Summary: Cassandra nodes startup time 20x more after upgarding to 
3.x
 Key: CASSANDRA-13215
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13215
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
 Environment: Cluster setup: two datacenters (dc-main, dc-backup).
dc-main - 9 servers, no vnodes
dc-backup - 6 servers, vnodes
Reporter: Viktor Kuzmin
 Attachments: simple-cache.patch

CompactionStrategyManage.getCompactionStrategyIndex is called on each sstable 
at startup. And this function calls StorageService.getDiskBoundaries. And 
getDiskBoundaries calls AbstractReplicationStrategy.getAddressRanges.
It appears that last function can be really slow. In our environment we have 
1545 tokens and with NetworkTopologyStrategy it can make 1545*1545 computations 
in worst case (maybe I'm wrong, but it really takes lot's of cpu).

Also this function can affect runtime later, cause it is called not only during 
startup.

I've tried to implement simple cache for getDiskBoundaries results and now 
startup time is about one minute instead of 25m, but I'm not sure if it's a 
good solution.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13211) cassandra shell script uses bad approach to write "Unable to find java" error to stderr, causing real issue to be masked by a permission error if user has changed

[Nate McCall (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863162#comment-15863162
 ] 

Nate McCall commented on CASSANDRA-13211:
-

I have a vague recollection of dealing with pseudoterminal ownership issues 
when doing {{su}}, so I could see this. [~brandon.williams] probably has a lot 
better bash foo than me. 

> cassandra shell script uses bad approach to write "Unable to find java" error 
> to stderr, causing real issue to be masked by a permission error if user has 
> changed user since logging in
> 
>
> Key: CASSANDRA-13211
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13211
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Max Bowsher
>
> The cassandra startup shell script contains this line:
> echo Unable to find java executable. Check JAVA_HOME and PATH environment 
> variables. > /dev/stderr
> The problem here is the construct "> /dev/stderr". If the user invoking 
> Cassandra has changed user (for example, by SSHing in as a personal user, and 
> then sudo-ing to an application user responsible for executing the Cassandra 
> daemon), then the attempt to open /dev/stderr will fail, because it will 
> point to a PTY node under /dev/pts/ owned by the original user.
> Ultimately this leads to the real problem being masked by the confusing error 
> message "bash: /dev/stderr: Permission denied".
> The correct technique is to replace "> /dev/stderr" with ">&2" which will 
> write to the already open stderr file descriptor, instead of resolving the 
> chain of symlinks starting at /dev/stderr, and attempting to reopen the 
> target by name.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13213) compactionstats not available, node eventually OOMs due to pending mutations

[Marcus Eriksson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863286#comment-15863286
 ] 

Marcus Eriksson commented on CASSANDRA-13213:
-

so, it hangs just by letting it run with autocompaction enabled? No 
repairs/relocatesstables/etc running at the same time?

> compactionstats not available, node eventually OOMs due to pending mutations
> 
>
> Key: CASSANDRA-13213
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13213
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction, Local Write-Read Paths
>Reporter: Zachary Girouard
>
> I'm seeing semi-frequent instances of nodetool compactionstats hanging 
> forever. While this is occurring none of the compaction metrics are available 
> via jmx/jconsole.
> Sometimes the node will eventually recover, but I'm seeing a pattern where a 
> node will exhibit this behavior, and then eventually pending mutations start 
> piling up and the node dies due to OOM. Sometimes pending gossip operations 
> starting piling up too, but I think this is due to the impending OOM causing 
> everything to bog down.
> As an experiment I turned auto compaction off on all the nodes and I haven't 
> seen this issue occur since I did that. Additionally, I'm running 
> relocatesstables on some nodes with unthrottled compaction and so far none of 
> them have had any issues handling it.
> I managed to get some stack traces from a dying node:
> All MutationStage threads look similar to this:
> {noformat}
> Name: MutationStage-10
>   
> State: WAITING
> Total blocked: 9  Total waited: 1,959,850
> Stack trace: 
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(Unknown Source)
> org.apache.cassandra.utils.concurrent.WaitQueue$AbstractSignal.awaitUninterruptibly(WaitQueue.java:279)
> org.apache.cassandra.utils.memory.MemtableAllocator$SubAllocator.allocate(MemtableAllocator.java:162)
> org.apache.cassandra.utils.memory.SlabAllocator.allocate(SlabAllocator.java:89)
> org.apache.cassandra.utils.memory.ContextAllocator.allocate(ContextAllocator.java:57)
> org.apache.cassandra.utils.memory.ContextAllocator.clone(ContextAllocator.java:47)
> org.apache.cassandra.db.rows.BufferCell.copy(BufferCell.java:122)
> org.apache.cassandra.utils.memory.AbstractAllocator$CloningBTreeRowBuilder.addCell(AbstractAllocator.java:72)
> org.apache.cassandra.db.rows.Rows.copy(Rows.java:51)
> org.apache.cassandra.db.partitions.AtomicBTreePartition$RowUpdater.apply(AtomicBTreePartition.java:332)
> org.apache.cassandra.db.partitions.AtomicBTreePartition$RowUpdater.apply(AtomicBTreePartition.java:295)
> org.apache.cassandra.utils.btree.NodeBuilder.addNewKey(NodeBuilder.java:323)
> org.apache.cassandra.utils.btree.NodeBuilder.update(NodeBuilder.java:184)
> org.apache.cassandra.utils.btree.TreeBuilder.update(TreeBuilder.java:95)
> org.apache.cassandra.utils.btree.BTree.update(BTree.java:182)
> org.apache.cassandra.db.partitions.AtomicBTreePartition.addAllWithSizeDelta(AtomicBTreePartition.java:156)
> org.apache.cassandra.db.Memtable.put(Memtable.java:284)org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1316)
> org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:618)
> org.apache.cassandra.db.Keyspace.applyFuture(Keyspace.java:425)
> org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:222)
> org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:68)
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
> java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
> java.lang.Thread.run(Unknown Source)
> {noformat}
> The Compaction threads
> {noformat}
> Name: CompactionExecutor:4
> State: RUNNABLE
> Total blocked: 32,781,277  Total waited: 549
> Stack trace: 
> org.apache.cassandra.io.sstable.format.SSTableReader.getTotalBytes(SSTableReader.java:661)
> org.apache.cassandra.db.compaction.LeveledManifest.getCandidatesFor(LeveledManifest.java:669)
> org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:385)
>- locked org.apache.cassandra.db.compaction.LeveledManifest@55c79600
>
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:119)
>
> 

[jira] [Commented] (CASSANDRA-12653) In-flight shadow round requests

[Joel Knighton (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863233#comment-15863233
 ] 

Joel Knighton commented on CASSANDRA-12653:
---

[~jjirsa] - yes. That said, I don't anticipate getting to it within the next 
couple days, so feel free to give it the final review if it is higher priority 
for you than that. 

I've given several passes of review to the core concepts and they seem good. I 
think a final code style/details pass is all that remains.

> In-flight shadow round requests
> ---
>
> Key: CASSANDRA-12653
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12653
> Project: Cassandra
>  Issue Type: Bug
>  Components: Distributed Metadata
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Minor
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x
>
> Attachments: 12653-2.2.patch, 12653-3.0.patch, 12653-trunk.patch
>
>
> Bootstrapping or replacing a node in the cluster requires to gather and check 
> some host IDs or tokens by doing a gossip "shadow round" once before joining 
> the cluster. This is done by sending a gossip SYN to all seeds until we 
> receive a response with the cluster state, from where we can move on in the 
> bootstrap process. Receiving a response will call the shadow round done and 
> calls {{Gossiper.resetEndpointStateMap}} for cleaning up the received state 
> again.
> The issue here is that at this point there might be other in-flight requests 
> and it's very likely that shadow round responses from other seeds will be 
> received afterwards, while the current state of the bootstrap process doesn't 
> expect this to happen (e.g. gossiper may or may not be enabled). 
> One side effect will be that MigrationTasks are spawned for each shadow round 
> reply except the first. Tasks might or might not execute based on whether at 
> execution time {{Gossiper.resetEndpointStateMap}} had been called, which 
> effects the outcome of {{FailureDetector.instance.isAlive(endpoint))}} at 
> start of the task. You'll see error log messages such as follows when this 
> happend:
> {noformat}
> INFO  [SharedPool-Worker-1] 2016-09-08 08:36:39,255 Gossiper.java:993 - 
> InetAddress /xx.xx.xx.xx is now UP
> ERROR [MigrationStage:1]2016-09-08 08:36:39,255 FailureDetector.java:223 
> - unknown endpoint /xx.xx.xx.xx
> {noformat}
> Although is isn't pretty, I currently don't see any serious harm from this, 
> but it would be good to get a second opinion (feel free to close as "wont 
> fix").
> /cc [~Stefania] [~thobbs]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13008) Add vm.max_map_count StartupCheck

[Stefania (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefania updated CASSANDRA-13008:
-
   Resolution: Fixed
Fix Version/s: (was: 3.0.x)
   3.11.0
   3.0.11
   Status: Resolved  (was: Patch Available)

Committed to 3.0 as 3a570d744e938077784d8a24b1334c7b567543ed and merged upwards.

> Add vm.max_map_count StartupCheck
> -
>
> Key: CASSANDRA-13008
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13008
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Configuration
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>Priority: Minor
> Fix For: 3.0.11, 3.11.0
>
> Attachments: 13008-3.0.txt
>
>
> It's recommended to set {{vm.max_map_count}} to 1048575 (CASSANDRA-3563)
> When the max_map_count is low, it throws OOM exception, which is hard to link 
> to the real issue of vm.max_map_count.
> The problem happened when we tried to remove one node, all the other nodes in 
> cluster crashed. As each node was trying to load more local SSTable files for 
> streaming.
> I would suggest to add a StartupCheck for max_map_count, at least it could 
> give a warning message to help the debug.
> {code}
> ERROR [STREAM-IN-] JVMStabilityInspector.java:140 - JVM state determined to 
> be unstable.  Exiting forcefully due to:
> java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_112]
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) 
> ~[na:1.8.0_112]
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:173)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:70) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:58) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.CompressedSegmentedFile.(CompressedSegmentedFile.java:47)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:193)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:276)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.access$600(BigTableWriter.java:50)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:313)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.sstable.format.SSTableWriter.finish(SSTableWriter.java:213)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.finish(SimpleSSTableMultiWriter.java:56)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.streaming.StreamReceiveTask.received(StreamReceiveTask.java:109)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:599) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:482)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:296)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-11479) BatchlogManager unit tests failing on truncate race condition

[Joel Knighton (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863235#comment-15863235
 ] 

Joel Knighton commented on CASSANDRA-11479:
---

More on my plate than Yuki's, I believe. The patch made sense to me, but I was 
waiting for a chance to dig deeper into the related compaction code before 
giving it a final OK. In the process, it slipped through the cracks quite 
badly. I'd be happy to do that, but it likely wouldn't happen in the next few 
days if you're interested in taking it on instead.

> BatchlogManager unit tests failing on truncate race condition
> -
>
> Key: CASSANDRA-11479
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11479
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Joel Knighton
>Assignee: Yuki Morishita
> Fix For: 2.2.x, 3.0.x, 3.11.x
>
> Attachments: 
> TEST-org.apache.cassandra.batchlog.BatchlogManagerTest.log
>
>
> Example on CI 
> [here|http://cassci.datastax.com/job/trunk_testall/818/testReport/junit/org.apache.cassandra.batchlog/BatchlogManagerTest/testLegacyReplay_compression/].
>  This seems to have only started happening relatively recently (within the 
> last month or two).
> As far as I can tell, this is only showing up on BatchlogManagerTests purely 
> because it is an aggressive user of truncate. The assertion is hit in the 
> setUp method, so it can happen before any of the test methods. The assertion 
> occurs because a compaction is happening when truncate wants to discard 
> SSTables; trace level logs suggest that this compaction is submitted after 
> the pause on the CompactionStrategyManager.
> This should be reproducible by running BatchlogManagerTest in a loop - it 
> takes up to half an hour in my experience. A trace-level log from such a run 
> is attached - grep for my added log message "SSTABLES COMPACTING WHEN 
> DISCARDING" to find when the assert is hit.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/6] cassandra git commit: Add vm.max_map_count StartupCheck

[stefania]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 51e3660c7 -> 3a570d744
  refs/heads/cassandra-3.11 ed45d5eb7 -> 01b91cce2
  refs/heads/trunk 1eaef3f9e -> ffc82b1ee


Add vm.max_map_count StartupCheck

patch by Jay Zhuang; reviewed by Stefania Alborghetti for CASSANDRA-13008


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3a570d74
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3a570d74
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3a570d74

Branch: refs/heads/cassandra-3.0
Commit: 3a570d744e938077784d8a24b1334c7b567543ed
Parents: 51e3660
Author: Jay Zhuang 
Authored: Thu Feb 9 17:20:19 2017 +
Committer: Stefania Alborghetti 
Committed: Mon Feb 13 10:57:30 2017 +

--
 CHANGES.txt |  1 +
 .../apache/cassandra/service/StartupChecks.java | 50 
 .../cassandra/service/StartupChecksTest.java|  7 +++
 3 files changed, 58 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/3a570d74/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index b720a0e..ac3d1ed 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.11
+ * Add vm.max_map_count StartupCheck (CASSANDRA-13008)
  * Hint related logging should include the IP address of the destination in 
addition to 
host ID (CASSANDRA-13205)
  * Reloading logback.xml does not work (CASSANDRA-13173)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/3a570d74/src/java/org/apache/cassandra/service/StartupChecks.java
--
diff --git a/src/java/org/apache/cassandra/service/StartupChecks.java 
b/src/java/org/apache/cassandra/service/StartupChecks.java
index ad6a104..756799e 100644
--- a/src/java/org/apache/cassandra/service/StartupChecks.java
+++ b/src/java/org/apache/cassandra/service/StartupChecks.java
@@ -17,6 +17,7 @@
  */
 package org.apache.cassandra.service;
 
+import java.io.BufferedReader;
 import java.io.File;
 import java.io.IOException;
 import java.nio.file.*;
@@ -30,6 +31,7 @@ import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import org.apache.cassandra.config.CFMetaData;
+import org.apache.cassandra.config.Config;
 import org.apache.cassandra.config.DatabaseDescriptor;
 import org.apache.cassandra.config.Schema;
 import org.apache.cassandra.db.*;
@@ -74,6 +76,7 @@ public class StartupChecks
   
inspectJvmOptions,
   
checkJnaInitialization,
   
initSigarLibrary,
+  
checkMaxMapCount,
   
checkDataDirs,
   
checkSSTablesFormat,
   
checkSystemKeyspaceState,
@@ -199,6 +202,53 @@ public class StartupChecks
 }
 };
 
+public static final StartupCheck checkMaxMapCount = new StartupCheck()
+{
+private final long EXPECTED_MAX_MAP_COUNT = 1048575;
+private final String MAX_MAP_COUNT_PATH = "/proc/sys/vm/max_map_count";
+
+private long getMaxMapCount()
+{
+final Path path = Paths.get(MAX_MAP_COUNT_PATH);
+try (final BufferedReader bufferedReader = 
Files.newBufferedReader(path))
+{
+final String data = bufferedReader.readLine();
+if (data != null)
+{
+try
+{
+return Long.parseLong(data);
+}
+catch (final NumberFormatException e)
+{
+logger.warn("Unable to parse {}.", path, e);
+}
+}
+}
+catch (final IOException e)
+{
+logger.warn("IO exception while reading file {}.", path, e);
+}
+return -1;
+}
+
+public void execute()
+{
+if (!FBUtilities.hasProcFS())
+return;
+
+if (DatabaseDescriptor.getDiskAccessMode() == 
Config.DiskAccessMode.standard &&
+DatabaseDescriptor.getIndexAccessMode() == 
Config.DiskAccessMode.standard)
+return; // no need to check if disk access mode is only 
standard and not mmap
+
+long maxMapCount = getMaxMapCount();
+if 

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[stefania]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/01b91cce
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/01b91cce
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/01b91cce

Branch: refs/heads/cassandra-3.11
Commit: 01b91cce2f4606eb4b1de327467833132c80f4e0
Parents: ed45d5e 3a570d7
Author: Stefania Alborghetti 
Authored: Mon Feb 13 10:59:02 2017 +
Committer: Stefania Alborghetti 
Committed: Mon Feb 13 10:59:42 2017 +

--
 CHANGES.txt |  1 +
 .../apache/cassandra/service/StartupChecks.java | 50 
 .../org/apache/cassandra/utils/FBUtilities.java |  1 +
 .../cassandra/service/StartupChecksTest.java|  7 +++
 4 files changed, 59 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/01b91cce/CHANGES.txt
--
diff --cc CHANGES.txt
index 5be6cef,ac3d1ed..67c45e8
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,13 -1,6 +1,14 @@@
 -3.0.11
 +3.11.0
 + * Obfuscate password in stress-graphs (CASSANDRA-12233)
 + * Move to FastThreadLocalThread and FastThreadLocal (CASSANDRA-13034)
 + * nodetool stopdaemon errors out (CASSANDRA-13030)
 + * Tables in system_distributed should not use gcgs of 0 (CASSANDRA-12954)
 + * Fix primary index calculation for SASI (CASSANDRA-12910)
 + * More fixes to the TokenAllocator (CASSANDRA-12990)
 + * NoReplicationTokenAllocator should work with zero replication factor 
(CASSANDRA-12983)
 +Merged from 3.0:
+  * Add vm.max_map_count StartupCheck (CASSANDRA-13008)
 - * Hint related logging should include the IP address of the destination in 
addition to 
 + * Hint related logging should include the IP address of the destination in 
addition to
 host ID (CASSANDRA-13205)
   * Reloading logback.xml does not work (CASSANDRA-13173)
   * Lightweight transactions temporarily fail after upgrade from 2.1 to 3.0 
(CASSANDRA-13109)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/01b91cce/src/java/org/apache/cassandra/service/StartupChecks.java
--
diff --cc src/java/org/apache/cassandra/service/StartupChecks.java
index 83971dd,756799e..2774931
--- a/src/java/org/apache/cassandra/service/StartupChecks.java
+++ b/src/java/org/apache/cassandra/service/StartupChecks.java
@@@ -30,12 -31,10 +31,13 @@@ import org.slf4j.Logger
  import org.slf4j.LoggerFactory;
  
  import org.apache.cassandra.config.CFMetaData;
+ import org.apache.cassandra.config.Config;
  import org.apache.cassandra.config.DatabaseDescriptor;
  import org.apache.cassandra.config.Schema;
 -import org.apache.cassandra.db.*;
 +import org.apache.cassandra.config.SchemaConstants;
 +import org.apache.cassandra.db.ColumnFamilyStore;
 +import org.apache.cassandra.db.Directories;
 +import org.apache.cassandra.db.SystemKeyspace;
  import org.apache.cassandra.exceptions.ConfigurationException;
  import org.apache.cassandra.exceptions.StartupException;
  import org.apache.cassandra.io.sstable.Descriptor;
@@@ -218,6 -202,53 +221,53 @@@ public class StartupCheck
  }
  };
  
+ public static final StartupCheck checkMaxMapCount = new StartupCheck()
+ {
+ private final long EXPECTED_MAX_MAP_COUNT = 1048575;
+ private final String MAX_MAP_COUNT_PATH = 
"/proc/sys/vm/max_map_count";
+ 
+ private long getMaxMapCount()
+ {
+ final Path path = Paths.get(MAX_MAP_COUNT_PATH);
+ try (final BufferedReader bufferedReader = 
Files.newBufferedReader(path))
+ {
+ final String data = bufferedReader.readLine();
+ if (data != null)
+ {
+ try
+ {
+ return Long.parseLong(data);
+ }
+ catch (final NumberFormatException e)
+ {
+ logger.warn("Unable to parse {}.", path, e);
+ }
+ }
+ }
+ catch (final IOException e)
+ {
+ logger.warn("IO exception while reading file {}.", path, e);
+ }
+ return -1;
+ }
+ 
+ public void execute()
+ {
 -if (!FBUtilities.hasProcFS())
++if (!FBUtilities.isLinux)
+ return;
+ 
+ if (DatabaseDescriptor.getDiskAccessMode() == 
Config.DiskAccessMode.standard &&
+ DatabaseDescriptor.getIndexAccessMode() == 
Config.DiskAccessMode.standard)
+ return; // no need to check if disk access mode is 

[3/6] cassandra git commit: Add vm.max_map_count StartupCheck

[stefania]

Add vm.max_map_count StartupCheck

patch by Jay Zhuang; reviewed by Stefania Alborghetti for CASSANDRA-13008


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3a570d74
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3a570d74
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3a570d74

Branch: refs/heads/trunk
Commit: 3a570d744e938077784d8a24b1334c7b567543ed
Parents: 51e3660
Author: Jay Zhuang 
Authored: Thu Feb 9 17:20:19 2017 +
Committer: Stefania Alborghetti 
Committed: Mon Feb 13 10:57:30 2017 +

--
 CHANGES.txt |  1 +
 .../apache/cassandra/service/StartupChecks.java | 50 
 .../cassandra/service/StartupChecksTest.java|  7 +++
 3 files changed, 58 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/3a570d74/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index b720a0e..ac3d1ed 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.11
+ * Add vm.max_map_count StartupCheck (CASSANDRA-13008)
  * Hint related logging should include the IP address of the destination in 
addition to 
host ID (CASSANDRA-13205)
  * Reloading logback.xml does not work (CASSANDRA-13173)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/3a570d74/src/java/org/apache/cassandra/service/StartupChecks.java
--
diff --git a/src/java/org/apache/cassandra/service/StartupChecks.java 
b/src/java/org/apache/cassandra/service/StartupChecks.java
index ad6a104..756799e 100644
--- a/src/java/org/apache/cassandra/service/StartupChecks.java
+++ b/src/java/org/apache/cassandra/service/StartupChecks.java
@@ -17,6 +17,7 @@
  */
 package org.apache.cassandra.service;
 
+import java.io.BufferedReader;
 import java.io.File;
 import java.io.IOException;
 import java.nio.file.*;
@@ -30,6 +31,7 @@ import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import org.apache.cassandra.config.CFMetaData;
+import org.apache.cassandra.config.Config;
 import org.apache.cassandra.config.DatabaseDescriptor;
 import org.apache.cassandra.config.Schema;
 import org.apache.cassandra.db.*;
@@ -74,6 +76,7 @@ public class StartupChecks
   
inspectJvmOptions,
   
checkJnaInitialization,
   
initSigarLibrary,
+  
checkMaxMapCount,
   
checkDataDirs,
   
checkSSTablesFormat,
   
checkSystemKeyspaceState,
@@ -199,6 +202,53 @@ public class StartupChecks
 }
 };
 
+public static final StartupCheck checkMaxMapCount = new StartupCheck()
+{
+private final long EXPECTED_MAX_MAP_COUNT = 1048575;
+private final String MAX_MAP_COUNT_PATH = "/proc/sys/vm/max_map_count";
+
+private long getMaxMapCount()
+{
+final Path path = Paths.get(MAX_MAP_COUNT_PATH);
+try (final BufferedReader bufferedReader = 
Files.newBufferedReader(path))
+{
+final String data = bufferedReader.readLine();
+if (data != null)
+{
+try
+{
+return Long.parseLong(data);
+}
+catch (final NumberFormatException e)
+{
+logger.warn("Unable to parse {}.", path, e);
+}
+}
+}
+catch (final IOException e)
+{
+logger.warn("IO exception while reading file {}.", path, e);
+}
+return -1;
+}
+
+public void execute()
+{
+if (!FBUtilities.hasProcFS())
+return;
+
+if (DatabaseDescriptor.getDiskAccessMode() == 
Config.DiskAccessMode.standard &&
+DatabaseDescriptor.getIndexAccessMode() == 
Config.DiskAccessMode.standard)
+return; // no need to check if disk access mode is only 
standard and not mmap
+
+long maxMapCount = getMaxMapCount();
+if (maxMapCount < EXPECTED_MAX_MAP_COUNT)
+logger.warn("Maximum number of memory map areas per process 
(vm.max_map_count) {} " +
+"is too low, 

[2/6] cassandra git commit: Add vm.max_map_count StartupCheck

[stefania]

Add vm.max_map_count StartupCheck

patch by Jay Zhuang; reviewed by Stefania Alborghetti for CASSANDRA-13008


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3a570d74
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3a570d74
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3a570d74

Branch: refs/heads/cassandra-3.11
Commit: 3a570d744e938077784d8a24b1334c7b567543ed
Parents: 51e3660
Author: Jay Zhuang 
Authored: Thu Feb 9 17:20:19 2017 +
Committer: Stefania Alborghetti 
Committed: Mon Feb 13 10:57:30 2017 +

--
 CHANGES.txt |  1 +
 .../apache/cassandra/service/StartupChecks.java | 50 
 .../cassandra/service/StartupChecksTest.java|  7 +++
 3 files changed, 58 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/3a570d74/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index b720a0e..ac3d1ed 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.11
+ * Add vm.max_map_count StartupCheck (CASSANDRA-13008)
  * Hint related logging should include the IP address of the destination in 
addition to 
host ID (CASSANDRA-13205)
  * Reloading logback.xml does not work (CASSANDRA-13173)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/3a570d74/src/java/org/apache/cassandra/service/StartupChecks.java
--
diff --git a/src/java/org/apache/cassandra/service/StartupChecks.java 
b/src/java/org/apache/cassandra/service/StartupChecks.java
index ad6a104..756799e 100644
--- a/src/java/org/apache/cassandra/service/StartupChecks.java
+++ b/src/java/org/apache/cassandra/service/StartupChecks.java
@@ -17,6 +17,7 @@
  */
 package org.apache.cassandra.service;
 
+import java.io.BufferedReader;
 import java.io.File;
 import java.io.IOException;
 import java.nio.file.*;
@@ -30,6 +31,7 @@ import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import org.apache.cassandra.config.CFMetaData;
+import org.apache.cassandra.config.Config;
 import org.apache.cassandra.config.DatabaseDescriptor;
 import org.apache.cassandra.config.Schema;
 import org.apache.cassandra.db.*;
@@ -74,6 +76,7 @@ public class StartupChecks
   
inspectJvmOptions,
   
checkJnaInitialization,
   
initSigarLibrary,
+  
checkMaxMapCount,
   
checkDataDirs,
   
checkSSTablesFormat,
   
checkSystemKeyspaceState,
@@ -199,6 +202,53 @@ public class StartupChecks
 }
 };
 
+public static final StartupCheck checkMaxMapCount = new StartupCheck()
+{
+private final long EXPECTED_MAX_MAP_COUNT = 1048575;
+private final String MAX_MAP_COUNT_PATH = "/proc/sys/vm/max_map_count";
+
+private long getMaxMapCount()
+{
+final Path path = Paths.get(MAX_MAP_COUNT_PATH);
+try (final BufferedReader bufferedReader = 
Files.newBufferedReader(path))
+{
+final String data = bufferedReader.readLine();
+if (data != null)
+{
+try
+{
+return Long.parseLong(data);
+}
+catch (final NumberFormatException e)
+{
+logger.warn("Unable to parse {}.", path, e);
+}
+}
+}
+catch (final IOException e)
+{
+logger.warn("IO exception while reading file {}.", path, e);
+}
+return -1;
+}
+
+public void execute()
+{
+if (!FBUtilities.hasProcFS())
+return;
+
+if (DatabaseDescriptor.getDiskAccessMode() == 
Config.DiskAccessMode.standard &&
+DatabaseDescriptor.getIndexAccessMode() == 
Config.DiskAccessMode.standard)
+return; // no need to check if disk access mode is only 
standard and not mmap
+
+long maxMapCount = getMaxMapCount();
+if (maxMapCount < EXPECTED_MAX_MAP_COUNT)
+logger.warn("Maximum number of memory map areas per process 
(vm.max_map_count) {} " +
+"is too 

[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

[stefania]

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ffc82b1e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ffc82b1e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ffc82b1e

Branch: refs/heads/trunk
Commit: ffc82b1ee495fcf068730d0d3c12f54afdf517a2
Parents: 1eaef3f 01b91cc
Author: Stefania Alborghetti 
Authored: Mon Feb 13 11:02:17 2017 +
Committer: Stefania Alborghetti 
Committed: Mon Feb 13 11:02:17 2017 +

--
 CHANGES.txt |  1 +
 .../apache/cassandra/service/StartupChecks.java | 50 
 .../org/apache/cassandra/utils/FBUtilities.java |  1 +
 .../cassandra/service/StartupChecksTest.java|  7 +++
 4 files changed, 59 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/ffc82b1e/CHANGES.txt
--
diff --cc CHANGES.txt
index 7c38295,67c45e8..39f9513
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -37,7 -7,7 +37,8 @@@
   * More fixes to the TokenAllocator (CASSANDRA-12990)
   * NoReplicationTokenAllocator should work with zero replication factor 
(CASSANDRA-12983)
  Merged from 3.0:
+  * Add vm.max_map_count StartupCheck (CASSANDRA-13008)
 + * Obfuscate password in stress-graphs (CASSANDRA-12233)
   * Hint related logging should include the IP address of the destination in 
addition to
 host ID (CASSANDRA-13205)
   * Reloading logback.xml does not work (CASSANDRA-13173)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ffc82b1e/src/java/org/apache/cassandra/service/StartupChecks.java
--
diff --cc src/java/org/apache/cassandra/service/StartupChecks.java
index 0458820,2774931..b519350
--- a/src/java/org/apache/cassandra/service/StartupChecks.java
+++ b/src/java/org/apache/cassandra/service/StartupChecks.java
@@@ -29,10 -30,11 +30,11 @@@ import com.google.common.collect.Iterab
  import org.slf4j.Logger;
  import org.slf4j.LoggerFactory;
  
 -import org.apache.cassandra.config.CFMetaData;
 +import org.apache.cassandra.schema.TableMetadata;
+ import org.apache.cassandra.config.Config;
  import org.apache.cassandra.config.DatabaseDescriptor;
 -import org.apache.cassandra.config.Schema;
 -import org.apache.cassandra.config.SchemaConstants;
 +import org.apache.cassandra.schema.Schema;
 +import org.apache.cassandra.schema.SchemaConstants;
  import org.apache.cassandra.db.ColumnFamilyStore;
  import org.apache.cassandra.db.Directories;
  import org.apache.cassandra.db.SystemKeyspace;

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ffc82b1e/test/unit/org/apache/cassandra/service/StartupChecksTest.java
--

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[stefania]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/01b91cce
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/01b91cce
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/01b91cce

Branch: refs/heads/trunk
Commit: 01b91cce2f4606eb4b1de327467833132c80f4e0
Parents: ed45d5e 3a570d7
Author: Stefania Alborghetti 
Authored: Mon Feb 13 10:59:02 2017 +
Committer: Stefania Alborghetti 
Committed: Mon Feb 13 10:59:42 2017 +

--
 CHANGES.txt |  1 +
 .../apache/cassandra/service/StartupChecks.java | 50 
 .../org/apache/cassandra/utils/FBUtilities.java |  1 +
 .../cassandra/service/StartupChecksTest.java|  7 +++
 4 files changed, 59 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/01b91cce/CHANGES.txt
--
diff --cc CHANGES.txt
index 5be6cef,ac3d1ed..67c45e8
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,13 -1,6 +1,14 @@@
 -3.0.11
 +3.11.0
 + * Obfuscate password in stress-graphs (CASSANDRA-12233)
 + * Move to FastThreadLocalThread and FastThreadLocal (CASSANDRA-13034)
 + * nodetool stopdaemon errors out (CASSANDRA-13030)
 + * Tables in system_distributed should not use gcgs of 0 (CASSANDRA-12954)
 + * Fix primary index calculation for SASI (CASSANDRA-12910)
 + * More fixes to the TokenAllocator (CASSANDRA-12990)
 + * NoReplicationTokenAllocator should work with zero replication factor 
(CASSANDRA-12983)
 +Merged from 3.0:
+  * Add vm.max_map_count StartupCheck (CASSANDRA-13008)
 - * Hint related logging should include the IP address of the destination in 
addition to 
 + * Hint related logging should include the IP address of the destination in 
addition to
 host ID (CASSANDRA-13205)
   * Reloading logback.xml does not work (CASSANDRA-13173)
   * Lightweight transactions temporarily fail after upgrade from 2.1 to 3.0 
(CASSANDRA-13109)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/01b91cce/src/java/org/apache/cassandra/service/StartupChecks.java
--
diff --cc src/java/org/apache/cassandra/service/StartupChecks.java
index 83971dd,756799e..2774931
--- a/src/java/org/apache/cassandra/service/StartupChecks.java
+++ b/src/java/org/apache/cassandra/service/StartupChecks.java
@@@ -30,12 -31,10 +31,13 @@@ import org.slf4j.Logger
  import org.slf4j.LoggerFactory;
  
  import org.apache.cassandra.config.CFMetaData;
+ import org.apache.cassandra.config.Config;
  import org.apache.cassandra.config.DatabaseDescriptor;
  import org.apache.cassandra.config.Schema;
 -import org.apache.cassandra.db.*;
 +import org.apache.cassandra.config.SchemaConstants;
 +import org.apache.cassandra.db.ColumnFamilyStore;
 +import org.apache.cassandra.db.Directories;
 +import org.apache.cassandra.db.SystemKeyspace;
  import org.apache.cassandra.exceptions.ConfigurationException;
  import org.apache.cassandra.exceptions.StartupException;
  import org.apache.cassandra.io.sstable.Descriptor;
@@@ -218,6 -202,53 +221,53 @@@ public class StartupCheck
  }
  };
  
+ public static final StartupCheck checkMaxMapCount = new StartupCheck()
+ {
+ private final long EXPECTED_MAX_MAP_COUNT = 1048575;
+ private final String MAX_MAP_COUNT_PATH = 
"/proc/sys/vm/max_map_count";
+ 
+ private long getMaxMapCount()
+ {
+ final Path path = Paths.get(MAX_MAP_COUNT_PATH);
+ try (final BufferedReader bufferedReader = 
Files.newBufferedReader(path))
+ {
+ final String data = bufferedReader.readLine();
+ if (data != null)
+ {
+ try
+ {
+ return Long.parseLong(data);
+ }
+ catch (final NumberFormatException e)
+ {
+ logger.warn("Unable to parse {}.", path, e);
+ }
+ }
+ }
+ catch (final IOException e)
+ {
+ logger.warn("IO exception while reading file {}.", path, e);
+ }
+ return -1;
+ }
+ 
+ public void execute()
+ {
 -if (!FBUtilities.hasProcFS())
++if (!FBUtilities.isLinux)
+ return;
+ 
+ if (DatabaseDescriptor.getDiskAccessMode() == 
Config.DiskAccessMode.standard &&
+ DatabaseDescriptor.getIndexAccessMode() == 
Config.DiskAccessMode.standard)
+ return; // no need to check if disk access mode is only