[jira] [Comment Edited] (CASSANDRA-10496) Make DTCS/TWCS split partitions based on time during compaction

[mck (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153049#comment-16153049
 ] 

mck edited comment on CASSANDRA-10496 at 9/5/17 2:28 AM:
-

[~iksaif],
a few comments:
 - i suspect [~krummas] is keen to see a patch that splits partitions. 
 - changing locations isn't supported. see how i paired it with the writer in 
my experiment above.
 - Marcus' original idea was to create only two sstables per TWCS window. is 
that still possible?
 - shouldn't the bucket be based of the maxTimestamp? see `getBuckets(..)` and 
`newestBucket(..)`
 - is it correct that the idea is as "old" sstables are split out they would 
later then get re-compacted with their original bucket, and the domino effect 
that this could cause re-compacting older buckets could be avoided by 
increasing minThreshold to 3?



was (Author: michaelsembwever):
[~iksaif],
a few comments:
 - i suspect [~krummas] is keen to see a patch that splits partitions. 
 - changing locations isn't supported. see how i paired it with the writer in 
my experiment above.
 - i don't think you want to create the SSTableWriters multiple times.
 - Marcus' original idea was to create only two sstables per TWCS window. is 
that still possible?
 - shouldn't the bucket be based of the maxTimestamp? see `getBuckets(..)` and 
`newestBucket(..)`
 - is it correct that the idea is as "old" sstables are split out they would 
later then get re-compacted with their original bucket, and the domino effect 
that this could cause re-compacting older buckets could be avoided by 
increasing minThreshold to 3?


> Make DTCS/TWCS split partitions based on time during compaction
> ---
>
> Key: CASSANDRA-10496
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10496
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Marcus Eriksson
>  Labels: dtcs
> Fix For: 4.x
>
>
> To avoid getting old data in new time windows with DTCS (or related, like 
> [TWCS|CASSANDRA-9666]), we need to split out old data into its own sstable 
> during compaction.
> My initial idea is to just create two sstables, when we create the compaction 
> task we state the start and end times for the window, and any data older than 
> the window will be put in its own sstable.
> By creating a single sstable with old data, we will incrementally get the 
> windows correct - say we have an sstable with these timestamps:
> {{[100, 99, 98, 97, 75, 50, 10]}}
> and we are compacting in window {{[100, 80]}} - we would create two sstables:
> {{[100, 99, 98, 97]}}, {{[75, 50, 10]}}, and the first window is now 
> 'correct'. The next compaction would compact in window {{[80, 60]}} and 
> create sstables {{[75]}}, {{[50, 10]}} etc.
> We will probably also want to base the windows on the newest data in the 
> sstables so that we actually have older data than the window.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-10496) Make DTCS/TWCS split partitions based on time during compaction

[mck (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153049#comment-16153049
 ] 

mck commented on CASSANDRA-10496:
-

[~iksaif],
a few comments:
 - i suspect [~krummas] is keen to see a patch that splits partitions. Even 
though a solution that doesn't still has a lot to offer.
 - changing locations isn't supported. see how i paired it with the writer in 
my experiment above.
 - i don't think you want to create the SSTableWriters multiple times.
 - Marcus' original idea was to create only two sstables per TWCS window. is 
that still possible?
 - shouldn't the bucket be based of the maxTimestamp? see `getBuckets(..)` and 
`newestBucket(..)`
 - is it correct that the idea is as "old" sstables are split out they would 
later then get re-compacted with their original bucket, and the domino effect 
that this could cause re-compacting older buckets could be avoided by 
increasing minThreshold to 3?


> Make DTCS/TWCS split partitions based on time during compaction
> ---
>
> Key: CASSANDRA-10496
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10496
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Marcus Eriksson
>  Labels: dtcs
> Fix For: 4.x
>
>
> To avoid getting old data in new time windows with DTCS (or related, like 
> [TWCS|CASSANDRA-9666]), we need to split out old data into its own sstable 
> during compaction.
> My initial idea is to just create two sstables, when we create the compaction 
> task we state the start and end times for the window, and any data older than 
> the window will be put in its own sstable.
> By creating a single sstable with old data, we will incrementally get the 
> windows correct - say we have an sstable with these timestamps:
> {{[100, 99, 98, 97, 75, 50, 10]}}
> and we are compacting in window {{[100, 80]}} - we would create two sstables:
> {{[100, 99, 98, 97]}}, {{[75, 50, 10]}}, and the first window is now 
> 'correct'. The next compaction would compact in window {{[80, 60]}} and 
> create sstables {{[75]}}, {{[50, 10]}} etc.
> We will probably also want to base the windows on the newest data in the 
> sstables so that we actually have older data than the window.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

cassandra git commit: fix logging context

[dbrosius]

Repository: cassandra
Updated Branches:
  refs/heads/trunk d32c474f6 -> 460360093


fix logging context


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/46036009
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/46036009
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/46036009

Branch: refs/heads/trunk
Commit: 46036009376eaba548bbb4ac4ddf2531c720ed92
Parents: d32c474
Author: Dave Brosius 
Authored: Mon Sep 4 22:28:23 2017 -0400
Committer: Dave Brosius 
Committed: Mon Sep 4 22:28:23 2017 -0400

--
 .../org/apache/cassandra/net/async/InboundHandshakeHandler.java| 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/46036009/src/java/org/apache/cassandra/net/async/InboundHandshakeHandler.java
--
diff --git 
a/src/java/org/apache/cassandra/net/async/InboundHandshakeHandler.java 
b/src/java/org/apache/cassandra/net/async/InboundHandshakeHandler.java
index 7a8303c..aa4f4ff 100644
--- a/src/java/org/apache/cassandra/net/async/InboundHandshakeHandler.java
+++ b/src/java/org/apache/cassandra/net/async/InboundHandshakeHandler.java
@@ -37,7 +37,7 @@ import org.apache.cassandra.streaming.messages.StreamMessage;
  */
 class InboundHandshakeHandler extends ByteToMessageDecoder
 {
-private static final Logger logger = 
LoggerFactory.getLogger(NettyFactory.class);
+private static final Logger logger = 
LoggerFactory.getLogger(InboundHandshakeHandler.class);
 
 enum State { START, AWAITING_HANDSHAKE_BEGIN, 
AWAIT_MESSAGING_START_RESPONSE, HANDSHAKE_COMPLETE, HANDSHAKE_FAIL }
 


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-10496) Make DTCS/TWCS split partitions based on time during compaction

[mck (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153049#comment-16153049
 ] 

mck edited comment on CASSANDRA-10496 at 9/5/17 2:27 AM:
-

[~iksaif],
a few comments:
 - i suspect [~krummas] is keen to see a patch that splits partitions. 
 - changing locations isn't supported. see how i paired it with the writer in 
my experiment above.
 - i don't think you want to create the SSTableWriters multiple times.
 - Marcus' original idea was to create only two sstables per TWCS window. is 
that still possible?
 - shouldn't the bucket be based of the maxTimestamp? see `getBuckets(..)` and 
`newestBucket(..)`
 - is it correct that the idea is as "old" sstables are split out they would 
later then get re-compacted with their original bucket, and the domino effect 
that this could cause re-compacting older buckets could be avoided by 
increasing minThreshold to 3?



was (Author: michaelsembwever):
[~iksaif],
a few comments:
 - i suspect [~krummas] is keen to see a patch that splits partitions. Even 
though a solution that doesn't still has a lot to offer.
 - changing locations isn't supported. see how i paired it with the writer in 
my experiment above.
 - i don't think you want to create the SSTableWriters multiple times.
 - Marcus' original idea was to create only two sstables per TWCS window. is 
that still possible?
 - shouldn't the bucket be based of the maxTimestamp? see `getBuckets(..)` and 
`newestBucket(..)`
 - is it correct that the idea is as "old" sstables are split out they would 
later then get re-compacted with their original bucket, and the domino effect 
that this could cause re-compacting older buckets could be avoided by 
increasing minThreshold to 3?


> Make DTCS/TWCS split partitions based on time during compaction
> ---
>
> Key: CASSANDRA-10496
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10496
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Marcus Eriksson
>  Labels: dtcs
> Fix For: 4.x
>
>
> To avoid getting old data in new time windows with DTCS (or related, like 
> [TWCS|CASSANDRA-9666]), we need to split out old data into its own sstable 
> during compaction.
> My initial idea is to just create two sstables, when we create the compaction 
> task we state the start and end times for the window, and any data older than 
> the window will be put in its own sstable.
> By creating a single sstable with old data, we will incrementally get the 
> windows correct - say we have an sstable with these timestamps:
> {{[100, 99, 98, 97, 75, 50, 10]}}
> and we are compacting in window {{[100, 80]}} - we would create two sstables:
> {{[100, 99, 98, 97]}}, {{[75, 50, 10]}}, and the first window is now 
> 'correct'. The next compaction would compact in window {{[80, 60]}} and 
> create sstables {{[75]}}, {{[50, 10]}} etc.
> We will probably also want to base the windows on the newest data in the 
> sstables so that we actually have older data than the window.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

cassandra git commit: add missing logging marker, to match parameter

[dbrosius]

Repository: cassandra
Updated Branches:
  refs/heads/trunk 10d5b7b2f -> d32c474f6


add missing logging marker, to match parameter


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d32c474f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d32c474f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d32c474f

Branch: refs/heads/trunk
Commit: d32c474f6d29efc6324886b08ac52b27d92c1434
Parents: 10d5b7b
Author: Dave Brosius 
Authored: Mon Sep 4 22:25:53 2017 -0400
Committer: Dave Brosius 
Committed: Mon Sep 4 22:25:53 2017 -0400

--
 src/java/org/apache/cassandra/net/async/MessageOutHandler.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/d32c474f/src/java/org/apache/cassandra/net/async/MessageOutHandler.java
--
diff --git a/src/java/org/apache/cassandra/net/async/MessageOutHandler.java 
b/src/java/org/apache/cassandra/net/async/MessageOutHandler.java
index b4ceb92..e88b56a 100644
--- a/src/java/org/apache/cassandra/net/async/MessageOutHandler.java
+++ b/src/java/org/apache/cassandra/net/async/MessageOutHandler.java
@@ -115,7 +115,7 @@ class MessageOutHandler extends ChannelDuplexHandler
 // the channel handlers are removed from the channel potentially saync 
from the close operation.
 if (!ctx.channel().isOpen())
 {
-logger.debug("attempting to process a message in the pipeline, but 
the channel is closed", ctx.channel().id());
+logger.debug("attempting to process a message in the pipeline, but 
channel {} is closed", ctx.channel().id());
 return;
 }
 


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13418) Allow TWCS to ignore overlaps when dropping fully expired sstables

[mck (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mck updated CASSANDRA-13418:

   Resolution: Fixed
Fix Version/s: (was: 3.11.x)
   (was: 4.x)
   4.0
   3.11.1
   Status: Resolved  (was: Patch Available)

committed.

> Allow TWCS to ignore overlaps when dropping fully expired sstables
> --
>
> Key: CASSANDRA-13418
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13418
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Compaction
>Reporter: Corentin Chary
>Assignee: Romain GERARD
>  Labels: twcs
> Fix For: 3.11.1, 4.0
>
> Attachments: twcs-cleanup.png
>
>
> http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html explains it well. If 
> you really want read-repairs you're going to have sstables blocking the 
> expiration of other fully expired SSTables because they overlap.
> You can set unchecked_tombstone_compaction = true or tombstone_threshold to a 
> very low value and that will purge the blockers of old data that should 
> already have expired, thus removing the overlaps and allowing the other 
> SSTables to expire.
> The thing is that this is rather CPU intensive and not optimal. If you have 
> time series, you might not care if all your data doesn't exactly expire at 
> the right time, or if data re-appears for some time, as long as it gets 
> deleted as soon as it can. And in this situation I believe it would be really 
> beneficial to allow users to simply ignore overlapping SSTables when looking 
> for fully expired ones.
> To the question: why would you need read-repairs ?
> - Full repairs basically take longer than the TTL of the data on my dataset, 
> so this isn't really effective.
> - Even with a 10% chances of doing a repair, we found out that this would be 
> enough to greatly reduce entropy of the most used data (and if you have 
> timeseries, you're likely to have a dashboard doing the same important 
> queries over and over again).
> - LOCAL_QUORUM is too expensive (need >3 replicas), QUORUM is too slow.
> I'll try to come up with a patch demonstrating how this would work, try it on 
> our system and report the effects.
> cc: [~adejanovski], [~rgerard] as I know you worked on similar issues already.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[2/3] cassandra git commit: CASSANDRA-13418 Allow to skip overlapings checks

[mck]

CASSANDRA-13418 Allow to skip overlapings checks

 patch by Romain GÉRARD; reviewed by Mick Semb Wever for CASSANDRA-13418


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/14d67d81
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/14d67d81
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/14d67d81

Branch: refs/heads/trunk
Commit: 14d67d81c57d6387c77bd85c57b342d285880835
Parents: 37d6730
Author: Romain GÉRARD 
Authored: Wed Aug 16 16:21:46 2017 +0200
Committer: Mick Semb Wever 
Committed: Tue Sep 5 08:33:25 2017 +1000

--
 CHANGES.txt |  1 +
 .../db/compaction/CompactionController.java | 67 --
 .../TimeWindowCompactionController.java | 49 +
 .../TimeWindowCompactionStrategy.java   | 10 +--
 .../TimeWindowCompactionStrategyOptions.java| 22 ++
 .../db/compaction/TimeWindowCompactionTask.java | 42 +++
 .../db/compaction/CompactionControllerTest.java |  5 ++
 .../TimeWindowCompactionStrategyTest.java   | 74 +++-
 8 files changed, 257 insertions(+), 13 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/14d67d81/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 1f63ced..9218d90 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.11.1
+ * Add a compaction option to TWCS to ignore sstables overlapping checks 
(CASSANDRA-13418)
  * BTree.Builder memory leak (CASSANDRA-13754)
  * Revert CASSANDRA-10368 of supporting non-pk column filtering due to 
correctness (CASSANDRA-13798)
  * Fix cassandra-stress hang issues when an error during cluster connection 
happens (CASSANDRA-12938)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/14d67d81/src/java/org/apache/cassandra/db/compaction/CompactionController.java
--
diff --git 
a/src/java/org/apache/cassandra/db/compaction/CompactionController.java 
b/src/java/org/apache/cassandra/db/compaction/CompactionController.java
index bf3647a..84aac09 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionController.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionController.java
@@ -20,6 +20,7 @@ package org.apache.cassandra.db.compaction;
 import java.util.*;
 import java.util.function.Predicate;
 
+import org.apache.cassandra.config.Config;
 import org.apache.cassandra.db.Memtable;
 import org.apache.cassandra.db.rows.UnfilteredRowIterator;
 
@@ -49,7 +50,8 @@ import static 
org.apache.cassandra.db.lifecycle.SSTableIntervalTree.buildInterva
 public class CompactionController implements AutoCloseable
 {
 private static final Logger logger = 
LoggerFactory.getLogger(CompactionController.class);
-static final boolean NEVER_PURGE_TOMBSTONES = 
Boolean.getBoolean("cassandra.never_purge_tombstones");
+private static final String NEVER_PURGE_TOMBSTONES_PROPERTY = 
Config.PROPERTY_PREFIX + "never_purge_tombstones";
+static final boolean NEVER_PURGE_TOMBSTONES = 
Boolean.getBoolean(NEVER_PURGE_TOMBSTONES_PROPERTY);
 
 public final ColumnFamilyStore cfs;
 private final boolean compactingRepaired;
@@ -98,7 +100,14 @@ public class CompactionController implements AutoCloseable
 {
 if (NEVER_PURGE_TOMBSTONES)
 {
-logger.debug("not refreshing overlaps - running with 
-Dcassandra.never_purge_tombstones=true");
+logger.debug("not refreshing overlaps - running with -D{}=true",
+NEVER_PURGE_TOMBSTONES_PROPERTY);
+return;
+}
+
+if (ignoreOverlaps())
+{
+logger.debug("not refreshing overlaps - running with 
ignoreOverlaps activated");
 return;
 }
 
@@ -120,7 +129,7 @@ public class CompactionController implements AutoCloseable
 if (this.overlappingSSTables != null)
 close();
 
-if (compacting == null)
+if (compacting == null || ignoreOverlaps())
 overlappingSSTables = 
Refs.tryRef(Collections.emptyList());
 else
 overlappingSSTables = 
cfs.getAndReferenceOverlappingLiveSSTables(compacting);
@@ -129,7 +138,7 @@ public class CompactionController implements AutoCloseable
 
 public Set getFullyExpiredSSTables()
 {
-return getFullyExpiredSSTables(cfs, compacting, overlappingSSTables, 
gcBefore);
+return getFullyExpiredSSTables(cfs, compacting, overlappingSSTables, 
gcBefore, ignoreOverlaps());
 }
 
 /**
@@ -146,20 +155,39 @@ public class CompactionController implements AutoCloseable
  * @param compacting we take the drop-candidates from this set, it is 

[1/3] cassandra git commit: CASSANDRA-13418 Allow to skip overlapings checks

[mck]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.11 37d67306a -> 14d67d81c
  refs/heads/trunk c8d15f04f -> 10d5b7b2f


CASSANDRA-13418 Allow to skip overlapings checks

 patch by Romain GÉRARD; reviewed by Mick Semb Wever for CASSANDRA-13418


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/14d67d81
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/14d67d81
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/14d67d81

Branch: refs/heads/cassandra-3.11
Commit: 14d67d81c57d6387c77bd85c57b342d285880835
Parents: 37d6730
Author: Romain GÉRARD 
Authored: Wed Aug 16 16:21:46 2017 +0200
Committer: Mick Semb Wever 
Committed: Tue Sep 5 08:33:25 2017 +1000

--
 CHANGES.txt |  1 +
 .../db/compaction/CompactionController.java | 67 --
 .../TimeWindowCompactionController.java | 49 +
 .../TimeWindowCompactionStrategy.java   | 10 +--
 .../TimeWindowCompactionStrategyOptions.java| 22 ++
 .../db/compaction/TimeWindowCompactionTask.java | 42 +++
 .../db/compaction/CompactionControllerTest.java |  5 ++
 .../TimeWindowCompactionStrategyTest.java   | 74 +++-
 8 files changed, 257 insertions(+), 13 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/14d67d81/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 1f63ced..9218d90 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.11.1
+ * Add a compaction option to TWCS to ignore sstables overlapping checks 
(CASSANDRA-13418)
  * BTree.Builder memory leak (CASSANDRA-13754)
  * Revert CASSANDRA-10368 of supporting non-pk column filtering due to 
correctness (CASSANDRA-13798)
  * Fix cassandra-stress hang issues when an error during cluster connection 
happens (CASSANDRA-12938)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/14d67d81/src/java/org/apache/cassandra/db/compaction/CompactionController.java
--
diff --git 
a/src/java/org/apache/cassandra/db/compaction/CompactionController.java 
b/src/java/org/apache/cassandra/db/compaction/CompactionController.java
index bf3647a..84aac09 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionController.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionController.java
@@ -20,6 +20,7 @@ package org.apache.cassandra.db.compaction;
 import java.util.*;
 import java.util.function.Predicate;
 
+import org.apache.cassandra.config.Config;
 import org.apache.cassandra.db.Memtable;
 import org.apache.cassandra.db.rows.UnfilteredRowIterator;
 
@@ -49,7 +50,8 @@ import static 
org.apache.cassandra.db.lifecycle.SSTableIntervalTree.buildInterva
 public class CompactionController implements AutoCloseable
 {
 private static final Logger logger = 
LoggerFactory.getLogger(CompactionController.class);
-static final boolean NEVER_PURGE_TOMBSTONES = 
Boolean.getBoolean("cassandra.never_purge_tombstones");
+private static final String NEVER_PURGE_TOMBSTONES_PROPERTY = 
Config.PROPERTY_PREFIX + "never_purge_tombstones";
+static final boolean NEVER_PURGE_TOMBSTONES = 
Boolean.getBoolean(NEVER_PURGE_TOMBSTONES_PROPERTY);
 
 public final ColumnFamilyStore cfs;
 private final boolean compactingRepaired;
@@ -98,7 +100,14 @@ public class CompactionController implements AutoCloseable
 {
 if (NEVER_PURGE_TOMBSTONES)
 {
-logger.debug("not refreshing overlaps - running with 
-Dcassandra.never_purge_tombstones=true");
+logger.debug("not refreshing overlaps - running with -D{}=true",
+NEVER_PURGE_TOMBSTONES_PROPERTY);
+return;
+}
+
+if (ignoreOverlaps())
+{
+logger.debug("not refreshing overlaps - running with 
ignoreOverlaps activated");
 return;
 }
 
@@ -120,7 +129,7 @@ public class CompactionController implements AutoCloseable
 if (this.overlappingSSTables != null)
 close();
 
-if (compacting == null)
+if (compacting == null || ignoreOverlaps())
 overlappingSSTables = 
Refs.tryRef(Collections.emptyList());
 else
 overlappingSSTables = 
cfs.getAndReferenceOverlappingLiveSSTables(compacting);
@@ -129,7 +138,7 @@ public class CompactionController implements AutoCloseable
 
 public Set getFullyExpiredSSTables()
 {
-return getFullyExpiredSSTables(cfs, compacting, overlappingSSTables, 
gcBefore);
+return getFullyExpiredSSTables(cfs, compacting, overlappingSSTables, 
gcBefore, ignoreOverlaps());
 }
 
 /**
@@ -146,20 

[3/3] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

[mck]

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/10d5b7b2
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/10d5b7b2
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/10d5b7b2

Branch: refs/heads/trunk
Commit: 10d5b7b2f77fb7c25e288f42f7fb64b3131fad35
Parents: c8d15f0 14d67d8
Author: Mick Semb Wever 
Authored: Tue Sep 5 08:36:12 2017 +1000
Committer: Mick Semb Wever 
Committed: Tue Sep 5 08:38:48 2017 +1000

--
 doc/cql3/CQL.textile| 36 +-
 doc/source/operating/compaction.rst |  8 ++-
 .../db/compaction/CompactionController.java | 67 --
 .../TimeWindowCompactionController.java | 49 +
 .../TimeWindowCompactionStrategy.java   | 10 +--
 .../TimeWindowCompactionStrategyOptions.java| 22 ++
 .../db/compaction/TimeWindowCompactionTask.java | 42 +++
 .../db/compaction/CompactionControllerTest.java |  5 ++
 .../TimeWindowCompactionStrategyTest.java   | 74 +++-
 9 files changed, 281 insertions(+), 32 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/10d5b7b2/doc/cql3/CQL.textile
--
diff --cc doc/cql3/CQL.textile
index 88d6694,f2f9bd8..db1ec22
--- a/doc/cql3/CQL.textile
+++ b/doc/cql3/CQL.textile
@@@ -347,24 -347,24 +347,24 @@@ h4(#compactionOptions). Compaction opti
  
  The @compaction@ property must at least define the @'class'@ sub-option, that 
defines the compaction strategy class to use. The default supported class are 
@'SizeTieredCompactionStrategy'@, @'LeveledCompactionStrategy'@, 
@'DateTieredCompactionStrategy'@ and @'TimeWindowCompactionStrategy'@. Custom 
strategy can be provided by specifying the full class name as a "string 
constant":#constants. The rest of the sub-options depends on the chosen class. 
The sub-options supported by the default classes are:
  
--|_. option |_. supported compaction strategy |_. 
default|_. description |
--| @enabled@| _all_   | true   
  | A boolean denoting whether compaction should be enabled or not.|
--| @tombstone_threshold@| _all_   | 0.2
  | A ratio such that if a sstable has more than this ratio of gcable 
tombstones over all contained columns, the sstable will be compacted (with no 
other sstables) for the purpose of purging those tombstones. |
--| @tombstone_compaction_interval@  | _all_   | 1 day  
  | The minimum time to wait after an sstable creation time before 
considering it for "tombstone compaction", where "tombstone compaction" is the 
compaction triggered if the sstable has more gcable tombstones than 
@tombstone_threshold@. |
--| @unchecked_tombstone_compaction@ | _all_   | false  
  | Setting this to true enables more aggressive tombstone compactions - 
single sstable tombstone compactions will run without checking how likely it is 
that they will be successful. |
--| @min_sstable_size@   | SizeTieredCompactionStrategy| 50MB   
  | The size tiered strategy groups SSTables to compact in buckets. A 
bucket groups SSTables that differs from less than 50% in size.  However, for 
small sizes, this would result in a bucketing that is too fine grained. 
@min_sstable_size@ defines a size threshold (in bytes) below which all SSTables 
belong to one unique bucket|
--| @min_threshold@  | SizeTieredCompactionStrategy| 4  
  | Minimum number of SSTables needed to start a minor compaction.|
--| @max_threshold@  | SizeTieredCompactionStrategy| 32 
  | Maximum number of SSTables processed by one minor compaction.|
--| @bucket_low@ | SizeTieredCompactionStrategy| 0.5
  | Size tiered consider sstables to be within the same bucket if their 
size is within [average_size * @bucket_low@, average_size * @bucket_high@ ] 
(i.e the default groups sstable whose sizes diverges by at most 50%)|
--| @bucket_high@| SizeTieredCompactionStrategy| 1.5
  | Size tiered consider sstables to be within the same bucket if their 
size is within [average_size * @bucket_low@, average_size * @bucket_high@ ] 
(i.e the default groups sstable whose sizes diverges by at most 50%).|
--| @sstable_size_in_mb@ | LeveledCompactionStrategy   | 5MB
  | The target size (in MB) for sstables in the leveled strategy. Note that 
while sstable sizes should stay less or equal to @sstable_size_in_mb@, it is 
possible to exceptionally have a larger 

[jira] [Updated] (CASSANDRA-13794) Fix short read protection logic for querying more rows

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko updated CASSANDRA-13794:
--
Status: Patch Available  (was: In Progress)

Marking the ticket as {{Patch Available}}, despite its lack of (new) tests, so 
that it can be reviewed first. Tests will be committed with the rest of the 
code.

> Fix short read protection logic for querying more rows
> --
>
> Key: CASSANDRA-13794
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13794
> Project: Cassandra
>  Issue Type: Bug
>  Components: Coordination
>Reporter: Benedict
>Assignee: Aleksey Yeschenko
> Fix For: 3.0.x, 3.11.x
>
>
> Discovered by [~benedict] while reviewing CASSANDRA-13747:
> {quote}
> While reviewing I got a little suspicious of the modified line 
> {{DataResolver}} :479, as it seemed that n and x were the wrong way around... 
> and, reading the comment of intent directly above, and reproducing the 
> calculation, they are indeed.
> This is probably a significant enough bug that it warrants its own ticket for 
> record keeping, though I'm fairly agnostic on that decision.
> I'm a little concerned about our current short read behaviour, as right now 
> it seems we should be requesting exactly one row, for any size of under-read, 
> which could mean extremely poor performance in case of large under-reads.
> I would suggest that the outer unconditional {{Math.max}} is a bad idea, has 
> been (poorly) insulating us from this error, and that we should first be 
> asserting that the calculation yields a value >= 0 before setting to 1.
> {quote}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13794) Fix short read protection logic for querying more rows

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152884#comment-16152884
 ] 

Aleksey Yeschenko commented on CASSANDRA-13794:
---

Work in progress branch 
[here|https://github.com/iamaleksey/cassandra/tree/13794-3.0]. Currently 
missing (new) tests, but I want to get the underlying logic reviewed and 
approved, first. Would add coverage before committing it.

A short summary of the issue: the code right now has two variables swapped, 
which ultimately results in us always fetching 1 extra row per short read 
protection requests, in a blocking manner, making it very inefficient. But upon 
closer look, there are some other inefficiencies here that can and should be 
addressed:

1. One of our stop conditions is {{lastCount == counter.counted()}}. It's 
supposed to abort a short read if our previous attempt to fetch more rows 
yielded 0 extra rows. It's not incorrect, but is only a special case of the 
more general scenario: our previous attempt to fetch more extra rows yielded 
fewer results than we requested for. That would mean there is no more rows to 
fetch at that replica, and allows us to abort earlier and more frequently.

2. Another of our stop conditions is {{!counter.isDoneForPartition()}}. Once 
again, it isn't incorrect, but it can be extended further. Due to the way 
{{isDoneForPartition()}} is defined ({{isDone() || rowInCurrentPartition >= 
perPartitionLimit}}) and because of that counter being counting-only, it is 
possible for us to have fetched enough rows total for other partitions short 
read retries previously to hit the global limit of rows in the counter. That 
would make {{isDone()}} return {{true}} always, and have 
{{isDoneForPartition()}} return false positives even if the partition currently 
processed only has a partition level deletion and/or tombstones. That can 
affect queries that set per partition limit explicitly or when running {{SELECT 
DISTINCT}} queries. Spotted that during CASSANDRA-13747 fixing.

3. Once we've swapped {{x}} and {{n}} in {{moreContents()}} to fix the logic 
error, we'd still have some issues. In degenerate cases where we have some 
nodes missing a fresh partition deletion, for example, the formula would fetch 
*a lot* of rows {{n * (n - 1)}}, with {{n}} growing exponentially with every 
attempt.

Upon closer inspection, the formula doesn't make 100% sense. It claims that we 
miss {{n - x}} rows - where {{n = counter.countedInCurrentPartition()}} and {{x 
= postReconciliationCounter.countedInCurrentPartition()}}, but the number we 
really miss is {{limit - postReconciliationCounter.counted()}} or 
{{perPartitionLimit - postReconciliationCounter.countedInCurrentPartition()}}. 
They might be the same on our first short read protection iteration, but will 
be diverging further and further with each request. In addition to that, it 
seems to assume a uniform distribution of tombstones (in the end result) rows 
in the source partition, which can't be true for most workloads.

I couldn't come up with some ideal heuristic that covers all workloads, so I 
stuck to something safe that respects client paging limits but still attempts 
to minimise the # of requests we make by fetching (in most cases) more rows 
than is minimally necessary. I'm not completely sure about it, but I welcome 
any ideas on how to make it better. Either way, anything we do should be 
significantly more efficient than what we have now.

I've also made some renames, refactorings, and moved a few things around to 
better understand the code myself, and make it clearer for future contributors 
- including future me. The most significant noticeable change is application of 
the per-response counter shift to {{mergeWithShortReadProtection()}} method, 
instead of overloading {{ShortReadRowProtection}} with responsibilities - I 
also like it to be next to the global counter creation, so you can see the 
contrast in arguments.

> Fix short read protection logic for querying more rows
> --
>
> Key: CASSANDRA-13794
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13794
> Project: Cassandra
>  Issue Type: Bug
>  Components: Coordination
>Reporter: Benedict
>Assignee: Aleksey Yeschenko
> Fix For: 3.0.x, 3.11.x
>
>
> Discovered by [~benedict] while reviewing CASSANDRA-13747:
> {quote}
> While reviewing I got a little suspicious of the modified line 
> {{DataResolver}} :479, as it seemed that n and x were the wrong way around... 
> and, reading the comment of intent directly above, and reproducing the 
> calculation, they are indeed.
> This is probably a significant enough bug that it warrants its own ticket for 
> record keeping, though I'm fairly agnostic on that decision.
> I'm a little concerned about our current short 

COMPACTING and REPAIRING : dependency between these two

[helloga...@gmail.com]

Hi,

We have come across high disk space usage for which we wanted to do COMPACTING.
And data consistency due to connectivity issues and for that we wanted to do 
REPAIRING.

We thought of having some automatic jobs for COMPACTING and REPAIRING. SO, can 
someone please suggest what would be the frequency that they should be running.

Is there any dependency between COMPACTING and REPAIRING? if so, which one need 
to be executed first in these both.

Thanks
G

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

compaction and repairing dependecy

[helloga...@gmail.com]

Hi,

we have observed a cases where disk space usage is going high for which we 
wanted to do COMPACTING.
And also data consistency on connectivity issues and for that we wanted to do 
REPAIRING.

So, can some one please suggest frequency of both COMPACTING and REPAIRING. And 
is there any dependency between both of them? Which one need to be performed 
first and then which one?

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13831) NettyFactoryTest is failing in trunk on MacOS

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152731#comment-16152731
 ] 

Aleksey Yeschenko commented on CASSANDRA-13831:
---

This makes the tests pass but still shits all over the logs.

{code}
[junit] WARN  [main] 2017-09-04 16:33:51,193 NettyFactory.java:98 - epoll 
not availble {}
[junit] java.lang.ExceptionInInitializerError: null
[junit] at io.netty.channel.epoll.Epoll.(Epoll.java:33) 
~[netty-all-4.1.14.Final.jar:4.1.14.Final]
[junit] at 
org.apache.cassandra.service.NativeTransportService.useEpoll(NativeTransportService.java:162)
 ~[main/:na]
[junit] at 
org.apache.cassandra.net.async.NettyFactoryTest.(NettyFactoryTest.java:65)
 ~[classes/:na]
[junit] at java.lang.Class.forName0(Native Method) ~[na:1.8.0_144]
[junit] at java.lang.Class.forName(Class.java:264) ~[na:1.8.0_144]
[junit] at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:380)
 ~[ant-junit.jar:na]
[junit] at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182)
 ~[ant-junit.jar:na]
[junit] at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033)
 ~[ant-junit.jar:na]
[junit] Caused by: java.lang.IllegalStateException: Only supported on Linux
[junit] at 
io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:189) 
~[netty-all-4.1.14.Final.jar:4.1.14.Final]
[junit] at io.netty.channel.epoll.Native.(Native.java:61) 
~[netty-all-4.1.14.Final.jar:4.1.14.Final]
[junit] ... 8 common frames omitted
{code}

Could you maybe modify {{NativeTransportService.useEpoll()}} to look at the OS 
first before trying to call {{Epoll.isAvailable()}}, or fix it some other way?

> NettyFactoryTest is failing in trunk on MacOS
> -
>
> Key: CASSANDRA-13831
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13831
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Aleksey Yeschenko
>Assignee: Jason Brown
>Priority: Minor
> Fix For: 4.x
>
>
> Example failure:
> {code}
> [junit] Testcase: 
> getEventLoopGroup_EpollWithoutIoRatioBoost(org.apache.cassandra.net.async.NettyFactoryTest):
> Caused an ERROR
> [junit] failed to load the required native library
> [junit] java.lang.UnsatisfiedLinkError: failed to load the required 
> native library
> [junit]   at 
> io.netty.channel.epoll.Epoll.ensureAvailability(Epoll.java:78)
> [junit]   at 
> io.netty.channel.epoll.EpollEventLoop.(EpollEventLoop.java:53)
> [junit]   at 
> io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:134)
> [junit]   at 
> io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:35)
> [junit]   at 
> io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:84)
> [junit]   at 
> io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:58)
> [junit]   at 
> io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:47)
> [junit]   at 
> io.netty.channel.MultithreadEventLoopGroup.(MultithreadEventLoopGroup.java:59)
> [junit]   at 
> io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:104)
> [junit]   at 
> io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:91)
> [junit]   at 
> io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:68)
> [junit]   at 
> org.apache.cassandra.net.async.NettyFactory.getEventLoopGroup(NettyFactory.java:175)
> [junit]   at 
> org.apache.cassandra.net.async.NettyFactoryTest.getEventLoopGroup_Epoll(NettyFactoryTest.java:187)
> [junit]   at 
> org.apache.cassandra.net.async.NettyFactoryTest.getEventLoopGroup_EpollWithoutIoRatioBoost(NettyFactoryTest.java:205)
> [junit] Caused by: java.lang.ExceptionInInitializerError
> [junit]   at io.netty.channel.epoll.Epoll.(Epoll.java:33)
> [junit]   at 
> org.apache.cassandra.service.NativeTransportService.useEpoll(NativeTransportService.java:162)
> [junit]   at 
> org.apache.cassandra.net.async.NettyFactory.(NettyFactory.java:94)
> [junit]   at 
> org.apache.cassandra.net.async.NettyFactoryTest.getEventLoopGroup_Nio(NettyFactoryTest.java:216)
> [junit]   at 
> org.apache.cassandra.net.async.NettyFactoryTest.getEventLoopGroup_NioWithoutIoRatioBoost(NettyFactoryTest.java:211)
> [junit] Caused by: java.lang.IllegalStateException: Only supported on 
> Linux
> [junit]   at 
> io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:189)
> [junit]   at io.netty.channel.epoll.Native.(Native.java:61)
> {code}
> It's 

[jira] [Updated] (CASSANDRA-13662) Remove unsupported CREDENTIALS message

[Jeremiah Jordan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeremiah Jordan updated CASSANDRA-13662:

Status: Ready to Commit  (was: Patch Available)

> Remove unsupported CREDENTIALS message
> --
>
> Key: CASSANDRA-13662
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13662
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Auth, CQL
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Minor
>  Labels: security
> Fix For: 4.x
>
>
> Remove CREDENTIAL message, as protocol v1 isn't supported anyways. Let's try 
> not to keep unused legacy classes around for any security relevant features. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13662) Remove unsupported CREDENTIALS message

[Jeremiah Jordan (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152651#comment-16152651
 ] 

Jeremiah Jordan commented on CASSANDRA-13662:
-

Yeah. Might be silly. Just asking the question as we have been wanting to focus 
on making sure new code has tests lately...

> Remove unsupported CREDENTIALS message
> --
>
> Key: CASSANDRA-13662
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13662
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Auth, CQL
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Minor
>  Labels: security
> Fix For: 4.x
>
>
> Remove CREDENTIAL message, as protocol v1 isn't supported anyways. Let's try 
> not to keep unused legacy classes around for any security relevant features. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13818) Add support for --hosts, --force, and subrange repair to incremental repair

[Marcus Eriksson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152636#comment-16152636
 ] 

Marcus Eriksson commented on CASSANDRA-13818:
-

Can't we run a normal repair over the unrepaired sstables to avoid the 
pointless anticompaction? I guess we would not be as consistent and might 
overstream a bit, but it can't hurt anything more than that right?

> Add support for --hosts, --force, and subrange repair to incremental repair
> ---
>
> Key: CASSANDRA-13818
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13818
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
> Fix For: 4.0
>
>
> It should be possible to run incremental repair with nodes down, we just 
> shouldn't promote the data to repaired afterwards



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13833) Failed compaction is not captured

[Marcus Eriksson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-13833:

   Resolution: Fixed
Fix Version/s: 4.0
   3.11.1
   3.0.15
   2.2.11
   Status: Resolved  (was: Patch Available)

committed as {{e80ede6d393460f22ee}}, thanks!

> Failed compaction is not captured
> -
>
> Key: CASSANDRA-13833
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13833
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
> Fix For: 2.2.11, 3.0.15, 3.11.1, 4.0
>
>
> Follow up for CASSANDRA-13785, when the compaction failed, it fails silently. 
> No error message is logged and exceptions metric is not updated. Basically, 
> it's unable to get the exception: 
> [CompactionManager.java:1491|https://github.com/apache/cassandra/blob/cassandra-2.2/src/java/org/apache/cassandra/db/compaction/CompactionManager.java#L1491]
> Here is the call stack:
> {noformat}
> at 
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:195)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at 
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:89)
> at 
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at 
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at 
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> There're 2 {{FutureTask}} in the call stack, for example 
> {{FutureTask1(FutureTask2))}}, If the call thrown an exception, 
> {{FutureTask2}} sets the status, save the exception and return. But 
> FutureTask1 doesn't get any exception, then set the status to normal. So 
> we're unable to get the exception in:
> [CompactionManager.java:1491|https://github.com/apache/cassandra/blob/cassandra-2.2/src/java/org/apache/cassandra/db/compaction/CompactionManager.java#L1491]
> 2.1.x is working fine, here is the call stack:
> {noformat}
> at 
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) 
> ~[main/:na]
> at 
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:73)
>  ~[main/:na]
> at 
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
>  ~[main/:na]
> at 
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264)
>  ~[main/:na]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> ~[na:1.8.0_141]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> ~[na:1.8.0_141]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  ~[na:1.8.0_141]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  [na:1.8.0_141]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_141]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[02/10] cassandra git commit: Fix compaction and flush exception not captured issue

[marcuse]

Fix compaction and flush exception not captured issue

patch by Jay Zhuang; reviewed by marcuse for CASSANDRA-13833


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e80ede6d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e80ede6d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e80ede6d

Branch: refs/heads/cassandra-3.0
Commit: e80ede6d393460f22ee2b313d4bac7e3fbbfe893
Parents: 4d90573
Author: Jay Zhuang 
Authored: Thu Aug 31 11:07:07 2017 -0700
Committer: Marcus Eriksson 
Committed: Mon Sep 4 15:01:02 2017 +0200

--
 CHANGES.txt |   1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  |   4 +-
 .../db/compaction/CompactionManager.java|   4 +-
 .../db/compaction/CompactionExecutorTest.java   | 131 +++
 4 files changed, 136 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/e80ede6d/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 4e68ddc..03a78fd 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.11
+ * Fix compaction and flush exception not captured (CASSANDRA-13833)
  * Make BatchlogManagerMBean.forceBatchlogReplay() blocking (CASSANDRA-13809)
  * Uncaught exceptions in Netty pipeline (CASSANDRA-13649)
  * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067) 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/e80ede6d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index 2e52eb2..7e36e11 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -906,9 +906,9 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 logFlush();
 Flush flush = new Flush(false);
 ListenableFutureTask flushTask = 
ListenableFutureTask.create(flush, null);
-flushExecutor.submit(flushTask);
+flushExecutor.execute(flushTask);
 ListenableFutureTask task = 
ListenableFutureTask.create(flush.postFlush);
-postFlushExecutor.submit(task);
+postFlushExecutor.execute(task);
 
 @SuppressWarnings("unchecked")
 ListenableFuture future = 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/e80ede6d/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java 
b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
index d21f1e8..cd50646 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
@@ -1457,7 +1457,7 @@ public class CompactionManager implements 
CompactionManagerMBean
 return CompactionMetrics.getCompactions().size();
 }
 
-private static class CompactionExecutor extends 
JMXEnabledThreadPoolExecutor
+static class CompactionExecutor extends JMXEnabledThreadPoolExecutor
 {
 protected CompactionExecutor(int minThreads, int maxThreads, String 
name, BlockingQueue queue)
 {
@@ -1537,7 +1537,7 @@ public class CompactionManager implements 
CompactionManagerMBean
 try
 {
 ListenableFutureTask ret = ListenableFutureTask.create(task);
-submit(ret);
+execute(ret);
 return ret;
 }
 catch (RejectedExecutionException ex)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/e80ede6d/test/unit/org/apache/cassandra/db/compaction/CompactionExecutorTest.java
--
diff --git 
a/test/unit/org/apache/cassandra/db/compaction/CompactionExecutorTest.java 
b/test/unit/org/apache/cassandra/db/compaction/CompactionExecutorTest.java
new file mode 100644
index 000..c6feb3f
--- /dev/null
+++ b/test/unit/org/apache/cassandra/db/compaction/CompactionExecutorTest.java
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a 

[05/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[marcuse]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b0eba5f9
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b0eba5f9
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b0eba5f9

Branch: refs/heads/cassandra-3.11
Commit: b0eba5f9c64db18840a4b0e4d56a589c5f2e08cd
Parents: f791c26 e80ede6
Author: Marcus Eriksson 
Authored: Mon Sep 4 15:02:53 2017 +0200
Committer: Marcus Eriksson 
Committed: Mon Sep 4 15:02:53 2017 +0200

--
 CHANGES.txt |   1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  |   2 +-
 .../db/compaction/CompactionManager.java|   4 +-
 .../db/compaction/CompactionExecutorTest.java   | 107 +++
 4 files changed, 111 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/b0eba5f9/CHANGES.txt
--
diff --cc CHANGES.txt
index b405fdf,03a78fd..3baa63b
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,37 -1,8 +1,38 @@@
 -2.2.11
 +3.0.15
 + * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626) 
 + * Fix race condition in read command serialization (CASSANDRA-13363)
 + * Enable segement creation before recovering commitlogs (CASSANDRA-13587)
 + * Fix AssertionError in short read protection (CASSANDRA-13747)
 + * Don't skip corrupted sstables on startup (CASSANDRA-13620)
 + * Fix the merging of cells with different user type versions 
(CASSANDRA-13776)
 + * Copy session properties on cqlsh.py do_login (CASSANDRA-13640)
 + * Potential AssertionError during ReadRepair of range tombstone and 
partition deletions (CASSANDRA-13719)
 + * Don't let stress write warmup data if n=0 (CASSANDRA-13773)
 + * Gossip thread slows down when using batch commit log (CASSANDRA-12966)
 + * Randomize batchlog endpoint selection with only 1 or 2 racks 
(CASSANDRA-12884)
 + * Fix digest calculation for counter cells (CASSANDRA-13750)
 + * Fix ColumnDefinition.cellValueType() for non-frozen collection and change 
SSTabledump to use type.toJSONString() (CASSANDRA-13573)
 + * Skip materialized view addition if the base table doesn't exist 
(CASSANDRA-13737)
 + * Drop table should remove corresponding entries in dropped_columns table 
(CASSANDRA-13730)
 + * Log warn message until legacy auth tables have been migrated 
(CASSANDRA-13371)
 + * Fix incorrect [2.1 <- 3.0] serialization of counter cells created in 2.0 
(CASSANDRA-13691)
 + * Fix invalid writetime for null cells (CASSANDRA-13711)
 + * Fix ALTER TABLE statement to atomically propagate changes to the table and 
its MVs (CASSANDRA-12952)
 + * Fixed ambiguous output of nodetool tablestats command (CASSANDRA-13722)
 + * JMXEnabledThreadPoolExecutor with corePoolSize equal to maxPoolSize 
(Backport CASSANDRA-13329)
 + * Fix Digest mismatch Exception if hints file has UnknownColumnFamily 
(CASSANDRA-13696)
 + * Purge tombstones created by expired cells (CASSANDRA-13643)
 + * Make concat work with iterators that have different subsets of columns 
(CASSANDRA-13482)
 + * Set test.runners based on cores and memory size (CASSANDRA-13078)
 + * Allow different NUMACTL_ARGS to be passed in (CASSANDRA-13557)
 + * Allow native function calls in CQLSSTableWriter (CASSANDRA-12606)
 + * Fix secondary index queries on COMPACT tables (CASSANDRA-13627)
 + * Nodetool listsnapshots output is missing a newline, if there are no 
snapshots (CASSANDRA-13568)
 + * sstabledump reports incorrect usage for argument order (CASSANDRA-13532)
 +Merged from 2.2:
+  * Fix compaction and flush exception not captured (CASSANDRA-13833)
 - * Make BatchlogManagerMBean.forceBatchlogReplay() blocking (CASSANDRA-13809)
   * Uncaught exceptions in Netty pipeline (CASSANDRA-13649)
 - * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067) 
 + * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067)
   * Fix queries with LIMIT and filtering on clustering columns 
(CASSANDRA-11223)
   * Fix potential NPE when resume bootstrap fails (CASSANDRA-13272)
   * Fix toJSONString for the UDT, tuple and collection types (CASSANDRA-13592)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/b0eba5f9/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --cc src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index 7251244,7e36e11..183176c
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@@ -837,12 -899,33 +837,12 @@@ public class ColumnFamilyStore implemen
  {
  synchronized (data)
  {
 -if (previousFlushFailure != null)
 -  

[09/10] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[marcuse]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/37d67306
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/37d67306
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/37d67306

Branch: refs/heads/trunk
Commit: 37d67306accb2fefed9cfc57856fcca4df93a407
Parents: bed7fa5 b0eba5f
Author: Marcus Eriksson 
Authored: Mon Sep 4 15:04:04 2017 +0200
Committer: Marcus Eriksson 
Committed: Mon Sep 4 15:04:04 2017 +0200

--
 CHANGES.txt |   1 +
 .../db/compaction/CompactionManager.java|   4 +-
 .../db/compaction/CompactionExecutorTest.java   | 109 +++
 3 files changed, 112 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/37d67306/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/37d67306/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/37d67306/test/unit/org/apache/cassandra/db/compaction/CompactionExecutorTest.java
--
diff --cc 
test/unit/org/apache/cassandra/db/compaction/CompactionExecutorTest.java
index 000,9b07da9..2f8b5b2
mode 00,100644..100644
--- a/test/unit/org/apache/cassandra/db/compaction/CompactionExecutorTest.java
+++ b/test/unit/org/apache/cassandra/db/compaction/CompactionExecutorTest.java
@@@ -1,0 -1,107 +1,109 @@@
+ /*
+  * Licensed to the Apache Software Foundation (ASF) under one
+  * or more contributor license agreements.  See the NOTICE file
+  * distributed with this work for additional information
+  * regarding copyright ownership.  The ASF licenses this file
+  * to you under the Apache License, Version 2.0 (the
+  * "License"); you may not use this file except in compliance
+  * with the License.  You may obtain a copy of the License at
+  *
+  * http://www.apache.org/licenses/LICENSE-2.0
+  *
+  * Unless required by applicable law or agreed to in writing, software
+  * distributed under the License is distributed on an "AS IS" BASIS,
+  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  * See the License for the specific language governing permissions and
+  * limitations under the License.
+  */
+ 
+ package org.apache.cassandra.db.compaction;
+ 
+ import java.util.concurrent.Future;
+ import java.util.concurrent.TimeUnit;
+ 
+ import org.junit.After;
+ import org.junit.Before;
+ import org.junit.Test;
+ import org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor;
++import org.apache.cassandra.config.DatabaseDescriptor;
+ 
+ import static org.junit.Assert.assertEquals;
+ import static org.junit.Assert.assertNotNull;
+ 
+ public class CompactionExecutorTest
+ {
+ static Throwable testTaskThrowable = null;
+ private static class TestTaskExecutor extends 
CompactionManager.CompactionExecutor
+ {
+ @Override
+ public void afterExecute(Runnable r, Throwable t)
+ {
+ if (t == null)
+ {
+ t = DebuggableThreadPoolExecutor.extractThrowable(r);
+ }
+ testTaskThrowable = t;
+ }
+ @Override
+ protected void beforeExecute(Thread t, Runnable r)
+ {
+ }
+ }
+ private CompactionManager.CompactionExecutor executor;
+ 
+ @Before
+ public void setup()
+ {
++DatabaseDescriptor.daemonInitialization();
+ executor = new TestTaskExecutor();
+ }
+ 
+ @After
+ public void destroy() throws Exception
+ {
+ executor.shutdown();
+ executor.awaitTermination(1, TimeUnit.MINUTES);
+ }
+ 
+ @Test
+ public void testFailedRunnable() throws Exception
+ {
+ testTaskThrowable = null;
+ Future tt = executor.submitIfRunning(
+ () -> { assert false : "testFailedRunnable"; }
+ , "compactionExecutorTest");
+ 
+ while (!tt.isDone())
+ Thread.sleep(10);
+ assertNotNull(testTaskThrowable);
+ assertEquals(testTaskThrowable.getMessage(), "testFailedRunnable");
+ }
+ 
+ @Test
+ public void testFailedCallable() throws Exception
+ {
+ testTaskThrowable = null;
+ Future tt = executor.submitIfRunning(
+ () -> { assert false : "testFailedCallable"; return 1; }
+ , "compactionExecutorTest");
+ 
+ while (!tt.isDone())
+ Thread.sleep(10);
+ assertNotNull(testTaskThrowable);
+ 

[03/10] cassandra git commit: Fix compaction and flush exception not captured issue

[marcuse]

Fix compaction and flush exception not captured issue

patch by Jay Zhuang; reviewed by marcuse for CASSANDRA-13833


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e80ede6d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e80ede6d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e80ede6d

Branch: refs/heads/cassandra-3.11
Commit: e80ede6d393460f22ee2b313d4bac7e3fbbfe893
Parents: 4d90573
Author: Jay Zhuang 
Authored: Thu Aug 31 11:07:07 2017 -0700
Committer: Marcus Eriksson 
Committed: Mon Sep 4 15:01:02 2017 +0200

--
 CHANGES.txt |   1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  |   4 +-
 .../db/compaction/CompactionManager.java|   4 +-
 .../db/compaction/CompactionExecutorTest.java   | 131 +++
 4 files changed, 136 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/e80ede6d/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 4e68ddc..03a78fd 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.11
+ * Fix compaction and flush exception not captured (CASSANDRA-13833)
  * Make BatchlogManagerMBean.forceBatchlogReplay() blocking (CASSANDRA-13809)
  * Uncaught exceptions in Netty pipeline (CASSANDRA-13649)
  * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067) 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/e80ede6d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index 2e52eb2..7e36e11 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -906,9 +906,9 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 logFlush();
 Flush flush = new Flush(false);
 ListenableFutureTask flushTask = 
ListenableFutureTask.create(flush, null);
-flushExecutor.submit(flushTask);
+flushExecutor.execute(flushTask);
 ListenableFutureTask task = 
ListenableFutureTask.create(flush.postFlush);
-postFlushExecutor.submit(task);
+postFlushExecutor.execute(task);
 
 @SuppressWarnings("unchecked")
 ListenableFuture future = 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/e80ede6d/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java 
b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
index d21f1e8..cd50646 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
@@ -1457,7 +1457,7 @@ public class CompactionManager implements 
CompactionManagerMBean
 return CompactionMetrics.getCompactions().size();
 }
 
-private static class CompactionExecutor extends 
JMXEnabledThreadPoolExecutor
+static class CompactionExecutor extends JMXEnabledThreadPoolExecutor
 {
 protected CompactionExecutor(int minThreads, int maxThreads, String 
name, BlockingQueue queue)
 {
@@ -1537,7 +1537,7 @@ public class CompactionManager implements 
CompactionManagerMBean
 try
 {
 ListenableFutureTask ret = ListenableFutureTask.create(task);
-submit(ret);
+execute(ret);
 return ret;
 }
 catch (RejectedExecutionException ex)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/e80ede6d/test/unit/org/apache/cassandra/db/compaction/CompactionExecutorTest.java
--
diff --git 
a/test/unit/org/apache/cassandra/db/compaction/CompactionExecutorTest.java 
b/test/unit/org/apache/cassandra/db/compaction/CompactionExecutorTest.java
new file mode 100644
index 000..c6feb3f
--- /dev/null
+++ b/test/unit/org/apache/cassandra/db/compaction/CompactionExecutorTest.java
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a 

[08/10] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[marcuse]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/37d67306
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/37d67306
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/37d67306

Branch: refs/heads/cassandra-3.11
Commit: 37d67306accb2fefed9cfc57856fcca4df93a407
Parents: bed7fa5 b0eba5f
Author: Marcus Eriksson 
Authored: Mon Sep 4 15:04:04 2017 +0200
Committer: Marcus Eriksson 
Committed: Mon Sep 4 15:04:04 2017 +0200

--
 CHANGES.txt |   1 +
 .../db/compaction/CompactionManager.java|   4 +-
 .../db/compaction/CompactionExecutorTest.java   | 109 +++
 3 files changed, 112 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/37d67306/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/37d67306/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/37d67306/test/unit/org/apache/cassandra/db/compaction/CompactionExecutorTest.java
--
diff --cc 
test/unit/org/apache/cassandra/db/compaction/CompactionExecutorTest.java
index 000,9b07da9..2f8b5b2
mode 00,100644..100644
--- a/test/unit/org/apache/cassandra/db/compaction/CompactionExecutorTest.java
+++ b/test/unit/org/apache/cassandra/db/compaction/CompactionExecutorTest.java
@@@ -1,0 -1,107 +1,109 @@@
+ /*
+  * Licensed to the Apache Software Foundation (ASF) under one
+  * or more contributor license agreements.  See the NOTICE file
+  * distributed with this work for additional information
+  * regarding copyright ownership.  The ASF licenses this file
+  * to you under the Apache License, Version 2.0 (the
+  * "License"); you may not use this file except in compliance
+  * with the License.  You may obtain a copy of the License at
+  *
+  * http://www.apache.org/licenses/LICENSE-2.0
+  *
+  * Unless required by applicable law or agreed to in writing, software
+  * distributed under the License is distributed on an "AS IS" BASIS,
+  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  * See the License for the specific language governing permissions and
+  * limitations under the License.
+  */
+ 
+ package org.apache.cassandra.db.compaction;
+ 
+ import java.util.concurrent.Future;
+ import java.util.concurrent.TimeUnit;
+ 
+ import org.junit.After;
+ import org.junit.Before;
+ import org.junit.Test;
+ import org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor;
++import org.apache.cassandra.config.DatabaseDescriptor;
+ 
+ import static org.junit.Assert.assertEquals;
+ import static org.junit.Assert.assertNotNull;
+ 
+ public class CompactionExecutorTest
+ {
+ static Throwable testTaskThrowable = null;
+ private static class TestTaskExecutor extends 
CompactionManager.CompactionExecutor
+ {
+ @Override
+ public void afterExecute(Runnable r, Throwable t)
+ {
+ if (t == null)
+ {
+ t = DebuggableThreadPoolExecutor.extractThrowable(r);
+ }
+ testTaskThrowable = t;
+ }
+ @Override
+ protected void beforeExecute(Thread t, Runnable r)
+ {
+ }
+ }
+ private CompactionManager.CompactionExecutor executor;
+ 
+ @Before
+ public void setup()
+ {
++DatabaseDescriptor.daemonInitialization();
+ executor = new TestTaskExecutor();
+ }
+ 
+ @After
+ public void destroy() throws Exception
+ {
+ executor.shutdown();
+ executor.awaitTermination(1, TimeUnit.MINUTES);
+ }
+ 
+ @Test
+ public void testFailedRunnable() throws Exception
+ {
+ testTaskThrowable = null;
+ Future tt = executor.submitIfRunning(
+ () -> { assert false : "testFailedRunnable"; }
+ , "compactionExecutorTest");
+ 
+ while (!tt.isDone())
+ Thread.sleep(10);
+ assertNotNull(testTaskThrowable);
+ assertEquals(testTaskThrowable.getMessage(), "testFailedRunnable");
+ }
+ 
+ @Test
+ public void testFailedCallable() throws Exception
+ {
+ testTaskThrowable = null;
+ Future tt = executor.submitIfRunning(
+ () -> { assert false : "testFailedCallable"; return 1; }
+ , "compactionExecutorTest");
+ 
+ while (!tt.isDone())
+ Thread.sleep(10);
+ assertNotNull(testTaskThrowable);
+ 

[06/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[marcuse]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b0eba5f9
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b0eba5f9
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b0eba5f9

Branch: refs/heads/cassandra-3.0
Commit: b0eba5f9c64db18840a4b0e4d56a589c5f2e08cd
Parents: f791c26 e80ede6
Author: Marcus Eriksson 
Authored: Mon Sep 4 15:02:53 2017 +0200
Committer: Marcus Eriksson 
Committed: Mon Sep 4 15:02:53 2017 +0200

--
 CHANGES.txt |   1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  |   2 +-
 .../db/compaction/CompactionManager.java|   4 +-
 .../db/compaction/CompactionExecutorTest.java   | 107 +++
 4 files changed, 111 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/b0eba5f9/CHANGES.txt
--
diff --cc CHANGES.txt
index b405fdf,03a78fd..3baa63b
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,37 -1,8 +1,38 @@@
 -2.2.11
 +3.0.15
 + * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626) 
 + * Fix race condition in read command serialization (CASSANDRA-13363)
 + * Enable segement creation before recovering commitlogs (CASSANDRA-13587)
 + * Fix AssertionError in short read protection (CASSANDRA-13747)
 + * Don't skip corrupted sstables on startup (CASSANDRA-13620)
 + * Fix the merging of cells with different user type versions 
(CASSANDRA-13776)
 + * Copy session properties on cqlsh.py do_login (CASSANDRA-13640)
 + * Potential AssertionError during ReadRepair of range tombstone and 
partition deletions (CASSANDRA-13719)
 + * Don't let stress write warmup data if n=0 (CASSANDRA-13773)
 + * Gossip thread slows down when using batch commit log (CASSANDRA-12966)
 + * Randomize batchlog endpoint selection with only 1 or 2 racks 
(CASSANDRA-12884)
 + * Fix digest calculation for counter cells (CASSANDRA-13750)
 + * Fix ColumnDefinition.cellValueType() for non-frozen collection and change 
SSTabledump to use type.toJSONString() (CASSANDRA-13573)
 + * Skip materialized view addition if the base table doesn't exist 
(CASSANDRA-13737)
 + * Drop table should remove corresponding entries in dropped_columns table 
(CASSANDRA-13730)
 + * Log warn message until legacy auth tables have been migrated 
(CASSANDRA-13371)
 + * Fix incorrect [2.1 <- 3.0] serialization of counter cells created in 2.0 
(CASSANDRA-13691)
 + * Fix invalid writetime for null cells (CASSANDRA-13711)
 + * Fix ALTER TABLE statement to atomically propagate changes to the table and 
its MVs (CASSANDRA-12952)
 + * Fixed ambiguous output of nodetool tablestats command (CASSANDRA-13722)
 + * JMXEnabledThreadPoolExecutor with corePoolSize equal to maxPoolSize 
(Backport CASSANDRA-13329)
 + * Fix Digest mismatch Exception if hints file has UnknownColumnFamily 
(CASSANDRA-13696)
 + * Purge tombstones created by expired cells (CASSANDRA-13643)
 + * Make concat work with iterators that have different subsets of columns 
(CASSANDRA-13482)
 + * Set test.runners based on cores and memory size (CASSANDRA-13078)
 + * Allow different NUMACTL_ARGS to be passed in (CASSANDRA-13557)
 + * Allow native function calls in CQLSSTableWriter (CASSANDRA-12606)
 + * Fix secondary index queries on COMPACT tables (CASSANDRA-13627)
 + * Nodetool listsnapshots output is missing a newline, if there are no 
snapshots (CASSANDRA-13568)
 + * sstabledump reports incorrect usage for argument order (CASSANDRA-13532)
 +Merged from 2.2:
+  * Fix compaction and flush exception not captured (CASSANDRA-13833)
 - * Make BatchlogManagerMBean.forceBatchlogReplay() blocking (CASSANDRA-13809)
   * Uncaught exceptions in Netty pipeline (CASSANDRA-13649)
 - * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067) 
 + * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067)
   * Fix queries with LIMIT and filtering on clustering columns 
(CASSANDRA-11223)
   * Fix potential NPE when resume bootstrap fails (CASSANDRA-13272)
   * Fix toJSONString for the UDT, tuple and collection types (CASSANDRA-13592)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/b0eba5f9/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --cc src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index 7251244,7e36e11..183176c
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@@ -837,12 -899,33 +837,12 @@@ public class ColumnFamilyStore implemen
  {
  synchronized (data)
  {
 -if (previousFlushFailure != null)
 -   

[04/10] cassandra git commit: Fix compaction and flush exception not captured issue

[marcuse]

Fix compaction and flush exception not captured issue

patch by Jay Zhuang; reviewed by marcuse for CASSANDRA-13833


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e80ede6d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e80ede6d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e80ede6d

Branch: refs/heads/trunk
Commit: e80ede6d393460f22ee2b313d4bac7e3fbbfe893
Parents: 4d90573
Author: Jay Zhuang 
Authored: Thu Aug 31 11:07:07 2017 -0700
Committer: Marcus Eriksson 
Committed: Mon Sep 4 15:01:02 2017 +0200

--
 CHANGES.txt |   1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  |   4 +-
 .../db/compaction/CompactionManager.java|   4 +-
 .../db/compaction/CompactionExecutorTest.java   | 131 +++
 4 files changed, 136 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/e80ede6d/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 4e68ddc..03a78fd 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.11
+ * Fix compaction and flush exception not captured (CASSANDRA-13833)
  * Make BatchlogManagerMBean.forceBatchlogReplay() blocking (CASSANDRA-13809)
  * Uncaught exceptions in Netty pipeline (CASSANDRA-13649)
  * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067) 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/e80ede6d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index 2e52eb2..7e36e11 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -906,9 +906,9 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 logFlush();
 Flush flush = new Flush(false);
 ListenableFutureTask flushTask = 
ListenableFutureTask.create(flush, null);
-flushExecutor.submit(flushTask);
+flushExecutor.execute(flushTask);
 ListenableFutureTask task = 
ListenableFutureTask.create(flush.postFlush);
-postFlushExecutor.submit(task);
+postFlushExecutor.execute(task);
 
 @SuppressWarnings("unchecked")
 ListenableFuture future = 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/e80ede6d/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java 
b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
index d21f1e8..cd50646 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
@@ -1457,7 +1457,7 @@ public class CompactionManager implements 
CompactionManagerMBean
 return CompactionMetrics.getCompactions().size();
 }
 
-private static class CompactionExecutor extends 
JMXEnabledThreadPoolExecutor
+static class CompactionExecutor extends JMXEnabledThreadPoolExecutor
 {
 protected CompactionExecutor(int minThreads, int maxThreads, String 
name, BlockingQueue queue)
 {
@@ -1537,7 +1537,7 @@ public class CompactionManager implements 
CompactionManagerMBean
 try
 {
 ListenableFutureTask ret = ListenableFutureTask.create(task);
-submit(ret);
+execute(ret);
 return ret;
 }
 catch (RejectedExecutionException ex)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/e80ede6d/test/unit/org/apache/cassandra/db/compaction/CompactionExecutorTest.java
--
diff --git 
a/test/unit/org/apache/cassandra/db/compaction/CompactionExecutorTest.java 
b/test/unit/org/apache/cassandra/db/compaction/CompactionExecutorTest.java
new file mode 100644
index 000..c6feb3f
--- /dev/null
+++ b/test/unit/org/apache/cassandra/db/compaction/CompactionExecutorTest.java
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the 

[07/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[marcuse]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b0eba5f9
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b0eba5f9
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b0eba5f9

Branch: refs/heads/trunk
Commit: b0eba5f9c64db18840a4b0e4d56a589c5f2e08cd
Parents: f791c26 e80ede6
Author: Marcus Eriksson 
Authored: Mon Sep 4 15:02:53 2017 +0200
Committer: Marcus Eriksson 
Committed: Mon Sep 4 15:02:53 2017 +0200

--
 CHANGES.txt |   1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  |   2 +-
 .../db/compaction/CompactionManager.java|   4 +-
 .../db/compaction/CompactionExecutorTest.java   | 107 +++
 4 files changed, 111 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/b0eba5f9/CHANGES.txt
--
diff --cc CHANGES.txt
index b405fdf,03a78fd..3baa63b
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,37 -1,8 +1,38 @@@
 -2.2.11
 +3.0.15
 + * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626) 
 + * Fix race condition in read command serialization (CASSANDRA-13363)
 + * Enable segement creation before recovering commitlogs (CASSANDRA-13587)
 + * Fix AssertionError in short read protection (CASSANDRA-13747)
 + * Don't skip corrupted sstables on startup (CASSANDRA-13620)
 + * Fix the merging of cells with different user type versions 
(CASSANDRA-13776)
 + * Copy session properties on cqlsh.py do_login (CASSANDRA-13640)
 + * Potential AssertionError during ReadRepair of range tombstone and 
partition deletions (CASSANDRA-13719)
 + * Don't let stress write warmup data if n=0 (CASSANDRA-13773)
 + * Gossip thread slows down when using batch commit log (CASSANDRA-12966)
 + * Randomize batchlog endpoint selection with only 1 or 2 racks 
(CASSANDRA-12884)
 + * Fix digest calculation for counter cells (CASSANDRA-13750)
 + * Fix ColumnDefinition.cellValueType() for non-frozen collection and change 
SSTabledump to use type.toJSONString() (CASSANDRA-13573)
 + * Skip materialized view addition if the base table doesn't exist 
(CASSANDRA-13737)
 + * Drop table should remove corresponding entries in dropped_columns table 
(CASSANDRA-13730)
 + * Log warn message until legacy auth tables have been migrated 
(CASSANDRA-13371)
 + * Fix incorrect [2.1 <- 3.0] serialization of counter cells created in 2.0 
(CASSANDRA-13691)
 + * Fix invalid writetime for null cells (CASSANDRA-13711)
 + * Fix ALTER TABLE statement to atomically propagate changes to the table and 
its MVs (CASSANDRA-12952)
 + * Fixed ambiguous output of nodetool tablestats command (CASSANDRA-13722)
 + * JMXEnabledThreadPoolExecutor with corePoolSize equal to maxPoolSize 
(Backport CASSANDRA-13329)
 + * Fix Digest mismatch Exception if hints file has UnknownColumnFamily 
(CASSANDRA-13696)
 + * Purge tombstones created by expired cells (CASSANDRA-13643)
 + * Make concat work with iterators that have different subsets of columns 
(CASSANDRA-13482)
 + * Set test.runners based on cores and memory size (CASSANDRA-13078)
 + * Allow different NUMACTL_ARGS to be passed in (CASSANDRA-13557)
 + * Allow native function calls in CQLSSTableWriter (CASSANDRA-12606)
 + * Fix secondary index queries on COMPACT tables (CASSANDRA-13627)
 + * Nodetool listsnapshots output is missing a newline, if there are no 
snapshots (CASSANDRA-13568)
 + * sstabledump reports incorrect usage for argument order (CASSANDRA-13532)
 +Merged from 2.2:
+  * Fix compaction and flush exception not captured (CASSANDRA-13833)
 - * Make BatchlogManagerMBean.forceBatchlogReplay() blocking (CASSANDRA-13809)
   * Uncaught exceptions in Netty pipeline (CASSANDRA-13649)
 - * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067) 
 + * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067)
   * Fix queries with LIMIT and filtering on clustering columns 
(CASSANDRA-11223)
   * Fix potential NPE when resume bootstrap fails (CASSANDRA-13272)
   * Fix toJSONString for the UDT, tuple and collection types (CASSANDRA-13592)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/b0eba5f9/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --cc src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index 7251244,7e36e11..183176c
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@@ -837,12 -899,33 +837,12 @@@ public class ColumnFamilyStore implemen
  {
  synchronized (data)
  {
 -if (previousFlushFailure != null)
 -throw 

[10/10] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

[marcuse]

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c8d15f04
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c8d15f04
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c8d15f04

Branch: refs/heads/trunk
Commit: c8d15f04f1efd37668e2ccbc681730ae6b2199da
Parents: e5f3bb6 37d6730
Author: Marcus Eriksson 
Authored: Mon Sep 4 15:04:18 2017 +0200
Committer: Marcus Eriksson 
Committed: Mon Sep 4 15:04:18 2017 +0200

--
 CHANGES.txt |   1 +
 .../db/compaction/CompactionManager.java|   4 +-
 .../db/compaction/CompactionExecutorTest.java   | 109 +++
 3 files changed, 112 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/c8d15f04/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c8d15f04/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[01/10] cassandra git commit: Fix compaction and flush exception not captured issue

[marcuse]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.2 4d90573c5 -> e80ede6d3
  refs/heads/cassandra-3.0 f791c2690 -> b0eba5f9c
  refs/heads/cassandra-3.11 bed7fa5ef -> 37d67306a
  refs/heads/trunk e5f3bb6e5 -> c8d15f04f


Fix compaction and flush exception not captured issue

patch by Jay Zhuang; reviewed by marcuse for CASSANDRA-13833


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e80ede6d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e80ede6d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e80ede6d

Branch: refs/heads/cassandra-2.2
Commit: e80ede6d393460f22ee2b313d4bac7e3fbbfe893
Parents: 4d90573
Author: Jay Zhuang 
Authored: Thu Aug 31 11:07:07 2017 -0700
Committer: Marcus Eriksson 
Committed: Mon Sep 4 15:01:02 2017 +0200

--
 CHANGES.txt |   1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  |   4 +-
 .../db/compaction/CompactionManager.java|   4 +-
 .../db/compaction/CompactionExecutorTest.java   | 131 +++
 4 files changed, 136 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/e80ede6d/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 4e68ddc..03a78fd 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.11
+ * Fix compaction and flush exception not captured (CASSANDRA-13833)
  * Make BatchlogManagerMBean.forceBatchlogReplay() blocking (CASSANDRA-13809)
  * Uncaught exceptions in Netty pipeline (CASSANDRA-13649)
  * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067) 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/e80ede6d/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index 2e52eb2..7e36e11 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -906,9 +906,9 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 logFlush();
 Flush flush = new Flush(false);
 ListenableFutureTask flushTask = 
ListenableFutureTask.create(flush, null);
-flushExecutor.submit(flushTask);
+flushExecutor.execute(flushTask);
 ListenableFutureTask task = 
ListenableFutureTask.create(flush.postFlush);
-postFlushExecutor.submit(task);
+postFlushExecutor.execute(task);
 
 @SuppressWarnings("unchecked")
 ListenableFuture future = 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/e80ede6d/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java 
b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
index d21f1e8..cd50646 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
@@ -1457,7 +1457,7 @@ public class CompactionManager implements 
CompactionManagerMBean
 return CompactionMetrics.getCompactions().size();
 }
 
-private static class CompactionExecutor extends 
JMXEnabledThreadPoolExecutor
+static class CompactionExecutor extends JMXEnabledThreadPoolExecutor
 {
 protected CompactionExecutor(int minThreads, int maxThreads, String 
name, BlockingQueue queue)
 {
@@ -1537,7 +1537,7 @@ public class CompactionManager implements 
CompactionManagerMBean
 try
 {
 ListenableFutureTask ret = ListenableFutureTask.create(task);
-submit(ret);
+execute(ret);
 return ret;
 }
 catch (RejectedExecutionException ex)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/e80ede6d/test/unit/org/apache/cassandra/db/compaction/CompactionExecutorTest.java
--
diff --git 
a/test/unit/org/apache/cassandra/db/compaction/CompactionExecutorTest.java 
b/test/unit/org/apache/cassandra/db/compaction/CompactionExecutorTest.java
new file mode 100644
index 000..c6feb3f
--- /dev/null
+++ b/test/unit/org/apache/cassandra/db/compaction/CompactionExecutorTest.java
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional 

[jira] [Commented] (CASSANDRA-8457) nio MessagingService

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152574#comment-16152574
 ] 

Stefan Podkowinski commented on CASSANDRA-8457:
---

[~jasobrown], did you forget to bump netty to 4.1.14 in build.xml?

> nio MessagingService
> 
>
> Key: CASSANDRA-8457
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8457
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Jonathan Ellis
>Assignee: Jason Brown
>Priority: Minor
>  Labels: netty, performance
> Fix For: 4.0
>
> Attachments: 8457-load.tgz
>
>
> Thread-per-peer (actually two each incoming and outbound) is a big 
> contributor to context switching, especially for larger clusters.  Let's look 
> at switching to nio, possibly via Netty.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13396) Cassandra 3.10: ClassCastException in ThreadAwareSecurityManager

[Eric Hubert (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152569#comment-16152569
 ] 

Eric Hubert commented on CASSANDRA-13396:
-

We faced the same underlying issue after upgrading from Cassandra 3.9 to 3.11.0 
when using Cassandra embedded for integration testing using JUnit. 
As our application uses a different logging backend and we did not want to 
switch it and provide appropriate redundant configuration for logback, we 
excluded logback dependencies and only provided our implementation to also 
avoid any warnings about duplicate bindings. This setup worked fine with 
Cassandra 3.9, but fails with Cassandra >= 3.10; the server does not startup, 
because of the missing classes. So in this case any patch working with 
instanceof checks still attempting to load those classes without specific 
try/catch would obviously also fail. 

In addition to SMAwareReconfigureOnChangeFilter in 
org.apache.cassandra.cql3.functions.ThreadAwareSecurityManager.install() using 
multiple logback internals (added with CASSANDRA-12535) I also found the change 
with CASSANDRA-12509 adding ch.qos.logback.core.hook.DelayingShutdownHook in 
StorageService#initServer problematic.
Would it be an alternative to handle all access to the underlying logging 
implementation via reflection? 
E.g. attempting to load logback classes and only if this does not fail, perform 
implementation specific actions via reflection (otherwise log a warning about 
missing logback presence, which can be ignored in integration test setups). We 
are mostly talking about one-time initialization, so the performance impact 
should be really negligible.
This solution would require users to properly exclude logback logging libs if 
they want to use other sf4j implementation bindings. Providing multiple logging 
implementations with sl4fj bindings anyway triggers a warning which should be 
handled.

> Cassandra 3.10: ClassCastException in ThreadAwareSecurityManager
> 
>
> Key: CASSANDRA-13396
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13396
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Edward Capriolo
>Assignee: Eugene Fedotov
>Priority: Minor
>
> https://www.mail-archive.com/user@cassandra.apache.org/msg51603.html



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13833) Failed compaction is not captured

[Marcus Eriksson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152553#comment-16152553
 ] 

Marcus Eriksson commented on CASSANDRA-13833:
-

I reran the failures locally and the ones that looked suspicious all pass 
(except for trunk which looks completely broken now)

I'll get it committed


> Failed compaction is not captured
> -
>
> Key: CASSANDRA-13833
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13833
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>
> Follow up for CASSANDRA-13785, when the compaction failed, it fails silently. 
> No error message is logged and exceptions metric is not updated. Basically, 
> it's unable to get the exception: 
> [CompactionManager.java:1491|https://github.com/apache/cassandra/blob/cassandra-2.2/src/java/org/apache/cassandra/db/compaction/CompactionManager.java#L1491]
> Here is the call stack:
> {noformat}
> at 
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:195)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at 
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:89)
> at 
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at 
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at 
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> There're 2 {{FutureTask}} in the call stack, for example 
> {{FutureTask1(FutureTask2))}}, If the call thrown an exception, 
> {{FutureTask2}} sets the status, save the exception and return. But 
> FutureTask1 doesn't get any exception, then set the status to normal. So 
> we're unable to get the exception in:
> [CompactionManager.java:1491|https://github.com/apache/cassandra/blob/cassandra-2.2/src/java/org/apache/cassandra/db/compaction/CompactionManager.java#L1491]
> 2.1.x is working fine, here is the call stack:
> {noformat}
> at 
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) 
> ~[main/:na]
> at 
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:73)
>  ~[main/:na]
> at 
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
>  ~[main/:na]
> at 
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264)
>  ~[main/:na]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> ~[na:1.8.0_141]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> ~[na:1.8.0_141]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  ~[na:1.8.0_141]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  [na:1.8.0_141]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_141]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13662) Remove unsupported CREDENTIALS message

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152525#comment-16152525
 ] 

Stefan Podkowinski commented on CASSANDRA-13662:


Thanks for taking a look at this. When it comes to unit testing, we could 
create a test to verify the correct opcode > UnsupportedMessageCodec mapping, 
which seems to be kinda silly. Same with writing a unit test for 
UnsupportedMessageCodec, which always just throws an exception for both 
methods. Writing a integration test would require to make the driver send 
CREDENTIAL, which I think would be non-trivial for this type of low-level 
operation, but I'm not a driver expert. 

> Remove unsupported CREDENTIALS message
> --
>
> Key: CASSANDRA-13662
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13662
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Auth, CQL
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Minor
>  Labels: security
> Fix For: 4.x
>
>
> Remove CREDENTIAL message, as protocol v1 isn't supported anyways. Let's try 
> not to keep unused legacy classes around for any security relevant features. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13815) RPM package for client tools - cqlsh + nodetool

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-13815:
---
Fix Version/s: (was: 3.11.x)
   (was: 3.10)
   4.x

> RPM package for client tools - cqlsh + nodetool
> ---
>
> Key: CASSANDRA-13815
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13815
> Project: Cassandra
>  Issue Type: Wish
>  Components: Packaging
>Reporter: Dennis
> Fix For: 4.x
>
>
> Feature request. 
> I see you guys are picking up on the RPM packages. 
> Thanks for that. That could even be improved if you could package the client 
> tools as a separate  or client-only package as well. That package could hold 
> cqlsh and nodetool for example. 
> That would support centralized, automated backup or other maintenance 
> processes. 
> Now the admin is forced to login to the box in order to use these tools, 
> which is not really best practice, security wise. The admin would need to 
> know an ssh account as well as the cassandra admin account. 
> So, benefits or usage of a client package (cqlsh+nodetool): 
> # Supports automated maintenance scripts (simply yum the client tools to a 
> temporary vm)
> # Better security, as the admin doesn't need to ssh into the instance host.
> Without having to pull the full Cassandra packages on the clients.
> Datastax does have such client packages, but they don't support the community 
> edition anymore, so I am hoping that you can do this going forward.
> Thanks! 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13692) CompactionAwareWriter_getWriteDirectory throws incompatible exceptions

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152476#comment-16152476
 ] 

Aleksey Yeschenko commented on CASSANDRA-13692:
---

That would explain it. It's a regression test - it's supposed to fail on all 
3.0/3.11/trunk branches that don't have CASSANDRA-13747 fix in them.

> CompactionAwareWriter_getWriteDirectory throws incompatible exceptions
> --
>
> Key: CASSANDRA-13692
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13692
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Hao Zhong
>Assignee: Dimitar Dimitrov
>  Labels: lhf
> Attachments: c13692-2.2-dtest-results.PNG, 
> c13692-2.2-testall-results.PNG, c13692-3.0-dtest-results.PNG, 
> c13692-3.0-testall-results.PNG, c13692-3.11-dtest-results.PNG, 
> c13692-3.11-testall-results.PNG, c13692-dtest-results.PNG, 
> c13692-testall-results.PNG
>
>
> The CompactionAwareWriter_getWriteDirectory throws RuntimeException:
> {code}
> public Directories.DataDirectory getWriteDirectory(Iterable 
> sstables, long estimatedWriteSize)
> {
> File directory = null;
> for (SSTableReader sstable : sstables)
> {
> if (directory == null)
> directory = sstable.descriptor.directory;
> if (!directory.equals(sstable.descriptor.directory))
> {
> logger.trace("All sstables not from the same disk - putting 
> results in {}", directory);
> break;
> }
> }
> Directories.DataDirectory d = 
> getDirectories().getDataDirectoryForFile(directory);
> if (d != null)
> {
> long availableSpace = d.getAvailableSpace();
> if (availableSpace < estimatedWriteSize)
> throw new RuntimeException(String.format("Not enough space to 
> write %s to %s (%s available)",
>  
> FBUtilities.prettyPrintMemory(estimatedWriteSize),
>  d.location,
>  
> FBUtilities.prettyPrintMemory(availableSpace)));
> logger.trace("putting compaction results in {}", directory);
> return d;
> }
> d = getDirectories().getWriteableLocation(estimatedWriteSize);
> if (d == null)
> throw new RuntimeException(String.format("Not enough disk space 
> to store %s",
>  
> FBUtilities.prettyPrintMemory(estimatedWriteSize)));
> return d;
> }
> {code}
> However, the thrown exception does not  trigger the failure policy. 
> CASSANDRA-11448 fixed a similar problem. The buggy code is:
> {code}
> protected Directories.DataDirectory getWriteDirectory(long writeSize)
> {
> Directories.DataDirectory directory = 
> getDirectories().getWriteableLocation(writeSize);
> if (directory == null)
> throw new RuntimeException("Insufficient disk space to write " + 
> writeSize + " bytes");
> return directory;
> }
> {code}
> The fixed code is:
> {code}
> protected Directories.DataDirectory getWriteDirectory(long writeSize)
> {
> Directories.DataDirectory directory = 
> getDirectories().getWriteableLocation(writeSize);
> if (directory == null)
> throw new FSWriteError(new IOException("Insufficient disk space 
> to write " + writeSize + " bytes"), "");
> return directory;
> }
> {code}
> The fixed code throws FSWE and triggers the failure policy.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-13841) Allow specific sources during rebuild

[Kurt Greaves (JIRA)]

Kurt Greaves created CASSANDRA-13841:


 Summary: Allow specific sources during rebuild
 Key: CASSANDRA-13841
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13841
 Project: Cassandra
  Issue Type: Bug
Reporter: Kurt Greaves
Assignee: Kurt Greaves
Priority: Minor


CASSANDRA-10406 introduced the ability to rebuild specific ranges, and 
CASSANDRA-9875 extended that to allow specifying a set of hosts to stream from. 
It's not incredibly clear why you would only want to stream a subset of ranges, 
but a possible use case for this functionality is to rebuild a node from 
targeted replicas. 

When doing a DC migration, if you are using racks==RF while rebuilding you can 
ensure you rebuild from each copy of a replica in the source datacenter by 
specifying all the hosts from a single rack to rebuild a single copy from. This 
can be repeated for each rack in the new datacenter to ensure you have each 
copy of the replica from the source DC, and thus maintaining consistency 
through rebuilds. 

For example, with the following topology for DC A and B with an RF of A:3 and 
B:3
||A   ||B||
||Node||Rack||Node||Rack||
|A1|rack1| B1|rack1|
|A2|rack2| B2|rack2|
|A3|rack3| B3|rack3|
The following set of actions will result in having exactly 1 copy of every 
replica in A in B, and B will be _at least_ as consistent as A.
Rebuild B1 from only A1
Rebuild B2 from only A2
Rebuild B3 from only A3

Unfortunately using this functionality is non-trivial at the moment, as you can 
only specify specific sources WITH the nodes set of tokens to rebuild from. To 
perform the above with vnodes/a large cluster, you will have to specify every 
token range in the -ts arg, which quickly gets unwieldy/impossible if you have 
a large cluster.

A solution to this is to simply filter on sources first, before processing 
ranges.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-13043) UnavailabeException caused by counter writes forwarded to leaders without complete cluster view

[Stefano Ortolani (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149262#comment-16149262
 ] 

Stefano Ortolani edited comment on CASSANDRA-13043 at 9/4/17 10:58 AM:
---

Some updates: added to ccm the ability to send a byteman rule when restarting a 
node (https://github.com/ostefano/ccm/tree/startup_byteman). Originally it was 
only possible to submit a rule _after_ the node started, which made reproducing 
this race quite impossible.

With that I could finally reproduce the bug. Here you can my branch with a new 
DTEST exercising the bug: 
https://github.com/ostefano/cassandra-dtest/tree/CASSANDRA-13043

Attached you can find a patch fixing the bug by removing nodes that are not RPC 
ready from a leader election. Also, I fixed the corner case where the CL 
required is local and the current DC does not have available nodes.

Here you can find the commit: 
https://github.com/ostefano/cassandra/commit/91cc9b4398009a3cee3004bc11a047c056fda6a6

update: unit tests are passing


was (Author: ostefano):
Some updates: added to ccm the ability to send a byteman rule when restarting a 
node (https://github.com/ostefano/ccm/tree/startup_byteman). Originally it was 
only possible to submit a rule _after_ the node started, which made reproducing 
this race quite impossible.

With that I could finally reproduce the bug. Here you can my branch with a new 
DTEST exercising the bug: 
https://github.com/ostefano/cassandra-dtest/tree/CASSANDRA-13043

Attached you can find a patch fixing the bug by removing nodes that are not RPC 
ready from a leader election. Also, I fixed the corner case where the CL 
required is local and the current DC does not have available nodes.

Here you can find the commit: 
https://github.com/ostefano/cassandra/commit/91cc9b4398009a3cee3004bc11a047c056fda6a6

> UnavailabeException caused by counter writes forwarded to leaders without 
> complete cluster view
> ---
>
> Key: CASSANDRA-13043
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13043
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: Debian
>Reporter: Catalin Alexandru Zamfir
> Attachments: patch.diff
>
>
> In version 3.9 of Cassandra, we get the following exceptions on the 
> system.log whenever booting an agent. They seem to grow in number with each 
> reboot. Any idea where they come from or what can we do about them? Note that 
> the cluster is healthy (has sufficient live nodes).
> {noformat}
> 2/14/2016 12:39:47 PMINFO  10:39:47 Updating topology for /10.136.64.120
> 12/14/2016 12:39:47 PMINFO  10:39:47 Updating topology for /10.136.64.120
> 12/14/2016 12:39:47 PMWARN  10:39:47 Uncaught exception on thread 
> Thread[CounterMutationStage-111,5,main]: {}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException: 
> Cannot achieve consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) 
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> ~[na:1.8.0_111]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
>  [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) 
> [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat java.lang.Thread.run(Thread.java:745) 
> [na:1.8.0_111]
> 12/14/2016 12:39:47 PMWARN  10:39:47 Uncaught exception on thread 
> 

[jira] [Comment Edited] (CASSANDRA-13043) UnavailabeException caused by counter writes forwarded to leaders without complete cluster view

[Stefano Ortolani (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149262#comment-16149262
 ] 

Stefano Ortolani edited comment on CASSANDRA-13043 at 9/4/17 10:44 AM:
---

Some updates: added to ccm the ability to send a byteman rule when restarting a 
node (https://github.com/ostefano/ccm/tree/startup_byteman). Originally it was 
only possible to submit a rule _after_ the node started, which made reproducing 
this race quite impossible.

With that I could finally reproduce the bug. Here you can my branch with a new 
DTEST exercising the bug: 
https://github.com/ostefano/cassandra-dtest/tree/CASSANDRA-13043

Attached you can find a patch fixing the bug by removing nodes that are not RPC 
ready from a leader election. Also, I fixed the corner case where the CL 
required is local and the current DC does not have available nodes.

Here you can find the commit: 
https://github.com/ostefano/cassandra/commit/91cc9b4398009a3cee3004bc11a047c056fda6a6


was (Author: ostefano):
Some updates: added to ccm the ability to send a byteman rule when restarting a 
node (https://github.com/ostefano/ccm/tree/startup_byteman). 

This allowed me to finally reproduce the bug: 
https://github.com/ostefano/cassandra-dtest/tree/CASSANDRA-13043

Attached a patch that try to fix it by removing from a leader election nodes 
that are not RPC ready.
Also, fixed corner case where the CL required is local.

> UnavailabeException caused by counter writes forwarded to leaders without 
> complete cluster view
> ---
>
> Key: CASSANDRA-13043
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13043
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: Debian
>Reporter: Catalin Alexandru Zamfir
> Attachments: patch.diff
>
>
> In version 3.9 of Cassandra, we get the following exceptions on the 
> system.log whenever booting an agent. They seem to grow in number with each 
> reboot. Any idea where they come from or what can we do about them? Note that 
> the cluster is healthy (has sufficient live nodes).
> {noformat}
> 2/14/2016 12:39:47 PMINFO  10:39:47 Updating topology for /10.136.64.120
> 12/14/2016 12:39:47 PMINFO  10:39:47 Updating topology for /10.136.64.120
> 12/14/2016 12:39:47 PMWARN  10:39:47 Uncaught exception on thread 
> Thread[CounterMutationStage-111,5,main]: {}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException: 
> Cannot achieve consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) 
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> ~[na:1.8.0_111]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
>  [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) 
> [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat java.lang.Thread.run(Thread.java:745) 
> [na:1.8.0_111]
> 12/14/2016 12:39:47 PMWARN  10:39:47 Uncaught exception on thread 
> Thread[CounterMutationStage-118,5,main]: {}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException: 
> Cannot achieve consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PMat 
> org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
>  ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat 
> 

[jira] [Updated] (CASSANDRA-13841) Allow specific sources during rebuild

[Kurt Greaves (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Greaves updated CASSANDRA-13841:
-
Status: Patch Available  (was: Open)

> Allow specific sources during rebuild
> -
>
> Key: CASSANDRA-13841
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13841
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Minor
>
> CASSANDRA-10406 introduced the ability to rebuild specific ranges, and 
> CASSANDRA-9875 extended that to allow specifying a set of hosts to stream 
> from. It's not incredibly clear why you would only want to stream a subset of 
> ranges, but a possible use case for this functionality is to rebuild a node 
> from targeted replicas. 
> When doing a DC migration, if you are using racks==RF while rebuilding you 
> can ensure you rebuild from each copy of a replica in the source datacenter 
> by specifying all the hosts from a single rack to rebuild a single copy from. 
> This can be repeated for each rack in the new datacenter to ensure you have 
> each copy of the replica from the source DC, and thus maintaining consistency 
> through rebuilds. 
> For example, with the following topology for DC A and B with an RF of A:3 and 
> B:3
> ||A   ||B||
> ||Node||Rack||Node||Rack||
> |A1|rack1| B1|rack1|
> |A2|rack2| B2|rack2|
> |A3|rack3| B3|rack3|
> The following set of actions will result in having exactly 1 copy of every 
> replica in A in B, and B will be _at least_ as consistent as A.
> {code:java}
> Rebuild B1 from only A1
> Rebuild B2 from only A2
> Rebuild B3 from only A3
> {code}
> Unfortunately using this functionality is non-trivial at the moment, as you 
> can only specify specific sources WITH the nodes set of tokens to rebuild 
> from. To perform the above with vnodes/a large cluster, you will have to 
> specify every token range in the -ts arg, which quickly gets 
> unwieldy/impossible if you have a large cluster.
> A solution to this is to simply filter on sources first, before processing 
> ranges.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13841) Allow specific sources during rebuild

[Kurt Greaves (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152455#comment-16152455
 ] 

Kurt Greaves commented on CASSANDRA-13841:
--

Patch for trunk here, however I'd like to get some opinions on fixing this in 
earlier versions as well. The patch for 3.11 is more or less the same and I 
don't see any reason why we couldn't apply this there either. More importantly 
I'd like to hear peoples thoughts on pushing this back to earlier versions, as 
it's a pretty effective way for consistent rebuilds, and thus "avoids" the need 
for a repair post rebuild (which may not always be possible). I've already 
backported it to 3.0, 2.2, and 2.1 and haven't encountered any issues in 
testing. Regardless we will probably backport it anyway to assist in some 
not-so-nice migrations that come up, but I figure if we find it useful 
potentially someone else would as well.
[trunk|https://github.com/apache/cassandra/compare/trunk...kgreav:rebuild-sources]

I haven't really touched the existing tests for this as I wanted to get 
opinions first, but this shouldn't break existing tests, and new tests should 
be relatively straightforward to write.

> Allow specific sources during rebuild
> -
>
> Key: CASSANDRA-13841
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13841
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Minor
>
> CASSANDRA-10406 introduced the ability to rebuild specific ranges, and 
> CASSANDRA-9875 extended that to allow specifying a set of hosts to stream 
> from. It's not incredibly clear why you would only want to stream a subset of 
> ranges, but a possible use case for this functionality is to rebuild a node 
> from targeted replicas. 
> When doing a DC migration, if you are using racks==RF while rebuilding you 
> can ensure you rebuild from each copy of a replica in the source datacenter 
> by specifying all the hosts from a single rack to rebuild a single copy from. 
> This can be repeated for each rack in the new datacenter to ensure you have 
> each copy of the replica from the source DC, and thus maintaining consistency 
> through rebuilds. 
> For example, with the following topology for DC A and B with an RF of A:3 and 
> B:3
> ||A   ||B||
> ||Node||Rack||Node||Rack||
> |A1|rack1| B1|rack1|
> |A2|rack2| B2|rack2|
> |A3|rack3| B3|rack3|
> The following set of actions will result in having exactly 1 copy of every 
> replica in A in B, and B will be _at least_ as consistent as A.
> {code:java}
> Rebuild B1 from only A1
> Rebuild B2 from only A2
> Rebuild B3 from only A3
> {code}
> Unfortunately using this functionality is non-trivial at the moment, as you 
> can only specify specific sources WITH the nodes set of tokens to rebuild 
> from. To perform the above with vnodes/a large cluster, you will have to 
> specify every token range in the -ts arg, which quickly gets 
> unwieldy/impossible if you have a large cluster.
> A solution to this is to simply filter on sources first, before processing 
> ranges.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13841) Allow specific sources during rebuild

[Kurt Greaves (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Greaves updated CASSANDRA-13841:
-
Description: 
CASSANDRA-10406 introduced the ability to rebuild specific ranges, and 
CASSANDRA-9875 extended that to allow specifying a set of hosts to stream from. 
It's not incredibly clear why you would only want to stream a subset of ranges, 
but a possible use case for this functionality is to rebuild a node from 
targeted replicas. 

When doing a DC migration, if you are using racks==RF while rebuilding you can 
ensure you rebuild from each copy of a replica in the source datacenter by 
specifying all the hosts from a single rack to rebuild a single copy from. This 
can be repeated for each rack in the new datacenter to ensure you have each 
copy of the replica from the source DC, and thus maintaining consistency 
through rebuilds. 

For example, with the following topology for DC A and B with an RF of A:3 and 
B:3
||A   ||B||
||Node||Rack||Node||Rack||
|A1|rack1| B1|rack1|
|A2|rack2| B2|rack2|
|A3|rack3| B3|rack3|
The following set of actions will result in having exactly 1 copy of every 
replica in A in B, and B will be _at least_ as consistent as A.

{code:java}
Rebuild B1 from only A1
Rebuild B2 from only A2
Rebuild B3 from only A3
{code}

Unfortunately using this functionality is non-trivial at the moment, as you can 
only specify specific sources WITH the nodes set of tokens to rebuild from. To 
perform the above with vnodes/a large cluster, you will have to specify every 
token range in the -ts arg, which quickly gets unwieldy/impossible if you have 
a large cluster.

A solution to this is to simply filter on sources first, before processing 
ranges.

  was:
CASSANDRA-10406 introduced the ability to rebuild specific ranges, and 
CASSANDRA-9875 extended that to allow specifying a set of hosts to stream from. 
It's not incredibly clear why you would only want to stream a subset of ranges, 
but a possible use case for this functionality is to rebuild a node from 
targeted replicas. 

When doing a DC migration, if you are using racks==RF while rebuilding you can 
ensure you rebuild from each copy of a replica in the source datacenter by 
specifying all the hosts from a single rack to rebuild a single copy from. This 
can be repeated for each rack in the new datacenter to ensure you have each 
copy of the replica from the source DC, and thus maintaining consistency 
through rebuilds. 

For example, with the following topology for DC A and B with an RF of A:3 and 
B:3
||A   ||B||
||Node||Rack||Node||Rack||
|A1|rack1| B1|rack1|
|A2|rack2| B2|rack2|
|A3|rack3| B3|rack3|
The following set of actions will result in having exactly 1 copy of every 
replica in A in B, and B will be _at least_ as consistent as A.
Rebuild B1 from only A1
Rebuild B2 from only A2
Rebuild B3 from only A3

Unfortunately using this functionality is non-trivial at the moment, as you can 
only specify specific sources WITH the nodes set of tokens to rebuild from. To 
perform the above with vnodes/a large cluster, you will have to specify every 
token range in the -ts arg, which quickly gets unwieldy/impossible if you have 
a large cluster.

A solution to this is to simply filter on sources first, before processing 
ranges.


> Allow specific sources during rebuild
> -
>
> Key: CASSANDRA-13841
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13841
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Minor
>
> CASSANDRA-10406 introduced the ability to rebuild specific ranges, and 
> CASSANDRA-9875 extended that to allow specifying a set of hosts to stream 
> from. It's not incredibly clear why you would only want to stream a subset of 
> ranges, but a possible use case for this functionality is to rebuild a node 
> from targeted replicas. 
> When doing a DC migration, if you are using racks==RF while rebuilding you 
> can ensure you rebuild from each copy of a replica in the source datacenter 
> by specifying all the hosts from a single rack to rebuild a single copy from. 
> This can be repeated for each rack in the new datacenter to ensure you have 
> each copy of the replica from the source DC, and thus maintaining consistency 
> through rebuilds. 
> For example, with the following topology for DC A and B with an RF of A:3 and 
> B:3
> ||A   ||B||
> ||Node||Rack||Node||Rack||
> |A1|rack1| B1|rack1|
> |A2|rack2| B2|rack2|
> |A3|rack3| B3|rack3|
> The following set of actions will result in having exactly 1 copy of every 
> replica in A in B, and B will be _at least_ as consistent as A.
> {code:java}
> Rebuild B1 from 

[jira] [Commented] (CASSANDRA-13692) CompactionAwareWriter_getWriteDirectory throws incompatible exceptions

[Dimitar Dimitrov (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152425#comment-16152425
 ] 

Dimitar Dimitrov commented on CASSANDRA-13692:
--

All observed {{dtest}} failures, including the 
consistency_test.TestConsistency.test_13747 failures, reproduce exactly the 
same on brand new copies of the cassandra-3.11 and trunk branches of my 
apache/cassandra fork.
My fork is by now tens of commits behind the origin, so I'll update the fork, 
and re-run the CI jobs for the 3.0, 3.11, and trunk branches.
I'm expecting to see a much more consistent picture this time.

> CompactionAwareWriter_getWriteDirectory throws incompatible exceptions
> --
>
> Key: CASSANDRA-13692
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13692
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Hao Zhong
>Assignee: Dimitar Dimitrov
>  Labels: lhf
> Attachments: c13692-2.2-dtest-results.PNG, 
> c13692-2.2-testall-results.PNG, c13692-3.0-dtest-results.PNG, 
> c13692-3.0-testall-results.PNG, c13692-3.11-dtest-results.PNG, 
> c13692-3.11-testall-results.PNG, c13692-dtest-results.PNG, 
> c13692-testall-results.PNG
>
>
> The CompactionAwareWriter_getWriteDirectory throws RuntimeException:
> {code}
> public Directories.DataDirectory getWriteDirectory(Iterable 
> sstables, long estimatedWriteSize)
> {
> File directory = null;
> for (SSTableReader sstable : sstables)
> {
> if (directory == null)
> directory = sstable.descriptor.directory;
> if (!directory.equals(sstable.descriptor.directory))
> {
> logger.trace("All sstables not from the same disk - putting 
> results in {}", directory);
> break;
> }
> }
> Directories.DataDirectory d = 
> getDirectories().getDataDirectoryForFile(directory);
> if (d != null)
> {
> long availableSpace = d.getAvailableSpace();
> if (availableSpace < estimatedWriteSize)
> throw new RuntimeException(String.format("Not enough space to 
> write %s to %s (%s available)",
>  
> FBUtilities.prettyPrintMemory(estimatedWriteSize),
>  d.location,
>  
> FBUtilities.prettyPrintMemory(availableSpace)));
> logger.trace("putting compaction results in {}", directory);
> return d;
> }
> d = getDirectories().getWriteableLocation(estimatedWriteSize);
> if (d == null)
> throw new RuntimeException(String.format("Not enough disk space 
> to store %s",
>  
> FBUtilities.prettyPrintMemory(estimatedWriteSize)));
> return d;
> }
> {code}
> However, the thrown exception does not  trigger the failure policy. 
> CASSANDRA-11448 fixed a similar problem. The buggy code is:
> {code}
> protected Directories.DataDirectory getWriteDirectory(long writeSize)
> {
> Directories.DataDirectory directory = 
> getDirectories().getWriteableLocation(writeSize);
> if (directory == null)
> throw new RuntimeException("Insufficient disk space to write " + 
> writeSize + " bytes");
> return directory;
> }
> {code}
> The fixed code is:
> {code}
> protected Directories.DataDirectory getWriteDirectory(long writeSize)
> {
> Directories.DataDirectory directory = 
> getDirectories().getWriteableLocation(writeSize);
> if (directory == null)
> throw new FSWriteError(new IOException("Insufficient disk space 
> to write " + writeSize + " bytes"), "");
> return directory;
> }
> {code}
> The fixed code throws FSWE and triggers the failure policy.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-10540) RangeAwareCompaction

[Marcus Eriksson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-10540:

Status: Open  (was: Patch Available)

removing patch available - making a few changes (we should flush to separate 
sstables if we don't use vnodes - skip 'L0')

> RangeAwareCompaction
> 
>
> Key: CASSANDRA-10540
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10540
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>  Labels: compaction, lcs, vnodes
> Fix For: 4.x
>
>
> Broken out from CASSANDRA-6696, we should split sstables based on ranges 
> during compaction.
> Requirements;
> * dont create tiny sstables - keep them bunched together until a single vnode 
> is big enough (configurable how big that is)
> * make it possible to run existing compaction strategies on the per-range 
> sstables
> We should probably add a global compaction strategy parameter that states 
> whether this should be enabled or not.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org