[
https://issues.apache.org/jira/browse/CASSANDRA-9661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14621576#comment-14621576
]
WeiFan commented on CASSANDRA-9661:
-----------------------------------
It seems that we can militate this kind of compaction by simply preventing DTCS
from submitting any tasks include only one SSTable. The patch is like this:
diff --git
a/src/java/org/apache/cassandra/db/compaction/DateTieredCompactionStrategy.java
b/src/java/org/apache/cassandra/db/compaction/DateTieredCompactionStrategy.java
index b7bb845..65769fc 100644
---
a/src/java/org/apache/cassandra/db/compaction/DateTieredCompactionStrategy.java
+++
b/src/java/org/apache/cassandra/db/compaction/DateTieredCompactionStrategy.java
@@ -86,6 +86,11 @@
logger.debug("Including expired sstables: {}", expired);
compactionCandidates.addAll(expired);
}
+
+ //issue 9661: never compact single SStable
+ if(compactionCandidates.size() == 1)
+ return Collections.emptyList();
+
return compactionCandidates;
}
> Endless compaction to a tiny, tombstoned SStable
> ------------------------------------------------
>
> Key: CASSANDRA-9661
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9661
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: WeiFan
> Assignee: Yuki Morishita
> Labels: compaction, dtcs
>
> We deployed a 3-nodes cluster (with 2.1.5) which worked under stable write
> requests ( about 2k wps) to a CF with DTCS, a default TTL as 43200s and
> gc_grace as 21600s. The CF contained inserted only, complete time series
> data. We found cassandra will occasionally keep writing logs like this:
> INFO [CompactionExecutor:30551] 2015-06-26 18:10:06,195
> CompactionTask.java:270 - Compacted 1 sstables to
> [/home/cassandra/workdata/data/sen_vaas_test/nodestatus-f96c7c50155811e589f69752ac9b06c7/sen_vaas_test-nodestatus-ka-2516270,].
> 449 bytes to 449 (~100% of original) in 12ms = 0.035683MB/s. 4 total
> partitions merged to 4. Partition merge counts were {1:4, }
> INFO [CompactionExecutor:30551] 2015-06-26 18:10:06,241
> CompactionTask.java:140 - Compacting
> [SSTableReader(path='/home/cassandra/workdata/data/sen_vaas_test/nodestatus-f96c7c50155811e589f69752ac9b06c7/sen_vaas_test-nodestatus-ka-2516270-Data.db')]
> INFO [CompactionExecutor:30551] 2015-06-26 18:10:06,253
> CompactionTask.java:270 - Compacted 1 sstables to
> [/home/cassandra/workdata/data/sen_vaas_test/nodestatus-f96c7c50155811e589f69752ac9b06c7/sen_vaas_test-nodestatus-ka-2516271,].
> 449 bytes to 449 (~100% of original) in 12ms = 0.035683MB/s. 4 total
> partitions merged to 4. Partition merge counts were {1:4, }
> It seems that cassandra kept doing compacting to a single SStable, serveral
> times per second, and lasted for many hours. Tons of logs were thrown and one
> CPU core exhausted during this time. The endless compacting finally end when
> another compaction started with a group of SStables (including previous one).
> All of our 3 nodes have been hit by this problem, but occurred in different
> time.
> We could not figure out how the problematic SStable come up because the log
> has wrapped around.
> We have dumped the records in the SStable and found it has the oldest data in
> our CF (again, our data was time series), and all of the record in this
> SStable have bben expired for more than 18 hours (12 hrs TTL + 6 hrs gc) so
> they should be dropped. However, c* do nothing to this SStable but compacting
> it again and again, until more SStable were out-dated enough to be considered
> for compacting together with this one by DTCS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
