[
https://issues.apache.org/jira/browse/CASSANDRA-9146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14487406#comment-14487406
]
Anuj commented on CASSANDRA-9146:
---------------------------------
Yes please analyze as we are facing it in 2.0.3.We may need this fix in 2.0
branch.Upgrading would take some time and scenario is not easily reproducible
but once it occurs in a cluster this burst of sstables grows with every
repair.We need to understand why this premature flushing is happenning? Whats
the work around till we upgrade? If its coldness issue which prevents auto
compaction then any advise to make all sstables hot ( we tried with reads but
without success)? Range scan doesnt contribute to hotness ..thats another open
issue I logged sometime back.
> Ever Growing Secondary Index sstables after every Repair
> --------------------------------------------------------
>
> Key: CASSANDRA-9146
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9146
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Anuj
>
> Cluster has reached a state where every "repair -pr" operation on CF results
> in numerous tiny sstables being flushed to disk. Most sstables are related to
> secondary indexes. Due to thousands of sstables, reads have started timing
> out. Even though compaction begins for one of the secondary index, sstable
> count after repair remains very high (thousands). Every repair adds thousands
> of sstables.
> Problems:
> 1. Why burst of tiny secondary index tables are flushed during repair ? What
> is triggering frequent/premature flush of secondary index sstable (more than
> hundred in every burst)? At max we see one ParNew GC pauses >200ms.
> 2. Why auto-compaction is not compacting all sstables. Is it related to
> coldness issue(CASSANDRA-8885) where compaction doesn't works even when
> cold_reads_to_omit=0 by default?
> If coldness is the issue, we are stuck in infinite loop: reads will
> trigger compaction but reads timeout as sstable count is in thousands
> 3. What's the way out if we face this issue in Prod?
> Is this issue fixed in latest production release 2.0.13? Issue looks similar
> to CASSANDRA-8641, but the issue is fixed in only 2.1.3. I think it should be
> fixed in 2.0 branch too.
> Configuration:
> Compaction Strategy: STCS
> memtable_flush_writers=4
> memtable_flush_queue_size=4
> in_memory_compaction_limit_in_mb=32
> concurrent_compactors=12
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
