[
https://issues.apache.org/jira/browse/CASSANDRA-12764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15562363#comment-15562363
]
Tom van der Woerdt commented on CASSANDRA-12764:
------------------------------------------------
Yesterday on IRC it was suggested to try a major compaction. Sadly, that
would've taken 20 days. :) It was at 0.21% when I aborted.
I ended up going for a solution along the lines of "move all data files out,
then in batches of 500 sstables move them back in, run 'nodetool refresh', wait
for compaction, then move the next batch".
I filed the ticket here so that hopefully compactions can speed up in the
future, so that if someone has the same kind of issue, they won't run into
these performance issues.
> Compaction performance issues with many sstables, during transaction commit
> phase
> ---------------------------------------------------------------------------------
>
> Key: CASSANDRA-12764
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12764
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Reporter: Tom van der Woerdt
>
> An issue with a script flooded my cluster with sstables. There is now a table
> with 100k sstables, all on the order of KBytes, and it's taking a long time
> (ETA 20 days) to compact, even though the table is only ~30GB.
> Stack trace :
> {noformat}
> "CompactionExecutor:308" #7541 daemon prio=1 os_prio=4 tid=0x00007fa22af35400
> nid=0x41eb runnable [0x00007fdbea48d000]
> java.lang.Thread.State: RUNNABLE
> at java.util.TimSort.countRunAndMakeAscending(TimSort.java:360)
> at java.util.TimSort.sort(TimSort.java:220)
> at java.util.Arrays.sort(Arrays.java:1438)
> at com.google.common.collect.Ordering.sortedCopy(Ordering.java:817)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:209)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:211)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:211)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:211)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:211)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:211)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:211)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:211)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:210)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:210)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:210)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:210)
> at org.apache.cassandra.utils.IntervalTree.<init>(IntervalTree.java:50)
> at
> org.apache.cassandra.db.lifecycle.SSTableIntervalTree.<init>(SSTableIntervalTree.java:40)
> at
> org.apache.cassandra.db.lifecycle.SSTableIntervalTree.build(SSTableIntervalTree.java:50)
> at org.apache.cassandra.db.lifecycle.View$4.apply(View.java:288)
> at org.apache.cassandra.db.lifecycle.View$4.apply(View.java:283)
> at
> com.google.common.base.Functions$FunctionComposition.apply(Functions.java:216)
> at org.apache.cassandra.db.lifecycle.Tracker.apply(Tracker.java:128)
> at org.apache.cassandra.db.lifecycle.Tracker.apply(Tracker.java:101)
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.checkpoint(LifecycleTransaction.java:307)
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.checkpoint(LifecycleTransaction.java:288)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:368)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:84)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:94)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:194)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:263)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> IntervalTree shows in a lot of stack traces I've taken on several nodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
