[
https://issues.apache.org/jira/browse/CASSANDRA-12764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15562395#comment-15562395
]
Benedict commented on CASSANDRA-12764:
--------------------------------------
bq. Sadly, that would've taken 20 days
Well, that sounds like a more pressing concern. These tickets shouldn't have
materially affected the ETA of a major compaction. It's possible it entered a
GC death spiral, as probably >6Gb of heap would have been needed for the reader
buffers alone (though it's been a while since I've checked our behaviours here;
it's possible they're lazily initialised and with tiny sstables they might not
all have been needed to be materialised at once).
Anyway, it sounds potentially worth filing a ticket to investigate that, as it
should be the get-out-of-jail free card for the problems you had.
Perhaps it's worth filing a ticket to explicitly make major compaction robust
to literally arbitrary numbers of files also. i.e. by performing multiple
rounds of "major compaction" if the file count is too high to do at once.
> Compaction performance issues with many sstables, during transaction commit
> phase
> ---------------------------------------------------------------------------------
>
> Key: CASSANDRA-12764
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12764
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Reporter: Tom van der Woerdt
>
> An issue with a script flooded my cluster with sstables. There is now a table
> with 100k sstables, all on the order of KBytes, and it's taking a long time
> (ETA 20 days) to compact, even though the table is only ~30GB.
> Stack trace :
> {noformat}
> "CompactionExecutor:308" #7541 daemon prio=1 os_prio=4 tid=0x00007fa22af35400
> nid=0x41eb runnable [0x00007fdbea48d000]
> java.lang.Thread.State: RUNNABLE
> at java.util.TimSort.countRunAndMakeAscending(TimSort.java:360)
> at java.util.TimSort.sort(TimSort.java:220)
> at java.util.Arrays.sort(Arrays.java:1438)
> at com.google.common.collect.Ordering.sortedCopy(Ordering.java:817)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:209)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:211)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:211)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:211)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:211)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:211)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:211)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:211)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:210)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:210)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:210)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:210)
> at org.apache.cassandra.utils.IntervalTree.<init>(IntervalTree.java:50)
> at
> org.apache.cassandra.db.lifecycle.SSTableIntervalTree.<init>(SSTableIntervalTree.java:40)
> at
> org.apache.cassandra.db.lifecycle.SSTableIntervalTree.build(SSTableIntervalTree.java:50)
> at org.apache.cassandra.db.lifecycle.View$4.apply(View.java:288)
> at org.apache.cassandra.db.lifecycle.View$4.apply(View.java:283)
> at
> com.google.common.base.Functions$FunctionComposition.apply(Functions.java:216)
> at org.apache.cassandra.db.lifecycle.Tracker.apply(Tracker.java:128)
> at org.apache.cassandra.db.lifecycle.Tracker.apply(Tracker.java:101)
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.checkpoint(LifecycleTransaction.java:307)
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.checkpoint(LifecycleTransaction.java:288)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:368)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:84)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:94)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:194)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:263)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> IntervalTree shows in a lot of stack traces I've taken on several nodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
