Arya Goudarzi created CASSANDRA-5119:
----------------------------------------
Summary: RejectedExecutionException During Compactions
Key: CASSANDRA-5119
URL: https://issues.apache.org/jira/browse/CASSANDRA-5119
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.6
Environment: Sun Java 1.6.0_37 64 Bit
Ubuntu 12.04
m1.xlarge EC2 instances
12 node cluster with 20Gb data on each node
Cassandra 1.1.6
Reporter: Arya Goudarzi
Hi, I frequently see the following ERROR and stack trace after
CompactionTask.java (line 109) has triggered a compaction job (These are
automatic minor compactions. We never trigger major compactions).
ERROR [CompactionExecutor:881] 2013-01-05 03:40:07,311
AbstractCassandraDaemon.java (line 135) Exception in thread
Thread[CompactionExecutor:881,1,RMI Runtime]
java.util.concurrent.RejectedExecutionException
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:215)
at
java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:397)
at
java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:470)
at
org.apache.cassandra.io.sstable.SSTableDeletingTask.schedule(SSTableDeletingTask.java:67)
at
org.apache.cassandra.io.sstable.SSTableReader.releaseReference(SSTableReader.java:826)
at
org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:358)
at org.apache.cassandra.db.DataTracker.postReplace(DataTracker.java:330)
at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:324)
at
org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:253)
at
org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:992)
at
org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:200)
at
org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:50)
at
org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:154)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Looking at the code, it seems the thread has been shutdown when it gets to
delayedExecute and isShutdown check would be true resulting a call to reject.
It is not clear to me why this happens. I am guessing the implication would be
the old SSTables would not get deleted which results in disk space usage to
keep going up. Restarting the nodes seem to make this go away for a while. I
could not find anything more useful in the logs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
