[
https://issues.apache.org/jira/browse/CASSANDRA-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis resolved CASSANDRA-5318.
---------------------------------------
Resolution: Duplicate
Duplicates CASSANDRA-1483. TLDR it's harmless.
> Exception in CompactionExecutor thread after nodetool drain
> -----------------------------------------------------------
>
> Key: CASSANDRA-5318
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5318
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.2.2
> Environment: I am running cluster of 6 nodes in amazon(2 nodes per
> availability zone in eu-west region) using EC2snith, Cassandra 1.2.2 on
> ubuntu 12.04, m1.xlarge and 4 x 450 GB of ephemeral volumes stripped together
> on each node.
> Reporter: Michael Czerwiński
>
> I am doing batch inserts around 5K rows per batch, 400 writes per second.
> I am using leveled compaction and multithreaded_compaction enabled.
> After I do "nodetool drain" I see in the logs memtables being flushed after
> which I see exception in CompactionExecutor Thread.
> INFO 15:20:17,471 Completed flushing
> /mnt/cassandra/data/OpsCenter/pdps/OpsCenter-pdps-ib-60-Data.db (316475
> bytes) for commitlog position ReplayPosition(segmentId=1362668614841,
> position=26678466)
> INFO 15:20:17,472 Writing Memtable-rollups300@466350667(305256/9115571
> serialized/live bytes, 12719 ops)
> INFO 15:20:17,568 Completed flushing
> /mnt/cassandra/data/OpsCenter/rollups300/OpsCenter-rollups300-ib-60-Data.db
> (275536 bytes) for commitlog position ReplayPosition(segmentId=1362668614841,
> position=26678466)
> INFO 15:20:17,687 DRAINED
> ERROR 15:20:18,271 Exception in thread Thread[CompactionExecutor:15,1,RMI
> Runtime]
> java.util.concurrent.RejectedExecutionException
> at
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:215)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:397)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:470)
> at
> org.apache.cassandra.io.sstable.SSTableDeletingTask.schedule(SSTableDeletingTask.java:63)
> at
> org.apache.cassandra.io.sstable.SSTableReader.releaseReference(SSTableReader.java:902)
> at
> org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:361)
> at
> org.apache.cassandra.db.DataTracker.postReplace(DataTracker.java:330)
> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:324)
> at
> org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:227)
> at
> org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:997)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:231)
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:62)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:191)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR 15:20:21,102 Exception in thread Thread[CompactionExecutor:14,1,RMI
> Runtime]
> java.util.concurrent.RejectedExecutionException
> at
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:215)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:397)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:470)
> at
> org.apache.cassandra.io.sstable.SSTableDeletingTask.schedule(SSTableDeletingTask.java:63)
> at
> org.apache.cassandra.io.sstable.SSTableReader.releaseReference(SSTableReader.java:902)
> at
> org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:361)
> at
> org.apache.cassandra.db.DataTracker.postReplace(DataTracker.java:330)
> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:324)
> at
> org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:227)
> at
> org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:997)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:231)
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:62)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:191)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR 15:20:22,760 Exception in thread Thread[CompactionExecutor:13,1,RMI
> Runtime]
> java.util.concurrent.RejectedExecutionException
> at
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:215)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:397)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:470)
> at
> org.apache.cassandra.io.sstable.SSTableDeletingTask.schedule(SSTableDeletingTask.java:63)
> at
> org.apache.cassandra.io.sstable.SSTableReader.releaseReference(SSTableReader.java:902)
> at
> org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:361)
> at
> org.apache.cassandra.db.DataTracker.postReplace(DataTracker.java:330)
> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:324)
> at
> org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:227)
> at
> org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:997)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:231)
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:62)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:191)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR 15:20:24,587 Exception in thread Thread[CompactionExecutor:16,1,RMI
> Runtime]
> java.util.concurrent.RejectedExecutionException
> at
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:215)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:397)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:470)
> at
> org.apache.cassandra.io.sstable.SSTableDeletingTask.schedule(SSTableDeletingTask.java:63)
> at
> org.apache.cassandra.io.sstable.SSTableReader.releaseReference(SSTableReader.java:902)
> at
> org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:361)
> at
> org.apache.cassandra.db.DataTracker.postReplace(DataTracker.java:330)
> at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:324)
> at
> org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:227)
> at
> org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:997)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:231)
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:62)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:191)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> This is replicable for me under this load by simply running nodetool drain.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
