On 19 Nov 2014, at 11:37, André Cruz <[email protected]> wrote:
>
> All the nodes were restarted on 21-23 October, for the upgrade (1.2.16 ->
> 1.2.19) I mentioned. The delete happened after. I should also point out that
> we were experiencing problems related to CASSANDRA-4206 and CASSANDRA-7808.
Another possible cause are these exceptions I found in the log as the nodes
were shutdown and brought up with the new version:
INFO [RMI TCP Connection(270364)-10.134.101.18] 2014-10-21 15:04:00,867
StorageService.java (line 939) DRAINED
ERROR [CompactionExecutor:15173] 2014-10-21 15:04:01,923 CassandraDaemon.java
(line 191) Exception in thread Thread[CompactionExecutor:15173,1,main]
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@1dad30c0
rejected from
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor@555b9c78[Terminated,
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 14052]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(Unknown
Source)
at java.util.concurrent.ScheduledThreadPoolExecutor.submit(Unknown
Source)
at
org.apache.cassandra.io.sstable.SSTableDeletingTask.schedule(SSTableDeletingTask.java:65)
at
org.apache.cassandra.io.sstable.SSTableReader.releaseReference(SSTableReader.java:976)
at
org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:370)
at org.apache.cassandra.db.DataTracker.postReplace(DataTracker.java:335)
at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:329)
at
org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
at
org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:995)
at
org.apache.cassandra.db.compaction.CompactionTask.replaceCompactedSSTables(CompactionTask.java:270)
at
org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:230)
at
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:208)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Each node has 4-9 of these exceptions as it is going down after being drained.
It seems Cassandra was trying to delete an sstable. Can this be related?
Best regards,
André Cruz
