[
https://issues.apache.org/jira/browse/CASSANDRA-8065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Eriksson resolved CASSANDRA-8065.
----------------------------------------
Resolution: Not a Problem
So that is the actual problem, you are running out of heap.
You probably need to bump your heap or reduce the size of the data you put in
(or both)
> Mutationstage error, ThreadPoolExecutor has shut down
> -----------------------------------------------------
>
> Key: CASSANDRA-8065
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8065
> Project: Cassandra
> Issue Type: Bug
> Environment: Casssandra Version: 2.0.9
> Total Clusters/Datacenters: 3
> Nodes in each cluster: 4
> Total # of Nodes: 12
> Replication Factor:3 for each data center
> Node Size of each cluster: C3.xlarge in EC2
> Reporter: Vishal Mehta
>
> 1. Writes running at about 120 qps w/ TTL at row level for 60 minutes (for 3
> column families)
> 2. GC set to run at every 10 minutes (so that we claim our space back from
> tombstones)
> 3. Other settings set as default.
> After some time (1 hour 30 minutes approximately of tests execution) 11 nodes
> across datacenters are Down. And write tests fails, with following exception
> in /var/log/cassandra/system.log.
> {noformat}
> ERROR [MutationStage:18098] 2014-10-04 01:05:36,525 CassandraDaemon.java
> (line 199) Exception in thread Thread[MutationStage:18098,5,main]
> java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
> down
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:61)
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:145)
> at
> java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110)
> at
> org.apache.cassandra.service.StorageProxy.submitHint(StorageProxy.java:893)
> at
> org.apache.cassandra.service.StorageProxy.access$300(StorageProxy.java:68)
> at
> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:1958)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> WARN [StorageServiceShutdownHook] 2014-10-04 01:05:36,525 StorageProxy.java
> (line 2031) Some hints were not written before shutdown. This is not
> supposed to happen. You should (a) run repair, and (b) file a bug report
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
