Vishal Mehta created CASSANDRA-8065:
---------------------------------------
Summary: Mutationstage error, ThreadPoolExecutor has shut down
Key: CASSANDRA-8065
URL: https://issues.apache.org/jira/browse/CASSANDRA-8065
Project: Cassandra
Issue Type: Bug
Environment: Casssandra Version: 2.0.9
Total Clusters/Datacenters: 3
Nodes in each cluster: 4
Total # of Nodes: 12
Replication Factor:3 for each data center
Node Size of each cluster: C3.xlarge in EC2
Reporter: Vishal Mehta
1. Writes running at about 120 qps w/ TTL at row level for 60 minutes (for 3
column families)
2. GC set to run at every 10 minutes (so that we claim our space back from
tombstones)
3. Other settings set as default.
After some time (1 hour 30 minutes approximately of tests execution) 11 nodes
across datacenters are Down. And write tests fails, with following exception in
/var/log/cassandra/system.log.
{noformat}
ERROR [MutationStage:18098] 2014-10-04 01:05:36,525 CassandraDaemon.java (line
199) Exception in thread Thread[MutationStage:18098,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
down
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:61)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:145)
at
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110)
at
org.apache.cassandra.service.StorageProxy.submitHint(StorageProxy.java:893)
at
org.apache.cassandra.service.StorageProxy.access$300(StorageProxy.java:68)
at
org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:1958)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
WARN [StorageServiceShutdownHook] 2014-10-04 01:05:36,525 StorageProxy.java
(line 2031) Some hints were not written before shutdown. This is not supposed
to happen. You should (a) run repair, and (b) file a bug report
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
