[
https://issues.apache.org/jira/browse/CASSANDRA-8065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14162524#comment-14162524
]
Vishal Mehta commented on CASSANDRA-8065:
-----------------------------------------
[~krummas] : yes my tests working fine now after I bumped up the memory but YG
size remains default.
Apologies for the noise.
> Mutationstage error, ThreadPoolExecutor has shut down
> -----------------------------------------------------
>
> Key: CASSANDRA-8065
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8065
> Project: Cassandra
> Issue Type: Bug
> Environment: Casssandra Version: 2.0.9
> Total Clusters/Datacenters: 3
> Nodes in each cluster: 4
> Total # of Nodes: 12
> Replication Factor:3 for each data center
> Node Size of each cluster: C3.xlarge in EC2
> Reporter: Vishal Mehta
>
> 1. Writes running at about 120 qps w/ TTL at row level for 60 minutes (for 3
> column families)
> 2. GC set to run at every 10 minutes (so that we claim our space back from
> tombstones)
> 3. Other settings set as default.
> After some time (1 hour 30 minutes approximately of tests execution) 11 nodes
> across datacenters are Down. And write tests fails, with following exception
> in /var/log/cassandra/system.log.
> {noformat}
> ERROR [MutationStage:18098] 2014-10-04 01:05:36,525 CassandraDaemon.java
> (line 199) Exception in thread Thread[MutationStage:18098,5,main]
> java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
> down
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:61)
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:145)
> at
> java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110)
> at
> org.apache.cassandra.service.StorageProxy.submitHint(StorageProxy.java:893)
> at
> org.apache.cassandra.service.StorageProxy.access$300(StorageProxy.java:68)
> at
> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:1958)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> WARN [StorageServiceShutdownHook] 2014-10-04 01:05:36,525 StorageProxy.java
> (line 2031) Some hints were not written before shutdown. This is not
> supposed to happen. You should (a) run repair, and (b) file a bug report
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
