[
https://issues.apache.org/jira/browse/CASSANDRA-8065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161952#comment-14161952
]
Vishal Mehta commented on CASSANDRA-8065:
-----------------------------------------
Since I've been running tests on and off after restarting Cassandra, I couldn't
find much of the previous logs, but I saw following information in other nodes.
Which suggests that it went *OOM*.
I've got 4gb or ram and 1gb of young generation ram (new ram), and data running
at this slow qps makes me think if it is still low :).
{noformat}
ERROR [Native-Transport-Requests:4699] 2014-10-07 06:46:33,322
ErrorMessage.java (line 222) Unexpected exception during request
java.lang.OutOfMemoryError: Java heap space
ERROR [Native-Transport-Requests:4698] 2014-10-07 06:46:33,310
ErrorMessage.java (line 222) Unexpected exception during request
java.lang.OutOfMemoryError: Java heap space
ERROR [MutationStage:24] 2014-10-07 06:46:33,307 CassandraDaemon.java (line
199) Exception in thread Thread[MutationStage:24,5,main]
java.lang.OutOfMemoryError: Java heap space
WARN [New I/O worker #6] 2014-10-07 06:46:11,661 Slf4JLogger.java (line 76)
Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: Java heap space
ERROR [pool-3-thread-1] 2014-10-07 06:46:01,791 CassandraDaemon.java (line 199)
Exception in thread Thread[pool-3-thread-1,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [MutationStage:21] 2014-10-07 06:44:23,041 CassandraDaemon.java (line
199) Exception in thread Thread[MutationStage:21,5,main]
java.lang.IllegalThreadStateException
at java.lang.Thread.start(Thread.java:684)
at
org.apache.cassandra.service.CassandraDaemon$2.uncaughtException(CassandraDaemon.java:205)
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.handleOrLog(DebuggableThreadPoolExecutor.java:220)
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.logExceptionsAfterExecute(DebuggableThreadPoolExecutor.java:203)
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:183)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
ERROR [MutationStage:3] 2014-10-07 06:44:23,041 CassandraDaemon.java (line 199)
Exception in thread Thread[MutationStage:3,5,main]
java.lang.OutOfMemoryError: Java heap space
INFO [GossipTasks:1] 2014-10-07 06:44:21,866 Gossiper.java (line 890)
InetAddress /54.225.19.49 is now DOWN
ERROR [Thread-31] 2014-10-07 06:44:20,370 CassandraDaemon.java (line 199)
Exception in thread Thread[Thread-31,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [MutationStage:28] 2014-10-07 06:44:15,210 CassandraDaemon.java (line
199) Exception in thread Thread[MutationStage:28,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [MutationStage:18] 2014-10-07 06:44:14,223 CassandraDaemon.java (line
199) Exception in thread Thread[MutationStage:18,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [MutationStage:19] 2014-10-07 06:42:31,591 CassandraDaemon.java (line
199) Exception in thread Thread[MutationStage:19,5,main]
java.lang.IllegalThreadStateException
at java.lang.Thread.start(Thread.java:684)
at
org.apache.cassandra.service.CassandraDaemon$2.uncaughtException(CassandraDaemon.java:205)
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.handleOrLog(DebuggableThreadPoolExecutor.java:220)
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.logExceptionsAfterExecute(DebuggableThreadPoolExecutor.java:203)
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:183)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
ERROR [MutationStage:28] 2014-10-07 06:46:46,375 CassandraDaemon.java (line
199) Exception in thread Thread[MutationStage:28,5,main]
java.lang.IllegalThreadStateException
at java.lang.Thread.start(Thread.java:684)
at
org.apache.cassandra.service.CassandraDaemon$2.uncaughtException(CassandraDaemon.java:205)
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.handleOrLog(DebuggableThreadPoolExecutor.java:220)
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.logExceptionsAfterExecute(DebuggableThreadPoolExecutor.java:203)
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:183)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
{noformat}
> Mutationstage error, ThreadPoolExecutor has shut down
> -----------------------------------------------------
>
> Key: CASSANDRA-8065
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8065
> Project: Cassandra
> Issue Type: Bug
> Environment: Casssandra Version: 2.0.9
> Total Clusters/Datacenters: 3
> Nodes in each cluster: 4
> Total # of Nodes: 12
> Replication Factor:3 for each data center
> Node Size of each cluster: C3.xlarge in EC2
> Reporter: Vishal Mehta
>
> 1. Writes running at about 120 qps w/ TTL at row level for 60 minutes (for 3
> column families)
> 2. GC set to run at every 10 minutes (so that we claim our space back from
> tombstones)
> 3. Other settings set as default.
> After some time (1 hour 30 minutes approximately of tests execution) 11 nodes
> across datacenters are Down. And write tests fails, with following exception
> in /var/log/cassandra/system.log.
> {noformat}
> ERROR [MutationStage:18098] 2014-10-04 01:05:36,525 CassandraDaemon.java
> (line 199) Exception in thread Thread[MutationStage:18098,5,main]
> java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
> down
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:61)
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:145)
> at
> java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110)
> at
> org.apache.cassandra.service.StorageProxy.submitHint(StorageProxy.java:893)
> at
> org.apache.cassandra.service.StorageProxy.access$300(StorageProxy.java:68)
> at
> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:1958)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> WARN [StorageServiceShutdownHook] 2014-10-04 01:05:36,525 StorageProxy.java
> (line 2031) Some hints were not written before shutdown. This is not
> supposed to happen. You should (a) run repair, and (b) file a bug report
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
