Hi, all
I have a C* cluster with 12 nodes. My cassandra version is 2.1.14; Just now
two nodes crashed and client fails to export data with read consistency QUORUM.
The following are logs of failed nodes:
ERROR [SharedPool-Worker-159] 2016-09-26 20:51:14,124 Message.java:538 -
Unexpected exception during request; channel = [id: 0xce43a388,
/13.13.13.80:55536 :> /13.13.13.149:9042]
java.lang.AssertionError: null
at
org.apache.cassandra.transport.ServerConnection.applyStateTransition(ServerConnection.java:100)
~[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:442)
[apache-cassandra-2.1.14.jar:2.1.14]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335)
[apache-cassandra-2.1.14.jar:2.1.14]
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_65]
at
org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
[apache-cassandra-2.1.14.jar:2.1.14]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-2.1.14.jar:2.1.14]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
ERROR [SharedPool-Worker-116] 2016-09-26 20:51:14,125
JVMStabilityInspector.java:117 - JVM state determined to be unstable. Exiting
forcefully due to:
java.lang.OutOfMemoryError: Java heap space
ERROR [SharedPool-Worker-121] 2016-09-26 20:51:14,125
JVMStabilityInspector.java:117 - JVM state determined to be unstable. Exiting
forcefully due to:
java.lang.OutOfMemoryError: Java heap space
ERROR [SharedPool-Worker-157] 2016-09-26 20:51:14,124 Message.java:538 -
Unexpected exception during request; channel = [id: 0xce43a388,
/13.13.13.80:55536 :> /13.13.13.149:9042]
My server has total 256G memory so I set the MAX_HEAP_SIZE 60G, the config in
cassandra-env.sh:
MAX_HEAP_SIZE="60G"
HEAP_NEWSIZE="20G"
How to solve such OOM?
