Chris Powell created CASSANDRA-12005:
----------------------------------------
Summary: Out of memory error in MessagingService
Key: CASSANDRA-12005
URL: https://issues.apache.org/jira/browse/CASSANDRA-12005
Project: Cassandra
Issue Type: Bug
Components: Streaming and Messaging
Environment: Ubuntu 14.04.4 LTS 3.13.0-79-generic #123-Ubuntu SMP
x86_64
Cassandra ReleaseVersion: 2.2.5
java version "1.8.0_31"
Java(TM) SE Runtime Environment (build 1.8.0_31-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.31-b07, mixed mode)
Reporter: Chris Powell
I am periodically loosing nodes due to the below OOM error. The nodes restart
perfectly fine. It appears intermittent and randomly affects nodes. There are
no other warnings or errors in the log files.
I am using the {{GCG1}} with the following options:
{quote}
JVM_OPTS="$JVM_OPTS -XX:+UseG1GC"
JVM_OPTS="$JVM_OPTS -XX:SurvivorRatio=8"
JVM_OPTS="$JVM_OPTS -XX:MaxTenuringThreshold=1"
JVM_OPTS="$JVM_OPTS -XX:+UseTLAB"
JVM_OPTS="$JVM_OPTS -XX:+AlwaysPreTouch"
JVM_OPTS="$JVM_OPTS -XX:-UseBiasedLocking"
JVM_OPTS="$JVM_OPTS -XX:+ResizeTLAB"
JVM_OPTS="$JVM_OPTS -XX:MaxGCPauseMillis=500"
JVM_OPTS="$JVM_OPTS -XX:G1RSetUpdatingPauseTimePercent=10"
JVM_OPTS="$JVM_OPTS -XX:InitiatingHeapOccupancyPercent=25"
{quote}
ERROR [MessagingService-Incoming-/10.184.11.109] 2016-06-14 13:00:20,237
CassandraDaemon.java:185 - Exception in thread
Thread[MessagingService-Incoming-/10.184.11.109,5,main]
java.lang.OutOfMemoryError: Java heap space
at
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:361)
~[apache-cassandra-2.2.5.jar:2.2.5]
at
org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:322)
~[apache-cassandra-2.2.5.jar:2.2.5]
at
org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:126)
~[apache-cassandra-2.2.5.jar:2.2.5]
at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:109)
~[apache-cassandra-2.2.5.jar:2.2.5]
at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:101)
~[apache-cassandra-2.2.5.jar:2.2.5]
at
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:109)
~[apache-cassandra-2.2.5.jar:2.2.5]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
~[apache-cassandra-2.2.5.jar:2.2.5]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
~[apache-cassandra-2.2.5.jar:2.2.5]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
~[apache-cassandra-2.2.5.jar:2.2.5]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
~[apache-cassandra-2.2.5.jar:2.2.5]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:200)
~[apache-cassandra-2.2.5.jar:2.2.5]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:177)
~[apache-cassandra-2.2.5.jar:2.2.5]
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:91)
~[apache-cassandra-2.2.5.jar:2.2.5]
ERROR [MessagingService-Incoming-/10.184.11.109] 2016-06-14 13:00:20,239
JVMStabilityInspector.java:117 - JVM state determined to be unstable. Exiting
forcefully due to:
java.lang.OutOfMemoryError: Java heap space
at
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:361)
~[apache-cassandra-2.2.5.jar:2.2.5]
at
org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:322)
~[apache-cassandra-2.2.5.jar:2.2.5]
at
org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:126)
~[apache-cassandra-2.2.5.jar:2.2.5]
at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:109)
~[apache-cassandra-2.2.5.jar:2.2.5]
at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:101)
~[apache-cassandra-2.2.5.jar:2.2.5]
at
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:109)
~[apache-cassandra-2.2.5.jar:2.2.5]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
~[apache-cassandra-2.2.5.jar:2.2.5]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
~[apache-cassandra-2.2.5.jar:2.2.5]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
~[apache-cassandra-2.2.5.jar:2.2.5]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
~[apache-cassandra-2.2.5.jar:2.2.5]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:200)
~[apache-cassandra-2.2.5.jar:2.2.5]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:177)
~[apache-cassandra-2.2.5.jar:2.2.5]
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:91)
~[apache-cassandra-2.2.5.jar:2.2.5]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
