[
https://issues.apache.org/jira/browse/CASSANDRA-12005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15364484#comment-15364484
]
Chris Powell commented on CASSANDRA-12005:
------------------------------------------
Thanks for the input! I have made some configuration changes to Cassandra and I
am no longer experiencing the problem. (Knock on wood). I specifically have
reduced the number of `concurrent_compactors` and moved the
`memtable_allocation_type` to `offheap_objects`.
> Out of memory error in MessagingService
> ---------------------------------------
>
> Key: CASSANDRA-12005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12005
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: Ubuntu 14.04.4 LTS 3.13.0-79-generic #123-Ubuntu SMP
> x86_64
> Cassandra ReleaseVersion: 2.2.5
> java version "1.8.0_31"
> Java(TM) SE Runtime Environment (build 1.8.0_31-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 25.31-b07, mixed mode)
> Reporter: Chris Powell
>
> I am periodically loosing nodes due to the below OOM error. The nodes restart
> perfectly fine. It appears intermittent and randomly affects nodes. There are
> no other warnings or errors in the log files.
> I am using the {{GCG1}} with the following options:
> {quote}
> JVM_OPTS="$JVM_OPTS -XX:+UseG1GC"
> JVM_OPTS="$JVM_OPTS -XX:SurvivorRatio=8"
> JVM_OPTS="$JVM_OPTS -XX:MaxTenuringThreshold=1"
> JVM_OPTS="$JVM_OPTS -XX:+UseTLAB"
> JVM_OPTS="$JVM_OPTS -XX:+AlwaysPreTouch"
> JVM_OPTS="$JVM_OPTS -XX:-UseBiasedLocking"
> JVM_OPTS="$JVM_OPTS -XX:+ResizeTLAB"
> JVM_OPTS="$JVM_OPTS -XX:MaxGCPauseMillis=500"
> JVM_OPTS="$JVM_OPTS -XX:G1RSetUpdatingPauseTimePercent=10"
> JVM_OPTS="$JVM_OPTS -XX:InitiatingHeapOccupancyPercent=25"
> {quote}
> ERROR [MessagingService-Incoming-/10.184.11.109] 2016-06-14 13:00:20,237
> CassandraDaemon.java:185 - Exception in thread
> Thread[MessagingService-Incoming-/10.184.11.109,5,main]
> java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:361)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:322)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:126)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:109)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:101)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:109)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:200)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:177)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:91)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> ERROR [MessagingService-Incoming-/10.184.11.109] 2016-06-14 13:00:20,239
> JVMStabilityInspector.java:117 - JVM state determined to be unstable.
> Exiting forcefully due to:
> java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:361)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:322)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:126)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:109)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:101)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:109)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:200)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:177)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:91)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
