[
https://issues.apache.org/jira/browse/CASSANDRA-16060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17184058#comment-17184058
]
Brandon Williams commented on CASSANDRA-16060:
----------------------------------------------
You don't have message coalescing enabled, so it should correspond to one
query. But, that's still quite large. Your commitlog segment size is the
default at 32MB, so you can't write a mutation larger than 16MB. Are you
seeing any errors in the logs before the OOM? Generally I wouldn't think a
schema problem causes this.
> Cassandra crashes with OutOfMemory Exception
> --------------------------------------------
>
> Key: CASSANDRA-16060
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16060
> Project: Cassandra
> Issue Type: Bug
> Reporter: Thomas De Keulenaer
> Priority: Normal
> Attachments: cassandra.yaml
>
>
> Out Cassandra instance has run perfectly for almost 5 years, but aout a month
> ago, the performance has dropped significantly. Reads are incredibly slow or
> time out, making the cluster almost useless. The load on the nodes has
> skyrocketed to almost 100% CPU usage. On top of that, the nodes crash with
> OutOfMemoryError.
> I also have [heap
> dumps|https://arcelormittal-my.sharepoint.com/:f:/g/personal/sidtdke_sidmar_be/Es7PaQq15jdKgIvDfMFfBRoBR6sr60t156AxbA0dFZvzJg?e=rohgHn].
>
> Cassandra version: 3.11.1
> Java version: OpenJDK 1.8.0_201
> We have upgraded our nodes to version 3.11.7 and applied the recommended
> OS/system setting, but this has not improved performance.
> {code:java}
> ERROR [ReadStage-9] 2020-08-10 14:56:01,399 JVMStabilityInspector.java:142 -
> JVM state determined to be unstable. Exiting forcefully due to:
> java.lang.OutOfMemoryError: Java heap space
> at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
> ~[na:1.8.0_201]
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_201]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.reallocate(DataOutputBuffer.java:159)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:413)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.rows.Cell$Serializer.serialize(Cell.java:210)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.lambda$serializeRowBody$0(UnfilteredSerializer.java:248)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer$$Lambda$120/205506350.accept(Unknown
> Source) ~[na:na]
> at
> org.apache.cassandra.utils.btree.BTree.applyForwards(BTree.java:1242)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.utils.btree.BTree.apply(BTree.java:1197)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.db.rows.BTreeRow.apply(BTreeRow.java:172)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.serializeRowBody(UnfilteredSerializer.java:236)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:205)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:137)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:125)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:137)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:92)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:79)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:308)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:167)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:160)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:156)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:346)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:50)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_201]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
