[
https://issues.apache.org/jira/browse/CASSANDRA-16060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas De Keulenaer updated CASSANDRA-16060:
--------------------------------------------
Description:
Out Cassandra instance has run perfectly for almost 5 years, but aout a month
ago, the performance has dropped significantly. Reads are incredibly slow or
time out, making the cluster almost useless. The load on the nodes has
skyrocketed to almost 100% CPU usage. On top of that, the nodes crash with
OutOfMemoryError.
I also have [heap
dumps|https://arcelormittal-my.sharepoint.com/:f:/g/personal/sidtdke_sidmar_be/Es7PaQq15jdKgIvDfMFfBRoBR6sr60t156AxbA0dFZvzJg?e=rohgHn].
Cassandra version: 3.11.1
Java version: OpenJDK 1.8.0_201
We have upgraded our nodes to version 3.11.7, but this has not improved
performance
{code:java}
ERROR [ReadStage-9] 2020-08-10 14:56:01,399 JVMStabilityInspector.java:142 -
JVM state determined to be unstable. Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
~[na:1.8.0_201]
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_201]
at
org.apache.cassandra.io.util.DataOutputBuffer.reallocate(DataOutputBuffer.java:159)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:413)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.rows.Cell$Serializer.serialize(Cell.java:210)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.rows.UnfilteredSerializer.lambda$serializeRowBody$0(UnfilteredSerializer.java:248)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.rows.UnfilteredSerializer$$Lambda$120/205506350.accept(Unknown
Source) ~[na:na]
at
org.apache.cassandra.utils.btree.BTree.applyForwards(BTree.java:1242)
~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.utils.btree.BTree.apply(BTree.java:1197)
~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.BTreeRow.apply(BTreeRow.java:172)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.rows.UnfilteredSerializer.serializeRowBody(UnfilteredSerializer.java:236)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:205)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:137)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:125)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:137)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:92)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:79)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:308)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:167)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:160)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:156)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:346)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:50)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_201]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
~[apache-cassandra-3.11.1.jar:3.11.1]
{code}
was:
Out Cassandra instance has run perfectly for almost 5 years, but aout a month
ago, the performance has dropped significantly. Reads are incredibly slow or
time out, making the cluster almost useless. The load on the nodes has
skyrocketed to almost 100% CPU usage. On top of that, the nodes crash with
OutOfMemoryError.
I also have [heap
dumps|https://arcelormittal-my.sharepoint.com/:f:/g/personal/sidtdke_sidmar_be/Es7PaQq15jdKgIvDfMFfBRoBR6sr60t156AxbA0dFZvzJg?e=rohgHn].
Cassandra version: 3.11.7
Java version: OpenJDK 1.8.0_201
{code:java}
ERROR [ReadStage-9] 2020-08-10 14:56:01,399 JVMStabilityInspector.java:142 -
JVM state determined to be unstable. Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
~[na:1.8.0_201]
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_201]
at
org.apache.cassandra.io.util.DataOutputBuffer.reallocate(DataOutputBuffer.java:159)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:413)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.rows.Cell$Serializer.serialize(Cell.java:210)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.rows.UnfilteredSerializer.lambda$serializeRowBody$0(UnfilteredSerializer.java:248)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.rows.UnfilteredSerializer$$Lambda$120/205506350.accept(Unknown
Source) ~[na:na]
at
org.apache.cassandra.utils.btree.BTree.applyForwards(BTree.java:1242)
~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.utils.btree.BTree.apply(BTree.java:1197)
~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.db.rows.BTreeRow.apply(BTreeRow.java:172)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.rows.UnfilteredSerializer.serializeRowBody(UnfilteredSerializer.java:236)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:205)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:137)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:125)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:137)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:92)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:79)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:308)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:167)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:160)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:156)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:346)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:50)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_201]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
~[apache-cassandra-3.11.1.jar:3.11.1]
{code}
> Cassandra crashes with OutOfMemory Exception
> --------------------------------------------
>
> Key: CASSANDRA-16060
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16060
> Project: Cassandra
> Issue Type: Bug
> Reporter: Thomas De Keulenaer
> Priority: Normal
>
> Out Cassandra instance has run perfectly for almost 5 years, but aout a month
> ago, the performance has dropped significantly. Reads are incredibly slow or
> time out, making the cluster almost useless. The load on the nodes has
> skyrocketed to almost 100% CPU usage. On top of that, the nodes crash with
> OutOfMemoryError.
> I also have [heap
> dumps|https://arcelormittal-my.sharepoint.com/:f:/g/personal/sidtdke_sidmar_be/Es7PaQq15jdKgIvDfMFfBRoBR6sr60t156AxbA0dFZvzJg?e=rohgHn].
>
> Cassandra version: 3.11.1
> Java version: OpenJDK 1.8.0_201
> We have upgraded our nodes to version 3.11.7, but this has not improved
> performance
> {code:java}
> ERROR [ReadStage-9] 2020-08-10 14:56:01,399 JVMStabilityInspector.java:142 -
> JVM state determined to be unstable. Exiting forcefully due to:
> java.lang.OutOfMemoryError: Java heap space
> at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
> ~[na:1.8.0_201]
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_201]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.reallocate(DataOutputBuffer.java:159)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:413)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.rows.Cell$Serializer.serialize(Cell.java:210)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.lambda$serializeRowBody$0(UnfilteredSerializer.java:248)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer$$Lambda$120/205506350.accept(Unknown
> Source) ~[na:na]
> at
> org.apache.cassandra.utils.btree.BTree.applyForwards(BTree.java:1242)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.utils.btree.BTree.apply(BTree.java:1197)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.db.rows.BTreeRow.apply(BTreeRow.java:172)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.serializeRowBody(UnfilteredSerializer.java:236)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:205)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:137)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:125)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:137)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:92)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:79)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:308)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:167)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:160)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:156)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:346)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:50)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_201]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
