I've found a huge partion (~9GB) in my cassandra cluster because I'm loosing 3 nodes recurrently due to OutOfMemoryError
> ERROR [SharedPool-Worker-12] 2019-08-12 11:07:45,735 > JVMStabilityInspector.java:140 - JVM state determined to be unstable. > Exiting forcefully due to: > java.lang.OutOfMemoryError: Java heap space > at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) ~[na:1.8.0_151] > at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_151] > at > org.apache.cassandra.io.util.DataOutputBuffer.reallocate(DataOutputBuffer.java:126) > ~[apache-cassandra-3.0.13.jar:3.0.13] > at > org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:86) > ~[apache-cassandra-3.0.13.jar:3.0.13] > at > org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132) > ~[apache-cassandra-3.0.13.jar:3.0.13] > at > org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151) > ~[apache-cassandra-3.0.13.jar:3.0.13] > at > org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:297) > ~[apache-cassandra-3.0.13.jar:3.0.13] > at > org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:373) > ~[apache-cassandra-3.0.13.jar:3.0.13] > at > org.apache.cassandra.db.rows.BufferCell$Serializer.serialize(BufferCell.java:267) > ~[apache-cassandra-3.0.13.jar:3.0.13] > at > org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:193) > ~[apache-cassandra-3.0.13.jar:3.0.13] > at > org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:109) > ~[apache-cassandra-3.0.13.jar:3.0.13] > at > org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:97) > ~[apache-cassandra-3.0.13.jar:3.0.13] > at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:132) > ~[apache-cassandra-3.0.13.jar:3.0.13] > at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87) > ~[apache-cassandra-3.0.13.jar:3.0.13] > at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:77) > ~[apache-cassandra-3.0.13.jar:3.0.13] > at > org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:301) > ~[apache-cassandra-3.0.13.jar:3.0.13] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:145) > ~[apache-cassandra-3.0.13.jar:3.0.13] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:138) > ~[apache-cassandra-3.0.13.jar:3.0.13] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:134) > ~[apache-cassandra-3.0.13.jar:3.0.13] > at > org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76) > ~[apache-cassandra-3.0.13.jar:3.0.13] > at > org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:321) > ~[apache-cassandra-3.0.13.jar:3.0.13] > at > org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:47) > ~[apache-cassandra-3.0.13.jar:3.0.13] > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) > ~[apache-cassandra-3.0.13.jar:3.0.13] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_151] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) > ~[apache-cassandra-3.0.13.jar:3.0.13] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) > [apache-cassandra-3.0.13.jar:3.0.13] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-3.0.13.jar:3.0.13] > at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151] > >From the stacktrace I assume that some client is try to read that partition (ReadResponse) so I may filter requests to this specific partition as a quick solution but I think the compaction will never be able to remove this partition (I already executed a DELETE). What can I do to delete this partition? May I delete the sstable directly? Or should I upgrade the node and give more heap to cassandra? Thanks.
