After a bulk load of writes to existing partition keys (with a higher
timestamp), I wanted to free disk space, suspecting that rows will be in
the highest levels and it would take a time until they were compacted.
I've started a major compaction, and the disk usage went from ~30% to ~40%
(as expected) but after ~10 hs Cassandra has crashed (*) and disk usage
continues being ~40%, even after restart.
How could I remove SSTables created during the failed compaction?
(*) From the logs, I understand that the JVM runs out of java heap space.
However, I don't understand why there are multiple OutOfMemory errors, it
shouldn't crashed with the first one?
ERROR [MessagingService-Incoming-/x.x.x.x] 2018-11-06 22:28:35,251
CassandraDaemon.java:207 - Exception in thread
Thread[MessagingService-Incoming-/x.x.x.x,5,main]
java.lang.OutOfMemoryError: Java heap space
at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:403)
~[apache-cassandra-3.0.13.jar:3.0.13]
at
org.apache.cassandra.utils.ByteBufferUtil.readWithVIntLength(ByteBufferUtil.java:341)
~[apache-cassandra-3.0.13.jar:3.0.13]
at
org.apache.cassandra.db.ReadResponse$Serializer.deserialize(ReadResponse.java:382)
~[apache-cassandra-3.0.13.jar:3.0.13]
--
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92)
~[apache-cassandra-3.0.13.jar:3.0.13]
INFO [HintsDispatcher:63] 2018-11-06 22:34:49,678
HintsDispatchExecutor.java:271 - Finished hinted handoff of file
9584b93c-f86e-464f-a9ba-3dd33134b7af-1541550621736-1.hints to endpoint
/y.y.y.y: 9584b93c-f86e-464f-a9ba-3dd33134b7af, partially
ERROR [MessagingService-Incoming-/z.z.z.z] 2018-11-06 22:37:11,099
CassandraDaemon.java:207 - Exception in thread
Thread[MessagingService-Incoming-/z.z.z.z,5,main]
java.lang.OutOfMemoryError: Java heap space
INFO [ScheduledTasks:1] 2018-11-06 22:37:34,716 StatusLogger.java:56
- Sampler 0 0 0
0 0
ERROR [MessagingService-Incoming-/y.y.y.y] 2018-11-06 22:39:39,860
CassandraDaemon.java:207 - Exception in thread
Thread[MessagingService-Incoming-/y.y.y.y,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [MessagingService-Incoming-/z.z.z.z] 2018-11-06 22:41:52,690
CassandraDaemon.java:207 - Exception in thread
Thread[MessagingService-Incoming-/z.z.z.z,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [MessagingService-Incoming-/a.a.a.a] 2018-11-06 22:42:23,498
CassandraDaemon.java:207 - Exception in thread
Thread[MessagingService-Incoming-/a.a.a.a,5,main]
java.lang.OutOfMemoryError: Java heap space
.....
ERROR [MessagingService-Incoming-/x.x.x.x] 2018-11-06 23:42:53,947
JVMStabilityInspector.java:140 - JVM state determined to be unstable.
Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space
at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:403)
~[apache-cassandra-3.0.13.jar:3.0.13]
at
org.apache.cassandra.utils.ByteBufferUtil.readWithVIntLength(ByteBufferUtil.java:341)
~[apache-cassandra-3.0.13.jar:3.0.13]
at
org.apache.cassandra.db.ReadResponse$Serializer.deserialize(ReadResponse.java:382)
~[apache-cassandra-3.0.13.jar:3.0.13]
--
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178)
~[apache-cassandra-3.0.13.jar:3.0.13]
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92)
~[apache-cassandra-3.0.13.jar:3.0.13]
ERROR [SharedPool-Worker-6] 2018-11-06 23:42:53,947
JVMStabilityInspector.java:140 - JVM state determined to be unstable.
Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space
ERROR [SharedPool-Worker-12] 2018-11-06 23:42:53,947
JVMStabilityInspector.java:140 - JVM state determined to be unstable.
Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space
ERROR [SharedPool-Worker-18] 2018-11-06 23:42:53,948
JVMStabilityInspector.java:140 - JVM state determined to be unstable.
Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space
Thanks.
