Robin Mahony created CASSANDRA-13624:
----------------------------------------
Summary: upgradesstables crashes with OOM when upgrading sstables
with lots of range tombstones
Key: CASSANDRA-13624
URL: https://issues.apache.org/jira/browse/CASSANDRA-13624
Project: Cassandra
Issue Type: Bug
Reporter: Robin Mahony
CASSANDRA-7953 can lead to range tombstones not properly being compacted. When
trying to upgrade from Cassandra 2.0.X to 2.1.X, running upgradesstables
(either via nodetool OR the offline version) does not work, as it crashes with
OOM. This essentially means if you are running a version of Cassandra 2.0.X for
a period of time where lots of range tombstones have been generated; you will
be unable to upgrade to Cassandra 2.1.X following the normal procedures. Hence
upgrade from 2.0.X to 2.1.X is essentially broken if you hit CASSANDRA-7953.
I hit this while trying to upgrade to Cassandra 2.1.17.
Offline Version:
# sstableupgrade storagegrid s3_usage_delta
Found 15 sstables that need upgrading.
Upgrading
SSTableReader(path='/var/local/cassandra/data/0/storagegrid/s3_usage_delta/storagegrid-s3_usage_delta-jb-20230-Data.db')
ERROR 22:38:24,626 LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@6821300e) to class
org.apache.cassandra.io.sstable.SSTableReader$InstanceTidier@244363601:/var/local/cassandra/data/0/storagegrid/s3_usage_delta/storagegrid-s3_usage_delta-ka-79531
was not released before the reference was garbage collected
ERROR 22:38:24,631 LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@2891457f) to class
org.apache.cassandra.io.sstable.SSTableReader$InstanceTidier@1230033823:/var/local/cassandra/data/0/storagegrid/s3_usage_delta/storagegrid-s3_usage_delta-ka-79532
was not released before the reference was garbage collected
Exception in thread "ScheduledTasks:1" Exception in thread
"metrics-meter-tick-thread-2" java.lang.OutOfMemoryError: Java heap space
at java.lang.Class.getName0(Native Method)
at java.lang.Class.getName(Class.java:642)
at java.lang.Throwable.toString(Throwable.java:479)
at java.lang.Throwable.<init>(Throwable.java:311)
at java.lang.Exception.<init>(Exception.java:102)
at java.util.concurrent.ExecutionException.<init>(ExecutionException.java:90)
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.extractThrowable(DebuggableThreadPoolExecutor.java:246)
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.logExceptionsAfterExecute(DebuggableThreadPoolExecutor.java:210)
at
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.afterExecute(DebuggableScheduledThreadPoolExecutor.java:89)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1150)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
ERROR 22:41:21,080 Error in ThreadPoolExecutor
java.lang.OutOfMemoryError: Java heap space
ERROR 22:41:21,080 JVM state determined to be unstable. Exiting forcefully due
to:
java.lang.OutOfMemoryError: Java heap space
root@DC1-SN-10-224-6-066:/var/local/cassandra/data/0 #
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
