[
https://issues.apache.org/jira/browse/CASSANDRA-13624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16056479#comment-16056479
]
Jeff Jirsa commented on CASSANDRA-13624:
----------------------------------------
Do you get the same OOM with the online version as you do with the offline?
> upgradesstables crashes with OOM when upgrading sstables with lots of range
> tombstones
> --------------------------------------------------------------------------------------
>
> Key: CASSANDRA-13624
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13624
> Project: Cassandra
> Issue Type: Bug
> Reporter: Robin Mahony
>
> CASSANDRA-7953 can lead to range tombstones not properly being compacted.
> When trying to upgrade from Cassandra 2.0.X to 2.1.X, running upgradesstables
> (either via nodetool OR the offline version) does not work, as it crashes
> with OOM. This essentially means if you are running a version of Cassandra
> 2.0.X for a period of time where lots of range tombstones have been
> generated; you will be unable to upgrade to Cassandra 2.1.X following the
> normal procedures. Hence upgrade from 2.0.X to 2.1.X is essentially broken if
> you hit CASSANDRA-7953.
> I hit this while trying to upgrade to Cassandra 2.1.17.
> Offline Version:
> # sstableupgrade storagegrid s3_usage_delta
> Found 15 sstables that need upgrading.
> Upgrading
> SSTableReader(path='/var/local/cassandra/data/0/storagegrid/s3_usage_delta/storagegrid-s3_usage_delta-jb-20230-Data.db')
>
> ERROR 22:38:24,626 LEAK DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@6821300e) to class
> org.apache.cassandra.io.sstable.SSTableReader$InstanceTidier@244363601:/var/local/cassandra/data/0/storagegrid/s3_usage_delta/storagegrid-s3_usage_delta-ka-79531
> was not released before the reference was garbage collected
> ERROR 22:38:24,631 LEAK DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@2891457f) to class
> org.apache.cassandra.io.sstable.SSTableReader$InstanceTidier@1230033823:/var/local/cassandra/data/0/storagegrid/s3_usage_delta/storagegrid-s3_usage_delta-ka-79532
> was not released before the reference was garbage collected
> Exception in thread "ScheduledTasks:1" Exception in thread
> "metrics-meter-tick-thread-2" java.lang.OutOfMemoryError: Java heap space
> at java.lang.Class.getName0(Native Method)
> at java.lang.Class.getName(Class.java:642)
> at java.lang.Throwable.toString(Throwable.java:479)
> at java.lang.Throwable.<init>(Throwable.java:311)
> at java.lang.Exception.<init>(Exception.java:102)
> at java.util.concurrent.ExecutionException.<init>(ExecutionException.java:90)
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.extractThrowable(DebuggableThreadPoolExecutor.java:246)
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.logExceptionsAfterExecute(DebuggableThreadPoolExecutor.java:210)
> at
> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.afterExecute(DebuggableScheduledThreadPoolExecutor.java:89)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1150)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
> java.lang.OutOfMemoryError: Java heap space
> java.lang.OutOfMemoryError: Java heap space
> ERROR 22:41:21,080 Error in ThreadPoolExecutor
> java.lang.OutOfMemoryError: Java heap space
> ERROR 22:41:21,080 JVM state determined to be unstable. Exiting forcefully
> due to:
> java.lang.OutOfMemoryError: Java heap space
> root@DC1-SN-10-224-6-066:/var/local/cassandra/data/0 #
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
