[
https://issues.apache.org/jira/browse/CASSANDRA-14308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16408844#comment-16408844
]
Jay Zhuang commented on CASSANDRA-14308:
----------------------------------------
Thanks for the information. You're right, I'm unable to reproduce the problem
locally with ccm if I just kill the process while compacting.
I'm trying to find more logs related to this, several clusters are having this
orphaned SSTables issue, but maybe they're accumulated for a long time.
On the other hand, should we just remove these SSTables without {{Stats}} (or
have an option to do that)? Just like remove SSTables without {{Data}}:
[{{ColumnFamilyStore.java:648}}|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L648]
> Remove invalid SSTables from interrupted compaction
> ---------------------------------------------------
>
> Key: CASSANDRA-14308
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14308
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Reporter: Jay Zhuang
> Assignee: Jay Zhuang
> Priority: Minor
>
> If the JVM crash while compaction is in progress, the incompleted SSTable
> won't be cleaned up, which causes "{{Stats component is missing for
> sstable}}" error in the startup log:
> {noformat}
> ERROR [SSTableBatchOpen:3] 2018-03-11 00:17:35,597 CassandraDaemon.java:207 -
> Exception in thread Thread[SSTableBatchOpen:3,5,main]
> java.lang.AssertionError: Stats component is missing for sstable
> /cassandra/data/keyspace/table-id/mc-12345-big
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:458)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:374)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:533)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_121]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_121]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_121]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_121]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> [apache-cassandra-3.0.14.jar:3.0.14]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
> {noformat}
> The accumulated incompleted SSTables could take lots of space, especially for
> STCS which could have very large SSTables.
> Here is the script we use to delete the SSTables after node is restarted:
> {noformat}
> grep 'Stats component is missing for sstable' $SYSTEM_LOG | awk '{print $8}'
> > ~/invalid_sstables ; for ss in `cat ~/invalid_sstables`; do echo == $ss; ll
> $ss*; sudo rm $ss* ; done
> {noformat}
> I would suggest to remove these incompleted SSTables while startup.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
