[
https://issues.apache.org/jira/browse/CASSANDRA-14308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16407273#comment-16407273
]
Jay Zhuang commented on CASSANDRA-14308:
----------------------------------------
Here is one proposal Option1: delete SSTables without stats component:
[14308-wip|https://github.com/cooldoger/cassandra/tree/14308-wip], but not sure
if it's safe to delete these SSTables and may cause data lost if there's a bug.
Option2: as the incompleted SSTables are generated by compaction, how about
storing the new SSTable in a temporary directory (e.g.:
{{data/keyspace/table-id/compaction/}}). Once the compaction is done, move the
new SSTable back to the data directory. So while Cassandra startup, it can
clean up SSTables in the temporary directory (if there's any interrupted
compaction). I'm not familar with compaction, not sure if it's possible or the
right way to do it. Any suggestions are welcomed.
cc @[~jasobrown], @[~jjirsa]
> Remove invalid SSTables from interrupted compaction
> ---------------------------------------------------
>
> Key: CASSANDRA-14308
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14308
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Reporter: Jay Zhuang
> Assignee: Jay Zhuang
> Priority: Minor
>
> If the JVM crash while compaction is in progress, the incompleted SSTable
> won't be cleaned up, which causes "{{Stats component is missing for
> sstable}}" error in the startup log:
> {noformat}
> ERROR [SSTableBatchOpen:3] 2018-03-11 00:17:35,597 CassandraDaemon.java:207 -
> Exception in thread Thread[SSTableBatchOpen:3,5,main]
> java.lang.AssertionError: Stats component is missing for sstable
> /cassandra/data/keyspace/table-id/mc-12345-big
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:458)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:374)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:533)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_121]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_121]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_121]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_121]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> [apache-cassandra-3.0.14.jar:3.0.14]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
> {noformat}
> The accumulated incompleted SSTables could take lots of space, especially for
> STCS which could have very large SSTables.
> Here is the script we use to delete the SSTables after node is restarted:
> {noformat}
> grep 'Stats component is missing for sstable' $SYSTEM_LOG | awk '{print $8}'
> > ~/invalid_sstables ; for ss in `cat ~/invalid_sstables`; do echo == $ss; ll
> $ss*; sudo rm $ss* ; done
> {noformat}
> I would suggest to remove these incompleted SSTables while startup.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
