[
https://issues.apache.org/jira/browse/CASSANDRA-6008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849672#comment-13849672
]
Nikolai Grigoriev commented on CASSANDRA-6008:
----------------------------------------------
I am wondering if it is possible that because of this problem I ended up with
this
(http://stackoverflow.com/questions/20589324/cassandra-2-0-3-endless-compactions-with-no-traffic)
issue.
I am constantly having this "This should never happen" problem with I restart
my 2.0.3 cluster. Out of 6 nodes, if I restart it now for sure at least 2 will
fail to start because of this condition. And to allow them to start I wipe the
contents of system.compactions_in_progress table and delete all
compactions_in_progress directories under my data directories on the node
affected.
> Getting 'This should never happen' error at startup due to sstables missing
> ---------------------------------------------------------------------------
>
> Key: CASSANDRA-6008
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6008
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: John Carrino
> Assignee: Tyler Hobbs
> Fix For: 2.0.4
>
> Attachments: 6008-2.0-part2.patch, 6008-2.0-v1.patch,
> 6008-trunk-v1.patch
>
>
> Exception encountered during startup: "Unfinished compactions reference
> missing sstables. This should never happen since compactions are marked
> finished before we start removing the old sstables"
> This happens when sstables that have been compacted away are removed, but
> they still have entries in the system.compactions_in_progress table.
> Normally this should not happen because the entries in
> system.compactions_in_progress are deleted before the old sstables are
> deleted.
> However at startup recovery time, old sstables are deleted (NOT BEFORE they
> are removed from the compactions_in_progress table) and then after that is
> done it does a truncate using SystemKeyspace.discardCompactionsInProgress
> We ran into a case where the disk filled up and the node died and was bounced
> and then failed to truncate this table on startup, and then got stuck hitting
> this exception in ColumnFamilyStore.removeUnfinishedCompactionLeftovers.
> Maybe on startup we can delete from this table incrementally as we clean
> stuff up in the same way that compactions delete from this table before they
> delete old sstables.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
