[
https://issues.apache.org/jira/browse/CASSANDRA-21649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Caleb Rackliffe updated CASSANDRA-21649:
----------------------------------------
Reviewers: Caleb Rackliffe
> Ensure SSTables are deleted in order of last-updated timestamp
> --------------------------------------------------------------
>
> Key: CASSANDRA-21649
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21649
> Project: Apache Cassandra
> Issue Type: Bug
> Reporter: Nivy Kani
> Assignee: Nivy Kani
> Priority: Normal
> Attachments: ci_summary.html, result_details.tar.gz
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When a node restarts after crashing, we need to clean up any
> partially-deleted SSTables, but we do a safety check on REMOVE log records to
> verify that the last-modified timestamps match the actual files on disk. If
> the timestamps don’t match, we throw an error that the log is corrupted, and
> the node doesn’t start.
>
> The comment in LogFile.verifyRecord() states: "Because we delete files from
> oldest to newest, the latest update time should always match.” But this isn’t
> always true; regular SSTable deletion after compaction always deletes DATA
> files first, and doesn’t explicitly set the deletion order of other files.
>
> So we can have this scenario:
> 1. SSTable files are written, and there’s some tiny delay between the
> written-timestamp of each file.
> 2. Compaction succeeds. SSTable files can be deleted now, and we write a
> REMOVE record with the timestamp of the most-recently-updated file of each
> SSTable.
> 3. SSTableTidier starts deleting the SSTable files. Problem: it can delete
> the files in non-timestamp order.
> 4. Halfway through, the node crashes. On restart, LogFile.verifyRecord() goes
> through each REMOVE record to compare timestamps. It fails because the
> last-written file was deleted, so the most-recent-timestamp is different now.
> 5. Now the node won’t start up, and some SSTable files are leaked.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
