Oleg Anastasyev created CASSANDRA-7872:
------------------------------------------
Summary: ensure compacted obsolete sstables are not open on node
restart and nodetool refresh, even on sstable reference miscounting or deletion
tasks are failed.
Key: CASSANDRA-7872
URL: https://issues.apache.org/jira/browse/CASSANDRA-7872
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Oleg Anastasyev
Since CASSANDRA-4436 compacted sstables are no more marked with
COMPACTED_MARKER file. Instead after they are compacted, DataTracker calls
SSTableReader.markObsolete(), but the actual deletion is happening later on
SSTableReader.releaseReference().
This reference counting is very fragile, it is very easy to introduce a
hard-to-catch and rare bug, so this reference count never reaches 0 ( like
CASSANDRA-6503 for example )
This means, that very rarely obsolete sstable files are not removed from disk
(but are not used anymore by cassandra to read data).
If more than gc grace time has passed since sstable file was not removed from
disk and operator issues either nodetool refresh or just reboots a node, these
obsolete files are being discovered and open for read by a node. So deleted
data is resurrected, being quickly spread by RR to whole cluster.
Because consequences are very serious (even a single not removed obsolete
sstable file could render your data useless) this patch makes sure no obsolete
sstable file can be open for read by:
1. Removing sstables on CFS init analyzing sstable generations (sstable is
removed, if there are another sstable, listing this as ancestor)
2. Reimplementing COMPACTED_MARKER file for sstable. This marker is created as
soon as markObsolete is called. This is neccessary b/c generation info can be
lost (when sstables compact to none)
3. To remove sstables sooner then restart - reimplemented the good old GC
phantom reference queue as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
