[
https://issues.apache.org/jira/browse/CASSANDRA-7872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14159188#comment-14159188
]
Oleg Anastasyev commented on CASSANDRA-7872:
--------------------------------------------
Do you mean stop the world full gc ? Phantom Queue is cleaned up at CMS cycle
on java 6.
Since java 7 (i am not sure about since what exact build number, but at least
on 7_60 it works) it is cleaned either on ParNew or CMS cycle - as soon as
reference is GCed.
Here is a log snippet from production demonstrating it on ParNew for example:
{code}
04:45:40,505 org.apache.cassandra.db.Memtable$FlushRunnable Completed flushing
/mnt/db
3/system/compactions_in_progress/system-compactions_in_progress-jb-1489409-Data.db
(42 bytes) for commitlog position ReplayPosition(segmentI
d=1410945220011, position=86495573)
04:45:40,670 org.apache.cassandra.db.Memtable$FlushRunnable Completed flushing
/mnt/db
3/system/compactions_in_progress/system-compactions_in_progress-jb-1489410-Data.db
(152 bytes) for commitlog position ReplayPosition(segment
Id=1410945220011, position=86503582)
04:45:40,672 org.apache.cassandra.db.compaction.CompactionTask Compacting
[SSTableRead
er(path='/mnt/db3/system/compactions_in_progress/system-compactions_in_progress-jb-1489409-Data.db'),
SSTableReader(path='/mnt/db3/system/co
mpactions_in_progress/system-compactions_in_progress-jb-1489410-Data.db'),
SSTableReader(path='/mnt/db3/system/compactions_in_progress/syste
m-compactions_in_progress-jb-1489407-Data.db'),
SSTableReader(path='/mnt/db1/system/compactions_in_progress/system-compactions_in_progress-j
b-1489408-Data.db')]
Oct 4 04:45:40 [GC[ParNew
Oct 4 04:45:40 Desired survivor size 268435456 bytes, new threshold 3 (max 3)
Oct 4 04:45:40 - age 1: 12869048 bytes, 12869048 total
Oct 4 04:45:40 - age 2: 3157328 bytes, 16026376 total
Oct 4 04:45:40 - age 3: 1676208 bytes, 17702584 total
Oct 4 04:45:40 : 4206407K->21161K(4718592K), 0.0428280 secs]
7744224K->3560919K(28835840K), 0.0462120
secs] [Times: user=0.55 sys=0.00, real=0.05 secs]
04:45:40,786
org.apache.cassandra.io.sstable.SSTableDeletingQueue$SSTableDeletingReference$1
Obsolete SSTable
/mnt/db3/system/compactions_in_progress/system-compactions_in_progress-jb-1489409
not used anymore. Its reference co
unting were broken or its deletion task is stuck. Forcing its remove now
04:45:40,787 org.apache.cassandra.io.sstable.SSTable Deleted
/mnt/db3/system/compactio
ns_in_progress/system-compactions_in_progress-jb-1489409
04:45:40,788
org.apache.cassandra.io.sstable.SSTableDeletingQueue$SSTableDeletingRefer
ence$1 Obsolete SSTable
/mnt/db3/system/compactions_in_progress/system-compactions_in_progress-jb-1489410
not used anymore. Its reference counting were broken or its deletion task is
stuck. Forcing its remove now
04:45:40,788 org.apache.cassandra.io.sstable.SSTable Deleted
/mnt/db3/system/compactio
ns_in_progress/system-compactions_in_progress-jb-1489410
{code}
compactions in progress ss table reader was created and dereferenced between 2
parnew collections, so its ref is cleaned on ParNew. If sstable reader
reference survives new gen collection its phantom will be cleaned on closes CMS
cycle, which clean its reference from heap.
> ensure compacted obsolete sstables are not open on node restart and nodetool
> refresh, even on sstable reference miscounting or deletion tasks are failed.
> ---------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-7872
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7872
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Oleg Anastasyev
> Assignee: Oleg Anastasyev
> Fix For: 2.0.11
>
> Attachments: EnsureNoObsoleteSSTables-7872-v2.0.txt
>
>
> Since CASSANDRA-4436 compacted sstables are no more marked with
> COMPACTED_MARKER file. Instead after they are compacted, DataTracker calls
> SSTableReader.markObsolete(), but the actual deletion is happening later on
> SSTableReader.releaseReference().
> This reference counting is very fragile, it is very easy to introduce a
> hard-to-catch and rare bug, so this reference count never reaches 0 ( like
> CASSANDRA-6503 for example )
> This means, that very rarely obsolete sstable files are not removed from disk
> (but are not used anymore by cassandra to read data).
> If more than gc grace time has passed since sstable file was not removed from
> disk and operator issues either nodetool refresh or just reboots a node,
> these obsolete files are being discovered and open for read by a node. So
> deleted data is resurrected, being quickly spread by RR to whole cluster.
> Because consequences are very serious (even a single not removed obsolete
> sstable file could render your data useless) this patch makes sure no
> obsolete sstable file can be open for read by:
> 1. Removing sstables on CFS init analyzing sstable generations (sstable is
> removed, if there are another sstable, listing this as ancestor)
> 2. Reimplementing COMPACTED_MARKER file for sstable. This marker is created
> as soon as markObsolete is called. This is neccessary b/c generation info can
> be lost (when sstables compact to none)
> 3. To remove sstables sooner then restart - reimplemented the good old GC
> phantom reference queue as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
