[
https://issues.apache.org/jira/browse/CASSANDRA-7872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160039#comment-14160039
]
Oleg Anastasyev edited comment on CASSANDRA-7872 at 10/6/14 7:01 AM:
---------------------------------------------------------------------
Pavel, this is not happens to work, this is just how CMS work since the very
beginning. The weak proc stage is explaned in this old blog post of Oracle back
in 2006. https://blogs.oracle.com/poonam/entry/understanding_cms_gc_logs for
example. If you google, you can find more. If you check out the code of the
hotspot, you can find it there. If youre curious, you can find this stage on
G1 mixed cycle as well.
I can hardly believe that Oracle will ever remove weak refs processing from
concurrent GC cycles. This is just meaningless.
Just because you did not made PhRefs work for some purpose in past does not
mean it cannot be used for all purposes ever. In your case, you arguing with,
this could be JVM with a bug or you may just had bug in your code, which
prevented PhRef to work as intended. Until you specify exact version of JDK you
was using, it is hard to tell was it bug, behavior of that exact JVM or just
bug in your code.
By the way, claiming that "FullGC never happens on C*" as an argument against
#3 is not legal as well, because you cannot point out some "spec like JMM",
that Full GC never happens on CMS as well. Moreover, it is explicitly known,
that it does happen on any CMS instance, regardless of how hard you try. So
even in worst case, using that JDK, version of which you cannot recon now, with
possibly broken standard JVM behavior, it will delete files from disk earlier
than restart of a node. And this is the exact purpose of #3 - just to remove it
earlier that restart.
Currently, on 2.0 on a 64 node cluster we have is a couple of tens sstable
removals performed by PhRef queue thread instead of a ref counting algo. (this
all happens at night repairs, so i suspect a bug there). So not having #3 you
are not and will not be even aware of the problem.
was (Author: m0nstermind):
Pavel, this is not happens to work, this is just how CMS work since the very
beginning. The weak proc stage is explaned in this old blog post of Oracle back
in 2006. https://blogs.oracle.com/poonam/entry/understanding_cms_gc_logs for
example. If you google, you can find more. If you check out the code of the
hotspot, you can find it there. If youre curious, you can find this stage on
G1 mixed cycle as well.
I can hardly believe that Oracle will ever remove weak refs processing from
concurrent GC cycles. This is just meaningless.
Just because you did not made PhRefs work for some purpose in past does not
mean it cannot be used for all purposes ever. In your case, you arguing with,
this could be JVM with a bug or you may just had bug in your code, which
prevented PhRef to work as intended. Until you specify exact version of JDK you
was using, it is hard to tell was it bug, behavior of that exact JVM or just
bug in your code.
By the way, claiming that "FullGC never happens on C*" as an argument against
#3 is not legal as well, because you cannot point out some "spec like JMM",
that Full GC never happens on CMS as well. Moreover, it is explicitly known,
that it does happen on any CMS instance, regardless of how hard you try. So
even in worst case, using that JDK, version of which you cannot recon now, with
possibly broken standard JVM behavior, it will delete files from disk earlier
than restart of a node. And this is the exact purpose of #3 - just to remove it
earlier that restart.
> ensure compacted obsolete sstables are not open on node restart and nodetool
> refresh, even on sstable reference miscounting or deletion tasks are failed.
> ---------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-7872
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7872
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Oleg Anastasyev
> Assignee: Oleg Anastasyev
> Fix For: 2.0.11
>
> Attachments: EnsureNoObsoleteSSTables-7872-v2.0.txt
>
>
> Since CASSANDRA-4436 compacted sstables are no more marked with
> COMPACTED_MARKER file. Instead after they are compacted, DataTracker calls
> SSTableReader.markObsolete(), but the actual deletion is happening later on
> SSTableReader.releaseReference().
> This reference counting is very fragile, it is very easy to introduce a
> hard-to-catch and rare bug, so this reference count never reaches 0 ( like
> CASSANDRA-6503 for example )
> This means, that very rarely obsolete sstable files are not removed from disk
> (but are not used anymore by cassandra to read data).
> If more than gc grace time has passed since sstable file was not removed from
> disk and operator issues either nodetool refresh or just reboots a node,
> these obsolete files are being discovered and open for read by a node. So
> deleted data is resurrected, being quickly spread by RR to whole cluster.
> Because consequences are very serious (even a single not removed obsolete
> sstable file could render your data useless) this patch makes sure no
> obsolete sstable file can be open for read by:
> 1. Removing sstables on CFS init analyzing sstable generations (sstable is
> removed, if there are another sstable, listing this as ancestor)
> 2. Reimplementing COMPACTED_MARKER file for sstable. This marker is created
> as soon as markObsolete is called. This is neccessary b/c generation info can
> be lost (when sstables compact to none)
> 3. To remove sstables sooner then restart - reimplemented the good old GC
> phantom reference queue as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
