[
https://issues.apache.org/jira/browse/CASSANDRA-9486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14565056#comment-14565056
]
Benedict commented on CASSANDRA-9486:
-------------------------------------
FTR, looking at the thread dump I have from the user encountering this, it
looks like the "nothing but RTs" is exactly the problem.
It appears that we only call {{add()}} with an atom in the _outer_ loop (in
ColumnIndex), never in getReduced. If the tombstones always shadow the live
data, we never end up calling it, so we accumulate every tombstone. i.e., an
infinite length sequence of (tombstone, data) is flattened into an infinite
accumulation of tombstone. The problem here is that this is exactly the
workload characteristic of collection overwrites, which is what this user is
doing (as are presumably many others).
> LazilyCompactedRow accumulates all expired RangeTombstones
> ----------------------------------------------------------
>
> Key: CASSANDRA-9486
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9486
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Benedict
> Assignee: Marcus Eriksson
> Fix For: 3.x, 2.1.x, 2.0.x, 2.2.x, 1.2.x
>
> Attachments: 0001-9486.patch
>
>
> LazilyCompactedRow initializes a ColumnIndex.Builder to use its
> RangeTombstone.Tracker, but it only calls update() with a RT argument, never
> an atom. The Tracker only ever _adds_ if it receives a RT, never removes. So
> all the RT ever seen for the partition (that have expired) remain in memory
> until the compaction completes. To make matters worse, this then forces a
> linear scan of all of these RT for each live cell we add, so this extra load
> hangs around for a long time, and compactions stall.
> This issue is biting one of our users badly (at least, it seems likely to be
> this issue), and there may be others. This user is not even making use of RT
> extensively themselves, only collections (presumably with a complete
> overwrite of the contents of the collection, resulting in a RT being
> generated).
> Probably the best solution is to make the RT addition itself remove any
> already present that are no longer helpful.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
