[
https://issues.apache.org/jira/browse/CASSANDRA-9486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14569721#comment-14569721
]
Brent Haines commented on CASSANDRA-9486:
-----------------------------------------
FWIW -- I created a patch from [~slebresne]'s proposed fix, tested it locally
and tried it on one of our impacted nodes in our data cluster. We are running
2.1.5, but the variance was easy to sift through.
Compactions that used to take a day finished in 30 seconds. We haven't seen any
spikes in heap space either, but that will take more time to evaluate. I do not
know if there are any residual or unexpected behaviors resulting from the patch
yet. We'll leave it in place for a few days to see. In the mean time, I thought
it might help to confirm that this does indeed seem to fix our issue.
Now, if we can stop creating so many RT's in the first place... :)
> LazilyCompactedRow accumulates all expired RangeTombstones
> ----------------------------------------------------------
>
> Key: CASSANDRA-9486
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9486
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Benedict
> Assignee: Sylvain Lebresne
> Priority: Critical
> Fix For: 3.x, 2.1.x, 2.0.x, 2.2.x, 1.2.x
>
> Attachments: 0001-9486.patch
>
>
> LazilyCompactedRow initializes a ColumnIndex.Builder to use its
> RangeTombstone.Tracker, but it only calls update() with a RT argument, never
> an atom. The Tracker only ever _adds_ if it receives a RT, never removes. So
> all the RT ever seen for the partition (that have expired) remain in memory
> until the compaction completes. To make matters worse, this then forces a
> linear scan of all of these RT for each live cell we add, so this extra load
> hangs around for a long time, and compactions stall.
> This issue is biting one of our users badly (at least, it seems likely to be
> this issue), and there may be others. This user is not even making use of RT
> extensively themselves, only collections (presumably with a complete
> overwrite of the contents of the collection, resulting in a RT being
> generated).
> Probably the best solution is to make the RT addition itself remove any
> already present that are no longer helpful.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
