[
https://issues.apache.org/jira/browse/CASSANDRA-11548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15246776#comment-15246776
]
Paulo Motta commented on CASSANDRA-11548:
-----------------------------------------
Good catch! I think this makes sense and there could be a race where an sstable
is removed from the compacting set during anti-compaction, and meanwhile is
removed from the {{DataTracker}} by a concurrent compaction, and when
{{cfs.getDataTracker().markCompactedSSTablesReplaced(successfullyAntiCompactedSSTables,
anticompactedSSTables, OperationType.ANTICOMPACTION);}} is called at the end
of the anti-compaction, the {{java.lang.AssertionError: Expecting new size of
95, got 96 while replacing XXX by XXX}} is thrown, causing the sstable to be
silently not replaced after compaction, but still picked up by a subsequent
repair and cause the {{received out of order AssertionError}} on CASSANDRA-9935.
On 2.2+ this is fixed by CASSANDRA-8568, that extended the transactional API to
sstables so no need to the apply the patch there.
I did some minor update on the patch (removed unnecessary map
{{successfullyAntiCompactedSSTables}}) and added a unit test to reproduce the
issue. CI tests will be available shortly below:
||2.1||
|[branch|https://github.com/apache/cassandra/compare/cassandra-2.1...pauloricardomg:2.1-11548]|
|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-2.1-11548-testall/lastCompletedBuild/testReport/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-2.1-11548-dtest/lastCompletedBuild/testReport/]|
[~krummas] could you have a quick look to double check this makes sense and we
haven't missed anything?
While this will fix the root cause of the issue, on CASSANDRA-9935 I will add a
protection to prevent the {{received out of order AssertionError}} when a bug
like this causes an sstable not to be removed correctly.
> Anticompaction not removing old sstables
> ----------------------------------------
>
> Key: CASSANDRA-11548
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11548
> Project: Cassandra
> Issue Type: Bug
> Environment: 2.1.13
> Reporter: Ruoran Wang
> Attachments: 0001-cassandra-2.1.13-potential-fix.patch
>
>
> 1. 12/29/15 https://issues.apache.org/jira/browse/CASSANDRA-10831
> Moved markCompactedSSTablesReplaced out of the loop ```for (SSTableReader
> sstable : repairedSSTables)```
> 2. 1/18/16 https://issues.apache.org/jira/browse/CASSANDRA-10829
> Added unmarkCompacting into the loop. ```for (SSTableReader sstable :
> repairedSSTables)```
> I think the effect of those above change might cause the
> markCompactedSSTablesReplaced fail on
> DataTracker.java
> {noformat}
> assert newSSTables.size() + newShadowed.size() == newSSTablesSize :
> String.format("Expecting new size of %d, got %d while
> replacing %s by %s in %s",
> newSSTablesSize, newSSTables.size() +
> newShadowed.size(), oldSSTables, replacements, this);
> {noformat}
> Since change CASSANDRA-10831 moved it out. This AssertError won't be caught,
> leaving the oldsstables not removed. (Then this might cause row out of order
> error when doing incremental repair if there are L1 un-repaired sstables.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
