[jira] [Commented] (CASSANDRA-11548) Anticompaction not removing old sstables
[
https://issues.apache.org/jira/browse/CASSANDRA-11548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256471#comment-15256471
]
Paulo Motta commented on CASSANDRA-11548:
-
vote was already called so it should be out in the next few days (this week for
sure)
> Anticompaction not removing old sstables
>
>
> Key: CASSANDRA-11548
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11548
> Project: Cassandra
> Issue Type: Bug
> Environment: 2.1.13
>Reporter: Ruoran Wang
>Assignee: Ruoran Wang
> Fix For: 2.1.14
>
> Attachments: 0001-cassandra-2.1.13-potential-fix.patch
>
>
> 1. 12/29/15 https://issues.apache.org/jira/browse/CASSANDRA-10831
> Moved markCompactedSSTablesReplaced out of the loop ```for (SSTableReader
> sstable : repairedSSTables)```
> 2. 1/18/16 https://issues.apache.org/jira/browse/CASSANDRA-10829
> Added unmarkCompacting into the loop. ```for (SSTableReader sstable :
> repairedSSTables)```
> I think the effect of those above change might cause the
> markCompactedSSTablesReplaced fail on
> DataTracker.java
> {noformat}
>assert newSSTables.size() + newShadowed.size() == newSSTablesSize :
> String.format("Expecting new size of %d, got %d while
> replacing %s by %s in %s",
> newSSTablesSize, newSSTables.size() +
> newShadowed.size(), oldSSTables, replacements, this);
> {noformat}
> Since change CASSANDRA-10831 moved it out. This AssertError won't be caught,
> leaving the oldsstables not removed. (Then this might cause row out of order
> error when doing incremental repair if there are L1 un-repaired sstables.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11548) Anticompaction not removing old sstables
[
https://issues.apache.org/jira/browse/CASSANDRA-11548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15251053#comment-15251053
]
Ruoran Wang commented on CASSANDRA-11548:
-
Thank you [~pauloricardomg]. May I ask about the release schedule for 2.1.14?
> Anticompaction not removing old sstables
>
>
> Key: CASSANDRA-11548
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11548
> Project: Cassandra
> Issue Type: Bug
> Environment: 2.1.13
>Reporter: Ruoran Wang
>Assignee: Ruoran Wang
> Fix For: 2.1.14
>
> Attachments: 0001-cassandra-2.1.13-potential-fix.patch
>
>
> 1. 12/29/15 https://issues.apache.org/jira/browse/CASSANDRA-10831
> Moved markCompactedSSTablesReplaced out of the loop ```for (SSTableReader
> sstable : repairedSSTables)```
> 2. 1/18/16 https://issues.apache.org/jira/browse/CASSANDRA-10829
> Added unmarkCompacting into the loop. ```for (SSTableReader sstable :
> repairedSSTables)```
> I think the effect of those above change might cause the
> markCompactedSSTablesReplaced fail on
> DataTracker.java
> {noformat}
>assert newSSTables.size() + newShadowed.size() == newSSTablesSize :
> String.format("Expecting new size of %d, got %d while
> replacing %s by %s in %s",
> newSSTablesSize, newSSTables.size() +
> newShadowed.size(), oldSSTables, replacements, this);
> {noformat}
> Since change CASSANDRA-10831 moved it out. This AssertError won't be caught,
> leaving the oldsstables not removed. (Then this might cause row out of order
> error when doing incremental repair if there are L1 un-repaired sstables.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11548) Anticompaction not removing old sstables
[
https://issues.apache.org/jira/browse/CASSANDRA-11548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15248413#comment-15248413
]
Paulo Motta commented on CASSANDRA-11548:
-
Thanks [~krummas]. Good job [~ruoranwang]!
> Anticompaction not removing old sstables
>
>
> Key: CASSANDRA-11548
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11548
> Project: Cassandra
> Issue Type: Bug
> Environment: 2.1.13
>Reporter: Ruoran Wang
>Assignee: Ruoran Wang
> Fix For: 2.1.14
>
> Attachments: 0001-cassandra-2.1.13-potential-fix.patch
>
>
> 1. 12/29/15 https://issues.apache.org/jira/browse/CASSANDRA-10831
> Moved markCompactedSSTablesReplaced out of the loop ```for (SSTableReader
> sstable : repairedSSTables)```
> 2. 1/18/16 https://issues.apache.org/jira/browse/CASSANDRA-10829
> Added unmarkCompacting into the loop. ```for (SSTableReader sstable :
> repairedSSTables)```
> I think the effect of those above change might cause the
> markCompactedSSTablesReplaced fail on
> DataTracker.java
> {noformat}
>assert newSSTables.size() + newShadowed.size() == newSSTablesSize :
> String.format("Expecting new size of %d, got %d while
> replacing %s by %s in %s",
> newSSTablesSize, newSSTables.size() +
> newShadowed.size(), oldSSTables, replacements, this);
> {noformat}
> Since change CASSANDRA-10831 moved it out. This AssertError won't be caught,
> leaving the oldsstables not removed. (Then this might cause row out of order
> error when doing incremental repair if there are L1 un-repaired sstables.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11548) Anticompaction not removing old sstables
[
https://issues.apache.org/jira/browse/CASSANDRA-11548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15247563#comment-15247563
]
Paulo Motta commented on CASSANDRA-11548:
-
I prepared for commit on 2.1.15 on two commits (to preserve authors), please
hold commit until there is a decision if this can still go on 2.1.14 (which is
already tagged).
Also, when you have time could you have a look on CASSANDRA-9935 which is
complementary to this?
Thank you [~krummas]!
> Anticompaction not removing old sstables
>
>
> Key: CASSANDRA-11548
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11548
> Project: Cassandra
> Issue Type: Bug
> Environment: 2.1.13
>Reporter: Ruoran Wang
> Attachments: 0001-cassandra-2.1.13-potential-fix.patch
>
>
> 1. 12/29/15 https://issues.apache.org/jira/browse/CASSANDRA-10831
> Moved markCompactedSSTablesReplaced out of the loop ```for (SSTableReader
> sstable : repairedSSTables)```
> 2. 1/18/16 https://issues.apache.org/jira/browse/CASSANDRA-10829
> Added unmarkCompacting into the loop. ```for (SSTableReader sstable :
> repairedSSTables)```
> I think the effect of those above change might cause the
> markCompactedSSTablesReplaced fail on
> DataTracker.java
> {noformat}
>assert newSSTables.size() + newShadowed.size() == newSSTablesSize :
> String.format("Expecting new size of %d, got %d while
> replacing %s by %s in %s",
> newSSTablesSize, newSSTables.size() +
> newShadowed.size(), oldSSTables, replacements, this);
> {noformat}
> Since change CASSANDRA-10831 moved it out. This AssertError won't be caught,
> leaving the oldsstables not removed. (Then this might cause row out of order
> error when doing incremental repair if there are L1 un-repaired sstables.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11548) Anticompaction not removing old sstables
[
https://issues.apache.org/jira/browse/CASSANDRA-11548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15247292#comment-15247292
]
Marcus Eriksson commented on CASSANDRA-11548:
-
It looks good, +1, could you prepare it for commit and I'll push it?
> Anticompaction not removing old sstables
>
>
> Key: CASSANDRA-11548
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11548
> Project: Cassandra
> Issue Type: Bug
> Environment: 2.1.13
>Reporter: Ruoran Wang
> Attachments: 0001-cassandra-2.1.13-potential-fix.patch
>
>
> 1. 12/29/15 https://issues.apache.org/jira/browse/CASSANDRA-10831
> Moved markCompactedSSTablesReplaced out of the loop ```for (SSTableReader
> sstable : repairedSSTables)```
> 2. 1/18/16 https://issues.apache.org/jira/browse/CASSANDRA-10829
> Added unmarkCompacting into the loop. ```for (SSTableReader sstable :
> repairedSSTables)```
> I think the effect of those above change might cause the
> markCompactedSSTablesReplaced fail on
> DataTracker.java
> {noformat}
>assert newSSTables.size() + newShadowed.size() == newSSTablesSize :
> String.format("Expecting new size of %d, got %d while
> replacing %s by %s in %s",
> newSSTablesSize, newSSTables.size() +
> newShadowed.size(), oldSSTables, replacements, this);
> {noformat}
> Since change CASSANDRA-10831 moved it out. This AssertError won't be caught,
> leaving the oldsstables not removed. (Then this might cause row out of order
> error when doing incremental repair if there are L1 un-repaired sstables.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11548) Anticompaction not removing old sstables
[
https://issues.apache.org/jira/browse/CASSANDRA-11548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15246776#comment-15246776
]
Paulo Motta commented on CASSANDRA-11548:
-
Good catch! I think this makes sense and there could be a race where an sstable
is removed from the compacting set during anti-compaction, and meanwhile is
removed from the {{DataTracker}} by a concurrent compaction, and when
{{cfs.getDataTracker().markCompactedSSTablesReplaced(successfullyAntiCompactedSSTables,
anticompactedSSTables, OperationType.ANTICOMPACTION);}} is called at the end
of the anti-compaction, the {{java.lang.AssertionError: Expecting new size of
95, got 96 while replacing XXX by XXX}} is thrown, causing the sstable to be
silently not replaced after compaction, but still picked up by a subsequent
repair and cause the {{received out of order AssertionError}} on CASSANDRA-9935.
On 2.2+ this is fixed by CASSANDRA-8568, that extended the transactional API to
sstables so no need to the apply the patch there.
I did some minor update on the patch (removed unnecessary map
{{successfullyAntiCompactedSSTables}}) and added a unit test to reproduce the
issue. CI tests will be available shortly below:
||2.1||
|[branch|https://github.com/apache/cassandra/compare/cassandra-2.1...pauloricardomg:2.1-11548]|
|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-2.1-11548-testall/lastCompletedBuild/testReport/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-2.1-11548-dtest/lastCompletedBuild/testReport/]|
[~krummas] could you have a quick look to double check this makes sense and we
haven't missed anything?
While this will fix the root cause of the issue, on CASSANDRA-9935 I will add a
protection to prevent the {{received out of order AssertionError}} when a bug
like this causes an sstable not to be removed correctly.
> Anticompaction not removing old sstables
>
>
> Key: CASSANDRA-11548
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11548
> Project: Cassandra
> Issue Type: Bug
> Environment: 2.1.13
>Reporter: Ruoran Wang
> Attachments: 0001-cassandra-2.1.13-potential-fix.patch
>
>
> 1. 12/29/15 https://issues.apache.org/jira/browse/CASSANDRA-10831
> Moved markCompactedSSTablesReplaced out of the loop ```for (SSTableReader
> sstable : repairedSSTables)```
> 2. 1/18/16 https://issues.apache.org/jira/browse/CASSANDRA-10829
> Added unmarkCompacting into the loop. ```for (SSTableReader sstable :
> repairedSSTables)```
> I think the effect of those above change might cause the
> markCompactedSSTablesReplaced fail on
> DataTracker.java
> {noformat}
>assert newSSTables.size() + newShadowed.size() == newSSTablesSize :
> String.format("Expecting new size of %d, got %d while
> replacing %s by %s in %s",
> newSSTablesSize, newSSTables.size() +
> newShadowed.size(), oldSSTables, replacements, this);
> {noformat}
> Since change CASSANDRA-10831 moved it out. This AssertError won't be caught,
> leaving the oldsstables not removed. (Then this might cause row out of order
> error when doing incremental repair if there are L1 un-repaired sstables.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
