[jira] [Commented] (CASSANDRA-12203) AssertionError on compaction after upgrade (2.1.9 -> 3.7)
[
https://issues.apache.org/jira/browse/CASSANDRA-12203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130277#comment-16130277
]
R1J2 commented on CASSANDRA-12203:
--
Can you list out the steps you took for this upgrade please ?
> AssertionError on compaction after upgrade (2.1.9 -> 3.7)
> -
>
> Key: CASSANDRA-12203
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12203
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 3.7 (upgrade from 2.1.9)
> Java version "1.8.0_91"
> Ubuntu 14.04.4 LTS (GNU/Linux 3.13.0-83-generic x86_64)
>Reporter: Roman S. Borschel
>Assignee: Yuki Morishita
>Priority: Critical
> Fix For: 3.0.11, 3.10
>
>
> After upgrading a Cassandra cluster from 2.1.9 to 3.7, one column family
> (using SizeTieredCompaction) repeatedly and continuously failed compaction
> (and thus also repair) across the cluster, with all nodes producing the
> following errors in the logs:
> {noformat}
> 016-07-14T09:29:47.96855 |srv=cassandra|ERROR: Exception in thread
> Thread[CompactionExecutor:3,1,main]
> 2016-07-14T09:29:47.96858 |srv=cassandra|java.lang.AssertionError: null
> 2016-07-14T09:29:47.96859 |srv=cassandra| at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$TombstoneTracker.openNew(UnfilteredDeserializer.java:650)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96860 |srv=cassandra| at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.hasNext(UnfilteredDeserializer.java:423)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96860 |srv=cassandra| at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.hasNext(UnfilteredDeserializer.java:298)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96860 |srv=cassandra| at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.readStaticRow(SSTableSimpleIterator.java:133)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96861 |srv=cassandra| at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:57)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96861 |srv=cassandra| at
> org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator$1.initializeIterator(BigTableScanner.java:334)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96862 |srv=cassandra| at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96862 |srv=cassandra| at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:70)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96863 |srv=cassandra| at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:109)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96863 |srv=cassandra| at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:100)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96864 |srv=cassandra| at
> org.apache.cassandra.utils.MergeIterator$Candidate.consume(MergeIterator.java:408)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96864 |srv=cassandra| at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:203)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96865 |srv=cassandra| at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:156)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96865 |srv=cassandra| at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96866 |srv=cassandra| at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$2.hasNext(UnfilteredPartitionIterators.java:150)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96866 |srv=cassandra| at
> org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:72)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96867 |srv=cassandra| at
> org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:226)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96867 |srv=cassandra| at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:182)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96867 |srv=cassandra| at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-3.7.jar:3.7]
>
[jira] [Commented] (CASSANDRA-12203) AssertionError on compaction after upgrade (2.1.9 -> 3.7)
[
https://issues.apache.org/jira/browse/CASSANDRA-12203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15823665#comment-15823665
]
Sylvain Lebresne commented on CASSANDRA-12203:
--
Thanks, +1.
> AssertionError on compaction after upgrade (2.1.9 -> 3.7)
> -
>
> Key: CASSANDRA-12203
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12203
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 3.7 (upgrade from 2.1.9)
> Java version "1.8.0_91"
> Ubuntu 14.04.4 LTS (GNU/Linux 3.13.0-83-generic x86_64)
>Reporter: Roman S. Borschel
>Assignee: Yuki Morishita
>Priority: Critical
> Fix For: 3.0.x, 3.x
>
>
> After upgrading a Cassandra cluster from 2.1.9 to 3.7, one column family
> (using SizeTieredCompaction) repeatedly and continuously failed compaction
> (and thus also repair) across the cluster, with all nodes producing the
> following errors in the logs:
> {noformat}
> 016-07-14T09:29:47.96855 |srv=cassandra|ERROR: Exception in thread
> Thread[CompactionExecutor:3,1,main]
> 2016-07-14T09:29:47.96858 |srv=cassandra|java.lang.AssertionError: null
> 2016-07-14T09:29:47.96859 |srv=cassandra| at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$TombstoneTracker.openNew(UnfilteredDeserializer.java:650)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96860 |srv=cassandra| at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.hasNext(UnfilteredDeserializer.java:423)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96860 |srv=cassandra| at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.hasNext(UnfilteredDeserializer.java:298)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96860 |srv=cassandra| at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.readStaticRow(SSTableSimpleIterator.java:133)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96861 |srv=cassandra| at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:57)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96861 |srv=cassandra| at
> org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator$1.initializeIterator(BigTableScanner.java:334)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96862 |srv=cassandra| at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96862 |srv=cassandra| at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:70)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96863 |srv=cassandra| at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:109)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96863 |srv=cassandra| at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:100)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96864 |srv=cassandra| at
> org.apache.cassandra.utils.MergeIterator$Candidate.consume(MergeIterator.java:408)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96864 |srv=cassandra| at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:203)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96865 |srv=cassandra| at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:156)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96865 |srv=cassandra| at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96866 |srv=cassandra| at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$2.hasNext(UnfilteredPartitionIterators.java:150)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96866 |srv=cassandra| at
> org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:72)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96867 |srv=cassandra| at
> org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:226)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96867 |srv=cassandra| at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:182)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96867 |srv=cassandra| at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96868 |srv=cassandra|
[jira] [Commented] (CASSANDRA-12203) AssertionError on compaction after upgrade (2.1.9 -> 3.7)
[
https://issues.apache.org/jira/browse/CASSANDRA-12203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822422#comment-15822422
]
Yuki Morishita commented on CASSANDRA-12203:
Thanks for taking a look at this.
Changed assertion as suggested.
||branch||testall||dtest||
|[12203-3.0|https://github.com/yukim/cassandra/tree/12203-3.0]|[testall|http://cassci.datastax.com/view/Dev/view/yukim/job/yukim-12203-3.0-testall/lastCompletedBuild/testReport/]|[dtest|http://cassci.datastax.com/view/Dev/view/yukim/job/yukim-12203-3.0-dtest/lastCompletedBuild/testReport/]|
Tests seem good.
> AssertionError on compaction after upgrade (2.1.9 -> 3.7)
> -
>
> Key: CASSANDRA-12203
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12203
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 3.7 (upgrade from 2.1.9)
> Java version "1.8.0_91"
> Ubuntu 14.04.4 LTS (GNU/Linux 3.13.0-83-generic x86_64)
>Reporter: Roman S. Borschel
>Assignee: Yuki Morishita
>Priority: Critical
> Fix For: 3.0.x, 3.x
>
>
> After upgrading a Cassandra cluster from 2.1.9 to 3.7, one column family
> (using SizeTieredCompaction) repeatedly and continuously failed compaction
> (and thus also repair) across the cluster, with all nodes producing the
> following errors in the logs:
> {noformat}
> 016-07-14T09:29:47.96855 |srv=cassandra|ERROR: Exception in thread
> Thread[CompactionExecutor:3,1,main]
> 2016-07-14T09:29:47.96858 |srv=cassandra|java.lang.AssertionError: null
> 2016-07-14T09:29:47.96859 |srv=cassandra| at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$TombstoneTracker.openNew(UnfilteredDeserializer.java:650)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96860 |srv=cassandra| at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.hasNext(UnfilteredDeserializer.java:423)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96860 |srv=cassandra| at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.hasNext(UnfilteredDeserializer.java:298)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96860 |srv=cassandra| at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.readStaticRow(SSTableSimpleIterator.java:133)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96861 |srv=cassandra| at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:57)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96861 |srv=cassandra| at
> org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator$1.initializeIterator(BigTableScanner.java:334)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96862 |srv=cassandra| at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96862 |srv=cassandra| at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:70)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96863 |srv=cassandra| at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:109)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96863 |srv=cassandra| at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:100)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96864 |srv=cassandra| at
> org.apache.cassandra.utils.MergeIterator$Candidate.consume(MergeIterator.java:408)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96864 |srv=cassandra| at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:203)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96865 |srv=cassandra| at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:156)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96865 |srv=cassandra| at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96866 |srv=cassandra| at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$2.hasNext(UnfilteredPartitionIterators.java:150)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96866 |srv=cassandra| at
> org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:72)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96867 |srv=cassandra| at
> org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:226)
>
[jira] [Commented] (CASSANDRA-12203) AssertionError on compaction after upgrade (2.1.9 -> 3.7)
[
https://issues.apache.org/jira/browse/CASSANDRA-12203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15817798#comment-15817798
]
Sylvain Lebresne commented on CASSANDRA-12203:
--
Actually, wait, I think I got that whole logic wrong more than we though and
the patch don't fully fix it.
As said above, the assertion is a sanity check that we have
{{!hasClosingMarkerBefore(tombstone))}} since that's a state assumption of this
{{openNew()}} method.
However,
{noformat}
!hasClosingMarkerBefore(tombstone)) == openTombstones.isEmpty() ||
!comparator.compare(openTombstones.first().stop.bound, tombstone.clustering())
< 0
{noformat}
and as we're in a branch where we know {{!openTombstones.isEmpty()}}, then
{noformat}
!hasClosingMarkerBefore(tombstone)) ==
!comparator.compare(openTombstones.first().stop.bound, tombstone.clustering())
< 0
==
comparator.compare(openTombstones.first().stop.bound, tombstone.clustering())
>= 0
==
comparator.compare(tombstone.clustering(), openTombstones.first().stop.bound)
<= 0
{noformat}
So, knowing that {{tombstone.clustering()}} above is equal to
{{tombstone.start.bound}} in practice, the assertion in {{openNew}} this ticket
is concerned above should be:
{noformat}
assert metadata.comparator.compare(tombstone.start.bound, first.stop.bound) <=
0;
{noformat}
and not
{noformat}
assert metadata.comparator.compare(tombstone.start.bound, first.stop.bound) > 0;
{noformat}
as is currently the case, nor even
{noformat}
assert metadata.comparator.compare(tombstone.start.bound, first.stop.bound) >=
0;
{noformat}
as the patch above changes it.
> AssertionError on compaction after upgrade (2.1.9 -> 3.7)
> -
>
> Key: CASSANDRA-12203
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12203
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 3.7 (upgrade from 2.1.9)
> Java version "1.8.0_91"
> Ubuntu 14.04.4 LTS (GNU/Linux 3.13.0-83-generic x86_64)
>Reporter: Roman S. Borschel
>Assignee: Yuki Morishita
>Priority: Critical
> Fix For: 3.0.x, 3.x
>
>
> After upgrading a Cassandra cluster from 2.1.9 to 3.7, one column family
> (using SizeTieredCompaction) repeatedly and continuously failed compaction
> (and thus also repair) across the cluster, with all nodes producing the
> following errors in the logs:
> {noformat}
> 016-07-14T09:29:47.96855 |srv=cassandra|ERROR: Exception in thread
> Thread[CompactionExecutor:3,1,main]
> 2016-07-14T09:29:47.96858 |srv=cassandra|java.lang.AssertionError: null
> 2016-07-14T09:29:47.96859 |srv=cassandra| at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$TombstoneTracker.openNew(UnfilteredDeserializer.java:650)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96860 |srv=cassandra| at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.hasNext(UnfilteredDeserializer.java:423)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96860 |srv=cassandra| at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.hasNext(UnfilteredDeserializer.java:298)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96860 |srv=cassandra| at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.readStaticRow(SSTableSimpleIterator.java:133)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96861 |srv=cassandra| at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:57)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96861 |srv=cassandra| at
> org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator$1.initializeIterator(BigTableScanner.java:334)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96862 |srv=cassandra| at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96862 |srv=cassandra| at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:70)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96863 |srv=cassandra| at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:109)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96863 |srv=cassandra| at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:100)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96864 |srv=cassandra| at
>
[jira] [Commented] (CASSANDRA-12203) AssertionError on compaction after upgrade (2.1.9 -> 3.7)
[
https://issues.apache.org/jira/browse/CASSANDRA-12203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15814558#comment-15814558
]
Sylvain Lebresne commented on CASSANDRA-12203:
--
+1, thanks.
> AssertionError on compaction after upgrade (2.1.9 -> 3.7)
> -
>
> Key: CASSANDRA-12203
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12203
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 3.7 (upgrade from 2.1.9)
> Java version "1.8.0_91"
> Ubuntu 14.04.4 LTS (GNU/Linux 3.13.0-83-generic x86_64)
>Reporter: Roman S. Borschel
>Assignee: Yuki Morishita
>Priority: Critical
> Fix For: 3.0.x, 3.x
>
>
> After upgrading a Cassandra cluster from 2.1.9 to 3.7, one column family
> (using SizeTieredCompaction) repeatedly and continuously failed compaction
> (and thus also repair) across the cluster, with all nodes producing the
> following errors in the logs:
> {noformat}
> 016-07-14T09:29:47.96855 |srv=cassandra|ERROR: Exception in thread
> Thread[CompactionExecutor:3,1,main]
> 2016-07-14T09:29:47.96858 |srv=cassandra|java.lang.AssertionError: null
> 2016-07-14T09:29:47.96859 |srv=cassandra| at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$TombstoneTracker.openNew(UnfilteredDeserializer.java:650)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96860 |srv=cassandra| at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.hasNext(UnfilteredDeserializer.java:423)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96860 |srv=cassandra| at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.hasNext(UnfilteredDeserializer.java:298)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96860 |srv=cassandra| at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.readStaticRow(SSTableSimpleIterator.java:133)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96861 |srv=cassandra| at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:57)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96861 |srv=cassandra| at
> org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator$1.initializeIterator(BigTableScanner.java:334)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96862 |srv=cassandra| at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96862 |srv=cassandra| at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:70)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96863 |srv=cassandra| at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:109)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96863 |srv=cassandra| at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:100)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96864 |srv=cassandra| at
> org.apache.cassandra.utils.MergeIterator$Candidate.consume(MergeIterator.java:408)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96864 |srv=cassandra| at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:203)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96865 |srv=cassandra| at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:156)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96865 |srv=cassandra| at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96866 |srv=cassandra| at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$2.hasNext(UnfilteredPartitionIterators.java:150)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96866 |srv=cassandra| at
> org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:72)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96867 |srv=cassandra| at
> org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:226)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96867 |srv=cassandra| at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:182)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96867 |srv=cassandra| at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96868 |srv=cassandra|
[jira] [Commented] (CASSANDRA-12203) AssertionError on compaction after upgrade (2.1.9 -> 3.7)
[
https://issues.apache.org/jira/browse/CASSANDRA-12203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15801620#comment-15801620
]
Yuki Morishita commented on CASSANDRA-12203:
Here is a trial patch.
||branch||testall||dtest||
|[12203-3.0|https://github.com/yukim/cassandra/tree/12203-3.0]|[testall|http://cassci.datastax.com/view/Dev/view/yukim/job/yukim-12203-3.0-testall/lastCompletedBuild/testReport/]|[dtest|http://cassci.datastax.com/view/Dev/view/yukim/job/yukim-12203-3.0-dtest/lastCompletedBuild/testReport/]|
I also added test to dtest's upgrade test:
https://github.com/yukim/cassandra-dtest/commit/6a86d2f70d2c4bed03ed24eeef3161f5e21f44ef
> AssertionError on compaction after upgrade (2.1.9 -> 3.7)
> -
>
> Key: CASSANDRA-12203
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12203
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 3.7 (upgrade from 2.1.9)
> Java version "1.8.0_91"
> Ubuntu 14.04.4 LTS (GNU/Linux 3.13.0-83-generic x86_64)
>Reporter: Roman S. Borschel
>Priority: Critical
> Fix For: 3.0.x, 3.x
>
>
> After upgrading a Cassandra cluster from 2.1.9 to 3.7, one column family
> (using SizeTieredCompaction) repeatedly and continuously failed compaction
> (and thus also repair) across the cluster, with all nodes producing the
> following errors in the logs:
> {noformat}
> 016-07-14T09:29:47.96855 |srv=cassandra|ERROR: Exception in thread
> Thread[CompactionExecutor:3,1,main]
> 2016-07-14T09:29:47.96858 |srv=cassandra|java.lang.AssertionError: null
> 2016-07-14T09:29:47.96859 |srv=cassandra| at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$TombstoneTracker.openNew(UnfilteredDeserializer.java:650)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96860 |srv=cassandra| at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.hasNext(UnfilteredDeserializer.java:423)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96860 |srv=cassandra| at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.hasNext(UnfilteredDeserializer.java:298)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96860 |srv=cassandra| at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.readStaticRow(SSTableSimpleIterator.java:133)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96861 |srv=cassandra| at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:57)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96861 |srv=cassandra| at
> org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator$1.initializeIterator(BigTableScanner.java:334)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96862 |srv=cassandra| at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96862 |srv=cassandra| at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:70)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96863 |srv=cassandra| at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:109)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96863 |srv=cassandra| at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:100)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96864 |srv=cassandra| at
> org.apache.cassandra.utils.MergeIterator$Candidate.consume(MergeIterator.java:408)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96864 |srv=cassandra| at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:203)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96865 |srv=cassandra| at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:156)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96865 |srv=cassandra| at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96866 |srv=cassandra| at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$2.hasNext(UnfilteredPartitionIterators.java:150)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96866 |srv=cassandra| at
> org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:72)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96867 |srv=cassandra| at
>
[jira] [Commented] (CASSANDRA-12203) AssertionError on compaction after upgrade (2.1.9 -> 3.7)
[
https://issues.apache.org/jira/browse/CASSANDRA-12203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15777938#comment-15777938
]
Sylvain Lebresne commented on CASSANDRA-12203:
--
bq. This assertion is not be able to be satisfied for the SSTable above. Maybe
we should change {{>}} to {{>=}} ?
Yes, you're right, that's what we should do. The assertion is a sanity check
that we have properly validated {{!hasClosingMarkerBefore(tombstone))}} before
calling this method (as called out in the method comment), but that method
doesn't exclude equality so we shouldn't exclude it. Want to submit the trivial
change, maybe with your repro steps as a dtest? Thanks for digging this up.
> AssertionError on compaction after upgrade (2.1.9 -> 3.7)
> -
>
> Key: CASSANDRA-12203
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12203
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 3.7 (upgrade from 2.1.9)
> Java version "1.8.0_91"
> Ubuntu 14.04.4 LTS (GNU/Linux 3.13.0-83-generic x86_64)
>Reporter: Roman S. Borschel
>Priority: Critical
> Fix For: 3.0.x, 3.x
>
>
> After upgrading a Cassandra cluster from 2.1.9 to 3.7, one column family
> (using SizeTieredCompaction) repeatedly and continuously failed compaction
> (and thus also repair) across the cluster, with all nodes producing the
> following errors in the logs:
> {noformat}
> 016-07-14T09:29:47.96855 |srv=cassandra|ERROR: Exception in thread
> Thread[CompactionExecutor:3,1,main]
> 2016-07-14T09:29:47.96858 |srv=cassandra|java.lang.AssertionError: null
> 2016-07-14T09:29:47.96859 |srv=cassandra| at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$TombstoneTracker.openNew(UnfilteredDeserializer.java:650)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96860 |srv=cassandra| at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.hasNext(UnfilteredDeserializer.java:423)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96860 |srv=cassandra| at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.hasNext(UnfilteredDeserializer.java:298)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96860 |srv=cassandra| at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.readStaticRow(SSTableSimpleIterator.java:133)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96861 |srv=cassandra| at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:57)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96861 |srv=cassandra| at
> org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator$1.initializeIterator(BigTableScanner.java:334)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96862 |srv=cassandra| at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96862 |srv=cassandra| at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:70)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96863 |srv=cassandra| at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:109)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96863 |srv=cassandra| at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:100)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96864 |srv=cassandra| at
> org.apache.cassandra.utils.MergeIterator$Candidate.consume(MergeIterator.java:408)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96864 |srv=cassandra| at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:203)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96865 |srv=cassandra| at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:156)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96865 |srv=cassandra| at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96866 |srv=cassandra| at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$2.hasNext(UnfilteredPartitionIterators.java:150)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96866 |srv=cassandra| at
> org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:72)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96867 |srv=cassandra| at
>
[jira] [Commented] (CASSANDRA-12203) AssertionError on compaction after upgrade (2.1.9 -> 3.7)
[
https://issues.apache.org/jira/browse/CASSANDRA-12203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15773824#comment-15773824
]
Yuki Morishita commented on CASSANDRA-12203:
I have one easily reproducable case for this. I tested with the following step
in cassandra-2.1 branch and updated to cassandra-3.0 branch, and got the same
exception as OP.
While in cassandra-2.1:
1. Create schema as follows:
{code:sql}
CREATE KEYSPACE IF NOT EXISTS ks WITH replication = {'class': 'SimpleStrategy',
'replication_factor': 1};
CREATE TABLE IF NOT EXISTS ks.test (
k ascii,
c1 ascii,
c2 int,
c3 int,
val text,
PRIMARY KEY (k, c1, c2, c3)
);
{code}
2. Create SSTable that contains several RangeTombstones:
{noformat}
$ ccm node1 cqlsh
Connected to 12203 at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 2.1.17-SNAPSHOT | CQL spec 3.2.1 | Native protocol v3]
Use HELP for help.
cqlsh> DELETE FROM ks.test WHERE k = 'a' AND c1 = 'a';
cqlsh> DELETE FROM ks.test WHERE k = 'a' AND c1 = 'a' AND c2 = 1;
cqlsh>
{noformat}
and then flush:
{noformat}
$ ccm node1 flush ks test
{noformat}
You should have SSTable like this:
{noformat}
$ ccm node1 json -k ks -c test
running
['/home/yuki/.ccm/12203/node1/data0/ks/test-17e3f3f0c95f11e69ed29dc952593398/ks-test-ka-1-Data.db']
-- ks-test-ka-1-Data.db -
[
{"key": "a",
"cells": [["a:_","a:1:_",1482532105311908,"t",1482532105],
["a:1:_","a:1:!",1482532109564587,"t",1482532109],
["a:1:!","a:!",1482532105311908,"t",1482532105]]}
]
{noformat}
Now upgrade to cassandra-3.0 branch.
This AE can be triggered in simple SELECT query, or any compaction/streaming
operation.
{noformat}
$ ccm node1 cqlsh
Connected to 12203 at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.0.11-SNAPSHOT | CQL spec 3.4.0 | Native protocol v4]
Use HELP for help.
cqlsh> SELECT * FROM ks.test;
ReadFailure: Error from server: code=1300 [Replica(s) failed to execute read]
message="Operation failed - received 0 responses and 1 failures"
info={'failures': 1, 'received_responses': 0, 'required_responses': 1,
'consistency': 'ONE'}
cqlsh>
{noformat}
In system.log you can see:
{noformat}
WARN [SharedPool-Worker-2] 2016-12-23 16:29:26,377
AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
Thread[SharedPool-Worker-2,5,main]: {}
java.lang.AssertionError: null
at
org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$TombstoneTracker.openNew(UnfilteredDeserializer.java:664)
~[main/:na]
at
org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.hasNext(UnfilteredDeserializer.java:427)
~[main/:na]
at
org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.hasNext(UnfilteredDeserializer.java:290)
~[main/:na]
at
org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.computeNext(SSTableIterator.java:126)
~[main/:na]
at
org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:153)
~[main/:na]
at
org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:340)
~[main/:na]
at
org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:219)
~[main/:na]
at
org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:32)
~[main/:na]
at
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
~[main/:na]
at
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
~[main/:na]
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[main/:na]
at
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
~[main/:na]
at
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
~[main/:na]
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[main/:na]
at
org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129)
~[main/:na]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:131)
~[main/:na]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87)
~[main/:na]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:77)
~[main/:na]
at
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:300)
~[main/:na]
at
[jira] [Commented] (CASSANDRA-12203) AssertionError on compaction after upgrade (2.1.9 -> 3.7)
[
https://issues.apache.org/jira/browse/CASSANDRA-12203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710374#comment-15710374
]
Simon Zhou commented on CASSANDRA-12203:
Also saw exactly the same issue after upgrading from 2.2.5 to 3.0.10.
> AssertionError on compaction after upgrade (2.1.9 -> 3.7)
> -
>
> Key: CASSANDRA-12203
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12203
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: Cassandra 3.7 (upgrade from 2.1.9)
> Java version "1.8.0_91"
> Ubuntu 14.04.4 LTS (GNU/Linux 3.13.0-83-generic x86_64)
>Reporter: Roman S. Borschel
> Fix For: 3.x
>
>
> After upgrading a Cassandra cluster from 2.1.9 to 3.7, one column family
> (using SizeTieredCompaction) repeatedly and continuously failed compaction
> (and thus also repair) across the cluster, with all nodes producing the
> following errors in the logs:
> {noformat}
> 016-07-14T09:29:47.96855 |srv=cassandra|ERROR: Exception in thread
> Thread[CompactionExecutor:3,1,main]
> 2016-07-14T09:29:47.96858 |srv=cassandra|java.lang.AssertionError: null
> 2016-07-14T09:29:47.96859 |srv=cassandra| at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$TombstoneTracker.openNew(UnfilteredDeserializer.java:650)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96860 |srv=cassandra| at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.hasNext(UnfilteredDeserializer.java:423)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96860 |srv=cassandra| at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.hasNext(UnfilteredDeserializer.java:298)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96860 |srv=cassandra| at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.readStaticRow(SSTableSimpleIterator.java:133)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96861 |srv=cassandra| at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:57)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96861 |srv=cassandra| at
> org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator$1.initializeIterator(BigTableScanner.java:334)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96862 |srv=cassandra| at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96862 |srv=cassandra| at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:70)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96863 |srv=cassandra| at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:109)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96863 |srv=cassandra| at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:100)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96864 |srv=cassandra| at
> org.apache.cassandra.utils.MergeIterator$Candidate.consume(MergeIterator.java:408)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96864 |srv=cassandra| at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:203)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96865 |srv=cassandra| at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:156)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96865 |srv=cassandra| at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96866 |srv=cassandra| at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$2.hasNext(UnfilteredPartitionIterators.java:150)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96866 |srv=cassandra| at
> org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:72)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96867 |srv=cassandra| at
> org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:226)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96867 |srv=cassandra| at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:182)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96867 |srv=cassandra| at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96868 |srv=cassandra| at
>
[jira] [Commented] (CASSANDRA-12203) AssertionError on compaction after upgrade (2.1.9 -> 3.7)
[
https://issues.apache.org/jira/browse/CASSANDRA-12203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15450347#comment-15450347
]
Adam Patacchiola commented on CASSANDRA-12203:
--
I'm seeing same issue, one table affected out of 6. Schema looks like:
CREATE TABLE chatmessages (
usernamevarchar,
spotnamevarchar,
id varint,
iv varchar,
fromuservarchar,
fromversion varchar,
touser varchar,
toversion varchar,
datetimetimestamp,
datavarchar,
mimetypevarchar,
datasizeint,
shareable boolean,
hashedboolean,
PRIMARY KEY (username, spotname, id)
);
> AssertionError on compaction after upgrade (2.1.9 -> 3.7)
> -
>
> Key: CASSANDRA-12203
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12203
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: Cassandra 3.7 (upgrade from 2.1.9)
> Java version "1.8.0_91"
> Ubuntu 14.04.4 LTS (GNU/Linux 3.13.0-83-generic x86_64)
>Reporter: Roman S. Borschel
> Fix For: 3.x
>
>
> After upgrading a Cassandra cluster from 2.1.9 to 3.7, one column family
> (using SizeTieredCompaction) repeatedly and continuously failed compaction
> (and thus also repair) across the cluster, with all nodes producing the
> following errors in the logs:
> {noformat}
> 016-07-14T09:29:47.96855 |srv=cassandra|ERROR: Exception in thread
> Thread[CompactionExecutor:3,1,main]
> 2016-07-14T09:29:47.96858 |srv=cassandra|java.lang.AssertionError: null
> 2016-07-14T09:29:47.96859 |srv=cassandra| at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$TombstoneTracker.openNew(UnfilteredDeserializer.java:650)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96860 |srv=cassandra| at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.hasNext(UnfilteredDeserializer.java:423)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96860 |srv=cassandra| at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.hasNext(UnfilteredDeserializer.java:298)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96860 |srv=cassandra| at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.readStaticRow(SSTableSimpleIterator.java:133)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96861 |srv=cassandra| at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:57)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96861 |srv=cassandra| at
> org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator$1.initializeIterator(BigTableScanner.java:334)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96862 |srv=cassandra| at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96862 |srv=cassandra| at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:70)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96863 |srv=cassandra| at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:109)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96863 |srv=cassandra| at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:100)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96864 |srv=cassandra| at
> org.apache.cassandra.utils.MergeIterator$Candidate.consume(MergeIterator.java:408)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96864 |srv=cassandra| at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:203)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96865 |srv=cassandra| at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:156)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96865 |srv=cassandra| at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96866 |srv=cassandra| at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$2.hasNext(UnfilteredPartitionIterators.java:150)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96866 |srv=cassandra| at
> org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:72)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96867 |srv=cassandra| at
> org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:226)
>
[jira] [Commented] (CASSANDRA-12203) AssertionError on compaction after upgrade (2.1.9 -> 3.7)
[
https://issues.apache.org/jira/browse/CASSANDRA-12203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383633#comment-15383633
]
Roman S. Borschel commented on CASSANDRA-12203:
---
Of course, issue description updated.
> AssertionError on compaction after upgrade (2.1.9 -> 3.7)
> -
>
> Key: CASSANDRA-12203
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12203
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: Cassandra 3.7 (upgrade from 2.1.9)
> Java version "1.8.0_91"
> Ubuntu 14.04.4 LTS (GNU/Linux 3.13.0-83-generic x86_64)
>Reporter: Roman S. Borschel
> Fix For: 3.x
>
>
> After upgrading a Cassandra cluster from 2.1.9 to 3.7, one column family
> (using SizeTieredCompaction) repeatedly and continuously failed compaction
> (and thus also repair) across the cluster, with all nodes producing the
> following errors in the logs:
> {noformat}
> 016-07-14T09:29:47.96855 |srv=cassandra|ERROR: Exception in thread
> Thread[CompactionExecutor:3,1,main]
> 2016-07-14T09:29:47.96858 |srv=cassandra|java.lang.AssertionError: null
> 2016-07-14T09:29:47.96859 |srv=cassandra| at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$TombstoneTracker.openNew(UnfilteredDeserializer.java:650)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96860 |srv=cassandra| at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.hasNext(UnfilteredDeserializer.java:423)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96860 |srv=cassandra| at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.hasNext(UnfilteredDeserializer.java:298)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96860 |srv=cassandra| at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.readStaticRow(SSTableSimpleIterator.java:133)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96861 |srv=cassandra| at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:57)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96861 |srv=cassandra| at
> org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator$1.initializeIterator(BigTableScanner.java:334)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96862 |srv=cassandra| at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96862 |srv=cassandra| at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:70)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96863 |srv=cassandra| at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:109)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96863 |srv=cassandra| at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:100)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96864 |srv=cassandra| at
> org.apache.cassandra.utils.MergeIterator$Candidate.consume(MergeIterator.java:408)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96864 |srv=cassandra| at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:203)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96865 |srv=cassandra| at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:156)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96865 |srv=cassandra| at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96866 |srv=cassandra| at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$2.hasNext(UnfilteredPartitionIterators.java:150)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96866 |srv=cassandra| at
> org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:72)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96867 |srv=cassandra| at
> org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:226)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96867 |srv=cassandra| at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:182)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96867 |srv=cassandra| at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96868 |srv=cassandra| at
>
[jira] [Commented] (CASSANDRA-12203) AssertionError on compaction after upgrade (2.1.9 -> 3.7)
[
https://issues.apache.org/jira/browse/CASSANDRA-12203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15382869#comment-15382869
]
Aleksey Yeschenko commented on CASSANDRA-12203:
---
Can you share the schema of the affected table with us, please?
> AssertionError on compaction after upgrade (2.1.9 -> 3.7)
> -
>
> Key: CASSANDRA-12203
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12203
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: Cassandra 3.7 (upgrade from 2.1.9)
> Java version "1.8.0_91"
> Ubuntu 14.04.4 LTS (GNU/Linux 3.13.0-83-generic x86_64)
>Reporter: Roman S. Borschel
> Fix For: 3.x
>
>
> After upgrading a Cassandra cluster from 2.1.9 to 3.7, one column family
> (using SizeTieredCompaction) repeatedly and continuously failed compaction
> (and thus also repair) across the cluster, with all nodes producing the
> following errors in the logs:
> {noformat}
> 016-07-14T09:29:47.96855 |srv=cassandra|ERROR: Exception in thread
> Thread[CompactionExecutor:3,1,main]
> 2016-07-14T09:29:47.96858 |srv=cassandra|java.lang.AssertionError: null
> 2016-07-14T09:29:47.96859 |srv=cassandra| at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$TombstoneTracker.openNew(UnfilteredDeserializer.java:650)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96860 |srv=cassandra| at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer$UnfilteredIterator.hasNext(UnfilteredDeserializer.java:423)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96860 |srv=cassandra| at
> org.apache.cassandra.db.UnfilteredDeserializer$OldFormatDeserializer.hasNext(UnfilteredDeserializer.java:298)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96860 |srv=cassandra| at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.readStaticRow(SSTableSimpleIterator.java:133)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96861 |srv=cassandra| at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:57)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96861 |srv=cassandra| at
> org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator$1.initializeIterator(BigTableScanner.java:334)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96862 |srv=cassandra| at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96862 |srv=cassandra| at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:70)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96863 |srv=cassandra| at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:109)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96863 |srv=cassandra| at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:100)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96864 |srv=cassandra| at
> org.apache.cassandra.utils.MergeIterator$Candidate.consume(MergeIterator.java:408)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96864 |srv=cassandra| at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:203)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96865 |srv=cassandra| at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:156)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96865 |srv=cassandra| at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96866 |srv=cassandra| at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$2.hasNext(UnfilteredPartitionIterators.java:150)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96866 |srv=cassandra| at
> org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:72)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96867 |srv=cassandra| at
> org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:226)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96867 |srv=cassandra| at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:182)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96867 |srv=cassandra| at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-3.7.jar:3.7]
> 2016-07-14T09:29:47.96868 |srv=cassandra| at
>
