[
https://issues.apache.org/jira/browse/CASSANDRA-7188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14154706#comment-14154706
]
Nicolas Lalevée edited comment on CASSANDRA-7188 at 10/1/14 12:00 PM:
----------------------------------------------------------------------
Our cluster was reverted back to 1.2.11. We upgraded last week to 1.2.19.
Successfully, without issues.
We tried to upgrade then to 2.0.10. Still a mess with the same error spreading
around as soon as we upgrade one node among 6.
Here are the kind of errors seen on the upgraded node:
{noformat}
ERROR [ReplicateOnWriteStage:136] 2014-10-01 11:19:37,778 CassandraDaemon.java
(line 199) Exception in thread Thread[ReplicateOnWriteSt
age:136,5,main]
java.lang.AssertionError: Wrong class type: class org.apache.cassandra.db.Column
at
org.apache.cassandra.db.CounterColumn.reconcile(CounterColumn.java:159)
at
org.apache.cassandra.db.filter.QueryFilter$1.reduce(QueryFilter.java:109)
at
org.apache.cassandra.db.filter.QueryFilter$1.reduce(QueryFilter.java:103)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:112)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:98)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at
org.apache.cassandra.db.filter.NamesQueryFilter.collectReducedColumns(NamesQueryFilter.java:98)
at
org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122)
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1547)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1376)
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:333)
at
org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:55)
at
org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:100)
at
org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1086)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1928)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
ERROR [MutationStage:136] 2014-10-01 11:19:36,843 CassandraDaemon.java (line
199) Exception in thread Thread[MutationStage:136,5,main]
java.lang.AssertionError: Wrong class type: class org.apache.cassandra.db.Column
at
org.apache.cassandra.db.CounterColumn.reconcile(CounterColumn.java:159)
at
org.apache.cassandra.db.AtomicSortedColumns$Holder.addColumn(AtomicSortedColumns.java:326)
at
org.apache.cassandra.db.AtomicSortedColumns.addAllWithSizeDelta(AtomicSortedColumns.java:191)
at org.apache.cassandra.db.Memtable.resolve(Memtable.java:218)
at org.apache.cassandra.db.Memtable.put(Memtable.java:165)
at
org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:900)
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:374)
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:339)
at org.apache.cassandra.db.RowMutation.apply(RowMutation.java:211)
at
org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:56)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{noformat}
And here is what can be found on the not upgraded nodes:
{noformat}
ERROR [MutationStage:147] 2014-10-01 11:26:57,270 CassandraDaemon.java (line
191) Exception in thread Thread[MutationStage:147,5,main]
java.lang.AssertionError: Wrong class type: class org.apache.cassandra.db.Column
at
org.apache.cassandra.db.CounterColumn.reconcile(CounterColumn.java:165)
at
org.apache.cassandra.db.AtomicSortedColumns$Holder.addColumn(AtomicSortedColumns.java:383)
at
org.apache.cassandra.db.AtomicSortedColumns.addColumn(AtomicSortedColumns.java:166)
at
org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:119)
at org.apache.cassandra.db.SuperColumn.addColumn(SuperColumn.java:218)
at org.apache.cassandra.db.SuperColumn.putColumn(SuperColumn.java:229)
at
org.apache.cassandra.db.ThreadSafeSortedColumns.addColumnInternal(ThreadSafeSortedColumns.java:108)
at
org.apache.cassandra.db.ThreadSafeSortedColumns.addAllWithSizeDelta(ThreadSafeSortedColumns.java:138)
at
org.apache.cassandra.db.AbstractColumnContainer.addAllWithSizeDelta(AbstractColumnContainer.java:99)
at org.apache.cassandra.db.Memtable.resolve(Memtable.java:205)
at org.apache.cassandra.db.Memtable.put(Memtable.java:168)
at
org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:749)
at org.apache.cassandra.db.Table.apply(Table.java:389)
at org.apache.cassandra.db.Table.apply(Table.java:354)
at org.apache.cassandra.db.RowMutation.apply(RowMutation.java:283)
at
org.apache.cassandra.db.CounterMutation.apply(CounterMutation.java:138)
at
org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:771)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1622)
at
org.apache.cassandra.service.StorageProxy$3.apply(StorageProxy.java:141)
at
org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:384)
at
org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:746)
at
org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:53)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
ERROR [ReadStage:73] 2014-10-01 11:27:03,218 CassandraDaemon.java (line 191)
Exception in thread Thread[ReadStage:73,5,main]
java.lang.AssertionError: Wrong class type: class org.apache.cassandra.db.Column
at
org.apache.cassandra.db.CounterColumn.reconcile(CounterColumn.java:165)
at
org.apache.cassandra.db.AtomicSortedColumns$Holder.addColumn(AtomicSortedColumns.java:383)
at
org.apache.cassandra.db.AtomicSortedColumns.addColumn(AtomicSortedColumns.java:166)
at
org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:119)
at org.apache.cassandra.db.SuperColumn.addColumn(SuperColumn.java:218)
at org.apache.cassandra.db.SuperColumn.putColumn(SuperColumn.java:229)
at
org.apache.cassandra.db.ArrayBackedSortedColumns.resolveAgainst(ArrayBackedSortedColumns.java:164)
at
org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:141)
at
org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:119)
at
org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:114)
at
org.apache.cassandra.db.filter.QueryFilter$1.reduce(QueryFilter.java:112)
at
org.apache.cassandra.db.filter.QueryFilter$1.reduce(QueryFilter.java:96)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:111)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:97)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at
org.apache.cassandra.db.filter.NamesQueryFilter.collectReducedColumns(NamesQueryFilter.java:103)
at
org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:136)
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:84)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:291)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1398)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1130)
at org.apache.cassandra.db.Table.getRow(Table.java:348)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:70)
at
org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:44)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{noformat}
was (Author: hibou):
Our cluster was reverted back to 1.2.11. We upgraded last week to 1.2.19.
Successfully, without issues.
We tried to upgrade then to 2.0.10. Still a mess with the same error spreading
around as soon as we upgrade one node among 6.
Here is the king of errors seen on the upgrade node:
{noformat}
ERROR [ReplicateOnWriteStage:136] 2014-10-01 11:19:37,778 CassandraDaemon.java
(line 199) Exception in thread Thread[ReplicateOnWriteSt
age:136,5,main]
java.lang.AssertionError: Wrong class type: class org.apache.cassandra.db.Column
at
org.apache.cassandra.db.CounterColumn.reconcile(CounterColumn.java:159)
at
org.apache.cassandra.db.filter.QueryFilter$1.reduce(QueryFilter.java:109)
at
org.apache.cassandra.db.filter.QueryFilter$1.reduce(QueryFilter.java:103)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:112)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:98)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at
org.apache.cassandra.db.filter.NamesQueryFilter.collectReducedColumns(NamesQueryFilter.java:98)
at
org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122)
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1547)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1376)
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:333)
at
org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:55)
at
org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:100)
at
org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1086)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1928)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
ERROR [MutationStage:136] 2014-10-01 11:19:36,843 CassandraDaemon.java (line
199) Exception in thread Thread[MutationStage:136,5,main]
java.lang.AssertionError: Wrong class type: class org.apache.cassandra.db.Column
at
org.apache.cassandra.db.CounterColumn.reconcile(CounterColumn.java:159)
at
org.apache.cassandra.db.AtomicSortedColumns$Holder.addColumn(AtomicSortedColumns.java:326)
at
org.apache.cassandra.db.AtomicSortedColumns.addAllWithSizeDelta(AtomicSortedColumns.java:191)
at org.apache.cassandra.db.Memtable.resolve(Memtable.java:218)
at org.apache.cassandra.db.Memtable.put(Memtable.java:165)
at
org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:900)
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:374)
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:339)
at org.apache.cassandra.db.RowMutation.apply(RowMutation.java:211)
at
org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:56)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{noformat}
And here is what can be found on the not upgraded ones:
{noformat}
ERROR [MutationStage:147] 2014-10-01 11:26:57,270 CassandraDaemon.java (line
191) Exception in thread Thread[MutationStage:147,5,main]
java.lang.AssertionError: Wrong class type: class org.apache.cassandra.db.Column
at
org.apache.cassandra.db.CounterColumn.reconcile(CounterColumn.java:165)
at
org.apache.cassandra.db.AtomicSortedColumns$Holder.addColumn(AtomicSortedColumns.java:383)
at
org.apache.cassandra.db.AtomicSortedColumns.addColumn(AtomicSortedColumns.java:166)
at
org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:119)
at org.apache.cassandra.db.SuperColumn.addColumn(SuperColumn.java:218)
at org.apache.cassandra.db.SuperColumn.putColumn(SuperColumn.java:229)
at
org.apache.cassandra.db.ThreadSafeSortedColumns.addColumnInternal(ThreadSafeSortedColumns.java:108)
at
org.apache.cassandra.db.ThreadSafeSortedColumns.addAllWithSizeDelta(ThreadSafeSortedColumns.java:138)
at
org.apache.cassandra.db.AbstractColumnContainer.addAllWithSizeDelta(AbstractColumnContainer.java:99)
at org.apache.cassandra.db.Memtable.resolve(Memtable.java:205)
at org.apache.cassandra.db.Memtable.put(Memtable.java:168)
at
org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:749)
at org.apache.cassandra.db.Table.apply(Table.java:389)
at org.apache.cassandra.db.Table.apply(Table.java:354)
at org.apache.cassandra.db.RowMutation.apply(RowMutation.java:283)
at
org.apache.cassandra.db.CounterMutation.apply(CounterMutation.java:138)
at
org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:771)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1622)
at
org.apache.cassandra.service.StorageProxy$3.apply(StorageProxy.java:141)
at
org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:384)
at
org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:746)
at
org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:53)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
ERROR [ReadStage:73] 2014-10-01 11:27:03,218 CassandraDaemon.java (line 191)
Exception in thread Thread[ReadStage:73,5,main]
java.lang.AssertionError: Wrong class type: class org.apache.cassandra.db.Column
at
org.apache.cassandra.db.CounterColumn.reconcile(CounterColumn.java:165)
at
org.apache.cassandra.db.AtomicSortedColumns$Holder.addColumn(AtomicSortedColumns.java:383)
at
org.apache.cassandra.db.AtomicSortedColumns.addColumn(AtomicSortedColumns.java:166)
at
org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:119)
at org.apache.cassandra.db.SuperColumn.addColumn(SuperColumn.java:218)
at org.apache.cassandra.db.SuperColumn.putColumn(SuperColumn.java:229)
at
org.apache.cassandra.db.ArrayBackedSortedColumns.resolveAgainst(ArrayBackedSortedColumns.java:164)
at
org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:141)
at
org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:119)
at
org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:114)
at
org.apache.cassandra.db.filter.QueryFilter$1.reduce(QueryFilter.java:112)
at
org.apache.cassandra.db.filter.QueryFilter$1.reduce(QueryFilter.java:96)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:111)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:97)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at
org.apache.cassandra.db.filter.NamesQueryFilter.collectReducedColumns(NamesQueryFilter.java:103)
at
org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:136)
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:84)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:291)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1398)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1130)
at org.apache.cassandra.db.Table.getRow(Table.java:348)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:70)
at
org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:44)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{noformat}
> Wrong class type: class org.apache.cassandra.db.Column in
> CounterColumn.reconcile
> ---------------------------------------------------------------------------------
>
> Key: CASSANDRA-7188
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7188
> Project: Cassandra
> Issue Type: Bug
> Reporter: Nicolas Lalevée
> Assignee: Michael Shuler
>
> When migrating a cluster of 6 nodes from 1.2.11 to 2.0.7, we started to see
> on the first migrated node this error:
> {noformat}
> ERROR [ReplicateOnWriteStage:1] 2014-05-07 11:26:59,779 CassandraDaemon.java
> (line 198) Exception in thread Thread[ReplicateOnWriteStage:1,5,main]
> java.lang.AssertionError: Wrong class type: class
> org.apache.cassandra.db.Column
> at
> org.apache.cassandra.db.CounterColumn.reconcile(CounterColumn.java:159)
> at
> org.apache.cassandra.db.filter.QueryFilter$1.reduce(QueryFilter.java:109)
> at
> org.apache.cassandra.db.filter.QueryFilter$1.reduce(QueryFilter.java:103)
> at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:112)
> at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:98)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at
> org.apache.cassandra.db.filter.NamesQueryFilter.collectReducedColumns(NamesQueryFilter.java:98)
> at
> org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122)
> at
> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)
> at
> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72)
> at
> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297)
> at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1540)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1369)
> at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:327)
> at
> org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:55)
> at
> org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:100)
> at
> org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1085)
> at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1916)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> {noformat}
> We then saw on the other 5 nodes, still on 1.2.x, this error:
> {noformat}
> ERROR [MutationStage:2793] 2014-05-07 11:46:12,301 CassandraDaemon.java (line
> 191) Exception in thread Thread[MutationStage:2793,5,main]
> java.lang.AssertionError: Wrong class type: class
> org.apache.cassandra.db.Column
> at
> org.apache.cassandra.db.CounterColumn.reconcile(CounterColumn.java:165)
> at
> org.apache.cassandra.db.AtomicSortedColumns$Holder.addColumn(AtomicSortedColumns.java:378)
> at
> org.apache.cassandra.db.AtomicSortedColumns.addColumn(AtomicSortedColumns.java:166)
> at
> org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:119)
> at org.apache.cassandra.db.SuperColumn.addColumn(SuperColumn.java:218)
> at org.apache.cassandra.db.SuperColumn.putColumn(SuperColumn.java:229)
> at
> org.apache.cassandra.db.ThreadSafeSortedColumns.addColumnInternal(ThreadSafeSortedColumns.java:108)
> at
> org.apache.cassandra.db.ThreadSafeSortedColumns.addAllWithSizeDelta(ThreadSafeSortedColumns.java:138)
> at
> org.apache.cassandra.db.AbstractColumnContainer.addAllWithSizeDelta(AbstractColumnContainer.java:99)
> at org.apache.cassandra.db.Memtable.resolve(Memtable.java:205)
> at org.apache.cassandra.db.Memtable.put(Memtable.java:168)
> at
> org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:742)
> at org.apache.cassandra.db.Table.apply(Table.java:388)
> at org.apache.cassandra.db.Table.apply(Table.java:353)
> at org.apache.cassandra.db.RowMutation.apply(RowMutation.java:280)
> at
> org.apache.cassandra.db.CounterMutation.apply(CounterMutation.java:137)
> at
> org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:773)
> at
> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:1651)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:679)
> {noformat}
> Here some other stack we also on the 5 unmigrated nodes:
> {noformat}
> ERROR [ReadStage:4242] 2014-05-07 11:46:12,259 CassandraDaemon.java (line
> 191) Exception in thread Thread[ReadStage:4242,5,main]
> java.lang.AssertionError: Wrong class type: class
> org.apache.cassandra.db.Column
> at
> org.apache.cassandra.db.CounterColumn.reconcile(CounterColumn.java:165)
> at
> org.apache.cassandra.db.AtomicSortedColumns$Holder.addColumn(AtomicSortedColumns.java:378)
> at
> org.apache.cassandra.db.AtomicSortedColumns.addColumn(AtomicSortedColumns.java:166)
> at
> org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:119)
> at org.apache.cassandra.db.SuperColumn.addColumn(SuperColumn.java:218)
> at org.apache.cassandra.db.SuperColumn.putColumn(SuperColumn.java:229)
> at
> org.apache.cassandra.db.ArrayBackedSortedColumns.resolveAgainst(ArrayBackedSortedColumns.java:164)
> at
> org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:141)
> at
> org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:119)
> at
> org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:114)
> at
> org.apache.cassandra.db.filter.QueryFilter$1.reduce(QueryFilter.java:112)
> at
> org.apache.cassandra.db.filter.QueryFilter$1.reduce(QueryFilter.java:96)
> at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:111)
> at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:97)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at
> org.apache.cassandra.db.filter.NamesQueryFilter.collectReducedColumns(NamesQueryFilter.java:103)
> at
> org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:136)
> at
> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:84)
> at
> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:291)
> at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1391)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1123)
> at org.apache.cassandra.db.Table.getRow(Table.java:347)
> at
> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:70)
> at
> org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:44)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:679)
> {noformat}
> And the client side, it is failing with:
> {noformat}
> Caused by: org.apache.cassandra.thrift.UnavailableException: null
> at
> org.apache.cassandra.thrift.Cassandra$get_slice_result.read(Cassandra.java:7866)
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
> at
> org.apache.cassandra.thrift.Cassandra$Client.recv_get_slice(Cassandra.java:594)
> at
> org.apache.cassandra.thrift.Cassandra$Client.get_slice(Cassandra.java:578)
> at
> me.prettyprint.cassandra.service.KeyspaceServiceImpl$7.execute(KeyspaceServiceImpl.java:274)
> {noformat}
> After seeing such errors, we just shut down the first migrated node, hoping
> it would avoid all these client errors. But errors continue to be logged,
> even if there were only the 5 1.2.x nodes in the ring.
> As the usual wild guess, let's reboot a node to fix it. At our damned
> surprise, it would restart and would fail with:
> {noformat}
> INFO 11:33:40,190 Initializing system.LocationInfo
> java.lang.AssertionError
> at
> org.apache.cassandra.cql3.CFDefinition.<init>(CFDefinition.java:162)
> at
> org.apache.cassandra.config.CFMetaData.updateCfDef(CFMetaData.java:1541)
> at
> org.apache.cassandra.config.CFMetaData.fromSchema(CFMetaData.java:1456)
> at
> org.apache.cassandra.config.KSMetaData.deserializeColumnFamilies(KSMetaData.java:306)
> at
> org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:287)
> at org.apache.cassandra.db.DefsTable.loadFromTable(DefsTable.java:154)
> at
> org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:574)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:253)
> at
> org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:381)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:212)
> Cannot load daemon
> Service exit with a return value of 3
> {noformat}
> From there we only had 4 running nodes, with errors spreading around. So we
> halted everything, put the first node back to 1.2.11 and restored the data
> which has been snapshot just before the first node was migrated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
