[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13781699#comment-13781699
]
Christopher Wirt commented on CASSANDRA-4687:
---------------------------------------------
Ran in to this assertion error after upgrading to 2.0.1 from 1.2.10 and
switching to Hsha.
We ran for a whole day on the sync server after the migration to 2.0.1 without
seeing this error. Once I'd switched over to HsHa I started to see this every
5-10minutes.
We were doing around 400writes/3000reads per second per machine, about half of
the writes would be increments to this counter CF.
We were running at INFO logging level.
Looks like it was only happening on the RelicateOnWrite stage for a counter CF
Compaction pre heat cache was set to true.
Worth mentioning also that we had two other DC running 1.2.10 which hadn't been
upgraded yet.
ERROR [ReplicateOnWriteStage:19] 2013-09-27 10:17:14,778 CassandraDaemon.java
(line 185) Exception in thread Thread[ReplicateOnWriteStage:19,5,main]
java.lang.AssertionError: DecoratedKey(-1754949563326053382,
a414b0c07f0547f8a75410555716ced6) != DecoratedKey(-1754949563326053382,
aeadcec8184445d4ab631ef4250927d0) in
/disk3/cassandra/data/struqrealtime/counters/struqrealtime-counters-jb-831953-Data.db
at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:114)
at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:62)
at
org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:87)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:62)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:249)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1468)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1294)
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:332)
at
org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:55)
at
org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:100)
at
org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1107)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1897)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
> Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
> -----------------------------------------------------------
>
> Key: CASSANDRA-4687
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
> node cluster
> Reporter: Leonid Shalupov
> Priority: Minor
> Attachments: 4687-debugging.txt
>
>
> Under heavy write load sometimes cassandra fails with assertion error.
> git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
> works fine if global key/row caches disabled in code.
> {quote}
> java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
> /var/lib/cassandra/data/...-he-1-Data.db
> at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:60)
> at
> org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
> at
> org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
> at
> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
> at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
> at org.apache.cassandra.db.Table.getRow(Table.java:378)
> at
> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
> at
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
> at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
