[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13465299#comment-13465299
]
Tyler Patterson commented on CASSANDRA-4687:
--------------------------------------------
I just spent a day trying to reproduce this error, to no avail. Here is what I
tried for reference:
I wrote a python "stress" script, then ran it in 50 threads from 3 different
EC2 instances, all stressing one EC2 node running cassandra-1.1.5. The
processor load on the cassandra node was pegged at 100% (or very close to) the
entire time each test was running.
The first version of the python stress script created text keys varying in
length up to 10000 chars. It inserted and read one key at a time, where keys
were randomly chosen from a pool of 10,000,000 keys. Cassandra had a key_cache
size of 10mb. This test used cql, and inserted and read one key and column at a
time.
The second version of the stress script worked the same as the first, but used
pycassa.
Version 3 also used pycassa, but did batch_insert and multiget, each with 1000
keys at a time. It also deleted a random key after every batch_insert and
multiget. For this test I also ran "cassandra-stress --operation=COUNTER_ADD"
and "cassandra-stress --operation-COUNTER_GET"
version 4 was the similar to version 3, but used int32 keys instead of text.
I played with some other settings like key_cache and compaction_strategy_class
during some of the tests. I never was able to get the error to happen.
> Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
> -----------------------------------------------------------
>
> Key: CASSANDRA-4687
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.5
> Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
> node cluster
> Reporter: Leonid Shalupov
> Assignee: Pavel Yaskevich
> Priority: Critical
> Fix For: 1.1.6
>
> Attachments: 4687-debugging.txt
>
>
> Under heavy write load sometimes cassandra fails with assertion error.
> git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
> works fine if global key/row caches disabled in code.
> {quote}
> java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
> /var/lib/cassandra/data/...-he-1-Data.db
> at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:60)
> at
> org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
> at
> org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
> at
> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
> at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
> at org.apache.cassandra.db.Table.getRow(Table.java:378)
> at
> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
> at
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
> at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
