[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14180211#comment-14180211
]
Michael Shuler commented on CASSANDRA-4687:
---
Do you have any progress to report, [~jfurmankiewicz]? Thanks!
(I'll close this ticket at some point, if there is no response, on my next pass
through triage)
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt,
apache-cassandra-1.2.13-SNAPSHOT.jar, guava-backed-cache.patch
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14180487#comment-14180487
]
Jacek Furmankiewicz commented on CASSANDRA-4687:
No, I don't.
We've had many GC issues with Cassandra under heavy load, which would cause
unexplained out of memory exceptions (e.g. heap jumps 5GB within one second),
so we had to disable all key and row caches.
Since we had done that this particular error from this issue has not reoccurred.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt,
apache-cassandra-1.2.13-SNAPSHOT.jar, guava-backed-cache.patch
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13866778#comment-13866778
]
Jacek Furmankiewicz commented on CASSANDRA-4687:
Hi, any responses from the Cassandra team? Can we expect this fix backported to
1.1?
1.2 seems to have other regressions with it (the unable to seek position
issue), so we're rather hesitant to upgrade to it.
Thank you, much appreciated
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt,
apache-cassandra-1.2.13-SNAPSHOT.jar, guava-backed-cache.patch
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13866791#comment-13866791
]
Jonathan Ellis commented on CASSANDRA-4687:
---
There will be no new 1.1 releases. You may or may not be relieved to learn
that the seek issue is the same as this one.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt,
apache-cassandra-1.2.13-SNAPSHOT.jar, guava-backed-cache.patch
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13866798#comment-13866798
]
Jacek Furmankiewicz commented on CASSANDRA-4687:
Sorry, somewhat confused here on that last statement.
The patch that we applied seemed to have fixed the DecoratedKey issue, but
introduced the seek issue instead.
So how can it be the same thing? Is it just a bug in the custom patch?
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt,
apache-cassandra-1.2.13-SNAPSHOT.jar, guava-backed-cache.patch
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13866803#comment-13866803
]
Jonathan Ellis commented on CASSANDRA-4687:
---
Your seek bug has been reported in conjunction with this one several times;
notably, disabling key cache fixes both. I see two possibilities:
# There are actually two separate bugs in the key cache
# The patch prevents bad cache entries from being created but does not expunge
existing bad saved entries, and you did not delete your saved caches
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt,
apache-cassandra-1.2.13-SNAPSHOT.jar, guava-backed-cache.patch
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13866804#comment-13866804
]
Jonathan Ellis commented on CASSANDRA-4687:
---
NB: if (1) is the case it is probably present in 1.1 as well.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt,
apache-cassandra-1.2.13-SNAPSHOT.jar, guava-backed-cache.patch
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13866814#comment-13866814
]
Jacek Furmankiewicz commented on CASSANDRA-4687:
I suspect 2 is not the case. I am pretty sure the customer nuked the entire
keyspace (drop keyspace in cassandra-cli) and started clean. This is what we
always do with them in their integration environment.
Unfortunately, (1) is definitely not occurring in 1.1.
Customer has been running with this version a long time and they just completed
their entire testing with our latest app version.
They never saw the seek error on 1.1 with this version of our app.
On the other hand, when running the same version of our app against the patched
1.2 they got the seek error the same day.
So if there are two separate bugs in the key cache, it definitely does not seem
to occur in 1.1
Seems to be in 1.2 only.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt,
apache-cassandra-1.2.13-SNAPSHOT.jar, guava-backed-cache.patch
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13867333#comment-13867333
]
Pavel Yaskevich commented on CASSANDRA-4687:
[~jfurmankiewicz] Can you please do one more test with 1.2 + patch: clear
saved-caches directory, set key_cache_save_period to 0 in cassandra.yaml and
try to load + read the data?
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt,
apache-cassandra-1.2.13-SNAPSHOT.jar, guava-backed-cache.patch
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863104#comment-13863104
]
Jacek Furmankiewicz commented on CASSANDRA-4687:
Hi, sorry for the delay, was out during the holidays.
So, I have good news and I have bad news.
The custom patch that was created seemed to have fixed the issue. The customer
was able to do the initial data load and complete it without Cassandra freezing
in the middle of it, as before. That is the good news.
Unfortunately, after a few hours of usage of this custom 1.2 version they
started seeing other errors they had never seen before:
ERROR [ReadStage:103] 2013-12-29 08:20:49,676 CassandraDaemon.java (line 191)
Exception in thread Thread[ReadStage:103,5,main]
java.lang.RuntimeException: java.lang.IllegalArgumentException: unable to seek
to position 315117685 in
/app/cassandra-int/data/SCHEDULE/TRIGGER_EVENT/SCHEDULE-TRIGGER_EVENT-ic-5-Data.db
(191774 bytes) in read-only mode
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1614)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.IllegalArgumentException: unable to seek to position
315117685 in
/app/cassandra-int/data/SCHEDULE/TRIGGER_EVENT/SCHEDULE-TRIGGER_EVENT-ic-5-Data.db
(191774 bytes) in read-only mode
at
org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:306)
at
org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:42)
at
org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1048)
at
org.apache.cassandra.db.columniterator.IndexedSliceReader.setToRowStart(IndexedSliceReader.java:130)
at
org.apache.cassandra.db.columniterator.IndexedSliceReader.init(IndexedSliceReader.java:91)
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:68)
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:44)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:104)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:272)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1397)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1213)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1129)
at org.apache.cassandra.db.Table.getRow(Table.java:344)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:70)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1058)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1610)
... 3 more
They never saw this errors with their current production 1.1.12 version.
So I am not sure if this problem is related to the patch or whether it is a new
bug in the 1.2 version that they are just happening to see.
We reverted back to 1.1.12 and disabled the key cache so we could continue
towards our live date.
But there is still something fishy going on here.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt,
apache-cassandra-1.2.13-SNAPSHOT.jar, guava-backed-cache.patch
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863113#comment-13863113
]
Balázs Póka commented on CASSANDRA-4687:
I'd like to point out that Jacek's last comment contains an exception which is
_very_ similar to mine, posted quite a while back. It suggests that this bug
manifests itself differently in Cassandra 1.1 vs 1.2, but I can imagine the
root cause to be the same or very similar. If that is true, we could infer that
the bug was not fixed with the customized 1.1 version.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt,
apache-cassandra-1.2.13-SNAPSHOT.jar, guava-backed-cache.patch
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863118#comment-13863118
]
Jacek Furmankiewicz commented on CASSANDRA-4687:
The version we were running was a customized 1.2. It had the patch + libsnappy
was downgraded from 0.5 to 0.4, so that it could still run on RHEL5.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt,
apache-cassandra-1.2.13-SNAPSHOT.jar, guava-backed-cache.patch
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863130#comment-13863130
]
Balázs Póka commented on CASSANDRA-4687:
I'm sorry, must have misunderstood something. My first statement may still be
valid, though. Since this stack was from 1.2, it's not a big surprise that it
looks like mine.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt,
apache-cassandra-1.2.13-SNAPSHOT.jar, guava-backed-cache.patch
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13861481#comment-13861481
]
Radim Kolar commented on CASSANDRA-4687:
Any experience from testing new version in customer test environment?
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt,
apache-cassandra-1.2.13-SNAPSHOT.jar, guava-backed-cache.patch
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13857955#comment-13857955
]
Jonathan Foley commented on CASSANDRA-4687:
---
I saw this in 2.01 writing at 800 ops/sec to a Counter CF, incremental updates
on 58k rows.
ERROR [ReplicateOnWriteStage:47] 2013-12-28 04:53:00,429 CassandraDaemon.java
(line 185) Exception in thread Thread[ReplicateOnWriteStage:47,5,main]
java.lang.AssertionError: DecoratedKey(-9223372036854775808, ) !=
DecoratedKey(-1114982523139082887, 52656e6f) in
/raid0/cassandra/data/Profiles_production_2/CityFeatures/Profile
I also saw a related error that has a different signature in that the hash
seems anomalously long:
java.lang.AssertionError: DecoratedKey(-5739285166491603123,
2d3230303504814336a50ee400240001a85e9880587011e389a8999c37e7417d0001000$
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt,
apache-cassandra-1.2.13-SNAPSHOT.jar, guava-backed-cache.patch
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13854478#comment-13854478
]
Jacek Furmankiewicz commented on CASSANDRA-4687:
Hi Pavel, I took a stock 1.1.12 install, removed the
apache-cassandra-1.1.12.jar from the /lib folder and replaced it with your
apache-cassandra-1.2.13-SNAPSHOT.jar.
When I try to run bin/cassandra I get
Error: Could not find or load main class
org.apache.cassandra.thrift.CassandraDaemon
Is there something else I need to do in order to integrate your JAR into a
stock 1.1.12 install?
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt,
apache-cassandra-1.2.13-SNAPSHOT.jar, guava-backed-cache.patch
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13854484#comment-13854484
]
Jonathan Ellis commented on CASSANDRA-4687:
---
Yes.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt,
apache-cassandra-1.2.13-SNAPSHOT.jar, guava-backed-cache.patch
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13854488#comment-13854488
]
Jacek Furmankiewicz commented on CASSANDRA-4687:
?
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt,
apache-cassandra-1.2.13-SNAPSHOT.jar, guava-backed-cache.patch
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13854498#comment-13854498
]
Pavel Yaskevich commented on CASSANDRA-4687:
How about you take a 1.2 install and replace the jar there?
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt,
apache-cassandra-1.2.13-SNAPSHOT.jar, guava-backed-cache.patch
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13854502#comment-13854502
]
Jacek Furmankiewicz commented on CASSANDRA-4687:
That is what I did. Downloaded 1.1.12 directly from the Apache site. Removed
the 1.1.2 jar from /lib and replaced it with your 1.1.3-SNAPSHOT jar.
This is what the /lib folder looks now:
ls -al
total 11908
drwxr-xr-x 3 jacekf jacekf4096 Dec 20 13:32 .
drwxr-xr-x 9 jacekf jacekf4096 Dec 20 13:31 ..
-rw-r--r-- 1 jacekf jacekf 1928009 May 23 2013 antlr-3.2.jar
-rw--- 1 jacekf jacekf 3143109 Dec 20 13:31
apache-cassandra-1.2.13-SNAPSHOT.jar
-rw-r--r-- 1 jacekf jacekf 35719 May 23 2013
apache-cassandra-clientutil-1.1.12.jar
-rw-r--r-- 1 jacekf jacekf 976978 May 23 2013
apache-cassandra-thrift-1.1.12.jar
-rw-r--r-- 1 jacekf jacekf 596381 May 23 2013 avro-1.4.0-fixes.jar
-rw-r--r-- 1 jacekf jacekf 276425 May 23 2013 avro-1.4.0-sources-fixes.jar
-rw-r--r-- 1 jacekf jacekf 36174 May 23 2013 commons-cli-1.1.jar
-rw-r--r-- 1 jacekf jacekf 30085 May 23 2013 commons-codec-1.2.jar
-rw-r--r-- 1 jacekf jacekf 261809 May 23 2013 commons-lang-2.4.jar
-rw-r--r-- 1 jacekf jacekf 25490 May 23 2013 compress-lzf-0.8.4.jar
-rw-r--r-- 1 jacekf jacekf 54345 May 23 2013
concurrentlinkedhashmap-lru-1.3.jar
-rw-r--r-- 1 jacekf jacekf 91982 May 23 2013 cql-internal-only-1.4.0.zip
-rw-r--r-- 1 jacekf jacekf 1112253 May 23 2013 guava-r08.jar
-rw-r--r-- 1 jacekf jacekf 96046 May 23 2013 high-scale-lib-1.1.2.jar
-rw-r--r-- 1 jacekf jacekf 228286 May 23 2013 jackson-core-asl-1.9.2.jar
-rw-r--r-- 1 jacekf jacekf 765648 May 23 2013 jackson-mapper-asl-1.9.2.jar
-rw-r--r-- 1 jacekf jacekf5792 May 23 2013 jamm-0.2.5.jar
-rw-r--r-- 1 jacekf jacekf 87325 May 23 2013 jline-0.9.94.jar
-rw-r--r-- 1 jacekf jacekf 16046 May 23 2013 json-simple-1.1.jar
-rw-r--r-- 1 jacekf jacekf 301547 May 23 2013 libthrift-0.7.0.jar
drwxr-xr-x 2 jacekf jacekf4096 Dec 20 13:31 licenses
-rw-r--r-- 1 jacekf jacekf 481534 May 23 2013 log4j-1.2.16.jar
-rw-r--r-- 1 jacekf jacekf 80800 May 23 2013 metrics-core-2.0.3.jar
-rw-r--r-- 1 jacekf jacekf 134133 May 23 2013 servlet-api-2.5-20081211.jar
-rw-r--r-- 1 jacekf jacekf 25496 May 23 2013 slf4j-api-1.6.1.jar
-rw-r--r-- 1 jacekf jacekf9753 May 23 2013 slf4j-log4j12-1.6.1.jar
-rw-r--r-- 1 jacekf jacekf 232121 May 23 2013 snakeyaml-1.6.jar
-rw-r--r-- 1 jacekf jacekf 995967 May 23 2013 snappy-java-1.0.4.1.jar
-rw-r--r-- 1 jacekf jacekf 55066 May 23 2013 snaptree-0.1.jar
-rw-r--r-- 1 jacekf jacekf 42854 May 23 2013
thrift-python-internal-only-0.7.0.zip
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt,
apache-cassandra-1.2.13-SNAPSHOT.jar, guava-backed-cache.patch
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13854509#comment-13854509
]
Jacek Furmankiewicz commented on CASSANDRA-4687:
Ah sorry, we had a misunderstanding. I cannot take the 1.2 version. We need a
1.1 version.
If you recall, we cannot run 1.2 on CentOS5 due to glibc issues. Client is also
on 1.1 and in the short term is not interested in upgrading a production system
to 1.2, so it would be a very hard sell.
That is why if we want to ask them to help us to reproduce the issue, we need
to be able to do it with a 1.1 version.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt,
apache-cassandra-1.2.13-SNAPSHOT.jar, guava-backed-cache.patch
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13854512#comment-13854512
]
Jacek Furmankiewicz commented on CASSANDRA-4687:
yes, sorry for the mistake. I was under the impression we could get a
customized 1.1 version
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt,
apache-cassandra-1.2.13-SNAPSHOT.jar, guava-backed-cache.patch
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13854510#comment-13854510
]
Pavel Yaskevich commented on CASSANDRA-4687:
Well, the version is 1.*2*.13 :)
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt,
apache-cassandra-1.2.13-SNAPSHOT.jar, guava-backed-cache.patch
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13854514#comment-13854514
]
Jacek Furmankiewicz commented on CASSANDRA-4687:
if you can give me a customized 1.2 version that starts on a stock RHEL5 (i.e.
bypasses the glibc issues in libsnappy) then we could try just in a test
environment with 1.2.
But it all hinges on getting 1.2 running on RHEL5
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt,
apache-cassandra-1.2.13-SNAPSHOT.jar, guava-backed-cache.patch
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13854519#comment-13854519
]
Pavel Yaskevich commented on CASSANDRA-4687:
Just replace the jar in latest 1.2 release with what I have attached and do
what Jonathan suggested:
bq. 1.2 will run on RHEL5 if you replace the snappy-1.0.5 jar with the
snappy-1.0.4 jar from C* 1.1.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt,
apache-cassandra-1.2.13-SNAPSHOT.jar, guava-backed-cache.patch
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13854520#comment-13854520
]
Jacek Furmankiewicz commented on CASSANDRA-4687:
OK, will do, thanks. Hopefully we should be able to try it in the customer's
test environment next week, I will keep you posted if we find anything
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt,
apache-cassandra-1.2.13-SNAPSHOT.jar, guava-backed-cache.patch
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13854522#comment-13854522
]
Pavel Yaskevich commented on CASSANDRA-4687:
Thanks!
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt,
apache-cassandra-1.2.13-SNAPSHOT.jar, guava-backed-cache.patch
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13854526#comment-13854526
]
Jacek Furmankiewicz commented on CASSANDRA-4687:
BTW, is there any particular reason why you upgraded snappy? if there are no
particularly important features, maybe it would be worth rolling it back for
the 1.2 releases.
The benefits of that minor change do not seem to be worth losing the ability to
run 1.2 on a stock RHEL5, which is still very common on production systems.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt,
apache-cassandra-1.2.13-SNAPSHOT.jar, guava-backed-cache.patch
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13845885#comment-13845885
]
Jacek Furmankiewicz commented on CASSANDRA-4687:
Hi Pavel, our customer run 'yum update' recently on those servers, so I am
guessing that version of glibc is as good as it is going to get.
What are out other options? We need to be able to test on RHEL5
Thank you
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13845914#comment-13845914
]
Jonathan Ellis commented on CASSANDRA-4687:
---
1.2 will run on RHEL5 if you replace the snappy-1.0.5 jar with the snappy-1.0.4
jar from C* 1.1.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13845917#comment-13845917
]
Jacek Furmankiewicz commented on CASSANDRA-4687:
Got it.
I could try to prepare a custom Franken Cassandra from both versions and give
it to our customer to try in a test environment.
Just let me know what customized jar you would want me to replace in a stock
1.2 installation.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13841205#comment-13841205
]
Jacek Furmankiewicz commented on CASSANDRA-4687:
We are seeing the same issue on 1.1.12 and RHEL 5. Major batch processing which
involves lots reads and writes and runs for a few hours...in the middle of it
Cassandra nodes start dying one by one IN THE ENTIRE CLUSTER.
Sooner or later, the entire cluster is dead in the water.
We can reproduce this every time.
This is stopping the live date for a major system.
BTW, this BUG IS NOT MINOR! This is a SHOWSTOPPER issue.
The entire cluster, ALL NODES, go down.
Distributed DB reliability is not supposed to allow something like this to
happen.
Only workaround is restarting every single node one by one...in what is
supposed to be a 24x7 system that has no right to go down.
Please escalate this to SHOWSTOPPER status.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13841216#comment-13841216
]
Jacek Furmankiewicz commented on CASSANDRA-4687:
Here's a sample of our logs from one of the failed logs before it stopped
responding altogether
{quote}
ERROR [ReadStage:658] 2013-12-05 19:05:13,414 AbstractCassandraDaemon.java
(line 132) Exception in thread Thread[ReadStage:658,5,main]
java.lang.AssertionError: DecoratedKey(-1, ) !=
DecoratedKey(84818348637167105300405141302451086940,
323031342d30332d31325f4c585f33363130) in
/app/cassandra-int/data/SCHEDULE/LEG/SCHEDULE-LEG-hf-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.init(SSTableNamesIterator.java:72)
at
org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:61)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectTimeOrderedData(CollationController.java:124)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1362)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1224)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1159)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:58)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:822)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1271)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
ERROR [ReadStage:605] 2013-12-05 19:05:13,413 AbstractCassandraDaemon.java
(line 132) Exception in thread Thread[ReadStage:605,5,main]
java.lang.AssertionError: DecoratedKey(165404967731480145169372669349926870701,
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13841437#comment-13841437
]
Radim Kolar commented on CASSANDRA-4687:
I agree with you, its serious problem. It is in C 1.1 up to 2.0. I do not
recommend anybody to run these versions for mission critical workloads.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13841468#comment-13841468
]
Balázs Póka commented on CASSANDRA-4687:
Did you try disabling the key cache? For me (also for others, see previous
messages) that is a valid workaround for the problem. I still consider this a
serious issue, though.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13841663#comment-13841663
]
Jacek Furmankiewicz commented on CASSANDRA-4687:
Yes, we have tried that and so far it seems OK.
However, we are not sure what the perf implications are.
Our app has very strict response time SLAs. Even a few ms extra could put us
in violation.
So this is a serious concern.
Very troubling to see a bug that can bring down an entire cluster of all nodes
is classified as Minor and has no one assigned to it.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13841677#comment-13841677
]
Pavel Yaskevich commented on CASSANDRA-4687:
I'm sorry Jacek, it's just we are not sure what is going on because nobody can
reproduce this locally. I have couple of theories regarding this, might be as
simple as bug in ConcurrentLinkedHashMap, as you can reproduce this behavior I
can give you a patch which would use other map implementation for key cache so
we can at least rule out that possibility, what do you think?...
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13841689#comment-13841689
]
Jacek Furmankiewicz commented on CASSANDRA-4687:
Sure thing, I am sure we could get our customer to try it out.
However, you would need to give me a complete Cassandra 1.1.X install that can
install on RHEL5.
We cannot even install 1.2 on RHEL5, since it complains about some issues with
snappy and glibc versions.
Also, the version of JNA on RHEL5 seems not to work with Cassandra and we have
to uninstall it
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13841786#comment-13841786
]
Pavel Yaskevich commented on CASSANDRA-4687:
Well, this is tough because I can only give you a jar at most, also I don't
think I can do those modifications on 1.1 release because it would require new
version of guava. So I guess the only way to do it would it in 1.2, is there
any way you can update glibc to the version snappy depends on? You can check it
by ldd -v. The problem is that cache needs replacement mechanism which in
Java world only done reliably by Guava and ConcurrentLinkedHashMap.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13781699#comment-13781699
]
Christopher Wirt commented on CASSANDRA-4687:
-
Ran in to this assertion error after upgrading to 2.0.1 from 1.2.10 and
switching to Hsha.
We ran for a whole day on the sync server after the migration to 2.0.1 without
seeing this error. Once I'd switched over to HsHa I started to see this every
5-10minutes.
We were doing around 400writes/3000reads per second per machine, about half of
the writes would be increments to this counter CF.
We were running at INFO logging level.
Looks like it was only happening on the RelicateOnWrite stage for a counter CF
Compaction pre heat cache was set to true.
Worth mentioning also that we had two other DC running 1.2.10 which hadn't been
upgraded yet.
ERROR [ReplicateOnWriteStage:19] 2013-09-27 10:17:14,778 CassandraDaemon.java
(line 185) Exception in thread Thread[ReplicateOnWriteStage:19,5,main]
java.lang.AssertionError: DecoratedKey(-1754949563326053382,
a414b0c07f0547f8a75410555716ced6) != DecoratedKey(-1754949563326053382,
aeadcec8184445d4ab631ef4250927d0) in
/disk3/cassandra/data/struqrealtime/counters/struqrealtime-counters-jb-831953-Data.db
at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:114)
at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.init(SSTableNamesIterator.java:62)
at
org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:87)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:62)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:249)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1468)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1294)
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:332)
at
org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:55)
at
org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:100)
at
org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1107)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1897)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13737508#comment-13737508
]
Pavel Yaskevich commented on CASSANDRA-4687:
Do you guys have compaction_preheat_key_cache set to true? [~pokabalazs] Can
you please elaborate what data migration program that was?
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725630#comment-13725630
]
Balázs Póka commented on CASSANDRA-4687:
I've just reproduced it with Cassandra 1.2.6 (on Ubuntu 12.04 OpenVZ box). 1
node cluster.
A have multiple big column families, and have been doing lots of reads/writes
while running a data migration program. Those CFs seem to be affected more
which get most of the reads and writes.
Log of the first exception and lines immediately before it:
INFO [OptionalTasks:1] 2013-07-23 16:36:08,651 ColumnFamilyStore.java (line
631) Enqueuing flush of Memtable-measuredata_201305@1749058828(11711/26488
serialized/live bytes, 276 ops)
INFO [FlushWriter:755] 2013-07-23 16:36:08,681 Memtable.java (line 461)
Writing Memtable-measuredata_201305@1749058828(11711/26488 serialized/live
bytes, 276 ops)
INFO [FlushWriter:755] 2013-07-23 16:36:08,789 Memtable.java (line 495)
Completed flushing
/mnt/db/cassandra/gps/measuredata_201305/gps-measuredata_201305-ic-1-Data.db
(7808 bytes) for commitlog position ReplayPosition(segmentId=136978869451
5, position=7660)
ERROR [ReadStage:55564] 2013-07-23 17:10:00,268 CassandraDaemon.java (line 175)
Exception in thread Thread[ReadStage:55564,5,main]
java.lang.RuntimeException: java.lang.IllegalArgumentException: unable to seek
to position 60965 in
/mnt/db/cassandra/gps/measuredata_201305/gps-measuredata_201305-ic-1-Data.db
(12801 bytes) in read-only mode
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1582)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.IllegalArgumentException: unable to seek to position 60965
in /mnt/db/cassandra/gps/measuredata_201305/gps-measuredata_201305-ic-1-Data.db
(12801 bytes) in read-only mode
at
org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:306)
at
org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:42)
at
org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:976)
at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.createFileDataInput(SSTableNamesIterator.java:94)
at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:112)
at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.init(SSTableNamesIterator.java:60)
at
org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
at
org.apache.cassandra.db.CollationController.collectTimeOrderedData(CollationController.java:133)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
at org.apache.cassandra.db.Table.getRow(Table.java:347)
at
org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1052)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1578)
... 3 more
Clearing the key cache with nodetool invalidatekeycache effectively fixed the
problem. I have not disabled it. Row cache is disabled.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13701709#comment-13701709
]
Andrew Bialecki commented on CASSANDRA-4687:
Just hit this while doing some load testing. I can provide a bit more
information about what we were doing and how we created it:
Running Cassandra 1.2.5 on Ubuntu 12.04 with the default config file. 3 node
cluster with the following Keyspace/CF:
CREATE KEYSPACE TestKeyspace1 WITH placement_strategy='SimpleStrategy'
AND strategy_options = { replication_factor: 3};
USE TestKeyspace1;
CREATE COLUMN FAMILY TestCounters1 WITH
default_validation_class=CounterColumnType AND
key_validation_class=UTF8Type
AND comparator=UTF8Type AND replicate_on_write=true
AND compaction_strategy=LeveledCompactionStrategy AND
compaction_strategy_options={sstable_size_in_mb: 5};
Used our equivalent of the load utility to use 40 threads to increment counters
across 1M keys with up to 100 columns randomly. Peak load with this
configuration was about ~9.5k writes/sec, no reads. Although reads were being
done as a read is part of a counter write.
We had been playing with different setting to benchmark and to clear the
keyspace/CF, we were running:
DROP KEYSPACE TestKeyspace1;
and then re-creating it with the script above. After a few drops/creates I
noticed the SSTable size didn't appear to adhere to the sstable size
(potentially another bug?). No errors in logs, but after a few minutes of
writes, we saw no sstables written, so we forced a flush with nodetool flush.
Upon doing so on one node, we ended up with a single sstable that was a 60M
file.
Right after the flush, we got a lot of errors in the log like:
java.lang.AssertionError: DecoratedKey(151132845099071093774085323081542634448,
636f6c756d6e34) != DecoratedKey(52766578155366297747310347893720908288,
726f77353035333134) in
/var/lib/cassandra/data/TestKeyspace1/TestCounters1/TestKeyspace1-TestCounters1-ic-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:119)
at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.init(SSTableNamesIterator.java:60)
at
org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:274)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
at org.apache.cassandra.db.Table.getRow(Table.java:347)
at
org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64)
at
org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:90)
at
org.apache.cassandra.service.StorageProxy$7$1.runMayThrow(StorageProxy.java:796)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1578)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
By the way, we changed how we dropped/recreated the keyspace from dropping the
keyspace and running:
nodetool clearsnapshot
nodetool cleanup
to dropping the keyspace and then running:
rm -rf /var/lib/cassandra/data/TestKeyspace1
nodetool invalidatekeycache
and now we're seeing the sstable size respected. Not sure if that's a bug, but
I can take the relevant bits from above and open a ticket if that's useful. Or
if it's better, when we have some free time we can try to simplify the steps to
reproduce.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666301#comment-13666301
]
Oleg Dulin commented on CASSANDRA-4687:
---
We just experienced this error. I can't reproduce it, but the workload was very
heavy read-before-write load where reads are slice queries. We were loading
about 6million data items resulting in updates to about a dozen column
families, so roughly speaking 30-40million writes, some interspersed with
read-before-write.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13644605#comment-13644605
]
Dirk Lachowski commented on CASSANDRA-4687:
---
I see this error on an older (longer in use) cf after adding a new column
definition via cqlsh and then adding values for existing keys for the new
column.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Minor
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13606336#comment-13606336
]
Jonathan Ellis commented on CASSANDRA-4687:
---
Addressing in CASSANDRA-5202.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Fix For: 1.2.4
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13606355#comment-13606355
]
Leonid Shalupov commented on CASSANDRA-4687:
Things in this issue happens without dropping or truncating keyspaces or column
families.
Is it really a duplicate?
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Fix For: 1.2.4
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13602957#comment-13602957
]
Matthew Goodson commented on CASSANDRA-4687:
An update. I replaced the node that was throwing the exception and the new node
doesn't have the same issue so...fixed?? I will report back if it happens
again.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Fix For: 1.2.4
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13601530#comment-13601530
]
Matthew Goodson commented on CASSANDRA-4687:
I have just started seeing this error. I get it 2 or 3 times a minute. I am
using a 3 node cluster running cassandra 1.0.6 which has been running for about
a year and only just started doing it. It is only happening to our largest
column family (~= 17G). I have disabled key caching for the CF, invalidated the
cache and removed the CF's saved cache. None of these workarounds have worked.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Fix For: 1.2.3
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13601537#comment-13601537
]
Dan Peebles commented on CASSANDRA-4687:
Matthew, what kind of use pattern do you have, and has it changed recently? In
my experience with this problem, I get it a lot when I truncate CFs or drop
keyspaces. The fact that clearing/disabling the key cache (are you using a row
cache?) doesn't help you suggests that your cause might be different from mine,
because that remedy did help me.
It's also worth noting that I saw someone on the IRC channel report this issue
last night, so it does seem to be affecting quite a few people.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Fix For: 1.2.3
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13601550#comment-13601550
]
Matthew Goodson commented on CASSANDRA-4687:
Hi Dan, I've only got one keyspace (other than system) and we never drop it. I
occasionally create new CF's but that's about it. It just runs. No row caching
is enabled just the key cache which was set to the default 20. It was prob
me that you saw report it on IRC and there was another guy who said that he
just started seeing it as well with version 1.1.?. It has just started
randomly, I don't think I have touched the cluster for about a month. I think
it might be time to replace the node and see if that makes the problem go away
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Fix For: 1.2.3
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13570586#comment-13570586
]
Ron Siemens commented on CASSANDRA-4687:
I am getting this error every few hours. I am using Titan 0.2.0 and Cassandra
1.1.6 on a single node. Titan creates the keyspace, so whatever defaults it is
using. It happens when I am loading data into the graph, it is doing many
small batches of work in a single thread, using Titan's graph.stopTransaction()
for each batch. It doesn't always happen, but has happened daily after a
couple hours usage. I am testing so I drop the keyspace and recreate it quite
a bit. My running cassandra is in this state right now. Once it gets in this
state, it doesn't clear up, until I shut down and restart.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Fix For: 1.2.2
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13570611#comment-13570611
]
Yuki Morishita commented on CASSANDRA-4687:
---
If you are creating and dropping the same CF often, you are hitting
CASSANDRA-5202.
I _think_ there are some other cases that cause this, but I don't have a clue
yet.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Fix For: 1.2.2
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13570656#comment-13570656
]
Ron Siemens commented on CASSANDRA-4687:
I'll manually invalidate the caches when dropping the keyspace - will report
back if it still occurs - thanks.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Fix For: 1.2.2
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13570661#comment-13570661
]
Pavel Yaskevich commented on CASSANDRA-4687:
If it does work with invalidated key cache or with caches turned off, that
means it could be a concurrency problem in cache implementation...
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Fix For: 1.2.2
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13567501#comment-13567501
]
Leonid Shalupov commented on CASSANDRA-4687:
User on SO has the same problem:
http://stackoverflow.com/questions/14131558/cql3-select-succeeds-for-some-keys-but-fails-for-others/14611903
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Fix For: 1.2.2
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13555143#comment-13555143
]
Sylvain Pognant commented on CASSANDRA-4687:
Just been hit by the same issue :
cassandra 1.2.0 cluster with 6 nodes, no write activity from clients, just one
client reading a specific row (replication factor 2)
I was randomly stopping/starting nodes to test the client read behavior.
At some point I restarted a node holding the row (the other replicating node
was down) and started to get 'unable to seek to position XX' errors on that
node.
- Tried restarting the node : no change
- Removed the content of saved_caches/ : fixed the problem
So, as commented above, the saved cache seems to be invalid.
Note that I always stopped the nodes cleanly.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Fix For: 1.2.2
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13501351#comment-13501351
]
Jeremy Hanna commented on CASSANDRA-4687:
-
As another data point, one user reported seeing the cache capacity misreporting
its value in the nodetool info output around the same time as seeing the
DecoratedKey exceptions in their log. Once they readjusted the cache capacity
using nodetool setcachecapacity, the exceptions went away. They set both key
and row cache to non-zero values, so it looks like disabling key cache, at
least for them, wasn't necessary.
So for example, they had 1024M for key cache and 1024M for row cache in the
cassandra.yaml. They saw values like the following when running nodetool info
on a few of the nodes:
{quote}
Key Cache: size 536870880 (bytes), capacity 536870880 (bytes), 81039812
hits, 95996835 requests, 0.879 recent hit rate, 14400 save period in seconds
Row Cache: size 237336 (bytes), capacity 238282 (bytes), 823752 hits,
1171017 requests, 0.687 recent hit rate, 0 save period in seconds
{quote}
Note that the capacity value is far below what they configured in their
cassandra.yaml.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.5
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Fix For: 1.1.7
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477136#comment-13477136
]
Chris Herron commented on CASSANDRA-4687:
-
Has anybody experienced Linux socket FD leakage alongside these errors? (see
CASSANDRA-4571).
To check this, you can run:
{{watch -n 10 sudo lsof -n | grep java | grep unix | wc -l}}
This number should stay at 1. If you see growth towards your limits
(/etc/security/limits.conf), then that suggests CASSANDRA-4571 might be a
side-effect of this problem.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.5
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Priority: Critical
Fix For: 1.1.7
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13470512#comment-13470512
]
T Jake Luciani commented on CASSANDRA-4687:
---
I am seeing something similar here.
{code}
ERROR [ReadStage:70] 2012-10-05 13:26:40,215 AbstractCassandraDaemon.java (line
135) Exception in thread Thread[ReadStage:70,5,main]
java.io.IOError: java.io.EOFException: unable to seek to position 165713136 in
/opt/data/cassandra/data/prod/market_data_state/prod-market_data_state-he-5-Data.db
(74622511 bytes) in read-only mode
at
org.apache.cassandra.io.util.CompressedSegmentedFile.getSegment(CompressedSegmentedFile.java:69)
at
org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:898)
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:50)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:51)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.EOFException: unable to seek to position 165713136 in
/opt/data/cassandra/data/prod/market_data_state/prod-market_data_state-he-5-Data.db
(74622511 bytes) in read-only mode
at
org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:253)
at
org.apache.cassandra.io.util.CompressedSegmentedFile.getSegment(CompressedSegmentedFile.java:64)
... 16 more
{code}
It seems to be hopping SSTables in compaction for the same row I'm trying to
access so could be serialization issue.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.5
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Priority: Critical
Fix For: 1.1.6
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13470517#comment-13470517
]
T Jake Luciani commented on CASSANDRA-4687:
---
On Pavel's advice I disabled the key cache and it resolved the issue for me.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.5
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Priority: Critical
Fix For: 1.1.6
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13470531#comment-13470531
]
T Jake Luciani commented on CASSANDRA-4687:
---
I've been loading data so it's 99% writes.
I found on the ML that someone else hit this and was bulk loading data...
http://mail-archives.apache.org/mod_mbox/cassandra-user/201208.mbox/%3c95ecc1abc58041c8a94503285e016...@ntoklo.com%3E
I am not using the bulk loader though...
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.5
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Priority: Critical
Fix For: 1.1.6
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13470596#comment-13470596
]
T Jake Luciani commented on CASSANDRA-4687:
---
One other thought.
This only happened on one of the nodes and I think that was the one node that
had restarted Cassandra.
So could possibly be the saved cache.
This was the one row I was searching previously before it broke.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.5
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Priority: Critical
Fix For: 1.1.6
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13470724#comment-13470724
]
Pavel Yaskevich commented on CASSANDRA-4687:
[~tpatterson] Can you try to test saved cache scenario? e.g. while inserting
try reading (to simulate write-heavy environment), make sure that cache was
saved (probably setting small save interval would be the best), restart the
node, try reading data.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.5
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Priority: Critical
Fix For: 1.1.6
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13465652#comment-13465652
]
Jonathan Ellis commented on CASSANDRA-4687:
---
I went ahead and committed my debugging patch for 1.1.6, so it will be easier
to troubleshoot. Note, I did drop the log level to TRACE since it's more than
even most DEBUG use cases call for.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.5
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Priority: Critical
Fix For: 1.1.6
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13465299#comment-13465299
]
Tyler Patterson commented on CASSANDRA-4687:
I just spent a day trying to reproduce this error, to no avail. Here is what I
tried for reference:
I wrote a python stress script, then ran it in 50 threads from 3 different
EC2 instances, all stressing one EC2 node running cassandra-1.1.5. The
processor load on the cassandra node was pegged at 100% (or very close to) the
entire time each test was running.
The first version of the python stress script created text keys varying in
length up to 1 chars. It inserted and read one key at a time, where keys
were randomly chosen from a pool of 10,000,000 keys. Cassandra had a key_cache
size of 10mb. This test used cql, and inserted and read one key and column at a
time.
The second version of the stress script worked the same as the first, but used
pycassa.
Version 3 also used pycassa, but did batch_insert and multiget, each with 1000
keys at a time. It also deleted a random key after every batch_insert and
multiget. For this test I also ran cassandra-stress --operation=COUNTER_ADD
and cassandra-stress --operation-COUNTER_GET
version 4 was the similar to version 3, but used int32 keys instead of text.
I played with some other settings like key_cache and compaction_strategy_class
during some of the tests. I never was able to get the error to happen.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.5
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Priority: Critical
Fix For: 1.1.6
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13464320#comment-13464320
]
Radim Kolar commented on CASSANDRA-4687:
Yes its with patch. I am not sure about compaction failing.
My paid work has priority over your demands to edit logfile for you. Its
hundreds MB long and i have just 2GB memory in computer.
If you want that entire logfile then buy it and i will spend half day uploading
it or if you want testsuite from our cassandra fork which can reproduce this
and various other errors, then my ask my company manager, if he is willing to
sell you that testsuite.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.5
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Priority: Critical
Fix For: 1.1.6
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13462621#comment-13462621
]
Pavel Yaskevich commented on CASSANDRA-4687:
lgtm, would be more prosaic then I have expected. Radim can you reproduce with
the patch?
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.5
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Priority: Critical
Fix For: 1.1.6
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13462628#comment-13462628
]
Sylvain Lebresne commented on CASSANDRA-4687:
-
bq. So either we're actually writing empty row keys, or we're seeking into the
file at a strange offset
Since Leonid has bisected it to the global cache patch, my vote would be for
the new key cache returning a bogus offset yielding to seeking in a wrong
position. Clearly it would be very helpful if either Radim or Lenoid can
reproduce with Jonathan's patch, but in the meantime we should probably start
looking that way.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.5
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Priority: Critical
Fix For: 1.1.6
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13462815#comment-13462815
]
Radim Kolar commented on CASSANDRA-4687:
how to turn on debug output from patch?
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.5
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Priority: Critical
Fix For: 1.1.6
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13462825#comment-13462825
]
Pavel Yaskevich commented on CASSANDRA-4687:
In server log4j config in conf/
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.5
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Priority: Critical
Fix For: 1.1.6
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13462966#comment-13462966
]
Jonathan Ellis commented on CASSANDRA-4687:
---
Radim, can you attach the log starting from server startup, so we get the debug
context?
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.5
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Priority: Critical
Fix For: 1.1.6
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13462970#comment-13462970
]
Jonathan Ellis commented on CASSANDRA-4687:
---
bq. my vote would be for the new key cache returning a bogus offset yielding
to seeking in a wrong position
We didn't actually change much there though (for key cache) -- instead of
{{Pairdescriptor, DK}} cache keys in a per-CFS Map, we have {{KeyCacheKey}}
objects (still basically descriptor + DK) in a global Map. Unless
descriptor.equals is broken I don't see the problem.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.5
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Priority: Critical
Fix For: 1.1.6
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13462998#comment-13462998
]
Radim Kolar commented on CASSANDRA-4687:
i dont have log anymore, deleted it. It was too huge ~ 200MB with all debugs on.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.5
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Priority: Critical
Fix For: 1.1.6
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13463029#comment-13463029
]
Jonathan Ellis commented on CASSANDRA-4687:
---
What we need is the history of the cache entry. So you'd start with the
assertion error, then grep backwards for the key in the debug lines. The
assertion error itself doesn't tell us anything new.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.5
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Priority: Critical
Fix For: 1.1.6
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13463133#comment-13463133
]
Radim Kolar commented on CASSANDRA-4687:
i am not interested to upload that huge file at my 128kBit unless you pay for
it.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.5
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Priority: Critical
Fix For: 1.1.6
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13463137#comment-13463137
]
Jonathan Ellis commented on CASSANDRA-4687:
---
I just explained how to isolate the relevant information without uploading the
entire file.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.5
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Priority: Critical
Fix For: 1.1.6
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13463352#comment-13463352
]
Pavel Yaskevich commented on CASSANDRA-4687:
Radim, were stack-traces you posted taken when you were running cassandra with
Jonathan's patch? also, can you tell if you have any compactions failing with
similar error?
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.5
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Priority: Critical
Fix For: 1.1.6
Attachments: 4687-debugging.txt
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13462447#comment-13462447
]
Jonathan Ellis commented on CASSANDRA-4687:
---
Possibly not new in 1.1.x? This looks like the same failure:
http://mail-archives.apache.org/mod_mbox/cassandra-user/201112.mbox/%3CCAGmOvD+ANvzpNBuWVuharEtb623Ook=agbnnnkgy62ngnzp...@mail.gmail.com%3E
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.5
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Priority: Critical
Fix For: 1.1.6
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13461104#comment-13461104
]
Radim Kolar commented on CASSANDRA-4687:
it fails with default 1.1 setup which is:
INFO [main] 2012-09-22 12:15:02,937 CacheService.java (line 96) Initializing
key cache with capacity of 50 MBs.
INFO [main] 2012-09-22 12:15:03,156 CacheService.java (line 107) Scheduling
key cache save to each 14400 seconds (going to save all keys).
INFO [main] 2012-09-22 12:15:03,187 CacheService.java (line 121) Initializing
row cache with capacity of 0 MBs and provider
org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider
INFO [main] 2012-09-22 12:15:03,203 CacheService.java (line 133) Scheduling
row cache save to each 0 seconds (going to save all keys).
Keyspace: test:
Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy
Durable Writes: true
Options: [datacenter1:2]
Column Families:
ColumnFamily: sipdb
phone calls routing information
Key Validation Class: org.apache.cassandra.db.marshal.IntegerType
Default column value validator: org.apache.cassandra.db.marshal.BytesType
Columns sorted by: org.apache.cassandra.db.marshal.AsciiType
GC grace seconds: 0
Compaction min/max thresholds: 4/32
Read repair chance: 0.0
DC Local Read repair chance: 0.0
Replicate on write: false
Caching: KEYS_ONLY
Bloom Filter FP chance: default
Built indexes: []
Column Metadata:
Column Name: kam
Validation Class: org.apache.cassandra.db.marshal.AsciiType
Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompacti
onStrategy
Compression Options:
sstable_compression: org.apache.cassandra.io.compress.SnappyCompressor
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.5
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Priority: Critical
Fix For: 1.1.6
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13461105#comment-13461105
]
Leonid Shalupov commented on CASSANDRA-4687:
It's different in my case:
{code}
[INFO ] 21 сен 2012 13:18:25 CacheService - Initializing key cache with
capacity of 50 MBs.
[INFO ] 21 сен 2012 13:18:25 CacheService - Scheduling key cache save
to each 14400 seconds (going to save all keys).
[INFO ] 21 сен 2012 13:18:25 CacheService - Initializing row cache with
capacity of 0 MBs and provider
org.apache.cassandra.cache.SerializingCacheProvider
[INFO ] 21 сен 2012 13:18:25 CacheService - Scheduling row cache save
to each 0 seconds (going to save all keys).
{code}
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.5
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Priority: Critical
Fix For: 1.1.6
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13461107#comment-13461107
]
Leonid Shalupov commented on CASSANDRA-4687:
Unfortunately, I can't reproduce it every time now like some weeks ago, when I
did git bisect.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.5
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Priority: Critical
Fix For: 1.1.6
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13461115#comment-13461115
]
Leonid Shalupov commented on CASSANDRA-4687:
Additional information:
- it happens after flushing
- they appear in pairs in different ReadStage threads
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.5
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Priority: Critical
Fix For: 1.1.6
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13461171#comment-13461171
]
Radim Kolar commented on CASSANDRA-4687:
I can reproduce it in about 40% of cases. Start with empty CF and then write
into it using batches of 1000 Keys in 20 threads. I am writing 2m keys, but
usually it triggers bug around 400k KEY.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.5
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Priority: Critical
Fix For: 1.1.6
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13461259#comment-13461259
]
Jonathan Ellis commented on CASSANDRA-4687:
---
You must be doing some reads as well?
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.5
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Priority: Critical
Fix For: 1.1.6
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13461273#comment-13461273
]
Pavel Yaskevich commented on CASSANDRA-4687:
Yes indeed there should be reads, I have a guess that it's somehow related to
the LRU part of the cache because, the interesting, is that it returns a valid
key position, testing this theory.
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.5
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Priority: Critical
Fix For: 1.1.6
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13460634#comment-13460634
]
Jonathan Ellis commented on CASSANDRA-4687:
---
Are you using row cache in practice? IOW, can we narrow the culprit down to
row cache (or vice versa)?
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.5
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Priority: Critical
Fix For: 1.1.6
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13460669#comment-13460669
]
Radim Kolar commented on CASSANDRA-4687:
i have set cache to KEYS_ONLY and snappy compression
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.5
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Priority: Critical
Fix For: 1.1.6
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
[
https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13460841#comment-13460841
]
Pavel Yaskevich commented on CASSANDRA-4687:
Leonid, can you please elaborate on what cache configuration do you use for
rows - is it serializing cache? Is so, have you tried switching to
ConcurrentLinkedHashCache? That would be very helpful to localize the problem...
Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
---
Key: CASSANDRA-4687
URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.5
Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single
node cluster
Reporter: Leonid Shalupov
Assignee: Pavel Yaskevich
Priority: Critical
Fix For: 1.1.6
Under heavy write load sometimes cassandra fails with assertion error.
git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
works fine if global key/row caches disabled in code.
{quote}
java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in
/var/lib/cassandra/data/...-he-1-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:60)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
