[
https://issues.apache.org/jira/browse/CASSANDRA-9126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14708008#comment-14708008
]
Dmitrii Saprykin edited comment on CASSANDRA-9126 at 8/22/15 12:00 PM:
-----------------------------------------------------------------------
I also experience the same issue. I did 'nodetool scrub' for affected
columnfamily without any errors. And issue persists after scrub
We use:
Cassandra 2.0.15, LCS, HSHA, 256 vnodes per node. nodetool repair -par -st
<start-range> -et <end-range>. I use subranges from describe_splits_ex to make
subrange repair.
Keyspace is 8 nodes, RF=3, ~300GB per node (~1TB of unique data without RF),
~10.000.000 keys
Before scrub we had:
{noformat}
ERROR [CompactionExecutor:499] 2015-08-21 20:26:24,438 CassandraDaemon.java
(line 258) Exception in thread Thread[CompactionExecutor:499,1,main]
java.lang.RuntimeException: Last written key DecoratedKey(1928386083417163406,
0313ad7e) >= current key DecoratedKey(1911131194341551311, 010b2fb4) writing
into /data/cassandra/storage/data/bigks/chunks/bigks-chunks-tmp-jb-26880-Data.db
at
org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:170)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
ERROR [CompactionExecutor:498] 2015-08-21 20:29:14,655 CassandraDaemon.java
(line 258) Exception in thread Thread[CompactionExecutor:498,1,main]
java.lang.RuntimeException: Last written key DecoratedKey(1928386083417163406,
0313ad7e) >= current key DecoratedKey(1911131194341551311, 010b2fb4) writing
into /data/cassandra/storage/data/bigks/chunks/bigks-chunks-tmp-jb-26954-Data.db
at
org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:170)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}
After scrub I see following errors during compaction:
{noformat}
ERROR [CompactionExecutor:815] 2015-08-22 03:16:17,153 CassandraDaemon.java
(line 258) Exception in thread Thread[CompactionExecutor:815,1,main]
java.lang.RuntimeException: Last written key DecoratedKey(1922197470632862656,
02c3a415) >= current key DecoratedKey(1916775706087412870, 013a445b) writing
into /data/cassandra/storage/data/bigks/chunks/bigks-chunks-tmp-jb-28982-Data.db
at
org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:170)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
ERROR [CompactionExecutor:819] 2015-08-22 03:18:10,152 CassandraDaemon.java
(line 258) Exception in thread Thread[CompactionExecutor:819,1,main]
java.lang.RuntimeException: Last written key DecoratedKey(1922197470632862656,
02c3a415) >= current key DecoratedKey(1916775706087412870, 013a445b) writing
into /data/cassandra/storage/data/bigks/chunks/bigks-chunks-tmp-jb-28999-Data.db
at
org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:170)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
ERROR [ValidationExecutor:592] 2015-08-22 03:20:33,374 Validator.java (line
255) Failed creating a merkle tree for [repair
#a76b0d50-48a6-11e5-849b-3bfefac09ede on bigks/chunks,
(1916658571049540871,1922266576662767350]], /130.14.18.93 (see log for details)
ERROR [ValidationExecutor:592] 2015-08-22 03:20:33,374 CassandraDaemon.java
(line 258) Exception in thread Thread[ValidationExecutor:592,1,main]
java.lang.AssertionError: row DecoratedKey(1916775706087412870, 013a445b)
received out of order wrt DecoratedKey(1922197470632862656, 02c3a415)
at org.apache.cassandra.repair.Validator.add(Validator.java:137)
at
org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:806)
at
org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:63)
at
org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:404)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
ERROR [ValidationExecutor:592] 2015-08-22 03:20:37,287 Validator.java (line
255) Failed creating a merkle tree for [repair
#aaab6b40-48a6-11e5-849b-3bfefac09ede on bigks/chunks,
(1924415850871700726,1932964198982195249]], /130.14.18.93 (see log for details)
ERROR [ValidationExecutor:592] 2015-08-22 03:20:37,287 CassandraDaemon.java
(line 258) Exception in thread Thread[ValidationExecutor:592,1,main]
java.lang.AssertionError: row DecoratedKey(1924416666703877076, 00e57790)
received out of order wrt DecoratedKey(1928782589289822441, 030cb3d9)
at org.apache.cassandra.repair.Validator.add(Validator.java:137)
at
org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:806)
at
org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:63)
at
org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:404)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
ERROR [CompactionExecutor:1005] 2015-08-22 06:07:15,676 CassandraDaemon.java
(line 258) Exception in thread Thread[CompactionExecutor:1005,1,main]
java.lang.RuntimeException: Last written key DecoratedKey(1922197470632862656,
02c3a415) >= current key DecoratedKey(1916775706087412870, 013a445b) writing
into /data/cassandra/storage/data/bigks/chunks/bigks-chunks-tmp-jb-29048-Data.db
at
org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:170)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
ERROR [CompactionExecutor:1009] 2015-08-22 06:11:24,670 CassandraDaemon.java
(line 258) Exception in thread Thread[CompactionExecutor:1009,1,main]
java.lang.RuntimeException: Last written key DecoratedKey(1922197470632862656,
02c3a415) >= current key DecoratedKey(1916775706087412870, 013a445b) writing
into /data/cassandra/storage/data/bigks/chunks/bigks-chunks-tmp-jb-29071-Data.db
at
org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:170)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}
was (Author: saprykin):
I also experience the same issue. I did 'nodetool scrub' for affected
columnfamily without any errors. And issue persists after scrub
We use:
Cassandra 2.0.15, LCS, HSHA.
Keyspace is 8 nodes, RF=3, ~300GB per node (~1TB of unique data without RF),
~10.000.000 keys
Before scrub we had:
{noformat}
ERROR [CompactionExecutor:499] 2015-08-21 20:26:24,438 CassandraDaemon.java
(line 258) Exception in thread Thread[CompactionExecutor:499,1,main]
java.lang.RuntimeException: Last written key DecoratedKey(1928386083417163406,
0313ad7e) >= current key DecoratedKey(1911131194341551311, 010b2fb4) writing
into /data/cassandra/storage/data/bigks/chunks/bigks-chunks-tmp-jb-26880-Data.db
at
org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:170)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
ERROR [CompactionExecutor:498] 2015-08-21 20:29:14,655 CassandraDaemon.java
(line 258) Exception in thread Thread[CompactionExecutor:498,1,main]
java.lang.RuntimeException: Last written key DecoratedKey(1928386083417163406,
0313ad7e) >= current key DecoratedKey(1911131194341551311, 010b2fb4) writing
into /data/cassandra/storage/data/bigks/chunks/bigks-chunks-tmp-jb-26954-Data.db
at
org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:170)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}
After scrub I see following errors during compaction:
{noformat}
ERROR [CompactionExecutor:815] 2015-08-22 03:16:17,153 CassandraDaemon.java
(line 258) Exception in thread Thread[CompactionExecutor:815,1,main]
java.lang.RuntimeException: Last written key DecoratedKey(1922197470632862656,
02c3a415) >= current key DecoratedKey(1916775706087412870, 013a445b) writing
into /data/cassandra/storage/data/bigks/chunks/bigks-chunks-tmp-jb-28982-Data.db
at
org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:170)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
ERROR [CompactionExecutor:819] 2015-08-22 03:18:10,152 CassandraDaemon.java
(line 258) Exception in thread Thread[CompactionExecutor:819,1,main]
java.lang.RuntimeException: Last written key DecoratedKey(1922197470632862656,
02c3a415) >= current key DecoratedKey(1916775706087412870, 013a445b) writing
into /data/cassandra/storage/data/bigks/chunks/bigks-chunks-tmp-jb-28999-Data.db
at
org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:170)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
ERROR [ValidationExecutor:592] 2015-08-22 03:20:33,374 Validator.java (line
255) Failed creating a merkle tree for [repair
#a76b0d50-48a6-11e5-849b-3bfefac09ede on bigks/chunks,
(1916658571049540871,1922266576662767350]], /130.14.18.93 (see log for details)
ERROR [ValidationExecutor:592] 2015-08-22 03:20:33,374 CassandraDaemon.java
(line 258) Exception in thread Thread[ValidationExecutor:592,1,main]
java.lang.AssertionError: row DecoratedKey(1916775706087412870, 013a445b)
received out of order wrt DecoratedKey(1922197470632862656, 02c3a415)
at org.apache.cassandra.repair.Validator.add(Validator.java:137)
at
org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:806)
at
org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:63)
at
org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:404)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
ERROR [ValidationExecutor:592] 2015-08-22 03:20:37,287 Validator.java (line
255) Failed creating a merkle tree for [repair
#aaab6b40-48a6-11e5-849b-3bfefac09ede on bigks/chunks,
(1924415850871700726,1932964198982195249]], /130.14.18.93 (see log for details)
ERROR [ValidationExecutor:592] 2015-08-22 03:20:37,287 CassandraDaemon.java
(line 258) Exception in thread Thread[ValidationExecutor:592,1,main]
java.lang.AssertionError: row DecoratedKey(1924416666703877076, 00e57790)
received out of order wrt DecoratedKey(1928782589289822441, 030cb3d9)
at org.apache.cassandra.repair.Validator.add(Validator.java:137)
at
org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:806)
at
org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:63)
at
org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:404)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
ERROR [CompactionExecutor:1005] 2015-08-22 06:07:15,676 CassandraDaemon.java
(line 258) Exception in thread Thread[CompactionExecutor:1005,1,main]
java.lang.RuntimeException: Last written key DecoratedKey(1922197470632862656,
02c3a415) >= current key DecoratedKey(1916775706087412870, 013a445b) writing
into /data/cassandra/storage/data/bigks/chunks/bigks-chunks-tmp-jb-29048-Data.db
at
org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:170)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
ERROR [CompactionExecutor:1009] 2015-08-22 06:11:24,670 CassandraDaemon.java
(line 258) Exception in thread Thread[CompactionExecutor:1009,1,main]
java.lang.RuntimeException: Last written key DecoratedKey(1922197470632862656,
02c3a415) >= current key DecoratedKey(1916775706087412870, 013a445b) writing
into /data/cassandra/storage/data/bigks/chunks/bigks-chunks-tmp-jb-29071-Data.db
at
org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:170)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}
> java.lang.RuntimeException: Last written key DecoratedKey >= current key
> DecoratedKey
> -------------------------------------------------------------------------------------
>
> Key: CASSANDRA-9126
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9126
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: srinivasu gottipati
> Priority: Critical
> Fix For: 2.0.x
>
>
> Cassandra V: 2.0.14,
> Getting the following exceptions while trying to compact (I see this issue
> was raised in earlier versions and marked as closed. However it still appears
> in 2.0.14). In our case, compaction is not getting succeeded and keep failing
> with this error.:
> {code}java.lang.RuntimeException: Last written key
> DecoratedKey(3462767860784856708,
> 354038323137333038305f3330325f31355f474d4543454f) >= current key
> DecoratedKey(3462334604624154281,
> 354036333036353334315f3336315f31355f474d4543454f) writing into {code}
> .......
> Stacktrace:{code}
> at
> org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
> at
> org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:167)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745){code}
> Any help is greatly appreciated
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
