[
https://issues.apache.org/jira/browse/CASSANDRA-9126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15304173#comment-15304173
]
jean carlo rivera ura edited comment on CASSANDRA-9126 at 6/1/16 8:18 AM:
--------------------------------------------------------------------------
[~mambocab] We are using cassandra 2.1.12
In production we got a first error doing a repair (-pr -par)
{code:none}
ERROR [ValidationExecutor:3283] 2016-05-26 09:37:37,911 Validator.java:245 -
Failed creating a merkle tree for [repair #b16c1f30-2314-11e6-97fe-dd058ca99653
on keyspace1/table1, (8699512854132214411,8702471448538509513]], /192.168.1.144
(see log for details)
ERROR [ValidationExecutor:3283] 2016-05-26 09:37:37,913
CassandraDaemon.java:227 - Exception in thread
Thread[ValidationExecutor:3283,1,main]
java.lang.AssertionError: row DecoratedKey(8699513492008207074,
434c503031303030303030303036363632323233) received out of order wrt
DecoratedKey(8702470199759211565,
4b454e4f42494a52432d43553130333031353232313032)
at org.apache.cassandra.repair.Validator.add(Validator.java:126)
~[apache-cassandra-2.1.12.jar:2.1.12]
at
org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1038)
~[apache-cassandra-2.1.12.jar:2.1.12]
at
org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:89)
~[apache-cassandra-2.1.12.jar:2.1.12]
at
org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:649)
~[apache-cassandra-2.1.12.jar:2.1.12]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_60]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_60]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
{code}
We decided to do another a repair over this token range
8699512854132214411,8702471448538509513 using the next command.
{code:none}
root@node021:~$ nodetool repair keyspace1 -st 8699512854132214411 -et
8702471448538509513
[2016-05-27 14:19:20,768] Starting repair command #37, repairing 1 ranges for
keyspace keyspace1 (parallelism=SEQUENTIAL, full=true)
[2016-05-27 14:19:38,489] Repair session 3dd2f250-2405-11e6-b33f-ab1665ddb9c0
for range (8699512854132214411,8702471448538509513] failed with error
org.apache.cassandra.exceptions.RepairException: [repair
#3dd2f250-2405-11e6-b33f-ab1665ddb9c0 on keyspace1/table2,
(8699512854132214411,8702471448538509513]] Validation failed in /192.168.2.137
[2016-05-27 14:19:38,489] Repair command #37 finished
error: nodetool failed, check server logs
-- StackTrace --
java.lang.RuntimeException: nodetool failed, check server logs
at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:292)
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:204)
{code}
Unlike the first time, this error comes from the node 192.168.2.137. Checking
the log, the node failed creating a merkle tree for the table table2. This
table is LCS
{code:none}
ERROR [ValidationExecutor:3427] 2016-05-27 14:19:38,466 Validator.java:245 -
Failed creating a merkle tree for [repair #3dd2f250-2405-11e6-b33f-ab1665ddb9c0
on keyspace1/table2, (8699512854132214411,8702471448538509513]], /192.168.1.154
(see log for details)
ERROR [ValidationExecutor:3427] 2016-05-27 14:19:38,467
CassandraDaemon.java:227 - Exception in thread
Thread[ValidationExecutor:3427,1,main]
java.lang.AssertionError: row DecoratedKey(8699513197702636918,
49442d5350502d3130302d54513944486f7333704e68492b387a6e786870653347614d766a6832597744446a5543446d4b397a393545)
received out of order wrt DecoratedKey(8702471248515353880,
49442d5350502d3130302d704a3063676b4446714b5075644b68654273672f686c6a4759767264724f4b6e664d3176765a7a70416d6b)
at org.apache.cassandra.repair.Validator.add(Validator.java:126)
~[apache-cassandra-2.1.12.jar:2.1.12]
at
org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1038)
~[apache-cassandra-2.1.12.jar:2.1.12]
at
org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:89)
~[apache-cassandra-2.1.12.jar:2.1.12]
at
org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:649)
~[apache-cassandra-2.1.12.jar:2.1.12]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_60]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_60]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
{code}
We tried to run again the same repair on the same node with the same tokenrange
and it finished without error
{code:none}
root@node021:~$ nodetool repair keyspace1 -st 8699512854132214411 -et
8702471448538509513
[2016-05-27 14:27:00,419] Starting repair command #38, repairing 1 ranges for
keyspace keyspace1 (parallelism=SEQUENTIAL, full=true)
[2016-05-27 14:36:18,239] Repair session 4fcda620-2406-11e6-b33f-ab1665ddb9c0
for range (8699512854132214411,8702471448538509513] finished
[2016-05-27 14:36:18,239] Repair command #38 finished
{code}
It seems this error comes up by hazard. We notice that our first repair -pr
-par has finished repairing all the tokenranges, but that one we got the error.
was (Author: carlo_4002):
[~mambocab] We are using cassandra 2.1.12
In production we got a first error doing a repair (-pr -par)
{code:none}
ERROR [ValidationExecutor:3283] 2016-05-26 09:37:37,911 Validator.java:245 -
Failed creating a merkle tree for [repair #b16c1f30-2314-11e6-97fe-dd058ca99653
on pns_fr_prod/pig, (8699512854132214411,8702471448538509513]], /10.98.255.144
(see log for details)
ERROR [ValidationExecutor:3283] 2016-05-26 09:37:37,913
CassandraDaemon.java:227 - Exception in thread
Thread[ValidationExecutor:3283,1,main]
node021.cassandra.prod.pns.s1.p.fti.net:
/var/opt/hosting/log/cassandra/system.log-java.lang.AssertionError: row
DecoratedKey(8699513492008207074, 434c503031303030303030303036363632323233)
received out of order wrt DecoratedKey(8702470199759211565,
4b454e4f42494a52432d435531303330313532323130
{code}
We decided to do another a repair over this token range
8699512854132214411,8702471448538509513 using the next command.
{code:none}
root@node021[SPH][PROD][PnS3]:~$ nodetool repair pns_fr_prod -st
8699512854132214411 -et 8702471448538509513
[2016-05-27 14:19:20,768] Starting repair command #37, repairing 1 ranges for
keyspace pns_fr_prod (parallelism=SEQUENTIAL, full=true)
[2016-05-27 14:19:38,489] Repair session 3dd2f250-2405-11e6-b33f-ab1665ddb9c0
for range (8699512854132214411,8702471448538509513] failed with error
org.apache.cassandra.exceptions.RepairException: [repair
#3dd2f250-2405-11e6-b33f-ab1665ddb9c0 on pns_fr_prod/syndic,
(8699512854132214411,8702471448538509513]] Validation failed in /10.234.72.137
[2016-05-27 14:19:38,489] Repair command #37 finished
error: nodetool failed, check server logs
-- StackTrace --
java.lang.RuntimeException: nodetool failed, check server logs
at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:292)
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:204)
{code}
Unlike the first time, this error comes from the node 10.234.72.137. Checking
the log, the node failed creating a merkle tree for the table syndic. This
table is LCS
{code:none}
ERROR [ValidationExecutor:3427] 2016-05-27 14:19:38,466 Validator.java:245 -
Failed creating a merkle tree for [repair #3dd2f250-2405-11e6-b33f-ab1665ddb9c0
on pns_fr_prod/syndic, (8699512854132214411,8702471448538509513]],
/10.98.255.154 (see log for details)
ERROR [ValidationExecutor:3427] 2016-05-27 14:19:38,467
CassandraDaemon.java:227 - Exception in thread
Thread[ValidationExecutor:3427,1,main]
java.lang.AssertionError: row DecoratedKey(8699513197702636918,
49442d5350502d3130302d54513944486f7333704e68492b387a6e786870653347614d766a6832597744446a5543446d4b397a393545)
received out of order wrt DecoratedKey(8702471248515353880,
49442d5350502d3130302d704a3063676b4446714b5075644b68654273672f686c6a4759767264724f4b6e664d3176765a7a70416d6b)
at org.apache.cassandra.repair.Validator.add(Validator.java:126)
~[apache-cassandra-2.1.12.jar:2.1.12]
at
org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1038)
~[apache-cassandra-2.1.12.jar:2.1.12]
at
org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:89)
~[apache-cassandra-2.1.12.jar:2.1.12]
at
org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:649)
~[apache-cassandra-2.1.12.jar:2.1.12]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_60]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_60]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
{code}
We tried to run again the same repair on the same node with the same tokenrange
and it finished without error
{code:none}
root@node021[SPH][PROD][PnS3]:~$ nodetool repair pns_fr_prod -st
8699512854132214411 -et 8702471448538509513
[2016-05-27 14:27:00,419] Starting repair command #38, repairing 1 ranges for
keyspace pns_fr_prod (parallelism=SEQUENTIAL, full=true)
[2016-05-27 14:36:18,239] Repair session 4fcda620-2406-11e6-b33f-ab1665ddb9c0
for range (8699512854132214411,8702471448538509513] finished
[2016-05-27 14:36:18,239] Repair command #38 finished
{code}
It seems this error comes up by hazard. We notice that our first repair -pr
-par has finished repairing all the tokenranges, but that one we got the error.
> java.lang.RuntimeException: Last written key DecoratedKey >= current key
> DecoratedKey
> -------------------------------------------------------------------------------------
>
> Key: CASSANDRA-9126
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9126
> Project: Cassandra
> Issue Type: Bug
> Reporter: srinivasu gottipati
> Priority: Critical
> Attachments: cassandra-system.log
>
>
> Cassandra V: 2.0.14,
> Getting the following exceptions while trying to compact (I see this issue
> was raised in earlier versions and marked as closed. However it still appears
> in 2.0.14). In our case, compaction is not getting succeeded and keep failing
> with this error.:
> {code}java.lang.RuntimeException: Last written key
> DecoratedKey(3462767860784856708,
> 354038323137333038305f3330325f31355f474d4543454f) >= current key
> DecoratedKey(3462334604624154281,
> 354036333036353334315f3336315f31355f474d4543454f) writing into {code}
> .......
> Stacktrace:{code}
> at
> org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
> at
> org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:167)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745){code}
> Any help is greatly appreciated
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
