[
https://issues.apache.org/jira/browse/CASSANDRA-10057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victor Trac updated CASSANDRA-10057:
------------------------------------
Description:
I bootstrapped a DC2 by restoring the snapshots from DC1 into equivalent nodes
in DC2. Everything comes up just fine, but when I tried to run a {code}repair
-dcpar -j4{code} in DC2, I got this error:
{code}
[root@cassandra-i-2677cce3 ~]# nodetool repair -dcpar -j4
[2015-08-12 15:56:05,682] Nothing to repair for keyspace 'system_auth'
[2015-08-12 15:56:05,949] Starting repair command #4, repairing keyspace crawl
with repair options (parallelism: dc_parallel, primary range: false,
incremental: true, job threads: 4, ColumnFamilies: [], dataCenters: [], hosts:
[], # of ranges: 2275)
[2015-08-12 15:59:33,050] Repair session 1b8d7810-410b-11e5-b71c-71288cf05b1d
for range (-1630840392403060839,-1622173360499444177] finished (progress: 0%)
[2015-08-12 15:59:33,284] Repair session 1b92a830-410b-11e5-b71c-71288cf05b1d
for range (-2766833977081486018,-2766120936176524808] failed with error Could
not create snapshot at /10.20.144.15 (progress: 0%)
[2015-08-12 15:59:35,543] Repair session 1b8fe910-410b-11e5-b71c-71288cf05b1d
for range (5127720400742928658,5138864412691114632] finished (progress: 0%)
[2015-08-12 15:59:36,040] Repair session 1b960390-410b-11e5-b71c-71288cf05b1d
for range (749871306972906628,751065038788146229] failed with error Could not
create snapshot at /10.20.144.15 (progress: 0%)
[2015-08-12 15:59:36,454] Repair session 1b9455e0-410b-11e5-b71c-71288cf05b1d
for range (-8769666365699147423,-8767955202550789015] finished (progress: 0%)
[2015-08-12 15:59:38,765] Repair session 1b97b140-410b-11e5-b71c-71288cf05b1d
for range (-4434580467371714601,-4433394767535421669] finished (progress: 0%)
[2015-08-12 15:59:41,520] Repair session 1b99d420-410b-11e5-b71c-71288cf05b1d
for range (-1085112943862424751,-1083156277882030877] finished (progress: 0%)
[2015-08-12 15:59:43,806] Repair session 1b9da4b0-410b-11e5-b71c-71288cf05b1d
for range (2125359121242932804,2126816999370470831] failed with error Could not
create snapshot at /10.20.144.15 (progress: 0%)
[2015-08-12 15:59:43,874] Repair session 1b9ba8e0-410b-11e5-b71c-71288cf05b1d
for range (-7469857353178912795,-7459624955099554284] finished (progress: 0%)
[2015-08-12 15:59:48,384] Repair session 1b9fa080-410b-11e5-b71c-71288cf05b1d
for range (-8005238987831093686,-8005057803798566519] finished (progress: 0%)
[2015-08-12 15:59:48,392] Repair session 1ba17540-410b-11e5-b71c-71288cf05b1d
for range (7291056720707652994,7292508243124389877] failed with error Could not
create snapshot at /10.20.144.15 (progress: 0%)
{code}
It seems like now that all 4 threads ran into an error, the repair process just
sits forever.
Looking at 10.20.144.15, I see this:
{code}
ERROR [AntiEntropyStage:2] 2015-08-12 15:59:35,965
RepairMessageVerbHandler.java:95 - Cannot start multiple repair sessions over
the same sstables
ERROR [AntiEntropyStage:2] 2015-08-12 15:59:35,966
RepairMessageVerbHandler.java:153 - Got error, removing parent repair session
ERROR [AntiEntropyStage:2] 2015-08-12 15:59:35,966 CassandraDaemon.java:182 -
Exception in thread Thread[AntiEntropyStage:2,5,main]
java.lang.RuntimeException: java.lang.RuntimeException: Cannot start multiple
repair sessions over the same sstables
at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:156)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_85]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_85]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_85]
Caused by: java.lang.RuntimeException: Cannot start multiple repair sessions
over the same sstables
at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:96)
~[apache-cassandra-2.2.0.jar:2.2.0]
... 4 common frames omitted
ERROR [AntiEntropyStage:3] 2015-08-12 15:59:38,722
RepairMessageVerbHandler.java:153 - Got error, removing parent repair session
ERROR [AntiEntropyStage:3] 2015-08-12 15:59:38,723 CassandraDaemon.java:182 -
Exception in thread Thread[AntiEntropyStage:3,5,main]
java.lang.RuntimeException: java.lang.NullPointerException
at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:156)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_85]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_85]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_85]
Caused by: java.lang.NullPointerException: null
at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:98)
~[apache-cassandra-2.2.0.jar:2.2.0]
... 4 common frames omitted
...
{code}
I thought maybe the -j4 was causing the issue, but when I run it without the -j
param, I get the same error but on a different node.
It seems like there are two problems:
* The error on the node: {code}RepairMessageVerbHandler.java:95 - Cannot start
multiple repair sessions over the same sstables{code}
* The repair process on the initiating node just sits there forever after the
threads ran into an error. It seems to me that it should exit out and report
back an error to the user.
was:
I bootstrapped a DC2 by restoring the snapshots from DC1 into equivalent nodes
in DC2. Everything comes up just fine, but when I tried to run a {code}repair
-dcpar -j4{code} in DC2, I got this error:
{code}
[root@cassandra-i-2677cce3 ~]# nodetool repair -dcpar -j4
[2015-08-12 15:56:05,682] Nothing to repair for keyspace 'system_auth'
[2015-08-12 15:56:05,949] Starting repair command #4, repairing keyspace crawl
with repair options (parallelism: dc_parallel, primary range: false,
incremental: true, job threads: 4, ColumnFamilies: [], dataCenters: [], hosts:
[], # of ranges: 2275)
[2015-08-12 15:59:33,050] Repair session 1b8d7810-410b-11e5-b71c-71288cf05b1d
for range (-1630840392403060839,-1622173360499444177] finished (progress: 0%)
[2015-08-12 15:59:33,284] Repair session 1b92a830-410b-11e5-b71c-71288cf05b1d
for range (-2766833977081486018,-2766120936176524808] failed with error Could
not create snapshot at /10.20.144.15 (progress: 0%)
[2015-08-12 15:59:35,543] Repair session 1b8fe910-410b-11e5-b71c-71288cf05b1d
for range (5127720400742928658,5138864412691114632] finished (progress: 0%)
[2015-08-12 15:59:36,040] Repair session 1b960390-410b-11e5-b71c-71288cf05b1d
for range (749871306972906628,751065038788146229] failed with error Could not
create snapshot at /10.20.144.15 (progress: 0%)
[2015-08-12 15:59:36,454] Repair session 1b9455e0-410b-11e5-b71c-71288cf05b1d
for range (-8769666365699147423,-8767955202550789015] finished (progress: 0%)
[2015-08-12 15:59:38,765] Repair session 1b97b140-410b-11e5-b71c-71288cf05b1d
for range (-4434580467371714601,-4433394767535421669] finished (progress: 0%)
[2015-08-12 15:59:41,520] Repair session 1b99d420-410b-11e5-b71c-71288cf05b1d
for range (-1085112943862424751,-1083156277882030877] finished (progress: 0%)
[2015-08-12 15:59:43,806] Repair session 1b9da4b0-410b-11e5-b71c-71288cf05b1d
for range (2125359121242932804,2126816999370470831] failed with error Could not
create snapshot at /10.20.144.15 (progress: 0%)
[2015-08-12 15:59:43,874] Repair session 1b9ba8e0-410b-11e5-b71c-71288cf05b1d
for range (-7469857353178912795,-7459624955099554284] finished (progress: 0%)
[2015-08-12 15:59:48,384] Repair session 1b9fa080-410b-11e5-b71c-71288cf05b1d
for range (-8005238987831093686,-8005057803798566519] finished (progress: 0%)
[2015-08-12 15:59:48,392] Repair session 1ba17540-410b-11e5-b71c-71288cf05b1d
for range (7291056720707652994,7292508243124389877] failed with error Could not
create snapshot at /10.20.144.15 (progress: 0%)
{code}
It seems like now that all 4 threads ran into an error, the repair process just
sits forever.
Looking at 10.20.144.15, I see this:
{code}
ERROR [AntiEntropyStage:2] 2015-08-12 15:59:35,965
RepairMessageVerbHandler.java:95 - Cannot start multiple repair sessions over
the same sstables
ERROR [AntiEntropyStage:2] 2015-08-12 15:59:35,966
RepairMessageVerbHandler.java:153 - Got error, removing parent repair session
ERROR [AntiEntropyStage:2] 2015-08-12 15:59:35,966 CassandraDaemon.java:182 -
Exception in thread Thread[AntiEntropyStage:2,5,main]
java.lang.RuntimeException: java.lang.RuntimeException: Cannot start multiple
repair sessions over the same sstables
at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:156)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_85]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_85]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_85]
Caused by: java.lang.RuntimeException: Cannot start multiple repair sessions
over the same sstables
at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:96)
~[apache-cassandra-2.2.0.jar:2.2.0]
... 4 common frames omitted
ERROR [AntiEntropyStage:3] 2015-08-12 15:59:38,722
RepairMessageVerbHandler.java:153 - Got error, removing parent repair session
ERROR [AntiEntropyStage:3] 2015-08-12 15:59:38,723 CassandraDaemon.java:182 -
Exception in thread Thread[AntiEntropyStage:3,5,main]
java.lang.RuntimeException: java.lang.NullPointerException
at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:156)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_85]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_85]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_85]
Caused by: java.lang.NullPointerException: null
at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:98)
~[apache-cassandra-2.2.0.jar:2.2.0]
... 4 common frames omitted
...
{code}
I thought maybe the -j4 was causing the issue, but when I run it without the -j
param, I get the same error but on a different node.
It seems like there are two problems:
# {code}RepairMessageVerbHandler.java:95 - Cannot start multiple repair
sessions over the same sstables{code}
# The repair process on the initiating node just sits there forever after the
threads ran into an error. It seems to me that it should exit out and report
back an error to the user.
> RepairMessageVerbHandler.java:95 - Cannot start multiple repair sessions over
> the same sstables
> -----------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-10057
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10057
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Amazon Linux: 3.14.48-33.39.amzn1.x86_64
> java version "1.7.0_85"
> OpenJDK Runtime Environment (amzn-2.6.1.3.61.amzn1-x86_64 u85-b01)
> OpenJDK 64-Bit Server VM (build 24.85-b03, mixed mode)
> Cassandra RPM: cassandra22-2.2.0-1.noarch
> Reporter: Victor Trac
> Fix For: 2.2.x
>
>
> I bootstrapped a DC2 by restoring the snapshots from DC1 into equivalent
> nodes in DC2. Everything comes up just fine, but when I tried to run a
> {code}repair -dcpar -j4{code} in DC2, I got this error:
> {code}
> [root@cassandra-i-2677cce3 ~]# nodetool repair -dcpar -j4
> [2015-08-12 15:56:05,682] Nothing to repair for keyspace 'system_auth'
> [2015-08-12 15:56:05,949] Starting repair command #4, repairing keyspace
> crawl with repair options (parallelism: dc_parallel, primary range: false,
> incremental: true, job threads: 4, ColumnFamilies: [], dataCenters: [],
> hosts: [], # of ranges: 2275)
> [2015-08-12 15:59:33,050] Repair session 1b8d7810-410b-11e5-b71c-71288cf05b1d
> for range (-1630840392403060839,-1622173360499444177] finished (progress: 0%)
> [2015-08-12 15:59:33,284] Repair session 1b92a830-410b-11e5-b71c-71288cf05b1d
> for range (-2766833977081486018,-2766120936176524808] failed with error Could
> not create snapshot at /10.20.144.15 (progress: 0%)
> [2015-08-12 15:59:35,543] Repair session 1b8fe910-410b-11e5-b71c-71288cf05b1d
> for range (5127720400742928658,5138864412691114632] finished (progress: 0%)
> [2015-08-12 15:59:36,040] Repair session 1b960390-410b-11e5-b71c-71288cf05b1d
> for range (749871306972906628,751065038788146229] failed with error Could not
> create snapshot at /10.20.144.15 (progress: 0%)
> [2015-08-12 15:59:36,454] Repair session 1b9455e0-410b-11e5-b71c-71288cf05b1d
> for range (-8769666365699147423,-8767955202550789015] finished (progress: 0%)
> [2015-08-12 15:59:38,765] Repair session 1b97b140-410b-11e5-b71c-71288cf05b1d
> for range (-4434580467371714601,-4433394767535421669] finished (progress: 0%)
> [2015-08-12 15:59:41,520] Repair session 1b99d420-410b-11e5-b71c-71288cf05b1d
> for range (-1085112943862424751,-1083156277882030877] finished (progress: 0%)
> [2015-08-12 15:59:43,806] Repair session 1b9da4b0-410b-11e5-b71c-71288cf05b1d
> for range (2125359121242932804,2126816999370470831] failed with error Could
> not create snapshot at /10.20.144.15 (progress: 0%)
> [2015-08-12 15:59:43,874] Repair session 1b9ba8e0-410b-11e5-b71c-71288cf05b1d
> for range (-7469857353178912795,-7459624955099554284] finished (progress: 0%)
> [2015-08-12 15:59:48,384] Repair session 1b9fa080-410b-11e5-b71c-71288cf05b1d
> for range (-8005238987831093686,-8005057803798566519] finished (progress: 0%)
> [2015-08-12 15:59:48,392] Repair session 1ba17540-410b-11e5-b71c-71288cf05b1d
> for range (7291056720707652994,7292508243124389877] failed with error Could
> not create snapshot at /10.20.144.15 (progress: 0%)
> {code}
> It seems like now that all 4 threads ran into an error, the repair process
> just sits forever.
> Looking at 10.20.144.15, I see this:
> {code}
> ERROR [AntiEntropyStage:2] 2015-08-12 15:59:35,965
> RepairMessageVerbHandler.java:95 - Cannot start multiple repair sessions over
> the same sstables
> ERROR [AntiEntropyStage:2] 2015-08-12 15:59:35,966
> RepairMessageVerbHandler.java:153 - Got error, removing parent repair session
> ERROR [AntiEntropyStage:2] 2015-08-12 15:59:35,966 CassandraDaemon.java:182 -
> Exception in thread Thread[AntiEntropyStage:2,5,main]
> java.lang.RuntimeException: java.lang.RuntimeException: Cannot start multiple
> repair sessions over the same sstables
> at
> org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:156)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[na:1.7.0_85]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[na:1.7.0_85]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_85]
> Caused by: java.lang.RuntimeException: Cannot start multiple repair sessions
> over the same sstables
> at
> org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:96)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> ... 4 common frames omitted
> ERROR [AntiEntropyStage:3] 2015-08-12 15:59:38,722
> RepairMessageVerbHandler.java:153 - Got error, removing parent repair session
> ERROR [AntiEntropyStage:3] 2015-08-12 15:59:38,723 CassandraDaemon.java:182 -
> Exception in thread Thread[AntiEntropyStage:3,5,main]
> java.lang.RuntimeException: java.lang.NullPointerException
> at
> org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:156)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[na:1.7.0_85]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[na:1.7.0_85]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_85]
> Caused by: java.lang.NullPointerException: null
> at
> org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:98)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> ... 4 common frames omitted
> ...
> {code}
> I thought maybe the -j4 was causing the issue, but when I run it without the
> -j param, I get the same error but on a different node.
> It seems like there are two problems:
> * The error on the node: {code}RepairMessageVerbHandler.java:95 - Cannot
> start multiple repair sessions over the same sstables{code}
> * The repair process on the initiating node just sits there forever after the
> threads ran into an error. It seems to me that it should exit out and report
> back an error to the user.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
