[
https://issues.apache.org/jira/browse/CASSANDRA-8316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14241147#comment-14241147
]
Alan Boudreault commented on CASSANDRA-8316:
--------------------------------------------
[~krummas] [~yukim] With some test runs, I cannot see the high CPU utilization
issue again. However, I still see the error message. Also I've noticed an
important changes between with and without the patch.
WITHOUT the patch: I can re-run the increment repairs. I might get again the
error message on the node that initially failed, but things will get OK after
the initial endpoints that failed are repaired.
WITH the patch: I cannot do an incremental repairs anymore, even after a
restart. This is what I get trying to run the repairs on my node:
{code}
aboudreault@kovarro:~/dev/cstar/8316$ ccm node1 nodetool -- repair -par -inc
[2014-12-10 09:00:42,767] Starting repair command #1, repairing 3 ranges for
keyspace r1 (parallelism=PARALLEL, full=false)
[2014-12-10 09:00:48,045] Repair session ee2a78c0-8074-11e4-9b59-bbfe19a8e904
for range (4611686018427387904,6917529027641081856] finished
[2014-12-10 09:00:48,046] Repair session ef77e050-8074-11e4-9b59-bbfe19a8e904
for range (2305843009213693952,4611686018427387904] finished
[2014-12-10 09:00:48,048] Repair session f06107d0-8074-11e4-9b59-bbfe19a8e904
for range (6917529027641081856,-9223372036854775808] finished
[2014-12-10 09:00:48,078] Repair command #1 finished
[2014-12-10 09:00:48,088] Nothing to repair for keyspace 'system'
[2014-12-10 09:00:48,104] Starting repair command #2, repairing 2 ranges for
keyspace system_traces (parallelism=PARALLEL, full=false)
[2014-12-10 09:00:58,916] Repair failed with error Did not get positive replies
from all endpoints. List of failed endpoint(s): [127.0.0.2]
aboudreault@kovarro:~/dev/cstar/8316$ ccm node2 nodetool -- repair -par -inc
[2014-12-10 09:01:07,233] Starting repair command #1, repairing 3 ranges for
keyspace r1 (parallelism=PARALLEL, full=false)
[2014-12-10 09:01:07,239] Repair failed with error Already repairing
SSTableReader(path='/home/aboudreault/.ccm/local/node2/data/r1/Standard1-c38dd6f0807111e494d8bbfe19a8e904/r1-Standard1-ka-5-Data.db'),
can not continue.
[2014-12-10 09:01:07,247] Nothing to repair for keyspace 'system'
[2014-12-10 09:01:07,252] Starting repair command #2, repairing 2 ranges for
keyspace system_traces (parallelism=PARALLEL, full=false)
[2014-12-10 09:01:07,254] Repair failed with error null
{code}
Does this help?
> "Did not get positive replies from all endpoints" error on incremental repair
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-8316
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8316
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: cassandra 2.1.2
> Reporter: Loic Lambiel
> Assignee: Marcus Eriksson
> Fix For: 2.1.3
>
> Attachments: 0001-patch.patch,
> CassandraDaemon-2014-11-25-2.snapshot.tar.gz, test.sh
>
>
> Hi,
> I've got an issue with incremental repairs on our production 15 nodes 2.1.2
> (new cluster, not yet loaded, RF=3)
> After having successfully performed an incremental repair (-par -inc) on 3
> nodes, I started receiving "Repair failed with error Did not get positive
> replies from all endpoints." from nodetool on all remaining nodes :
> [2014-11-14 09:12:36,488] Starting repair command #3, repairing 108 ranges
> for keyspace xxxx (seq=false, full=false)
> [2014-11-14 09:12:47,919] Repair failed with error Did not get positive
> replies from all endpoints.
> All the nodes are up and running and the local system log shows that the
> repair commands got started and that's it.
> I've also noticed that soon after the repair, several nodes started having
> more cpu load indefinitely without any particular reason (no tasks / queries,
> nothing in the logs). I then restarted C* on these nodes and retried the
> repair on several nodes, which were successful until facing the issue again.
> I tried to repro on our 3 nodes preproduction cluster without success
> It looks like I'm not the only one having this issue:
> http://www.mail-archive.com/user%40cassandra.apache.org/msg39145.html
> Any idea?
> Thanks
> Loic
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
