The verify and scrub went without any error on the keyspace. I ran it again
with trace mode and still the same issue
[2017-06-29 21:37:45,578] Parsing UPDATE
system_distributed.parent_repair_history SET finished_at =
toTimestamp(now()), successful_ranges = {'....} WHERE
parent_id=f1f10af0-5d12-11e7-8df9-59d19ef3dd23
[2017-06-29 21:37:45,580] Preparing statement
[2017-06-29 21:37:45,580] Determining replicas for mutation
[2017-06-29 21:37:45,580] Sending MUTATION message to /xx.xx.xx.95
[2017-06-29 21:37:45,580] Sending MUTATION message to /xx.xx.xx.94
[2017-06-29 21:37:45,580] Sending MUTATION message to /xx.xx.xx.93
[2017-06-29 21:37:45,581] REQUEST_RESPONSE message received from
/xx.xx.xx.93
[2017-06-29 21:37:45,581] REQUEST_RESPONSE message received from
/xx.xx.xx.94
[2017-06-29 21:37:45,581] Processing response from /xx.xx.xx.93
[2017-06-29 21:37:45,581] /xx.xx.xx.94: MUTATION message received from
/xx.xx.xx.91
[2017-06-29 21:37:45,582] Processing response from /xx.xx.xx.94
[2017-06-29 21:37:45,582] /xx.xx.xx.93: MUTATION message received from
/xx.xx.xx.91
[2017-06-29 21:37:45,582] /xx.xx.xx.95: MUTATION message received from
/xx.xx.xx.91
[2017-06-29 21:37:45,582] /xx.xx.xx.94: Appending to commitlog
[2017-06-29 21:37:45,582] /xx.xx.xx.94: Adding to parent_repair_history
memtable
[2017-06-29 21:37:45,582] Some repair failed
[2017-06-29 21:37:45,582] Repair command #3 finished in 1 minute 44 seconds
error: Repair job has failed with the error message: [2017-06-29
21:37:45,582] Some repair failed
-- StackTrace --
java.lang.RuntimeException: Repair job has failed with the error message:
[2017-06-29 21:37:45,582] Some repair failed
at org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:116)
at
org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:583)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:533)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108)
On Thu, Jun 29, 2017 at 1:36 PM, Subroto Barua <sbarua...@yahoo.com.invalid>
wrote:
> Balaji,
>
> Are you repairing a specific keyspace/table? if the failure is tied to a
> table, try 'verify' and 'scrub' options on .91...see if you get any errors.
>
>
>
>
> On Thursday, June 29, 2017, 12:12:14 PM PDT, Balaji Venkatesan <
> venkatesan.bal...@gmail.com> wrote:
>
>
> Thanks. I tried with trace option and there is not much info. Here are the
> few log lines just before it failed.
>
>
> [2017-06-29 19:01:54,969] /xx.xx.xx.93: Sending REPAIR_MESSAGE message to
> /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Appending to commitlog
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Adding to repair_history memtable
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Enqueuing response to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message
> to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message
> to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message
> to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message
> to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message
> to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message
> to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message
> to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message
> to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message
> to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message
> to /xx.xx.xx.91
> [2017-06-29 19:01:54,969] /xx.xx.xx.92: Sending REQUEST_RESPONSE message
> to /xx.xx.xx.91
> [2017-06-29 19:02:04,842] Some repair failed
> [2017-06-29 19:02:04,848] Repair command #1 finished in 1 minute 2 seconds
> error: Repair job has failed with the error message: [2017-06-29
> 19:02:04,842] Some repair failed
> -- StackTrace --
> java.lang.RuntimeException: Repair job has failed with the error message:
> [2017-06-29 19:02:04,842] Some repair failed
> at org.apache.cassandra.tools. RepairRunner.progress(
> RepairRunner.java:116)
> at org.apache.cassandra.utils. progress.jmx.
> JMXNotificationProgressListene r.handleNotification(
> JMXNotificationProgressListene r.java:77)
> at com.sun.jmx.remote.internal. ClientNotifForwarder$ NotifFetcher.
> dispatchNotification( ClientNotifForwarder.java:583)
> at com.sun.jmx.remote.internal. ClientNotifForwarder$ NotifFetcher.doRun(
> ClientNotifForwarder.java:533)
> at com.sun.jmx.remote.internal. ClientNotifForwarder$ NotifFetcher.run(
> ClientNotifForwarder.java:452)
> at com.sun.jmx.remote.internal. ClientNotifForwarder$
> LinearExecutor$1.run( ClientNotifForwarder.java:108)
>
>
>
> FYI I am running repair from xx.xx.xx.91 node and its a 5 node cluster
> xx.xx.xx.91-xx.xx.xx.95
>
> On Wed, Jun 28, 2017 at 5:16 PM, Akhil Mehra <akhilme...@gmail.com> wrote:
>
> nodetool repair has a trace option
>
> nodetool repair -tr yourkeyspacename
>
> see if that provides you with additional information.
>
> Regards,
> Akhil
>
> On 28/06/2017, at 2:25 AM, Balaji Venkatesan <venkatesan.bal...@gmail.com>
> wrote:
>
>
> We use Apache Cassandra 3.10-13
>
> On Jun 26, 2017 8:41 PM, "Michael Shuler" <mich...@pbandjelly.org> wrote:
>
> What version of Cassandra?
>
> --
> Michael
>
> On 06/26/2017 09:53 PM, Balaji Venkatesan wrote:
> > Hi All,
> >
> > When I run nodetool repair on a keyspace I constantly get "Some repair
> > failed" error, there are no sufficient info to debug more. Any help?
> >
> > Here is the stacktrace
> >
> > ============================== ============================== ==========
> > [2017-06-27 02:44:34,275] Some repair failed
> > [2017-06-27 02:44:34,279] Repair command #3 finished in 33 seconds
> > error: Repair job has failed with the error message: [2017-06-27
> > 02:44:34,275] Some repair failed
> > -- StackTrace --
> > java.lang.RuntimeException: Repair job has failed with the error
> > message: [2017-06-27 02:44:34,275] Some repair failed
> > at org.apache.cassandra.tools.Rep airRunner.progress(RepairRunne
> r.java:116)
> > at
> > org.apache.cassandra.utils.pro gress.jmx.JMXNotificationProgr
> essListener. handleNotification(JMXNotifica tionProgressListener.java:77)
> > at
> > com.sun.jmx.remote.internal.Cl ientNotifForwarder$NotifFetche
> r.dispatchNotification(ClientN otifForwarder.java:583)
> > at
> > com.sun.jmx.remote.internal.Cl ientNotifForwarder$NotifFetche
> r.doRun(ClientNotifForwarder. java:533)
> > at
> > com.sun.jmx.remote.internal.Cl ientNotifForwarder$NotifFetche
> r.run(ClientNotifForwarder. java:452)
> > at
> > com.sun.jmx.remote.internal.Cl ientNotifForwarder$LinearExecu
> tor$1.run(ClientNotifForwarder .java:108)
> > ============================== ============================== ==========
> >
> >
> > --
> > Thanks,
> > Balaji Venkatesan.
>
>
> ------------------------------ ------------------------------ ---------
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apa che.org
> <user-unsubscr...@cassandra.apache.org>
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>
>
>
>
>
> --
> Thanks,
> Balaji Venkatesan.
>
--
Thanks,
Balaji Venkatesan.
