[
https://issues.apache.org/jira/browse/CASSANDRA-10070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15147751#comment-15147751
]
Paulo Motta edited comment on CASSANDRA-10070 at 2/15/16 8:07 PM:
------------------------------------------------------------------
Starting with a single repair per dc and adding support for parallel repair
sessions later sounds like a good idea.
bq. I agree and we could probably store the parent repair session id in an
extra column of the lock table and have a thread wake up periodically to see if
there are repair sessions without locks.
Do we intend to reuse the lock table for other maintenance tasks as well? If
so, we must add a generic "holder" column to the lock table so we can reuse to
identify resources other than the parent repair session in the future. We could
also add an "attributes" map in the lock table to store additional attributes
such as status, or have a separate table to maintain status to keep the lock
table simple.
bq. But then we must somehow be able to differentiate user-defined and
automatically scheduled repair sessions. It could be done by having all repairs
go through this scheduling interface, which also would reduce user mistakes
with multiple repairs in parallel. Another alternative is to have a custom flag
in the parent repair that makes the garbage collector ignore it if it's
user-defined. I think that the garbage collector/cancel repairs when unable to
lock feature is something that should be included in the first pass.
Ideally all repairs would go through this interface, but this would probably
add complexity at this stage. So we should probably just add a "lockResource"
attribute to each repair session object, and each node would go through all
repairs currently running checking if it still holds the lock in case the
"lockResource" field is set.
bq. The most basic failure scenarios should be covered by retrying a repair if
it fails and log a warning/error based on how many times it failed. Could the
retry behaviour cause some unexpected consequences?
It would probably be safe to abort ongoing validation and stream background
tasks and cleanup repair state on all involved nodes before starting a new
repair session in the same ranges. This doesn't seem to be done currently. As
far as I understood, if there are nodes A, B, C running repair, A is the
coordinator. If validation or streaming fails on node B, the coordinator (A) is
notified and fails the repair session, but node C will remain doing validation
and/or streaming, what could cause problems (or increased load) if we start
another repair session on the same range.
We will probably need to extend the repair protocol to perform this
cleanup/abort step on failure. We already have a legacy cleanup message that
doesn't seem to be used in the current protocol that we could maybe reuse to
cleanup repair state after a failure. This repair abortion will probably have
intersection with CASSANDRA-3486. In any case, this is a separate (but related)
issue and we should address it in an independent ticket, and make this ticket
dependent on that.
Another unrelated option that we should probably include in the future is a
timeout, and abort repair sessions running longer than that.
was (Author: pauloricardomg):
Starting with a single repair per dc and adding support for parallel repair
sessions later sounds like a good idea.
bq. I agree and we could probably store the parent repair session id in an
extra column of the lock table and have a thread wake up periodically to see if
there are repair sessions without locks.
Do we intend to reuse the lock table for other maintenance tasks as well? If
so, we must add a generic "holder" column to the lock table so we can reuse to
identify resources other than the parent repair session in the future. We could
also add an "attributes" map in the lock table to store additional attributes
such as status, or have a separate table to maintain status to keep the lock
table simple.
bq. But then we must somehow be able to differentiate user-defined and
automatically scheduled repair sessions. It could be done by having all repairs
go through this scheduling interface, which also would reduce user mistakes
with multiple repairs in parallel. Another alternative is to have a custom flag
in the parent repair that makes the garbage collector ignore it if it's
user-defined. I think that the garbage collector/cancel repairs when unable to
lock feature is something that should be included in the first pass.
Ideally all repairs would go through this interface, but this would probably
add complexity at this stage. So we should probably just add a "lockResource"
attribute to each local repair session object (as opposed to only the parent
repair object), and each node would go through all repairs currently running
checking if it still holds the lock if the "lockResource" field is set.
bq. The most basic failure scenarios should be covered by retrying a repair if
it fails and log a warning/error based on how many times it failed. Could the
retry behaviour cause some unexpected consequences?
It would probably be safe to abort ongoing validation and stream background
tasks and cleanup repair state on all involved nodes before starting a new
repair session in the same ranges. This doesn't seem to be done currently. As
far as I understood, if there are nodes A, B, C running repair, A is the
coordinator. If validation or streaming fails on node B, the coordinator (A) is
notified and fails the repair session, but node C will remain doing validation
and/or streaming, what could cause problems (or increased load) if we start
another repair session on the same range.
We will probably need to extend the repair protocol to perform this
cleanup/abort step on failure. We already have a legacy cleanup message that
doesn't seem to be used in the current protocol that we could maybe reuse to
cleanup repair state after a failure. This repair abortion will probably have
intersection with CASSANDRA-3486. In any case, this is a separate (but related)
issue and we should address it in an independent ticket, and make this ticket
dependent on that.
Another unrelated option that we should probably include in the future is a
timeout, and abort repair sessions running longer than that.
> Automatic repair scheduling
> ---------------------------
>
> Key: CASSANDRA-10070
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10070
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Marcus Olsson
> Assignee: Marcus Olsson
> Priority: Minor
> Fix For: 3.x
>
> Attachments: Distributed Repair Scheduling.doc
>
>
> Scheduling and running repairs in a Cassandra cluster is most often a
> required task, but this can both be hard for new users and it also requires a
> bit of manual configuration. There are good tools out there that can be used
> to simplify things, but wouldn't this be a good feature to have inside of
> Cassandra? To automatically schedule and run repairs, so that when you start
> up your cluster it basically maintains itself in terms of normal
> anti-entropy, with the possibility for manual configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
