[
https://issues.apache.org/jira/browse/IGNITE-17056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mirza Aliev updated IGNITE-17056:
---------------------------------
Epic Link: IGNITE-14209
> Implement rebalance cancel mechanism
> ------------------------------------
>
> Key: IGNITE-17056
> URL: https://issues.apache.org/jira/browse/IGNITE-17056
> Project: Ignite
> Issue Type: Task
> Reporter: Mirza Aliev
> Priority: Major
> Labels: ignite-3
>
> There are cases when a current leader cannot perform rebalance on specified
> set of nodes, for example, when some node from the raft group permanently
> fails with \{{RaftError#ECATCHUP}}. For such scenario retry mechanism is
> implemented in IGNITE-16801, but we cannot retry rebalance intent infinitely,
> so there should be implemented mechanism for canceling a rebalance.
> Naive canceling could be implemented by removing {{pending key}} and
> replacing it with {{planned key}}. But this approach has several crucial
> limitations and may cause inconsistency in the current rebalance protocol,
> for example, when there is a race between cancel and applying new assignment
> to the {{stable key}} from the new leader. We can remove {{pending key}}
> right before applying new assignment to the {{stable key}}, so we cannot
> resolve peers to ClusterIds, which is made on a union of pending and stable
> keys.
> Also there is a case, when we can lost planned rebalance:
> # Current leader retries failed rebalance
> # Current leader stops being leader for some reasons and sleeps
> # New leader performs rebalance and calls
> {{RebalanceRaftGroupEventsListener#onNewPeersConfigurationApplied}}
> # At this moment old leader wakes up and cancels the current rebalance, so
> it removes pending and writes to it planned key.
> # At this moment we receive
> {{RebalanceRaftGroupEventsListener#onNewPeersConfigurationApplied}} from the
> new leader, see that planned is empty, so we just delete pending key, but
> this is not correct to delete this key as far as the rebalance that is
> associated to the removed key hasn't been performed yet.
> Also we should consider separating scenarios for recoverable and
> unrecoverable errors, because it might be useless to retry rebalance, if some
> participating node fails with unrecoverable error.
> Seems like we should properly think about introducing some failure handling
> for such exceptional scenarios.
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
