[
https://issues.apache.org/jira/browse/IGNITE-23587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17894673#comment-17894673
]
Mikhail Efremov commented on IGNITE-23587:
------------------------------------------
This ticket is required to fix possible livelocks on rebalance through primary
replica
> Make PlacementDriver able to choose lease candidate from pendings
> -----------------------------------------------------------------
>
> Key: IGNITE-23587
> URL: https://issues.apache.org/jira/browse/IGNITE-23587
> Project: Ignite
> Issue Type: Improvement
> Reporter: Mikhail Efremov
> Assignee: Mikhail Efremov
> Priority: Blocker
> Labels: ignite-3, lease
>
> *Description*
> While working under IGNITE-22036 there was found 2 exceptional scenarios:
> _Recovery case_
> {code:java}
> nodes = [A, B]
> parts = 1
> replicas = 1
> | starting nodes
> v
> 13_part_0 stable = [B]
> B hosts replicaGrpId=13_part_0
> B becomes PR
> |
> v
> A starts
> | <nothing happens>
> v
> B is shutdowned
> | <do reset on pending = [A]>
> v
> A::TableManager::handlePending fails (A not a primary)
> A starts voting and became a leader term=2
> | ReplicaStateManager fails (A not a primary)
> v
> LeaseUpdater#updateBatchLeaseInternal
> <---------------------------------------\
> | iterate over assignments == [B]
> |
> v
> |
> topologyTracker#nodeByConsistentId
> |
> | returns null
> |
> v
> |
> leaseUpdateStatistics#onLeaseWithoutCandidate();
> |
> | continue
> |
> v
> |
> LOG.debug("No leases to update found.") // don't see in log because debug lvl
> |
> |
> |
> \-----------------------------------------------------------------------------/
> {code}
> _Stable dies, change peers wasn't done_
> {code:java}
> nodes = [A, B]
> parts = 1
> replicas = 1
> | starting nodes
> v
> 13_part_0 stable = [B]
> B hosts replicaGrpId=13_part_0
> B becomes PR
> |
> v
> A starts
> | <filter={A}>
> v
> A::TM#handleChangePendingsAssignmentsEvent
> A::ReplicaManager#startReplica
> A::RaftGroupService#updatePeersAndLearners
> | <ready for catch up>
> v
> B::TM#handleChangePendingsAssignmentsEvent
> B::TM#sendChangePeersAndLearnersReplicaRequest
> B::ReplicaPartitionListener#processChangePeersAndLearnersReplicaRequest
> B::RaftGroupService#changePeersAndLearnersAsync
> | <B dies>
> v
> <No primary => no one would process failover>
> <All TM#handleChangePendingsAssignmentsEvent are finished previously =>
> => no one would send ChangePeersAndLearnersReplicaRequest>
> |
> v
> LeaseUpdater#updateBatchLeaseInternal
> <---------------------------------------\
> | iterate over assignments == [B]
> |
> v
> |
> topologyTracker#nodeByConsistentId
> |
> | returns null
> |
> v
> |
> leaseUpdateStatistics#onLeaseWithoutCandidate();
> |
> | continue
> |
> v
> |
> LOG.debug("No leases to update found.") // don't see in log because debug lvl
> |
> |
> |
> \-----------------------------------------------------------------------------/
> {code}
> As we can see, in both cases we have a situation where we can't re-grant
> lease and then neither send {{ChangePeersAndLearnersReplicaRequest}} nor
> trigger {{onLeaderElected}} callback that was hooked after
> {{PRIMARY_REPLICA_ELECTED}}. And if in the first scenario there is possible
> workaround through sending the replica request with ignoring of primary
> replica nod colocation, but in the second scenario all possible pending
> handlers are already finished and wouldn't be called further.
> This situation leads us to an idea that we may take a closer look on lease
> granting process. For now we're checking only stable assignments for a new
> lease holder candidate. So, while we didn't finished assignments stable
> switch (call of {{onNewPeersConfigurationApplied}} after full RAFT rebalabce
> is done), we will nave {{null}} candidate and stucks in live lock.
> A possible solution is to check pendings assignments too in case if on stable
> assignments we haven't got any possibe lease candidate. But there we have an
> assumption that granting lease for a candidate from pending assignments won't
> be a problem, because we may accept lease only if candidate replica can
> retrieve replication group leader (see
> {{ReplicaImpl#processLeaseGrantedMessage}}, mostly the first action is to get
> the leader), then we're almost able to process requests: at least we will
> propose leader-colocated replica or (see the 2nd case) will waiting if leader
> will be elected.
> There should be a note about about the 2nd case: if B node dies before catch
> up is done and before RAFT group de-facto changed a configuration and then A
> didn't and unable to elect itself as a leader, we have a situation as now
> when partition is broken because the only replica is unavailable.
> *Motivation*
> The main motivation there is to make IGNITE-22036 and a whole epic
> IGNITE-22313 be able to handle rebalance even if we have no any active node
> in stable assignments because assignments stable switch unable to be called.
> *Definition of done*
> # {{AssignmentsTracker}} is able to track both stable and pending assignments.
> # While {{LeaseUpdater#Updater#updateLeaseBatchInternal}} if on lease
> expiration {{LeaseUpdater#nextLeaseHolder}} returns {{null}}, then it now
> should gets pending assignments from {{AssignmentsTracker}} and choose a
> lease candidate among them.
> # Before IGNITE-22036 is done this changes shouldn't breaks any tests.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
