[
https://issues.apache.org/jira/browse/IGNITE-23587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Efremov updated IGNITE-23587:
-------------------------------------
Description:
*Description*
While working under IGNITE-22036 there was found 2 exceptional scenarios:
_Recovery case_
{code:java}
nodes = [A, B]
parts = 1
replicas = 1
| starting nodes
v
13_part_0 stable = [B]
B hosts replicaGrpId=13_part_0
B becomes PR
|
v
A starts
| <nothing happens>
v
B is shutdowned
| <do reset on pending = [A]>
v
A::TableManager::handlePending fails (A not a primary)
A starts voting and became a leader term=2
| ReplicaStateManager fails (A not a primary)
v
LeaseUpdater#updateBatchLeaseInternal <---------------------------------------\
| iterate over assignments == [B] |
v |
topologyTracker#nodeByConsistentId |
| returns null |
v |
leaseUpdateStatistics#onLeaseWithoutCandidate(); |
| continue |
v |
LOG.debug("No leases to update found.") // don't see in log because debug lvl |
| |
\-----------------------------------------------------------------------------/
{code}
_Stable dies, change peers wasn't done_
{code:java}
nodes = [A, B]
parts = 1
replicas = 1
| starting nodes
v
13_part_0 stable = [B]
B hosts replicaGrpId=13_part_0
B becomes PR
|
v
A starts
| <filter={A}>
v
A::TM#handleChangePendingsAssignmentsEvent
A::ReplicaManager#startReplica
A::RaftGroupService#updatePeersAndLearners
| <ready for catch up>
v
B::TM#handleChangePendingsAssignmentsEvent
B::TM#sendChangePeersAndLearnersReplicaRequest
B::ReplicaPartitionListener#processChangePeersAndLearnersReplicaRequest
B::RaftGroupService#changePeersAndLearnersAsync
| <B dies>
v
<No primary => no one would process failover>
<All TM#handleChangePendingsAssignmentsEvent are finished previously =>
=> no one would send ChangePeersAndLearnersReplicaRequest>
|
v
LeaseUpdater#updateBatchLeaseInternal <---------------------------------------\
| iterate over assignments == [B] |
v |
topologyTracker#nodeByConsistentId |
| returns null |
v |
leaseUpdateStatistics#onLeaseWithoutCandidate(); |
| continue |
v |
LOG.debug("No leases to update found.") // don't see in log because debug lvl |
| |
\-----------------------------------------------------------------------------/
{code}
As we can see, in both cases we have a situation where we can't re-grant lease
and then neither send {{ChangePeersAndLearnersReplicaRequest}} nor trigger
{{onLeaderElected}} callback that was hooked after {{PRIMARY_REPLICA_ELECTED}}.
And if in the first scenario there is possible workaround through sending the
replica request with ignoring of primary replica nod colocation, but in the
second scenario all possible pending handlers are already finished and wouldn't
be called further.
This situation leads us to an idea that we may take a closer look on lease
granting process. For now we're checking only stable assignments for a new
lease holder candidate. So, while we didn't finished assignments stable switch
(call of {{onNewPeersConfigurationApplied}} after full RAFT rebalabce is done),
we will nave {{null}} candidate and stucks in live lock.
A possible solution is to check pendings assignments too in case if on stable
assignments we haven't got any possibe lease candidate. But there we have an
assumption that granting lease for a candidate from pending assignments won't
be a problem, because we may accept lease only if candidate replica can
retrieve replication group leader (see
{{ReplicaImpl#processLeaseGrantedMessage}}, mostly the first action is to get
the leader), then we're almost able to process requests: at least we will
propose leader-colocated replica or (see the 2nd case) will waiting if leader
will be elected.
There should be a note about about the 2nd case: if B node dies before catch up
is done and before RAFT group de-facto changed a configuration and then A
didn't and unable to elect itself as a leader, we have a situation as now when
partition is broken because the only replica is unavailable.
*Motivation*
The main motivation there is to make IGNITE-22036 and a whole epic IGNITE-22313
be able to handle rebalance even if we have no any active node in stable
assignments because assignments stable switch unable to be called.
*Definition of done*
# {{AssignmentsTracker}} is able to track both stable and pending assignments.
# While {{LeaseUpdater#Updater#updateLeaseBatchInternal}} if on lease
expiration {{LeaseUpdater#nextLeaseHolder}} returns {{null}}, then it now
should gets pending assignments from {{AssignmentsTracker}} and choose a lease
candidate among them.
# Before IGNITE-22036 is done this changes shouldn't break any test.
was:
*Description*
While working under IGNITE-22036 there was found 2 exceptional scenarios:
_Recovery case_
{code:java}
nodes = [A, B]
parts = 1
replicas = 1
| starting nodes
v
13_part_0 stable = [B]
B hosts replicaGrpId=13_part_0
B becomes PR
|
v
A starts
| <nothing happens>
v
B is shutdowned
| <do reset on pending = [A]>
v
A::TableManager::handlePending fails (A not a primary)
A starts voting and became a leader term=2
| ReplicaStateManager fails (A not a primary)
v
LeaseUpdater#updateBatchLeaseInternal <---------------------------------------\
| iterate over assignments == [B] |
v |
topologyTracker#nodeByConsistentId |
| returns null |
v |
leaseUpdateStatistics#onLeaseWithoutCandidate(); |
| continue |
v |
LOG.debug("No leases to update found.") // don't see in log because debug lvl |
| |
\-----------------------------------------------------------------------------/
{code}
_Stable dies, change peers wasn't done_
{code:java}
nodes = [A, B]
parts = 1
replicas = 1
| starting nodes
v
13_part_0 stable = [B]
B hosts replicaGrpId=13_part_0
B becomes PR
|
v
A starts
| <filter={A}>
v
A::TM#handleChangePendingsAssignmentsEvent
A::ReplicaManager#startReplica
A::RaftGroupService#updatePeersAndLearners
| <ready for catch up>
v
B::TM#handleChangePendingsAssignmentsEvent
B::TM#sendChangePeersAndLearnersReplicaRequest
B::ReplicaPartitionListener#processChangePeersAndLearnersReplicaRequest
B::RaftGroupService#changePeersAndLearnersAsync
| <B dies>
v
<No primary => no one would process failover>
<All TM#handleChangePendingsAssignmentsEvent are finished previously =>
=> no one would send ChangePeersAndLearnersReplicaRequest>
|
v
LeaseUpdater#updateBatchLeaseInternal <---------------------------------------\
| iterate over assignments == [B] |
v |
topologyTracker#nodeByConsistentId |
| returns null |
v |
leaseUpdateStatistics#onLeaseWithoutCandidate(); |
| continue |
v |
LOG.debug("No leases to update found.") // don't see in log because debug lvl |
| |
\-----------------------------------------------------------------------------/
{code}
As we can see, in both cases we have a situation where we can't re-grant lease
and then neither send {{ChangePeersAndLearnersReplicaRequest}} nor trigger
{{onLeaderElected}} callback that was hooked after {{PRIMARY_REPLICA_ELECTED}}.
And if in the first scenario there is possible workaround through sending the
replica request with ignoring of primary replica nod colocation, but in the
second scenario all possible pending handlers are already finished and wouldn't
be called further.
This situation leads us to an idea that we may take a closer look on lease
granting process. For now we're checking only stable assignments for a new
lease holder candidate. So, while we didn't finished assignments stable switch
(call of {{onNewPeersConfigurationApplied}} after full RAFT rebalabce is done),
we will nave {{null}} candidate and stucks in live lock.
A possible solution is to check pendings assignments too in case if on stable
assignments we haven't got any possibe lease candidate. But there we have an
assumption that granting lease for a candidate from pending assignments won't
be a problem, because we may accept lease only if candidate replica can
retrieve replication group leader (see
{{ReplicaImpl#processLeaseGrantedMessage}}, mostly the first action is to get
the leader), then we're almost able to process requests: at least we will
propose leader-colocated replica or (see the 2nd case) will waiting if leader
will be elected.
There should be a note about about the 2nd case: if B node dies before catch up
is done and before RAFT group de-facto changed a configuration and then A
didn't and unable to elect itself as a leader, we have a situation as now when
partition is broken because the only replica is unavailable.
*Motivation*
The main motivation there is to make IGNITE-22036 and a whole epic IGNITE-22313
be able to handle rebalance even if we have no any active node in stable
assignments because assignments stable switch unable to be called.
*Definition of done*
# {{AssignmentsTracker}} is able to track both stable and pending assignments.
# While {{LeaseUpdater#Updater#updateLeaseBatchInternal}} if on lease
expiration {{LeaseUpdater#nextLeaseHolder}} returns {{null}}, then it now
should gets pending assignments from {{AssignmentsTracker}} and choose a lease
candidate among them.
# Before IGNITE-22036 is done this changes shouldn't breaks any tests.
> Make PlacementDriver able to choose lease candidate from pendings
> -----------------------------------------------------------------
>
> Key: IGNITE-23587
> URL: https://issues.apache.org/jira/browse/IGNITE-23587
> Project: Ignite
> Issue Type: Improvement
> Reporter: Mikhail Efremov
> Assignee: Mikhail Efremov
> Priority: Blocker
> Labels: ignite-3, lease
>
> *Description*
> While working under IGNITE-22036 there was found 2 exceptional scenarios:
> _Recovery case_
> {code:java}
> nodes = [A, B]
> parts = 1
> replicas = 1
> | starting nodes
> v
> 13_part_0 stable = [B]
> B hosts replicaGrpId=13_part_0
> B becomes PR
> |
> v
> A starts
> | <nothing happens>
> v
> B is shutdowned
> | <do reset on pending = [A]>
> v
> A::TableManager::handlePending fails (A not a primary)
> A starts voting and became a leader term=2
> | ReplicaStateManager fails (A not a primary)
> v
> LeaseUpdater#updateBatchLeaseInternal
> <---------------------------------------\
> | iterate over assignments == [B]
> |
> v
> |
> topologyTracker#nodeByConsistentId
> |
> | returns null
> |
> v
> |
> leaseUpdateStatistics#onLeaseWithoutCandidate();
> |
> | continue
> |
> v
> |
> LOG.debug("No leases to update found.") // don't see in log because debug lvl
> |
> |
> |
> \-----------------------------------------------------------------------------/
> {code}
> _Stable dies, change peers wasn't done_
> {code:java}
> nodes = [A, B]
> parts = 1
> replicas = 1
> | starting nodes
> v
> 13_part_0 stable = [B]
> B hosts replicaGrpId=13_part_0
> B becomes PR
> |
> v
> A starts
> | <filter={A}>
> v
> A::TM#handleChangePendingsAssignmentsEvent
> A::ReplicaManager#startReplica
> A::RaftGroupService#updatePeersAndLearners
> | <ready for catch up>
> v
> B::TM#handleChangePendingsAssignmentsEvent
> B::TM#sendChangePeersAndLearnersReplicaRequest
> B::ReplicaPartitionListener#processChangePeersAndLearnersReplicaRequest
> B::RaftGroupService#changePeersAndLearnersAsync
> | <B dies>
> v
> <No primary => no one would process failover>
> <All TM#handleChangePendingsAssignmentsEvent are finished previously =>
> => no one would send ChangePeersAndLearnersReplicaRequest>
> |
> v
> LeaseUpdater#updateBatchLeaseInternal
> <---------------------------------------\
> | iterate over assignments == [B]
> |
> v
> |
> topologyTracker#nodeByConsistentId
> |
> | returns null
> |
> v
> |
> leaseUpdateStatistics#onLeaseWithoutCandidate();
> |
> | continue
> |
> v
> |
> LOG.debug("No leases to update found.") // don't see in log because debug lvl
> |
> |
> |
> \-----------------------------------------------------------------------------/
> {code}
> As we can see, in both cases we have a situation where we can't re-grant
> lease and then neither send {{ChangePeersAndLearnersReplicaRequest}} nor
> trigger {{onLeaderElected}} callback that was hooked after
> {{PRIMARY_REPLICA_ELECTED}}. And if in the first scenario there is possible
> workaround through sending the replica request with ignoring of primary
> replica nod colocation, but in the second scenario all possible pending
> handlers are already finished and wouldn't be called further.
> This situation leads us to an idea that we may take a closer look on lease
> granting process. For now we're checking only stable assignments for a new
> lease holder candidate. So, while we didn't finished assignments stable
> switch (call of {{onNewPeersConfigurationApplied}} after full RAFT rebalabce
> is done), we will nave {{null}} candidate and stucks in live lock.
> A possible solution is to check pendings assignments too in case if on stable
> assignments we haven't got any possibe lease candidate. But there we have an
> assumption that granting lease for a candidate from pending assignments won't
> be a problem, because we may accept lease only if candidate replica can
> retrieve replication group leader (see
> {{ReplicaImpl#processLeaseGrantedMessage}}, mostly the first action is to get
> the leader), then we're almost able to process requests: at least we will
> propose leader-colocated replica or (see the 2nd case) will waiting if leader
> will be elected.
> There should be a note about about the 2nd case: if B node dies before catch
> up is done and before RAFT group de-facto changed a configuration and then A
> didn't and unable to elect itself as a leader, we have a situation as now
> when partition is broken because the only replica is unavailable.
> *Motivation*
> The main motivation there is to make IGNITE-22036 and a whole epic
> IGNITE-22313 be able to handle rebalance even if we have no any active node
> in stable assignments because assignments stable switch unable to be called.
> *Definition of done*
> # {{AssignmentsTracker}} is able to track both stable and pending assignments.
> # While {{LeaseUpdater#Updater#updateLeaseBatchInternal}} if on lease
> expiration {{LeaseUpdater#nextLeaseHolder}} returns {{null}}, then it now
> should gets pending assignments from {{AssignmentsTracker}} and choose a
> lease candidate among them.
> # Before IGNITE-22036 is done this changes shouldn't break any test.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
