[
https://issues.apache.org/jira/browse/IGNITE-22904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Puchkovskiy updated IGNITE-22904:
---------------------------------------
Summary: Do not allow old MG majority to hijack leadership (was: Disallow
old MG majority to hijack leadership)
> Do not allow old MG majority to hijack leadership
> -------------------------------------------------
>
> Key: IGNITE-22904
> URL: https://issues.apache.org/jira/browse/IGNITE-22904
> Project: Ignite
> Issue Type: Improvement
> Reporter: Roman Puchkovskiy
> Assignee: Roman Puchkovskiy
> Priority: Major
> Labels: iep-128, ignite-3
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> If some node did not see Metastorage repair, it will be migrated to the new
> cluster using the Migrate REST/CLI command. Such a node (judging by its local
> MG Raft log) might still think it's a member of the voting set, so it might
> propose itself as a candidate, and it can win the election if there are
> enough such nodes. This will result in the leadership being hijacked by the
> 'old' majority, which will mess the repaired Metastorage up. This has to be
> avoided.
> To do so, the following should be done:
> # In the CMG, add a property called metastorageRepairClusterId (empty in the
> blank cluster)
> # When, during MG repair (IGNITE-22899), we choose new metastorageNodes and
> save them to the CMG (which happens before resetPeers() is called), we write
> metastorageRepairClusterId together with metastorageNodes to the CMG
> # We add a property called witnessedMetastorageRepairClusterId to the Vault.
> This property will store clusterId for the incarnation of the cluster in
> which the node witnessed MG repair (either it participated in the repair, or
> it was migrated and successfully performed the 'MG reentry' procedure, see
> below. This property is empty on a blank node
> # When a node handles
> MetastorageIndexTermRequestMessage, it writes current clusterId to its
> Vault.witnessedMetastorageRepairClusterId. As a result, every node
> participating in the MG repair will be marked as a witness of the repair and
> we'll not need to do 'MG reentry' for them
> # On node start, before starting the MG, Ignite node gets from the CMG
> leader metastorageNodes and metastorageRepairClusterId. If it's not null and
> Vault.witnessedMetastorageRepairClusterId is absent or differs from
> metastorageRepairClusterId, then the node has to perform the 'MG reentry'
> procedure.
> # The 'MG reentry' procedure is as follows:
> ## The node destroys all 3 Raft storages for MG (these are meta, log,
> snapshot storage) as well as Metastorage KV storage
> ## Writes current clusterId to Vault.witnessedMetastorageRepairClusterId
> ## Then starts the MG Raft server as usual
> Another potential issue is that, when a node reenters and still has just a
> partial log, the part is has might tell it that it's a voting member. If at
> this moment the leader fails, the reentering node (which is not a member of
> the voting set in the latest configuration) might believe it IS such a member
> (as it only sees a part of the log) and hijack the leadership.
> To prevent this:
> # We'll add yet another property to the CMG, that is, metastorageRepairIndex.
> # When writing metastorageNodes to the CMG during repair, we also write
> metastorageRepairIndex
> # When doing the 'MG reentry' procedure, in Nodeimpl#init(), we store
> metastorageRepairIndex to a volatile field and, if the latest local config
> index is less than this index, we change the volatile config to the current
> metastorage nodes from the CMG (to prevent the reentered Raft node becoming a
> leader); when configuration is updated (coming from AppendEntries or
> installed snapshot), we check whether its index reached
> metastorageRepairIndex, and if no, we change the volatile configuration again.
> This also solves a potential ABA problem.
> h2. Old description (would not work)
> -If, during a join (on getting the fresh cluster state from the CMG), a node
> detects that, according to the MG configuration saved in the MG on this node,
> this node is the member of the voting set (i.e. it’s a peer, not a learner),
> and this node is NOT one of the metastorageNodes in the CMG, then, before
> starting its MG Raft member, it raises a flag that disallows its Raft node
> becoming a candidate.-
> -(This flag does not exist in JRaft, we need to introduce it there; the flag
> is not persisted).-
> -As soon as the Raft node applies a new Raft configuration (coming from the
> new leader), this flag is cleared.-
> -After this, the Raft node is ‘converted’ to the new MG and cannot hijack the
> leadership.-
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
