YifanZhang created KUDU-3098:
--------------------------------
Summary: leader change during 'add_peer' process for a tablet may
lead to an isolate replica
Key: KUDU-3098
URL: https://issues.apache.org/jira/browse/KUDU-3098
Project: Kudu
Issue Type: Bug
Components: consensus, master
Affects Versions: 1.10.1
Reporter: YifanZhang
Lately we found some tablets in a cluster with a very large
"time_since_last_leader_heartbeat" metric, they are LEARNER/NON_VOTER and seems
couldn't become VOTER for a long time.
These replicas created during the rebalance/tablet_copy process. After
beginning a new copy session from leader to the new added NON_VOTER peer,
leadership changed, old leader aborted uncommited CHANGE_CONFIG_OP operation.
Finally the tablet_copy session ended but new leader knew nothing about the new
peer.
Master didn't delete this new added replica because it has a larger opid_index
than the latest reported committed config. See the comments in
CatalogManager::ProcessTabletReport
{code:java}
// 5. Tombstone a replica that is no longer part of the Raft config (and
// not already tombstoned or deleted outright).
//
// If the report includes a committed raft config, we only tombstone if
// the opid_index is strictly less than the latest reported committed
// config. This prevents us from spuriously deleting replicas that have
// just been added to the committed config and are in the process of copying.
{code}
Maybe we shouldn't use opid_index to determine if replicas are in the process
of copying.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
