Marton Greber has posted comments on this change. ( http://gerrit.cloudera.org:8080/24630 )
Change subject: [tools/master] make move of a leader replica recoverable ...................................................................... Patch Set 2: (5 comments) Thanks for working on this! Overall looks good, just some observations. Thanks! http://gerrit.cloudera.org:8080/#/c/24630/2/src/kudu/integration-tests/raft_consensus_nonvoter-itest.cc File src/kudu/integration-tests/raft_consensus_nonvoter-itest.cc: http://gerrit.cloudera.org:8080/#/c/24630/2/src/kudu/integration-tests/raft_consensus_nonvoter-itest.cc@1340 PS2, Line 1340: EXCLUDE_HEALTH_REPORT, &cs)); nit: the disabled-case relies on a fixed SleepFor(5s) to assert the master never evicts. That is inherently the right shape for a negative test, but it fixes a 5s floor on this parameter instance. Fine to leave; just flagging the cost. http://gerrit.cloudera.org:8080/#/c/24630/2/src/kudu/master/catalog_manager.cc File src/kudu/master/catalog_manager.cc: http://gerrit.cloudera.org:8080/#/c/24630/2/src/kudu/master/catalog_manager.cc@5517 PS2, Line 5517: bool AsyncLeaderStepDownForReplacementTask::SendRequest(int attempt) { q: If the 'replace'-marked leader steps down but is not yet evicted, nothing prevents it from winning a subsequent election (the 'replace' attr does not exclude it from candidacy). If it re-becomes leader before eviction commits, this task path re-triggers. Presumably eventual eviction wins the race and this is the same self-healing loop the client path also relies on -- just confirming that's the intended/acceptable behavior rather than a potential ping-pong. http://gerrit.cloudera.org:8080/#/c/24630/2/src/kudu/master/catalog_manager.cc@5900 PS2, Line 5900: rpcs.emplace_back(new AsyncLeaderStepDownForReplacementTask( q: Unlike the AsyncEvictReplicaTask / AsyncAddReplicaTask branches, a leader step-down does not create a pending config, so this branch is not suppressed by the enclosing `!cstate.has_pending_config()` guard. That means while the graceful leadership transfer is in flight the leader still reports itself as leader marked 'replace' on every heartbeat, and each report schedules a fresh AsyncLeaderStepDownForReplacementTask. It looks self-limiting (redundant tasks get ServiceUnavailable "transfer already in progress" and retry, then MarkComplete once SendRequest/HandleResponse observe the leader changed), but at low --heartbeat_interval_ms this can spawn a burst of tasks and repeated GRACEFUL step-down RPCs. Is it worth gating on an already-in-flight step-down for this tablet (or otherwise deduping) to avoid the churn? http://gerrit.cloudera.org:8080/#/c/24630/2/src/kudu/tools/tool_replica_util.cc File src/kudu/tools/tool_replica_util.cc: http://gerrit.cloudera.org:8080/#/c/24630/2/src/kudu/tools/tool_replica_util.cc@638 PS2, Line 638: nit: IsReplicaMoveScheduled() runs GetTabletLeader()/GetConsensusState() with no retry, and it is now the first thing MoveReplica() does. If the tablet is transiently leaderless -- e.g. exactly during the recovery step-down this feature detects -- the command fails outright instead of retrying (the old WaitForCleanKsck path retried for 5s). Minor, since move_replica is a manual command, but worth a thought. http://gerrit.cloudera.org:8080/#/c/24630/2/src/kudu/tools/tool_replica_util.cc@668 PS2, Line 668: (!from_ts_uuid_in_config || from_ts_uuid_has_replace); q: In the 3-4-3 branch, `scheduled` becomes true when `to` is present and `from` is absent (!from_ts_uuid_in_config). If a caller passes a from_ts_uuid that is not actually a current replica while `to` coincidentally is, this reports "already scheduled" and MoveReplica() skips scheduling and drops straight into WaitForMoveToComplete -- whereas the previous flow would have surfaced an error via ScheduleReplicaMove(). Is validating that `from` was a genuine replica worth it here, or is that input out of scope? -- To view, visit http://gerrit.cloudera.org:8080/24630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic7b28ca4e41355bd82c76d50e137f06197b402bc Gerrit-Change-Number: 24630 Gerrit-PatchSet: 2 Gerrit-Owner: Yifan Zhang <[email protected]> Gerrit-Reviewer: Gabriella Lotz <[email protected]> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Marton Greber <[email protected]> Gerrit-Reviewer: Yifan Zhang <[email protected]> Gerrit-Comment-Date: Thu, 30 Jul 2026 13:20:13 +0000 Gerrit-HasComments: Yes
