[
https://issues.apache.org/jira/browse/KUDU-3789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18092650#comment-18092650
]
Gabriella Lotz commented on KUDU-3789:
--------------------------------------
After investigating all of the tests, the latest dist-test run shows 999/1000
pass in TSAN -
[http://dist-test.cloudera.org/job?job_id=root.1782821335.244594.]
The 1 failing run is a pre-existing flakiness that has nothing to do with the
auto-rebalancer. It's a DCHECK(has_responded_) abort in LeaderElection's
destructor during cluster teardown (leader_election.cc:206).
Consensus/leader_election.cc isn't touched on this branch; the auto-rebalancer
test just happens to trigger it via election churn.
I'll upload the fixes in two changes to Gerrit:
1. Production fix (only change that touches production code). Commit
[2f16608ab|https://gerrit.cloudera.org/c/21073/] "Auto-rebalancer: clear
replace marker on failed moves" introduced clearing the source replica's
replace marker when a move fails, but it only tried once. When something goes
wrong at that moment and the leader is busy with concurrent config work, the
single BulkChangeConfig gets rejected, the marker stays set, and
TestRemoveReplaceFlagIfMoveFails fails. The fix makes the cleanup durable: an
inline retry, treating NotFound/InvalidArgument as already-cleared, and a
pending_replace_clears_ queue that's retried at the top of every rebalancer
loop (even when rebalancing is disabled).
2. Test-only changes (no production impact):
OnlyLeaderDoesAutoRebalancing / NextLeaderResumesAutoRebalancing: assert over a
stable-leadership window that only the leader's iteration counts grow, instead
of asserting non-leaders are == 0 (the initial election can flicker and briefly
leave a transient leader's count > 0).
NoRebalancingIfReplicasRecovering / TestDeletedTables: replaced the fixed
SleepFor(tserver_unresponsive_timeout_ms) after killing a tserver with a
deterministic wait until the master actually reports the tserver dead
(WaitForLiveTServerCount); set tserver_unresponsive_timeout_ms only after the
cluster and tables are up.
PreferFollowerRebalancingTest: lighter workload and a longer
CheckSomeMovesScheduled timeout.
ExecuteMovesCASRejectionDropsMoveGracefully: retry until the intended
CAS-rejection race is actually observed (a heartbeat can refresh the catalog's
opid before ExecuteMoves runs).
TestMaxMovesPerServer: longer CheckSomeMovesScheduled timeout.
RebalancerMetricsFollowerMoves: relaxed ASSERT_EQ(0, leader) to
ASSERT_GE(follower, leader).
> Investigate and fix flaky tests
> -------------------------------
>
> Key: KUDU-3789
> URL: https://issues.apache.org/jira/browse/KUDU-3789
> Project: Kudu
> Issue Type: Sub-task
> Reporter: Gabriella Lotz
> Assignee: Gabriella Lotz
> Priority: Major
>
> # In auto_rebalancer-test.cc
> # In auto_leader_rebalancer-test.cc
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
