Gabriella Lotz has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24591
Change subject: KUDU-3789 Deflake auto_leader_rebalancer-test ...................................................................... KUDU-3789 Deflake auto_leader_rebalancer-test Under TSAN this test suite was very flaky (about 488 of 1000 runs failed at one baseline). The causes were all test-side timing assumptions on a cluster whose leadership keeps moving, not product bugs. Fixed retry loops (run N rounds, then assert balanced) raced the cluster's ongoing leadership moves. Replaced them with a RunUntilLeaderBalanced() helper that retries CheckLeaderBalance() over a TSAN-aware time budget via AssertEventually. AddTserver and RestartTserver used 59 tablets. That many raft groups move leadership around through natural elections and keep the cluster from ever reaching an exactly balanced snapshot. Reduced to 24 (divisible by the tserver count for a clean balance target). Note that stopping the replica rebalancer to remove that movement makes things worse: on a static cluster the leader rebalancer gets stuck moving the same leaders back and forth and never balances, so the replica rebalancer is left running. Placement-sensitive tests (MultiTableLeaderBalance, FilterSoftDeletedTableTest, TestMaintenanceMode) were disrupted by extra leader elections when TSAN CPU starvation exceeded the default leader failure timeout. Raised leader_failure_max_missed_heartbeat_periods from its default of 3.0 to 15.0 for those tests (none shut a tserver down, so slower failover does no harm), and reset it to the default in TearDown so the failover-based tests are unaffected under test reordering. Relaxed the ignored soft-deleted table check in FilterSoftDeletedTableTest to "not fully balanced" rather than requiring every leader count to differ from 3, which natural elections could trip. Rewrote TestMaintenanceMode to arrange the imbalance directly with MakeLeaderDistribution and assert the rebalancer never places a leader on the maintenance-mode tserver, instead of restarting the tserver and racing on election timing. Test-only. No production behavior change. Change-Id: I00224fddfead0fb462e793f085d935b597792aaa --- M src/kudu/master/auto_leader_rebalancer-test.cc 1 file changed, 140 insertions(+), 169 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/91/24591/1 -- To view, visit http://gerrit.cloudera.org:8080/24591 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I00224fddfead0fb462e793f085d935b597792aaa Gerrit-Change-Number: 24591 Gerrit-PatchSet: 1 Gerrit-Owner: Gabriella Lotz <[email protected]>
