[
https://issues.apache.org/jira/browse/KUDU-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18086984#comment-18086984
]
Gabriella Lotz commented on KUDU-3732:
--------------------------------------
I have investigated the TSAN flakiness. I ran auto_rebalancer-test under TSAN
on dist-test (1000x) to validate this change. Baseline failure rate was ~32%
(323/1000) - [http://dist-test.cloudera.org/job?job_id=root.1780911928.28709] -
dominated by ThreadSanitizer data races (exit code 66). Two distinct races were
found:
# (Pre-existing) TSDescriptor::AssignLocationForTesting() wrote location_
without holding lock_, while the auto-rebalancer thread reads it via
TSDescriptor::location() (which takes a shared_lock) from
AutoRebalancerTask::BuildClusterRawInfo(). This race exists independently of
this change.
# (Introduced by this patch) The new test
ExecuteMovesCASRejectionDropsMoveGracefully drove ExecuteMoves() and read
moves_per_tserver_ directly from the test thread while the live auto-rebalancer
thread was running. moves_per_tserver_ is unsynchronized by design (only ever
touched by the single rebalancer thread in production), so the cross-thread
access from the test raced RunLoop().
Both are fixed (see Gerrit). After the fixes a 1000x TSAN run shows 0 data
races and a ~1.1% failure rate (11/1000) -
[http://dist-test.cloudera.org/job?job_id=root.1780921959.68408] - all from two
unrelated pre-existing flakes that are not data races and not caused by this
change:
* NextLeaderResumesAutoRebalancing: assertion is too strict under early
leadership churn (10/1000).
* NoRebalancingIfReplicasRecovering: creates an RF=3 workload table during a
window where only 2 tservers are live, which TestWorkload::Setup() treats as
fatal (1/1000).
I am planning to fix these flaky tests after this patch has been merged.
> Get consensus state from CatalogManager
> ---------------------------------------
>
> Key: KUDU-3732
> URL: https://issues.apache.org/jira/browse/KUDU-3732
> Project: Kudu
> Issue Type: Sub-task
> Reporter: Gabriella Lotz
> Assignee: Gabriella Lotz
> Priority: Major
>
> h2. Address the two existing TODOs
> Both are in
> {{{}[src/kudu/master/auto_rebalancer.cc](src/kudu/master/auto_rebalancer.cc){}}}:
> {*}TODO 1 (line 478){*}: {{remove moves that fail to be scheduled from
> 'replica_moves'}}
> Currently {{ExecuteMoves()}} iterates all moves but does not remove
> failed-to-schedule moves. Failed moves stay in the in-progress set, blocking
> future rounds from reconsidering those tablets. Fix: collect scheduling
> failures and erase them from {{replica_moves}} before returning, so
> {{CheckReplicaMovesCompleted}} only waits on actually-submitted moves.
> {*}TODO 2 (line 853){*}: {{Retrieve consensus state from CatalogManager
> instead}}
> {{CheckMoveCompleted()}} currently mirrors the tool-side
> {{CheckCompleteMove()}} — it issues RPCs. The better path reads consensus
> state from the master's in-memory catalog (already available via
> {{{}CatalogManager{}}}), eliminating the extra RPC round-trip per move. The
> implementation pattern exists in {{{}CatalogManager::GetTabletLocations{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
