[
https://issues.apache.org/jira/browse/KUDU-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18135495#comment-18135495
]
ASF subversion and git services commented on KUDU-3733:
-------------------------------------------------------
Commit 953a2c5c83cb88d875823730c04e1bba2dc9a65b in kudu's branch
refs/heads/master from Gabriella Lotz
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=953a2c5c8 ]
KUDU-3733 run auto-leader-rebalancer at startup
RunLoop() slept for --auto_leader_rebalancing_interval_seconds before
its first pass, leaving the auto-leader-rebalancer idle for up to an
hour after master startup or failover.
Running a pass immediately isn't enough on its own: the thread is
created while the catalog manager is still initializing, so that pass
no-ops on the leadership check and the loop sleeps out the interval
anyway. The interval says how often to rebalance, not how long to
sleep, so separate the two. The loop now polls every 5 seconds and
decides whether a pass is due from when the last one ran; only a pass
that got past the leadership check advances that timestamp.
While here, clamp the interval to a second. The flag has no validator
and is runtime settable, so 0 is reachable, and at 0 the loop never
sleeps and spins on RunLeaderRebalancer() (~14k passes in 3 seconds,
measured). That spin predates this patch and isn't made worse by it,
but the clamp is cheap, and reading the interval once for both the
poll and the due check beats reading the flag in two places.
Adds LeaderRebalancerTest.FirstPassDoesNotWaitOutFullInterval, which
fails against the old loop, and ZeroIntervalDoesNotSpin, which fails
without the clamp.
Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
Co-authored-by: Cursor <[email protected]>
Reviewed-on: http://gerrit.cloudera.org:8080/24946
Reviewed-by: Marton Greber <[email protected]>
Tested-by: Marton Greber <[email protected]>
Reviewed-by: Alexey Serbin <[email protected]>
> Restructure RunLoop() in auto_leader_rebalancer
> -----------------------------------------------
>
> Key: KUDU-3733
> URL: https://issues.apache.org/jira/browse/KUDU-3733
> Project: Kudu
> Issue Type: Sub-task
> Reporter: Gabriella Lotz
> Assignee: Gabriella Lotz
> Priority: Major
>
> h3. Context:
> RunLoop() currently sleeps for the full
> --auto_leader_rebalancing_interval_seconds (default: 1 hour) before executing
> the first rebalance pass. This means that on master restart or failover, the
> rebalancer is effectively inactive for up to an hour, which is unacceptable
> in production. The loop should run once immediately on startup, then sleep
> between subsequent iterations.
> h3. Current behavior:
> sleep → run → sleep → run → ...
> h3. Expected behavior:
> run → sleep → run → sleep → ...
> The fix is a small restructure of RunLoop() in auto_leader_rebalancer.cc from
> a while(!WaitFor(...)) pattern to a do \{ run } while(!WaitFor(...)) pattern.
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)