[
https://issues.apache.org/jira/browse/HDDS-17124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18128630#comment-18128630
]
Chia-Chuan Ho commented on HDDS-17124:
--------------------------------------
Hi [~smeng]
I tested the leader-hint changes and would appreciate your thoughts on an
additional observation.
I used a MiniOzoneHACluster with five OMs and five SCMs, measuring three
successive server-side transferLeadership calls for each service with the
default Ratis retry policy. Cluster startup and additional test sleeps were
excluded.
Before the first measurement, I ensured that the current leader was not the
first peer, using a direct RaftServer call to avoid warming the client leader
cache during preparation. Subsequent measurements could benefit from a warm
cache. I compared the same harness with and without the changes, capturing
request destinations and timestamps. The existing three-node shell integration
tests also passed.
With the hints, the initial priority-setting phase improved from approximately
3.5–3.8 seconds to 43–45 ms, and no NotLeaderException replies were observed.
In these runs, however, resetting priorities immediately after a successful
transfer received LeaderNotReadyException from the target. This incurred
another retry sleep, leaving the overall operation at approximately 2.9–7.6
seconds.
For the readiness-related delay, one possible approach would be to give the
dedicated priority-reset client a small, bounded number of short-interval
retries before falling back to the configured retry policy, while honoring the
existing retry limits and timeout.
This could avoid a full multi-second sleep during a brief leader-readiness
transition. The initial priority update and leadership-transfer requests would
continue to use their existing retry policy.
I would appreciate your thoughts on whether this would be a suitable approach
for the remaining delay.
> Leadership transfer is slow because RatisHelper#transferRatisLeadership
> creates its RaftClient without a leader hint
> --------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17124
> URL: https://issues.apache.org/jira/browse/HDDS-17124
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: OM HA, SCM HA
> Reporter: Siyao Meng
> Assignee: Chia-Chuan Ho
> Priority: Major
>
> {{RatisHelper#transferRatisLeadership}} (used by
> {{OzoneManager#transferLeadership}} and
> {{SCMClientProtocolServer#transferLeadership}}) builds a RaftClient with
> {{leaderId = null}}. Two requests then usually go to a non-leader first, and
> each miss costs a full Ratis client retry sleep:
> # {{setConfiguration}} (priority change) is sent to the first peer of the
> group. That peer is a follower with probability (N-1)/N, so 4/5 of the time
> in a 5-node ring.
> # {{resetPriorities}} runs after the transfer, when the client still has the
> *old* leader cached, so it always misses once.
> Measured in 5-node MiniOzoneCluster tests:
> * OM: "Start transferring leadership" 17.520 -> setConfiguration first sent
> to omNode-5 (follower) 17.570 -> reached the leader omNode-1 only at 24.088
> (6.5 s); resetPriorities took another ~6 s.
> * SCM: setConfiguration hit the leader first (fast), resetPriorities took 4.1
> s.
> So {{ozone admin om transfer}} / {{ozone admin scm transfer}} take ~4-13 s
> instead of well under a second. With a short client RPC timeout this also
> triggers the retry problem in HDDS-17125.
> *Proposal:* the caller always runs on the current leader, so pass the local
> peer id as the client's leader, and send {{resetPriorities}} to the new
> leader (the transfer target).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
