1230fahid opened a new pull request, #22934:
URL: https://github.com/apache/kafka/pull/22934
## Summary
The KRaft controller's preferred-leader balancer today elects a recovering
broker as partition leader the moment it unfences, before it has caught up to
the ISR. This causes a produce latency spike (typically 3-10× P99) for the
duration of the catch-up window because the new leader must fetch data from
followers before it can serve producers.
This PR introduces a **`wait-for-sync` gate algorithm** that withholds
preferred-leader elections for a recovering broker until it has rejoined the
ISR for enough of its partitions, eliminating the spike at the cost of a
slightly longer rebalance window.
---
## What Changed
**New configs** (all static, read at broker startup, same lifecycle as
`leader.imbalance.check.interval.seconds`):
| Config | Default | Description |
|---|---|---|
| `leader.imbalance.election.algorithm` | `immediate` | `immediate`
(existing behaviour) or `wait-for-sync` |
| `leader.imbalance.election.wait.for.sync.threshold.percent` | `0` | Gate
fires when OOS% exceeds this threshold |
| `leader.imbalance.election.wait.for.sync.max.wait.ms` | `1800000` | Escape
hatch: force-release gate after this many ms (0 = disabled) |
| `leader.imbalance.election.max.per.run` | _(unbounded)_ | Cap elections
per periodic run; enables round-robin across recovering brokers |
| `leader.imbalance.election.immediate.period.ms` | `10` | Reschedule delay
(ms) when a capped run signals continuation |
**Gate behaviour (`wait-for-sync`):**
- A broker is gated when `outOfSyncPartitions / totalPreferredPartitions >
threshold%`
- Gate releases only after **2 consecutive ungated runs** (hysteresis -
prevents flapping)
- If a broker stays gated longer than `max.wait.ms`, the **escape hatch**
fires: elections proceed
for any partitions where the broker IS in the ISR, and
`PreferredLeaderElectionEscapeHatchCount`increments
**Round-robin scheduling:** When `max.per.run` is set, elections are
distributed across all recovering brokers in a single run rather than
exhausting one broker's queue before moving to the next.
**New JMX metrics** (all under `kafka.controller:type=KafkaController`):
| Metric | Type | Description |
|---|---|---|
| `PreferredLeaderElectionsPerRun` | Gauge | Elections performed in the last
run |
| `GatedPreferredLeaderBrokerCount` | Gauge | Brokers currently blocked by
the gate (returns to 0 when resolved) |
| `PreferredLeaderElectionThrottledRunCount` | Gauge | Cumulative runs that
hit the `max.per.run` cap |
| `PreferredLeaderElectionEscapeHatchCount` | Gauge | Cumulative
escape-hatch releases |
| `OutOfSyncPreferredPartitionCount{broker=N}` | Gauge (dynamic per-broker)
| OOS preferred partition count per gated broker; deregisters when broker
recovers |
---
## Backward Compatibility
- Default algorithm is `immediate`, **no behaviour change** unless the new
configs are explicitly set.
- Mixed-version rolling upgrade: the active controller's config determines
behaviour; standby controllers are unaffected.
- Downgrade: older controllers ignore the new configs; the feature silently
turns off. No new metadata record types or metadata version changes.
---
## Test Plan
- [x] `ReplicationControlManagerTest` - 9 unit tests covering: gate on/off,
hysteresis (2-run release), escape hatch, threshold boundary (4.8% vs 5.3%),
immediate backward-compat, round-robin fairness, `brokerOutOfSyncCounts()` data
pipeline
- [x] `PeriodicTaskControlManagerTest` - 3 tests covering throttled
reschedule delay, no contamination of other tasks, immediate-period fallback
- [x] `QuorumControllerMetricsTest` - 2 tests covering all 4 new scalar
gauges and the dynamic per-broker OOS gauge lifecycle
- [x] `QuorumControllerPreferredLeaderIntegrationTest` - 4 full-stack
integration tests covering wait-for-sync gating, round-robin, escape hatch, and
OOS metric wiring
- [ ] **System / load test** _(reviewer action required)_: On a KRaft
cluster with ≥10,000 partitions, restart one broker and verify:
- Produce P99 latency over the 10-minute post-unfencing window stays
within **1.5× steady-state P99**
- No produce timeouts for clients with `request.timeout.ms ≥ 30000`
- Total recovery time (unfenced → leader of ≥99% preferred partitions)
is at most **2× today's baseline**
- [ ] **Rolling upgrade / downgrade** _(reviewer action required)_: Verify
cluster remains functional in a mixed-version controller quorum; verify
downgrade leaves cluster functional with feature silently disabled
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
