lucasbru opened a new pull request, #23608:
URL: https://github.com/apache/kafka/pull/23608
This ports the scale-out warm-up scenario from
HighAvailabilityTaskAssignorIntegrationTest, which exercises the classic
protocol's client-side HighAvailabilityTaskAssignor, to the streams group
protocol. Under the streams group protocol, warm-up derivation happens
broker-side via an AssignmentRefiner, which defaults to a no-op implementation,
so the test enables the real AssignmentRefinerImpl explicitly to get coverage
of the actual warm-up behavior against a live cluster with real KafkaStreams
clients, which did not exist before.
The test proves the same property as the original: when a stateful app
scales out, the new instance is first handed a warm-up task rather than being
promoted to active immediately, and is only promoted once its local state has
caught up, at which point the promotion happens with zero real state
restoration because the warm-up's state is reused directly.
Two mechanisms from the original test don't carry over and needed
replacements. There is no client-side assignment-listener equivalent under the
streams group protocol, since that callback is wired into the classic
protocol's client-side assignor, so warm-up assignment and later promotion are
instead detected by polling KafkaStreams.metadataForLocalThreads() for a task
moving from the standby set into the active set. On the target-assignment side,
the streams group protocol currently only ships the sticky assignor, which
balances active task load across processes rather than doing plain round robin,
but its load-balancing quota math still forces one of the two tasks in this
scenario onto the new, empty instance's target assignment, which is exactly the
mismatch the refiner needs in order to stage a warm-up. So the test works with
the sticky assignor as is, and does not need to wait for a more
round-robin-like assignor.
The rack-aware parameterization from the original test was dropped, since
the streams group protocol's sticky assignor has no rack-aware placement logic
today, so that axis would not exercise anything meaningful. The in-memory
versus persistent store parameterization was kept, since it is a pure
client-side choice orthogonal to the group protocol.
### Test plan
- Ran both parameterized variants locally against a live 3-broker embedded
KRaft cluster; both pass.
- Confirmed via logs that the new instance is assigned a warm-up task before
promotion (`warm-up tasks {0_1}`, no active task), and that the state-restore
listener reports zero restored records at promotion time.
- `./gradlew spotlessApply checkstyleMain checkstyleTest` is clean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
