sanghyeok An created KAFKA-21179:
------------------------------------
Summary: Support rack-aware task assignment in the Streams group
protocol
Key: KAFKA-21179
URL: https://issues.apache.org/jira/browse/KAFKA-21179
Project: Kafka
Issue Type: Improvement
Reporter: sanghyeok An
Assignee: sanghyeok An
The broker-side Streams assignor API exposes client rack information, but does
not expose the topic partitions associated with each task or the racks hosting
their replicas. Although this metadata is available on the broker, custom
assignors cannot use it through the current API to make locality-aware
assignment decisions.
When partition replicas are available only in a subset of racks, assigning a
task to a client outside those racks can introduce avoidable cross-rack
traffic. Depending on the deployment topology, this may increase read latency
and network costs.
Classic Streams already supports this optimization through KIP-925, balancing
partition locality against task distribution and assignment stability. KIP-1071
also explicitly identifies KIP-925-style optimization as future work for the
Streams rebalance protocol.
We should close this gap by making the necessary partition and replica rack
metadata available to broker-side Streams assignors and supporting
locality-aware assignment. The initial scope can focus on active tasks and
their input partitions, while preserving balanced task distribution and
avoiding unnecessary task movement.
Validation should cover deployments where replicas do not span all client
racks, missing rack information, and the trade-off between locality
improvements and task movement. Actual traffic reduction should be evaluated
with rack-aware fetching configured appropriately.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
