[
https://issues.apache.org/jira/browse/KAFKA-21179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-21179:
------------------------------------
Component/s: group-coordinator
streams
> Support rack-aware task assignment in the Streams group protocol in terms of
> partition rack
> -------------------------------------------------------------------------------------------
>
> Key: KAFKA-21179
> URL: https://issues.apache.org/jira/browse/KAFKA-21179
> Project: Kafka
> Issue Type: Improvement
> Components: group-coordinator, streams
> Reporter: sanghyeok An
> Assignee: sanghyeok An
> Priority: Minor
>
> The broker-side Streams assignor API exposes client rack information, but
> does not expose the topic partitions associated with each task or the racks
> hosting their replicas. Although this metadata is available on the broker,
> custom assignors cannot use it through the current API to make locality-aware
> assignment decisions.
> When partition replicas are available only in a subset of racks, assigning a
> task to a client outside those racks can introduce avoidable cross-rack
> traffic. Depending on the deployment topology, this may increase read latency
> and network costs.
> Classic Streams already supports this optimization through KIP-925, balancing
> partition locality against task distribution and assignment stability.
> KIP-1071 also explicitly identifies KIP-925-style optimization as future work
> for the Streams rebalance protocol.
> We should close this gap by making the necessary partition and replica rack
> metadata available to broker-side Streams assignors and supporting
> locality-aware assignment. The initial scope can focus on active tasks and
> their input partitions, while preserving balanced task distribution and
> avoiding unnecessary task movement.
> Validation should cover deployments where replicas do not span all client
> racks, missing rack information, and the trade-off between locality
> improvements and task movement. Actual traffic reduction should be evaluated
> with rack-aware fetching configured appropriately.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
