[
https://issues.apache.org/jira/browse/KAFKA-21179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18133370#comment-18133370
]
Matthias J. Sax commented on KAFKA-21179:
-----------------------------------------
Based on the learning from KIP-1071 implementation, I am not sure if we should
really support KIP-925 with "streams" protocol. The implemented algorithm are
rather compute expensive and putting this type of load on the group-coordinator
might not be a good thing.
I am actually not even sure, how many people use/rely on KIP-925... Before we
pick up this work, I would like to get broader user feedback that there is high
demand for it. Otherwise, it might not be worth doing it.
> Support rack-aware task assignment in the Streams group protocol in terms of
> partition rack
> -------------------------------------------------------------------------------------------
>
> Key: KAFKA-21179
> URL: https://issues.apache.org/jira/browse/KAFKA-21179
> Project: Kafka
> Issue Type: Improvement
> Components: group-coordinator, streams
> Reporter: sanghyeok An
> Assignee: sanghyeok An
> Priority: Minor
>
> The broker-side Streams assignor API exposes client rack information, but
> does not expose the topic partitions associated with each task or the racks
> hosting their replicas. Although this metadata is available on the broker,
> custom assignors cannot use it through the current API to make locality-aware
> assignment decisions.
> When partition replicas are available only in a subset of racks, assigning a
> task to a client outside those racks can introduce avoidable cross-rack
> traffic. Depending on the deployment topology, this may increase read latency
> and network costs.
> Classic Streams already supports this optimization through KIP-925, balancing
> partition locality against task distribution and assignment stability.
> KIP-1071 also explicitly identifies KIP-925-style optimization as future work
> for the Streams rebalance protocol.
> We should close this gap by making the necessary partition and replica rack
> metadata available to broker-side Streams assignors and supporting
> locality-aware assignment. The initial scope can focus on active tasks and
> their input partitions, while preserving balanced task distribution and
> avoiding unnecessary task movement.
> Validation should cover deployments where replicas do not span all client
> racks, missing rack information, and the trade-off between locality
> improvements and task movement. Actual traffic reduction should be evaluated
> with rack-aware fetching configured appropriately.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
