[
https://issues.apache.org/jira/browse/KAFKA-21179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18134701#comment-18134701
]
sanghyeok An commented on KAFKA-21179:
--------------------------------------
[~gabriellafu]
Thanks, Ziyun. I agree that the goals are different, although there may be
overlap in the assignor code and rack-related metadata. Based on Matthias’s
feedback, we’ll first need to establish user demand before deciding whether to
proceed with this ticket, so there’s no need to hold up your PRs on its
account. Please go ahead with your planned changes, and if this work moves
forward, I’ll coordinate with you and build on the relevant changes from your
work!
> Support rack-aware task assignment in the Streams group protocol in terms of
> partition rack
> -------------------------------------------------------------------------------------------
>
> Key: KAFKA-21179
> URL: https://issues.apache.org/jira/browse/KAFKA-21179
> Project: Kafka
> Issue Type: Improvement
> Components: group-coordinator, streams
> Reporter: sanghyeok An
> Assignee: sanghyeok An
> Priority: Minor
> Labels: needs-kip
>
> The broker-side Streams assignor API exposes client rack information, but
> does not expose the topic partitions associated with each task or the racks
> hosting their replicas. Although this metadata is available on the broker,
> custom assignors cannot use it through the current API to make locality-aware
> assignment decisions.
> When partition replicas are available only in a subset of racks, assigning a
> task to a client outside those racks can introduce avoidable cross-rack
> traffic. Depending on the deployment topology, this may increase read latency
> and network costs.
> Classic Streams already supports this optimization through KIP-925, balancing
> partition locality against task distribution and assignment stability.
> KIP-1071 also explicitly identifies KIP-925-style optimization as future work
> for the Streams rebalance protocol.
> We should close this gap by making the necessary partition and replica rack
> metadata available to broker-side Streams assignors and supporting
> locality-aware assignment. The initial scope can focus on active tasks and
> their input partitions, while preserving balanced task distribution and
> avoiding unnecessary task movement.
> Validation should cover deployments where replicas do not span all client
> racks, missing rack information, and the trade-off between locality
> improvements and task movement. Actual traffic reduction should be evaluated
> with rack-aware fetching configured appropriately.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
