[
https://issues.apache.org/jira/browse/KAFKA-21179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18120830#comment-18120830
]
Ziyun Fu commented on KAFKA-21179:
----------------------------------
Hi, this ticket looks related to two tickets I'm working on: KAFKA-20999
(KIP-708 tag-based standby placement for the streams group protocol, PR coming
soon) and its follow-up KAFKA-16867 (using client.rack as an implicit tag). I
think the goals are different, yours is active-task locality and mine is
standby spreading, but we'd both be changing StickyTaskAssignor and the
rack-related metadata. Could you take a look at those two and we can sync on
scope so we don't step on each other?
> Support rack-aware task assignment in the Streams group protocol in terms of
> partition rack
> -------------------------------------------------------------------------------------------
>
> Key: KAFKA-21179
> URL: https://issues.apache.org/jira/browse/KAFKA-21179
> Project: Kafka
> Issue Type: Improvement
> Reporter: sanghyeok An
> Assignee: sanghyeok An
> Priority: Minor
> Labels: streams
>
> The broker-side Streams assignor API exposes client rack information, but
> does not expose the topic partitions associated with each task or the racks
> hosting their replicas. Although this metadata is available on the broker,
> custom assignors cannot use it through the current API to make locality-aware
> assignment decisions.
> When partition replicas are available only in a subset of racks, assigning a
> task to a client outside those racks can introduce avoidable cross-rack
> traffic. Depending on the deployment topology, this may increase read latency
> and network costs.
> Classic Streams already supports this optimization through KIP-925, balancing
> partition locality against task distribution and assignment stability.
> KIP-1071 also explicitly identifies KIP-925-style optimization as future work
> for the Streams rebalance protocol.
> We should close this gap by making the necessary partition and replica rack
> metadata available to broker-side Streams assignors and supporting
> locality-aware assignment. The initial scope can focus on active tasks and
> their input partitions, while preserving balanced task distribution and
> avoiding unnecessary task movement.
> Validation should cover deployments where replicas do not span all client
> racks, missing rack information, and the trade-off between locality
> improvements and task movement. Actual traffic reduction should be evaluated
> with rack-aware fetching configured appropriately.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
