[
https://issues.apache.org/jira/browse/KAFKA-21180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18120779#comment-18120779
]
sanghyeok An commented on KAFKA-21180:
--------------------------------------
When using Kafka Streams, I have occasionally encountered situations where a
particular task required significantly more resources than others, which could
make the node hosting that task unstable.
This made me wonder whether it might be useful to allow users to assign a
weight to each task and have the task assignor take those weights into account
so that the workload can be distributed more evenly across instances. If this
idea sounds reasonable, I would be interested in writing a KIP and discussing
it further with the community.
> Support user-defined processing weights for Kafka Streams task assignment
> -------------------------------------------------------------------------
>
> Key: KAFKA-21180
> URL: https://issues.apache.org/jira/browse/KAFKA-21180
> Project: Kafka
> Issue Type: Improvement
> Reporter: sanghyeok An
> Assignee: sanghyeok An
> Priority: Minor
> Labels: streams
>
> *Description*
> Processing costs can vary across stages within a Kafka Streams topology.
> Simple filtering or transformation may require relatively little CPU, while
> complex aggregations or windowed operations may require more processing
> resources. As a result, assigning an equal number of tasks to each instance
> does not necessarily produce a balanced workload.
> Automatically measuring these differences at runtime and incorporating them
> into assignment decisions on the client or broker is challenging. Observed
> load depends on input traffic and the execution environment, and temporary
> traffic spikes need to be distinguished from sustained differences in
> processing cost. Reassigning tasks in response to short-lived fluctuations
> may introduce unnecessary task movement and state restoration costs.
> Application developers and operators, however, may understand the relative
> processing costs through knowledge of the application logic and experience
> from long-term monitoring. It would be useful to express this knowledge as
> user-defined weights and incorporate them into task assignment.
>
> *Proposal*
> Allow users to specify relative processing weights through the builder APIs
> used to construct a topology, or a similar mechanism, and make these weights
> available to the task assignor.
> For example, users could assign a lower weight to a simple filtering stage
> and a higher weight to a complex aggregation or windowed operation. The
> weights would represent relative estimated processing costs, rather than
> execution priorities. Individual tasks would remain the unit of assignment.
> The granularity and API for declaring weights, how weights from multiple
> processing stages contribute to a task’s estimated cost, how this information
> is passed to the assignor, and its interaction with existing assignment
> policies would be discussed in a KIP.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
