[
https://issues.apache.org/jira/browse/KAFKA-21180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-21180:
------------------------------------
Labels: (was: streams)
> Support user-defined processing weights for Kafka Streams task assignment
> -------------------------------------------------------------------------
>
> Key: KAFKA-21180
> URL: https://issues.apache.org/jira/browse/KAFKA-21180
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: sanghyeok An
> Assignee: sanghyeok An
> Priority: Minor
>
> *Description*
> Processing costs can vary across stages within a Kafka Streams topology.
> Simple filtering or transformation may require relatively little CPU, while
> complex aggregations or windowed operations may require more processing
> resources. As a result, assigning an equal number of tasks to each instance
> does not necessarily produce a balanced workload.
> Automatically measuring these differences at runtime and incorporating them
> into assignment decisions on the client or broker is challenging. Observed
> load depends on input traffic and the execution environment, and temporary
> traffic spikes need to be distinguished from sustained differences in
> processing cost. Reassigning tasks in response to short-lived fluctuations
> may introduce unnecessary task movement and state restoration costs.
> Application developers and operators, however, may understand the relative
> processing costs through knowledge of the application logic and experience
> from long-term monitoring. It would be useful to express this knowledge as
> user-defined weights and incorporate them into task assignment.
>
> *Proposal*
> Allow users to specify relative processing weights through the builder APIs
> used to construct a topology, or a similar mechanism, and make these weights
> available to the task assignor.
> For example, users could assign a lower weight to a simple filtering stage
> and a higher weight to a complex aggregation or windowed operation. The
> weights would represent relative estimated processing costs, rather than
> execution priorities. Individual tasks would remain the unit of assignment.
> The granularity and API for declaring weights, how weights from multiple
> processing stages contribute to a task’s estimated cost, how this information
> is passed to the assignor, and its interaction with existing assignment
> policies would be discussed in a KIP.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
