sanghyeok An created KAFKA-21180:
------------------------------------
Summary: Support user-defined processing weights for Kafka Streams
task assignment
Key: KAFKA-21180
URL: https://issues.apache.org/jira/browse/KAFKA-21180
Project: Kafka
Issue Type: Improvement
Reporter: sanghyeok An
Assignee: sanghyeok An
*Description*
Processing costs can vary across stages within a Kafka Streams topology. Simple
filtering or transformation may require relatively little CPU, while complex
aggregations or windowed operations may require more processing resources. As a
result, assigning an equal number of tasks to each instance does not
necessarily produce a balanced workload.
Automatically measuring these differences at runtime and incorporating them
into assignment decisions on the client or broker is challenging. Observed load
depends on input traffic and the execution environment, and temporary traffic
spikes need to be distinguished from sustained differences in processing cost.
Reassigning tasks in response to short-lived fluctuations may introduce
unnecessary task movement and state restoration costs.
Application developers and operators, however, may understand the relative
processing costs through knowledge of the application logic and experience from
long-term monitoring. It would be useful to express this knowledge as
user-defined weights and incorporate them into task assignment.
*Proposal*
Allow users to specify relative processing weights through the builder APIs
used to construct a topology, or a similar mechanism, and make these weights
available to the task assignor.
For example, users could assign a lower weight to a simple filtering stage and
a higher weight to a complex aggregation or windowed operation. The weights
would represent relative estimated processing costs, rather than execution
priorities. Individual tasks would remain the unit of assignment.
The granularity and API for declaring weights, how weights from multiple
processing stages contribute to a task’s estimated cost, how this information
is passed to the assignor, and its interaction with existing assignment
policies would be discussed in a KIP.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
