[
https://issues.apache.org/jira/browse/KAFKA-4696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16372081#comment-16372081
]
Guozhang Wang commented on KAFKA-4696:
--------------------------------------
[~Yohan123] That is a good question, and here are the initial thoughts: we will
enrich the rebalance protocol metadata (today it already contain the prev tasks
in order to enable sticky assignment) to include the information about the
number of state stores for each task, and use that as the approximate "weight"
of the task.
If we want to go even further, we can encode each task's "restoration progress
indicator", which will be computed as the sum of {{log-end-offset -
checkpointed-offset}} across all state stores of that task, and this indicate
how far the task need to be restored before it is ready to be processed, and
hence can be used as a more accurate "weight" of the tasks.
> Streams standby task assignment should be state-store aware
> -----------------------------------------------------------
>
> Key: KAFKA-4696
> URL: https://issues.apache.org/jira/browse/KAFKA-4696
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
> Affects Versions: 0.10.2.0, 0.11.0.0
> Reporter: Damian Guy
> Priority: Major
>
> Task Assignment is currently not aware of which tasks have State Stores. This
> can result in uneven balance of standby task assignment as all tasks are
> assigned, but only those tasks with state-stores are ever created by
> {{StreamThread}}. So what seems like an optimal strategy during assignment
> time could be sub-optimal post-assignment.
> For example, lets say we have 4 tasks (2 with state-stores), 2 clients,
> numStandbyReplicas = 1. Each client would get 2 active and 2 standby tasks.
> One of the clients may end up with both state-store tasks, while the other
> has none.
> Further to this, standby task configuration is currently "all or nothing". It
> might make sense to allow more fine grained configuration, i.e., the ability
> to specify the number of standby replicas individually for each stateful
> operator.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
