[ https://issues.apache.org/jira/browse/KAFKA-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890935#comment-15890935 ]
Jeff Widman commented on KAFKA-4677: ------------------------------------ Is KIP-54 related since it also implements a sticky partition assigner, albeit within the Kafka consumer? I haven't worked directly with streams, but I thought they piggy-backed on top of KafKa consumer. If so, wouldn't it be better to just inherit the KIP-54 implementation? Again, I don't really understand internals, so not sure. > Avoid unnecessary task movement across threads during rebalance > --------------------------------------------------------------- > > Key: KAFKA-4677 > URL: https://issues.apache.org/jira/browse/KAFKA-4677 > Project: Kafka > Issue Type: Bug > Components: streams > Affects Versions: 0.10.2.0 > Reporter: Damian Guy > Assignee: Damian Guy > Fix For: 0.10.3.0 > > > StreamPartitionAssigner tries to follow a sticky assignment policy to avoid > expensive task migration. Currently, it does this in a best-effort approach. > We could observe a case, for which tasks did migrate for no good reason, thus > we assume that the current implementation could be improved to be more sticky. > The concrete scenario is as follows: > assume we have topology with 3 tasks, A, B, C > assume we have 3 threads, each executing one task: 1-A, 2-B, 3-C > for some reason, thread 1 goes down and a rebalance gets triggered > thread 2 and 3 get their partitions revoked > sometimes (not sure what the exact condition for this is), the new assignment > flips the assignment for task B and C (task A is newly assigned to either > thread 2 or 3) > > possible new assignment 2(A,C) and 3-B > There is no obvious reason (like load-balancing) why the task assignment for > B and C does change to the other thread resulting in unnecessary task > migration. -- This message was sent by Atlassian JIRA (v6.3.15#6346)