[
https://issues.apache.org/jira/browse/KAFKA-21391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ziyun Fu updated KAFKA-21391:
-----------------------------
Description:
When rack.aware.assignment.tags is set, the sticky assignor ranks rack
diversity above stickiness. It places standbys in four steps, and each step
runs for all tasks before the next one starts:
1. to a previous member (active or standby) of the task that makes the task
more rack-diverse
2. to the least-loaded member that makes the task more rack-diverse
3. to a previous member of the task
4. to the least-loaded member
The least-loaded pick in step 2 only looks at load. It doesn't know that the
member it picks is a previous member of another task that is still waiting for
step 3. For example:
- Members A and B make task1 equally rack-diverse, and neither has task1's
state. A an B have the same load only depending on which is in the front, so
step 2 puts task1 on A.
- A is also a previous standby member of task2, but it doesn't make task2
more rack-diverse (e.g. it is in the rack of task2's active), so task2 can only
go back to A in step 3.
- If task1 took A's last free slot, step 3 can't put task2 on A. Step 4 then
puts task2 on a member without its state, which has to restore it.
was:
In the rack-aware sticky assignor, we are planning to priorize rack ahead of
stickiness.
There will be 4 steps,
# assign the standby task to previous holder who is rack-diverse
# assign the standby task to the least-loaded rack-diverse member
# assign it to a previous holder
# assignor it to the least-loaded member
In step 2, member A and B are both the same level rack diverse to task1, if A
has least-loaded then B, we assign task1 to A. But there's cases that A is
actually the previous member for task2, then in step 3 for task2, A could not
able to hold back task2 because it's holding task1. If we are able to assign
task1 to B then we will not lose rack diversity and at the same time improve
stickiness
> Improve standby stickiness of rack-aware standby placement in the sticky task
> assignor
> ----------------------------------------------------------------------------------------
>
> Key: KAFKA-21391
> URL: https://issues.apache.org/jira/browse/KAFKA-21391
> Project: Kafka
> Issue Type: Improvement
> Components: group-coordinator, streams
> Reporter: Ziyun Fu
> Assignee: Ziyun Fu
> Priority: Major
>
> When rack.aware.assignment.tags is set, the sticky assignor ranks rack
> diversity above stickiness. It places standbys in four steps, and each step
> runs for all tasks before the next one starts:
>
>
>
> 1. to a previous member (active or standby) of the task that makes the task
> more rack-diverse
> 2. to the least-loaded member that makes the task more rack-diverse
>
> 3. to a previous member of the task
>
> 4. to the least-loaded member
>
>
> The least-loaded pick in step 2 only looks at load. It doesn't know that the
> member it picks is a previous member of another task that is still waiting
> for step 3. For example:
>
>
>
> - Members A and B make task1 equally rack-diverse, and neither has task1's
> state. A an B have the same load only depending on which is in the front, so
> step 2 puts task1 on A.
> - A is also a previous standby member of task2, but it doesn't make task2
> more rack-diverse (e.g. it is in the rack of task2's active), so task2 can
> only go back to A in step 3.
>
> - If task1 took A's last free slot, step 3 can't put task2 on A. Step 4
> then puts task2 on a member without its state, which has to restore it.
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
