[
https://issues.apache.org/jira/browse/KAFKA-21308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ziyun Fu updated KAFKA-21308:
-----------------------------
Component/s: streams
> StickyTaskAssignor: a full previous member hides other previous members that
> have room
> ----------------------------------------------------------------------------------------
>
> Key: KAFKA-21308
> URL: https://issues.apache.org/jira/browse/KAFKA-21308
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Ziyun Fu
> Assignee: Ziyun Fu
> Priority: Major
>
> {{StickyTaskAssignor#findPrevMemberWithLeastLoad}} picks one previous member
> by load, and the caller checks that member's quota only afterwards. If the
> picked member is at its quota, the lookup returns no member, even when
> another previous member of the task is below its quota. The task then goes to
> the least-loaded process, which may not have the task's state.
> The candidate is replaced only when another member is strictly lower on both
> process load and member load, so a full member stays picked easily.
> *Example:* task {{t}} has previous standby members {{a1}} (process A) and
> {{b1}} (process B), with {{a1}} first in the list.
> * {{a1}} is at the per-member quota. A's other member {{a2}} has no tasks,
> so A's process load is lower than B's.
> * {{b1}} is below the quota, but B's process load is not lower than A's, so
> {{a1}} stays picked.
> * The quota check on {{a1}} fails, and the standby of {{t}} goes to the
> least-loaded process (for example {{{}a2{}}}, which has no state), not to
> {{{}b1{}}}.
> *Affected call sites:*
> * Active task assignment from previous standby members
> ({{{}hasUnfulfilledActiveTaskQuota{}}} checked after the lookup)
> * Standby task assignment from previous standby members
> ({{{}hasUnfulfilledTaskQuota{}}} checked after the lookup)
> * The rack-aware standby steps of KAFKA-20999, which use the same lookup
> *Proposed fix:* check the quota inside {{{}findPrevMemberWithLeastLoad{}}},
> skipping members that are at their quota, so the least-loaded previous member
> with room is picked. This changes some assignments, since more tasks stay on
> members that have their state.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
