Ziyun Fu created KAFKA-21308:
--------------------------------
Summary: StickyTaskAssignor: a full previous member hides other
previous members that have room
Key: KAFKA-21308
URL: https://issues.apache.org/jira/browse/KAFKA-21308
Project: Kafka
Issue Type: Improvement
Reporter: Ziyun Fu
{{StickyTaskAssignor#findPrevMemberWithLeastLoad}} picks one previous member by
load, and the caller checks that member's quota only afterwards. If the picked
member is at its quota, the lookup returns no member, even when another
previous member of the task is below its quota. The task then goes to the
least-loaded process, which may not have the task's state.
The candidate is replaced only when another member is strictly lower on both
process load and member load, so a full member stays picked easily.
*Example:* task {{t}} has previous standby members {{a1}} (process A) and
{{b1}} (process B), with {{a1}} first in the list.
* {{a1}} is at the per-member quota. A's other member {{a2}} has no tasks, so
A's process load is lower than B's.
* {{b1}} is below the quota, but B's process load is not lower than A's, so
{{a1}} stays picked.
* The quota check on {{a1}} fails, and the standby of {{t}} goes to the
least-loaded process (for example {{{}a2{}}}, which has no state), not to
{{{}b1{}}}.
*Affected call sites:*
* Active task assignment from previous standby members
({{{}hasUnfulfilledActiveTaskQuota{}}} checked after the lookup)
* Standby task assignment from previous standby members
({{{}hasUnfulfilledTaskQuota{}}} checked after the lookup)
* The rack-aware standby steps of KAFKA-20999, which use the same lookup
*Proposed fix:* check the quota inside {{{}findPrevMemberWithLeastLoad{}}},
skipping members that are at their quota, so the least-loaded previous member
with room is picked. This changes some assignments, since more tasks stay on
members that have their state.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
