Gavin Wang created KAFKA-21194:
----------------------------------
Summary: Speed up the balanced task assignor and register it as a
built-in assignor
Key: KAFKA-21194
URL: https://issues.apache.org/jira/browse/KAFKA-21194
Project: Kafka
Issue Type: Sub-task
Components: streams
Affects Versions: 4.5.0
Reporter: Gavin Wang
Assignee: Gavin Wang
KAFKA-20998 added \{{BalancedTaskAssignor}}, a one-to-one translation of the
classic \{{HighAvailabilityTaskAssignor}} placement, not yet registered as a
built-in. Its fix-up loop tests every pair of processes on each pass, so it
costs Θ(P²) per pass even when nothing moves: up to 26x the sticky assignor's
time in JMH (12.4 ms for 1000 single-member processes).
Plan:
# Index-based implementation and an O(P) early exit for passes that cannot move
a task. Same assignment.
# Place active tasks with the Sainte-Laguë divisor method (next task to the
process with the smallest (2n+1)/c), read from a precomputed period, instead of
round-robin plus fix-up. Same placement for equal member counts; for unequal
ones the same per-member balance and a more even spread of each subtopology.
Separate PR, as it changes placement.
# Register \{{balanced}} in
\{{GroupCoordinatorConfig.STREAMS_GROUP_BUILTIN_ASSIGNORS}} once KIP-1071 is
amended from \{{highly_available}}.
Prototypes: 1 is 6-10x faster, 1+2 is 12-19x, about a quarter of the sticky
assignor's time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
