[
https://issues.apache.org/jira/browse/KAFKA-20998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gavin Wang updated KAFKA-20998:
-------------------------------
Description:
The "classic" protocol offers two built-in assignors, "sticky" and
"ha_assignor". So far, "streams" only offers the sticky assignor.
KIP-1071 defines both "sticky" and "ha_assignor" as options, too, and we can
now close this gap.
However, we should amend the KIP slightly, and call the second built-in
assignor "balanced" (instead of "ha_assignor" as currently specified on the
KIP). The reason is, that high-availability features are delivered
independently on the assignment strategy with "streams" protocol, ie, "warmup
tasks" are inserted by the new refiner step, not the task assignor. Thus, the
new assignor is not a 1:1 of the classic HA-assignor, but only need to deliver
a different ("balanced") task assignment compare to sticky assignor.
We can start with a 1:1 translation of he existing task assignment algorithm,
but should use JMH benchmarks, and refine as needed, similar to what we did for
the sticky assignor. Because task assignment is running inside the GC, we need
to ensure that the task assignor performance is good, so it doesn't put too
much load on the broker.
The classic HA-assignor also supports rack-aware standby task assignment, which
we should exclude in this ticket. We rather want so deliver rack-aware standby
task assignment for both "sticky" and "balanced" assignor, and are tracking
this in it's own sub-task.
was:
The "classic" protocol offers two built-in assignors, "sticky" and
"ha_assignor". So far, "streams" only offers the sticky assignor.
KIP-1017 defines both "sticky" and "ha_assignor" as options, too, and we can
now close this gap.
However, we should amend the KIP slightly, and call the second built-in
assignor "balanced" (instead of "ha_assignor" as currently specified on the
KIP). The reason is, that high-availability features are delivered
independently on the assignment strategy with "streams" protocol, ie, "warmup
tasks" are inserted by the new refiner step, not the task assignor. Thus, the
new assignor is not a 1:1 of the classic HA-assignor, but only need to deliver
a different ("balanced") task assignment compare to sticky assignor.
We can start with a 1:1 translation of he existing task assignment algorithm,
but should use JMH benchmarks, and refine as needed, similar to what we did for
the sticky assignor. Because task assignment is running inside the GC, we need
to ensure that the task assignor performance is good, so it doesn't put too
much load on the broker.
The classic HA-assignor also supports rack-aware standby task assignment, which
we should exclude in this ticket. We rather want so deliver rack-aware standby
task assignment for both "sticky" and "balanced" assignor, and are tracking
this in it's own sub-task.
> Add HA task assignor
> --------------------
>
> Key: KAFKA-20998
> URL: https://issues.apache.org/jira/browse/KAFKA-20998
> Project: Kafka
> Issue Type: Sub-task
> Components: group-coordinator, streams
> Reporter: Matthias J. Sax
> Assignee: Gavin Wang
> Priority: Major
>
> The "classic" protocol offers two built-in assignors, "sticky" and
> "ha_assignor". So far, "streams" only offers the sticky assignor.
> KIP-1071 defines both "sticky" and "ha_assignor" as options, too, and we can
> now close this gap.
> However, we should amend the KIP slightly, and call the second built-in
> assignor "balanced" (instead of "ha_assignor" as currently specified on the
> KIP). The reason is, that high-availability features are delivered
> independently on the assignment strategy with "streams" protocol, ie, "warmup
> tasks" are inserted by the new refiner step, not the task assignor. Thus, the
> new assignor is not a 1:1 of the classic HA-assignor, but only need to
> deliver a different ("balanced") task assignment compare to sticky assignor.
> We can start with a 1:1 translation of he existing task assignment algorithm,
> but should use JMH benchmarks, and refine as needed, similar to what we did
> for the sticky assignor. Because task assignment is running inside the GC, we
> need to ensure that the task assignor performance is good, so it doesn't put
> too much load on the broker.
> The classic HA-assignor also supports rack-aware standby task assignment,
> which we should exclude in this ticket. We rather want so deliver rack-aware
> standby task assignment for both "sticky" and "balanced" assignor, and are
> tracking this in it's own sub-task.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
