[
https://issues.apache.org/jira/browse/KAFKA-16869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17956779#comment-17956779
]
Lorcan commented on KAFKA-16869:
--------------------------------
Hi [~ableegoldman], I've got a proof of concept of this change to work in a
local branch. I have also ported over some of the existing tests to use the new
HighAvailabilityTaskAssignor that implements the TaskAssignor interface and
these pass.
My initial goal was to use my own HighAvailabilityClientState objects during
the rebalancing, which would store only the information needed during the
rebalance, rather than all the extra information that the ClientState class
holds.
A full replacement (at this stage) was not possible because, for example, the
standbyTaskAssignor assign() and isAllowedTaskMovement() functions use the
ClientState object. I wanted to focus on the HighAvailabilityTaskAssignor
change and not start rewriting the Standby classes too.
The optimizeRackAwareActiveTasks() function uses the KafkaStreamsAssignment
objects for rack optimisations during the assignment.
This has led to a complicated series of synchronisation steps between these
three data objects due to the different dependencies in this assignment class.
Given the complexity of synchronising the different state objects mentioned,
could you provide advice as to whether this is the correct approach to take for
this work? Thanks
> Rewrite HighAvailabilityTaskAssignor to implement the new TaskAssignor
> interface
> --------------------------------------------------------------------------------
>
> Key: KAFKA-16869
> URL: https://issues.apache.org/jira/browse/KAFKA-16869
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
> Reporter: A. Sophie Blee-Goldman
> Assignee: Lorcan
> Priority: Major
>
> We need to add a new HighAvailabilityTaskAssignor that implements the new
> TaskAssignor interface. Once we have that, we need to remember to also make
> these related changes:
> # Change the StreamsConfig.TASK_ASSIGNOR_CLASS_CONFIG default from null to
> the new HAAssignor
> # Check for this new HAAssignor type when evaluating the OptionalInt
> rack-aware assignment configs in the public AssignmentConfigs class. If these
> configs are Optional.empty() and the new HAAssignor is used, they should be
> overridden to the HAAssignor-specific default values. This code already
> exists but should be updated to check for the new HAAssignor class name
> instead of "null"
> # Until the old HAAssignor and old internal task assignor config can be
> removed completely, make sure the new HAAssignor is used by default when a
> TaskAssignor is selected in StreamsPartitionAssignor
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
