[
https://issues.apache.org/jira/browse/KAFKA-17755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jerry Cai updated KAFKA-17755:
------------------------------
Description:
currently rack-aware partition assignor's design have obvious two significant
flaws:
# Rely on the Kafka broker side to enable the configuration:
replica.selector.class -> RackAwareReplicaSelector. This breaks the agreement
that the Partition Assignor can be customized independently by the client.
# It destroys the read-write consistency principle of Kafka, leading to load
imbalance and many other side effects.
This improvement of merge request is based on the following ideas:
Always read messages from the leader of the broker, that is, do not enable
replica.selector.class
, it can be aligned based only on the rack information of the leader, so that
balance can be achieved first. In the case of balance, the same rack partition
allocation strategy as the leader should be given priority as much as possible.
was:
During my local test and debug, I noticed that the below logical is in correct,
it needs to change
from !racksPerPartition.values().stream().allMatch(partitionRacks::equals)
to racksPerPartition.values().stream().allMatch(partitionRacks::equals)
current logical
{code:java}
protected boolean useRackAwareAssignment(Set<String> consumerRacks, Set<String>
partitionRacks, Map<TopicPartition, Set<String>> racksPerPartition) {
if (consumerRacks.isEmpty() || Collections.disjoint(consumerRacks,
partitionRacks))
return false;
else if (preferRackAwareLogic)
return true;
else {
return
!racksPerPartition.values().stream().allMatch(partitionRacks::equals);
}
}
{code}
expected logical
{code:java}
protected boolean useRackAwareAssignment(Set<String> consumerRacks, Set<String>
partitionRacks, Map<TopicPartition, Set<String>> racksPerPartition) {
if (consumerRacks.isEmpty() || Collections.disjoint(consumerRacks,
partitionRacks))
return false;
else if (preferRackAwareLogic)
return true;
else {
return
racksPerPartition.values().stream().allMatch(partitionRacks::equals);
}
}
{code}
Due Date: 1/May/25 (was: 12/Oct/24)
Issue Type: Improvement (was: Bug)
Priority: Major (was: Critical)
> AbstractPartitionAssignor can not enable RackAwareAssignment base on lead
> rack mode
> -----------------------------------------------------------------------------------
>
> Key: KAFKA-17755
> URL: https://issues.apache.org/jira/browse/KAFKA-17755
> Project: Kafka
> Issue Type: Improvement
> Components: consumer
> Affects Versions: 3.7.0, 3.8.0, 3.7.1
> Reporter: Jerry Cai
> Assignee: Jerry Cai
> Priority: Major
>
> currently rack-aware partition assignor's design have obvious two significant
> flaws:
> # Rely on the Kafka broker side to enable the configuration:
> replica.selector.class -> RackAwareReplicaSelector. This breaks the agreement
> that the Partition Assignor can be customized independently by the client.
> # It destroys the read-write consistency principle of Kafka, leading to load
> imbalance and many other side effects.
> This improvement of merge request is based on the following ideas:
> Always read messages from the leader of the broker, that is, do not enable
> replica.selector.class
> , it can be aligned based only on the rack information of the leader, so that
> balance can be achieved first. In the case of balance, the same rack
> partition allocation strategy as the leader should be given priority as much
> as possible.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
