Re: [PR] KAFKA-19909: Add rack awareness assignment to UniformHomogeneousAssignmentBuilder [kafka]
github-actions[bot] commented on PR #2: URL: https://github.com/apache/kafka/pull/2#issuecomment-5041907114 This PR is being marked as stale since it has not had any activity in 90 days. If you would like to keep this PR alive, please leave a comment asking for a review. If the PR has merge conflicts, update it with the latest from the base branch. If you are having difficulty finding a reviewer, please reach out on the [mailing list](https://kafka.apache.org/contact). If this PR is no longer valid or desired, please feel free to close it. If no activity occurs in the next 30 days, it will be automatically closed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19909: Add rack awareness assignment to UniformHomogeneousAssignmentBuilder [kafka]
github-actions[bot] commented on PR #2: URL: https://github.com/apache/kafka/pull/2#issuecomment-4293501561 This PR is being marked as stale since it has not had any activity in 90 days. If you would like to keep this PR alive, please leave a comment asking for a review. If the PR has merge conflicts, update it with the latest from the base branch. If you are having difficulty finding a reviewer, please reach out on the [mailing list](https://kafka.apache.org/contact). If this PR is no longer valid or desired, please feel free to close it. If no activity occurs in the next 30 days, it will be automatically closed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19909: Add rack awareness assignment to UniformHomogeneousAssignmentBuilder [kafka]
squah-confluent commented on PR #2: URL: https://github.com/apache/kafka/pull/2#issuecomment-3707344977 @FrankYang0529 Thanks for collecting the new benchmark results. The non-rack aware numbers look good. The rack aware number are better but still a little slow. It's not ideal to be blocking the group coordinator thread for 150 ms. Maybe this won't be too bad in practice, since 1. most groups won't be as large 2. I'm working on a KIP to reduce the impact of slow assignors If we really want to, I think it's possible to improve performance further by re-designing the `SubscribedTopicDescriber.racksForPartition` interface, but maybe it's best left to a separate PR. `jmh-benchmarks/README.md` has instructions for running the benchmarks with libasyncProfiler which will generate a flame graph of the assignor run. Separately I have some concerns about stickiness when static members are replaced. The group coordinator assigns the new static member a new member id and keeps the previous assignment, so the order of member ids is not stable (I'm aware the existing range assignors also have this problem). How expensive would it be to track the previous owner of partitions in `maybeRevokePartitions` and maybe add a new pass in between `assignRackAwarenessRemainingPartitions` and `assignRemainingPartitions` to restore those partitions to their preferred sticky members? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19909: Add rack awareness assignment to UniformHomogeneousAssignmentBuilder [kafka]
FrankYang0529 commented on PR #2: URL: https://github.com/apache/kafka/pull/2#issuecomment-3633043052 Performance for skipping partition racks collection if member racks is empty. PR: Uniform Homogeneous rack awareness vs non rack awareness ``` Benchmark (assignmentType) (assignorType) (isRackAware) (memberCount) (partitionsToMemberRatio) (subscriptionType) (topicCount) Mode CntScoreError Units ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true100 5 HOMOGENEOUS 10 avgt30.076 ± 0.006 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true100 5 HOMOGENEOUS 100 avgt30.104 ± 0.006 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true100 5 HOMOGENEOUS 1000 avgt30.202 ± 0.007 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true100 10 HOMOGENEOUS 10 avgt30.147 ± 0.017 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true100 10 HOMOGENEOUS 100 avgt30.203 ± 0.015 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true100 10 HOMOGENEOUS 1000 avgt30.484 ± 0.016 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true100 50 HOMOGENEOUS 10 avgt30.987 ± 0.152 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true100 50 HOMOGENEOUS 100 avgt30.928 ± 0.100 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true100 50 HOMOGENEOUS 1000 avgt31.584 ± 0.103 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true500 5 HOMOGENEOUS 10 avgt30.495 ± 0.049 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true500 5 HOMOGENEOUS 100 avgt30.652 ± 0.078 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true500 5 HOMOGENEOUS 1000 avgt30.980 ± 0.091 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true500 10 HOMOGENEOUS 10 avgt31.014 ± 0.489 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true500 10 HOMOGENEOUS 100 avgt31.181 ± 0.143 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true500 10 HOMOGENEOUS 1000 avgt31.744 ± 0.149 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true500 50 HOMOGENEOUS 10 avgt34.959 ± 0.106 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true500 50 HOMOGENEOUS 100 avgt35.699 ± 0.293 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true500 50 HOMOGENEOUS 1000 avgt36.002 ± 0.093 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true 1000 5 HOMOGENEOUS 10 avgt31.153 ± 0.350 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true 1000 5 HOMOGENEOUS 100 avgt31.266 ± 0.278 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true 1000 5 HOMOGENEOUS 1000 avgt32.034 ± 0.052 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true 1000 10 HOMOGENEOUS 10
Re: [PR] KAFKA-19909: Add rack awareness assignment to UniformHomogeneousAssignmentBuilder [kafka]
FrankYang0529 commented on code in PR #2:
URL: https://github.com/apache/kafka/pull/2#discussion_r2598805118
##
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/UniformHomogeneousAssignmentBuilder.java:
##
@@ -101,6 +113,24 @@ public class UniformHomogeneousAssignmentBuilder {
this.unassignedPartitions = new ArrayList<>();
this.targetAssignment = new HashMap<>();
+this.partitionRacks = new HashMap<>();
+
+Set allMemberRacks = new HashSet<>();
+for (String memberId : groupSpec.memberIds()) {
+
groupSpec.memberSubscription(memberId).rackId().ifPresent(allMemberRacks::add);
+}
+
+Set allPartitionRacks = new HashSet<>();
+for (Uuid topicId : this.subscribedTopicIds) {
+int partitionCount =
subscribedTopicDescriber.numPartitions(topicId);
+for (int partitionId = 0; partitionId < partitionCount;
partitionId++) {
+Set racks =
subscribedTopicDescriber.racksForPartition(topicId, partitionId);
+partitionRacks.put(new TopicIdPartition(topicId, partitionId),
racks);
+allPartitionRacks.addAll(racks);
+}
+}
Review Comment:
> @dajac mentioned we could try shortcutting the rack-aware part if no
members have racks.
Yes, good suggestion. I changed the code to skip gathering the racks of all
partitions if no members have racks. However, for rack-awareness assignment, it
still needs to gather information and the worst case may cost `293.988` ms for
500,000 topic-partitions to 10,000 members. Can we afford this cost in server
side? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19909: Add rack awareness assignment to UniformHomogeneousAssignmentBuilder [kafka]
Copilot commented on code in PR #2:
URL: https://github.com/apache/kafka/pull/2#discussion_r2598156989
##
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/AssignorHelpers.java:
##
@@ -69,4 +71,34 @@ static HashMap newHashMap(int numMappings) {
static HashSet newHashSet(int numElements) {
return new HashSet<>((int) (((numElements + 1) / 0.75f) + 1));
}
+
+
+/**
+ * Checks if the member's rack matches any of the partition's racks.
+ * @param memberRackId The member's rack id.
+ * @param partitionRackIds The partition's rack ids.
+ * @return True if the member's rack matches any of the partition's racks,
false otherwise.
+ */
+public static boolean isRackMatch(Optional memberRackId,
Set partitionRackIds) {
+return memberRackId.isPresent() &&
partitionRackIds.contains(memberRackId.get());
+}
+
+/**
+ * Determines whether rack-aware assignment should be used based on the
provided racks.
+ * @param allMemberRacks The set of all member racks.
+ * @param allPartitionRacks The set of all partition racks.
+ * @param racksPerPartition A map of partitions to their respective racks.
+ * @return True if member racks and partition racks overlap and not all
partitions have the same set of racks, false otherwise.
+ */
+public static boolean useRackAwareAssignment(
+Set allMemberRacks,
+Set allPartitionRacks,
+Map> racksPerPartition
+) {
+if (allMemberRacks.isEmpty() || Collections.disjoint(allMemberRacks,
allPartitionRacks))
+return false;
+else {
+return
!racksPerPartition.values().stream().allMatch(allPartitionRacks::equals);
+}
Review Comment:
[nitpick] The if-else statement could be simplified by removing the else
block and directly returning the condition. Consider:
```java
return !allMemberRacks.isEmpty()
&& !Collections.disjoint(allMemberRacks, allPartitionRacks)
&&
!racksPerPartition.values().stream().allMatch(allPartitionRacks::equals);
```
This is more concise and avoids the unnecessary else block.
```suggestion
return !allMemberRacks.isEmpty()
&& !Collections.disjoint(allMemberRacks, allPartitionRacks)
&&
!racksPerPartition.values().stream().allMatch(allPartitionRacks::equals);
```
##
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/assignor/CommonAssignorTests.java:
##
@@ -123,9 +124,14 @@ public static void testAssignmentReuse(PartitionAssignor
assignor, SubscriptionT
);
for (String memberId : members.keySet()) {
-// The assignment map from the assignor must be the same as the
immutable assignment map
-// that went in.
-assertSame(membersWithAssignment.get(memberId).partitions(),
secondAssignment.members().get(memberId).partitions());
+if (rackAware) {
+// With rack awareness, the assignment maps may be mutable
cause of revoking non-matched partitions.
Review Comment:
Grammatical error: "cause of" should be "because of".
```suggestion
// With rack awareness, the assignment maps may be mutable
because of revoking non-matched partitions.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19909: Add rack awareness assignment to UniformHomogeneousAssignmentBuilder [kafka]
squah-confluent commented on code in PR #2:
URL: https://github.com/apache/kafka/pull/2#discussion_r2594744175
##
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/UniformHomogeneousAssignmentBuilder.java:
##
@@ -101,6 +113,24 @@ public class UniformHomogeneousAssignmentBuilder {
this.unassignedPartitions = new ArrayList<>();
this.targetAssignment = new HashMap<>();
+this.partitionRacks = new HashMap<>();
+
+Set allMemberRacks = new HashSet<>();
+for (String memberId : groupSpec.memberIds()) {
+
groupSpec.memberSubscription(memberId).rackId().ifPresent(allMemberRacks::add);
+}
+
+Set allPartitionRacks = new HashSet<>();
+for (Uuid topicId : this.subscribedTopicIds) {
+int partitionCount =
subscribedTopicDescriber.numPartitions(topicId);
+for (int partitionId = 0; partitionId < partitionCount;
partitionId++) {
+Set racks =
subscribedTopicDescriber.racksForPartition(topicId, partitionId);
+partitionRacks.put(new TopicIdPartition(topicId, partitionId),
racks);
+allPartitionRacks.addAll(racks);
+}
+}
Review Comment:
A lot of the constructor cost is in gathering the racks of all partitions.
@dajac mentioned we could try shortcutting the rack-aware part if no members
have racks.
Or we could also check if no members have racks matching any broker, since
clients can misconfigure their racks. The number of racks is <= the number of
brokers, which is small and we can collect the set of broker racks cheaply from
the `MetadataImage` underlying the `SubscribedTopicDescriber`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19909: Add rack awareness assignment to UniformHomogeneousAssignmentBuilder [kafka]
squah-confluent commented on code in PR #2:
URL: https://github.com/apache/kafka/pull/2#discussion_r2585436106
##
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/UniformHomogeneousAssignmentBuilder.java:
##
@@ -240,13 +277,75 @@ private void maybeRevokePartitions() {
}
}
+/**
+ * Assign the unassigned partitions to the unfilled members if member and
partition racks are matched.
+ */
+private void assignRackAwarenessRemainingPartitions() {
+// Assign partitions to members with descending order. This avoids the
cost of shifting elements.
+for (int i = unassignedPartitions.size() - 1; i >= 0; i--) {
+TopicIdPartition tip = unassignedPartitions.get(i);
+boolean isPartitionAssigned = false;
+
+for (var unfilledMembersIter = unfilledMembers.iterator();
unfilledMembersIter.hasNext(); ) {
+MemberWithRemainingQuota unfilledMember =
unfilledMembersIter.next();
+if (unfilledMember.remainingQuota() == 0 &&
remainingMembersToGetAnExtraPartition == 0) {
+unfilledMembersIter.remove();
+continue;
+}
+
+String memberId = unfilledMember.memberId;
+if
(!AssignorHelpers.isRackMatch(groupSpec.memberSubscription(memberId).rackId(),
+partitionRacks.getOrDefault(tip, Set.of( {
+continue;
+}
+
+Map> newAssignment =
targetAssignment.get(memberId).partitions();
+if (AssignorHelpers.isImmutableMap(newAssignment)) {
+// If the new assignment is immutable, we must create a
deep copy of it
+// before altering it.
+newAssignment =
AssignorHelpers.deepCopyAssignment(newAssignment);
+targetAssignment.put(memberId, new
MemberAssignmentImpl(newAssignment));
Review Comment:
The fix has been merged
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19909: Add rack awareness assignment to UniformHomogeneousAssignmentBuilder [kafka]
squah-confluent commented on code in PR #2:
URL: https://github.com/apache/kafka/pull/2#discussion_r2582160658
##
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/UniformHomogeneousAssignmentBuilder.java:
##
@@ -240,13 +277,75 @@ private void maybeRevokePartitions() {
}
}
+/**
+ * Assign the unassigned partitions to the unfilled members if member and
partition racks are matched.
+ */
+private void assignRackAwarenessRemainingPartitions() {
+// Assign partitions to members with descending order. This avoids the
cost of shifting elements.
+for (int i = unassignedPartitions.size() - 1; i >= 0; i--) {
+TopicIdPartition tip = unassignedPartitions.get(i);
+boolean isPartitionAssigned = false;
+
+for (var unfilledMembersIter = unfilledMembers.iterator();
unfilledMembersIter.hasNext(); ) {
+MemberWithRemainingQuota unfilledMember =
unfilledMembersIter.next();
+if (unfilledMember.remainingQuota() == 0 &&
remainingMembersToGetAnExtraPartition == 0) {
+unfilledMembersIter.remove();
+continue;
+}
+
+String memberId = unfilledMember.memberId;
+if
(!AssignorHelpers.isRackMatch(groupSpec.memberSubscription(memberId).rackId(),
+partitionRacks.getOrDefault(tip, Set.of( {
+continue;
+}
+
+Map> newAssignment =
targetAssignment.get(memberId).partitions();
+if (AssignorHelpers.isImmutableMap(newAssignment)) {
+// If the new assignment is immutable, we must create a
deep copy of it
+// before altering it.
+newAssignment =
AssignorHelpers.deepCopyAssignment(newAssignment);
+targetAssignment.put(memberId, new
MemberAssignmentImpl(newAssignment));
Review Comment:
I filed https://issues.apache.org/jira/browse/KAFKA-19955 and opened
https://github.com/apache/kafka/pull/21058 to fix it. It might improve the
rack-aware times by a bit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19909: Add rack awareness assignment to UniformHomogeneousAssignmentBuilder [kafka]
squah-confluent commented on code in PR #2:
URL: https://github.com/apache/kafka/pull/2#discussion_r2582004284
##
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/UniformHomogeneousAssignmentBuilder.java:
##
@@ -240,13 +277,75 @@ private void maybeRevokePartitions() {
}
}
+/**
+ * Assign the unassigned partitions to the unfilled members if member and
partition racks are matched.
+ */
+private void assignRackAwarenessRemainingPartitions() {
+// Assign partitions to members with descending order. This avoids the
cost of shifting elements.
+for (int i = unassignedPartitions.size() - 1; i >= 0; i--) {
+TopicIdPartition tip = unassignedPartitions.get(i);
+boolean isPartitionAssigned = false;
+
+for (var unfilledMembersIter = unfilledMembers.iterator();
unfilledMembersIter.hasNext(); ) {
+MemberWithRemainingQuota unfilledMember =
unfilledMembersIter.next();
+if (unfilledMember.remainingQuota() == 0 &&
remainingMembersToGetAnExtraPartition == 0) {
+unfilledMembersIter.remove();
+continue;
+}
+
+String memberId = unfilledMember.memberId;
+if
(!AssignorHelpers.isRackMatch(groupSpec.memberSubscription(memberId).rackId(),
+partitionRacks.getOrDefault(tip, Set.of( {
+continue;
+}
+
+Map> newAssignment =
targetAssignment.get(memberId).partitions();
+if (AssignorHelpers.isImmutableMap(newAssignment)) {
+// If the new assignment is immutable, we must create a
deep copy of it
+// before altering it.
+newAssignment =
AssignorHelpers.deepCopyAssignment(newAssignment);
+targetAssignment.put(memberId, new
MemberAssignmentImpl(newAssignment));
Review Comment:
https://github.com/apache/kafka/pull/20097 introduced a performance
regression. After that change `MemberAssignmentImpl` always wraps the new
assignment in another immutable map, so we will always deep copy here. Can we
fix the regression in a separate PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19909: Add rack awareness assignment to UniformHomogeneousAssignmentBuilder [kafka]
FrankYang0529 commented on PR #2: URL: https://github.com/apache/kafka/pull/2#issuecomment-3595983833 Hi @dajac, following is summary for trunk & PR comparison. The difference between trunk and PR for non-rackaware is from checking whether to use rack awareness strategy (Step 1). Current PR implementation needs `useRackStrategy` variable to skip rack awareness check if users don't need it. The performance check for step 1 takes about `95.415 ± 2.039 ms/op`. If we add this part to trunk result, the final results between trunk and PR for non-rackaware are similar. Most of the cost are used to collect `allMemberRacks`, `partitionRacks`, and `allPartitionRacks`. I'm wondering do we want to add another configuration to consumer group to indicate whether to enable rack aware check? If user doesn't need rack aware calculation, we can get similar result as trunk branch. https://github.com/apache/kafka/pull/2/files#diff-a1046bd3b6306ea23bdb5cba83c0302d8137ed9bc0d6ecc1368caf4bbca2612dR116-R133 | assignmentType | memberCount | partitionsToMemberRatio | subscriptionType | topicCount | Mode | Cnt | Trunk & Non-RackAware | PR & Non-RackAware | PR & RackAware | ||-|-|--||--|-|---||| | FULL | 1 | 50 | HOMOGENEOUS | 1000 | avgt | 3 | 12.305 ± 1.321 ms/op | 110.267 ± 20.565 ms/op | 293.988 ± 38.140 ms/op | | INCREMENTAL| 1 | 50 | HOMOGENEOUS | 1000 | avgt | 3 | 5.778 ± 0.643 ms/op | 101.283 ± 1.971 ms/op | 158.401 ± 20.146 ms/op | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19909: Add rack awareness assignment to UniformHomogeneousAssignmentBuilder [kafka]
FrankYang0529 commented on PR #2: URL: https://github.com/apache/kafka/pull/2#issuecomment-3588016615 PR: Uniform Homogeneous rack awareness vs non rack awareness ``` Benchmark (assignmentType) (assignorType) (isRackAware) (memberCount) (partitionsToMemberRatio) (subscriptionType) (topicCount) Mode CntScore Error Units ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true100 5 HOMOGENEOUS 10 avgt30.108 ± 0.009 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true100 5 HOMOGENEOUS 100 avgt30.152 ± 0.009 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true100 5 HOMOGENEOUS 1000 avgt30.188 ± 0.015 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true100 10 HOMOGENEOUS 10 avgt30.245 ± 0.036 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true100 10 HOMOGENEOUS 100 avgt30.355 ± 0.025 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true100 10 HOMOGENEOUS 1000 avgt30.470 ± 0.031 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true100 50 HOMOGENEOUS 10 avgt32.356 ± 0.203 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true100 50 HOMOGENEOUS 100 avgt33.085 ± 0.226 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true100 50 HOMOGENEOUS 1000 avgt34.696 ± 0.068 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true500 5 HOMOGENEOUS 10 avgt30.631 ± 0.010 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true500 5 HOMOGENEOUS 100 avgt30.858 ± 0.082 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true500 5 HOMOGENEOUS 1000 avgt31.015 ± 0.073 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true500 10 HOMOGENEOUS 10 avgt31.367 ± 0.165 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true500 10 HOMOGENEOUS 100 avgt31.864 ± 0.094 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true500 10 HOMOGENEOUS 1000 avgt33.050 ± 0.294 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true500 50 HOMOGENEOUS 10 avgt3 11.298 ± 0.354 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true500 50 HOMOGENEOUS 100 avgt3 15.842 ± 2.087 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true500 50 HOMOGENEOUS 1000 avgt3 14.337 ± 0.292 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true 1000 5 HOMOGENEOUS 10 avgt31.516 ± 0.233 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true 1000 5 HOMOGENEOUS 100 avgt31.668 ± 0.199 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true 1000 5 HOMOGENEOUS 1000 avgt32.543 ± 0.209 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true 1000 10 HOMOGENEOUS 10 avgt32.801 ± 0.179 ms/op
Re: [PR] KAFKA-19909: Add rack awareness assignment to UniformHomogeneousAssignmentBuilder [kafka]
FrankYang0529 commented on PR #2: URL: https://github.com/apache/kafka/pull/2#issuecomment-3576097871 New commit improves performance a little, but overall it's still not good. I will try other ways tomorrow. PR: Uniform Homogeneous rack awareness vs non rack awareness ``` Benchmark (assignmentType) (assignorType) (isRackAware) (memberCount) (partitionsToMemberRatio) (subscriptionType) (topicCount) Mode Cnt Score Error Units ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true100 5 HOMOGENEOUS 10 avgt3 0.109 ± 0.004 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true100 5 HOMOGENEOUS 100 avgt3 0.147 ± 0.014 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true100 5 HOMOGENEOUS 1000 avgt3 0.189 ± 0.011 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true100 10 HOMOGENEOUS 10 avgt3 0.247 ± 0.020 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true100 10 HOMOGENEOUS 100 avgt3 0.358 ± 0.007 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true100 10 HOMOGENEOUS 1000 avgt3 0.717 ± 0.060 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true100 50 HOMOGENEOUS 10 avgt3 2.699 ± 0.152 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true100 50 HOMOGENEOUS 100 avgt3 2.406 ± 0.148 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true100 50 HOMOGENEOUS 1000 avgt3 4.689 ± 0.283 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true500 5 HOMOGENEOUS 10 avgt3 0.618 ± 0.051 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true500 5 HOMOGENEOUS 100 avgt3 0.831 ± 0.041 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true500 5 HOMOGENEOUS 1000 avgt3 1.007 ± 0.009 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true500 10 HOMOGENEOUS 10 avgt3 1.377 ± 0.138 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true500 10 HOMOGENEOUS 100 avgt3 1.806 ± 0.043 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true500 10 HOMOGENEOUS 1000 avgt3 3.020 ± 0.192 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true500 50 HOMOGENEOUS 10 avgt3 11.132 ± 0.252 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true500 50 HOMOGENEOUS 100 avgt3 15.998 ± 1.364 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true500 50 HOMOGENEOUS 1000 avgt3 14.395 ± 0.723 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true 1000 5 HOMOGENEOUS 10 avgt3 1.525 ± 0.012 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true 1000 5 HOMOGENEOUS 100 avgt3 1.700 ± 0.080 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true 1000 5 HOMOGENEOUS 1000
Re: [PR] KAFKA-19909: Add rack awareness assignment to UniformHomogeneousAssignmentBuilder [kafka]
FrankYang0529 commented on PR #2: URL: https://github.com/apache/kafka/pull/2#issuecomment-3571298783 I did some tests, but the performance is not good. I will try to improve it tomorrow. PR: Uniform Homogeneous rack awareness vs non rack awareness ``` Benchmark (assignmentType) (assignorType) (isRackAware) (memberCount) (partitionsToMemberRatio) (subscriptionType) (topicCount) Mode Cnt Score Error Units ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true100 5 HOMOGENEOUS 10 avgt3 0.165 ± 0.018 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true100 5 HOMOGENEOUS 100 avgt3 0.226 ± 0.014 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true100 5 HOMOGENEOUS 1000 avgt3 0.193 ± 0.002 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true100 10 HOMOGENEOUS 10 avgt3 0.356 ± 0.004 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true100 10 HOMOGENEOUS 100 avgt3 0.579 ± 0.065 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true100 10 HOMOGENEOUS 1000 avgt3 0.751 ± 0.027 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true100 50 HOMOGENEOUS 10 avgt3 3.419 ± 0.731 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true100 50 HOMOGENEOUS 100 avgt3 3.103 ± 0.156 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true100 50 HOMOGENEOUS 1000 avgt3 5.613 ± 0.173 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true500 5 HOMOGENEOUS 10 avgt3 0.918 ± 0.091 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true500 5 HOMOGENEOUS 100 avgt3 1.061 ± 0.041 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true500 5 HOMOGENEOUS 1000 avgt3 1.446 ± 0.122 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true500 10 HOMOGENEOUS 10 avgt3 2.013 ± 0.088 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true500 10 HOMOGENEOUS 100 avgt3 2.159 ± 0.014 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true500 10 HOMOGENEOUS 1000 avgt3 3.517 ± 0.118 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true500 50 HOMOGENEOUS 10 avgt3 15.173 ± 1.107 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true500 50 HOMOGENEOUS 100 avgt3 14.322 ± 0.220 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true500 50 HOMOGENEOUS 1000 avgt3 17.719 ± 0.493 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true 1000 5 HOMOGENEOUS 10 avgt3 1.901 ± 0.022 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true 1000 5 HOMOGENEOUS 100 avgt3 2.160 ± 0.064 ms/op ServerSideAssignorBenchmark.doAssignment FULL UNIFORM true 1000 5 HOMOGENEOUS 1000 avgt3 3.138 ±
