lucasbru commented on code in PR #23165:
URL: https://github.com/apache/kafka/pull/23165#discussion_r4152867643
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##########
@@ -2268,9 +2266,10 @@ private CoordinatorResult<StreamsGroupHeartbeatResult,
CoordinatorRecord> stream
assignmentUpdate = AssignmentUpdate.RECOMPUTE;
}
- // Check if assignment configurations have changed
- Map<String, String> currentAssignmentConfigs =
streamsGroupAssignmentConfigs(groupId);
- Map<String, String> storedAssignmentConfigs =
group.lastAssignmentConfigs();
+ // Check if assignment configurations have changed. A group with no
recorded configs holds the defaults,
+ // so upgrading the broker does not rebalance every group that has
nothing configured.
+ AssignmentConfigsImpl currentAssignmentConfigs =
streamsGroupAssignmentConfigs(groupId);
+ AssignmentConfigsImpl storedAssignmentConfigs =
group.lastAssignmentConfigs();
if (assignmentUpdate == AssignmentUpdate.NONE &&
!currentAssignmentConfigs.equals(storedAssignmentConfigs)) {
Review Comment:
DEFAULT is hardcoded to the compiled default (0 / ""), but
`streamsGroupAssignmentConfigs()` compares against the live broker config
value, which can differ if `group.streams.num.standby.replicas` (or the
rack-tags config) is set cluster-wide. Wouldn't that still cause a spurious
rebalance for a group with no recorded config in that case - just gated on
"broker default customized" instead of "never recorded"?
##########
group-coordinator/group-coordinator-api/src/main/java/org/apache/kafka/coordinator/group/api/streams/assignor/GroupAssignment.java:
##########
@@ -36,11 +35,11 @@ public class GroupAssignment {
public GroupAssignment(
Map<String, MemberAssignment> members
) {
- this.members = Objects.requireNonNull(members);
+ this.members = Map.copyOf(members);
Review Comment:
Why this change? Map.copyOf() will NPE if any value in the map is null, not
just the map reference itself - a bit stricter than before for custom
TaskAssignor implementations. Is that intentional? Also, every caller here
already builds a fresh, private map just for this constructor, so the copy is
doing a bit of extra work on top of that.
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##########
@@ -6388,17 +6386,16 @@ public void replay(
streamsGroup.setStoredDescriptionTopologyEpoch(value.storedDescriptionTopologyEpoch());
streamsGroup.setFailedDescriptionTopologyEpoch(value.failedDescriptionTopologyEpoch());
- if (value.lastAssignmentConfigs() != null) {
- streamsGroup.setLastAssignmentConfigs(
- value.lastAssignmentConfigs().stream()
- .collect(Collectors.toMap(
-
StreamsGroupMetadataValue.LastAssignmentConfig::key,
-
StreamsGroupMetadataValue.LastAssignmentConfig::value
- ))
- );
- } else {
- streamsGroup.setLastAssignmentConfigs(Map.of());
- }
+ // A record without configs (written before they were persisted)
yields the defaults, so the next
+ // heartbeat only rebalances the group if any effective config
differs from its default.
+ Map<String, String> lastAssignmentConfigs =
value.lastAssignmentConfigs() == null
+ ? Map.of()
+ : value.lastAssignmentConfigs().stream()
+ .collect(Collectors.toMap(
+ StreamsGroupMetadataValue.LastAssignmentConfig::key,
+ StreamsGroupMetadataValue.LastAssignmentConfig::value
+ ));
+
streamsGroup.setLastAssignmentConfigs(AssignmentConfigsImpl.fromMap(lastAssignmentConfigs));
Review Comment:
This builds a Map just to immediately hand it to fromMap() - could probably
construct the AssignmentConfigsImpl directly from the LastAssignmentConfig list
here instead. Maybe worth the optimization, since replay will be called on
every GC failover for all groups.
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/streams/assignor/AssignmentConfigsImpl.java:
##########
@@ -52,20 +53,18 @@ public record AssignmentConfigsImpl(
}
/**
- * Converts the raw assignment configs computed for the group into the
typed configs passed to the assignor.
+ * Converts the raw assignment configs recorded for the group into the
typed configs passed to the assignor.
*/
public static AssignmentConfigsImpl fromMap(Map<String, String> configs) {
- // The map is empty when it was replayed from a group metadata record
written before the last assignment
- // configs were persisted.
- if (configs.isEmpty()) {
- return DEFAULT;
- }
- // The rack-aware assignment tags are only recorded when any are
configured, so an absent value means the
- // configuration is at its default.
- String rackAwareAssignmentTags = configs.getOrDefault(
- RACK_AWARE_ASSIGNMENT_TAGS_CONFIG,
GroupCoordinatorConfig.STREAMS_GROUP_RACK_AWARE_ASSIGNMENT_TAGS_DEFAULT);
+ // Configs are only recorded when set, so every absent key means the
configuration is at its default.
+ String numStandbyReplicasConfig =
configs.get(NUM_STANDBY_REPLICAS_CONFIG);
+ int numStandbyReplicas = numStandbyReplicasConfig != null
+ ? Integer.parseInt(numStandbyReplicasConfig)
+ :
GroupCoordinatorConfig.STREAMS_GROUP_NUM_STANDBY_REPLICAS_DEFAULT;
Review Comment:
Could this just reference DEFAULT.numStandbyReplicas() instead of the raw
constant? Minor, but avoids having two places that need to agree on the default
value.
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/streams/assignor/AssignmentConfigsImpl.java:
##########
@@ -52,20 +53,18 @@ public record AssignmentConfigsImpl(
}
/**
- * Converts the raw assignment configs computed for the group into the
typed configs passed to the assignor.
+ * Converts the raw assignment configs recorded for the group into the
typed configs passed to the assignor.
*/
public static AssignmentConfigsImpl fromMap(Map<String, String> configs) {
- // The map is empty when it was replayed from a group metadata record
written before the last assignment
- // configs were persisted.
- if (configs.isEmpty()) {
- return DEFAULT;
- }
- // The rack-aware assignment tags are only recorded when any are
configured, so an absent value means the
- // configuration is at its default.
- String rackAwareAssignmentTags = configs.getOrDefault(
- RACK_AWARE_ASSIGNMENT_TAGS_CONFIG,
GroupCoordinatorConfig.STREAMS_GROUP_RACK_AWARE_ASSIGNMENT_TAGS_DEFAULT);
+ // Configs are only recorded when set, so every absent key means the
configuration is at its default.
Review Comment:
This comment says every absent key means default, but toMap always writes
num.standby.replicas even when it's at its default.
Which one is it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]