dajac commented on code in PR #24108:
URL: https://github.com/apache/kafka/pull/24108#discussion_r4513055581
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##########
@@ -8789,27 +8789,39 @@ private CoordinatorResult<HeartbeatResponseData,
CoordinatorRecord> classicGroup
String groupId = request.groupId();
String memberId = request.memberId();
String instanceId = request.groupInstanceId();
+ List<CoordinatorRecord> records = new ArrayList<>();
ConsumerGroupMember member = validateConsumerGroupMember(group,
memberId, instanceId);
throwIfMemberDoesNotUseClassicProtocol(member);
throwIfGenerationIdUnmatched(memberId, member.memberEpoch(),
request.generationId());
scheduleConsumerGroupSessionTimeout(groupId, memberId,
member.classicProtocolSessionTimeout().get());
+ // Update the target assignment if the group epoch is larger than the
target assignment epoch.
+ // The delta between the existing and the new target assignment is
persisted to the partition.
+ UpdateTargetAssignmentResult<Assignment> updateTargetAssignmentResult
= maybeUpdateTargetAssignment(
+ group,
+ group.groupEpoch(),
+ member,
+ member,
+ group.subscriptionType(),
+ records
+ );
+
Errors error = Errors.NONE;
// The member should rejoin if any of the following conditions is met.
// 1) The target assignment epoch is bumped so the member needs to
rejoin to catch up.
// 2) The member needs to revoke some partitions and rejoin to
reconcile with the new epoch.
// 3) The member's partitions pending assignment are free, so it can
rejoin to get the complete assignment.
- if (member.memberEpoch() < group.assignmentEpoch() ||
+ if (member.memberEpoch() <
updateTargetAssignmentResult.targetAssignmentEpoch() ||
Review Comment:
nit: The Javadoc of this method still says that the member epoch is compared
with the group epoch. Could we update it to say the target assignment epoch,
and mention that the heartbeat may update the target assignment?
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##########
@@ -8789,27 +8789,39 @@ private CoordinatorResult<HeartbeatResponseData,
CoordinatorRecord> classicGroup
String groupId = request.groupId();
String memberId = request.memberId();
String instanceId = request.groupInstanceId();
+ List<CoordinatorRecord> records = new ArrayList<>();
ConsumerGroupMember member = validateConsumerGroupMember(group,
memberId, instanceId);
throwIfMemberDoesNotUseClassicProtocol(member);
throwIfGenerationIdUnmatched(memberId, member.memberEpoch(),
request.generationId());
scheduleConsumerGroupSessionTimeout(groupId, memberId,
member.classicProtocolSessionTimeout().get());
+ // Update the target assignment if the group epoch is larger than the
target assignment epoch.
+ // The delta between the existing and the new target assignment is
persisted to the partition.
+ UpdateTargetAssignmentResult<Assignment> updateTargetAssignmentResult
= maybeUpdateTargetAssignment(
Review Comment:
Should we also refresh the metadata here when
`group.hasMetadataExpired(currentTimeMs)` is true, like
`consumerGroupHeartbeat` and `classicGroupJoinToConsumerGroup` do? When all the
members use the classic protocol, nothing bumps the group epoch when a
subscribed topic gets new partitions or is created. The classic members don't
detect it either because none of them is the leader. I tried it with a
partition increase of `foo` in a group with a single classic member: the
heartbeats keep returning `NONE` and the group epoch does not change. Calling
`updateSubscriptionMetadata(group, false, member, member, records)` before
`maybeUpdateTargetAssignment` fixes it. This was already the case before
KAFKA-20272, so a separate Jira and PR is fine with me.
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java:
##########
@@ -17847,6 +17848,138 @@ public void
testClassicGroupHeartbeatToConsumerGroupRebalanceInProgress() throws
});
}
+ @Test
+ public void
testClassicGroupHeartbeatToConsumerGroupUpdatesTargetAssignment() {
+ String groupId = "fooup";
+ String memberId1 = Uuid.randomUuid().toString();
+ String memberId2 = Uuid.randomUuid().toString();
+
+ Uuid fooTopicId = Uuid.randomUuid();
+ String fooTopicName = "foo";
+
+ MockPartitionAssignor assignor = new MockPartitionAssignor("range");
+
+ List<ConsumerGroupMemberMetadataValue.ClassicProtocol> protocols =
List.of(
+ new ConsumerGroupMemberMetadataValue.ClassicProtocol()
+ .setName("range")
+
.setMetadata(Utils.toArray(ConsumerProtocol.serializeSubscription(new
ConsumerPartitionAssignor.Subscription(
+ List.of(fooTopicName),
+ null,
+ List.of(
+ new TopicPartition(fooTopicName, 0),
+ new TopicPartition(fooTopicName, 1),
+ new TopicPartition(fooTopicName, 2)
+ )
+ ))))
+ );
+
+ ConsumerGroupMember member1 = new
ConsumerGroupMember.Builder(memberId1)
+ .setState(MemberState.STABLE)
+ .setMemberEpoch(10)
+ .setPreviousMemberEpoch(9)
+ .setClientId(DEFAULT_CLIENT_ID)
+ .setClientHost(DEFAULT_CLIENT_ADDRESS.toString())
+ .setSubscribedTopicNames(List.of(fooTopicName))
+ .setServerAssignorName("range")
+ .setRebalanceTimeoutMs(45000)
+ .setClassicMemberMetadata(
+ new ConsumerGroupMemberMetadataValue.ClassicMemberMetadata()
+ .setSessionTimeoutMs(5000)
+ .setSupportedProtocols(protocols)
+ )
+ .setAssignedPartitions(toAssignmentWithEpochs(mkAssignment(
+ mkTopicAssignment(fooTopicId, 0, 1, 2)), 10))
+ .build();
+ ConsumerGroupMember member2 = new
ConsumerGroupMember.Builder(memberId2)
+ .setState(MemberState.STABLE)
+ .setMemberEpoch(10)
+ .setPreviousMemberEpoch(9)
+ .setClientId(DEFAULT_CLIENT_ID)
+ .setClientHost(DEFAULT_CLIENT_ADDRESS.toString())
+ .setSubscribedTopicNames(List.of(fooTopicName))
+ .setServerAssignorName("range")
+ .setRebalanceTimeoutMs(45000)
+ .setAssignedPartitions(toAssignmentWithEpochs(mkAssignment(
+ mkTopicAssignment(fooTopicId, 3, 4, 5)), 10))
+ .build();
+
+ CoordinatorMetadataImage metadataImage = new MetadataImageBuilder()
+ .addTopic(fooTopicId, fooTopicName, 6)
+ .addRacks()
+ .buildCoordinatorMetadataImage();
+
+ // Consumer group with two members.
+ // Member 1 uses the classic protocol and member 2 uses the consumer
protocol.
+ // The migration policy does not allow the group to be downgraded.
+ GroupMetadataManagerTestContext context = new
GroupMetadataManagerTestContext.Builder()
+
.withConfig(GroupCoordinatorConfig.CONSUMER_GROUP_MIGRATION_POLICY_CONFIG,
ConsumerGroupMigrationPolicy.UPGRADE.toString())
+
.withConfig(GroupCoordinatorConfig.CONSUMER_GROUP_ASSIGNORS_CONFIG,
List.of(assignor))
+ .withMetadataImage(metadataImage)
+ .withConsumerGroup(new ConsumerGroupBuilder(groupId, 10)
+ .withMember(member1)
+ .withMember(member2)
+ .withAssignment(memberId1, mkAssignment(
+ mkTopicAssignment(fooTopicId, 0, 1, 2)))
+ .withAssignment(memberId2, mkAssignment(
+ mkTopicAssignment(fooTopicId, 3, 4, 5)))
+ .withAssignmentEpoch(10)
+ .withMetadataHash(computeGroupHash(Map.of(
+ fooTopicName, computeTopicHash(fooTopicName, metadataImage)
+ ))))
+ .build();
+
+ // Member 2 leaves the group, bumping the group epoch. The new
assignment is not computed
+ // yet and no members using the consumer protocol remain to trigger it.
+ CoordinatorResult<ConsumerGroupHeartbeatResponseData,
CoordinatorRecord> leaveResult = context.consumerGroupHeartbeat(
+ new ConsumerGroupHeartbeatRequestData()
+ .setGroupId(groupId)
+ .setMemberId(memberId2)
+ .setMemberEpoch(LEAVE_GROUP_MEMBER_EPOCH));
+
+ assertResponseEquals(
+ new ConsumerGroupHeartbeatResponseData()
+ .setMemberId(memberId2)
+ .setMemberEpoch(LEAVE_GROUP_MEMBER_EPOCH)
+ .setHeartbeatIntervalMs(0),
+ leaveResult.response()
+ );
+
+ assertRecordsEquals(
+ List.of(
+
GroupCoordinatorRecordHelpers.newConsumerGroupCurrentAssignmentTombstoneRecord(groupId,
memberId2),
+
GroupCoordinatorRecordHelpers.newConsumerGroupTargetAssignmentTombstoneRecord(groupId,
memberId2),
+
GroupCoordinatorRecordHelpers.newConsumerGroupMemberSubscriptionTombstoneRecord(groupId,
memberId2),
+
GroupCoordinatorRecordHelpers.newConsumerGroupEpochRecord(groupId, 11,
computeGroupHash(Map.of(
+ fooTopicName, computeTopicHash(fooTopicName, metadataImage)
+ )))
+ ),
+ leaveResult.records()
+ );
+
+ // Member 1 heartbeats, triggers a new assignment and is told to
rebalance.
+ assignor.prepareGroupAssignment(new GroupAssignment(Map.of(
+ memberId1, new MemberAssignmentImpl(mkAssignment(
+ mkTopicAssignment(fooTopicId, 0, 1, 2, 3, 4, 5)
+ ))
+ )));
+ CoordinatorResult<HeartbeatResponseData, CoordinatorRecord> result =
context.sendClassicGroupHeartbeat(
Review Comment:
nit: `sendClassicGroupHeartbeat` does not replay the returned records,
unlike `consumerGroupHeartbeat`. Now that the heartbeat can write records,
should the helper replay them? Otherwise a second heartbeat in a test computes
the target assignment again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]