dajac commented on code in PR #24113:
URL: https://github.com/apache/kafka/pull/24113#discussion_r4513043602
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##########
@@ -3915,11 +3915,12 @@ private CoordinatorResult<Void, CoordinatorRecord>
handleRegularExpressionsResul
if (group.numSubscribedMembers(regex) == 0) continue;
ResolvedRegularExpression newResolvedRegularExpression =
entry.getValue();
- ResolvedRegularExpression oldResolvedRegularExpression = group
- .resolvedRegularExpression(regex)
- .orElse(ResolvedRegularExpression.EMPTY);
+ Optional<ResolvedRegularExpression> previousResolution =
group.resolvedRegularExpression(regex);
+ ResolvedRegularExpression oldResolvedRegularExpression =
previousResolution.orElse(ResolvedRegularExpression.EMPTY);
- if
(!oldResolvedRegularExpression.topics().equals(newResolvedRegularExpression.topics()))
{
+ // The first resolution must trigger assignment computation
even when no topics match.
Review Comment:
Could we keep the code as it is on trunk and handle the first resolution in
its own branch? The change would then be limited to the case that it fixes.
Something like:
```java
ResolvedRegularExpression newResolvedRegularExpression =
entry.getValue();
ResolvedRegularExpression oldResolvedRegularExpression =
group
.resolvedRegularExpression(regex)
.orElse(ResolvedRegularExpression.EMPTY);
if
(!oldResolvedRegularExpression.topics().equals(newResolvedRegularExpression.topics()))
{
...
} else if (group.resolvedRegularExpression(regex).isEmpty())
{
// The heartbeat does not bump the group epoch when a
member subscribes to a
// regex that is not resolved yet, so the first
resolution bumps it even when
// the regex matches no topic.
bumpGroupEpoch = true;
}
```
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java:
##########
@@ -27104,8 +27104,10 @@ memberId2, new MemberAssignmentImpl(mkAssignment(
);
}
- @Test
- public void testConsumerGroupMemberJoinsWithNewRegex() {
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ public void testConsumerGroupMemberJoinsWithNewRegex(boolean matchesTopic)
{
Review Comment:
This covers a new member joining with a regex that matches no topic, which
has no partitions to give back. Could we also add a test for the case from the
Jira? Two members subscribe to `foo*`, one switches to a regex that matches no
topic, the resolution bumps the group epoch and the other member gets the
partition at its next heartbeat. Today only the integration test covers it.
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java:
##########
@@ -27207,6 +27209,55 @@ fooTopicName, computeTopicHash(fooTopicName,
metadataImage)
);
}
+ @Test
+ public void testConsumerGroupRefreshOfNonMatchingRegexDoesNotBumpEpoch() {
+ String groupId = "regex-refresh-group";
+ String memberId = Uuid.randomUuid().toString();
+ String regex = "nothing-matches-this.*";
+ long groupMetadataHash = computeGroupHash(Map.of());
+
+ GroupMetadataManagerTestContext context = new
GroupMetadataManagerTestContext.Builder()
+
.withConfig(GroupCoordinatorConfig.CONSUMER_GROUP_REGEX_REFRESH_INTERVAL_MS_CONFIG,
10000)
+ .withMetadataImage(new MetadataImageBuilder()
+ .addTopic(Uuid.randomUuid(), "foo", 2)
+ .buildCoordinatorMetadataImage(12345L))
+ .withConsumerGroup(new ConsumerGroupBuilder(groupId, 10)
+ .withMember(new ConsumerGroupMember.Builder(memberId)
+ .setState(MemberState.STABLE)
+ .setMemberEpoch(10)
+ .setPreviousMemberEpoch(10)
+ .setClientId(DEFAULT_CLIENT_ID)
+ .setClientHost(DEFAULT_CLIENT_ADDRESS.toString())
+ .setRebalanceTimeoutMs(5000)
+ .setSubscribedTopicRegex(regex)
+ .build())
+ .withResolvedRegularExpression(regex, new
ResolvedRegularExpression(Set.of(), 12345L, 0L))
+ .withAssignment(memberId, Map.of())
+ .withAssignmentEpoch(10)
+ .withMetadataHash(groupMetadataHash))
+ .build();
+
+ // Refresh an already resolved empty result after the batching
interval.
+ context.sleep(11000L);
+ context.consumerGroupHeartbeat(new ConsumerGroupHeartbeatRequestData()
+ .setGroupId(groupId)
+ .setMemberId(memberId)
+ .setMemberEpoch(10));
+
+ assertTrue(context.executor.isScheduled(groupId + "-regex"));
+ List<MockCoordinatorExecutor.ExecutorResult<CoordinatorRecord>> tasks
= context.processTasks();
+ assertEquals(1, tasks.size());
+ assertRecordsEquals(
+
List.of(GroupCoordinatorRecordHelpers.newConsumerGroupRegularExpressionRecord(
+ groupId,
+ regex,
+ new ResolvedRegularExpression(Set.of(), 12345L,
context.time.milliseconds())
+ )),
+ tasks.get(0).result().records()
+ );
+ assertEquals(10,
context.groupMetadataManager.consumerGroup(groupId).groupEpoch());
Review Comment:
nit: Could we assert `tasks` in one assertion, as in
`testConsumerGroupMemberJoinsWithNewRegex`? `processTasks` replays the records,
so the missing epoch record already shows that the group epoch did not change.
```suggestion
List<MockCoordinatorExecutor.ExecutorResult<CoordinatorRecord>>
tasks = context.processTasks();
assertEquals(
List.of(
new MockCoordinatorExecutor.ExecutorResult<>(
groupId + "-regex",
new CoordinatorResult<>(List.of(
// The resolution is persisted but the group epoch
is not bumped.
GroupCoordinatorRecordHelpers.newConsumerGroupRegularExpressionRecord(
groupId,
regex,
new ResolvedRegularExpression(Set.of(), 12345L,
context.time.milliseconds())
)
))
)
),
tasks
);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]