[
https://issues.apache.org/jira/browse/KAFKA-21244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18126749#comment-18126749
]
sanghyeok An commented on KAFKA-21244:
--------------------------------------
[~lianetm]
Thanks for the detailed explanation!
That makes sense. I agree that the issue should be addressed on the broker side
first, since a client-side fix alone could leave the assignment inconsistent
with the group coordinator's view. I'll keep an eye on the broker-side work and
Happy to help with any follow-up work if needed.
> Server-side consumer regex subscriptions do not honor
> `exclude.internal.topics`
> -------------------------------------------------------------------------------
>
> Key: KAFKA-21244
> URL: https://issues.apache.org/jira/browse/KAFKA-21244
> Project: Kafka
> Issue Type: Bug
> Components: group-coordinator
> Affects Versions: 4.0.0
> Reporter: sanghyeok An
> Assignee: sanghyeok An
> Priority: Minor
> Labels: consumer, group-coordinator
>
> *Description*
> When using {{SubscriptionPattern}} with {{{}*group.protocol=consumer*{}}},
> server-side regex subscriptions do not honor
> {{{}*exclude.internal.topics=true*{}}}. The same issue occurs when the
> configuration is omitted, since its default value is {{{}true{}}}.
> *Java {{Pattern}}* subscriptions correctly exclude internal topics under the
> same consumer protocol. This creates inconsistent configuration behavior
> between the two regex subscription APIs.
>
> *Conditions*
> * {{group.protocol=consumer}}
> * {{{}exclude.internal.topics=true{}}}, or the configuration is omitted
> * A regex that also matches internal topic names, such as
> {{SubscriptionPattern(".*")}}
> * An internal topic, such as {{{}__consumer_offsets{}}}, exists in the
> coordinator metadata
> * The consumer has the required permissions to describe and read the topic
> Example configuration and subscription:
> {code:java}
> props.put("group.protocol", "consumer");
> props.put("exclude.internal.topics", "true");
> consumer.subscribe(new SubscriptionPattern(".*")); {code}
>
> *Expected Behavior*
> Internal topics should be excluded from the subscription and assignment even
> when they match the regex.
> With {{{}exclude.internal.topics=false{}}}, matching internal topics should
> remain eligible for subscription. Explicit subscriptions by topic name should
> also preserve the existing contract: internal topics may be explicitly
> subscribed to even when the setting is {{{}true{}}}.
>
> *Observed Behavior and Cause*
> For Java {{Pattern}} subscriptions, the client evaluates the regex against
> metadata that excludes internal topics and sends the matching topic names in
> the heartbeat's {{SubscribedTopicNames}} field.
> For {{SubscriptionPattern}} subscriptions, the client sends the regex string
> in {{{}SubscribedTopicRegex{}}}. The current heartbeat schema has no field
> for conveying the internal-topic exclusion setting, and
> {{TopicRegexResolver}} evaluates the regex against all topic names without
> excluding internal topics.
> Additionally, {{ConsumerMetadata.retainTopic(...)}} retains metadata for
> topic IDs assigned through server-side regex subscriptions using the
> following condition:
> {code:java}
> return retainTopic(topicName, isInternal, nowMs)
> || subscription.isAssignedFromRe2j(topicId); {code}
> Consequently, metadata is retained for a topic ID assigned by the server even
> when the internal-topic exclusion check returns {{{}false{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)