[
https://issues.apache.org/jira/browse/KAFKA-21244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18125022#comment-18125022
]
sanghyeok An commented on KAFKA-21244:
--------------------------------------
[~lianetm] Hi!
My understanding is that *{{SubscriptionPattern}}* was introduced as part of
KIP-848.
Could you please clarify whether it is intended to honor the client-side
*{{exclude.internal.topics}}* configuration?
If it is, it seems that the client would need to include this setting in the
API request sent to the server.
I would appreciate your thoughts on this! :)
> Server-side consumer regex subscriptions do not honor
> `exclude.internal.topics`
> -------------------------------------------------------------------------------
>
> Key: KAFKA-21244
> URL: https://issues.apache.org/jira/browse/KAFKA-21244
> Project: Kafka
> Issue Type: Bug
> Reporter: sanghyeok An
> Assignee: sanghyeok An
> Priority: Minor
> Labels: consumer, group-coordinator
>
> *Description*
> When using {{SubscriptionPattern}} with {{{}*group.protocol=consumer*{}}},
> server-side regex subscriptions do not honor
> {{{}*exclude.internal.topics=true*{}}}. The same issue occurs when the
> configuration is omitted, since its default value is {{{}true{}}}.
> *Java {{Pattern}}* subscriptions correctly exclude internal topics under the
> same consumer protocol. This creates inconsistent configuration behavior
> between the two regex subscription APIs.
>
> *Conditions*
> * {{group.protocol=consumer}}
> * {{{}exclude.internal.topics=true{}}}, or the configuration is omitted
> * A regex that also matches internal topic names, such as
> {{SubscriptionPattern(".*")}}
> * An internal topic, such as {{{}__consumer_offsets{}}}, exists in the
> coordinator metadata
> * The consumer has the required permissions to describe and read the topic
> Example configuration and subscription:
> {code:java}
> props.put("group.protocol", "consumer");
> props.put("exclude.internal.topics", "true");
> consumer.subscribe(new SubscriptionPattern(".*")); {code}
>
> *Expected Behavior*
> Internal topics should be excluded from the subscription and assignment even
> when they match the regex.
> With {{{}exclude.internal.topics=false{}}}, matching internal topics should
> remain eligible for subscription. Explicit subscriptions by topic name should
> also preserve the existing contract: internal topics may be explicitly
> subscribed to even when the setting is {{{}true{}}}.
>
> *Observed Behavior and Cause*
> For Java {{Pattern}} subscriptions, the client evaluates the regex against
> metadata that excludes internal topics and sends the matching topic names in
> the heartbeat's {{SubscribedTopicNames}} field.
> For {{SubscriptionPattern}} subscriptions, the client sends the regex string
> in {{{}SubscribedTopicRegex{}}}. The current heartbeat schema has no field
> for conveying the internal-topic exclusion setting, and
> {{TopicRegexResolver}} evaluates the regex against all topic names without
> excluding internal topics.
> Additionally, {{ConsumerMetadata.retainTopic(...)}} retains metadata for
> topic IDs assigned through server-side regex subscriptions using the
> following condition:
> {code:java}
> return retainTopic(topicName, isInternal, nowMs)
> || subscription.isAssignedFromRe2j(topicId); {code}
> Consequently, metadata is retained for a topic ID assigned by the server even
> when the internal-topic exclusion check returns {{{}false{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)