[ 
https://issues.apache.org/jira/browse/KAFKA-21244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18125022#comment-18125022
 ] 

sanghyeok An commented on KAFKA-21244:
--------------------------------------

[~lianetm] Hi!

My understanding is that *{{SubscriptionPattern}}* was introduced as part of 
KIP-848.
Could you please clarify whether it is intended to honor the client-side 
*{{exclude.internal.topics}}* configuration?

If it is, it seems that the client would need to include this setting in the 
API request sent to the server.
I would appreciate your thoughts on this! :)

> Server-side consumer regex subscriptions do not honor 
> `exclude.internal.topics`
> -------------------------------------------------------------------------------
>
>                 Key: KAFKA-21244
>                 URL: https://issues.apache.org/jira/browse/KAFKA-21244
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: sanghyeok An
>            Assignee: sanghyeok An
>            Priority: Minor
>              Labels: consumer, group-coordinator
>
> *Description*
> When using {{SubscriptionPattern}} with {{{}*group.protocol=consumer*{}}}, 
> server-side regex subscriptions do not honor 
> {{{}*exclude.internal.topics=true*{}}}. The same issue occurs when the 
> configuration is omitted, since its default value is {{{}true{}}}.
> *Java {{Pattern}}* subscriptions correctly exclude internal topics under the 
> same consumer protocol. This creates inconsistent configuration behavior 
> between the two regex subscription APIs.
>  
> *Conditions*
>  * {{group.protocol=consumer}}
>  * {{{}exclude.internal.topics=true{}}}, or the configuration is omitted
>  * A regex that also matches internal topic names, such as 
> {{SubscriptionPattern(".*")}}
>  * An internal topic, such as {{{}__consumer_offsets{}}}, exists in the 
> coordinator metadata
>  * The consumer has the required permissions to describe and read the topic
> Example configuration and subscription:
> {code:java}
> props.put("group.protocol", "consumer");
> props.put("exclude.internal.topics", "true");
> consumer.subscribe(new SubscriptionPattern(".*")); {code}
>  
> *Expected Behavior*
> Internal topics should be excluded from the subscription and assignment even 
> when they match the regex.
> With {{{}exclude.internal.topics=false{}}}, matching internal topics should 
> remain eligible for subscription. Explicit subscriptions by topic name should 
> also preserve the existing contract: internal topics may be explicitly 
> subscribed to even when the setting is {{{}true{}}}.
>  
> *Observed Behavior and Cause*
> For Java {{Pattern}} subscriptions, the client evaluates the regex against 
> metadata that excludes internal topics and sends the matching topic names in 
> the heartbeat's {{SubscribedTopicNames}} field.
> For {{SubscriptionPattern}} subscriptions, the client sends the regex string 
> in {{{}SubscribedTopicRegex{}}}. The current heartbeat schema has no field 
> for conveying the internal-topic exclusion setting, and 
> {{TopicRegexResolver}} evaluates the regex against all topic names without 
> excluding internal topics.
> Additionally, {{ConsumerMetadata.retainTopic(...)}} retains metadata for 
> topic IDs assigned through server-side regex subscriptions using the 
> following condition:
> {code:java}
> return retainTopic(topicName, isInternal, nowMs)
>     || subscription.isAssignedFromRe2j(topicId); {code}
> Consequently, metadata is retained for a topic ID assigned by the server even 
> when the internal-topic exclusion check returns {{{}false{}}}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to