[ 
https://issues.apache.org/jira/browse/KAFKA-21244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18126756#comment-18126756
 ] 

David Jacot commented on KAFKA-21244:
-------------------------------------

[~chickenchickenlove] As a first step, we could filter out the internal topics 
when the regex is resolved on the server side. I think that it is a better 
default behavior than the current implementation. We could backport this fix to 
previous versions too. Then, we can discuss whether we want to bring the config 
back when we discuss the design of KAFKA-21254. Would you be interested by 
doing this first step?

> Server-side consumer regex subscriptions do not honor 
> `exclude.internal.topics`
> -------------------------------------------------------------------------------
>
>                 Key: KAFKA-21244
>                 URL: https://issues.apache.org/jira/browse/KAFKA-21244
>             Project: Kafka
>          Issue Type: Bug
>          Components: group-coordinator
>    Affects Versions: 4.0.0
>            Reporter: sanghyeok An
>            Assignee: sanghyeok An
>            Priority: Minor
>              Labels: consumer, group-coordinator
>
> *Description*
> When using {{SubscriptionPattern}} with {{{}*group.protocol=consumer*{}}}, 
> server-side regex subscriptions do not honor 
> {{{}*exclude.internal.topics=true*{}}}. The same issue occurs when the 
> configuration is omitted, since its default value is {{{}true{}}}.
> *Java {{Pattern}}* subscriptions correctly exclude internal topics under the 
> same consumer protocol. This creates inconsistent configuration behavior 
> between the two regex subscription APIs.
>  
> *Conditions*
>  * {{group.protocol=consumer}}
>  * {{{}exclude.internal.topics=true{}}}, or the configuration is omitted
>  * A regex that also matches internal topic names, such as 
> {{SubscriptionPattern(".*")}}
>  * An internal topic, such as {{{}__consumer_offsets{}}}, exists in the 
> coordinator metadata
>  * The consumer has the required permissions to describe and read the topic
> Example configuration and subscription:
> {code:java}
> props.put("group.protocol", "consumer");
> props.put("exclude.internal.topics", "true");
> consumer.subscribe(new SubscriptionPattern(".*")); {code}
>  
> *Expected Behavior*
> Internal topics should be excluded from the subscription and assignment even 
> when they match the regex.
> With {{{}exclude.internal.topics=false{}}}, matching internal topics should 
> remain eligible for subscription. Explicit subscriptions by topic name should 
> also preserve the existing contract: internal topics may be explicitly 
> subscribed to even when the setting is {{{}true{}}}.
>  
> *Observed Behavior and Cause*
> For Java {{Pattern}} subscriptions, the client evaluates the regex against 
> metadata that excludes internal topics and sends the matching topic names in 
> the heartbeat's {{SubscribedTopicNames}} field.
> For {{SubscriptionPattern}} subscriptions, the client sends the regex string 
> in {{{}SubscribedTopicRegex{}}}. The current heartbeat schema has no field 
> for conveying the internal-topic exclusion setting, and 
> {{TopicRegexResolver}} evaluates the regex against all topic names without 
> excluding internal topics.
> Additionally, {{ConsumerMetadata.retainTopic(...)}} retains metadata for 
> topic IDs assigned through server-side regex subscriptions using the 
> following condition:
> {code:java}
> return retainTopic(topicName, isInternal, nowMs)
>     || subscription.isAssignedFromRe2j(topicId); {code}
> Consequently, metadata is retained for a topic ID assigned by the server even 
> when the internal-topic exclusion check returns {{{}false{}}}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to