[ 
https://issues.apache.org/jira/browse/KAFKA-21244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18125564#comment-18125564
 ] 

Lianet Magrans commented on KAFKA-21244:
----------------------------------------

Hey [~chickenchickenlove] , interesting case, thanks for reporting!

Agree with the gap, on the client we're ignoring that config when the 
assignment comes from re2j (not intentional, just a miss), but this is bigger 
than that, the assignment path for the regex needs to consider it and fix at 
the source (with client side drop only the topic would remain assigned to this 
member from the GC pov, wrong).

There is some ongoing work around the broker-side regex that may include this 
(they linked their jira). I would say we follow that broker-side work first, 
and then we can assess if there is anything else we may need to do. Makes sense?

Thanks, good catch!

> Server-side consumer regex subscriptions do not honor 
> `exclude.internal.topics`
> -------------------------------------------------------------------------------
>
>                 Key: KAFKA-21244
>                 URL: https://issues.apache.org/jira/browse/KAFKA-21244
>             Project: Kafka
>          Issue Type: Bug
>          Components: group-coordinator
>    Affects Versions: 4.0.0
>            Reporter: sanghyeok An
>            Assignee: sanghyeok An
>            Priority: Minor
>              Labels: consumer, group-coordinator
>
> *Description*
> When using {{SubscriptionPattern}} with {{{}*group.protocol=consumer*{}}}, 
> server-side regex subscriptions do not honor 
> {{{}*exclude.internal.topics=true*{}}}. The same issue occurs when the 
> configuration is omitted, since its default value is {{{}true{}}}.
> *Java {{Pattern}}* subscriptions correctly exclude internal topics under the 
> same consumer protocol. This creates inconsistent configuration behavior 
> between the two regex subscription APIs.
>  
> *Conditions*
>  * {{group.protocol=consumer}}
>  * {{{}exclude.internal.topics=true{}}}, or the configuration is omitted
>  * A regex that also matches internal topic names, such as 
> {{SubscriptionPattern(".*")}}
>  * An internal topic, such as {{{}__consumer_offsets{}}}, exists in the 
> coordinator metadata
>  * The consumer has the required permissions to describe and read the topic
> Example configuration and subscription:
> {code:java}
> props.put("group.protocol", "consumer");
> props.put("exclude.internal.topics", "true");
> consumer.subscribe(new SubscriptionPattern(".*")); {code}
>  
> *Expected Behavior*
> Internal topics should be excluded from the subscription and assignment even 
> when they match the regex.
> With {{{}exclude.internal.topics=false{}}}, matching internal topics should 
> remain eligible for subscription. Explicit subscriptions by topic name should 
> also preserve the existing contract: internal topics may be explicitly 
> subscribed to even when the setting is {{{}true{}}}.
>  
> *Observed Behavior and Cause*
> For Java {{Pattern}} subscriptions, the client evaluates the regex against 
> metadata that excludes internal topics and sends the matching topic names in 
> the heartbeat's {{SubscribedTopicNames}} field.
> For {{SubscriptionPattern}} subscriptions, the client sends the regex string 
> in {{{}SubscribedTopicRegex{}}}. The current heartbeat schema has no field 
> for conveying the internal-topic exclusion setting, and 
> {{TopicRegexResolver}} evaluates the regex against all topic names without 
> excluding internal topics.
> Additionally, {{ConsumerMetadata.retainTopic(...)}} retains metadata for 
> topic IDs assigned through server-side regex subscriptions using the 
> following condition:
> {code:java}
> return retainTopic(topicName, isInternal, nowMs)
>     || subscription.isAssignedFromRe2j(topicId); {code}
> Consequently, metadata is retained for a topic ID assigned by the server even 
> when the internal-topic exclusion check returns {{{}false{}}}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to