[ 
https://issues.apache.org/jira/browse/KAFKA-21244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18127023#comment-18127023
 ] 

sanghyeok An commented on KAFKA-21244:
--------------------------------------

[~dajac] 

Yes, I'd be happy to work on this!

Considering Ismael's feedback, should we also consider introducing a dynamic 
broker configuration? For example, we could add a server-side config such as 
{{{}exclude.internal.topics{}}}, defaulting to {{{}true{}}}, and allow users to 
set it to {{false}} via the Dynamic Config API. (However, this might require us 
to consider another deprecation sequence when we refactor this as part of 
KAFKA-21254.)

If that adds too much complexity, I could start with a simpler implementation 
that always filters out internal topics without introducing a new configuration.

> Server-side consumer regex subscriptions do not honor 
> `exclude.internal.topics`
> -------------------------------------------------------------------------------
>
>                 Key: KAFKA-21244
>                 URL: https://issues.apache.org/jira/browse/KAFKA-21244
>             Project: Kafka
>          Issue Type: Bug
>          Components: group-coordinator
>    Affects Versions: 4.0.0
>            Reporter: sanghyeok An
>            Assignee: sanghyeok An
>            Priority: Minor
>              Labels: consumer, group-coordinator
>
> *Description*
> When using {{SubscriptionPattern}} with {{{}*group.protocol=consumer*{}}}, 
> server-side regex subscriptions do not honor 
> {{{}*exclude.internal.topics=true*{}}}. The same issue occurs when the 
> configuration is omitted, since its default value is {{{}true{}}}.
> *Java {{Pattern}}* subscriptions correctly exclude internal topics under the 
> same consumer protocol. This creates inconsistent configuration behavior 
> between the two regex subscription APIs.
>  
> *Conditions*
>  * {{group.protocol=consumer}}
>  * {{{}exclude.internal.topics=true{}}}, or the configuration is omitted
>  * A regex that also matches internal topic names, such as 
> {{SubscriptionPattern(".*")}}
>  * An internal topic, such as {{{}__consumer_offsets{}}}, exists in the 
> coordinator metadata
>  * The consumer has the required permissions to describe and read the topic
> Example configuration and subscription:
> {code:java}
> props.put("group.protocol", "consumer");
> props.put("exclude.internal.topics", "true");
> consumer.subscribe(new SubscriptionPattern(".*")); {code}
>  
> *Expected Behavior*
> Internal topics should be excluded from the subscription and assignment even 
> when they match the regex.
> With {{{}exclude.internal.topics=false{}}}, matching internal topics should 
> remain eligible for subscription. Explicit subscriptions by topic name should 
> also preserve the existing contract: internal topics may be explicitly 
> subscribed to even when the setting is {{{}true{}}}.
>  
> *Observed Behavior and Cause*
> For Java {{Pattern}} subscriptions, the client evaluates the regex against 
> metadata that excludes internal topics and sends the matching topic names in 
> the heartbeat's {{SubscribedTopicNames}} field.
> For {{SubscriptionPattern}} subscriptions, the client sends the regex string 
> in {{{}SubscribedTopicRegex{}}}. The current heartbeat schema has no field 
> for conveying the internal-topic exclusion setting, and 
> {{TopicRegexResolver}} evaluates the regex against all topic names without 
> excluding internal topics.
> Additionally, {{ConsumerMetadata.retainTopic(...)}} retains metadata for 
> topic IDs assigned through server-side regex subscriptions using the 
> following condition:
> {code:java}
> return retainTopic(topicName, isInternal, nowMs)
>     || subscription.isAssignedFromRe2j(topicId); {code}
> Consequently, metadata is retained for a topic ID assigned by the server even 
> when the internal-topic exclusion check returns {{{}false{}}}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to