sanghyeok An created KAFKA-21244:
------------------------------------
Summary: Server-side consumer regex subscriptions do not honor
`exclude.internal.topics`
Key: KAFKA-21244
URL: https://issues.apache.org/jira/browse/KAFKA-21244
Project: Kafka
Issue Type: Bug
Reporter: sanghyeok An
Assignee: sanghyeok An
*Description*
When using {{SubscriptionPattern}} with {{{}*group.protocol=consumer*{}}},
server-side regex subscriptions do not honor
{{{}*exclude.internal.topics=true*{}}}. The same issue occurs when the
configuration is omitted, since its default value is {{{}true{}}}.
*Java {{Pattern}}* subscriptions correctly exclude internal topics under the
same consumer protocol. This creates inconsistent configuration behavior
between the two regex subscription APIs.
*Conditions*
* {{group.protocol=consumer}}
* {{{}exclude.internal.topics=true{}}}, or the configuration is omitted
* A regex that also matches internal topic names, such as
{{SubscriptionPattern(".*")}}
* An internal topic, such as {{{}__consumer_offsets{}}}, exists in the
coordinator metadata
* The consumer has the required permissions to describe and read the topic
Example configuration and subscription:
{code:java}
props.put("group.protocol", "consumer");
props.put("exclude.internal.topics", "true");
consumer.subscribe(new SubscriptionPattern(".*")); {code}
*Expected Behavior*
Internal topics should be excluded from the subscription and assignment even
when they match the regex.
With {{{}exclude.internal.topics=false{}}}, matching internal topics should
remain eligible for subscription. Explicit subscriptions by topic name should
also preserve the existing contract: internal topics may be explicitly
subscribed to even when the setting is {{{}true{}}}.
*Observed Behavior and Cause*
For Java {{Pattern}} subscriptions, the client evaluates the regex against
metadata that excludes internal topics and sends the matching topic names in
the heartbeat's {{SubscribedTopicNames}} field.
For {{SubscriptionPattern}} subscriptions, the client sends the regex string in
{{{}SubscribedTopicRegex{}}}. The current heartbeat schema has no field for
conveying the internal-topic exclusion setting, and {{TopicRegexResolver}}
evaluates the regex against all topic names without excluding internal topics.
Additionally, {{ConsumerMetadata.retainTopic(...)}} retains metadata for topic
IDs assigned through server-side regex subscriptions using the following
condition:
{code:java}
return retainTopic(topicName, isInternal, nowMs)
|| subscription.isAssignedFromRe2j(topicId); {code}
Consequently, metadata is retained for a topic ID assigned by the server even
when the internal-topic exclusion check returns {{{}false{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
