Andras Katona created KAFKA-21557:
-------------------------------------

             Summary: Connect internal topics can be auto-created by the broker 
with the wrong cleanup policy
                 Key: KAFKA-21557
                 URL: https://issues.apache.org/jira/browse/KAFKA-21557
             Project: Kafka
          Issue Type: Bug
          Components: connect
            Reporter: Andras Katona
            Assignee: Andras Katona


When a Connect worker starts, it tries to create its internal topics (config, 
offsets, status) with {color:#0747A6}cleanup.policy=compact{color}. If that 
CreateTopics call fails with an authorization error, Connect logs it at DEBUG 
and carries on. It assumes the topic exists or will be auto-created.

Next, {color:#0747A6}KafkaBasedLog.start(){color} calls 
{color:#0747A6}consumer.partitionsFor(topic){color}. The internal consumer uses 
the default {color:#0747A6}allow.auto.create.topics=true{color}. So if the 
broker has {color:#0747A6}auto.create.topics.enable=true{color}, this metadata 
request creates the topic with the broker's defaults, usually 
{color:#0747A6}cleanup.policy=delete{color} and 
{color:#0747A6}num.partitions=1{color}.

The cleanup policy check doesn't catch this on the worker that triggered the 
auto-create. It runs before the topic exists, finds no config, and only logs at 
INFO. That worker keeps running on a non-compacted topic. Its data, such as 
source offsets, can be removed by retention. Every other worker then fails at 
startup with:

{color:#0747A6}ConfigException: Topic '...' supplied via the 
'offset.storage.topic' property is required to have 'cleanup.policy=compact' 
... but found the topic currently has 'cleanup.policy=delete'.{color}

We hit this when the worker's CreateTopics call was briefly denied: the 
authorizer was still loading its policies at startup.

The connector-specific offsets topics (KIP-618) have the same problem.

Proposed fix: set {color:#0747A6}allow.auto.create.topics=false{color} on the 
consumers that read Connect's internal topics:
- {color:#0747A6}KafkaBasedLog.createConsumer(){color}, for the worker's 
config, offsets and status topics
- {color:#0747A6}Worker.regularSourceOffsetsConsumerConfigs{color} and 
{color:#0747A6}Worker.exactlyOnceSourceOffsetsConsumerConfigs{color}, for 
connector-specific offsets topics

The topic should always be created by Connect with the right settings. If 
Connect can't create it, the worker should fail rather than run on a topic with 
the wrong config.

Related, possibly separate:
- {color:#0747A6}TopicAdmin.createOrFindTopics{color} logs a denied create at 
DEBUG only. It should at least be a WARN.
- The MirrorMaker 2 internal topics (offset-syncs, checkpoints, heartbeats) are 
compacted too, and their consumers have the same default.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to