Andras Katona created KAFKA-21557:
-------------------------------------
Summary: Connect internal topics can be auto-created by the broker
with the wrong cleanup policy
Key: KAFKA-21557
URL: https://issues.apache.org/jira/browse/KAFKA-21557
Project: Kafka
Issue Type: Bug
Components: connect
Reporter: Andras Katona
Assignee: Andras Katona
When a Connect worker starts, it tries to create its internal topics (config,
offsets, status) with {color:#0747A6}cleanup.policy=compact{color}. If that
CreateTopics call fails with an authorization error, Connect logs it at DEBUG
and carries on. It assumes the topic exists or will be auto-created.
Next, {color:#0747A6}KafkaBasedLog.start(){color} calls
{color:#0747A6}consumer.partitionsFor(topic){color}. The internal consumer uses
the default {color:#0747A6}allow.auto.create.topics=true{color}. So if the
broker has {color:#0747A6}auto.create.topics.enable=true{color}, this metadata
request creates the topic with the broker's defaults, usually
{color:#0747A6}cleanup.policy=delete{color} and
{color:#0747A6}num.partitions=1{color}.
The cleanup policy check doesn't catch this on the worker that triggered the
auto-create. It runs before the topic exists, finds no config, and only logs at
INFO. That worker keeps running on a non-compacted topic. Its data, such as
source offsets, can be removed by retention. Every other worker then fails at
startup with:
{color:#0747A6}ConfigException: Topic '...' supplied via the
'offset.storage.topic' property is required to have 'cleanup.policy=compact'
... but found the topic currently has 'cleanup.policy=delete'.{color}
We hit this when the worker's CreateTopics call was briefly denied: the
authorizer was still loading its policies at startup.
The connector-specific offsets topics (KIP-618) have the same problem.
Proposed fix: set {color:#0747A6}allow.auto.create.topics=false{color} on the
consumers that read Connect's internal topics:
- {color:#0747A6}KafkaBasedLog.createConsumer(){color}, for the worker's
config, offsets and status topics
- {color:#0747A6}Worker.regularSourceOffsetsConsumerConfigs{color} and
{color:#0747A6}Worker.exactlyOnceSourceOffsetsConsumerConfigs{color}, for
connector-specific offsets topics
The topic should always be created by Connect with the right settings. If
Connect can't create it, the worker should fail rather than run on a topic with
the wrong config.
Related, possibly separate:
- {color:#0747A6}TopicAdmin.createOrFindTopics{color} logs a denied create at
DEBUG only. It should at least be a WARN.
- The MirrorMaker 2 internal topics (offset-syncs, checkpoints, heartbeats) are
compacted too, and their consumers have the same default.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)