Sagar Rao created KAFKA-15041:
---------------------------------
Summary: Source Connector auto topic creation fails when topic is
deleted and brokers don't support auto topic creation
Key: KAFKA-15041
URL: https://issues.apache.org/jira/browse/KAFKA-15041
Project: Kafka
Issue Type: Bug
Components: KafkaConnect
Reporter: Sagar Rao
Assignee: Sagar Rao
[KIP-158|https://cwiki.apache.org/confluence/display/KAFKA/KIP-158%3A+Kafka+Connect+should+allow+source+connectors+to+set+topic-specific+settings+for+new+topics]
allows the source connectors to create topics even when the broker doesn't
allow to do so. It does so by checking for every record if a topic needs to be
created
[https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractWorkerSourceTask.java#L500.]
To not always keep checking for topic presence via admin topics, it also
maintains a cache of the topics that it has created and doesn't create those
anymore. This helps to create topics when brokers don't support automatic topic
creation.
However, lets say the topic gets created initially and later on gets deleted
while the connector is still running and the brokers don't support automatic
topic creation. For such cases, the connector has cached the topic it has
already created and wouldn't recreate it because the cache never updates and
since the broker doesn't support topic creation, the logs would just be full of
messages like
```
Error while fetching metadata with correlation id 3260 :
\{connect-test=UNKNOWN_TOPIC_OR_PARTITION}
```
This can become a problem on enviroments where brokers don't allow topic
creation. We need a way to refresh the topics cache for such cases.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
