[
https://issues.apache.org/jira/browse/KAFKA-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sagar Rao updated KAFKA-15041:
------------------------------
Description:
[KIP-158|https://cwiki.apache.org/confluence/display/KAFKA/KIP-158%3A+Kafka+Connect+should+allow+source+connectors+to+set+topic-specific+settings+for+new+topics]
allows the source connectors to create topics even when the broker doesn't
allow to do so. It does so by checking for every record if a topic needs to be
created
[https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractWorkerSourceTask.java#L500.]
To not always keep checking for topic presence via admin topics, it also
maintains a cache of the topics that it has created and doesn't create those
anymore. This helps to create topics when brokers don't support automatic topic
creation.
However, lets say the topic gets created initially and later on gets deleted
while the connector is still running and the brokers don't support automatic
topic creation. For such cases, the connector has cached the topic it has
already created and wouldn't recreate it because the cache never updates and
since the broker doesn't support topic creation, the logs would just be full of
messages like
{code:java}
Error while fetching metadata with correlation id 3260 :
{connect-test=UNKNOWN_TOPIC_OR_PARTITION}{code}
This can become a problem on environments where brokers don't allow topic
creation. We need a way to refresh the topics cache for such cases.
was:
[KIP-158|https://cwiki.apache.org/confluence/display/KAFKA/KIP-158%3A+Kafka+Connect+should+allow+source+connectors+to+set+topic-specific+settings+for+new+topics]
allows the source connectors to create topics even when the broker doesn't
allow to do so. It does so by checking for every record if a topic needs to be
created
[https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractWorkerSourceTask.java#L500.]
To not always keep checking for topic presence via admin topics, it also
maintains a cache of the topics that it has created and doesn't create those
anymore. This helps to create topics when brokers don't support automatic topic
creation.
However, lets say the topic gets created initially and later on gets deleted
while the connector is still running and the brokers don't support automatic
topic creation. For such cases, the connector has cached the topic it has
already created and wouldn't recreate it because the cache never updates and
since the broker doesn't support topic creation, the logs would just be full of
messages like
{code:java}
Error while fetching metadata with correlation id 3260 :
{connect-test=UNKNOWN_TOPIC_OR_PARTITION}{code}
This can become a problem on enviroments where brokers don't allow topic
creation. We need a way to refresh the topics cache for such cases.
> Source Connector auto topic creation fails when topic is deleted and brokers
> don't support auto topic creation
> --------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-15041
> URL: https://issues.apache.org/jira/browse/KAFKA-15041
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
> Reporter: Sagar Rao
> Assignee: Sagar Rao
> Priority: Major
>
> [KIP-158|https://cwiki.apache.org/confluence/display/KAFKA/KIP-158%3A+Kafka+Connect+should+allow+source+connectors+to+set+topic-specific+settings+for+new+topics]
> allows the source connectors to create topics even when the broker doesn't
> allow to do so. It does so by checking for every record if a topic needs to
> be created
> [https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractWorkerSourceTask.java#L500.]
> To not always keep checking for topic presence via admin topics, it also
> maintains a cache of the topics that it has created and doesn't create those
> anymore. This helps to create topics when brokers don't support automatic
> topic creation.
> However, lets say the topic gets created initially and later on gets deleted
> while the connector is still running and the brokers don't support automatic
> topic creation. For such cases, the connector has cached the topic it has
> already created and wouldn't recreate it because the cache never updates and
> since the broker doesn't support topic creation, the logs would just be full
> of messages like
>
> {code:java}
> Error while fetching metadata with correlation id 3260 :
> {connect-test=UNKNOWN_TOPIC_OR_PARTITION}{code}
>
> This can become a problem on environments where brokers don't allow topic
> creation. We need a way to refresh the topics cache for such cases.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
