[
https://issues.apache.org/jira/browse/KAFKA-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sagar Rao resolved KAFKA-15041.
-------------------------------
Resolution: Won't Fix
For now, setting the config `producer.override.max.block.ms` at a connector
config level or `producer.max.block.ms` at a worker config level to a lower
value should fix this value. The problem is that the default value for the
above config is[ set to Long.MAX_VALUE
|https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Worker.java#L820]in
the configs and when topics are deleted manually, there's really no signal
that is received to indicate the same. We could add some heuristics like
checking if a topic is present or not periodically and refreshing the cache, or
check the source topic metrics to see if the records are just being buffered
and not being sent but that's outside the scope of runtime.
> Source Connector auto topic creation fails when topic is deleted and brokers
> don't support auto topic creation
> --------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-15041
> URL: https://issues.apache.org/jira/browse/KAFKA-15041
> Project: Kafka
> Issue Type: Bug
> Components: connect
> Reporter: Sagar Rao
> Assignee: Sagar Rao
> Priority: Major
>
> [KIP-158|https://cwiki.apache.org/confluence/display/KAFKA/KIP-158%3A+Kafka+Connect+should+allow+source+connectors+to+set+topic-specific+settings+for+new+topics]
> allows the source connectors to create topics even when the broker doesn't
> allow to do so. It does so by checking for every record if a topic needs to
> be created
> [https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractWorkerSourceTask.java#L500.]
> To not always keep checking for topic presence via admin topics, it also
> maintains a cache of the topics that it has created and doesn't create those
> anymore. This helps to create topics when brokers don't support automatic
> topic creation.
> However, lets say the topic gets created initially and later on gets deleted
> while the connector is still running and the brokers don't support automatic
> topic creation. For such cases, the connector has cached the topic it has
> already created and wouldn't recreate it because the cache never updates and
> since the broker doesn't support topic creation, the logs would just be full
> of messages like
>
> {code:java}
> Error while fetching metadata with correlation id 3260 :
> {connect-test=UNKNOWN_TOPIC_OR_PARTITION}{code}
>
> This can become a problem on environments where brokers don't allow topic
> creation. We need a way to refresh the topics cache for such cases.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
