[
https://issues.apache.org/jira/browse/KAFKA-10340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17174105#comment-17174105
]
Luke Chen edited comment on KAFKA-10340 at 8/10/20, 6:38 AM:
-------------------------------------------------------------
After investigation, will improve the logging in this area.
Proposed to add 2 logs:
1. When the topic creation is disabled or topic is existed:
log.debug({color:#008000}"The topic creation setting is disabled or the topic
name {} is already created. " {color}+
{color:#008000}"If the topic doesn't exist, we'll rely on the
auto.create.topics.enable setting in broker side " {color}+
{color:#008000}"to see if the topic can be auto created or not"{color}, topic);
2. Before the producer send the record:
log.trace({color:#008000}"{} is going to send record to {}"{color},
WorkerSourceTask.{color:#000080}this{color}, topic);
So, for the request to let user know it's stuck waiting for the destination
topic to be created is now basically cannot know from client side because the
producer.send will block on waitOnMetadata method, which will keep trying until
timeout. There are many possible reasons for this timeout. It's hard to tell.
And the dynamically {{describeConfigs}} to get the broker setting is also not
easy because the broker name (we need broker name to describeConfig) is not
kept in the config, and also there's no other places in Kafka to check the
broker setting before doing something. I'd prefer to keep it as is because this
behavior applied for all kafka, not only for connectors.
Thanks.
was (Author: showuon):
Will improve the logging in this area.
Proposed to add 2 logs:
1. When the topic creation is disabled or topic is existed:
log.debug({color:#008000}"The topic creation setting is disabled or the topic
name {} is already created. " {color}+
{color:#008000}"If the topic doesn't exist, we'll rely on the
auto.create.topics.enable setting in broker side " {color}+
{color:#008000}"to see if the topic can be auto created or not"{color}, topic);
2. Before the producer send the record:
log.trace({color:#008000}"{} is going to send record to {}"{color},
WorkerSourceTask.{color:#000080}this{color}, topic);
So, for the request to let user know it's stuck waiting for the destination
topic to be created is now basically cannot know from client side because the
producer.send will block on waitOnMetadata method, which will keep trying until
timeout. There are many possible reasons for this timeout. It's hard to tell.
And the dynamically {{describeConfigs}} to get the broker setting is also not
easy because the broker name (we need broker name to describeConfig) is not
kept in the config, and also there's no other places in Kafka to check the
broker setting before doing something. I'd prefer to keep it as is because this
behavior applied for all kafka, not only for connectors.
Thanks.
> Source connectors should report error when trying to produce records to
> non-existent topics instead of hanging forever
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-10340
> URL: https://issues.apache.org/jira/browse/KAFKA-10340
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
> Reporter: Arjun Satish
> Assignee: Luke Chen
> Priority: Major
>
> Currently, a source connector will blindly attempt to write a record to a
> Kafka topic. When the topic does not exist, its creation is controlled by the
> {{auto.create.topics.enable}} config on the brokers. When auto.create is
> disabled, the producer.send() call on the Connect worker will hang
> indefinitely (due to the "infinite retries" configuration for said producer).
> In setups where this config is usually disabled, the source connector simply
> appears to hang and not produce any output.
> It is desirable to either log an info or an error message (or inform the user
> somehow) that the connector is simply stuck waiting for the destination topic
> to be created. When the worker has permissions to inspect the broker
> settings, it can use the {{listTopics}} and {{describeConfigs}} API in
> AdminClient to check if the topic exists, the broker can
> {{auto.create.topics.enable}} topics, and if these cases do not exist, either
> throw an error.
> With the recently merged
> [KIP-158|https://cwiki.apache.org/confluence/display/KAFKA/KIP-158%3A+Kafka+Connect+should+allow+source+connectors+to+set+topic-specific+settings+for+new+topics],
> this becomes even more specific a corner case: when topic creation settings
> are enabled, the worker should handle the corner case where topic creation is
> disabled, {{auto.create.topics.enable}} is set to false and topic does not
> exist.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
