[jira] [Commented] (KAFKA-5704) Auto topic creation causes failure with older clusters
[ https://issues.apache.org/jira/browse/KAFKA-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119354#comment-16119354 ] ASF GitHub Bot commented on KAFKA-5704: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3641 > Auto topic creation causes failure with older clusters > -- > > Key: KAFKA-5704 > URL: https://issues.apache.org/jira/browse/KAFKA-5704 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Affects Versions: 0.11.0.0 >Reporter: Ewen Cheslack-Postava >Assignee: Randall Hauch > Fix For: 0.11.0.1, 1.0.0 > > > The new automatic internal topic creation always tries to check the topic and > create it if missing. However, older brokers that we should still be > compatible with don't support some requests that are used. This results in an > UnsupportedVersionException which some of the TopicAdmin code notes that it > can throw but then isn't caught in the initializers, causing the entire > process to fail. > We should probably just catch it, log a message, and allow things to proceed > hoping that the user has already created the topics correctly (as we used to > do). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5704) Auto topic creation causes failure with older clusters
[ https://issues.apache.org/jira/browse/KAFKA-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117125#comment-16117125 ] ASF GitHub Bot commented on KAFKA-5704: --- GitHub user rhauch opened a pull request: https://github.com/apache/kafka/pull/3641 KAFKA-5704 Corrected Connect distributed startup behavior to allow older brokers to auto-create topics When a Connect distributed worker starts up talking with broker versions 0.10.1.0 and later, it will use the AdminClient to look for the internal topics and attempt to create them if they are missing. Although the AdminClient was added in 0.11.0.0, the AdminClient uses APIs to create topics that existed in 0.10.1.0 and later. This feature works as expected when Connect uses a broker version 0.10.1.0 or later. However, when a Connect distributed worker starts up using a broker older than 0.10.1.0, the AdminClient is not able to find the required APIs and thus will throw an UnsupportedVersionException. Unfortunately, this exception is not caught and instead causes the Connect worker to fail even when the topics already exist. This change handles the UnsupportedVersionException by logging a debug message and doing nothing. The existing producer logic will get information about the topics, which will cause the broker to create them if they don’t exist and broker auto-creation of topics is enabled. This is the same behavior that existed prior to 0.11.0.0, and so this change restores that behavior for brokers older than 0.10.1.0. This change also adds a system test that verifies Connect works with a variety of brokers and is able to run source and sink connectors. The test verifies that Connect can read from the internal topics when the connectors are restarted. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rhauch/kafka kafka-5704 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3641.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3641 commit 0d45fd113eeaff3844742181191db2cc508353fd Author: Randall Hauch Date: 2017-08-07T19:32:29Z KAFKA-5704 Corrected Connect distributed startup behavior to allow older brokers to auto-create topics When a Connect distributed worker starts up talking with broker versions 0.10.1.0 and later, it will use the AdminClient to look for the internal topics and attempt to create them if they are missing. Although the AdminClient was added in 0.11.0.0, the AdminClient uses APIs to create topics that existed in 0.10.1.0 and later. This feature works as expected when Connect uses a broker version 0.10.1.0 or later. However, when a Connect distributed worker starts up using a broker older than 0.10.1.0, the AdminClient is not able to find the required APIs and thus will throw an UnsupportedVersionException. Unfortunately, this exception is not caught and instead causes the Connect worker to fail even when the topics already exist. This change handles the UnsupportedVersionException by logging a debug message and doing nothing. The existing producer logic will get information about the topics, which will cause the broker to create them if they don’t exist and broker auto-creation of topics is enabled. This is the same behavior that existed prior to 0.11.0.0, and so this change restores that behavior for brokers older than 0.10.1.0. This change also adds a system test that verifies Connect works with a variety of brokers and is able to run source and sink connectors. The test verifies that Connect can read from the internal topics when the connectors are restarted. > Auto topic creation causes failure with older clusters > -- > > Key: KAFKA-5704 > URL: https://issues.apache.org/jira/browse/KAFKA-5704 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Affects Versions: 0.11.0.0 >Reporter: Ewen Cheslack-Postava >Assignee: Randall Hauch > > The new automatic internal topic creation always tries to check the topic and > create it if missing. However, older brokers that we should still be > compatible with don't support some requests that are used. This results in an > UnsupportedVersionException which some of the TopicAdmin code notes that it > can throw but then isn't caught in the initializers, causing the entire > process to fail. > We should probably just catch it, log a message, and allow things to proceed > hoping that the user has already created the topics correctly (as we used to > do). -- This message was sent by Atlassian JIRA (v6.4.14#64
[jira] [Commented] (KAFKA-5704) Auto topic creation causes failure with older clusters
[ https://issues.apache.org/jira/browse/KAFKA-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16115067#comment-16115067 ] Randall Hauch commented on KAFKA-5704: -- The fix is straightforward: [this block|https://github.com/apache/kafka/blob/0.11.0.0/connect/runtime/src/main/java/org/apache/kafka/connect/util/TopicAdmin.java#L235] should log the appropriate message and return an empty set. Then, the initializer in [KafkaBaseLog|https://github.com/apache/kafka/blob/0.11.0.0/connect/runtime/src/main/java/org/apache/kafka/connect/util/KafkaBasedLog.java#L126] will complete by doing nothing, and the producer will get the topic metadata that will result in the broker auto-creating the topic if missing and if so configured. > Auto topic creation causes failure with older clusters > -- > > Key: KAFKA-5704 > URL: https://issues.apache.org/jira/browse/KAFKA-5704 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Affects Versions: 0.11.0.0 >Reporter: Ewen Cheslack-Postava >Assignee: Randall Hauch > > The new automatic internal topic creation always tries to check the topic and > create it if missing. However, older brokers that we should still be > compatible with don't support some requests that are used. This results in an > UnsupportedVersionException which some of the TopicAdmin code notes that it > can throw but then isn't caught in the initializers, causing the entire > process to fail. > We should probably just catch it, log a message, and allow things to proceed > hoping that the user has already created the topics correctly (as we used to > do). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5704) Auto topic creation causes failure with older clusters
[
https://issues.apache.org/jira/browse/KAFKA-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16115063#comment-16115063
]
Randall Hauch commented on KAFKA-5704:
--
As of Kafka 0.11.0.0, the Kafka Connect distributed worker will automatically
create the internal topics (see KAFKA-4667) using the new KafkaAdminClient.
However, the KafkaAdminClient will throw an UnsupportedVersionException if the
broker version doesn't support the necessary API (which in this case is the
[{{CREATE_TOPICS}}
API|https://github.com/apache/kafka/blob/0.10.1/clients/src/main/java/org/apache/kafka/common/protocol/ApiKeys.java#L42]
added in 0.10.1.0). That means contrary to the [notes in
KAFKA-4667|https://issues.apache.org/jira/browse/KAFKA-4667?focusedCommentId=15998675&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15998675],
Kafka Connect can indeed work with brokers older than 0.10.1.0, will receive
the exception in such a case, and will not properly handle the exception.
> Auto topic creation causes failure with older clusters
> --
>
> Key: KAFKA-5704
> URL: https://issues.apache.org/jira/browse/KAFKA-5704
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 0.11.0.0
>Reporter: Ewen Cheslack-Postava
>Assignee: Randall Hauch
>
> The new automatic internal topic creation always tries to check the topic and
> create it if missing. However, older brokers that we should still be
> compatible with don't support some requests that are used. This results in an
> UnsupportedVersionException which some of the TopicAdmin code notes that it
> can throw but then isn't caught in the initializers, causing the entire
> process to fail.
> We should probably just catch it, log a message, and allow things to proceed
> hoping that the user has already created the topics correctly (as we used to
> do).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
