[
https://issues.apache.org/jira/browse/KAFKA-10440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandini Anagondi updated KAFKA-10440:
-------------------------------------
Description:
I found an issue while running MM2 using Kconnect Framework. Based on the
configuration provided in
[link|[https://github.com/apache/kafka/blob/trunk/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorConnectorConfig.java]]
*source : localhost:9092, Alias : A*
*target : localhost:9093, Alias : B*
*Issue Faced:*
Topics from the source cluster are created at target cluster but instead of
producing data of source topics to target it is producing to source cluster
topics and those topics are getting created at source because the
*allowAutoTopicCreation=true.*
*How the issue caused:*
Kconnect is running on source cluster and connectors are configured from A->B.
Connect framework uses connect-distributed.properties to initialise
producerConfig, consumerConfig. So, producer and consumer are initialised with
source bootstrap servers i.e., with
localhost:9092("source.cluster.producer.bootstrap.servers" : "localhost:9092”,
"source.cluster.consumer.bootstrap.servers" : "localhost:9092”).
This configuration works perfectly fine with all traditional connectors because
they produce and consume from that cluster itself. But MM2 connectors are
contrary to this. Let's see this example
Ideally MM2 should work like this
A B
test-mm2 A.test-mm2
Indeed it is working like this and creating topics recursively. Please refer to
the attachments.
A B
A.test-mm2 A.A.test-mm2
test-mm2 A.test-mm2
This applies to all the topics where data need to be copied. Suppose if it
doesn't have any data it won't cause this issue.
*Fixing the issue:*
MirrorSourceConnector should produce to destination cluster topics while
copying data. This issue can be fixed by overriding bootstrap.servers. To do
this connector.client.config.override.policy should be set to "All" in
connect-distributed.properties & (producer.override.bootstrap.servers,
consumer.override.bootstrap.servers) should match with
target.cluster.bootstrap.servers and this need to be provided in connectors
payload.So, connectors now produce the data to target cluster.
A B
test-mm2 A.test-mm2
List of the topics in both cases: [^list_of_topics.txt]
You can find the config in both cases here:
[^MirrorSourceConnector-config.json] - Config that matches with
MirrorConnectorConfig
[^MirrorSourceConnector-override-config.json]Config that works
To imitate the scenario: Start the kconnect at source cluster, Configure
connector with MirrorSourceConnector-config.json, create a topic and produce
data to it. Check the list of topics in 5-10seconds.
was:
I found an issue while running MM2 using Kconnect Framework. Based on the
configuration provided in [link
title|https://issues.apache.org/jira/browse/KAFKA-10440?jql=project%20%3D%20KAFKA%20AND%20component%20%3D%20KafkaConnect]
*source : localhost:9092, Alias : A*
*target : localhost:9093, Alias : B*
*Issue Faced:*
Topics from the source cluster are created at target cluster but instead of
producing data of source topics to target it is producing to source cluster
topics and those topics are getting created at source because the
*allowAutoTopicCreation=true.*
*How the issue caused:*
Kconnect is running on source cluster and connectors are configured from A->B.
Connect framework uses connect-distributed.properties to initialise
producerConfig, consumerConfig. So, producer and consumer are initialised with
source bootstrap servers i.e., with
localhost:9092("source.cluster.producer.bootstrap.servers" : "localhost:9092”,
"source.cluster.consumer.bootstrap.servers" : "localhost:9092”).
This configuration works perfectly fine with all traditional connectors because
they produce and consume from that cluster itself. But MM2 connectors are
contrary to this. Let's see this example
Ideally MM2 should work like this
A B
test-mm2 A.test-mm2
Indeed it is working like this and creating topics recursively. Please refer to
the attachments.
A B
A.test-mm2 A.A.test-mm2
test-mm2 A.test-mm2
This applies to all the topics where data need to be copied. Suppose if it
doesn't have any data it won't cause this issue.
*Fixing the issue:*
MirrorSourceConnector should produce to destination cluster topics while
copying data. This issue can be fixed by overriding bootstrap.servers. To do
this connector.client.config.override.policy should be set to "All" in
connect-distributed.properties & (producer.override.bootstrap.servers,
consumer.override.bootstrap.servers) should match with
target.cluster.bootstrap.servers and this need to be provided in connectors
payload.So, connectors now produce the data to target cluster.
A B
test-mm2 A.test-mm2
List of the topics in both cases: [^list_of_topics.txt]
You can find the config in both cases here:
[^MirrorSourceConnector-config.json] - Config that matches with
MirrorConnectorConfig
[^MirrorSourceConnector-override-config.json]Config that works
To imitate the scenario: Start the kconnect at source cluster, Configure
connector with MirrorSourceConnector-config.json, create a topic and produce
data to it. Check the list of topics in 5-10seconds.
> MM2 using Kconnect Framework Creates topics recursively on source cluster
> -------------------------------------------------------------------------
>
> Key: KAFKA-10440
> URL: https://issues.apache.org/jira/browse/KAFKA-10440
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
> Affects Versions: 2.5.0
> Reporter: Nandini Anagondi
> Priority: Major
> Fix For: 2.5.0
>
> Attachments: MirrorSourceConnector-config.json,
> MirrorSourceConnector-override-config.json, list_of_topics.txt
>
>
> I found an issue while running MM2 using Kconnect Framework. Based on the
> configuration provided in
> [link|[https://github.com/apache/kafka/blob/trunk/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorConnectorConfig.java]]
> *source : localhost:9092, Alias : A*
> *target : localhost:9093, Alias : B*
> *Issue Faced:*
> Topics from the source cluster are created at target cluster but instead of
> producing data of source topics to target it is producing to source cluster
> topics and those topics are getting created at source because the
> *allowAutoTopicCreation=true.*
> *How the issue caused:*
> Kconnect is running on source cluster and connectors are configured from
> A->B. Connect framework uses connect-distributed.properties to initialise
> producerConfig, consumerConfig. So, producer and consumer are initialised
> with source bootstrap servers i.e., with
> localhost:9092("source.cluster.producer.bootstrap.servers" :
> "localhost:9092”, "source.cluster.consumer.bootstrap.servers" :
> "localhost:9092”).
> This configuration works perfectly fine with all traditional connectors
> because they produce and consume from that cluster itself. But MM2 connectors
> are contrary to this. Let's see this example
> Ideally MM2 should work like this
> A B
> test-mm2 A.test-mm2
> Indeed it is working like this and creating topics recursively. Please refer
> to the attachments.
> A B
> A.test-mm2 A.A.test-mm2
> test-mm2 A.test-mm2
>
> This applies to all the topics where data need to be copied. Suppose if it
> doesn't have any data it won't cause this issue.
> *Fixing the issue:*
> MirrorSourceConnector should produce to destination cluster topics while
> copying data. This issue can be fixed by overriding bootstrap.servers. To do
> this connector.client.config.override.policy should be set to "All" in
> connect-distributed.properties & (producer.override.bootstrap.servers,
> consumer.override.bootstrap.servers) should match with
> target.cluster.bootstrap.servers and this need to be provided in connectors
> payload.So, connectors now produce the data to target cluster.
> A B
> test-mm2 A.test-mm2
>
> List of the topics in both cases: [^list_of_topics.txt]
> You can find the config in both cases here:
> [^MirrorSourceConnector-config.json] - Config that matches with
> MirrorConnectorConfig
> [^MirrorSourceConnector-override-config.json]Config that works
>
> To imitate the scenario: Start the kconnect at source cluster, Configure
> connector with MirrorSourceConnector-config.json, create a topic and produce
> data to it. Check the list of topics in 5-10seconds.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
