[jira] [Commented] (KAFKA-13255) Mirrormaker config property config.properties.exclude is not working as expected
[
https://issues.apache.org/jira/browse/KAFKA-13255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17529986#comment-17529986
]
Edoardo Comar commented on KAFKA-13255:
---
Thanks Tom !!!
> Mirrormaker config property config.properties.exclude is not working as
> expected
> -
>
> Key: KAFKA-13255
> URL: https://issues.apache.org/jira/browse/KAFKA-13255
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 2.8.0
>Reporter: Anamika Nadkarni
>Assignee: Ed Berezitsky
>Priority: Major
> Fix For: 3.2.0, 3.1.1
>
>
> Objective - Use MM2 (kafka connect in distributed cluster) for data migration
> between cluster hosted in private data center and aws msk cluster.
> Steps performed -
> # Started kafka-connect service.
> # Created 3 MM2 connectors (i.e. source connector, checkpoint connector and
> heartbeat connector). Curl commands used to create connectors are in the
> attached file. To exclude certain config properties while topic replication,
> we are using the 'config.properties.exclude' property in the MM2 source
> connector.
> Expected -
> Source topic 'dev.portlandDc.anamika.helloMsk' should be successfully created
> in destination cluster.
> Actual -
> Creation of the source topic 'dev.portlandDc.anamika.helloMsk' in destination
> cluster fails with an error. Error is
> {code:java}
> [2021-08-06 06:13:40,944] WARN [mm2-msc|worker] Could not create topic
> dev.portlandDc.anamika.helloMsk.
> (org.apache.kafka.connect.mirror.MirrorSourceConnector:371)
> org.apache.kafka.common.errors.InvalidConfigurationException: Unknown topic
> config name: confluent.value.schema.validation{code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (KAFKA-13255) Mirrormaker config property config.properties.exclude is not working as expected
[
https://issues.apache.org/jira/browse/KAFKA-13255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17529834#comment-17529834
]
Tom Bentley commented on KAFKA-13255:
-
OK [~ecomar] I've cherry-picked it to 3.1
> Mirrormaker config property config.properties.exclude is not working as
> expected
> -
>
> Key: KAFKA-13255
> URL: https://issues.apache.org/jira/browse/KAFKA-13255
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 2.8.0
>Reporter: Anamika Nadkarni
>Assignee: Ed Berezitsky
>Priority: Major
> Fix For: 3.2.0, 3.1.1
>
>
> Objective - Use MM2 (kafka connect in distributed cluster) for data migration
> between cluster hosted in private data center and aws msk cluster.
> Steps performed -
> # Started kafka-connect service.
> # Created 3 MM2 connectors (i.e. source connector, checkpoint connector and
> heartbeat connector). Curl commands used to create connectors are in the
> attached file. To exclude certain config properties while topic replication,
> we are using the 'config.properties.exclude' property in the MM2 source
> connector.
> Expected -
> Source topic 'dev.portlandDc.anamika.helloMsk' should be successfully created
> in destination cluster.
> Actual -
> Creation of the source topic 'dev.portlandDc.anamika.helloMsk' in destination
> cluster fails with an error. Error is
> {code:java}
> [2021-08-06 06:13:40,944] WARN [mm2-msc|worker] Could not create topic
> dev.portlandDc.anamika.helloMsk.
> (org.apache.kafka.connect.mirror.MirrorSourceConnector:371)
> org.apache.kafka.common.errors.InvalidConfigurationException: Unknown topic
> config name: confluent.value.schema.validation{code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (KAFKA-13255) Mirrormaker config property config.properties.exclude is not working as expected
[
https://issues.apache.org/jira/browse/KAFKA-13255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17529541#comment-17529541
]
Edoardo Comar commented on KAFKA-13255:
---
Would have been really nice to have this fix in 3.1
> Mirrormaker config property config.properties.exclude is not working as
> expected
> -
>
> Key: KAFKA-13255
> URL: https://issues.apache.org/jira/browse/KAFKA-13255
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 2.8.0
>Reporter: Anamika Nadkarni
>Assignee: Ed Berezitsky
>Priority: Major
> Fix For: 3.2.0
>
>
> Objective - Use MM2 (kafka connect in distributed cluster) for data migration
> between cluster hosted in private data center and aws msk cluster.
> Steps performed -
> # Started kafka-connect service.
> # Created 3 MM2 connectors (i.e. source connector, checkpoint connector and
> heartbeat connector). Curl commands used to create connectors are in the
> attached file. To exclude certain config properties while topic replication,
> we are using the 'config.properties.exclude' property in the MM2 source
> connector.
> Expected -
> Source topic 'dev.portlandDc.anamika.helloMsk' should be successfully created
> in destination cluster.
> Actual -
> Creation of the source topic 'dev.portlandDc.anamika.helloMsk' in destination
> cluster fails with an error. Error is
> {code:java}
> [2021-08-06 06:13:40,944] WARN [mm2-msc|worker] Could not create topic
> dev.portlandDc.anamika.helloMsk.
> (org.apache.kafka.connect.mirror.MirrorSourceConnector:371)
> org.apache.kafka.common.errors.InvalidConfigurationException: Unknown topic
> config name: confluent.value.schema.validation{code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (KAFKA-13255) Mirrormaker config property config.properties.exclude is not working as expected
[
https://issues.apache.org/jira/browse/KAFKA-13255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460354#comment-17460354
]
Luke Chen commented on KAFKA-13255:
---
Thanks for your response. In that case, I agree we don't need to include this
fix in v3.1.0. Thank you.
> Mirrormaker config property config.properties.exclude is not working as
> expected
> -
>
> Key: KAFKA-13255
> URL: https://issues.apache.org/jira/browse/KAFKA-13255
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 2.8.0
>Reporter: Anamika Nadkarni
>Assignee: Ed Berezitsky
>Priority: Major
> Fix For: 3.2.0
>
>
> Objective - Use MM2 (kafka connect in distributed cluster) for data migration
> between cluster hosted in private data center and aws msk cluster.
> Steps performed -
> # Started kafka-connect service.
> # Created 3 MM2 connectors (i.e. source connector, checkpoint connector and
> heartbeat connector). Curl commands used to create connectors are in the
> attached file. To exclude certain config properties while topic replication,
> we are using the 'config.properties.exclude' property in the MM2 source
> connector.
> Expected -
> Source topic 'dev.portlandDc.anamika.helloMsk' should be successfully created
> in destination cluster.
> Actual -
> Creation of the source topic 'dev.portlandDc.anamika.helloMsk' in destination
> cluster fails with an error. Error is
> {code:java}
> [2021-08-06 06:13:40,944] WARN [mm2-msc|worker] Could not create topic
> dev.portlandDc.anamika.helloMsk.
> (org.apache.kafka.connect.mirror.MirrorSourceConnector:371)
> org.apache.kafka.common.errors.InvalidConfigurationException: Unknown topic
> config name: confluent.value.schema.validation{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (KAFKA-13255) Mirrormaker config property config.properties.exclude is not working as expected
[
https://issues.apache.org/jira/browse/KAFKA-13255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460163#comment-17460163
]
Ed Berezitsky commented on KAFKA-13255:
---
the workaround would be to pre-create topics on a target cluster. Also, this
bug is critical (show stopper) only if MM2 configured to replicate topics
cross-flavors cluster, as reported in the original case above (Confluent to
MSK, or to vanilla Apache Kafka). Once topic is created on target, this bug
won't affect data mirroring process.
> Mirrormaker config property config.properties.exclude is not working as
> expected
> -
>
> Key: KAFKA-13255
> URL: https://issues.apache.org/jira/browse/KAFKA-13255
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 2.8.0
>Reporter: Anamika Nadkarni
>Assignee: Ed Berezitsky
>Priority: Major
> Fix For: 3.2.0
>
>
> Objective - Use MM2 (kafka connect in distributed cluster) for data migration
> between cluster hosted in private data center and aws msk cluster.
> Steps performed -
> # Started kafka-connect service.
> # Created 3 MM2 connectors (i.e. source connector, checkpoint connector and
> heartbeat connector). Curl commands used to create connectors are in the
> attached file. To exclude certain config properties while topic replication,
> we are using the 'config.properties.exclude' property in the MM2 source
> connector.
> Expected -
> Source topic 'dev.portlandDc.anamika.helloMsk' should be successfully created
> in destination cluster.
> Actual -
> Creation of the source topic 'dev.portlandDc.anamika.helloMsk' in destination
> cluster fails with an error. Error is
> {code:java}
> [2021-08-06 06:13:40,944] WARN [mm2-msc|worker] Could not create topic
> dev.portlandDc.anamika.helloMsk.
> (org.apache.kafka.connect.mirror.MirrorSourceConnector:371)
> org.apache.kafka.common.errors.InvalidConfigurationException: Unknown topic
> config name: confluent.value.schema.validation{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (KAFKA-13255) Mirrormaker config property config.properties.exclude is not working as expected
[
https://issues.apache.org/jira/browse/KAFKA-13255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460107#comment-17460107
]
Mickael Maison commented on KAFKA-13255:
It's not a regression from 3.0 so it's probably too late for 3.1.0. David Jacot
is the release manager and has the final say. You can ask him if you think this
should be included.
I'm not sure if there's a workaround apart from not mirroring topics with
custom configurations
> Mirrormaker config property config.properties.exclude is not working as
> expected
> -
>
> Key: KAFKA-13255
> URL: https://issues.apache.org/jira/browse/KAFKA-13255
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 2.8.0
>Reporter: Anamika Nadkarni
>Assignee: Ed Berezitsky
>Priority: Major
> Fix For: 3.2.0
>
>
> Objective - Use MM2 (kafka connect in distributed cluster) for data migration
> between cluster hosted in private data center and aws msk cluster.
> Steps performed -
> # Started kafka-connect service.
> # Created 3 MM2 connectors (i.e. source connector, checkpoint connector and
> heartbeat connector). Curl commands used to create connectors are in the
> attached file. To exclude certain config properties while topic replication,
> we are using the 'config.properties.exclude' property in the MM2 source
> connector.
> Expected -
> Source topic 'dev.portlandDc.anamika.helloMsk' should be successfully created
> in destination cluster.
> Actual -
> Creation of the source topic 'dev.portlandDc.anamika.helloMsk' in destination
> cluster fails with an error. Error is
> {code:java}
> [2021-08-06 06:13:40,944] WARN [mm2-msc|worker] Could not create topic
> dev.portlandDc.anamika.helloMsk.
> (org.apache.kafka.connect.mirror.MirrorSourceConnector:371)
> org.apache.kafka.common.errors.InvalidConfigurationException: Unknown topic
> config name: confluent.value.schema.validation{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (KAFKA-13255) Mirrormaker config property config.properties.exclude is not working as expected
[
https://issues.apache.org/jira/browse/KAFKA-13255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17459930#comment-17459930
]
Luke Chen commented on KAFKA-13255:
---
[~mimaison] [~bdesert] , do you know if there's any workaround for this issue?
If no, do you think we should get this fix cherry-picked back to v3.1.0, since
it blocked users using MM2?
Thank you.
> Mirrormaker config property config.properties.exclude is not working as
> expected
> -
>
> Key: KAFKA-13255
> URL: https://issues.apache.org/jira/browse/KAFKA-13255
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 2.8.0
>Reporter: Anamika Nadkarni
>Assignee: Ed Berezitsky
>Priority: Major
> Fix For: 3.2.0
>
>
> Objective - Use MM2 (kafka connect in distributed cluster) for data migration
> between cluster hosted in private data center and aws msk cluster.
> Steps performed -
> # Started kafka-connect service.
> # Created 3 MM2 connectors (i.e. source connector, checkpoint connector and
> heartbeat connector). Curl commands used to create connectors are in the
> attached file. To exclude certain config properties while topic replication,
> we are using the 'config.properties.exclude' property in the MM2 source
> connector.
> Expected -
> Source topic 'dev.portlandDc.anamika.helloMsk' should be successfully created
> in destination cluster.
> Actual -
> Creation of the source topic 'dev.portlandDc.anamika.helloMsk' in destination
> cluster fails with an error. Error is
> {code:java}
> [2021-08-06 06:13:40,944] WARN [mm2-msc|worker] Could not create topic
> dev.portlandDc.anamika.helloMsk.
> (org.apache.kafka.connect.mirror.MirrorSourceConnector:371)
> org.apache.kafka.common.errors.InvalidConfigurationException: Unknown topic
> config name: confluent.value.schema.validation{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (KAFKA-13255) Mirrormaker config property config.properties.exclude is not working as expected
[
https://issues.apache.org/jira/browse/KAFKA-13255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17429026#comment-17429026
]
Ed Berezitsky commented on KAFKA-13255:
---
[~ryannedolan], I recommend to fix this bug report:
Subject: Mirrormaker doesn't honor exclude filters for new topics
The bug was indeed introduced in 2.8, when MM started creating topics WITH
configs. Prior to this, MM would create new topic on target without any config
from source, and then `syncTopicConfigs` would replicate config while honoring
the exclude filter defined in `DefaultConfigPropertyFilter` or explicitly
specified in MM property `config.properties.exclude`.
After the update in 2.8, MM started creating new topics along with their
configs from source cluster, but the configs aren't being filtered against
exclude param.
I've submitted PR with a little update to use the same method as used for
`syncTopicConfigs` to apply the filter.
> Mirrormaker config property config.properties.exclude is not working as
> expected
> -
>
> Key: KAFKA-13255
> URL: https://issues.apache.org/jira/browse/KAFKA-13255
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 2.8.0
>Reporter: Anamika Nadkarni
>Priority: Major
>
> Objective - Use MM2 (kafka connect in distributed cluster) for data migration
> between cluster hosted in private data center and aws msk cluster.
> Steps performed -
> # Started kafka-connect service.
> # Created 3 MM2 connectors (i.e. source connector, checkpoint connector and
> heartbeat connector). Curl commands used to create connectors are in the
> attached file. To exclude certain config properties while topic replication,
> we are using the 'config.properties.exclude' property in the MM2 source
> connector.
> Expected -
> Source topic 'dev.portlandDc.anamika.helloMsk' should be successfully created
> in destination cluster.
> Actual -
> Creation of the source topic 'dev.portlandDc.anamika.helloMsk' in destination
> cluster fails with an error. Error is
> {code:java}
> [2021-08-06 06:13:40,944] WARN [mm2-msc|worker] Could not create topic
> dev.portlandDc.anamika.helloMsk.
> (org.apache.kafka.connect.mirror.MirrorSourceConnector:371)
> org.apache.kafka.common.errors.InvalidConfigurationException: Unknown topic
> config name: confluent.value.schema.validation{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-13255) Mirrormaker config property config.properties.exclude is not working as expected
[
https://issues.apache.org/jira/browse/KAFKA-13255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17418867#comment-17418867
]
Anamika Nadkarni commented on KAFKA-13255:
--
[~ryannedolan] How much time usually it takes to get PR approved on kafka?
https://github.com/apache/kafka/pull/11328
> Mirrormaker config property config.properties.exclude is not working as
> expected
> -
>
> Key: KAFKA-13255
> URL: https://issues.apache.org/jira/browse/KAFKA-13255
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 2.8.0
>Reporter: Anamika Nadkarni
>Priority: Major
>
> Objective - Use MM2 (kafka connect in distributed cluster) for data migration
> between cluster hosted in private data center and aws msk cluster.
> Steps performed -
> # Started kafka-connect service.
> # Created 3 MM2 connectors (i.e. source connector, checkpoint connector and
> heartbeat connector). Curl commands used to create connectors are in the
> attached file. To exclude certain config properties while topic replication,
> we are using the 'config.properties.exclude' property in the MM2 source
> connector.
> Expected -
> Source topic 'dev.portlandDc.anamika.helloMsk' should be successfully created
> in destination cluster.
> Actual -
> Creation of the source topic 'dev.portlandDc.anamika.helloMsk' in destination
> cluster fails with an error. Error is
> {code:java}
> [2021-08-06 06:13:40,944] WARN [mm2-msc|worker] Could not create topic
> dev.portlandDc.anamika.helloMsk.
> (org.apache.kafka.connect.mirror.MirrorSourceConnector:371)
> org.apache.kafka.common.errors.InvalidConfigurationException: Unknown topic
> config name: confluent.value.schema.validation{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-13255) Mirrormaker config property config.properties.exclude is not working as expected
[
https://issues.apache.org/jira/browse/KAFKA-13255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415812#comment-17415812
]
Anamika Nadkarni commented on KAFKA-13255:
--
[~ryannedolan] Submitted PR. Thank you !!
> Mirrormaker config property config.properties.exclude is not working as
> expected
> -
>
> Key: KAFKA-13255
> URL: https://issues.apache.org/jira/browse/KAFKA-13255
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 2.8.0
>Reporter: Anamika Nadkarni
>Priority: Major
>
> Objective - Use MM2 (kafka connect in distributed cluster) for data migration
> between cluster hosted in private data center and aws msk cluster.
> Steps performed -
> # Started kafka-connect service.
> # Created 3 MM2 connectors (i.e. source connector, checkpoint connector and
> heartbeat connector). Curl commands used to create connectors are in the
> attached file. To exclude certain config properties while topic replication,
> we are using the 'config.properties.exclude' property in the MM2 source
> connector.
> Expected -
> Source topic 'dev.portlandDc.anamika.helloMsk' should be successfully created
> in destination cluster.
> Actual -
> Creation of the source topic 'dev.portlandDc.anamika.helloMsk' in destination
> cluster fails with an error. Error is
> {code:java}
> [2021-08-06 06:13:40,944] WARN [mm2-msc|worker] Could not create topic
> dev.portlandDc.anamika.helloMsk.
> (org.apache.kafka.connect.mirror.MirrorSourceConnector:371)
> org.apache.kafka.common.errors.InvalidConfigurationException: Unknown topic
> config name: confluent.value.schema.validation{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-13255) Mirrormaker config property config.properties.exclude is not working as expected
[
https://issues.apache.org/jira/browse/KAFKA-13255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17407742#comment-17407742
]
Ryanne Dolan commented on KAFKA-13255:
--
hey [~anadkarni], you'll need to create a pull request on github
(https://github.com/apache/kafka) so that we can review and merge the changes.
Let me know if you need help.
> Mirrormaker config property config.properties.exclude is not working as
> expected
> -
>
> Key: KAFKA-13255
> URL: https://issues.apache.org/jira/browse/KAFKA-13255
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 2.8.0
>Reporter: Anamika Nadkarni
>Priority: Major
>
> Objective - Use MM2 (kafka connect in distributed cluster) for data migration
> between cluster hosted in private data center and aws msk cluster.
> Steps performed -
> # Started kafka-connect service.
> # Created 3 MM2 connectors (i.e. source connector, checkpoint connector and
> heartbeat connector). Curl commands used to create connectors are in the
> attached file. To exclude certain config properties while topic replication,
> we are using the 'config.properties.exclude' property in the MM2 source
> connector.
> Expected -
> Source topic 'dev.portlandDc.anamika.helloMsk' should be successfully created
> in destination cluster.
> Actual -
> Creation of the source topic 'dev.portlandDc.anamika.helloMsk' in destination
> cluster fails with an error. Error is
> {code:java}
> [2021-08-06 06:13:40,944] WARN [mm2-msc|worker] Could not create topic
> dev.portlandDc.anamika.helloMsk.
> (org.apache.kafka.connect.mirror.MirrorSourceConnector:371)
> org.apache.kafka.common.errors.InvalidConfigurationException: Unknown topic
> config name: confluent.value.schema.validation{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-13255) Mirrormaker config property config.properties.exclude is not working as expected
[
https://issues.apache.org/jira/browse/KAFKA-13255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17407528#comment-17407528
]
Anamika Nadkarni commented on KAFKA-13255:
--
I have fixed this issue and would like to contribute my changes to apache kafka
code.
> Mirrormaker config property config.properties.exclude is not working as
> expected
> -
>
> Key: KAFKA-13255
> URL: https://issues.apache.org/jira/browse/KAFKA-13255
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 2.8.0
>Reporter: Anamika Nadkarni
>Priority: Major
>
> Objective - Use MM2 (kafka connect in distributed cluster) for data migration
> between cluster hosted in private data center and aws msk cluster.
> Steps performed -
> # Started kafka-connect service.
> # Created 3 MM2 connectors (i.e. source connector, checkpoint connector and
> heartbeat connector). Curl commands used to create connectors are in the
> attached file. To exclude certain config properties while topic replication,
> we are using the 'config.properties.exclude' property in the MM2 source
> connector.
> Expected -
> Source topic 'dev.portlandDc.anamika.helloMsk' should be successfully created
> in destination cluster.
> Actual -
> Creation of the source topic 'dev.portlandDc.anamika.helloMsk' in destination
> cluster fails with an error. Error is
> {code:java}
> [2021-08-06 06:13:40,944] WARN [mm2-msc|worker] Could not create topic
> dev.portlandDc.anamika.helloMsk.
> (org.apache.kafka.connect.mirror.MirrorSourceConnector:371)
> org.apache.kafka.common.errors.InvalidConfigurationException: Unknown topic
> config name: confluent.value.schema.validation{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-13255) Mirrormaker config property config.properties.exclude is not working as expected
[
https://issues.apache.org/jira/browse/KAFKA-13255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17407122#comment-17407122
]
Anamika Nadkarni commented on KAFKA-13255:
--
To replicate source topic in destination cluster with no prefix
(), I am using custom replication policy while creating source
connector. You can generate custom replication policy jar from below link. I am
getting 'HTTP - 403 Forbidden' , when I am trying to attach the jar.
https://github.com/aws-samples/mirrormaker2-msk-migration/tree/master/CustomMM2ReplicationPolicy
> Mirrormaker config property config.properties.exclude is not working as
> expected
> -
>
> Key: KAFKA-13255
> URL: https://issues.apache.org/jira/browse/KAFKA-13255
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 2.8.0
>Reporter: Anamika Nadkarni
>Priority: Major
>
> Objective - Use MM2 (kafka connect in distributed cluster) for data migration
> between cluster hosted in private data center and aws msk cluster.
> Steps performed -
> # Started kafka-connect service.
> # Created 3 MM2 connectors (i.e. source connector, checkpoint connector and
> heartbeat connector). Curl commands used to create connectors are in the
> attached file. To exclude certain config properties while topic replication,
> we are using the 'config.properties.exclude' property in the MM2 source
> connector.
> Expected -
> Source topic 'dev.portlandDc.anamika.helloMsk' should be successfully created
> in destination cluster.
> Actual -
> Creation of the source topic 'dev.portlandDc.anamika.helloMsk' in destination
> cluster fails with an error. Error is
> {code:java}
> [2021-08-06 06:13:40,944] WARN [mm2-msc|worker] Could not create topic
> dev.portlandDc.anamika.helloMsk.
> (org.apache.kafka.connect.mirror.MirrorSourceConnector:371)
> org.apache.kafka.common.errors.InvalidConfigurationException: Unknown topic
> config name: confluent.value.schema.validation{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-13255) Mirrormaker config property config.properties.exclude is not working as expected
[
https://issues.apache.org/jira/browse/KAFKA-13255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17407116#comment-17407116
]
Anamika Nadkarni commented on KAFKA-13255:
--
I keep getting HTTP error 403 when trying to attach file with curl commands to
this jira. As a work around I am pasting the file contents below.
{code:java}
curl -X POST http://10.xx.xx.xx:8083/connectors -H "Content-Type:
application/json" -d '{
"name":"mm2-msc",
"config":{
"connector.class":"org.apache.kafka.connect.mirror.MirrorSourceConnector",
"clusters":"mysource,mydestination",
"source.cluster.alias":"mysource",
"target.cluster.alias":"mydestination",
"replication.policy.class":"com.amazonaws.kafka.samples.CustomMM2ReplicationPolicy",
"target.cluster.bootstrap.servers":"b-2.mydestination.amazonaws.com:9096,b-3.mydestination.amazonaws.com:9096,b-1.mydestination.amazonaws.com:9096",
"source.cluster.bootstrap.servers":"10.xx.xx.xx:9093,10.xx.xx.xx:9093,10.xx.xx.xx:9093,10.xx.xx.xx:9093,10.xx.xx.xx:9093",
"topics":"dev.portlandDc.anamika.helloMsk",
"config.properties.exclude":"^confluent\\..*$",
"tasks.max":"1",
"key.converter":" org.apache.kafka.connect.converters.ByteArrayConverter",
"value.converter":"org.apache.kafka.connect.converters.ByteArrayConverter",
"replication.factor":"3",
"offset-syncs.topic.replication.factor":"3",
"sync.topic.acls.interval.seconds":"20",
"sync.topic.configs.interval.seconds":"20",
"refresh.topics.interval.seconds":"20",
"refresh.groups.interval.seconds":"20",
"consumer.group.id":"mm2-msc-cons",
"producer.enable.idempotence":"true",
"source.cluster.security.protocol":"SASL_SSL",
"source.cluster.sasl.mechanism":"PLAIN",
"source.cluster.ssl.truststore.location":"/home/ec2-user/dc/kafka.client.truststore.jks",
"source.cluster.ssl.truststore.password":"x",
"source.cluster.ssl.endpoint.identification.algorithm":"",
"source.cluster.sasl.jaas.config":"org.apache.kafka.common.security.plain.PlainLoginModule
required username=\"x\" password=\"x\";",
"target.cluster.security.protocol":"SASL_SSL",
"target.cluster.sasl.mechanism":"SCRAM-SHA-512",
"target.cluster.ssl.truststore.location":"/home/ec2-user/kafka_2.13-2.8.0/config/msk/kafka.client.truststore.jks",
"target.cluster.sasl.jaas.config":"org.apache.kafka.common.security.scram.ScramLoginModule
required username=\"x\" password=\"x\";"
}
}' | jq .
curl -X POST http://10.xx.xx.xx:8083/connectors -H "Content-Type:
application/json" -d '{
"name":"mm2-cpc",
"config":{
"connector.class":"org.apache.kafka.connect.mirror.MirrorCheckpointConnector",
"clusters":"mysource,mydestination",
"source.cluster.alias":"mysource",
"target.cluster.alias":"mydestination",
"config.properties.exclude":"confluent.value.schema.validation",
"config.properties.blacklist":"confluent.value.schema.validation",
"target.cluster.bootstrap.servers":"b-2.mydestination.amazonaws.com:9096,b-3.mydestination.amazonaws.com:9096,b-1.mydestination.amazonaws.com:9096",
"source.cluster.bootstrap.servers":"10.xx.xx.xx:9093,10.xx.xx.xx:9093,10.xx.xx.xx:9093,10.xx.xx.xx:9093,10.xx.xx.xx:9093",
"tasks.max":"1",
"key.converter":" org.apache.kafka.connect.converters.ByteArrayConverter",
"value.converter":"org.apache.kafka.connect.converters.ByteArrayConverter",
"replication.policy.class":"com.amazonaws.kafka.samples.CustomMM2ReplicationPolicy",
"replication.factor":"3",
"sync.topic.configs.enabled":"false",
"checkpoints.topic.replication.factor":"3",
"emit.checkpoints.interval.seconds":"20",
"source.cluster.security.protocol":"SASL_SSL",
"source.cluster.sasl.mechanism":"PLAIN",
"source.cluster.ssl.truststore.location":"/home/ec2-user/dc/kafka.client.truststore.jks",
"source.cluster.ssl.truststore.password":"x",
"source.cluster.ssl.endpoint.identification.algorithm":"",
"source.cluster.sasl.jaas.config":"org.apache.kafka.common.security.plain.PlainLoginModule
required username=\"x\" password=\"x\";",
"target.cluster.security.protocol":"SASL_SSL",
"target.cluster.sasl.mechanism":"SCRAM-SHA-512",
"target.cluster.ssl.truststore.location":"/home/ec2-user/kafka_2.13-2.8.0/config/msk/kafka.client.truststore.jks",
"target.cluster.sasl.jaas.config":"org.apache.kafka.common.security.scram.ScramLoginModule
required username=\"x\" password=\"x\";"
}
}' | jq .
curl -X POST http://10.xx.xx.xx:8083/connectors -H "Content-Type:
application/json" -d '{
"name":"mm2-hbc",
"config":{
"connector.class":"org.apache.kafka.connect.mirror.MirrorHeartbeatConnector",
"clusters":"mysource,mydestination",
