[jira] [Commented] (KAFKA-14740) Missing source tag on MirrorSource metrics
[ https://issues.apache.org/jira/browse/KAFKA-14740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692252#comment-17692252 ] Ryanne Dolan commented on KAFKA-14740: -- [~mimaison] the topic name usually includes the source cluster already, so I figured it was redundant. With identity replication you don't get that, but you presumably know what the source is in such cases. I don't have any objections to adding it tho. > Missing source tag on MirrorSource metrics > -- > > Key: KAFKA-14740 > URL: https://issues.apache.org/jira/browse/KAFKA-14740 > Project: Kafka > Issue Type: Improvement > Components: mirrormaker >Reporter: Mickael Maison >Priority: Major > > The metrics defined in MirrorSourceMetrics have the following tags "target", > "topic", "partition". It would be good to also have a "source" tag with the > source cluster alias. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[ https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17500205#comment-17500205 ] Ryanne Dolan commented on KAFKA-7500: - mm2 will not skip records. > MirrorMaker 2.0 (KIP-382) > - > > Key: KAFKA-7500 > URL: https://issues.apache.org/jira/browse/KAFKA-7500 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect, mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Major > Labels: pull-request-available, ready-to-commit > Fix For: 2.4.0 > > Attachments: Active-Active XDCR setup.png > > > Implement a drop-in replacement for MirrorMaker leveraging the Connect > framework. > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0] > [https://github.com/apache/kafka/pull/6295] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[ https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17499835#comment-17499835 ] Ryanne Dolan commented on KAFKA-7500: - [~gsavinov] mm2 cannot guarantee that the downstream topics are the same as the upstream topics if there are other producers sending to those topics, obviously. But if that isn't a concern, there is no technical reason you can't have other producers. The offset mappings will still work, for example. > MirrorMaker 2.0 (KIP-382) > - > > Key: KAFKA-7500 > URL: https://issues.apache.org/jira/browse/KAFKA-7500 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect, mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Major > Labels: pull-request-available, ready-to-commit > Fix For: 2.4.0 > > Attachments: Active-Active XDCR setup.png > > > Implement a drop-in replacement for MirrorMaker leveraging the Connect > framework. > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0] > [https://github.com/apache/kafka/pull/6295] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (KAFKA-13255) Mirrormaker config property config.properties.exclude is not working as expected
[
https://issues.apache.org/jira/browse/KAFKA-13255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17407742#comment-17407742
]
Ryanne Dolan commented on KAFKA-13255:
--
hey [~anadkarni], you'll need to create a pull request on github
(https://github.com/apache/kafka) so that we can review and merge the changes.
Let me know if you need help.
> Mirrormaker config property config.properties.exclude is not working as
> expected
> -
>
> Key: KAFKA-13255
> URL: https://issues.apache.org/jira/browse/KAFKA-13255
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 2.8.0
>Reporter: Anamika Nadkarni
>Priority: Major
>
> Objective - Use MM2 (kafka connect in distributed cluster) for data migration
> between cluster hosted in private data center and aws msk cluster.
> Steps performed -
> # Started kafka-connect service.
> # Created 3 MM2 connectors (i.e. source connector, checkpoint connector and
> heartbeat connector). Curl commands used to create connectors are in the
> attached file. To exclude certain config properties while topic replication,
> we are using the 'config.properties.exclude' property in the MM2 source
> connector.
> Expected -
> Source topic 'dev.portlandDc.anamika.helloMsk' should be successfully created
> in destination cluster.
> Actual -
> Creation of the source topic 'dev.portlandDc.anamika.helloMsk' in destination
> cluster fails with an error. Error is
> {code:java}
> [2021-08-06 06:13:40,944] WARN [mm2-msc|worker] Could not create topic
> dev.portlandDc.anamika.helloMsk.
> (org.apache.kafka.connect.mirror.MirrorSourceConnector:371)
> org.apache.kafka.common.errors.InvalidConfigurationException: Unknown topic
> config name: confluent.value.schema.validation{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KAFKA-13038) document IdentityReplicationPolicy
[ https://issues.apache.org/jira/browse/KAFKA-13038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryanne Dolan reassigned KAFKA-13038: Assignee: Ryanne Dolan > document IdentityReplicationPolicy > -- > > Key: KAFKA-13038 > URL: https://issues.apache.org/jira/browse/KAFKA-13038 > Project: Kafka > Issue Type: Task > Components: mirrormaker >Affects Versions: 3.0.0 >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Minor > > We should add something to Geo-Replication section of the docs to introduce > IdentityReplicationPolicy. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-13038) document IdentityReplicationPolicy
[ https://issues.apache.org/jira/browse/KAFKA-13038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryanne Dolan updated KAFKA-13038: - Issue Type: Task (was: Bug) > document IdentityReplicationPolicy > -- > > Key: KAFKA-13038 > URL: https://issues.apache.org/jira/browse/KAFKA-13038 > Project: Kafka > Issue Type: Task > Components: mirrormaker >Affects Versions: 3.0.0 >Reporter: Ryanne Dolan >Priority: Minor > > We should add something to Geo-Replication section of the docs to introduce > IdentityReplicationPolicy. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-13038) document IdentityReplicationPolicy
Ryanne Dolan created KAFKA-13038: Summary: document IdentityReplicationPolicy Key: KAFKA-13038 URL: https://issues.apache.org/jira/browse/KAFKA-13038 Project: Kafka Issue Type: Bug Components: mirrormaker Affects Versions: 3.0.0 Reporter: Ryanne Dolan We should add something to Geo-Replication section of the docs to introduce IdentityReplicationPolicy. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-9726) LegacyReplicationPolicy for MM2 to mimic MM1
[ https://issues.apache.org/jira/browse/KAFKA-9726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryanne Dolan updated KAFKA-9726: Fix Version/s: 3.0.0 > LegacyReplicationPolicy for MM2 to mimic MM1 > > > Key: KAFKA-9726 > URL: https://issues.apache.org/jira/browse/KAFKA-9726 > Project: Kafka > Issue Type: Improvement > Components: mirrormaker >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Minor > Fix For: 3.0.0 > > > Per KIP-382, we should support MM2 in "legacy mode", i.e. with behavior > similar to MM1. A key requirement for this is a ReplicationPolicy that does > not rename topics. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-9726) LegacyReplicationPolicy for MM2 to mimic MM1
[ https://issues.apache.org/jira/browse/KAFKA-9726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryanne Dolan updated KAFKA-9726: Priority: Major (was: Minor) > LegacyReplicationPolicy for MM2 to mimic MM1 > > > Key: KAFKA-9726 > URL: https://issues.apache.org/jira/browse/KAFKA-9726 > Project: Kafka > Issue Type: Improvement > Components: mirrormaker >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Major > Fix For: 3.0.0 > > > Per KIP-382, we should support MM2 in "legacy mode", i.e. with behavior > similar to MM1. A key requirement for this is a ReplicationPolicy that does > not rename topics. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9726) IdentityReplicationPolicy for MM2 to mimic MM1
[ https://issues.apache.org/jira/browse/KAFKA-9726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17357014#comment-17357014 ] Ryanne Dolan commented on KAFKA-9726: - PR ready for merge. > IdentityReplicationPolicy for MM2 to mimic MM1 > -- > > Key: KAFKA-9726 > URL: https://issues.apache.org/jira/browse/KAFKA-9726 > Project: Kafka > Issue Type: Improvement > Components: mirrormaker >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Major > Fix For: 3.0.0 > > > Per KIP-382, we should support MM2 in "legacy mode", i.e. with behavior > similar to MM1. A key requirement for this is a ReplicationPolicy that does > not rename topics. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-9726) IdentityReplicationPolicy for MM2 to mimic MM1
[ https://issues.apache.org/jira/browse/KAFKA-9726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryanne Dolan updated KAFKA-9726: Summary: IdentityReplicationPolicy for MM2 to mimic MM1 (was: LegacyReplicationPolicy for MM2 to mimic MM1) > IdentityReplicationPolicy for MM2 to mimic MM1 > -- > > Key: KAFKA-9726 > URL: https://issues.apache.org/jira/browse/KAFKA-9726 > Project: Kafka > Issue Type: Improvement > Components: mirrormaker >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Major > Fix For: 3.0.0 > > > Per KIP-382, we should support MM2 in "legacy mode", i.e. with behavior > similar to MM1. A key requirement for this is a ReplicationPolicy that does > not rename topics. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KAFKA-9726) LegacyReplicationPolicy for MM2 to mimic MM1
[ https://issues.apache.org/jira/browse/KAFKA-9726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryanne Dolan reassigned KAFKA-9726: --- Assignee: Ryanne Dolan (was: Matthew de Detrich) > LegacyReplicationPolicy for MM2 to mimic MM1 > > > Key: KAFKA-9726 > URL: https://issues.apache.org/jira/browse/KAFKA-9726 > Project: Kafka > Issue Type: Improvement > Components: mirrormaker >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Minor > > Per KIP-382, we should support MM2 in "legacy mode", i.e. with behavior > similar to MM1. A key requirement for this is a ReplicationPolicy that does > not rename topics. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-7815) SourceTask should expose ACK'd offsets, metadata
[ https://issues.apache.org/jira/browse/KAFKA-7815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryanne Dolan resolved KAFKA-7815. - Fix Version/s: 2.4.0 Resolution: Fixed Equivalent functionality was included as part of KIP-382. > SourceTask should expose ACK'd offsets, metadata > > > Key: KAFKA-7815 > URL: https://issues.apache.org/jira/browse/KAFKA-7815 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Minor > Labels: needs-kip > Fix For: 2.4.0 > > Original Estimate: 24h > Remaining Estimate: 24h > > Add a new callback method, recordLogged(), to notify SourceTasks when a > record is ACK'd by the downstream broker. Include offsets and metadata of > ACK'd record. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12436) deprecate MirrorMaker v1
[ https://issues.apache.org/jira/browse/KAFKA-12436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355841#comment-17355841 ] Ryanne Dolan commented on KAFKA-12436: -- [~ijuma] can I get a review please? > deprecate MirrorMaker v1 > > > Key: KAFKA-12436 > URL: https://issues.apache.org/jira/browse/KAFKA-12436 > Project: Kafka > Issue Type: Improvement > Components: mirrormaker >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Minor > Fix For: 3.0.0 > > > Per KIP-382, the old MirrorMaker code (MM1) should be deprecated and > subsequently removed. Targeting upcoming release 3.0.0, we should mark > mirror-maker as deprecated, but leave code in place until subsequent major > release (4.0.0). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-12436) deprecate MirrorMaker v1
[ https://issues.apache.org/jira/browse/KAFKA-12436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryanne Dolan updated KAFKA-12436: - Reviewer: Ismael Juma (was: Konstantine Karantasis) > deprecate MirrorMaker v1 > > > Key: KAFKA-12436 > URL: https://issues.apache.org/jira/browse/KAFKA-12436 > Project: Kafka > Issue Type: Improvement > Components: mirrormaker >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Minor > Fix For: 3.0.0 > > > Per KIP-382, the old MirrorMaker code (MM1) should be deprecated and > subsequently removed. Targeting upcoming release 3.0.0, we should mark > mirror-maker as deprecated, but leave code in place until subsequent major > release (4.0.0). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12430) emit.heartbeats.enabled = false should disable heartbeats topic creation
[
https://issues.apache.org/jira/browse/KAFKA-12430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17351104#comment-17351104
]
Ryanne Dolan commented on KAFKA-12430:
--
Hmm. I guess the downside to not creating the topics is that upstreamClusters
etc won't work. They don't depend on actual records, just the topics. I don't
have any objections but it's a consideration.
> emit.heartbeats.enabled = false should disable heartbeats topic creation
>
>
> Key: KAFKA-12430
> URL: https://issues.apache.org/jira/browse/KAFKA-12430
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Reporter: Ivan Yurchenko
>Assignee: Matthew de Detrich
>Priority: Minor
>
> Currently, whether MirrorMaker 2's {{MirrorHeartbeatConnector}} emits
> heartbeats or not is based on {{emit.heartbeats.enabled}} setting. However,
> {{heartbeats}} topic is created unconditionally. It seems that the same
> setting should really disable the topic creation as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-12838) Kafka Broker - Request threads inefficiently blocking during produce
[
https://issues.apache.org/jira/browse/KAFKA-12838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350606#comment-17350606
]
Ryanne Dolan commented on KAFKA-12838:
--
Would it help to significantly increase the number of partitions you're writing
to?
> Kafka Broker - Request threads inefficiently blocking during produce
>
>
> Key: KAFKA-12838
> URL: https://issues.apache.org/jira/browse/KAFKA-12838
> Project: Kafka
> Issue Type: Improvement
> Components: core
>Affects Versions: 2.7.0, 2.8.0
>Reporter: Ryan Cabral
>Priority: Major
>
> Hello, I have been using Kafka brokers for a bit and have run into a problem
> with the way a kafka broker handles produce requests. If there are multiple
> producers to the same topic and partition, any request handler threads
> handling the produce for that topic and partition become blocked until all
> requests before it are done. Request handler threads for the entire broker
> can become exhausted waiting on the same partition lock, blocking requests
> for other partitions that would not have needed the same lock.
> Once that starts happening, requests start to back up, queued requests can
> reach its maximum and network threads begin to be paused cascading the
> problem a bit more. Overall performance ends up being degraded. I'm not so
> focused on the cascade at the moment as I am the initial contention.
> Intuitively I would expect locking contention on a single partition to ONLY
> affect throughput on that partition and not the entire broker.
>
> The append call within the request handler originates here:
> [https://github.com/apache/kafka/blob/2.8.0/core/src/main/scala/kafka/server/KafkaApis.scala#L638]
> Further down the stack the lock during append is created here:
> [https://github.com/apache/kafka/blob/2.8.0/core/src/main/scala/kafka/log/Log.scala#L1165]
> At this point the first request will hold the lock during append and future
> requests on the same partition will block, waiting for the lock, tying up an
> io thread (request handler).
> At first glance, it seems like it would make the most sense to (via config?)
> be able to funnel (produce) requests for the same partition through its own
> request queue of sorts and dispatch them such that at most one io thread is
> tied up at a time for a given partition. There are a number of reasons the
> lock could be held elsewhere too but this should at least help mitigate the
> issue a bit. I'm assuming this is easier said than done though and likely
> requires significant refactoring to properly achieve but hoping this is
> something that could end up on some sort of long term roadmap.
>
> Snippet from jstack. Almost all request handlers threads (there are 256 of
> them, up from 25 to mitigate the issue) in the jstack are blocked waiting on
> the same lock due to the number of producers we have.
>
> {noformat}
> "data-plane-kafka-request-handler-254" #335 daemon prio=5 os_prio=0
> tid=0x7fb1c9f13000 nid=0x53f1 runnable [0x7fad35796000]
> java.lang.Thread.State: RUNNABLE
> at
> org.apache.kafka.common.record.KafkaLZ4BlockOutputStream.(KafkaLZ4BlockOutputStream.java:82)
> at
> org.apache.kafka.common.record.KafkaLZ4BlockOutputStream.(KafkaLZ4BlockOutputStream.java:125)
> at
> org.apache.kafka.common.record.CompressionType$4.wrapForOutput(CompressionType.java:101)
> at
> org.apache.kafka.common.record.MemoryRecordsBuilder.(MemoryRecordsBuilder.java:134)
> at
> org.apache.kafka.common.record.MemoryRecordsBuilder.(MemoryRecordsBuilder.java:170)
> at
> org.apache.kafka.common.record.MemoryRecords.builder(MemoryRecords.java:508)
> at
> kafka.log.LogValidator$.buildRecordsAndAssignOffsets(LogValidator.scala:500)
> at
> kafka.log.LogValidator$.validateMessagesAndAssignOffsetsCompressed(LogValidator.scala:455)
> at
> kafka.log.LogValidator$.validateMessagesAndAssignOffsets(LogValidator.scala:106)
> at kafka.log.Log.$anonfun$append$2(Log.scala:1126)
> - locked <0x0004c9a6fd60> (a java.lang.Object)
> at kafka.log.Log.append(Log.scala:2387)
> at kafka.log.Log.appendAsLeader(Log.scala:1050)
> at
> kafka.cluster.Partition.$anonfun$appendRecordsToLeader$1(Partition.scala:1079)
> at kafka.cluster.Partition.appendRecordsToLeader(Partition.scala:1067)
> at
> kafka.server.ReplicaManager.$anonfun$appendToLocalLog$4(ReplicaManager.scala:953)
> at kafka.server.ReplicaManager$$Lambda$1078/1017241486.apply(Unknown
> Source)
> at
> scala.collection.StrictOptimizedMapOps.map(StrictOptimizedMapOps.scala:28)
> at
> scala.collection.StrictOptimizedMapOps.map$(StrictOptimizedMapOps.scala:27)
> at
[jira] [Commented] (KAFKA-12726) misbehaving Task.stop() can prevent other Tasks from stopping
[ https://issues.apache.org/jira/browse/KAFKA-12726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17338575#comment-17338575 ] Ryanne Dolan commented on KAFKA-12726: -- Relevant tests here: https://github.com/apache/kafka/pull/10629 > misbehaving Task.stop() can prevent other Tasks from stopping > - > > Key: KAFKA-12726 > URL: https://issues.apache.org/jira/browse/KAFKA-12726 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Affects Versions: 2.8.0 >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Minor > > We've observed a misbehaving Task fail to stop in a timely manner (e.g. stuck > in a retry loop). Despite Connect supporting a property > task.shutdown.graceful.timeout.ms, this is currently not enforced – tasks can > take as long as they want to stop, and the only consequence is an error > message. > We've seen a Worker's "task-count" metric double following a rebalance, which > we think is due to Tasks not getting cleaned up when Task.stop() is stuck. > While the Connector implementation is ultimately to blame here – a Task > probably shouldn't loop forever in stop() – we believe the Connect runtime > should handle this situation more gracefully. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-12726) misbehaving Task.stop() can prevent other Tasks from stopping
[ https://issues.apache.org/jira/browse/KAFKA-12726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryanne Dolan resolved KAFKA-12726. -- Resolution: Fixed Closing as duplicate. > misbehaving Task.stop() can prevent other Tasks from stopping > - > > Key: KAFKA-12726 > URL: https://issues.apache.org/jira/browse/KAFKA-12726 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Affects Versions: 2.8.0 >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Minor > > We've observed a misbehaving Task fail to stop in a timely manner (e.g. stuck > in a retry loop). Despite Connect supporting a property > task.shutdown.graceful.timeout.ms, this is currently not enforced – tasks can > take as long as they want to stop, and the only consequence is an error > message. > We've seen a Worker's "task-count" metric double following a rebalance, which > we think is due to Tasks not getting cleaned up when Task.stop() is stuck. > While the Connector implementation is ultimately to blame here – a Task > probably shouldn't loop forever in stop() – we believe the Connect runtime > should handle this situation more gracefully. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-12726) misbehaving Task.stop() can prevent other Tasks from stopping
[ https://issues.apache.org/jira/browse/KAFKA-12726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryanne Dolan updated KAFKA-12726: - Description: We've observed a misbehaving Task fail to stop in a timely manner (e.g. stuck in a retry loop). Despite Connect supporting a property task.shutdown.graceful.timeout.ms, this is currently not enforced – tasks can take as long as they want to stop, and the only consequence is an error message. We've seen a Worker's "task-count" metric double following a rebalance, which we think is due to Tasks not getting cleaned up when Task.stop() is stuck. While the Connector implementation is ultimately to blame here – a Task probably shouldn't loop forever in stop() – we believe the Connect runtime should handle this situation more gracefully. was: We've observed a misbehaving Task fail to stop in a timely manner (e.g. stuck in a retry loop). Despite Connect supporting a property task.shutdown.graceful.timeout.ms, this is currently not enforced -- tasks can take as long as they want to stop, and the only consequence is an error message. Unfortunately, Workers stop Tasks sequentially, meaning that a stuck Task can prevent any further Tasks from stopping. Moreover, after a rebalance, these lingering tasks can persist along with their replacements. For example, we've seen a Worker's "task-count" metric double following a rebalance. While the Connector implementation is ultimately to blame here -- a Task probably shouldn't loop forever in stop() -- we believe the Connect runtime should handle this situation more gracefully. > misbehaving Task.stop() can prevent other Tasks from stopping > - > > Key: KAFKA-12726 > URL: https://issues.apache.org/jira/browse/KAFKA-12726 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Affects Versions: 2.8.0 >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Minor > > We've observed a misbehaving Task fail to stop in a timely manner (e.g. stuck > in a retry loop). Despite Connect supporting a property > task.shutdown.graceful.timeout.ms, this is currently not enforced – tasks can > take as long as they want to stop, and the only consequence is an error > message. > We've seen a Worker's "task-count" metric double following a rebalance, which > we think is due to Tasks not getting cleaned up when Task.stop() is stuck. > While the Connector implementation is ultimately to blame here – a Task > probably shouldn't loop forever in stop() – we believe the Connect runtime > should handle this situation more gracefully. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12726) misbehaving Task.stop() can prevent other Tasks from stopping
[ https://issues.apache.org/jira/browse/KAFKA-12726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337603#comment-17337603 ] Ryanne Dolan commented on KAFKA-12726: -- Thanks for the detailed analysis [~ChrisEgerton]. I added some tests to BlockingConnectorTests that try to recreate the issue, but the tests all pass against trunk. Checking with me team, turns out we were running an experimental SourceConnector at the time. So I think it's very likely that this was fixed in KAFKA-10792 after all. I'll close this ticket and the PR to avoid confusion, and will open a new PR for the additional tests in a bit. > misbehaving Task.stop() can prevent other Tasks from stopping > - > > Key: KAFKA-12726 > URL: https://issues.apache.org/jira/browse/KAFKA-12726 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Affects Versions: 2.8.0 >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Minor > > We've observed a misbehaving Task fail to stop in a timely manner (e.g. stuck > in a retry loop). Despite Connect supporting a property > task.shutdown.graceful.timeout.ms, this is currently not enforced -- tasks > can take as long as they want to stop, and the only consequence is an error > message. > Unfortunately, Workers stop Tasks sequentially, meaning that a stuck Task can > prevent any further Tasks from stopping. Moreover, after a rebalance, these > lingering tasks can persist along with their replacements. For example, we've > seen a Worker's "task-count" metric double following a rebalance. > While the Connector implementation is ultimately to blame here -- a Task > probably shouldn't loop forever in stop() -- we believe the Connect runtime > should handle this situation more gracefully. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12726) misbehaving Task.stop() can prevent other Tasks from stopping
[ https://issues.apache.org/jira/browse/KAFKA-12726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17336550#comment-17336550 ] Ryanne Dolan commented on KAFKA-12726: -- [~ChrisEgerton] Yeah, the problem is that we were seeing, say, 100 tasks on each Worker, then a rebalance, then 200 tasks per Worker (as reported by the tasks-count metric) with nothing to do but restart each Worker -- which, ofc, would cause further rebalances! I'm not proposing we interrupt any threads here. I agree with you that it's reasonable to just leak a thread if a Task impl is stuck indefinitely. But we can leak a stuck thread while cleaning up everything around it. I'm proposing we continue with the WorkerTask shutdown after the grace period, which includes removing the WorkerTask from the list of current tasks (and thus the tasks-count metric). > misbehaving Task.stop() can prevent other Tasks from stopping > - > > Key: KAFKA-12726 > URL: https://issues.apache.org/jira/browse/KAFKA-12726 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Affects Versions: 2.8.0 >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Minor > > We've observed a misbehaving Task fail to stop in a timely manner (e.g. stuck > in a retry loop). Despite Connect supporting a property > task.shutdown.graceful.timeout.ms, this is currently not enforced -- tasks > can take as long as they want to stop, and the only consequence is an error > message. > Unfortunately, Workers stop Tasks sequentially, meaning that a stuck Task can > prevent any further Tasks from stopping. Moreover, after a rebalance, these > lingering tasks can persist along with their replacements. For example, we've > seen a Worker's "task-count" metric double following a rebalance. > While the Connector implementation is ultimately to blame here -- a Task > probably shouldn't loop forever in stop() -- we believe the Connect runtime > should handle this situation more gracefully. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12726) misbehaving Task.stop() can prevent other Tasks from stopping
[ https://issues.apache.org/jira/browse/KAFKA-12726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17335592#comment-17335592 ] Ryanne Dolan commented on KAFKA-12726: -- [~ChrisEgerton] ah, indeed I have conflated WorkerTask.stop() and Task.stop(). If I'm (re-)reading this correctly, Task.stop() is called from WorkerTask.close() [here|https://github.com/apache/kafka/blob/f9de25f046452b2a6d916e6bca41e31d49bbdecf/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L169] (not WorkTask.stop()...) which is called at the end of the WorkerTask's main loop [here|https://github.com/apache/kafka/blob/f9de25f046452b2a6d916e6bca41e31d49bbdecf/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerTask.java#L197]. wrt KAFKA-10792, we've observed the problem while only running SinkConnectors, so I don't think that particular fix can be related. So it appears the problem still exists -- a stuck Task.stop() will prevent the WorkerTask from closing, as we've observed in production -- but it's clear I'm mistaken about 1) my remark that Task.stop()s are sequential (they are not) and 2) where this fix needs to go :) Lemme relocate this logic to WorkerTask.doClose(). However, this begs the question: should we be doing this for every Task method? Seems any stuck method would yield the same behavior. wrt BlockingConnectorTest, that is indeed where I started my investigation, but the existing tests don't seem to capture this issue. I'll see if I can add a test to repro and show it failing. My understanding is that BlockingConnectorTest is only testing whether subsequent Tasks can created and run, but doesn't test that stopped Tasks are ever actually stopped: https://github.com/apache/kafka/blob/f9de25f046452b2a6d916e6bca41e31d49bbdecf/connect/runtime/src/test/java/org/apache/kafka/connect/integration/BlockingConnectorTest.java#L219 In that test, I believe the blocked Task will be leaked, which is what we're observing in production. > misbehaving Task.stop() can prevent other Tasks from stopping > - > > Key: KAFKA-12726 > URL: https://issues.apache.org/jira/browse/KAFKA-12726 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Affects Versions: 2.8.0 >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Minor > > We've observed a misbehaving Task fail to stop in a timely manner (e.g. stuck > in a retry loop). Despite Connect supporting a property > task.shutdown.graceful.timeout.ms, this is currently not enforced -- tasks > can take as long as they want to stop, and the only consequence is an error > message. > Unfortunately, Workers stop Tasks sequentially, meaning that a stuck Task can > prevent any further Tasks from stopping. Moreover, after a rebalance, these > lingering tasks can persist along with their replacements. For example, we've > seen a Worker's "task-count" metric double following a rebalance. > While the Connector implementation is ultimately to blame here -- a Task > probably shouldn't loop forever in stop() -- we believe the Connect runtime > should handle this situation more gracefully. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KAFKA-12726) misbehaving Task.stop() can prevent other Tasks from stopping
[ https://issues.apache.org/jira/browse/KAFKA-12726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryanne Dolan reassigned KAFKA-12726: Assignee: Ryanne Dolan > misbehaving Task.stop() can prevent other Tasks from stopping > - > > Key: KAFKA-12726 > URL: https://issues.apache.org/jira/browse/KAFKA-12726 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Affects Versions: 2.8.0 >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Minor > > We've observed a misbehaving Task fail to stop in a timely manner (e.g. stuck > in a retry loop). Despite Connect supporting a property > task.shutdown.graceful.timeout.ms, this is currently not enforced -- tasks > can take as long as they want to stop, and the only consequence is an error > message. > Unfortunately, Workers stop Tasks sequentially, meaning that a stuck Task can > prevent any further Tasks from stopping. Moreover, after a rebalance, these > lingering tasks can persist along with their replacements. For example, we've > seen a Worker's "task-count" metric double following a rebalance. > While the Connector implementation is ultimately to blame here -- a Task > probably shouldn't loop forever in stop() -- we believe the Connect runtime > should handle this situation more gracefully. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-12726) misbehaving Task.stop() can prevent other Tasks from stopping
Ryanne Dolan created KAFKA-12726: Summary: misbehaving Task.stop() can prevent other Tasks from stopping Key: KAFKA-12726 URL: https://issues.apache.org/jira/browse/KAFKA-12726 Project: Kafka Issue Type: Bug Components: KafkaConnect Affects Versions: 2.8.0 Reporter: Ryanne Dolan We've observed a misbehaving Task fail to stop in a timely manner (e.g. stuck in a retry loop). Despite Connect supporting a property task.shutdown.graceful.timeout.ms, this is currently not enforced -- tasks can take as long as they want to stop, and the only consequence is an error message. Unfortunately, Workers stop Tasks sequentially, meaning that a stuck Task can prevent any further Tasks from stopping. Moreover, after a rebalance, these lingering tasks can persist along with their replacements. For example, we've seen a Worker's "task-count" metric double following a rebalance. While the Connector implementation is ultimately to blame here -- a Task probably shouldn't loop forever in stop() -- we believe the Connect runtime should handle this situation more gracefully. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KAFKA-12558) MM2 may not sync partition offsets correctly
[
https://issues.apache.org/jira/browse/KAFKA-12558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryanne Dolan reassigned KAFKA-12558:
Assignee: Ryanne Dolan
> MM2 may not sync partition offsets correctly
>
>
> Key: KAFKA-12558
> URL: https://issues.apache.org/jira/browse/KAFKA-12558
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 2.7.0, 2.6.1
>Reporter: Alan
>Assignee: Ryanne Dolan
>Priority: Major
>
> There is a race condition in {{MirrorSourceTask}} where certain partition
> offsets may never be sent. The bug occurs when the [outstandingOffsetSync
> semaphore is
> full|https://github.com/apache/kafka/blob/trunk/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceTask.java#L207].
> In this case, the sendOffsetSync [will silently
> fail|https://github.com/apache/kafka/blob/trunk/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceTask.java#L207].
> This failure is normally acceptable since offset sync will retry frequently.
> However, {{maybeSyncOffsets}} has a bug where it will [mutate the partition
> state|https://github.com/apache/kafka/blob/trunk/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceTask.java#L199]
> prior to confirming the result of {{sendOffsetSync}}. The end result is that
> the partition state is mutated prematurely, and prevent future offset syncs
> to recover.
> Since {{MAX_OUTSTANDING_OFFSET_SYNCS}} is 10, this bug happens when you
> assign more than 10 partitions to each task.
> In my test cases where I had over 100 partitions per task, the majority of
> the offsets were wrong. Here's an example of such a failure.
> https://issues.apache.org/jira/browse/KAFKA-12468?focusedCommentId=17308308=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17308308
> During my troubleshooting, I customized the {{MirrorSourceTask}} to confirm
> that all partitions that have the wrong offset were failing to acquire the
> initial semaphore. The condition [can be trapped
> here|https://github.com/apache/kafka/blob/trunk/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceTask.java#L208].
> *Possible Fix:*
> A possible fix is to create a {{shouldUpdate}} method in {{PartitionState}}.
> This method should be read-only and return true if {{sendOffsetSync}} is
> needed. Once {{sendOffsetSync}} is successful, only then {{update}} should be
> called.
> Here's some pseudocode
> {code:java}
> private void maybeSyncOffsets(TopicPartition topicPartition, long
> upstreamOffset,
> long downstreamOffset) {
> PartitionState partitionState =
> partitionStates.computeIfAbsent(topicPartition, x -> new
> PartitionState(maxOffsetLag));
> if (partitionState.shouldUpdate(upstreamOffset, downstreamOffset)) {
> if(sendOffsetSync(topicPartition, upstreamOffset, downstreamOffset)) {
> partitionState.update(upstreamOffset, downstreamOffset)
> }
> }
> }
> {code}
>
> *Workaround:*
> For those who are experiencing this issue, the workaround is to make sure you
> have less than or equal to 10 partitions per task. Set your `tasks.max` value
> accordingly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-12645) KIP-731: Record Rate Limiting for Kafka Connect
Ryanne Dolan created KAFKA-12645: Summary: KIP-731: Record Rate Limiting for Kafka Connect Key: KAFKA-12645 URL: https://issues.apache.org/jira/browse/KAFKA-12645 Project: Kafka Issue Type: Improvement Reporter: Ryanne Dolan Assignee: Ryanne Dolan https://cwiki.apache.org/confluence/display/KAFKA/KIP-731%3A+Record+Rate+Limiting+for+Kafka+Connect -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12563) Something wrong with MM2 metrics
[ https://issues.apache.org/jira/browse/KAFKA-12563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309447#comment-17309447 ] Ryanne Dolan commented on KAFKA-12563: -- You can look at the lag of the internal consumers to get an idea of how far behind the connectors are from real time. But I agree that a metric like you describe would be useful. Are you interested in proposing a KIP? Can you close this ticket? > Something wrong with MM2 metrics > > > Key: KAFKA-12563 > URL: https://issues.apache.org/jira/browse/KAFKA-12563 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 2.7.0 >Reporter: Bui Thanh MInh >Priority: Major > Attachments: Screen Shot 2021-03-26 at 12.10.12.png > > > The metric > _*`adt_2dc_c1_kafka_connect_mirror_source_connector_replication_latency_ms_avg`*_ > shows that value of latency is a very large number but the amount of > messages in two DC are the same. > View details in the attachment. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12563) Something wrong with MM2 metrics
[ https://issues.apache.org/jira/browse/KAFKA-12563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309191#comment-17309191 ] Ryanne Dolan commented on KAFKA-12563: -- The metric is calculated based on the timestamp of each replicated record, which can be set by client code. e.g. a producer can set a timestamp of zero, which would yield the replication latency you observe. > Something wrong with MM2 metrics > > > Key: KAFKA-12563 > URL: https://issues.apache.org/jira/browse/KAFKA-12563 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 2.7.0 >Reporter: Bui Thanh MInh >Priority: Major > Attachments: Screen Shot 2021-03-26 at 12.10.12.png > > > The metric > _*`adt_2dc_c1_kafka_connect_mirror_source_connector_replication_latency_ms_avg`*_ > shows that value of latency is a very large number but the amount of > messages in two DC are the same. > View details in the attachment. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12558) MM2 may not sync partition offsets correctly
[
https://issues.apache.org/jira/browse/KAFKA-12558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308968#comment-17308968
]
Ryanne Dolan commented on KAFKA-12558:
--
This sounds right. Updating the partition store without sending an offset sync
means additional offset syncs are unlikely to arrive for a long time, if ever.
I'm happy to fix but will leave this unassigned for a bit in case someone else
wants to take this on.
> MM2 may not sync partition offsets correctly
>
>
> Key: KAFKA-12558
> URL: https://issues.apache.org/jira/browse/KAFKA-12558
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 2.7.0, 2.6.1
>Reporter: Alan
>Priority: Major
>
> There is a race condition in {{MirrorSourceTask}} where certain partition
> offsets may never be sent. The bug occurs when the [outstandingOffsetSync
> semaphore is
> full|https://github.com/apache/kafka/blob/trunk/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceTask.java#L207].
> In this case, the sendOffsetSync [will silently
> fail|https://github.com/apache/kafka/blob/trunk/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceTask.java#L207].
> This failure is normally acceptable since offset sync will retry frequently.
> However, {{maybeSyncOffsets}} has a bug where it will [mutate the partition
> state|https://github.com/apache/kafka/blob/trunk/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceTask.java#L199]
> prior to confirming the result of {{sendOffsetSync}}. The end result is that
> the partition state is mutated prematurely, and prevent future offset syncs
> to recover.
> Since {{MAX_OUTSTANDING_OFFSET_SYNCS}} is 10, this bug happens when you
> assign more than 10 partitions to each task.
> In my test cases where I had over 100 partitions per task, the majority of
> the offsets were wrong. Here's an example of such a failure.
> https://issues.apache.org/jira/browse/KAFKA-12468?focusedCommentId=17308308=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17308308
> During my troubleshooting, I customized the {{MirrorSourceTask}} to confirm
> that all partitions that have the wrong offset were failing to acquire the
> initial semaphore. The condition [can be trapped
> here|https://github.com/apache/kafka/blob/trunk/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceTask.java#L208].
> *Possible Fix:*
> A possible fix is to create a {{shouldUpdate}} method in {{PartitionState}}.
> This method should be read-only and return true if {{sendOffsetSync}} is
> needed. Once {{sendOffsetSync}} is successful, only then {{update}} should be
> called.
> Here's some pseudocode
> {code:java}
> private void maybeSyncOffsets(TopicPartition topicPartition, long
> upstreamOffset,
> long downstreamOffset) {
> PartitionState partitionState =
> partitionStates.computeIfAbsent(topicPartition, x -> new
> PartitionState(maxOffsetLag));
> if (partitionState.shouldUpdate(upstreamOffset, downstreamOffset)) {
> if(sendOffsetSync(topicPartition, upstreamOffset, downstreamOffset)) {
> partitionState.update(upstreamOffset, downstreamOffset)
> }
> }
> }
> {code}
>
> *Workaround:*
> For those who are experiencing this issue, the workaround is to make sure you
> have less than or equal to 10 partitions per task. Set your `tasks.max` value
> accordingly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-12436) deprecate MirrorMaker v1
[ https://issues.apache.org/jira/browse/KAFKA-12436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryanne Dolan updated KAFKA-12436: - Description: Per KIP-382, the old MirrorMaker code (MM1) should be deprecated and subsequently removed. Targeting upcoming release 3.0.0, we should mark mirror-maker as deprecated, but leave code in place until subsequent major release (4.0.0). (was: Per KIP-382, the old MirrorMaker code (MM1) should be deprecated and subsequently removed. Targeting upcoming release 3.0.0, we should mark mirror-maker as deprecated, but leave code in place until subsequent release (ideally 3.1.0).) > deprecate MirrorMaker v1 > > > Key: KAFKA-12436 > URL: https://issues.apache.org/jira/browse/KAFKA-12436 > Project: Kafka > Issue Type: Improvement > Components: mirrormaker >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Minor > Fix For: 3.0.0 > > > Per KIP-382, the old MirrorMaker code (MM1) should be deprecated and > subsequently removed. Targeting upcoming release 3.0.0, we should mark > mirror-maker as deprecated, but leave code in place until subsequent major > release (4.0.0). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12436) deprecate MirrorMaker v1
[ https://issues.apache.org/jira/browse/KAFKA-12436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296773#comment-17296773 ] Ryanne Dolan commented on KAFKA-12436: -- Thanks I'll update the ticket. > deprecate MirrorMaker v1 > > > Key: KAFKA-12436 > URL: https://issues.apache.org/jira/browse/KAFKA-12436 > Project: Kafka > Issue Type: Improvement > Components: mirrormaker >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Minor > Fix For: 3.0.0 > > > Per KIP-382, the old MirrorMaker code (MM1) should be deprecated and > subsequently removed. Targeting upcoming release 3.0.0, we should mark > mirror-maker as deprecated, but leave code in place until subsequent release > (ideally 3.1.0). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-12436) deprecate MirrorMaker v1
Ryanne Dolan created KAFKA-12436: Summary: deprecate MirrorMaker v1 Key: KAFKA-12436 URL: https://issues.apache.org/jira/browse/KAFKA-12436 Project: Kafka Issue Type: Improvement Components: mirrormaker Reporter: Ryanne Dolan Assignee: Ryanne Dolan Fix For: 3.0.0 Per KIP-382, the old MirrorMaker code (MM1) should be deprecated and subsequently removed. Targeting upcoming release 3.0.0, we should mark mirror-maker as deprecated, but leave code in place until subsequent release (ideally 3.1.0). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10370) WorkerSinkTask: IllegalStateException cased by consumer.seek(tp, offsets) when (tp, offsets) are supplied by WorkerSinkTaskContext
[
https://issues.apache.org/jira/browse/KAFKA-10370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17179220#comment-17179220
]
Ryanne Dolan commented on KAFKA-10370:
--
Thanks [~yangguo1220]. I believe that using assign() instead of subscribe()
will have unexpected side effects, e.g. no rebalances and no auto-detection of
new topics/partitions. As you know, in MirrorSourceTask, we explicitly avoided
using subscribe() and instead handle rebalances and new topic-partitions
explicitly, for efficiency purposes. (Rebalances were a major problem in
production deployments of MM1.) But I'm not sure it would be appropriate to
change the behavior of WorkerSinkTask here, especially as a side-effect of
calling SinkTaskContext.offset().
IIRC, the root cause of the exception is actually that subscribe() results in
partitions being assigned asynchronously, so if you subscribe() and then seek()
you'll likely have zero assignments at that point. I believe the correct way to
deal with this is to register a RebalanceListener, which can rewind the offsets
of a partition _after_ the partition is assigned.
It may be possible for WorkerSinkTask to do this automatically. There are
basically two scenarios:
- SinkTaskContext.offset() is called _after_ a partition is assigned, in which
case the existing implementation should work. Unfortunately, it seems
impossible for a SinkTask to know whether the partition is assigned or not.
This to me seems like a bug in the API.
- SinkTaskContext.offset() is called _before_ a partition is assigned, which
would result in the exception you're seeing. In this case, WorkerSinkTask could
store the offsets and seek() asynchronously using a RebalanceListener. This
essentially defers the seek() until _after_ the partition is actually assigned,
thus avoiding the exception.
It's possible this bug only exists in the edge-case of calling
WorkerSinkTask.offsets() within the SinktTask.start() method. We could possibly
handle that case specially: if offsets() is called during start(),
WorkerSinkTask could use the RebalanceListener to defer the seek() until the
partitions are actually assigned.
> WorkerSinkTask: IllegalStateException cased by consumer.seek(tp, offsets)
> when (tp, offsets) are supplied by WorkerSinkTaskContext
> --
>
> Key: KAFKA-10370
> URL: https://issues.apache.org/jira/browse/KAFKA-10370
> Project: Kafka
> Issue Type: New Feature
> Components: KafkaConnect
>Affects Versions: 2.5.0
>Reporter: Ning Zhang
>Assignee: Ning Zhang
>Priority: Major
> Fix For: 2.7.0
>
>
> In
> [WorkerSinkTask.java|https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java],
> when we want the consumer to consume from certain offsets, rather than from
> the last committed offset,
> [WorkerSinkTaskContext|https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTaskContext.java#L63-L66]
> provided a way to supply the offsets from external (e.g. implementation of
> SinkTask) to rewind the consumer.
> In the [poll()
> method|https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L312],
> it first call
> [rewind()|https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L615-L633]
> to (1) read the offsets from WorkerSinkTaskContext, if the offsets are not
> empty, (2) consumer.seek(tp, offset) to rewind the consumer.
> As a part of [WorkerSinkTask
> initialization|https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L290-L307],
> when the [SinkTask
> starts|https://github.com/apache/kafka/blob/trunk/connect/api/src/main/java/org/apache/kafka/connect/sink/SinkTask.java#L83-L88],
> we can supply the specific offsets by +"context.offset(supplied_offsets);+"
> in start() method, so that when the consumer does the first poll, it should
> rewind to the specific offsets in rewind() method. However in practice, we
> saw the following IllegalStateException when running consumer.seek(tp,
> offsets);
> {code:java}
> [2020-08-07 23:53:55,752] INFO WorkerSinkTask{id=MirrorSinkConnector-0}
> Rewind test-1 to offset 3
> (org.apache.kafka.connect.runtime.WorkerSinkTask:648)
> [2020-08-07 23:53:55,752] INFO [Consumer
> clientId=connector-consumer-MirrorSinkConnector-0,
> groupId=connect-MirrorSinkConnector] Seeking to offset 3 for partition test-1
> (org.apache.kafka.clients.consumer.KafkaConsumer:1592)
> [2020-08-07 23:53:55,752] ERROR
[jira] [Commented] (KAFKA-9726) LegacyReplicationPolicy for MM2 to mimic MM1
[ https://issues.apache.org/jira/browse/KAFKA-9726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17179133#comment-17179133 ] Ryanne Dolan commented on KAFKA-9726: - [~ivanyu] that'd be great! > LegacyReplicationPolicy for MM2 to mimic MM1 > > > Key: KAFKA-9726 > URL: https://issues.apache.org/jira/browse/KAFKA-9726 > Project: Kafka > Issue Type: Improvement > Components: mirrormaker >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Minor > > Per KIP-382, we should support MM2 in "legacy mode", i.e. with behavior > similar to MM1. A key requirement for this is a ReplicationPolicy that does > not rename topics. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10339) MirrorMaker2 Exactly-once Semantics
[ https://issues.apache.org/jira/browse/KAFKA-10339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17171642#comment-17171642 ] Ryanne Dolan commented on KAFKA-10339: -- I believe your proposed task.loadOffsets() is taken care of by the existing SinkTaskContext.offsets() method actually. This mechanism is used in other SinkTasks where the offsets are stored in the downstream system. I think we may already have all the interfaces required to make this work. > MirrorMaker2 Exactly-once Semantics > --- > > Key: KAFKA-10339 > URL: https://issues.apache.org/jira/browse/KAFKA-10339 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect >Reporter: Ning Zhang >Assignee: Ning Zhang >Priority: Major > Labels: needs-kip > > MirrorMaker2 is currently implemented on Kafka Connect Framework, more > specifically the Source Connector / Task, which do not provide exactly-once > semantics (EOS) out-of-the-box, as discussed in > https://github.com/confluentinc/kafka-connect-jdbc/issues/461, > https://github.com/apache/kafka/pull/5553, > https://issues.apache.org/jira/browse/KAFKA-6080 and > https://issues.apache.org/jira/browse/KAFKA-3821. Therefore MirrorMaker2 > currently does not provide EOS. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10339) MirrorMaker2 Exactly-once Semantics
[ https://issues.apache.org/jira/browse/KAFKA-10339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17171519#comment-17171519 ] Ryanne Dolan commented on KAFKA-10339: -- I think what you mean is that a read-process-write loop cannot span clusters, since a transaction coordinator on one cluster cannot commit offsets on another cluster. But I don't think we actually need that -- we can just store offsets on the target cluster instead. I think what we need is something along these lines: - we manage offsets ourselves -- we don't rely on Connect's internal offsets tracking or __consumer_offsets on the source cluster. - we only write to the target cluster. - offsets are stored on the target cluster using a "fake" consumer group. I say "fake" because there would be no actual records being consumed by the group, just offsets being stored in __consumer_offsets topic. - we write all records in a transaction, just as the KIP currently describes. - in addition, we call addOffsetsToTransaction in order to commit offsets to the "fake" consumer group on the target cluster. - when MirrorSourceTask starts, it loads initial offsets from __consumer_offsets on the target cluster. Result: - if the transaction succeeds, the __consumer_offsets topic on the target cluster is updated. - if the transaction aborts, all data records are dropped, and the __consumer_offsets topic is not updated. - when MirrorSourceTask starts/restarts, it resumes at the last committed offsets, as recorded in the target cluster. Thoughts? > MirrorMaker2 Exactly-once Semantics > --- > > Key: KAFKA-10339 > URL: https://issues.apache.org/jira/browse/KAFKA-10339 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect >Reporter: Ning Zhang >Assignee: Ning Zhang >Priority: Major > Labels: needs-kip > > MirrorMaker2 is currently implemented on Kafka Connect Framework, more > specifically the Source Connector / Task, which do not provide exactly-once > semantics (EOS) out-of-the-box, as discussed in > https://github.com/confluentinc/kafka-connect-jdbc/issues/461, > https://github.com/apache/kafka/pull/5553, > https://issues.apache.org/jira/browse/KAFKA-6080 and > https://issues.apache.org/jira/browse/KAFKA-3821. Therefore MirrorMaker2 > currently does not provide EOS. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10339) MirrorMaker2 Exactly-once Semantics
[ https://issues.apache.org/jira/browse/KAFKA-10339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17170425#comment-17170425 ] Ryanne Dolan commented on KAFKA-10339: -- Yeah, I guess it would be possible to switch from MirrorSourceConnector to MirrorSinkConnector in the connect-mirror-maker "driver", but external tooling would notice (e.g. JMX metrics would change), so we'd need to put that behind a flag or something so users could opt-in to the SinkConnector in order to get EOS. But even without changing the "driver', an EOS MirrorSinkConnector would be very useful to many organizations that run MM2 on existing Connect clusters. > MirrorMaker2 Exactly-once Semantics > --- > > Key: KAFKA-10339 > URL: https://issues.apache.org/jira/browse/KAFKA-10339 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect >Reporter: Ning Zhang >Assignee: Ning Zhang >Priority: Major > Labels: needs-kip > > MirrorMaker2 is currently implemented on Kafka Connect Framework, more > specifically the Source Connector / Task, which do not provide exactly-once > semantics (EOS) out-of-the-box, as discussed in > https://github.com/confluentinc/kafka-connect-jdbc/issues/461, > https://github.com/apache/kafka/pull/5553, > https://issues.apache.org/jira/browse/KAFKA-6080 and > https://issues.apache.org/jira/browse/KAFKA-3821. Therefore MirrorMaker2 > currently does not provide EOS. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10339) MirrorMaker2 Exactly-once Semantics
[ https://issues.apache.org/jira/browse/KAFKA-10339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17170377#comment-17170377 ] Ryanne Dolan commented on KAFKA-10339: -- [~yangguo1220] this is awesome, thanks! My team has looked into this multiple times, and we came to the same conclusion around WorkerSourceTask requiring a lot of changes to support transactions. I believe it would be easier to start from scratch with a new SourceWorker than to adapt it to support transactions, and we'd probably need to deprecate a number of APIs in the process. I wouldn't rule it out, but it would be difficult. Love how your KIP "kills two birds with one stone" -- we get a MirrorSinkConnector _and_ EOS. > MirrorMaker2 Exactly-once Semantics > --- > > Key: KAFKA-10339 > URL: https://issues.apache.org/jira/browse/KAFKA-10339 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect >Reporter: Ning Zhang >Assignee: Ning Zhang >Priority: Major > Labels: needs-kip > > MirrorMaker2 is currently implemented on Kafka Connect Framework, more > specifically the Source Connector / Task, which do not provide exactly-once > semantics (EOS) out-of-the-box, as discussed in > https://github.com/confluentinc/kafka-connect-jdbc/issues/461, > https://github.com/apache/kafka/pull/5553, > https://issues.apache.org/jira/browse/KAFKA-6080 and > https://issues.apache.org/jira/browse/KAFKA-3821. Therefore MirrorMaker2 > currently does not provide EOS. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[ https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17140849#comment-17140849 ] Ryanne Dolan commented on KAFKA-7500: - [~har5havardhan] I think that is unlikely, given all the missing Admin APIs. You can certainly disable offending features (e.g. topic config sync) and get MM2 to work with older versions of Kafka, but I have not tried with 0.10.0.0 specifically. > MirrorMaker 2.0 (KIP-382) > - > > Key: KAFKA-7500 > URL: https://issues.apache.org/jira/browse/KAFKA-7500 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect, mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Major > Labels: pull-request-available, ready-to-commit > Fix For: 2.4.0 > > Attachments: Active-Active XDCR setup.png > > > Implement a drop-in replacement for MirrorMaker leveraging the Connect > framework. > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0] > [https://github.com/apache/kafka/pull/6295] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10159) MirrorSourceConnector don`t work on connect-distributed.sh
[
https://issues.apache.org/jira/browse/KAFKA-10159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17135817#comment-17135817
]
Ryanne Dolan commented on KAFKA-10159:
--
I think you are missing target.cluster.bootstrap.servers. Seems like a bug tho
-- should be a required property and should have a better error message.
> MirrorSourceConnector don`t work on connect-distributed.sh
> --
>
> Key: KAFKA-10159
> URL: https://issues.apache.org/jira/browse/KAFKA-10159
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 2.4.1
> Environment: centos7
>Reporter: cosmozhu
>Priority: Major
> Fix For: 2.4.1
>
> Attachments: connectDistributed.out
>
>
> hi
> I want to run a MirrorSourceConnector with connect-distributed .
> the connector config like this :
> ```
> {
> "name" : "cosmo-source",
> "config" :
> { "connector.class" :
> "org.apache.kafka.connect.mirror.MirrorSourceConnector",
> "source.cluster.alias" : "cosmo", "target.cluster.alias" : "nana",
> "source.cluster.bootstrap.servers" :
> "192.168.4.42:9092,192.168.4.42:9093,192.168.4.42:9094", "topics" : ".*" }
> }
> ```
> when I post the rest requestion, it returns to me
> ```
> {"name":"cosmo-source","config":{"connector.class":"org.apache.kafka.connect.mirror.MirrorSourceConnector","target.cluster.alias":"nana","topics":".*","source.cluster.alias":"cosmo","name":"cosmo-source","source.cluster.bootstrap.servers":"192.168.4.42:9092,192.168.4.42:9093,192.168.4.42:9094"},"tasks":[],"type":"source"}
> ```
> the task array is empty.
> It's obvious that something's wrong here.
> in connectDistributed.out
> ```
> org.apache.kafka.common.config.ConfigException: Missing required
> configuration "bootstrap.servers" which has no default value.
> ```
> full logs in the attachment.
> thanks for any help.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-10160) Kafka MM2 consumer configuration
[ https://issues.apache.org/jira/browse/KAFKA-10160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17135815#comment-17135815 ] Ryanne Dolan commented on KAFKA-10160: -- Agree this is a bug. This property is hard-coded but doesn't need to be. MM2 should probably default to earliest, but shouldn't override the property if it's manually configured, IMO. > Kafka MM2 consumer configuration > > > Key: KAFKA-10160 > URL: https://issues.apache.org/jira/browse/KAFKA-10160 > Project: Kafka > Issue Type: Bug >Affects Versions: 2.5.0, 2.4.1 >Reporter: Pavol Ipoth >Priority: Major > Labels: configuration, kafka, mirror-maker > > [https://github.com/apache/kafka/blob/trunk/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorMakerConfig.java#L51,] > according this producer/consumer level properties should be configured as > e.g. somesource->sometarget.consumer.client.id, i try to set > somesource->sometarget.consumer.auto.offset.reset=latest, but without > success, consumer always tries to fetch earliest, not sure if bug or my > misconfiguration, but then at least some update to docu would be useful -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[ https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17114309#comment-17114309 ] Ryanne Dolan commented on KAFKA-7500: - [~qq619618919] I usually recommend using a load balancers and health checks for this purpose, but you can also get away with just two VIPs (dc1-kafka, dc2-kafka) and KIP-302. > MirrorMaker 2.0 (KIP-382) > - > > Key: KAFKA-7500 > URL: https://issues.apache.org/jira/browse/KAFKA-7500 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect, mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Major > Labels: pull-request-available, ready-to-commit > Fix For: 2.4.0 > > Attachments: Active-Active XDCR setup.png > > > Implement a drop-in replacement for MirrorMaker leveraging the Connect > framework. > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0] > [https://github.com/apache/kafka/pull/6295] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9981) Running a dedicated mm2 cluster with more than one nodes,When the configuration is updated the task is not aware and will lose the update operation.
[
https://issues.apache.org/jira/browse/KAFKA-9981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108701#comment-17108701
]
Ryanne Dolan commented on KAFKA-9981:
-
Configuration updates only come from the REST API, afaik, which doesn't exist
when running with connect-mirror-maker.sh. So I'm not sure what would be
triggering the logic in the PR. In order to configuration changes to be picked
up at all, the leader must be restarted. Generally you can't know which nodes
are leaders of which flows, so generally it makes sense to just restart
everything with a new config.
This would change if we added back the REST API to connect-mirror-maker.sh (it
is purposefully turned off at present). If we had a REST API, _then_
configuration could change and workers would need to notify their leaders. But
that is not the case now.
> Running a dedicated mm2 cluster with more than one nodes,When the
> configuration is updated the task is not aware and will lose the update
> operation.
>
>
> Key: KAFKA-9981
> URL: https://issues.apache.org/jira/browse/KAFKA-9981
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 2.4.0, 2.5.0, 2.4.1
>Reporter: victor
>Priority: Major
>
> DistributedHerder.reconfigureConnector induction config update as follows:
> {code:java}
> if (changed) {
> List> rawTaskProps = reverseTransform(connName,
> configState, taskProps);
> if (isLeader()) {
> configBackingStore.putTaskConfigs(connName, rawTaskProps);
> cb.onCompletion(null, null);
> } else {
> // We cannot forward the request on the same thread because this
> reconfiguration can happen as a result of connector
> // addition or removal. If we blocked waiting for the response from
> leader, we may be kicked out of the worker group.
> forwardRequestExecutor.submit(new Runnable() {
> @Override
> public void run() {
> try {
> String leaderUrl = leaderUrl();
> if (leaderUrl == null || leaderUrl.trim().isEmpty()) {
> cb.onCompletion(new ConnectException("Request to
> leader to " +
> "reconfigure connector tasks failed " +
> "because the URL of the leader's REST
> interface is empty!"), null);
> return;
> }
> String reconfigUrl = RestServer.urlJoin(leaderUrl,
> "/connectors/" + connName + "/tasks");
> log.trace("Forwarding task configurations for connector
> {} to leader", connName);
> RestClient.httpRequest(reconfigUrl, "POST", null,
> rawTaskProps, null, config, sessionKey, requestSignatureAlgorithm);
> cb.onCompletion(null, null);
> } catch (ConnectException e) {
> log.error("Request to leader to reconfigure connector
> tasks failed", e);
> cb.onCompletion(e, null);
> }
> }
> });
> }
> }
> {code}
> KafkaConfigBackingStore task checks for configuration updates,such as topic
> whitelist update.If KafkaConfigBackingStore task is not running on leader
> node,an HTTP request will be send to notify the leader of the configuration
> update.However,dedicated mm2 cluster does not have the HTTP server turned
> on,so the request will fail to be sent,causing the update operation to be
> lost.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[ https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17082634#comment-17082634 ] Ryanne Dolan commented on KAFKA-7500: - [~Lothar] you have a small misunderstanding: after failover producers do _not_ produce to K1.TOPIC1, as that is a remote topic and by definition has only _replicated_ records from some other cluster. Instead they just produce to TOPIC1 (a normal topic) on the new cluster, which is then replicated back to K1 as K2.TOPIC1. > MirrorMaker 2.0 (KIP-382) > - > > Key: KAFKA-7500 > URL: https://issues.apache.org/jira/browse/KAFKA-7500 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect, mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Major > Labels: pull-request-available, ready-to-commit > Fix For: 2.4.0 > > Attachments: Active-Active XDCR setup.png > > > Implement a drop-in replacement for MirrorMaker leveraging the Connect > framework. > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0] > [https://github.com/apache/kafka/pull/6295] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9746) MM2 produces messages in the wrong cluster
[
https://issues.apache.org/jira/browse/KAFKA-9746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071927#comment-17071927
]
Ryanne Dolan commented on KAFKA-9746:
-
This is because MM2 hands off records to Connect to produce to the target
cluster. If you need to produce to a cluster that is not the same cluster
Connect is producing to, you'd need to have a MirrorSinkConnector (not yet
released) in the middle, _or_ you can override Connect's producer config to
point to your target cluster.
> MM2 produces messages in the wrong cluster
> --
>
> Key: KAFKA-9746
> URL: https://issues.apache.org/jira/browse/KAFKA-9746
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 2.4.0, 2.4.1
>Reporter: Marcel Renders
>Priority: Major
>
> When the MirrorSourceConnector (MM2) is used to get messages from a remote
> Kafka cluster to the Kafka cluster that is connected to Kafka Connect, MM2 is
> working fine. However, when the connected cluster is being copied to the
> remote Kafka cluster, the messages are sent to the connected cluster, causing
> an infinite loop.
> Note that the topics are in fact created as expected in the remote Kafka
> cluster and no errors are thrown except flush failures due to the infinite
> loop.
> In short:
> * Connected target -> unconnected source: OK
> * Connected source -> unconnected target: NOK
> *Desired situation*
> The connected Kafka instance should only be used by Kafka Connect to store
> configs, offsets and status. It should be possible to mirror one Kafka
> cluster to another while neither are connected to the Kafka Connect instance.
> Typically the Kafka connectors are either sink or source, but the MM2 is
> different because the connector settings require both target and source
> configurations.
> *Reproducing the issue*
> This issue occurs with the following configuration, where localhost:9092 is
> used for Kafka Connect:
> {code:java}
> {
> "name": "MM_TEST",
> "connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector",
> "errors.log.enable": "true",
> "enable.auto.commit": "true",
> "auto.commit.interval.ms": "2",
> "refresh.topics.interval.seconds": "60",
> "replication.factor": "1",
> "sync.topic.acls.enabled": "false",
> "checkpoints.topic.replication.factor": "1",
> "heartbeats.topic.replication.factor": "1",
> "offset-syncs.topic.replication.factor": "1",
> "replication.policy.separator": "",
> "topics": "'test1'",
> "tasks.max": "1",
> "key.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
> "value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
> "source.cluster.bootstrap.servers": "localhost:9092",
> "source.cluster.alias": "",
> "source.cluster.security.protocol": "PLAINTEXT",
> "source.cluster.group.id": "consumer_mm_test",
> "target.cluster.bootstrap.servers": "localhost:9093",
> "target.cluster.alias": "",
> "target.cluster.security.protocol": "PLAINTEXT"
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-9726) LegacyReplicationPolicy for MM2 to mimic MM1
Ryanne Dolan created KAFKA-9726: --- Summary: LegacyReplicationPolicy for MM2 to mimic MM1 Key: KAFKA-9726 URL: https://issues.apache.org/jira/browse/KAFKA-9726 Project: Kafka Issue Type: Improvement Components: mirrormaker Reporter: Ryanne Dolan Assignee: Ryanne Dolan Per KIP-382, we should support MM2 in "legacy mode", i.e. with behavior similar to MM1. A key requirement for this is a ReplicationPolicy that does not rename topics. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9376) Plugin class loader not found using MM2
[
https://issues.apache.org/jira/browse/KAFKA-9376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039327#comment-17039327
]
Ryanne Dolan commented on KAFKA-9376:
-
Is it possible you all have the MM2 Connectors in your Connect plugin.path? I
wonder if you hand-copied the jars at one point, and then Connect is trying to
load the Connectors from both locations? I suspect that would cause this sort
of behavior.
> Plugin class loader not found using MM2
> ---
>
> Key: KAFKA-9376
> URL: https://issues.apache.org/jira/browse/KAFKA-9376
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 2.4.0
>Reporter: Sinóros-Szabó Péter
>Priority: Minor
>
> I am using MM2 (release 2.4.0 with scala 2.12) I geta bunch of classloader
> errors. MM2 seems to be working, but I do not know if all of it components
> are working as expected as this is the first time I use MM2.
> I run MM2 with the following command:
> {code:java}
> ./bin/connect-mirror-maker.sh config/connect-mirror-maker.properties
> {code}
> Errors are:
> {code:java}
> [2020-01-07 15:06:17,892] ERROR Plugin class loader for connector:
> 'org.apache.kafka.connect.mirror.MirrorHeartbeatConnector' was not found.
> Returning:
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@6ebf0f36
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:165)
> [2020-01-07 15:06:17,889] ERROR Plugin class loader for connector:
> 'org.apache.kafka.connect.mirror.MirrorHeartbeatConnector' was not found.
> Returning:
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@6ebf0f36
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:165)
> [2020-01-07 15:06:17,904] INFO ConnectorConfig values:
> config.action.reload = restart
> connector.class = org.apache.kafka.connect.mirror.MirrorHeartbeatConnector
> errors.log.enable = false
> errors.log.include.messages = false
> errors.retry.delay.max.ms = 6
> errors.retry.timeout = 0
> errors.tolerance = none
> header.converter = null
> key.converter = null
> name = MirrorHeartbeatConnector
> tasks.max = 1
> transforms = []
> value.converter = null
> (org.apache.kafka.connect.runtime.ConnectorConfig:347)
> [2020-01-07 15:06:17,904] INFO EnrichedConnectorConfig values:
> config.action.reload = restart
> connector.class = org.apache.kafka.connect.mirror.MirrorHeartbeatConnector
> errors.log.enable = false
> errors.log.include.messages = false
> errors.retry.delay.max.ms = 6
> errors.retry.timeout = 0
> errors.tolerance = none
> header.converter = null
> key.converter = null
> name = MirrorHeartbeatConnector
> tasks.max = 1
> transforms = []
> value.converter = null
>
> (org.apache.kafka.connect.runtime.ConnectorConfig$EnrichedConnectorConfig:347)
> [2020-01-07 15:06:17,905] INFO TaskConfig values:
> task.class = class org.apache.kafka.connect.mirror.MirrorHeartbeatTask
> (org.apache.kafka.connect.runtime.TaskConfig:347)
> [2020-01-07 15:06:17,905] INFO Instantiated task MirrorHeartbeatConnector-0
> with version 1 of type org.apache.kafka.connect.mirror.MirrorHeartbeatTask
> (org.apache.kafka.connect.runtime.Worker:434){code}
> After a while, these errors are not logged any more.
> Config is:
> {code:java}
> clusters = eucmain, euwbackup
> eucmain.bootstrap.servers = kafka1:9092,kafka2:9092
> euwbackup.bootstrap.servers = 172.30.197.203:9092,172.30.213.104:9092
> eucmain->euwbackup.enabled = true
> eucmain->euwbackup.topics = .*
> eucmain->euwbackup.topics.blacklist = ^(kafka|kmf|__|pricing).*
> eucmain->euwbackup.rename.topics = false
> rename.topics = false
> eucmain->euwbackup.sync.topic.acls.enabled = false
> sync.topic.acls.enabled = false{code}
> Using OpenJDK 8 or 11, I get the same error.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9013) Flaky Test MirrorConnectorsIntegrationTest#testReplication
[
https://issues.apache.org/jira/browse/KAFKA-9013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034850#comment-17034850
]
Ryanne Dolan commented on KAFKA-9013:
-
I'm seeing loglines like `Found 11 topic-partitions on primary. 0 are new. 1
were removed. Previously had 11.` That doesn't make sense. It looks like the
log message is wrong actually (it shows the same count "11" twice -- confirmed
as wrong in the code), but more importantly it means the tasks are being
re-configured repeatedly, which might account for the flaky test results.
[~ecomar] can you take a look at that? Do we know why MirrorSourceConnector
would think that upstreamTargetTopicPartitions - knownSourceTopicPartitions is
non-empty?
> Flaky Test MirrorConnectorsIntegrationTest#testReplication
> --
>
> Key: KAFKA-9013
> URL: https://issues.apache.org/jira/browse/KAFKA-9013
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Bruno Cadonna
>Priority: Major
> Labels: flaky-test
>
> h1. Stacktrace:
> {code:java}
> java.lang.AssertionError: Condition not met within timeout 2. Offsets not
> translated downstream to primary cluster.
> at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:377)
> at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:354)
> at
> org.apache.kafka.connect.mirror.MirrorConnectorsIntegrationTest.testReplication(MirrorConnectorsIntegrationTest.java:239)
> {code}
> h1. Standard Error
> {code}
> Standard Error
> Oct 09, 2019 11:32:00 PM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource
> registered in SERVER runtime does not implement any provider interfaces
> applicable in the SERVER runtime. Due to constraint configuration problems
> the provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource will
> be ignored.
> Oct 09, 2019 11:32:00 PM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A provider
> org.apache.kafka.connect.runtime.rest.resources.LoggingResource registered in
> SERVER runtime does not implement any provider interfaces applicable in the
> SERVER runtime. Due to constraint configuration problems the provider
> org.apache.kafka.connect.runtime.rest.resources.LoggingResource will be
> ignored.
> Oct 09, 2019 11:32:00 PM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource registered
> in SERVER runtime does not implement any provider interfaces applicable in
> the SERVER runtime. Due to constraint configuration problems the provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource will be
> ignored.
> Oct 09, 2019 11:32:00 PM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A provider
> org.apache.kafka.connect.runtime.rest.resources.RootResource registered in
> SERVER runtime does not implement any provider interfaces applicable in the
> SERVER runtime. Due to constraint configuration problems the provider
> org.apache.kafka.connect.runtime.rest.resources.RootResource will be ignored.
> Oct 09, 2019 11:32:01 PM org.glassfish.jersey.internal.Errors logErrors
> WARNING: The following warnings have been detected: WARNING: The
> (sub)resource method listLoggers in
> org.apache.kafka.connect.runtime.rest.resources.LoggingResource contains
> empty path annotation.
> WARNING: The (sub)resource method listConnectors in
> org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains
> empty path annotation.
> WARNING: The (sub)resource method createConnector in
> org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains
> empty path annotation.
> WARNING: The (sub)resource method listConnectorPlugins in
> org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource
> contains empty path annotation.
> WARNING: The (sub)resource method serverInfo in
> org.apache.kafka.connect.runtime.rest.resources.RootResource contains empty
> path annotation.
> Oct 09, 2019 11:32:02 PM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource
> registered in SERVER runtime does not implement any provider interfaces
> applicable in the SERVER runtime. Due to constraint configuration problems
> the provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource will
> be ignored.
> Oct 09, 2019 11:32:02 PM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A
[jira] [Commented] (KAFKA-9013) Flaky Test MirrorConnectorsIntegrationTest#testReplication
[
https://issues.apache.org/jira/browse/KAFKA-9013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026700#comment-17026700
]
Ryanne Dolan commented on KAFKA-9013:
-
Sounds reasonable. I'll take a look as well.
> Flaky Test MirrorConnectorsIntegrationTest#testReplication
> --
>
> Key: KAFKA-9013
> URL: https://issues.apache.org/jira/browse/KAFKA-9013
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Bruno Cadonna
>Priority: Major
> Labels: flaky-test
>
> h1. Stacktrace:
> {code:java}
> java.lang.AssertionError: Condition not met within timeout 2. Offsets not
> translated downstream to primary cluster.
> at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:377)
> at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:354)
> at
> org.apache.kafka.connect.mirror.MirrorConnectorsIntegrationTest.testReplication(MirrorConnectorsIntegrationTest.java:239)
> {code}
> h1. Standard Error
> {code}
> Standard Error
> Oct 09, 2019 11:32:00 PM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource
> registered in SERVER runtime does not implement any provider interfaces
> applicable in the SERVER runtime. Due to constraint configuration problems
> the provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource will
> be ignored.
> Oct 09, 2019 11:32:00 PM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A provider
> org.apache.kafka.connect.runtime.rest.resources.LoggingResource registered in
> SERVER runtime does not implement any provider interfaces applicable in the
> SERVER runtime. Due to constraint configuration problems the provider
> org.apache.kafka.connect.runtime.rest.resources.LoggingResource will be
> ignored.
> Oct 09, 2019 11:32:00 PM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource registered
> in SERVER runtime does not implement any provider interfaces applicable in
> the SERVER runtime. Due to constraint configuration problems the provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource will be
> ignored.
> Oct 09, 2019 11:32:00 PM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A provider
> org.apache.kafka.connect.runtime.rest.resources.RootResource registered in
> SERVER runtime does not implement any provider interfaces applicable in the
> SERVER runtime. Due to constraint configuration problems the provider
> org.apache.kafka.connect.runtime.rest.resources.RootResource will be ignored.
> Oct 09, 2019 11:32:01 PM org.glassfish.jersey.internal.Errors logErrors
> WARNING: The following warnings have been detected: WARNING: The
> (sub)resource method listLoggers in
> org.apache.kafka.connect.runtime.rest.resources.LoggingResource contains
> empty path annotation.
> WARNING: The (sub)resource method listConnectors in
> org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains
> empty path annotation.
> WARNING: The (sub)resource method createConnector in
> org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains
> empty path annotation.
> WARNING: The (sub)resource method listConnectorPlugins in
> org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource
> contains empty path annotation.
> WARNING: The (sub)resource method serverInfo in
> org.apache.kafka.connect.runtime.rest.resources.RootResource contains empty
> path annotation.
> Oct 09, 2019 11:32:02 PM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource
> registered in SERVER runtime does not implement any provider interfaces
> applicable in the SERVER runtime. Due to constraint configuration problems
> the provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource will
> be ignored.
> Oct 09, 2019 11:32:02 PM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource registered
> in SERVER runtime does not implement any provider interfaces applicable in
> the SERVER runtime. Due to constraint configuration problems the provider
> org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource will be
> ignored.
> Oct 09, 2019 11:32:02 PM org.glassfish.jersey.internal.inject.Providers
> checkProviderRuntime
> WARNING: A provider
>
[jira] [Closed] (KAFKA-9351) Higher count in destination cluster using Kafka MM2
[ https://issues.apache.org/jira/browse/KAFKA-9351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryanne Dolan closed KAFKA-9351. --- > Higher count in destination cluster using Kafka MM2 > --- > > Key: KAFKA-9351 > URL: https://issues.apache.org/jira/browse/KAFKA-9351 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 2.4.0 >Reporter: Nitish Goyal >Priority: Minor > > I have setup replication between cluster across different data centres. After > setting up replication, at times, I am observing higher event count in > destination cluster > Below are counts in source and destination cluster > > *Source Cluster* > ``` > > events_4:0:51048 > events_4:1:52250 > events_4:2:51526 > ``` > > *Destination Cluster* > ``` > nm5.events_4:0:53289 > nm5.events_4:1:54569 > nm5.events_4:2:53733 > ``` > > This is a blocker for us to start using MM2 replicatior -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-9351) Higher count in destination cluster using Kafka MM2
[ https://issues.apache.org/jira/browse/KAFKA-9351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryanne Dolan resolved KAFKA-9351. - Resolution: Information Provided > Higher count in destination cluster using Kafka MM2 > --- > > Key: KAFKA-9351 > URL: https://issues.apache.org/jira/browse/KAFKA-9351 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 2.4.0 >Reporter: Nitish Goyal >Priority: Minor > > I have setup replication between cluster across different data centres. After > setting up replication, at times, I am observing higher event count in > destination cluster > Below are counts in source and destination cluster > > *Source Cluster* > ``` > > events_4:0:51048 > events_4:1:52250 > events_4:2:51526 > ``` > > *Destination Cluster* > ``` > nm5.events_4:0:53289 > nm5.events_4:1:54569 > nm5.events_4:2:53733 > ``` > > This is a blocker for us to start using MM2 replicatior -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9351) Higher count in destination cluster using Kafka MM2
[ https://issues.apache.org/jira/browse/KAFKA-9351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010033#comment-17010033 ] Ryanne Dolan commented on KAFKA-9351: - [~nitishgoyal13] retries in the producer wouldn't normally cause that many dupes. It's more likely that a task was killed before committing cleanly. The tasks will send a commit record during a clean shutdown, but that's not guaranteed, obviously. A failure to commit could result in a large number of dupes like that. Exactly-once semantics would take care of this scenario as well. > Higher count in destination cluster using Kafka MM2 > --- > > Key: KAFKA-9351 > URL: https://issues.apache.org/jira/browse/KAFKA-9351 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 2.4.0 >Reporter: Nitish Goyal >Priority: Minor > > I have setup replication between cluster across different data centres. After > setting up replication, at times, I am observing higher event count in > destination cluster > Below are counts in source and destination cluster > > *Source Cluster* > ``` > > events_4:0:51048 > events_4:1:52250 > events_4:2:51526 > ``` > > *Destination Cluster* > ``` > nm5.events_4:0:53289 > nm5.events_4:1:54569 > nm5.events_4:2:53733 > ``` > > This is a blocker for us to start using MM2 replicatior -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-9376) Plugin class loader not found using MM2
[
https://issues.apache.org/jira/browse/KAFKA-9376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryanne Dolan updated KAFKA-9376:
Priority: Minor (was: Major)
> Plugin class loader not found using MM2
> ---
>
> Key: KAFKA-9376
> URL: https://issues.apache.org/jira/browse/KAFKA-9376
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 2.4.0
>Reporter: Sinóros-Szabó Péter
>Priority: Minor
>
> I am using MM2 (release 2.4.0 with scala 2.12) I geta bunch of classloader
> errors. MM2 seems to be working, but I do not know if all of it components
> are working as expected as this is the first time I use MM2.
> I run MM2 with the following command:
> {code:java}
> ./bin/connect-mirror-maker.sh config/connect-mirror-maker.properties
> {code}
> Errors are:
> {code:java}
> [2020-01-07 15:06:17,892] ERROR Plugin class loader for connector:
> 'org.apache.kafka.connect.mirror.MirrorHeartbeatConnector' was not found.
> Returning:
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@6ebf0f36
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:165)
> [2020-01-07 15:06:17,889] ERROR Plugin class loader for connector:
> 'org.apache.kafka.connect.mirror.MirrorHeartbeatConnector' was not found.
> Returning:
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@6ebf0f36
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:165)
> [2020-01-07 15:06:17,904] INFO ConnectorConfig values:
> config.action.reload = restart
> connector.class = org.apache.kafka.connect.mirror.MirrorHeartbeatConnector
> errors.log.enable = false
> errors.log.include.messages = false
> errors.retry.delay.max.ms = 6
> errors.retry.timeout = 0
> errors.tolerance = none
> header.converter = null
> key.converter = null
> name = MirrorHeartbeatConnector
> tasks.max = 1
> transforms = []
> value.converter = null
> (org.apache.kafka.connect.runtime.ConnectorConfig:347)
> [2020-01-07 15:06:17,904] INFO EnrichedConnectorConfig values:
> config.action.reload = restart
> connector.class = org.apache.kafka.connect.mirror.MirrorHeartbeatConnector
> errors.log.enable = false
> errors.log.include.messages = false
> errors.retry.delay.max.ms = 6
> errors.retry.timeout = 0
> errors.tolerance = none
> header.converter = null
> key.converter = null
> name = MirrorHeartbeatConnector
> tasks.max = 1
> transforms = []
> value.converter = null
>
> (org.apache.kafka.connect.runtime.ConnectorConfig$EnrichedConnectorConfig:347)
> [2020-01-07 15:06:17,905] INFO TaskConfig values:
> task.class = class org.apache.kafka.connect.mirror.MirrorHeartbeatTask
> (org.apache.kafka.connect.runtime.TaskConfig:347)
> [2020-01-07 15:06:17,905] INFO Instantiated task MirrorHeartbeatConnector-0
> with version 1 of type org.apache.kafka.connect.mirror.MirrorHeartbeatTask
> (org.apache.kafka.connect.runtime.Worker:434){code}
> After a while, these errors are not logged any more.
> Config is:
> {code:java}
> clusters = eucmain, euwbackup
> eucmain.bootstrap.servers = kafka1:9092,kafka2:9092
> euwbackup.bootstrap.servers = 172.30.197.203:9092,172.30.213.104:9092
> eucmain->euwbackup.enabled = true
> eucmain->euwbackup.topics = .*
> eucmain->euwbackup.topics.blacklist = ^(kafka|kmf|__|pricing).*
> eucmain->euwbackup.rename.topics = false
> rename.topics = false
> eucmain->euwbackup.sync.topic.acls.enabled = false
> sync.topic.acls.enabled = false{code}
> Using OpenJDK 8 or 11, I get the same error.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9376) Plugin class loader not found using MM2
[
https://issues.apache.org/jira/browse/KAFKA-9376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009904#comment-17009904
]
Ryanne Dolan commented on KAFKA-9376:
-
Thanks for the report. I'm unable to reproduce this so far, but I'll continue
to investigate. Given that the MirrorHeartbeatConnector seems to work despite
this error, I'll downgrade the priority to minor for now.
> Plugin class loader not found using MM2
> ---
>
> Key: KAFKA-9376
> URL: https://issues.apache.org/jira/browse/KAFKA-9376
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 2.4.0
>Reporter: Sinóros-Szabó Péter
>Priority: Major
>
> I am using MM2 (release 2.4.0 with scala 2.12) I geta bunch of classloader
> errors. MM2 seems to be working, but I do not know if all of it components
> are working as expected as this is the first time I use MM2.
> I run MM2 with the following command:
> {code:java}
> ./bin/connect-mirror-maker.sh config/connect-mirror-maker.properties
> {code}
> Errors are:
> {code:java}
> [2020-01-07 15:06:17,892] ERROR Plugin class loader for connector:
> 'org.apache.kafka.connect.mirror.MirrorHeartbeatConnector' was not found.
> Returning:
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@6ebf0f36
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:165)
> [2020-01-07 15:06:17,889] ERROR Plugin class loader for connector:
> 'org.apache.kafka.connect.mirror.MirrorHeartbeatConnector' was not found.
> Returning:
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@6ebf0f36
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:165)
> [2020-01-07 15:06:17,904] INFO ConnectorConfig values:
> config.action.reload = restart
> connector.class = org.apache.kafka.connect.mirror.MirrorHeartbeatConnector
> errors.log.enable = false
> errors.log.include.messages = false
> errors.retry.delay.max.ms = 6
> errors.retry.timeout = 0
> errors.tolerance = none
> header.converter = null
> key.converter = null
> name = MirrorHeartbeatConnector
> tasks.max = 1
> transforms = []
> value.converter = null
> (org.apache.kafka.connect.runtime.ConnectorConfig:347)
> [2020-01-07 15:06:17,904] INFO EnrichedConnectorConfig values:
> config.action.reload = restart
> connector.class = org.apache.kafka.connect.mirror.MirrorHeartbeatConnector
> errors.log.enable = false
> errors.log.include.messages = false
> errors.retry.delay.max.ms = 6
> errors.retry.timeout = 0
> errors.tolerance = none
> header.converter = null
> key.converter = null
> name = MirrorHeartbeatConnector
> tasks.max = 1
> transforms = []
> value.converter = null
>
> (org.apache.kafka.connect.runtime.ConnectorConfig$EnrichedConnectorConfig:347)
> [2020-01-07 15:06:17,905] INFO TaskConfig values:
> task.class = class org.apache.kafka.connect.mirror.MirrorHeartbeatTask
> (org.apache.kafka.connect.runtime.TaskConfig:347)
> [2020-01-07 15:06:17,905] INFO Instantiated task MirrorHeartbeatConnector-0
> with version 1 of type org.apache.kafka.connect.mirror.MirrorHeartbeatTask
> (org.apache.kafka.connect.runtime.Worker:434){code}
> After a while, these errors are not logged any more.
> Config is:
> {code:java}
> clusters = eucmain, euwbackup
> eucmain.bootstrap.servers = kafka1:9092,kafka2:9092
> euwbackup.bootstrap.servers = 172.30.197.203:9092,172.30.213.104:9092
> eucmain->euwbackup.enabled = true
> eucmain->euwbackup.topics = .*
> eucmain->euwbackup.topics.blacklist = ^(kafka|kmf|__|pricing).*
> eucmain->euwbackup.rename.topics = false
> rename.topics = false
> eucmain->euwbackup.sync.topic.acls.enabled = false
> sync.topic.acls.enabled = false{code}
> Using OpenJDK 8 or 11, I get the same error.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9345) Deploy Mirror Maker 2.0 and dynamically modify the topic whitelist through REST API
[ https://issues.apache.org/jira/browse/KAFKA-9345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009040#comment-17009040 ] Ryanne Dolan commented on KAFKA-9345: - [~xinzhuxianshenger] no inconvenience. I'll close this ticket, as I don't believe it represents a bug. > Deploy Mirror Maker 2.0 and dynamically modify the topic whitelist through > REST API > --- > > Key: KAFKA-9345 > URL: https://issues.apache.org/jira/browse/KAFKA-9345 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect, mirrormaker >Affects Versions: 2.4.0 > Environment: runtime env: > source-cluster: kafka 2.2.1 > target-cluster: kafka 2.2.1 > Mirror Maker 2.0 : kafka 2.4.0 >Reporter: yzhou >Assignee: yzhou >Priority: Minor > > 1. Which is the best way to deploy mirror maker 2.0? (a dedicated mm2 > cluster or running mm2 in a connect cluster) . Could you tell me the > difference between them? > 2. According to the blog or wiki > ([https://blog.cloudera.com/a-look-inside-kafka-mirrormaker-2/] , > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0#KIP-382:MirrorMaker2.0-Config,ACLSync] > , > [https://github.com/apache/kafka/blob/cae2a5e1f0779a0889f6cb43b523ebc8a812f4c2/connect/mirror/README.md] > ). Mirror Maker 2.0 topic supports dynamic modification of the whielist, > but I cannot figure out how to make it. Could you tell me how to solve this > problem? > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-9345) Deploy Mirror Maker 2.0 and dynamically modify the topic whitelist through REST API
[ https://issues.apache.org/jira/browse/KAFKA-9345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryanne Dolan resolved KAFKA-9345. - Resolution: Information Provided > Deploy Mirror Maker 2.0 and dynamically modify the topic whitelist through > REST API > --- > > Key: KAFKA-9345 > URL: https://issues.apache.org/jira/browse/KAFKA-9345 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect, mirrormaker >Affects Versions: 2.4.0 > Environment: runtime env: > source-cluster: kafka 2.2.1 > target-cluster: kafka 2.2.1 > Mirror Maker 2.0 : kafka 2.4.0 >Reporter: yzhou >Assignee: yzhou >Priority: Minor > > 1. Which is the best way to deploy mirror maker 2.0? (a dedicated mm2 > cluster or running mm2 in a connect cluster) . Could you tell me the > difference between them? > 2. According to the blog or wiki > ([https://blog.cloudera.com/a-look-inside-kafka-mirrormaker-2/] , > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0#KIP-382:MirrorMaker2.0-Config,ACLSync] > , > [https://github.com/apache/kafka/blob/cae2a5e1f0779a0889f6cb43b523ebc8a812f4c2/connect/mirror/README.md] > ). Mirror Maker 2.0 topic supports dynamic modification of the whielist, > but I cannot figure out how to make it. Could you tell me how to solve this > problem? > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KAFKA-9345) Deploy Mirror Maker 2.0 and dynamically modify the topic whitelist through REST API
[ https://issues.apache.org/jira/browse/KAFKA-9345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryanne Dolan closed KAFKA-9345. --- > Deploy Mirror Maker 2.0 and dynamically modify the topic whitelist through > REST API > --- > > Key: KAFKA-9345 > URL: https://issues.apache.org/jira/browse/KAFKA-9345 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect, mirrormaker >Affects Versions: 2.4.0 > Environment: runtime env: > source-cluster: kafka 2.2.1 > target-cluster: kafka 2.2.1 > Mirror Maker 2.0 : kafka 2.4.0 >Reporter: yzhou >Assignee: yzhou >Priority: Minor > > 1. Which is the best way to deploy mirror maker 2.0? (a dedicated mm2 > cluster or running mm2 in a connect cluster) . Could you tell me the > difference between them? > 2. According to the blog or wiki > ([https://blog.cloudera.com/a-look-inside-kafka-mirrormaker-2/] , > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0#KIP-382:MirrorMaker2.0-Config,ACLSync] > , > [https://github.com/apache/kafka/blob/cae2a5e1f0779a0889f6cb43b523ebc8a812f4c2/connect/mirror/README.md] > ). Mirror Maker 2.0 topic supports dynamic modification of the whielist, > but I cannot figure out how to make it. Could you tell me how to solve this > problem? > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-9351) Higher count in destination cluster using Kafka MM2
[ https://issues.apache.org/jira/browse/KAFKA-9351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryanne Dolan updated KAFKA-9351: Priority: Minor (was: Blocker) > Higher count in destination cluster using Kafka MM2 > --- > > Key: KAFKA-9351 > URL: https://issues.apache.org/jira/browse/KAFKA-9351 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 2.4.0 >Reporter: Nitish Goyal >Priority: Minor > > I have setup replication between cluster across different data centres. After > setting up replication, at times, I am observing higher event count in > destination cluster > Below are counts in source and destination cluster > > *Source Cluster* > ``` > > events_4:0:51048 > events_4:1:52250 > events_4:2:51526 > ``` > > *Destination Cluster* > ``` > nm5.events_4:0:53289 > nm5.events_4:1:54569 > nm5.events_4:2:53733 > ``` > > This is a blocker for us to start using MM2 replicatior -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9351) Higher count in destination cluster using Kafka MM2
[ https://issues.apache.org/jira/browse/KAFKA-9351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009035#comment-17009035 ] Ryanne Dolan commented on KAFKA-9351: - [~nitishgoyal13] are those event counts or offsets? MM2 cannot guarantee that downstream offsets or event counts match exactly -- it only offers at-least-once semantics. So you can be sure that records are not dropped, and they maintain approximately the same order, but there may be dupes due to retries in the producer. I'm working on a PoC for exactly-once semantics, but currently the Connect framework makes this difficult to get right. > Higher count in destination cluster using Kafka MM2 > --- > > Key: KAFKA-9351 > URL: https://issues.apache.org/jira/browse/KAFKA-9351 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 2.4.0 >Reporter: Nitish Goyal >Priority: Blocker > > I have setup replication between cluster across different data centres. After > setting up replication, at times, I am observing higher event count in > destination cluster > Below are counts in source and destination cluster > > *Source Cluster* > ``` > > events_4:0:51048 > events_4:1:52250 > events_4:2:51526 > ``` > > *Destination Cluster* > ``` > nm5.events_4:0:53289 > nm5.events_4:1:54569 > nm5.events_4:2:53733 > ``` > > This is a blocker for us to start using MM2 replicatior -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9345) Deploy Mirror Maker 2.0 and dynamically modify the topic whitelist through REST API
[ https://issues.apache.org/jira/browse/KAFKA-9345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005402#comment-17005402 ] Ryanne Dolan commented on KAFKA-9345: - [~xinzhuxianshenger] can you ask these questions on the user list? Re dynamic topic whitelists: the Cloudera distribution bundles a few plugins, including one that enables dynamic whitelists. The TopicFilter interface enables you to implement this. Alternatively, you can dynamically reconfigure the individual connectors if you run them on a Connect cluster with the Connect REST API. > Deploy Mirror Maker 2.0 and dynamically modify the topic whitelist through > REST API > --- > > Key: KAFKA-9345 > URL: https://issues.apache.org/jira/browse/KAFKA-9345 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect, mirrormaker >Affects Versions: 2.4.0 > Environment: runtime env: > source-cluster: kafka 2.2.1 > target-cluster: kafka 2.2.1 > Mirror Maker 2.0 : kafka 2.4.0 >Reporter: yzhou >Priority: Minor > > 1. Which is the best way to deploy mirror maker 2.0? (a dedicated mm2 > cluster or running mm2 in a connect cluster) . Could you tell me the > difference between them? > 2. According to the blog or wiki > ([https://blog.cloudera.com/a-look-inside-kafka-mirrormaker-2/] , > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0#KIP-382:MirrorMaker2.0-Config,ACLSync] > , > [https://github.com/apache/kafka/blob/cae2a5e1f0779a0889f6cb43b523ebc8a812f4c2/connect/mirror/README.md] > ). Mirror Maker 2.0 topic supports dynamic modification of the whielist, > but I cannot figure out how to make it. Could you tell me how to solve this > problem? > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-9221) Kafka REST Proxy wrongly converts quotes in message when sending json
[
https://issues.apache.org/jira/browse/KAFKA-9221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryanne Dolan resolved KAFKA-9221.
-
Resolution: Invalid
not relevant to this project
> Kafka REST Proxy wrongly converts quotes in message when sending json
> -
>
> Key: KAFKA-9221
> URL: https://issues.apache.org/jira/browse/KAFKA-9221
> Project: Kafka
> Issue Type: Bug
> Components: tools
>Affects Versions: 2.3.0
> Environment: Linux redhat
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Major
>
> Kafka REST Proxy has a problem when sending/converting json files (e.g.
> json.new) into Kafka protocol. For example JSON file:
> {code:java}
> {"records":[{"value":"rest.kafka.testmetric,host=server.com,partition=8,topic=my_topic,url=http:--localhost:7071-metrics
> 1337 1572276922"}]}
> {code}
> is sent using call to Kafka REST Proxy on localhost:8073:
> {code:java}
> curl -X POST -H "Content-Type: application/vnd.kafka.json.v2+json" -H
> "Accept: application/vnd.kafka.v2+json" --data @json.new
> http://localhost:8073/topics/somple_topic -i
> {code}
> in Kafka in some_topic we got:
> {code:java}
> "rest.kafka.testmetric,host=server.com,partition=8,topic=my_topic,url=http:--localhost:7071-metrics
> 1337 1572276922"
> {code}
> but expected is that message has no quotes:
> {code:java}
> rest.kafka.testmetric,host=server.com,partition=8,topic=my_topic,url=http:--localhost:7071-metrics
> 1337 1572276922
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9175) MirrorMaker 2 emits invalid topic partition metrics
[
https://issues.apache.org/jira/browse/KAFKA-9175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973597#comment-16973597
]
Ryanne Dolan commented on KAFKA-9175:
-
Thanks [~ecomar], [~mimaison], lgtm.
> MirrorMaker 2 emits invalid topic partition metrics
> ---
>
> Key: KAFKA-9175
> URL: https://issues.apache.org/jira/browse/KAFKA-9175
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.4.0
>Reporter: Mickael Maison
>Assignee: Mickael Maison
>Priority: Blocker
> Attachments: image-2019-11-12-17-42-45-773.png
>
>
> While looking at MirrorMaker 2 metrics with [~ecomar], we noticed the topic
> partition metrics were invalid.
> !image-2019-11-12-17-42-45-773.png|width=822,height=615!
> There is no traffic on the topic spp.hello but its metrics are constantly
> updating.
> The issue is in {{MirrorMetrics.PartitionMetrics}}. In the constructor,
> Sensors are built using {{metrics.sensor()}} with a name that does not
> include the topic partition. The method {{metrics.sensor()}} does not always
> create a new Sensor but can return an existing Sensor if one exists for the
> specified name. So in practice, if a Task is handling many topic partitions,
> they all share the same Sensors!
> This renders the topic partition metrics unusable and really prevents running
> MirrorMaker 2 in a production environment.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9121) Mirror Maker 2.0 doesn't handle the topic names in consumer checkpoints properly when topic name contain separator
[
https://issues.apache.org/jira/browse/KAFKA-9121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16963519#comment-16963519
]
Ryanne Dolan commented on KAFKA-9121:
-
Hmm, I think we'll always run into such issues if the separator is used in
replicated topics. But I wouldn't be surprised if we could fix this particular
case, perhaps in DefaultReplicationPolicy, which is responsible for creating
and interpreting remote topics.
> Mirror Maker 2.0 doesn't handle the topic names in consumer checkpoints
> properly when topic name contain separator
> --
>
> Key: KAFKA-9121
> URL: https://issues.apache.org/jira/browse/KAFKA-9121
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 2.4.0
>Reporter: Jakub Scholz
>Priority: Major
>
> I was trying the Kafka Mirror Maker 2.0 and run into the following situation:
> 1) I have 2 Kafka clusters with topic {{kafka-test-apps}} topic
> 2) I configured Mirror Maker with {{replication.policy.separator=-}} and with
> mirroring between cluster {{a}} and {{b}}.
> 3) When running Mirror Maker the mirroring of topics works fine. But when I
> use the {{RemoteClusterUtils}} to recover the offsets, the names of the
> topics for which the offsets are found are {{a-kafka-test-apps}} and
> {{apps}}. While the expected topic names would be {{a-kafka-test-apps}} and
> {{kafka-test-apps}}.
> I tried to find the issue, but didn't found it so far. But it doesn't seem to
> be in {{RemoteClusterUtils}} because the topic names seem to be wrong already
> in {{checkpoints.internal}} topic. So it is probably already processed in the
> wrong way in the source cluster.
> When I use {{.}} as the separator, it seems to work fine for me. It looks
> like the problem is only when the topci names contain already the separator
> in the original topic name. But using the right separator might not be a
> solution for this, because you migth have topics with different characters
> and always have this problem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9076) MirrorMaker 2.0 automated consumer offset sync
[ https://issues.apache.org/jira/browse/KAFKA-9076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16957563#comment-16957563 ] Ryanne Dolan commented on KAFKA-9076: - [~yangguo1220] I took a look at the PR. Very cool! I see that you only sync offsets for consumers that are inactive and behind the checkpoint, which makes a lot of sense. This seems to allay my concerns above. I'm glad to hear this is working for you internally. I'm onboard, but you'll need a small KIP to make these changes, since the public API is affected. > MirrorMaker 2.0 automated consumer offset sync > -- > > Key: KAFKA-9076 > URL: https://issues.apache.org/jira/browse/KAFKA-9076 > Project: Kafka > Issue Type: Improvement > Components: mirrormaker >Affects Versions: 2.4.0 >Reporter: Ning Zhang >Priority: Major > Labels: mirrormaker, pull-request-available > Fix For: 2.5.0 > > > To calculate the translated consumer offset in the target cluster, currently > `Mirror-client` provides a function called "remoteConsumerOffsets()" that is > used by "RemoteClusterUtils" for one-time purpose. > In order to make the consumer and stream applications migrate from source to > target cluster transparently and conveniently, e.g. in event of source > cluster failure, a background job is proposed to periodically sync the > consumer offsets from the source to target cluster, so that when the consumer > and stream applications switch to the target cluster, it will resume to > consume from where it left off at source cluster. > > https://github.com/apache/kafka/pull/7577 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9076) MirrorMaker 2.0 automated consumer offset sync
[ https://issues.apache.org/jira/browse/KAFKA-9076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16957016#comment-16957016 ] Ryanne Dolan commented on KAFKA-9076: - [~yangguo1220] I agree this would be an improvement, but there is some complexity here: - in order to write offsets for a consumer group, we need to know that the group is not already running on the target cluster. Otherwise we'd be stepping on that group's current offsets. The group coordinator won't allow this afaik. - we could kick out a target group and force it to seek to the new offsets by revoking group membership and forcing a rebalance etc. But we wouldn't want to do this periodically. - we could write offsets to a new group ID, eg. us-west group1, just like we do with topics, s.t. we avoid the above issues. Then migrating groups would involve changing the group ID. That works fine, but consumers would need a way to determine which group ID to use. Translating group ID like that is more cumbersome than translating offsets, since offsets can be altered using existing tools, but there is no way to tell a consumer to change its group ID. I think there are scenarios where automatically writing offsets as you propose might make sense, e.g. in an active/standby scenario where consumers only connect to one cluster at a time. But if you are automating that behavior, you might as well automate the offset translation via RemoteClusterUtils, IMO. My team has built external tooling using RemoteClusterUtils that works with existing consumers. It's possible to fully automate failover and failback this way. I'm skeptical that automatically writing offsets as you propose would make this process simpler. An alternative to automatically writing offsets is to provide such tooling, e.g. as part of kafka-consumer-groups command. In addition to resetting consumers to a particular timestamp or offset, the tool could cause consumers to seek to the latest MM2 checkpoint. > MirrorMaker 2.0 automated consumer offset sync > -- > > Key: KAFKA-9076 > URL: https://issues.apache.org/jira/browse/KAFKA-9076 > Project: Kafka > Issue Type: Improvement > Components: mirrormaker >Affects Versions: 2.4.0 >Reporter: Ning Zhang >Priority: Major > Labels: mirrormaker, pull-request-available > Fix For: 2.5.0 > > > To calculate the translated consumer offset in the target cluster, currently > `Mirror-client` provides a function called "remoteConsumerOffsets()" that is > used by "RemoteClusterUtils" for one-time purpose. > In order to make the consumer migration from source to target cluster > transparent and convenient, e.g. in event of source cluster failure, it is > better to have a background job to continuously and periodically sync the > consumer offsets from the source to target cluster, so that when the consumer > switches to the target cluster, it will resume to consume from where it left > off at source cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-6080) Transactional EoS for source connectors
[ https://issues.apache.org/jira/browse/KAFKA-6080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16952954#comment-16952954 ] Ryanne Dolan commented on KAFKA-6080: - My goal is to eliminate duplicates downstream of a SourceConnector. The records returned by poll() should be stored in Kafka exactly as they come -- in the same order, exactly once, no additional dupes introduced by the worker. This is mostly straightforward. Here's my working hypothesis: - Workers get producer IDs from their Herder, based on the task ID. - Workers send, flush, commit transactionally. - When a Task fails, the Worker aborts the transaction. - When a Worker fails, any outstanding transactions time out No API changes are required for this, afaict. However, I may end up completely reimplementing WorkerSourceTask to get this right. I'm happy to collaborate or yield if you guys want to pick this up: otherwise I'll put a KIP together very soon. > Transactional EoS for source connectors > --- > > Key: KAFKA-6080 > URL: https://issues.apache.org/jira/browse/KAFKA-6080 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect >Reporter: Antony Stubbs >Assignee: Ryanne Dolan >Priority: Major > Labels: needs-kip > > Exactly once (eos) message production for source connectors. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KAFKA-6080) Transactional EoS for source connectors
[ https://issues.apache.org/jira/browse/KAFKA-6080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryanne Dolan reassigned KAFKA-6080: --- Assignee: Ryanne Dolan > Transactional EoS for source connectors > --- > > Key: KAFKA-6080 > URL: https://issues.apache.org/jira/browse/KAFKA-6080 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect >Reporter: Antony Stubbs >Assignee: Ryanne Dolan >Priority: Major > Labels: needs-kip > > Exactly once (eos) message production for source connectors. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-6080) Transactional EoS for source connectors
[ https://issues.apache.org/jira/browse/KAFKA-6080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16952205#comment-16952205 ] Ryanne Dolan commented on KAFKA-6080: - I'm thinking about this differently than you guys [~rhauch] [~sliebau], but I'd like to take this on, if there are no objections. I'll put a KIP together in the near future. Given this has been idle for a while, I'll go ahead and assign this to myself. > Transactional EoS for source connectors > --- > > Key: KAFKA-6080 > URL: https://issues.apache.org/jira/browse/KAFKA-6080 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect >Reporter: Antony Stubbs >Priority: Major > Labels: needs-kip > > Exactly once (eos) message production for source connectors. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8929) MM2 system tests
[ https://issues.apache.org/jira/browse/KAFKA-8929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943741#comment-16943741 ] Ryanne Dolan commented on KAFKA-8929: - I've got ducktape system tests implemented to match the existing legacy mirror-maker tests. Will have a PR ready soonish. > MM2 system tests > > > Key: KAFKA-8929 > URL: https://issues.apache.org/jira/browse/KAFKA-8929 > Project: Kafka > Issue Type: Improvement > Components: mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Minor > Labels: test > Fix For: 2.4.0 > > > Add system tests for MM2 driver. Should resemble existing mirror-maker system > tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-8929) MM2 system tests
Ryanne Dolan created KAFKA-8929: --- Summary: MM2 system tests Key: KAFKA-8929 URL: https://issues.apache.org/jira/browse/KAFKA-8929 Project: Kafka Issue Type: Improvement Components: mirrormaker Affects Versions: 2.4.0 Reporter: Ryanne Dolan Assignee: Ryanne Dolan Fix For: 2.4.0 Add system tests for MM2 driver. Should resemble existing mirror-maker system tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-8930) MM2 documentation
Ryanne Dolan created KAFKA-8930: --- Summary: MM2 documentation Key: KAFKA-8930 URL: https://issues.apache.org/jira/browse/KAFKA-8930 Project: Kafka Issue Type: Improvement Components: documentation, mirrormaker Affects Versions: 2.4.0 Reporter: Ryanne Dolan Assignee: Ryanne Dolan Fix For: 2.4.0 Expand javadocs for new MirrorMaker (entrypoint) and MirrorMakerConfig classes. Include example usage and example configuration. Expand javadocs for MirrorSourceConnector, MirrorCheckpointConnector, and MirrorHeartbeatConnector, including example configuration for running on Connect w/o mm2 driver. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[ https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932658#comment-16932658 ] Ryanne Dolan commented on KAFKA-7500: - [~qihong] the direct passthrough described there is what MM2 currently does -- there are dedicated workers consuming from each source cluster and writing to each target cluster, without requiring a bunch of Connect clusters, and without requiring hops through an intermediate Kafka cluster. I think the confusion here is that "sink" does not imply "SinkConnector". Sink in this context is the target cluster. > MirrorMaker 2.0 (KIP-382) > - > > Key: KAFKA-7500 > URL: https://issues.apache.org/jira/browse/KAFKA-7500 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect, mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Assignee: Manikumar >Priority: Major > Labels: pull-request-available, ready-to-commit > Fix For: 2.4.0 > > Attachments: Active-Active XDCR setup.png > > > Implement a drop-in replacement for MirrorMaker leveraging the Connect > framework. > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0] > [https://github.com/apache/kafka/pull/6295] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[ https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931776#comment-16931776 ] Ryanne Dolan commented on KAFKA-7500: - [~qihong] That's all correct :) In fact, KIP-382 mentions a "primary" cluster (in your case "C") and a single Connect cluster being used to replicate data between any number of other clusters. One way to do this is with a SinkConnector (part of the KIP and coming later) s.t. records travel _through_ the primary cluster to other clusters. I believe you could also override the Connect Worker's internal Producer config to write directly to another cluster, s.t., as you say, state is stored in one cluster but records go to another. I've never tried that, ymmv, but I suspect it'd work as you've described. Notice that, generally, you will end up with multiple Connect clusters anyway -- one in each DC -- for performance and HA reasons. At that point you are back to managing multiple Connectors on multiple Connect clusters. MM2's top-level driver manages that complexity for you by automatically spinning up and configuring all the required Workers. re a REST API, the MM2 driver essentially turns off the REST service inside each Herder. This is because the current Connect REST API doesn't support having multiple Herders or Worker configs, so we'd need to sorta abuse the Connect REST API to get it to work. However, there was much discussion around KIP-382 re an MM2 REST API, and there are several good ideas floating around. These were ultimately deferred, but not ruled out. Also, fwiw, I have successfully run an MM2 cluster with the Connect REST API turned on, with each Herder's endpoints wrapped in a higher-level API. I have done this successfully with a reverse proxy as well as with a fork of Connect's RestServer. This enables you to start/stop/configure individual connectors within the MM2 cluster, if you so wish. And finally: MM2 is very pluggable. For example, you can drop in a TopicFilter that grabs dynamic whitelists from somewhere. I happen to know this works very well :) > MirrorMaker 2.0 (KIP-382) > - > > Key: KAFKA-7500 > URL: https://issues.apache.org/jira/browse/KAFKA-7500 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect, mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Assignee: Manikumar >Priority: Major > Labels: pull-request-available, ready-to-commit > Fix For: 2.4.0 > > Attachments: Active-Active XDCR setup.png > > > Implement a drop-in replacement for MirrorMaker leveraging the Connect > framework. > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0] > [https://github.com/apache/kafka/pull/6295] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[ https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931597#comment-16931597 ] Ryanne Dolan commented on KAFKA-7500: - [~chridtian.hagel] we try to be smart about what config properties are replicated and which are left as defaults: - we have a config property blacklist: https://github.com/apache/kafka/pull/6295/files#diff-1ae39c06c52ded296030121b13d4b791R33 - we don't replicate a config property that is inherited from the cluster default or from the static broker config, i.e. we only replicate properties that are explicitly set for a topic. - we don't replicate read-only or sensitive properties for obvious reasons. cleanup.policy is one that _should_ be replicated, generally, unless you are expecting the default value to be replicated. Also, be advised that config sync is only periodic, with a default interval of 10 minutes. > MirrorMaker 2.0 (KIP-382) > - > > Key: KAFKA-7500 > URL: https://issues.apache.org/jira/browse/KAFKA-7500 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect, mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Assignee: Manikumar >Priority: Major > Labels: pull-request-available, ready-to-commit > Fix For: 2.4.0 > > Attachments: Active-Active XDCR setup.png > > > Implement a drop-in replacement for MirrorMaker leveraging the Connect > framework. > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0] > [https://github.com/apache/kafka/pull/6295] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[ https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929359#comment-16929359 ] Ryanne Dolan commented on KAFKA-7500: - [~chridtian.hagel] thanks for giving it a spin. The "failed to flush" errors are probably due to WorkerSourceTask being unable to send the 5942 messages within the default flush timeout, which I believe is 5 seconds. There are various reasons this might be the case: - tasks.max could be 1 (the default), which means a single Producer is sending records across the entire Herder. Try increasing this considerably. This can be as high as the total number of partitions being replicated, at the cost of more overhead per partition, obviously. If you configure this too high, MM2 just uses one task per partition. - The producer lag may be high, which is detrimental to throughput. Make sure the MM2 driver is running close to the target cluster to minimize this latency. If you are replicating between multiple DCs, consider running a few MM2 nodes in each DC, with `--clusters` argument to hint which clusters are nearby. That way, drivers will consume from other DCs but only produce locally. - You may need to use more MM2 nodes. - You may need to increase the 5 second flush timeout. Re: duplicated messages, you are correct that MM2 will send dupes if containers are bounced like that. Generally, this is okay -- occasional dupes are a fact of life in most Kafka pipelines. That said, I am working on a PoC and KIP for exactly-once replication with MM2, which will eliminate these dupes. > MirrorMaker 2.0 (KIP-382) > - > > Key: KAFKA-7500 > URL: https://issues.apache.org/jira/browse/KAFKA-7500 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect, mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Assignee: Manikumar >Priority: Major > Labels: pull-request-available, ready-to-commit > Fix For: 2.4.0 > > Attachments: Active-Active XDCR setup.png > > > Implement a drop-in replacement for MirrorMaker leveraging the Connect > framework. > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0] > [https://github.com/apache/kafka/pull/6295] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[ https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927978#comment-16927978 ] Ryanne Dolan commented on KAFKA-7500: - [~qihong] thanks for the questions. > But couldn't find any consumer groups on dr2 related to consumer group > test1grp. MM2 does not create consumer groups for you or attempt to keep them in sync -- it only produces checkpoints (dr1.checkpoints.internal) that encode the state of remote consumer groups. You must then do something with these checkpoints, depending on your use-case. The RemoteClusterUtils class will read checkpoints for you, which you can then use in interesting ways. For example, you can use RemoteClusterUtils.translateOffsets() and the kafka-consumer-groups.sh --reset-offsets tool to create a consumer group in dr2 based on MM2's checkpoints from dr1. Or, you can use RemoteClusterUtils in your Consumer code to failover/failback automatically. Both require a bit of code, but nothing too sophisticated. Looking ahead a bit, this will be a ton easier when KIP-396 is merged (KAFKA-7689). Once consumer offsets can be controlled from the Admin API, it will be possible to consume checkpoints and update offsets directly. That will enable the behavior you were expecting. > By the way, how to set up and run this in a Kafka connect cluster? MM2's Connectors are just plain-old Connectors. You can run them with connect-standalone.sh or connect-distributed.sh as with any other Connector. To do so, you need a worker config and a connector config as usual. The worker config must include whatever client settings are required to connect to the _target_ cluster (i.e. bootstrap servers, security settings), since the Worker is what is actually producing downstream records. The connector configs, on the other hand, need connection settings for _both_ source and target clusters (e.g. source.cluster.bootstrap.servers, target.cluster.bootstrap.servers). The Connectors use both source and target clusters when syncing topic configuration etc. There is an example Connector configuration here: https://github.com/apache/kafka/pull/6295#issuecomment-522074048 > MirrorMaker 2.0 (KIP-382) > - > > Key: KAFKA-7500 > URL: https://issues.apache.org/jira/browse/KAFKA-7500 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect, mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Assignee: Manikumar >Priority: Major > Labels: pull-request-available, ready-to-commit > Fix For: 2.4.0 > > Attachments: Active-Active XDCR setup.png > > > Implement a drop-in replacement for MirrorMaker leveraging the Connect > framework. > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0] > [https://github.com/apache/kafka/pull/6295] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[ https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryanne Dolan reassigned KAFKA-7500: --- Assignee: Manikumar > MirrorMaker 2.0 (KIP-382) > - > > Key: KAFKA-7500 > URL: https://issues.apache.org/jira/browse/KAFKA-7500 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect, mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Assignee: Manikumar >Priority: Major > Labels: pull-request-available, ready-to-commit > Fix For: 2.4.0 > > Attachments: Active-Active XDCR setup.png > > > Implement a drop-in replacement for MirrorMaker leveraging the Connect > framework. > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0] > [https://github.com/apache/kafka/pull/6295] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Resolved] (KAFKA-8568) MirrorMaker 2.0 resource leak
[
https://issues.apache.org/jira/browse/KAFKA-8568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryanne Dolan resolved KAFKA-8568.
-
Resolution: Fixed
> MirrorMaker 2.0 resource leak
> -
>
> Key: KAFKA-8568
> URL: https://issues.apache.org/jira/browse/KAFKA-8568
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect, mirrormaker
>Affects Versions: 2.2.2
>Reporter: Péter Gergő Barna
>Assignee: Ryanne Dolan
>Priority: Major
>
> This issue produced by the branch KIP-382 (I am not sure which version is
> affected by that branch).
> While MirrorMaker 2.0 is running, the following command returns a number that
> is getting larger and larger.
>
> {noformat}
> lsof -p | grep ESTABLISHED | wc -l{noformat}
>
> Meanwhile, in the error log, NullPointers pop up from the
> MirrorSourceTask.cleanup, because either the consumer or the producer is null
> when the cleanup method tries to close them.
>
> {noformat}
> Exception in thread "Thread-790" java.lang.NullPointerException
> at
> org.apache.kafka.connect.mirror.MirrorSourceTask.cleanup(MirrorSourceTask.java:110)
> at java.lang.Thread.run(Thread.java:748)
> Exception in thread "Thread-792" java.lang.NullPointerException
> at
> org.apache.kafka.connect.mirror.MirrorSourceTask.cleanup(MirrorSourceTask.java:110)
> at java.lang.Thread.run(Thread.java:748)
> Exception in thread "Thread-791" java.lang.NullPointerException
> at
> org.apache.kafka.connect.mirror.MirrorSourceTask.cleanup(MirrorSourceTask.java:116)
> at java.lang.Thread.run(Thread.java:748)
> Exception in thread "Thread-793" java.lang.NullPointerException
> at
> org.apache.kafka.connect.mirror.MirrorSourceTask.cleanup(MirrorSourceTask.java:110)
> at java.lang.Thread.run(Thread.java:748){noformat}
> When the number of the established connections (returned by lsof) reaches a
> certain limit, new exceptions start to pop up in the logs: Too many open files
> {noformat}
> [2019-06-19 12:56:43,949] ERROR
> WorkerSourceTask{id=MirrorHeartbeatConnector-0} failed to send record to
> heartbeats: {} (org.apache.kafka.connect.runtime.WorkerSourceTask)
> org.apache.kafka.common.errors.SaslAuthenticationException: An error:
> (java.security.PrivilegedActionException: javax.security.sasl.SaslException:
> GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Too many open
> files)]) occurred when evaluating SASL token received from the Kafka Broker.
> Kafka Client will go to A
> UTHENTICATION_FAILED state.
> Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Too many open
> files)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> at
> org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.lambda$createSaslToken$1(SaslClientAuthenticator.java:461)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslToken(SaslClientAuthenticator.java:461)
> at
> org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendSaslClientToken(SaslClientAuthenticator.java:370)
> at
> org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendInitialToken(SaslClientAuthenticator.java:290)
> at
> org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.authenticate(SaslClientAuthenticator.java:230)
> at
> org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:173)
> at
> org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:536)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:472)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
> at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
> at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: GSSException: No valid credentials provided (Mechanism level: Too
> many open files)
> at
> sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:775)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
> ... 14 more
> Caused by: java.net.SocketException: Too many open files
> at java.net.Socket.createImpl(Socket.java:460)
>
[jira] [Commented] (KAFKA-8568) MirrorMaker 2.0 resource leak
[
https://issues.apache.org/jira/browse/KAFKA-8568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922927#comment-16922927
]
Ryanne Dolan commented on KAFKA-8568:
-
This was fixed a while back. MM2 was re-using some top-level configuration
properties for internal clients, causing a huge number of extraneous
MetricsReporters, ConfigProviders etc to be created. This was resolved by
limiting which top-level properties are applied to internal clients.
> MirrorMaker 2.0 resource leak
> -
>
> Key: KAFKA-8568
> URL: https://issues.apache.org/jira/browse/KAFKA-8568
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect, mirrormaker
>Affects Versions: 2.2.2
>Reporter: Péter Gergő Barna
>Assignee: Ryanne Dolan
>Priority: Major
>
> This issue produced by the branch KIP-382 (I am not sure which version is
> affected by that branch).
> While MirrorMaker 2.0 is running, the following command returns a number that
> is getting larger and larger.
>
> {noformat}
> lsof -p | grep ESTABLISHED | wc -l{noformat}
>
> Meanwhile, in the error log, NullPointers pop up from the
> MirrorSourceTask.cleanup, because either the consumer or the producer is null
> when the cleanup method tries to close them.
>
> {noformat}
> Exception in thread "Thread-790" java.lang.NullPointerException
> at
> org.apache.kafka.connect.mirror.MirrorSourceTask.cleanup(MirrorSourceTask.java:110)
> at java.lang.Thread.run(Thread.java:748)
> Exception in thread "Thread-792" java.lang.NullPointerException
> at
> org.apache.kafka.connect.mirror.MirrorSourceTask.cleanup(MirrorSourceTask.java:110)
> at java.lang.Thread.run(Thread.java:748)
> Exception in thread "Thread-791" java.lang.NullPointerException
> at
> org.apache.kafka.connect.mirror.MirrorSourceTask.cleanup(MirrorSourceTask.java:116)
> at java.lang.Thread.run(Thread.java:748)
> Exception in thread "Thread-793" java.lang.NullPointerException
> at
> org.apache.kafka.connect.mirror.MirrorSourceTask.cleanup(MirrorSourceTask.java:110)
> at java.lang.Thread.run(Thread.java:748){noformat}
> When the number of the established connections (returned by lsof) reaches a
> certain limit, new exceptions start to pop up in the logs: Too many open files
> {noformat}
> [2019-06-19 12:56:43,949] ERROR
> WorkerSourceTask{id=MirrorHeartbeatConnector-0} failed to send record to
> heartbeats: {} (org.apache.kafka.connect.runtime.WorkerSourceTask)
> org.apache.kafka.common.errors.SaslAuthenticationException: An error:
> (java.security.PrivilegedActionException: javax.security.sasl.SaslException:
> GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Too many open
> files)]) occurred when evaluating SASL token received from the Kafka Broker.
> Kafka Client will go to A
> UTHENTICATION_FAILED state.
> Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Too many open
> files)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> at
> org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.lambda$createSaslToken$1(SaslClientAuthenticator.java:461)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslToken(SaslClientAuthenticator.java:461)
> at
> org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendSaslClientToken(SaslClientAuthenticator.java:370)
> at
> org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendInitialToken(SaslClientAuthenticator.java:290)
> at
> org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.authenticate(SaslClientAuthenticator.java:230)
> at
> org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:173)
> at
> org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:536)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:472)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
> at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
> at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: GSSException: No valid credentials provided (Mechanism level: Too
> many open files)
> at
> sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:775)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
> at
>
[jira] [Updated] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[ https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryanne Dolan updated KAFKA-7500: Labels: pull-request-available ready-to-commit (was: ) > MirrorMaker 2.0 (KIP-382) > - > > Key: KAFKA-7500 > URL: https://issues.apache.org/jira/browse/KAFKA-7500 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect, mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Priority: Major > Labels: pull-request-available, ready-to-commit > Fix For: 2.4.0 > > Attachments: Active-Active XDCR setup.png > > > Implement a drop-in replacement for MirrorMaker leveraging the Connect > framework. > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0] > [https://github.com/apache/kafka/pull/6295] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[ https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryanne Dolan updated KAFKA-7500: Priority: Major (was: Minor) > MirrorMaker 2.0 (KIP-382) > - > > Key: KAFKA-7500 > URL: https://issues.apache.org/jira/browse/KAFKA-7500 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect, mirrormaker >Reporter: Ryanne Dolan >Priority: Major > Fix For: 2.4.0 > > Attachments: Active-Active XDCR setup.png > > > Implement a drop-in replacement for MirrorMaker leveraging the Connect > framework. > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0] > [https://github.com/apache/kafka/pull/6295] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[ https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryanne Dolan updated KAFKA-7500: Affects Version/s: 2.4.0 > MirrorMaker 2.0 (KIP-382) > - > > Key: KAFKA-7500 > URL: https://issues.apache.org/jira/browse/KAFKA-7500 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect, mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Priority: Major > Fix For: 2.4.0 > > Attachments: Active-Active XDCR setup.png > > > Implement a drop-in replacement for MirrorMaker leveraging the Connect > framework. > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0] > [https://github.com/apache/kafka/pull/6295] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Comment Edited] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[ https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886520#comment-16886520 ] Ryanne Dolan edited comment on KAFKA-7500 at 8/21/19 7:57 PM: -- [~apriceq] I would recommend pulling down the PR and building it. Something along these lines should work: $ git clone github.com/ryannedolan/kafka $ git checkout KIP-382 $ gradle :connect:mirror:build Of course, you can also fork my repo. I welcome PRs! was (Author: ryannedolan): [~apriceq] I would recommend pulling down the PR and building it. Something along these lines should work: $ git clone github.com/ryannedolan/kafka $ git checkout KIP-382 $ git :connect:mirror:build Of course, you can also fork my repo. I welcome PRs! > MirrorMaker 2.0 (KIP-382) > - > > Key: KAFKA-7500 > URL: https://issues.apache.org/jira/browse/KAFKA-7500 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect, mirrormaker >Reporter: Ryanne Dolan >Priority: Minor > Fix For: 2.4.0 > > Attachments: Active-Active XDCR setup.png > > > Implement a drop-in replacement for MirrorMaker leveraging the Connect > framework. > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0] > [https://github.com/apache/kafka/pull/6295] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[ https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16912637#comment-16912637 ] Ryanne Dolan commented on KAFKA-7500: - [~shoxdid] This is now fixed, thanks for reporting! > MirrorMaker 2.0 (KIP-382) > - > > Key: KAFKA-7500 > URL: https://issues.apache.org/jira/browse/KAFKA-7500 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect, mirrormaker >Reporter: Ryanne Dolan >Priority: Minor > Fix For: 2.4.0 > > Attachments: Active-Active XDCR setup.png > > > Implement a drop-in replacement for MirrorMaker leveraging the Connect > framework. > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0] > [https://github.com/apache/kafka/pull/6295] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[ https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886520#comment-16886520 ] Ryanne Dolan commented on KAFKA-7500: - [~apriceq] I would recommend pulling down the PR and building it. Something along these lines should work: $ git clone github.com/ryannedolan/kafka $ git checkout KIP-382 $ git :connect:mirror:build Of course, you can also fork my repo. I welcome PRs! > MirrorMaker 2.0 (KIP-382) > - > > Key: KAFKA-7500 > URL: https://issues.apache.org/jira/browse/KAFKA-7500 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect, mirrormaker >Reporter: Ryanne Dolan >Priority: Minor > Fix For: 2.4.0 > > Attachments: Active-Active XDCR setup.png > > > Implement a drop-in replacement for MirrorMaker leveraging the Connect > framework. > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0] > [https://github.com/apache/kafka/pull/6295] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[ https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886515#comment-16886515 ] Ryanne Dolan commented on KAFKA-7500: - [~sdeokulecluster] Kafka releases are time-based, every 4 months. MM2 has been tested with several versions of Kafka and should work with 0.10.2.1, though I have not tested that specific version. > MirrorMaker 2.0 (KIP-382) > - > > Key: KAFKA-7500 > URL: https://issues.apache.org/jira/browse/KAFKA-7500 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect, mirrormaker >Reporter: Ryanne Dolan >Priority: Minor > Fix For: 2.4.0 > > Attachments: Active-Active XDCR setup.png > > > Implement a drop-in replacement for MirrorMaker leveraging the Connect > framework. > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0] > [https://github.com/apache/kafka/pull/6295] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[ https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886512#comment-16886512 ] Ryanne Dolan commented on KAFKA-7500: - @Munamala I discuss this a bit in the readme: https://github.com/apache/kafka/blob/cae2a5e1f0779a0889f6cb43b523ebc8a812f4c2/connect/mirror/README.md Please let me know if anything is unclear. > MirrorMaker 2.0 (KIP-382) > - > > Key: KAFKA-7500 > URL: https://issues.apache.org/jira/browse/KAFKA-7500 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect, mirrormaker >Reporter: Ryanne Dolan >Priority: Minor > Fix For: 2.4.0 > > Attachments: Active-Active XDCR setup.png > > > Implement a drop-in replacement for MirrorMaker leveraging the Connect > framework. > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0] > [https://github.com/apache/kafka/pull/6295] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Comment Edited] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[ https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886512#comment-16886512 ] Ryanne Dolan edited comment on KAFKA-7500 at 7/16/19 10:19 PM: --- [~Munamala] I discuss this a bit in the readme: https://github.com/apache/kafka/blob/cae2a5e1f0779a0889f6cb43b523ebc8a812f4c2/connect/mirror/README.md Please let me know if anything is unclear. was (Author: ryannedolan): @Munamala I discuss this a bit in the readme: https://github.com/apache/kafka/blob/cae2a5e1f0779a0889f6cb43b523ebc8a812f4c2/connect/mirror/README.md Please let me know if anything is unclear. > MirrorMaker 2.0 (KIP-382) > - > > Key: KAFKA-7500 > URL: https://issues.apache.org/jira/browse/KAFKA-7500 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect, mirrormaker >Reporter: Ryanne Dolan >Priority: Minor > Fix For: 2.4.0 > > Attachments: Active-Active XDCR setup.png > > > Implement a drop-in replacement for MirrorMaker leveraging the Connect > framework. > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0] > [https://github.com/apache/kafka/pull/6295] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[ https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878308#comment-16878308 ] Ryanne Dolan commented on KAFKA-7500: - [~Munamala] Yes, the translateOffsets() works both ways. When K1 comes back online, you can "failback" from K2 to K1 once a checkpoint for TOPIC_GROUP is emitted upstream to K1. The checkpoint will have offsets for TOPIC1 on K1 (translated from K1.TOPIC1 on K2). You can then seek() to skip over the records TOPIC_GROUP already consumed in K2. The tricky part here is that you need to make sure MM2 is configured to emit checkpoints both from K1->K2 and K2->K1. Configure the whitelists like: K1->K2.topics = TOPIC1, K2.TOPIC1 K2->K1.topics = TOPIC1, K1.TOPIC1 K1->K2.groups = TOPIC_GROUP K2->K1.groups = TOPIC_GROUP Otherwise, you won't see checkpoints for TOPIC1 going from K2 to K1. Ryanne > MirrorMaker 2.0 (KIP-382) > - > > Key: KAFKA-7500 > URL: https://issues.apache.org/jira/browse/KAFKA-7500 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect, mirrormaker >Reporter: Ryanne Dolan >Priority: Minor > Fix For: 2.4.0 > > > Implement a drop-in replacement for MirrorMaker leveraging the Connect > framework. > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0] > [https://github.com/apache/kafka/pull/6295] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KAFKA-8568) MirrorMaker 2.0 resource leak
[
https://issues.apache.org/jira/browse/KAFKA-8568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryanne Dolan reassigned KAFKA-8568:
---
Assignee: Ryanne Dolan
> MirrorMaker 2.0 resource leak
> -
>
> Key: KAFKA-8568
> URL: https://issues.apache.org/jira/browse/KAFKA-8568
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect, mirrormaker
>Affects Versions: 2.2.2
>Reporter: Péter Gergő Barna
>Assignee: Ryanne Dolan
>Priority: Major
>
> This issue produced by the branch KIP-382 (I am not sure which version is
> affected by that branch).
> While MirrorMaker 2.0 is running, the following command returns a number that
> is getting larger and larger.
>
> {noformat}
> lsof -p | grep ESTABLISHED | wc -l{noformat}
>
> Meanwhile, in the error log, NullPointers pop up from the
> MirrorSourceTask.cleanup, because either the consumer or the producer is null
> when the cleanup method tries to close them.
>
> {noformat}
> Exception in thread "Thread-790" java.lang.NullPointerException
> at
> org.apache.kafka.connect.mirror.MirrorSourceTask.cleanup(MirrorSourceTask.java:110)
> at java.lang.Thread.run(Thread.java:748)
> Exception in thread "Thread-792" java.lang.NullPointerException
> at
> org.apache.kafka.connect.mirror.MirrorSourceTask.cleanup(MirrorSourceTask.java:110)
> at java.lang.Thread.run(Thread.java:748)
> Exception in thread "Thread-791" java.lang.NullPointerException
> at
> org.apache.kafka.connect.mirror.MirrorSourceTask.cleanup(MirrorSourceTask.java:116)
> at java.lang.Thread.run(Thread.java:748)
> Exception in thread "Thread-793" java.lang.NullPointerException
> at
> org.apache.kafka.connect.mirror.MirrorSourceTask.cleanup(MirrorSourceTask.java:110)
> at java.lang.Thread.run(Thread.java:748){noformat}
> When the number of the established connections (returned by lsof) reaches a
> certain limit, new exceptions start to pop up in the logs: Too many open files
> {noformat}
> [2019-06-19 12:56:43,949] ERROR
> WorkerSourceTask{id=MirrorHeartbeatConnector-0} failed to send record to
> heartbeats: {} (org.apache.kafka.connect.runtime.WorkerSourceTask)
> org.apache.kafka.common.errors.SaslAuthenticationException: An error:
> (java.security.PrivilegedActionException: javax.security.sasl.SaslException:
> GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Too many open
> files)]) occurred when evaluating SASL token received from the Kafka Broker.
> Kafka Client will go to A
> UTHENTICATION_FAILED state.
> Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Too many open
> files)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> at
> org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.lambda$createSaslToken$1(SaslClientAuthenticator.java:461)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslToken(SaslClientAuthenticator.java:461)
> at
> org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendSaslClientToken(SaslClientAuthenticator.java:370)
> at
> org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendInitialToken(SaslClientAuthenticator.java:290)
> at
> org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.authenticate(SaslClientAuthenticator.java:230)
> at
> org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:173)
> at
> org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:536)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:472)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
> at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
> at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: GSSException: No valid credentials provided (Mechanism level: Too
> many open files)
> at
> sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:775)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
> ... 14 more
> Caused by: java.net.SocketException: Too many open files
> at java.net.Socket.createImpl(Socket.java:460)
[jira] [Commented] (KAFKA-7077) KIP-318: Make Kafka Connect Source idempotent
[ https://issues.apache.org/jira/browse/KAFKA-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16875021#comment-16875021 ] Ryanne Dolan commented on KAFKA-7077: - [~vpernin] yes, this is a frequent ask for MM2. Would love to see this merged. Ryanne > KIP-318: Make Kafka Connect Source idempotent > - > > Key: KAFKA-7077 > URL: https://issues.apache.org/jira/browse/KAFKA-7077 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Affects Versions: 2.0.0 >Reporter: Stephane Maarek >Assignee: Stephane Maarek >Priority: Major > > KIP Link: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-318%3A+Make+Kafka+Connect+Source+idempotent -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[
https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869925#comment-16869925
]
Ryanne Dolan commented on KAFKA-7500:
-
[~abellemare] Thanks for trying it out.
> 1) Are the offset translation topics included... KAFKA-7915
Yes, I've included the required changes to SourceTask in PR-6295.
> 2) ...switching a consumer from one cluster to another...
So glad you asked :)
The key is the RemoteClusterUtils.translateOffsets() method. This consumes from
the checkpoint topic (not the offset-sync topic directly), which has both
upstream and downstream offsets for each consumer group. The downstream offsets
are calculated based on the offset-sync stream, of course, but
MirrorCheckpointConnector does the translation for you. This makes the
translateOffsets() method rather straightforward -- it just finds the most
recent checkpoint for a given group.
The translateOffsets() method works both ways: you can translate from a source
topic ("topic1") to a remote topic ("us-east.topic1") and vice versa, which
means your failover and failback logic is identical. In both cases you just
migrate all your consumer groups from one cluster to another.
Also note that migration only requires information that is already stored on
the target cluster (the checkpoints), so you do not need to connect to a failed
cluster in order to failover from it. Obviously that would defeat the purpose!
Based on translateOffsets(), you can do several nifty things wrt
failover/failback, e.g. build scripts that bulk-migrate consumer groups from
one cluster to another, or add consumer client logic that automatically
failsover to a secondary cluster as needed.
In the former case, you can use kafka-consumer-groups.sh to "reset offsets" to
those returned by translateOffsets(). This will cause consumer state to be
transferred to the target cluster, in effect. In the latter, you can use
translateOffsets() with KafkaConsumer.seek().
There are more advanced operations and architectures you can build using MM2 as
well. Some are outlined in the following talk (by me):
https://www.slideshare.net/ConfluentInc/disaster-recovery-with-mirrormaker-20-ryanne-dolan-cloudera-kafka-summit-london-2019
> 3) Do you use timestamps at all for failing over from one cluster to another?
MM2 preserves the timestamps of replicated records, but otherwise does not care
about timestamps. Nor does failover need to involve any timestamps.
Ryanne
> MirrorMaker 2.0 (KIP-382)
> -
>
> Key: KAFKA-7500
> URL: https://issues.apache.org/jira/browse/KAFKA-7500
> Project: Kafka
> Issue Type: New Feature
> Components: KafkaConnect, mirrormaker
>Reporter: Ryanne Dolan
>Priority: Minor
> Fix For: 2.4.0
>
>
> Implement a drop-in replacement for MirrorMaker leveraging the Connect
> framework.
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0]
> [https://github.com/apache/kafka/pull/6295]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[ https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847632#comment-16847632 ] Ryanne Dolan commented on KAFKA-7500: - [~Munamala] thanks for trying it out! > The topics are created with a replication factor 1 There is a property `replication.factor` that you can change as needed to prevent this issue. This can be set across all clusters or for a specific cluster, e.g. `my-cluster.replication.factor = 4`, and MM2 will use this value when creating topics. [~arunmathew88] suggested we bump the default replication factor to 2. It would be cool if MM2 could figure this value out based on the downstream cluster's min ISRs -- maybe a future KIP? > How are the topics created: source.checkpoints.internal and heartbeats? Currently they are not being actively created -- we sorta assume the downstream brokers are configured to auto-create topics, or that the internal topics are manually created. It's a good idea to fix this, thanks. > Is this the right forum to ask questions during the evaluation? Yes, thanks! Ryanne > MirrorMaker 2.0 (KIP-382) > - > > Key: KAFKA-7500 > URL: https://issues.apache.org/jira/browse/KAFKA-7500 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect, mirrormaker >Reporter: Ryanne Dolan >Priority: Minor > Fix For: 2.4 > > > Implement a drop-in replacement for MirrorMaker leveraging the Connect > framework. > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0] > [https://github.com/apache/kafka/pull/6295] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[ https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryanne Dolan reassigned KAFKA-7500: --- Assignee: Colin P. McCabe (was: Ryanne Dolan) Reviewer: Colin P. McCabe [https://github.com/apache/kafka/pull/6295] > MirrorMaker 2.0 (KIP-382) > - > > Key: KAFKA-7500 > URL: https://issues.apache.org/jira/browse/KAFKA-7500 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect, mirrormaker >Reporter: Ryanne Dolan >Assignee: Colin P. McCabe >Priority: Minor > Fix For: 2.3.0 > > > Implement a drop-in replacement for MirrorMaker leveraging the Connect > framework. > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0] > [https://github.com/apache/kafka/pull/6295] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[ https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryanne Dolan updated KAFKA-7500: Fix Version/s: 2.3.0 > MirrorMaker 2.0 (KIP-382) > - > > Key: KAFKA-7500 > URL: https://issues.apache.org/jira/browse/KAFKA-7500 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect, mirrormaker >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Minor > Fix For: 2.3.0 > > > Implement a drop-in replacement for MirrorMaker leveraging the Connect > framework. > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0] > [https://github.com/apache/kafka/pull/6295] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[ https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryanne Dolan updated KAFKA-7500: Description: Implement a drop-in replacement for MirrorMaker leveraging the Connect framework. [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0] [https://github.com/apache/kafka/pull/6295] was: Implement a drop-in replacement for MirrorMaker leveraging the Connect framework. https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0 > MirrorMaker 2.0 (KIP-382) > - > > Key: KAFKA-7500 > URL: https://issues.apache.org/jira/browse/KAFKA-7500 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect, mirrormaker >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Minor > > Implement a drop-in replacement for MirrorMaker leveraging the Connect > framework. > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0] > [https://github.com/apache/kafka/pull/6295] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[
https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836058#comment-16836058
]
Ryanne Dolan edited comment on KAFKA-7500 at 5/9/19 2:41 PM:
-
[~cloudfrog], glad you are taking a look. I'm looking forward to hearing about
your experience.
> Is this expected to work with kafka 1.1.1 clusters?
Yes, I believe it will work with 0.11.0.0 or higher, but maybe you can test it
to verify :)
> will prefix remote topic names ... be configurable?
Yes, you can implement your own ReplicationPolicy to define remote topics
however you like:
{code:java}
replication.policy.class = my.SuffixReplicationPolicy
{code}
Also, MM2 doesn't care how existing source topics are named. If your topics are
suffixed with their local DC (a common pattern), you can leave them as-is
without breaking anything. By default you'd get topics like "dc1.topic1-dc1",
so you might consider implementing a ReplicationPolicy that strips the suffix
during replication so you get just "dc1.topic1".
Ryanne
was (Author: ryannedolan):
[~cloudfrog], glad you are taking a look. I'm looking forward to hearing about
your experience.
> Is this expected to work with kafka 1.1.1 clusters?
Yes, I believe it will work with 0.11.0.0 or higher, but maybe you can test it
to verify :)
> will prefix remote topic names ... be configurable?
Yes, you can implement your own ReplicationPolicy to define remote topics
however you like:
{code:java}
replication.policy.class = my.SuffixReplicationPolicy
{code}
Also, MM2 doesn't care how existing source topics are named. If your topics are
prefixed with their local DC (a common pattern), you can leave them as-is
without breaking anything. By default you'd get topics like "dc1.topic1-dc1",
so you might consider implementing a ReplicationPolicy that strips the suffix
during replication so you get just "dc1.topic1".
Ryanne
> MirrorMaker 2.0 (KIP-382)
> -
>
> Key: KAFKA-7500
> URL: https://issues.apache.org/jira/browse/KAFKA-7500
> Project: Kafka
> Issue Type: New Feature
> Components: KafkaConnect, mirrormaker
>Reporter: Ryanne Dolan
>Assignee: Ryanne Dolan
>Priority: Minor
>
> Implement a drop-in replacement for MirrorMaker leveraging the Connect
> framework.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[
https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836058#comment-16836058
]
Ryanne Dolan commented on KAFKA-7500:
-
[~cloudfrog], glad you are taking a look. I'm looking forward to hearing about
your experience.
> Is this expected to work with kafka 1.1.1 clusters?
Yes, I believe it will work with 0.11.0.0 or higher, but maybe you can test it
to verify :)
> will prefix remote topic names ... be configurable?
Yes, you can implement your own ReplicationPolicy to define remote topics
however you like:
{code:java}
replication.policy.class = my.SuffixReplicationPolicy
{code}
Also, MM2 doesn't care how existing source topics are named. If your topics are
prefixed with their local DC (a common pattern), you can leave them as-is
without breaking anything. By default you'd get topics like "dc1.topic1-dc1",
so you might consider implementing a ReplicationPolicy that strips the suffix
during replication so you get just "dc1.topic1".
Ryanne
> MirrorMaker 2.0 (KIP-382)
> -
>
> Key: KAFKA-7500
> URL: https://issues.apache.org/jira/browse/KAFKA-7500
> Project: Kafka
> Issue Type: New Feature
> Components: KafkaConnect, mirrormaker
>Reporter: Ryanne Dolan
>Assignee: Ryanne Dolan
>Priority: Minor
>
> Implement a drop-in replacement for MirrorMaker leveraging the Connect
> framework.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[
https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825346#comment-16825346
]
Ryanne Dolan commented on KAFKA-7500:
-
[~terence.yi] the most trivial scenario would be something like this:
{{clusters = primary, secondary}}
{{primary.bootstrap.servers = primary1.host:9092, primary2.host:9092,
primary3.host:9092}}
{{secondary.bootstrap.servers = secondary1.host:9092, secondary2.host:9092,
secondary3.host:9092}}
{{primary->secondary.enabled = true}}
{{}}{{secondary->primary.enabled = true}}
{{primary->secondary.topics = .*}}
{{secondary->primary.topics = .*}}
... where there are two clusters replicating each other. Then some simple
things you can test include:
# create a topic on either cluster; a remote topic should show up on the other
# send records to a topic on one cluster and verify the same records appear on
the other
# change a topic's configuration, e.g. retention.policy=compact, and verify
the config is sync'd to the other
# change a topic's ACL and verify the ACL is sync'd to the other
# consume from a topic in one cluster and verify that checkpoints are emitted
for the group to the other
# use RemoteClusterUtils to translate a consumer's offsets from one cluster to
the other and verify no records are skipped
etc
I'm putting together documentation for more advanced scenarios and will share
when ready.
> MirrorMaker 2.0 (KIP-382)
> -
>
> Key: KAFKA-7500
> URL: https://issues.apache.org/jira/browse/KAFKA-7500
> Project: Kafka
> Issue Type: New Feature
> Components: KafkaConnect, mirrormaker
>Reporter: Ryanne Dolan
>Assignee: Ryanne Dolan
>Priority: Minor
>
> Implement a drop-in replacement for MirrorMaker leveraging the Connect
> framework.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
