Hmm, sounds like something is wrong with advertised.listeners on the brokers. Even if (as you mentioned) you can reach the brokers over tcp, the brokers must be configured to advertise the right addresses and ports. It sounds like mm can reach the bootstrap servers, but then these brokers are providing the wrong listeners (in metadata responses). This would explain why mm can create topics but not replicate.
Ryanne On Mon, Oct 25, 2021, 3:38 PM Rijo Roy <rjo_...@yahoo.com.invalid> wrote: > Hi Ryanne, > > Thanks for responding! > > I am using only one node to run the mm atm. I found some luck in enabling > the connect.log and now it is printing a lot of lines like below - > > [2021-10-25 20:13:24,477] WARN [Worker clientId=connect-2, groupId=A-mm2] > 51 partitions have leader brokers without a matching listener, including > [__consumer_offsets-0, __consumer_offsets-10, __consumer_offsets-20, > __consumer_offsets-40, __consumer_offsets-30, __consumer_offsets-9, > __consumer_offsets-39, __consumer_offsets-11, __consumer_offsets-31, > __consumer_offsets-13] > (org.apache.kafka.clients.NetworkClient:1070)[2021-10-25 20:13:24,512] INFO > AbstractConfig > values: (org.apache.kafka.common.config.AbstractConfig:347)[2021-10-25 > 20:13:24,515] INFO Connector B->A configured. > (org.apache.kafka.connect.mirror.MirrorMaker:212)org.apache.kafka.connect.runtime.rest.errors.BadRequestException: > Connector configuration is invalid and contains the following 1 > error(s):Invalid value > io.strimzi.kafka.connect.mirror.IdentityReplicationPolicy for configuration > replication.policy.class: Class > io.strimzi.kafka.connect.mirror.IdentityReplicationPolicy could not be > found. > > but mostly this one - [2021-10-25 20:23:48,430] WARN [Worker > clientId=connect-2, groupId=A-mm2] 51 partitions have leader brokers > without a matching listener, including [__consumer_offsets-0, > __consumer_offsets-10, __consumer_offsets-20, __consumer_offsets-40, > __consumer_offsets-30, __consumer_offsets-9, __consumer_offsets-39, > __consumer_offsets-11, __consumer_offsets-31, __consumer_offsets-13] > (org.apache.kafka.clients.NetworkClient:1070) > > Could you help me understand the WARN. Now I have stopped it as this > looked confusing [2021-10-25 20:13:24,515] INFO Connector B->A configured. > (org.apache.kafka.connect.mirror.MirrorMaker:212) as my data is on A which > needs to be mirrored to B and that is what I have set in my mm2.properties > (pasting it again fyr) > A->B.enabled = true > > B->A.enabled = false > > Thanks again for your help! > > > ~ > > On Tuesday, 26 October, 2021, 01:46:54 am IST, Ryanne Dolan < > ryannedo...@gmail.com> wrote: > > Hello, how many mm nodes are you using? Try starting with one and adding > more after the first starts working. There is a known race that effects > some people trying to start multiple nodes at once. > > Also check for any potential auth problems. The driver isn't very verbose > when it comes to auth problems, and it's possible for some nodes to have > the right certs while others don't (due to the distributed nature of > Connect). For example, I've often seen the leader node have the proper > configuration while the other Workers don't have the same creds in the same > place. > > Ryanne > > On Mon, Oct 25, 2021, 2:42 PM Rijo Roy <rjo_...@yahoo.com.invalid> wrote: > > > Hi, > > > > I was very much relieved to see MirrorMaker2 working in my sandbox for a > > major project where we are migrating out of cluster A to cluster B for a > > better hardware. To my surprise, it fails to migrate any user topics > when I > > used it in one of our real pre-prod environments where we have real > topics. > > Let me give a brief about my environment - > > > > ENV A > > > > OS: Ubuntu 18 > > 3 node Kafka cluster > > 3 node ZooKeeper cluster > > > > ENV B > > OS: Ubuntu 18 > > 3 node Kafka cluster > > 3 node ZooKeeper cluster > > > > Kafka version: Confluent kafka 5.5 > > Zk version: 3.5.8 > > > > here is the mm2.properties I used in my sandbox fyr where replication is > > working fine - > > # mm2.propertiesclusters = A, BA.bootstrap.servers = Ahost1:port, > > Ahost2:port, Ahost3:portB.bootstrap.servers = > > Bhost1:port, Bhost2:port, Bhost3:portA->B.enabled = trueB->A.enabled = > > > falsereplication.factor=3checkpoints.topic.replication.factor=3heartbeats.topic.replication.factor=3offset-syncs.topic.replication.factor=3replication.policy.separator=source.cluster.alias=target.cluster.alias=replication.policy.class= > > io.strimzi.kafka.connect.mirror.IdentityReplicationPolicytopics = > .*groups > > = .*emit.checkpoints.interval.seconds = > > > 10security.protocol=SASL_PLAINTEXTsasl.mechanism=PLAINsasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule > > required \ username="vvvvv" \ password="ccccccccc"; > > > > I used the same configuration in my pre-prod environment where I have to > > successfully test the cutover procedure inorder to move to PROD within > the > > speculated tight timeline but it fails to replicate any data. The only > job > > it is successfully doing is creating these 3 below topics in both A and B > > clusters - > > mm2-configs.n.internalmm2-offsets.n.internalmm2-status.n.internal > > where n is A in source and B in target > > And the kafka logs does not have anything to help me with troubleshooting > > neither have any monitoring tools to track the sync progress if at all it > > is happening. > > > > Could you please help me here - > > 1. How to enable logging for mirrormaker > > 2. How can I successfully monitor it > > 3. How can I make it work > > > > Note: Its not a network traffic issue as I have verified telnet from > > A<-->B and it is working. > > > > Thanks! > > >
