Re: Apache Kafka cluster to cluster
Is there documentation or example for mirror maker 2.0 ? On 4/29/20, 9:04 PM, "Liam Clarke-Hutchinson" wrote: [External] Hi Blake, Replicator is, AFAIK, not FOSS - however, Mirror Maker 2.0, which is built along very similar lines (i.e., on top of Kafka Connect) is, as is Mirror Maker 1.0. On Thu, Apr 30, 2020 at 6:51 AM Blake Miller wrote: > Oh, and it looks like Confluent has released a newer replacement for > MirrorMaker called Replicator > > > https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.confluent.io%2Fcurrent%2Fmulti-dc-deployments%2Freplicator%2Fmigrate-replicator.htmldata=02%7C01%7CManoj.Agrawal2%40cognizant.com%7Cdd5c2b32b1aa4567421e08d7ecbb9def%7Cde08c40719b9427d9fe8edf254300ca7%7C0%7C0%7C637238162871927627sdata=f6zWwsX%2FEjBxwpiT%2FuaOBB5RSi6HnFcN13S9iukAIqA%3Dreserved=0 > > > > On Wed, Apr 29, 2020 at 6:49 PM Blake Miller > wrote: > > > Hi Vishnu, > > > > Check out MirrorMaker > > > https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D27846330data=02%7C01%7CManoj.Agrawal2%40cognizant.com%7Cdd5c2b32b1aa4567421e08d7ecbb9def%7Cde08c40719b9427d9fe8edf254300ca7%7C0%7C0%7C637238162871927627sdata=ey5jIPUG%2BVgEGCshvlMlXdmU%2B54E787oWiF6AZ6jj%2FU%3Dreserved=0 > > > > This can do what you want. Note that the offsets are not copied, nor are > > the message timestamps. > > > > HTH > > > > > > On Wed, Apr 29, 2020 at 6:47 PM vishnu murali < > vishnumurali9...@gmail.com> > > wrote: > > > >> Hi Guys, > >> > >> I am having two separate Kafka cluster running in two independent > >> zookeeper > >> > >> I need to send a set of data from one topic from cluster A to cluster B > >> with the same topic name with all data also.. > >> > >> How can I achieve this > >> > >> Done anyone have any idea ?? > >> > > > This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored.
Re: Apache Kafka cluster to cluster
Hi Vishnu, I can see you're using Mirror Maker 1, unlike Mirror Maker 2, this doesn't auto-create topics on the target cluster. Does your target cluster on port 9091 have the same topics created on it that exist on the source cluster on port 9092? Also note that a Kafka consumer defaults to "latest" for auto offset reset - so when starting MM 1 it won't automatically replicate all existing data, only data created after it begins consuming. You can override this by setting the consumer property "auto.offset.reset" to "earliest". Hope that helps, Liam Clarke-Hutchinson On Thu, Apr 30, 2020 at 3:56 PM vishnu murali wrote: > i start the kafka mirror using the below configuration > > .\bin\windows\kafka-mirror-maker.bat --consumer.config > .\config\consumer.properties --producer.config .\config\producer.properties > --whitelist=".*" > > *Consumer.properties:* > # format: host1:port1,host2:port2 ... > bootstrap.servers=localhost:9092 > > # consumer group id > group.id=test-consumer-group > > *Producer.properties:* > # format: host1:port1,host2:port2 ... > bootstrap.servers=localhost:9091 > > # specify the compression codec for all data generated: none, gzip, snappy, > lz4, zstd > compression.type=none > > > I get this kind of warning statement and it will be still present in that > statement of more than 30 minutes.No data will be transfered > > WARNING: The default partition assignment strategy of the mirror maker will > change from 'range' to 'roundrobin' in an upcoming release (so that better > load balancing can be achieved). If you prefer to make this switch in > advance of that release add the following to the corresponding config: > > 'partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor' > > can anyone clarify why i am getting this and what i am doing wrong? > > On Thu, Apr 30, 2020 at 12:22 AM vishnu murali > > wrote: > > > Thanks Blake.. > > > > More over can we use any connector types for this situation? > > > > Like source is a topic and sink is also an another topic > > > > Is this possible... > > > > On Thu, Apr 30, 2020, 00:19 Blake Miller wrote: > > > >> Hi Vishnu, > >> > >> Check out MirrorMaker > >> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330 > >> > >> This can do what you want. Note that the offsets are not copied, nor are > >> the message timestamps. > >> > >> HTH > >> > >> > >> On Wed, Apr 29, 2020 at 6:47 PM vishnu murali < > vishnumurali9...@gmail.com > >> > > >> wrote: > >> > >> > Hi Guys, > >> > > >> > I am having two separate Kafka cluster running in two independent > >> zookeeper > >> > > >> > I need to send a set of data from one topic from cluster A to cluster > B > >> > with the same topic name with all data also.. > >> > > >> > How can I achieve this > >> > > >> > Done anyone have any idea ?? > >> > > >> > > >
Re: Apache Kafka cluster to cluster
Hi Blake, Replicator is, AFAIK, not FOSS - however, Mirror Maker 2.0, which is built along very similar lines (i.e., on top of Kafka Connect) is, as is Mirror Maker 1.0. On Thu, Apr 30, 2020 at 6:51 AM Blake Miller wrote: > Oh, and it looks like Confluent has released a newer replacement for > MirrorMaker called Replicator > > > https://docs.confluent.io/current/multi-dc-deployments/replicator/migrate-replicator.html > > > > On Wed, Apr 29, 2020 at 6:49 PM Blake Miller > wrote: > > > Hi Vishnu, > > > > Check out MirrorMaker > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330 > > > > This can do what you want. Note that the offsets are not copied, nor are > > the message timestamps. > > > > HTH > > > > > > On Wed, Apr 29, 2020 at 6:47 PM vishnu murali < > vishnumurali9...@gmail.com> > > wrote: > > > >> Hi Guys, > >> > >> I am having two separate Kafka cluster running in two independent > >> zookeeper > >> > >> I need to send a set of data from one topic from cluster A to cluster B > >> with the same topic name with all data also.. > >> > >> How can I achieve this > >> > >> Done anyone have any idea ?? > >> > > >
Re: Apache Kafka cluster to cluster
i start the kafka mirror using the below configuration .\bin\windows\kafka-mirror-maker.bat --consumer.config .\config\consumer.properties --producer.config .\config\producer.properties --whitelist=".*" *Consumer.properties:* # format: host1:port1,host2:port2 ... bootstrap.servers=localhost:9092 # consumer group id group.id=test-consumer-group *Producer.properties:* # format: host1:port1,host2:port2 ... bootstrap.servers=localhost:9091 # specify the compression codec for all data generated: none, gzip, snappy, lz4, zstd compression.type=none I get this kind of warning statement and it will be still present in that statement of more than 30 minutes.No data will be transfered WARNING: The default partition assignment strategy of the mirror maker will change from 'range' to 'roundrobin' in an upcoming release (so that better load balancing can be achieved). If you prefer to make this switch in advance of that release add the following to the corresponding config: 'partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor' can anyone clarify why i am getting this and what i am doing wrong? On Thu, Apr 30, 2020 at 12:22 AM vishnu murali wrote: > Thanks Blake.. > > More over can we use any connector types for this situation? > > Like source is a topic and sink is also an another topic > > Is this possible... > > On Thu, Apr 30, 2020, 00:19 Blake Miller wrote: > >> Hi Vishnu, >> >> Check out MirrorMaker >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330 >> >> This can do what you want. Note that the offsets are not copied, nor are >> the message timestamps. >> >> HTH >> >> >> On Wed, Apr 29, 2020 at 6:47 PM vishnu murali > > >> wrote: >> >> > Hi Guys, >> > >> > I am having two separate Kafka cluster running in two independent >> zookeeper >> > >> > I need to send a set of data from one topic from cluster A to cluster B >> > with the same topic name with all data also.. >> > >> > How can I achieve this >> > >> > Done anyone have any idea ?? >> > >> >
Re: Apache Kafka cluster to cluster
Thanks Blake.. More over can we use any connector types for this situation? Like source is a topic and sink is also an another topic Is this possible... On Thu, Apr 30, 2020, 00:19 Blake Miller wrote: > Hi Vishnu, > > Check out MirrorMaker > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330 > > This can do what you want. Note that the offsets are not copied, nor are > the message timestamps. > > HTH > > > On Wed, Apr 29, 2020 at 6:47 PM vishnu murali > wrote: > > > Hi Guys, > > > > I am having two separate Kafka cluster running in two independent > zookeeper > > > > I need to send a set of data from one topic from cluster A to cluster B > > with the same topic name with all data also.. > > > > How can I achieve this > > > > Done anyone have any idea ?? > > >
Re: Apache Kafka cluster to cluster
Oh, and it looks like Confluent has released a newer replacement for MirrorMaker called Replicator https://docs.confluent.io/current/multi-dc-deployments/replicator/migrate-replicator.html On Wed, Apr 29, 2020 at 6:49 PM Blake Miller wrote: > Hi Vishnu, > > Check out MirrorMaker > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330 > > This can do what you want. Note that the offsets are not copied, nor are > the message timestamps. > > HTH > > > On Wed, Apr 29, 2020 at 6:47 PM vishnu murali > wrote: > >> Hi Guys, >> >> I am having two separate Kafka cluster running in two independent >> zookeeper >> >> I need to send a set of data from one topic from cluster A to cluster B >> with the same topic name with all data also.. >> >> How can I achieve this >> >> Done anyone have any idea ?? >> >
Re: Apache Kafka cluster to cluster
Hi Vishnu, Check out MirrorMaker https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330 This can do what you want. Note that the offsets are not copied, nor are the message timestamps. HTH On Wed, Apr 29, 2020 at 6:47 PM vishnu murali wrote: > Hi Guys, > > I am having two separate Kafka cluster running in two independent zookeeper > > I need to send a set of data from one topic from cluster A to cluster B > with the same topic name with all data also.. > > How can I achieve this > > Done anyone have any idea ?? >
Apache Kafka cluster to cluster
Hi Guys, I am having two separate Kafka cluster running in two independent zookeeper I need to send a set of data from one topic from cluster A to cluster B with the same topic name with all data also.. How can I achieve this Done anyone have any idea ??
