Re: How to Setup MirrorMaker in Generalized way

2014-12-12 Thread Neha Narkhede
Is there any solution to avoid this duplicated entry in target cluster? I am using Kafka That is the expected behavior when the mirror maker process is killed. It is expected that in most cases, it be shutdown cleanly. If it is killed, it doesn't get a chance to checkpoint it's offset, which

Re: How to Setup MirrorMaker in Generalized way

2014-12-11 Thread Madhukar Bharti
Hi Neha, Thanks for your reply. Now using MM tool to replicate data between Kafka clusters, But I am facing one problem, Messages gets duplicated if MM killed forcefully[ *kill -9* ]. Is there any solution to avoid this duplicated entry in target cluster? I am using Kafka *8.1.1.* On Mon,

How to Setup MirrorMaker in Generalized way

2014-12-08 Thread Madhukar Bharti
Hi, I am going to setup Kafka clusters having 3 brokers in Datacenter 1. Topics can be created time to time. Each topic can have varying partitions mostly 1,10 or 20. Each application might have different partitioning algorithm that we don't know(let it be hidden from ops team). We want to setup

Re: How to Setup MirrorMaker in Generalized way

2014-12-08 Thread Neha Narkhede
Hi Madhukar, From the same documentation link you referred to - The source and destination clusters are completely independent entities: they can have different numbers of partitions and the offsets will not be the same. For this reason the mirror cluster is not really intended as a