For now (0.8.2) if you have a hard crash then you will likely have duplicates or even data loss unfortunately, I do not have a good solution to this on top of my head.
On Wed, Jan 21, 2015 at 10:14 PM, Madhukar Bharti <bhartimadhu...@gmail.com> wrote: > Thanks Guozhang for your reply. > > Checked the details as mentioned above. Issue KAFKA-1650 has been fixed but > it will come with 0.8.3. :-( > > By "MM Killed", it is hard-failed(kill -9 )/crashed due to system reboot or > any other reason. In that case some of the messages get duplicated. > > Can you please suggest a way to do inter DC replication for time being > without loss/duplication of messages? > > > > On Thu, Jan 22, 2015 at 6:39 AM, Guozhang Wang <wangg...@gmail.com> wrote: > > > Hi Madhukar, > > > > We are currently working on improving MM to avoid data loss / duplicates, > > details can be found in KAFKA-1650 / 1839 / 1840. > > > > For your concern, by saying "MM killed" do you mean it is shutdown or > > hard-failed / crashed that can cause duplicates? > > > > Guozhang > > > > On Mon, Jan 19, 2015 at 10:51 PM, Madhukar Bharti < > > bhartimadhu...@gmail.com> > > wrote: > > > > > Hi, > > > > > > I want to setup inter DC replication b/w Kafka clusters. Is there any > > > inbuilt tool to do this? > > > > > > I already have tried MirrorMaker tool but the problem is, if MM killed > > then > > > some messages get duplicated. I don't want to duplicate the messages. > > > > > > Please suggest a way to do this. Please share your experience how you > > have > > > setup this. > > > > > > > > > Thanks in Advance, > > > Madhukar > > > > > > > > > > > -- > > -- Guozhang > > > > > > -- > Thanks and Regards, > Madhukar Bharti > -- -- Guozhang
