I wouldn't use synchronous replication between two datacenters. If your network link ever goes down all Kafka writes will fail. If you ever need to do maintenance you'll either need to somehow turn this off or all kafka writes will fail. Plus, as Hans mentions, this will slow down your throughput quite a bit.
-Dave -----Original Message----- From: Hans Jespersen [mailto:h...@confluent.io] Sent: Friday, October 28, 2016 11:36 AM To: Mudit Agarwal Cc: users@kafka.apache.org Subject: Re: Kafka Multi DataCenter HA/Failover Are you willing to have a maximum throughput of 6.67 messages per second? -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Fri, Oct 28, 2016 at 9:07 AM, Mudit Agarwal <mudit...@yahoo.com> wrote: > Hi Hans, > > The latency between my two DC is 150ms.And yes I'm looking for > synchronous replication.Is that possible? > > Thanks, > Mudit > > > ------------------------------ > *From:* Hans Jespersen <h...@confluent.io> > *To:* users@kafka.apache.org; Mudit Agarwal <mudit...@yahoo.com> > *Sent:* Friday, 28 October 2016 4:34 PM > *Subject:* Re: Kafka Multi DataCenter HA/Failover > > > What is the latency between the two datacenters? I ask because unless > they are very close, you probably don’t want to do any form of > synchronous replication. > The Confluent Replicator (coming very soon in Confluent Enterprise > 3.1) will do async replication of both messages and configuration > metadata between datacenters. > It’s still up to you to monitor for what your app considers acceptable > lag between the two datacenters but at least now that is possible > using timestamps and the new offsetsForTimes() capability added in 0.10.1. > Accurate message timestamp based offset lookup is necessary because > the offset numbers for a given message will not match in both datacenters. > > -hans > > On Oct 28, 2016, at 8:08 AM, Mudit Agarwal > <mudit...@yahoo.com.INVALID> > wrote: > > Thanks dave. > Any ways for how we can achieve HA/Failover in kafka across two DC? > Thanks,Mudit > > From: "Tauzell, Dave" <dave.tauz...@surescripts.com> > To: "users@kafka.apache.org" <users@kafka.apache.org>; Mudit Agarwal < > mudit...@yahoo.com> > Sent: Friday, 28 October 2016 4:02 PM > Subject: RE: Kafka Multi DataCenter HA/Failover > > without any lag > > > You are going to have some lag at some point between datacenters. > > I haven't used this but from taking to them they are working or have > created a replacement for MirrorMaker using the Connect framework > which will fix a number of MirrorMaker issues. I haven't talked to > anybody about Kafka failoer. > > -Dave > > -----Original Message----- > From: Mudit Agarwal [mailto:mudit...@yahoo.com.INVALID > <mudit...@yahoo.com.INVALID>] > Sent: Friday, October 28, 2016 9:38 AM > To: Users > Subject: Kafka Multi DataCenter HA/Failover > > Hi, > I learned that Confluent Enterprise provides Multi DC failover and HA > synchronously and without any lag.I'm looking to learn further > information and more detailed documentation on this.I have gone > thorugh the white paper and it just talks about Replicator. > Any pointers for more information will be helpful. > Thanks,Mudit > This e-mail and any files transmitted with it are confidential, may > contain sensitive information, and are intended solely for the use of > the individual or entity to whom they are addressed. If you have > received this e-mail in error, please notify the sender by reply > e-mail immediately and destroy all copies of the e-mail and any attachments. > > > > > > This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.
