Things like consumer rebalances on the cluster you copy from, and brokers going down on the cluster your writing down can cause duplications. The default settings are set to prevent data loss, making data duplication more likely to happen in case of error. You could possibly make a simple consumer to check for duplicates, depending on your data.
On Mon, Jul 25, 2016 at 8:38 AM Sathyakumar Seshachalam < sathyakumar_seshacha...@trimble.com> wrote: > Am trying to mirror from a production Kafka cluster to a DR cluster. > > However the offsets between topics (retrieved with GetOffsetShell > < > https://cwiki.apache.org/confluence/display/KAFKA/System+Tools#SystemTools-GetOffsetShell > >) > on these two clusters do not always match. > > While a lesser offset is the "mirrored" cluster is understandable assuming > mirror maker is still matching, what is the theoretical possibility of > offset being higher in the mirrored cluster ? (As this is what is happening > in my case. > > Thanks, > Sathya >
