Bhavesh, For keyed data, the mirror maker will just distribute data based on hash(key)%num_partitions. If num_partitions is different in the target DC (which it is), a message that lived in partition 0 in the source cluster might end up in partition 10 in the target cluster.
Thanks, Neha On Mon, Aug 11, 2014 at 7:23 PM, Bhavesh Mistry <mistry.p.bhav...@gmail.com> wrote: > Hi Guozhang, > > We are using Kafka 0.8.1 for all producer consumer and MM. > > We have 32 partition in source (local) per DC and we have 100 in target > (Central) DC. > > Is there any configuration on MM for this etc ? > > Thanks, > > Bhavesh > > > On Mon, Aug 11, 2014 at 4:33 PM, Guozhang Wang <wangg...@gmail.com> wrote: > > > Hi Bhavesh, > > > > What is the number of partitions on the source and target clusters, and > > what version of Kafka MM are you using? > > > > Guozhang > > > > > > On Mon, Aug 11, 2014 at 1:21 PM, Bhavesh Mistry < > > mistry.p.bhav...@gmail.com> > > wrote: > > > > > HI Kafka Dev Team, > > > > > > > > > > > > We have to aggregate events (count) per DC and across DCs for one of > > topic. > > > We have standard Linked-in data pipe line producers --> Local Brokers > --> > > > MM --> Center Brokers. > > > > > > > > > > > > So I would like to know How MM handles messages when custom > partitioning > > > logic is used as below and number of partition in target DC is SAME vs > > > different > > > than the source DC ? > > > > > > > > > > > > If we have key based messages and custom partitioning logic ( hash(key) > > % > > > number of partition per topic source topic) we want to count event > > > similar > > > event by hashing to same partition and count events, and but when same > > > event is MM to target DC will it go to same partition even though > number > > of > > > partition is different in target DC (meaning does MM will use hash(key > > > message) % number of partition) ? > > > > > > > > > > > > According to this reference, I do not have way to configure this or to > > > control which partitioning logic to use when MM data ? > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330 > > > > > > > > > Thanks, > > > > > > > > > > > > Bhavesh > > > > > > > > > > > -- > > -- Guozhang > > >
