Hi, We are using mirrormaker to replicate data between two kafka clusters in difference datacenter, a fiber connected them, 1G bandwidth, ~35ms lantency.
we found that the MessageIn in JMX beans with the broker in destination cluster greater than broker in source cluster. Our configuration like this: destination & source cluster both deployed kafka 0.7.0 source clusters has 6 brokers with 32 partitions each,target clusters has 12 brokers with 32 partitions each and 6 mirrormakers doing replication,replicate only one topic. mirrormaker configuration like this: --num.streams 128 --num.producers 16 ================================= consumer.properties: shallowiterator.enable=true socket.buffersize=31457280 fetch.size=1048576 ================================= producer.properties: queue.enqueueTimeout.ms <http://queue.enqueuetimeout.ms/>=-1 # the number of messages batched at the producer batch.size=4000 #default 5000 connect.timeout.ms=10000 #default 30000 socket.timeout.ms=300000 ================================== test result: MessageIn: source cluster => 252,560,169, target cluster => 253,128,669 another problem, sometimes one or more broker in source clusters has not be consumered during the mirrormakers replicate other brokers well. Does anyone have an idea to resolve them.
