Any exceptions in the mirror maker logs? may be you can enable mirror maker trace logs. maybe all messages are having same key? Can you recheck partition count on target cluster?
On Thu, Nov 2, 2017 at 2:45 AM, Chris Neal <cwn...@gmail.com> wrote: > Apologies for bumping my own post, but really hoping someone has experience > with MirrorMaker who might be able to help with my question below. :) > > Thanks! > Chris > > On Fri, Oct 27, 2017 at 10:59 AM, Chris Neal <cwn...@gmail.com> wrote: > > > Hi all, > > > > I've been struggling with this for awhile and need some help please. :) > > > > I have a source cluster running Kafka 0.8.1.1 with about 900 topics. > > I have a target cluster running Kafka 0.10.2 with those same topics > > pre-created with the same number of partitions per topic as the source > > cluster. > > > > I'm running Mirror Maker from the 0.8.1.1 Kafka version, and cannot seem > > to get the produced records to go anywhere besides partition 0 of all > these > > topics. :( > > > > I found this page to be helpful WRT performance tuning, but it did not > > solve my partition producer problem: > > > > https://community.hortonworks.com/articles/79891/kafka- > > mirror-maker-best-practices.html > > > > My consumer.config: > > ====================================== > > zookeeper.connect=zk1:2181,zk2:2181,zk3:2181 > > group.id=prod_perf_mirrormaker2_new1 > > num.consumer.fetchers=1100 > > client.id=MirrorMakerConsumer_perf > > ====================================== > > > > My producer config: > > ====================================== > > metadata.broker.list=kafka1:9092,kafka2:9092,kafka3:9092 > > compression.codec=gzip > > producer.type=async > > batch.num.messages=1000 > > message.send.max.retries=2 > > client.id=MirrorMakerProducer_perf > > ====================================== > > > > My start script: > > ====================================== > > #!/bin/sh > > > > export KAFKA_HEAP_OPTS="-Xmx2G -Xms2G" > > > > /home/chris.neal/kafka_2.9.2-0.8.1.1/bin/kafka-run-class.sh > > kafka.tools.MirrorMaker \ > > --producer.config /home/chris.neal/mirror_maker/ > prod_perf/producer.properties > > \ > > --consumer.config /home/chris.neal/mirror_maker/ > prod_perf/consumer.properties > > \ > > --num.streams=10 \ > > --whitelist '<snip long list of about 60 topics>' > > ====================================== > > > > I do know that there is some combination of "num.streams" and > > "num.consumer.fetchers" that should get it working, but I can't figure > out > > what that combination is. > > > > All topics have the same setup of 10 partitions per topic. > > > > Can anyone shed some light on how to make this go across all 10 > > partitions, and not just partition 0? > > > > Thank you SO much for your time and help. > > Chris > > >
