thanks, but can you please tell which metrics could highlight the factor causing slow data migration by MirrorMaker?
Regards, Nitin Kumar Sharma. On Tue, Apr 7, 2015 at 10:10 PM, tao xiao <xiaotao...@gmail.com> wrote: > You may need to look into the consumer metrics and producer metrics to > identify the root cause. metrics in kafka.consumer and kafka.producer > categories will help you find out the problems. > > This link gives instruction how to read the metrics > http://kafka.apache.org/documentation.html#monitoring > > > On Wed, Apr 8, 2015 at 3:39 AM, nitin sharma <kumarsharma.ni...@gmail.com> > wrote: > > > hi, > > > > sorry for late response. ... i have been able to fix the issue .. problem > > was in my approach. I got confused between my source and target system > > while defining consumer & producer property file .. it is fixed now > > > > Now new issue.. the rate at which data is migrated is very very slow... i > > mean it took 5 min to copy only 15Kb.. :( .. here are the property for > > producer and consumer.. there is no network latency between Source and > > Destination clusters as such. > > > > > > #### Producer ########### > > metadata.broker.list=<broker1IP>:9092,<broker2IP>:9092 > > serializer.class=kafka.serializer.DefaultEncoder > > auto.create.topics.enable=true > > request.required.acks=1 > > request.required.acks=1 > > producer.type=async > > batch.num.messages=3000 > > queue.buffering.max.ms=5000 > > queue.buffering.max.messages=100000 > > queue.enqueue.timeout.ms=-1 > > socket.send.buffer.bytes=5282880 > > > > ######### Consumer ########### > > > zookeeper.connect=<zk1hostname>:2181,<zk2hostname>:2181,<zk3hostname>:2181 > > group.id=KafkaMaker > > auto.create.topics.enable=true > > socket.receive.buffer.bytes=5243880 > > zookeeper.connection.timeout.ms=1000000 > > num.consumer.fetchers=20 > > fetch.message.max.bytes=5243880 > > > > > > > > > > Regards, > > Nitin Kumar Sharma. > > > > > > On Tue, Mar 31, 2015 at 12:36 PM, tao xiao <xiaotao...@gmail.com> wrote: > > > > > Can you attach your mirror maker log? > > > > > > On Wed, Apr 1, 2015 at 12:28 AM, nitin sharma < > > kumarsharma.ni...@gmail.com > > > > > > > wrote: > > > > > > > i tried with auto.offset.reset=smallest, but still not working.. > > > > > > > > there is data in my source cluster > > > > > > > > Regards, > > > > Nitin Kumar Sharma. > > > > > > > > > > > > On Mon, Mar 30, 2015 at 10:30 PM, tao xiao <xiaotao...@gmail.com> > > wrote: > > > > > > > > > Do you have data sending to *testtopic? *By default mirror maker > only > > > > > consumes data being sent after it taps into the topic. you need to > > keep > > > > > sending data to the topic after mirror maker connection is > > established. > > > > If > > > > > you want to change the behavior you can set > > auto.offset.reset=smallest > > > so > > > > > that any new mirror maker coming to the topic will start from the > > > > smallest > > > > > offset > > > > > > > > > > On Tue, Mar 31, 2015 at 3:53 AM, nitin sharma < > > > > kumarsharma.ni...@gmail.com > > > > > > > > > > > wrote: > > > > > > > > > > > thanks Xiao > > > > > > > > > > > > I tried MirrorMaker option in my test environment but failed. I > am > > > not > > > > > > able to see the log getting copied to destination cluster. I see > in > > > the > > > > > log > > > > > > of MirrorMaker process that connection is successfully > established > > > > > between > > > > > > source and destination cluster but still not sure what i causing > > the > > > > > problem > > > > > > > > > > > > Env. Setup ==> > > > > > > > > > > > > I). Source Cluster (Qenv02) -- i have 2 > > broker(Qenv02kf01,Qenv02kf02) > > > > and > > > > > > 3 zk(Qenv02zk01,Qenv02zk02 and Qenv02zk03). > > > > > > Destination Clustern (Qenv05) -- i have 2 broker > > > > (Qenv05kf01,Qenv05kf02) > > > > > > and 3 zk(Qenv05zk01,Qenv05zk02 and Qenv05zk03). > > > > > > > > > > > > II). i have kept consumer and producer properties file in one of > > the > > > > > > source kafka broker config folder. > > > > > > > > > > > > III).i have executed following command from the same kafka broker > > to > > > > > > start the process.. log are attached : > > > > > > > > > > > > /app/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker > > > > > -consumer.config > > > > > > /app/kafka/config/consumer1.properties --num.streams=2 > > > > --producer.config > > > > > > /app/kafka/config/producer1.properties --whitelist testtopic > > > > > > > > > > > > > > > > > > IV). I tried Consumer offset tracker tool also, while Mirror > Maker > > > > > running > > > > > > . I tried by launching second session of same broker where mirror > > > maker > > > > > is > > > > > > running. I got error message that "*NoNode for > > > > > > /consumers/KafkaMaker/offsets/testtopic/0*" .Complete log > attached. > > > > > > > > > > > > > > > > > > Regards, > > > > > > Nitin Kumar Sharma. > > > > > > > > > > > > > > > > > > On Thu, Mar 26, 2015 at 11:24 AM, tao xiao <xiaotao...@gmail.com > > > > > > wrote: > > > > > > > > > > > >> Both consumer-1 and consumer-2 are properties of source clusters > > > > mirror > > > > > >> maker transfers data from. Mirror maker is designed to be able > to > > > > > consume > > > > > >> data from N sources (N >= 1) and transfer data to one > destination > > > > > cluster. > > > > > >> You are free to supply as many consumer properties as you want > to > > > > > instruct > > > > > >> mirror maker where to consumer data from. > > > > > >> > > > > > >> On Thu, Mar 26, 2015 at 9:50 PM, nitin sharma < > > > > > >> kumarsharma.ni...@gmail.com> > > > > > >> wrote: > > > > > >> > > > > > >> > thanks Mayuresh and Jiangjie for your response. > > > > > >> > > > > > > >> > I have actually not understood Mirror maker clearly and hence > > bit > > > > > >> skeptical > > > > > >> > if i will be able to execute it effectively. > > > > > >> > > > > > > >> > Online i hv seen the following command to be execute, but not > > > > > understood > > > > > >> > what is consumer-1 & -2.properties here? do i need to copy > from > > > my > > > > > >> > consumer code? also, any reason why to provide consumer > > property? > > > > > >> > > > > > > >> > bin/kafka-run-class.sh kafka.tools.MirrorMaker > --consumer.config > > > > > >> > consumer-1.properties --consumer.config consumer-2.properties > > > > > >> > --producer.config producer.properties --whitelist my-topic > > > > > >> > > > > > > >> > > > > > > >> > Regards, > > > > > >> > Nitin Kumar Sharma. > > > > > >> > > > > > > >> > > > > > > >> > On Wed, Mar 25, 2015 at 8:57 PM, Mayuresh Gharat < > > > > > >> > gharatmayures...@gmail.com > > > > > >> > > wrote: > > > > > >> > > > > > > >> > > You can use the Mirror maker to move data from one data > center > > > to > > > > > >> other > > > > > >> > and > > > > > >> > > once all the data has been moved you can shut down the > source > > > data > > > > > >> center > > > > > >> > > by doing a controlled shutdown. > > > > > >> > > > > > > > >> > > Thanks, > > > > > >> > > > > > > > >> > > Mayuresh > > > > > >> > > > > > > > >> > > On Wed, Mar 25, 2015 at 2:35 PM, Jiangjie Qin > > > > > >> <j...@linkedin.com.invalid > > > > > >> > > > > > > > >> > > wrote: > > > > > >> > > > > > > > >> > > > If you want to do a seamless migration. I think a better > way > > > is > > > > to > > > > > >> > build > > > > > >> > > a > > > > > >> > > > cross datacenter Kafka cluster temporarily. So the process > > is: > > > > > >> > > > 1. Add several new Kafka brokers in your new datacenter > and > > > add > > > > > >> them to > > > > > >> > > > the old cluster. > > > > > >> > > > 2. Use replica assignment tool to reassign all the > > partitions > > > to > > > > > >> > brokers > > > > > >> > > > in new datacenter. > > > > > >> > > > 3. Perform controlled shutdown on the brokers in old > > > datacenter. > > > > > >> > > > > > > > > >> > > > Jiangjie (Becket) Qin > > > > > >> > > > > > > > > >> > > > On 3/25/15, 2:01 PM, "nitin sharma" < > > > > kumarsharma.ni...@gmail.com> > > > > > >> > wrote: > > > > > >> > > > > > > > > >> > > > >Hi Team, > > > > > >> > > > > > > > > > >> > > > >in my project, we have built a new datacenter for Kafka > > > brokers > > > > > and > > > > > >> > > wants > > > > > >> > > > >to migrate from current datacenter to new one. > > > > > >> > > > > > > > > > >> > > > >Switching producers and consumers wont be a problem > > provided > > > > New > > > > > >> > > > >Datacenter > > > > > >> > > > >has all the messages of existing Datacenter. > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > >i have only 1 topic with 2 partition that need to be > > > > migrated... > > > > > >> that > > > > > >> > > too > > > > > >> > > > >it is only 1 time activity. > > > > > >> > > > > > > > > > >> > > > >Kindly suggest the best way to deal with this situation. > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > >Regards, > > > > > >> > > > >Nitin Kumar Sharma. > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > >> > > -- > > > > > >> > > -Regards, > > > > > >> > > Mayuresh R. Gharat > > > > > >> > > (862) 250-7125 > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > > > > > >> -- > > > > > >> Regards, > > > > > >> Tao > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Regards, > > > > > Tao > > > > > > > > > > > > > > > > > > > > > -- > > > Regards, > > > Tao > > > > > > > > > -- > Regards, > Tao >
