how about the consumer lag of mirror maker? On Mon, Apr 13, 2015 at 1:33 PM, nitin sharma <kumarsharma.ni...@gmail.com> wrote:
> i just tested that too and below is the stats.. it is clear that with > "kafka-consumer-perf-test.sh", i am able to get a high throughput. around > 44.0213 MB/sec. > > Seriously some configuration needs to be tweaked in MirrorMaker > configuration for speedy processing... Can you think of something ? > > > start.time, end.time, fetch.size, data.consumed.in.MB, MB.sec, > data.consumed.in.nMsg, nMsg.sec > 2015-04-13 20:29:29:070, 2015-04-13 20:29:34:596, 1048576, 23.1552, > 44.0213, 16683, 31716.7300 > > Regards, > Nitin Kumar Sharma. > > > On Mon, Apr 13, 2015 at 3:51 PM, tao xiao <xiaotao...@gmail.com> wrote: > > > num.consumer.fetchers means the max number of fetcher threads that can be > > spawned. it doesn't necessarily mean you can get as many fetcher threads > as > > you specify. > > > > To me the metrics are suggesting a very slow consumption rate only 18.21 > > bytes/minute. Here is the benchmark Linkedin does > > > > > > > http://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines > > > > You should check if 18.21 bytes/minute is the max throughput you can get > on > > your machine with bin/kafka-consumer-perf-test.sh if this is case you > > definitely need to tune it > > > > On Mon, Apr 13, 2015 at 12:43 PM, nitin sharma < > > kumarsharma.ni...@gmail.com> > > wrote: > > > > > hi Xiao, > > > > > > i have finally got JMX monitoring enabled for my kafka nodes in test > > > envrionment and here is what i observed. > > > i was monitoring mbeans under "kafka.consumer" domain of JVM running > > "Kafka > > > Mirror Maker" process. > > > > > > ========================= > > > AllTopicsBytes ===> 18.21 bytes/minute > > > FetchRequestRateAndTimeMs ===> 9.69 Request/min and 99th Percentile is > > > 104.13ms. > > > ====================== > > > > > > Interesting thing is that I have specified "num.consumer.fetchers=200" > in > > > my consumer property file but i can see only 8 threads of type : > > > > > > > > > > > > "kafka.consumer":name="KafkaMaker1-ConsumerFetcherThread-KafkaMaker1_<<zkhost>>-1428952277321-5e044226-138-1-host_<<brokerhostname>>-port_9092-FetchRequestRateAndTimeMs",type="FetchRequestAndResponseMetrics" > > > > > > > > > Could this be the issue? > > > > > > note, my JVM is set to 1GB and only 30MB is utilized most of the time. > > > > > > > > > Regards, > > > Nitin Kumar Sharma. > > > > > > > > > On Wed, Apr 8, 2015 at 10:48 PM, tao xiao <xiaotao...@gmail.com> > wrote: > > > > > > > Metrics like Bytepersec, FetchRequestRateAndTimeMs can help you to > > check > > > if > > > > the consumer has problem processing messages > > > > > > > > On Thu, Apr 9, 2015 at 2:40 AM, nitin sharma < > > > kumarsharma.ni...@gmail.com> > > > > wrote: > > > > > > > > > thanks, but can you please tell which metrics could highlight the > > > factor > > > > > causing slow data migration by MirrorMaker? > > > > > > > > > > Regards, > > > > > Nitin Kumar Sharma. > > > > > > > > > > > > > > > On Tue, Apr 7, 2015 at 10:10 PM, tao xiao <xiaotao...@gmail.com> > > > wrote: > > > > > > > > > > > You may need to look into the consumer metrics and producer > metrics > > > to > > > > > > identify the root cause. metrics in kafka.consumer and > > kafka.producer > > > > > > categories will help you find out the problems. > > > > > > > > > > > > This link gives instruction how to read the metrics > > > > > > http://kafka.apache.org/documentation.html#monitoring > > > > > > > > > > > > > > > > > > On Wed, Apr 8, 2015 at 3:39 AM, nitin sharma < > > > > > kumarsharma.ni...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > hi, > > > > > > > > > > > > > > sorry for late response. ... i have been able to fix the issue > .. > > > > > problem > > > > > > > was in my approach. I got confused between my source and target > > > > system > > > > > > > while defining consumer & producer property file .. it is fixed > > now > > > > > > > > > > > > > > Now new issue.. the rate at which data is migrated is very very > > > > > slow... i > > > > > > > mean it took 5 min to copy only 15Kb.. :( .. here are the > > property > > > > for > > > > > > > producer and consumer.. there is no network latency between > > Source > > > > and > > > > > > > Destination clusters as such. > > > > > > > > > > > > > > > > > > > > > #### Producer ########### > > > > > > > metadata.broker.list=<broker1IP>:9092,<broker2IP>:9092 > > > > > > > serializer.class=kafka.serializer.DefaultEncoder > > > > > > > auto.create.topics.enable=true > > > > > > > request.required.acks=1 > > > > > > > request.required.acks=1 > > > > > > > producer.type=async > > > > > > > batch.num.messages=3000 > > > > > > > queue.buffering.max.ms=5000 > > > > > > > queue.buffering.max.messages=100000 > > > > > > > queue.enqueue.timeout.ms=-1 > > > > > > > socket.send.buffer.bytes=5282880 > > > > > > > > > > > > > > ######### Consumer ########### > > > > > > > > > > > > > > > > > > > > > > > > > > > > zookeeper.connect=<zk1hostname>:2181,<zk2hostname>:2181,<zk3hostname>:2181 > > > > > > > group.id=KafkaMaker > > > > > > > auto.create.topics.enable=true > > > > > > > socket.receive.buffer.bytes=5243880 > > > > > > > zookeeper.connection.timeout.ms=1000000 > > > > > > > num.consumer.fetchers=20 > > > > > > > fetch.message.max.bytes=5243880 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > Nitin Kumar Sharma. > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 31, 2015 at 12:36 PM, tao xiao < > xiaotao...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > > > Can you attach your mirror maker log? > > > > > > > > > > > > > > > > On Wed, Apr 1, 2015 at 12:28 AM, nitin sharma < > > > > > > > kumarsharma.ni...@gmail.com > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > i tried with auto.offset.reset=smallest, but still not > > > working.. > > > > > > > > > > > > > > > > > > there is data in my source cluster > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > Nitin Kumar Sharma. > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 30, 2015 at 10:30 PM, tao xiao < > > > xiaotao...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Do you have data sending to *testtopic? *By default > mirror > > > > maker > > > > > > only > > > > > > > > > > consumes data being sent after it taps into the topic. > you > > > need > > > > > to > > > > > > > keep > > > > > > > > > > sending data to the topic after mirror maker connection > is > > > > > > > established. > > > > > > > > > If > > > > > > > > > > you want to change the behavior you can set > > > > > > > auto.offset.reset=smallest > > > > > > > > so > > > > > > > > > > that any new mirror maker coming to the topic will start > > from > > > > the > > > > > > > > > smallest > > > > > > > > > > offset > > > > > > > > > > > > > > > > > > > > On Tue, Mar 31, 2015 at 3:53 AM, nitin sharma < > > > > > > > > > kumarsharma.ni...@gmail.com > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > thanks Xiao > > > > > > > > > > > > > > > > > > > > > > I tried MirrorMaker option in my test environment but > > > > failed. I > > > > > > am > > > > > > > > not > > > > > > > > > > > able to see the log getting copied to destination > > cluster. > > > I > > > > > see > > > > > > in > > > > > > > > the > > > > > > > > > > log > > > > > > > > > > > of MirrorMaker process that connection is successfully > > > > > > established > > > > > > > > > > between > > > > > > > > > > > source and destination cluster but still not sure what > i > > > > > causing > > > > > > > the > > > > > > > > > > problem > > > > > > > > > > > > > > > > > > > > > > Env. Setup ==> > > > > > > > > > > > > > > > > > > > > > > I). Source Cluster (Qenv02) -- i have 2 > > > > > > > broker(Qenv02kf01,Qenv02kf02) > > > > > > > > > and > > > > > > > > > > > 3 zk(Qenv02zk01,Qenv02zk02 and Qenv02zk03). > > > > > > > > > > > Destination Clustern (Qenv05) -- i have 2 broker > > > > > > > > > (Qenv05kf01,Qenv05kf02) > > > > > > > > > > > and 3 zk(Qenv05zk01,Qenv05zk02 and Qenv05zk03). > > > > > > > > > > > > > > > > > > > > > > II). i have kept consumer and producer properties file > in > > > one > > > > > of > > > > > > > the > > > > > > > > > > > source kafka broker config folder. > > > > > > > > > > > > > > > > > > > > > > III).i have executed following command from the same > > kafka > > > > > broker > > > > > > > to > > > > > > > > > > > start the process.. log are attached : > > > > > > > > > > > > > > > > > > > > > > /app/kafka/bin/kafka-run-class.sh > kafka.tools.MirrorMaker > > > > > > > > > > -consumer.config > > > > > > > > > > > /app/kafka/config/consumer1.properties --num.streams=2 > > > > > > > > > --producer.config > > > > > > > > > > > /app/kafka/config/producer1.properties --whitelist > > > testtopic > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IV). I tried Consumer offset tracker tool also, while > > > Mirror > > > > > > Maker > > > > > > > > > > running > > > > > > > > > > > . I tried by launching second session of same broker > > where > > > > > mirror > > > > > > > > maker > > > > > > > > > > is > > > > > > > > > > > running. I got error message that "*NoNode for > > > > > > > > > > > /consumers/KafkaMaker/offsets/testtopic/0*" .Complete > log > > > > > > attached. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > Nitin Kumar Sharma. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Mar 26, 2015 at 11:24 AM, tao xiao < > > > > > xiaotao...@gmail.com > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > >> Both consumer-1 and consumer-2 are properties of > source > > > > > clusters > > > > > > > > > mirror > > > > > > > > > > >> maker transfers data from. Mirror maker is designed to > > be > > > > able > > > > > > to > > > > > > > > > > consume > > > > > > > > > > >> data from N sources (N >= 1) and transfer data to one > > > > > > destination > > > > > > > > > > cluster. > > > > > > > > > > >> You are free to supply as many consumer properties as > > you > > > > want > > > > > > to > > > > > > > > > > instruct > > > > > > > > > > >> mirror maker where to consumer data from. > > > > > > > > > > >> > > > > > > > > > > >> On Thu, Mar 26, 2015 at 9:50 PM, nitin sharma < > > > > > > > > > > >> kumarsharma.ni...@gmail.com> > > > > > > > > > > >> wrote: > > > > > > > > > > >> > > > > > > > > > > >> > thanks Mayuresh and Jiangjie for your response. > > > > > > > > > > >> > > > > > > > > > > > >> > I have actually not understood Mirror maker clearly > > and > > > > > hence > > > > > > > bit > > > > > > > > > > >> skeptical > > > > > > > > > > >> > if i will be able to execute it effectively. > > > > > > > > > > >> > > > > > > > > > > > >> > Online i hv seen the following command to be > execute, > > > but > > > > > not > > > > > > > > > > understood > > > > > > > > > > >> > what is consumer-1 & -2.properties here? do i need > to > > > > copy > > > > > > from > > > > > > > > my > > > > > > > > > > >> > consumer code? also, any reason why to provide > > consumer > > > > > > > property? > > > > > > > > > > >> > > > > > > > > > > > >> > bin/kafka-run-class.sh kafka.tools.MirrorMaker > > > > > > --consumer.config > > > > > > > > > > >> > consumer-1.properties --consumer.config > > > > > consumer-2.properties > > > > > > > > > > >> > --producer.config producer.properties --whitelist > > > my-topic > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > Regards, > > > > > > > > > > >> > Nitin Kumar Sharma. > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > On Wed, Mar 25, 2015 at 8:57 PM, Mayuresh Gharat < > > > > > > > > > > >> > gharatmayures...@gmail.com > > > > > > > > > > >> > > wrote: > > > > > > > > > > >> > > > > > > > > > > > >> > > You can use the Mirror maker to move data from one > > > data > > > > > > center > > > > > > > > to > > > > > > > > > > >> other > > > > > > > > > > >> > and > > > > > > > > > > >> > > once all the data has been moved you can shut down > > the > > > > > > source > > > > > > > > data > > > > > > > > > > >> center > > > > > > > > > > >> > > by doing a controlled shutdown. > > > > > > > > > > >> > > > > > > > > > > > > >> > > Thanks, > > > > > > > > > > >> > > > > > > > > > > > > >> > > Mayuresh > > > > > > > > > > >> > > > > > > > > > > > > >> > > On Wed, Mar 25, 2015 at 2:35 PM, Jiangjie Qin > > > > > > > > > > >> <j...@linkedin.com.invalid > > > > > > > > > > >> > > > > > > > > > > > > >> > > wrote: > > > > > > > > > > >> > > > > > > > > > > > > >> > > > If you want to do a seamless migration. I think > a > > > > better > > > > > > way > > > > > > > > is > > > > > > > > > to > > > > > > > > > > >> > build > > > > > > > > > > >> > > a > > > > > > > > > > >> > > > cross datacenter Kafka cluster temporarily. So > the > > > > > process > > > > > > > is: > > > > > > > > > > >> > > > 1. Add several new Kafka brokers in your new > > > > datacenter > > > > > > and > > > > > > > > add > > > > > > > > > > >> them to > > > > > > > > > > >> > > > the old cluster. > > > > > > > > > > >> > > > 2. Use replica assignment tool to reassign all > the > > > > > > > partitions > > > > > > > > to > > > > > > > > > > >> > brokers > > > > > > > > > > >> > > > in new datacenter. > > > > > > > > > > >> > > > 3. Perform controlled shutdown on the brokers in > > old > > > > > > > > datacenter. > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > Jiangjie (Becket) Qin > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > On 3/25/15, 2:01 PM, "nitin sharma" < > > > > > > > > > kumarsharma.ni...@gmail.com> > > > > > > > > > > >> > wrote: > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > >Hi Team, > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > >in my project, we have built a new datacenter > for > > > > Kafka > > > > > > > > brokers > > > > > > > > > > and > > > > > > > > > > >> > > wants > > > > > > > > > > >> > > > >to migrate from current datacenter to new one. > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > >Switching producers and consumers wont be a > > problem > > > > > > > provided > > > > > > > > > New > > > > > > > > > > >> > > > >Datacenter > > > > > > > > > > >> > > > >has all the messages of existing Datacenter. > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > >i have only 1 topic with 2 partition that need > to > > > be > > > > > > > > > migrated... > > > > > > > > > > >> that > > > > > > > > > > >> > > too > > > > > > > > > > >> > > > >it is only 1 time activity. > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > >Kindly suggest the best way to deal with this > > > > > situation. > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > >Regards, > > > > > > > > > > >> > > > >Nitin Kumar Sharma. > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > -- > > > > > > > > > > >> > > -Regards, > > > > > > > > > > >> > > Mayuresh R. Gharat > > > > > > > > > > >> > > (862) 250-7125 > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> -- > > > > > > > > > > >> Regards, > > > > > > > > > > >> Tao > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Regards, > > > > > > > > > > Tao > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Regards, > > > > > > > > Tao > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Regards, > > > > > > Tao > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Regards, > > > > Tao > > > > > > > > > > > > > > > -- > > Regards, > > Tao > > > -- Regards, Tao
