Hi Vineet, Sounds great, will up the number of partitions! I basically have one KafkaBolt per topic partition, so I am going from 2 topics X 10 partitions = 20 executors to 2 topics X 20 partitions = 40 executors.
--John On Sun, Jan 10, 2016 at 1:39 PM, Vineet Mishra <[email protected]> wrote: > Hi John, > > Increasing the Kafka partition should be helpful. > > Additionally what is the KafkaBolt parallelism you are keeping? > > Cheers! > VM > On Jan 10, 2016 11:33 PM, "John Yost" <[email protected]> wrote: > >> Hi Everyone, >> >> I am attempting to use the KafkaBolt for output from my Storm topology. >> Whereas the throughput from the KafkaSpout is screaming fast, output to >> Kafka via the KafkaBolt is comparatively really slow. I've tried outputting >> with request.required.acks set to 0,1,-1 which does not affect throughput >> significantly. >> >> I am thinking that one issue is that the KafkaSpout is reading off of a >> topic with 20 partitions, whereas the KafkaBolt is writing to a topic with >> 10 partitions. I am going to expand the write topic to twenty partitions >> tomorrow and retry as I think that increasing the producer group size via a >> greater number of partitions should help. >> >> Any other ideas/guidance would be great--thanks! >> >> --John >> >
