Sorry, I m not sure about the max or ideal cap for number of threads for Kafka bolt. But my guess is that, it will be more of a function of traffic than mere number of topics and partition. Moreover, in my set up of kafkabolt, each bolt handles traffic of one topic which then eventually handled by different storm bolts. Can you please explain why do you think the number of thread should be specific for a kafka bolt or how did you arrived to that formula ?
That may help understanding the approach more. On Wed, Jan 13, 2016 at 4:50 AM, John Yost <[email protected]> wrote: > Hi Millind, > > I apologize, I need to ask my question more precisely. :) I know that I > set the number of executors for storm, but the question is specifically in > regards to the KafkaBolt. I am thinking that the max number of KafkaBolt > executors should be no more than the number of topics times the number of > partitions. In the example I presented in this thread, that would be 2 > topics * 10 partitions per topic = 20. > > Just wondering if my logic makes sense and/or if there is a better > parallelism strategy for KafkaBolts. > > Thanks > > --John > > On Wed, Jan 13, 2016 at 2:44 AM, Milind Vaidya <[email protected]> wrote: > >> Hi John, >> >> No. It is not driven by number of topics or partitions. This is a number >> you can configure while setting the bolt into the topology builder. >> >> Here is useful link to clarify more : >> http://storm.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html >> >> >> >> On Tue, Jan 12, 2016 at 3:00 PM, John Yost <[email protected]> wrote: >> >>> Hi Everyone, >>> >>> Is the number of KafkaBolt executors driven by number of topics X number >>> of partitions? For example, if I have two topics, each of which has ten >>> partitions, does 20 KafkaBolt executors make sense? Just trying to figure >>> out the optimal parallelism. >>> >>> Any insights much appreciated--thanks! >>> >>> --John >>> >> >> >
