Nikos, Thanks for your reply.
I tried many value from low to high of max.spout.pending but not getting throughput more than 7000/s. I tried turning internal buffer but no improvement. Did you tried any benchmark test on storm kafkaspout ? how much you got? given the code so that I can run same test in my environment. Regards, -Anandh Kumar On Sat, Apr 16, 2016 at 8:44 PM, Nikos R. Katsipoulakis < [email protected]> wrote: > Hello Anandh, > > Why do you keep the max.spout.pending this low? I suggest increasing it to > a higher value, because this might be the cause of your low throughput. In > fact, it makes sense if you consider the following scenario: > > The KafkaSpout sends out 250 messages that try to fill out the Netty > buffers before they are sent to the next node. When those are send, the > Spout stops sending any additional tuples, until the ones "on-flight" are > fully ACKed. Meanwhile, the Netty buffers are not full, therefore some time > is spent up until they are sent out. Therefore, your throughput is > implicitly "throttled" > > If I were you, I would keep the default values on the buffers (or even > increase them a little - common practice for high throughput) and also try > with a bigger value on max.spout.pending. > > Cheers, > Nikos > > On Sat, Apr 16, 2016 at 11:06 AM, John Yost <[email protected]> wrote: > >> Hi Anandh, >> >> I recommend profiling your workers with something like jvisualvm to see >> where your Bolt executors are spending their time. >> >> --John >> >> On Sat, Apr 16, 2016 at 10:01 AM, Anandh Kumar <[email protected]> >> wrote: >> >>> Hi Team, >>> >>> I benchmark the storm 0.10.0 with KafkaSpout and simple bolt not getting >>> throughput more than 7000 per seconds. >>> >>> Right now getting following throughput. >>> >>> Ack thread: 4 >>> max.spout.pending: 250 >>> >>> With KafkaSpout with simple bolt >>> >>> Throughput - 7000/s message >>> >>> ====================================================== >>> >>> Ack thread: 4 >>> max.spout.pending: 250 >>> >>> With KafkaSpout with* no bolt * >>> >>> Throughput - 100000/s message >>> >>> ==================================================== >>> >>> Why with bolt I am getting very less throughput? >>> >>> Not getting thoughput more than 7000/s. Tried to tune all the buffer >>> parameters( send,receive buffer) not use. >>> >>> Please help me. >>> >>> Regards, >>> -Anandh Kumar >>> >> >> > > > -- > Nikos R. Katsipoulakis, > Department of Computer Science > University of Pittsburgh >
