Hello again, I am not using kafka-spouts and I did not run any benchmarks on my setup. Can I ask what is your parallelism hint on the kafka-spouts? Because, it might be the case that 7000 t/s might be the ceiling for your parallelism hint and you may have to increase that in order to achieve higher throughput.
Cheers, Nikos On Mon, Apr 18, 2016 at 1:56 AM, Anandh Kumar <[email protected]> wrote: > Nikos, > > Thanks for your reply. > > I tried many value from low to high of max.spout.pending but not getting > throughput more than 7000/s. > > I tried turning internal buffer but no improvement. > > Did you tried any benchmark test on storm kafkaspout ? how much you got? > given the code so that I can run same test in my environment. > > Regards, > -Anandh Kumar > > On Sat, Apr 16, 2016 at 8:44 PM, Nikos R. Katsipoulakis < > [email protected]> wrote: > >> Hello Anandh, >> >> Why do you keep the max.spout.pending this low? I suggest increasing it >> to a higher value, because this might be the cause of your low throughput. >> In fact, it makes sense if you consider the following scenario: >> >> The KafkaSpout sends out 250 messages that try to fill out the Netty >> buffers before they are sent to the next node. When those are send, the >> Spout stops sending any additional tuples, until the ones "on-flight" are >> fully ACKed. Meanwhile, the Netty buffers are not full, therefore some time >> is spent up until they are sent out. Therefore, your throughput is >> implicitly "throttled" >> >> If I were you, I would keep the default values on the buffers (or even >> increase them a little - common practice for high throughput) and also try >> with a bigger value on max.spout.pending. >> >> Cheers, >> Nikos >> >> On Sat, Apr 16, 2016 at 11:06 AM, John Yost <[email protected]> wrote: >> >>> Hi Anandh, >>> >>> I recommend profiling your workers with something like jvisualvm to see >>> where your Bolt executors are spending their time. >>> >>> --John >>> >>> On Sat, Apr 16, 2016 at 10:01 AM, Anandh Kumar <[email protected]> >>> wrote: >>> >>>> Hi Team, >>>> >>>> I benchmark the storm 0.10.0 with KafkaSpout and simple bolt not >>>> getting throughput more than 7000 per seconds. >>>> >>>> Right now getting following throughput. >>>> >>>> Ack thread: 4 >>>> max.spout.pending: 250 >>>> >>>> With KafkaSpout with simple bolt >>>> >>>> Throughput - 7000/s message >>>> >>>> ====================================================== >>>> >>>> Ack thread: 4 >>>> max.spout.pending: 250 >>>> >>>> With KafkaSpout with* no bolt * >>>> >>>> Throughput - 100000/s message >>>> >>>> ==================================================== >>>> >>>> Why with bolt I am getting very less throughput? >>>> >>>> Not getting thoughput more than 7000/s. Tried to tune all the buffer >>>> parameters( send,receive buffer) not use. >>>> >>>> Please help me. >>>> >>>> Regards, >>>> -Anandh Kumar >>>> >>> >>> >> >> >> -- >> Nikos R. Katsipoulakis, >> Department of Computer Science >> University of Pittsburgh >> > > -- Nikos R. Katsipoulakis, Department of Computer Science University of Pittsburgh
