It is a 5 machines os 4 core 8 gb. Parallelism is set to 100. Whether I have 1000 messages or 100, I see greeter than 10-15ms from the time bolt emits and the next bolt consumes... :(
Vidya, thanks. Will check the ITaskHook. What is the typical latency you guys see? Thanks Kashyap On Jun 22, 2015 22:50, "Srividhya Shanmugam" <[email protected]> wrote: > Another approach is to add a ITaskHook implementation to understand the > actual latency in your bolt for every ack..An example using ITaskHook can > be found at > http://iamparv.blogspot.com/2013/06/storm-itaskhook-implementation.html > > > > Using the BoltAckInfo, we can find the actuals. > > > > *From:* Seungtack Baek [mailto:[email protected]] > *Sent:* Monday, June 22, 2015 2:31 PM > *To:* [email protected] > *Subject:* Re: How fast can bolt access emitted data > > > > You might want to check your parallelism. Are all those "10" messages in > bolt B,C,etc are being processed in parallel? How many workers are you > using? How many cores do you have for each node? > > > [image: Image removed by sender.] > > *Seungtack Baek | Precocity, LLC* > > Tel/Direct: (972) 378-1030 | Mobile: (214) 477-5715 > > *[email protected] <[email protected]>* | > www.precocityllc.com > > > > This is the end of this message. > > -- > > > > On Mon, Jun 22, 2015 at 11:30 AM, Kashyap Mhaisekar <[email protected]> > wrote: > > Has no one faced this issue? > > Thanks > Kashyap > > On Jun 21, 2015 14:57, "Kashyap Mhaisekar" <[email protected]> wrote: > > Hi, > > I have a topology like the below - > > A (Spout) ->(Emits a number say 1000) -> B (bolt) [Receives this number > and divides this into 10 emits of 100 each) -> C (bolt) [Recieves these > emits and divides this to 10 emits of 10 numbers) -> D (bolt) [Does some > computation on the number and emits one message] -> E (bolt) [Aggregates > all the data and confirms if all the 1000 messages are processed) > > > > Every bolt takes under 3 msec to complete and as a result, I estimated > that the end to end processing for 1000 takes not more than 50 msec > including any latencies. But what I noticed is that its takes close to 200 > msec and the Emit from a spout/bolt and getting it to bolt takes close to 6 > ms on a an average. I am not able to figure out why it takes so much time > to come between a spout/bolt to next bolt. I understand that the spout/bolt > buffers the data into a queue and then the subsequent bolt consumes from > there. > > > > Any pointers on this will help. > > > > Thanks > > Kashyap > > > > This email and any files transmitted with it are confidential, proprietary > and intended solely for the individual or entity to whom they are > addressed. If you have received this email in error please delete it > immediately. >
