Hi Jungtaek and Roshan, Thank you for your replies and the suggestions.
I just did a more detailed evaluation for this messaging delay issue. 1. I use ping and iPerf and get the numbers of 0.193ms and 934Mbps. 2. I use scp to transmit the image m0.png(1.7MB) with the delay of 0.072s. 3. I convert two m0.png to a single byte array with the length of 9331200(8.89MB). 4. I write a udp socket with the buffer size of 1024 to transmit the 8.89MB buffer in 80ms. 5. I send the 8.89MB buffer from the spout to a bolt in 93ms. The storm message will introduce 10ms more delay. I may get previous numbers because the heavy computing overhead and the tuple is buffered in the queue. Thanks a lot and motivate me to find this. I will try to consider encode/decode images to reduce the transmission latency. Best Regards, Wuyang ᐧ On Wed, Jan 31, 2018 at 7:18 PM, Roshan Naik <[email protected]> wrote: > Continuing with Jungtaek’s line of thinking… would also like to know > > - What latencies are you able to achieve when directly > transmitting a 5mb image between two nodes (not using Storm) ? And > similarly,… within two processes on the same node. ? > > - And how are you measuring it ? > > -roshan > > > > *From: *Jungtaek Lim <[email protected]> > *Reply-To: *"[email protected]" <[email protected]> > *Date: *Wednesday, January 31, 2018 at 3:38 PM > *To: *"[email protected]" <[email protected]> > *Subject: *Re: Apache Storm High Messaging Delay When Passing 5MB Images > > > > I meant "easily seen" as "not exposed" and we are easy to miss to consider. > > > > 2018년 2월 1일 (목) 오전 8:36, Jungtaek Lim <[email protected]>님이 작성: > > I'm not clear whether you're saying message transfer for each bolt took > 200ms, or summation of 4 or 5 times network transfer latencies were 200 ms. > > > > Why I say I'm not clear is that if it's latter and there's 5 times network > transfers, it is ideal latency in theory, since 1Gbps is 125MBps (1000Mbps, > not 1024Mbps) and 5M/125MBps = 40ms per each transfer. (I'd rather suspect > how it was possible in this case.) > > > > Even there's 4 times network transfers, we may need to take this to > account: the latency calculation above is in theory, and there're many > overheads other than messaging which is not easily seen, so the latency may > not be from messaging overhead. > > > > If it took 200 ms for each transfer that can be a thing to talk about. > Please let me know which is your case. > > > > Thanks, > > Jungtaek Lim (HeartSaVioR) > > > > 2018년 2월 1일 (목) 오전 8:11, Wuyang Zhang <[email protected]>님이 작성: > > I am playing with Apache Storm for a real-time image processing > application which requires ultra low latency. In the topology definition, a > single spout will emit raw images(5MB) in every 1s and a few bolts will > process them. The processing latency of each bolt is acceptable and the > overall computing delay can be around 150ms. > > *However, I find that the message passing delay between workers on the > different nodes is really high. The overall such delay on the 5 successive > bolts is around 200ms.* To calculate this delay, I subtract all the task > latencies from the end-to-end latency. Moreover, I implement a timer bolt > and other processing bolts will register in this timer bolt to record the > timestamp before starting the real processing. By comparing the timestamps > of the bolts, I find the delay between each bolt is high as I previously > noticed. > > To analyze the source of this high additional delay, I firstly reduce the > sending interval to 1s and thus there should be no queuing delay due to the > high computing overheads. Also, from the Storm UI, I find none bolt is in > high CPU utilization. > > Then, I checked the network delay. I am using a 1Gbps network testbed and > test the network by RTT and bandwidth. The network latency should not be > that high to send a 5MB image. > > Finally, I am thinking about the buffer delay. I find each thread > maintains its own sending buffer and transfer the data to the worker's > sending buffer. I am not sure how long it takes before the receiver bolt > can get this sending message. As suggested by the community, I increase the > sender/receiver buffer size to 16384, modify STORM_NETTY_MESSAGE_BATCH_SIZE > to 32768. However, it did not help. > > *My question is that how to remove/reduce the messaging overheads between > bolts?(inter workers)* It is possible to synchronize the communication > between bolts and have the receiver got the sending messages immediately > without any delay? > > > > ᐧ > >
