Continuing with Jungtaek’s line of thinking… would also like to know - What latencies are you able to achieve when directly transmitting a 5mb image between two nodes (not using Storm) ? And similarly,… within two processes on the same node. ?
- And how are you measuring it ? -roshan From: Jungtaek Lim <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Wednesday, January 31, 2018 at 3:38 PM To: "[email protected]" <[email protected]> Subject: Re: Apache Storm High Messaging Delay When Passing 5MB Images I meant "easily seen" as "not exposed" and we are easy to miss to consider. 2018년 2월 1일 (목) 오전 8:36, Jungtaek Lim <[email protected]<mailto:[email protected]>>님이 작성: I'm not clear whether you're saying message transfer for each bolt took 200ms, or summation of 4 or 5 times network transfer latencies were 200 ms. Why I say I'm not clear is that if it's latter and there's 5 times network transfers, it is ideal latency in theory, since 1Gbps is 125MBps (1000Mbps, not 1024Mbps) and 5M/125MBps = 40ms per each transfer. (I'd rather suspect how it was possible in this case.) Even there's 4 times network transfers, we may need to take this to account: the latency calculation above is in theory, and there're many overheads other than messaging which is not easily seen, so the latency may not be from messaging overhead. If it took 200 ms for each transfer that can be a thing to talk about. Please let me know which is your case. Thanks, Jungtaek Lim (HeartSaVioR) 2018년 2월 1일 (목) 오전 8:11, Wuyang Zhang <[email protected]<mailto:[email protected]>>님이 작성: I am playing with Apache Storm for a real-time image processing application which requires ultra low latency. In the topology definition, a single spout will emit raw images(5MB) in every 1s and a few bolts will process them. The processing latency of each bolt is acceptable and the overall computing delay can be around 150ms. However, I find that the message passing delay between workers on the different nodes is really high. The overall such delay on the 5 successive bolts is around 200ms. To calculate this delay, I subtract all the task latencies from the end-to-end latency. Moreover, I implement a timer bolt and other processing bolts will register in this timer bolt to record the timestamp before starting the real processing. By comparing the timestamps of the bolts, I find the delay between each bolt is high as I previously noticed. To analyze the source of this high additional delay, I firstly reduce the sending interval to 1s and thus there should be no queuing delay due to the high computing overheads. Also, from the Storm UI, I find none bolt is in high CPU utilization. Then, I checked the network delay. I am using a 1Gbps network testbed and test the network by RTT and bandwidth. The network latency should not be that high to send a 5MB image. Finally, I am thinking about the buffer delay. I find each thread maintains its own sending buffer and transfer the data to the worker's sending buffer. I am not sure how long it takes before the receiver bolt can get this sending message. As suggested by the community, I increase the sender/receiver buffer size to 16384, modify STORM_NETTY_MESSAGE_BATCH_SIZE to 32768. However, it did not help. My question is that how to remove/reduce the messaging overheads between bolts?(inter workers) It is possible to synchronize the communication between bolts and have the receiver got the sending messages immediately without any delay? ᐧ
