Yes, it is the latter choice. I also insert the timestamps into the transmission layer and notice the overhead of seriliazation/deserilization.(10ms)
Right now passing images of large size is required when to pipeline the process. I will look for solutions to reduce the overall latency. Thanks again Jungtaek for your follow up discussion. On Jan 31, 2018 11:12 PM, "Jungtaek Lim" <[email protected]> wrote: > 4. I write a udp socket with the buffer size of 1024 to transmit the 8.89MB buffer in 80ms. > 5. I send the 8.89MB buffer from the spout to a bolt in 93ms. 1. The exact comparison should be testing with TCP instead of UDP: I think it will be more closer. 2. How did you measure the latency between spout and bolt only for message layer? This is most important part on current topic since you're focusing latency from only message layer. Did you modify storm core logic and rebuild to include timestamp in transport layer? Or measured via after emit() in Spout to starting of execute() in Bolt? If it's latter, it is including serialization/deserialization cost which we consider as expensive part. 2018년 2월 1일 (목) 오후 12:55, Wuyang Zhang <[email protected]>님이 작성: > Also, I would like to ask that when the target application demands ultra > low latency and removes any possible messaging delay, do you think setting > the batch size = 1 can help in this case? According to the source code, it > will flush the queue when its size comes to a predefined batch size or goes > after an interval. So to reduce the queue delay, I try to set the batch > size to 1 and the flush interval to 1ms. Meanwhile, I modify the waiting > strategy to yield. > > Any other suggestions to reduce messaging delay introduced inside Storm? > ᐧ > > On Wed, Jan 31, 2018 at 10:20 PM, Wuyang Zhang <[email protected]> > wrote: > >> Hi Jungtaek and Roshan, >> >> Thank you for your replies and the suggestions. >> >> I just did a more detailed evaluation for this messaging delay issue. >> >> 1. I use ping and iPerf and get the numbers of 0.193ms and 934Mbps. >> 2. I use scp to transmit the image m0.png(1.7MB) with the delay of >> 0.072s. >> 3. I convert two m0.png to a single byte array with the length of >> 9331200(8.89MB). >> 4. I write a udp socket with the buffer size of 1024 to transmit the >> 8.89MB buffer in 80ms. >> 5. I send the 8.89MB buffer from the spout to a bolt in 93ms. >> >> The storm message will introduce 10ms more delay. >> >> I may get previous numbers because the heavy computing overhead and the >> tuple is buffered in the queue. >> >> Thanks a lot and motivate me to find this. I will try to consider >> encode/decode images to reduce the transmission latency. >> >> Best Regards, >> Wuyang >> ᐧ >> >> On Wed, Jan 31, 2018 at 7:18 PM, Roshan Naik <[email protected]> >> wrote: >> >>> Continuing with Jungtaek’s line of thinking… would also like to know >>> >>> - What latencies are you able to achieve when directly >>> transmitting a 5mb image between two nodes (not using Storm) ? And >>> similarly,… within two processes on the same node. ? >>> >>> - And how are you measuring it ? >>> >>> -roshan >>> >>> >>> >>> *From: *Jungtaek Lim <[email protected]> >>> *Reply-To: *"[email protected]" <[email protected]> >>> *Date: *Wednesday, January 31, 2018 at 3:38 PM >>> *To: *"[email protected]" <[email protected]> >>> *Subject: *Re: Apache Storm High Messaging Delay When Passing 5MB Images >>> >>> >>> >>> I meant "easily seen" as "not exposed" and we are easy to miss to >>> consider. >>> >>> >>> >>> 2018년 2월 1일 (목) 오전 8:36, Jungtaek Lim <[email protected]>님이 작성: >>> >>> I'm not clear whether you're saying message transfer for each bolt took >>> 200ms, or summation of 4 or 5 times network transfer latencies were 200 ms. >>> >>> >>> >>> Why I say I'm not clear is that if it's latter and there's 5 times >>> network transfers, it is ideal latency in theory, since 1Gbps is 125MBps >>> (1000Mbps, not 1024Mbps) and 5M/125MBps = 40ms per each transfer. (I'd >>> rather suspect how it was possible in this case.) >>> >>> >>> >>> Even there's 4 times network transfers, we may need to take this to >>> account: the latency calculation above is in theory, and there're many >>> overheads other than messaging which is not easily seen, so the latency may >>> not be from messaging overhead. >>> >>> >>> >>> If it took 200 ms for each transfer that can be a thing to talk about. >>> Please let me know which is your case. >>> >>> >>> >>> Thanks, >>> >>> Jungtaek Lim (HeartSaVioR) >>> >>> >>> >>> 2018년 2월 1일 (목) 오전 8:11, Wuyang Zhang <[email protected]>님이 작성: >>> >>> I am playing with Apache Storm for a real-time image processing >>> application which requires ultra low latency. In the topology definition, a >>> single spout will emit raw images(5MB) in every 1s and a few bolts will >>> process them. The processing latency of each bolt is acceptable and the >>> overall computing delay can be around 150ms. >>> >>> *However, I find that the message passing delay between workers on the >>> different nodes is really high. The overall such delay on the 5 successive >>> bolts is around 200ms.* To calculate this delay, I subtract all the >>> task latencies from the end-to-end latency. Moreover, I implement a timer >>> bolt and other processing bolts will register in this timer bolt to record >>> the timestamp before starting the real processing. By comparing the >>> timestamps of the bolts, I find the delay between each bolt is high as I >>> previously noticed. >>> >>> To analyze the source of this high additional delay, I firstly reduce >>> the sending interval to 1s and thus there should be no queuing delay due to >>> the high computing overheads. Also, from the Storm UI, I find none bolt is >>> in high CPU utilization. >>> >>> Then, I checked the network delay. I am using a 1Gbps network testbed >>> and test the network by RTT and bandwidth. The network latency should not >>> be that high to send a 5MB image. >>> >>> Finally, I am thinking about the buffer delay. I find each thread >>> maintains its own sending buffer and transfer the data to the worker's >>> sending buffer. I am not sure how long it takes before the receiver bolt >>> can get this sending message. As suggested by the community, I increase the >>> sender/receiver buffer size to 16384, modify STORM_NETTY_MESSAGE_BATCH_SIZE >>> to 32768. However, it did not help. >>> >>> *My question is that how to remove/reduce the messaging overheads >>> between bolts?(inter workers)* It is possible to synchronize the >>> communication between bolts and have the receiver got the sending messages >>> immediately without any delay? >>> >>> >>> >>> ᐧ >>> >>> >> >
