Algoby, When you say transfer queue, which queue do you mean exactly. In storm there are a lot of queues currently and they sometimes have confusing names.
There is the receive queue, which holds tuples to be processed by a specific executor. Then there is the send queue, or some times called the batch transfer queue. All of the emit calls from an executor go into this and then a second thread handles batching the massages and routing them to where they need to go. The there is the transfer queue. This queue gets all of the tuples that need to be sent outside this worker. We have looked at supporting all of these different queues for back pressure. The receive queue is the big one as it is where most of the user code likely executes. The send queue tends to back up if the time taken to serialize an object is more then the processing needed to produce the object. This is not that common, but I have seen it where a single large message gets split up into many messages, each that may be kind of difficult to serialize. I thought we had a patch to include the send queue as part of back pressure, but I don't know what happened to it. The transfer queue is much less likely to back up, but the consequences are much worse when it does backup. The thread that reads from the transfer queue only routes messages to clients that are buffering the messages and sending them to other workers. There is not much work happening here. The clients themselves don't have any back pressure built in either. So if the transfer queue is backing up then your worker likely is writing messages into memory as fast as it can, and you are going to get an OOM some time soon. To really make this work you would need some kind of back pressure in the netty clients that could also be involved with this. A patch that we will likely merge into 2.x shortly https://github.com/apache/storm/pull/2241 has done all of this and also redesigned back pressure to not go off of high/low water marks with signals through zookeeper, but instead to push back to the upstream component when a queue is full. The only downside there is that we will only be able to support DAGs for processing. No loops in user code, or you could deadlock. Until we get 2.x out the door and stable (which I really want to do in Q1 2018) you are probably going to have to live with some of these issues. - Bobby On Thu, Dec 21, 2017 at 9:50 AM Waleed Al-Gobi <[email protected]> wrote: > Dear All, > > My concern is about on which queue Storm relies to for back-pressure. > > I did simple test for back-pressure supported by Storm. > Each instance (executor) maintains incoming(receive) Q and > outgoing(transfer) Q, and according to min and max threshold on these > queues, a back-pressure works to slow down the spout in case of queue > buildup. > > The purpose I wanted to make sure in case of link bottleneck whether > back-pressure still helps or not. > The conclusion, it helps only in case of queue buildup due to CPU > bottleneck. I guess the reason for which why it could not make it for link > bottleneck, because back-pessure relies only on the executor receive Q. > > Does this make sense? If so, could we anyway make the back-pressure also > working if ececutor transfer Q is full in case of link bottleneck? > > Thanks! > > Best, > Algoby >
