Dear Bobby,
Thank you very much for detailed explanation.
Best Regards,Algoby
From: Bobby Evans <[email protected]>
To: [email protected]
Cc: [email protected]
Sent: Friday, December 22, 2017 12:00 AM
Subject: Re: Back-pressure Mechamism
Algoby,
When you say transfer queue, which queue do you mean exactly. In storm there
are a lot of queues currently and they sometimes have confusing names.
There is the receive queue, which holds tuples to be processed by a specific
executor. Then there is the send queue, or some times called the batch
transfer queue. All of the emit calls from an executor go into this and then a
second thread handles batching the massages and routing them to where they need
to go. The there is the transfer queue. This queue gets all of the tuples
that need to be sent outside this worker.
We have looked at supporting all of these different queues for back pressure.
The receive queue is the big one as it is where most of the user code likely
executes. The send queue tends to back up if the time taken to serialize an
object is more then the processing needed to produce the object. This is not
that common, but I have seen it where a single large message gets split up into
many messages, each that may be kind of difficult to serialize. I thought we
had a patch to include the send queue as part of back pressure, but I don't
know what happened to it.
The transfer queue is much less likely to back up, but the consequences are
much worse when it does backup. The thread that reads from the transfer queue
only routes messages to clients that are buffering the messages and sending
them to other workers. There is not much work happening here. The clients
themselves don't have any back pressure built in either. So if the transfer
queue is backing up then your worker likely is writing messages into memory as
fast as it can, and you are going to get an OOM some time soon. To really make
this work you would need some kind of back pressure in the netty clients that
could also be involved with this.
A patch that we will likely merge into 2.x shortly
https://github.com/apache/storm/pull/2241 has done all of this and also
redesigned back pressure to not go off of high/low water marks with signals
through zookeeper, but instead to push back to the upstream component when a
queue is full. The only downside there is that we will only be able to support
DAGs for processing. No loops in user code, or you could deadlock. Until we
get 2.x out the door and stable (which I really want to do in Q1 2018) you are
probably going to have to live with some of these issues.
- Bobby
On Thu, Dec 21, 2017 at 9:50 AM Waleed Al-Gobi <[email protected]> wrote:
Dear All,
My concern is about on which queue Storm relies to for back-pressure.
I did simple test for back-pressure supported by Storm.
Each instance (executor) maintains incoming(receive) Q and
outgoing(transfer) Q, and according to min and max threshold on these
queues, a back-pressure works to slow down the spout in case of queue
buildup.
The purpose I wanted to make sure in case of link bottleneck whether
back-pressure still helps or not.
The conclusion, it helps only in case of queue buildup due to CPU
bottleneck. I guess the reason for which why it could not make it for link
bottleneck, because back-pessure relies only on the executor receive Q.
Does this make sense? If so, could we anyway make the back-pressure also
working if executor transfer Q is full in case of link bottleneck?
Thanks!
Best,
Algoby
