It’s kind of implied from prior responses but just wanted to emphasize .. it is not used if ACKing is disabled (I.e acker count = 0)
Sent from Yahoo Mail for iPhone On Wednesday, April 3, 2019, 11:50 AM, Stig Rohde Døssing <[email protected]> wrote: Supplementing the other great answers here, Storm checks max spout pending before it invokes nextTuple on the spout. The spout executor calls nextTuple in a loop. If the spout task has more pending tuples (i.e. emitted but not acked or failed or timed out) than max spout pending, the executor won't call nextTuple, but instead wait for a bit before it checks again. The relevant code is here https://github.com/apache/storm/blob/9680928a0c6e26a8a2d3e5ceb9a632579546353d/storm-client/src/jvm/org/apache/storm/executor/spout/SpoutExecutor.java#L190 Den ons. 3. apr. 2019 kl. 15.00 skrev Alessio Pagliari <[email protected]>: Hi Jayant, The MAX_SPOUT_PENDING is applied in each of the spouts. Each spout take count of all the tuples it sends, increasing it for each emit and decreasing it when it receive the final ack message for a “pending tuple” (= a tuple that is sent but it’s still waiting an ack). If the number of pending tuples raises above the value you set, let’s say 500, the spouts will automatically slow down, until they reach an emission rate such that the number of pending tuples stabilise under 500.This mechanism is applied per spout, meaning that each spout can be limited in different times, and the MAX_SPOUT_PENDING represents the maximum number of non-acked tuples that a single spout will allow. Alessio On 3 Apr 2019, at 14:45, Joshua Martell <[email protected]> wrote: I’m pretty sure it’s per spout task. And they’re not coordinated. Each task gets its own count. Joshua On Wed, Apr 3, 2019 at 4:50 AM Jayant Sharma <[email protected]> wrote: Hi, Can someone please explain at what point storm checks and applies MAX_SPOUT_PENDING limit. Also, is this limit applied per executor(or Task if that's the case) of spout or aggregated and applied over all the executors of spout.Suppose I have 3 executors and 3 tasks of spouts, each of them fetch 1 message from input source. If my MAX_SPOUT_PENDING is 2, will only 2 executors be able to send the tuple forward or all 3 will send and further nextTuple calls on all will be blocked? Thanks,Jayant Sharma
