Supplementing the other great answers here, Storm checks max spout pending before it invokes nextTuple on the spout. The spout executor calls nextTuple in a loop. If the spout task has more pending tuples (i.e. emitted but not acked or failed or timed out) than max spout pending, the executor won't call nextTuple, but instead wait for a bit before it checks again. The relevant code is here https://github.com/apache/storm/blob/9680928a0c6e26a8a2d3e5ceb9a632579546353d/storm-client/src/jvm/org/apache/storm/executor/spout/SpoutExecutor.java#L190
Den ons. 3. apr. 2019 kl. 15.00 skrev Alessio Pagliari < pagli...@i3s.unice.fr>: > Hi Jayant, > > The MAX_SPOUT_PENDING is applied in each of the spouts. > > Each spout take count of all the tuples it sends, increasing it for each > emit and decreasing it when it receive the final ack message for a “pending > tuple” (= a tuple that is sent but it’s still waiting an ack). > If the number of pending tuples raises above the value you set, let’s say > 500, the spouts will automatically slow down, until they reach an emission > rate such that the number of pending tuples stabilise under 500. > This mechanism is applied per spout, meaning that each spout can be > limited in different times, and the MAX_SPOUT_PENDING represents the > maximum number of non-acked tuples that a single spout will allow. > > Alessio > > On 3 Apr 2019, at 14:45, Joshua Martell <joshua.mart...@gmail.com> wrote: > > I’m pretty sure it’s per spout task. And they’re not coordinated. Each > task gets its own count. > > Joshua > On Wed, Apr 3, 2019 at 4:50 AM Jayant Sharma <sharmajayan...@gmail.com> > wrote: > >> Hi, >> >> Can someone please explain at what point storm checks and applies >> MAX_SPOUT_PENDING limit. Also, is this limit applied per executor(or Task >> if that's the case) of spout or aggregated and applied over all the >> executors of spout. >> Suppose I have 3 executors and 3 tasks of spouts, each of them fetch 1 >> message from input source. If my MAX_SPOUT_PENDING is 2, will only 2 >> executors be able to send the tuple forward or all 3 will send and further >> nextTuple calls on all will be blocked? >> >> Thanks, >> Jayant Sharma >> > >
