Re: When is the MAX_SPOUT_PENDING limit applied

[Roshan Naik]

It’s kind of implied from prior responses but just wanted to emphasize  .. it 
is not used if ACKing is disabled (I.e acker count = 0)


Sent from Yahoo Mail for iPhone


On Wednesday, April 3, 2019, 11:50 AM, Stig Rohde Døssing 
 wrote:

Supplementing the other great answers here, Storm checks max spout pending 
before it invokes nextTuple on the spout. The spout executor calls nextTuple in 
a loop. If the spout task has more pending tuples (i.e. emitted but not acked 
or failed or timed out) than max spout pending, the executor won't call 
nextTuple, but instead wait for a bit before it checks again. The relevant code 
is here 
https://github.com/apache/storm/blob/9680928a0c6e26a8a2d3e5ceb9a632579546353d/storm-client/src/jvm/org/apache/storm/executor/spout/SpoutExecutor.java#L190

Den ons. 3. apr. 2019 kl. 15.00 skrev Alessio Pagliari :

Hi Jayant,
The MAX_SPOUT_PENDING is applied in each of the spouts. 
Each spout take count of all the tuples it sends, increasing it for each emit 
and decreasing it when it receive the final ack message for a “pending tuple” 
(= a tuple that is sent but it’s still waiting an ack). If the number of 
pending tuples raises above the value you set, let’s say 500, the spouts will 
automatically slow down, until they reach an emission rate such that the number 
of pending tuples stabilise under 500.This mechanism is applied per spout, 
meaning that each spout can be limited in different times, and the 
MAX_SPOUT_PENDING represents the maximum number of non-acked tuples that a 
single spout will allow.
Alessio

On 3 Apr 2019, at 14:45, Joshua Martell  wrote:
I’m pretty sure it’s per spout task. And they’re not coordinated. Each task 
gets its own count. 

Joshua
On Wed, Apr 3, 2019 at 4:50 AM Jayant Sharma  wrote:

Hi,

Can someone please explain at what point storm checks and applies 
MAX_SPOUT_PENDING limit. Also, is this limit applied per executor(or Task if 
that's the case) of spout or aggregated and applied over all the executors of 
spout.Suppose I have 3 executors and 3 tasks of spouts, each of them fetch 1 
message from input source. If my MAX_SPOUT_PENDING is 2, will only 2 executors 
be able to send the tuple forward or all 3 will send and further nextTuple 
calls on all will be blocked?
Thanks,Jayant Sharma

Re: When is the MAX_SPOUT_PENDING limit applied

[Stig Rohde Døssing]

Supplementing the other great answers here, Storm checks max spout pending
before it invokes nextTuple on the spout. The spout executor calls
nextTuple in a loop. If the spout task has more pending tuples (i.e.
emitted but not acked or failed or timed out) than max spout pending, the
executor won't call nextTuple, but instead wait for a bit before it checks
again. The relevant code is here
https://github.com/apache/storm/blob/9680928a0c6e26a8a2d3e5ceb9a632579546353d/storm-client/src/jvm/org/apache/storm/executor/spout/SpoutExecutor.java#L190

Den ons. 3. apr. 2019 kl. 15.00 skrev Alessio Pagliari <
pagli...@i3s.unice.fr>:

> Hi Jayant,
>
> The MAX_SPOUT_PENDING is applied in each of the spouts.
>
> Each spout take count of all the tuples it sends, increasing it for each
> emit and decreasing it when it receive the final ack message for a “pending
> tuple” (= a tuple that is sent but it’s still waiting an ack).
> If the number of pending tuples raises above the value you set, let’s say
> 500, the spouts will automatically slow down, until they reach an emission
> rate such that the number of pending tuples stabilise under 500.
> This mechanism is applied per spout, meaning that each spout can be
> limited in different times, and the MAX_SPOUT_PENDING represents the
> maximum number of non-acked tuples that a single spout will allow.
>
> Alessio
>
> On 3 Apr 2019, at 14:45, Joshua Martell  wrote:
>
> I’m pretty sure it’s per spout task. And they’re not coordinated. Each
> task gets its own count.
>
> Joshua
> On Wed, Apr 3, 2019 at 4:50 AM Jayant Sharma 
> wrote:
>
>> Hi,
>>
>> Can someone please explain at what point storm checks and applies
>> MAX_SPOUT_PENDING limit. Also, is this limit applied per executor(or Task
>> if that's the case) of spout or aggregated and applied over all the
>> executors of spout.
>> Suppose I have 3 executors and 3 tasks of spouts, each of them fetch 1
>> message from input source. If my MAX_SPOUT_PENDING is 2, will only 2
>> executors be able to send the tuple forward or all 3 will send and further
>> nextTuple calls on all will be blocked?
>>
>> Thanks,
>> Jayant Sharma
>>
>
>

Re: When is the MAX_SPOUT_PENDING limit applied

[Alessio Pagliari]

Hi Jayant,

The MAX_SPOUT_PENDING is applied in each of the spouts. 

Each spout take count of all the tuples it sends, increasing it for each emit 
and decreasing it when it receive the final ack message for a “pending tuple” 
(= a tuple that is sent but it’s still waiting an ack). 
If the number of pending tuples raises above the value you set, let’s say 500, 
the spouts will automatically slow down, until they reach an emission rate such 
that the number of pending tuples stabilise under 500.
This mechanism is applied per spout, meaning that each spout can be limited in 
different times, and the MAX_SPOUT_PENDING represents the maximum number of 
non-acked tuples that a single spout will allow.

Alessio

> On 3 Apr 2019, at 14:45, Joshua Martell  wrote:
> 
> I’m pretty sure it’s per spout task. And they’re not coordinated. Each task 
> gets its own count. 
> 
> Joshua
> On Wed, Apr 3, 2019 at 4:50 AM Jayant Sharma  > wrote:
> Hi,
> 
> Can someone please explain at what point storm checks and applies 
> MAX_SPOUT_PENDING limit. Also, is this limit applied per executor(or Task if 
> that's the case) of spout or aggregated and applied over all the executors of 
> spout.
> Suppose I have 3 executors and 3 tasks of spouts, each of them fetch 1 
> message from input source. If my MAX_SPOUT_PENDING is 2, will only 2 
> executors be able to send the tuple forward or all 3 will send and further 
> nextTuple calls on all will be blocked?
> 
> Thanks,
> Jayant Sharma

Re: When is the MAX_SPOUT_PENDING limit applied

[Joshua Martell]

I’m pretty sure it’s per spout task. And they’re not coordinated. Each task
gets its own count.

Joshua
On Wed, Apr 3, 2019 at 4:50 AM Jayant Sharma 
wrote:

> Hi,
>
> Can someone please explain at what point storm checks and applies
> MAX_SPOUT_PENDING limit. Also, is this limit applied per executor(or Task
> if that's the case) of spout or aggregated and applied over all the
> executors of spout.
> Suppose I have 3 executors and 3 tasks of spouts, each of them fetch 1
> message from input source. If my MAX_SPOUT_PENDING is 2, will only 2
> executors be able to send the tuple forward or all 3 will send and further
> nextTuple calls on all will be blocked?
>
> Thanks,
> Jayant Sharma
>