Thanks for getting back to me so fast.
I have just tried the following tests:
maxPendingSpout tuples per second ackers
35 5k
3
3500 5-6k
3
10000 5.5-6k 3
10000 5.5k
10
They all seem to emit at the same speed. As if I have hit a wall in performance.
The cpu is not under a lot of strain for the spout, so my only conclusion can
be the networking performance?
Someone mentioned that is could be JVM related, to do with the GC or available
memory? Im not sure about this though.
These values are very similar to the ones I get on a c4.large, which maxes out
at 3.5-4k per second.
Interestingly I found this when the topology starts:
[INFO] Spout tuples per second: 1
[INFO] Spout tuples per second: 10000
[INFO] Spout tuples per second: 936
[INFO] Spout tuples per second: 1851
[INFO] Spout tuples per second: 4480
[INFO] Spout tuples per second: 5174
[INFO] Spout tuples per second: 6399
It then holds at about 6500 from here on.
Seems like it emits the 10000 I specify in the maxPending, then it drops.
Part of me is wondering if the bolts are to blame, but the UI and my timers say
they take less that a ms to perform.
That can leave only the network speed, right?
Thanks for any help!
> On 22 Mar 2016, at 22:01, Brian Candler <[email protected]> wrote:
>
> On 22/03/2016 20:13, Matthew Lowe wrote:
>> The maxTuplePending is set to 35 (During testing we found 3 ackers and 35
>> max pending was “good")
> Have you tried increasing this now you have built your spout-bolt-bolt
> topology?
>> When testing with only the Spout (no bolts) I was emitting about 70,000 per
>> second.
>> When adding the first bolt this it then dropped to about 10,000 - 14,000 per
>> second
>> Finally, adding the final bolt dropped me down to about 5,000 per second.
>>
>> When looking at Storm ui the spout takes about 11ms (complete latency, when
>> running full topology)
>> And each spout takes about 0.2ms to execute.
>> When running only the Spout the complete latency is a lot less, maybe 2-3ms.
> Once maxTuplePending is reached the spout stops sending, and won't send
> another message until a previous message has been fully processed, all the
> way to the end.
>
> You have set a limit of 35 messages "in flight" and each message takes 11ms
> end to end. This gives you an expected throughput of 35 x (1000/11) ~= 3,200
> messages per second.
>
> When running only the spout with 2ms latency you would expect 35 x (1000/2) =
> 17,500 messages per second.
>
> Those numbers don't tie exactly with what you see, but gives a rough idea of
> how maxTuplePending might be limiting your throughput. At least, that's my
> guess as to what's happening :-)
>
> Regards,
>
> Brian.
>
