On 23/03/2016 08:45, Matthew Lowe wrote:
The cpu is not under a lot of strain for the spout
On each VM, have you tried running "top" and then hitting "1"? This will
show you the CPU utilisation per core. If on any VM you see one core at
100% then you need to increase parallelism by more threads (executors).
, so my only conclusion can be the networking performance?
Are your tuples large? A 1G NIC has a usable capacity of about
112MiB/sec. So a limit of 6000/sec would imply 19KB/tuple.
Someone mentioned that is could be JVM related, to do with the GC or available
memory? Im not sure about this though.
These values are very similar to the ones I get on a c4.large, which maxes out
at 3.5-4k per second.
Interestingly I found this when the topology starts:
[INFO] Spout tuples per second: 1
[INFO] Spout tuples per second: 10000
[INFO] Spout tuples per second: 936
[INFO] Spout tuples per second: 1851
[INFO] Spout tuples per second: 4480
[INFO] Spout tuples per second: 5174
[INFO] Spout tuples per second: 6399
It then holds at about 6500 from here on.
I expect the spout will send as fast as it can until it reaches
maxTuplePending, then stop and wait for tuples to be acked, and
eventually will stabilise at the achievable throughput.
Regards,
Brian.
