Thanks again for the feedback. When running `htop` I see that the CPU utilisation for each core is <25%, this is true for all components. I have a c4.xlarge instance that that 4 vCPUs (2 hyper thread intel cores). We are getting a json formatted item from our Kestrel Message queue server. Is there a way I can check the tuple size? When I save the JSON file on my mac, the file size is about 450 bytes.
Additional information about the spout: We grab a batch of kestrel items (4000) and then emit them one by one in `nextTuple`. This reduces the need to consistently communicate with kestrel. We also batch the confirmations back to kestrel (which is done when we have ack’d half the original batch(2000)). Adding this batching has increased performance. //Matt > On 23 Mar 2016, at 10:12, Brian Candler <[email protected]> wrote: > > On 23/03/2016 08:45, Matthew Lowe wrote: >> The cpu is not under a lot of strain for the spout > On each VM, have you tried running "top" and then hitting "1"? This will show > you the CPU utilisation per core. If on any VM you see one core at 100% then > you need to increase parallelism by more threads (executors). > >> , so my only conclusion can be the networking performance? > Are your tuples large? A 1G NIC has a usable capacity of about 112MiB/sec. So > a limit of 6000/sec would imply 19KB/tuple. > >> Someone mentioned that is could be JVM related, to do with the GC or >> available memory? Im not sure about this though. >> These values are very similar to the ones I get on a c4.large, which maxes >> out at 3.5-4k per second. >> >> Interestingly I found this when the topology starts: >> [INFO] Spout tuples per second: 1 >> [INFO] Spout tuples per second: 10000 >> [INFO] Spout tuples per second: 936 >> [INFO] Spout tuples per second: 1851 >> [INFO] Spout tuples per second: 4480 >> [INFO] Spout tuples per second: 5174 >> [INFO] Spout tuples per second: 6399 >> It then holds at about 6500 from here on. > I expect the spout will send as fast as it can until it reaches > maxTuplePending, then stop and wait for tuples to be acked, and eventually > will stabilise at the achievable throughput. > > Regards, > > Brian. >
