Hi Matt Some ideas (not sure it will help) - could you check the capacity kpi on the ui. Afair, this is an average on the last 10 min? - are you sure your data is evenly distributed across the bolts (when doing the field grouping) ?
Oliv On Wednesday, 23 March 2016, Matthew Lowe <[email protected]> wrote: > Thanks again for the feedback. > > When running `htop` I see that the CPU utilisation for each core is <25%, > this is true for all components. > I have a c4.xlarge instance that that 4 vCPUs (2 hyper thread intel cores). > We are getting a json formatted item from our Kestrel Message queue > server. Is there a way I can check the tuple size? > When I save the JSON file on my mac, the file size is about 450 bytes. > > Additional information about the spout: > We grab a batch of kestrel items (4000) and then emit them one by one in > `nextTuple`. This reduces the need to consistently communicate with kestrel. > We also batch the confirmations back to kestrel (which is done when we > have ack’d half the original batch(2000)). > > Adding this batching has increased performance. > > //Matt > > > On 23 Mar 2016, at 10:12, Brian Candler <[email protected] > <javascript:;>> wrote: > > > > On 23/03/2016 08:45, Matthew Lowe wrote: > >> The cpu is not under a lot of strain for the spout > > On each VM, have you tried running "top" and then hitting "1"? This will > show you the CPU utilisation per core. If on any VM you see one core at > 100% then you need to increase parallelism by more threads (executors). > > > >> , so my only conclusion can be the networking performance? > > Are your tuples large? A 1G NIC has a usable capacity of about > 112MiB/sec. So a limit of 6000/sec would imply 19KB/tuple. > > > >> Someone mentioned that is could be JVM related, to do with the GC or > available memory? Im not sure about this though. > >> These values are very similar to the ones I get on a c4.large, which > maxes out at 3.5-4k per second. > >> > >> Interestingly I found this when the topology starts: > >> [INFO] Spout tuples per second: 1 > >> [INFO] Spout tuples per second: 10000 > >> [INFO] Spout tuples per second: 936 > >> [INFO] Spout tuples per second: 1851 > >> [INFO] Spout tuples per second: 4480 > >> [INFO] Spout tuples per second: 5174 > >> [INFO] Spout tuples per second: 6399 > >> It then holds at about 6500 from here on. > > I expect the spout will send as fast as it can until it reaches > maxTuplePending, then stop and wait for tuples to be acked, and eventually > will stabilise at the achievable throughput. > > > > Regards, > > > > Brian. > > > >
