Mark Payne created NIFI-5466:
--------------------------------
Summary: Improve efficiency of RingBufferEventRepository
Key: NIFI-5466
URL: https://issues.apache.org/jira/browse/NIFI-5466
Project: Apache NiFi
Issue Type: Improvement
Components: Core Framework
Reporter: Mark Payne
Assignee: Mark Payne
The implementation of the FlowFileEventRepository (i.e., the
RingBufferEventRepository) is implemented in such a way that 300 buckets are
kept for each component. Each bucket holds a set of 16+ stats (bytes read,
bytes written, flowfiles in, flowfiles out, etc. as well as counters)
Each time that a user refreshes stats, we calculate the 5-minute window for
each component by iterating over all 300 buckets, then summing together the
values of the stats. So, for every 1,000 processors, we have 1000 * 300 * 16 =
4.8 MMÂ calculations being performed each time that the user refreshes.
We can improve this significantly. To do this, we can keep a 'running tally'
for each stat. Every time that we update a bucket in the
FlowFileEventRepository, we should similarly update the running tally. Each
time that we 'replace' a bucket in the FlowFileEventRepository, we should
subtract from the running tally the values in the bucket, in addition to adding
in the new value. This would result in amortizing the cost of calculation over
the 5-minute period and means that every time that we get the processor stats
we could do so by simply performing a 'get' operation without any calculations.
But wait, there's more! The EventSumValue class has an {{add(FlowFileEvent)}}
method. If the EventSumValue were to simply keep a boolean value indicating
whether or not it was empty, then when adding to an empty bucket we could avoid
the calculations all together, and when replacing an empty bucket we could
avoid the calculation against the running tally as well.
Similarly, in the VolatileComponentStatusRepository, when we perform a
'capture' we should detect any 'empty' stats objects and use a single static
instance for this, which would significantly reduce the amount of heap that is
used to store metrics.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
