[
https://issues.apache.org/jira/browse/NIFI-5466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16587613#comment-16587613
]
ASF subversion and git services commented on NIFI-5466:
-------------------------------------------------------
Commit 7bbb5a823aa3d338cecf45bd683f9f34a1338d02 in nifi's branch
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=7bbb5a8 ]
NIFI-5466: Keep a running total of stats for each component. Refactored
FlowFileEvent and repository in order to provide more efficient storage of
objects on Java heap by allowing the same 'EMPTY' object to be reused
- Refactored VolatileComponentStatusRepository to avoid holding on to
ProcessorStatus objects, etc, and only keep what they need
- Updated VolatileComponentStatusRepository to ensure that we are efficiently
storing metrics for processors, etc. that are not running
This closes #2939
Signed-off-by: Mark Payne <[email protected]>
> Improve efficiency of RingBufferEventRepository
> -----------------------------------------------
>
> Key: NIFI-5466
> URL: https://issues.apache.org/jira/browse/NIFI-5466
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Reporter: Mark Payne
> Assignee: Mark Payne
> Priority: Major
> Fix For: 1.8.0
>
>
> The implementation of the FlowFileEventRepository (i.e., the
> RingBufferEventRepository) is implemented in such a way that 300 buckets are
> kept for each component. Each bucket holds a set of 16+ stats (bytes read,
> bytes written, flowfiles in, flowfiles out, etc. as well as counters)
> Each time that a user refreshes stats, we calculate the 5-minute window for
> each component by iterating over all 300 buckets, then summing together the
> values of the stats. So, for every 1,000 processors, we have 1000 * 300 * 16
> = 4.8 MMÂ calculations being performed each time that the user refreshes.
> We can improve this significantly. To do this, we can keep a 'running tally'
> for each stat. Every time that we update a bucket in the
> FlowFileEventRepository, we should similarly update the running tally. Each
> time that we 'replace' a bucket in the FlowFileEventRepository, we should
> subtract from the running tally the values in the bucket, in addition to
> adding in the new value. This would result in amortizing the cost of
> calculation over the 5-minute period and means that every time that we get
> the processor stats we could do so by simply performing a 'get' operation
> without any calculations.
> But wait, there's more! The EventSumValue class has an {{add(FlowFileEvent)}}
> method. If the EventSumValue were to simply keep a boolean value indicating
> whether or not it was empty, then when adding to an empty bucket we could
> avoid the calculations all together, and when replacing an empty bucket we
> could avoid the calculation against the running tally as well.
> Similarly, in the VolatileComponentStatusRepository, when we perform a
> 'capture' we should detect any 'empty' stats objects and use a single static
> instance for this, which would significantly reduce the amount of heap that
> is used to store metrics.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
