Matt Burgess created NIFI-12389:
-----------------------------------
Summary: Add variance and standard deviation to
AttributeRollingWindow
Key: NIFI-12389
URL: https://issues.apache.org/jira/browse/NIFI-12389
Project: Apache NiFi
Issue Type: Improvement
Components: Extensions
Reporter: Matt Burgess
AttributeRollingWindow currently uses state to calculate metrics like sum,
count, and mean (average) for a rolling time window (optionally using
sub-windows to use smaller aggregates to estimate metrics for a larger total
time window as it stores the window values in state).
Variance and its positive square root (standard deviation) are other helpful
univariate statistics. Using "online" algorithms such as Welford's method adds
no extra memory requirement and runs in constant time. Apache Commons Math3 has
such an implementation and could be used to add attributes to the FlowFile for
variance and standard deviation. These metrics can be used for filtering out
noise downstream, for example RouteOnAttribute and/or stateful UpdateAttribute
where the current attribute value is more than 2 standard deviations away from
the mean.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
