[
https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yolanda M. Davis updated NIFI-6510:
-----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
This feature was merged 9/9/2019
> Predictive Analytics for NiFi Metrics
> -------------------------------------
>
> Key: NIFI-6510
> URL: https://issues.apache.org/jira/browse/NIFI-6510
> Project: Apache NiFi
> Issue Type: New Feature
> Reporter: Andrew Christianson
> Assignee: Yolanda M. Davis
> Priority: Major
> Fix For: 1.10.0
>
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> From Yolanda's email to the list:
>
> {noformat}
> Currently NiFi has lots of metrics available for areas including jvm and flow
> component usage (via component status) as well as provenance data which NiFi
> makes available either through the UI or reporting tasks (for consumption by
> other systems). Past discussions in the community cite users shipping this
> data to applications such as Prometheus, ELK stacks, or Ambari metrics for
> further analysis in order to capture/review performance issues, detect
> anomalies, and send alerts or notifications. These systems are efficient in
> capturing and helping to analyze these metrics however it requires
> customization work and knowledge of NiFi operations to provide meaningful
> analytics within a flow context.
> In speaking with Matt Burgess and Andy Christianson on this topic we feel
> that there is an opportunity to introduce an analytics framework that could
> provide users reasonable predictions on key performance indicators for flows,
> such as back pressure and flow rate, to help administrators improve
> operational management of NiFi clusters. This framework could offer several
> key features:
> - Provide a flexible internal analytics engine and model api which supports
> the addition of or enhancement to onboard models
> - Support integration of remote or cloud based ML models
> - Support both traditional and online (incremental) learning methods
> - Provide support for model caching (perhaps later inclusion into a model
> repository or registry)
> - UI enhancements to display prediction information either in existing
> summary data, new data visualizations, or directly within the flow/canvas
> (where applicable)
> For an initial target we thought that back pressure prediction would be a
> good starting point for this initiative, given that back pressure detection
> is a key indicator of flow performance and many of the metrics currently
> available would provide enough data points to create a reasonable performing
> model. We have some ideas on how this could be achieved however we wanted to
> discuss this more with the community to get thoughts about tackling this
> work, especially if there are specific use cases or other factors that should
> be considered.{noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
