Andrew Christianson created NIFI-6510:
-----------------------------------------
Summary: Predictive Analytics for NiFi Metrics
Key: NIFI-6510
URL: https://issues.apache.org/jira/browse/NIFI-6510
Project: Apache NiFi
Issue Type: Improvement
Reporter: Andrew Christianson
>From Yolanda's email to the list:
{noformat}
Currently NiFi has lots of metrics available for areas including jvm and flow
component usage (via component status) as well as provenance data which NiFi
makes available either through the UI or reporting tasks (for consumption by
other systems). Past discussions in the community cite users shipping this data
to applications such as Prometheus, ELK stacks, or Ambari metrics for further
analysis in order to capture/review performance issues, detect anomalies, and
send alerts or notifications. These systems are efficient in capturing and
helping to analyze these metrics however it requires customization work and
knowledge of NiFi operations to provide meaningful analytics within a flow
context.
In speaking with Matt Burgess and Andy Christianson on this topic we feel that
there is an opportunity to introduce an analytics framework that could provide
users reasonable predictions on key performance indicators for flows, such as
back pressure and flow rate, to help administrators improve operational
management of NiFi clusters. This framework could offer several key features:
- Provide a flexible internal analytics engine and model api which supports the
addition of or enhancement to onboard models
- Support integration of remote or cloud based ML models
- Support both traditional and online (incremental) learning methods
- Provide support for model caching (perhaps later inclusion into a model
repository or registry)
- UI enhancements to display prediction information either in existing summary
data, new data visualizations, or directly within the flow/canvas (where
applicable)
For an initial target we thought that back pressure prediction would be a good
starting point for this initiative, given that back pressure detection is a key
indicator of flow performance and many of the metrics currently available would
provide enough data points to create a reasonable performing model. We have
some ideas on how this could be achieved however we wanted to discuss this more
with the community to get thoughts about tackling this work, especially if
there are specific use cases or other factors that should be
considered.{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
