[ https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16919803#comment-16919803 ]
ASF subversion and git services commented on NIFI-6510: ------------------------------------------------------- Commit 7e6eddd0570ecceeb8e14e927cf257c731375fc9 in nifi's branch refs/heads/analytics-framework from Yolanda M. Davis [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=7e6eddd ] NIFI-6510 - documentation updates for enable/disable property > Predictive Analytics for NiFi Metrics > ------------------------------------- > > Key: NIFI-6510 > URL: https://issues.apache.org/jira/browse/NIFI-6510 > Project: Apache NiFi > Issue Type: Improvement > Reporter: Andrew Christianson > Assignee: Yolanda M. Davis > Priority: Major > Fix For: 1.10.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > From Yolanda's email to the list: > > {noformat} > Currently NiFi has lots of metrics available for areas including jvm and flow > component usage (via component status) as well as provenance data which NiFi > makes available either through the UI or reporting tasks (for consumption by > other systems). Past discussions in the community cite users shipping this > data to applications such as Prometheus, ELK stacks, or Ambari metrics for > further analysis in order to capture/review performance issues, detect > anomalies, and send alerts or notifications. These systems are efficient in > capturing and helping to analyze these metrics however it requires > customization work and knowledge of NiFi operations to provide meaningful > analytics within a flow context. > In speaking with Matt Burgess and Andy Christianson on this topic we feel > that there is an opportunity to introduce an analytics framework that could > provide users reasonable predictions on key performance indicators for flows, > such as back pressure and flow rate, to help administrators improve > operational management of NiFi clusters. This framework could offer several > key features: > - Provide a flexible internal analytics engine and model api which supports > the addition of or enhancement to onboard models > - Support integration of remote or cloud based ML models > - Support both traditional and online (incremental) learning methods > - Provide support for model caching (perhaps later inclusion into a model > repository or registry) > - UI enhancements to display prediction information either in existing > summary data, new data visualizations, or directly within the flow/canvas > (where applicable) > For an initial target we thought that back pressure prediction would be a > good starting point for this initiative, given that back pressure detection > is a key indicator of flow performance and many of the metrics currently > available would provide enough data points to create a reasonable performing > model. We have some ideas on how this could be achieved however we wanted to > discuss this more with the community to get thoughts about tackling this > work, especially if there are specific use cases or other factors that should > be considered.{noformat} -- This message was sent by Atlassian Jira (v8.3.2#803003)