Hello All, I just wanted to follow up on the discussion we started a couple of weeks ago concerning an analytics framework for NiFi metrics. Working with Andy Christianson and Matt Burgess we shaped our ideas and drafted a proposal for this feature on the Apache NiFi Wiki [1] . We've also begun implementing some of these ideas in a feature branch (which is work in progress) [2]. We’d appreciate any questions or feedback you may have.
Thanks, -yolanda [1] - https://cwiki.apache.org/confluence/display/NIFI/Operational+Analytics+Framework+for+NiFi [2] - https://github.com/apache/nifi/commits/analytics-framework On Wed, Jul 31, 2019 at 9:58 AM Andy Christianson <aichr...@protonmail.com.invalid> wrote: > As someone who operated a 24/7 mission-critical NiFi flow, this feature > would have been a life saver. If I'm heading home on a Friday, it would be > great to have some blinking red lights to let me know that the system > predicts that it is going to experience backpressure sometime over the > weekend, so that corrective action could be taken before leaving. > > Since there is support in the community for this, I created a JIRA to > track the effort: > > https://issues.apache.org/jira/browse/NIFI-6510 > > I also created a JIRA to track the remote protocol: > > https://issues.apache.org/jira/browse/NIFI-6511 > > > Regards, > > Andy > > > Sent from ProtonMail, Swiss-based encrypted email. > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > On Wednesday, July 31, 2019 6:57 AM, Arpad Boda <ab...@apache.org> wrote: > > > If you could share a bit more details about your OPC and Modbus usage, > that > > would be highly appreciated! > > > > On Wed, Jul 31, 2019 at 12:01 PM Craig Knell craig.kn...@gmail.com > wrote: > > > > > Sounds. Great > > > Let me know if you need some help > > > Best regards > > > Craig > > > > > > > On 31 Jul 2019, at 17:31, Arpad Boda ab...@cloudera.com.invalid > wrote: > > > > Craig, > > > > OPC ( https://issues.apache.org/jira/browse/MINIFICPP-819 ) and > Modbus ( > > > > https://issues.apache.org/jira/browse/MINIFICPP-897 ) are on the > way for > > > > MiNiFi c++, hopefully both will be part of next release (0.7.0). > > > > It's gonna be legen... wait for it! :) > > > > Regards, > > > > Arpad > > > > > > > > > On Wed, Jul 31, 2019 at 2:30 AM Craig Knell craig.kn...@gmail.com > > > > > wrote: > > > > > > > > > Hi Folks > > > > > That's our use case now. All our Models are run in python. > > > > > Currently we send events to the ML via http, although this is not > > > > > optimal > > > > > > > > > Our use case is edge ML where we want a light weight wrapper for > > > > > Python code base. > > > > > Jython however does not work with the code base > > > > > I'm think of changing the interface to some thing like REDIS for > pub/sub > > > > > Id also like this to be a push deployment via minifi > > > > > Also support for sensors via protocols via Modbus and OPC would be > great > > > > > Craig > > > > > > > > > > > On Wed, Jul 31, 2019 at 1:43 AM Joe Witt joe.w...@gmail.com > wrote: > > > > > > Definitely something that I think would really help the > community. It > > > > > > might make sense to frame/structure these APIs such that an > internal > > > > > > option > > > > > > could be available to reduce dependencies and get up and running > but > > > > > > that > > > > > > > > > > also just as easily a remote implementation where the engine > lives and > > > > > > is > > > > > > > > > > managed externally could also be supported. > > > > > > Thanks > > > > > > On Tue, Jul 30, 2019 at 1:40 PM Andy LoPresto > alopre...@apache.org > > > > > > wrote: > > > > > > > > > > > > > Yolanda, > > > > > > > I think this sounds like a great idea and will be very useful > to > > > > > > > admins/users, as well as enabling some interesting next-level > > > > > > > functionality > > > > > > > > > > > > > and insight generation. Thanks for putting this out there. > > > > > > > Andy LoPresto > > > > > > > alopre...@apache.org > > > > > > > alopresto.apa...@gmail.com > > > > > > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D > EF69 > > > > > > > > > > > > > > > On Jul 30, 2019, at 5:55 AM, Yolanda Davis < > > > > > > > > yolanda.m.da...@gmail.com> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hello Everyone, > > > > > > > > I wanted to reach out to the community to discuss potentially > > > > > > > > enhancing > > > > > > > > > > > > > > NiFi to include predictive analytics that can help users > assess and > > > > > > > > predict > > > > > > > > NiFi behavior and performance. Currently NiFi has lots of > metrics > > > > > > > > available > > > > > > > > for areas including jvm and flow component usage (via > component > > > > > > > > status) > > > > > > > > > > > > > as > > > > > > > > > > > > > > > well as provenance data which NiFi makes available either > through > > > > > > > > the UI > > > > > > > > > > > > > or > > > > > > > > > > > > > > > reporting tasks (for consumption by other systems). Past > discussions > > > > > > > > in > > > > > > > > > > > > > the > > > > > > > > > > > > > > > community cite users shipping this data to applications such > as > > > > > > > > Prometheus, > > > > > > > > ELK stacks, or Ambari metrics for further analysis in order > to > > > > > > > > capture/review performance issues, detect anomalies, and > send alerts > > > > > > > > or > > > > > > > > > > > > > > notifications. These systems are efficient in capturing and > helping > > > > > > > > to > > > > > > > > > > > > > > analyze these metrics however it requires customization work > and > > > > > > > > knowledge > > > > > > > > of NiFi operations to provide meaningful analytics within a > flow > > > > > > > > context. > > > > > > > > > > > > > > In speaking with Matt Burgess and Andy Christianson on this > topic we > > > > > > > > feel > > > > > > > > > > > > > > that there is an opportunity to introduce an analytics > framework that > > > > > > > > could > > > > > > > > provide users reasonable predictions on key performance > indicators > > > > > > > > for > > > > > > > > > > > > > > flows, such as back pressure and flow rate, to help > administrators > > > > > > > > improve > > > > > > > > operational management of NiFi clusters. This framework > could offer > > > > > > > > several key features: > > > > > > > > > > > > > > > > - Provide a flexible internal analytics engine and model > api which > > > > > > > > supports the addition of or enhancement to onboard models > > > > > > > > > > > > > > > > - Support integration of remote or cloud based ML models > > > > > > > > - Support both traditional and online (incremental) > learning > > > > > > > > methods > > > > > > > > > > > > > > > > > > > > > > - Provide support for model caching (perhaps later > inclusion into > > > > > > > > a > > > > > > > > > > > > > > > > > > > > > > model repository or registry) > > > > > > > > > > > > > > > > - UI enhancements to display prediction information either > in > > > > > > > > existing > > > > > > > > > > > > > > > > > > > > > > summary data, new data visualizations, or directly within the > > > > > > > > flow/canvas > > > > > > > > (where applicable) > > > > > > > > For an initial target we thought that back pressure > prediction would > > > > > > > > be a > > > > > > > > > > > > > > good starting point for this initiative, given that back > pressure > > > > > > > > detection > > > > > > > > is a key indicator of flow performance and many of the > metrics > > > > > > > > currently > > > > > > > > > > > > > > available would provide enough data points to create a > reasonable > > > > > > > > performing model. We have some ideas on how this could be > achieved > > > > > > > > however > > > > > > > > we wanted to discuss this more with the community to get > thoughts > > > > > > > > about > > > > > > > > > > > > > > tackling this work, especially if there are specific use > cases or > > > > > > > > other > > > > > > > > > > > > > > factors that should be considered. > > > > > > > > Looking forward to everyone's thoughts and input. > > > > > > > > Thanks, > > > > > > > > -yolanda > > > > > > > > -- > > > > > > > > yolanda.m.da...@gmail.com > > > > > > > > @YolandaMDavis > > > > > > > > > > -- > > > > > Regards > > > > > Craig Knell > > > > > Mobile: +61 402 128 615 > > > > > Skype: craigknell > > > -- -- yolanda.m.da...@gmail.com @YolandaMDavis
