Hi Craig, Thanks for your feedback and insight on your use cases. What version of MiNiFi are you running? Concerning performing edge ML this may be possible for you with MiNiFi C++ version 0.6.0. That release supports the creation of python processors which can be added to your flow to execute models. Andy Christianson sent me info from a blog written by Marc Parisi on this topic here: https://www.parisi.io/index.php/2019/03/27/hey-bro/
In creating an analytics framework for models we may look to simplify things further where instead of creating a processor for models you could perhaps just implement a simple interface and rely on the engine to execute things as needed. But for now perhaps the python processsor could help fill the gap for you? -yolanda On Wed, Jul 31, 2019 at 6:01 AM Craig Knell <craig.kn...@gmail.com> wrote: > Sounds. Great > > Let me know if you need some help > > Best regards > > Craig > > > > > On 31 Jul 2019, at 17:31, Arpad Boda <ab...@cloudera.com.invalid> wrote: > > > > Craig, > > > > OPC ( https://issues.apache.org/jira/browse/MINIFICPP-819 ) and Modbus ( > > https://issues.apache.org/jira/browse/MINIFICPP-897 ) are on the way for > > MiNiFi c++, hopefully both will be part of next release (0.7.0). > > It's gonna be legen... wait for it! :) > > > > Regards, > > Arpad > > > >> On Wed, Jul 31, 2019 at 2:30 AM Craig Knell <craig.kn...@gmail.com> > wrote: > >> > >> Hi Folks > >> > >> That's our use case now. All our Models are run in python. > >> Currently we send events to the ML via http, although this is not > optimal > >> > >> Our use case is edge ML where we want a light weight wrapper for > >> Python code base. > >> Jython however does not work with the code base > >> I'm think of changing the interface to some thing like REDIS for pub/sub > >> Id also like this to be a push deployment via minifi > >> > >> Also support for sensors via protocols via Modbus and OPC would be great > >> > >> Craig > >> > >>> On Wed, Jul 31, 2019 at 1:43 AM Joe Witt <joe.w...@gmail.com> wrote: > >>> > >>> Definitely something that I think would really help the community. It > >>> might make sense to frame/structure these APIs such that an internal > >> option > >>> could be available to reduce dependencies and get up and running but > that > >>> also just as easily a remote implementation where the engine lives and > is > >>> managed externally could also be supported. > >>> > >>> Thanks > >>> > >>> > >>> On Tue, Jul 30, 2019 at 1:40 PM Andy LoPresto <alopre...@apache.org> > >> wrote: > >>> > >>>> Yolanda, > >>>> > >>>> I think this sounds like a great idea and will be very useful to > >>>> admins/users, as well as enabling some interesting next-level > >> functionality > >>>> and insight generation. Thanks for putting this out there. > >>>> > >>>> Andy LoPresto > >>>> alopre...@apache.org > >>>> alopresto.apa...@gmail.com > >>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > >>>> > >>>>> On Jul 30, 2019, at 5:55 AM, Yolanda Davis < > >> yolanda.m.da...@gmail.com> > >>>> wrote: > >>>>> > >>>>> Hello Everyone, > >>>>> > >>>>> I wanted to reach out to the community to discuss potentially > >> enhancing > >>>>> NiFi to include predictive analytics that can help users assess and > >>>> predict > >>>>> NiFi behavior and performance. Currently NiFi has lots of metrics > >>>> available > >>>>> for areas including jvm and flow component usage (via component > >> status) > >>>> as > >>>>> well as provenance data which NiFi makes available either through > >> the UI > >>>> or > >>>>> reporting tasks (for consumption by other systems). Past discussions > >> in > >>>> the > >>>>> community cite users shipping this data to applications such as > >>>> Prometheus, > >>>>> ELK stacks, or Ambari metrics for further analysis in order to > >>>>> capture/review performance issues, detect anomalies, and send alerts > >> or > >>>>> notifications. These systems are efficient in capturing and helping > >> to > >>>>> analyze these metrics however it requires customization work and > >>>> knowledge > >>>>> of NiFi operations to provide meaningful analytics within a flow > >> context. > >>>>> > >>>>> In speaking with Matt Burgess and Andy Christianson on this topic we > >> feel > >>>>> that there is an opportunity to introduce an analytics framework that > >>>> could > >>>>> provide users reasonable predictions on key performance indicators > >> for > >>>>> flows, such as back pressure and flow rate, to help administrators > >>>> improve > >>>>> operational management of NiFi clusters. This framework could offer > >>>>> several key features: > >>>>> > >>>>> - Provide a flexible internal analytics engine and model api which > >>>>> supports the addition of or enhancement to onboard models > >>>>> - Support integration of remote or cloud based ML models > >>>>> - Support both traditional and online (incremental) learning > >> methods > >>>>> - Provide support for model caching (perhaps later inclusion into > >> a > >>>>> model repository or registry) > >>>>> - UI enhancements to display prediction information either in > >> existing > >>>>> summary data, new data visualizations, or directly within the > >>>> flow/canvas > >>>>> (where applicable) > >>>>> > >>>>> For an initial target we thought that back pressure prediction would > >> be a > >>>>> good starting point for this initiative, given that back pressure > >>>> detection > >>>>> is a key indicator of flow performance and many of the metrics > >> currently > >>>>> available would provide enough data points to create a reasonable > >>>>> performing model. We have some ideas on how this could be achieved > >>>> however > >>>>> we wanted to discuss this more with the community to get thoughts > >> about > >>>>> tackling this work, especially if there are specific use cases or > >> other > >>>>> factors that should be considered. > >>>>> > >>>>> Looking forward to everyone's thoughts and input. > >>>>> > >>>>> Thanks, > >>>>> > >>>>> -yolanda > >>>>> > >>>>> -- > >>>>> yolanda.m.da...@gmail.com > >>>>> @YolandaMDavis > >>>> > >>>> > >> > >> > >> > >> -- > >> Regards > >> > >> Craig Knell > >> Mobile: +61 402 128 615 > >> Skype: craigknell > >> > -- -- yolanda.m.da...@gmail.com @YolandaMDavis
