Hi Marc, thanks to you also for your reply and your thoughts. I cannot comment to the C++ development as we are at home in the java world and for our use cases the footprint of MiNiFis Java Runtime is fine. But Indeed, your exemplary question is pretty close to what we hear day to day, and all sorts of related questions like “for long has it been down”, “how often in the last x days”, … .
Regarding your Sensor Example, this is pretty close to where I come from, and last week Chris, one of the plc4x commiters created a NiFi integration for plc4x which allows to query PLCs and ingest the data to NiFi or MiNiFi (java) Dataflows, which is kind of an extension / complement to your Code, as I understand it. What you write seems to be in good accordance to what we reason about a lot, building blocks for analytics with powerful and adequate semantics for analyzing this kind of data. In fact, we have some of this already achieved in our “home made” framework which we want to join with an eco system to leverage its abilities. I think you answered some of the questions I didn’t ask which is extremely helpful for me and as I see it you are working on problems related to ours which gives me the impression that MiNiFi / NiFi can’t be that wrong of a tool. To your last question, you are absolutely right, keeping and managing state is the core of what we are doing (we used to refer to it as stateful stream processing, but nowadays this term seem to be overused a bit). We used Apache Flink for that but for Edge applications this is overkill. Best Julian PS.: I hope that I find the time to look at the video posted by Joe tomorrow to deepen my understanding of NiFi Von: Marc Parisi <[email protected]> Antworten an: "[email protected]" <[email protected]> Datum: Montag, 1. Oktober 2018 um 19:25 An: "[email protected]" <[email protected]> Betreff: Re: NiFi for IoT Analytics Julian, Very good question. One of the issues I've been tackling has been 'how to ease development' within MiNiFi C++. Users and developers will likely have similar questions to yours, a common one I've heard on forums is "[h]ow can I monitor for pull-down failure in this circuit?" There are a variety of ways we could answer that and a variety of ways we could get that information, but it's perfectly reasonable that consumers want this answer to be sent to NiFi. One thing I've wrestled with is the fact that any given library you have to support sensor or device management is going to fit every use case [1] -- so how can we ease the analysis of that type of information without creating individual stovepipes of flows. I've been thinking a lot about enabling those who wish to use MiNIFi as a library, supporting a variety of sensor/control libraries in a variety of a languages using MiNIFi's C library as an interconnect. This might help you capture that bit and monitor how long it's been there, facilitating the endpoints around it to store that information. This piggybacks on what Joe said, but I think enabling the python scripts, c programs, etc that already exist might help improve the development process while also giving the capabilities that people need. This brings me to some other ideas we've been working on -- create useful building blocks to facilitate this type of development. This may mean providing facilities for the languages to keep and compute state -- as consumables for these specific purposes. A great example might be to monitor a sensor in python, capture and store that information with a few library functions to send back through NiFi where you may perform further evaluation/analysis on all sensor devices. I know I haven't answered your question, but I'm trying to selfishly gauge interest with this type of work. Some of it already exists, but knowing the value of these capabilities is supremely helpful. In the end is the state management of most importance to your evaluation or am I completely off base? [1]https://github.com/apache/nifi-minifi-cpp/tree/master/extensions/sensors -- specific to certain sensors and servos and likely not wildly helpful to others. Thanks, Marc On Sun, Sep 30, 2018 at 7:58 AM Julian Feinauer <[email protected]<mailto:[email protected]>> wrote: Hi Nifi-team, I’m from the incubating plc4x project [1] and I am looking for a framework which is suitable for the management of IoT Datastreams and do some edge computing. As nifi is often times mentioned in relation with IoT I tried to find out what nifi realy does and how it would fit with our ideas (and also the MiNiFi Project seems to fit into this). From what I understood from the Docs and some Videos NiFi looks for me a bit like Apache Camel [2] as it is able to (dynamically) integrate different systems and manage the dataflow between them. So what I did not get exactly I how the payloads are managed between these Endpoints and how much of processing Nifi does itself and how much it delegates to other components (like e.g. Service Activater in EIP). What I am looking for is a framework which does some analysis of data streams coming from controllers that, e.g., control machines or robots. chrisdutz already prepared the first version of an NiFi Endpoint in th Plc4x Repo so we are already able to stream these datasets to NiFi. Whats unclear to me is how we could tackle some of the questions like “how long was this bit set” or “notify me when this signal is below a certain threshold for more than 30s” or so. Is this in the scope of NiFi or is NiFi more of an integration / data-flow layer which is absolutely agnostic of these processing blocks? I hope my questions are not too dumb or I’m not missing NiFis core too much with my current knowledge. I would be happy for some answers or some ideas about how to approach the questions stated above by some experienced users. Best Julian [1] http://plc4x.incubator.apache.org/ [2] https://camel.apache.org/ [3] https://github.com/apache/incubator-plc4x/tree/master/integrations/apache-nifi
