Hi Joe, I am not sure whether I sounds stupid. I was also researching on the option 2. I felt nifi lack the capability to process the big data jobs where data need to be processed with locality.
Initially I was thinking about making the modification to run nifi on yarn. But this approach also will not solve many of the problems like splitting input for parallel execution etc. Of course we can write processor to do that. But I wish the frame work should take a call on that, not processor. I got lost when I thought about running nifi on other framework like flink. But I feel beam is a better option . I also feel that nifi as a beam runner is not a good idea. May be I am not aware what we will gain out of this. Earlier I used aws pipeline and Azure data factory, I felt the concept of data source node is a good and easy to use. Do we have a plan to introduce the data source concept. Tijo On 06-Sep-2016 7:38 pm, "Joe Witt" <[email protected]> wrote: > Gunjan > > No plans at this point. What sort of beam related integration are you > envisioning? There are a couple scenarios that come to mind: > 1) NiFi as a beam runner. Probably not a great fit as by design we'd > not be attempting to address and important cross section of the types > of processing you could define in a beam API. Storm/Flink/Spark > already have those or are under way and we'd just integrate there. > > 2) NiFi being able to submit processing flows to Beam. No idea what > this would really mean yet. > > Thanks > Joe > > On Tue, Sep 6, 2016 at 10:02 AM, Gunjan Dave <[email protected]> > wrote: > > Thank Joe, > > w.r.t to NiFi externals, like we currently have NiFi spark and storm > > externals, any plans for NiFi beam external? Reason is I saw your name as > > one of the contributors there in incubation proposal. > > I was studying beam in more depth hence the question. > > > > > > On Tue, Sep 6, 2016, 6:42 PM Joe Witt <[email protected]> wrote: > >> > >> Gunjan > >> > >> The best indicator of areas of focus I think are listed here > >> https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals > >> > >> In a roadmap discussion thread back in January of this year the items > >> mentioned specifically as trailing the 1.0 release were extension and > >> variable registries. > >> > >> Which items will get addressed in which order is perhaps less precise > >> and more a function of where contributions occur. > >> > >> Thanks > >> Joe > >> > >> On Tue, Sep 6, 2016 at 9:03 AM, Gunjan Dave <[email protected] > > > >> wrote: > >> > Hello NiFi team, now that version 1.0.0 is out in open, what are the > >> > next > >> > big ticket plans? > >> > I saw 1.1.0 jiras but those are mostly bug fixes and minor > enhacements, > >> > but > >> > what are the larger plans from the existing roadmap? > >> > >
