Gunjan/Joe, I was looking into this at one point... I think the idea would be to implement a Beam source/sink (whatever their terminology is) for NiFi, and this would allow someone to write a single Beam job pulling/pushing data to/from NiFi, and then that single job could be executed in Flink, Storm, or whatever underlying providers are available to Beam.
For example, here is their Kafka integration: https://github.com/apache/incubator-beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java -Bryan On Tue, Sep 6, 2016 at 10:07 AM, Joe Witt <[email protected]> wrote: > Gunjan > > No plans at this point. What sort of beam related integration are you > envisioning? There are a couple scenarios that come to mind: > 1) NiFi as a beam runner. Probably not a great fit as by design we'd > not be attempting to address and important cross section of the types > of processing you could define in a beam API. Storm/Flink/Spark > already have those or are under way and we'd just integrate there. > > 2) NiFi being able to submit processing flows to Beam. No idea what > this would really mean yet. > > Thanks > Joe > > On Tue, Sep 6, 2016 at 10:02 AM, Gunjan Dave <[email protected]> > wrote: > > Thank Joe, > > w.r.t to NiFi externals, like we currently have NiFi spark and storm > > externals, any plans for NiFi beam external? Reason is I saw your name as > > one of the contributors there in incubation proposal. > > I was studying beam in more depth hence the question. > > > > > > On Tue, Sep 6, 2016, 6:42 PM Joe Witt <[email protected]> wrote: > >> > >> Gunjan > >> > >> The best indicator of areas of focus I think are listed here > >> https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals > >> > >> In a roadmap discussion thread back in January of this year the items > >> mentioned specifically as trailing the 1.0 release were extension and > >> variable registries. > >> > >> Which items will get addressed in which order is perhaps less precise > >> and more a function of where contributions occur. > >> > >> Thanks > >> Joe > >> > >> On Tue, Sep 6, 2016 at 9:03 AM, Gunjan Dave <[email protected] > > > >> wrote: > >> > Hello NiFi team, now that version 1.0.0 is out in open, what are the > >> > next > >> > big ticket plans? > >> > I saw 1.1.0 jiras but those are mostly bug fixes and minor > enhacements, > >> > but > >> > what are the larger plans from the existing roadmap? > >> > >
