Re: What next with NiFi
Hi Joe, I am not sure whether I sounds stupid. I was also researching on the option 2. I felt nifi lack the capability to process the big data jobs where data need to be processed with locality. Initially I was thinking about making the modification to run nifi on yarn. But this approach also will not solve many of the problems like splitting input for parallel execution etc. Of course we can write processor to do that. But I wish the frame work should take a call on that, not processor. I got lost when I thought about running nifi on other framework like flink. But I feel beam is a better option . I also feel that nifi as a beam runner is not a good idea. May be I am not aware what we will gain out of this. Earlier I used aws pipeline and Azure data factory, I felt the concept of data source node is a good and easy to use. Do we have a plan to introduce the data source concept. Tijo On 06-Sep-2016 7:38 pm, "Joe Witt"wrote: > Gunjan > > No plans at this point. What sort of beam related integration are you > envisioning? There are a couple scenarios that come to mind: > 1) NiFi as a beam runner. Probably not a great fit as by design we'd > not be attempting to address and important cross section of the types > of processing you could define in a beam API. Storm/Flink/Spark > already have those or are under way and we'd just integrate there. > > 2) NiFi being able to submit processing flows to Beam. No idea what > this would really mean yet. > > Thanks > Joe > > On Tue, Sep 6, 2016 at 10:02 AM, Gunjan Dave > wrote: > > Thank Joe, > > w.r.t to NiFi externals, like we currently have NiFi spark and storm > > externals, any plans for NiFi beam external? Reason is I saw your name as > > one of the contributors there in incubation proposal. > > I was studying beam in more depth hence the question. > > > > > > On Tue, Sep 6, 2016, 6:42 PM Joe Witt wrote: > >> > >> Gunjan > >> > >> The best indicator of areas of focus I think are listed here > >> https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals > >> > >> In a roadmap discussion thread back in January of this year the items > >> mentioned specifically as trailing the 1.0 release were extension and > >> variable registries. > >> > >> Which items will get addressed in which order is perhaps less precise > >> and more a function of where contributions occur. > >> > >> Thanks > >> Joe > >> > >> On Tue, Sep 6, 2016 at 9:03 AM, Gunjan Dave > > >> wrote: > >> > Hello NiFi team, now that version 1.0.0 is out in open, what are the > >> > next > >> > big ticket plans? > >> > I saw 1.1.0 jiras but those are mostly bug fixes and minor > enhacements, > >> > but > >> > what are the larger plans from the existing roadmap? > >> > >
Re: What next with NiFi
Ram, There was some design discussion about this here [1], see the part about "getAuthorizationHistory", but I don't think we currently have any JIRAs defined. The file-based authorizer provided in the 1.0.0 release does not provide this capability yet, so you would need to implement a custom authorizer that wrapped the file-based authorizer and then did something with the results. Another option is to use Ranger as the authorizer for NiFi, Ranger provides auditing and typically stores the audit logs in Solr. -Bryan [1] https://cwiki.apache.org/confluence/display/NIFI/Support+Authorizer+API On Tue, Sep 6, 2016 at 10:24 AM, Nathamuni, Ramanujam <rnatham...@tiaa.org> wrote: > Hi Bryan, > > > > Thanks, Yes. How do we get the audit log from NiFi? I need to integrate > the audit log with IBM aveksa or IBM Guradium every day. > > > > Thanks, > > Ram > > > > *From:* Bryan Bende [mailto:bbe...@gmail.com] > *Sent:* Tuesday, September 06, 2016 10:20 AM > > *To:* users@nifi.apache.org > *Subject:* Re: What next with NiFi > > > > Ram, > > > > Are you talking about isolation from a security perspective? If so, that > is now available in the 1.0.0 release. > > > > You can restrict access to any portion of the flow through a policy model. > > > > -Bryan > > > > On Tue, Sep 6, 2016 at 10:16 AM, Nathamuni, Ramanujam <rnatham...@tiaa.org> > wrote: > > Hi Joe and Team: > > Enterprise needs very high data security and audit trial. What is the > vision to have canvas/processor/processor group ...etc. isolated for > project/LOBs? > > Thanks, > Ram > > > -Original Message- > From: Joe Witt [mailto:joe.w...@gmail.com] > Sent: Tuesday, September 06, 2016 9:12 AM > To: users@nifi.apache.org > Subject: Re: What next with NiFi > > Gunjan > > The best indicator of areas of focus I think are listed here > https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals > > In a roadmap discussion thread back in January of this year the items > mentioned specifically as trailing the 1.0 release were extension and > variable registries. > > Which items will get addressed in which order is perhaps less precise and > more a function of where contributions occur. > > Thanks > Joe > > On Tue, Sep 6, 2016 at 9:03 AM, Gunjan Dave <gunjanpiyushd...@gmail.com> > wrote: > > Hello NiFi team, now that version 1.0.0 is out in open, what are the > > next big ticket plans? > > I saw 1.1.0 jiras but those are mostly bug fixes and minor > > enhacements, but what are the larger plans from the existing roadmap? > > > > * > This e-mail may contain confidential or privileged information. > If you are not the intended recipient, please notify the sender > immediately and then delete it. > > TIAA > * > > > > > * > This e-mail may contain confidential or privileged information. > If you are not the intended recipient, please notify the sender > immediately and then delete it. > > TIAA > * >
Re: What next with NiFi
Ram, Are you talking about isolation from a security perspective? If so, that is now available in the 1.0.0 release. You can restrict access to any portion of the flow through a policy model. -Bryan On Tue, Sep 6, 2016 at 10:16 AM, Nathamuni, Ramanujam <rnatham...@tiaa.org> wrote: > Hi Joe and Team: > > Enterprise needs very high data security and audit trial. What is the > vision to have canvas/processor/processor group ...etc. isolated for > project/LOBs? > > Thanks, > Ram > > -Original Message- > From: Joe Witt [mailto:joe.w...@gmail.com] > Sent: Tuesday, September 06, 2016 9:12 AM > To: users@nifi.apache.org > Subject: Re: What next with NiFi > > Gunjan > > The best indicator of areas of focus I think are listed here > https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals > > In a roadmap discussion thread back in January of this year the items > mentioned specifically as trailing the 1.0 release were extension and > variable registries. > > Which items will get addressed in which order is perhaps less precise and > more a function of where contributions occur. > > Thanks > Joe > > On Tue, Sep 6, 2016 at 9:03 AM, Gunjan Dave <gunjanpiyushd...@gmail.com> > wrote: > > Hello NiFi team, now that version 1.0.0 is out in open, what are the > > next big ticket plans? > > I saw 1.1.0 jiras but those are mostly bug fixes and minor > > enhacements, but what are the larger plans from the existing roadmap? > > > > * > This e-mail may contain confidential or privileged information. > If you are not the intended recipient, please notify the sender > immediately and then delete it. > > TIAA > * >
RE: What next with NiFi
Hi Joe and Team: Enterprise needs very high data security and audit trial. What is the vision to have canvas/processor/processor group ...etc. isolated for project/LOBs? Thanks, Ram -Original Message- From: Joe Witt [mailto:joe.w...@gmail.com] Sent: Tuesday, September 06, 2016 9:12 AM To: users@nifi.apache.org Subject: Re: What next with NiFi Gunjan The best indicator of areas of focus I think are listed here https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals In a roadmap discussion thread back in January of this year the items mentioned specifically as trailing the 1.0 release were extension and variable registries. Which items will get addressed in which order is perhaps less precise and more a function of where contributions occur. Thanks Joe On Tue, Sep 6, 2016 at 9:03 AM, Gunjan Dave <gunjanpiyushd...@gmail.com> wrote: > Hello NiFi team, now that version 1.0.0 is out in open, what are the > next big ticket plans? > I saw 1.1.0 jiras but those are mostly bug fixes and minor > enhacements, but what are the larger plans from the existing roadmap? > * This e-mail may contain confidential or privileged information. If you are not the intended recipient, please notify the sender immediately and then delete it. TIAA *
Re: What next with NiFi
Gunjan/Joe, I was looking into this at one point... I think the idea would be to implement a Beam source/sink (whatever their terminology is) for NiFi, and this would allow someone to write a single Beam job pulling/pushing data to/from NiFi, and then that single job could be executed in Flink, Storm, or whatever underlying providers are available to Beam. For example, here is their Kafka integration: https://github.com/apache/incubator-beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java -Bryan On Tue, Sep 6, 2016 at 10:07 AM, Joe Wittwrote: > Gunjan > > No plans at this point. What sort of beam related integration are you > envisioning? There are a couple scenarios that come to mind: > 1) NiFi as a beam runner. Probably not a great fit as by design we'd > not be attempting to address and important cross section of the types > of processing you could define in a beam API. Storm/Flink/Spark > already have those or are under way and we'd just integrate there. > > 2) NiFi being able to submit processing flows to Beam. No idea what > this would really mean yet. > > Thanks > Joe > > On Tue, Sep 6, 2016 at 10:02 AM, Gunjan Dave > wrote: > > Thank Joe, > > w.r.t to NiFi externals, like we currently have NiFi spark and storm > > externals, any plans for NiFi beam external? Reason is I saw your name as > > one of the contributors there in incubation proposal. > > I was studying beam in more depth hence the question. > > > > > > On Tue, Sep 6, 2016, 6:42 PM Joe Witt wrote: > >> > >> Gunjan > >> > >> The best indicator of areas of focus I think are listed here > >> https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals > >> > >> In a roadmap discussion thread back in January of this year the items > >> mentioned specifically as trailing the 1.0 release were extension and > >> variable registries. > >> > >> Which items will get addressed in which order is perhaps less precise > >> and more a function of where contributions occur. > >> > >> Thanks > >> Joe > >> > >> On Tue, Sep 6, 2016 at 9:03 AM, Gunjan Dave > > >> wrote: > >> > Hello NiFi team, now that version 1.0.0 is out in open, what are the > >> > next > >> > big ticket plans? > >> > I saw 1.1.0 jiras but those are mostly bug fixes and minor > enhacements, > >> > but > >> > what are the larger plans from the existing roadmap? > >> > >
Re: What next with NiFi
Gunjan No plans at this point. What sort of beam related integration are you envisioning? There are a couple scenarios that come to mind: 1) NiFi as a beam runner. Probably not a great fit as by design we'd not be attempting to address and important cross section of the types of processing you could define in a beam API. Storm/Flink/Spark already have those or are under way and we'd just integrate there. 2) NiFi being able to submit processing flows to Beam. No idea what this would really mean yet. Thanks Joe On Tue, Sep 6, 2016 at 10:02 AM, Gunjan Davewrote: > Thank Joe, > w.r.t to NiFi externals, like we currently have NiFi spark and storm > externals, any plans for NiFi beam external? Reason is I saw your name as > one of the contributors there in incubation proposal. > I was studying beam in more depth hence the question. > > > On Tue, Sep 6, 2016, 6:42 PM Joe Witt wrote: >> >> Gunjan >> >> The best indicator of areas of focus I think are listed here >> https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals >> >> In a roadmap discussion thread back in January of this year the items >> mentioned specifically as trailing the 1.0 release were extension and >> variable registries. >> >> Which items will get addressed in which order is perhaps less precise >> and more a function of where contributions occur. >> >> Thanks >> Joe >> >> On Tue, Sep 6, 2016 at 9:03 AM, Gunjan Dave >> wrote: >> > Hello NiFi team, now that version 1.0.0 is out in open, what are the >> > next >> > big ticket plans? >> > I saw 1.1.0 jiras but those are mostly bug fixes and minor enhacements, >> > but >> > what are the larger plans from the existing roadmap? >> >
Re: What next with NiFi
Thank Joe, w.r.t to NiFi externals, like we currently have NiFi spark and storm externals, any plans for NiFi beam external? Reason is I saw your name as one of the contributors there in incubation proposal. I was studying beam in more depth hence the question. On Tue, Sep 6, 2016, 6:42 PM Joe Wittwrote: > Gunjan > > The best indicator of areas of focus I think are listed here > https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals > > In a roadmap discussion thread back in January of this year the items > mentioned specifically as trailing the 1.0 release were extension and > variable registries. > > Which items will get addressed in which order is perhaps less precise > and more a function of where contributions occur. > > Thanks > Joe > > On Tue, Sep 6, 2016 at 9:03 AM, Gunjan Dave > wrote: > > Hello NiFi team, now that version 1.0.0 is out in open, what are the next > > big ticket plans? > > I saw 1.1.0 jiras but those are mostly bug fixes and minor enhacements, > but > > what are the larger plans from the existing roadmap? > > >
Re: What next with NiFi
Gunjan The best indicator of areas of focus I think are listed here https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals In a roadmap discussion thread back in January of this year the items mentioned specifically as trailing the 1.0 release were extension and variable registries. Which items will get addressed in which order is perhaps less precise and more a function of where contributions occur. Thanks Joe On Tue, Sep 6, 2016 at 9:03 AM, Gunjan Davewrote: > Hello NiFi team, now that version 1.0.0 is out in open, what are the next > big ticket plans? > I saw 1.1.0 jiras but those are mostly bug fixes and minor enhacements, but > what are the larger plans from the existing roadmap? >