Thats what I was looking for. That sounds awesome because it is decoupled from the application itself!
Finally, is there any example or some pointer you can point me to that shows how Nifi takes cares in case of different protocols (for now you can assume all standard protocol) so say a simplest case An HTTP based Application talks to cassandra or some other database Via CQL V4 or psql in case of postgress . How can I add Nifi ? or Any other simple use case would be great! I searched to see if there is any book I can read on Nifi but I couldn't find one. On Wed, Dec 7, 2016 at 12:19 PM, Oleg Zhurakousky < [email protected]> wrote: > Ohh no, not at all. Just tell it the host/port combination and off you go, > basically the second part of what you said > > Oleg > > On Dec 7, 2016, at 3:08 PM, kant kodali <[email protected]> wrote: > > so do I need to import PostHTTP processor, GetHTTP processor Libraries in > my application code? or can I just build a HTTP Server and tell Nifi JVM > that "hey here is my Application (it runs on specific IP and port) and it > uses HTTP". > > On Wed, Dec 7, 2016 at 12:00 PM, Oleg Zhurakousky < > [email protected]> wrote: > >> Kant >> >> So, yes NiFi already provides basic integration with HTTP by virtue of >> PostHTTP processor (post a content of the FlowFile to HTTP endpoint) and >> the same in reverse with GetHTTP. So for you case your integration will be >> as simple as posting or getting from your micro service >> >> And what makes it even more interesting is that NIFi in the microservices >> architecture can play an even more significant role as coordinator, >> orchestrator and mediator of heterogeneous environment of many micro >> services. >> >> Hope that answers it. >> Cheers >> Oleg >> >> >> On Dec 7, 2016, at 2:48 PM, kant kodali <[email protected]> wrote: >> >> Sorry I should have been more clear. My question is even more simpler and >> naive. Say I am writing a HTTP based microservice (Now I assume Nifi has >> integration with HTTP since it is a well known protocol ). Now, how would I >> integrate Nifi with my HTTP based server? >> >> On Wed, Dec 7, 2016 at 4:55 AM, Oleg Zhurakousky < >> [email protected]> wrote: >> >>> Kant >>> >>> There are couple of questions here so, let me try one at the time. >>> 1. “why Nifi would act as a server?”. Well, NiFi is a runtime >>> environment where things are *happening*. To be more exact; NiFi is a >>> runtime environment where things are *triggered*. The big distinction >>> here is trigger vs happening. In other words you may choose (as most >>> processors do) to have NiFi act as an execution container and run your >>> code, or you may chose for the NiFi to be a triggering container that >>> triggers to run your code elsewhere. Example of the later one is the >>> SpringContextProcessor which while still runs in the same JVM delegates >>> execution to Spring application that runs in its own container (which could >>> as well be a separate JVM or nah other container - ala micro services). So >>> in this case NiFi still manages the orchestration, mediation, security etc. >>> >>> 2. “...do I need to worry about how my service would talk to Nifi or do >>> I have the freedom of just focusing on my application and using whatever >>> protocol I want and In the end just plugin to Nifi which would take care?” >>> That is a loaded question and I must admit not fully understood, but i’ll >>> give it a try. When integrating with NiFi you use one of the integration >>> points such as Processor and/or ControllerService. Those are NiFi known >>> strategies (interfaces) and so your custom Processor would need to >>> implement such strategy and obviously be compliant with its contract. >>> However, the other part of your question is about implementing something >>> “independent” and just plug-in and if t’s possible. My answer is still YES >>> as long as you design it that way. As an example of such design you may >>> want to look at AMQP support where there are a pair of processors >>> (PublishAMQP/ConsumeAMQP), but when you look at the code of these >>> processors you’ll see that neither has any AMQP dependencies. Instead they >>> depend on another case (let’s call it Worker) and that class is completely >>> independent of NiFi. So in summary your protocol-specific Processor is >>> independent of the protocol and your protocol-specific Worker is >>> independent of NiFi and the two delegate between one another through common >>> to both object (in this case bye[]). The Spring processor mentioned above >>> implements the same pattern. >>> >>> And one more point about Microservices. I am willing to go as far as >>> saying that NiFi is a variation of Microservices container. I am basing it >>> n the fact that NiFi components (i.e., processors) implement a fundamental >>> micro services pattern - certain independence from it’s runtime and >>> persistence of its results - which allows each and every component to be >>> managed (start/stopped/reconfigured/changed) independently of >>> upstream/downstream components. >>> >>> Ok, that is a load to process, so I’ll stop ;) >>> >>> Look forward to more questions >>> >>> Cheers >>> Oleg >>> >>> >>> On Dec 7, 2016, at 4:52 AM, kant kodali <[email protected]> wrote: >>> >>> I am also confused a little bit since I am new to Nifi. I wonder why >>> Nifi would act as a server? isn't Nifi a routing layer between systems? >>> because this brings in another question about Nifi in general. >>> >>> When I write my applications/microservices do I need to worry about how >>> my service would talk to Nifi or do I have the freedom of just focusing on >>> my application and using whatever protocol I want and In the end just >>> plugin to Nifi which would take care? other words is Nifi a tight >>> integration with applications such that I always have to import a Nifi >>> Library within my application/microservice ? other words do I need to worry >>> about Nifi at programming/development time of an Application/Microservice >>> or at deployment time? >>> >>> Sorry if these are naive questions. But answers to those will help >>> greatly and prevent me from asking more questions! >>> >>> Thanks much! >>> kant >>> >>> >>> >>> >>> On Wed, Dec 7, 2016 at 1:23 AM, kant kodali <[email protected]> wrote: >>> >>>> Hi Koji, >>>> >>>> That is an awesome explanation! I expected processors for HTTP2 at very >>>> least since it is widely used ( the entire GRPC stack runs on that). I am >>>> not sure how easy or hard it is to build one? >>>> >>>> Thanks! >>>> >>>> On Wed, Dec 7, 2016 at 1:08 AM, Koji Kawamura <[email protected]> >>>> wrote: >>>> >>>>> Hi Kant, >>>>> >>>>> Although I'm not aware of existing processor for HTTP2 or NSQ, NiFi >>>>> has a set of processors for WebSocket since 1.1.0. >>>>> It enables NiFi to act as a WebSocket client to communicate with a >>>>> remote WebSocket server, or makes NiFi a WebSocket server so that >>>>> remote clients access to it via WebSocket protocol. >>>>> >>>>> I've written a blog post about how to use it, I hope it will be useful >>>>> for your use case: >>>>> http://ijokarumawak.github.io/nifi/2016/11/04/nifi-websocket/ >>>>> >>>>> Thanks, >>>>> Koji >>>>> >>>>> >>>>> >>>>> On Wed, Dec 7, 2016 at 3:23 PM, kant kodali <[email protected]> >>>>> wrote: >>>>> > Thanks a ton guys! Didn't expect Nifi community to be so good! >>>>> (Another >>>>> > convincing reason!) >>>>> > >>>>> > Coming back to the problem, We use NSQ a lot (although not my >>>>> favorite) and >>>>> > want to be able integrate Nifi with NSQ to other systems such as >>>>> Kafka, >>>>> > Spark, Cassandra, ElasticSearch, some micro services that uses HTTP2 >>>>> and >>>>> > some micro service that uses Websockets >>>>> > >>>>> > (which brings in other question if Nifi has HTTP2 and Websocket >>>>> processors?) >>>>> > >>>>> > Also below is NSQ protocol spec. They have Java client library as >>>>> well. >>>>> > >>>>> > http://nsq.io/clients/tcp_protocol_spec.html >>>>> > >>>>> > >>>>> > On Tue, Dec 6, 2016 at 5:14 PM, Oleg Zhurakousky >>>>> > <[email protected]> wrote: >>>>> >> >>>>> >> Hi Kant >>>>> >> >>>>> >> What you’re trying to accomplish is definitely possible, however >>>>> more >>>>> >> information may be needed from you. >>>>> >> For example, the way I understand your statement about “integration >>>>> with >>>>> >> many systems” is something like JMS, Kafka, TCP, FTP etc…. If that >>>>> is the >>>>> >> case such integration is definitely possible with your “custom >>>>> system” by >>>>> >> developing custom Processor and/or ControllerService. >>>>> >> Processors and ControllerServices are the two main integration >>>>> points >>>>> >> within NiFi >>>>> >> You can definitely find may examples by looking at some of the >>>>> processors >>>>> >> (i.e., PublishKafka or ConsumeKafka, PublishJMS or ConsumeJMS etc.) >>>>> >> >>>>> >> Let us know if you need more help to guide you through the process. >>>>> >> >>>>> >> Cheers >>>>> >> Oleg >>>>> >> >>>>> >> > On Dec 6, 2016, at 7:46 PM, kant kodali <[email protected]> >>>>> wrote: >>>>> >> > >>>>> >> > HI All, >>>>> >> > >>>>> >> > I understand that Apache Nifi has integration with many systems >>>>> but what >>>>> >> > If I have an application that talks a custom protocol ? How do I >>>>> integrate >>>>> >> > Apache Nifi with the custom protocol? >>>>> >> > >>>>> >> > Thanks, >>>>> >> > kant >>>>> >> >>>>> > >>>>> >>>> >>>> >>> >>> >> >> > >
