so do I need to import PostHTTP processor, GetHTTP processor Libraries in my application code? or can I just build a HTTP Server and tell Nifi JVM that "hey here is my Application (it runs on specific IP and port) and it uses HTTP".
On Wed, Dec 7, 2016 at 12:00 PM, Oleg Zhurakousky < [email protected]> wrote: > Kant > > So, yes NiFi already provides basic integration with HTTP by virtue of > PostHTTP processor (post a content of the FlowFile to HTTP endpoint) and > the same in reverse with GetHTTP. So for you case your integration will be > as simple as posting or getting from your micro service > > And what makes it even more interesting is that NIFi in the microservices > architecture can play an even more significant role as coordinator, > orchestrator and mediator of heterogeneous environment of many micro > services. > > Hope that answers it. > Cheers > Oleg > > > On Dec 7, 2016, at 2:48 PM, kant kodali <[email protected]> wrote: > > Sorry I should have been more clear. My question is even more simpler and > naive. Say I am writing a HTTP based microservice (Now I assume Nifi has > integration with HTTP since it is a well known protocol ). Now, how would I > integrate Nifi with my HTTP based server? > > On Wed, Dec 7, 2016 at 4:55 AM, Oleg Zhurakousky < > [email protected]> wrote: > >> Kant >> >> There are couple of questions here so, let me try one at the time. >> 1. “why Nifi would act as a server?”. Well, NiFi is a runtime environment >> where things are *happening*. To be more exact; NiFi is a runtime >> environment where things are *triggered*. The big distinction here is >> trigger vs happening. In other words you may choose (as most processors do) >> to have NiFi act as an execution container and run your code, or you may >> chose for the NiFi to be a triggering container that triggers to run your >> code elsewhere. Example of the later one is the SpringContextProcessor >> which while still runs in the same JVM delegates execution to Spring >> application that runs in its own container (which could as well be a >> separate JVM or nah other container - ala micro services). So in this case >> NiFi still manages the orchestration, mediation, security etc. >> >> 2. “...do I need to worry about how my service would talk to Nifi or do I >> have the freedom of just focusing on my application and using whatever >> protocol I want and In the end just plugin to Nifi which would take care?” >> That is a loaded question and I must admit not fully understood, but i’ll >> give it a try. When integrating with NiFi you use one of the integration >> points such as Processor and/or ControllerService. Those are NiFi known >> strategies (interfaces) and so your custom Processor would need to >> implement such strategy and obviously be compliant with its contract. >> However, the other part of your question is about implementing something >> “independent” and just plug-in and if t’s possible. My answer is still YES >> as long as you design it that way. As an example of such design you may >> want to look at AMQP support where there are a pair of processors >> (PublishAMQP/ConsumeAMQP), but when you look at the code of these >> processors you’ll see that neither has any AMQP dependencies. Instead they >> depend on another case (let’s call it Worker) and that class is completely >> independent of NiFi. So in summary your protocol-specific Processor is >> independent of the protocol and your protocol-specific Worker is >> independent of NiFi and the two delegate between one another through common >> to both object (in this case bye[]). The Spring processor mentioned above >> implements the same pattern. >> >> And one more point about Microservices. I am willing to go as far as >> saying that NiFi is a variation of Microservices container. I am basing it >> n the fact that NiFi components (i.e., processors) implement a fundamental >> micro services pattern - certain independence from it’s runtime and >> persistence of its results - which allows each and every component to be >> managed (start/stopped/reconfigured/changed) independently of >> upstream/downstream components. >> >> Ok, that is a load to process, so I’ll stop ;) >> >> Look forward to more questions >> >> Cheers >> Oleg >> >> >> On Dec 7, 2016, at 4:52 AM, kant kodali <[email protected]> wrote: >> >> I am also confused a little bit since I am new to Nifi. I wonder why Nifi >> would act as a server? isn't Nifi a routing layer between systems? because >> this brings in another question about Nifi in general. >> >> When I write my applications/microservices do I need to worry about how >> my service would talk to Nifi or do I have the freedom of just focusing on >> my application and using whatever protocol I want and In the end just >> plugin to Nifi which would take care? other words is Nifi a tight >> integration with applications such that I always have to import a Nifi >> Library within my application/microservice ? other words do I need to worry >> about Nifi at programming/development time of an Application/Microservice >> or at deployment time? >> >> Sorry if these are naive questions. But answers to those will help >> greatly and prevent me from asking more questions! >> >> Thanks much! >> kant >> >> >> >> >> On Wed, Dec 7, 2016 at 1:23 AM, kant kodali <[email protected]> wrote: >> >>> Hi Koji, >>> >>> That is an awesome explanation! I expected processors for HTTP2 at very >>> least since it is widely used ( the entire GRPC stack runs on that). I am >>> not sure how easy or hard it is to build one? >>> >>> Thanks! >>> >>> On Wed, Dec 7, 2016 at 1:08 AM, Koji Kawamura <[email protected]> >>> wrote: >>> >>>> Hi Kant, >>>> >>>> Although I'm not aware of existing processor for HTTP2 or NSQ, NiFi >>>> has a set of processors for WebSocket since 1.1.0. >>>> It enables NiFi to act as a WebSocket client to communicate with a >>>> remote WebSocket server, or makes NiFi a WebSocket server so that >>>> remote clients access to it via WebSocket protocol. >>>> >>>> I've written a blog post about how to use it, I hope it will be useful >>>> for your use case: >>>> http://ijokarumawak.github.io/nifi/2016/11/04/nifi-websocket/ >>>> >>>> Thanks, >>>> Koji >>>> >>>> >>>> >>>> On Wed, Dec 7, 2016 at 3:23 PM, kant kodali <[email protected]> wrote: >>>> > Thanks a ton guys! Didn't expect Nifi community to be so good! >>>> (Another >>>> > convincing reason!) >>>> > >>>> > Coming back to the problem, We use NSQ a lot (although not my >>>> favorite) and >>>> > want to be able integrate Nifi with NSQ to other systems such as >>>> Kafka, >>>> > Spark, Cassandra, ElasticSearch, some micro services that uses HTTP2 >>>> and >>>> > some micro service that uses Websockets >>>> > >>>> > (which brings in other question if Nifi has HTTP2 and Websocket >>>> processors?) >>>> > >>>> > Also below is NSQ protocol spec. They have Java client library as >>>> well. >>>> > >>>> > http://nsq.io/clients/tcp_protocol_spec.html >>>> > >>>> > >>>> > On Tue, Dec 6, 2016 at 5:14 PM, Oleg Zhurakousky >>>> > <[email protected]> wrote: >>>> >> >>>> >> Hi Kant >>>> >> >>>> >> What you’re trying to accomplish is definitely possible, however more >>>> >> information may be needed from you. >>>> >> For example, the way I understand your statement about “integration >>>> with >>>> >> many systems” is something like JMS, Kafka, TCP, FTP etc…. If that >>>> is the >>>> >> case such integration is definitely possible with your “custom >>>> system” by >>>> >> developing custom Processor and/or ControllerService. >>>> >> Processors and ControllerServices are the two main integration points >>>> >> within NiFi >>>> >> You can definitely find may examples by looking at some of the >>>> processors >>>> >> (i.e., PublishKafka or ConsumeKafka, PublishJMS or ConsumeJMS etc.) >>>> >> >>>> >> Let us know if you need more help to guide you through the process. >>>> >> >>>> >> Cheers >>>> >> Oleg >>>> >> >>>> >> > On Dec 6, 2016, at 7:46 PM, kant kodali <[email protected]> >>>> wrote: >>>> >> > >>>> >> > HI All, >>>> >> > >>>> >> > I understand that Apache Nifi has integration with many systems >>>> but what >>>> >> > If I have an application that talks a custom protocol ? How do I >>>> integrate >>>> >> > Apache Nifi with the custom protocol? >>>> >> > >>>> >> > Thanks, >>>> >> > kant >>>> >> >>>> > >>>> >>> >>> >> >> > >
