Hi Andy, I am a bit confused again. In my example my deployment is fixed right
Node1 has my HTTP service and my Node 2 has Cassandra. The communication between Node1 and Node2 is through CQL3. A client will initiate the request to the my service on Node 1 via HTTP and you can assume a client is a browser. so if you are saying place Nifi on Node 2 and receive incoming HTTP request ? it doesn't quite make sense to me because the only communication protocol that is allowed between Node1 and Node 2 is CQL3 so I am not sure how one can receive HTTP requests in node 2 ? Thanks! On Wed, Dec 7, 2016 at 5:58 PM, Andy LoPresto <[email protected]> wrote: > Hi Kant, > > Before we get too much farther, I think the Overview [1] and Getting > Started [2] guide and Joe Witt’s OSCON talk [3] might be valuable because > they really underline the problem(s) NiFi solves and the behaviors it uses > to do that. > > In the scenario you described, you only need one Nifi node (forgive the > ASCII diagram). > > Node 1 Node 2 > micro service ---HTTP-—> NiFi ListenHTTP processor > success cxn > NiFi PutCassandraQL processor ---CQL--> > Cassandra instance > > NiFi will receive the HTTP messages from your micro service and ingest > each incoming request, generating a flowfile for each request, placing the > request body into the content claim, and various header data into > attributes. Then you will create a connection between the ListenHTTP (or > HandleHTTPRequest processor if it’s more complex) and a PutCassandraQL > processor. This processor will write the contents of the flowfile to the > Cassandra instance over CQL3. Now, the PutCassandraQL processor expects the > content of the incoming flowfile to be a CQL command, so let’s use the > example where the command is a simple INSERT statement but the values > depend on what is being generated by the micro service. In that case, you > could use a ReplaceText processor between ListenHTTP and PutCassandraQL to > format the query correctly. If the HTTP content was JSON, for example, you > could use an EvaluateJsonPath processor to extract the values you care > about from the content to attributes, and ReplaceText will populate the > templated expression with the proper values using the NiFi Expression > Language [4]. > > For more examples, I’d suggest you look at the NiFi Template Registry [5] > or watch some of the demo videos [6]. The User Guide [7] is also valuable. > Hopefully all of these resources and our continuing conversations will help > you understand and apply NiFi to solve your data flow challenges. > > [1] https://nifi.apache.org/docs/nifi-docs/html/overview.html > [2] https://nifi.apache.org/docs/nifi-docs/html/getting-started.html > [3] https://youtu.be/sQCgtCoZyFQ > [4] https://nifi.apache.org/docs/nifi-docs/html/ > expression-language-guide.html > [5] https://cwiki.apache.org/confluence/display/NIFI/ > Example+Dataflow+Templates > [6] https://nifi.apache.org/videos.html > [7] https://nifi.apache.org/docs/nifi-docs/html/user-guide.html > > Andy LoPresto > [email protected] > *[email protected] <[email protected]>* > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Dec 7, 2016, at 5:31 PM, kant kodali <[email protected]> wrote: > > Hi Koji, > > > Thanks again for your response. > > "Should an application/micro service worry about how it talk with NiFi? > My answer would be no. The service should focus and use the best > protocol or technology for its purpose." > > This sounds great. This is what we want! I also understand the other > capabilities of Nifi you mentioned! > > I have the following question. Say I have a HTTP based micro service on > node 1 and a single process Cassandra on node 2 and say my micro service > talks to cassandra using CQL3 protocol. so how would I integrate NiFi ? > Here is what I understood so far from this thread. > > 1. Run a Nifi JVM on node 1 and node 2. > 2. Configure Nifi JVM on node 1 to use HTTP protocol for incoming messages > and for output messages configure Nifi JVM to use CQL3 protocol. > 3. Configure Nifi JVM on node 2 to use CQL3 protocol messages for both > incoming and outgoing messages. > > Is this correct so far? am I missing something? > > Thanks! > > > > > > > On Wed, Dec 7, 2016 at 4:47 PM, Koji Kawamura <[email protected]> > wrote: > >> Hi Kant, >> >> Glad to know that you liked the explanation, thanks :) >> >> By reading discussion in this thread, I assume what you'd like to do >> with NiFi is, get messages from a system amd do some routing/filtering >> on NiFi, then send it to another system, using a custom protocol. >> If so, you can write two custom NiFi processors, typically named as >> GetXXXX and PutXXXX (XXXX is a name of the protocol). >> >> Based on this architecture, NiFi as a routing layer, if you need to >> store received data into another datastore to extend the use of data >> further, then NiFi will be really useful, because it already support >> variety of data source and protocol integrations. Also, if data schema >> changed at different rate between these systems, then NiFi can be a >> schema migration point. >> >> Data ingested into NiFi called FlowFiles, which has opaque binary >> Content, and string key/value pairs called Attributes, very generic >> data format. >> Once the data is converted to a FlowFile, there's no big difference >> where the FlowFile came from via what protocol. >> So, If NiFi can act as a server for different protocols, then your >> NiFi data flow can support broader type of clients. >> >> There's many existing processors, act as server for certain protocol, >> such as ListenTCP, ListenSyslog, ListenHTTP/HandleHTTPRequest ... etc. >> Once you created PutXXXX for your custom protocol, then you can use >> NiFi to support TCP, HTTP, WebSocket ... protocols to receive data to >> integrate with the destination using XXXX protocol, e.g. ListenHTTP -> >> some data processing on NiFi -> PutXXXX. >> >> Also, MiNiFi can be helpful to bring data integration to the edge. >> >> Should an application/micro service worry about how it talk with NiFi? >> My answer would be no. The service should focus and use the best >> protocol or technology for its purpose. >> But knowing what NiFi can help extending use cases around the service, >> would help you to focus more on the app/service itself. >> >> Thanks, >> Koji >> >> On Wed, Dec 7, 2016 at 6:52 PM, kant kodali <[email protected]> wrote: >> > I am also confused a little bit since I am new to Nifi. I wonder why >> Nifi >> > would act as a server? isn't Nifi a routing layer between systems? >> because >> > this brings in another question about Nifi in general. >> > >> > When I write my applications/microservices do I need to worry about how >> my >> > service would talk to Nifi or do I have the freedom of just focusing on >> my >> > application and using whatever protocol I want and In the end just >> plugin to >> > Nifi which would take care? other words is Nifi a tight integration with >> > applications such that I always have to import a Nifi Library within my >> > application/microservice ? other words do I need to worry about Nifi at >> > programming/development time of an Application/Microservice or at >> deployment >> > time? >> > >> > Sorry if these are naive questions. But answers to those will help >> greatly >> > and prevent me from asking more questions! >> > >> > Thanks much! >> > kant >> > >> > >> > >> > >> > On Wed, Dec 7, 2016 at 1:23 AM, kant kodali <[email protected]> wrote: >> >> >> >> Hi Koji, >> >> >> >> That is an awesome explanation! I expected processors for HTTP2 at very >> >> least since it is widely used ( the entire GRPC stack runs on that). I >> am >> >> not sure how easy or hard it is to build one? >> >> >> >> Thanks! >> >> >> >> On Wed, Dec 7, 2016 at 1:08 AM, Koji Kawamura <[email protected]> >> >> wrote: >> >>> >> >>> Hi Kant, >> >>> >> >>> Although I'm not aware of existing processor for HTTP2 or NSQ, NiFi >> >>> has a set of processors for WebSocket since 1.1.0. >> >>> It enables NiFi to act as a WebSocket client to communicate with a >> >>> remote WebSocket server, or makes NiFi a WebSocket server so that >> >>> remote clients access to it via WebSocket protocol. >> >>> >> >>> I've written a blog post about how to use it, I hope it will be useful >> >>> for your use case: >> >>> http://ijokarumawak.github.io/nifi/2016/11/04/nifi-websocket/ >> >>> >> >>> Thanks, >> >>> Koji >> >>> >> >>> >> >>> >> >>> On Wed, Dec 7, 2016 at 3:23 PM, kant kodali <[email protected]> >> wrote: >> >>> > Thanks a ton guys! Didn't expect Nifi community to be so good! >> (Another >> >>> > convincing reason!) >> >>> > >> >>> > Coming back to the problem, We use NSQ a lot (although not my >> favorite) >> >>> > and >> >>> > want to be able integrate Nifi with NSQ to other systems such as >> Kafka, >> >>> > Spark, Cassandra, ElasticSearch, some micro services that uses HTTP2 >> >>> > and >> >>> > some micro service that uses Websockets >> >>> > >> >>> > (which brings in other question if Nifi has HTTP2 and Websocket >> >>> > processors?) >> >>> > >> >>> > Also below is NSQ protocol spec. They have Java client library as >> well. >> >>> > >> >>> > http://nsq.io/clients/tcp_protocol_spec.html >> >>> > >> >>> > >> >>> > On Tue, Dec 6, 2016 at 5:14 PM, Oleg Zhurakousky >> >>> > <[email protected]> wrote: >> >>> >> >> >>> >> Hi Kant >> >>> >> >> >>> >> What you’re trying to accomplish is definitely possible, however >> more >> >>> >> information may be needed from you. >> >>> >> For example, the way I understand your statement about “integration >> >>> >> with >> >>> >> many systems” is something like JMS, Kafka, TCP, FTP etc…. If that >> is >> >>> >> the >> >>> >> case such integration is definitely possible with your “custom >> system” >> >>> >> by >> >>> >> developing custom Processor and/or ControllerService. >> >>> >> Processors and ControllerServices are the two main integration >> points >> >>> >> within NiFi >> >>> >> You can definitely find may examples by looking at some of the >> >>> >> processors >> >>> >> (i.e., PublishKafka or ConsumeKafka, PublishJMS or ConsumeJMS etc.) >> >>> >> >> >>> >> Let us know if you need more help to guide you through the process. >> >>> >> >> >>> >> Cheers >> >>> >> Oleg >> >>> >> >> >>> >> > On Dec 6, 2016, at 7:46 PM, kant kodali <[email protected]> >> wrote: >> >>> >> > >> >>> >> > HI All, >> >>> >> > >> >>> >> > I understand that Apache Nifi has integration with many systems >> but >> >>> >> > what >> >>> >> > If I have an application that talks a custom protocol ? How do I >> >>> >> > integrate >> >>> >> > Apache Nifi with the custom protocol? >> >>> >> > >> >>> >> > Thanks, >> >>> >> > kant >> >>> >> >> >>> > >> >> >> >> >> > >> > > >
