I'd also keep in mind how partitioning of kafka topics works with consumers, and whether or not you have a clustered nifi (most people would for production).
If you have a topic with 1 partition and 3 nifi nodes, only one of the nifi nodes has a consumer assigned to this partition (assuming 1 processor, 1 concurrent task), and you wouldn't know which node to call ListenHttp/HandleHttpRequest on. If you have a topic with 3 partitions and 3 nifi nodes, most likely 1 partition is assigned to the consumer on each node, so then you could call the end-point on any node, but you'd only be consuming from 1 partition, not all the messages available in the topic. On Thu, Feb 4, 2021 at 10:05 AM David Handermann <[email protected]> wrote: > Thanks Chris, after looking more closely at the ConsumeKafka processors, I > realized that they don't accept incoming flow files. > > It makes the flow more complex, and subject to potential queuing volume > concerns, but another option might be to use the Wait and Notify > processors. At a basic level, ConsumeKafka would retrieve files and queue > them up with the Wait processor. The HandleHttpRequest processor would > trigger the Notify processor, which would allow the Wait processor to > release the files for HandleHttpResponse. Depending on the volume of > messages from Kafka, and the interval at which the REST client connects, > this could be problematic, but it is one potential option. > > Regards, > David Handermann > > On Thu, Feb 4, 2021 at 8:50 AM Ralph Vercauteren <[email protected]> wrote: > >> Thanks, Chris, >> >> This is the same problem I figured, >> >> Is it possible to create an executeScript reading Kafka? >> >> With regards, >> Mit freundlichem Gruß, >> Met vriendelijke groet, >> >> *Ralph Vercauteren* >> Technical Architect QAD Automation Solutions >> Mobile NL: +31 6 5397 7230 >> [email protected] >> >> This e-mail may contain QAD proprietary information and should be treated >> as confidential. >> >> >> On Thu, Feb 4, 2021 at 3:45 PM Chris Sampson <[email protected]> >> wrote: >> >>> Suspect the problem there would be that the ConsumeKafka processors >>> (according to the documentation) don't accept incoming connections, so >>> you've no way of sending the FlowFile from InvokeHttp/HandleHttpRequest to >>> ConsumeKafka to fetch the messages (which you'd possibly then want to >>> MergeContent/MergeRecord to send everything back as a single response). >>> >>> I'm not very familiar with Kafka processors though, so hopefully someone >>> else can suggest an alternative (or correct me and the NiFi docs if >>> ConsumeKafka isn't just a source processor). >>> >>> >>> >>> --- >>> *Chris Sampson* >>> IT Consultant >>> [email protected] >>> <https://www.naimuri.com/> >>> >>> >>> On Thu, 4 Feb 2021 at 14:26, David Handermann < >>> [email protected]> wrote: >>> >>>> Ralph, >>>> >>>> Thanks for the reply and additional details. I may be >>>> misunderstanding, but just for clarification, in your diagram, are you >>>> expecting NiFi to act as the REST server? If that is the case, then you >>>> should be able to use one of the ConsumeKafka processors to retrieve >>>> messages and return them to the REST client. At a very basic level, the >>>> flow would go from HandleHttpRequest to ConsumeKafka to >>>> HandleHttpResponse. Depending on the format of the messages retrieved from >>>> Kafka, you may need to introduce additional processors prior to >>>> HandleHttpResponse in order to format the output for delivery to the REST >>>> client. >>>> >>>> Regards, >>>> David Handermann >>>> >>>> On Thu, Feb 4, 2021 at 7:47 AM Ralph Vercauteren <[email protected]> wrote: >>>> >>>>> Hi David, >>>>> >>>>> Thanks for responding. >>>>> >>>>> I understand that part, still have the issue to see how we get the >>>>> kafka messages from a topic. >>>>> >>>>> [image: image.png] >>>>> >>>>> With regards, >>>>> Mit freundlichem Gruß, >>>>> Met vriendelijke groet, >>>>> >>>>> *Ralph Vercauteren* >>>>> Technical Architect QAD Automation Solutions >>>>> Mobile NL: +31 6 5397 7230 >>>>> [email protected] >>>>> >>>>> This e-mail may contain QAD proprietary information and should be >>>>> treated as confidential. >>>>> >>>>> >>>>> On Thu, Feb 4, 2021 at 2:21 PM David Handermann < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi Ralph, >>>>>> >>>>>> If you are looking to create a REST service for receiving and >>>>>> processing messages in NiFi, take a look at the ListenHTTP processor, as >>>>>> well as the HandleHttpRequest and HandleHttpResponse processors. The >>>>>> ListenHTTP processor supports receiving HTTP requests and sending the >>>>>> contents to other processors. The ListenHTTP processor provides the >>>>>> easiest approach for getting started. If you need to perform validation >>>>>> or >>>>>> additional processing before sending back a response to REST clients, >>>>>> then >>>>>> using HandleHttpRequest and HandleHttpResponse with other processors in >>>>>> between can support that approach. When using HandleHttpRequest and >>>>>> HandleHttpResponse, it is important to optimize intermediate as much as >>>>>> possible to avoid HTTP clients timing out while waiting for a response. >>>>>> Both processors support a number of configuration properties, so be sure >>>>>> to >>>>>> read through the documentation for each one in order to determine the >>>>>> optimal settings for your configuration. >>>>>> >>>>>> Regards, >>>>>> David Handermann >>>>>> >>>>>> On Thu, Feb 4, 2021 at 3:32 AM Ralph Vercauteren <[email protected]> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> Not sure if I need to ask this in this mailing list or the list of >>>>>>> the developers. >>>>>>> >>>>>>> I want to create a REST server with NIFI. >>>>>>> >>>>>>> I have a Kafka queue storing messages. When something is connected >>>>>>> to the REST server, the Kafka queue has to be read and all waiting >>>>>>> messages >>>>>>> should be collected and send back as a response to the REST client, >>>>>>> >>>>>>> I tried to google for a hint or solution, but I couldn't find it. >>>>>>> >>>>>>> With regards, >>>>>>> Mit freundlichem Gruß, >>>>>>> Met vriendelijke groet, >>>>>>> >>>>>>> *Ralph Vercauteren* >>>>>>> Technical Architect QAD Automation Solutions >>>>>>> Mobile NL: +31 6 5397 7230 >>>>>>> [email protected] >>>>>>> >>>>>>> This e-mail may contain QAD proprietary information and should be >>>>>>> treated as confidential. >>>>>>> >>>>>>
