Got it, thanks for the clarification. On Mon, 3 Jun 2024, 18:00 Michael Moser, <moser...@gmail.com> wrote:
> When ControlRate yields, it will yield for the full Yield Duration. So > your scenario, Alexis, will allow 10 files through in the first > half-second, and when the 11th arrives it will yield for 1 second (if Yield > Duration = 1 sec). After that yield ends, it will allow the next 10 files > through immediately (assuming at least 10 have been waiting) and yield > again for 1 second. > > -- Mike > > On Mon, Jun 3, 2024 at 9:41 AM Alexis Sarda-Espinosa < > sarda.espin...@gmail.com> wrote: > >> Hi Jim, >> >> Thanks for the prompt response. I think that could work, but now that I >> look at it, the documentation says that accuracy can be increased "by >> decreasing the Yield Duration" - if I want, say, 10 files per second and 20 >> arrive evenly spread out within a second, does that mean that after half a >> second the processor would yield? If yes, would it yield for another half a >> second, or for the whole Yield Duration that's configured? >> >> Regards, >> Alexis. >> >> Am Mo., 3. Juni 2024 um 15:08 Uhr schrieb Jim Steinebrey < >> jrsteineb...@gmail.com>: >> >>> Hi Alexis, >>> Yes, what you see in your experiments is the expected behavior. >>> The NiFi documentation for FlowFileCurrency says: >>> SINGLE_BATCH_PER_NODE >>> <https://javadoc.io/static/org.apache.nifi/nifi-framework-core-api/1.17.0/org/apache/nifi/groups/FlowFileConcurrency.html#SINGLE_BATCH_PER_NODE> >>> When an Input Port is triggered to run, it will pull *all* FlowFiles >>> from its input queues into the Process Group as a single batch of FlowFiles. >>> >>> In addition, back pressure only applies to the processor BEFORE the >>> connection queue, not the input port AFTER the connection queue. >>> >>> Would it work for you to add a ControlRate processor before input port >>> to separate the flow files into the max batch size you seek? >>> >>> Regards, >>> Jim >>> >>> >>> On Jun 3, 2024, at 5:41 AM, Alexis Sarda-Espinosa < >>> sarda.espin...@gmail.com> wrote: >>> >>> Hello, >>> >>> I am using NiFi 1.26.0. I created a processor group and configured its >>> FlowFile Concurrency as "Single Batch Per Node". The group has a single >>> input port that connects to only one processor. I configured the queue >>> between the input port and the processor to generate back pressure, hoping >>> that if the Back Pressure Object Threshold is X, the maximum batch size per >>> node would be equal to X, but based on my experiments it seems that the >>> input port will consume as much data as it can without considering the back >>> pressure from the queues connected to it. Is this expected? >>> >>> Regards, >>> Alexis. >>> >>> >>>
