Joseph Percivall commented on NIFI-4871:

This should probably be handled in the same way was funnels, which has a while 
loop[1] that transfers all the files in a single onTrigger but in 100 FlowFile 



> Data Queuing at Process Group Input Port
> ----------------------------------------
>                 Key: NIFI-4871
>                 URL: https://issues.apache.org/jira/browse/NIFI-4871
>             Project: Apache NiFi
>          Issue Type: Bug
>            Reporter: Brandon DeVries
>            Priority: Major
> When a flow is under heavy load, we've seen data queue on the way in to a 
> process group.  There is nothing queued *inside* the process group, and no 
> back pressure... it simply isn't moving files fast enough.  As a workaround, 
> multiple input ports were added... but that's ugly.
> The first issue may be the limit of 100 files transferred on any call to 
> onTrigger()[1].  100 seems low, ideally it would be nice to fill to the back 
> pressure point (or some reasonable limit if set to "0").  This may also be 
> compounded by some scheduling issues we've seen, but not reliably enough to 
> actually diagnose.  But the combination of not transferring enough files on 
> each run and not running as often as it "should" is definitely causing 
> issues.  The scheduling issue may be complex, but transferring more files 
> should be pretty straightforward...
>  [1] 
> [https://github.com/apache/nifi/blob/d838f61291d2582592754a37314911b701c6891b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/connectable/LocalPort.java#L76]

This message was sent by Atlassian JIRA

Reply via email to