[ https://issues.apache.org/jira/browse/NIFI-4871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361231#comment-16361231 ]
Joseph Percivall commented on NIFI-4871: ---------------------------------------- This should probably be handled in the same way was funnels, which has a while loop[1] that transfers all the files in a single onTrigger but in 100 FlowFile batches. [1] https://github.com/apache/nifi/blob/d838f61291d2582592754a37314911b701c6891b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/StandardFunnel.java#L365 > Data Queuing at Process Group Input Port > ---------------------------------------- > > Key: NIFI-4871 > URL: https://issues.apache.org/jira/browse/NIFI-4871 > Project: Apache NiFi > Issue Type: Bug > Reporter: Brandon DeVries > Priority: Major > > When a flow is under heavy load, we've seen data queue on the way in to a > process group. There is nothing queued *inside* the process group, and no > back pressure... it simply isn't moving files fast enough. As a workaround, > multiple input ports were added... but that's ugly. > The first issue may be the limit of 100 files transferred on any call to > onTrigger()[1]. 100 seems low, ideally it would be nice to fill to the back > pressure point (or some reasonable limit if set to "0"). This may also be > compounded by some scheduling issues we've seen, but not reliably enough to > actually diagnose. But the combination of not transferring enough files on > each run and not running as often as it "should" is definitely causing > issues. The scheduling issue may be complex, but transferring more files > should be pretty straightforward... > > [1] > [https://github.com/apache/nifi/blob/d838f61291d2582592754a37314911b701c6891b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/connectable/LocalPort.java#L76] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)