Brandon DeVries created NIFI-4871: ------------------------------------- Summary: Data Queuing at Process Group Input Port Key: NIFI-4871 URL: https://issues.apache.org/jira/browse/NIFI-4871 Project: Apache NiFi Issue Type: Bug Reporter: Brandon DeVries
When a flow is under heavy load, we've seen data queue on the way in to a process group. There is nothing queued *inside* the process group, and no back pressure... it simply isn't moving files fast enough. As a workaround, multiple input ports were added... but that's ugly. The first issue may be the limit of 100 files transferred on any call to onTrigger()[1]. 100 seems low, ideally it would be nice to fill to the back pressure point (or some reasonable limit if set to "0"). This may also be compounded by some scheduling issues we've seen, but not reliably enough to actually diagnose. But the combination of not transferring enough files on each run and not running as often as it "should" is definitely causing issues. The scheduling issue may be complex, but transferring more files should be pretty straightforward... [1] [https://github.com/apache/nifi/blob/d838f61291d2582592754a37314911b701c6891b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/connectable/LocalPort.java#L76] -- This message was sent by Atlassian JIRA (v7.6.3#76005)