Brandon DeVries created NIFI-4871:
-------------------------------------
Summary: Data Queuing at Process Group Input Port
Key: NIFI-4871
URL: https://issues.apache.org/jira/browse/NIFI-4871
Project: Apache NiFi
Issue Type: Bug
Reporter: Brandon DeVries
When a flow is under heavy load, we've seen data queue on the way in to a
process group. There is nothing queued *inside* the process group, and no back
pressure... it simply isn't moving files fast enough. As a workaround,
multiple input ports were added... but that's ugly.
The first issue may be the limit of 100 files transferred on any call to
onTrigger()[1]. 100 seems low, ideally it would be nice to fill to the back
pressure point (or some reasonable limit if set to "0"). This may also be
compounded by some scheduling issues we've seen, but not reliably enough to
actually diagnose. But the combination of not transferring enough files on
each run and not running as often as it "should" is definitely causing issues.
The scheduling issue may be complex, but transferring more files should be
pretty straightforward...
[1]
[https://github.com/apache/nifi/blob/d838f61291d2582592754a37314911b701c6891b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/connectable/LocalPort.java#L76]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
